2024-03-06T07:10:09.419 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-06T07:10:09.419 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} --archive /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060 --name leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi --owner scheduled_leonidus@teuthology --verbose -- /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/orig.config.yaml 2024-03-06T07:10:09.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:10:09.556 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060 branch: wip-lusov-qdb description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} email: null first_in_suite: false job_id: '7583060' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-qdb ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-qdb sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5072 sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd sleep_before_teardown: 0 subset: 246/10000 suite: fs suite_branch: wip-lusov-qdb suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCtvxMT4pfpfCRFYMBCFiNMlviQr8bJ6S6yeXVqeLhb68KzR5NS0ZCgwYjKleIYo7G/6Gl9YgE2jNprIz1lZFac= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJCTiRBUCAsvCfWLVyHQLrRgxHEjeAHsL1fnPo17LH6ZJ0FHIDmxIoBRs5NN378dcLl9HGNxpNaBHagISm2z7PM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-06_07:04:29 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-06T07:10:09.556 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa; will attempt to use it 2024-03-06T07:10:09.557 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks 2024-03-06T07:10:09.557 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-06T07:10:09.558 INFO:teuthology.task.internal:Checking packages... 2024-03-06T07:10:09.583 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd' 2024-03-06T07:10:09.583 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-06T07:10:09.583 INFO:teuthology.packaging:ref: None 2024-03-06T07:10:09.583 INFO:teuthology.packaging:tag: None 2024-03-06T07:10:09.583 INFO:teuthology.packaging:branch: wip-lusov-qdb 2024-03-06T07:10:09.583 INFO:teuthology.packaging:sha1: 1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:10:09.584 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-qdb 2024-03-06T07:10:09.813 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1739.g1bc250c7 2024-03-06T07:10:09.814 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-06T07:10:09.819 INFO:teuthology.task.internal:no buildpackages task found 2024-03-06T07:10:09.820 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-06T07:10:09.845 INFO:teuthology.task.internal:Saving configuration 2024-03-06T07:10:09.861 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-06T07:10:09.867 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:10:09.891 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:40.609193', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCtvxMT4pfpfCRFYMBCFiNMlviQr8bJ6S6yeXVqeLhb68KzR5NS0ZCgwYjKleIYo7G/6Gl9YgE2jNprIz1lZFac='} 2024-03-06T07:10:09.912 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:40.610607', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJCTiRBUCAsvCfWLVyHQLrRgxHEjeAHsL1fnPo17LH6ZJ0FHIDmxIoBRs5NN378dcLl9HGNxpNaBHagISm2z7PM='} 2024-03-06T07:10:09.912 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-06T07:10:09.918 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-06T07:10:09.918 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-06T07:10:09.918 INFO:teuthology.run_tasks:Running task console_log... 2024-03-06T07:10:10.000 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6842c6f70>, signals=[15]) 2024-03-06T07:10:10.000 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-06T07:10:10.005 INFO:teuthology.task.internal:Opening connections... 2024-03-06T07:10:10.005 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:10:10.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:10.103 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:10:10.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:10:10.191 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-06T07:10:10.197 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-03-06T07:10:10.215 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-03-06T07:10:10.215 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:10.271 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:10.272 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:10.273 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-03-06T07:10:10.294 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-03-06T07:10:10.312 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-03-06T07:10:10.312 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-03-06T07:10:10.369 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-03-06T07:10:10.369 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-03-06T07:10:10.369 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-03-06T07:10:10.369 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-03-06T07:10:10.369 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-06T07:10:10.370 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-06T07:10:10.370 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-03-06T07:10:10.395 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-06T07:10:10.403 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-06T07:10:10.411 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-06T07:10:10.411 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:10.414 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-03-06T07:10:10.430 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-06T07:10:10.437 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-06T07:10:10.437 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:10.473 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-03-06T07:10:10.508 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-06T07:10:10.562 INFO:teuthology.run_tasks:Running task kernel... 2024-03-06T07:10:10.577 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-06T07:10:10.577 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-06T07:10:10.577 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:10.578 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-06T07:10:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:10.593 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-06T07:10:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:10:10.595 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-06T07:10:10.649 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:10.649 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2024-03-06T07:10:10.650 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-03-06T07:10:10.652 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-03-06T07:10:10.652 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-361.el9.x86_64 2024-03-06T07:10:10.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-03-06T07:10:12.827 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-06T07:10:13.657 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-06T07:10:14.210 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-06T07:10:15.066 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-06T07:10:15.259 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-06T07:10:16.134 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-06T07:10:18.085 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-06T07:10:19.262 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 19 MB 00:01 2024-03-06T07:10:21.961 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-06T07:10:23.380 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-06T07:10:23.579 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-06T07:10:24.832 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2024-03-06T07:10:25.842 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-03-06T07:10:26.795 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-06T07:10:30.755 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-06T07:10:31.695 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-06T07:10:32.937 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:32.937 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:33.043 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-06T07:10:33.055 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-03-06T07:10:33.056 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:10:33.057 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 100 M 2024-03-06T07:10:33.057 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 123 M 2024-03-06T07:10:33.057 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-06T07:10:33.448 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-06T07:10:33.781 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 34 MB/s | 21 MB 00:00 2024-03-06T07:10:33.953 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-06T07:10:33.953 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-06T07:10:34.060 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:10:34.075 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-06T07:10:34.076 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-06T07:10:34.076 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-03-06T07:10:34.076 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:10:34.076 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 100 M 2024-03-06T07:10:34.076 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-03-06T07:10:34.077 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-06T07:10:34.165 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 47 MB/s | 33 MB 00:00 2024-03-06T07:10:34.457 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 18 MB/s | 6.0 MB 00:00 2024-03-06T07:10:34.607 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 27 MB/s | 39 MB 00:01 2024-03-06T07:10:34.607 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:34.607 INFO:teuthology.orchestra.run.smithi146.stdout:Total 64 MB/s | 100 MB 00:01 2024-03-06T07:10:35.091 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-03-06T07:10:35.493 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-06T07:10:35.891 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-06T07:10:35.891 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-06T07:10:35.974 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-03-06T07:10:36.708 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 15 MB/s | 33 MB 00:02 2024-03-06T07:10:36.708 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:10:36.708 INFO:teuthology.orchestra.run.smithi072.stdout:Total 38 MB/s | 100 MB 00:02 2024-03-06T07:10:37.267 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-06T07:10:37.268 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-06T07:10:37.595 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-06T07:10:37.986 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-06T07:10:37.987 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-06T07:10:39.333 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-06T07:10:39.333 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-06T07:10:39.970 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-06T07:10:40.627 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:10:40.757 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:42.045 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-06T07:10:42.778 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:10:42.964 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:43.568 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:43.758 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:45.779 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:10:46.050 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:48.514 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:48.643 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:10:50.808 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:10:50.966 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:10:53.410 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:10:55.732 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:22.522 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:22.523 INFO:teuthology.orchestra.run.smithi146.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:22.523 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:11:22.607 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:23.089 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:23.089 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:23.089 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:23.089 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:11:23.552 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-06T07:11:24.097 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:24.235 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:24.236 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:24.236 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:24.237 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:24.237 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:24.238 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-06T07:11:24.238 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:24.238 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:24.359 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:24.359 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:24.359 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-03-06T07:11:24.393 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-03-06T07:11:24.393 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:24.394 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:11:24.395 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:24.396 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:24.396 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:24.396 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:25.018 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-03-06T07:11:26.503 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:26.535 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-03-06T07:11:26.537 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-06T07:11:26.552 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.LH6itGDTWl 2024-03-06T07:11:26.552 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LH6itGDTWl 2024-03-06T07:11:26.627 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.LH6itGDTWl 2024-03-06T07:11:26.647 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:26.647 INFO:teuthology.orchestra.run.smithi072.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-06T07:11:26.647 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:11:26.731 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:26.771 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.LH6itGDTWl is 6KB 2024-03-06T07:11:26.819 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.LH6itGDTWl 2024-03-06T07:11:26.833 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:26.901 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:26.901 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:26.901 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:26.901 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:26.902 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:27.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-03-06T07:11:27.233 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:27.233 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-06T07:11:27.233 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-06T07:11:27.233 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-06T07:11:27.709 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-06T07:11:27.709 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:11:27.709 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-06T07:11:27.709 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:27.709 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:27.710 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-06T07:11:27.710 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-06T07:11:27.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:11:27.710 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-06T07:11:28.458 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:28.610 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:28.610 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:28.610 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-06T07:11:28.610 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-06T07:11:28.611 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-06T07:11:28.611 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-06T07:11:28.611 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-03-06T07:11:28.611 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-06T07:11:28.731 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-06T07:11:28.732 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-06T07:11:28.732 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-03-06T07:11:28.765 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-03-06T07:11:28.765 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-03-06T07:11:28.765 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-03-06T07:11:28.765 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-03-06T07:11:28.766 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:11:28.767 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-06T07:11:28.767 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-06T07:11:28.767 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-06T07:11:28.767 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-03-06T07:11:28.767 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:11:28.767 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-03-06T07:11:28.768 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-06T07:11:28.768 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-427.el9.x86_64 2024-03-06T07:11:28.768 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-06T07:11:29.407 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-03-06T07:11:30.976 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-06T07:11:31.002 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-03-06T07:11:31.004 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-06T07:11:31.019 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.BnAt4hfTmK 2024-03-06T07:11:31.019 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BnAt4hfTmK 2024-03-06T07:11:31.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.BnAt4hfTmK 2024-03-06T07:11:31.233 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.BnAt4hfTmK is 6KB 2024-03-06T07:11:31.283 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.BnAt4hfTmK 2024-03-06T07:11:31.297 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-03-06T07:11:31.361 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-06T07:11:31.361 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-06T07:11:31.361 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-06T07:11:31.361 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-06T07:11:31.363 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-06T07:11:31.502 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-03-06T07:11:57.076 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-06T07:11:57.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:11:57.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:01.509 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-06T07:12:01.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-06T07:12:01.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:31.549 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-06T07:12:40.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:12:40.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:12:58.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-06T07:13:01.516 DEBUG:teuthology.orchestra.remote:timed out 2024-03-06T07:13:10.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-06T07:13:10.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:10.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:13:10.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:14.016 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-03-06T07:13:18.093 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-06T07:13:18.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-06T07:13:18.713 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:18.713 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-06T07:13:18.727 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:18.727 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:18.727 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:18.728 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:19.729 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-06T07:13:19.729 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:19.847 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2024-03-06T07:13:19.886 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:29.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:13:29.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:13:29.417 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-06T07:13:29.817 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:13:29.817 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-06T07:13:29.817 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-03-06T07:13:29.832 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-427.el9.x86_64 2024-03-06T07:13:29.833 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-06T07:13:29.833 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-06T07:13:29.833 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-06T07:13:30.834 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-06T07:13:30.835 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-06T07:13:30.914 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-03-06T07:13:30.953 DEBUG:teuthology.parallel:result is None 2024-03-06T07:13:30.953 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-06T07:13:30.961 INFO:teuthology.task.internal:Creating test directory... 2024-03-06T07:13:30.961 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:13:30.965 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-06T07:13:30.983 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-06T07:13:30.995 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-06T07:13:31.031 INFO:teuthology.task.internal:Creating archive directory... 2024-03-06T07:13:31.031 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:13:31.034 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-06T07:13:31.079 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-06T07:13:31.086 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-06T07:13:31.086 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:13:31.102 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-06T07:13:31.133 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:31.147 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:31.154 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:31.167 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-06T07:13:31.169 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-06T07:13:31.177 INFO:teuthology.task.internal:Configuring sudo... 2024-03-06T07:13:31.177 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:13:31.191 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-06T07:13:31.238 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-06T07:13:31.248 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-06T07:13:31.249 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:13:31.261 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-06T07:13:31.295 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:31.362 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:31.419 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:31.502 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:31.544 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:13:31.544 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:13:31.612 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:31.657 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-06T07:13:31.714 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:31.800 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-06T07:13:31.844 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:13:31.844 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-06T07:13:31.914 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-03-06T07:13:31.916 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-03-06T07:13:31.955 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:13:32.000 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:13:32.494 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-06T07:13:32.503 INFO:teuthology.task.internal:Starting timer... 2024-03-06T07:13:32.504 INFO:teuthology.run_tasks:Running task pcp... 2024-03-06T07:13:32.524 INFO:teuthology.run_tasks:Running task selinux... 2024-03-06T07:13:32.556 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-03-06T07:13:32.657 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-03-06T07:13:32.659 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-03-06T07:13:32.716 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-03-06T07:13:32.718 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-06T07:13:32.718 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-03-06T07:13:32.745 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-03-06T07:13:32.745 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-03-06T07:13:32.769 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-03-06T07:13:32.769 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-03-06T07:13:32.769 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:13:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:13:32.815 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:13:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:13:32.844 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-06T07:13:32.844 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:13:32.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-03-06T07:13:32.946 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-06T07:13:32.957 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-06T07:13:32.976 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-06T07:13:32.978 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-06T07:17:51.856 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-03-06T07:17:51.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-06T07:17:51.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:51.949 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-06T07:17:52.032 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-06T07:17:52.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:17:52.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-06T07:17:52.122 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-03-06T07:17:52.206 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-03-06T07:17:52.207 INFO:teuthology.run_tasks:Running task clock... 2024-03-06T07:17:52.218 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-06T07:17:52.218 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:17:52.219 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:17:52.221 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-06T07:17:52.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:17:52.250 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:17:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:17:52.277 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-06T07:17:52.293 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-06T07:17:52.296 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-03-06T07:17:52.310 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-03-06T07:17:52.323 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-03-06T07:17:52.326 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:17:52.337 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-03-06T07:17:52.342 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:17:52.353 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-06T07:17:52.370 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-06T07:17:52.401 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:17:52.405 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:17:52.405 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-06T07:17:52.405 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.406 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.406 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.406 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.422 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:17:52.426 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:17:52.426 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-06T07:17:52.426 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.426 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.426 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.426 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:17:52.427 INFO:teuthology.run_tasks:Running task install... 2024-03-06T07:17:52.437 DEBUG:teuthology.task.install:project ceph 2024-03-06T07:17:52.437 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}} 2024-03-06T07:17:52.437 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'} 2024-03-06T07:17:52.437 INFO:teuthology.task.install:Using flavor: default 2024-03-06T07:17:52.443 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-06T07:17:52.443 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-06T07:17:52.444 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:17:52.444 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:17:52.446 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'tag': None, 'wait_for_package': False} 2024-03-06T07:17:52.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:17:52.591 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:17:52.592 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:17:52.679 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/ 2024-03-06T07:17:52.680 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1739.g1bc250c7 2024-03-06T07:17:52.920 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:17:52.920 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:17:52.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:17:52.949 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:17:52.950 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:17:53.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:17:53.088 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-qdb/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-06T07:17:53.088 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:17:53.088 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-06T07:17:53.097 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:17:53.119 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-06T07:17:53.119 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-06T07:17:53.164 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-03-06T07:17:53.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-03-06T07:17:53.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-06T07:17:53.271 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-06T07:17:53.339 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-03-06T07:17:53.340 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-03-06T07:17:53.414 INFO:teuthology.orchestra.run.smithi146.stdout:74 files removed 2024-03-06T07:17:53.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:17:53.575 INFO:teuthology.orchestra.run.smithi072.stdout:74 files removed 2024-03-06T07:17:53.602 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-06T07:17:54.699 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 88 kB/s | 83 kB 00:00 2024-03-06T07:17:54.968 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 79 kB/s | 83 kB 00:01 2024-03-06T07:17:55.530 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-06T07:17:55.810 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-06T07:17:56.289 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-06T07:17:56.687 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-06T07:17:56.967 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-06T07:17:57.273 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-06T07:17:58.417 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-06T07:17:58.716 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-06T07:17:59.724 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-06T07:18:00.984 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-06T07:18:07.378 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 19 MB 00:05 2024-03-06T07:18:11.563 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.1 MB 00:00 2024-03-06T07:18:12.882 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 15 kB 00:00 2024-03-06T07:18:14.847 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-06T07:18:19.745 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-06T07:18:22.283 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:22.284 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:22.298 INFO:teuthology.orchestra.run.smithi072.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:18:22.299 INFO:teuthology.orchestra.run.smithi072.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:18:22.300 INFO:teuthology.orchestra.run.smithi072.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:18:22.301 INFO:teuthology.orchestra.run.smithi072.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:22.303 INFO:teuthology.orchestra.run.smithi072.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:22.306 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:18:22.307 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:18:22.388 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:18:22.413 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:18:22.414 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:18:22.415 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:18:22.416 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:18:22.417 INFO:teuthology.orchestra.run.smithi072.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:18:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:18:22.419 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:18:22.420 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:18:22.421 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:18:22.422 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:18:22.423 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:18:22.424 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:18:22.425 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-06T07:18:22.426 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-06T07:18:22.427 INFO:teuthology.orchestra.run.smithi072.stdout:Install 150 Packages 2024-03-06T07:18:22.427 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 26 Packages 2024-03-06T07:18:22.427 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:18:22.427 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 224 M 2024-03-06T07:18:22.427 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-06T07:18:23.490 INFO:teuthology.orchestra.run.smithi072.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-03-06T07:18:25.923 INFO:teuthology.orchestra.run.smithi072.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 365 kB/s | 888 kB 00:02 2024-03-06T07:18:26.325 INFO:teuthology.orchestra.run.smithi072.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 361 kB/s | 145 kB 00:00 2024-03-06T07:18:28.130 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 704 kB/s | 19 MB 00:27 2024-03-06T07:18:28.559 INFO:teuthology.orchestra.run.smithi072.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 994 kB/s | 5.3 MB 00:05 2024-03-06T07:18:30.393 INFO:teuthology.orchestra.run.smithi072.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 834 kB/s | 1.5 MB 00:01 2024-03-06T07:18:31.268 INFO:teuthology.orchestra.run.smithi072.stdout:(6/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 479 kB/s | 2.3 MB 00:04 2024-03-06T07:18:32.514 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.1 MB 00:00 2024-03-06T07:18:34.265 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-03-06T07:18:36.177 INFO:teuthology.orchestra.run.smithi072.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 830 kB/s | 4.7 MB 00:05 2024-03-06T07:18:38.124 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 21 MB 00:03 2024-03-06T07:18:42.974 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-06T07:18:45.161 INFO:teuthology.orchestra.run.smithi072.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 1.1 MB/s | 9.8 MB 00:08 2024-03-06T07:18:45.253 INFO:teuthology.orchestra.run.smithi072.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 275 kB/s | 25 kB 00:00 2024-03-06T07:18:45.473 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:45.474 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-06T07:18:45.485 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-06T07:18:45.486 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-06T07:18:45.487 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-06T07:18:45.488 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:45.490 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-06T07:18:45.492 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-06T07:18:45.493 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-06T07:18:45.575 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 6.6 k 2024-03-06T07:18:45.598 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.3 M 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 888 k 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 145 k 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 1.5 M 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 138 k 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 1.8 M 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 7.4 M 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 49 k 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 9.8 M 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 47 M 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 264 k 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 748 k 2024-03-06T07:18:45.599 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 220 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 27 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 18 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 31 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 747 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 127 k 2024-03-06T07:18:45.600 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:3.0.7-27.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 162 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 321 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 298 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 99 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 85 k 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:45.601 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 171 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager x86_64 1:1.46.0-1.el9 CentOS-BaseOS 2.3 M 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 CentOS-BaseOS 1.8 M 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 CentOS-BaseOS 41 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 CentOS-BaseOS 247 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:45.602 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.4 M 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 3.1 M 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: openssl x86_64 1:3.0.7-27.el9 CentOS-BaseOS 1.2 M 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs x86_64 1:3.0.7-27.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:45.603 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 20 M 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 31 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 2.3 M 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 248 k 2024-03-06T07:18:45.604 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 4.7 M 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 17 M 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1739.g1bc250c7.el9 ceph-noarch 15 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 25 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-06T07:18:45.605 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 163 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 501 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 5.1 M 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-06T07:18:45.606 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-06T07:18:45.607 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 45 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:19.0.0-1739.g1bc250c7.el9 ceph 131 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-06T07:18:45.608 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-06T07:18:45.609 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-06T07:18:45.610 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-06T07:18:45.611 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-06T07:18:45.612 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout:Install 150 Packages 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 26 Packages 2024-03-06T07:18:45.613 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:18:45.614 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 224 M 2024-03-06T07:18:45.614 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-03-06T07:18:46.292 INFO:teuthology.orchestra.run.smithi146.stdout:(1/176): ceph-19.0.0-1739.g1bc250c7.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-03-06T07:18:48.159 INFO:teuthology.orchestra.run.smithi146.stdout:(2/176): ceph-fuse-19.0.0-1739.g1bc250c7.el9.x8 476 kB/s | 888 kB 00:01 2024-03-06T07:18:48.475 INFO:teuthology.orchestra.run.smithi146.stdout:(3/176): ceph-immutable-object-cache-19.0.0-173 459 kB/s | 145 kB 00:00 2024-03-06T07:18:50.534 INFO:teuthology.orchestra.run.smithi146.stdout:(4/176): ceph-base-19.0.0-1739.g1bc250c7.el9.x8 1.2 MB/s | 5.3 MB 00:04 2024-03-06T07:18:52.134 INFO:teuthology.orchestra.run.smithi146.stdout:(5/176): ceph-mgr-19.0.0-1739.g1bc250c7.el9.x86 956 kB/s | 1.5 MB 00:01 2024-03-06T07:18:52.759 INFO:teuthology.orchestra.run.smithi146.stdout:(6/176): ceph-mds-19.0.0-1739.g1bc250c7.el9.x86 552 kB/s | 2.3 MB 00:04 2024-03-06T07:18:57.284 INFO:teuthology.orchestra.run.smithi146.stdout:(7/176): ceph-mon-19.0.0-1739.g1bc250c7.el9.x86 932 kB/s | 4.7 MB 00:05 2024-03-06T07:18:59.728 INFO:teuthology.orchestra.run.smithi072.stdout:(10/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 605 kB/s | 17 MB 00:28 2024-03-06T07:18:59.904 INFO:teuthology.orchestra.run.smithi072.stdout:(11/176): ceph-common-19.0.0-1739.g1bc250c7.el9 567 kB/s | 20 MB 00:36 2024-03-06T07:19:00.029 INFO:teuthology.orchestra.run.smithi072.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 216 kB/s | 27 kB 00:00 2024-03-06T07:19:00.157 INFO:teuthology.orchestra.run.smithi072.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 247 kB/s | 31 kB 00:00 2024-03-06T07:19:00.265 INFO:teuthology.orchestra.run.smithi072.stdout:(14/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 411 kB/s | 220 kB 00:00 2024-03-06T07:19:00.566 INFO:teuthology.orchestra.run.smithi072.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 544 kB/s | 163 kB 00:00 2024-03-06T07:19:00.841 INFO:teuthology.orchestra.run.smithi072.stdout:(16/176): librados-devel-19.0.0-1739.g1bc250c7. 460 kB/s | 127 kB 00:00 2024-03-06T07:19:01.100 INFO:teuthology.orchestra.run.smithi072.stdout:(17/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 792 kB/s | 747 kB 00:00 2024-03-06T07:19:01.592 INFO:teuthology.orchestra.run.smithi072.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 668 kB/s | 501 kB 00:00 2024-03-06T07:19:01.712 INFO:teuthology.orchestra.run.smithi072.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 386 kB/s | 45 kB 00:00 2024-03-06T07:19:01.943 INFO:teuthology.orchestra.run.smithi072.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 562 kB/s | 131 kB 00:00 2024-03-06T07:19:02.269 INFO:teuthology.orchestra.run.smithi072.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 497 kB/s | 162 kB 00:00 2024-03-06T07:19:02.769 INFO:teuthology.orchestra.run.smithi072.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 641 kB/s | 321 kB 00:00 2024-03-06T07:19:03.212 INFO:teuthology.orchestra.run.smithi072.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 674 kB/s | 298 kB 00:00 2024-03-06T07:19:03.437 INFO:teuthology.orchestra.run.smithi072.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 441 kB/s | 99 kB 00:00 2024-03-06T07:19:03.638 INFO:teuthology.orchestra.run.smithi072.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 425 kB/s | 85 kB 00:00 2024-03-06T07:19:05.680 INFO:teuthology.orchestra.run.smithi072.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 1.1 MB/s | 5.1 MB 00:04 2024-03-06T07:19:05.888 INFO:teuthology.orchestra.run.smithi072.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 819 kB/s | 171 kB 00:00 2024-03-06T07:19:06.007 INFO:teuthology.orchestra.run.smithi072.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 265 kB/s | 31 kB 00:00 2024-03-06T07:19:06.216 INFO:teuthology.orchestra.run.smithi072.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 662 kB/s | 138 kB 00:00 2024-03-06T07:19:08.016 INFO:teuthology.orchestra.run.smithi072.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 723 kB/s | 3.1 MB 00:04 2024-03-06T07:19:08.159 INFO:teuthology.orchestra.run.smithi072.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 939 kB/s | 1.8 MB 00:01 2024-03-06T07:19:08.476 INFO:teuthology.orchestra.run.smithi072.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 784 kB/s | 248 kB 00:00 2024-03-06T07:19:08.593 INFO:teuthology.orchestra.run.smithi072.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 422 kB/s | 49 kB 00:00 2024-03-06T07:19:08.710 INFO:teuthology.orchestra.run.smithi072.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 131 kB/s | 15 kB 00:00 2024-03-06T07:19:09.019 INFO:teuthology.orchestra.run.smithi072.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 855 kB/s | 264 kB 00:00 2024-03-06T07:19:09.786 INFO:teuthology.orchestra.run.smithi072.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 975 kB/s | 748 kB 00:00 2024-03-06T07:19:09.895 INFO:teuthology.orchestra.run.smithi072.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 168 kB/s | 18 kB 00:00 2024-03-06T07:19:10.046 INFO:teuthology.orchestra.run.smithi072.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 662 kB 00:00 2024-03-06T07:19:10.104 INFO:teuthology.orchestra.run.smithi072.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-06T07:19:10.147 INFO:teuthology.orchestra.run.smithi072.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-06T07:19:10.189 INFO:teuthology.orchestra.run.smithi072.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 490 kB/s | 20 kB 00:00 2024-03-06T07:19:10.216 INFO:teuthology.orchestra.run.smithi072.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 11 MB/s | 307 kB 00:00 2024-03-06T07:19:10.250 INFO:teuthology.orchestra.run.smithi072.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 940 kB/s | 32 kB 00:00 2024-03-06T07:19:10.367 INFO:teuthology.orchestra.run.smithi072.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-06T07:19:10.401 INFO:teuthology.orchestra.run.smithi072.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 509 kB/s | 17 kB 00:00 2024-03-06T07:19:10.435 INFO:teuthology.orchestra.run.smithi072.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 300 kB/s | 10 kB 00:00 2024-03-06T07:19:10.477 INFO:teuthology.orchestra.run.smithi072.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-06T07:19:10.511 INFO:teuthology.orchestra.run.smithi072.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-06T07:19:10.561 INFO:teuthology.orchestra.run.smithi072.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-06T07:19:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-06T07:19:10.646 INFO:teuthology.orchestra.run.smithi072.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 727 kB/s | 24 kB 00:00 2024-03-06T07:19:10.680 INFO:teuthology.orchestra.run.smithi072.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-06T07:19:10.714 INFO:teuthology.orchestra.run.smithi072.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.2 MB/s | 282 kB 00:00 2024-03-06T07:19:10.764 INFO:teuthology.orchestra.run.smithi072.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2024-03-06T07:19:10.798 INFO:teuthology.orchestra.run.smithi072.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-06T07:19:10.999 INFO:teuthology.orchestra.run.smithi072.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-03-06T07:19:11.109 INFO:teuthology.orchestra.run.smithi072.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 27 MB/s | 3.0 MB 00:00 2024-03-06T07:19:11.185 INFO:teuthology.orchestra.run.smithi072.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-06T07:19:11.244 INFO:teuthology.orchestra.run.smithi072.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-06T07:19:11.303 INFO:teuthology.orchestra.run.smithi072.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-03-06T07:19:11.486 INFO:teuthology.orchestra.run.smithi072.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-06T07:19:11.570 INFO:teuthology.orchestra.run.smithi072.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-03-06T07:19:11.613 INFO:teuthology.orchestra.run.smithi072.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-06T07:19:11.708 INFO:teuthology.orchestra.run.smithi146.stdout:(8/176): ceph-radosgw-19.0.0-1739.g1bc250c7.el9 694 kB/s | 9.8 MB 00:14 2024-03-06T07:19:11.797 INFO:teuthology.orchestra.run.smithi072.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 5.8 MB 00:00 2024-03-06T07:19:11.808 INFO:teuthology.orchestra.run.smithi146.stdout:(9/176): ceph-selinux-19.0.0-1739.g1bc250c7.el9 253 kB/s | 25 kB 00:00 2024-03-06T07:19:11.839 INFO:teuthology.orchestra.run.smithi072.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 5.3 MB/s | 227 kB 00:00 2024-03-06T07:19:11.890 INFO:teuthology.orchestra.run.smithi072.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 486 kB/s | 24 kB 00:00 2024-03-06T07:19:11.923 INFO:teuthology.orchestra.run.smithi072.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 538 kB/s | 18 kB 00:00 2024-03-06T07:19:11.966 INFO:teuthology.orchestra.run.smithi072.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-06T07:19:11.999 INFO:teuthology.orchestra.run.smithi072.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 935 kB/s | 32 kB 00:00 2024-03-06T07:19:12.159 INFO:teuthology.orchestra.run.smithi072.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-06T07:19:12.194 INFO:teuthology.orchestra.run.smithi072.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.1 MB/s | 217 kB 00:00 2024-03-06T07:19:12.228 INFO:teuthology.orchestra.run.smithi072.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-06T07:19:12.262 INFO:teuthology.orchestra.run.smithi072.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:12.304 INFO:teuthology.orchestra.run.smithi072.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 976 kB/s | 41 kB 00:00 2024-03-06T07:19:12.363 INFO:teuthology.orchestra.run.smithi072.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-06T07:19:12.430 INFO:teuthology.orchestra.run.smithi072.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-06T07:19:12.489 INFO:teuthology.orchestra.run.smithi072.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-06T07:19:12.548 INFO:teuthology.orchestra.run.smithi072.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 804 kB/s | 47 kB 00:00 2024-03-06T07:19:12.599 INFO:teuthology.orchestra.run.smithi072.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 849 kB/s | 43 kB 00:00 2024-03-06T07:19:13.266 INFO:teuthology.orchestra.run.smithi072.stdout:(80/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-03-06T07:19:13.333 INFO:teuthology.orchestra.run.smithi072.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 578 kB/s | 39 kB 00:00 2024-03-06T07:19:13.400 INFO:teuthology.orchestra.run.smithi072.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 613 kB/s | 41 kB 00:00 2024-03-06T07:19:13.468 INFO:teuthology.orchestra.run.smithi072.stdout:(83/176): qatlib-23.11.0-1.el9.x86_64.rpm 3.1 MB/s | 215 kB 00:00 2024-03-06T07:19:13.526 INFO:teuthology.orchestra.run.smithi072.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 577 kB/s | 34 kB 00:00 2024-03-06T07:19:13.578 INFO:teuthology.orchestra.run.smithi072.stdout:(85/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 837 kB/s | 43 kB 00:00 2024-03-06T07:19:13.637 INFO:teuthology.orchestra.run.smithi072.stdout:(86/176): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-06T07:19:13.680 INFO:teuthology.orchestra.run.smithi072.stdout:(87/176): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-06T07:19:13.714 INFO:teuthology.orchestra.run.smithi072.stdout:(88/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.5 MB/s | 87 kB 00:00 2024-03-06T07:19:13.756 INFO:teuthology.orchestra.run.smithi072.stdout:(89/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-06T07:19:13.798 INFO:teuthology.orchestra.run.smithi072.stdout:(90/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-06T07:19:13.832 INFO:teuthology.orchestra.run.smithi072.stdout:(91/176): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-06T07:19:13.874 INFO:teuthology.orchestra.run.smithi072.stdout:(92/176): jq-1.6-16.el9.x86_64.rpm 4.3 MB/s | 186 kB 00:00 2024-03-06T07:19:13.933 INFO:teuthology.orchestra.run.smithi072.stdout:(93/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:13.976 INFO:teuthology.orchestra.run.smithi072.stdout:(94/176): libquadmath-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 194 kB 00:00 2024-03-06T07:19:14.018 INFO:teuthology.orchestra.run.smithi072.stdout:(95/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-06T07:19:14.060 INFO:teuthology.orchestra.run.smithi072.stdout:(96/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-06T07:19:14.127 INFO:teuthology.orchestra.run.smithi072.stdout:(97/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-06T07:19:14.161 INFO:teuthology.orchestra.run.smithi072.stdout:(98/176): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-06T07:19:14.203 INFO:teuthology.orchestra.run.smithi072.stdout:(99/176): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-06T07:19:14.247 INFO:teuthology.orchestra.run.smithi072.stdout:(100/176): python3-requests-2.25.1-8.el9.noarch 2.5 MB/s | 113 kB 00:00 2024-03-06T07:19:14.281 INFO:teuthology.orchestra.run.smithi072.stdout:(101/176): python3-urllib3-1.26.5-5.el9.noarch. 5.5 MB/s | 187 kB 00:00 2024-03-06T07:19:14.865 INFO:teuthology.orchestra.run.smithi072.stdout:(102/176): python3-devel-3.9.17-2.el9.x86_64.rp 420 kB/s | 245 kB 00:00 2024-03-06T07:19:14.982 INFO:teuthology.orchestra.run.smithi072.stdout:(103/176): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-06T07:19:15.316 INFO:teuthology.orchestra.run.smithi072.stdout:(104/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-06T07:19:15.633 INFO:teuthology.orchestra.run.smithi072.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-06T07:19:15.717 INFO:teuthology.orchestra.run.smithi072.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 3.6 MB/s | 309 kB 00:00 2024-03-06T07:19:15.776 INFO:teuthology.orchestra.run.smithi072.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 331 kB/s | 19 kB 00:00 2024-03-06T07:19:16.035 INFO:teuthology.orchestra.run.smithi072.stdout:(108/176): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-06T07:19:16.094 INFO:teuthology.orchestra.run.smithi072.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-03-06T07:19:16.153 INFO:teuthology.orchestra.run.smithi072.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 830 kB/s | 49 kB 00:00 2024-03-06T07:19:16.174 INFO:teuthology.orchestra.run.smithi146.stdout:(10/176): ceph-common-19.0.0-1739.g1bc250c7.el9 692 kB/s | 20 MB 00:30 2024-03-06T07:19:16.212 INFO:teuthology.orchestra.run.smithi072.stdout:(111/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-06T07:19:16.371 INFO:teuthology.orchestra.run.smithi072.stdout:(112/176): luarocks-3.9.2-1.el9.noarch.rpm 952 kB/s | 151 kB 00:00 2024-03-06T07:19:16.425 INFO:teuthology.orchestra.run.smithi146.stdout:(11/176): cephfs-mirror-19.0.0-1739.g1bc250c7.e 882 kB/s | 220 kB 00:00 2024-03-06T07:19:16.525 INFO:teuthology.orchestra.run.smithi146.stdout:(12/176): cephfs-shell-19.0.0-1739.g1bc250c7.el 271 kB/s | 27 kB 00:00 2024-03-06T07:19:16.588 INFO:teuthology.orchestra.run.smithi072.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-06T07:19:16.617 INFO:teuthology.orchestra.run.smithi146.stdout:(13/176): libcephfs-devel-19.0.0-1739.g1bc250c7 342 kB/s | 31 kB 00:00 2024-03-06T07:19:16.682 INFO:teuthology.orchestra.run.smithi072.stdout:(114/176): python3-asyncssh-2.13.2-4.el9.noarch 5.7 MB/s | 548 kB 00:00 2024-03-06T07:19:16.757 INFO:teuthology.orchestra.run.smithi072.stdout:(115/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 576 kB/s | 43 kB 00:00 2024-03-06T07:19:16.833 INFO:teuthology.orchestra.run.smithi072.stdout:(116/176): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-06T07:19:16.892 INFO:teuthology.orchestra.run.smithi072.stdout:(117/176): python3-certifi-2021.10.8-2.el9.noar 255 kB/s | 15 kB 00:00 2024-03-06T07:19:16.959 INFO:teuthology.orchestra.run.smithi072.stdout:(118/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-06T07:19:17.026 INFO:teuthology.orchestra.run.smithi072.stdout:(119/176): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-03-06T07:19:17.102 INFO:teuthology.orchestra.run.smithi072.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-06T07:19:17.169 INFO:teuthology.orchestra.run.smithi072.stdout:(121/176): python3-colorama-0.4.4-12.el9.noarch 527 kB/s | 35 kB 00:00 2024-03-06T07:19:17.236 INFO:teuthology.orchestra.run.smithi072.stdout:(122/176): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-06T07:19:17.292 INFO:teuthology.orchestra.run.smithi146.stdout:(14/176): libcephfs2-19.0.0-1739.g1bc250c7.el9. 1.1 MB/s | 747 kB 00:00 2024-03-06T07:19:17.362 INFO:teuthology.orchestra.run.smithi072.stdout:(123/176): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-06T07:19:17.421 INFO:teuthology.orchestra.run.smithi072.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-06T07:19:17.451 INFO:teuthology.orchestra.run.smithi146.stdout:(15/176): libcephsqlite-19.0.0-1739.g1bc250c7.e 1.0 MB/s | 163 kB 00:00 2024-03-06T07:19:17.479 INFO:teuthology.orchestra.run.smithi072.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-03-06T07:19:17.539 INFO:teuthology.orchestra.run.smithi072.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-06T07:19:17.614 INFO:teuthology.orchestra.run.smithi072.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-03-06T07:19:17.675 INFO:teuthology.orchestra.run.smithi072.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 323 kB/s | 19 kB 00:00 2024-03-06T07:19:17.684 INFO:teuthology.orchestra.run.smithi146.stdout:(16/176): ceph-osd-19.0.0-1739.g1bc250c7.el9.x8 691 kB/s | 17 MB 00:24 2024-03-06T07:19:17.710 INFO:teuthology.orchestra.run.smithi146.stdout:(17/176): librados-devel-19.0.0-1739.g1bc250c7. 490 kB/s | 127 kB 00:00 2024-03-06T07:19:17.742 INFO:teuthology.orchestra.run.smithi072.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 293 kB/s | 20 kB 00:00 2024-03-06T07:19:17.826 INFO:teuthology.orchestra.run.smithi072.stdout:(130/176): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-06T07:19:17.885 INFO:teuthology.orchestra.run.smithi072.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 790 kB/s | 46 kB 00:00 2024-03-06T07:19:17.944 INFO:teuthology.orchestra.run.smithi072.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-06T07:19:18.003 INFO:teuthology.orchestra.run.smithi072.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 985 kB/s | 58 kB 00:00 2024-03-06T07:19:18.070 INFO:teuthology.orchestra.run.smithi072.stdout:(134/176): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-06T07:19:18.137 INFO:teuthology.orchestra.run.smithi072.stdout:(135/176): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-03-06T07:19:18.204 INFO:teuthology.orchestra.run.smithi072.stdout:(136/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-06T07:19:18.271 INFO:teuthology.orchestra.run.smithi072.stdout:(137/176): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-03-06T07:19:18.330 INFO:teuthology.orchestra.run.smithi072.stdout:(138/176): python3-repoze-lru-0.7-16.el9.noarch 525 kB/s | 31 kB 00:00 2024-03-06T07:19:18.376 INFO:teuthology.orchestra.run.smithi146.stdout:(18/176): libradosstriper1-19.0.0-1739.g1bc250c 725 kB/s | 501 kB 00:00 2024-03-06T07:19:18.398 INFO:teuthology.orchestra.run.smithi072.stdout:(139/176): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-06T07:19:18.468 INFO:teuthology.orchestra.run.smithi146.stdout:(19/176): python3-ceph-argparse-19.0.0-1739.g1b 495 kB/s | 45 kB 00:00 2024-03-06T07:19:18.531 INFO:teuthology.orchestra.run.smithi072.stdout:(140/176): ceph-mgr-diskprediction-local-19.0.0 722 kB/s | 7.4 MB 00:10 2024-03-06T07:19:18.557 INFO:teuthology.orchestra.run.smithi072.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2024-03-06T07:19:18.616 INFO:teuthology.orchestra.run.smithi072.stdout:(142/176): python3-typing-extensions-4.4.0-2.el 873 kB/s | 51 kB 00:00 2024-03-06T07:19:18.668 INFO:teuthology.orchestra.run.smithi072.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 263 kB/s | 36 kB 00:00 2024-03-06T07:19:18.702 INFO:teuthology.orchestra.run.smithi072.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.6 MB/s | 230 kB 00:00 2024-03-06T07:19:18.710 INFO:teuthology.orchestra.run.smithi146.stdout:(20/176): python3-ceph-common-19.0.0-1739.g1bc2 543 kB/s | 131 kB 00:00 2024-03-06T07:19:18.778 INFO:teuthology.orchestra.run.smithi072.stdout:(145/176): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-06T07:19:18.803 INFO:teuthology.orchestra.run.smithi072.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 665 kB/s | 90 kB 00:00 2024-03-06T07:19:18.837 INFO:teuthology.orchestra.run.smithi072.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 339 kB/s | 20 kB 00:00 2024-03-06T07:19:18.938 INFO:teuthology.orchestra.run.smithi072.stdout:(148/176): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-06T07:19:18.944 INFO:teuthology.orchestra.run.smithi146.stdout:(21/176): python3-cephfs-19.0.0-1739.g1bc250c7. 693 kB/s | 162 kB 00:00 2024-03-06T07:19:18.971 INFO:teuthology.orchestra.run.smithi072.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-06T07:19:19.352 INFO:teuthology.orchestra.run.smithi146.stdout:(22/176): python3-rados-19.0.0-1739.g1bc250c7.e 786 kB/s | 321 kB 00:00 2024-03-06T07:19:19.769 INFO:teuthology.orchestra.run.smithi146.stdout:(23/176): python3-rbd-19.0.0-1739.g1bc250c7.el9 715 kB/s | 298 kB 00:00 2024-03-06T07:19:19.945 INFO:teuthology.orchestra.run.smithi146.stdout:(24/176): python3-rgw-19.0.0-1739.g1bc250c7.el9 567 kB/s | 99 kB 00:00 2024-03-06T07:19:20.387 INFO:teuthology.orchestra.run.smithi146.stdout:(25/176): rbd-fuse-19.0.0-1739.g1bc250c7.el9.x8 193 kB/s | 85 kB 00:00 2024-03-06T07:19:23.036 INFO:teuthology.orchestra.run.smithi146.stdout:(26/176): librgw2-19.0.0-1739.g1bc250c7.el9.x86 984 kB/s | 5.1 MB 00:05 2024-03-06T07:19:23.295 INFO:teuthology.orchestra.run.smithi146.stdout:(27/176): rbd-nbd-19.0.0-1739.g1bc250c7.el9.x86 661 kB/s | 171 kB 00:00 2024-03-06T07:19:23.379 INFO:teuthology.orchestra.run.smithi146.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1739.g 378 kB/s | 31 kB 00:00 2024-03-06T07:19:23.562 INFO:teuthology.orchestra.run.smithi146.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1739.g1bc250c 753 kB/s | 138 kB 00:00 2024-03-06T07:19:23.838 INFO:teuthology.orchestra.run.smithi072.stdout:(150/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 659 kB/s | 3.1 MB 00:04 2024-03-06T07:19:23.880 INFO:teuthology.orchestra.run.smithi072.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2024-03-06T07:19:23.989 INFO:teuthology.orchestra.run.smithi072.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:19:24.090 INFO:teuthology.orchestra.run.smithi072.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 18 MB/s | 1.8 MB 00:00 2024-03-06T07:19:24.124 INFO:teuthology.orchestra.run.smithi072.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:24.166 INFO:teuthology.orchestra.run.smithi072.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:24.200 INFO:teuthology.orchestra.run.smithi072.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:24.234 INFO:teuthology.orchestra.run.smithi072.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:24.277 INFO:teuthology.orchestra.run.smithi072.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.3 MB/s | 191 kB 00:00 2024-03-06T07:19:24.320 INFO:teuthology.orchestra.run.smithi072.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.9 MB/s | 254 kB 00:00 2024-03-06T07:19:24.378 INFO:teuthology.orchestra.run.smithi072.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:24.429 INFO:teuthology.orchestra.run.smithi146.stdout:(30/176): rbd-mirror-19.0.0-1739.g1bc250c7.el9. 783 kB/s | 3.1 MB 00:04 2024-03-06T07:19:24.446 INFO:teuthology.orchestra.run.smithi072.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 6.3 MB/s | 433 kB 00:00 2024-03-06T07:19:24.480 INFO:teuthology.orchestra.run.smithi072.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 656 kB/s | 22 kB 00:00 2024-03-06T07:19:24.522 INFO:teuthology.orchestra.run.smithi072.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-06T07:19:24.564 INFO:teuthology.orchestra.run.smithi072.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 154 kB 00:00 2024-03-06T07:19:24.606 INFO:teuthology.orchestra.run.smithi072.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.1 MB/s | 135 kB 00:00 2024-03-06T07:19:24.640 INFO:teuthology.orchestra.run.smithi072.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.8 MB/s | 63 kB 00:00 2024-03-06T07:19:24.674 INFO:teuthology.orchestra.run.smithi072.stdout:(167/176): libuuid-2.37.4-18.el9.x86_64.rpm 822 kB/s | 28 kB 00:00 2024-03-06T07:19:24.741 INFO:teuthology.orchestra.run.smithi072.stdout:(168/176): openssl-3.0.7-27.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-03-06T07:19:24.833 INFO:teuthology.orchestra.run.smithi072.stdout:(169/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 23 MB/s | 2.2 MB 00:00 2024-03-06T07:19:24.926 INFO:teuthology.orchestra.run.smithi072.stdout:(170/176): util-linux-2.37.4-18.el9.x86_64.rpm 24 MB/s | 2.2 MB 00:00 2024-03-06T07:19:24.976 INFO:teuthology.orchestra.run.smithi072.stdout:(171/176): util-linux-core-2.37.4-18.el9.x86_64 8.5 MB/s | 438 kB 00:00 2024-03-06T07:19:25.012 INFO:teuthology.orchestra.run.smithi072.stdout:(172/176): util-linux-user-2.37.4-18.el9.x86_64 858 kB/s | 30 kB 00:00 2024-03-06T07:19:25.062 INFO:teuthology.orchestra.run.smithi072.stdout:(173/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:19:25.129 INFO:teuthology.orchestra.run.smithi072.stdout:(174/176): librados2-19.0.0-1739.g1bc250c7.el9. 561 kB/s | 3.4 MB 00:06 2024-03-06T07:19:25.172 INFO:teuthology.orchestra.run.smithi072.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.7 MB/s | 1.1 MB 00:00 2024-03-06T07:19:25.488 INFO:teuthology.orchestra.run.smithi146.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1739.g1bc25 947 kB/s | 1.8 MB 00:01 2024-03-06T07:19:25.796 INFO:teuthology.orchestra.run.smithi146.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1739.g1b 806 kB/s | 248 kB 00:00 2024-03-06T07:19:25.880 INFO:teuthology.orchestra.run.smithi146.stdout:(33/176): ceph-mgr-rook-19.0.0-1739.g1bc250c7.e 592 kB/s | 49 kB 00:00 2024-03-06T07:19:25.963 INFO:teuthology.orchestra.run.smithi146.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1739.g1 184 kB/s | 15 kB 00:00 2024-03-06T07:19:26.305 INFO:teuthology.orchestra.run.smithi146.stdout:(35/176): ceph-volume-19.0.0-1739.g1bc250c7.el9 772 kB/s | 264 kB 00:00 2024-03-06T07:19:27.014 INFO:teuthology.orchestra.run.smithi146.stdout:(36/176): cephadm-19.0.0-1739.g1bc250c7.el9.noa 1.0 MB/s | 748 kB 00:00 2024-03-06T07:19:27.097 INFO:teuthology.orchestra.run.smithi146.stdout:(37/176): cephfs-top-19.0.0-1739.g1bc250c7.el9. 219 kB/s | 18 kB 00:00 2024-03-06T07:19:27.239 INFO:teuthology.orchestra.run.smithi146.stdout:(38/176): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-06T07:19:27.298 INFO:teuthology.orchestra.run.smithi146.stdout:(39/176): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-06T07:19:27.365 INFO:teuthology.orchestra.run.smithi146.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-06T07:19:27.365 INFO:teuthology.orchestra.run.smithi146.stdout:(41/176): elfutils-libelf-devel-0.190-2.el9.x86 823 kB/s | 20 kB 00:00 2024-03-06T07:19:27.399 INFO:teuthology.orchestra.run.smithi146.stdout:(42/176): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-06T07:19:27.424 INFO:teuthology.orchestra.run.smithi146.stdout:(43/176): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-06T07:19:27.519 INFO:teuthology.orchestra.run.smithi146.stdout:(44/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-06T07:19:27.545 INFO:teuthology.orchestra.run.smithi146.stdout:(45/176): flexiblas-openblas-openmp-3.0.4-8.el9 599 kB/s | 17 kB 00:00 2024-03-06T07:19:27.600 INFO:teuthology.orchestra.run.smithi146.stdout:(46/176): libaio-devel-0.3.111-13.el9.x86_64.rp 184 kB/s | 10 kB 00:00 2024-03-06T07:19:27.633 INFO:teuthology.orchestra.run.smithi146.stdout:(47/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-06T07:19:27.659 INFO:teuthology.orchestra.run.smithi146.stdout:(48/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-06T07:19:27.734 INFO:teuthology.orchestra.run.smithi146.stdout:(49/176): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-06T07:19:27.776 INFO:teuthology.orchestra.run.smithi146.stdout:(50/176): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-06T07:19:27.810 INFO:teuthology.orchestra.run.smithi146.stdout:(51/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 732 kB/s | 24 kB 00:00 2024-03-06T07:19:27.835 INFO:teuthology.orchestra.run.smithi146.stdout:(52/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-06T07:19:27.868 INFO:teuthology.orchestra.run.smithi146.stdout:(53/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-06T07:19:27.911 INFO:teuthology.orchestra.run.smithi146.stdout:(54/176): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-06T07:19:27.936 INFO:teuthology.orchestra.run.smithi146.stdout:(55/176): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-06T07:19:28.103 INFO:teuthology.orchestra.run.smithi146.stdout:(56/176): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-06T07:19:28.211 INFO:teuthology.orchestra.run.smithi146.stdout:(57/176): openssl-devel-3.0.7-27.el9.x86_64.rpm 28 MB/s | 3.0 MB 00:00 2024-03-06T07:19:28.278 INFO:teuthology.orchestra.run.smithi146.stdout:(58/176): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-06T07:19:28.354 INFO:teuthology.orchestra.run.smithi146.stdout:(59/176): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-06T07:19:28.396 INFO:teuthology.orchestra.run.smithi146.stdout:(60/176): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-06T07:19:28.571 INFO:teuthology.orchestra.run.smithi146.stdout:(61/176): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-06T07:19:28.638 INFO:teuthology.orchestra.run.smithi146.stdout:(62/176): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-06T07:19:28.680 INFO:teuthology.orchestra.run.smithi146.stdout:(63/176): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-06T07:19:28.872 INFO:teuthology.orchestra.run.smithi146.stdout:(64/176): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 5.8 MB 00:00 2024-03-06T07:19:28.914 INFO:teuthology.orchestra.run.smithi146.stdout:(65/176): python3-jinja2-2.11.3-5.el9.noarch.rp 5.3 MB/s | 227 kB 00:00 2024-03-06T07:19:28.947 INFO:teuthology.orchestra.run.smithi146.stdout:(66/176): python3-jsonpatch-1.21-16.el9.noarch. 732 kB/s | 24 kB 00:00 2024-03-06T07:19:28.981 INFO:teuthology.orchestra.run.smithi146.stdout:(67/176): python3-jsonpointer-2.0-4.el9.noarch. 542 kB/s | 18 kB 00:00 2024-03-06T07:19:29.015 INFO:teuthology.orchestra.run.smithi146.stdout:(68/176): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-06T07:19:29.040 INFO:teuthology.orchestra.run.smithi146.stdout:(69/176): python3-markupsafe-1.1.1-12.el9.x86_6 1.2 MB/s | 32 kB 00:00 2024-03-06T07:19:29.174 INFO:teuthology.orchestra.run.smithi146.stdout:(70/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-06T07:19:29.216 INFO:teuthology.orchestra.run.smithi146.stdout:(71/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-06T07:19:29.258 INFO:teuthology.orchestra.run.smithi146.stdout:(72/176): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-06T07:19:29.300 INFO:teuthology.orchestra.run.smithi146.stdout:(73/176): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 69 kB 00:00 2024-03-06T07:19:29.333 INFO:teuthology.orchestra.run.smithi146.stdout:(74/176): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:29.375 INFO:teuthology.orchestra.run.smithi146.stdout:(75/176): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-06T07:19:29.409 INFO:teuthology.orchestra.run.smithi146.stdout:(76/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-06T07:19:29.442 INFO:teuthology.orchestra.run.smithi146.stdout:(77/176): python3-pyasn1-modules-0.4.8-6.el9.no 6.1 MB/s | 211 kB 00:00 2024-03-06T07:19:29.467 INFO:teuthology.orchestra.run.smithi146.stdout:(78/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-06T07:19:29.493 INFO:teuthology.orchestra.run.smithi146.stdout:(79/176): python3-requests-oauthlib-1.3.0-12.el 1.7 MB/s | 43 kB 00:00 2024-03-06T07:19:29.818 INFO:teuthology.orchestra.run.smithi146.stdout:(80/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-06T07:19:29.874 INFO:teuthology.orchestra.run.smithi146.stdout:(81/176): python3-toml-0.10.2-6.el9.noarch.rpm 695 kB/s | 39 kB 00:00 2024-03-06T07:19:29.902 INFO:teuthology.orchestra.run.smithi146.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.4 MB/s | 41 kB 00:00 2024-03-06T07:19:29.944 INFO:teuthology.orchestra.run.smithi146.stdout:(83/176): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 215 kB 00:00 2024-03-06T07:19:29.969 INFO:teuthology.orchestra.run.smithi146.stdout:(84/176): qatlib-service-23.11.0-1.el9.x86_64.r 1.3 MB/s | 34 kB 00:00 2024-03-06T07:19:29.995 INFO:teuthology.orchestra.run.smithi146.stdout:(85/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-03-06T07:19:30.037 INFO:teuthology.orchestra.run.smithi146.stdout:(86/176): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-06T07:19:30.062 INFO:teuthology.orchestra.run.smithi146.stdout:(87/176): uuid-1.6.2-55.el9.x86_64.rpm 2.2 MB/s | 56 kB 00:00 2024-03-06T07:19:30.096 INFO:teuthology.orchestra.run.smithi146.stdout:(88/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.5 MB/s | 87 kB 00:00 2024-03-06T07:19:30.121 INFO:teuthology.orchestra.run.smithi146.stdout:(89/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-06T07:19:30.146 INFO:teuthology.orchestra.run.smithi146.stdout:(90/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-06T07:19:30.179 INFO:teuthology.orchestra.run.smithi146.stdout:(91/176): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-06T07:19:30.213 INFO:teuthology.orchestra.run.smithi146.stdout:(92/176): jq-1.6-16.el9.x86_64.rpm 5.4 MB/s | 186 kB 00:00 2024-03-06T07:19:30.272 INFO:teuthology.orchestra.run.smithi146.stdout:(93/176): libgfortran-11.4.1-3.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:30.314 INFO:teuthology.orchestra.run.smithi146.stdout:(94/176): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 194 kB 00:00 2024-03-06T07:19:30.356 INFO:teuthology.orchestra.run.smithi146.stdout:(95/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-06T07:19:30.391 INFO:teuthology.orchestra.run.smithi146.stdout:(96/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 7.1 MB/s | 241 kB 00:00 2024-03-06T07:19:30.457 INFO:teuthology.orchestra.run.smithi146.stdout:(97/176): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-06T07:19:30.490 INFO:teuthology.orchestra.run.smithi146.stdout:(98/176): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-06T07:19:30.524 INFO:teuthology.orchestra.run.smithi146.stdout:(99/176): python3-pycparser-2.20-6.el9.noarch.r 3.6 MB/s | 124 kB 00:00 2024-03-06T07:19:30.566 INFO:teuthology.orchestra.run.smithi146.stdout:(100/176): python3-requests-2.25.1-8.el9.noarch 2.7 MB/s | 113 kB 00:00 2024-03-06T07:19:30.600 INFO:teuthology.orchestra.run.smithi146.stdout:(101/176): python3-urllib3-1.26.5-5.el9.noarch. 5.5 MB/s | 187 kB 00:00 2024-03-06T07:19:30.742 INFO:teuthology.orchestra.run.smithi146.stdout:(102/176): ceph-mgr-diskprediction-local-19.0.0 1.2 MB/s | 7.4 MB 00:06 2024-03-06T07:19:30.850 INFO:teuthology.orchestra.run.smithi146.stdout:(103/176): python3-devel-3.9.17-2.el9.x86_64.rp 981 kB/s | 245 kB 00:00 2024-03-06T07:19:31.050 INFO:teuthology.orchestra.run.smithi146.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 72 kB/s | 22 kB 00:00 2024-03-06T07:19:31.384 INFO:teuthology.orchestra.run.smithi146.stdout:(105/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-03-06T07:19:31.468 INFO:teuthology.orchestra.run.smithi146.stdout:(106/176): gperftools-libs-2.9.1-2.el9.x86_64.r 3.6 MB/s | 309 kB 00:00 2024-03-06T07:19:31.518 INFO:teuthology.orchestra.run.smithi146.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-03-06T07:19:31.676 INFO:teuthology.orchestra.run.smithi146.stdout:(108/176): protobuf-compiler-3.14.0-13.el9.x86_ 1.0 MB/s | 863 kB 00:00 2024-03-06T07:19:31.735 INFO:teuthology.orchestra.run.smithi146.stdout:(109/176): libarrow-9.0.0-10.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2024-03-06T07:19:31.785 INFO:teuthology.orchestra.run.smithi146.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 977 kB/s | 49 kB 00:00 2024-03-06T07:19:31.819 INFO:teuthology.orchestra.run.smithi146.stdout:(111/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 183 kB/s | 26 kB 00:00 2024-03-06T07:19:31.844 INFO:teuthology.orchestra.run.smithi146.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-06T07:19:31.911 INFO:teuthology.orchestra.run.smithi146.stdout:(113/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-06T07:19:31.970 INFO:teuthology.orchestra.run.smithi146.stdout:(114/176): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-06T07:19:32.003 INFO:teuthology.orchestra.run.smithi146.stdout:(115/176): python3-asyncssh-2.13.2-4.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-06T07:19:32.037 INFO:teuthology.orchestra.run.smithi146.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 648 kB/s | 43 kB 00:00 2024-03-06T07:19:32.062 INFO:teuthology.orchestra.run.smithi146.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-06T07:19:32.104 INFO:teuthology.orchestra.run.smithi146.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-06T07:19:32.138 INFO:teuthology.orchestra.run.smithi146.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-06T07:19:32.213 INFO:teuthology.orchestra.run.smithi146.stdout:(120/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.0 MB/s | 303 kB 00:00 2024-03-06T07:19:32.238 INFO:teuthology.orchestra.run.smithi146.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-03-06T07:19:32.264 INFO:teuthology.orchestra.run.smithi146.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 703 kB/s | 35 kB 00:00 2024-03-06T07:19:32.305 INFO:teuthology.orchestra.run.smithi146.stdout:(123/176): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-06T07:19:32.364 INFO:teuthology.orchestra.run.smithi146.stdout:(124/176): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-03-06T07:19:32.389 INFO:teuthology.orchestra.run.smithi146.stdout:(125/176): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-06T07:19:32.423 INFO:teuthology.orchestra.run.smithi146.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-03-06T07:19:32.449 INFO:teuthology.orchestra.run.smithi146.stdout:(127/176): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-06T07:19:32.474 INFO:teuthology.orchestra.run.smithi146.stdout:(128/176): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-06T07:19:32.491 INFO:teuthology.orchestra.run.smithi146.stdout:(129/176): python3-jaraco-functools-3.5.0-2.el9 464 kB/s | 19 kB 00:00 2024-03-06T07:19:32.516 INFO:teuthology.orchestra.run.smithi146.stdout:(130/176): python3-jaraco-text-3.2.0-6.el9.noar 470 kB/s | 20 kB 00:00 2024-03-06T07:19:32.566 INFO:teuthology.orchestra.run.smithi146.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 929 kB/s | 46 kB 00:00 2024-03-06T07:19:32.625 INFO:teuthology.orchestra.run.smithi146.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-06T07:19:32.658 INFO:teuthology.orchestra.run.smithi146.stdout:(133/176): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-03-06T07:19:32.675 INFO:teuthology.orchestra.run.smithi146.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-06T07:19:32.734 INFO:teuthology.orchestra.run.smithi146.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-06T07:19:32.759 INFO:teuthology.orchestra.run.smithi146.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-03-06T07:19:32.793 INFO:teuthology.orchestra.run.smithi146.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-06T07:19:32.818 INFO:teuthology.orchestra.run.smithi146.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-06T07:19:32.843 INFO:teuthology.orchestra.run.smithi146.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-03-06T07:19:32.877 INFO:teuthology.orchestra.run.smithi146.stdout:(140/176): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-06T07:19:32.902 INFO:teuthology.orchestra.run.smithi146.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-06T07:19:32.936 INFO:teuthology.orchestra.run.smithi146.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 614 kB/s | 36 kB 00:00 2024-03-06T07:19:32.961 INFO:teuthology.orchestra.run.smithi146.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 869 kB/s | 51 kB 00:00 2024-03-06T07:19:33.003 INFO:teuthology.orchestra.run.smithi146.stdout:(144/176): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-06T07:19:33.037 INFO:teuthology.orchestra.run.smithi146.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-06T07:19:33.087 INFO:teuthology.orchestra.run.smithi146.stdout:(146/176): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-06T07:19:33.137 INFO:teuthology.orchestra.run.smithi146.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-03-06T07:19:33.163 INFO:teuthology.orchestra.run.smithi146.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-06T07:19:33.246 INFO:teuthology.orchestra.run.smithi146.stdout:(149/176): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-06T07:19:36.330 INFO:teuthology.orchestra.run.smithi146.stdout:(150/176): librbd1-19.0.0-1739.g1bc250c7.el9.x8 1.0 MB/s | 3.1 MB 00:03 2024-03-06T07:19:36.386 INFO:teuthology.orchestra.run.smithi146.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 68 kB 00:00 2024-03-06T07:19:36.497 INFO:teuthology.orchestra.run.smithi146.stdout:(152/176): NetworkManager-1.46.0-1.el9.x86_64.r 21 MB/s | 2.3 MB 00:00 2024-03-06T07:19:36.597 INFO:teuthology.orchestra.run.smithi146.stdout:(153/176): NetworkManager-libnm-1.46.0-1.el9.x8 18 MB/s | 1.8 MB 00:00 2024-03-06T07:19:36.630 INFO:teuthology.orchestra.run.smithi146.stdout:(154/176): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 41 kB 00:00 2024-03-06T07:19:36.673 INFO:teuthology.orchestra.run.smithi146.stdout:(155/176): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 247 kB 00:00 2024-03-06T07:19:36.706 INFO:teuthology.orchestra.run.smithi146.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-03-06T07:19:36.740 INFO:teuthology.orchestra.run.smithi146.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-03-06T07:19:36.782 INFO:teuthology.orchestra.run.smithi146.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-03-06T07:19:36.824 INFO:teuthology.orchestra.run.smithi146.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-03-06T07:19:36.883 INFO:teuthology.orchestra.run.smithi146.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-03-06T07:19:36.933 INFO:teuthology.orchestra.run.smithi146.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.5 MB/s | 433 kB 00:00 2024-03-06T07:19:36.967 INFO:teuthology.orchestra.run.smithi146.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 656 kB/s | 22 kB 00:00 2024-03-06T07:19:37.008 INFO:teuthology.orchestra.run.smithi146.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 107 kB 00:00 2024-03-06T07:19:37.059 INFO:teuthology.orchestra.run.smithi146.stdout:(164/176): librados2-19.0.0-1739.g1bc250c7.el9. 892 kB/s | 3.4 MB 00:03 2024-03-06T07:19:37.084 INFO:teuthology.orchestra.run.smithi146.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 154 kB 00:00 2024-03-06T07:19:37.109 INFO:teuthology.orchestra.run.smithi146.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-03-06T07:19:37.143 INFO:teuthology.orchestra.run.smithi146.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 63 kB 00:00 2024-03-06T07:19:37.168 INFO:teuthology.orchestra.run.smithi146.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 475 kB/s | 28 kB 00:00 2024-03-06T07:19:37.210 INFO:teuthology.orchestra.run.smithi146.stdout:(169/176): openssl-3.0.7-27.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-03-06T07:19:37.286 INFO:teuthology.orchestra.run.smithi146.stdout:(170/176): openssl-libs-3.0.7-27.el9.x86_64.rpm 19 MB/s | 2.2 MB 00:00 2024-03-06T07:19:37.344 INFO:teuthology.orchestra.run.smithi146.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 17 MB/s | 2.2 MB 00:00 2024-03-06T07:19:37.370 INFO:teuthology.orchestra.run.smithi146.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.3 MB/s | 438 kB 00:00 2024-03-06T07:19:37.395 INFO:teuthology.orchestra.run.smithi146.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 611 kB/s | 30 kB 00:00 2024-03-06T07:19:37.420 INFO:teuthology.orchestra.run.smithi146.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-06T07:19:37.470 INFO:teuthology.orchestra.run.smithi146.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-03-06T07:19:39.821 INFO:teuthology.orchestra.run.smithi072.stdout:(176/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 889 kB/s | 47 MB 00:54 2024-03-06T07:19:39.826 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:39.826 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.9 MB/s | 224 MB 01:17 2024-03-06T07:19:39.876 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-06T07:19:40.188 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-06T07:19:40.188 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-06T07:19:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-06T07:19:42.231 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-06T07:19:44.753 INFO:teuthology.orchestra.run.smithi146.stdout:(176/176): ceph-test-19.0.0-1739.g1bc250c7.el9. 1.4 MB/s | 47 MB 00:32 2024-03-06T07:19:44.757 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-03-06T07:19:44.757 INFO:teuthology.orchestra.run.smithi146.stdout:Total 3.8 MB/s | 224 MB 00:59 2024-03-06T07:19:44.803 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-03-06T07:19:45.116 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-03-06T07:19:45.116 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-03-06T07:19:47.052 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-03-06T07:19:47.052 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-03-06T07:19:49.404 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-06T07:19:49.528 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:19:49.565 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:19:49.603 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:49.651 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:49.699 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:19:49.737 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:19:49.792 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:19:49.835 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:19:49.857 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:19:49.912 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:19:50.239 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:19:50.371 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:50.497 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:50.523 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:50.613 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:50.652 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:50.809 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:19:50.870 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:19:50.899 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:51.003 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:51.025 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:51.091 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:51.130 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:19:51.164 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:51.179 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:51.366 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:51.397 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:51.438 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:19:51.832 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:19:51.933 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:51.993 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:52.195 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:52.255 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:52.474 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:52.578 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:19:52.672 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:19:52.705 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:19:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:19:52.828 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:19:52.870 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:19:52.918 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:19:52.956 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:19:53.026 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:19:53.063 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:19:53.107 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:19:53.153 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:19:53.185 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:19:53.262 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:19:53.301 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:19:53.345 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:19:53.374 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:19:53.442 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:19:53.490 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:19:53.562 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:19:53.698 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:19:53.754 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:19:53.986 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-03-06T07:19:54.091 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/202 2024-03-06T07:19:54.125 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-03-06T07:19:54.172 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:54.200 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-03-06T07:19:54.259 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-03-06T07:19:54.269 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:19:54.298 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-03-06T07:19:54.352 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-03-06T07:19:54.360 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:19:54.384 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-03-06T07:19:54.419 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-03-06T07:19:54.462 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-03-06T07:19:54.467 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:19:54.518 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:54.526 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:54.565 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:19:54.604 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:19:54.654 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:19:54.770 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:19:54.797 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-03-06T07:19:54.817 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:19:54.844 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:54.949 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:19:54.978 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:55.070 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:19:55.072 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:19:55.102 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:19:55.252 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:19:55.265 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-03-06T07:19:55.323 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-03-06T07:19:55.338 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:19:55.348 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:55.382 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:19:55.444 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:19:55.466 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:55.513 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:19:55.531 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 17/202 2024-03-06T07:19:55.565 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:19:55.570 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-03-06T07:19:55.601 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:55.616 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-03-06T07:19:55.619 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:19:55.692 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:19:55.727 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:19:55.764 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:19:55.793 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:55.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:19:55.823 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-03-06T07:19:55.847 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:19:55.855 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-03-06T07:19:55.891 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:19:55.934 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:19:56.061 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-03-06T07:19:56.065 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:19:56.110 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:19:56.149 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:19:56.191 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:19:56.198 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:56.258 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/202 2024-03-06T07:19:56.284 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:19:56.326 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:19:56.368 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:19:56.374 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:56.395 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:19:56.420 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:19:56.427 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:56.498 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:19:56.631 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:19:56.652 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/202 2024-03-06T07:19:56.731 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:19:56.756 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-03-06T07:19:56.776 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:19:56.833 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:19:56.850 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-03-06T07:19:56.882 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-03-06T07:19:56.882 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:19:56.948 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-03-06T07:19:56.995 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-03-06T07:19:57.027 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-03-06T07:19:57.070 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-03-06T07:19:57.071 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:19:57.118 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-03-06T07:19:57.127 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:19:57.181 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-03-06T07:19:57.217 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-03-06T07:19:57.260 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-03-06T07:19:57.261 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:19:57.293 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-03-06T07:19:57.301 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:19:57.321 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-03-06T07:19:57.345 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:19:57.377 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:19:57.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-03-06T07:19:57.413 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:19:57.434 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 39/202 2024-03-06T07:19:57.444 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:19:57.476 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:19:57.478 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 40/202 2024-03-06T07:19:57.506 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 41/202 2024-03-06T07:19:57.507 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:19:57.547 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:19:57.574 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-03-06T07:19:57.589 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:19:57.620 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:19:57.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-03-06T07:19:57.693 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-03-06T07:19:57.695 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:19:57.731 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:19:57.798 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:19:57.825 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-03-06T07:19:57.841 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:19:57.873 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-03-06T07:19:57.875 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:19:57.964 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:19:58.055 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:19:58.101 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:19:58.142 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:19:58.189 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:19:58.286 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:19:58.346 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:19:58.386 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-03-06T07:19:58.467 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-03-06T07:19:58.549 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-03-06T07:19:58.588 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:58.596 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-03-06T07:19:58.635 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-03-06T07:19:58.672 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 52/202 2024-03-06T07:19:58.713 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-03-06T07:19:58.829 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-03-06T07:19:58.908 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-03-06T07:19:59.064 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:19:59.116 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:19:59.150 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:19:59.179 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-03-06T07:19:59.198 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:19:59.296 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:19:59.338 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:19:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:19:59.536 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-03-06T07:19:59.544 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:19:59.622 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:19:59.792 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-03-06T07:19:59.836 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-03-06T07:19:59.979 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-03-06T07:20:00.048 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/202 2024-03-06T07:20:00.102 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-03-06T07:20:00.166 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-03-06T07:20:00.206 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-03-06T07:20:00.239 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:00.243 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-03-06T07:20:00.289 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-03-06T07:20:00.325 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-03-06T07:20:00.364 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:00.368 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-03-06T07:20:00.411 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-03-06T07:20:00.544 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-03-06T07:20:00.591 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-03-06T07:20:00.621 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-03-06T07:20:00.663 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-03-06T07:20:00.757 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-03-06T07:20:00.804 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 75/202 2024-03-06T07:20:00.858 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-03-06T07:20:00.887 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-03-06T07:20:00.911 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-03-06T07:20:01.000 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-03-06T07:20:01.129 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-03-06T07:20:01.235 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-03-06T07:20:01.277 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-03-06T07:20:01.326 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-03-06T07:20:01.409 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-03-06T07:20:01.566 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:01.598 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-03-06T07:20:01.649 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-03-06T07:20:01.663 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:01.699 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:01.783 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-03-06T07:20:01.822 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 88/202 2024-03-06T07:20:01.854 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-03-06T07:20:01.886 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-03-06T07:20:01.921 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-03-06T07:20:01.960 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-03-06T07:20:01.989 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-03-06T07:20:02.009 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:02.022 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-03-06T07:20:02.048 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:02.062 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-03-06T07:20:02.093 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-03-06T07:20:02.123 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-03-06T07:20:02.148 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:02.179 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:02.199 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-03-06T07:20:02.225 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:02.233 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-03-06T07:20:02.267 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-03-06T07:20:02.315 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-03-06T07:20:02.358 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-03-06T07:20:02.447 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-03-06T07:20:02.537 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-03-06T07:20:02.597 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-03-06T07:20:02.629 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-03-06T07:20:02.676 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-03-06T07:20:02.678 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:02.704 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:02.772 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-03-06T07:20:02.775 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:02.800 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:02.830 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-03-06T07:20:03.557 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-03-06T07:20:03.601 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-03-06T07:20:03.635 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-03-06T07:20:03.684 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-03-06T07:20:03.771 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-03-06T07:20:03.811 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-03-06T07:20:03.999 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-03-06T07:20:04.034 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-03-06T07:20:04.112 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-03-06T07:20:04.765 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-03-06T07:20:04.882 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-03-06T07:20:05.090 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:05.355 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:05.897 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:05.945 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:06.039 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:06.039 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:06.039 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:06.039 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:06.081 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:06.096 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-03-06T07:20:06.171 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-03-06T07:20:06.197 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-03-06T07:20:06.514 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-03-06T07:20:06.544 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:06.634 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 125/202 2024-03-06T07:20:06.665 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 126/202 2024-03-06T07:20:06.702 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-03-06T07:20:07.162 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-03-06T07:20:07.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:07.260 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 129/202 2024-03-06T07:20:07.290 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 130/202 2024-03-06T07:20:09.527 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:09.562 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:10.108 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 131/202 2024-03-06T07:20:10.156 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:10.247 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 132/202 2024-03-06T07:20:10.247 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-06T07:20:10.247 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-06T07:20:10.248 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:10.291 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:20.912 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-03-06T07:20:20.913 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:21.078 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:21.094 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:21.094 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:21.094 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:21.094 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:21.946 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 133/202 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-03-06T07:20:21.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:22.101 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 134/202 2024-03-06T07:20:22.115 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.115 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-06T07:20:22.115 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:22.594 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:22.620 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:22.620 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.620 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:22.620 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:22.620 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:22.620 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:22.837 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:22.859 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:22.860 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:22.860 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:23.305 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:23.327 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:23.328 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.328 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:23.328 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:23.328 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:23.328 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:23.418 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:23.620 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:23.703 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 135/202 2024-03-06T07:20:23.704 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.704 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-06T07:20:23.704 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:23.704 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-06T07:20:23.704 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:23.829 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:23.863 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:23.879 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:23.901 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 136/202 2024-03-06T07:20:23.902 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:23.902 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-06T07:20:23.902 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:23.902 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-06T07:20:23.902 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:23.903 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:23.963 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:23.987 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:24.212 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:24.247 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:24.365 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:24.387 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 137/202 2024-03-06T07:20:24.387 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:24.387 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-06T07:20:24.387 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:24.387 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-06T07:20:24.387 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:24.491 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-03-06T07:20:24.843 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:24.873 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 139/202 2024-03-06T07:20:24.904 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 140/202 2024-03-06T07:20:24.965 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 141/202 2024-03-06T07:20:24.988 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:25.201 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:25.217 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 142/202 2024-03-06T07:20:25.231 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:25.247 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:25.332 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:25.495 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:25.528 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:25.551 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:25.552 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:25.552 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:25.552 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:25.552 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:25.552 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:25.605 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:25.616 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:25.672 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:26.223 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 143/202 2024-03-06T07:20:26.255 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:26.356 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 144/202 2024-03-06T07:20:26.528 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:26.529 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 145/202 2024-03-06T07:20:26.558 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:26.558 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:26.580 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 146/202 2024-03-06T07:20:26.580 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.581 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-06T07:20:26.581 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:26.581 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-06T07:20:26.581 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:26.582 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:26.582 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.582 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:26.582 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:26.583 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:26.583 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:26.624 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:26.627 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:26.635 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 147/202 2024-03-06T07:20:26.648 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:26.648 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.648 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:26.648 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:26.688 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 148/202 2024-03-06T07:20:26.696 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:26.718 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:26.718 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:26.718 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:26.718 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:26.719 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:26.719 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:27.039 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:27.061 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:27.061 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:27.061 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:27.061 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:27.062 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:27.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:27.579 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-03-06T07:20:27.610 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:27.633 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 150/202 2024-03-06T07:20:27.633 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:27.633 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-06T07:20:27.633 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:27.633 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-06T07:20:27.633 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:27.677 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:27.699 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 151/202 2024-03-06T07:20:27.699 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:27.699 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-06T07:20:27.700 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:27.761 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:27.783 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 152/202 2024-03-06T07:20:27.783 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:27.783 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-06T07:20:27.784 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:27.784 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-06T07:20:27.784 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:28.103 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:28.125 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:28.125 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-06T07:20:28.125 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-06T07:20:28.125 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:28.125 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-06T07:20:28.125 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:31.475 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:31.525 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:31.555 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:31.583 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:31.772 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:31.895 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:31.937 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:32.014 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:32.074 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:32.166 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:32.203 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:32.249 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:32.280 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:32.310 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:32.377 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:32.436 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:32.480 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:32.649 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 154/202 2024-03-06T07:20:32.690 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 155/202 2024-03-06T07:20:32.721 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 156/202 2024-03-06T07:20:32.748 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-03-06T07:20:32.942 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 158/202 2024-03-06T07:20:33.054 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:33.066 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-03-06T07:20:33.102 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:33.122 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:33.122 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 160/202 2024-03-06T07:20:33.173 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-03-06T07:20:33.229 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-03-06T07:20:33.311 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-03-06T07:20:33.365 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-03-06T07:20:33.408 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-03-06T07:20:33.447 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 166/202 2024-03-06T07:20:33.477 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 167/202 2024-03-06T07:20:33.567 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 168/202 2024-03-06T07:20:33.621 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-03-06T07:20:33.659 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-03-06T07:20:33.934 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:34.020 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:34.061 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:34.185 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:34.212 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:34.246 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:34.246 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:34.272 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:34.272 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:34.290 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:34.309 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-03-06T07:20:34.367 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:34.394 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:34.425 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:34.425 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:34.511 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:34.551 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:34.582 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:34.648 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:34.690 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:34.726 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:34.759 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:34.811 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:34.854 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:34.879 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:34.904 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:34.938 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:34.972 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:34.978 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:34.978 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:35.119 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 172/202 2024-03-06T07:20:35.169 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:35.170 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:35.215 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 173/202 2024-03-06T07:20:35.253 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:35.259 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-03-06T07:20:35.293 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:35.320 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:35.346 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:35.346 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-03-06T07:20:35.372 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:35.379 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-03-06T07:20:35.405 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-03-06T07:20:35.430 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-03-06T07:20:35.447 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:35.457 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-03-06T07:20:35.457 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:35.533 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-03-06T07:20:35.564 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-03-06T07:20:35.599 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-03-06T07:20:35.600 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:35.668 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-03-06T07:20:35.694 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-03-06T07:20:35.727 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-03-06T07:20:35.752 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-03-06T07:20:35.782 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-03-06T07:20:35.820 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-03-06T07:20:35.854 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-03-06T07:20:35.893 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-03-06T07:20:35.925 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-03-06T07:20:35.950 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-03-06T07:20:35.984 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-03-06T07:20:36.018 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:36.043 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:36.049 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:36.237 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-03-06T07:20:36.237 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:36.320 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-03-06T07:20:36.352 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:36.379 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-03-06T07:20:36.404 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-03-06T07:20:36.430 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-03-06T07:20:36.504 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:41.474 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:41.474 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:41.475 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:41.476 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:41.477 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:41.478 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:41.479 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:41.480 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:41.481 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:41.482 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:41.483 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:41.484 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:41.485 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:41.486 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:41.487 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:41.488 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:41.489 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:41.490 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:41.491 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 1/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 3/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 4/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1739.g1bc25 5/202 2024-03-06T07:20:43.922 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 6/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 7/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 8/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 9/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 10/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 11/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 12/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 13/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 14/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_ 15/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 16/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 17/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_6 18/202 2024-03-06T07:20:43.923 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86 19/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 20/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el 21/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9. 22/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_6 23/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 24/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 25/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 26/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 27/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 28/202 2024-03-06T07:20:43.924 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 29/202 2024-03-06T07:20:43.940 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7. 30/202 2024-03-06T07:20:43.940 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noa 31/202 2024-03-06T07:20:43.940 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.n 32/202 2024-03-06T07:20:43.940 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc 33/202 2024-03-06T07:20:43.940 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el 34/202 2024-03-06T07:20:43.940 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 35/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.e 36/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 37/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 38/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 39/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-03-06T07:20:43.941 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 59/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-03-06T07:20:43.942 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-03-06T07:20:43.943 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-03-06T07:20:43.944 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-03-06T07:20:43.945 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/202 2024-03-06T07:20:43.946 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-03-06T07:20:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-03-06T07:20:43.948 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 151/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 153/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/202 2024-03-06T07:20:43.949 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-03-06T07:20:43.950 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-03-06T07:20:43.951 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 189/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 191/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-03-06T07:20:43.952 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-03-06T07:20:43.953 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:44.456 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.457 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.458 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.459 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:44.460 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.461 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:44.462 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:44.463 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:44.464 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:44.465 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:44.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:44.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:44.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:44.469 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:44.470 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:44.470 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:44.470 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-03-06T07:20:45.150 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:46.984 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: acl-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: iproute-6.7.0-2.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-03-06T07:20:46.985 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.986 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: automake-1.16.2-8.el9.noarch 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: bison-3.7.4-5.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.987 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.988 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top-2:19.0.0-1739.g1bc250c7.el9.noarch 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: flex-2.6.4-9.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2024-03-06T07:20:46.989 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-06T07:20:46.990 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-06T07:20:46.991 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-13.11-1.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.992 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-06T07:20:46.993 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-06T07:20:46.994 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-06T07:20:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-06T07:20:46.996 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-1739.g1bc250c7.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-06T07:20:46.997 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-06T07:20:46.998 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-06T07:20:46.998 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:46.998 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-06T07:20:47.594 DEBUG:teuthology.parallel:result is None 2024-03-06T07:20:47.594 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:47.814 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:47.839 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:47.839 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:47.840 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:47.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd 2024-03-06T07:20:47.988 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-06T07:20:48.014 INFO:teuthology.orchestra.run.smithi146.stdout:19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:48.014 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1739.g1bc250c7.el9 2024-03-06T07:20:48.014 INFO:teuthology.task.install:The correct ceph version 19.0.0-1739.g1bc250c7 is installed. 2024-03-06T07:20:48.016 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-06T07:20:48.016 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:48.016 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:48.050 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:48.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-06T07:20:48.090 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-06T07:20:48.091 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:48.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:48.121 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:48.189 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:48.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-03-06T07:20:48.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-06T07:20:48.292 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-06T07:20:48.292 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:48.293 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:48.322 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:48.391 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:48.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-06T07:20:48.422 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-06T07:20:48.493 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-06T07:20:48.494 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:48.494 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:48.521 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:48.589 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:48.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-03-06T07:20:48.619 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-06T07:20:48.689 INFO:teuthology.run_tasks:Running task ceph... 2024-03-06T07:20:48.772 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-06T07:20:48.772 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:48.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-03-06T07:20:48.803 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-06T07:20:48.804 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:48.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-06T07:20:48.872 INFO:tasks.ceph:Creating extra log directories... 2024-03-06T07:20:48.873 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:48.911 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-06T07:20:48.949 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-06T07:20:48.949 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-06T07:20:48.949 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-06T07:20:48.949 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:48.950 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:48.992 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:49.111 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:49.138 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:49.138 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-06T07:20:49.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:49.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-06T07:20:49.309 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-06T07:20:49.309 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-06T07:20:49.310 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060', 'branch': 'wip-lusov-qdb', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays}', 'email': None, 'first_in_suite': False, 'job_id': '7583060', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-qdb'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-qdb', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5072, 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'sleep_before_teardown': 0, 'subset': '246/10000', 'suite': 'fs', 'suite_branch': 'wip-lusov-qdb', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCtvxMT4pfpfCRFYMBCFiNMlviQr8bJ6S6yeXVqeLhb68KzR5NS0ZCgwYjKleIYo7G/6Gl9YgE2jNprIz1lZFac=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJCTiRBUCAsvCfWLVyHQLrRgxHEjeAHsL1fnPo17LH6ZJ0FHIDmxIoBRs5NN378dcLl9HGNxpNaBHagISm2z7PM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-06_07:04:29', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-06T07:20:49.310 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:49.312 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-06T07:20:49.359 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:49.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-03-06T07:20:49.426 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:49.427 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:49.446 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:49.446 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:49.500 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:49.500 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.500 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-06T07:20:49.501 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.501 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.501 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-06 07:20:35.784843551 +0000 2024-03-06T07:20:49.501 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-06 07:17:28.700245489 +0000 2024-03-06T07:20:49.501 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-06 07:17:28.700245489 +0000 2024-03-06T07:20:49.501 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-06 07:17:28.700245489 +0000 2024-03-06T07:20:49.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:49.567 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-06T07:20:49.567 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-06T07:20:49.567 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000323092 s, 1.6 MB/s 2024-03-06T07:20:49.568 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:49.623 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-06 07:20:35.784843551 +0000 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-06 07:17:29.040238116 +0000 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-06 07:17:29.040238116 +0000 2024-03-06T07:20:49.677 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-06 07:17:29.040238116 +0000 2024-03-06T07:20:49.678 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:49.742 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-06T07:20:49.742 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-06T07:20:49.742 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000205733 s, 2.5 MB/s 2024-03-06T07:20:49.743 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:49.798 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-06 07:20:35.784843551 +0000 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-06 07:17:29.384230657 +0000 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-06 07:17:29.384230657 +0000 2024-03-06T07:20:49.852 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-06 07:17:29.384230657 +0000 2024-03-06T07:20:49.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-06T07:20:49.917 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.00018298 s, 2.8 MB/s 2024-03-06T07:20:49.919 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:49.974 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:50.028 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:50.028 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-06 07:20:35.785843529 +0000 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-06 07:17:29.720223371 +0000 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-06 07:17:29.720223371 +0000 2024-03-06T07:20:50.029 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-06 07:17:29.720223371 +0000 2024-03-06T07:20:50.029 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:50.094 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-06T07:20:50.094 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-06T07:20:50.094 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000174013 s, 2.9 MB/s 2024-03-06T07:20:50.095 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:50.151 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:50.151 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:50.151 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-03-06T07:20:50.170 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-06T07:20:50.170 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-03-06T07:20:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-06T07:20:50.228 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.228 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-06T07:20:50.228 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.228 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.228 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-06 07:20:36.887867157 +0000 2024-03-06T07:20:50.229 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-06 07:17:24.111076620 +0000 2024-03-06T07:20:50.229 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-06 07:17:24.111076620 +0000 2024-03-06T07:20:50.229 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-06 07:17:24.111076620 +0000 2024-03-06T07:20:50.229 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-06T07:20:50.298 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-06T07:20:50.299 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-06T07:20:50.299 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000199294 s, 2.6 MB/s 2024-03-06T07:20:50.300 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-06T07:20:50.358 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-03-06T07:20:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-06T07:20:50.415 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.415 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-06T07:20:50.415 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.415 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.415 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-06 07:20:36.887867157 +0000 2024-03-06T07:20:50.416 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-06 07:17:24.445071539 +0000 2024-03-06T07:20:50.416 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-06 07:17:24.445071539 +0000 2024-03-06T07:20:50.416 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-06 07:17:24.445071539 +0000 2024-03-06T07:20:50.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-06T07:20:50.484 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-06T07:20:50.484 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-06T07:20:50.484 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000198747 s, 2.6 MB/s 2024-03-06T07:20:50.485 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-06T07:20:50.543 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-06 07:20:36.887867157 +0000 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-06 07:17:24.978063432 +0000 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-06 07:17:24.978063432 +0000 2024-03-06T07:20:50.599 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-06 07:17:24.978063432 +0000 2024-03-06T07:20:50.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-06T07:20:50.666 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-06T07:20:50.666 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-06T07:20:50.666 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000269624 s, 1.9 MB/s 2024-03-06T07:20:50.668 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-06T07:20:50.725 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-03-06T07:20:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-06T07:20:50.780 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-06T07:20:50.780 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-06T07:20:50.780 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-06T07:20:50.781 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-03-06T07:20:50.781 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-03-06 07:20:36.888867142 +0000 2024-03-06T07:20:50.781 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-03-06 07:17:25.328058109 +0000 2024-03-06T07:20:50.781 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-03-06 07:17:25.328058109 +0000 2024-03-06T07:20:50.781 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-03-06 07:17:25.328058109 +0000 2024-03-06T07:20:50.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-06T07:20:50.848 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-03-06T07:20:50.848 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-03-06T07:20:50.848 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.0001951 s, 2.6 MB/s 2024-03-06T07:20:50.849 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-06T07:20:50.905 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:50.906 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:50.906 INFO:tasks.ceph:Generating config... 2024-03-06T07:20:50.907 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-06T07:20:50.907 INFO:tasks.ceph:[client] debug client = 20 2024-03-06T07:20:50.907 INFO:tasks.ceph:[client] debug ms = 1 2024-03-06T07:20:50.907 INFO:tasks.ceph:[client] pid file = 2024-03-06T07:20:50.907 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-06T07:20:50.907 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-06T07:20:50.907 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-06T07:20:50.907 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-06T07:20:50.907 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-06T07:20:50.908 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-06T07:20:50.908 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-06T07:20:50.908 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-06T07:20:50.909 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-06T07:20:50.910 INFO:tasks.ceph:Setting up mon.a... 2024-03-06T07:20:50.910 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-06T07:20:50.963 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.keyring 2024-03-06T07:20:50.965 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-06T07:20:51.010 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:51.037 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.72'), ('mon.b', '172.21.15.146'), ('mon.c', '[v2:172.21.15.146:3301,v1:172.21.15.146:6790]')] 2024-03-06T07:20:51.040 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-06T07:20:51.041 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:51.041 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:51.092 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.72 --add b 172.21.15.146 --addv c '[v2:172.21.15.146:3301,v1:172.21.15.146:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:51.177 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:51.177 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: generated fsid 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 0 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:fsid 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:last_changed 2024-03-06T07:20:51.176522+0000 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-03-06T07:20:51.176522+0000 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:min_mon_release 17 (quincy) 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:election_strategy: 1 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-03-06T07:20:51.178 INFO:teuthology.orchestra.run.smithi072.stdout:1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-03-06T07:20:51.179 INFO:teuthology.orchestra.run.smithi072.stdout:2: [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] mon.c 2024-03-06T07:20:51.179 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-06T07:20:51.180 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-06T07:20:51.195 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0927e84a-942e-4592-98d9-697c1cf6243f... 2024-03-06T07:20:51.196 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:51.237 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:51.287 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.288 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:20:51.289 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:51.290 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.291 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:20:51.292 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:20:51.293 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:20:51.294 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.295 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:51.296 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:20:51.297 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:20:51.298 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:20:51.299 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:20:51.301 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:20:51.302 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:20:51.303 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:20:51.304 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:20:51.305 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.306 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:20:51.307 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:20:51.308 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:20:51.309 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:20:51.310 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:20:51.311 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:20:51.311 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:20:51.311 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:20:51.311 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:20:51.311 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:20:51.312 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-06T07:20:51.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-06T07:20:51.409 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-06T07:20:51.409 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:51.409 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-06T07:20:51.423 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:51.423 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-06T07:20:51.478 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:20:51.479 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:51.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:20:51.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:51.559 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:20:51.560 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:51.614 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:20:51.614 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:51.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-06T07:20:51.614 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-06T07:20:51.654 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:20:51.655 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:51.710 INFO:tasks.ceph:Setting up mon nodes... 2024-03-06T07:20:51.710 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-06T07:20:51.710 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:20:51.765 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-06T07:20:51.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:20:51.827 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-06T07:20:51.830 INFO:tasks.ceph:Setting up mds nodes... 2024-03-06T07:20:51.830 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:20:51.886 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-06T07:20:51.889 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-06T07:20:51.916 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:20:52.010 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-06T07:20:52.013 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-06T07:20:52.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:20:52.094 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-06T07:20:52.096 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-06T07:20:52.123 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:20:52.218 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-06T07:20:52.220 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-06T07:20:52.247 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-06T07:20:52.248 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-06T07:20:52.289 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-06T07:20:52.305 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-06T07:20:52.386 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-06T07:20:52.401 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-06T07:20:52.443 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-06T07:20:52.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-06T07:20:52.541 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-06T07:20:52.560 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-06T07:20:52.560 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-06T07:20:52.560 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:52.587 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:52.588 INFO:tasks.ceph:role: osd.0 2024-03-06T07:20:52.588 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:20:52.588 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:20:52.661 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:52.661 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:52.661 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:52.661 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:52.661 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:52.661 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:52.662 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:52.662 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:52.662 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:52.662 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:53.174 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-06T07:20:53.176 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-06T07:20:53.176 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:53.325 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:53.361 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:53.428 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:53.428 INFO:tasks.ceph:role: osd.1 2024-03-06T07:20:53.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:20:53.429 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:20:53.503 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:53.503 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:53.503 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:53.503 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:53.503 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:53.504 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:53.504 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:53.504 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:53.504 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:53.504 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:53.952 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-06T07:20:53.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-06T07:20:53.954 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:54.057 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-06T07:20:54.088 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:54.156 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:54.157 INFO:tasks.ceph:role: osd.2 2024-03-06T07:20:54.157 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:20:54.157 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:54.231 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:54.232 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:54.232 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:54.232 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:54.715 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-06T07:20:54.717 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-06T07:20:54.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:54.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-06T07:20:54.890 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:54.959 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-06T07:20:54.959 INFO:tasks.ceph:role: osd.3 2024-03-06T07:20:54.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:20:54.960 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:20:55.035 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:20:55.036 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:20:55.526 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-06T07:20:55.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-06T07:20:55.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:55.669 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-06T07:20:55.705 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:55.829+0000 7f6d6d78a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-06T07:20:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:55.829+0000 7f6d6d78a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-06T07:20:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:55.829+0000 7f6d6d78a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:55.829+0000 7f6d6d78a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:55.829+0000 7f6d6d78a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-06T07:20:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:55.830+0000 7f6d6d78a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-06T07:20:58.681 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-06T07:20:58.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:20:58.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:58.798+0000 7f14decd9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-06T07:20:58.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:58.799+0000 7f14decd9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-06T07:20:58.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:58.799+0000 7f14decd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:20:58.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:58.799+0000 7f14decd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:20:58.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:58.799+0000 7f14decd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-06T07:20:58.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:20:58.799+0000 7f14decd9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-06T07:21:01.653 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-06T07:21:01.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:01.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:01.774+0000 7f0feaa17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-06T07:21:01.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:01.774+0000 7f0feaa17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-06T07:21:01.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:01.774+0000 7f0feaa17640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:01.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:01.774+0000 7f0feaa17640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:01.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:01.774+0000 7f0feaa17640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-06T07:21:01.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:01.774+0000 7f0feaa17640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-06T07:21:04.625 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-06T07:21:04.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:04.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:04.744+0000 7f8438e9d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-06T07:21:04.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:04.745+0000 7f8438e9d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-06T07:21:04.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:04.745+0000 7f8438e9d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:04.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:04.745+0000 7f8438e9d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:04.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:04.745+0000 7f8438e9d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-06T07:21:04.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:04.745+0000 7f8438e9d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-06T07:21:07.600 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-06T07:21:07.631 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:07.663 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:07.663 INFO:tasks.ceph:role: osd.4 2024-03-06T07:21:07.663 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:21:07.663 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-06T07:21:07.740 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:07.740 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:07.740 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:07.740 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:07.740 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:07.740 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:07.741 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:07.741 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:07.741 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:07.741 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:08.253 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-06T07:21:08.255 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-06T07:21:08.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:08.411 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:08.444 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:08.512 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:08.512 INFO:tasks.ceph:role: osd.5 2024-03-06T07:21:08.512 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:21:08.512 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-06T07:21:08.586 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:08.586 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:08.587 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:09.036 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-06T07:21:09.038 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-06T07:21:09.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:09.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:09.192 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:09.260 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:09.260 INFO:tasks.ceph:role: osd.6 2024-03-06T07:21:09.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:21:09.260 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-06T07:21:09.334 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:09.335 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:09.866 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-06T07:21:09.868 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-06T07:21:09.868 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:09.982 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:10.014 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:10.082 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-06T07:21:10.083 INFO:tasks.ceph:role: osd.7 2024-03-06T07:21:10.083 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:21:10.083 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-06T07:21:10.157 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-06T07:21:10.158 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-06T07:21:10.635 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-03-06T07:21:10.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-03-06T07:21:10.637 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:10.769 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:10.802 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:10.915 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:10.913+0000 7fa8da2f9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-06T07:21:10.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:10.913+0000 7fa8da2f9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-06T07:21:10.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:10.913+0000 7fa8da2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:10.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:10.913+0000 7fa8da2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:10.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:10.913+0000 7fa8da2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-06T07:21:10.916 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:10.914+0000 7fa8da2f9640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-06T07:21:13.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-06T07:21:13.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:13.885 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:13.883+0000 7f319ea15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-06T07:21:13.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:13.883+0000 7f319ea15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-06T07:21:13.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:13.883+0000 7f319ea15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:13.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:13.883+0000 7f319ea15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:13.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:13.883+0000 7f319ea15640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-06T07:21:13.886 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:13.884+0000 7f319ea15640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-06T07:21:16.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-06T07:21:16.756 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:16.843 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:16.841+0000 7f462d215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-06T07:21:16.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:16.841+0000 7f462d215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-06T07:21:16.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:16.841+0000 7f462d215640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:16.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:16.841+0000 7f462d215640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:16.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:16.841+0000 7f462d215640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-06T07:21:16.844 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:16.842+0000 7f462d215640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-06T07:21:19.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:19.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:19.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-06T07:21:19.717 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:19.802+0000 7f3ab4385640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-06T07:21:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:19.802+0000 7f3ab4385640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-06T07:21:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:19.802+0000 7f3ab4385640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:19.802+0000 7f3ab4385640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:19.802+0000 7f3ab4385640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-06T07:21:19.804 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:19.803+0000 7f3ab4385640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-06T07:21:22.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-06T07:21:22.678 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-06T07:21:22.678 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:22.678 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-06T07:21:22.710 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:22.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-06T07:21:22.781 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:22.781 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-06T07:21:22.851 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:22.851 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-06T07:21:22.920 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:22.920 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-06T07:21:22.988 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:22.988 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-06T07:21:23.057 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:23.057 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-06T07:21:23.124 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.124 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-06T07:21:23.156 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.156 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-06T07:21:23.227 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-06T07:21:23.296 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-06T07:21:23.365 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-06T07:21:23.433 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-06T07:21:23.501 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-06T07:21:23.568 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:23.568 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-06T07:21:23.583 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:23.584 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-06T07:21:23.639 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.639 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-06T07:21:23.653 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:23.654 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-06T07:21:23.709 INFO:tasks.ceph:Adding keys to all mons... 2024-03-06T07:21:23.709 DEBUG:teuthology.orchestra.run.smithi072:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:23.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.y] 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng== 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg== 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q== 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.0] 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg== 2024-03-06T07:21:23.752 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.1] 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw== 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.2] 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg== 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.3] 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w== 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.x] 2024-03-06T07:21:23.753 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.4] 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.5] 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.6] 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.7] 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ== 2024-03-06T07:21:23.754 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout:[client.2] 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q== 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout:[client.1] 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg== 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout:[client.3] 2024-03-06T07:21:23.755 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ== 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng== 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg== 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q== 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-03-06T07:21:23.774 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A== 2024-03-06T07:21:23.775 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[client.2] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg== 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout:[client.3] 2024-03-06T07:21:23.776 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ== 2024-03-06T07:21:23.777 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:23.798 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:23.869 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.872 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.927 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.930 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:23.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:23.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.038 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.040 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.092 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.148 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.150 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.202 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:24.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-06T07:21:24.256 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:24.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:24.313 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:24.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-06T07:21:24.370 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.372 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.427 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.534 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-06T07:21:24.588 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.590 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.644 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.701 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.703 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.758 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.761 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-06T07:21:24.816 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-06T07:21:24.816 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:24.842 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:24.939 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-06T07:21:24.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:24.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:25.089 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-06T07:21:25.119 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:25.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-06T07:21:25.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-06T07:21:25.306 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:25.309 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-06T07:21:25.361 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-06T07:21:25.361 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-06T07:21:25.362 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-06T07:21:25.364 INFO:tasks.ceph.mon.a:Started 2024-03-06T07:21:25.364 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-06T07:21:25.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-06T07:21:25.404 INFO:tasks.ceph.mon.b:Started 2024-03-06T07:21:25.404 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-06T07:21:25.404 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-06T07:21:25.406 INFO:tasks.ceph.mon.c:Started 2024-03-06T07:21:25.407 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-06T07:21:25.407 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-06T07:21:25.407 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-06T07:21:25.409 INFO:tasks.ceph.mgr.y:Started 2024-03-06T07:21:25.409 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-06T07:21:25.409 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-06T07:21:25.411 INFO:tasks.ceph.mgr.x:Started 2024-03-06T07:21:25.411 INFO:tasks.ceph:Setting crush tunables to default 2024-03-06T07:21:25.411 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd crush tunables default 2024-03-06T07:21:25.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.510+0000 7fae87c42640 1 Processor -- start 2024-03-06T07:21:25.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 -- start start 2024-03-06T07:21:25.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae800cca00 con 0x7fae800b2eb0 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8015c2b0 con 0x7fae800d1cd0 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae87c42640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae800d2160 con 0x7fae8015bed0 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae851dd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae851dd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53392/0 (socket says 172.21.15.72:53392) 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.511+0000 7fae851dd640 1 -- 172.21.15.72:0/2167203190 learned_addr learned my addr 172.21.15.72:0/2167203190 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae851dd640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 msgr2=0x7fae800d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae851dd640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 msgr2=0x7fae800d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae851dd640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae851dd640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae861df640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae859de640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae859de640 1 -- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 msgr2=0x7fae800cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae859de640 1 -- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 msgr2=0x7fae800cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae859de640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae859de640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae861df640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 msgr2=0x7fae800ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae861df640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 msgr2=0x7fae800ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae861df640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.512+0000 7fae861df640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae851dd640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.72:3300/0 conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae861df640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae859de640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae861df640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 msgr2=0x7fae800ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae861df640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 msgr2=0x7fae800ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae861df640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae859de640 1 -- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 msgr2=0x7fae800cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae861df640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae859de640 1 -- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 msgr2=0x7fae800cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-06T07:21:25.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae859de640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-06T07:21:25.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae859de640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-06T07:21:25.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae851dd640 1 -- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 msgr2=0x7fae800cf9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:25.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae851dd640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae851dd640 1 -- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 msgr2=0x7fae800ce4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:25.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae851dd640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.713+0000 7fae851dd640 1 -- 172.21.15.72:0/2167203190 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800d0330 con 0x7fae800b2eb0 2024-03-06T07:21:25.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.714+0000 7fae851dd640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae800b2eb0 0x7fae800d1790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009100 tx=0x7fae7c02bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f944018dda9b10c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.714+0000 7fae849dc640 1 -- 172.21.15.72:0/2167203190 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c039070 con 0x7fae800b2eb0 2024-03-06T07:21:25.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.714+0000 7fae849dc640 1 -- 172.21.15.72:0/2167203190 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c02ca00 con 0x7fae800b2eb0 2024-03-06T07:21:25.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.714+0000 7fae849dc640 1 -- 172.21.15.72:0/2167203190 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00f040 con 0x7fae800b2eb0 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 -- 172.21.15.72:0/2167203190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae800b2eb0 msgr2=0x7fae800d1790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae800b2eb0 0x7fae800d1790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fae7c009100 tx=0x7fae7c02bc50 comp rx=0 tx=0).stop 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 -- 172.21.15.72:0/2167203190 shutdown_connections 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 --2- 172.21.15.72:0/2167203190 >> v2:172.21.15.146:3300/0 conn(0x7fae800d1cd0 0x7fae800ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae800b2eb0 0x7fae800d1790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 --2- 172.21.15.72:0/2167203190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae8015bed0 0x7fae800cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 -- 172.21.15.72:0/2167203190 >> 172.21.15.72:0/2167203190 conn(0x7fae8009a9f0 msgr2=0x7fae8009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 -- 172.21.15.72:0/2167203190 shutdown_connections 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.715+0000 7fae87c42640 1 -- 172.21.15.72:0/2167203190 wait complete. 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.716+0000 7fae87c42640 1 Processor -- start 2024-03-06T07:21:25.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 -- start start 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae800d1cd0 0x7fae800b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 0x7fae8015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae800b6030 0x7fae80157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae800cc5d0 con 0x7fae8015bed0 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae800d08f0 con 0x7fae800b6030 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae87c42640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae800cc750 con 0x7fae800d1cd0 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae851dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 0x7fae8015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae851dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 0x7fae8015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53418/0 (socket says 172.21.15.72:53418) 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae851dd640 1 -- 172.21.15.72:0/709518445 learned_addr learned my addr 172.21.15.72:0/709518445 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae859de640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae800d1cd0 0x7fae800b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.717+0000 7fae861df640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae800b6030 0x7fae80157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae851dd640 1 -- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae800d1cd0 msgr2=0x7fae800b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae851dd640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae800d1cd0 0x7fae800b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae851dd640 1 -- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae800b6030 msgr2=0x7fae80157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae851dd640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae800b6030 0x7fae80157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae851dd640 1 -- 172.21.15.72:0/709518445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae800b5260 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae851dd640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 0x7fae8015d6f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae7c037040 tx=0x7fae7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c044020 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7c003610 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae800b5540 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7c039040 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.718+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae800b4e50 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.719+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae800b5a60 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.720+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fae7c047020 con 0x7fae8015bed0 2024-03-06T07:21:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.720+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae7c02d170 con 0x7fae8015bed0 2024-03-06T07:21:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.722+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fae800b5a60 con 0x7fae8015bed0 2024-03-06T07:21:25.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.790+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fae8004f330 con 0x7fae8015bed0 2024-03-06T07:21:25.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.793+0000 7fae767fc640 1 -- 172.21.15.72:0/709518445 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fae7c04e030 con 0x7fae8015bed0 2024-03-06T07:21:25.795 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 msgr2=0x7fae8015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 0x7fae8015d6f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae7c037040 tx=0x7fae7c004730 comp rx=0 tx=0).stop 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 shutdown_connections 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fae800b6030 0x7fae80157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae8015bed0 0x7fae8015d6f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 --2- 172.21.15.72:0/709518445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fae800d1cd0 0x7fae800b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:25.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 >> 172.21.15.72:0/709518445 conn(0x7fae8009a9f0 msgr2=0x7fae800ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:25.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 shutdown_connections 2024-03-06T07:21:25.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:25.795+0000 7fae87c42640 1 -- 172.21.15.72:0/709518445 wait complete. 2024-03-06T07:21:25.807 INFO:tasks.ceph:check_enable_crimson: False 2024-03-06T07:21:25.807 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-06T07:21:25.808 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:25.808 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-06T07:21:25.834 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:25.834 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-06T07:21:25.901 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:25.901 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-06T07:21:25.970 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:21:25.971 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-06T07:21:26.038 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:26.038 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-06T07:21:26.065 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:26.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-06T07:21:26.132 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:26.133 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-06T07:21:26.190 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:26.189+0000 7f1eb2821040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:26.199 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:21:26.199 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-06T07:21:26.205 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:26.203+0000 7fb2ffbb9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-06T07:21:26.247 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:26.245+0000 7f1eb2821040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:26.258 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:26.257+0000 7fb2ffbb9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-06T07:21:26.267 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new e430101d-cbf3-4e3e-a12e-d68c57daef06 0 2024-03-06T07:21:26.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.406+0000 7f539a75c640 1 Processor -- start 2024-03-06T07:21:26.408 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.406+0000 7f539a75c640 1 -- start start 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f539a75c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940cf3c0 0x7f53940d4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f539a75c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f53940b7760 0x7f53940b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f539a75c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940b6cc0 con 0x7f53940d07e0 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f539a75c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53940b72d0 con 0x7f53940b7760 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f539a75c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53940cf970 con 0x7f53940cf3c0 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f53937fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940cf3c0 0x7f53940d4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f5393fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f53940d07e0 0x7f53940d0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53100/0 (socket says 172.21.15.146:53100) 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f5393fff640 1 -- 172.21.15.146:0/752801558 learned_addr learned my addr 172.21.15.146:0/752801558 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f5398cf9640 1 --2- 172.21.15.146:0/752801558 >> v2:172.21.15.72:3300/0 conn(0x7f53940b7760 0x7f53940b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f5392ffd640 1 -- 172.21.15.146:0/752801558 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895878676 0 0) 0x7f53940b6cc0 con 0x7f53940d07e0 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f5392ffd640 1 -- 172.21.15.146:0/752801558 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c003520 con 0x7f53940d07e0 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f53937fe640 1 -- 172.21.15.146:0/752801558 >> v2:172.21.15.72:3300/0 conn(0x7f53940b7760 msgr2=0x7f53940b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f53937fe640 1 --2- 172.21.15.146:0/752801558 >> v2:172.21.15.72:3300/0 conn(0x7f53940b7760 0x7f53940b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f53937fe640 1 -- 172.21.15.146:0/752801558 >> v1:172.21.15.146:6789/0 conn(0x7f53940d07e0 legacy=0x7f53940d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.407+0000 7f53937fe640 1 -- 172.21.15.146:0/752801558 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940b6f60 con 0x7f53940cf3c0 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.408+0000 7f53937fe640 1 --2- 172.21.15.146:0/752801558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940cf3c0 0x7f53940d4710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f53840097a0 tx=0x7f538402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2103ee2c8e044387 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.408+0000 7f5392ffd640 1 -- 172.21.15.146:0/752801558 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384035020 con 0x7f53940cf3c0 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.408+0000 7f5392ffd640 1 -- 172.21.15.146:0/752801558 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5384031830 con 0x7f53940cf3c0 2024-03-06T07:21:26.411 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.408+0000 7f5392ffd640 1 -- 172.21.15.146:0/752801558 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5384031c00 con 0x7f53940cf3c0 2024-03-06T07:21:26.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 -- 172.21.15.146:0/752801558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940cf3c0 msgr2=0x7f53940d4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 --2- 172.21.15.146:0/752801558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940cf3c0 0x7f53940d4710 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f53840097a0 tx=0x7f538402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 -- 172.21.15.146:0/752801558 shutdown_connections 2024-03-06T07:21:26.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 --2- 172.21.15.146:0/752801558 >> v2:172.21.15.72:3300/0 conn(0x7f53940b7760 0x7f53940b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 --2- 172.21.15.146:0/752801558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940cf3c0 0x7f53940d4710 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 -- 172.21.15.146:0/752801558 >> 172.21.15.146:0/752801558 conn(0x7f539409d020 msgr2=0x7f53940d3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 -- 172.21.15.146:0/752801558 shutdown_connections 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.409+0000 7f539a75c640 1 -- 172.21.15.146:0/752801558 wait complete. 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.410+0000 7f539a75c640 1 Processor -- start 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.410+0000 7f539a75c640 1 -- start start 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f539a75c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 0x7f53940cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f539a75c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53940cf3c0 0x7f53941f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f539a75c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53940d07e0 0x7f53940d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f539a75c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394161820 con 0x7f53940cf3c0 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f539a75c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53940d4c50 con 0x7f53940d07e0 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f539a75c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53941619a0 con 0x7f53940b7760 2024-03-06T07:21:26.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5393fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 0x7f53940cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5393fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 0x7f53940cbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:34232/0 (socket says 172.21.15.146:34232) 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53940d07e0 0x7f53940d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5393fff640 1 -- 172.21.15.146:0/484171174 learned_addr learned my addr 172.21.15.146:0/484171174 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f53937fe640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53940cf3c0 0x7f53941f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 -- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 msgr2=0x7f53940cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 0x7f53940cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 -- 172.21.15.146:0/484171174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53940cf3c0 msgr2=0x7f53941f4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53940cf3c0 0x7f53941f4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 -- 172.21.15.146:0/484171174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53941f4b80 con 0x7f53940d07e0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5393fff640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 0x7f53940cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.411+0000 7f5398cf9640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53940d07e0 0x7f53940d1ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f538c00eba0 tx=0x7f538c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c018020 con 0x7f53940d07e0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c00cd20 con 0x7f53940d07e0 2024-03-06T07:21:26.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53940cc4f0 con 0x7f53940d07e0 2024-03-06T07:21:26.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0146c0 con 0x7f53940d07e0 2024-03-06T07:21:26.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53940ccab0 con 0x7f53940d07e0 2024-03-06T07:21:26.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f538c005d10 con 0x7f53940d07e0 2024-03-06T07:21:26.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.412+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f538c01dd50 con 0x7f53940d07e0 2024-03-06T07:21:26.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.413+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53940a1b00 con 0x7f53940d07e0 2024-03-06T07:21:26.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.415+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f538c022030 con 0x7f53940d07e0 2024-03-06T07:21:26.489 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.487+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e430101d-cbf3-4e3e-a12e-d68c57daef06", "id": 0} v 0) v1 -- 0x7f5394068ff0 con 0x7f53940d07e0 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.493+0000 7f5390ff9640 1 -- 172.21.15.146:0/484171174 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e430101d-cbf3-4e3e-a12e-d68c57daef06", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f53941f4b80 con 0x7f53940d07e0 2024-03-06T07:21:26.495 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-03-06T07:21:26.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53940d07e0 msgr2=0x7f53940d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.496 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53940d07e0 0x7f53940d1ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f538c00eba0 tx=0x7f538c00c490 comp rx=0 tx=0).stop 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 shutdown_connections 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53940d07e0 0x7f53940d1ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53940cf3c0 0x7f53941f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 --2- 172.21.15.146:0/484171174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53940b7760 0x7f53940cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 >> 172.21.15.146:0/484171174 conn(0x7f539409d020 msgr2=0x7f53940b8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 shutdown_connections 2024-03-06T07:21:26.497 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.494+0000 7f539a75c640 1 -- 172.21.15.146:0/484171174 wait complete. 2024-03-06T07:21:26.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 7cd0d2fb-bff1-41c8-b382-01b0b22d939b 1 2024-03-06T07:21:26.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.606+0000 7fdfb8cb1640 1 Processor -- start 2024-03-06T07:21:26.608 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.606+0000 7fdfb8cb1640 1 -- start start 2024-03-06T07:21:26.609 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb8cb1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdfb406bd80 0x7fdfb406c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb8cb1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb4072e90 con 0x7fdfb406b9a0 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb8cb1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfb4073170 con 0x7fdfb40b5e20 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb8cb1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfb41619c0 con 0x7fdfb406bd80 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb1d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fdfb40b5e20 0x7fdfb4072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53106/0 (socket says 172.21.15.146:53106) 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb1d9b640 1 -- 172.21.15.146:0/3169696807 learned_addr learned my addr 172.21.15.146:0/3169696807 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb2d9d640 1 --2- 172.21.15.146:0/3169696807 >> v2:172.21.15.72:3300/0 conn(0x7fdfb406bd80 0x7fdfb406c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183422286 0 0) 0x7fdfb4073170 con 0x7fdfb40b5e20 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf9c003520 con 0x7fdfb40b5e20 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.607+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114866704 0 0) 0x7fdfb4072e90 con 0x7fdfb406b9a0 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.608+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb4073170 con 0x7fdfb406b9a0 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.608+0000 7fdfb2d9d640 1 -- 172.21.15.146:0/3169696807 >> v1:172.21.15.146:6789/0 conn(0x7fdfb40b5e20 legacy=0x7fdfb4072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.608+0000 7fdfb2d9d640 1 -- 172.21.15.146:0/3169696807 >> v1:172.21.15.72:6789/0 conn(0x7fdfb406b9a0 legacy=0x7fdfb40ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.610 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.608+0000 7fdfb2d9d640 1 -- 172.21.15.146:0/3169696807 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb4073450 con 0x7fdfb406bd80 2024-03-06T07:21:26.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.608+0000 7fdfb2d9d640 1 --2- 172.21.15.146:0/3169696807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb406bd80 0x7fdfb406c1f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009d90 tx=0x7fdfac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48f82650143e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.608+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac039070 con 0x7fdfb406bd80 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac02cc60 con 0x7fdfb406bd80 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb159a640 1 -- 172.21.15.146:0/3169696807 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac034660 con 0x7fdfb406bd80 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/3169696807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb406bd80 msgr2=0x7fdfb406c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 --2- 172.21.15.146:0/3169696807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb406bd80 0x7fdfb406c1f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009d90 tx=0x7fdfac02beb0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/3169696807 shutdown_connections 2024-03-06T07:21:26.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 --2- 172.21.15.146:0/3169696807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb406bd80 0x7fdfb406c1f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/3169696807 >> 172.21.15.146:0/3169696807 conn(0x7fdfb40b4fe0 msgr2=0x7fdfb40b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/3169696807 shutdown_connections 2024-03-06T07:21:26.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.609+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/3169696807 wait complete. 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.610+0000 7fdfb8cb1640 1 Processor -- start 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.610+0000 7fdfb8cb1640 1 -- start start 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb8cb1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfb406b9a0 0x7fdfb414f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb8cb1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 0x7fdfb415aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb8cb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb40b5e20 0x7fdfb40ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb8cb1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb406c730 con 0x7fdfb40b5e20 2024-03-06T07:21:26.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb8cb1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40cfed0 con 0x7fdfb406bd80 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb8cb1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb40d0640 con 0x7fdfb406b9a0 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb1d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 0x7fdfb415aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb259c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfb406b9a0 0x7fdfb414f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb1d9b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 0x7fdfb415aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34070/0 (socket says 172.21.15.146:34070) 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb1d9b640 1 -- 172.21.15.146:0/2353487167 learned_addr learned my addr 172.21.15.146:0/2353487167 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.611+0000 7fdfb2d9d640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb40b5e20 0x7fdfb40ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb1d9b640 1 -- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfb406b9a0 msgr2=0x7fdfb414f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb1d9b640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfb406b9a0 0x7fdfb414f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb1d9b640 1 -- 172.21.15.146:0/2353487167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb40b5e20 msgr2=0x7fdfb40ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb1d9b640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb40b5e20 0x7fdfb40ae1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb1d9b640 1 -- 172.21.15.146:0/2353487167 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb414f790 con 0x7fdfb406bd80 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb259c640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfb406b9a0 0x7fdfb414f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb1d9b640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 0x7fdfb415aaf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400c3e0 tx=0x7fdfa400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4017070 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfa4005ce0 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfa40072d0 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb40cf120 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb40cf6e0 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdfa401a070 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.612+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fdfa40127b0 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.613+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb406f0b0 con 0x7fdfb406bd80 2024-03-06T07:21:26.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.615+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdfb406f0b0 con 0x7fdfb406bd80 2024-03-06T07:21:26.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.688+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7cd0d2fb-bff1-41c8-b382-01b0b22d939b", "id": 1} v 0) v1 -- 0x7fdfb409e4e0 con 0x7fdfb406bd80 2024-03-06T07:21:26.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.696+0000 7fdfa2ffd640 1 -- 172.21.15.146:0/2353487167 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7cd0d2fb-bff1-41c8-b382-01b0b22d939b", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdfa4007510 con 0x7fdfb406bd80 2024-03-06T07:21:26.698 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.696+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 msgr2=0x7fdfb415aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.696+0000 7fdfb8cb1640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 0x7fdfb415aaf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdfa400c3e0 tx=0x7fdfa400c8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 shutdown_connections 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdfb40b5e20 0x7fdfb40ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdfb406bd80 0x7fdfb415aaf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 --2- 172.21.15.146:0/2353487167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdfb406b9a0 0x7fdfb414f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 >> 172.21.15.146:0/2353487167 conn(0x7fdfb40b4fe0 msgr2=0x7fdfb406e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 shutdown_connections 2024-03-06T07:21:26.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.697+0000 7fdfb8cb1640 1 -- 172.21.15.146:0/2353487167 wait complete. 2024-03-06T07:21:26.709 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new a076999a-a24b-4415-beb9-6ca9c85bbf86 2 2024-03-06T07:21:26.807 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:26.805+0000 7f1eb2821040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:26.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.808+0000 7fa25c7b6640 1 Processor -- start 2024-03-06T07:21:26.810 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.808+0000 7fa25c7b6640 1 -- start start 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.809+0000 7fa25c7b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 0x7fa2540b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.809+0000 7fa25c7b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa25406f0b0 0x7fa25406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.809+0000 7fa25c7b6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa25415c470 con 0x7fa25415fb70 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.809+0000 7fa25c7b6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2540cddc0 con 0x7fa25406f0b0 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.809+0000 7fa25c7b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25415ffd0 con 0x7fa25415c090 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.809+0000 7fa25a552640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 0x7fa2540b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25a552640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 0x7fa2540b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:34242/0 (socket says 172.21.15.146:34242) 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25a552640 1 -- 172.21.15.146:0/885678840 learned_addr learned my addr 172.21.15.146:0/885678840 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25ad53640 1 --2- 172.21.15.146:0/885678840 >> v2:172.21.15.72:3300/0 conn(0x7fa25406f0b0 0x7fa25406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.812 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25a552640 1 -- 172.21.15.146:0/885678840 >> v2:172.21.15.72:3300/0 conn(0x7fa25406f0b0 msgr2=0x7fa25406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:26.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25a552640 1 --2- 172.21.15.146:0/885678840 >> v2:172.21.15.72:3300/0 conn(0x7fa25406f0b0 0x7fa25406f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25a552640 1 -- 172.21.15.146:0/885678840 >> v1:172.21.15.72:6789/0 conn(0x7fa25415fb70 legacy=0x7fa2540cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.810+0000 7fa25a552640 1 -- 172.21.15.146:0/885678840 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2541602a0 con 0x7fa25415c090 2024-03-06T07:21:26.813 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:26.810+0000 7fb2ffbb9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25a552640 1 --2- 172.21.15.146:0/885678840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 0x7fa2540b42f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa244009870 tx=0x7fa24402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ea5b8738989907 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa259550640 1 -- 172.21.15.146:0/885678840 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244034070 con 0x7fa25415c090 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa259550640 1 -- 172.21.15.146:0/885678840 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa244030840 con 0x7fa25415c090 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa259550640 1 -- 172.21.15.146:0/885678840 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa244030c10 con 0x7fa25415c090 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25c7b6640 1 -- 172.21.15.146:0/885678840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 msgr2=0x7fa2540b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/885678840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 0x7fa2540b42f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa244009870 tx=0x7fa24402b980 comp rx=0 tx=0).stop 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25c7b6640 1 -- 172.21.15.146:0/885678840 shutdown_connections 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/885678840 >> v2:172.21.15.72:3300/0 conn(0x7fa25406f0b0 0x7fa25406f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/885678840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415c090 0x7fa2540b42f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.811+0000 7fa25c7b6640 1 -- 172.21.15.146:0/885678840 >> 172.21.15.146:0/885678840 conn(0x7fa2540b3800 msgr2=0x7fa2540b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.812+0000 7fa25c7b6640 1 -- 172.21.15.146:0/885678840 shutdown_connections 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.812+0000 7fa25c7b6640 1 -- 172.21.15.146:0/885678840 wait complete. 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 Processor -- start 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 -- start start 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa25406f0b0 0x7fa25406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa25415c090 0x7fa25415d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 0x7fa25414f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540cf2f0 con 0x7fa25415c090 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa254160860 con 0x7fa25406f0b0 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.813+0000 7fa25c7b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2540cf470 con 0x7fa25415fb70 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25a552640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa25406f0b0 0x7fa25406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 0x7fa25414f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 0x7fa25414f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:34254/0 (socket says 172.21.15.146:34254) 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 -- 172.21.15.146:0/3706323453 learned_addr learned my addr 172.21.15.146:0/3706323453 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa259d51640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa25415c090 0x7fa25415d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 -- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa25406f0b0 msgr2=0x7fa25406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa25406f0b0 0x7fa25406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 -- 172.21.15.146:0/3706323453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa25415c090 msgr2=0x7fa25415d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa25415c090 0x7fa25415d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 -- 172.21.15.146:0/3706323453 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25415d570 con 0x7fa25415fb70 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25ad53640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 0x7fa25414f5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00efc0 tx=0x7fa24c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c018020 con 0x7fa25415fb70 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa24c014480 con 0x7fa25415fb70 2024-03-06T07:21:26.819 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2540b4970 con 0x7fa25415fb70 2024-03-06T07:21:26.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c014870 con 0x7fa25415fb70 2024-03-06T07:21:26.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25415dae0 con 0x7fa25415fb70 2024-03-06T07:21:26.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa24c005d10 con 0x7fa25415fb70 2024-03-06T07:21:26.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.814+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fa24c013800 con 0x7fa25415fb70 2024-03-06T07:21:26.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.815+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25409fe60 con 0x7fa25415fb70 2024-03-06T07:21:26.820 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.817+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa24c022070 con 0x7fa25415fb70 2024-03-06T07:21:26.864 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:26.862+0000 7f1eb2821040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:26.866 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:26.864+0000 7fb2ffbb9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-06T07:21:26.892 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.890+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a076999a-a24b-4415-beb9-6ca9c85bbf86", "id": 2} v 0) v1 -- 0x7fa2540b4b40 con 0x7fa25415fb70 2024-03-06T07:21:26.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.896+0000 7fa24affd640 1 -- 172.21.15.146:0/3706323453 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a076999a-a24b-4415-beb9-6ca9c85bbf86", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa24c010040 con 0x7fa25415fb70 2024-03-06T07:21:26.898 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 msgr2=0x7fa25414f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 0x7fa25414f5d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa24c00efc0 tx=0x7fa24c00c490 comp rx=0 tx=0).stop 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 shutdown_connections 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa25415fb70 0x7fa25414f5d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa25415c090 0x7fa25415d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 --2- 172.21.15.146:0/3706323453 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa25406f0b0 0x7fa25406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 >> 172.21.15.146:0/3706323453 conn(0x7fa2540b3800 msgr2=0x7fa2540cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 shutdown_connections 2024-03-06T07:21:26.899 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:26.897+0000 7fa25c7b6640 1 -- 172.21.15.146:0/3706323453 wait complete. 2024-03-06T07:21:26.909 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 79189137-86fa-4d90-8872-0a05f646a09d 3 2024-03-06T07:21:26.911 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:26.909+0000 7fb2ffbb9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:26.912 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:26.910+0000 7f1eb2821040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-06T07:21:26.964 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:26.962+0000 7fb2ffbb9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:26.971 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:26.967+0000 7f1eb2821040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-06T07:21:27.006 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.004+0000 7fb2ffbb9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.008+0000 7f48b1039640 1 Processor -- start 2024-03-06T07:21:27.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.008+0000 7f48b1039640 1 -- start start 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48b1039640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f48ac1614d0 0x7f48ac0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48b1039640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac1618b0 con 0x7f48ac0d1910 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48b1039640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac0cf400 con 0x7f48ac0d24b0 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48b1039640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ac0cf6a0 con 0x7f48ac1614d0 2024-03-06T07:21:27.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48ab59e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f48ac0d24b0 0x7f48ac0b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53116/0 (socket says 172.21.15.146:53116) 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48ab59e640 1 -- 172.21.15.146:0/3476036362 learned_addr learned my addr 172.21.15.146:0/3476036362 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48aad9d640 1 --2- 172.21.15.146:0/3476036362 >> v2:172.21.15.72:3300/0 conn(0x7f48ac1614d0 0x7f48ac0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140955702 0 0) 0x7f48ac0cf400 con 0x7f48ac0d24b0 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4894003520 con 0x7f48ac0d24b0 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562407281 0 0) 0x7f48ac1618b0 con 0x7f48ac0d1910 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac0cf400 con 0x7f48ac0d1910 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48aad9d640 1 -- 172.21.15.146:0/3476036362 >> v1:172.21.15.146:6789/0 conn(0x7f48ac0d24b0 legacy=0x7f48ac0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48aad9d640 1 -- 172.21.15.146:0/3476036362 >> v1:172.21.15.72:6789/0 conn(0x7f48ac0d1910 legacy=0x7f48ac0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.009+0000 7f48aad9d640 1 -- 172.21.15.146:0/3476036362 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0cf980 con 0x7f48ac1614d0 2024-03-06T07:21:27.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.010+0000 7f48aad9d640 1 --2- 172.21.15.146:0/3476036362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac1614d0 0x7f48ac0d4860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f48a00097a0 tx=0x7f48a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad2e6d70d18127f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.010+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0034070 con 0x7f48ac1614d0 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.010+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a0030610 con 0x7f48ac1614d0 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.010+0000 7f48a9d9b640 1 -- 172.21.15.146:0/3476036362 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a00309e0 con 0x7f48ac1614d0 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 -- 172.21.15.146:0/3476036362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac1614d0 msgr2=0x7f48ac0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 --2- 172.21.15.146:0/3476036362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac1614d0 0x7f48ac0d4860 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f48a00097a0 tx=0x7f48a002b960 comp rx=0 tx=0).stop 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 -- 172.21.15.146:0/3476036362 shutdown_connections 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 --2- 172.21.15.146:0/3476036362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac1614d0 0x7f48ac0d4860 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 -- 172.21.15.146:0/3476036362 >> 172.21.15.146:0/3476036362 conn(0x7f48ac0d3d40 msgr2=0x7f48ac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 -- 172.21.15.146:0/3476036362 shutdown_connections 2024-03-06T07:21:27.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.011+0000 7f48b1039640 1 -- 172.21.15.146:0/3476036362 wait complete. 2024-03-06T07:21:27.014 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.012+0000 7f1eb2821040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.012+0000 7f48b1039640 1 Processor -- start 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.012+0000 7f48b1039640 1 -- start start 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48b1039640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48ac0d1910 0x7f48ac0b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48b1039640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac0d24b0 0x7f48ac152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48b1039640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 0x7f48ac1f3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48b1039640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac0b1540 con 0x7f48ac0d24b0 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48b1039640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac0b0910 con 0x7f48ac1614d0 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48b1039640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac0b16c0 con 0x7f48ac0d1910 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 0x7f48ac1f3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48aad9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48ac0d1910 0x7f48ac0b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 0x7f48ac1f3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34090/0 (socket says 172.21.15.146:34090) 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 -- 172.21.15.146:0/1860896339 learned_addr learned my addr 172.21.15.146:0/1860896339 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48aa59c640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac0d24b0 0x7f48ac152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 -- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48ac0d1910 msgr2=0x7f48ac0b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48ac0d1910 0x7f48ac0b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 -- 172.21.15.146:0/1860896339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac0d24b0 msgr2=0x7f48ac152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac0d24b0 0x7f48ac152150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48ab59e640 1 -- 172.21.15.146:0/1860896339 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0b7010 con 0x7f48ac1614d0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48aa59c640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac0d24b0 0x7f48ac152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.013+0000 7f48aad9d640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48ac0d1910 0x7f48ac0b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48ab59e640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 0x7f48ac1f3cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48a400cbc0 tx=0x7f48a400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4017070 con 0x7f48ac1614d0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a400ad60 con 0x7f48ac1614d0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac0b72f0 con 0x7f48ac1614d0 2024-03-06T07:21:27.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a4012640 con 0x7f48ac1614d0 2024-03-06T07:21:27.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac0b8260 con 0x7f48ac1614d0 2024-03-06T07:21:27.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f48a401a070 con 0x7f48ac1614d0 2024-03-06T07:21:27.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.014+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f48a4007500 con 0x7f48ac1614d0 2024-03-06T07:21:27.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.015+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac15ba50 con 0x7f48ac1614d0 2024-03-06T07:21:27.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.018+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f48a4021070 con 0x7f48ac1614d0 2024-03-06T07:21:27.093 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.091+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "79189137-86fa-4d90-8872-0a05f646a09d", "id": 3} v 0) v1 -- 0x7f48ac0b8460 con 0x7f48ac1614d0 2024-03-06T07:21:27.098 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.097+0000 7f48937fe640 1 -- 172.21.15.146:0/1860896339 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79189137-86fa-4d90-8872-0a05f646a09d", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f48a4007070 con 0x7f48ac1614d0 2024-03-06T07:21:27.099 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 msgr2=0x7f48ac1f3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 0x7f48ac1f3cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f48a400cbc0 tx=0x7f48a400a430 comp rx=0 tx=0).stop 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 shutdown_connections 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48ac1614d0 0x7f48ac1f3cc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48ac0d24b0 0x7f48ac152150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 --2- 172.21.15.146:0/1860896339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48ac0d1910 0x7f48ac0b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 >> 172.21.15.146:0/1860896339 conn(0x7f48ac0d3d40 msgr2=0x7f48ac06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 shutdown_connections 2024-03-06T07:21:27.100 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.098+0000 7f48b1039640 1 -- 172.21.15.146:0/1860896339 wait complete. 2024-03-06T07:21:27.110 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 86e81164-5d05-4c30-a056-850177548536 4 2024-03-06T07:21:27.121 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.119+0000 7fb2ffbb9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:27.138 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.136+0000 7f1eb2821040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-06T07:21:27.190 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.187+0000 7fb2ffbb9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:27.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.209+0000 7f7fa70ed640 1 Processor -- start 2024-03-06T07:21:27.210 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.209+0000 7f7fa70ed640 1 -- start start 2024-03-06T07:21:27.211 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.209+0000 7f1eb2821040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.209+0000 7f7fa70ed640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7fa00b42f0 0x7f7fa015fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.209+0000 7f7fa70ed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa015c470 con 0x7f7fa006f0b0 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7fa70ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa01602a0 con 0x7f7fa015c090 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7fa70ed640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa00cddc0 con 0x7f7fa00b42f0 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7f9ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7fa00b42f0 0x7f7fa015fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7f9ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7fa00b42f0 0x7f7fa015fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34094/0 (socket says 172.21.15.146:34094) 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7fa4e89640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7fa015c090 0x7f7fa00cebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53132/0 (socket says 172.21.15.146:53132) 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7f9ffff640 1 -- 172.21.15.146:0/1096432412 learned_addr learned my addr 172.21.15.146:0/1096432412 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7f9ffff640 1 -- 172.21.15.146:0/1096432412 >> v1:172.21.15.146:6789/0 conn(0x7f7fa015c090 legacy=0x7f7fa00cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7f9ffff640 1 -- 172.21.15.146:0/1096432412 >> v1:172.21.15.72:6789/0 conn(0x7f7fa006f0b0 legacy=0x7f7fa00cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.212 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.210+0000 7f7f9ffff640 1 -- 172.21.15.146:0/1096432412 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa0160580 con 0x7f7fa00b42f0 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7f9ffff640 1 --2- 172.21.15.146:0/1096432412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa00b42f0 0x7f7fa015fb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c009870 tx=0x7f7f8c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ceb690e72d081a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7f9f7fe640 1 -- 172.21.15.146:0/1096432412 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c02f840 con 0x7f7fa00b42f0 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7f9f7fe640 1 -- 172.21.15.146:0/1096432412 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c02f9e0 con 0x7f7fa00b42f0 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7f9f7fe640 1 -- 172.21.15.146:0/1096432412 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c02fdb0 con 0x7f7fa00b42f0 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7fa70ed640 1 -- 172.21.15.146:0/1096432412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa00b42f0 msgr2=0x7f7fa015fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7fa70ed640 1 --2- 172.21.15.146:0/1096432412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa00b42f0 0x7f7fa015fb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c009870 tx=0x7f7f8c02ba20 comp rx=0 tx=0).stop 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7fa70ed640 1 -- 172.21.15.146:0/1096432412 shutdown_connections 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7fa70ed640 1 --2- 172.21.15.146:0/1096432412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa00b42f0 0x7f7fa015fb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.211+0000 7f7fa70ed640 1 -- 172.21.15.146:0/1096432412 >> 172.21.15.146:0/1096432412 conn(0x7f7fa00b3800 msgr2=0x7f7fa00b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.212+0000 7f7fa70ed640 1 -- 172.21.15.146:0/1096432412 shutdown_connections 2024-03-06T07:21:27.214 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.212+0000 7f7fa70ed640 1 -- 172.21.15.146:0/1096432412 wait complete. 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.213+0000 7f7fa70ed640 1 Processor -- start 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.213+0000 7f7fa70ed640 1 -- start start 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.213+0000 7f7fa70ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa006f0b0 0x7f7fa00b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.213+0000 7f7fa70ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 0x7f7fa00b5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.213+0000 7f7fa70ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa015c090 0x7f7fa00b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa70ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00cf470 con 0x7f7fa006f0b0 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa70ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00cf2f0 con 0x7f7fa015c090 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa70ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00ccd70 con 0x7f7fa00b42f0 2024-03-06T07:21:27.217 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 0x7f7fa00b5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa568a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa015c090 0x7f7fa00b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 0x7f7fa00b5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:34264/0 (socket says 172.21.15.146:34264) 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 -- 172.21.15.146:0/763422282 learned_addr learned my addr 172.21.15.146:0/763422282 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa4e89640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa006f0b0 0x7f7fa00b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 -- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa015c090 msgr2=0x7f7fa00b6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa015c090 0x7f7fa00b6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 -- 172.21.15.146:0/763422282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa006f0b0 msgr2=0x7f7fa00b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa006f0b0 0x7f7fa00b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 -- 172.21.15.146:0/763422282 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00b7010 con 0x7f7fa00b42f0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa568a640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa015c090 0x7f7fa00b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9ffff640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 0x7f7fa00b5bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c000c00 tx=0x7f7f8c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c041400 con 0x7f7fa00b42f0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c046070 con 0x7f7fa00b42f0 2024-03-06T07:21:27.218 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa00b6770 con 0x7f7fa00b42f0 2024-03-06T07:21:27.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c041800 con 0x7f7fa00b42f0 2024-03-06T07:21:27.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.214+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa00b6d30 con 0x7f7fa00b42f0 2024-03-06T07:21:27.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.215+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7f8c040070 con 0x7f7fa00b42f0 2024-03-06T07:21:27.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.215+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f7f8c04bb00 con 0x7f7fa00b42f0 2024-03-06T07:21:27.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.215+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa009fe60 con 0x7f7fa00b42f0 2024-03-06T07:21:27.219 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.217+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f7f8c033030 con 0x7f7fa00b42f0 2024-03-06T07:21:27.236 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.234+0000 7fb2ffbb9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:27.259 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.257+0000 7f1eb2821040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-06T07:21:27.288 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.286+0000 7fb2ffbb9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:27.290 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.288+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "86e81164-5d05-4c30-a056-850177548536", "id": 4} v 0) v1 -- 0x7f7fa0068ff0 con 0x7f7fa00b42f0 2024-03-06T07:21:27.295 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.294+0000 7f7f9d7fa640 1 -- 172.21.15.146:0/763422282 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "86e81164-5d05-4c30-a056-850177548536", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7f8c02f8e0 con 0x7f7fa00b42f0 2024-03-06T07:21:27.295 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-03-06T07:21:27.296 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.294+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 msgr2=0x7f7fa00b5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.294+0000 7f7fa70ed640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 0x7f7fa00b5bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c000c00 tx=0x7f7f8c004730 comp rx=0 tx=0).stop 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 shutdown_connections 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa015c090 0x7f7fa00b6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa00b42f0 0x7f7fa00b5bd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 --2- 172.21.15.146:0/763422282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa006f0b0 0x7f7fa00b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 >> 172.21.15.146:0/763422282 conn(0x7f7fa00b3800 msgr2=0x7f7fa006f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 shutdown_connections 2024-03-06T07:21:27.297 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.295+0000 7f7fa70ed640 1 -- 172.21.15.146:0/763422282 wait complete. 2024-03-06T07:21:27.307 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 351b9f65-50b0-442c-a118-e53c5589de7c 5 2024-03-06T07:21:27.313 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.312+0000 7f1eb2821040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-06T07:21:27.335 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.333+0000 7fb2ffbb9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:27.363 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.362+0000 7f1eb2821040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-06T07:21:27.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.411+0000 7f9e5713e640 1 Processor -- start 2024-03-06T07:21:27.413 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.411+0000 7f9e5713e640 1 -- start start 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e5713e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9e50072d00 0x7f9e50073170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e5713e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5006ba60 con 0x7f9e5015f2a0 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e5713e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5006bd00 con 0x7f9e500b5ed0 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e5713e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e50072590 con 0x7f9e50072d00 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e4ffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9e500b5ed0 0x7f9e500b09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:53148/0 (socket says 172.21.15.146:53148) 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e556db640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9e50072d00 0x7f9e50073170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e4ffff640 1 -- 172.21.15.146:0/106506885 learned_addr learned my addr 172.21.15.146:0/106506885 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e4f7fe640 1 -- 172.21.15.146:0/106506885 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077615979 0 0) 0x7f9e5006bd00 con 0x7f9e500b5ed0 2024-03-06T07:21:27.414 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e4f7fe640 1 -- 172.21.15.146:0/106506885 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e40003520 con 0x7f9e500b5ed0 2024-03-06T07:21:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e556db640 1 -- 172.21.15.146:0/106506885 >> v1:172.21.15.146:6789/0 conn(0x7f9e500b5ed0 legacy=0x7f9e500b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e556db640 1 -- 172.21.15.146:0/106506885 >> v1:172.21.15.72:6789/0 conn(0x7f9e5015f2a0 legacy=0x7f9e500d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.415 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.412+0000 7f9e556db640 1 -- 172.21.15.146:0/106506885 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5006bfe0 con 0x7f9e50072d00 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.413+0000 7f9e556db640 1 --2- 172.21.15.146:0/106506885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e50073170 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009d90 tx=0x7f9e4802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38f23bcb6a9f2909 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.413+0000 7f9e4f7fe640 1 -- 172.21.15.146:0/106506885 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48039070 con 0x7f9e50072d00 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.413+0000 7f9e4f7fe640 1 -- 172.21.15.146:0/106506885 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4802cd10 con 0x7f9e50072d00 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.413+0000 7f9e4f7fe640 1 -- 172.21.15.146:0/106506885 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48034660 con 0x7f9e50072d00 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 -- 172.21.15.146:0/106506885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 msgr2=0x7f9e50073170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 --2- 172.21.15.146:0/106506885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e50073170 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009d90 tx=0x7f9e4802bf60 comp rx=0 tx=0).stop 2024-03-06T07:21:27.416 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 -- 172.21.15.146:0/106506885 shutdown_connections 2024-03-06T07:21:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 --2- 172.21.15.146:0/106506885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e50073170 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 -- 172.21.15.146:0/106506885 >> 172.21.15.146:0/106506885 conn(0x7f9e500b4fe0 msgr2=0x7f9e500b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 -- 172.21.15.146:0/106506885 shutdown_connections 2024-03-06T07:21:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.414+0000 7f9e5713e640 1 -- 172.21.15.146:0/106506885 wait complete. 2024-03-06T07:21:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.415+0000 7f9e5713e640 1 Processor -- start 2024-03-06T07:21:27.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.415+0000 7f9e5713e640 1 -- start start 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e5713e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e501591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e5713e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e500b5ed0 0x7f9e5009d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e5713e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e5015f2a0 0x7f9e5009d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e5713e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e500b1270 con 0x7f9e5015f2a0 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e5713e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e500b10f0 con 0x7f9e50072d00 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e5713e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e500af370 con 0x7f9e500b5ed0 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e54eda640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e501591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e4ffff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e500b5ed0 0x7f9e5009d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e54eda640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e501591d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:34128/0 (socket says 172.21.15.146:34128) 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e54eda640 1 -- 172.21.15.146:0/2344881363 learned_addr learned my addr 172.21.15.146:0/2344881363 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.416+0000 7f9e556db640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e5015f2a0 0x7f9e5009d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e54eda640 1 -- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e500b5ed0 msgr2=0x7f9e5009d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e54eda640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e500b5ed0 0x7f9e5009d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e54eda640 1 -- 172.21.15.146:0/2344881363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e5015f2a0 msgr2=0x7f9e5009d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e54eda640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e5015f2a0 0x7f9e5009d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e54eda640 1 -- 172.21.15.146:0/2344881363 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5009c400 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e4ffff640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e500b5ed0 0x7f9e5009d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e54eda640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e501591d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400c340 tx=0x7f9e4400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44017070 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e440052d0 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e44003850 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5009c740 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5009cf80 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9e4401a070 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.417+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f9e44011ae0 con 0x7f9e50072d00 2024-03-06T07:21:27.421 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.418+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e500b46d0 con 0x7f9e50072d00 2024-03-06T07:21:27.422 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.420+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f9e500b46d0 con 0x7f9e50072d00 2024-03-06T07:21:27.495 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.493+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "351b9f65-50b0-442c-a118-e53c5589de7c", "id": 5} v 0) v1 -- 0x7f9e50068ff0 con 0x7f9e50072d00 2024-03-06T07:21:27.500 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.498+0000 7f9e4d7fa640 1 -- 172.21.15.146:0/2344881363 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "351b9f65-50b0-442c-a118-e53c5589de7c", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9e44021020 con 0x7f9e50072d00 2024-03-06T07:21:27.500 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 msgr2=0x7f9e501591d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e501591d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400c340 tx=0x7f9e4400c800 comp rx=0 tx=0).stop 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 shutdown_connections 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e5015f2a0 0x7f9e5009d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e500b5ed0 0x7f9e5009d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 --2- 172.21.15.146:0/2344881363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e50072d00 0x7f9e501591d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.501 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 >> 172.21.15.146:0/2344881363 conn(0x7f9e500b4fe0 msgr2=0x7f9e500cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 shutdown_connections 2024-03-06T07:21:27.502 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.499+0000 7f9e5713e640 1 -- 172.21.15.146:0/2344881363 wait complete. 2024-03-06T07:21:27.511 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 524bb873-aab9-45af-b6b4-41fb9a95709a 6 2024-03-06T07:21:27.612 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.610+0000 7f3e3b415640 1 Processor -- start 2024-03-06T07:21:27.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 -- start start 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e3406b220 0x7f3e340b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 0x7f3e34072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e3415ecb0 0x7f3e34157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e340b3a00 con 0x7f3e3406b220 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e340b3ce0 con 0x7f3e3415ecb0 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.611+0000 7f3e3b415640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3415e4b0 con 0x7f3e3406b600 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e3415ecb0 0x7f3e34157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e389b0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 0x7f3e34072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e3415ecb0 0x7f3e34157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41900/0 (socket says 172.21.15.146:41900) 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e389b0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 0x7f3e34072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:58460/0 (socket says 172.21.15.146:58460) 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 -- 172.21.15.146:0/2694633546 learned_addr learned my addr 172.21.15.146:0/2694633546 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e391b1640 1 --2- 172.21.15.146:0/2694633546 >> v2:172.21.15.72:3300/0 conn(0x7f3e3406b220 0x7f3e340b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 -- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 msgr2=0x7f3e34072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 --2- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 0x7f3e34072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 -- 172.21.15.146:0/2694633546 >> v2:172.21.15.72:3300/0 conn(0x7f3e3406b220 msgr2=0x7f3e340b58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 --2- 172.21.15.146:0/2694633546 >> v2:172.21.15.72:3300/0 conn(0x7f3e3406b220 0x7f3e340b58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e399b2640 1 -- 172.21.15.146:0/2694633546 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3415e790 con 0x7f3e3415ecb0 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.612+0000 7f3e389b0640 1 --2- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 0x7f3e34072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.613+0000 7f3e399b2640 1 --2- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3415ecb0 0x7f3e34157e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3e3406fb00 tx=0x7f3e2c02be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4fa845c02292404 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.613+0000 7f3e2bfff640 1 -- 172.21.15.146:0/2694633546 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c02c9e0 con 0x7f3e3415ecb0 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.613+0000 7f3e2bfff640 1 -- 172.21.15.146:0/2694633546 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c02cb80 con 0x7f3e3415ecb0 2024-03-06T07:21:27.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.613+0000 7f3e2bfff640 1 -- 172.21.15.146:0/2694633546 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0343f0 con 0x7f3e3415ecb0 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 -- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3415ecb0 msgr2=0x7f3e34157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 --2- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3415ecb0 0x7f3e34157e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3e3406fb00 tx=0x7f3e2c02be40 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 -- 172.21.15.146:0/2694633546 shutdown_connections 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 --2- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3415ecb0 0x7f3e34157e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 --2- 172.21.15.146:0/2694633546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3406b600 0x7f3e34072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 --2- 172.21.15.146:0/2694633546 >> v2:172.21.15.72:3300/0 conn(0x7f3e3406b220 0x7f3e340b58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 -- 172.21.15.146:0/2694633546 >> 172.21.15.146:0/2694633546 conn(0x7f3e3406c0c0 msgr2=0x7f3e3406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 -- 172.21.15.146:0/2694633546 shutdown_connections 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.614+0000 7f3e3b415640 1 -- 172.21.15.146:0/2694633546 wait complete. 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.615+0000 7f3e3b415640 1 Processor -- start 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.615+0000 7f3e3b415640 1 -- start start 2024-03-06T07:21:27.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e3b415640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e3406b220 0x7f3e34159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e3b415640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 0x7f3e341525b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e3b415640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3415ecb0 0x7f3e3415b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e3b415640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e340b4140 con 0x7f3e3406b220 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e3b415640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e340b3fc0 con 0x7f3e3406b600 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e3b415640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3415f120 con 0x7f3e3415ecb0 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 0x7f3e341525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 0x7f3e341525b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41912/0 (socket says 172.21.15.146:41912) 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 -- 172.21.15.146:0/454545508 learned_addr learned my addr 172.21.15.146:0/454545508 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e399b2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3415ecb0 0x7f3e3415b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e391b1640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e3406b220 0x7f3e34159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 -- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3415ecb0 msgr2=0x7f3e3415b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3415ecb0 0x7f3e3415b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 -- 172.21.15.146:0/454545508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e3406b220 msgr2=0x7f3e34159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e3406b220 0x7f3e34159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 -- 172.21.15.146:0/454545508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e34152e50 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e391b1640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e3406b220 0x7f3e34159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e399b2640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3415ecb0 0x7f3e3415b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.616+0000 7f3e389b0640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 0x7f3e341525b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000ebe0 tx=0x7f3e3000c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30017070 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3000cc30 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3415b090 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30013470 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e341e8860 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3e3001a070 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.617+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f3e3001ca90 con 0x7f3e3406b600 2024-03-06T07:21:27.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.618+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e340733b0 con 0x7f3e3406b600 2024-03-06T07:21:27.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.620+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3e34152e50 con 0x7f3e3406b600 2024-03-06T07:21:27.694 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.693+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "524bb873-aab9-45af-b6b4-41fb9a95709a", "id": 6} v 0) v1 -- 0x7f3e34068180 con 0x7f3e3406b600 2024-03-06T07:21:27.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.698+0000 7f3e29ffb640 1 -- 172.21.15.146:0/454545508 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "524bb873-aab9-45af-b6b4-41fb9a95709a", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e3001c4a0 con 0x7f3e3406b600 2024-03-06T07:21:27.700 INFO:teuthology.orchestra.run.smithi146.stdout:6 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 msgr2=0x7f3e341525b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 0x7f3e341525b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000ebe0 tx=0x7f3e3000c520 comp rx=0 tx=0).stop 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 shutdown_connections 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e3415ecb0 0x7f3e3415b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e3406b600 0x7f3e341525b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 --2- 172.21.15.146:0/454545508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e3406b220 0x7f3e34159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 >> 172.21.15.146:0/454545508 conn(0x7f3e3406c0c0 msgr2=0x7f3e340af020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 shutdown_connections 2024-03-06T07:21:27.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.699+0000 7f3e3b415640 1 -- 172.21.15.146:0/454545508 wait complete. 2024-03-06T07:21:27.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new dbeeecd3-529d-4582-9aa7-8e26cbf8683a 7 2024-03-06T07:21:27.712 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.710+0000 7fb2ffbb9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:27.764 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.762+0000 7f1eb2821040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-06T07:21:27.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.811+0000 7f6d88264640 1 Processor -- start 2024-03-06T07:21:27.813 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d88264640 1 -- start start 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d88264640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d8015aec0 0x7f6d800ad720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d88264640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6d800b0930 0x7f6d8006b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d88264640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d800b0f70 con 0x7f6d8006c5b0 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d88264640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8015c520 con 0x7f6d800b0930 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d88264640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d800adc60 con 0x7f6d8015aec0 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d86000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d8015aec0 0x7f6d800ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d857ff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f6d8006c5b0 0x7f6d800b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:41988/0 (socket says 172.21.15.146:41988) 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d86000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d8015aec0 0x7f6d800ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41926/0 (socket says 172.21.15.146:41926) 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.812+0000 7f6d857ff640 1 -- 172.21.15.146:0/2513717071 learned_addr learned my addr 172.21.15.146:0/2513717071 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-03-06T07:21:27.815 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.813+0000 7f6d86801640 1 --2- 172.21.15.146:0/2513717071 >> v2:172.21.15.72:3300/0 conn(0x7f6d800b0930 0x7f6d8006b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.813+0000 7f6d86000640 1 -- 172.21.15.146:0/2513717071 >> v2:172.21.15.72:3300/0 conn(0x7f6d800b0930 msgr2=0x7f6d8006b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:27.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.813+0000 7f6d86000640 1 --2- 172.21.15.146:0/2513717071 >> v2:172.21.15.72:3300/0 conn(0x7f6d800b0930 0x7f6d8006b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.813+0000 7f6d86000640 1 -- 172.21.15.146:0/2513717071 >> v1:172.21.15.146:6789/0 conn(0x7f6d8006c5b0 legacy=0x7f6d800b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.813+0000 7f6d84ffe640 1 -- 172.21.15.146:0/2513717071 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592829947 0 0) 0x7f6d800b0f70 con 0x7f6d8006c5b0 2024-03-06T07:21:27.816 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.813+0000 7f6d86000640 1 -- 172.21.15.146:0/2513717071 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8015bf60 con 0x7f6d8015aec0 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.814+0000 7f6d86000640 1 --2- 172.21.15.146:0/2513717071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d800ad720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6d80071fa0 tx=0x7f6d7002e0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ce5c0330bfd057 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.814+0000 7f6d84ffe640 1 -- 172.21.15.146:0/2513717071 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7003b070 con 0x7f6d8015aec0 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.814+0000 7f6d84ffe640 1 -- 172.21.15.146:0/2513717071 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7002ee50 con 0x7f6d8015aec0 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.814+0000 7f6d84ffe640 1 -- 172.21.15.146:0/2513717071 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70036660 con 0x7f6d8015aec0 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.814+0000 7f6d88264640 1 -- 172.21.15.146:0/2513717071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 msgr2=0x7f6d800ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.814+0000 7f6d88264640 1 --2- 172.21.15.146:0/2513717071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d800ad720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6d80071fa0 tx=0x7f6d7002e0a0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.815+0000 7f6d88264640 1 -- 172.21.15.146:0/2513717071 shutdown_connections 2024-03-06T07:21:27.817 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.815+0000 7f6d88264640 1 --2- 172.21.15.146:0/2513717071 >> v2:172.21.15.72:3300/0 conn(0x7f6d800b0930 0x7f6d8006b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.815+0000 7f6d88264640 1 --2- 172.21.15.146:0/2513717071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d800ad720 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.815+0000 7f6d88264640 1 -- 172.21.15.146:0/2513717071 >> 172.21.15.146:0/2513717071 conn(0x7f6d800b4fe0 msgr2=0x7f6d800b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.815+0000 7f6d88264640 1 -- 172.21.15.146:0/2513717071 shutdown_connections 2024-03-06T07:21:27.818 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.815+0000 7f6d88264640 1 -- 172.21.15.146:0/2513717071 wait complete. 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.816+0000 7f6d88264640 1 Processor -- start 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.816+0000 7f6d88264640 1 -- start start 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d8006c5b0 0x7f6d80151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d800b0930 0x7f6d8006e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d80155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b5db0 con 0x7f6d8006c5b0 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8006b150 con 0x7f6d8015aec0 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d800b5f30 con 0x7f6d800b0930 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d857ff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d800b0930 0x7f6d8006e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86801640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d80155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d8006c5b0 0x7f6d80151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:27.821 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86801640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d80155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:41928/0 (socket says 172.21.15.146:41928) 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86801640 1 -- 172.21.15.146:0/3783083873 learned_addr learned my addr 172.21.15.146:0/3783083873 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 -- 172.21.15.146:0/3783083873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d800b0930 msgr2=0x7f6d8006e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d800b0930 0x7f6d8006e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 -- 172.21.15.146:0/3783083873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 msgr2=0x7f6d80155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d80155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 -- 172.21.15.146:0/3783083873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d801efd60 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d86000640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d8006c5b0 0x7f6d80151d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6d70035000 tx=0x7f6d700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70046070 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70004030 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7003b040 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d801f0040 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.817+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d801f0600 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.818+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6d70049070 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.818+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f6d70003ca0 con 0x7f6d8006c5b0 2024-03-06T07:21:27.822 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.818+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d800af8b0 con 0x7f6d8006c5b0 2024-03-06T07:21:27.823 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.821+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6d800af8b0 con 0x7f6d8006c5b0 2024-03-06T07:21:27.848 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.846+0000 7fb2ffbb9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:27.895 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.893+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dbeeecd3-529d-4582-9aa7-8e26cbf8683a", "id": 7} v 0) v1 -- 0x7f6d8009e4e0 con 0x7f6d8006c5b0 2024-03-06T07:21:27.896 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.895+0000 7f6d767fc640 1 -- 172.21.15.146:0/3783083873 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dbeeecd3-529d-4582-9aa7-8e26cbf8683a", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6d70004400 con 0x7f6d8006c5b0 2024-03-06T07:21:27.897 INFO:teuthology.orchestra.run.smithi146.stdout:7 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d8006c5b0 msgr2=0x7f6d80151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d8006c5b0 0x7f6d80151d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6d70035000 tx=0x7f6d700046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 shutdown_connections 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d8015aec0 0x7f6d80155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d800b0930 0x7f6d8006e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 --2- 172.21.15.146:0/3783083873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d8006c5b0 0x7f6d80151d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 >> 172.21.15.146:0/3783083873 conn(0x7f6d800b4fe0 msgr2=0x7f6d800d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 shutdown_connections 2024-03-06T07:21:27.898 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:21:27.896+0000 7f6d88264640 1 -- 172.21.15.146:0/3783083873 wait complete. 2024-03-06T07:21:27.906 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:27.904+0000 7f1eb2821040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-06T07:21:27.908 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-06T07:21:27.909 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-06T07:21:27.911 INFO:tasks.ceph.osd.0:Started 2024-03-06T07:21:27.911 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-06T07:21:27.911 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-06T07:21:27.913 INFO:tasks.ceph.osd.1:Started 2024-03-06T07:21:27.913 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-06T07:21:27.913 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-06T07:21:27.915 INFO:tasks.ceph.osd.2:Started 2024-03-06T07:21:27.915 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-06T07:21:27.915 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-06T07:21:27.918 INFO:tasks.ceph.osd.3:Started 2024-03-06T07:21:27.918 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-06T07:21:27.918 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-06T07:21:27.920 INFO:tasks.ceph.osd.4:Started 2024-03-06T07:21:27.920 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-06T07:21:27.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-06T07:21:27.922 INFO:tasks.ceph.osd.5:Started 2024-03-06T07:21:27.923 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-06T07:21:27.923 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-06T07:21:27.924 INFO:tasks.ceph.osd.6:Started 2024-03-06T07:21:27.925 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-06T07:21:27.925 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-06T07:21:27.927 INFO:tasks.ceph.osd.7:Started 2024-03-06T07:21:27.928 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:27.989 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:27.987+0000 7fb2ffbb9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:28.041 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.039+0000 7f1eb2821040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-06T07:21:28.042 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:28.040+0000 7fb2ffbb9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:28.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.045+0000 7f2191615640 1 Processor -- start 2024-03-06T07:21:28.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.045+0000 7f2191615640 1 -- start start 2024-03-06T07:21:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2191615640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c06c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2191615640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f218c15e180 0x7f218c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2191615640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f218c0d1090 con 0x7f218c0b0aa0 2024-03-06T07:21:28.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2191615640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218c0d06c0 con 0x7f218c15e180 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2191615640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c15f380 con 0x7f218c06c680 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2183fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f218c15e180 0x7f218c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f218b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f218c0b0aa0 0x7f218c06ca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39566/0 (socket says 172.21.15.72:39566) 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f218b7fe640 1 -- 172.21.15.72:0/817811957 learned_addr learned my addr 172.21.15.72:0/817811957 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f218affd640 1 --2- 172.21.15.72:0/817811957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c06c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2183fff640 1 -- 172.21.15.72:0/817811957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 msgr2=0x7f218c06c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:28.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2183fff640 1 --2- 172.21.15.72:0/817811957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c06c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2183fff640 1 -- 172.21.15.72:0/817811957 >> v1:172.21.15.72:6789/0 conn(0x7f218c0b0aa0 legacy=0x7f218c06ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f2183fff640 1 -- 172.21.15.72:0/817811957 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0d0070 con 0x7f218c15e180 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.046+0000 7f218a7fc640 1 -- 172.21.15.72:0/817811957 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118821775 0 0) 0x7f218c0d1090 con 0x7f218c0b0aa0 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.047+0000 7f2183fff640 1 --2- 172.21.15.72:0/817811957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c0d0b50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f21740097a0 tx=0x7f217402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546ba4a0fc5d56bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.047+0000 7f218a7fc640 1 -- 172.21.15.72:0/817811957 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174035020 con 0x7f218c15e180 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.047+0000 7f218a7fc640 1 -- 172.21.15.72:0/817811957 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2174031810 con 0x7f218c15e180 2024-03-06T07:21:28.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.047+0000 7f218a7fc640 1 -- 172.21.15.72:0/817811957 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174031be0 con 0x7f218c15e180 2024-03-06T07:21:28.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 -- 172.21.15.72:0/817811957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 msgr2=0x7f218c0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 --2- 172.21.15.72:0/817811957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c0d0b50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f21740097a0 tx=0x7f217402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 -- 172.21.15.72:0/817811957 shutdown_connections 2024-03-06T07:21:28.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 --2- 172.21.15.72:0/817811957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c0d0b50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 --2- 172.21.15.72:0/817811957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c06c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 -- 172.21.15.72:0/817811957 >> 172.21.15.72:0/817811957 conn(0x7f218c0d1b20 msgr2=0x7f218c0d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 -- 172.21.15.72:0/817811957 shutdown_connections 2024-03-06T07:21:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.048+0000 7f2191615640 1 -- 172.21.15.72:0/817811957 wait complete. 2024-03-06T07:21:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.049+0000 7f2191615640 1 Processor -- start 2024-03-06T07:21:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.049+0000 7f2191615640 1 -- start start 2024-03-06T07:21:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2191615640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c0c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2191615640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f218c0b0aa0 0x7f218c15a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2191615640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c09b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2191615640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0cfe30 con 0x7f218c15e180 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2191615640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218c0cfcb0 con 0x7f218c0b0aa0 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2191615640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218c15efa0 con 0x7f218c06c680 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c09b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218affd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c0c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c09b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53450/0 (socket says 172.21.15.72:53450) 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 -- 172.21.15.72:0/1091631099 learned_addr learned my addr 172.21.15.72:0/1091631099 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f2183fff640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f218c0b0aa0 0x7f218c15a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 -- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 msgr2=0x7f218c0c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c0c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 -- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f218c0b0aa0 msgr2=0x7f218c15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f218c0b0aa0 0x7f218c15a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 -- 172.21.15.72:0/1091631099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f218c0b50c0 con 0x7f218c15e180 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.050+0000 7f218b7fe640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c09b900 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f218400c7a0 tx=0x7f218400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184018070 con 0x7f218c15e180 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218400a640 con 0x7f218c15e180 2024-03-06T07:21:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218c15a830 con 0x7f218c15e180 2024-03-06T07:21:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218400aab0 con 0x7f218c15e180 2024-03-06T07:21:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218c0c7c50 con 0x7f218c15e180 2024-03-06T07:21:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f218401b070 con 0x7f218c15e180 2024-03-06T07:21:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.051+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f21840078a0 con 0x7f218c15e180 2024-03-06T07:21:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.052+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218c04f330 con 0x7f218c15e180 2024-03-06T07:21:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.054+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f218400ac50 con 0x7f218c15e180 2024-03-06T07:21:28.088 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.086+0000 7f1eb2821040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-06T07:21:28.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.113+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f218c06e850 con 0x7f218c15e180 2024-03-06T07:21:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.114+0000 7f21837fe640 1 -- 172.21.15.72:0/1091631099 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f21840070d0 con 0x7f218c15e180 2024-03-06T07:21:28.116 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:28.116 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":10,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:27.895074+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:28.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 msgr2=0x7f218c09b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:28.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c09b900 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f218400c7a0 tx=0x7f218400cc60 comp rx=0 tx=0).stop 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 shutdown_connections 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f218c15e180 0x7f218c09b900 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f218c0b0aa0 0x7f218c15a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 --2- 172.21.15.72:0/1091631099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f218c06c680 0x7f218c0c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 >> 172.21.15.72:0/1091631099 conn(0x7f218c0d1b20 msgr2=0x7f218c0d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 shutdown_connections 2024-03-06T07:21:28.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:28.115+0000 7f2191615640 1 -- 172.21.15.72:0/1091631099 wait complete. 2024-03-06T07:21:28.127 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-06T07:21:28.127 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-06T07:21:28.158 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:28.156+0000 7fb2ffbb9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:28.200 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.198+0000 7f1eb2821040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:28.253 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:28.251+0000 7fb2ffbb9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:28.295 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.293+0000 7f1eb2821040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-06T07:21:28.408 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:28.406+0000 7fb2ffbb9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:28.449 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.448+0000 7f1eb2821040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-06T07:21:28.450 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:28.448+0000 7fb2ffbb9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:28.492 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.490+0000 7f1eb2821040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-06T07:21:28.520 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-06T07:21:28.518+0000 7f313b014640 -1 Falling back to public interface 2024-03-06T07:21:28.523 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-03-06T07:21:28.521+0000 7f16cb80d640 -1 Falling back to public interface 2024-03-06T07:21:28.526 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-06T07:21:28.524+0000 7fb875c06640 -1 Falling back to public interface 2024-03-06T07:21:28.530 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-03-06T07:21:28.528+0000 7f61f5410640 -1 Falling back to public interface 2024-03-06T07:21:28.536 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-06T07:21:28.534+0000 7f65d6e0d640 -1 Falling back to public interface 2024-03-06T07:21:28.536 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-03-06T07:21:28.534+0000 7fe2e8811640 -1 Falling back to public interface 2024-03-06T07:21:28.537 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-06T07:21:28.536+0000 7f65d56ed640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6822/0: (98) Address already in use 2024-03-06T07:21:28.552 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-06T07:21:28.550+0000 7fb57b373640 -1 Falling back to public interface 2024-03-06T07:21:28.553 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-06T07:21:28.551+0000 7ffafe79e640 -1 Falling back to public interface 2024-03-06T07:21:28.610 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:28.608+0000 7fb2ffbb9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:28.652 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:28.650+0000 7f1eb2821040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-06T07:21:28.932 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:29.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.022+0000 7fbe6a393640 1 Processor -- start 2024-03-06T07:21:29.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.022+0000 7fbe6a393640 1 -- start start 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe6a393640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe640ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe6a393640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe6406e140 0x7fbe6406aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe6a393640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6406f990 con 0x7fbe640cc6f0 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe6a393640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe6406dcb0 con 0x7fbe6406e140 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe6a393640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406f3d0 con 0x7fbe640ccad0 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe640ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe640ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33038/0 (socket says 172.21.15.72:33038) 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 -- 172.21.15.72:0/1608380655 learned_addr learned my addr 172.21.15.72:0/1608380655 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:29.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe68930640 1 --2- 172.21.15.72:0/1608380655 >> v2:172.21.15.146:3300/0 conn(0x7fbe6406e140 0x7fbe6406aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.028 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 -- 172.21.15.72:0/1608380655 >> v2:172.21.15.146:3300/0 conn(0x7fbe6406e140 msgr2=0x7fbe6406aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:29.028 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 --2- 172.21.15.72:0/1608380655 >> v2:172.21.15.146:3300/0 conn(0x7fbe6406e140 0x7fbe6406aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.028 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 -- 172.21.15.72:0/1608380655 >> v1:172.21.15.146:6789/0 conn(0x7fbe640cc6f0 legacy=0x7fbe6406ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.028 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe62ffd640 1 -- 172.21.15.72:0/1608380655 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011715638 0 0) 0x7fbe6406f990 con 0x7fbe640cc6f0 2024-03-06T07:21:29.028 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe637fe640 1 -- 172.21.15.72:0/1608380655 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6406f6b0 con 0x7fbe640ccad0 2024-03-06T07:21:29.028 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.023+0000 7fbe68930640 1 --2- 172.21.15.72:0/1608380655 >> v2:172.21.15.146:3300/0 conn(0x7fbe6406e140 0x7fbe6406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:29.031 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.024+0000 7fbe637fe640 1 --2- 172.21.15.72:0/1608380655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe640ce830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbe580098a0 tx=0x7fbe5802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36d2f504cc5d1dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.031 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.024+0000 7fbe62ffd640 1 -- 172.21.15.72:0/1608380655 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5802f830 con 0x7fbe640ccad0 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.024+0000 7fbe62ffd640 1 -- 172.21.15.72:0/1608380655 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5802f9d0 con 0x7fbe640ccad0 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.024+0000 7fbe62ffd640 1 -- 172.21.15.72:0/1608380655 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5802fda0 con 0x7fbe640ccad0 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 -- 172.21.15.72:0/1608380655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 msgr2=0x7fbe640ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 --2- 172.21.15.72:0/1608380655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe640ce830 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbe580098a0 tx=0x7fbe5802ba10 comp rx=0 tx=0).stop 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 -- 172.21.15.72:0/1608380655 shutdown_connections 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 --2- 172.21.15.72:0/1608380655 >> v2:172.21.15.146:3300/0 conn(0x7fbe6406e140 0x7fbe6406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 --2- 172.21.15.72:0/1608380655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe640ce830 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 -- 172.21.15.72:0/1608380655 >> 172.21.15.72:0/1608380655 conn(0x7fbe640cd670 msgr2=0x7fbe640cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 -- 172.21.15.72:0/1608380655 shutdown_connections 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.025+0000 7fbe6a393640 1 -- 172.21.15.72:0/1608380655 wait complete. 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.026+0000 7fbe6a393640 1 Processor -- start 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.026+0000 7fbe6a393640 1 -- start start 2024-03-06T07:21:29.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe6406e140 0x7fbe6415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 0x7fbe641f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe64146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406e770 con 0x7fbe640cc6f0 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406e5f0 con 0x7fbe6406e140 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6406b530 con 0x7fbe640ccad0 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 0x7fbe641f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 0x7fbe641f6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53454/0 (socket says 172.21.15.72:53454) 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 -- 172.21.15.72:0/2696180694 learned_addr learned my addr 172.21.15.72:0/2696180694 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe68930640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe64146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe63fff640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe6406e140 0x7fbe6415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 -- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 msgr2=0x7fbe64146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe64146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 -- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe6406e140 msgr2=0x7fbe6415bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe6406e140 0x7fbe6415bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 -- 172.21.15.72:0/2696180694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe641e6800 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe637fe640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 0x7fbe641f6820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbe580090b0 tx=0x7fbe58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58047070 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58037bc0 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe641e6ae0 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.027+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004030 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.028+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641442c0 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.028+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbe5804a070 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.028+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbe58042430 con 0x7fbe640cc6f0 2024-03-06T07:21:29.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.028+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6404f330 con 0x7fbe640cc6f0 2024-03-06T07:21:29.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.030+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbe58033030 con 0x7fbe640cc6f0 2024-03-06T07:21:29.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.092+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe6406b900 con 0x7fbe640cc6f0 2024-03-06T07:21:29.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.093+0000 7fbe60ff9640 1 -- 172.21.15.72:0/2696180694 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fbe6406b900 con 0x7fbe640cc6f0 2024-03-06T07:21:29.095 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-06T07:21:29.095 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:27.895074+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.093+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 msgr2=0x7fbe641f6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.093+0000 7fbe6a393640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 0x7fbe641f6820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbe580090b0 tx=0x7fbe58004730 comp rx=0 tx=0).stop 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 shutdown_connections 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe640ccad0 0x7fbe64146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe640cc6f0 0x7fbe641f6820 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 --2- 172.21.15.72:0/2696180694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe6406e140 0x7fbe6415bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 >> 172.21.15.72:0/2696180694 conn(0x7fbe640cd670 msgr2=0x7fbe640adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 shutdown_connections 2024-03-06T07:21:29.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:29.094+0000 7fbe6a393640 1 -- 172.21.15.72:0/2696180694 wait complete. 2024-03-06T07:21:29.105 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:29.483 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:29.481+0000 7fb2ffbb9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:29.483 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:29.482+0000 7f1eb2821040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-06T07:21:29.759 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:29.757+0000 7fb2ffbb9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:29.762 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:29.761+0000 7f1eb2821040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-06T07:21:29.811 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:29.808+0000 7fb2ffbb9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:29.813 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:29.811+0000 7f1eb2821040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-06T07:21:29.928 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:29.926+0000 7fb2ffbb9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:29.929 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-03-06T07:21:29.927+0000 7f1eb2821040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-06T07:21:30.952 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:30.950+0000 7fb29f464640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-06T07:21:30.953 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-06T07:21:30.950+0000 7fb29f464640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-06T07:21:31.982 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-03-06T07:21:31.980+0000 7f16cb80d640 -1 osd.1 0 log_to_monitors true 2024-03-06T07:21:32.006 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-06T07:21:32.004+0000 7f313b014640 -1 osd.5 0 log_to_monitors true 2024-03-06T07:21:32.028 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-06T07:21:32.027+0000 7fb875c06640 -1 osd.6 0 log_to_monitors true 2024-03-06T07:21:32.037 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-03-06T07:21:32.035+0000 7f61f5410640 -1 osd.3 0 log_to_monitors true 2024-03-06T07:21:32.049 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-03-06T07:21:32.047+0000 7fe2e8811640 -1 osd.2 0 log_to_monitors true 2024-03-06T07:21:32.052 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-06T07:21:32.050+0000 7f65d6e0d640 -1 osd.0 0 log_to_monitors true 2024-03-06T07:21:32.053 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-06T07:21:32.052+0000 7ffafe79e640 -1 osd.4 0 log_to_monitors true 2024-03-06T07:21:32.064 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-06T07:21:32.063+0000 7fb57b373640 -1 osd.7 0 log_to_monitors true 2024-03-06T07:21:35.913 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:35.922 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-06T07:21:35.919+0000 7fb871379640 -1 osd.6 0 waiting for initial osdmap 2024-03-06T07:21:35.940 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-03-06T07:21:35.938+0000 7fb8699ab640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:35.996 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-06T07:21:35.994+0000 7f65d1eb1640 -1 osd.0 0 waiting for initial osdmap 2024-03-06T07:21:36.014 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-06T07:21:36.012+0000 7f65ca4e3640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.046 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-03-06T07:21:36.044+0000 7f61f04a3640 -1 osd.3 0 waiting for initial osdmap 2024-03-06T07:21:36.050 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.048+0000 7f5acb1d0640 1 Processor -- start 2024-03-06T07:21:36.051 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.048+0000 7f5acb1d0640 1 -- start start 2024-03-06T07:21:36.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5acb1d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ac40d0790 0x7f5ac40b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5acb1d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ac40ceab0 0x7f5ac40b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5acb1d0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac415c350 con 0x7f5ac415bf70 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5acb1d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac40d25c0 con 0x7f5ac40ceab0 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5acb1d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40b0f90 con 0x7f5ac40d0790 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5ac8f6c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5ac415bf70 0x7f5ac40d0080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43856/0 (socket says 172.21.15.72:43856) 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.049+0000 7f5ac8f6c640 1 -- 172.21.15.72:0/3292116798 learned_addr learned my addr 172.21.15.72:0/3292116798 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346037061 0 0) 0x7f5ac415c350 con 0x7f5ac415bf70 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4003520 con 0x7f5ac415bf70 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135244198 0 0) 0x7f5ab4003520 con 0x7f5ac415bf70 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ac40d0790 msgr2=0x7f5ac40b19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ac40d0790 0x7f5ac40b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 >> v2:172.21.15.146:3300/0 conn(0x7f5ac40ceab0 msgr2=0x7f5ac40b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 --2- 172.21.15.72:0/3292116798 >> v2:172.21.15.146:3300/0 conn(0x7f5ac40ceab0 0x7f5ac40b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40d2810 con 0x7f5ac415bf70 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f5ab8001670 con 0x7f5ac415bf70 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 >> v1:172.21.15.72:6789/0 conn(0x7f5ac415bf70 legacy=0x7f5ac40d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40039b0 0x7f5ab4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ac40d0790 0x7f5ab4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab4004d90 0x7f5ab4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40d25c0 con 0x7f5ab4004d90 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40b0f90 con 0x7f5ab40039b0 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab400bcc0 con 0x7f5ac40d0790 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac976d640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab4004d90 0x7f5ab4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac8f6c640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40039b0 0x7f5ab4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac976d640 1 -- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ac40d0790 msgr2=0x7f5ab4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac976d640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ac40d0790 0x7f5ab4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.055 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac976d640 1 -- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40039b0 msgr2=0x7f5ab4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:36.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.050+0000 7f5ac976d640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ab40039b0 0x7f5ab4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.051+0000 7f5ac976d640 1 -- 172.21.15.72:0/3292116798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4003520 con 0x7f5ab4004d90 2024-03-06T07:21:36.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.052+0000 7f5ac976d640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab4004d90 0x7f5ab4005210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009e40 tx=0x7f5ab4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af892da6700626e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.052+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc039070 con 0x7f5ab4004d90 2024-03-06T07:21:36.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.052+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc006e50 con 0x7f5ab4004d90 2024-03-06T07:21:36.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.052+0000 7f5ac37fe640 1 -- 172.21.15.72:0/3292116798 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc009a10 con 0x7f5ab4004d90 2024-03-06T07:21:36.057 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-06T07:21:36.053+0000 7ffafb749640 -1 osd.4 0 waiting for initial osdmap 2024-03-06T07:21:36.057 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5acb1d0640 1 -- 172.21.15.72:0/3292116798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab4004d90 msgr2=0x7f5ab4005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/3292116798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ab4004d90 0x7f5ab4005210 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009e40 tx=0x7f5ab4005830 comp rx=0 tx=0).stop 2024-03-06T07:21:36.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5ac976d640 1 -- 172.21.15.72:0/3292116798 reap_dead start 2024-03-06T07:21:36.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5acb1d0640 1 -- 172.21.15.72:0/3292116798 shutdown_connections 2024-03-06T07:21:36.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5acb1d0640 1 -- 172.21.15.72:0/3292116798 >> 172.21.15.72:0/3292116798 conn(0x7f5ac40d1aa0 msgr2=0x7f5ac40d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5acb1d0640 1 -- 172.21.15.72:0/3292116798 shutdown_connections 2024-03-06T07:21:36.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.053+0000 7f5acb1d0640 1 -- 172.21.15.72:0/3292116798 wait complete. 2024-03-06T07:21:36.058 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-03-06T07:21:36.054+0000 7fe2e39b0640 -1 osd.2 0 waiting for initial osdmap 2024-03-06T07:21:36.059 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.055+0000 7f5acb1d0640 1 Processor -- start 2024-03-06T07:21:36.059 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.055+0000 7f5acb1d0640 1 -- start start 2024-03-06T07:21:36.059 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f61e8ad5640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5acb1d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ab4004d90 0x7f5ac40b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5acb1d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ac414fb50 0x7f5ac40b6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5acb1d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 0x7f5ac40c7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.060 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f16c699d640 -1 osd.1 0 waiting for initial osdmap 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5ac976d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 0x7f5ac40c7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5ac976d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 0x7f5ac40c7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53444/0 (socket says 172.21.15.72:53444) 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5ac976d640 1 -- 172.21.15.72:0/2840775523 learned_addr learned my addr 172.21.15.72:0/2840775523 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40cd830 con 0x7f5ac40c7970 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40b1550 con 0x7f5ac414fb50 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.057+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac40cd9b0 con 0x7f5ab4004d90 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac8f6c640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ab4004d90 0x7f5ac40b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac976d640 1 -- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ab4004d90 msgr2=0x7f5ac40b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac976d640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ab4004d90 0x7f5ac40b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac976d640 1 -- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ac414fb50 msgr2=0x7f5ac40b6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac976d640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ac414fb50 0x7f5ac40b6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac976d640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac40c5e50 con 0x7f5ac40c7970 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.058+0000 7f5ac976d640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 0x7f5ac40c7e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009f60 tx=0x7f5abc004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.059+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc039070 con 0x7f5ac40c7970 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.059+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac40c6130 con 0x7f5ac40c7970 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.059+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc046070 con 0x7f5ac40c7970 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.059+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc009a10 con 0x7f5ac40c7970 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.059+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac40b49a0 con 0x7f5ac40c7970 2024-03-06T07:21:36.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.060+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac404f330 con 0x7f5ac40c7970 2024-03-06T07:21:36.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.061+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) v1 ==== 89724+0+0 (secure 0 0 0) 0x7f5abc0046b0 con 0x7f5ac40c7970 2024-03-06T07:21:36.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.061+0000 7f5ac17fa640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5aa806b440 0x7f5aa806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:36.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.061+0000 7f5ac8f6c640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5aa806b440 0x7f5aa806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:36.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.061+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f5abc0ab980 con 0x7f5ac40c7970 2024-03-06T07:21:36.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.062+0000 7f5ac8f6c640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5aa806b440 0x7f5aa806d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000fb0 tx=0x7f5ab8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:36.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.066+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5abc008030 con 0x7f5ac40c7970 2024-03-06T07:21:36.070 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-03-06T07:21:36.067+0000 7ffaf2543640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.070 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-03-06T07:21:36.069+0000 7fe2dbfe2640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.074 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-06T07:21:36.071+0000 7f313613e640 -1 osd.5 0 waiting for initial osdmap 2024-03-06T07:21:36.075 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-03-06T07:21:36.073+0000 7f16befcf640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.085 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-03-06T07:21:36.083+0000 7f312e770640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.092 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-06T07:21:36.089+0000 7fb57831e640 -1 osd.7 0 waiting for initial osdmap 2024-03-06T07:21:36.101 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-03-06T07:21:36.099+0000 7fb56f118640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-06T07:21:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.241+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ac41e3280 con 0x7f5ac40c7970 2024-03-06T07:21:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.242+0000 7f5ac17fa640 1 -- 172.21.15.72:0/2840775523 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f5abc07b780 con 0x7f5ac40c7970 2024-03-06T07:21:36.244 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-06T07:21:36.244 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":12,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:33.956717+0000","last_up_change":"0.000000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5aa806b440 msgr2=0x7f5aa806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5aa806b440 0x7f5aa806d900 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8000fb0 tx=0x7f5ab8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 msgr2=0x7f5ac40c7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 0x7f5ac40c7e20 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009f60 tx=0x7f5abc004b50 comp rx=0 tx=0).stop 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 shutdown_connections 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5aa806b440 0x7f5aa806d900 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ac40c7970 0x7f5ac40c7e20 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ac414fb50 0x7f5ac40b6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 --2- 172.21.15.72:0/2840775523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ab4004d90 0x7f5ac40b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 >> 172.21.15.72:0/2840775523 conn(0x7f5ac40d1aa0 msgr2=0x7f5ac41e3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 shutdown_connections 2024-03-06T07:21:36.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:36.244+0000 7f5acb1d0640 1 -- 172.21.15.72:0/2840775523 wait complete. 2024-03-06T07:21:36.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-06T07:21:43.063 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:43.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.160+0000 7f1bb5d32640 1 Processor -- start 2024-03-06T07:21:43.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.160+0000 7f1bb5d32640 1 -- start start 2024-03-06T07:21:43.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1bb5d32640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1bb015bc10 0x7f1bb006c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1bb5d32640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb00b1be0 con 0x7f1bb006c640 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1bb5d32640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bb00b1ec0 con 0x7f1bb00d07f0 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1bb5d32640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bb00b2510 con 0x7f1bb015bc10 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1baeffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1bb006c640 0x7f1bb00d00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41746/0 (socket says 172.21.15.72:41746) 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1baeffd640 1 -- 172.21.15.72:0/2796006026 learned_addr learned my addr 172.21.15.72:0/2796006026 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826880206 0 0) 0x7f1bb00b1be0 con 0x7f1bb006c640 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b98003520 con 0x7f1bb006c640 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.161+0000 7f1baf7fe640 1 --2- 172.21.15.72:0/2796006026 >> v2:172.21.15.146:3300/0 conn(0x7f1bb015bc10 0x7f1bb006c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591551463 0 0) 0x7f1b98003520 con 0x7f1bb006c640 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 >> v2:172.21.15.146:3300/0 conn(0x7f1bb015bc10 msgr2=0x7f1bb006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 --2- 172.21.15.72:0/2796006026 >> v2:172.21.15.146:3300/0 conn(0x7f1bb015bc10 0x7f1bb006c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 >> v1:172.21.15.146:6789/0 conn(0x7f1bb00d07f0 legacy=0x7f1bb00b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb00b21a0 con 0x7f1bb006c640 2024-03-06T07:21:43.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1ba4001670 con 0x7f1bb006c640 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 >> v1:172.21.15.72:6789/0 conn(0x7f1bb006c640 legacy=0x7f1bb00d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b980039b0 0x7f1b98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b98004800 0x7f1b98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b980051c0 0x7f1b9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb00b2510 con 0x7f1b98004800 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9800c060 con 0x7f1b980051c0 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9800c260 con 0x7f1b980039b0 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baeffd640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b98004800 0x7f1b98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baf7fe640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b980039b0 0x7f1b98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baffff640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b980051c0 0x7f1b9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baeffd640 1 -- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b980039b0 msgr2=0x7f1b98003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baeffd640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b980039b0 0x7f1b98003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baeffd640 1 -- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b980051c0 msgr2=0x7f1b9800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baeffd640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b980051c0 0x7f1b9800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baffff640 1 -- 172.21.15.72:0/2796006026 reap_dead start 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.162+0000 7f1baeffd640 1 -- 172.21.15.72:0/2796006026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98003520 con 0x7f1b98004800 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.163+0000 7f1baeffd640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b98004800 0x7f1b98004c80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1bb006ca20 tx=0x7f1b98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d885016aa62f989 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.163+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4007d30 con 0x7f1b98004800 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.163+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba4030070 con 0x7f1b98004800 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.163+0000 7f1bae7fc640 1 -- 172.21.15.72:0/2796006026 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4005d70 con 0x7f1b98004800 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 -- 172.21.15.72:0/2796006026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b98004800 msgr2=0x7f1b98004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b98004800 0x7f1b98004c80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1bb006ca20 tx=0x7f1b98005830 comp rx=0 tx=0).stop 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 -- 172.21.15.72:0/2796006026 shutdown_connections 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/2796006026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b98004800 0x7f1b98004c80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 -- 172.21.15.72:0/2796006026 >> 172.21.15.72:0/2796006026 conn(0x7f1bb009a9e0 msgr2=0x7f1bb009adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 -- 172.21.15.72:0/2796006026 shutdown_connections 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.164+0000 7f1bb5d32640 1 -- 172.21.15.72:0/2796006026 wait complete. 2024-03-06T07:21:43.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.165+0000 7f1bb5d32640 1 Processor -- start 2024-03-06T07:21:43.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.165+0000 7f1bb5d32640 1 -- start start 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1bb5d32640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b98004800 0x7f1bb015c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1bb5d32640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb01551b0 0x7f1bb014b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1bb5d32640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 0x7f1bb00afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1bb5d32640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb006c490 con 0x7f1bb00c5650 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1bb5d32640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb006c310 con 0x7f1b98004800 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1bb5d32640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb00d00e0 con 0x7f1bb01551b0 2024-03-06T07:21:43.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 0x7f1bb00afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 0x7f1bb00afc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34690/0 (socket says 172.21.15.72:34690) 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 -- 172.21.15.72:0/1864377636 learned_addr learned my addr 172.21.15.72:0/1864377636 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baf7fe640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b98004800 0x7f1bb015c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baeffd640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb01551b0 0x7f1bb014b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 -- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb01551b0 msgr2=0x7f1bb014b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb01551b0 0x7f1bb014b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 -- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b98004800 msgr2=0x7f1bb015c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b98004800 0x7f1bb015c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 -- 172.21.15.72:0/1864377636 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb015c590 con 0x7f1bb00c5650 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baf7fe640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b98004800 0x7f1bb015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.166+0000 7f1baffff640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 0x7f1bb00afc40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0072870 tx=0x7f1ba800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.167+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8003b80 con 0x7f1bb00c5650 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.167+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ba8018070 con 0x7f1bb00c5650 2024-03-06T07:21:43.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.167+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb015c830 con 0x7f1bb00c5650 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.167+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80050d0 con 0x7f1bb00c5650 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.167+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb015cdf0 con 0x7f1bb00c5650 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.168+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb004f330 con 0x7f1bb00c5650 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.168+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ba8005270 con 0x7f1bb00c5650 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.168+0000 7f1b97fff640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b9c06b4d0 0x7f1b9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.168+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ba8089c20 con 0x7f1bb00c5650 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.168+0000 7f1baf7fe640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b9c06b4d0 0x7f1b9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.169+0000 7f1baf7fe640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b9c06b4d0 0x7f1b9c06d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009b30 tx=0x7f1ba0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.173+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ba8058df0 con 0x7f1bb00c5650 2024-03-06T07:21:43.348 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.346+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bb00b1820 con 0x7f1bb00c5650 2024-03-06T07:21:43.348 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.346+0000 7f1b97fff640 1 -- 172.21.15.72:0/1864377636 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7f1ba801be40 con 0x7f1bb00c5650 2024-03-06T07:21:43.349 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-06T07:21:43.349 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":17,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:41.025880+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.348+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b9c06b4d0 msgr2=0x7f1b9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.348+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b9c06b4d0 0x7f1b9c06d990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0009b30 tx=0x7f1ba0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.348+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 msgr2=0x7f1bb00afc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.348+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 0x7f1bb00afc40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0072870 tx=0x7f1ba800fc00 comp rx=0 tx=0).stop 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 shutdown_connections 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b9c06b4d0 0x7f1b9c06d990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bb00c5650 0x7f1bb00afc40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bb01551b0 0x7f1bb014b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 --2- 172.21.15.72:0/1864377636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b98004800 0x7f1bb015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 >> 172.21.15.72:0/1864377636 conn(0x7f1bb009a9e0 msgr2=0x7f1bb006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 shutdown_connections 2024-03-06T07:21:43.352 INFO:teuthology.misc.health.smithi072.stderr:2024-03-06T07:21:43.349+0000 7f1bb5d32640 1 -- 172.21.15.72:0/1864377636 wait complete. 2024-03-06T07:21:43.360 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-06T07:21:43.360 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-06T07:21:43.361 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:21:43.361 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:21:43.363 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:21:43.363 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:21:43.363 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:21:43.368 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:21:43.368 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:21:43.369 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:21:43.371 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:21:43.371 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:21:43.371 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:21:43.373 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:21:43.373 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-06T07:21:43.376 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:43.438 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:21:43.439 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:21:43.439 INFO:tasks.ceph.mds.a.smithi072.stderr:2024-03-06T07:21:43.437+0000 7f8dcf3c9640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-06T07:21:43.456 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:21:43.456 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:21:43.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 Processor -- start 2024-03-06T07:21:43.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 -- start start 2024-03-06T07:21:43.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f26e01621c0 0x7f26e00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f26e00d4920 0x7f26e00d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e015dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00cf690 con 0x7f26e00d4920 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00ceac0 con 0x7f26e01621c0 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26e51c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00ced60 con 0x7f26e00d2bf0 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.480+0000 7f26de59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f26e00d4920 0x7f26e00d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f26e00d4920 0x7f26e00d26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34734/0 (socket says 172.21.15.72:34734) 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 -- 172.21.15.72:0/2035727898 learned_addr learned my addr 172.21.15.72:0/2035727898 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26ded9d640 1 --2- 172.21.15.72:0/2035727898 >> v2:172.21.15.146:3300/0 conn(0x7f26e01621c0 0x7f26e00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26df59e640 1 --2- 172.21.15.72:0/2035727898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e015dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 -- 172.21.15.72:0/2035727898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 msgr2=0x7f26e015dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 --2- 172.21.15.72:0/2035727898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e015dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 -- 172.21.15.72:0/2035727898 >> v2:172.21.15.146:3300/0 conn(0x7f26e01621c0 msgr2=0x7f26e00cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 --2- 172.21.15.72:0/2035727898 >> v2:172.21.15.146:3300/0 conn(0x7f26e01621c0 0x7f26e00cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 -- 172.21.15.72:0/2035727898 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00cf040 con 0x7f26e00d4920 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26ded9d640 1 --2- 172.21.15.72:0/2035727898 >> v2:172.21.15.146:3300/0 conn(0x7f26e01621c0 0x7f26e00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.481+0000 7f26de59c640 1 --2- 172.21.15.72:0/2035727898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e00d4920 0x7f26e00d26b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f26e00d3ce0 tx=0x7f26c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da15dd903ac81f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26ddd9b640 1 -- 172.21.15.72:0/2035727898 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8035020 con 0x7f26e00d4920 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26ddd9b640 1 -- 172.21.15.72:0/2035727898 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c8031870 con 0x7f26e00d4920 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26ddd9b640 1 -- 172.21.15.72:0/2035727898 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8031c40 con 0x7f26e00d4920 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 -- 172.21.15.72:0/2035727898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e00d4920 msgr2=0x7f26e00d26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 --2- 172.21.15.72:0/2035727898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e00d4920 0x7f26e00d26b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f26e00d3ce0 tx=0x7f26c802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 -- 172.21.15.72:0/2035727898 shutdown_connections 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 --2- 172.21.15.72:0/2035727898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e015dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 --2- 172.21.15.72:0/2035727898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e00d4920 0x7f26e00d26b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 --2- 172.21.15.72:0/2035727898 >> v2:172.21.15.146:3300/0 conn(0x7f26e01621c0 0x7f26e00cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 -- 172.21.15.72:0/2035727898 >> 172.21.15.72:0/2035727898 conn(0x7f26e00d3d10 msgr2=0x7f26e00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 -- 172.21.15.72:0/2035727898 shutdown_connections 2024-03-06T07:21:43.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.482+0000 7f26e51c9640 1 -- 172.21.15.72:0/2035727898 wait complete. 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.483+0000 7f26e51c9640 1 Processor -- start 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.483+0000 7f26e51c9640 1 -- start start 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26e51c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e014c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26e51c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e00d4920 0x7f26e0160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26e51c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 0x7f26e015db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26e51c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0162720 con 0x7f26e01621c0 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26e51c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01625a0 con 0x7f26e00d4920 2024-03-06T07:21:43.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26e51c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e01628a0 con 0x7f26e00d2bf0 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26df59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 0x7f26e015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26df59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 0x7f26e015db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34750/0 (socket says 172.21.15.72:34750) 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26df59e640 1 -- 172.21.15.72:0/3631895708 learned_addr learned my addr 172.21.15.72:0/3631895708 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.484+0000 7f26de59c640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e00d4920 0x7f26e0160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26ded9d640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e014c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26df59e640 1 -- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 msgr2=0x7f26e014c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26df59e640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e014c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26df59e640 1 -- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e00d4920 msgr2=0x7f26e0160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26df59e640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e00d4920 0x7f26e0160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26df59e640 1 -- 172.21.15.72:0/3631895708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00ccab0 con 0x7f26e01621c0 2024-03-06T07:21:43.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26ded9d640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e014c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26df59e640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 0x7f26e015db10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26d8006b10 tx=0x7f26d800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8015070 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d8004220 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e00ccd90 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d8003610 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.485+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e00cd870 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.486+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e004f330 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.487+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26d80037b0 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.487+0000 7f26d77fe640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26c006b4d0 0x7f26c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.487+0000 7f26ded9d640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26c006b4d0 0x7f26c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.487+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f26d8089bd0 con 0x7f26e01621c0 2024-03-06T07:21:43.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.487+0000 7f26ded9d640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26c006b4d0 0x7f26c006d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f26d0007f40 tx=0x7f26d0007490 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.491+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26d8058da0 con 0x7f26e01621c0 2024-03-06T07:21:43.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.664+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e00b50b0 con 0x7f26e01621c0 2024-03-06T07:21:43.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.665+0000 7f26d77fe640 1 -- 172.21.15.72:0/3631895708 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7f26d80586a0 con 0x7f26e01621c0 2024-03-06T07:21:43.667 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:43.667 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:41.025880+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26c006b4d0 msgr2=0x7f26c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26c006b4d0 0x7f26c006d990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f26d0007f40 tx=0x7f26d0007490 comp rx=0 tx=0).stop 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 msgr2=0x7f26e015db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 0x7f26e015db10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f26d8006b10 tx=0x7f26d800ca20 comp rx=0 tx=0).stop 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 shutdown_connections 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26c006b4d0 0x7f26c006d990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f26e01621c0 0x7f26e015db10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f26e00d4920 0x7f26e0160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 --2- 172.21.15.72:0/3631895708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26e00d2bf0 0x7f26e014c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 >> 172.21.15.72:0/3631895708 conn(0x7f26e00d3d10 msgr2=0x7f26e00b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 shutdown_connections 2024-03-06T07:21:43.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.667+0000 7f26e51c9640 1 -- 172.21.15.72:0/3631895708 wait complete. 2024-03-06T07:21:43.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:43.681 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.782+0000 7f4067cc9640 1 Processor -- start 2024-03-06T07:21:43.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.782+0000 7f4067cc9640 1 -- start start 2024-03-06T07:21:43.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4067cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 0x7f40600aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4067cc9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40600d01a0 0x7f40600b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4067cc9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4060158ac0 con 0x7f40600aef80 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4067cc9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406015b2a0 con 0x7f40600d01a0 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4067cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406015b540 con 0x7f40601586e0 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4065264640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f40600aef80 0x7f40600cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41748/0 (socket says 172.21.15.72:41748) 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4065264640 1 -- 172.21.15.72:0/3755193703 learned_addr learned my addr 172.21.15.72:0/3755193703 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677932454 0 0) 0x7f4060158ac0 con 0x7f40600aef80 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f404c003520 con 0x7f40600aef80 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4066266640 1 --2- 172.21.15.72:0/3755193703 >> v2:172.21.15.146:3300/0 conn(0x7f40600d01a0 0x7f40600b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4065a65640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 0x7f40600aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013273010 0 0) 0x7f404c003520 con 0x7f40600aef80 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 msgr2=0x7f40600aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 0x7f40600aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 >> v2:172.21.15.146:3300/0 conn(0x7f40600d01a0 msgr2=0x7f40600b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 --2- 172.21.15.72:0/3755193703 >> v2:172.21.15.146:3300/0 conn(0x7f40600d01a0 0x7f40600b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406015b820 con 0x7f40600aef80 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f4050001670 con 0x7f40600aef80 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 >> v1:172.21.15.72:6789/0 conn(0x7f40600aef80 legacy=0x7f40600cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f404c0039b0 0x7f404c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 0x7f404c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404c004d40 0x7f404c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406015b2a0 con 0x7f404c0039b0 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406015b540 con 0x7f404c004d40 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.783+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404c00ba70 con 0x7f40601586e0 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4066266640 1 --2- 172.21.15.72:0/3755193703 >> v2:172.21.15.146:3300/0 conn(0x7f40600d01a0 0x7f40600b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065a65640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f404c0039b0 0x7f404c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065264640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 0x7f404c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4066266640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404c004d40 0x7f404c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065a65640 1 -- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 msgr2=0x7f404c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065a65640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40601586e0 0x7f404c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065a65640 1 -- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404c004d40 msgr2=0x7f404c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065a65640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f404c004d40 0x7f404c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.784+0000 7f4065a65640 1 -- 172.21.15.72:0/3755193703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404c003520 con 0x7f404c0039b0 2024-03-06T07:21:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4065a65640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f404c0039b0 0x7f404c003dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ba60 tx=0x7f404c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d56b862fb6ca76c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c036020 con 0x7f404c0039b0 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c032630 con 0x7f404c0039b0 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4064a63640 1 -- 172.21.15.72:0/3755193703 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c00b600 con 0x7f404c0039b0 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4067cc9640 1 -- 172.21.15.72:0/3755193703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f404c0039b0 msgr2=0x7f404c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4067cc9640 1 --2- 172.21.15.72:0/3755193703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f404c0039b0 0x7f404c003dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ba60 tx=0x7f404c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4066266640 1 -- 172.21.15.72:0/3755193703 reap_dead start 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4067cc9640 1 -- 172.21.15.72:0/3755193703 shutdown_connections 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4067cc9640 1 -- 172.21.15.72:0/3755193703 >> 172.21.15.72:0/3755193703 conn(0x7f406006b190 msgr2=0x7f406006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4067cc9640 1 -- 172.21.15.72:0/3755193703 shutdown_connections 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.785+0000 7f4067cc9640 1 -- 172.21.15.72:0/3755193703 wait complete. 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.786+0000 7f4067cc9640 1 Processor -- start 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.786+0000 7f4067cc9640 1 -- start start 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4067cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f404c0039b0 0x7f40601f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4067cc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40601f43c0 0x7f40601521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4067cc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 0x7f4060152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4067cc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4060071350 con 0x7f40601f4840 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4067cc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f406015bde0 con 0x7f40601f43c0 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4067cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40600714d0 con 0x7f404c0039b0 2024-03-06T07:21:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 0x7f4060152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 0x7f4060152720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34768/0 (socket says 172.21.15.72:34768) 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 -- 172.21.15.72:0/2752215912 learned_addr learned my addr 172.21.15.72:0/2752215912 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4065264640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40601f43c0 0x7f40601521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4065a65640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f404c0039b0 0x7f40601f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 -- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f404c0039b0 msgr2=0x7f40601f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f404c0039b0 0x7f40601f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 -- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40601f43c0 msgr2=0x7f40601521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40601f43c0 0x7f40601521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4066266640 1 -- 172.21.15.72:0/2752215912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406014fe30 con 0x7f40601f4840 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4065264640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40601f43c0 0x7f40601521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.787+0000 7f4065a65640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f404c0039b0 0x7f40601f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.788+0000 7f4066266640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 0x7f4060152720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f406006bf00 tx=0x7f4058007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.788+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058003ea0 con 0x7f40601f4840 2024-03-06T07:21:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.788+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4058019070 con 0x7f40601f4840 2024-03-06T07:21:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.788+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4060150110 con 0x7f40601f4840 2024-03-06T07:21:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.788+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4058014630 con 0x7f40601f4840 2024-03-06T07:21:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.788+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40601506d0 con 0x7f40601f4840 2024-03-06T07:21:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.789+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40600732b0 con 0x7f40601f4840 2024-03-06T07:21:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.789+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40580147d0 con 0x7f40601f4840 2024-03-06T07:21:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.789+0000 7f40567fc640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f404406b4d0 0x7f404406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.790+0000 7f4065a65640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f404406b4d0 0x7f404406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.790+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f405808a380 con 0x7f40601f4840 2024-03-06T07:21:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.790+0000 7f4065a65640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f404406b4d0 0x7f404406d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ba30 tx=0x7f405c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.794+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40580595e0 con 0x7f40601f4840 2024-03-06T07:21:43.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.965+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40601508a0 con 0x7f40601f4840 2024-03-06T07:21:43.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.965+0000 7f40567fc640 1 -- 172.21.15.72:0/2752215912 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4058058ee0 con 0x7f40601f4840 2024-03-06T07:21:43.967 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f404406b4d0 msgr2=0x7f404406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f404406b4d0 0x7f404406d990 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ba30 tx=0x7f405c030000 comp rx=0 tx=0).stop 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 msgr2=0x7f4060152720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 0x7f4060152720 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f406006bf00 tx=0x7f4058007700 comp rx=0 tx=0).stop 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 shutdown_connections 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f404406b4d0 0x7f404406d990 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40601f4840 0x7f4060152720 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40601f43c0 0x7f40601521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 --2- 172.21.15.72:0/2752215912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f404c0039b0 0x7f40601f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:43.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 >> 172.21.15.72:0/2752215912 conn(0x7f406006b190 msgr2=0x7f406006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:43.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 shutdown_connections 2024-03-06T07:21:43.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:43.967+0000 7f4067cc9640 1 -- 172.21.15.72:0/2752215912 wait complete. 2024-03-06T07:21:43.979 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:44.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 Processor -- start 2024-03-06T07:21:44.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 -- start start 2024-03-06T07:21:44.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f797815bb20 0x7f79780d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79780cfff0 0x7f79780d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79780ce4b0 0x7f79780ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7978161980 con 0x7f79780cfff0 2024-03-06T07:21:44.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7978161c60 con 0x7f797815bb20 2024-03-06T07:21:44.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.079+0000 7f797eb79640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7978161f40 con 0x7f79780ce4b0 2024-03-06T07:21:44.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79780cfff0 0x7f79780d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79780cfff0 0x7f79780d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34776/0 (socket says 172.21.15.72:34776) 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 -- 172.21.15.72:0/2121036538 learned_addr learned my addr 172.21.15.72:0/2121036538 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 -- 172.21.15.72:0/2121036538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79780ce4b0 msgr2=0x7f79780ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 --2- 172.21.15.72:0/2121036538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79780ce4b0 0x7f79780ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 -- 172.21.15.72:0/2121036538 >> v2:172.21.15.146:3300/0 conn(0x7f797815bb20 msgr2=0x7f79780d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 --2- 172.21.15.72:0/2121036538 >> v2:172.21.15.146:3300/0 conn(0x7f797815bb20 0x7f79780d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 -- 172.21.15.72:0/2121036538 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978162220 con 0x7f79780cfff0 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.080+0000 7f7977fff640 1 --2- 172.21.15.72:0/2121036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79780d04a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f79780d2bc0 tx=0x7f796402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c91fb34d5f1d15 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f79777fe640 1 -- 172.21.15.72:0/2121036538 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964035020 con 0x7f79780cfff0 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f79777fe640 1 -- 172.21.15.72:0/2121036538 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964031870 con 0x7f79780cfff0 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f79777fe640 1 -- 172.21.15.72:0/2121036538 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964031c40 con 0x7f79780cfff0 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 -- 172.21.15.72:0/2121036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 msgr2=0x7f79780d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 --2- 172.21.15.72:0/2121036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79780d04a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f79780d2bc0 tx=0x7f796402b910 comp rx=0 tx=0).stop 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 -- 172.21.15.72:0/2121036538 shutdown_connections 2024-03-06T07:21:44.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 --2- 172.21.15.72:0/2121036538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79780ce4b0 0x7f79780ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 --2- 172.21.15.72:0/2121036538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79780d04a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 --2- 172.21.15.72:0/2121036538 >> v2:172.21.15.146:3300/0 conn(0x7f797815bb20 0x7f79780d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 -- 172.21.15.72:0/2121036538 >> 172.21.15.72:0/2121036538 conn(0x7f797809a9f0 msgr2=0x7f797809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 -- 172.21.15.72:0/2121036538 shutdown_connections 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.081+0000 7f797eb79640 1 -- 172.21.15.72:0/2121036538 wait complete. 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.082+0000 7f797eb79640 1 Processor -- start 2024-03-06T07:21:44.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.082+0000 7f797eb79640 1 -- start start 2024-03-06T07:21:44.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f797eb79640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79780ce4b0 0x7f79781f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f797eb79640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79781552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f797eb79640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f797815bb20 0x7f79780ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f797eb79640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79780d24b0 con 0x7f79780cfff0 2024-03-06T07:21:44.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f797eb79640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79780d2330 con 0x7f79780ce4b0 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f797eb79640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79780d1980 con 0x7f797815bb20 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f7977fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79781552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f7977fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79781552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34792/0 (socket says 172.21.15.72:34792) 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.083+0000 7f7977fff640 1 -- 172.21.15.72:0/2896629037 learned_addr learned my addr 172.21.15.72:0/2896629037 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f797d116640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f797815bb20 0x7f79780ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f7977fff640 1 -- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f797815bb20 msgr2=0x7f79780ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f797c915640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79780ce4b0 0x7f79781f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f7977fff640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f797815bb20 0x7f79780ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f7977fff640 1 -- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79780ce4b0 msgr2=0x7f79781f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f7977fff640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79780ce4b0 0x7f79781f7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f7977fff640 1 -- 172.21.15.72:0/2896629037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79780cb060 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f797d116640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f797815bb20 0x7f79780ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f797c915640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79780ce4b0 0x7f79781f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f7977fff640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79781552b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7964000c00 tx=0x7f79640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f796403ad60 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79640443f0 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79640447e0 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79780cb340 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.084+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79780c8b90 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.085+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f797804f330 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.085+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f796404b060 con 0x7f79780cfff0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.086+0000 7f79757fa640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f795c06b500 0x7f795c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.086+0000 7f797c915640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f795c06b500 0x7f795c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:44.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.086+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f79640ac110 con 0x7f79780cfff0 2024-03-06T07:21:44.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.086+0000 7f797c915640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f795c06b500 0x7f795c06d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79780b7a10 tx=0x7f796c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:44.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.090+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f796403f070 con 0x7f79780cfff0 2024-03-06T07:21:44.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.296+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f797815dfb0 con 0x7f79780cfff0 2024-03-06T07:21:44.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.297+0000 7f79757fa640 1 -- 172.21.15.72:0/2896629037 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f796407b2e0 con 0x7f79780cfff0 2024-03-06T07:21:44.299 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:44.299 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:44.299 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 2 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f795c06b500 msgr2=0x7f795c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f795c06b500 0x7f795c06d9c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f79780b7a10 tx=0x7f796c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 msgr2=0x7f79781552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79781552b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7964000c00 tx=0x7f79640046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 shutdown_connections 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f795c06b500 0x7f795c06d9c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f797815bb20 0x7f79780ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79780cfff0 0x7f79781552b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 --2- 172.21.15.72:0/2896629037 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79780ce4b0 0x7f79781f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 >> 172.21.15.72:0/2896629037 conn(0x7f797809a9f0 msgr2=0x7f797809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:44.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.298+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 shutdown_connections 2024-03-06T07:21:44.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:44.299+0000 7f797eb79640 1 -- 172.21.15.72:0/2896629037 wait complete. 2024-03-06T07:21:46.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b79c6f640 1 Processor -- start 2024-03-06T07:21:46.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b79c6f640 1 -- start start 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b79c6f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4b740d2710 0x7f4b740b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b79c6f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b740b3430 con 0x7f4b740d22b0 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b79c6f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b740b3710 con 0x7f4b7415dfe0 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b79c6f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b740b31e0 con 0x7f4b740d2710 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b73fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4b740d2710 0x7f4b740b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b6affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4b740d22b0 0x7f4b740b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41764/0 (socket says 172.21.15.72:41764) 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.422+0000 7f4b6affd640 1 -- 172.21.15.72:0/3480610141 learned_addr learned my addr 172.21.15.72:0/3480610141 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.423+0000 7f4b73fff640 1 -- 172.21.15.72:0/3480610141 >> v1:172.21.15.146:6789/0 conn(0x7f4b7415dfe0 legacy=0x7f4b740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.423+0000 7f4b73fff640 1 -- 172.21.15.72:0/3480610141 >> v1:172.21.15.72:6789/0 conn(0x7f4b740d22b0 legacy=0x7f4b740b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.423+0000 7f4b73fff640 1 -- 172.21.15.72:0/3480610141 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b740b39f0 con 0x7f4b740d2710 2024-03-06T07:21:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.423+0000 7f4b73fff640 1 --2- 172.21.15.72:0/3480610141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b740b4900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009dc0 tx=0x7f4b6c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f7ced8407e5e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b72ffd640 1 -- 172.21.15.72:0/3480610141 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c039070 con 0x7f4b740d2710 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b72ffd640 1 -- 172.21.15.72:0/3480610141 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6c02cc40 con 0x7f4b740d2710 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b72ffd640 1 -- 172.21.15.72:0/3480610141 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c034440 con 0x7f4b740d2710 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 -- 172.21.15.72:0/3480610141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 msgr2=0x7f4b740b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/3480610141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b740b4900 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c009dc0 tx=0x7f4b6c02be90 comp rx=0 tx=0).stop 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 -- 172.21.15.72:0/3480610141 shutdown_connections 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/3480610141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b740b4900 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 -- 172.21.15.72:0/3480610141 >> 172.21.15.72:0/3480610141 conn(0x7f4b740d3cd0 msgr2=0x7f4b740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 -- 172.21.15.72:0/3480610141 shutdown_connections 2024-03-06T07:21:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.424+0000 7f4b79c6f640 1 -- 172.21.15.72:0/3480610141 wait complete. 2024-03-06T07:21:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.425+0000 7f4b79c6f640 1 Processor -- start 2024-03-06T07:21:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 -- start start 2024-03-06T07:21:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b740d22b0 0x7f4b7415bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b741f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b7415dfe0 0x7f4b740b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740cfa00 con 0x7f4b740d2710 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740b3d60 con 0x7f4b740d22b0 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b79c6f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b740d1c70 con 0x7f4b7415dfe0 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b6affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b741f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b6affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b741f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34812/0 (socket says 172.21.15.72:34812) 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.426+0000 7f4b6affd640 1 -- 172.21.15.72:0/6837046 learned_addr learned my addr 172.21.15.72:0/6837046 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b737fe640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b740d22b0 0x7f4b7415bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b6affd640 1 -- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b7415dfe0 msgr2=0x7f4b740b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b73fff640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b7415dfe0 0x7f4b740b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b6affd640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b7415dfe0 0x7f4b740b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b6affd640 1 -- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b740d22b0 msgr2=0x7f4b7415bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b6affd640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b740d22b0 0x7f4b7415bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b6affd640 1 -- 172.21.15.72:0/6837046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7415c500 con 0x7f4b740d2710 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b737fe640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b740d22b0 0x7f4b7415bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b6affd640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b741f6fa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00c430 tx=0x7f4b5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c017070 con 0x7f4b740d2710 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c005ce0 con 0x7f4b740d2710 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b741f78a0 con 0x7f4b740d2710 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c0072f0 con 0x7f4b740d2710 2024-03-06T07:21:46.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.427+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b74160250 con 0x7f4b740d2710 2024-03-06T07:21:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.428+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7404f330 con 0x7f4b740d2710 2024-03-06T07:21:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.428+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4b5c01a070 con 0x7f4b740d2710 2024-03-06T07:21:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.429+0000 7f4b70ff9640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4b5406b4d0 0x7f4b5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.429+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b5c089240 con 0x7f4b740d2710 2024-03-06T07:21:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.429+0000 7f4b737fe640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4b5406b4d0 0x7f4b5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.429+0000 7f4b737fe640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4b5406b4d0 0x7f4b5406d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400a830 tx=0x7f4b64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.433+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4b5c0584a0 con 0x7f4b740d2710 2024-03-06T07:21:46.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.638+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b741f7dc0 con 0x7f4b740d2710 2024-03-06T07:21:46.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.639+0000 7f4b70ff9640 1 -- 172.21.15.72:0/6837046 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7f4b5c057da0 con 0x7f4b740d2710 2024-03-06T07:21:46.641 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:46.641 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-06T07:21:46.641 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 3 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4b5406b4d0 msgr2=0x7f4b5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4b5406b4d0 0x7f4b5406d990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400a830 tx=0x7f4b64008040 comp rx=0 tx=0).stop 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 msgr2=0x7f4b741f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b741f6fa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00c430 tx=0x7f4b5c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 shutdown_connections 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4b5406b4d0 0x7f4b5406d990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b7415dfe0 0x7f4b740b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b740d2710 0x7f4b741f6fa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 --2- 172.21.15.72:0/6837046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b740d22b0 0x7f4b7415bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 >> 172.21.15.72:0/6837046 conn(0x7f4b740d3cd0 msgr2=0x7f4b7406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 shutdown_connections 2024-03-06T07:21:46.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.641+0000 7f4b79c6f640 1 -- 172.21.15.72:0/6837046 wait complete. 2024-03-06T07:21:46.653 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:46.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.751+0000 7f8fd8fa0640 1 Processor -- start 2024-03-06T07:21:46.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.751+0000 7f8fd8fa0640 1 -- start start 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd8fa0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd41515e0 0x7f8fd40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd8fa0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8fd40cd7a0 0x7f8fd40cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd8fa0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd4071f40 con 0x7f8fd415b430 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd8fa0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd415b8b0 con 0x7f8fd40cd7a0 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd8fa0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4070a70 con 0x7f8fd41515e0 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8fd40cd7a0 0x7f8fd40cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8fd40cd7a0 0x7f8fd40cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34816/0 (socket says 172.21.15.72:34816) 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd2d9d640 1 -- 172.21.15.72:0/464180181 learned_addr learned my addr 172.21.15.72:0/464180181 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.752+0000 7f8fd259c640 1 --2- 172.21.15.72:0/464180181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd41515e0 0x7f8fd40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd2d9d640 1 -- 172.21.15.72:0/464180181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd41515e0 msgr2=0x7f8fd40cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd2d9d640 1 --2- 172.21.15.72:0/464180181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd41515e0 0x7f8fd40cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd2d9d640 1 -- 172.21.15.72:0/464180181 >> v1:172.21.15.146:6789/0 conn(0x7f8fd415b430 legacy=0x7f8fd4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd2d9d640 1 -- 172.21.15.72:0/464180181 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd4070d50 con 0x7f8fd40cd7a0 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd2d9d640 1 --2- 172.21.15.72:0/464180181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd40cdbd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009870 tx=0x7f8fcc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40e200e267f2384 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd159a640 1 -- 172.21.15.72:0/464180181 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc034070 con 0x7f8fd40cd7a0 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd159a640 1 -- 172.21.15.72:0/464180181 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc030840 con 0x7f8fd40cd7a0 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.753+0000 7f8fd159a640 1 -- 172.21.15.72:0/464180181 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc030c10 con 0x7f8fd40cd7a0 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/464180181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd40cd7a0 msgr2=0x7f8fd40cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/464180181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd40cdbd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009870 tx=0x7f8fcc02b980 comp rx=0 tx=0).stop 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/464180181 shutdown_connections 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/464180181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd40cdbd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/464180181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd41515e0 0x7f8fd40cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/464180181 >> 172.21.15.72:0/464180181 conn(0x7f8fd40cf8a0 msgr2=0x7f8fd40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/464180181 shutdown_connections 2024-03-06T07:21:46.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.754+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/464180181 wait complete. 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.755+0000 7f8fd8fa0640 1 Processor -- start 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.755+0000 7f8fd8fa0640 1 -- start start 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd8fa0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd8fa0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 0x7f8fd4072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd8fa0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd415b430 0x7f8fd4072650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd8fa0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40ce110 con 0x7f8fd41515e0 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd8fa0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4071310 con 0x7f8fd40cd7a0 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd8fa0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40b3f90 con 0x7f8fd415b430 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 0x7f8fd4072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 0x7f8fd4072110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34832/0 (socket says 172.21.15.72:34832) 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 -- 172.21.15.72:0/2650229666 learned_addr learned my addr 172.21.15.72:0/2650229666 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd259c640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd2d9d640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd415b430 0x7f8fd4072650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 -- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd415b430 msgr2=0x7f8fd4072650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd415b430 0x7f8fd4072650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 -- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd40cd7a0 msgr2=0x7f8fd414b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.756+0000 7f8fd1d9b640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd414b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fd1d9b640 1 -- 172.21.15.72:0/2650229666 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd40c71f0 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fd259c640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd414b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fd1d9b640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 0x7f8fd4072110 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c3c0 tx=0x7f8fc800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80052d0 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc8005490 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8007260 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd40c74d0 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.757+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd40c7a90 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.758+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd4068840 con 0x7f8fd41515e0 2024-03-06T07:21:46.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.758+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fc8007400 con 0x7f8fd41515e0 2024-03-06T07:21:46.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.758+0000 7f8fc2ffd640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:46.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.759+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8fc8089300 con 0x7f8fd41515e0 2024-03-06T07:21:46.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.759+0000 7f8fd259c640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:46.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.759+0000 7f8fd259c640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fb406b4d0 0x7f8fb406d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40074c0 tx=0x7f8fc400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:46.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.763+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fc8058560 con 0x7f8fd41515e0 2024-03-06T07:21:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.933+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd404f330 con 0x7f8fd41515e0 2024-03-06T07:21:46.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.934+0000 7f8fc2ffd640 1 -- 172.21.15.72:0/2650229666 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12896 (secure 0 0 0) 0x7f8fc8057e60 con 0x7f8fd41515e0 2024-03-06T07:21:46.936 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:46.936 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":17,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:41.025880+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:46.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fb406b4d0 msgr2=0x7f8fb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fb406b4d0 0x7f8fb406d990 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40074c0 tx=0x7f8fc400b040 comp rx=0 tx=0).stop 2024-03-06T07:21:46.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 msgr2=0x7f8fd4072110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 0x7f8fd4072110 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c3c0 tx=0x7f8fc800c880 comp rx=0 tx=0).stop 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 shutdown_connections 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fb406b4d0 0x7f8fb406d990 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd415b430 0x7f8fd4072650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd41515e0 0x7f8fd4072110 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 --2- 172.21.15.72:0/2650229666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd40cd7a0 0x7f8fd414b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 >> 172.21.15.72:0/2650229666 conn(0x7f8fd40cf8a0 msgr2=0x7f8fd4073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 shutdown_connections 2024-03-06T07:21:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:46.936+0000 7f8fd8fa0640 1 -- 172.21.15.72:0/2650229666 wait complete. 2024-03-06T07:21:46.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:46.949 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:47.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.051+0000 7f1ae75d7640 1 Processor -- start 2024-03-06T07:21:47.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.051+0000 7f1ae75d7640 1 -- start start 2024-03-06T07:21:47.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae75d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae00d4860 0x7f1ae00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae75d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1ae00b0e30 0x7f1ae00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae75d7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae01618b0 con 0x7f1ae01614d0 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae75d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae00d4cc0 con 0x7f1ae00b0e30 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae75d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00cf400 con 0x7f1ae00d4860 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5373640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1ae01614d0 0x7f1ae00d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41770/0 (socket says 172.21.15.72:41770) 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5373640 1 -- 172.21.15.72:0/2093945996 learned_addr learned my addr 172.21.15.72:0/2093945996 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5b74640 1 --2- 172.21.15.72:0/2093945996 >> v2:172.21.15.72:3300/0 conn(0x7f1ae00b0e30 0x7f1ae00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ad7fff640 1 -- 172.21.15.72:0/2093945996 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836087443 0 0) 0x7f1ae01618b0 con 0x7f1ae01614d0 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae4b72640 1 --2- 172.21.15.72:0/2093945996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae00d4860 0x7f1ae00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ad7fff640 1 -- 172.21.15.72:0/2093945996 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1acc003520 con 0x7f1ae01614d0 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5b74640 1 -- 172.21.15.72:0/2093945996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae00d4860 msgr2=0x7f1ae00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5b74640 1 --2- 172.21.15.72:0/2093945996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae00d4860 0x7f1ae00d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5b74640 1 -- 172.21.15.72:0/2093945996 >> v1:172.21.15.72:6789/0 conn(0x7f1ae01614d0 legacy=0x7f1ae00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.052+0000 7f1ae5b74640 1 -- 172.21.15.72:0/2093945996 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00cf6a0 con 0x7f1ae00b0e30 2024-03-06T07:21:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae5b74640 1 --2- 172.21.15.72:0/2093945996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00b0e30 0x7f1ae00b12c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009d90 tx=0x7f1ad802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f8f891092e10c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ad7fff640 1 -- 172.21.15.72:0/2093945996 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8039070 con 0x7f1ae00b0e30 2024-03-06T07:21:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ad7fff640 1 -- 172.21.15.72:0/2093945996 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad802cc50 con 0x7f1ae00b0e30 2024-03-06T07:21:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ad7fff640 1 -- 172.21.15.72:0/2093945996 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8034440 con 0x7f1ae00b0e30 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae75d7640 1 -- 172.21.15.72:0/2093945996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00b0e30 msgr2=0x7f1ae00b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/2093945996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00b0e30 0x7f1ae00b12c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009d90 tx=0x7f1ad802bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae75d7640 1 -- 172.21.15.72:0/2093945996 shutdown_connections 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/2093945996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00b0e30 0x7f1ae00b12c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/2093945996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae00d4860 0x7f1ae00d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.053+0000 7f1ae75d7640 1 -- 172.21.15.72:0/2093945996 >> 172.21.15.72:0/2093945996 conn(0x7f1ae00d3d40 msgr2=0x7f1ae00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.054+0000 7f1ae75d7640 1 -- 172.21.15.72:0/2093945996 shutdown_connections 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.054+0000 7f1ae75d7640 1 -- 172.21.15.72:0/2093945996 wait complete. 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 Processor -- start 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 -- start start 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae00b0e30 0x7f1ae015f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 0x7f1ae00cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae01614d0 0x7f1ae01f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00b0910 con 0x7f1ae00d4860 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00cfc60 con 0x7f1ae00b0e30 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae75d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00d2560 con 0x7f1ae01614d0 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae4b72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 0x7f1ae00cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae4b72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 0x7f1ae00cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34846/0 (socket says 172.21.15.72:34846) 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.055+0000 7f1ae4b72640 1 -- 172.21.15.72:0/3650214947 learned_addr learned my addr 172.21.15.72:0/3650214947 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae5b74640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae01614d0 0x7f1ae01f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae5373640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae00b0e30 0x7f1ae015f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae4b72640 1 -- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae01614d0 msgr2=0x7f1ae01f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae4b72640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae01614d0 0x7f1ae01f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae4b72640 1 -- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae00b0e30 msgr2=0x7f1ae015f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae4b72640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae00b0e30 0x7f1ae015f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae4b72640 1 -- 172.21.15.72:0/3650214947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00c89b0 con 0x7f1ae00d4860 2024-03-06T07:21:47.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae4b72640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 0x7f1ae00cc330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000d830 tx=0x7f1ad000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000b6c0 con 0x7f1ae00d4860 2024-03-06T07:21:47.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0019070 con 0x7f1ae00d4860 2024-03-06T07:21:47.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad000ba80 con 0x7f1ae00d4860 2024-03-06T07:21:47.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae00c8c90 con 0x7f1ae00d4860 2024-03-06T07:21:47.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.056+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae015f580 con 0x7f1ae00d4860 2024-03-06T07:21:47.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.057+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae0068ff0 con 0x7f1ae00d4860 2024-03-06T07:21:47.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.057+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ad001e460 con 0x7f1ae00d4860 2024-03-06T07:21:47.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.058+0000 7f1ad5ffb640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.058+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ad008a360 con 0x7f1ae00d4860 2024-03-06T07:21:47.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.058+0000 7f1ae5373640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.058+0000 7f1ae5373640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1adc005880 tx=0x7f1adc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.062+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ad0058f70 con 0x7f1ae00d4860 2024-03-06T07:21:47.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.233+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ae01f6c80 con 0x7f1ae00d4860 2024-03-06T07:21:47.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.234+0000 7f1ad5ffb640 1 -- 172.21.15.72:0/3650214947 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ad005d010 con 0x7f1ae00d4860 2024-03-06T07:21:47.236 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:21:47.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ac406b4d0 msgr2=0x7f1ac406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ac406b4d0 0x7f1ac406d990 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1adc005880 tx=0x7f1adc009040 comp rx=0 tx=0).stop 2024-03-06T07:21:47.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 msgr2=0x7f1ae00cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 0x7f1ae00cc330 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000d830 tx=0x7f1ad000dcf0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 shutdown_connections 2024-03-06T07:21:47.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ac406b4d0 0x7f1ac406d990 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae01614d0 0x7f1ae01f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ae00d4860 0x7f1ae00cc330 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 --2- 172.21.15.72:0/3650214947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae00b0e30 0x7f1ae015f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 >> 172.21.15.72:0/3650214947 conn(0x7f1ae00d3d40 msgr2=0x7f1ae006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 shutdown_connections 2024-03-06T07:21:47.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.236+0000 7f1ae75d7640 1 -- 172.21.15.72:0/3650214947 wait complete. 2024-03-06T07:21:47.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:21:47.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.350+0000 7f5e79847640 1 Processor -- start 2024-03-06T07:21:47.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.350+0000 7f5e79847640 1 -- start start 2024-03-06T07:21:47.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e79847640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e740d22b0 0x7f5e740d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e79847640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e740b41f0 0x7f5e740b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e79847640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740b3430 con 0x7f5e7415dfe0 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e79847640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e740d2c50 con 0x7f5e740b41f0 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e79847640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e740b31e0 con 0x7f5e740d22b0 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e740b41f0 0x7f5e740b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e740b41f0 0x7f5e740b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34862/0 (socket says 172.21.15.72:34862) 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 -- 172.21.15.72:0/2338129279 learned_addr learned my addr 172.21.15.72:0/2338129279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e727fc640 1 --2- 172.21.15.72:0/2338129279 >> v2:172.21.15.146:3300/0 conn(0x7f5e740d22b0 0x7f5e740d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 -- 172.21.15.72:0/2338129279 >> v2:172.21.15.146:3300/0 conn(0x7f5e740d22b0 msgr2=0x7f5e740d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 --2- 172.21.15.72:0/2338129279 >> v2:172.21.15.146:3300/0 conn(0x7f5e740d22b0 0x7f5e740d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 -- 172.21.15.72:0/2338129279 >> v1:172.21.15.146:6789/0 conn(0x7f5e7415dfe0 legacy=0x7f5e740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.351+0000 7f5e737fe640 1 -- 172.21.15.72:0/2338129279 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740b3710 con 0x7f5e740b41f0 2024-03-06T07:21:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.352+0000 7f5e737fe640 1 --2- 172.21.15.72:0/2338129279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e740b4680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009e50 tx=0x7f5e6c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59a5da57c8e9067 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.352+0000 7f5e71ffb640 1 -- 172.21.15.72:0/2338129279 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c02cb20 con 0x7f5e740b41f0 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.352+0000 7f5e71ffb640 1 -- 172.21.15.72:0/2338129279 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c02ccc0 con 0x7f5e740b41f0 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.352+0000 7f5e71ffb640 1 -- 172.21.15.72:0/2338129279 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c034610 con 0x7f5e740b41f0 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 -- 172.21.15.72:0/2338129279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 msgr2=0x7f5e740b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 --2- 172.21.15.72:0/2338129279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e740b4680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009e50 tx=0x7f5e6c02bf80 comp rx=0 tx=0).stop 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 -- 172.21.15.72:0/2338129279 shutdown_connections 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 --2- 172.21.15.72:0/2338129279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e740b4680 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 --2- 172.21.15.72:0/2338129279 >> v2:172.21.15.146:3300/0 conn(0x7f5e740d22b0 0x7f5e740d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 -- 172.21.15.72:0/2338129279 >> 172.21.15.72:0/2338129279 conn(0x7f5e740d3cd0 msgr2=0x7f5e740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 -- 172.21.15.72:0/2338129279 shutdown_connections 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.353+0000 7f5e79847640 1 -- 172.21.15.72:0/2338129279 wait complete. 2024-03-06T07:21:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.354+0000 7f5e79847640 1 Processor -- start 2024-03-06T07:21:47.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.354+0000 7f5e79847640 1 -- start start 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e7415c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e740d22b0 0x7f5e740b85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7415dfe0 0x7f5e741603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740cfa00 con 0x7f5e740b41f0 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740b3d60 con 0x7f5e7415dfe0 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740d1c70 con 0x7f5e740d22b0 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e7415c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e7415c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34866/0 (socket says 172.21.15.72:34866) 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 -- 172.21.15.72:0/3918937400 learned_addr learned my addr 172.21.15.72:0/3918937400 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e737fe640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7415dfe0 0x7f5e741603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e727fc640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e740d22b0 0x7f5e740b85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 -- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e740d22b0 msgr2=0x7f5e740b85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e740d22b0 0x7f5e740b85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 -- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7415dfe0 msgr2=0x7f5e741603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7415dfe0 0x7f5e741603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 -- 172.21.15.72:0/3918937400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740cd170 con 0x7f5e740b41f0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e72ffd640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e7415c340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400c3c0 tx=0x7f5e6400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e640052d0 con 0x7f5e740b41f0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e64005490 con 0x7f5e740b41f0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e64007260 con 0x7f5e740b41f0 2024-03-06T07:21:47.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e740cd450 con 0x7f5e740b41f0 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.355+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e740cda10 con 0x7f5e740b41f0 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.356+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7404f330 con 0x7f5e740b41f0 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.357+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e64007400 con 0x7f5e740b41f0 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.357+0000 7f5e5bfff640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.357+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e64089350 con 0x7f5e740b41f0 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.357+0000 7f5e727fc640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.358+0000 7f5e727fc640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e740b3150 tx=0x7f5e68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.361+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e64058520 con 0x7f5e740b41f0 2024-03-06T07:21:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.536+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5e7415d2e0 con 0x7f5e740b41f0 2024-03-06T07:21:47.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.536+0000 7f5e5bfff640 1 -- 172.21.15.72:0/3918937400 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5e64057e20 con 0x7f5e740b41f0 2024-03-06T07:21:47.539 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:47.539 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:21:47.540 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:21:47.540 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:21:47.540 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:21:47.540 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:21:47.540 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 msgr2=0x7f5e6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e740b3150 tx=0x7f5e68008040 comp rx=0 tx=0).stop 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 msgr2=0x7f5e7415c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e7415c340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5e6400c3c0 tx=0x7f5e6400c880 comp rx=0 tx=0).stop 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 shutdown_connections 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e7415dfe0 0x7f5e741603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e740d22b0 0x7f5e740b85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 --2- 172.21.15.72:0/3918937400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740b41f0 0x7f5e7415c340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 >> 172.21.15.72:0/3918937400 conn(0x7f5e740d3cd0 msgr2=0x7f5e7406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 shutdown_connections 2024-03-06T07:21:47.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.538+0000 7f5e79847640 1 -- 172.21.15.72:0/3918937400 wait complete. 2024-03-06T07:21:47.551 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:21:47.551 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:21:47.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.653+0000 7f6a91881640 1 Processor -- start 2024-03-06T07:21:47.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.653+0000 7f6a91881640 1 -- start start 2024-03-06T07:21:47.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a91881640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a8c15dfa0 0x7f6a8c0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a91881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a8c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a91881640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a8c15e380 con 0x7f6a8c0d2820 2024-03-06T07:21:47.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a91881640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8c0d2100 con 0x7f6a8c15dfa0 2024-03-06T07:21:47.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a91881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0d1e30 con 0x7f6a8c0d4630 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6a8c0d2820 0x7f6a8c0d05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41772/0 (socket says 172.21.15.72:41772) 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8b7fe640 1 -- 172.21.15.72:0/3960241366 learned_addr learned my addr 172.21.15.72:0/3960241366 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8affd640 1 --2- 172.21.15.72:0/3960241366 >> v2:172.21.15.146:3300/0 conn(0x7f6a8c15dfa0 0x7f6a8c0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a83fff640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a8c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578932729 0 0) 0x7f6a8c15e380 con 0x7f6a8c0d2820 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a78003520 con 0x7f6a8c0d2820 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2021682854 0 0) 0x7f6a78003520 con 0x7f6a8c0d2820 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 msgr2=0x7f6a8c0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a8c0d22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 >> v2:172.21.15.146:3300/0 conn(0x7f6a8c15dfa0 msgr2=0x7f6a8c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 --2- 172.21.15.72:0/3960241366 >> v2:172.21.15.146:3300/0 conn(0x7f6a8c15dfa0 0x7f6a8c0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c0cf470 con 0x7f6a8c0d2820 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6a840019f0 con 0x7f6a8c0d2820 2024-03-06T07:21:47.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a83fff640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a8c0d22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 >> v1:172.21.15.72:6789/0 conn(0x7f6a8c0d2820 legacy=0x7f6a8c0d05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a780039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a78003ef0 0x7f6a78004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a78004d40 0x7f6a780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.654+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0d2100 con 0x7f6a78003ef0 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0d1e30 con 0x7f6a78004d40 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7800b9c0 con 0x7f6a8c0d4630 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a8affd640 1 --2- 172.21.15.72:0/3960241366 >> v2:172.21.15.146:3300/0 conn(0x7f6a8c15dfa0 0x7f6a8c0d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a78003ef0 0x7f6a78004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a8b7fe640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a78004d40 0x7f6a780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a8affd640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a780039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 -- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 msgr2=0x7f6a780039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a8c0d4630 0x7f6a780039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 -- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a78004d40 msgr2=0x7f6a780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a78004d40 0x7f6a780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 -- 172.21.15.72:0/3960241366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a78003520 con 0x7f6a78003ef0 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.655+0000 7f6a83fff640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a78003ef0 0x7f6a78004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009c90 tx=0x7f6a78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61fb50a5ed17c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74039070 con 0x7f6a78003ef0 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7402c8f0 con 0x7f6a78003ef0 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a8a7fc640 1 -- 172.21.15.72:0/3960241366 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a74009860 con 0x7f6a78003ef0 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a91881640 1 -- 172.21.15.72:0/3960241366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a78003ef0 msgr2=0x7f6a78004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a91881640 1 --2- 172.21.15.72:0/3960241366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a78003ef0 0x7f6a78004800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009c90 tx=0x7f6a78005830 comp rx=0 tx=0).stop 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a8b7fe640 1 -- 172.21.15.72:0/3960241366 reap_dead start 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a91881640 1 -- 172.21.15.72:0/3960241366 shutdown_connections 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a91881640 1 -- 172.21.15.72:0/3960241366 >> 172.21.15.72:0/3960241366 conn(0x7f6a8c09d000 msgr2=0x7f6a8c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a91881640 1 -- 172.21.15.72:0/3960241366 shutdown_connections 2024-03-06T07:21:47.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.656+0000 7f6a91881640 1 -- 172.21.15.72:0/3960241366 wait complete. 2024-03-06T07:21:47.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.657+0000 7f6a91881640 1 Processor -- start 2024-03-06T07:21:47.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.657+0000 7f6a91881640 1 -- start start 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a91881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a78003ef0 0x7f6a8c0a0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a91881640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a8c09f670 0x7f6a8c09dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a91881640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 0x7f6a8c06c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a91881640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0cc1a0 con 0x7f6a8c153080 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a91881640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0cfa30 con 0x7f6a8c09f670 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a91881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8c0cc320 con 0x7f6a78003ef0 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 0x7f6a8c06c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 0x7f6a8c06c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34882/0 (socket says 172.21.15.72:34882) 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 -- 172.21.15.72:0/653065290 learned_addr learned my addr 172.21.15.72:0/653065290 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 -- 172.21.15.72:0/653065290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a78003ef0 msgr2=0x7f6a8c0a0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a78003ef0 0x7f6a8c0a0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 -- 172.21.15.72:0/653065290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a8c09f670 msgr2=0x7f6a8c09dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a8c09f670 0x7f6a8c09dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 -- 172.21.15.72:0/653065290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8c09e1e0 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.658+0000 7f6a8b7fe640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 0x7f6a8c06c9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a84000f80 tx=0x7f6a8400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.659+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8400ee20 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.659+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84018070 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.659+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8c09fe30 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.659+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a84013660 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.659+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8c0a03f0 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.660+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8c04f330 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.660+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a84013800 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.660+0000 7f6a88ff9640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a6c06b500 0x7f6a6c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.660+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a8408a1f0 con 0x7f6a8c153080 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.660+0000 7f6a8affd640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a6c06b500 0x7f6a6c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.661+0000 7f6a8affd640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a6c06b500 0x7f6a6c06d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009b20 tx=0x7f6a7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:47.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.665+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a840593c0 con 0x7f6a8c153080 2024-03-06T07:21:47.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:47.838+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6a8c0ccde0 con 0x7f6a8c153080 2024-03-06T07:21:48.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.048+0000 7f6a88ff9640 1 -- 172.21.15.72:0/653065290 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6a84058cc0 con 0x7f6a8c153080 2024-03-06T07:21:48.051 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.051+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a6c06b500 msgr2=0x7f6a6c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.051+0000 7f6a91881640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a6c06b500 0x7f6a6c06d9c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c009b20 tx=0x7f6a7c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 msgr2=0x7f6a8c06c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 0x7f6a8c06c9a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6a84000f80 tx=0x7f6a8400e530 comp rx=0 tx=0).stop 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 shutdown_connections 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a6c06b500 0x7f6a6c06d9c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a8c153080 0x7f6a8c06c9a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a8c09f670 0x7f6a8c09dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 --2- 172.21.15.72:0/653065290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a78003ef0 0x7f6a8c0a0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 >> 172.21.15.72:0/653065290 conn(0x7f6a8c09d000 msgr2=0x7f6a8c0cd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 shutdown_connections 2024-03-06T07:21:48.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.052+0000 7f6a91881640 1 -- 172.21.15.72:0/653065290 wait complete. 2024-03-06T07:21:48.068 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:21:48.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.172+0000 7f5bfa237640 1 Processor -- start 2024-03-06T07:21:48.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.172+0000 7f5bfa237640 1 -- start start 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bfa237640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bfa237640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf40d2480 0x7f5bf40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bfa237640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bf41618b0 con 0x7f5bf40d1910 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bfa237640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bf40cf400 con 0x7f5bf40d2480 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bfa237640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40cf6a0 con 0x7f5bf41614d0 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bf3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf40d2480 0x7f5bf40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bf2ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5bf40d1910 0x7f5bf40d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:48076/0 (socket says 172.21.15.72:48076) 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bf3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bf40d2480 0x7f5bf40b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:48440/0 (socket says 172.21.15.72:48440) 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf2ffd640 1 -- 172.21.15.72:0/282163074 learned_addr learned my addr 172.21.15.72:0/282163074 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.173+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf40d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:57728/0 (socket says 172.21.15.72:57728) 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf27fc640 1 -- 172.21.15.72:0/282163074 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996857669 0 0) 0x7f5bf41618b0 con 0x7f5bf40d1910 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf27fc640 1 -- 172.21.15.72:0/282163074 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bdc003520 con 0x7f5bf40d1910 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf3fff640 1 -- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 msgr2=0x7f5bf40d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf3fff640 1 --2- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf40d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf3fff640 1 -- 172.21.15.72:0/282163074 >> v1:172.21.15.146:6789/0 conn(0x7f5bf40d1910 legacy=0x7f5bf40d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf3fff640 1 -- 172.21.15.72:0/282163074 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf40cf980 con 0x7f5bf40d2480 2024-03-06T07:21:48.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.174+0000 7f5bf37fe640 1 --2- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bf3fff640 1 --2- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40b0e30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009d90 tx=0x7f5bec02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7929c590e62e1371 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bf27fc640 1 -- 172.21.15.72:0/282163074 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec039070 con 0x7f5bf40d2480 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bf27fc640 1 -- 172.21.15.72:0/282163074 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec02cc50 con 0x7f5bf40d2480 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bf27fc640 1 -- 172.21.15.72:0/282163074 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec034440 con 0x7f5bf40d2480 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 -- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 msgr2=0x7f5bf40b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 --2- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40b0e30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5bec009d90 tx=0x7f5bec02bea0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 -- 172.21.15.72:0/282163074 shutdown_connections 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 --2- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40b0e30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 --2- 172.21.15.72:0/282163074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf40d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 -- 172.21.15.72:0/282163074 >> 172.21.15.72:0/282163074 conn(0x7f5bf40d3d40 msgr2=0x7f5bf40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 -- 172.21.15.72:0/282163074 shutdown_connections 2024-03-06T07:21:48.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.175+0000 7f5bfa237640 1 -- 172.21.15.72:0/282163074 wait complete. 2024-03-06T07:21:48.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.176+0000 7f5bfa237640 1 Processor -- start 2024-03-06T07:21:48.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 -- start start 2024-03-06T07:21:48.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 0x7f5bf40cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40b1370 con 0x7f5bf40d1910 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40b0910 con 0x7f5bf40d2480 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bfa237640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bf40b14f0 con 0x7f5bf41614d0 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.177+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 0x7f5bf40cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 0x7f5bf40cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34890/0 (socket says 172.21.15.72:34890) 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 -- 172.21.15.72:0/1901076614 learned_addr learned my addr 172.21.15.72:0/1901076614 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf2ffd640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf3fff640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 -- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 msgr2=0x7f5bf415b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf415b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 -- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 msgr2=0x7f5bf40cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 -- 172.21.15.72:0/1901076614 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf40b8650 con 0x7f5bf40d1910 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf2ffd640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf3fff640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf415b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bf37fe640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 0x7f5bf40cd990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e960 tx=0x7f5be400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be4017070 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5be400ca00 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5be400cdd0 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bf40b8930 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.178+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bf40ccdb0 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.179+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf40cbe60 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.180+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5be401a070 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.180+0000 7f5bdbfff640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5be006b500 0x7f5be006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.180+0000 7f5bf2ffd640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5be006b500 0x7f5be006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.180+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f5be4089e60 con 0x7f5bf40d1910 2024-03-06T07:21:48.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.180+0000 7f5bf2ffd640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5be006b500 0x7f5be006d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5be8007fd0 tx=0x7f5be8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:48.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.185+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5be4058f50 con 0x7f5bf40d1910 2024-03-06T07:21:48.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:48.357+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5bf40b25c0 con 0x7f5bf40d1910 2024-03-06T07:21:49.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.053+0000 7f5bdbfff640 1 -- 172.21.15.72:0/1901076614 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5be4058850 con 0x7f5bf40d1910 2024-03-06T07:21:49.055 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.056+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5be006b500 msgr2=0x7f5be006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.056+0000 7f5bfa237640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5be006b500 0x7f5be006d9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5be8007fd0 tx=0x7f5be8009040 comp rx=0 tx=0).stop 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.056+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 msgr2=0x7f5bf40cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.056+0000 7f5bfa237640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 0x7f5bf40cd990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5be400e960 tx=0x7f5be400ee20 comp rx=0 tx=0).stop 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.056+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 shutdown_connections 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5be006b500 0x7f5be006d9c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bf41614d0 0x7f5bf415b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bf40d2480 0x7f5bf40cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 --2- 172.21.15.72:0/1901076614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bf40d1910 0x7f5bf40cd990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 >> 172.21.15.72:0/1901076614 conn(0x7f5bf40d3d40 msgr2=0x7f5bf40b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 shutdown_connections 2024-03-06T07:21:49.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.057+0000 7f5bfa237640 1 -- 172.21.15.72:0/1901076614 wait complete. 2024-03-06T07:21:49.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:21:49.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9892d65640 1 Processor -- start 2024-03-06T07:21:49.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9892d65640 1 -- start start 2024-03-06T07:21:49.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9892d65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 0x7f988c0cc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9892d65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c06ee00 con 0x7f988c0ceb80 2024-03-06T07:21:49.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9892d65640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988c06f0e0 con 0x7f988c0d0060 2024-03-06T07:21:49.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9892d65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c06f3c0 con 0x7f988c06be00 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9891302640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f988c0ceb80 0x7f988c0ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41786/0 (socket says 172.21.15.72:41786) 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.181+0000 7f9891302640 1 -- 172.21.15.72:0/3357632910 learned_addr learned my addr 172.21.15.72:0/3357632910 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252217285 0 0) 0x7f988c06ee00 con 0x7f988c0ceb80 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988bfff640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 0x7f988c0cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f987c003520 con 0x7f988c0ceb80 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3733622155 0 0) 0x7f987c003520 con 0x7f988c0ceb80 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 msgr2=0x7f988c0cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 0x7f988c0cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 >> v1:172.21.15.146:6789/0 conn(0x7f988c0d0060 legacy=0x7f988c0d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c06f6a0 con 0x7f988c0ceb80 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f98840019f0 con 0x7f988c0ceb80 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 >> v1:172.21.15.72:6789/0 conn(0x7f988c0ceb80 legacy=0x7f988c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f987c0039e0 0x7f987c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c004800 0x7f987c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 0x7f987c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c06f3c0 con 0x7f987c0039e0 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c00bb30 con 0x7f987c004800 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c00bd30 con 0x7f988c06be00 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9890b01640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f987c0039e0 0x7f987c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988bfff640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c004800 0x7f987c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9891302640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 0x7f987c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9890b01640 1 -- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 msgr2=0x7f987c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9890b01640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c06be00 0x7f987c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9890b01640 1 -- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c004800 msgr2=0x7f987c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9890b01640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c004800 0x7f987c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f9890b01640 1 -- 172.21.15.72:0/3357632910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c003520 con 0x7f987c0039e0 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.182+0000 7f988bfff640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c004800 0x7f987c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.183+0000 7f9890b01640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f987c0039e0 0x7f987c003e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9880001720 tx=0x7f987c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e95b476b2ab1b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.183+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880001370 con 0x7f987c0039e0 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.183+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98800052d0 con 0x7f987c0039e0 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.183+0000 7f988b7fe640 1 -- 172.21.15.72:0/3357632910 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880003830 con 0x7f987c0039e0 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9892d65640 1 -- 172.21.15.72:0/3357632910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f987c0039e0 msgr2=0x7f987c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9892d65640 1 --2- 172.21.15.72:0/3357632910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f987c0039e0 0x7f987c003e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9880001720 tx=0x7f987c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9891302640 1 -- 172.21.15.72:0/3357632910 reap_dead start 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9892d65640 1 -- 172.21.15.72:0/3357632910 shutdown_connections 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9892d65640 1 -- 172.21.15.72:0/3357632910 >> 172.21.15.72:0/3357632910 conn(0x7f988c0b35b0 msgr2=0x7f988c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9892d65640 1 -- 172.21.15.72:0/3357632910 shutdown_connections 2024-03-06T07:21:49.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.184+0000 7f9892d65640 1 -- 172.21.15.72:0/3357632910 wait complete. 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.185+0000 7f9892d65640 1 Processor -- start 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.185+0000 7f9892d65640 1 -- start start 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9892d65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c0039e0 0x7f988c0c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9892d65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 0x7f988c0b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9892d65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c0c7600 0x7f988c0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9892d65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c15e3f0 con 0x7f988c0cab20 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9892d65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c15e270 con 0x7f987c0039e0 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9892d65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0cc870 con 0x7f988c0c7600 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 0x7f988c0b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 0x7f988c0b7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34908/0 (socket says 172.21.15.72:34908) 2024-03-06T07:21:49.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 -- 172.21.15.72:0/3138629950 learned_addr learned my addr 172.21.15.72:0/3138629950 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9890b01640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c0039e0 0x7f988c0c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9891302640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c0c7600 0x7f988c0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 -- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c0c7600 msgr2=0x7f988c0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c0c7600 0x7f988c0c7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 -- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c0039e0 msgr2=0x7f988c0c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c0039e0 0x7f988c0c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 -- 172.21.15.72:0/3138629950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0c6f40 con 0x7f988c0cab20 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f9890b01640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c0039e0 0x7f988c0c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f988bfff640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 0x7f988c0b7100 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9878011510 tx=0x7f98780119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878015870 con 0x7f988c0cab20 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.186+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9878015a10 con 0x7f988c0cab20 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.187+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c0c66a0 con 0x7f988c0cab20 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.187+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878015dc0 con 0x7f988c0cab20 2024-03-06T07:21:49.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.187+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c0c6c60 con 0x7f988c0cab20 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.188+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c04f330 con 0x7f988c0cab20 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.188+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f987801d400 con 0x7f988c0cab20 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.188+0000 7f98897fa640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f987006b4d0 0x7f987006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.188+0000 7f9890b01640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f987006b4d0 0x7f987006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.188+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f987808a990 con 0x7f988c0cab20 2024-03-06T07:21:49.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.189+0000 7f9890b01640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f987006b4d0 0x7f987006d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9880033000 tx=0x7f9880009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:49.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.193+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9878019030 con 0x7f988c0cab20 2024-03-06T07:21:49.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:49.398+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f988c0b0f30 con 0x7f988c0cab20 2024-03-06T07:21:49.405 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:49.407 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:21:50.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.067+0000 7f98897fa640 1 -- 172.21.15.72:0/3138629950 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9878059870 con 0x7f988c0cab20 2024-03-06T07:21:50.069 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 2 and data pool 3 2024-03-06T07:21:50.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.071+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f987006b4d0 msgr2=0x7f987006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.071+0000 7f9892d65640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f987006b4d0 0x7f987006d990 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9880033000 tx=0x7f9880009030 comp rx=0 tx=0).stop 2024-03-06T07:21:50.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.071+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 msgr2=0x7f988c0b7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.071+0000 7f9892d65640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 0x7f988c0b7100 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9878011510 tx=0x7f98780119d0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.072+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 shutdown_connections 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.072+0000 7f9892d65640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f987006b4d0 0x7f987006d990 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.072+0000 7f9892d65640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f988c0c7600 0x7f988c0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.072+0000 7f9892d65640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c0cab20 0x7f988c0b7100 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.072+0000 7f9892d65640 1 --2- 172.21.15.72:0/3138629950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f987c0039e0 0x7f988c0c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.072+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 >> 172.21.15.72:0/3138629950 conn(0x7f988c0b35b0 msgr2=0x7f988c0b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.073+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 shutdown_connections 2024-03-06T07:21:50.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.073+0000 7f9892d65640 1 -- 172.21.15.72:0/3138629950 wait complete. 2024-03-06T07:21:50.096 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:50.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 Processor -- start 2024-03-06T07:21:50.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 -- start start 2024-03-06T07:21:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f64cc13dc10 0x7f64cc0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f64cc15bf30 0x7f64cc0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc15e850 0x7f64cc15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc0cf3f0 con 0x7f64cc13dc10 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc15b210 con 0x7f64cc15bf30 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.197+0000 7f64d4713640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15f580 con 0x7f64cc15e850 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f64cc13dc10 0x7f64cc0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f64cc13dc10 0x7f64cc0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34916/0 (socket says 172.21.15.72:34916) 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 -- 172.21.15.72:0/1897451716 learned_addr learned my addr 172.21.15.72:0/1897451716 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d2cb0640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc15e850 0x7f64cc15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d1cae640 1 --2- 172.21.15.72:0/1897451716 >> v2:172.21.15.146:3300/0 conn(0x7f64cc15bf30 0x7f64cc0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 -- 172.21.15.72:0/1897451716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc15e850 msgr2=0x7f64cc15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc15e850 0x7f64cc15eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 -- 172.21.15.72:0/1897451716 >> v2:172.21.15.146:3300/0 conn(0x7f64cc15bf30 msgr2=0x7f64cc0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 --2- 172.21.15.72:0/1897451716 >> v2:172.21.15.146:3300/0 conn(0x7f64cc15bf30 0x7f64cc0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d24af640 1 -- 172.21.15.72:0/1897451716 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc15b450 con 0x7f64cc13dc10 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d2cb0640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc15e850 0x7f64cc15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.198+0000 7f64d1cae640 1 --2- 172.21.15.72:0/1897451716 >> v2:172.21.15.146:3300/0 conn(0x7f64cc15bf30 0x7f64cc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d24af640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc13dc10 0x7f64cc0b2950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f64cc06c790 tx=0x7f64c802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754b6a857f50dba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d14ad640 1 -- 172.21.15.72:0/1897451716 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c80317d0 con 0x7f64cc13dc10 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d14ad640 1 -- 172.21.15.72:0/1897451716 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8037070 con 0x7f64cc13dc10 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d14ad640 1 -- 172.21.15.72:0/1897451716 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8031b90 con 0x7f64cc13dc10 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 -- 172.21.15.72:0/1897451716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc13dc10 msgr2=0x7f64cc0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc13dc10 0x7f64cc0b2950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f64cc06c790 tx=0x7f64c802b870 comp rx=0 tx=0).stop 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 -- 172.21.15.72:0/1897451716 shutdown_connections 2024-03-06T07:21:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc15e850 0x7f64cc15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 --2- 172.21.15.72:0/1897451716 >> v2:172.21.15.146:3300/0 conn(0x7f64cc15bf30 0x7f64cc0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 --2- 172.21.15.72:0/1897451716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc13dc10 0x7f64cc0b2950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 -- 172.21.15.72:0/1897451716 >> 172.21.15.72:0/1897451716 conn(0x7f64cc0cf8a0 msgr2=0x7f64cc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 -- 172.21.15.72:0/1897451716 shutdown_connections 2024-03-06T07:21:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.199+0000 7f64d4713640 1 -- 172.21.15.72:0/1897451716 wait complete. 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.200+0000 7f64d4713640 1 Processor -- start 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.200+0000 7f64d4713640 1 -- start start 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d4713640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc13dc10 0x7f64cc13fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d4713640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64cc15bf30 0x7f64cc13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d4713640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 0x7f64cc159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d4713640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15f1e0 con 0x7f64cc15e850 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d4713640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc13dff0 con 0x7f64cc15bf30 2024-03-06T07:21:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d4713640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc15f360 con 0x7f64cc13dc10 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d2cb0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 0x7f64cc159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d2cb0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 0x7f64cc159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34932/0 (socket says 172.21.15.72:34932) 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d2cb0640 1 -- 172.21.15.72:0/4286077306 learned_addr learned my addr 172.21.15.72:0/4286077306 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d1cae640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64cc15bf30 0x7f64cc13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.201+0000 7f64d24af640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc13dc10 0x7f64cc13fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d2cb0640 1 -- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc13dc10 msgr2=0x7f64cc13fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d2cb0640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc13dc10 0x7f64cc13fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d2cb0640 1 -- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64cc15bf30 msgr2=0x7f64cc13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d2cb0640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64cc15bf30 0x7f64cc13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d2cb0640 1 -- 172.21.15.72:0/4286077306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc13af40 con 0x7f64cc15e850 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d2cb0640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 0x7f64cc159fa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64c400cef0 tx=0x7f64c400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c401a070 con 0x7f64cc15e850 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c400d860 con 0x7f64cc15e850 2024-03-06T07:21:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c400dc50 con 0x7f64cc15e850 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc13b220 con 0x7f64cc15e850 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.202+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc13b7e0 con 0x7f64cc15e850 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.203+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc04f330 con 0x7f64cc15e850 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.203+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64c401d070 con 0x7f64cc15e850 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.204+0000 7f64beffd640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f64b006b4d0 0x7f64b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.204+0000 7f64d24af640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f64b006b4d0 0x7f64b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.204+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64c408a9c0 con 0x7f64cc15e850 2024-03-06T07:21:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.204+0000 7f64d24af640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f64b006b4d0 0x7f64b006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64c8036000 tx=0x7f64c803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.208+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64c408ce20 con 0x7f64cc15e850 2024-03-06T07:21:50.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.382+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64cc06eb00 con 0x7f64cc15e850 2024-03-06T07:21:50.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.384+0000 7f64beffd640 1 -- 172.21.15.72:0/4286077306 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f64c40592b0 con 0x7f64cc15e850 2024-03-06T07:21:50.386 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:50.386 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:49.400261+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f64b006b4d0 msgr2=0x7f64b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f64b006b4d0 0x7f64b006d990 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64c8036000 tx=0x7f64c803d040 comp rx=0 tx=0).stop 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 msgr2=0x7f64cc159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 0x7f64cc159fa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64c400cef0 tx=0x7f64c400a710 comp rx=0 tx=0).stop 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 shutdown_connections 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f64b006b4d0 0x7f64b006d990 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64cc15e850 0x7f64cc159fa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64cc15bf30 0x7f64cc13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 --2- 172.21.15.72:0/4286077306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64cc13dc10 0x7f64cc13fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 >> 172.21.15.72:0/4286077306 conn(0x7f64cc0cf8a0 msgr2=0x7f64cc0b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 shutdown_connections 2024-03-06T07:21:50.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.386+0000 7f64d4713640 1 -- 172.21.15.72:0/4286077306 wait complete. 2024-03-06T07:21:50.399 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:50.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.522+0000 7f17a4a08640 1 Processor -- start 2024-03-06T07:21:50.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.522+0000 7f17a4a08640 1 -- start start 2024-03-06T07:21:50.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f17a4a08640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a015ddf0 0x7f17a00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f17a4a08640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f17a015e1d0 0x7f17a00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f17a4a08640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a00d06d0 0x7f17a00d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f17a4a08640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a00d1080 con 0x7f17a015ddf0 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f17a4a08640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a00d27c0 con 0x7f17a015e1d0 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f17a4a08640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00d18d0 con 0x7f17a00d06d0 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a015ddf0 0x7f17a00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a015ddf0 0x7f17a00cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34948/0 (socket says 172.21.15.72:34948) 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 -- 172.21.15.72:0/2584673780 learned_addr learned my addr 172.21.15.72:0/2584673780 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 -- 172.21.15.72:0/2584673780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a00d06d0 msgr2=0x7f17a00d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 --2- 172.21.15.72:0/2584673780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a00d06d0 0x7f17a00d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 -- 172.21.15.72:0/2584673780 >> v2:172.21.15.146:3300/0 conn(0x7f17a015e1d0 msgr2=0x7f17a00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f1797fff640 1 --2- 172.21.15.72:0/2584673780 >> v2:172.21.15.146:3300/0 conn(0x7f17a015e1d0 0x7f17a00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 --2- 172.21.15.72:0/2584673780 >> v2:172.21.15.146:3300/0 conn(0x7f17a015e1d0 0x7f17a00cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.523+0000 7f179e59c640 1 -- 172.21.15.72:0/2584673780 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a00d1bb0 con 0x7f17a015ddf0 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.524+0000 7f179e59c640 1 --2- 172.21.15.72:0/2584673780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a015ddf0 0x7f17a00cf440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f17a00d39f0 tx=0x7f179002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19215a1ed4c24564 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.524+0000 7f179dd9b640 1 -- 172.21.15.72:0/2584673780 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790035020 con 0x7f17a015ddf0 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.524+0000 7f179dd9b640 1 -- 172.21.15.72:0/2584673780 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1790031870 con 0x7f17a015ddf0 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.524+0000 7f179dd9b640 1 -- 172.21.15.72:0/2584673780 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790031c40 con 0x7f17a015ddf0 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 -- 172.21.15.72:0/2584673780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a015ddf0 msgr2=0x7f17a00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 --2- 172.21.15.72:0/2584673780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a015ddf0 0x7f17a00cf440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f17a00d39f0 tx=0x7f179002b910 comp rx=0 tx=0).stop 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 -- 172.21.15.72:0/2584673780 shutdown_connections 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 --2- 172.21.15.72:0/2584673780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a00d06d0 0x7f17a00d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 --2- 172.21.15.72:0/2584673780 >> v2:172.21.15.146:3300/0 conn(0x7f17a015e1d0 0x7f17a00cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 --2- 172.21.15.72:0/2584673780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a015ddf0 0x7f17a00cf440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 -- 172.21.15.72:0/2584673780 >> 172.21.15.72:0/2584673780 conn(0x7f17a009cff0 msgr2=0x7f17a00d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 -- 172.21.15.72:0/2584673780 shutdown_connections 2024-03-06T07:21:50.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.525+0000 7f17a4a08640 1 -- 172.21.15.72:0/2584673780 wait complete. 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.526+0000 7f17a4a08640 1 Processor -- start 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.526+0000 7f17a4a08640 1 -- start start 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 0x7f17a009da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17a015ddf0 0x7f17a015aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a015e1d0 0x7f17a00cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00d2450 con 0x7f17a00d06d0 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00d2170 con 0x7f17a015ddf0 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00d2c50 con 0x7f17a015e1d0 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 0x7f17a009da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 0x7f17a009da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34960/0 (socket says 172.21.15.72:34960) 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 -- 172.21.15.72:0/3854182939 learned_addr learned my addr 172.21.15.72:0/3854182939 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:50.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f1797fff640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17a015ddf0 0x7f17a015aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179ed9d640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a015e1d0 0x7f17a00cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 -- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a015e1d0 msgr2=0x7f17a00cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a015e1d0 0x7f17a00cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 -- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17a015ddf0 msgr2=0x7f17a015aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17a015ddf0 0x7f17a015aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 -- 172.21.15.72:0/3854182939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a015b020 con 0x7f17a00d06d0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f179e59c640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 0x7f17a009da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f179002b950 tx=0x7f17900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790040070 con 0x7f17a00d06d0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179003a420 con 0x7f17a00d06d0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a00c8850 con 0x7f17a00d06d0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790035040 con 0x7f17a00d06d0 2024-03-06T07:21:50.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.527+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a00c8e10 con 0x7f17a00d06d0 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.528+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a004f330 con 0x7f17a00d06d0 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.529+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1790038020 con 0x7f17a00d06d0 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.529+0000 7f1796ffd640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f178006b4d0 0x7f178006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.529+0000 7f1797fff640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f178006b4d0 0x7f178006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.529+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17900ac0e0 con 0x7f17a00d06d0 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.530+0000 7f1797fff640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f178006b4d0 0x7f178006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17a0160180 tx=0x7f1788008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.534+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f179007a9a0 con 0x7f17a00d06d0 2024-03-06T07:21:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.710+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a00747a0 con 0x7f17a00d06d0 2024-03-06T07:21:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.711+0000 7f1796ffd640 1 -- 172.21.15.72:0/3854182939 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f179007ea40 con 0x7f17a00d06d0 2024-03-06T07:21:50.714 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:50.714 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:49.400261+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:50.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f178006b4d0 msgr2=0x7f178006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f178006b4d0 0x7f178006d990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f17a0160180 tx=0x7f1788008040 comp rx=0 tx=0).stop 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 msgr2=0x7f17a009da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 0x7f17a009da60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f179002b950 tx=0x7f17900046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 shutdown_connections 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f178006b4d0 0x7f178006d990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17a015e1d0 0x7f17a00cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17a015ddf0 0x7f17a015aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 --2- 172.21.15.72:0/3854182939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d06d0 0x7f17a009da60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 >> 172.21.15.72:0/3854182939 conn(0x7f17a009cff0 msgr2=0x7f17a015bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 shutdown_connections 2024-03-06T07:21:50.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.713+0000 7f17a4a08640 1 -- 172.21.15.72:0/3854182939 wait complete. 2024-03-06T07:21:50.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:21:50.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.828+0000 7f83a00af640 1 Processor -- start 2024-03-06T07:21:50.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.828+0000 7f83a00af640 1 -- start start 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f83a00af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f83980d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f83a00af640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f83980b0e30 0x7f83980b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f83a00af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83981618b0 con 0x7f83981614d0 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f83a00af640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83980d4cc0 con 0x7f83980b0e30 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f83a00af640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83980cf400 con 0x7f83980d4860 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f839de4b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f83981614d0 0x7f83980d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41792/0 (socket says 172.21.15.72:41792) 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f839de4b640 1 -- 172.21.15.72:0/2287739906 learned_addr learned my addr 172.21.15.72:0/2287739906 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f839e64c640 1 --2- 172.21.15.72:0/2287739906 >> v2:172.21.15.146:3300/0 conn(0x7f83980b0e30 0x7f83980b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f839d64a640 1 --2- 172.21.15.72:0/2287739906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f83980d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f839ce49640 1 -- 172.21.15.72:0/2287739906 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286468172 0 0) 0x7f83981618b0 con 0x7f83981614d0 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.829+0000 7f839ce49640 1 -- 172.21.15.72:0/2287739906 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8384003520 con 0x7f83981614d0 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839ce49640 1 -- 172.21.15.72:0/2287739906 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3314640252 0 0) 0x7f8384003520 con 0x7f83981614d0 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839d64a640 1 -- 172.21.15.72:0/2287739906 >> v2:172.21.15.146:3300/0 conn(0x7f83980b0e30 msgr2=0x7f83980b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839d64a640 1 --2- 172.21.15.72:0/2287739906 >> v2:172.21.15.146:3300/0 conn(0x7f83980b0e30 0x7f83980b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839d64a640 1 -- 172.21.15.72:0/2287739906 >> v1:172.21.15.72:6789/0 conn(0x7f83981614d0 legacy=0x7f83980d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839d64a640 1 -- 172.21.15.72:0/2287739906 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980cf6a0 con 0x7f83980d4860 2024-03-06T07:21:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839e64c640 1 --2- 172.21.15.72:0/2287739906 >> v2:172.21.15.146:3300/0 conn(0x7f83980b0e30 0x7f83980b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:50.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.830+0000 7f839d64a640 1 --2- 172.21.15.72:0/2287739906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f83980d2020 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83880097a0 tx=0x7f838802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b3b184f7cbbdce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f839ce49640 1 -- 172.21.15.72:0/2287739906 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388037070 con 0x7f83980d4860 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f839ce49640 1 -- 172.21.15.72:0/2287739906 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8388030510 con 0x7f83980d4860 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f839ce49640 1 -- 172.21.15.72:0/2287739906 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394001670 con 0x7f83980d4860 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 -- 172.21.15.72:0/2287739906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 msgr2=0x7f83980d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 --2- 172.21.15.72:0/2287739906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f83980d2020 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83880097a0 tx=0x7f838802b790 comp rx=0 tx=0).stop 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 -- 172.21.15.72:0/2287739906 shutdown_connections 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 --2- 172.21.15.72:0/2287739906 >> v2:172.21.15.146:3300/0 conn(0x7f83980b0e30 0x7f83980b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 --2- 172.21.15.72:0/2287739906 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f83980d2020 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 -- 172.21.15.72:0/2287739906 >> 172.21.15.72:0/2287739906 conn(0x7f83980d3d40 msgr2=0x7f83980d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 -- 172.21.15.72:0/2287739906 shutdown_connections 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.831+0000 7f83a00af640 1 -- 172.21.15.72:0/2287739906 wait complete. 2024-03-06T07:21:50.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.832+0000 7f83a00af640 1 Processor -- start 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.832+0000 7f83a00af640 1 -- start start 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f83a00af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83980b0e30 0x7f83980b7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f83a00af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f839815a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f83a00af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 0x7f83980cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f83a00af640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980b0910 con 0x7f83981614d0 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f83a00af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83980cfc60 con 0x7f83980b0e30 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f83a00af640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83980d2560 con 0x7f83980d4860 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 0x7f83980cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 0x7f83980cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34974/0 (socket says 172.21.15.72:34974) 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 -- 172.21.15.72:0/3086975966 learned_addr learned my addr 172.21.15.72:0/3086975966 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839de4b640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83980b0e30 0x7f83980b7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839d64a640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f839815a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 -- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 msgr2=0x7f839815a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f839815a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 -- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83980b0e30 msgr2=0x7f83980b7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83980b0e30 0x7f83980b7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839e64c640 1 -- 172.21.15.72:0/3086975966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839815ac40 con 0x7f83981614d0 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.833+0000 7f839d64a640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f839815a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.834+0000 7f839e64c640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 0x7f83980cce20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f839000efc0 tx=0x7f8390009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.834+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390018030 con 0x7f83981614d0 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.834+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f839000cde0 con 0x7f83981614d0 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.834+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83980c8100 con 0x7f83981614d0 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.834+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390003bb0 con 0x7f83981614d0 2024-03-06T07:21:50.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.834+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83980c86c0 con 0x7f83981614d0 2024-03-06T07:21:50.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.835+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8398068ff0 con 0x7f83981614d0 2024-03-06T07:21:50.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.835+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8390005d10 con 0x7f83981614d0 2024-03-06T07:21:50.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.835+0000 7f838e7fc640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f837c06b4d0 0x7f837c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:50.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.836+0000 7f839de4b640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f837c06b4d0 0x7f837c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:50.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.836+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f839008ab80 con 0x7f83981614d0 2024-03-06T07:21:50.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.836+0000 7f839de4b640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f837c06b4d0 0x7f837c06d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8394001320 tx=0x7f83940012b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:50.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:50.840+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8390022070 con 0x7f83981614d0 2024-03-06T07:21:51.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:51.064+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f83980d0040 con 0x7f83981614d0 2024-03-06T07:21:52.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.057+0000 7f838e7fc640 1 -- 172.21.15.72:0/3086975966 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f839815ac40 con 0x7f83981614d0 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f837c06b4d0 msgr2=0x7f837c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f837c06b4d0 0x7f837c06d990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8394001320 tx=0x7f83940012b0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 msgr2=0x7f83980cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 0x7f83980cce20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f839000efc0 tx=0x7f8390009f90 comp rx=0 tx=0).stop 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 shutdown_connections 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f837c06b4d0 0x7f837c06d990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83981614d0 0x7f83980cce20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83980d4860 0x7f839815a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 --2- 172.21.15.72:0/3086975966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83980b0e30 0x7f83980b7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 >> 172.21.15.72:0/3086975966 conn(0x7f83980d3d40 msgr2=0x7f839806bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 shutdown_connections 2024-03-06T07:21:52.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.059+0000 7f83a00af640 1 -- 172.21.15.72:0/3086975966 wait complete. 2024-03-06T07:21:52.072 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:21:52.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:21:52.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.173+0000 7ff122913640 1 Processor -- start 2024-03-06T07:21:52.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.173+0000 7ff122913640 1 -- start start 2024-03-06T07:21:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff122913640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff11c15c050 0x7ff11c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff122913640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c0d0af0 0x7ff11c0cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff122913640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11c0cd4b0 con 0x7ff11c06fb40 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff122913640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11c06ffb0 con 0x7ff11c15c050 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff122913640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c0cd790 con 0x7ff11c0d0af0 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff11c15c050 0x7ff11c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff11c06fb40 0x7ff11c0cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41800/0 (socket says 172.21.15.72:41800) 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11b7fe640 1 -- 172.21.15.72:0/230089704 learned_addr learned my addr 172.21.15.72:0/230089704 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11bfff640 1 -- 172.21.15.72:0/230089704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c0d0af0 msgr2=0x7ff11c0cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11bfff640 1 --2- 172.21.15.72:0/230089704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c0d0af0 0x7ff11c0cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11bfff640 1 -- 172.21.15.72:0/230089704 >> v1:172.21.15.72:6789/0 conn(0x7ff11c06fb40 legacy=0x7ff11c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11bfff640 1 -- 172.21.15.72:0/230089704 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0cda70 con 0x7ff11c15c050 2024-03-06T07:21:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.174+0000 7ff11affd640 1 -- 172.21.15.72:0/230089704 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138704860 0 0) 0x7ff11c0cd4b0 con 0x7ff11c06fb40 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.175+0000 7ff11bfff640 1 --2- 172.21.15.72:0/230089704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c15c050 0x7ff11c06f600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0097a0 tx=0x7ff10c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2bf5e0b9894eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.175+0000 7ff11affd640 1 -- 172.21.15.72:0/230089704 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c0315a0 con 0x7ff11c15c050 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.175+0000 7ff11affd640 1 -- 172.21.15.72:0/230089704 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10c037070 con 0x7ff11c15c050 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.175+0000 7ff11affd640 1 -- 172.21.15.72:0/230089704 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10c031960 con 0x7ff11c15c050 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 -- 172.21.15.72:0/230089704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c15c050 msgr2=0x7ff11c06f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 --2- 172.21.15.72:0/230089704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c15c050 0x7ff11c06f600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff10c0097a0 tx=0x7ff10c02b840 comp rx=0 tx=0).stop 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 -- 172.21.15.72:0/230089704 shutdown_connections 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 --2- 172.21.15.72:0/230089704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c0d0af0 0x7ff11c0cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 --2- 172.21.15.72:0/230089704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c15c050 0x7ff11c06f600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 -- 172.21.15.72:0/230089704 >> 172.21.15.72:0/230089704 conn(0x7ff11c0b3800 msgr2=0x7ff11c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:52.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 -- 172.21.15.72:0/230089704 shutdown_connections 2024-03-06T07:21:52.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.176+0000 7ff122913640 1 -- 172.21.15.72:0/230089704 wait complete. 2024-03-06T07:21:52.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.177+0000 7ff122913640 1 Processor -- start 2024-03-06T07:21:52.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.177+0000 7ff122913640 1 -- start start 2024-03-06T07:21:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c06fb40 0x7ff11c0b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 0x7ff11c14d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c15c050 0x7ff11c0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c165c30 con 0x7ff11c0d0af0 2024-03-06T07:21:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c165ab0 con 0x7ff11c15c050 2024-03-06T07:21:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11c165db0 con 0x7ff11c06fb40 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 0x7ff11c14d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 0x7ff11c14d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34990/0 (socket says 172.21.15.72:34990) 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 -- 172.21.15.72:0/218636687 learned_addr learned my addr 172.21.15.72:0/218636687 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11bfff640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c06fb40 0x7ff11c0b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff120eb0640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c15c050 0x7ff11c0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 -- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c06fb40 msgr2=0x7ff11c0b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c06fb40 0x7ff11c0b6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 -- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c15c050 msgr2=0x7ff11c0b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c15c050 0x7ff11c0b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 -- 172.21.15.72:0/218636687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0b5d80 con 0x7ff11c0d0af0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff11b7fe640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 0x7ff11c14d4f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff10400c460 tx=0x7ff10400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104018070 con 0x7ff11c0d0af0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff104007500 con 0x7ff11c0d0af0 2024-03-06T07:21:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11c1590e0 con 0x7ff11c0d0af0 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff104004a20 con 0x7ff11c0d0af0 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.178+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11c0b6300 con 0x7ff11c0d0af0 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.179+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11c09fe60 con 0x7ff11c0d0af0 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.180+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff10401b070 con 0x7ff11c0d0af0 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.180+0000 7ff118ff9640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff0fc06b4d0 0x7ff0fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.180+0000 7ff11bfff640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff0fc06b4d0 0x7ff0fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.180+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1040893f0 con 0x7ff11c0d0af0 2024-03-06T07:21:52.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.181+0000 7ff11bfff640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff0fc06b4d0 0x7ff0fc06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff10c036000 tx=0x7ff10c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:52.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.185+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff104057c20 con 0x7ff11c0d0af0 2024-03-06T07:21:52.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:52.395+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff11c0ce820 con 0x7ff11c0d0af0 2024-03-06T07:21:53.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.069+0000 7ff118ff9640 1 -- 172.21.15.72:0/218636687 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff10405bcc0 con 0x7ff11c0d0af0 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff0fc06b4d0 msgr2=0x7ff0fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff0fc06b4d0 0x7ff0fc06d990 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff10c036000 tx=0x7ff10c008030 comp rx=0 tx=0).stop 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 msgr2=0x7ff11c14d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 0x7ff11c14d4f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff10400c460 tx=0x7ff10400c920 comp rx=0 tx=0).stop 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 shutdown_connections 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff0fc06b4d0 0x7ff0fc06d990 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff11c15c050 0x7ff11c0b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff11c0d0af0 0x7ff11c14d4f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 --2- 172.21.15.72:0/218636687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff11c06fb40 0x7ff11c0b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 >> 172.21.15.72:0/218636687 conn(0x7ff11c0b3800 msgr2=0x7ff11c0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 shutdown_connections 2024-03-06T07:21:53.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.071+0000 7ff122913640 1 -- 172.21.15.72:0/218636687 wait complete. 2024-03-06T07:21:53.087 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:21:53.087 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.184+0000 7f73305d7640 1 Processor -- start 2024-03-06T07:21:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.184+0000 7f73305d7640 1 -- start start 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f73305d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f73280d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f73305d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73280cd8c0 0x7f73280d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f73305d7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7328161980 con 0x7f73280cfff0 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f73305d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7328161c60 con 0x7f73280cd8c0 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f73305d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7328161f40 con 0x7f732815bb20 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f732eb74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73280cd8c0 0x7f73280d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f732db72640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f73280cfff0 0x7f73280ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49084/0 (socket says 172.21.15.72:49084) 2024-03-06T07:21:53.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f732db72640 1 -- 172.21.15.72:0/707822382 learned_addr learned my addr 172.21.15.72:0/707822382 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f732eb74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73280cd8c0 0x7f73280d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54218/0 (socket says 172.21.15.72:54218) 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.185+0000 7f732e373640 1 --2- 172.21.15.72:0/707822382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f73280d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.186+0000 7f732eb74640 1 -- 172.21.15.72:0/707822382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 msgr2=0x7f73280d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.186+0000 7f732eb74640 1 --2- 172.21.15.72:0/707822382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f73280d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.186+0000 7f732eb74640 1 -- 172.21.15.72:0/707822382 >> v1:172.21.15.72:6789/0 conn(0x7f73280cfff0 legacy=0x7f73280ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.186+0000 7f732eb74640 1 -- 172.21.15.72:0/707822382 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328162220 con 0x7f73280cd8c0 2024-03-06T07:21:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.186+0000 7f732d371640 1 -- 172.21.15.72:0/707822382 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993862246 0 0) 0x7f7328161980 con 0x7f73280cfff0 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f732eb74640 1 --2- 172.21.15.72:0/707822382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f73280d0470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f73200097a0 tx=0x7f732002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfe63270e495318 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f732d371640 1 -- 172.21.15.72:0/707822382 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73200315a0 con 0x7f73280cd8c0 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f732d371640 1 -- 172.21.15.72:0/707822382 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7320037070 con 0x7f73280cd8c0 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f732d371640 1 -- 172.21.15.72:0/707822382 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320031960 con 0x7f73280cd8c0 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f73305d7640 1 -- 172.21.15.72:0/707822382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 msgr2=0x7f73280d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f73305d7640 1 --2- 172.21.15.72:0/707822382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f73280d0470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f73200097a0 tx=0x7f732002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f73305d7640 1 -- 172.21.15.72:0/707822382 shutdown_connections 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f73305d7640 1 --2- 172.21.15.72:0/707822382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f73280d0470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f73305d7640 1 --2- 172.21.15.72:0/707822382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f73280d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.187+0000 7f73305d7640 1 -- 172.21.15.72:0/707822382 >> 172.21.15.72:0/707822382 conn(0x7f732809a9f0 msgr2=0x7f732809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.188+0000 7f73305d7640 1 -- 172.21.15.72:0/707822382 shutdown_connections 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.188+0000 7f73305d7640 1 -- 172.21.15.72:0/707822382 wait complete. 2024-03-06T07:21:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.189+0000 7f73305d7640 1 Processor -- start 2024-03-06T07:21:53.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.189+0000 7f73305d7640 1 -- start start 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f73305d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f7328156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f73305d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73280cfff0 0x7f732815d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f73305d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f732815d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f73305d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73280d24b0 con 0x7f73280cd8c0 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f73305d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73280d2330 con 0x7f73280cfff0 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f73305d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73280d1980 con 0x7f732815bb20 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f7328156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f7328156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54220/0 (socket says 172.21.15.72:54220) 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 -- 172.21.15.72:0/493790847 learned_addr learned my addr 172.21.15.72:0/493790847 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732eb74640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f732815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732db72640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73280cfff0 0x7f732815d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 -- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 msgr2=0x7f732815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f732815d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 -- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73280cfff0 msgr2=0x7f732815d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73280cfff0 0x7f732815d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 -- 172.21.15.72:0/493790847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328157480 con 0x7f73280cd8c0 2024-03-06T07:21:53.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732eb74640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f732815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:53.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f732e373640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f7328156f40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f732400efc0 tx=0x7f732400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324018020 con 0x7f73280cd8c0 2024-03-06T07:21:53.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.190+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7324014480 con 0x7f73280cd8c0 2024-03-06T07:21:53.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.191+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732815e910 con 0x7f73280cd8c0 2024-03-06T07:21:53.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.191+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7324014850 con 0x7f73280cd8c0 2024-03-06T07:21:53.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.191+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7328160fc0 con 0x7f73280cd8c0 2024-03-06T07:21:53.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.192+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732804f330 con 0x7f73280cd8c0 2024-03-06T07:21:53.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.192+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7324005d10 con 0x7f73280cd8c0 2024-03-06T07:21:53.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.192+0000 7f731effd640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.193+0000 7f732db72640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.193+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f732408ac60 con 0x7f73280cd8c0 2024-03-06T07:21:53.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.193+0000 7f732db72640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730c06b4d0 0x7f730c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f73280b0a10 tx=0x7f7318007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.197+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7324022070 con 0x7f73280cd8c0 2024-03-06T07:21:53.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.451+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7328160aa0 con 0x7f73280cd8c0 2024-03-06T07:21:53.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.451+0000 7f731effd640 1 -- 172.21.15.72:0/493790847 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f7324010040 con 0x7f73280cd8c0 2024-03-06T07:21:53.453 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:53.454 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:21:53.454 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:21:53.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730c06b4d0 msgr2=0x7f730c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730c06b4d0 0x7f730c06d990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f73280b0a10 tx=0x7f7318007c00 comp rx=0 tx=0).stop 2024-03-06T07:21:53.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 msgr2=0x7f7328156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f7328156f40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f732400efc0 tx=0x7f732400c490 comp rx=0 tx=0).stop 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 shutdown_connections 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730c06b4d0 0x7f730c06d990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f732815bb20 0x7f732815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73280cfff0 0x7f732815d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 --2- 172.21.15.72:0/493790847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73280cd8c0 0x7f7328156f40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 >> 172.21.15.72:0/493790847 conn(0x7f732809a9f0 msgr2=0x7f732815a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 shutdown_connections 2024-03-06T07:21:53.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.454+0000 7f73305d7640 1 -- 172.21.15.72:0/493790847 wait complete. 2024-03-06T07:21:53.463 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-03-06T07:21:53.461+0000 7f6d9d8fb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:21:53.467 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:53.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.564+0000 7f9b2a08d640 1 Processor -- start 2024-03-06T07:21:53.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.564+0000 7f9b2a08d640 1 -- start start 2024-03-06T07:21:53.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b2a08d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b240cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b2a08d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b240ceac0 con 0x7f9b241621c0 2024-03-06T07:21:53.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b2a08d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b240d2dc0 con 0x7f9b241625a0 2024-03-06T07:21:53.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b2a08d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240cf690 con 0x7f9b240d4920 2024-03-06T07:21:53.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b237fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9b241621c0 0x7f9b240d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49096/0 (socket says 172.21.15.72:49096) 2024-03-06T07:21:53.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b237fe640 1 -- 172.21.15.72:0/2536516559 learned_addr learned my addr 172.21.15.72:0/2536516559 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b22ffd640 1 --2- 172.21.15.72:0/2536516559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b240cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b227fc640 1 -- 172.21.15.72:0/2536516559 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998225433 0 0) 0x7f9b240ceac0 con 0x7f9b241621c0 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b227fc640 1 -- 172.21.15.72:0/2536516559 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b0c003520 con 0x7f9b241621c0 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b22ffd640 1 -- 172.21.15.72:0/2536516559 >> v1:172.21.15.146:6789/0 conn(0x7f9b241625a0 legacy=0x7f9b2415dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b227fc640 1 -- 172.21.15.72:0/2536516559 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1443711067 0 0) 0x7f9b0c003520 con 0x7f9b241621c0 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b22ffd640 1 -- 172.21.15.72:0/2536516559 >> v1:172.21.15.72:6789/0 conn(0x7f9b241621c0 legacy=0x7f9b240d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.566+0000 7f9b22ffd640 1 -- 172.21.15.72:0/2536516559 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240ced60 con 0x7f9b240d4920 2024-03-06T07:21:53.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.567+0000 7f9b22ffd640 1 --2- 172.21.15.72:0/2536516559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b240cfb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9b140097d0 tx=0x7f9b1402b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13880d0e44028761 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.567+0000 7f9b227fc640 1 -- 172.21.15.72:0/2536516559 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14037070 con 0x7f9b240d4920 2024-03-06T07:21:53.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.567+0000 7f9b227fc640 1 -- 172.21.15.72:0/2536516559 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b14030510 con 0x7f9b240d4920 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.567+0000 7f9b227fc640 1 -- 172.21.15.72:0/2536516559 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1402be60 con 0x7f9b240d4920 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 -- 172.21.15.72:0/2536516559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 msgr2=0x7f9b240cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/2536516559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b240cfb20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9b140097d0 tx=0x7f9b1402b660 comp rx=0 tx=0).stop 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 -- 172.21.15.72:0/2536516559 shutdown_connections 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/2536516559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b240cfb20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 -- 172.21.15.72:0/2536516559 >> 172.21.15.72:0/2536516559 conn(0x7f9b240d3d10 msgr2=0x7f9b240d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 -- 172.21.15.72:0/2536516559 shutdown_connections 2024-03-06T07:21:53.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.568+0000 7f9b2a08d640 1 -- 172.21.15.72:0/2536516559 wait complete. 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.569+0000 7f9b2a08d640 1 Processor -- start 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.569+0000 7f9b2a08d640 1 -- start start 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b2a08d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b24151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b2a08d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b241621c0 0x7f9b240b8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b2a08d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 0x7f9b2415aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b2a08d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2415e450 con 0x7f9b241625a0 2024-03-06T07:21:53.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b2a08d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b240cf320 con 0x7f9b241621c0 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b2a08d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2415e5d0 con 0x7f9b240d4920 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 0x7f9b2415aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 0x7f9b2415aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54222/0 (socket says 172.21.15.72:54222) 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 -- 172.21.15.72:0/1035413918 learned_addr learned my addr 172.21.15.72:0/1035413918 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 -- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 msgr2=0x7f9b24151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b24151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 -- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b241621c0 msgr2=0x7f9b240b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b22ffd640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b241621c0 0x7f9b240b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b241621c0 0x7f9b240b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.570+0000 7f9b23fff640 1 -- 172.21.15.72:0/1035413918 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b241f7dd0 con 0x7f9b241625a0 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.571+0000 7f9b23fff640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 0x7f9b2415aa60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00c8c0 tx=0x7f9b1c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.571+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c017070 con 0x7f9b241625a0 2024-03-06T07:21:53.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.571+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1c003da0 con 0x7f9b241625a0 2024-03-06T07:21:53.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.571+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1c005290 con 0x7f9b241625a0 2024-03-06T07:21:53.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.571+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b24152310 con 0x7f9b241625a0 2024-03-06T07:21:53.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.571+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b241f72f0 con 0x7f9b241625a0 2024-03-06T07:21:53.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.572+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2404f330 con 0x7f9b241625a0 2024-03-06T07:21:53.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.572+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b1c01a070 con 0x7f9b241625a0 2024-03-06T07:21:53.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.573+0000 7f9b0bfff640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1006b4d0 0x7f9b1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.573+0000 7f9b237fe640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1006b4d0 0x7f9b1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.573+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b1c089970 con 0x7f9b241625a0 2024-03-06T07:21:53.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.573+0000 7f9b237fe640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1006b4d0 0x7f9b1006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b18000c00 tx=0x7f9b18001270 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.577+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b1c058880 con 0x7f9b241625a0 2024-03-06T07:21:53.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.778+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b240d2680 con 0x7f9b241625a0 2024-03-06T07:21:53.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.779+0000 7f9b0bfff640 1 -- 172.21.15.72:0/1035413918 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f9b1c058180 con 0x7f9b241625a0 2024-03-06T07:21:53.781 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:53.781 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:53.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1006b4d0 msgr2=0x7f9b1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1006b4d0 0x7f9b1006d990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b18000c00 tx=0x7f9b18001270 comp rx=0 tx=0).stop 2024-03-06T07:21:53.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 msgr2=0x7f9b2415aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 0x7f9b2415aa60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9b1c00c8c0 tx=0x7f9b1c00cd80 comp rx=0 tx=0).stop 2024-03-06T07:21:53.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 shutdown_connections 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1006b4d0 0x7f9b1006d990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b241625a0 0x7f9b2415aa60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b241621c0 0x7f9b240b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 --2- 172.21.15.72:0/1035413918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b240d4920 0x7f9b24151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 >> 172.21.15.72:0/1035413918 conn(0x7f9b240d3d10 msgr2=0x7f9b2415ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 shutdown_connections 2024-03-06T07:21:53.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.781+0000 7f9b2a08d640 1 -- 172.21.15.72:0/1035413918 wait complete. 2024-03-06T07:21:53.797 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:21:53.798 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:53.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.895+0000 7efd6f2c9640 1 Processor -- start 2024-03-06T07:21:53.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.895+0000 7efd6f2c9640 1 -- start start 2024-03-06T07:21:53.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6f2c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd680ce450 0x7efd680d2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6f2c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd6806f380 0x7efd6806f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6f2c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd680cd370 con 0x7efd680cf0a0 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6f2c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680cd650 con 0x7efd680ce450 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6f2c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd680cd930 con 0x7efd6806f380 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6d065640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd680ce450 0x7efd680d2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6c864640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efd680cf0a0 0x7efd680ce830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49108/0 (socket says 172.21.15.72:49108) 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.896+0000 7efd6c864640 1 -- 172.21.15.72:0/409981964 learned_addr learned my addr 172.21.15.72:0/409981964 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d065640 1 -- 172.21.15.72:0/409981964 >> v2:172.21.15.146:3300/0 conn(0x7efd6806f380 msgr2=0x7efd6806f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d866640 1 --2- 172.21.15.72:0/409981964 >> v2:172.21.15.146:3300/0 conn(0x7efd6806f380 0x7efd6806f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d065640 1 --2- 172.21.15.72:0/409981964 >> v2:172.21.15.146:3300/0 conn(0x7efd6806f380 0x7efd6806f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d065640 1 -- 172.21.15.72:0/409981964 >> v1:172.21.15.72:6789/0 conn(0x7efd680cf0a0 legacy=0x7efd680ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d065640 1 -- 172.21.15.72:0/409981964 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6806eb80 con 0x7efd680ce450 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d866640 1 --2- 172.21.15.72:0/409981964 >> v2:172.21.15.146:3300/0 conn(0x7efd6806f380 0x7efd6806f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:53.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd6d065640 1 --2- 172.21.15.72:0/409981964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680ce450 0x7efd680d2580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd68074e90 tx=0x7efd6402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa37719b9365d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.897+0000 7efd5ffff640 1 -- 172.21.15.72:0/409981964 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64035020 con 0x7efd680ce450 2024-03-06T07:21:53.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd5ffff640 1 -- 172.21.15.72:0/409981964 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd64031830 con 0x7efd680ce450 2024-03-06T07:21:53.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd5ffff640 1 -- 172.21.15.72:0/409981964 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd64031c00 con 0x7efd680ce450 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 -- 172.21.15.72:0/409981964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680ce450 msgr2=0x7efd680d2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/409981964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680ce450 0x7efd680d2580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd68074e90 tx=0x7efd6402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 -- 172.21.15.72:0/409981964 shutdown_connections 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/409981964 >> v2:172.21.15.146:3300/0 conn(0x7efd6806f380 0x7efd6806f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/409981964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680ce450 0x7efd680d2580 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 -- 172.21.15.72:0/409981964 >> 172.21.15.72:0/409981964 conn(0x7efd680d1a60 msgr2=0x7efd680d1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 -- 172.21.15.72:0/409981964 shutdown_connections 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.898+0000 7efd6f2c9640 1 -- 172.21.15.72:0/409981964 wait complete. 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.899+0000 7efd6f2c9640 1 Processor -- start 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.899+0000 7efd6f2c9640 1 -- start start 2024-03-06T07:21:53.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6f2c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6806f380 0x7efd680c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6f2c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd680ce450 0x7efd6806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6f2c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 0x7efd6806d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6f2c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6815f590 con 0x7efd680cf0a0 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6f2c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd680cd060 con 0x7efd6806f380 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6f2c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6815f710 con 0x7efd680ce450 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 0x7efd6806d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 0x7efd6806d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54244/0 (socket says 172.21.15.72:54244) 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 -- 172.21.15.72:0/3202245740 learned_addr learned my addr 172.21.15.72:0/3202245740 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 -- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd680ce450 msgr2=0x7efd6806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd680ce450 0x7efd6806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 -- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6806f380 msgr2=0x7efd680c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d065640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6806f380 0x7efd680c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6806f380 0x7efd680c7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 -- 172.21.15.72:0/3202245740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd680cb830 con 0x7efd680cf0a0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.900+0000 7efd6d866640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 0x7efd6806d610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efd6000cd80 tx=0x7efd6000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.901+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6000f680 con 0x7efd680cf0a0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.901+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd6000f840 con 0x7efd680cf0a0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.901+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd680b67d0 con 0x7efd680cf0a0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.901+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd6000fc30 con 0x7efd680cf0a0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.901+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd680b6a70 con 0x7efd680cf0a0 2024-03-06T07:21:53.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.902+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6804f330 con 0x7efd680cf0a0 2024-03-06T07:21:53.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.903+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd600213f0 con 0x7efd680cf0a0 2024-03-06T07:21:53.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.903+0000 7efd5dffb640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd4c06b4d0 0x7efd4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:53.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.903+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd6008aee0 con 0x7efd680cf0a0 2024-03-06T07:21:53.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.903+0000 7efd6d065640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd4c06b4d0 0x7efd4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:53.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.903+0000 7efd6d065640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd4c06b4d0 0x7efd4c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd64036000 tx=0x7efd64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:53.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:53.907+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd60014030 con 0x7efd680cf0a0 2024-03-06T07:21:54.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.083+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd680cdc80 con 0x7efd4c06b4d0 2024-03-06T07:21:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.090+0000 7efd5dffb640 1 -- 172.21.15.72:0/3202245740 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251964 (secure 0 0 0) 0x7efd680cdc80 con 0x7efd4c06b4d0 2024-03-06T07:21:54.095 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd4c06b4d0 msgr2=0x7efd4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd4c06b4d0 0x7efd4c06d990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efd64036000 tx=0x7efd64008030 comp rx=0 tx=0).stop 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 msgr2=0x7efd6806d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 0x7efd6806d610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efd6000cd80 tx=0x7efd6000a650 comp rx=0 tx=0).stop 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 shutdown_connections 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd4c06b4d0 0x7efd4c06d990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd680cf0a0 0x7efd6806d610 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd680ce450 0x7efd6806d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 --2- 172.21.15.72:0/3202245740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6806f380 0x7efd680c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 >> 172.21.15.72:0/3202245740 conn(0x7efd680d1a60 msgr2=0x7efd6815e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 shutdown_connections 2024-03-06T07:21:54.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.093+0000 7efd6f2c9640 1 -- 172.21.15.72:0/3202245740 wait complete. 2024-03-06T07:21:54.106 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:21:53.948772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219076,"kb_used_data":2884,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755644,"statfs":{"total":773094113280,"available":772869779456,"internally_reserved":0,"allocated":2953216,"data_stored":1771496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.898584"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060871+0000","last_change":"2024-03-06T07:21:50.086544+0000","last_active":"2024-03-06T07:21:51.060871+0000","last_peered":"2024-03-06T07:21:51.060871+0000","last_clean":"2024-03-06T07:21:51.060871+0000","last_became_active":"2024-03-06T07:21:50.085415+0000","last_became_peered":"2024-03-06T07:21:50.085415+0000","last_unstale":"2024-03-06T07:21:51.060871+0000","last_undegraded":"2024-03-06T07:21:51.060871+0000","last_fullsized":"2024-03-06T07:21:51.060871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059336+0000","last_change":"2024-03-06T07:21:50.074454+0000","last_active":"2024-03-06T07:21:51.059336+0000","last_peered":"2024-03-06T07:21:51.059336+0000","last_clean":"2024-03-06T07:21:51.059336+0000","last_became_active":"2024-03-06T07:21:50.074183+0000","last_became_peered":"2024-03-06T07:21:50.074183+0000","last_unstale":"2024-03-06T07:21:51.059336+0000","last_undegraded":"2024-03-06T07:21:51.059336+0000","last_fullsized":"2024-03-06T07:21:51.059336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333626+0000","last_change":"2024-03-06T07:21:50.063938+0000","last_active":"2024-03-06T07:21:51.333626+0000","last_peered":"2024-03-06T07:21:51.333626+0000","last_clean":"2024-03-06T07:21:51.333626+0000","last_became_active":"2024-03-06T07:21:50.063674+0000","last_became_peered":"2024-03-06T07:21:50.063674+0000","last_unstale":"2024-03-06T07:21:51.333626+0000","last_undegraded":"2024-03-06T07:21:51.333626+0000","last_fullsized":"2024-03-06T07:21:51.333626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061678+0000","last_change":"2024-03-06T07:21:50.075505+0000","last_active":"2024-03-06T07:21:51.061678+0000","last_peered":"2024-03-06T07:21:51.061678+0000","last_clean":"2024-03-06T07:21:51.061678+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.061678+0000","last_undegraded":"2024-03-06T07:21:51.061678+0000","last_fullsized":"2024-03-06T07:21:51.061678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061574+0000","last_change":"2024-03-06T07:21:50.082456+0000","last_active":"2024-03-06T07:21:51.061574+0000","last_peered":"2024-03-06T07:21:51.061574+0000","last_clean":"2024-03-06T07:21:51.061574+0000","last_became_active":"2024-03-06T07:21:50.081900+0000","last_became_peered":"2024-03-06T07:21:50.081900+0000","last_unstale":"2024-03-06T07:21:51.061574+0000","last_undegraded":"2024-03-06T07:21:51.061574+0000","last_fullsized":"2024-03-06T07:21:51.061574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:30.952471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061465+0000","last_change":"2024-03-06T07:21:50.075967+0000","last_active":"2024-03-06T07:21:51.061465+0000","last_peered":"2024-03-06T07:21:51.061465+0000","last_clean":"2024-03-06T07:21:51.061465+0000","last_became_active":"2024-03-06T07:21:50.075668+0000","last_became_peered":"2024-03-06T07:21:50.075668+0000","last_unstale":"2024-03-06T07:21:51.061465+0000","last_undegraded":"2024-03-06T07:21:51.061465+0000","last_fullsized":"2024-03-06T07:21:51.061465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060628+0000","last_change":"2024-03-06T07:21:50.081328+0000","last_active":"2024-03-06T07:21:51.060628+0000","last_peered":"2024-03-06T07:21:51.060628+0000","last_clean":"2024-03-06T07:21:51.060628+0000","last_became_active":"2024-03-06T07:21:50.080941+0000","last_became_peered":"2024-03-06T07:21:50.080941+0000","last_unstale":"2024-03-06T07:21:51.060628+0000","last_undegraded":"2024-03-06T07:21:51.060628+0000","last_fullsized":"2024-03-06T07:21:51.060628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334303+0000","last_change":"2024-03-06T07:21:50.073822+0000","last_active":"2024-03-06T07:21:51.334303+0000","last_peered":"2024-03-06T07:21:51.334303+0000","last_clean":"2024-03-06T07:21:51.334303+0000","last_became_active":"2024-03-06T07:21:50.073421+0000","last_became_peered":"2024-03-06T07:21:50.073421+0000","last_unstale":"2024-03-06T07:21:51.334303+0000","last_undegraded":"2024-03-06T07:21:51.334303+0000","last_fullsized":"2024-03-06T07:21:51.334303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061065+0000","last_change":"2024-03-06T07:21:50.087355+0000","last_active":"2024-03-06T07:21:51.061065+0000","last_peered":"2024-03-06T07:21:51.061065+0000","last_clean":"2024-03-06T07:21:51.061065+0000","last_became_active":"2024-03-06T07:21:50.086854+0000","last_became_peered":"2024-03-06T07:21:50.086854+0000","last_unstale":"2024-03-06T07:21:51.061065+0000","last_undegraded":"2024-03-06T07:21:51.061065+0000","last_fullsized":"2024-03-06T07:21:51.061065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075461+0000","last_change":"2024-03-06T07:21:50.079046+0000","last_active":"2024-03-06T07:21:51.075461+0000","last_peered":"2024-03-06T07:21:51.075461+0000","last_clean":"2024-03-06T07:21:51.075461+0000","last_became_active":"2024-03-06T07:21:50.078365+0000","last_became_peered":"2024-03-06T07:21:50.078365+0000","last_unstale":"2024-03-06T07:21:51.075461+0000","last_undegraded":"2024-03-06T07:21:51.075461+0000","last_fullsized":"2024-03-06T07:21:51.075461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075358+0000","last_change":"2024-03-06T07:21:50.078291+0000","last_active":"2024-03-06T07:21:51.075358+0000","last_peered":"2024-03-06T07:21:51.075358+0000","last_clean":"2024-03-06T07:21:51.075358+0000","last_became_active":"2024-03-06T07:21:50.078024+0000","last_became_peered":"2024-03-06T07:21:50.078024+0000","last_unstale":"2024-03-06T07:21:51.075358+0000","last_undegraded":"2024-03-06T07:21:51.075358+0000","last_fullsized":"2024-03-06T07:21:51.075358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334132+0000","last_change":"2024-03-06T07:21:50.081726+0000","last_active":"2024-03-06T07:21:51.334132+0000","last_peered":"2024-03-06T07:21:51.334132+0000","last_clean":"2024-03-06T07:21:51.334132+0000","last_became_active":"2024-03-06T07:21:50.081458+0000","last_became_peered":"2024-03-06T07:21:50.081458+0000","last_unstale":"2024-03-06T07:21:51.334132+0000","last_undegraded":"2024-03-06T07:21:51.334132+0000","last_fullsized":"2024-03-06T07:21:51.334132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:41.946707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333563+0000","last_change":"2024-03-06T07:21:50.070195+0000","last_active":"2024-03-06T07:21:51.333563+0000","last_peered":"2024-03-06T07:21:51.333563+0000","last_clean":"2024-03-06T07:21:51.333563+0000","last_became_active":"2024-03-06T07:21:50.069887+0000","last_became_peered":"2024-03-06T07:21:50.069887+0000","last_unstale":"2024-03-06T07:21:51.333563+0000","last_undegraded":"2024-03-06T07:21:51.333563+0000","last_fullsized":"2024-03-06T07:21:51.333563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333869+0000","last_change":"2024-03-06T07:21:50.066290+0000","last_active":"2024-03-06T07:21:51.333869+0000","last_peered":"2024-03-06T07:21:51.333869+0000","last_clean":"2024-03-06T07:21:51.333869+0000","last_became_active":"2024-03-06T07:21:50.066008+0000","last_became_peered":"2024-03-06T07:21:50.066008+0000","last_unstale":"2024-03-06T07:21:51.333869+0000","last_undegraded":"2024-03-06T07:21:51.333869+0000","last_fullsized":"2024-03-06T07:21:51.333869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059103+0000","last_change":"2024-03-06T07:21:50.074011+0000","last_active":"2024-03-06T07:21:51.059103+0000","last_peered":"2024-03-06T07:21:51.059103+0000","last_clean":"2024-03-06T07:21:51.059103+0000","last_became_active":"2024-03-06T07:21:50.073679+0000","last_became_peered":"2024-03-06T07:21:50.073679+0000","last_unstale":"2024-03-06T07:21:51.059103+0000","last_undegraded":"2024-03-06T07:21:51.059103+0000","last_fullsized":"2024-03-06T07:21:51.059103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060633+0000","last_change":"2024-03-06T07:21:50.076532+0000","last_active":"2024-03-06T07:21:51.060633+0000","last_peered":"2024-03-06T07:21:51.060633+0000","last_clean":"2024-03-06T07:21:51.060633+0000","last_became_active":"2024-03-06T07:21:50.075857+0000","last_became_peered":"2024-03-06T07:21:50.075857+0000","last_unstale":"2024-03-06T07:21:51.060633+0000","last_undegraded":"2024-03-06T07:21:51.060633+0000","last_fullsized":"2024-03-06T07:21:51.060633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062233+0000","last_change":"2024-03-06T07:21:50.086644+0000","last_active":"2024-03-06T07:21:51.062233+0000","last_peered":"2024-03-06T07:21:51.062233+0000","last_clean":"2024-03-06T07:21:51.062233+0000","last_became_active":"2024-03-06T07:21:50.084602+0000","last_became_peered":"2024-03-06T07:21:50.084602+0000","last_unstale":"2024-03-06T07:21:51.062233+0000","last_undegraded":"2024-03-06T07:21:51.062233+0000","last_fullsized":"2024-03-06T07:21:51.062233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334765+0000","last_change":"2024-03-06T07:21:50.075838+0000","last_active":"2024-03-06T07:21:51.334765+0000","last_peered":"2024-03-06T07:21:51.334765+0000","last_clean":"2024-03-06T07:21:51.334765+0000","last_became_active":"2024-03-06T07:21:50.075531+0000","last_became_peered":"2024-03-06T07:21:50.075531+0000","last_unstale":"2024-03-06T07:21:51.334765+0000","last_undegraded":"2024-03-06T07:21:51.334765+0000","last_fullsized":"2024-03-06T07:21:51.334765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:00.903565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061683+0000","last_change":"2024-03-06T07:21:50.083575+0000","last_active":"2024-03-06T07:21:51.061683+0000","last_peered":"2024-03-06T07:21:51.061683+0000","last_clean":"2024-03-06T07:21:51.061683+0000","last_became_active":"2024-03-06T07:21:50.083016+0000","last_became_peered":"2024-03-06T07:21:50.083016+0000","last_unstale":"2024-03-06T07:21:51.061683+0000","last_undegraded":"2024-03-06T07:21:51.061683+0000","last_fullsized":"2024-03-06T07:21:51.061683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:20.854195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075317+0000","last_change":"2024-03-06T07:21:50.082493+0000","last_active":"2024-03-06T07:21:51.075317+0000","last_peered":"2024-03-06T07:21:51.075317+0000","last_clean":"2024-03-06T07:21:51.075317+0000","last_became_active":"2024-03-06T07:21:50.082217+0000","last_became_peered":"2024-03-06T07:21:50.082217+0000","last_unstale":"2024-03-06T07:21:51.075317+0000","last_undegraded":"2024-03-06T07:21:51.075317+0000","last_fullsized":"2024-03-06T07:21:51.075317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333683+0000","last_change":"2024-03-06T07:21:50.082417+0000","last_active":"2024-03-06T07:21:51.333683+0000","last_peered":"2024-03-06T07:21:51.333683+0000","last_clean":"2024-03-06T07:21:51.333683+0000","last_became_active":"2024-03-06T07:21:50.082162+0000","last_became_peered":"2024-03-06T07:21:50.082162+0000","last_unstale":"2024-03-06T07:21:51.333683+0000","last_undegraded":"2024-03-06T07:21:51.333683+0000","last_fullsized":"2024-03-06T07:21:51.333683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061987+0000","last_change":"2024-03-06T07:21:50.085327+0000","last_active":"2024-03-06T07:21:51.061987+0000","last_peered":"2024-03-06T07:21:51.061987+0000","last_clean":"2024-03-06T07:21:51.061987+0000","last_became_active":"2024-03-06T07:21:50.085019+0000","last_became_peered":"2024-03-06T07:21:50.085019+0000","last_unstale":"2024-03-06T07:21:51.061987+0000","last_undegraded":"2024-03-06T07:21:51.061987+0000","last_fullsized":"2024-03-06T07:21:51.061987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333521+0000","last_change":"2024-03-06T07:21:50.080803+0000","last_active":"2024-03-06T07:21:51.333521+0000","last_peered":"2024-03-06T07:21:51.333521+0000","last_clean":"2024-03-06T07:21:51.333521+0000","last_became_active":"2024-03-06T07:21:50.080460+0000","last_became_peered":"2024-03-06T07:21:50.080460+0000","last_unstale":"2024-03-06T07:21:51.333521+0000","last_undegraded":"2024-03-06T07:21:51.333521+0000","last_fullsized":"2024-03-06T07:21:51.333521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060865+0000","last_change":"2024-03-06T07:21:50.071650+0000","last_active":"2024-03-06T07:21:51.060865+0000","last_peered":"2024-03-06T07:21:51.060865+0000","last_clean":"2024-03-06T07:21:51.060865+0000","last_became_active":"2024-03-06T07:21:50.071383+0000","last_became_peered":"2024-03-06T07:21:50.071383+0000","last_unstale":"2024-03-06T07:21:51.060865+0000","last_undegraded":"2024-03-06T07:21:51.060865+0000","last_fullsized":"2024-03-06T07:21:51.060865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333966+0000","last_change":"2024-03-06T07:21:50.077589+0000","last_active":"2024-03-06T07:21:51.333966+0000","last_peered":"2024-03-06T07:21:51.333966+0000","last_clean":"2024-03-06T07:21:51.333966+0000","last_became_active":"2024-03-06T07:21:50.077302+0000","last_became_peered":"2024-03-06T07:21:50.077302+0000","last_unstale":"2024-03-06T07:21:51.333966+0000","last_undegraded":"2024-03-06T07:21:51.333966+0000","last_fullsized":"2024-03-06T07:21:51.333966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075478+0000","last_change":"2024-03-06T07:21:50.085031+0000","last_active":"2024-03-06T07:21:51.075478+0000","last_peered":"2024-03-06T07:21:51.075478+0000","last_clean":"2024-03-06T07:21:51.075478+0000","last_became_active":"2024-03-06T07:21:50.084758+0000","last_became_peered":"2024-03-06T07:21:50.084758+0000","last_unstale":"2024-03-06T07:21:51.075478+0000","last_undegraded":"2024-03-06T07:21:51.075478+0000","last_fullsized":"2024-03-06T07:21:51.075478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059511+0000","last_change":"2024-03-06T07:21:50.072164+0000","last_active":"2024-03-06T07:21:51.059511+0000","last_peered":"2024-03-06T07:21:51.059511+0000","last_clean":"2024-03-06T07:21:51.059511+0000","last_became_active":"2024-03-06T07:21:50.071795+0000","last_became_peered":"2024-03-06T07:21:50.071795+0000","last_unstale":"2024-03-06T07:21:51.059511+0000","last_undegraded":"2024-03-06T07:21:51.059511+0000","last_fullsized":"2024-03-06T07:21:51.059511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334595+0000","last_change":"2024-03-06T07:21:50.078217+0000","last_active":"2024-03-06T07:21:51.334595+0000","last_peered":"2024-03-06T07:21:51.334595+0000","last_clean":"2024-03-06T07:21:51.334595+0000","last_became_active":"2024-03-06T07:21:50.077809+0000","last_became_peered":"2024-03-06T07:21:50.077809+0000","last_unstale":"2024-03-06T07:21:51.334595+0000","last_undegraded":"2024-03-06T07:21:51.334595+0000","last_fullsized":"2024-03-06T07:21:51.334595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333826+0000","last_change":"2024-03-06T07:21:50.065564+0000","last_active":"2024-03-06T07:21:51.333826+0000","last_peered":"2024-03-06T07:21:51.333826+0000","last_clean":"2024-03-06T07:21:51.333826+0000","last_became_active":"2024-03-06T07:21:50.065294+0000","last_became_peered":"2024-03-06T07:21:50.065294+0000","last_unstale":"2024-03-06T07:21:51.333826+0000","last_undegraded":"2024-03-06T07:21:51.333826+0000","last_fullsized":"2024-03-06T07:21:51.333826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062062+0000","last_change":"2024-03-06T07:21:50.073038+0000","last_active":"2024-03-06T07:21:51.062062+0000","last_peered":"2024-03-06T07:21:51.062062+0000","last_clean":"2024-03-06T07:21:51.062062+0000","last_became_active":"2024-03-06T07:21:50.072603+0000","last_became_peered":"2024-03-06T07:21:50.072603+0000","last_unstale":"2024-03-06T07:21:51.062062+0000","last_undegraded":"2024-03-06T07:21:51.062062+0000","last_fullsized":"2024-03-06T07:21:51.062062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334317+0000","last_change":"2024-03-06T07:21:50.069962+0000","last_active":"2024-03-06T07:21:51.334317+0000","last_peered":"2024-03-06T07:21:51.334317+0000","last_clean":"2024-03-06T07:21:51.334317+0000","last_became_active":"2024-03-06T07:21:50.069700+0000","last_became_peered":"2024-03-06T07:21:50.069700+0000","last_unstale":"2024-03-06T07:21:51.334317+0000","last_undegraded":"2024-03-06T07:21:51.334317+0000","last_fullsized":"2024-03-06T07:21:51.334317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075081+0000","last_change":"2024-03-06T07:21:50.077457+0000","last_active":"2024-03-06T07:21:51.075081+0000","last_peered":"2024-03-06T07:21:51.075081+0000","last_clean":"2024-03-06T07:21:51.075081+0000","last_became_active":"2024-03-06T07:21:50.077173+0000","last_became_peered":"2024-03-06T07:21:50.077173+0000","last_unstale":"2024-03-06T07:21:51.075081+0000","last_undegraded":"2024-03-06T07:21:51.075081+0000","last_fullsized":"2024-03-06T07:21:51.075081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333933+0000","last_change":"2024-03-06T07:21:50.075503+0000","last_active":"2024-03-06T07:21:51.333933+0000","last_peered":"2024-03-06T07:21:51.333933+0000","last_clean":"2024-03-06T07:21:51.333933+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.333933+0000","last_undegraded":"2024-03-06T07:21:51.333933+0000","last_fullsized":"2024-03-06T07:21:51.333933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:48.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333798+0000","last_change":"2024-03-06T07:21:49.073461+0000","last_active":"2024-03-06T07:21:51.333798+0000","last_peered":"2024-03-06T07:21:51.333798+0000","last_clean":"2024-03-06T07:21:51.333798+0000","last_became_active":"2024-03-06T07:21:49.073212+0000","last_became_peered":"2024-03-06T07:21:49.073212+0000","last_unstale":"2024-03-06T07:21:51.333798+0000","last_undegraded":"2024-03-06T07:21:51.333798+0000","last_fullsized":"2024-03-06T07:21:51.333798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333964+0000","last_change":"2024-03-06T07:21:50.083293+0000","last_active":"2024-03-06T07:21:51.333964+0000","last_peered":"2024-03-06T07:21:51.333964+0000","last_clean":"2024-03-06T07:21:51.333964+0000","last_became_active":"2024-03-06T07:21:50.082931+0000","last_became_peered":"2024-03-06T07:21:50.082931+0000","last_unstale":"2024-03-06T07:21:51.333964+0000","last_undegraded":"2024-03-06T07:21:51.333964+0000","last_fullsized":"2024-03-06T07:21:51.333964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075166+0000","last_change":"2024-03-06T07:21:49.072875+0000","last_active":"2024-03-06T07:21:51.075166+0000","last_peered":"2024-03-06T07:21:51.075166+0000","last_clean":"2024-03-06T07:21:51.075166+0000","last_became_active":"2024-03-06T07:21:49.071548+0000","last_became_peered":"2024-03-06T07:21:49.071548+0000","last_unstale":"2024-03-06T07:21:51.075166+0000","last_undegraded":"2024-03-06T07:21:51.075166+0000","last_fullsized":"2024-03-06T07:21:51.075166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333606+0000","last_change":"2024-03-06T07:21:49.075539+0000","last_active":"2024-03-06T07:21:51.333606+0000","last_peered":"2024-03-06T07:21:51.333606+0000","last_clean":"2024-03-06T07:21:51.333606+0000","last_became_active":"2024-03-06T07:21:49.075192+0000","last_became_peered":"2024-03-06T07:21:49.075192+0000","last_unstale":"2024-03-06T07:21:51.333606+0000","last_undegraded":"2024-03-06T07:21:51.333606+0000","last_fullsized":"2024-03-06T07:21:51.333606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334442+0000","last_change":"2024-03-06T07:21:50.078018+0000","last_active":"2024-03-06T07:21:51.334442+0000","last_peered":"2024-03-06T07:21:51.334442+0000","last_clean":"2024-03-06T07:21:51.334442+0000","last_became_active":"2024-03-06T07:21:50.077638+0000","last_became_peered":"2024-03-06T07:21:50.077638+0000","last_unstale":"2024-03-06T07:21:51.334442+0000","last_undegraded":"2024-03-06T07:21:51.334442+0000","last_fullsized":"2024-03-06T07:21:51.334442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060973+0000","last_change":"2024-03-06T07:21:49.070076+0000","last_active":"2024-03-06T07:21:51.060973+0000","last_peered":"2024-03-06T07:21:51.060973+0000","last_clean":"2024-03-06T07:21:51.060973+0000","last_became_active":"2024-03-06T07:21:49.069491+0000","last_became_peered":"2024-03-06T07:21:49.069491+0000","last_unstale":"2024-03-06T07:21:51.060973+0000","last_undegraded":"2024-03-06T07:21:51.060973+0000","last_fullsized":"2024-03-06T07:21:51.060973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059076+0000","last_change":"2024-03-06T07:21:50.080260+0000","last_active":"2024-03-06T07:21:51.059076+0000","last_peered":"2024-03-06T07:21:51.059076+0000","last_clean":"2024-03-06T07:21:51.059076+0000","last_became_active":"2024-03-06T07:21:50.079987+0000","last_became_peered":"2024-03-06T07:21:50.079987+0000","last_unstale":"2024-03-06T07:21:51.059076+0000","last_undegraded":"2024-03-06T07:21:51.059076+0000","last_fullsized":"2024-03-06T07:21:51.059076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075328+0000","last_change":"2024-03-06T07:21:49.071143+0000","last_active":"2024-03-06T07:21:51.075328+0000","last_peered":"2024-03-06T07:21:51.075328+0000","last_clean":"2024-03-06T07:21:51.075328+0000","last_became_active":"2024-03-06T07:21:49.070872+0000","last_became_peered":"2024-03-06T07:21:49.070872+0000","last_unstale":"2024-03-06T07:21:51.075328+0000","last_undegraded":"2024-03-06T07:21:51.075328+0000","last_fullsized":"2024-03-06T07:21:51.075328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061550+0000","last_change":"2024-03-06T07:21:50.078307+0000","last_active":"2024-03-06T07:21:51.061550+0000","last_peered":"2024-03-06T07:21:51.061550+0000","last_clean":"2024-03-06T07:21:51.061550+0000","last_became_active":"2024-03-06T07:21:50.078025+0000","last_became_peered":"2024-03-06T07:21:50.078025+0000","last_unstale":"2024-03-06T07:21:51.061550+0000","last_undegraded":"2024-03-06T07:21:51.061550+0000","last_fullsized":"2024-03-06T07:21:51.061550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061672+0000","last_change":"2024-03-06T07:21:49.066235+0000","last_active":"2024-03-06T07:21:51.061672+0000","last_peered":"2024-03-06T07:21:51.061672+0000","last_clean":"2024-03-06T07:21:51.061672+0000","last_became_active":"2024-03-06T07:21:49.065968+0000","last_became_peered":"2024-03-06T07:21:49.065968+0000","last_unstale":"2024-03-06T07:21:51.061672+0000","last_undegraded":"2024-03-06T07:21:51.061672+0000","last_fullsized":"2024-03-06T07:21:51.061672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333774+0000","last_change":"2024-03-06T07:21:50.067265+0000","last_active":"2024-03-06T07:21:51.333774+0000","last_peered":"2024-03-06T07:21:51.333774+0000","last_clean":"2024-03-06T07:21:51.333774+0000","last_became_active":"2024-03-06T07:21:50.066634+0000","last_became_peered":"2024-03-06T07:21:50.066634+0000","last_unstale":"2024-03-06T07:21:51.333774+0000","last_undegraded":"2024-03-06T07:21:51.333774+0000","last_fullsized":"2024-03-06T07:21:51.333774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059350+0000","last_change":"2024-03-06T07:21:49.070187+0000","last_active":"2024-03-06T07:21:51.059350+0000","last_peered":"2024-03-06T07:21:51.059350+0000","last_clean":"2024-03-06T07:21:51.059350+0000","last_became_active":"2024-03-06T07:21:49.069894+0000","last_became_peered":"2024-03-06T07:21:49.069894+0000","last_unstale":"2024-03-06T07:21:51.059350+0000","last_undegraded":"2024-03-06T07:21:51.059350+0000","last_fullsized":"2024-03-06T07:21:51.059350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334133+0000","last_change":"2024-03-06T07:21:50.086859+0000","last_active":"2024-03-06T07:21:51.334133+0000","last_peered":"2024-03-06T07:21:51.334133+0000","last_clean":"2024-03-06T07:21:51.334133+0000","last_became_active":"2024-03-06T07:21:50.085246+0000","last_became_peered":"2024-03-06T07:21:50.085246+0000","last_unstale":"2024-03-06T07:21:51.334133+0000","last_undegraded":"2024-03-06T07:21:51.334133+0000","last_fullsized":"2024-03-06T07:21:51.334133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075355+0000","last_change":"2024-03-06T07:21:49.074802+0000","last_active":"2024-03-06T07:21:51.075355+0000","last_peered":"2024-03-06T07:21:51.075355+0000","last_clean":"2024-03-06T07:21:51.075355+0000","last_became_active":"2024-03-06T07:21:49.074143+0000","last_became_peered":"2024-03-06T07:21:49.074143+0000","last_unstale":"2024-03-06T07:21:51.075355+0000","last_undegraded":"2024-03-06T07:21:51.075355+0000","last_fullsized":"2024-03-06T07:21:51.075355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061356+0000","last_change":"2024-03-06T07:21:50.083325+0000","last_active":"2024-03-06T07:21:51.061356+0000","last_peered":"2024-03-06T07:21:51.061356+0000","last_clean":"2024-03-06T07:21:51.061356+0000","last_became_active":"2024-03-06T07:21:50.082675+0000","last_became_peered":"2024-03-06T07:21:50.082675+0000","last_unstale":"2024-03-06T07:21:51.061356+0000","last_undegraded":"2024-03-06T07:21:51.061356+0000","last_fullsized":"2024-03-06T07:21:51.061356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334058+0000","last_change":"2024-03-06T07:21:49.066349+0000","last_active":"2024-03-06T07:21:51.334058+0000","last_peered":"2024-03-06T07:21:51.334058+0000","last_clean":"2024-03-06T07:21:51.334058+0000","last_became_active":"2024-03-06T07:21:49.066024+0000","last_became_peered":"2024-03-06T07:21:49.066024+0000","last_unstale":"2024-03-06T07:21:51.334058+0000","last_undegraded":"2024-03-06T07:21:51.334058+0000","last_fullsized":"2024-03-06T07:21:51.334058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333454+0000","last_change":"2024-03-06T07:21:50.082035+0000","last_active":"2024-03-06T07:21:51.333454+0000","last_peered":"2024-03-06T07:21:51.333454+0000","last_clean":"2024-03-06T07:21:51.333454+0000","last_became_active":"2024-03-06T07:21:50.081731+0000","last_became_peered":"2024-03-06T07:21:50.081731+0000","last_unstale":"2024-03-06T07:21:51.333454+0000","last_undegraded":"2024-03-06T07:21:51.333454+0000","last_fullsized":"2024-03-06T07:21:51.333454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:57.652112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062117+0000","last_change":"2024-03-06T07:21:49.074426+0000","last_active":"2024-03-06T07:21:51.062117+0000","last_peered":"2024-03-06T07:21:51.062117+0000","last_clean":"2024-03-06T07:21:51.062117+0000","last_became_active":"2024-03-06T07:21:49.074117+0000","last_became_peered":"2024-03-06T07:21:49.074117+0000","last_unstale":"2024-03-06T07:21:51.062117+0000","last_undegraded":"2024-03-06T07:21:51.062117+0000","last_fullsized":"2024-03-06T07:21:51.062117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061727+0000","last_change":"2024-03-06T07:21:50.082545+0000","last_active":"2024-03-06T07:21:51.061727+0000","last_peered":"2024-03-06T07:21:51.061727+0000","last_clean":"2024-03-06T07:21:51.061727+0000","last_became_active":"2024-03-06T07:21:50.082197+0000","last_became_peered":"2024-03-06T07:21:50.082197+0000","last_unstale":"2024-03-06T07:21:51.061727+0000","last_undegraded":"2024-03-06T07:21:51.061727+0000","last_fullsized":"2024-03-06T07:21:51.061727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:57.476299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075070+0000","last_change":"2024-03-06T07:21:49.073223+0000","last_active":"2024-03-06T07:21:51.075070+0000","last_peered":"2024-03-06T07:21:51.075070+0000","last_clean":"2024-03-06T07:21:51.075070+0000","last_became_active":"2024-03-06T07:21:49.071609+0000","last_became_peered":"2024-03-06T07:21:49.071609+0000","last_unstale":"2024-03-06T07:21:51.075070+0000","last_undegraded":"2024-03-06T07:21:51.075070+0000","last_fullsized":"2024-03-06T07:21:51.075070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075085+0000","last_change":"2024-03-06T07:21:50.077209+0000","last_active":"2024-03-06T07:21:51.075085+0000","last_peered":"2024-03-06T07:21:51.075085+0000","last_clean":"2024-03-06T07:21:51.075085+0000","last_became_active":"2024-03-06T07:21:50.076936+0000","last_became_peered":"2024-03-06T07:21:50.076936+0000","last_unstale":"2024-03-06T07:21:51.075085+0000","last_undegraded":"2024-03-06T07:21:51.075085+0000","last_fullsized":"2024-03-06T07:21:51.075085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334123+0000","last_change":"2024-03-06T07:21:49.072833+0000","last_active":"2024-03-06T07:21:51.334123+0000","last_peered":"2024-03-06T07:21:51.334123+0000","last_clean":"2024-03-06T07:21:51.334123+0000","last_became_active":"2024-03-06T07:21:49.072543+0000","last_became_peered":"2024-03-06T07:21:49.072543+0000","last_unstale":"2024-03-06T07:21:51.334123+0000","last_undegraded":"2024-03-06T07:21:51.334123+0000","last_fullsized":"2024-03-06T07:21:51.334123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060781+0000","last_change":"2024-03-06T07:21:50.081698+0000","last_active":"2024-03-06T07:21:51.060781+0000","last_peered":"2024-03-06T07:21:51.060781+0000","last_clean":"2024-03-06T07:21:51.060781+0000","last_became_active":"2024-03-06T07:21:50.081294+0000","last_became_peered":"2024-03-06T07:21:50.081294+0000","last_unstale":"2024-03-06T07:21:51.060781+0000","last_undegraded":"2024-03-06T07:21:51.060781+0000","last_fullsized":"2024-03-06T07:21:51.060781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334204+0000","last_change":"2024-03-06T07:21:49.069462+0000","last_active":"2024-03-06T07:21:51.334204+0000","last_peered":"2024-03-06T07:21:51.334204+0000","last_clean":"2024-03-06T07:21:51.334204+0000","last_became_active":"2024-03-06T07:21:49.069161+0000","last_became_peered":"2024-03-06T07:21:49.069161+0000","last_unstale":"2024-03-06T07:21:51.334204+0000","last_undegraded":"2024-03-06T07:21:51.334204+0000","last_fullsized":"2024-03-06T07:21:51.334204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:48.201301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061018+0000","last_change":"2024-03-06T07:21:50.082841+0000","last_active":"2024-03-06T07:21:51.061018+0000","last_peered":"2024-03-06T07:21:51.061018+0000","last_clean":"2024-03-06T07:21:51.061018+0000","last_became_active":"2024-03-06T07:21:50.082543+0000","last_became_peered":"2024-03-06T07:21:50.082543+0000","last_unstale":"2024-03-06T07:21:51.061018+0000","last_undegraded":"2024-03-06T07:21:51.061018+0000","last_fullsized":"2024-03-06T07:21:51.061018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334727+0000","last_change":"2024-03-06T07:21:49.070603+0000","last_active":"2024-03-06T07:21:51.334727+0000","last_peered":"2024-03-06T07:21:51.334727+0000","last_clean":"2024-03-06T07:21:51.334727+0000","last_became_active":"2024-03-06T07:21:49.070341+0000","last_became_peered":"2024-03-06T07:21:49.070341+0000","last_unstale":"2024-03-06T07:21:51.334727+0000","last_undegraded":"2024-03-06T07:21:51.334727+0000","last_fullsized":"2024-03-06T07:21:51.334727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061230+0000","last_change":"2024-03-06T07:21:50.082050+0000","last_active":"2024-03-06T07:21:51.061230+0000","last_peered":"2024-03-06T07:21:51.061230+0000","last_clean":"2024-03-06T07:21:51.061230+0000","last_became_active":"2024-03-06T07:21:50.081723+0000","last_became_peered":"2024-03-06T07:21:50.081723+0000","last_unstale":"2024-03-06T07:21:51.061230+0000","last_undegraded":"2024-03-06T07:21:51.061230+0000","last_fullsized":"2024-03-06T07:21:51.061230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059449+0000","last_change":"2024-03-06T07:21:49.069397+0000","last_active":"2024-03-06T07:21:51.059449+0000","last_peered":"2024-03-06T07:21:51.059449+0000","last_clean":"2024-03-06T07:21:51.059449+0000","last_became_active":"2024-03-06T07:21:49.069101+0000","last_became_peered":"2024-03-06T07:21:49.069101+0000","last_unstale":"2024-03-06T07:21:51.059449+0000","last_undegraded":"2024-03-06T07:21:51.059449+0000","last_fullsized":"2024-03-06T07:21:51.059449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061985+0000","last_change":"2024-03-06T07:21:50.072534+0000","last_active":"2024-03-06T07:21:51.061985+0000","last_peered":"2024-03-06T07:21:51.061985+0000","last_clean":"2024-03-06T07:21:51.061985+0000","last_became_active":"2024-03-06T07:21:50.072252+0000","last_became_peered":"2024-03-06T07:21:50.072252+0000","last_unstale":"2024-03-06T07:21:51.061985+0000","last_undegraded":"2024-03-06T07:21:51.061985+0000","last_fullsized":"2024-03-06T07:21:51.061985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334565+0000","last_change":"2024-03-06T07:21:49.071113+0000","last_active":"2024-03-06T07:21:51.334565+0000","last_peered":"2024-03-06T07:21:51.334565+0000","last_clean":"2024-03-06T07:21:51.334565+0000","last_became_active":"2024-03-06T07:21:49.070854+0000","last_became_peered":"2024-03-06T07:21:49.070854+0000","last_unstale":"2024-03-06T07:21:51.334565+0000","last_undegraded":"2024-03-06T07:21:51.334565+0000","last_fullsized":"2024-03-06T07:21:51.334565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061519+0000","last_change":"2024-03-06T07:21:50.087871+0000","last_active":"2024-03-06T07:21:51.061519+0000","last_peered":"2024-03-06T07:21:51.061519+0000","last_clean":"2024-03-06T07:21:51.061519+0000","last_became_active":"2024-03-06T07:21:50.087545+0000","last_became_peered":"2024-03-06T07:21:50.087545+0000","last_unstale":"2024-03-06T07:21:51.061519+0000","last_undegraded":"2024-03-06T07:21:51.061519+0000","last_fullsized":"2024-03-06T07:21:51.061519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:46.267182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:21:51.061213+0000","last_peered":"2024-03-06T07:21:51.061213+0000","last_clean":"2024-03-06T07:21:51.061213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:21:51.061213+0000","last_undegraded":"2024-03-06T07:21:51.061213+0000","last_fullsized":"2024-03-06T07:21:51.061213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061289+0000","last_change":"2024-03-06T07:21:49.064484+0000","last_active":"2024-03-06T07:21:51.061289+0000","last_peered":"2024-03-06T07:21:51.061289+0000","last_clean":"2024-03-06T07:21:51.061289+0000","last_became_active":"2024-03-06T07:21:49.064202+0000","last_became_peered":"2024-03-06T07:21:49.064202+0000","last_unstale":"2024-03-06T07:21:51.061289+0000","last_undegraded":"2024-03-06T07:21:51.061289+0000","last_fullsized":"2024-03-06T07:21:51.061289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333973+0000","last_change":"2024-03-06T07:21:50.074427+0000","last_active":"2024-03-06T07:21:51.333973+0000","last_peered":"2024-03-06T07:21:51.333973+0000","last_clean":"2024-03-06T07:21:51.333973+0000","last_became_active":"2024-03-06T07:21:50.074131+0000","last_became_peered":"2024-03-06T07:21:50.074131+0000","last_unstale":"2024-03-06T07:21:51.333973+0000","last_undegraded":"2024-03-06T07:21:51.333973+0000","last_fullsized":"2024-03-06T07:21:51.333973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061151+0000","last_change":"2024-03-06T07:21:49.075192+0000","last_active":"2024-03-06T07:21:51.061151+0000","last_peered":"2024-03-06T07:21:51.061151+0000","last_clean":"2024-03-06T07:21:51.061151+0000","last_became_active":"2024-03-06T07:21:49.074904+0000","last_became_peered":"2024-03-06T07:21:49.074904+0000","last_unstale":"2024-03-06T07:21:51.061151+0000","last_undegraded":"2024-03-06T07:21:51.061151+0000","last_fullsized":"2024-03-06T07:21:51.061151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:59.897349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333341+0000","last_change":"2024-03-06T07:21:50.074328+0000","last_active":"2024-03-06T07:21:51.333341+0000","last_peered":"2024-03-06T07:21:51.333341+0000","last_clean":"2024-03-06T07:21:51.333341+0000","last_became_active":"2024-03-06T07:21:50.074021+0000","last_became_peered":"2024-03-06T07:21:50.074021+0000","last_unstale":"2024-03-06T07:21:51.333341+0000","last_undegraded":"2024-03-06T07:21:51.333341+0000","last_fullsized":"2024-03-06T07:21:51.333341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:14.753301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334402+0000","last_change":"2024-03-06T07:21:49.068516+0000","last_active":"2024-03-06T07:21:51.334402+0000","last_peered":"2024-03-06T07:21:51.334402+0000","last_clean":"2024-03-06T07:21:51.334402+0000","last_became_active":"2024-03-06T07:21:49.068184+0000","last_became_peered":"2024-03-06T07:21:49.068184+0000","last_unstale":"2024-03-06T07:21:51.334402+0000","last_undegraded":"2024-03-06T07:21:51.334402+0000","last_fullsized":"2024-03-06T07:21:51.334402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059118+0000","last_change":"2024-03-06T07:21:50.071162+0000","last_active":"2024-03-06T07:21:51.059118+0000","last_peered":"2024-03-06T07:21:51.059118+0000","last_clean":"2024-03-06T07:21:51.059118+0000","last_became_active":"2024-03-06T07:21:50.070848+0000","last_became_peered":"2024-03-06T07:21:50.070848+0000","last_unstale":"2024-03-06T07:21:51.059118+0000","last_undegraded":"2024-03-06T07:21:51.059118+0000","last_fullsized":"2024-03-06T07:21:51.059118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062173+0000","last_change":"2024-03-06T07:21:49.071017+0000","last_active":"2024-03-06T07:21:51.062173+0000","last_peered":"2024-03-06T07:21:51.062173+0000","last_clean":"2024-03-06T07:21:51.062173+0000","last_became_active":"2024-03-06T07:21:49.070588+0000","last_became_peered":"2024-03-06T07:21:49.070588+0000","last_unstale":"2024-03-06T07:21:51.062173+0000","last_undegraded":"2024-03-06T07:21:51.062173+0000","last_fullsized":"2024-03-06T07:21:51.062173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334053+0000","last_change":"2024-03-06T07:21:50.082287+0000","last_active":"2024-03-06T07:21:51.334053+0000","last_peered":"2024-03-06T07:21:51.334053+0000","last_clean":"2024-03-06T07:21:51.334053+0000","last_became_active":"2024-03-06T07:21:50.081767+0000","last_became_peered":"2024-03-06T07:21:50.081767+0000","last_unstale":"2024-03-06T07:21:51.334053+0000","last_undegraded":"2024-03-06T07:21:51.334053+0000","last_fullsized":"2024-03-06T07:21:51.334053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:09.878872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059265+0000","last_change":"2024-03-06T07:21:49.069978+0000","last_active":"2024-03-06T07:21:51.059265+0000","last_peered":"2024-03-06T07:21:51.059265+0000","last_clean":"2024-03-06T07:21:51.059265+0000","last_became_active":"2024-03-06T07:21:49.069223+0000","last_became_peered":"2024-03-06T07:21:49.069223+0000","last_unstale":"2024-03-06T07:21:51.059265+0000","last_undegraded":"2024-03-06T07:21:51.059265+0000","last_fullsized":"2024-03-06T07:21:51.059265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062012+0000","last_change":"2024-03-06T07:21:50.076147+0000","last_active":"2024-03-06T07:21:51.062012+0000","last_peered":"2024-03-06T07:21:51.062012+0000","last_clean":"2024-03-06T07:21:51.062012+0000","last_became_active":"2024-03-06T07:21:50.075840+0000","last_became_peered":"2024-03-06T07:21:50.075840+0000","last_unstale":"2024-03-06T07:21:51.062012+0000","last_undegraded":"2024-03-06T07:21:51.062012+0000","last_fullsized":"2024-03-06T07:21:51.062012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061682+0000","last_change":"2024-03-06T07:21:49.077962+0000","last_active":"2024-03-06T07:21:51.061682+0000","last_peered":"2024-03-06T07:21:51.061682+0000","last_clean":"2024-03-06T07:21:51.061682+0000","last_became_active":"2024-03-06T07:21:49.077633+0000","last_became_peered":"2024-03-06T07:21:49.077633+0000","last_unstale":"2024-03-06T07:21:51.061682+0000","last_undegraded":"2024-03-06T07:21:51.061682+0000","last_fullsized":"2024-03-06T07:21:51.061682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:13.747084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059377+0000","last_change":"2024-03-06T07:21:50.071233+0000","last_active":"2024-03-06T07:21:51.059377+0000","last_peered":"2024-03-06T07:21:51.059377+0000","last_clean":"2024-03-06T07:21:51.059377+0000","last_became_active":"2024-03-06T07:21:50.070925+0000","last_became_peered":"2024-03-06T07:21:50.070925+0000","last_unstale":"2024-03-06T07:21:51.059377+0000","last_undegraded":"2024-03-06T07:21:51.059377+0000","last_fullsized":"2024-03-06T07:21:51.059377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.074529+0000","last_change":"2024-03-06T07:21:49.075388+0000","last_active":"2024-03-06T07:21:51.074529+0000","last_peered":"2024-03-06T07:21:51.074529+0000","last_clean":"2024-03-06T07:21:51.074529+0000","last_became_active":"2024-03-06T07:21:49.075115+0000","last_became_peered":"2024-03-06T07:21:49.075115+0000","last_unstale":"2024-03-06T07:21:51.074529+0000","last_undegraded":"2024-03-06T07:21:51.074529+0000","last_fullsized":"2024-03-06T07:21:51.074529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334111+0000","last_change":"2024-03-06T07:21:50.084419+0000","last_active":"2024-03-06T07:21:51.334111+0000","last_peered":"2024-03-06T07:21:51.334111+0000","last_clean":"2024-03-06T07:21:51.334111+0000","last_became_active":"2024-03-06T07:21:50.084022+0000","last_became_peered":"2024-03-06T07:21:50.084022+0000","last_unstale":"2024-03-06T07:21:51.334111+0000","last_undegraded":"2024-03-06T07:21:51.334111+0000","last_fullsized":"2024-03-06T07:21:51.334111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075351+0000","last_change":"2024-03-06T07:21:49.072877+0000","last_active":"2024-03-06T07:21:51.075351+0000","last_peered":"2024-03-06T07:21:51.075351+0000","last_clean":"2024-03-06T07:21:51.075351+0000","last_became_active":"2024-03-06T07:21:49.072559+0000","last_became_peered":"2024-03-06T07:21:49.072559+0000","last_unstale":"2024-03-06T07:21:51.075351+0000","last_undegraded":"2024-03-06T07:21:51.075351+0000","last_fullsized":"2024-03-06T07:21:51.075351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333756+0000","last_change":"2024-03-06T07:21:50.069013+0000","last_active":"2024-03-06T07:21:51.333756+0000","last_peered":"2024-03-06T07:21:51.333756+0000","last_clean":"2024-03-06T07:21:51.333756+0000","last_became_active":"2024-03-06T07:21:50.068753+0000","last_became_peered":"2024-03-06T07:21:50.068753+0000","last_unstale":"2024-03-06T07:21:51.333756+0000","last_undegraded":"2024-03-06T07:21:51.333756+0000","last_fullsized":"2024-03-06T07:21:51.333756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060433+0000","last_change":"2024-03-06T07:21:49.077101+0000","last_active":"2024-03-06T07:21:51.060433+0000","last_peered":"2024-03-06T07:21:51.060433+0000","last_clean":"2024-03-06T07:21:51.060433+0000","last_became_active":"2024-03-06T07:21:49.076900+0000","last_became_peered":"2024-03-06T07:21:49.076900+0000","last_unstale":"2024-03-06T07:21:51.060433+0000","last_undegraded":"2024-03-06T07:21:51.060433+0000","last_fullsized":"2024-03-06T07:21:51.060433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:37.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334046+0000","last_change":"2024-03-06T07:21:50.072349+0000","last_active":"2024-03-06T07:21:51.334046+0000","last_peered":"2024-03-06T07:21:51.334046+0000","last_clean":"2024-03-06T07:21:51.334046+0000","last_became_active":"2024-03-06T07:21:50.072052+0000","last_became_peered":"2024-03-06T07:21:50.072052+0000","last_unstale":"2024-03-06T07:21:51.334046+0000","last_undegraded":"2024-03-06T07:21:51.334046+0000","last_fullsized":"2024-03-06T07:21:51.334046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:59.429356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059318+0000","last_change":"2024-03-06T07:21:49.072417+0000","last_active":"2024-03-06T07:21:51.059318+0000","last_peered":"2024-03-06T07:21:51.059318+0000","last_clean":"2024-03-06T07:21:51.059318+0000","last_became_active":"2024-03-06T07:21:49.072153+0000","last_became_peered":"2024-03-06T07:21:49.072153+0000","last_unstale":"2024-03-06T07:21:51.059318+0000","last_undegraded":"2024-03-06T07:21:51.059318+0000","last_fullsized":"2024-03-06T07:21:51.059318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334590+0000","last_change":"2024-03-06T07:21:50.068165+0000","last_active":"2024-03-06T07:21:51.334590+0000","last_peered":"2024-03-06T07:21:51.334590+0000","last_clean":"2024-03-06T07:21:51.334590+0000","last_became_active":"2024-03-06T07:21:50.067278+0000","last_became_peered":"2024-03-06T07:21:50.067278+0000","last_unstale":"2024-03-06T07:21:51.334590+0000","last_undegraded":"2024-03-06T07:21:51.334590+0000","last_fullsized":"2024-03-06T07:21:51.334590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059227+0000","last_change":"2024-03-06T07:21:49.074974+0000","last_active":"2024-03-06T07:21:51.059227+0000","last_peered":"2024-03-06T07:21:51.059227+0000","last_clean":"2024-03-06T07:21:51.059227+0000","last_became_active":"2024-03-06T07:21:49.074697+0000","last_became_peered":"2024-03-06T07:21:49.074697+0000","last_unstale":"2024-03-06T07:21:51.059227+0000","last_undegraded":"2024-03-06T07:21:51.059227+0000","last_fullsized":"2024-03-06T07:21:51.059227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333690+0000","last_change":"2024-03-06T07:21:50.078075+0000","last_active":"2024-03-06T07:21:51.333690+0000","last_peered":"2024-03-06T07:21:51.333690+0000","last_clean":"2024-03-06T07:21:51.333690+0000","last_became_active":"2024-03-06T07:21:50.077794+0000","last_became_peered":"2024-03-06T07:21:50.077794+0000","last_unstale":"2024-03-06T07:21:51.333690+0000","last_undegraded":"2024-03-06T07:21:51.333690+0000","last_fullsized":"2024-03-06T07:21:51.333690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061819+0000","last_change":"2024-03-06T07:21:49.073724+0000","last_active":"2024-03-06T07:21:51.061819+0000","last_peered":"2024-03-06T07:21:51.061819+0000","last_clean":"2024-03-06T07:21:51.061819+0000","last_became_active":"2024-03-06T07:21:49.073443+0000","last_became_peered":"2024-03-06T07:21:49.073443+0000","last_unstale":"2024-03-06T07:21:51.061819+0000","last_undegraded":"2024-03-06T07:21:51.061819+0000","last_fullsized":"2024-03-06T07:21:51.061819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:56.645896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075154+0000","last_change":"2024-03-06T07:21:50.081464+0000","last_active":"2024-03-06T07:21:51.075154+0000","last_peered":"2024-03-06T07:21:51.075154+0000","last_clean":"2024-03-06T07:21:51.075154+0000","last_became_active":"2024-03-06T07:21:50.081190+0000","last_became_peered":"2024-03-06T07:21:50.081190+0000","last_unstale":"2024-03-06T07:21:51.075154+0000","last_undegraded":"2024-03-06T07:21:51.075154+0000","last_fullsized":"2024-03-06T07:21:51.075154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061935+0000","last_change":"2024-03-06T07:21:50.074879+0000","last_active":"2024-03-06T07:21:51.061935+0000","last_peered":"2024-03-06T07:21:51.061935+0000","last_clean":"2024-03-06T07:21:51.061935+0000","last_became_active":"2024-03-06T07:21:50.074633+0000","last_became_peered":"2024-03-06T07:21:50.074633+0000","last_unstale":"2024-03-06T07:21:51.061935+0000","last_undegraded":"2024-03-06T07:21:51.061935+0000","last_fullsized":"2024-03-06T07:21:51.061935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061004+0000","last_change":"2024-03-06T07:21:49.078149+0000","last_active":"2024-03-06T07:21:51.061004+0000","last_peered":"2024-03-06T07:21:51.061004+0000","last_clean":"2024-03-06T07:21:51.061004+0000","last_became_active":"2024-03-06T07:21:49.077845+0000","last_became_peered":"2024-03-06T07:21:49.077845+0000","last_unstale":"2024-03-06T07:21:51.061004+0000","last_undegraded":"2024-03-06T07:21:51.061004+0000","last_fullsized":"2024-03-06T07:21:51.061004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061862+0000","last_change":"2024-03-06T07:21:50.073658+0000","last_active":"2024-03-06T07:21:51.061862+0000","last_peered":"2024-03-06T07:21:51.061862+0000","last_clean":"2024-03-06T07:21:51.061862+0000","last_became_active":"2024-03-06T07:21:50.073396+0000","last_became_peered":"2024-03-06T07:21:50.073396+0000","last_unstale":"2024-03-06T07:21:51.061862+0000","last_undegraded":"2024-03-06T07:21:51.061862+0000","last_fullsized":"2024-03-06T07:21:51.061862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061590+0000","last_change":"2024-03-06T07:21:49.078569+0000","last_active":"2024-03-06T07:21:51.061590+0000","last_peered":"2024-03-06T07:21:51.061590+0000","last_clean":"2024-03-06T07:21:51.061590+0000","last_became_active":"2024-03-06T07:21:49.078274+0000","last_became_peered":"2024-03-06T07:21:49.078274+0000","last_unstale":"2024-03-06T07:21:51.061590+0000","last_undegraded":"2024-03-06T07:21:51.061590+0000","last_fullsized":"2024-03-06T07:21:51.061590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:23.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333637+0000","last_change":"2024-03-06T07:21:50.067267+0000","last_active":"2024-03-06T07:21:51.333637+0000","last_peered":"2024-03-06T07:21:51.333637+0000","last_clean":"2024-03-06T07:21:51.333637+0000","last_became_active":"2024-03-06T07:21:50.067014+0000","last_became_peered":"2024-03-06T07:21:50.067014+0000","last_unstale":"2024-03-06T07:21:51.333637+0000","last_undegraded":"2024-03-06T07:21:51.333637+0000","last_fullsized":"2024-03-06T07:21:51.333637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334431+0000","last_change":"2024-03-06T07:21:49.066664+0000","last_active":"2024-03-06T07:21:51.334431+0000","last_peered":"2024-03-06T07:21:51.334431+0000","last_clean":"2024-03-06T07:21:51.334431+0000","last_became_active":"2024-03-06T07:21:49.066350+0000","last_became_peered":"2024-03-06T07:21:49.066350+0000","last_unstale":"2024-03-06T07:21:51.334431+0000","last_undegraded":"2024-03-06T07:21:51.334431+0000","last_fullsized":"2024-03-06T07:21:51.334431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334333+0000","last_change":"2024-03-06T07:21:50.079262+0000","last_active":"2024-03-06T07:21:51.334333+0000","last_peered":"2024-03-06T07:21:51.334333+0000","last_clean":"2024-03-06T07:21:51.334333+0000","last_became_active":"2024-03-06T07:21:50.078776+0000","last_became_peered":"2024-03-06T07:21:50.078776+0000","last_unstale":"2024-03-06T07:21:51.334333+0000","last_undegraded":"2024-03-06T07:21:51.334333+0000","last_fullsized":"2024-03-06T07:21:51.334333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333867+0000","last_change":"2024-03-06T07:21:49.070371+0000","last_active":"2024-03-06T07:21:51.333867+0000","last_peered":"2024-03-06T07:21:51.333867+0000","last_clean":"2024-03-06T07:21:51.333867+0000","last_became_active":"2024-03-06T07:21:49.070100+0000","last_became_peered":"2024-03-06T07:21:49.070100+0000","last_unstale":"2024-03-06T07:21:51.333867+0000","last_undegraded":"2024-03-06T07:21:51.333867+0000","last_fullsized":"2024-03-06T07:21:51.333867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:44.026321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:54.110 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.245+0000 7f21d066b640 1 Processor -- start 2024-03-06T07:21:54.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.245+0000 7f21d066b640 1 -- start start 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21d066b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f21c815ab00 0x7f21c80b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21d066b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f21c80b48f0 0x7f21c80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21d066b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c80b4e30 con 0x7f21c80afc00 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21d066b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c80b0060 con 0x7f21c80b48f0 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21d066b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c80b5110 con 0x7f21c815ab00 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f21c80b48f0 0x7f21c80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f21c80b48f0 0x7f21c80af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54252/0 (socket says 172.21.15.72:54252) 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 -- 172.21.15.72:0/962586973 learned_addr learned my addr 172.21.15.72:0/962586973 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 -- 172.21.15.72:0/962586973 >> v2:172.21.15.146:3300/0 conn(0x7f21c815ab00 msgr2=0x7f21c80b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 --2- 172.21.15.72:0/962586973 >> v2:172.21.15.146:3300/0 conn(0x7f21c815ab00 0x7f21c80b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 -- 172.21.15.72:0/962586973 >> v1:172.21.15.146:6789/0 conn(0x7f21c80afc00 legacy=0x7f21c806e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.246+0000 7f21cdc06640 1 -- 172.21.15.72:0/962586973 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c815b310 con 0x7f21c80b48f0 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.247+0000 7f21cdc06640 1 --2- 172.21.15.72:0/962586973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80b48f0 0x7f21c80af6c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21c80732b0 tx=0x7f21b802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b10acd93b475bea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.247+0000 7f21cd405640 1 -- 172.21.15.72:0/962586973 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b803b070 con 0x7f21c80b48f0 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.247+0000 7f21cd405640 1 -- 172.21.15.72:0/962586973 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b8036440 con 0x7f21c80b48f0 2024-03-06T07:21:54.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.247+0000 7f21cd405640 1 -- 172.21.15.72:0/962586973 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8036810 con 0x7f21c80b48f0 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 -- 172.21.15.72:0/962586973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80b48f0 msgr2=0x7f21c80af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 --2- 172.21.15.72:0/962586973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80b48f0 0x7f21c80af6c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f21c80732b0 tx=0x7f21b802e0e0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 -- 172.21.15.72:0/962586973 shutdown_connections 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 --2- 172.21.15.72:0/962586973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80b48f0 0x7f21c80af6c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 --2- 172.21.15.72:0/962586973 >> v2:172.21.15.146:3300/0 conn(0x7f21c815ab00 0x7f21c80b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 -- 172.21.15.72:0/962586973 >> 172.21.15.72:0/962586973 conn(0x7f21c806c0c0 msgr2=0x7f21c806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 -- 172.21.15.72:0/962586973 shutdown_connections 2024-03-06T07:21:54.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.248+0000 7f21d066b640 1 -- 172.21.15.72:0/962586973 wait complete. 2024-03-06T07:21:54.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.250+0000 7f21d066b640 1 Processor -- start 2024-03-06T07:21:54.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.250+0000 7f21d066b640 1 -- start start 2024-03-06T07:21:54.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21d066b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 0x7f21c81423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21d066b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21c80b48f0 0x7f21c80cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21d066b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21c815ab00 0x7f21c815d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21d066b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c8071ca0 con 0x7f21c80afc00 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21d066b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c8071b20 con 0x7f21c80b48f0 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21d066b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c815aee0 con 0x7f21c815ab00 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21ce407640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 0x7f21c81423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21ce407640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 0x7f21c81423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54260/0 (socket says 172.21.15.72:54260) 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.251+0000 7f21ce407640 1 -- 172.21.15.72:0/2402607780 learned_addr learned my addr 172.21.15.72:0/2402607780 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21cdc06640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21c80b48f0 0x7f21c80cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21ce407640 1 -- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21c815ab00 msgr2=0x7f21c815d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21cec08640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21c815ab00 0x7f21c815d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21ce407640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21c815ab00 0x7f21c815d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21ce407640 1 -- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21c80b48f0 msgr2=0x7f21c80cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21ce407640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21c80b48f0 0x7f21c80cf020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21ce407640 1 -- 172.21.15.72:0/2402607780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c815dd50 con 0x7f21c80afc00 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21cdc06640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21c80b48f0 0x7f21c80cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21cec08640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21c815ab00 0x7f21c815d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21ce407640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 0x7f21c81423c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21c000b440 tx=0x7f21c000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0015020 con 0x7f21c80afc00 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c0011890 con 0x7f21c80afc00 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0011c80 con 0x7f21c80afc00 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c80cdca0 con 0x7f21c80afc00 2024-03-06T07:21:54.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.252+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c80ce260 con 0x7f21c80afc00 2024-03-06T07:21:54.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.253+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c804f330 con 0x7f21c80afc00 2024-03-06T07:21:54.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.254+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21c0018020 con 0x7f21c80afc00 2024-03-06T07:21:54.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.254+0000 7f21beffd640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21ac06b500 0x7f21ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.254+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21c008a2c0 con 0x7f21c80afc00 2024-03-06T07:21:54.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.254+0000 7f21cdc06640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21ac06b500 0x7f21ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.254+0000 7f21cdc06640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21ac06b500 0x7f21ac06d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21b8038000 tx=0x7f21b8032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.258+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f21c001f020 con 0x7f21c80afc00 2024-03-06T07:21:54.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.432+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21c8157b70 con 0x7f21ac06b500 2024-03-06T07:21:54.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.438+0000 7f21beffd640 1 -- 172.21.15.72:0/2402607780 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251964 (secure 0 0 0) 0x7f21c8157b70 con 0x7f21ac06b500 2024-03-06T07:21:54.442 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:54.444 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:21:54.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21ac06b500 msgr2=0x7f21ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21ac06b500 0x7f21ac06d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f21b8038000 tx=0x7f21b8032000 comp rx=0 tx=0).stop 2024-03-06T07:21:54.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 msgr2=0x7f21c81423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 0x7f21c81423c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f21c000b440 tx=0x7f21c000b900 comp rx=0 tx=0).stop 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 shutdown_connections 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21ac06b500 0x7f21ac06d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21c815ab00 0x7f21c815d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21c80b48f0 0x7f21c80cf020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 --2- 172.21.15.72:0/2402607780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21c80afc00 0x7f21c81423c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 >> 172.21.15.72:0/2402607780 conn(0x7f21c806c0c0 msgr2=0x7f21c80b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 shutdown_connections 2024-03-06T07:21:54.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.441+0000 7f21d066b640 1 -- 172.21.15.72:0/2402607780 wait complete. 2024-03-06T07:21:54.453 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-03-06T07:21:53.948772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219076,"kb_used_data":2884,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755644,"statfs":{"total":773094113280,"available":772869779456,"internally_reserved":0,"allocated":2953216,"data_stored":1771496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.898584"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060871+0000","last_change":"2024-03-06T07:21:50.086544+0000","last_active":"2024-03-06T07:21:51.060871+0000","last_peered":"2024-03-06T07:21:51.060871+0000","last_clean":"2024-03-06T07:21:51.060871+0000","last_became_active":"2024-03-06T07:21:50.085415+0000","last_became_peered":"2024-03-06T07:21:50.085415+0000","last_unstale":"2024-03-06T07:21:51.060871+0000","last_undegraded":"2024-03-06T07:21:51.060871+0000","last_fullsized":"2024-03-06T07:21:51.060871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059336+0000","last_change":"2024-03-06T07:21:50.074454+0000","last_active":"2024-03-06T07:21:51.059336+0000","last_peered":"2024-03-06T07:21:51.059336+0000","last_clean":"2024-03-06T07:21:51.059336+0000","last_became_active":"2024-03-06T07:21:50.074183+0000","last_became_peered":"2024-03-06T07:21:50.074183+0000","last_unstale":"2024-03-06T07:21:51.059336+0000","last_undegraded":"2024-03-06T07:21:51.059336+0000","last_fullsized":"2024-03-06T07:21:51.059336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333626+0000","last_change":"2024-03-06T07:21:50.063938+0000","last_active":"2024-03-06T07:21:51.333626+0000","last_peered":"2024-03-06T07:21:51.333626+0000","last_clean":"2024-03-06T07:21:51.333626+0000","last_became_active":"2024-03-06T07:21:50.063674+0000","last_became_peered":"2024-03-06T07:21:50.063674+0000","last_unstale":"2024-03-06T07:21:51.333626+0000","last_undegraded":"2024-03-06T07:21:51.333626+0000","last_fullsized":"2024-03-06T07:21:51.333626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061678+0000","last_change":"2024-03-06T07:21:50.075505+0000","last_active":"2024-03-06T07:21:51.061678+0000","last_peered":"2024-03-06T07:21:51.061678+0000","last_clean":"2024-03-06T07:21:51.061678+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.061678+0000","last_undegraded":"2024-03-06T07:21:51.061678+0000","last_fullsized":"2024-03-06T07:21:51.061678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061574+0000","last_change":"2024-03-06T07:21:50.082456+0000","last_active":"2024-03-06T07:21:51.061574+0000","last_peered":"2024-03-06T07:21:51.061574+0000","last_clean":"2024-03-06T07:21:51.061574+0000","last_became_active":"2024-03-06T07:21:50.081900+0000","last_became_peered":"2024-03-06T07:21:50.081900+0000","last_unstale":"2024-03-06T07:21:51.061574+0000","last_undegraded":"2024-03-06T07:21:51.061574+0000","last_fullsized":"2024-03-06T07:21:51.061574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:30.952471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061465+0000","last_change":"2024-03-06T07:21:50.075967+0000","last_active":"2024-03-06T07:21:51.061465+0000","last_peered":"2024-03-06T07:21:51.061465+0000","last_clean":"2024-03-06T07:21:51.061465+0000","last_became_active":"2024-03-06T07:21:50.075668+0000","last_became_peered":"2024-03-06T07:21:50.075668+0000","last_unstale":"2024-03-06T07:21:51.061465+0000","last_undegraded":"2024-03-06T07:21:51.061465+0000","last_fullsized":"2024-03-06T07:21:51.061465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060628+0000","last_change":"2024-03-06T07:21:50.081328+0000","last_active":"2024-03-06T07:21:51.060628+0000","last_peered":"2024-03-06T07:21:51.060628+0000","last_clean":"2024-03-06T07:21:51.060628+0000","last_became_active":"2024-03-06T07:21:50.080941+0000","last_became_peered":"2024-03-06T07:21:50.080941+0000","last_unstale":"2024-03-06T07:21:51.060628+0000","last_undegraded":"2024-03-06T07:21:51.060628+0000","last_fullsized":"2024-03-06T07:21:51.060628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334303+0000","last_change":"2024-03-06T07:21:50.073822+0000","last_active":"2024-03-06T07:21:51.334303+0000","last_peered":"2024-03-06T07:21:51.334303+0000","last_clean":"2024-03-06T07:21:51.334303+0000","last_became_active":"2024-03-06T07:21:50.073421+0000","last_became_peered":"2024-03-06T07:21:50.073421+0000","last_unstale":"2024-03-06T07:21:51.334303+0000","last_undegraded":"2024-03-06T07:21:51.334303+0000","last_fullsized":"2024-03-06T07:21:51.334303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061065+0000","last_change":"2024-03-06T07:21:50.087355+0000","last_active":"2024-03-06T07:21:51.061065+0000","last_peered":"2024-03-06T07:21:51.061065+0000","last_clean":"2024-03-06T07:21:51.061065+0000","last_became_active":"2024-03-06T07:21:50.086854+0000","last_became_peered":"2024-03-06T07:21:50.086854+0000","last_unstale":"2024-03-06T07:21:51.061065+0000","last_undegraded":"2024-03-06T07:21:51.061065+0000","last_fullsized":"2024-03-06T07:21:51.061065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075461+0000","last_change":"2024-03-06T07:21:50.079046+0000","last_active":"2024-03-06T07:21:51.075461+0000","last_peered":"2024-03-06T07:21:51.075461+0000","last_clean":"2024-03-06T07:21:51.075461+0000","last_became_active":"2024-03-06T07:21:50.078365+0000","last_became_peered":"2024-03-06T07:21:50.078365+0000","last_unstale":"2024-03-06T07:21:51.075461+0000","last_undegraded":"2024-03-06T07:21:51.075461+0000","last_fullsized":"2024-03-06T07:21:51.075461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075358+0000","last_change":"2024-03-06T07:21:50.078291+0000","last_active":"2024-03-06T07:21:51.075358+0000","last_peered":"2024-03-06T07:21:51.075358+0000","last_clean":"2024-03-06T07:21:51.075358+0000","last_became_active":"2024-03-06T07:21:50.078024+0000","last_became_peered":"2024-03-06T07:21:50.078024+0000","last_unstale":"2024-03-06T07:21:51.075358+0000","last_undegraded":"2024-03-06T07:21:51.075358+0000","last_fullsized":"2024-03-06T07:21:51.075358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334132+0000","last_change":"2024-03-06T07:21:50.081726+0000","last_active":"2024-03-06T07:21:51.334132+0000","last_peered":"2024-03-06T07:21:51.334132+0000","last_clean":"2024-03-06T07:21:51.334132+0000","last_became_active":"2024-03-06T07:21:50.081458+0000","last_became_peered":"2024-03-06T07:21:50.081458+0000","last_unstale":"2024-03-06T07:21:51.334132+0000","last_undegraded":"2024-03-06T07:21:51.334132+0000","last_fullsized":"2024-03-06T07:21:51.334132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:41.946707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333563+0000","last_change":"2024-03-06T07:21:50.070195+0000","last_active":"2024-03-06T07:21:51.333563+0000","last_peered":"2024-03-06T07:21:51.333563+0000","last_clean":"2024-03-06T07:21:51.333563+0000","last_became_active":"2024-03-06T07:21:50.069887+0000","last_became_peered":"2024-03-06T07:21:50.069887+0000","last_unstale":"2024-03-06T07:21:51.333563+0000","last_undegraded":"2024-03-06T07:21:51.333563+0000","last_fullsized":"2024-03-06T07:21:51.333563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333869+0000","last_change":"2024-03-06T07:21:50.066290+0000","last_active":"2024-03-06T07:21:51.333869+0000","last_peered":"2024-03-06T07:21:51.333869+0000","last_clean":"2024-03-06T07:21:51.333869+0000","last_became_active":"2024-03-06T07:21:50.066008+0000","last_became_peered":"2024-03-06T07:21:50.066008+0000","last_unstale":"2024-03-06T07:21:51.333869+0000","last_undegraded":"2024-03-06T07:21:51.333869+0000","last_fullsized":"2024-03-06T07:21:51.333869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059103+0000","last_change":"2024-03-06T07:21:50.074011+0000","last_active":"2024-03-06T07:21:51.059103+0000","last_peered":"2024-03-06T07:21:51.059103+0000","last_clean":"2024-03-06T07:21:51.059103+0000","last_became_active":"2024-03-06T07:21:50.073679+0000","last_became_peered":"2024-03-06T07:21:50.073679+0000","last_unstale":"2024-03-06T07:21:51.059103+0000","last_undegraded":"2024-03-06T07:21:51.059103+0000","last_fullsized":"2024-03-06T07:21:51.059103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060633+0000","last_change":"2024-03-06T07:21:50.076532+0000","last_active":"2024-03-06T07:21:51.060633+0000","last_peered":"2024-03-06T07:21:51.060633+0000","last_clean":"2024-03-06T07:21:51.060633+0000","last_became_active":"2024-03-06T07:21:50.075857+0000","last_became_peered":"2024-03-06T07:21:50.075857+0000","last_unstale":"2024-03-06T07:21:51.060633+0000","last_undegraded":"2024-03-06T07:21:51.060633+0000","last_fullsized":"2024-03-06T07:21:51.060633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062233+0000","last_change":"2024-03-06T07:21:50.086644+0000","last_active":"2024-03-06T07:21:51.062233+0000","last_peered":"2024-03-06T07:21:51.062233+0000","last_clean":"2024-03-06T07:21:51.062233+0000","last_became_active":"2024-03-06T07:21:50.084602+0000","last_became_peered":"2024-03-06T07:21:50.084602+0000","last_unstale":"2024-03-06T07:21:51.062233+0000","last_undegraded":"2024-03-06T07:21:51.062233+0000","last_fullsized":"2024-03-06T07:21:51.062233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334765+0000","last_change":"2024-03-06T07:21:50.075838+0000","last_active":"2024-03-06T07:21:51.334765+0000","last_peered":"2024-03-06T07:21:51.334765+0000","last_clean":"2024-03-06T07:21:51.334765+0000","last_became_active":"2024-03-06T07:21:50.075531+0000","last_became_peered":"2024-03-06T07:21:50.075531+0000","last_unstale":"2024-03-06T07:21:51.334765+0000","last_undegraded":"2024-03-06T07:21:51.334765+0000","last_fullsized":"2024-03-06T07:21:51.334765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:00.903565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061683+0000","last_change":"2024-03-06T07:21:50.083575+0000","last_active":"2024-03-06T07:21:51.061683+0000","last_peered":"2024-03-06T07:21:51.061683+0000","last_clean":"2024-03-06T07:21:51.061683+0000","last_became_active":"2024-03-06T07:21:50.083016+0000","last_became_peered":"2024-03-06T07:21:50.083016+0000","last_unstale":"2024-03-06T07:21:51.061683+0000","last_undegraded":"2024-03-06T07:21:51.061683+0000","last_fullsized":"2024-03-06T07:21:51.061683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:20.854195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075317+0000","last_change":"2024-03-06T07:21:50.082493+0000","last_active":"2024-03-06T07:21:51.075317+0000","last_peered":"2024-03-06T07:21:51.075317+0000","last_clean":"2024-03-06T07:21:51.075317+0000","last_became_active":"2024-03-06T07:21:50.082217+0000","last_became_peered":"2024-03-06T07:21:50.082217+0000","last_unstale":"2024-03-06T07:21:51.075317+0000","last_undegraded":"2024-03-06T07:21:51.075317+0000","last_fullsized":"2024-03-06T07:21:51.075317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333683+0000","last_change":"2024-03-06T07:21:50.082417+0000","last_active":"2024-03-06T07:21:51.333683+0000","last_peered":"2024-03-06T07:21:51.333683+0000","last_clean":"2024-03-06T07:21:51.333683+0000","last_became_active":"2024-03-06T07:21:50.082162+0000","last_became_peered":"2024-03-06T07:21:50.082162+0000","last_unstale":"2024-03-06T07:21:51.333683+0000","last_undegraded":"2024-03-06T07:21:51.333683+0000","last_fullsized":"2024-03-06T07:21:51.333683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061987+0000","last_change":"2024-03-06T07:21:50.085327+0000","last_active":"2024-03-06T07:21:51.061987+0000","last_peered":"2024-03-06T07:21:51.061987+0000","last_clean":"2024-03-06T07:21:51.061987+0000","last_became_active":"2024-03-06T07:21:50.085019+0000","last_became_peered":"2024-03-06T07:21:50.085019+0000","last_unstale":"2024-03-06T07:21:51.061987+0000","last_undegraded":"2024-03-06T07:21:51.061987+0000","last_fullsized":"2024-03-06T07:21:51.061987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333521+0000","last_change":"2024-03-06T07:21:50.080803+0000","last_active":"2024-03-06T07:21:51.333521+0000","last_peered":"2024-03-06T07:21:51.333521+0000","last_clean":"2024-03-06T07:21:51.333521+0000","last_became_active":"2024-03-06T07:21:50.080460+0000","last_became_peered":"2024-03-06T07:21:50.080460+0000","last_unstale":"2024-03-06T07:21:51.333521+0000","last_undegraded":"2024-03-06T07:21:51.333521+0000","last_fullsized":"2024-03-06T07:21:51.333521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060865+0000","last_change":"2024-03-06T07:21:50.071650+0000","last_active":"2024-03-06T07:21:51.060865+0000","last_peered":"2024-03-06T07:21:51.060865+0000","last_clean":"2024-03-06T07:21:51.060865+0000","last_became_active":"2024-03-06T07:21:50.071383+0000","last_became_peered":"2024-03-06T07:21:50.071383+0000","last_unstale":"2024-03-06T07:21:51.060865+0000","last_undegraded":"2024-03-06T07:21:51.060865+0000","last_fullsized":"2024-03-06T07:21:51.060865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333966+0000","last_change":"2024-03-06T07:21:50.077589+0000","last_active":"2024-03-06T07:21:51.333966+0000","last_peered":"2024-03-06T07:21:51.333966+0000","last_clean":"2024-03-06T07:21:51.333966+0000","last_became_active":"2024-03-06T07:21:50.077302+0000","last_became_peered":"2024-03-06T07:21:50.077302+0000","last_unstale":"2024-03-06T07:21:51.333966+0000","last_undegraded":"2024-03-06T07:21:51.333966+0000","last_fullsized":"2024-03-06T07:21:51.333966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075478+0000","last_change":"2024-03-06T07:21:50.085031+0000","last_active":"2024-03-06T07:21:51.075478+0000","last_peered":"2024-03-06T07:21:51.075478+0000","last_clean":"2024-03-06T07:21:51.075478+0000","last_became_active":"2024-03-06T07:21:50.084758+0000","last_became_peered":"2024-03-06T07:21:50.084758+0000","last_unstale":"2024-03-06T07:21:51.075478+0000","last_undegraded":"2024-03-06T07:21:51.075478+0000","last_fullsized":"2024-03-06T07:21:51.075478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059511+0000","last_change":"2024-03-06T07:21:50.072164+0000","last_active":"2024-03-06T07:21:51.059511+0000","last_peered":"2024-03-06T07:21:51.059511+0000","last_clean":"2024-03-06T07:21:51.059511+0000","last_became_active":"2024-03-06T07:21:50.071795+0000","last_became_peered":"2024-03-06T07:21:50.071795+0000","last_unstale":"2024-03-06T07:21:51.059511+0000","last_undegraded":"2024-03-06T07:21:51.059511+0000","last_fullsized":"2024-03-06T07:21:51.059511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334595+0000","last_change":"2024-03-06T07:21:50.078217+0000","last_active":"2024-03-06T07:21:51.334595+0000","last_peered":"2024-03-06T07:21:51.334595+0000","last_clean":"2024-03-06T07:21:51.334595+0000","last_became_active":"2024-03-06T07:21:50.077809+0000","last_became_peered":"2024-03-06T07:21:50.077809+0000","last_unstale":"2024-03-06T07:21:51.334595+0000","last_undegraded":"2024-03-06T07:21:51.334595+0000","last_fullsized":"2024-03-06T07:21:51.334595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333826+0000","last_change":"2024-03-06T07:21:50.065564+0000","last_active":"2024-03-06T07:21:51.333826+0000","last_peered":"2024-03-06T07:21:51.333826+0000","last_clean":"2024-03-06T07:21:51.333826+0000","last_became_active":"2024-03-06T07:21:50.065294+0000","last_became_peered":"2024-03-06T07:21:50.065294+0000","last_unstale":"2024-03-06T07:21:51.333826+0000","last_undegraded":"2024-03-06T07:21:51.333826+0000","last_fullsized":"2024-03-06T07:21:51.333826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062062+0000","last_change":"2024-03-06T07:21:50.073038+0000","last_active":"2024-03-06T07:21:51.062062+0000","last_peered":"2024-03-06T07:21:51.062062+0000","last_clean":"2024-03-06T07:21:51.062062+0000","last_became_active":"2024-03-06T07:21:50.072603+0000","last_became_peered":"2024-03-06T07:21:50.072603+0000","last_unstale":"2024-03-06T07:21:51.062062+0000","last_undegraded":"2024-03-06T07:21:51.062062+0000","last_fullsized":"2024-03-06T07:21:51.062062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334317+0000","last_change":"2024-03-06T07:21:50.069962+0000","last_active":"2024-03-06T07:21:51.334317+0000","last_peered":"2024-03-06T07:21:51.334317+0000","last_clean":"2024-03-06T07:21:51.334317+0000","last_became_active":"2024-03-06T07:21:50.069700+0000","last_became_peered":"2024-03-06T07:21:50.069700+0000","last_unstale":"2024-03-06T07:21:51.334317+0000","last_undegraded":"2024-03-06T07:21:51.334317+0000","last_fullsized":"2024-03-06T07:21:51.334317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075081+0000","last_change":"2024-03-06T07:21:50.077457+0000","last_active":"2024-03-06T07:21:51.075081+0000","last_peered":"2024-03-06T07:21:51.075081+0000","last_clean":"2024-03-06T07:21:51.075081+0000","last_became_active":"2024-03-06T07:21:50.077173+0000","last_became_peered":"2024-03-06T07:21:50.077173+0000","last_unstale":"2024-03-06T07:21:51.075081+0000","last_undegraded":"2024-03-06T07:21:51.075081+0000","last_fullsized":"2024-03-06T07:21:51.075081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333933+0000","last_change":"2024-03-06T07:21:50.075503+0000","last_active":"2024-03-06T07:21:51.333933+0000","last_peered":"2024-03-06T07:21:51.333933+0000","last_clean":"2024-03-06T07:21:51.333933+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.333933+0000","last_undegraded":"2024-03-06T07:21:51.333933+0000","last_fullsized":"2024-03-06T07:21:51.333933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:48.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333798+0000","last_change":"2024-03-06T07:21:49.073461+0000","last_active":"2024-03-06T07:21:51.333798+0000","last_peered":"2024-03-06T07:21:51.333798+0000","last_clean":"2024-03-06T07:21:51.333798+0000","last_became_active":"2024-03-06T07:21:49.073212+0000","last_became_peered":"2024-03-06T07:21:49.073212+0000","last_unstale":"2024-03-06T07:21:51.333798+0000","last_undegraded":"2024-03-06T07:21:51.333798+0000","last_fullsized":"2024-03-06T07:21:51.333798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333964+0000","last_change":"2024-03-06T07:21:50.083293+0000","last_active":"2024-03-06T07:21:51.333964+0000","last_peered":"2024-03-06T07:21:51.333964+0000","last_clean":"2024-03-06T07:21:51.333964+0000","last_became_active":"2024-03-06T07:21:50.082931+0000","last_became_peered":"2024-03-06T07:21:50.082931+0000","last_unstale":"2024-03-06T07:21:51.333964+0000","last_undegraded":"2024-03-06T07:21:51.333964+0000","last_fullsized":"2024-03-06T07:21:51.333964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075166+0000","last_change":"2024-03-06T07:21:49.072875+0000","last_active":"2024-03-06T07:21:51.075166+0000","last_peered":"2024-03-06T07:21:51.075166+0000","last_clean":"2024-03-06T07:21:51.075166+0000","last_became_active":"2024-03-06T07:21:49.071548+0000","last_became_peered":"2024-03-06T07:21:49.071548+0000","last_unstale":"2024-03-06T07:21:51.075166+0000","last_undegraded":"2024-03-06T07:21:51.075166+0000","last_fullsized":"2024-03-06T07:21:51.075166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333606+0000","last_change":"2024-03-06T07:21:49.075539+0000","last_active":"2024-03-06T07:21:51.333606+0000","last_peered":"2024-03-06T07:21:51.333606+0000","last_clean":"2024-03-06T07:21:51.333606+0000","last_became_active":"2024-03-06T07:21:49.075192+0000","last_became_peered":"2024-03-06T07:21:49.075192+0000","last_unstale":"2024-03-06T07:21:51.333606+0000","last_undegraded":"2024-03-06T07:21:51.333606+0000","last_fullsized":"2024-03-06T07:21:51.333606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334442+0000","last_change":"2024-03-06T07:21:50.078018+0000","last_active":"2024-03-06T07:21:51.334442+0000","last_peered":"2024-03-06T07:21:51.334442+0000","last_clean":"2024-03-06T07:21:51.334442+0000","last_became_active":"2024-03-06T07:21:50.077638+0000","last_became_peered":"2024-03-06T07:21:50.077638+0000","last_unstale":"2024-03-06T07:21:51.334442+0000","last_undegraded":"2024-03-06T07:21:51.334442+0000","last_fullsized":"2024-03-06T07:21:51.334442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060973+0000","last_change":"2024-03-06T07:21:49.070076+0000","last_active":"2024-03-06T07:21:51.060973+0000","last_peered":"2024-03-06T07:21:51.060973+0000","last_clean":"2024-03-06T07:21:51.060973+0000","last_became_active":"2024-03-06T07:21:49.069491+0000","last_became_peered":"2024-03-06T07:21:49.069491+0000","last_unstale":"2024-03-06T07:21:51.060973+0000","last_undegraded":"2024-03-06T07:21:51.060973+0000","last_fullsized":"2024-03-06T07:21:51.060973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059076+0000","last_change":"2024-03-06T07:21:50.080260+0000","last_active":"2024-03-06T07:21:51.059076+0000","last_peered":"2024-03-06T07:21:51.059076+0000","last_clean":"2024-03-06T07:21:51.059076+0000","last_became_active":"2024-03-06T07:21:50.079987+0000","last_became_peered":"2024-03-06T07:21:50.079987+0000","last_unstale":"2024-03-06T07:21:51.059076+0000","last_undegraded":"2024-03-06T07:21:51.059076+0000","last_fullsized":"2024-03-06T07:21:51.059076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075328+0000","last_change":"2024-03-06T07:21:49.071143+0000","last_active":"2024-03-06T07:21:51.075328+0000","last_peered":"2024-03-06T07:21:51.075328+0000","last_clean":"2024-03-06T07:21:51.075328+0000","last_became_active":"2024-03-06T07:21:49.070872+0000","last_became_peered":"2024-03-06T07:21:49.070872+0000","last_unstale":"2024-03-06T07:21:51.075328+0000","last_undegraded":"2024-03-06T07:21:51.075328+0000","last_fullsized":"2024-03-06T07:21:51.075328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061550+0000","last_change":"2024-03-06T07:21:50.078307+0000","last_active":"2024-03-06T07:21:51.061550+0000","last_peered":"2024-03-06T07:21:51.061550+0000","last_clean":"2024-03-06T07:21:51.061550+0000","last_became_active":"2024-03-06T07:21:50.078025+0000","last_became_peered":"2024-03-06T07:21:50.078025+0000","last_unstale":"2024-03-06T07:21:51.061550+0000","last_undegraded":"2024-03-06T07:21:51.061550+0000","last_fullsized":"2024-03-06T07:21:51.061550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061672+0000","last_change":"2024-03-06T07:21:49.066235+0000","last_active":"2024-03-06T07:21:51.061672+0000","last_peered":"2024-03-06T07:21:51.061672+0000","last_clean":"2024-03-06T07:21:51.061672+0000","last_became_active":"2024-03-06T07:21:49.065968+0000","last_became_peered":"2024-03-06T07:21:49.065968+0000","last_unstale":"2024-03-06T07:21:51.061672+0000","last_undegraded":"2024-03-06T07:21:51.061672+0000","last_fullsized":"2024-03-06T07:21:51.061672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333774+0000","last_change":"2024-03-06T07:21:50.067265+0000","last_active":"2024-03-06T07:21:51.333774+0000","last_peered":"2024-03-06T07:21:51.333774+0000","last_clean":"2024-03-06T07:21:51.333774+0000","last_became_active":"2024-03-06T07:21:50.066634+0000","last_became_peered":"2024-03-06T07:21:50.066634+0000","last_unstale":"2024-03-06T07:21:51.333774+0000","last_undegraded":"2024-03-06T07:21:51.333774+0000","last_fullsized":"2024-03-06T07:21:51.333774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059350+0000","last_change":"2024-03-06T07:21:49.070187+0000","last_active":"2024-03-06T07:21:51.059350+0000","last_peered":"2024-03-06T07:21:51.059350+0000","last_clean":"2024-03-06T07:21:51.059350+0000","last_became_active":"2024-03-06T07:21:49.069894+0000","last_became_peered":"2024-03-06T07:21:49.069894+0000","last_unstale":"2024-03-06T07:21:51.059350+0000","last_undegraded":"2024-03-06T07:21:51.059350+0000","last_fullsized":"2024-03-06T07:21:51.059350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334133+0000","last_change":"2024-03-06T07:21:50.086859+0000","last_active":"2024-03-06T07:21:51.334133+0000","last_peered":"2024-03-06T07:21:51.334133+0000","last_clean":"2024-03-06T07:21:51.334133+0000","last_became_active":"2024-03-06T07:21:50.085246+0000","last_became_peered":"2024-03-06T07:21:50.085246+0000","last_unstale":"2024-03-06T07:21:51.334133+0000","last_undegraded":"2024-03-06T07:21:51.334133+0000","last_fullsized":"2024-03-06T07:21:51.334133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075355+0000","last_change":"2024-03-06T07:21:49.074802+0000","last_active":"2024-03-06T07:21:51.075355+0000","last_peered":"2024-03-06T07:21:51.075355+0000","last_clean":"2024-03-06T07:21:51.075355+0000","last_became_active":"2024-03-06T07:21:49.074143+0000","last_became_peered":"2024-03-06T07:21:49.074143+0000","last_unstale":"2024-03-06T07:21:51.075355+0000","last_undegraded":"2024-03-06T07:21:51.075355+0000","last_fullsized":"2024-03-06T07:21:51.075355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061356+0000","last_change":"2024-03-06T07:21:50.083325+0000","last_active":"2024-03-06T07:21:51.061356+0000","last_peered":"2024-03-06T07:21:51.061356+0000","last_clean":"2024-03-06T07:21:51.061356+0000","last_became_active":"2024-03-06T07:21:50.082675+0000","last_became_peered":"2024-03-06T07:21:50.082675+0000","last_unstale":"2024-03-06T07:21:51.061356+0000","last_undegraded":"2024-03-06T07:21:51.061356+0000","last_fullsized":"2024-03-06T07:21:51.061356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334058+0000","last_change":"2024-03-06T07:21:49.066349+0000","last_active":"2024-03-06T07:21:51.334058+0000","last_peered":"2024-03-06T07:21:51.334058+0000","last_clean":"2024-03-06T07:21:51.334058+0000","last_became_active":"2024-03-06T07:21:49.066024+0000","last_became_peered":"2024-03-06T07:21:49.066024+0000","last_unstale":"2024-03-06T07:21:51.334058+0000","last_undegraded":"2024-03-06T07:21:51.334058+0000","last_fullsized":"2024-03-06T07:21:51.334058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333454+0000","last_change":"2024-03-06T07:21:50.082035+0000","last_active":"2024-03-06T07:21:51.333454+0000","last_peered":"2024-03-06T07:21:51.333454+0000","last_clean":"2024-03-06T07:21:51.333454+0000","last_became_active":"2024-03-06T07:21:50.081731+0000","last_became_peered":"2024-03-06T07:21:50.081731+0000","last_unstale":"2024-03-06T07:21:51.333454+0000","last_undegraded":"2024-03-06T07:21:51.333454+0000","last_fullsized":"2024-03-06T07:21:51.333454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:57.652112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062117+0000","last_change":"2024-03-06T07:21:49.074426+0000","last_active":"2024-03-06T07:21:51.062117+0000","last_peered":"2024-03-06T07:21:51.062117+0000","last_clean":"2024-03-06T07:21:51.062117+0000","last_became_active":"2024-03-06T07:21:49.074117+0000","last_became_peered":"2024-03-06T07:21:49.074117+0000","last_unstale":"2024-03-06T07:21:51.062117+0000","last_undegraded":"2024-03-06T07:21:51.062117+0000","last_fullsized":"2024-03-06T07:21:51.062117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061727+0000","last_change":"2024-03-06T07:21:50.082545+0000","last_active":"2024-03-06T07:21:51.061727+0000","last_peered":"2024-03-06T07:21:51.061727+0000","last_clean":"2024-03-06T07:21:51.061727+0000","last_became_active":"2024-03-06T07:21:50.082197+0000","last_became_peered":"2024-03-06T07:21:50.082197+0000","last_unstale":"2024-03-06T07:21:51.061727+0000","last_undegraded":"2024-03-06T07:21:51.061727+0000","last_fullsized":"2024-03-06T07:21:51.061727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:57.476299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075070+0000","last_change":"2024-03-06T07:21:49.073223+0000","last_active":"2024-03-06T07:21:51.075070+0000","last_peered":"2024-03-06T07:21:51.075070+0000","last_clean":"2024-03-06T07:21:51.075070+0000","last_became_active":"2024-03-06T07:21:49.071609+0000","last_became_peered":"2024-03-06T07:21:49.071609+0000","last_unstale":"2024-03-06T07:21:51.075070+0000","last_undegraded":"2024-03-06T07:21:51.075070+0000","last_fullsized":"2024-03-06T07:21:51.075070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075085+0000","last_change":"2024-03-06T07:21:50.077209+0000","last_active":"2024-03-06T07:21:51.075085+0000","last_peered":"2024-03-06T07:21:51.075085+0000","last_clean":"2024-03-06T07:21:51.075085+0000","last_became_active":"2024-03-06T07:21:50.076936+0000","last_became_peered":"2024-03-06T07:21:50.076936+0000","last_unstale":"2024-03-06T07:21:51.075085+0000","last_undegraded":"2024-03-06T07:21:51.075085+0000","last_fullsized":"2024-03-06T07:21:51.075085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334123+0000","last_change":"2024-03-06T07:21:49.072833+0000","last_active":"2024-03-06T07:21:51.334123+0000","last_peered":"2024-03-06T07:21:51.334123+0000","last_clean":"2024-03-06T07:21:51.334123+0000","last_became_active":"2024-03-06T07:21:49.072543+0000","last_became_peered":"2024-03-06T07:21:49.072543+0000","last_unstale":"2024-03-06T07:21:51.334123+0000","last_undegraded":"2024-03-06T07:21:51.334123+0000","last_fullsized":"2024-03-06T07:21:51.334123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060781+0000","last_change":"2024-03-06T07:21:50.081698+0000","last_active":"2024-03-06T07:21:51.060781+0000","last_peered":"2024-03-06T07:21:51.060781+0000","last_clean":"2024-03-06T07:21:51.060781+0000","last_became_active":"2024-03-06T07:21:50.081294+0000","last_became_peered":"2024-03-06T07:21:50.081294+0000","last_unstale":"2024-03-06T07:21:51.060781+0000","last_undegraded":"2024-03-06T07:21:51.060781+0000","last_fullsized":"2024-03-06T07:21:51.060781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334204+0000","last_change":"2024-03-06T07:21:49.069462+0000","last_active":"2024-03-06T07:21:51.334204+0000","last_peered":"2024-03-06T07:21:51.334204+0000","last_clean":"2024-03-06T07:21:51.334204+0000","last_became_active":"2024-03-06T07:21:49.069161+0000","last_became_peered":"2024-03-06T07:21:49.069161+0000","last_unstale":"2024-03-06T07:21:51.334204+0000","last_undegraded":"2024-03-06T07:21:51.334204+0000","last_fullsized":"2024-03-06T07:21:51.334204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:48.201301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061018+0000","last_change":"2024-03-06T07:21:50.082841+0000","last_active":"2024-03-06T07:21:51.061018+0000","last_peered":"2024-03-06T07:21:51.061018+0000","last_clean":"2024-03-06T07:21:51.061018+0000","last_became_active":"2024-03-06T07:21:50.082543+0000","last_became_peered":"2024-03-06T07:21:50.082543+0000","last_unstale":"2024-03-06T07:21:51.061018+0000","last_undegraded":"2024-03-06T07:21:51.061018+0000","last_fullsized":"2024-03-06T07:21:51.061018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334727+0000","last_change":"2024-03-06T07:21:49.070603+0000","last_active":"2024-03-06T07:21:51.334727+0000","last_peered":"2024-03-06T07:21:51.334727+0000","last_clean":"2024-03-06T07:21:51.334727+0000","last_became_active":"2024-03-06T07:21:49.070341+0000","last_became_peered":"2024-03-06T07:21:49.070341+0000","last_unstale":"2024-03-06T07:21:51.334727+0000","last_undegraded":"2024-03-06T07:21:51.334727+0000","last_fullsized":"2024-03-06T07:21:51.334727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061230+0000","last_change":"2024-03-06T07:21:50.082050+0000","last_active":"2024-03-06T07:21:51.061230+0000","last_peered":"2024-03-06T07:21:51.061230+0000","last_clean":"2024-03-06T07:21:51.061230+0000","last_became_active":"2024-03-06T07:21:50.081723+0000","last_became_peered":"2024-03-06T07:21:50.081723+0000","last_unstale":"2024-03-06T07:21:51.061230+0000","last_undegraded":"2024-03-06T07:21:51.061230+0000","last_fullsized":"2024-03-06T07:21:51.061230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059449+0000","last_change":"2024-03-06T07:21:49.069397+0000","last_active":"2024-03-06T07:21:51.059449+0000","last_peered":"2024-03-06T07:21:51.059449+0000","last_clean":"2024-03-06T07:21:51.059449+0000","last_became_active":"2024-03-06T07:21:49.069101+0000","last_became_peered":"2024-03-06T07:21:49.069101+0000","last_unstale":"2024-03-06T07:21:51.059449+0000","last_undegraded":"2024-03-06T07:21:51.059449+0000","last_fullsized":"2024-03-06T07:21:51.059449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061985+0000","last_change":"2024-03-06T07:21:50.072534+0000","last_active":"2024-03-06T07:21:51.061985+0000","last_peered":"2024-03-06T07:21:51.061985+0000","last_clean":"2024-03-06T07:21:51.061985+0000","last_became_active":"2024-03-06T07:21:50.072252+0000","last_became_peered":"2024-03-06T07:21:50.072252+0000","last_unstale":"2024-03-06T07:21:51.061985+0000","last_undegraded":"2024-03-06T07:21:51.061985+0000","last_fullsized":"2024-03-06T07:21:51.061985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334565+0000","last_change":"2024-03-06T07:21:49.071113+0000","last_active":"2024-03-06T07:21:51.334565+0000","last_peered":"2024-03-06T07:21:51.334565+0000","last_clean":"2024-03-06T07:21:51.334565+0000","last_became_active":"2024-03-06T07:21:49.070854+0000","last_became_peered":"2024-03-06T07:21:49.070854+0000","last_unstale":"2024-03-06T07:21:51.334565+0000","last_undegraded":"2024-03-06T07:21:51.334565+0000","last_fullsized":"2024-03-06T07:21:51.334565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061519+0000","last_change":"2024-03-06T07:21:50.087871+0000","last_active":"2024-03-06T07:21:51.061519+0000","last_peered":"2024-03-06T07:21:51.061519+0000","last_clean":"2024-03-06T07:21:51.061519+0000","last_became_active":"2024-03-06T07:21:50.087545+0000","last_became_peered":"2024-03-06T07:21:50.087545+0000","last_unstale":"2024-03-06T07:21:51.061519+0000","last_undegraded":"2024-03-06T07:21:51.061519+0000","last_fullsized":"2024-03-06T07:21:51.061519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:46.267182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:21:51.061213+0000","last_peered":"2024-03-06T07:21:51.061213+0000","last_clean":"2024-03-06T07:21:51.061213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:21:51.061213+0000","last_undegraded":"2024-03-06T07:21:51.061213+0000","last_fullsized":"2024-03-06T07:21:51.061213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061289+0000","last_change":"2024-03-06T07:21:49.064484+0000","last_active":"2024-03-06T07:21:51.061289+0000","last_peered":"2024-03-06T07:21:51.061289+0000","last_clean":"2024-03-06T07:21:51.061289+0000","last_became_active":"2024-03-06T07:21:49.064202+0000","last_became_peered":"2024-03-06T07:21:49.064202+0000","last_unstale":"2024-03-06T07:21:51.061289+0000","last_undegraded":"2024-03-06T07:21:51.061289+0000","last_fullsized":"2024-03-06T07:21:51.061289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333973+0000","last_change":"2024-03-06T07:21:50.074427+0000","last_active":"2024-03-06T07:21:51.333973+0000","last_peered":"2024-03-06T07:21:51.333973+0000","last_clean":"2024-03-06T07:21:51.333973+0000","last_became_active":"2024-03-06T07:21:50.074131+0000","last_became_peered":"2024-03-06T07:21:50.074131+0000","last_unstale":"2024-03-06T07:21:51.333973+0000","last_undegraded":"2024-03-06T07:21:51.333973+0000","last_fullsized":"2024-03-06T07:21:51.333973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061151+0000","last_change":"2024-03-06T07:21:49.075192+0000","last_active":"2024-03-06T07:21:51.061151+0000","last_peered":"2024-03-06T07:21:51.061151+0000","last_clean":"2024-03-06T07:21:51.061151+0000","last_became_active":"2024-03-06T07:21:49.074904+0000","last_became_peered":"2024-03-06T07:21:49.074904+0000","last_unstale":"2024-03-06T07:21:51.061151+0000","last_undegraded":"2024-03-06T07:21:51.061151+0000","last_fullsized":"2024-03-06T07:21:51.061151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:59.897349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333341+0000","last_change":"2024-03-06T07:21:50.074328+0000","last_active":"2024-03-06T07:21:51.333341+0000","last_peered":"2024-03-06T07:21:51.333341+0000","last_clean":"2024-03-06T07:21:51.333341+0000","last_became_active":"2024-03-06T07:21:50.074021+0000","last_became_peered":"2024-03-06T07:21:50.074021+0000","last_unstale":"2024-03-06T07:21:51.333341+0000","last_undegraded":"2024-03-06T07:21:51.333341+0000","last_fullsized":"2024-03-06T07:21:51.333341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:14.753301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334402+0000","last_change":"2024-03-06T07:21:49.068516+0000","last_active":"2024-03-06T07:21:51.334402+0000","last_peered":"2024-03-06T07:21:51.334402+0000","last_clean":"2024-03-06T07:21:51.334402+0000","last_became_active":"2024-03-06T07:21:49.068184+0000","last_became_peered":"2024-03-06T07:21:49.068184+0000","last_unstale":"2024-03-06T07:21:51.334402+0000","last_undegraded":"2024-03-06T07:21:51.334402+0000","last_fullsized":"2024-03-06T07:21:51.334402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059118+0000","last_change":"2024-03-06T07:21:50.071162+0000","last_active":"2024-03-06T07:21:51.059118+0000","last_peered":"2024-03-06T07:21:51.059118+0000","last_clean":"2024-03-06T07:21:51.059118+0000","last_became_active":"2024-03-06T07:21:50.070848+0000","last_became_peered":"2024-03-06T07:21:50.070848+0000","last_unstale":"2024-03-06T07:21:51.059118+0000","last_undegraded":"2024-03-06T07:21:51.059118+0000","last_fullsized":"2024-03-06T07:21:51.059118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062173+0000","last_change":"2024-03-06T07:21:49.071017+0000","last_active":"2024-03-06T07:21:51.062173+0000","last_peered":"2024-03-06T07:21:51.062173+0000","last_clean":"2024-03-06T07:21:51.062173+0000","last_became_active":"2024-03-06T07:21:49.070588+0000","last_became_peered":"2024-03-06T07:21:49.070588+0000","last_unstale":"2024-03-06T07:21:51.062173+0000","last_undegraded":"2024-03-06T07:21:51.062173+0000","last_fullsized":"2024-03-06T07:21:51.062173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334053+0000","last_change":"2024-03-06T07:21:50.082287+0000","last_active":"2024-03-06T07:21:51.334053+0000","last_peered":"2024-03-06T07:21:51.334053+0000","last_clean":"2024-03-06T07:21:51.334053+0000","last_became_active":"2024-03-06T07:21:50.081767+0000","last_became_peered":"2024-03-06T07:21:50.081767+0000","last_unstale":"2024-03-06T07:21:51.334053+0000","last_undegraded":"2024-03-06T07:21:51.334053+0000","last_fullsized":"2024-03-06T07:21:51.334053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:09.878872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059265+0000","last_change":"2024-03-06T07:21:49.069978+0000","last_active":"2024-03-06T07:21:51.059265+0000","last_peered":"2024-03-06T07:21:51.059265+0000","last_clean":"2024-03-06T07:21:51.059265+0000","last_became_active":"2024-03-06T07:21:49.069223+0000","last_became_peered":"2024-03-06T07:21:49.069223+0000","last_unstale":"2024-03-06T07:21:51.059265+0000","last_undegraded":"2024-03-06T07:21:51.059265+0000","last_fullsized":"2024-03-06T07:21:51.059265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062012+0000","last_change":"2024-03-06T07:21:50.076147+0000","last_active":"2024-03-06T07:21:51.062012+0000","last_peered":"2024-03-06T07:21:51.062012+0000","last_clean":"2024-03-06T07:21:51.062012+0000","last_became_active":"2024-03-06T07:21:50.075840+0000","last_became_peered":"2024-03-06T07:21:50.075840+0000","last_unstale":"2024-03-06T07:21:51.062012+0000","last_undegraded":"2024-03-06T07:21:51.062012+0000","last_fullsized":"2024-03-06T07:21:51.062012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061682+0000","last_change":"2024-03-06T07:21:49.077962+0000","last_active":"2024-03-06T07:21:51.061682+0000","last_peered":"2024-03-06T07:21:51.061682+0000","last_clean":"2024-03-06T07:21:51.061682+0000","last_became_active":"2024-03-06T07:21:49.077633+0000","last_became_peered":"2024-03-06T07:21:49.077633+0000","last_unstale":"2024-03-06T07:21:51.061682+0000","last_undegraded":"2024-03-06T07:21:51.061682+0000","last_fullsized":"2024-03-06T07:21:51.061682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:13.747084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059377+0000","last_change":"2024-03-06T07:21:50.071233+0000","last_active":"2024-03-06T07:21:51.059377+0000","last_peered":"2024-03-06T07:21:51.059377+0000","last_clean":"2024-03-06T07:21:51.059377+0000","last_became_active":"2024-03-06T07:21:50.070925+0000","last_became_peered":"2024-03-06T07:21:50.070925+0000","last_unstale":"2024-03-06T07:21:51.059377+0000","last_undegraded":"2024-03-06T07:21:51.059377+0000","last_fullsized":"2024-03-06T07:21:51.059377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.074529+0000","last_change":"2024-03-06T07:21:49.075388+0000","last_active":"2024-03-06T07:21:51.074529+0000","last_peered":"2024-03-06T07:21:51.074529+0000","last_clean":"2024-03-06T07:21:51.074529+0000","last_became_active":"2024-03-06T07:21:49.075115+0000","last_became_peered":"2024-03-06T07:21:49.075115+0000","last_unstale":"2024-03-06T07:21:51.074529+0000","last_undegraded":"2024-03-06T07:21:51.074529+0000","last_fullsized":"2024-03-06T07:21:51.074529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334111+0000","last_change":"2024-03-06T07:21:50.084419+0000","last_active":"2024-03-06T07:21:51.334111+0000","last_peered":"2024-03-06T07:21:51.334111+0000","last_clean":"2024-03-06T07:21:51.334111+0000","last_became_active":"2024-03-06T07:21:50.084022+0000","last_became_peered":"2024-03-06T07:21:50.084022+0000","last_unstale":"2024-03-06T07:21:51.334111+0000","last_undegraded":"2024-03-06T07:21:51.334111+0000","last_fullsized":"2024-03-06T07:21:51.334111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075351+0000","last_change":"2024-03-06T07:21:49.072877+0000","last_active":"2024-03-06T07:21:51.075351+0000","last_peered":"2024-03-06T07:21:51.075351+0000","last_clean":"2024-03-06T07:21:51.075351+0000","last_became_active":"2024-03-06T07:21:49.072559+0000","last_became_peered":"2024-03-06T07:21:49.072559+0000","last_unstale":"2024-03-06T07:21:51.075351+0000","last_undegraded":"2024-03-06T07:21:51.075351+0000","last_fullsized":"2024-03-06T07:21:51.075351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333756+0000","last_change":"2024-03-06T07:21:50.069013+0000","last_active":"2024-03-06T07:21:51.333756+0000","last_peered":"2024-03-06T07:21:51.333756+0000","last_clean":"2024-03-06T07:21:51.333756+0000","last_became_active":"2024-03-06T07:21:50.068753+0000","last_became_peered":"2024-03-06T07:21:50.068753+0000","last_unstale":"2024-03-06T07:21:51.333756+0000","last_undegraded":"2024-03-06T07:21:51.333756+0000","last_fullsized":"2024-03-06T07:21:51.333756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060433+0000","last_change":"2024-03-06T07:21:49.077101+0000","last_active":"2024-03-06T07:21:51.060433+0000","last_peered":"2024-03-06T07:21:51.060433+0000","last_clean":"2024-03-06T07:21:51.060433+0000","last_became_active":"2024-03-06T07:21:49.076900+0000","last_became_peered":"2024-03-06T07:21:49.076900+0000","last_unstale":"2024-03-06T07:21:51.060433+0000","last_undegraded":"2024-03-06T07:21:51.060433+0000","last_fullsized":"2024-03-06T07:21:51.060433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:37.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334046+0000","last_change":"2024-03-06T07:21:50.072349+0000","last_active":"2024-03-06T07:21:51.334046+0000","last_peered":"2024-03-06T07:21:51.334046+0000","last_clean":"2024-03-06T07:21:51.334046+0000","last_became_active":"2024-03-06T07:21:50.072052+0000","last_became_peered":"2024-03-06T07:21:50.072052+0000","last_unstale":"2024-03-06T07:21:51.334046+0000","last_undegraded":"2024-03-06T07:21:51.334046+0000","last_fullsized":"2024-03-06T07:21:51.334046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:59.429356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059318+0000","last_change":"2024-03-06T07:21:49.072417+0000","last_active":"2024-03-06T07:21:51.059318+0000","last_peered":"2024-03-06T07:21:51.059318+0000","last_clean":"2024-03-06T07:21:51.059318+0000","last_became_active":"2024-03-06T07:21:49.072153+0000","last_became_peered":"2024-03-06T07:21:49.072153+0000","last_unstale":"2024-03-06T07:21:51.059318+0000","last_undegraded":"2024-03-06T07:21:51.059318+0000","last_fullsized":"2024-03-06T07:21:51.059318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334590+0000","last_change":"2024-03-06T07:21:50.068165+0000","last_active":"2024-03-06T07:21:51.334590+0000","last_peered":"2024-03-06T07:21:51.334590+0000","last_clean":"2024-03-06T07:21:51.334590+0000","last_became_active":"2024-03-06T07:21:50.067278+0000","last_became_peered":"2024-03-06T07:21:50.067278+0000","last_unstale":"2024-03-06T07:21:51.334590+0000","last_undegraded":"2024-03-06T07:21:51.334590+0000","last_fullsized":"2024-03-06T07:21:51.334590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059227+0000","last_change":"2024-03-06T07:21:49.074974+0000","last_active":"2024-03-06T07:21:51.059227+0000","last_peered":"2024-03-06T07:21:51.059227+0000","last_clean":"2024-03-06T07:21:51.059227+0000","last_became_active":"2024-03-06T07:21:49.074697+0000","last_became_peered":"2024-03-06T07:21:49.074697+0000","last_unstale":"2024-03-06T07:21:51.059227+0000","last_undegraded":"2024-03-06T07:21:51.059227+0000","last_fullsized":"2024-03-06T07:21:51.059227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333690+0000","last_change":"2024-03-06T07:21:50.078075+0000","last_active":"2024-03-06T07:21:51.333690+0000","last_peered":"2024-03-06T07:21:51.333690+0000","last_clean":"2024-03-06T07:21:51.333690+0000","last_became_active":"2024-03-06T07:21:50.077794+0000","last_became_peered":"2024-03-06T07:21:50.077794+0000","last_unstale":"2024-03-06T07:21:51.333690+0000","last_undegraded":"2024-03-06T07:21:51.333690+0000","last_fullsized":"2024-03-06T07:21:51.333690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061819+0000","last_change":"2024-03-06T07:21:49.073724+0000","last_active":"2024-03-06T07:21:51.061819+0000","last_peered":"2024-03-06T07:21:51.061819+0000","last_clean":"2024-03-06T07:21:51.061819+0000","last_became_active":"2024-03-06T07:21:49.073443+0000","last_became_peered":"2024-03-06T07:21:49.073443+0000","last_unstale":"2024-03-06T07:21:51.061819+0000","last_undegraded":"2024-03-06T07:21:51.061819+0000","last_fullsized":"2024-03-06T07:21:51.061819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:56.645896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075154+0000","last_change":"2024-03-06T07:21:50.081464+0000","last_active":"2024-03-06T07:21:51.075154+0000","last_peered":"2024-03-06T07:21:51.075154+0000","last_clean":"2024-03-06T07:21:51.075154+0000","last_became_active":"2024-03-06T07:21:50.081190+0000","last_became_peered":"2024-03-06T07:21:50.081190+0000","last_unstale":"2024-03-06T07:21:51.075154+0000","last_undegraded":"2024-03-06T07:21:51.075154+0000","last_fullsized":"2024-03-06T07:21:51.075154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061935+0000","last_change":"2024-03-06T07:21:50.074879+0000","last_active":"2024-03-06T07:21:51.061935+0000","last_peered":"2024-03-06T07:21:51.061935+0000","last_clean":"2024-03-06T07:21:51.061935+0000","last_became_active":"2024-03-06T07:21:50.074633+0000","last_became_peered":"2024-03-06T07:21:50.074633+0000","last_unstale":"2024-03-06T07:21:51.061935+0000","last_undegraded":"2024-03-06T07:21:51.061935+0000","last_fullsized":"2024-03-06T07:21:51.061935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061004+0000","last_change":"2024-03-06T07:21:49.078149+0000","last_active":"2024-03-06T07:21:51.061004+0000","last_peered":"2024-03-06T07:21:51.061004+0000","last_clean":"2024-03-06T07:21:51.061004+0000","last_became_active":"2024-03-06T07:21:49.077845+0000","last_became_peered":"2024-03-06T07:21:49.077845+0000","last_unstale":"2024-03-06T07:21:51.061004+0000","last_undegraded":"2024-03-06T07:21:51.061004+0000","last_fullsized":"2024-03-06T07:21:51.061004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061862+0000","last_change":"2024-03-06T07:21:50.073658+0000","last_active":"2024-03-06T07:21:51.061862+0000","last_peered":"2024-03-06T07:21:51.061862+0000","last_clean":"2024-03-06T07:21:51.061862+0000","last_became_active":"2024-03-06T07:21:50.073396+0000","last_became_peered":"2024-03-06T07:21:50.073396+0000","last_unstale":"2024-03-06T07:21:51.061862+0000","last_undegraded":"2024-03-06T07:21:51.061862+0000","last_fullsized":"2024-03-06T07:21:51.061862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061590+0000","last_change":"2024-03-06T07:21:49.078569+0000","last_active":"2024-03-06T07:21:51.061590+0000","last_peered":"2024-03-06T07:21:51.061590+0000","last_clean":"2024-03-06T07:21:51.061590+0000","last_became_active":"2024-03-06T07:21:49.078274+0000","last_became_peered":"2024-03-06T07:21:49.078274+0000","last_unstale":"2024-03-06T07:21:51.061590+0000","last_undegraded":"2024-03-06T07:21:51.061590+0000","last_fullsized":"2024-03-06T07:21:51.061590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:23.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333637+0000","last_change":"2024-03-06T07:21:50.067267+0000","last_active":"2024-03-06T07:21:51.333637+0000","last_peered":"2024-03-06T07:21:51.333637+0000","last_clean":"2024-03-06T07:21:51.333637+0000","last_became_active":"2024-03-06T07:21:50.067014+0000","last_became_peered":"2024-03-06T07:21:50.067014+0000","last_unstale":"2024-03-06T07:21:51.333637+0000","last_undegraded":"2024-03-06T07:21:51.333637+0000","last_fullsized":"2024-03-06T07:21:51.333637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334431+0000","last_change":"2024-03-06T07:21:49.066664+0000","last_active":"2024-03-06T07:21:51.334431+0000","last_peered":"2024-03-06T07:21:51.334431+0000","last_clean":"2024-03-06T07:21:51.334431+0000","last_became_active":"2024-03-06T07:21:49.066350+0000","last_became_peered":"2024-03-06T07:21:49.066350+0000","last_unstale":"2024-03-06T07:21:51.334431+0000","last_undegraded":"2024-03-06T07:21:51.334431+0000","last_fullsized":"2024-03-06T07:21:51.334431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334333+0000","last_change":"2024-03-06T07:21:50.079262+0000","last_active":"2024-03-06T07:21:51.334333+0000","last_peered":"2024-03-06T07:21:51.334333+0000","last_clean":"2024-03-06T07:21:51.334333+0000","last_became_active":"2024-03-06T07:21:50.078776+0000","last_became_peered":"2024-03-06T07:21:50.078776+0000","last_unstale":"2024-03-06T07:21:51.334333+0000","last_undegraded":"2024-03-06T07:21:51.334333+0000","last_fullsized":"2024-03-06T07:21:51.334333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333867+0000","last_change":"2024-03-06T07:21:49.070371+0000","last_active":"2024-03-06T07:21:51.333867+0000","last_peered":"2024-03-06T07:21:51.333867+0000","last_clean":"2024-03-06T07:21:51.333867+0000","last_became_active":"2024-03-06T07:21:49.070100+0000","last_became_peered":"2024-03-06T07:21:49.070100+0000","last_unstale":"2024-03-06T07:21:51.333867+0000","last_undegraded":"2024-03-06T07:21:51.333867+0000","last_fullsized":"2024-03-06T07:21:51.333867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:44.026321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":69305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":660311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:54.457 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:21:54.457 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-06T07:21:54.457 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-06T07:21:54.457 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-06T07:21:54.458 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-06T07:21:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.596+0000 7f231c901640 1 Processor -- start 2024-03-06T07:21:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.596+0000 7f231c901640 1 -- start start 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231c901640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 0x7f231806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231c901640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f231806bc00 0x7f231806d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231c901640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180cc090 con 0x7f231815bd10 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231c901640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23180ae170 con 0x7f231806bc00 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231c901640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23180cc330 con 0x7f231806c5c0 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f231815bd10 0x7f231806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49122/0 (socket says 172.21.15.72:49122) 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f231659c640 1 -- 172.21.15.72:0/2806857878 learned_addr learned my addr 172.21.15.72:0/2806857878 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2316d9d640 1 --2- 172.21.15.72:0/2806857878 >> v2:172.21.15.146:3300/0 conn(0x7f231806bc00 0x7f231806d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f230ffff640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 0x7f231806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578879572 0 0) 0x7f23180cc090 con 0x7f231815bd10 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304003520 con 0x7f231815bd10 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666403122 0 0) 0x7f2304003520 con 0x7f231815bd10 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 msgr2=0x7f231806b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 0x7f231806b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 >> v2:172.21.15.146:3300/0 conn(0x7f231806bc00 msgr2=0x7f231806d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 --2- 172.21.15.72:0/2806857878 >> v2:172.21.15.146:3300/0 conn(0x7f231806bc00 0x7f231806d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180cc610 con 0x7f231815bd10 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2308001670 con 0x7f231815bd10 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 >> v1:172.21.15.72:6789/0 conn(0x7f231815bd10 legacy=0x7f231806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.597+0000 7f2315d9b640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 0x7f23040039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2315d9b640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2304003f30 0x7f2304004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2315d9b640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2304004d40 0x7f23040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180ae170 con 0x7f2304004d40 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180cc330 con 0x7f2304003f30 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230400b9c0 con 0x7f231806c5c0 2024-03-06T07:21:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2316d9d640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2304004d40 0x7f23040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f231659c640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 0x7f23040039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f230ffff640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2304003f30 0x7f2304004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2316d9d640 1 -- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 msgr2=0x7f23040039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2316d9d640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231806c5c0 0x7f23040039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2316d9d640 1 -- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2304003f30 msgr2=0x7f2304004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2316d9d640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2304003f30 0x7f2304004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.598+0000 7f2316d9d640 1 -- 172.21.15.72:0/2806857878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304003520 con 0x7f2304004d40 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f2316d9d640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2304004d40 0x7f23040051c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2310009fc0 tx=0x7f2304005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76bb5e9233d23a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310039070 con 0x7f2304004d40 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231002cc40 con 0x7f2304004d40 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f2315d9b640 1 -- 172.21.15.72:0/2806857878 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310009c10 con 0x7f2304004d40 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f231c901640 1 -- 172.21.15.72:0/2806857878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2304004d40 msgr2=0x7f23040051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f231c901640 1 --2- 172.21.15.72:0/2806857878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2304004d40 0x7f23040051c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2310009fc0 tx=0x7f2304005830 comp rx=0 tx=0).stop 2024-03-06T07:21:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f2316d9d640 1 -- 172.21.15.72:0/2806857878 reap_dead start 2024-03-06T07:21:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f231c901640 1 -- 172.21.15.72:0/2806857878 shutdown_connections 2024-03-06T07:21:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f231c901640 1 -- 172.21.15.72:0/2806857878 >> 172.21.15.72:0/2806857878 conn(0x7f23180d00e0 msgr2=0x7f23180d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f231c901640 1 -- 172.21.15.72:0/2806857878 shutdown_connections 2024-03-06T07:21:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.599+0000 7f231c901640 1 -- 172.21.15.72:0/2806857878 wait complete. 2024-03-06T07:21:54.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 Processor -- start 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 -- start start 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 0x7f23180c5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23181f5c70 0x7f23181f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231814ed80 0x7f231814f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231815c270 con 0x7f23181f5890 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231815c0f0 con 0x7f23181f5c70 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231c901640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231806b6c0 con 0x7f231814ed80 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 0x7f23180c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 0x7f23180c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54278/0 (socket says 172.21.15.72:54278) 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.601+0000 7f231659c640 1 -- 172.21.15.72:0/4284318582 learned_addr learned my addr 172.21.15.72:0/4284318582 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f2316d9d640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231814ed80 0x7f231814f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f230ffff640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23181f5c70 0x7f23181f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231659c640 1 -- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231814ed80 msgr2=0x7f231814f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231659c640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231814ed80 0x7f231814f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231659c640 1 -- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23181f5c70 msgr2=0x7f23181f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231659c640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23181f5c70 0x7f23181f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231659c640 1 -- 172.21.15.72:0/4284318582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231814f720 con 0x7f23181f5890 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f2316d9d640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231814ed80 0x7f231814f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f230ffff640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23181f5c70 0x7f23181f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231659c640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 0x7f23180c5530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f23080017e0 tx=0x7f2308007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230800fa00 con 0x7f23181f5890 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2308018070 con 0x7f23181f5890 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230800fe20 con 0x7f23181f5890 2024-03-06T07:21:54.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23180b3000 con 0x7f23181f5890 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.602+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23180b35c0 con 0x7f23181f5890 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.603+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23180b5fe0 con 0x7f23181f5890 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.603+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f230801d460 con 0x7f23181f5890 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.604+0000 7f230effd640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f22f806b500 0x7f22f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.604+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f230808a470 con 0x7f23181f5890 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.604+0000 7f230ffff640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f22f806b500 0x7f22f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.604+0000 7f230ffff640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f22f806b500 0x7f22f806d9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2318158cf0 tx=0x7f2300008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.608+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f23180b5fe0 con 0x7f23181f5890 2024-03-06T07:21:54.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.828+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f231804f330 con 0x7f23181f5890 2024-03-06T07:21:54.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.831+0000 7f230effd640 1 -- 172.21.15.72:0/4284318582 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171780 (secure 0 0 0) 0x7f23080592f0 con 0x7f23181f5890 2024-03-06T07:21:54.835 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:54.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.833+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f22f806b500 msgr2=0x7f22f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f22f806b500 0x7f22f806d9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2318158cf0 tx=0x7f2300008040 comp rx=0 tx=0).stop 2024-03-06T07:21:54.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 msgr2=0x7f23180c5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 0x7f23180c5530 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f23080017e0 tx=0x7f2308007720 comp rx=0 tx=0).stop 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 shutdown_connections 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f22f806b500 0x7f22f806d9c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f231814ed80 0x7f231814f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23181f5c70 0x7f23181f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 --2- 172.21.15.72:0/4284318582 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23181f5890 0x7f23180c5530 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 >> 172.21.15.72:0/4284318582 conn(0x7f23180d00e0 msgr2=0x7f23180b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 shutdown_connections 2024-03-06T07:21:54.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.834+0000 7f231c901640 1 -- 172.21.15.72:0/4284318582 wait complete. 2024-03-06T07:21:54.846 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":333891523},{"type":"v1","addr":"172.21.15.72:6833","nonce":333891523}]},"active_addr":"172.21.15.72:6833/333891523","active_change":"2024-03-06T07:21:29.939962+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4111,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":591181799}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3322720046}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1294430828}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":899705246}]}]} 2024-03-06T07:21:54.849 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-06T07:21:54.849 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-06T07:21:54.850 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:54.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.951+0000 7f8e8f34b640 1 Processor -- start 2024-03-06T07:21:54.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.951+0000 7f8e8f34b640 1 -- start start 2024-03-06T07:21:54.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8f34b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8e8815bdd0 0x7f8e880c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8f34b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880cebe0 con 0x7f8e880d16e0 2024-03-06T07:21:54.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8f34b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880ceec0 con 0x7f8e8815fa10 2024-03-06T07:21:54.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8f34b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e880ce980 con 0x7f8e8815bdd0 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8d0e7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8e8815bdd0 0x7f8e880c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8d8e8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8e880d16e0 0x7f8e880d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49124/0 (socket says 172.21.15.72:49124) 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8d8e8640 1 -- 172.21.15.72:0/21640020 learned_addr learned my addr 172.21.15.72:0/21640020 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8d0e7640 1 -- 172.21.15.72:0/21640020 >> v1:172.21.15.146:6789/0 conn(0x7f8e8815fa10 legacy=0x7f8e880ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8d0e7640 1 -- 172.21.15.72:0/21640020 >> v1:172.21.15.72:6789/0 conn(0x7f8e880d16e0 legacy=0x7f8e880d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e7ffff640 1 -- 172.21.15.72:0/21640020 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388761569 0 0) 0x7f8e880cebe0 con 0x7f8e880d16e0 2024-03-06T07:21:54.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.952+0000 7f8e8d0e7640 1 -- 172.21.15.72:0/21640020 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e880cf1a0 con 0x7f8e8815bdd0 2024-03-06T07:21:54.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.953+0000 7f8e8d0e7640 1 --2- 172.21.15.72:0/21640020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e880c5610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8e840097a0 tx=0x7f8e8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa29992d796b4b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.953+0000 7f8e7ffff640 1 -- 172.21.15.72:0/21640020 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e840315a0 con 0x7f8e8815bdd0 2024-03-06T07:21:54.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.953+0000 7f8e7ffff640 1 -- 172.21.15.72:0/21640020 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e84037070 con 0x7f8e8815bdd0 2024-03-06T07:21:54.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.953+0000 7f8e7ffff640 1 -- 172.21.15.72:0/21640020 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e84031960 con 0x7f8e8815bdd0 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 -- 172.21.15.72:0/21640020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 msgr2=0x7f8e880c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/21640020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e880c5610 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8e840097a0 tx=0x7f8e8402b840 comp rx=0 tx=0).stop 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 -- 172.21.15.72:0/21640020 shutdown_connections 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/21640020 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e880c5610 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 -- 172.21.15.72:0/21640020 >> 172.21.15.72:0/21640020 conn(0x7f8e8809a9f0 msgr2=0x7f8e8809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 -- 172.21.15.72:0/21640020 shutdown_connections 2024-03-06T07:21:54.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.954+0000 7f8e8f34b640 1 -- 172.21.15.72:0/21640020 wait complete. 2024-03-06T07:21:54.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.956+0000 7f8e8f34b640 1 Processor -- start 2024-03-06T07:21:54.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.956+0000 7f8e8f34b640 1 -- start start 2024-03-06T07:21:54.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8f34b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880d16e0 0x7f8e8815edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8f34b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e8815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8f34b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8815fa10 0x7f8e880c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8f34b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880cc5d0 con 0x7f8e8815bdd0 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8f34b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880c5b50 con 0x7f8e880d16e0 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8f34b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8815c1b0 con 0x7f8e8815fa10 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8c8e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e8815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8c8e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e8815c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54292/0 (socket says 172.21.15.72:54292) 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8c8e6640 1 -- 172.21.15.72:0/4033057943 learned_addr learned my addr 172.21.15.72:0/4033057943 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8d0e7640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880d16e0 0x7f8e8815edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.957+0000 7f8e8d8e8640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8815fa10 0x7f8e880c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8c8e6640 1 -- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8815fa10 msgr2=0x7f8e880c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8c8e6640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8815fa10 0x7f8e880c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8c8e6640 1 -- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880d16e0 msgr2=0x7f8e8815edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:54.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8c8e6640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880d16e0 0x7f8e8815edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8c8e6640 1 -- 172.21.15.72:0/4033057943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8815cd30 con 0x7f8e8815bdd0 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8d8e8640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8815fa10 0x7f8e880c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8c8e6640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e8815c3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800b3c0 tx=0x7f8e7800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e78011830 con 0x7f8e8815bdd0 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e78016070 con 0x7f8e8815bdd0 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8815d010 con 0x7f8e8815bdd0 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e78011c50 con 0x7f8e8815bdd0 2024-03-06T07:21:54.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.958+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8815f300 con 0x7f8e8815bdd0 2024-03-06T07:21:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.959+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8804f330 con 0x7f8e8815bdd0 2024-03-06T07:21:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.960+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8e7801b460 con 0x7f8e8815bdd0 2024-03-06T07:21:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.960+0000 7f8e7dffb640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8e6c06b4d0 0x7f8e6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.960+0000 7f8e8d0e7640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8e6c06b4d0 0x7f8e6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.960+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e78089c30 con 0x7f8e8815bdd0 2024-03-06T07:21:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.960+0000 7f8e8d0e7640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8e6c06b4d0 0x7f8e6c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8e84036000 tx=0x7f8e84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:54.965+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8e78020020 con 0x7f8e8815bdd0 2024-03-06T07:21:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.215+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e880b7a40 con 0x7f8e8815bdd0 2024-03-06T07:21:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.216+0000 7f8e7dffb640 1 -- 172.21.15.72:0/4033057943 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f8e78058ab0 con 0x7f8e8815bdd0 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:55.219 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8e6c06b4d0 msgr2=0x7f8e6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8e6c06b4d0 0x7f8e6c06d990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8e84036000 tx=0x7f8e84008030 comp rx=0 tx=0).stop 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 msgr2=0x7f8e8815c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e8815c3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800b3c0 tx=0x7f8e7800b880 comp rx=0 tx=0).stop 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 shutdown_connections 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8e6c06b4d0 0x7f8e6c06d990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8815fa10 0x7f8e880c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e8815bdd0 0x7f8e8815c3c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 --2- 172.21.15.72:0/4033057943 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e880d16e0 0x7f8e8815edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 >> 172.21.15.72:0/4033057943 conn(0x7f8e8809a9f0 msgr2=0x7f8e8806f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.218+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 shutdown_connections 2024-03-06T07:21:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.219+0000 7f8e8f34b640 1 -- 172.21.15.72:0/4033057943 wait complete. 2024-03-06T07:21:55.232 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-06T07:21:55.232 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:55.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.329+0000 7f1fabc8f640 1 Processor -- start 2024-03-06T07:21:55.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 -- start start 2024-03-06T07:21:55.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1fa415bdd0 0x7f1fa40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fa40d16e0 0x7f1fa40d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa40cebe0 con 0x7f1fa415bdd0 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa40ceec0 con 0x7f1fa40d16e0 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fabc8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40cf1a0 con 0x7f1fa415fa10 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fa9a2b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1fa415bdd0 0x7f1fa40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fa9a2b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1fa415bdd0 0x7f1fa40c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54302/0 (socket says 172.21.15.72:54302) 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.330+0000 7f1fa9a2b640 1 -- 172.21.15.72:0/2097245837 learned_addr learned my addr 172.21.15.72:0/2097245837 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1faa22c640 1 --2- 172.21.15.72:0/2097245837 >> v2:172.21.15.146:3300/0 conn(0x7f1fa40d16e0 0x7f1fa40d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa922a640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa9a2b640 1 -- 172.21.15.72:0/2097245837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 msgr2=0x7f1fa40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa40ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa9a2b640 1 -- 172.21.15.72:0/2097245837 >> v2:172.21.15.146:3300/0 conn(0x7f1fa40d16e0 msgr2=0x7f1fa40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/2097245837 >> v2:172.21.15.146:3300/0 conn(0x7f1fa40d16e0 0x7f1fa40d1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa9a2b640 1 -- 172.21.15.72:0/2097245837 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40cc8e0 con 0x7f1fa415bdd0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa922a640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa40ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa40c5610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40729e0 tx=0x7f1fa002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f787db1a12781ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.331+0000 7f1fa8a29640 1 -- 172.21.15.72:0/2097245837 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa002e070 con 0x7f1fa415bdd0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fa8a29640 1 -- 172.21.15.72:0/2097245837 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa0037070 con 0x7f1fa415bdd0 2024-03-06T07:21:55.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fa8a29640 1 -- 172.21.15.72:0/2097245837 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0031a00 con 0x7f1fa415bdd0 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 -- 172.21.15.72:0/2097245837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 msgr2=0x7f1fa40c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa40c5610 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40729e0 tx=0x7f1fa002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 -- 172.21.15.72:0/2097245837 shutdown_connections 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/2097245837 >> v2:172.21.15.146:3300/0 conn(0x7f1fa40d16e0 0x7f1fa40d1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa40ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/2097245837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa40c5610 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 -- 172.21.15.72:0/2097245837 >> 172.21.15.72:0/2097245837 conn(0x7f1fa409a9f0 msgr2=0x7f1fa409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 -- 172.21.15.72:0/2097245837 shutdown_connections 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.332+0000 7f1fabc8f640 1 -- 172.21.15.72:0/2097245837 wait complete. 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.333+0000 7f1fabc8f640 1 Processor -- start 2024-03-06T07:21:55.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.333+0000 7f1fabc8f640 1 -- start start 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fabc8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fa40d16e0 0x7f1fa409b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fabc8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa415aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fabc8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa415b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fabc8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40cc5d0 con 0x7f1fa415bdd0 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fabc8f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40d2070 con 0x7f1fa40d16e0 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fabc8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa415c1b0 con 0x7f1fa415fa10 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa415aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa415aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54316/0 (socket says 172.21.15.72:54316) 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 -- 172.21.15.72:0/1884130200 learned_addr learned my addr 172.21.15.72:0/1884130200 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1faa22c640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa415b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fa40d16e0 0x7f1fa409b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 -- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 msgr2=0x7f1fa415b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa415b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 -- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fa40d16e0 msgr2=0x7f1fa409b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fa40d16e0 0x7f1fa409b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 -- 172.21.15.72:0/1884130200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa415c9f0 con 0x7f1fa415bdd0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1faa22c640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa415b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fa40d16e0 0x7f1fa409b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.334+0000 7f1fa922a640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa415aac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400c7f0 tx=0x7f1f9400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.335+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94015070 con 0x7f1fa415bdd0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.335+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f94007a30 con 0x7f1fa415bdd0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.335+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa415ccd0 con 0x7f1fa415bdd0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.335+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94007e00 con 0x7f1fa415bdd0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.335+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa409be40 con 0x7f1fa415bdd0 2024-03-06T07:21:55.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.336+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa404f330 con 0x7f1fa415bdd0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.337+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f94018070 con 0x7f1fa415bdd0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.337+0000 7f1f9a7fc640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f8806b4d0 0x7f1f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.337+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f9408a450 con 0x7f1fa415bdd0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.337+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f8806b4d0 0x7f1f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.337+0000 7f1fa9a2b640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f8806b4d0 0x7f1f8806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0036000 tx=0x7f1fa003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.340+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f94059440 con 0x7f1fa415bdd0 2024-03-06T07:21:55.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.520+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fa40c79e0 con 0x7f1fa415bdd0 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.521+0000 7f1f9a7fc640 1 -- 172.21.15.72:0/1884130200 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f1f94058d40 con 0x7f1fa415bdd0 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:55.523 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f8806b4d0 msgr2=0x7f1f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f8806b4d0 0x7f1f8806d990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0036000 tx=0x7f1fa003d040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 msgr2=0x7f1fa415aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa415aac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400c7f0 tx=0x7f1f9400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 shutdown_connections 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f8806b4d0 0x7f1f8806d990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fa415fa10 0x7f1fa415b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fa415bdd0 0x7f1fa415aac0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 --2- 172.21.15.72:0/1884130200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fa40d16e0 0x7f1fa409b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 >> 172.21.15.72:0/1884130200 conn(0x7f1fa409a9f0 msgr2=0x7f1fa409add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 shutdown_connections 2024-03-06T07:21:55.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.523+0000 7f1fabc8f640 1 -- 172.21.15.72:0/1884130200 wait complete. 2024-03-06T07:21:55.536 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-06T07:21:55.537 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-06T07:21:55.537 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-06T07:21:55.538 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-06T07:21:55.538 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-06T07:21:55.538 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-06T07:21:55.539 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-06T07:21:55.539 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-06T07:21:55.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.706+0000 7f0ce8cdf640 1 Processor -- start 2024-03-06T07:21:55.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.707+0000 7fd8484c9640 1 Processor -- start 2024-03-06T07:21:55.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.707+0000 7fc889bd7640 1 Processor -- start 2024-03-06T07:21:55.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.707+0000 7fc889bd7640 1 -- start start 2024-03-06T07:21:55.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.707+0000 7f0ce8cdf640 1 -- start start 2024-03-06T07:21:55.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce8cdf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce40b8020 0x7f0ce416ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce8cdf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce406c530 con 0x7f0ce40b5760 2024-03-06T07:21:55.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce8cdf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40b0950 con 0x7f0ce406b9e0 2024-03-06T07:21:55.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce8cdf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce40b0c30 con 0x7f0ce40b8020 2024-03-06T07:21:55.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc889bd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c005e90 0x7fc87c0062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc889bd7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc87c006810 0x7fc87c006ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc889bd7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc87c053720 con 0x7fc87c0ead00 2024-03-06T07:21:55.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f664dbe5640 1 Processor -- start 2024-03-06T07:21:55.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f664dbe5640 1 -- start start 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0ce40b5760 0x7f0ce40b5b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49126/0 (socket says 172.21.15.72:49126) 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce259c640 1 -- 172.21.15.72:0/3202220540 learned_addr learned my addr 172.21.15.72:0/3202220540 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7f0ce2d9d640 1 --2- 172.21.15.72:0/3202220540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce40b8020 0x7f0ce416ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311416584 0 0) 0x7f0ce40b0950 con 0x7f0ce406b9e0 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd0003520 con 0x7f0ce406b9e0 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532558507 0 0) 0x7f0ce406c530 con 0x7f0ce40b5760 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce40b0950 con 0x7f0ce40b5760 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce2d9d640 1 -- 172.21.15.72:0/3202220540 >> v1:172.21.15.146:6789/0 conn(0x7f0ce406b9e0 legacy=0x7f0ce406be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce2d9d640 1 -- 172.21.15.72:0/3202220540 >> v1:172.21.15.72:6789/0 conn(0x7f0ce40b5760 legacy=0x7f0ce40b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f0ce2d9d640 1 -- 172.21.15.72:0/3202220540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40b0f10 con 0x7f0ce40b8020 2024-03-06T07:21:55.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f2c57be2640 1 Processor -- start 2024-03-06T07:21:55.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fd8484c9640 1 -- start start 2024-03-06T07:21:55.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd8484c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd838048100 0x7fd838006f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd8484c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd838005ec0 con 0x7fd838007460 2024-03-06T07:21:55.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd8484c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd838053280 con 0x7fd838054560 2024-03-06T07:21:55.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd8484c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380e7b90 con 0x7fd838048100 2024-03-06T07:21:55.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc883fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc87c006810 0x7fc87c006ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc883fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc87c006810 0x7fc87c006ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54322/0 (socket says 172.21.15.72:54322) 2024-03-06T07:21:55.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc883fff640 1 -- 172.21.15.72:0/3513029114 learned_addr learned my addr 172.21.15.72:0/3513029114 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.708+0000 7fc882ffd640 1 --2- 172.21.15.72:0/3513029114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c005e90 0x7fc87c0062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd845a64640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd838007460 0x7fd838053e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49140/0 (socket says 172.21.15.72:49140) 2024-03-06T07:21:55.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd845a64640 1 -- 172.21.15.72:0/884784120 learned_addr learned my addr 172.21.15.72:0/884784120 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:55.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87c054330 con 0x7fc87c006810 2024-03-06T07:21:55.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c053a00 con 0x7fc87c005e90 2024-03-06T07:21:55.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fc8827fc640 1 -- 172.21.15.72:0/3513029114 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459077491 0 0) 0x7fc87c053720 con 0x7fc87c0ead00 2024-03-06T07:21:55.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7f0ce2d9d640 1 --2- 172.21.15.72:0/3202220540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce40b8020 0x7f0ce416ce20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009dc0 tx=0x7f0cdc02bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b61cb345542f531 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850601417 0 0) 0x7fd838053280 con 0x7fd838054560 2024-03-06T07:21:55.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd82c003520 con 0x7fd838054560 2024-03-06T07:21:55.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693214093 0 0) 0x7fd838005ec0 con 0x7fd838007460 2024-03-06T07:21:55.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7f664dbe5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66400e7690 0x7f6640052c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd838053280 con 0x7fd838007460 2024-03-06T07:21:55.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc039070 con 0x7f0ce40b8020 2024-03-06T07:21:55.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664dbe5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6640051bc0 con 0x7f6640038e60 2024-03-06T07:21:55.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664dbe5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6640051ea0 con 0x7f664004a3e0 2024-03-06T07:21:55.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664dbe5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66400527d0 con 0x7f66400e7690 2024-03-06T07:21:55.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc02cd50 con 0x7f0ce40b8020 2024-03-06T07:21:55.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664cbe3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66400e7690 0x7f6640052c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.710+0000 7f0ce1d9b640 1 -- 172.21.15.72:0/3202220540 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc034660 con 0x7f0ce40b8020 2024-03-06T07:21:55.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fc8827fc640 1 -- 172.21.15.72:0/3513029114 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc874003520 con 0x7fc87c0ead00 2024-03-06T07:21:55.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664cbe3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66400e7690 0x7f6640052c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42238/0 (socket says 172.21.15.72:42238) 2024-03-06T07:21:55.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664cbe3640 1 -- 172.21.15.72:0/1587565876 learned_addr learned my addr 172.21.15.72:0/1587565876 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377139483 0 0) 0x7fd838053280 con 0x7fd838007460 2024-03-06T07:21:55.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fc883fff640 1 -- 172.21.15.72:0/3513029114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c005e90 msgr2=0x7fc87c0062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fc883fff640 1 --2- 172.21.15.72:0/3513029114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c005e90 0x7fc87c0062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fc883fff640 1 -- 172.21.15.72:0/3513029114 >> v1:172.21.15.146:6789/0 conn(0x7fc87c0ead00 legacy=0x7fc87c04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fc883fff640 1 -- 172.21.15.72:0/3513029114 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c053ce0 con 0x7fc87c006810 2024-03-06T07:21:55.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.709+0000 7f2c57be2640 1 -- start start 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 >> v2:172.21.15.146:3300/0 conn(0x7fd838048100 msgr2=0x7fd838006f20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 --2- 172.21.15.72:0/884784120 >> v2:172.21.15.146:3300/0 conn(0x7fd838048100 0x7fd838006f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 >> v1:172.21.15.146:6789/0 conn(0x7fd838054560 legacy=0x7fd83804f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838006160 con 0x7fd838007460 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f9086765640 1 Processor -- start 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd83c001520 con 0x7fd838007460 2024-03-06T07:21:55.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 >> v1:172.21.15.72:6789/0 conn(0x7fd838007460 legacy=0x7fd838053e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c004bd0 0x7fd82c004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0037a0 0x7fd82c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664cbe3640 1 -- 172.21.15.72:0/1587565876 >> v1:172.21.15.146:6789/0 conn(0x7f664004a3e0 legacy=0x7f6640038750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7fd845263640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd82c004160 0x7fd82c00ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f664cbe3640 1 -- 172.21.15.72:0/1587565876 >> v1:172.21.15.72:6789/0 conn(0x7f6640038e60 legacy=0x7f6640007be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f664cbe3640 1 -- 172.21.15.72:0/1587565876 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640052180 con 0x7f66400e7690 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/3202220540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce40b8020 msgr2=0x7f0ce416ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f0ce8cdf640 1 --2- 172.21.15.72:0/3202220540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce40b8020 0x7f0ce416ce20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009dc0 tx=0x7f0cdc02bfa0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8380e7b90 con 0x7fd82c004160 2024-03-06T07:21:55.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c00c150 con 0x7fd82c0037a0 2024-03-06T07:21:55.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd846a66640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd82c004160 0x7fd82c00ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/3202220540 shutdown_connections 2024-03-06T07:21:55.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f2c57be2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c480ead00 0x7f2c4804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f2c57be2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c4804ffd0 0x7f2c48037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f2c57be2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c48038110 0x7f2c480385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f2c57be2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c48054330 con 0x7f2c48038110 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f2c57be2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c48053720 con 0x7f2c4804ffd0 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f2c57be2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48053a00 con 0x7f2c480ead00 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd82c00c350 con 0x7fd82c004bd0 2024-03-06T07:21:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845a64640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0037a0 0x7fd82c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f2c5617f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c48038110 0x7f2c480385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd846265640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c004bd0 0x7fd82c004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f2c5617f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c48038110 0x7f2c480385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54332/0 (socket says 172.21.15.72:54332) 2024-03-06T07:21:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f0ce8cdf640 1 --2- 172.21.15.72:0/3202220540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce40b8020 0x7f0ce416ce20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/3202220540 >> 172.21.15.72:0/3202220540 conn(0x7f0ce40b91c0 msgr2=0x7f0ce40b95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f2c5617f640 1 -- 172.21.15.72:0/2218896283 learned_addr learned my addr 172.21.15.72:0/2218896283 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/3202220540 shutdown_connections 2024-03-06T07:21:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845a64640 1 -- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c004bd0 msgr2=0x7fd82c004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845a64640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c004bd0 0x7fd82c004fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845a64640 1 -- 172.21.15.72:0/884784120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd82c004160 msgr2=0x7fd82c00ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fc883fff640 1 --2- 172.21.15.72:0/3513029114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c006810 0x7fc87c006ca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc870009870 tx=0x7fc87002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9560ea29c5d992e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7fc8827fc640 1 -- 172.21.15.72:0/3513029114 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870034070 con 0x7fc87c006810 2024-03-06T07:21:55.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7fc8827fc640 1 -- 172.21.15.72:0/3513029114 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc870030860 con 0x7fc87c006810 2024-03-06T07:21:55.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7fc8827fc640 1 -- 172.21.15.72:0/3513029114 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870030c30 con 0x7fc87c006810 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.711+0000 7f9086765640 1 -- start start 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9086765640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90800afae0 0x7f908006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9086765640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908006f290 0x7f908006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9086765640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90800b3a90 con 0x7f9080161ed0 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9086765640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90800b3d70 con 0x7f90800afae0 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9086765640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90800b57a0 con 0x7f908006f290 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90800afae0 0x7f908006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90800afae0 0x7f908006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54356/0 (socket says 172.21.15.72:54356) 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 -- 172.21.15.72:0/1096940063 learned_addr learned my addr 172.21.15.72:0/1096940063 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9085f64640 1 --2- 172.21.15.72:0/1096940063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908006f290 0x7f908006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845a64640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd82c004160 0x7fd82c00ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7fd845a64640 1 -- 172.21.15.72:0/884784120 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838053280 con 0x7fd82c0037a0 2024-03-06T07:21:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 -- 172.21.15.72:0/1096940063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908006f290 msgr2=0x7f908006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 --2- 172.21.15.72:0/1096940063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908006f290 0x7f908006b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 -- 172.21.15.72:0/1096940063 >> v1:172.21.15.72:6789/0 conn(0x7f9080161ed0 legacy=0x7f90801622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f9084f62640 1 -- 172.21.15.72:0/1096940063 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90801629c0 con 0x7f90800afae0 2024-03-06T07:21:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/3202220540 wait complete. 2024-03-06T07:21:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f2c5617f640 1 -- 172.21.15.72:0/2218896283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c480ead00 msgr2=0x7f2c4804fa90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7f664cbe3640 1 --2- 172.21.15.72:0/1587565876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66400e7690 0x7f6640052c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f663c009870 tx=0x7f663c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=336d3a8d428a0d74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.712+0000 7f2c5617f640 1 --2- 172.21.15.72:0/2218896283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c480ead00 0x7f2c4804fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f2c5617f640 1 -- 172.21.15.72:0/2218896283 >> v2:172.21.15.146:3300/0 conn(0x7f2c4804ffd0 msgr2=0x7f2c48037bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-06T07:21:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f2c5617f640 1 --2- 172.21.15.72:0/2218896283 >> v2:172.21.15.146:3300/0 conn(0x7f2c4804ffd0 0x7f2c48037bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7f2c5617f640 1 -- 172.21.15.72:0/2218896283 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c48053ce0 con 0x7f2c48038110 2024-03-06T07:21:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c006810 msgr2=0x7fc87c006ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fc889bd7640 1 --2- 172.21.15.72:0/3513029114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c006810 0x7fc87c006ca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc870009870 tx=0x7fc87002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7f66477fe640 1 -- 172.21.15.72:0/1587565876 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c035020 con 0x7f66400e7690 2024-03-06T07:21:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.713+0000 7fd846265640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c004bd0 0x7fd82c004fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fd846a66640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd82c004160 0x7fd82c00ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fd846a66640 1 -- 172.21.15.72:0/884784120 reap_dead start 2024-03-06T07:21:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 shutdown_connections 2024-03-06T07:21:55.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7f66477fe640 1 -- 172.21.15.72:0/1587565876 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c031870 con 0x7f66400e7690 2024-03-06T07:21:55.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fc889bd7640 1 --2- 172.21.15.72:0/3513029114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c006810 0x7fc87c006ca0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7fc889bd7640 1 --2- 172.21.15.72:0/3513029114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c005e90 0x7fc87c0062d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7f9084f62640 1 --2- 172.21.15.72:0/1096940063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800afae0 0x7f908006afb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9078009e50 tx=0x7f907802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7023ac18ad0e45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f0ce8cdf640 1 Processor -- start 2024-03-06T07:21:55.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f66477fe640 1 -- 172.21.15.72:0/1587565876 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c031c40 con 0x7f66400e7690 2024-03-06T07:21:55.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 >> 172.21.15.72:0/3513029114 conn(0x7fc87c009fb0 msgr2=0x7fc87c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.714+0000 7f2c5617f640 1 --2- 172.21.15.72:0/2218896283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480385a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c480524d0 tx=0x7f2c4002bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778ca81c88b0f09c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f0ce8cdf640 1 -- start start 2024-03-06T07:21:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f2c5497c640 1 -- 172.21.15.72:0/2218896283 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40039070 con 0x7f2c48038110 2024-03-06T07:21:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f2c5497c640 1 -- 172.21.15.72:0/2218896283 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4002c750 con 0x7f2c48038110 2024-03-06T07:21:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7fd845a64640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0037a0 0x7fd82c003c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd83c007ef0 tx=0x7fd82c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=414197cf6da9af8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f9077fff640 1 -- 172.21.15.72:0/1096940063 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907802ca70 con 0x7f90800afae0 2024-03-06T07:21:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f9077fff640 1 -- 172.21.15.72:0/1096940063 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907802cc10 con 0x7f90800afae0 2024-03-06T07:21:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f9077fff640 1 -- 172.21.15.72:0/1096940063 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90780343f0 con 0x7f90800afae0 2024-03-06T07:21:55.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c007b10 con 0x7fd82c0037a0 2024-03-06T07:21:55.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c030070 con 0x7fd82c0037a0 2024-03-06T07:21:55.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 shutdown_connections 2024-03-06T07:21:55.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7fd845263640 1 -- 172.21.15.72:0/884784120 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c005d30 con 0x7fd82c0037a0 2024-03-06T07:21:55.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.715+0000 7f2c5497c640 1 -- 172.21.15.72:0/2218896283 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4002cb20 con 0x7f2c48038110 2024-03-06T07:21:55.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f0ce8cdf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce406b9e0 0x7f0ce4160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 -- 172.21.15.72:0/2218896283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 msgr2=0x7f2c480385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f0ce8cdf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ce40b5760 0x7f0ce41611f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 --2- 172.21.15.72:0/2218896283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480385a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2c480524d0 tx=0x7f2c4002bbb0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f0ce8cdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ce40b8020 0x7f0ce409da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 -- 172.21.15.72:0/2218896283 shutdown_connections 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f0ce8cdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce4073380 con 0x7f0ce40b8020 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f0ce8cdf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce40b14d0 con 0x7f0ce40b5760 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f0ce8cdf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce4073500 con 0x7f0ce406b9e0 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7fd8484c9640 1 -- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0037a0 msgr2=0x7fd82c003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7fd8484c9640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0037a0 0x7fd82c003c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd83c007ef0 tx=0x7fd82c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7fc889bd7640 1 -- 172.21.15.72:0/3513029114 wait complete. 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 --2- 172.21.15.72:0/2218896283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480385a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 --2- 172.21.15.72:0/2218896283 >> v2:172.21.15.146:3300/0 conn(0x7f2c4804ffd0 0x7f2c48037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 --2- 172.21.15.72:0/2218896283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c480ead00 0x7f2c4804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 -- 172.21.15.72:0/2218896283 >> 172.21.15.72:0/2218896283 conn(0x7f2c48009fb0 msgr2=0x7f2c4800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 -- 172.21.15.72:0/2218896283 shutdown_connections 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f9086765640 1 -- 172.21.15.72:0/1096940063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800afae0 msgr2=0x7f908006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f9086765640 1 --2- 172.21.15.72:0/1096940063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800afae0 0x7f908006afb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9078009e50 tx=0x7f907802bed0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f2c57be2640 1 -- 172.21.15.72:0/2218896283 wait complete. 2024-03-06T07:21:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f9086765640 1 -- 172.21.15.72:0/1096940063 shutdown_connections 2024-03-06T07:21:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f9086765640 1 --2- 172.21.15.72:0/1096940063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f908006f290 0x7f908006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f9086765640 1 --2- 172.21.15.72:0/1096940063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800afae0 0x7f908006afb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.716+0000 7f9086765640 1 -- 172.21.15.72:0/1096940063 >> 172.21.15.72:0/1096940063 conn(0x7f908006c0c0 msgr2=0x7f908006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f9086765640 1 -- 172.21.15.72:0/1096940063 shutdown_connections 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f9086765640 1 -- 172.21.15.72:0/1096940063 wait complete. 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7fd8484c9640 1 -- 172.21.15.72:0/884784120 shutdown_connections 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7fd8484c9640 1 --2- 172.21.15.72:0/884784120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd82c0037a0 0x7fd82c003c20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7fd8484c9640 1 -- 172.21.15.72:0/884784120 >> 172.21.15.72:0/884784120 conn(0x7fd838009fe0 msgr2=0x7fd83800a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce406b9e0 0x7f0ce4160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce406b9e0 0x7f0ce4160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33644/0 (socket says 172.21.15.72:33644) 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 -- 172.21.15.72:0/762448269 learned_addr learned my addr 172.21.15.72:0/762448269 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce2d9d640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ce40b8020 0x7f0ce409da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 -- 172.21.15.72:0/762448269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ce40b5760 msgr2=0x7f0ce41611f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ce40b5760 0x7f0ce41611f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 -- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ce40b8020 msgr2=0x7f0ce409da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ce40b8020 0x7f0ce409da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 -- 172.21.15.72:0/762448269 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40a1660 con 0x7f0ce406b9e0 2024-03-06T07:21:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.717+0000 7f0ce259c640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce406b9e0 0x7f0ce4160cb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400c490 tx=0x7f0cd400c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4017070 con 0x7f0ce406b9e0 2024-03-06T07:21:55.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd4005ce0 con 0x7f0ce406b9e0 2024-03-06T07:21:55.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4007270 con 0x7f0ce406b9e0 2024-03-06T07:21:55.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7f2c57be2640 1 Processor -- start 2024-03-06T07:21:55.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7f2c57be2640 1 -- start start 2024-03-06T07:21:55.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7fc889bd7640 1 Processor -- start 2024-03-06T07:21:55.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7fd8484c9640 1 -- 172.21.15.72:0/884784120 shutdown_connections 2024-03-06T07:21:55.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 -- 172.21.15.72:0/1587565876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66400e7690 msgr2=0x7f6640052c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 --2- 172.21.15.72:0/1587565876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66400e7690 0x7f6640052c60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f663c009870 tx=0x7f663c02b910 comp rx=0 tx=0).stop 2024-03-06T07:21:55.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 -- 172.21.15.72:0/1587565876 shutdown_connections 2024-03-06T07:21:55.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 --2- 172.21.15.72:0/1587565876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66400e7690 0x7f6640052c60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 -- 172.21.15.72:0/1587565876 >> 172.21.15.72:0/1587565876 conn(0x7f6640053750 msgr2=0x7f6640053b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7fc889bd7640 1 -- start start 2024-03-06T07:21:55.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.718+0000 7f9086765640 1 Processor -- start 2024-03-06T07:21:55.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c57be2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c57be2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4804ffd0 0x7f2c4804caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c57be2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c480ead00 0x7f2c48006e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c57be2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48054940 con 0x7f2c48038110 2024-03-06T07:21:55.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c57be2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c480547c0 con 0x7f2c480ead00 2024-03-06T07:21:55.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c57be2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4804f550 con 0x7f2c4804ffd0 2024-03-06T07:21:55.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7fd8484c9640 1 -- 172.21.15.72:0/884784120 wait complete. 2024-03-06T07:21:55.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 -- 172.21.15.72:0/1587565876 shutdown_connections 2024-03-06T07:21:55.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f664dbe5640 1 -- 172.21.15.72:0/1587565876 wait complete. 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f9086765640 1 -- start start 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f9086765640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006f290 0x7f908012cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f9086765640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90800afae0 0x7f90800b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f9086765640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9080161ed0 0x7f9080148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f9086765640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90800b5540 con 0x7f908006f290 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f9086765640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90800b53c0 con 0x7f9080161ed0 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f9086765640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908006e380 con 0x7f90800afae0 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c5597e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c5597e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54366/0 (socket says 172.21.15.72:54366) 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.719+0000 7f2c5597e640 1 -- 172.21.15.72:0/1120810565 learned_addr learned my addr 172.21.15.72:0/1120810565 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f2c5597e640 1 -- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4804ffd0 msgr2=0x7f2c4804caa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f2c5597e640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4804ffd0 0x7f2c4804caa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f2c5597e640 1 -- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c480ead00 msgr2=0x7f2c48006e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f2c5597e640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c480ead00 0x7f2c48006e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f2c5597e640 1 -- 172.21.15.72:0/1120810565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c480cdfe0 con 0x7f2c48038110 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fc889bd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c005e90 0x7fc87c180970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/762448269 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce414c7e0 con 0x7f0ce406b9e0 2024-03-06T07:21:55.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fc889bd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c006810 0x7fc87c04cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fc8837fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c005e90 0x7fc87c180970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/762448269 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce409df70 con 0x7f0ce406b9e0 2024-03-06T07:21:55.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fd8484c9640 1 Processor -- start 2024-03-06T07:21:55.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fd8484c9640 1 -- start start 2024-03-06T07:21:55.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f2c5597e640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480cd790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2c500500e0 tx=0x7f2c500683c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7f625987d640 1 Processor -- start 2024-03-06T07:21:55.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.721+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50092270 con 0x7f2c48038110 2024-03-06T07:21:55.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.721+0000 7f625987d640 1 -- start start 2024-03-06T07:21:55.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.721+0000 7f2c57be2640 1 -- 172.21.15.72:0/1120810565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c480ce2c0 con 0x7f2c48038110 2024-03-06T07:21:55.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.721+0000 7f664dbe5640 1 Processor -- start 2024-03-06T07:21:55.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.721+0000 7f664dbe5640 1 -- start start 2024-03-06T07:21:55.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fc8837fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c005e90 0x7fc87c180970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54398/0 (socket says 172.21.15.72:54398) 2024-03-06T07:21:55.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc882ffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c006810 0x7fc87c04cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc882ffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c006810 0x7fc87c04cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33652/0 (socket says 172.21.15.72:33652) 2024-03-06T07:21:55.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.720+0000 7fc889bd7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc87c0ead00 0x7fc87c0e2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc8837fe640 1 -- 172.21.15.72:0/3215294663 learned_addr learned my addr 172.21.15.72:0/3215294663 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd8484c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c0037a0 0x7fd8380d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0547c0 con 0x7fc87c005e90 2024-03-06T07:21:55.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c0501a0 con 0x7fc87c0ead00 2024-03-06T07:21:55.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd8484c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8380451a0 0x7fd8380499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c054940 con 0x7fc87c006810 2024-03-06T07:21:55.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc883fff640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc87c0ead00 0x7fc87c0e2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd8484c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8380436b0 0x7fd838049ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625987d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 0x7f625415c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625987d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62540ae7d0 0x7f62540aec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625987d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f625415cc30 con 0x7f62540cff00 2024-03-06T07:21:55.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625987d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f625415cf10 con 0x7f62540ae7d0 2024-03-06T07:21:55.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625987d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f625415d1f0 con 0x7f6254156250 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd8484c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838006720 con 0x7fd8380436b0 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd8484c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838006a90 con 0x7fd8380451a0 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd845a64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8380451a0 0x7fd8380499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd845a64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8380451a0 0x7fd8380499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42310/0 (socket says 172.21.15.72:42310) 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd845a64640 1 -- 172.21.15.72:0/617664262 learned_addr learned my addr 172.21.15.72:0/617664262 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fd8484c9640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd838006f20 con 0x7fd82c0037a0 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fd846265640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c0037a0 0x7fd8380d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/762448269 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ce40a1c30 con 0x7f0ce406b9e0 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c50091070 con 0x7f2c48038110 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625887b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 0x7f625415c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625907c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62540ae7d0 0x7f62540aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f6253fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f62540cff00 0x7f625406d100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49158/0 (socket says 172.21.15.72:49158) 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f6253fff640 1 -- 172.21.15.72:0/2130258078 learned_addr learned my addr 172.21.15.72:0/2130258078 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f625887b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 0x7f625415c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33656/0 (socket says 172.21.15.72:33656) 2024-03-06T07:21:55.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f625887b640 1 -- 172.21.15.72:0/2130258078 >> v2:172.21.15.146:3300/0 conn(0x7f62540ae7d0 msgr2=0x7f62540aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc882ffd640 1 -- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc87c0ead00 msgr2=0x7fc87c0e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc882ffd640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc87c0ead00 0x7fc87c0e2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7fc882ffd640 1 -- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c005e90 msgr2=0x7fc87c180970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fc882ffd640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c005e90 0x7fc87c180970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fc882ffd640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c04d640 con 0x7fc87c006810 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fc8837fe640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc87c005e90 0x7fc87c180970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:55.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fc882ffd640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c006810 0x7fc87c04cbe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc884000b60 tx=0x7fc8840683c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fd845a64640 1 -- 172.21.15.72:0/617664262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c0037a0 msgr2=0x7fd8380d2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fd845a64640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd82c0037a0 0x7fd8380d2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fd845a64640 1 -- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8380436b0 msgr2=0x7fd838049ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fd845a64640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8380436b0 0x7fd838049ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7fd845a64640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8380d3200 con 0x7fd8380451a0 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f664dbe5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6640038e60 0x7f664003bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f664dbe5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f664004a3e0 0x7f66400e4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f664dbe5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66400e7690 0x7f66400396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f664dbe5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66400531a0 con 0x7f6640038e60 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f664dbe5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6640055bf0 con 0x7f664004a3e0 2024-03-06T07:21:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f664dbe5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66400517e0 con 0x7f66400e7690 2024-03-06T07:21:55.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0cd401a070 con 0x7f0ce406b9e0 2024-03-06T07:21:55.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f9085763640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006f290 0x7f908012cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6640038e60 0x7f664003bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7fd845a64640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8380451a0 0x7fd8380499b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00c110 tx=0x7fd838006a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6640038e60 0x7f664003bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54404/0 (socket says 172.21.15.72:54404) 2024-03-06T07:21:55.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50092610 con 0x7f2c48038110 2024-03-06T07:21:55.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c500927f0 con 0x7f2c48038110 2024-03-06T07:21:55.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 -- 172.21.15.72:0/368175909 learned_addr learned my addr 172.21.15.72:0/368175909 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f2c57be2640 1 -- 172.21.15.72:0/1120810565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c480cc1d0 con 0x7f2c48038110 2024-03-06T07:21:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664d3e4640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66400e7690 0x7f66400396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f625887b640 1 --2- 172.21.15.72:0/2130258078 >> v2:172.21.15.146:3300/0 conn(0x7f62540ae7d0 0x7f62540aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f625887b640 1 -- 172.21.15.72:0/2130258078 >> v1:172.21.15.72:6789/0 conn(0x7f62540cff00 legacy=0x7f625406d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f625887b640 1 -- 172.21.15.72:0/2130258078 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625406c900 con 0x7f6254156250 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f9084f62640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90800afae0 0x7f90800b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f9084f62640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90800afae0 0x7f90800b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33684/0 (socket says 172.21.15.72:33684) 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.722+0000 7f9084f62640 1 -- 172.21.15.72:0/1999966883 learned_addr learned my addr 172.21.15.72:0/1999966883 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.723+0000 7f9085f64640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9080161ed0 0x7f9080148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f9085763640 1 -- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90800afae0 msgr2=0x7f90800b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f9085763640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90800afae0 0x7f90800b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f9085763640 1 -- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9080161ed0 msgr2=0x7f9080148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f9085763640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9080161ed0 0x7f9080148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f9085763640 1 -- 172.21.15.72:0/1999966883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908012dba0 con 0x7f908006f290 2024-03-06T07:21:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c007480 con 0x7fd8380451a0 2024-03-06T07:21:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83c007620 con 0x7fd8380451a0 2024-03-06T07:21:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c02c8c0 con 0x7fd8380451a0 2024-03-06T07:21:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f0cdaffd640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0cc406b5a0 0x7f0cc406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7fd8484c9640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8380d34a0 con 0x7fd8380451a0 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7fd8484c9640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83804f430 con 0x7fd8380451a0 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 -- 172.21.15.72:0/368175909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66400e7690 msgr2=0x7f66400396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66400e7690 0x7f66400396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 -- 172.21.15.72:0/368175909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f664004a3e0 msgr2=0x7f66400e4c40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f664004a3e0 0x7f66400e4c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f664cbe3640 1 -- 172.21.15.72:0/368175909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664004ea00 con 0x7f6640038e60 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cd40894a0 con 0x7f0ce406b9e0 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f0cdaffd640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f0cc40737c0 0x7f0cc4075be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f0cdbfff640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0cc406b5a0 0x7f0cc406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cc4076290 con 0x7f0cc40737c0 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cd4090050 con 0x7f0ce406b9e0 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f0ce2d9d640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f0cc40737c0 0x7f0cc4075be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f0cdbfff640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0cc406b5a0 0x7f0cc406da60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc001110 tx=0x7f0ccc007950 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f2c467fc640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c3406b580 0x7f2c3406da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f2c5517d640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c3406b580 0x7f2c3406da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd828001030 con 0x7fd8380451a0 2024-03-06T07:21:55.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f664cbe3640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6640038e60 0x7f664003bbb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f663c009990 tx=0x7f663c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c0443f0 con 0x7f6640038e60 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c044590 con 0x7f6640038e60 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c044980 con 0x7f6640038e60 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664004ece0 con 0x7f6640038e60 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66400e40a0 con 0x7f6640038e60 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.724+0000 7f9085763640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006f290 0x7f908012cc90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f907c00b3e0 tx=0x7f907c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c015020 con 0x7f908006f290 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.725+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c011830 con 0x7f908006f290 2024-03-06T07:21:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7f0ce2d9d640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f0cc40737c0 0x7f0cc4075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884073660 con 0x7fc87c006810 2024-03-06T07:21:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc884094440 con 0x7fc87c006810 2024-03-06T07:21:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884094810 con 0x7fc87c006810 2024-03-06T07:21:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c011c40 con 0x7f908006f290 2024-03-06T07:21:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c04d920 con 0x7fc87c006810 2024-03-06T07:21:55.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c04dee0 con 0x7fc87c006810 2024-03-06T07:21:55.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7f9086765640 1 -- 172.21.15.72:0/1999966883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f908013eac0 con 0x7f908006f290 2024-03-06T07:21:55.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7f9086765640 1 -- 172.21.15.72:0/1999966883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f908012d5e0 con 0x7f908006f290 2024-03-06T07:21:55.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.726+0000 7f625887b640 1 --2- 172.21.15.72:0/2130258078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 0x7f625415c2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f62440097a0 tx=0x7f624402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa076fa3144a80b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.727+0000 7f62537fe640 1 -- 172.21.15.72:0/2130258078 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244035020 con 0x7f6254156250 2024-03-06T07:21:55.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.727+0000 7f62537fe640 1 -- 172.21.15.72:0/2130258078 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6244031830 con 0x7f6254156250 2024-03-06T07:21:55.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.727+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== osd.3 v2:172.21.15.72:6808/4245662075 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0cc4076290 con 0x7f0cc40737c0 2024-03-06T07:21:55.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.727+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc87c005390 con 0x7fc87c006810 2024-03-06T07:21:55.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9080068f80 con 0x7f908006f290 2024-03-06T07:21:55.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f625987d640 1 -- 172.21.15.72:0/2130258078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 msgr2=0x7f625415c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f625987d640 1 --2- 172.21.15.72:0/2130258078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 0x7f625415c2e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f62440097a0 tx=0x7f624402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c2c001030 con 0x7f2c48038110 2024-03-06T07:21:55.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f2c5517d640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c3406b580 0x7f2c3406da40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c480dea10 tx=0x7f2c4c006bd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c501041a0 con 0x7f2c48038110 2024-03-06T07:21:55.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f2c467fc640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f2c34073720 0x7f2c34075b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c50091070 con 0x7f2c34073720 2024-03-06T07:21:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.728+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f663c04b060 con 0x7f6640038e60 2024-03-06T07:21:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7f2c5617f640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f2c34073720 0x7f2c34075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7f66457fa640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f662c06b5a0 0x7f662c06da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f663c0ac3f0 con 0x7f6640038e60 2024-03-06T07:21:55.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c50109060 con 0x7f2c48038110 2024-03-06T07:21:55.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8840949b0 con 0x7fc87c006810 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7fc86ffff640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc86806b520 0x7fc86806d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8841051b0 con 0x7fc87c006810 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7fc8837fe640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc86806b520 0x7fc86806d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fc86ffff640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7fc868073740 0x7fc868075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc868076210 con 0x7fc868073740 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc88410c050 con 0x7fc87c006810 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f2c5617f640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f2c34073720 0x7f2c34075b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fc883fff640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7fc868073740 0x7fc868075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.729+0000 7f6647fff640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f662c06b5a0 0x7f662c06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f664dbe5640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f66400e5180 0x7f6640053b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fc8837fe640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc86806b520 0x7fc86806d9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc878007fd0 tx=0x7fc878009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66400e4790 con 0x7f66400e5180 2024-03-06T07:21:55.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f664d3e4640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f66400e5180 0x7f6640053b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f907c018020 con 0x7f908006f290 2024-03-06T07:21:55.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f625987d640 1 -- 172.21.15.72:0/2130258078 shutdown_connections 2024-03-06T07:21:55.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f625987d640 1 --2- 172.21.15.72:0/2130258078 >> v2:172.21.15.146:3300/0 conn(0x7f62540ae7d0 0x7f62540aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f625987d640 1 --2- 172.21.15.72:0/2130258078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6254156250 0x7f625415c2e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f625987d640 1 -- 172.21.15.72:0/2130258078 >> 172.21.15.72:0/2130258078 conn(0x7f625406afb0 msgr2=0x7f625406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f625987d640 1 -- 172.21.15.72:0/2130258078 shutdown_connections 2024-03-06T07:21:55.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fc883fff640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7fc868073740 0x7fc868075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7f625987d640 1 -- 172.21.15.72:0/2130258078 wait complete. 2024-03-06T07:21:55.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f6647fff640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f662c06b5a0 0x7f662c06da60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66400eaed0 tx=0x7f66340079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.730+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd83c02ca60 con 0x7fd8380451a0 2024-03-06T07:21:55.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fd836ffd640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd82406b4d0 0x7fd82406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== osd.6 v2:172.21.15.146:6808/1113968375 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2c50091070 con 0x7f2c34073720 2024-03-06T07:21:55.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f664d3e4640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f66400e5180 0x7f6640053b30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f9075ffb640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906406b520 0x7f906406d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f907c08a5d0 con 0x7f908006f290 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f9075ffb640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f9064073740 0x7f9064075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f907c025ec0 con 0x7f9064073740 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f907c08a980 con 0x7f908006f290 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7f9084f62640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906406b520 0x7f906406d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== osd.4 v2:172.21.15.146:6820/3084745005 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc868076210 con 0x7fc868073740 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd83c0ac2d0 con 0x7fd8380451a0 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fd846265640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd82406b4d0 0x7fd82406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fd836ffd640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7fd824073750 0x7fd824075b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd83c044ec0 con 0x7fd824073750 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.731+0000 7fd846265640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd82406b4d0 0x7fd82406d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd83000a950 tx=0x7fd830008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7fd846a66640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7fd824073750 0x7fd824075b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd83c0ac6c0 con 0x7fd8380451a0 2024-03-06T07:21:55.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== osd.2 v2:172.21.15.72:6817/3054266462 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f66400e4790 con 0x7f66400e5180 2024-03-06T07:21:55.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7f625987d640 1 Processor -- start 2024-03-06T07:21:55.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7f625987d640 1 -- start start 2024-03-06T07:21:55.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.733+0000 7fd846a66640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7fd824073750 0x7fd824075b70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7f9085f64640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f9064073740 0x7f9064075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.732+0000 7f9084f62640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906406b520 0x7f906406d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f907802bf10 tx=0x7f9078030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.733+0000 7f9085f64640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f9064073740 0x7f9064075b60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625987d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62540ae7d0 0x7f6254159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625987d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62540cff00 0x7f62541497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625987d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6254156250 0x7f625414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f6253fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62540cff00 0x7f62541497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd83c044ec0 con 0x7fd824073750 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f6253fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62540cff00 0x7f62541497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33706/0 (socket says 172.21.15.72:33706) 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625887b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62540ae7d0 0x7f6254159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625887b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62540ae7d0 0x7f6254159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54416/0 (socket says 172.21.15.72:54416) 2024-03-06T07:21:55.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== osd.5 v2:172.21.15.146:6800/2545401618 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f907c025ec0 con 0x7f9064073740 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f6253fff640 1 -- 172.21.15.72:0/3881120821 learned_addr learned my addr 172.21.15.72:0/3881120821 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625987d640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625406d990 con 0x7f62540ae7d0 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625987d640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625406d810 con 0x7f6254156250 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.734+0000 7f625987d640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6254156630 con 0x7f62540cff00 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.735+0000 7f625907c640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6254156250 0x7f625414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.735+0000 7f6253fff640 1 -- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6254156250 msgr2=0x7f625414dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.735+0000 7f6253fff640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6254156250 0x7f625414dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.735+0000 7f6253fff640 1 -- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62540ae7d0 msgr2=0x7f6254159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f6253fff640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62540ae7d0 0x7f6254159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f6253fff640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f625414e680 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f625887b640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62540ae7d0 0x7f6254159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f6253fff640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62540cff00 0x7f62541497e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f624c00cba0 tx=0x7f624c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c017070 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f624c00ad40 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.736+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624c012600 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.738+0000 7f625987d640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f625414e960 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.738+0000 7f625987d640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f625414ef20 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.740+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f624c01a070 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.741+0000 7f62517fa640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f624006b5a0 0x7f624006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.741+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f625415a430 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.741+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f624c089fc0 con 0x7f62540cff00 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.741+0000 7f62517fa640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f62400737a0 0x7f6240075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.741+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6240076270 con 0x7f62400737a0 2024-03-06T07:21:55.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.741+0000 7f625907c640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f62400737a0 0x7f6240075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.742+0000 7f625887b640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f624006b5a0 0x7f624006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.743+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f624c054a70 con 0x7f62540cff00 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.743+0000 7f625907c640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f62400737a0 0x7f6240075bc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.743+0000 7f625887b640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f624006b5a0 0x7f624006da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6244036000 tx=0x7f624403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.744+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== osd.7 v2:172.21.15.146:6816/1306818419 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6240076270 con 0x7f62400737a0 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.757+0000 7f0ce8cdf640 1 -- 172.21.15.72:0/762448269 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0ce40681b0 con 0x7f0cc40737c0 2024-03-06T07:21:55.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.758+0000 7f0cdaffd640 1 -- 172.21.15.72:0/762448269 <== osd.3 v2:172.21.15.72:6808/4245662075 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0ce40681b0 con 0x7f0cc40737c0 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f0cc40737c0 msgr2=0x7f0cc4075be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0cd8ff9640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f0cc40737c0 0x7f0cc4075be0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0cc406b5a0 msgr2=0x7f0cc406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0cd8ff9640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0cc406b5a0 0x7f0cc406da60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc001110 tx=0x7f0ccc007950 comp rx=0 tx=0).stop 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce406b9e0 msgr2=0x7f0ce4160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0cd8ff9640 1 --2- 172.21.15.72:0/762448269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ce406b9e0 0x7f0ce4160cb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0cd400c490 tx=0x7f0cd400c950 comp rx=0 tx=0).stop 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.760+0000 7f0ce2d9d640 1 -- 172.21.15.72:0/762448269 reap_dead start 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.762+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc87c039440 con 0x7fc868073740 2024-03-06T07:21:55.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.763+0000 7fc86ffff640 1 -- 172.21.15.72:0/3215294663 <== osd.4 v2:172.21.15.146:6820/3084745005 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc87c039440 con 0x7fc868073740 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.763+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7fc868073740 msgr2=0x7fc868075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.763+0000 7fc889bd7640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7fc868073740 0x7fc868075b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 shutdown_connections 2024-03-06T07:21:55.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 >> 172.21.15.72:0/762448269 conn(0x7f0ce40b91c0 msgr2=0x7f0ce40b3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f908014d4e0 con 0x7f9064073740 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc86806b520 msgr2=0x7fc86806d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 shutdown_connections 2024-03-06T07:21:55.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7f0cd8ff9640 1 -- 172.21.15.72:0/762448269 wait complete. 2024-03-06T07:21:55.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc889bd7640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc86806b520 0x7fc86806d9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc878007fd0 tx=0x7fc878009040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7f9075ffb640 1 -- 172.21.15.72:0/1999966883 <== osd.5 v2:172.21.15.146:6800/2545401618 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f908014d4e0 con 0x7f9064073740 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c006810 msgr2=0x7fc87c04cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc889bd7640 1 --2- 172.21.15.72:0/3215294663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc87c006810 0x7fc87c04cbe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc884000b60 tx=0x7fc8840683c0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc883fff640 1 -- 172.21.15.72:0/3215294663 reap_dead start 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 shutdown_connections 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.764+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 >> 172.21.15.72:0/3215294663 conn(0x7fc87c009fb0 msgr2=0x7fc87c0e9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.765+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 shutdown_connections 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.765+0000 7fc889bd7640 1 -- 172.21.15.72:0/3215294663 wait complete. 2024-03-06T07:21:55.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.765+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66400057e0 con 0x7f66400e5180 2024-03-06T07:21:55.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.766+0000 7f66457fa640 1 -- 172.21.15.72:0/368175909 <== osd.2 v2:172.21.15.72:6817/3054266462 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f66400057e0 con 0x7f66400e5180 2024-03-06T07:21:55.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.766+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f9064073740 msgr2=0x7f9064075b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.766+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd828002dd0 con 0x7fd824073750 2024-03-06T07:21:55.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f90637fe640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f9064073740 0x7f9064075b60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f66400e5180 msgr2=0x7f6640053b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f664dbe5640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f66400e5180 0x7f6640053b30 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7fd836ffd640 1 -- 172.21.15.72:0/617664262 <== osd.0 v2:172.21.15.72:6816/4178159476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd828002dd0 con 0x7fd824073750 2024-03-06T07:21:55.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906406b520 msgr2=0x7f906406d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f90637fe640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906406b520 0x7f906406d9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f907802bf10 tx=0x7f9078030000 comp rx=0 tx=0).stop 2024-03-06T07:21:55.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006f290 msgr2=0x7f908012cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.767+0000 7f90637fe640 1 --2- 172.21.15.72:0/1999966883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006f290 0x7f908012cc90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f907c00b3e0 tx=0x7f907c00b8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f9085f64640 1 -- 172.21.15.72:0/1999966883 reap_dead start 2024-03-06T07:21:55.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7fd824073750 msgr2=0x7fd824075b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7fd824073750 0x7fd824075b70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f662c06b5a0 msgr2=0x7f662c06da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f664dbe5640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f662c06b5a0 0x7f662c06da60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f66400eaed0 tx=0x7f66340079b0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6640038e60 msgr2=0x7f664003bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f664dbe5640 1 --2- 172.21.15.72:0/368175909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6640038e60 0x7f664003bbb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f663c009990 tx=0x7f663c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f664d3e4640 1 -- 172.21.15.72:0/368175909 reap_dead start 2024-03-06T07:21:55.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd82406b4d0 msgr2=0x7fd82406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd82406b4d0 0x7fd82406d990 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd83000a950 tx=0x7fd830008040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8380451a0 msgr2=0x7fd8380499b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 --2- 172.21.15.72:0/617664262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8380451a0 0x7fd8380499b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00c110 tx=0x7fd838006a00 comp rx=0 tx=0).stop 2024-03-06T07:21:55.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 shutdown_connections 2024-03-06T07:21:55.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd846a66640 1 -- 172.21.15.72:0/617664262 reap_dead start 2024-03-06T07:21:55.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 >> 172.21.15.72:0/1999966883 conn(0x7f908006c0c0 msgr2=0x7f9080163c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 shutdown_connections 2024-03-06T07:21:55.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 >> 172.21.15.72:0/617664262 conn(0x7fd838009fe0 msgr2=0x7fd83804f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 shutdown_connections 2024-03-06T07:21:55.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7f90637fe640 1 -- 172.21.15.72:0/1999966883 wait complete. 2024-03-06T07:21:55.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 shutdown_connections 2024-03-06T07:21:55.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.768+0000 7fd834ff9640 1 -- 172.21.15.72:0/617664262 wait complete. 2024-03-06T07:21:55.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.769+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 shutdown_connections 2024-03-06T07:21:55.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.769+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 >> 172.21.15.72:0/368175909 conn(0x7f6640053750 msgr2=0x7f664004ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.769+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6254156af0 con 0x7f62400737a0 2024-03-06T07:21:55.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.769+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 shutdown_connections 2024-03-06T07:21:55.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.769+0000 7f664dbe5640 1 -- 172.21.15.72:0/368175909 wait complete. 2024-03-06T07:21:55.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.769+0000 7f62517fa640 1 -- 172.21.15.72:0/3881120821 <== osd.7 v2:172.21.15.146:6816/1306818419 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6254156af0 con 0x7f62400737a0 2024-03-06T07:21:55.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.770+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f62400737a0 msgr2=0x7f6240075bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.770+0000 7f623affd640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f62400737a0 0x7f6240075bc0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.770+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f624006b5a0 msgr2=0x7f624006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.770+0000 7f623affd640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f624006b5a0 0x7f624006da60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6244036000 tx=0x7f624403c040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.771+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62540cff00 msgr2=0x7f62541497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.771+0000 7f623affd640 1 --2- 172.21.15.72:0/3881120821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62540cff00 0x7f62541497e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f624c00cba0 tx=0x7f624c00a430 comp rx=0 tx=0).stop 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.771+0000 7f625907c640 1 -- 172.21.15.72:0/3881120821 reap_dead start 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.771+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 shutdown_connections 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.771+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 >> 172.21.15.72:0/3881120821 conn(0x7f625406afb0 msgr2=0x7f625415c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.771+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 shutdown_connections 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.772+0000 7f623affd640 1 -- 172.21.15.72:0/3881120821 wait complete. 2024-03-06T07:21:55.867 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:55.868 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-06T07:21:55.869 INFO:teuthology.orchestra.run.smithi072.stdout:55834574855 2024-03-06T07:21:55.869 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-06T07:21:55.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.787+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c2c002dd0 con 0x7f2c34073720 2024-03-06T07:21:55.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.788+0000 7f2c467fc640 1 -- 172.21.15.72:0/1120810565 <== osd.6 v2:172.21.15.146:6808/1113968375 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2c2c002dd0 con 0x7f2c34073720 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.788+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f2c34073720 msgr2=0x7f2c34075b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.788+0000 7f2c33fff640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f2c34073720 0x7f2c34075b40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c3406b580 msgr2=0x7f2c3406da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c3406b580 0x7f2c3406da40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2c480dea10 tx=0x7f2c4c006bd0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 msgr2=0x7f2c480cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 --2- 172.21.15.72:0/1120810565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c48038110 0x7f2c480cd790 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2c500500e0 tx=0x7f2c500683c0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c5617f640 1 -- 172.21.15.72:0/1120810565 reap_dead start 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 shutdown_connections 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 >> 172.21.15.72:0/1120810565 conn(0x7f2c48009fb0 msgr2=0x7f2c48039380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 shutdown_connections 2024-03-06T07:21:55.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.789+0000 7f2c33fff640 1 -- 172.21.15.72:0/1120810565 wait complete. 2024-03-06T07:21:55.872 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:55.872 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-06T07:21:55.873 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:55.873 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-06T07:21:55.874 INFO:teuthology.orchestra.run.smithi072.stdout:55834574853 2024-03-06T07:21:55.874 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-06T07:21:55.875 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:55.875 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-06T07:21:55.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.796+0000 7f50c63a0640 1 Processor -- start 2024-03-06T07:21:55.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.797+0000 7f50c63a0640 1 -- start start 2024-03-06T07:21:55.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.797+0000 7f50c63a0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f50c00b23a0 0x7f50c006b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.797+0000 7f50c63a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b5c30 0x7f50c00acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.797+0000 7f50c63a0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c00b2bf0 con 0x7f50c015dc30 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.797+0000 7f50c63a0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c006c950 con 0x7f50c00b23a0 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.797+0000 7f50c63a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c006b8f0 con 0x7f50c00b5c30 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c4b9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f50c00b23a0 0x7f50c006b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c539e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f50c015dc30 0x7f50c00b1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49168/0 (socket says 172.21.15.72:49168) 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c539e640 1 -- 172.21.15.72:0/701001285 learned_addr learned my addr 172.21.15.72:0/701001285 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c5b9f640 1 --2- 172.21.15.72:0/701001285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b5c30 0x7f50c00acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50b7fff640 1 -- 172.21.15.72:0/701001285 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303353343 0 0) 0x7f50bc001110 con 0x7f50c015dc30 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50b7fff640 1 -- 172.21.15.72:0/701001285 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ac003520 con 0x7f50c015dc30 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c4b9d640 1 -- 172.21.15.72:0/701001285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b5c30 msgr2=0x7f50c00acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c4b9d640 1 --2- 172.21.15.72:0/701001285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b5c30 0x7f50c00acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c4b9d640 1 -- 172.21.15.72:0/701001285 >> v1:172.21.15.72:6789/0 conn(0x7f50c015dc30 legacy=0x7f50c00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.798+0000 7f50c4b9d640 1 -- 172.21.15.72:0/701001285 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c006bb90 con 0x7f50c00b23a0 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.799+0000 7f50c4b9d640 1 --2- 172.21.15.72:0/701001285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b23a0 0x7f50c006b250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f50b00097a0 tx=0x7f50b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af8bc33d435c2d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.799+0000 7f50b7fff640 1 -- 172.21.15.72:0/701001285 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00315a0 con 0x7f50c00b23a0 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.799+0000 7f50b7fff640 1 -- 172.21.15.72:0/701001285 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0037070 con 0x7f50c00b23a0 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.799+0000 7f50b7fff640 1 -- 172.21.15.72:0/701001285 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0031940 con 0x7f50c00b23a0 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 -- 172.21.15.72:0/701001285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b23a0 msgr2=0x7f50c006b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 --2- 172.21.15.72:0/701001285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b23a0 0x7f50c006b250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f50b00097a0 tx=0x7f50b002b840 comp rx=0 tx=0).stop 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 -- 172.21.15.72:0/701001285 shutdown_connections 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 --2- 172.21.15.72:0/701001285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b5c30 0x7f50c00acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 --2- 172.21.15.72:0/701001285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b23a0 0x7f50c006b250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 -- 172.21.15.72:0/701001285 >> 172.21.15.72:0/701001285 conn(0x7f50c00b4e60 msgr2=0x7f50c00b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 -- 172.21.15.72:0/701001285 shutdown_connections 2024-03-06T07:21:55.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.800+0000 7f50c63a0640 1 -- 172.21.15.72:0/701001285 wait complete. 2024-03-06T07:21:55.880 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:55.880 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.801+0000 7f50c63a0640 1 Processor -- start 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.801+0000 7f50c63a0640 1 -- start start 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b23a0 0x7f50c00cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b5c30 0x7f50c0149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c015dc30 0x7f50c0148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c006c430 con 0x7f50c00b5c30 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c006c150 con 0x7f50c015dc30 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c015e010 con 0x7f50c00b23a0 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b5c30 0x7f50c0149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b5c30 0x7f50c0149790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54428/0 (socket says 172.21.15.72:54428) 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 -- 172.21.15.72:0/72006647 learned_addr learned my addr 172.21.15.72:0/72006647 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:55.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c539e640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b23a0 0x7f50c00cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c5b9f640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c015dc30 0x7f50c0148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 -- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b23a0 msgr2=0x7f50c00cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c00b23a0 0x7f50c00cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 -- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c015dc30 msgr2=0x7f50c0148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c015dc30 0x7f50c0148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 -- 172.21.15.72:0/72006647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0156080 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c5b9f640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c015dc30 0x7f50c0148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c4b9d640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b5c30 0x7f50c0149790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f50b0034040 tx=0x7f50b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00317a0 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0031940 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c00c97c0 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.803+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0031cf0 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.804+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c00c9d80 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.804+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50c00ca160 con 0x7f50c00b5c30 2024-03-06T07:21:55.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.805+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50b0032430 con 0x7f50c00b5c30 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.805+0000 7f50b5ffb640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f50a406b4d0 0x7f50a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50b002f030 con 0x7f50c00b5c30 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50c539e640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f50a406b4d0 0x7f50a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50b5ffb640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f50a4073670 0x7f50a4075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50b000af30 con 0x7f50a4073670 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50b00acd10 con 0x7f50c00b5c30 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50c5b9f640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f50a4073670 0x7f50a4075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50c539e640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f50a406b4d0 0x7f50a406d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50bc005880 tx=0x7f50bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.806+0000 7f50c5b9f640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f50a4073670 0x7f50a4075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.807+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== osd.1 v2:172.21.15.72:6800/2878132174 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f50b000af30 con 0x7f50a4073670 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.821+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50c0068f80 con 0x7f50a4073670 2024-03-06T07:21:55.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.821+0000 7f50b5ffb640 1 -- 172.21.15.72:0/72006647 <== osd.1 v2:172.21.15.72:6800/2878132174 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f50c0068f80 con 0x7f50a4073670 2024-03-06T07:21:55.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f50a4073670 msgr2=0x7f50a4075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f50a4073670 0x7f50a4075a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f50a406b4d0 msgr2=0x7f50a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f50a406b4d0 0x7f50a406d990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50bc005880 tx=0x7f50bc009040 comp rx=0 tx=0).stop 2024-03-06T07:21:55.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b5c30 msgr2=0x7f50c0149790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:55.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 --2- 172.21.15.72:0/72006647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50c00b5c30 0x7f50c0149790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f50b0034040 tx=0x7f50b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:55.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c5b9f640 1 -- 172.21.15.72:0/72006647 reap_dead start 2024-03-06T07:21:55.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 shutdown_connections 2024-03-06T07:21:55.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 >> 172.21.15.72:0/72006647 conn(0x7f50c00b4e60 msgr2=0x7f50c015cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:55.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 shutdown_connections 2024-03-06T07:21:55.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:55.822+0000 7f50c63a0640 1 -- 172.21.15.72:0/72006647 wait complete. 2024-03-06T07:21:55.889 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:55.889 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-06T07:21:56.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.076+0000 7f3252c50640 1 Processor -- start 2024-03-06T07:21:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.076+0000 7f3252c50640 1 -- start start 2024-03-06T07:21:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.081+0000 7fd6e3fff640 1 Processor -- start 2024-03-06T07:21:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7f3252c50640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c0af7d0 0x7f324c0d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7f3252c50640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0d0a50 0x7f324c163f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7f3252c50640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f324c06b730 con 0x7f324c0d0130 2024-03-06T07:21:56.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7f3252c50640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f324c15dd50 con 0x7f324c0af7d0 2024-03-06T07:21:56.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7f3252c50640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324c06c340 con 0x7f324c0d0a50 2024-03-06T07:21:56.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.083+0000 7f324bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c0af7d0 0x7f324c0d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.083+0000 7f324bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f324c0af7d0 0x7f324c0d0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42332/0 (socket says 172.21.15.72:42332) 2024-03-06T07:21:56.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.083+0000 7f324bfff640 1 -- 172.21.15.72:0/1017023155 learned_addr learned my addr 172.21.15.72:0/1017023155 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.081+0000 7fd6e3fff640 1 -- start start 2024-03-06T07:21:56.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7fd6e3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd6dc005e80 0x7fd6dc0062c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7fd6e3fff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc053710 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7fd6e3fff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc0539f0 con 0x7fd6dc006800 2024-03-06T07:21:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.082+0000 7fd6e3fff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc054320 con 0x7fd6dc005e80 2024-03-06T07:21:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.083+0000 7fd6e27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd6dc005e80 0x7fd6dc0062c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.083+0000 7fd6e27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd6dc005e80 0x7fd6dc0062c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54440/0 (socket says 172.21.15.72:54440) 2024-03-06T07:21:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.083+0000 7fd6e27fc640 1 -- 172.21.15.72:0/4163331055 learned_addr learned my addr 172.21.15.72:0/4163331055 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.084+0000 7fd6e27fc640 1 -- 172.21.15.72:0/4163331055 >> v1:172.21.15.146:6789/0 conn(0x7fd6dc006800 legacy=0x7fd6dc006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.085+0000 7fccd9e92640 1 Processor -- start 2024-03-06T07:21:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.085+0000 7fccd9e92640 1 -- start start 2024-03-06T07:21:56.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd9e92640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 0x7fcccc052ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd9e92640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcccc03aa10 0x7fcccc03ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd9e92640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc009560 con 0x7fcccc048c30 2024-03-06T07:21:56.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd9e92640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc035960 con 0x7fcccc03aa10 2024-03-06T07:21:56.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd9e92640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc035c20 con 0x7fcccc009180 2024-03-06T07:21:56.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7f1ac9b93640 1 Processor -- start 2024-03-06T07:21:56.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.084+0000 7fd6e27fc640 1 -- 172.21.15.72:0/4163331055 >> v1:172.21.15.72:6789/0 conn(0x7fd6dc0eacf0 legacy=0x7fd6dc04fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.084+0000 7fd6e27fc640 1 -- 172.21.15.72:0/4163331055 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc053cd0 con 0x7fd6dc005e80 2024-03-06T07:21:56.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd8e90640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 0x7fcccc052ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd8e90640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 0x7fcccc052ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33738/0 (socket says 172.21.15.72:33738) 2024-03-06T07:21:56.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd8e90640 1 -- 172.21.15.72:0/3358618813 learned_addr learned my addr 172.21.15.72:0/3358618813 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7f1ac9b93640 1 -- start start 2024-03-06T07:21:56.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd8e90640 1 -- 172.21.15.72:0/3358618813 >> v2:172.21.15.146:3300/0 conn(0x7fcccc03aa10 msgr2=0x7fcccc03ae20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:56.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.087+0000 7fccd8e90640 1 --2- 172.21.15.72:0/3358618813 >> v2:172.21.15.146:3300/0 conn(0x7fcccc03aa10 0x7fcccc03ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7fccd8e90640 1 -- 172.21.15.72:0/3358618813 >> v1:172.21.15.146:6789/0 conn(0x7fcccc048c30 legacy=0x7fcccc049050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7fccd8e90640 1 -- 172.21.15.72:0/3358618813 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc035f00 con 0x7fcccc009180 2024-03-06T07:21:56.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f1ac9b93640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1ac415e000 0x7f1ac40d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f1ac9b93640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40d25e0 con 0x7f1ac40d4810 2024-03-06T07:21:56.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f1ac9b93640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40d2880 con 0x7f1ac40cf7d0 2024-03-06T07:21:56.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7fd6e27fc640 1 --2- 172.21.15.72:0/4163331055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc005e80 0x7fd6dc0062c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429cde880739363d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f1ac9b93640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40c7c50 con 0x7f1ac415e000 2024-03-06T07:21:56.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f1ac3fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1ac40cf7d0 0x7f1ac40c87f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:38958/0 (socket says 172.21.15.72:38958) 2024-03-06T07:21:56.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f1ac3fff640 1 -- 172.21.15.72:0/1124263448 learned_addr learned my addr 172.21.15.72:0/1124263448 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:56.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7fd6e1ffb640 1 -- 172.21.15.72:0/4163331055 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4034070 con 0x7fd6dc005e80 2024-03-06T07:21:56.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f32511ed640 1 --2- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0d0a50 0x7f324c163f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7fd6e1ffb640 1 -- 172.21.15.72:0/4163331055 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4030860 con 0x7fd6dc005e80 2024-03-06T07:21:56.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7fd6e1ffb640 1 -- 172.21.15.72:0/4163331055 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4030c30 con 0x7fd6dc005e80 2024-03-06T07:21:56.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7fccd8e90640 1 --2- 172.21.15.72:0/3358618813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 0x7fcccc052ef0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fccd407a850 tx=0x7fccd40a69b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a37a8bc132a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f324bfff640 1 -- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0d0a50 msgr2=0x7f324c163f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f1ac37fe640 1 --2- 172.21.15.72:0/1124263448 >> v2:172.21.15.146:3300/0 conn(0x7f1ac415e000 0x7f1ac40d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f324bfff640 1 --2- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0d0a50 0x7f324c163f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f9aa8b65640 1 Processor -- start 2024-03-06T07:21:56.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.088+0000 7f9aa8b65640 1 -- start start 2024-03-06T07:21:56.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f324bfff640 1 -- 172.21.15.72:0/1017023155 >> v1:172.21.15.146:6789/0 conn(0x7f324c0d0130 legacy=0x7f324c0cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f324bfff640 1 -- 172.21.15.72:0/1017023155 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c06ba10 con 0x7f324c0af7d0 2024-03-06T07:21:56.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f9aa8b65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa414bb10 0x7f9aa4165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f9aa8b65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9aa406afb0 0x7f9aa406b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f9aa8b65640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa406e3c0 con 0x7f9aa40ce8d0 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f9aa8b65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa406b960 con 0x7f9aa406afb0 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f9aa8b65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa41666d0 con 0x7f9aa414bb10 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa1d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9aa40ce8d0 0x7f9aa40cbae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:38968/0 (socket says 172.21.15.72:38968) 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa1d9b640 1 -- 172.21.15.72:0/1512146901 learned_addr learned my addr 172.21.15.72:0/1512146901 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1512146901 >> v2:172.21.15.72:3300/0 conn(0x7f9aa406afb0 0x7f9aa406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512532936 0 0) 0x7f9aa406e3c0 con 0x7f9aa40ce8d0 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a8c003520 con 0x7f9aa40ce8d0 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa259c640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa414bb10 0x7f9aa4165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f1ac37fe640 1 -- 172.21.15.72:0/1124263448 >> v1:172.21.15.146:6789/0 conn(0x7f1ac40cf7d0 legacy=0x7f1ac40c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f1ac37fe640 1 -- 172.21.15.72:0/1124263448 >> v1:172.21.15.72:6789/0 conn(0x7f1ac40d4810 legacy=0x7f1ac40c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.089+0000 7f1ac37fe640 1 -- 172.21.15.72:0/1124263448 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40d2b40 con 0x7f1ac415e000 2024-03-06T07:21:56.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7fd6e3fff640 1 -- 172.21.15.72:0/4163331055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc005e80 msgr2=0x7fd6dc0062c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/4163331055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc005e80 0x7fd6dc0062c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7fd6e3fff640 1 -- 172.21.15.72:0/4163331055 shutdown_connections 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 279054782 0 0) 0x7f9a8c003520 con 0x7f9aa40ce8d0 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa414bb10 msgr2=0x7f9aa4165fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa414bb10 0x7f9aa4165fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 >> v2:172.21.15.72:3300/0 conn(0x7f9aa406afb0 msgr2=0x7f9aa406b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 --2- 172.21.15.72:0/1512146901 >> v2:172.21.15.72:3300/0 conn(0x7f9aa406afb0 0x7f9aa406b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa41669b0 con 0x7f9aa40ce8d0 2024-03-06T07:21:56.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/4163331055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc005e80 0x7fd6dc0062c0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9a98001670 con 0x7f9aa40ce8d0 2024-03-06T07:21:56.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1512146901 >> v2:172.21.15.72:3300/0 conn(0x7f9aa406afb0 0x7f9aa406b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:56.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 >> v1:172.21.15.146:6789/0 conn(0x7f9aa40ce8d0 legacy=0x7f9aa40cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c0039b0 0x7f9a8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f324bfff640 1 --2- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0af7d0 0x7f324c0d0510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3240009870 tx=0x7f324002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a93238339b6547d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a8c004800 0x7f9a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0051c0 0x7f9a8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa406b960 con 0x7f9a8c0039b0 2024-03-06T07:21:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa41666d0 con 0x7f9a8c0051c0 2024-03-06T07:21:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c00c090 con 0x7f9a8c004800 2024-03-06T07:21:56.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7fd6e3fff640 1 -- 172.21.15.72:0/4163331055 >> 172.21.15.72:0/4163331055 conn(0x7fd6dc009fa0 msgr2=0x7fd6dc00a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.090+0000 7f1ac37fe640 1 --2- 172.21.15.72:0/1124263448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40d0680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80097a0 tx=0x7f1ab802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb8a10f65c9adee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7fd6e3fff640 1 -- 172.21.15.72:0/4163331055 shutdown_connections 2024-03-06T07:21:56.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f324b7fe640 1 -- 172.21.15.72:0/1017023155 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240034070 con 0x7f324c0af7d0 2024-03-06T07:21:56.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f324b7fe640 1 -- 172.21.15.72:0/1017023155 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3240030840 con 0x7f324c0af7d0 2024-03-06T07:21:56.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7fcccb7fe640 1 -- 172.21.15.72:0/3358618813 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40af070 con 0x7fcccc009180 2024-03-06T07:21:56.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7fcccb7fe640 1 -- 172.21.15.72:0/3358618813 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd40ab930 con 0x7fcccc009180 2024-03-06T07:21:56.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7fcccb7fe640 1 -- 172.21.15.72:0/3358618813 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd40abda0 con 0x7fcccc009180 2024-03-06T07:21:56.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f324b7fe640 1 -- 172.21.15.72:0/1017023155 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240034070 con 0x7f324c0af7d0 2024-03-06T07:21:56.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa1d9b640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a8c004800 0x7f9a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0051c0 0x7f9a8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f1ac27fc640 1 -- 172.21.15.72:0/1124263448 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8034070 con 0x7f1ac415e000 2024-03-06T07:21:56.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa259c640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c0039b0 0x7f9a8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f1ac27fc640 1 -- 172.21.15.72:0/1124263448 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab8030610 con 0x7f1ac415e000 2024-03-06T07:21:56.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f1ac27fc640 1 -- 172.21.15.72:0/1124263448 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80309e0 con 0x7f1ac415e000 2024-03-06T07:21:56.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa1d9b640 1 -- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0051c0 msgr2=0x7f9a8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa1d9b640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0051c0 0x7f9a8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa1d9b640 1 -- 172.21.15.72:0/1512146901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c0039b0 msgr2=0x7f9a8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa1d9b640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c0039b0 0x7f9a8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa1d9b640 1 -- 172.21.15.72:0/1512146901 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa41669b0 con 0x7f9a8c004800 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a8c0051c0 0x7f9a8c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 -- 172.21.15.72:0/3358618813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 msgr2=0x7fcccc052ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 --2- 172.21.15.72:0/3358618813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 0x7fcccc052ef0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fccd407a850 tx=0x7fccd40a69b0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 -- 172.21.15.72:0/3358618813 shutdown_connections 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 --2- 172.21.15.72:0/3358618813 >> v2:172.21.15.146:3300/0 conn(0x7fcccc03aa10 0x7fcccc03ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 --2- 172.21.15.72:0/3358618813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc009180 0x7fcccc052ef0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 -- 172.21.15.72:0/3358618813 >> 172.21.15.72:0/3358618813 conn(0x7fcccc01ef30 msgr2=0x7fcccc055c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 -- 172.21.15.72:0/3358618813 shutdown_connections 2024-03-06T07:21:56.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fccd9e92640 1 -- 172.21.15.72:0/3358618813 wait complete. 2024-03-06T07:21:56.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7f1ac9b93640 1 -- 172.21.15.72:0/1124263448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 msgr2=0x7f1ac40d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/1124263448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40d0680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80097a0 tx=0x7f1ab802b960 comp rx=0 tx=0).stop 2024-03-06T07:21:56.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7f1ac9b93640 1 -- 172.21.15.72:0/1124263448 shutdown_connections 2024-03-06T07:21:56.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/1124263448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40d0680 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.091+0000 7fd6e3fff640 1 -- 172.21.15.72:0/4163331055 wait complete. 2024-03-06T07:21:56.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7fd6e3fff640 1 Processor -- start 2024-03-06T07:21:56.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7f1ac9b93640 1 -- 172.21.15.72:0/1124263448 >> 172.21.15.72:0/1124263448 conn(0x7f1ac40d3cf0 msgr2=0x7f1ac40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.092+0000 7f1ac9b93640 1 -- 172.21.15.72:0/1124263448 shutdown_connections 2024-03-06T07:21:56.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7f9aa1d9b640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a8c004800 0x7f9a8c004c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c260 tx=0x7f9a8c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4312b53cf627e608 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7fd6e3fff640 1 -- start start 2024-03-06T07:21:56.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7f1ac9b93640 1 -- 172.21.15.72:0/1124263448 wait complete. 2024-03-06T07:21:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98007d70 con 0x7f9a8c004800 2024-03-06T07:21:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7f9aa159a640 1 -- 172.21.15.72:0/1512146901 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a98030070 con 0x7f9a8c004800 2024-03-06T07:21:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a8c004800 msgr2=0x7f9a8c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7f9aa8b65640 1 --2- 172.21.15.72:0/1512146901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a8c004800 0x7f9a8c004c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a9800c260 tx=0x7f9a8c005830 comp rx=0 tx=0).stop 2024-03-06T07:21:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc006800 0x7fd6dc0393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7f9aa2d9d640 1 -- 172.21.15.72:0/1512146901 reap_dead start 2024-03-06T07:21:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 0x7fd6dc0e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7f1ac9b93640 1 Processor -- start 2024-03-06T07:21:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e3fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6dc046f10 0x7fd6dc0d7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1512146901 shutdown_connections 2024-03-06T07:21:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1512146901 >> 172.21.15.72:0/1512146901 conn(0x7f9aa40b1830 msgr2=0x7f9aa40b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e3fff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0547b0 con 0x7fd6dc006800 2024-03-06T07:21:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e27fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 0x7fd6dc0e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1512146901 shutdown_connections 2024-03-06T07:21:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7f1ac9b93640 1 -- start start 2024-03-06T07:21:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7fccd9e92640 1 Processor -- start 2024-03-06T07:21:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.093+0000 7fccd9e92640 1 -- start start 2024-03-06T07:21:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9e92640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcccc009180 0x7fcccc007e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9e92640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc03aa10 0x7fcccc1893d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9e92640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcccc048c30 0x7fcccc189910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e27fc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 0x7fd6dc0e9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42352/0 (socket says 172.21.15.72:42352) 2024-03-06T07:21:56.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.094+0000 7fd6e27fc640 1 -- 172.21.15.72:0/2161099208 learned_addr learned my addr 172.21.15.72:0/2161099208 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fd6e2ffd640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc006800 0x7fd6dc0393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9e92640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0367a0 con 0x7fcccc048c30 2024-03-06T07:21:56.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fd6e37fe640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6dc046f10 0x7fd6dc0d7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc050190 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9e92640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc0364c0 con 0x7fcccc009180 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9e92640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc054090 con 0x7fcccc03aa10 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9691640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcccc048c30 0x7fcccc189910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9691640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcccc048c30 0x7fcccc189910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54476/0 (socket says 172.21.15.72:54476) 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd9691640 1 -- 172.21.15.72:0/4280117927 learned_addr learned my addr 172.21.15.72:0/4280117927 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fccd8e90640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcccc009180 0x7fcccc007e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc054930 con 0x7fd6dc046f10 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7fcccbfff640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc03aa10 0x7fcccc1893d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fcccbfff640 1 -- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcccc009180 msgr2=0x7fcccc007e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fcccbfff640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcccc009180 0x7fcccc007e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fd6e27fc640 1 -- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6dc046f10 msgr2=0x7fd6dc0d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fd6e27fc640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6dc046f10 0x7fd6dc0d7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fd6e27fc640 1 -- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc006800 msgr2=0x7fd6dc0393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fd6e27fc640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc006800 0x7fd6dc0393a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fd6e27fc640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc043e70 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fcccbfff640 1 -- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcccc048c30 msgr2=0x7fcccc189910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fcccbfff640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcccc048c30 0x7fcccc189910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fcccbfff640 1 -- 172.21.15.72:0/4280117927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc0e5250 con 0x7fcccc03aa10 2024-03-06T07:21:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.096+0000 7fcccbfff640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc03aa10 0x7fcccc1893d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c390 tx=0x7fccc400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.095+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1512146901 wait complete. 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f9aa8b65640 1 Processor -- start 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4017070 con 0x7fcccc03aa10 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc0e4c90 con 0x7fcccc03aa10 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc0329c0 con 0x7fcccc03aa10 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc40052d0 con 0x7fcccc03aa10 2024-03-06T07:21:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc4003830 con 0x7fcccc03aa10 2024-03-06T07:21:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f9aa8b65640 1 -- start start 2024-03-06T07:21:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f3252c50640 1 -- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0af7d0 msgr2=0x7f324c0d0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f3252c50640 1 --2- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0af7d0 0x7f324c0d0510 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3240009870 tx=0x7f324002b980 comp rx=0 tx=0).stop 2024-03-06T07:21:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa8b65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c004800 0x7f9aa413f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa8b65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa4142d70 0x7f9aa40cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa8b65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 0x7f9aa40cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa8b65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4071770 con 0x7f9a8c004800 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa8b65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4071ac0 con 0x7f9aa4142d70 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa8b65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa414bef0 con 0x7f9aa41431f0 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 0x7f9aa40cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa2d9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 0x7f9aa40cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33808/0 (socket says 172.21.15.72:33808) 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f9aa2d9d640 1 -- 172.21.15.72:0/1958076243 learned_addr learned my addr 172.21.15.72:0/1958076243 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa2d9d640 1 -- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa4142d70 msgr2=0x7f9aa40cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa4142d70 0x7f9aa40cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa2d9d640 1 -- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c004800 msgr2=0x7f9aa413f940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa259c640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c004800 0x7f9aa413f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c004800 0x7f9aa413f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa2d9d640 1 -- 172.21.15.72:0/1958076243 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa41f8310 con 0x7f9aa41431f0 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f9aa259c640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c004800 0x7f9aa413f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f3252c50640 1 -- 172.21.15.72:0/1017023155 shutdown_connections 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f3252c50640 1 --2- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0d0a50 0x7f324c163f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f3252c50640 1 --2- 172.21.15.72:0/1017023155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0af7d0 0x7f324c0d0510 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f3252c50640 1 -- 172.21.15.72:0/1017023155 >> 172.21.15.72:0/1017023155 conn(0x7f324c072520 msgr2=0x7f324c072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f3252c50640 1 -- 172.21.15.72:0/1017023155 shutdown_connections 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f3252c50640 1 -- 172.21.15.72:0/1017023155 wait complete. 2024-03-06T07:21:56.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fdd72556640 1 Processor -- start 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fdd72556640 1 -- start start 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7fdd72556640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c0ce9e0 0x7fdd6c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7fd6e27fc640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 0x7fd6dc0e9cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4048070 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc044150 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc044710 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4039d10 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4034040 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f1ac9b93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac40cf7d0 0x7f1ac40cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f1ac9b93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ac40d4810 0x7f1ac4158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f1ac9b93640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.097+0000 7f1ac9b93640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac415e3e0 con 0x7f1ac40cf7d0 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f1ac9b93640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40d22d0 con 0x7f1ac415e000 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.098+0000 7f1ac9b93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40d0bc0 con 0x7f1ac40d4810 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f1ac3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f1ac3fff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42382/0 (socket says 172.21.15.72:42382) 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f1ac3fff640 1 -- 172.21.15.72:0/745022948 learned_addr learned my addr 172.21.15.72:0/745022948 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.099+0000 7f1ac2ffd640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ac40d4810 0x7f1ac4158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f9aa2d9d640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 0x7f9aa40cfdc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406e740 tx=0x7f9a9c007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f1ac3fff640 1 -- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ac40d4810 msgr2=0x7f1ac4158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f3252c50640 1 Processor -- start 2024-03-06T07:21:56.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c009c80 con 0x7f9aa41431f0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd72556640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c06e7b0 con 0x7fdd6c0cedc0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd72556640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd6c071fd0 con 0x7fdd6c15dda0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd72556640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c0722b0 con 0x7fdd6c0ce9e0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd6bfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c0ce9e0 0x7fdd6c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd6b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdd6c0cedc0 0x7fdd6c06e0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49196/0 (socket says 172.21.15.72:49196) 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd6b7fe640 1 -- 172.21.15.72:0/1433709286 learned_addr learned my addr 172.21.15.72:0/1433709286 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1958076243 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa41437c0 con 0x7f9aa41431f0 2024-03-06T07:21:56.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f3252c50640 1 -- start start 2024-03-06T07:21:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f9aa8b65640 1 -- 172.21.15.72:0/1958076243 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa4140990 con 0x7f9aa41431f0 2024-03-06T07:21:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f1ac3fff640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ac40d4810 0x7f1ac4158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f1ac3fff640 1 -- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac40cf7d0 msgr2=0x7f1ac40cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f1ac3fff640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac40cf7d0 0x7f1ac40cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3041792920 0 0) 0x7fdd60001110 con 0x7fdd6c0cedc0 2024-03-06T07:21:56.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd54003520 con 0x7fdd6c0cedc0 2024-03-06T07:21:56.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f1ac3fff640 1 -- 172.21.15.72:0/745022948 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac41599b0 con 0x7f1ac415e000 2024-03-06T07:21:56.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccc401a070 con 0x7fcccc03aa10 2024-03-06T07:21:56.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7f1ac3fff640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40cd9e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00caa0 tx=0x7f1abc00cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1588066975 0 0) 0x7fdd54003520 con 0x7fdd6c0cedc0 2024-03-06T07:21:56.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.100+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4005290 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c0ce9e0 msgr2=0x7fdd6c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c0ce9e0 0x7fdd6c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 >> v1:172.21.15.146:6789/0 conn(0x7fdd6c15dda0 legacy=0x7fdd6c15e230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c072590 con 0x7fdd6c0cedc0 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fdd60001840 con 0x7fdd6c0cedc0 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 >> v1:172.21.15.72:6789/0 conn(0x7fdd6c0cedc0 legacy=0x7fdd6c06e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd540039e0 0x7fdd54003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd540043b0 0x7fdd54004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd54005120 0x7fdd5400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c0722b0 con 0x7fdd54005120 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5400c030 con 0x7fdd540043b0 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5400c230 con 0x7fdd540039e0 2024-03-06T07:21:56.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd70af3640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd54005120 0x7fdd5400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4005290 con 0x7fcccc03aa10 2024-03-06T07:21:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd6bfff640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd540039e0 0x7fdd54003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc017070 con 0x7f1ac415e000 2024-03-06T07:21:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd70af3640 1 -- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd540039e0 msgr2=0x7fdd54003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd70af3640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd540039e0 0x7fdd54003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd70af3640 1 -- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd540043b0 msgr2=0x7fdd54004880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.101+0000 7fdd70af3640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd540043b0 0x7fdd54004880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7fdd70af3640 1 -- 172.21.15.72:0/1433709286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c072590 con 0x7fdd54005120 2024-03-06T07:21:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c01a070 con 0x7f9aa41431f0 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f3252c50640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0af7d0 0x7f324c0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f3252c50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 0x7f324c1f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f3252c50640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0d0a50 0x7f324c15a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f3252c50640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c0d0ee0 con 0x7f324c0d0130 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f3252c50640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324c06bfd0 con 0x7f324c0d0a50 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f3252c50640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324c06c7d0 con 0x7f324c0af7d0 2024-03-06T07:21:56.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd6d404b070 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7fccc97fa640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fccc006b5a0 0x7fccc006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccc40895d0 con 0x7fcccc03aa10 2024-03-06T07:21:56.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7fccd8e90640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fccc006b5a0 0x7fccc006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1abc00ac00 con 0x7f1ac415e000 2024-03-06T07:21:56.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c015620 con 0x7f9aa41431f0 2024-03-06T07:21:56.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7fd6cbfff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd6d006b5d0 0x7fd6d006da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa40ce430 con 0x7f9aa41431f0 2024-03-06T07:21:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.102+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc017070 con 0x7f1ac415e000 2024-03-06T07:21:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a9c015840 con 0x7f9aa41431f0 2024-03-06T07:21:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7fccd8e90640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fccc006b5a0 0x7fccc006da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009730 tx=0x7fccd406d030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac409ddc0 con 0x7f1ac415e000 2024-03-06T07:21:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f32511ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0d0a50 0x7f324c15a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 0x7f324c1f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 0x7f324c1f75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54508/0 (socket says 172.21.15.72:54508) 2024-03-06T07:21:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 -- 172.21.15.72:0/911278173 learned_addr learned my addr 172.21.15.72:0/911278173 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f32509ec640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0af7d0 0x7f324c0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f9a92ffd640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b5d0 0x7f9a8406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7fdd70af3640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd54005120 0x7fdd5400b9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009800 tx=0x7fdd54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189076398641f3f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 -- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0af7d0 msgr2=0x7f324c0cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0af7d0 0x7f324c0cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 -- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0d0a50 msgr2=0x7f324c15a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0d0a50 0x7f324c15a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.103+0000 7f324bfff640 1 -- 172.21.15.72:0/911278173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f324c15bc20 con 0x7f324c0d0130 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40cc170 con 0x7f1ac415e000 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a9c08a850 con 0x7f9aa41431f0 2024-03-06T07:21:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f9aa259c640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b5d0 0x7f9a8406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c037070 con 0x7fdd54005120 2024-03-06T07:21:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f324bfff640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 0x7f324c1f75a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f324002b9c0 tx=0x7f3240004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c032600 con 0x7fdd54005120 2024-03-06T07:21:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7fdd6affd640 1 -- 172.21.15.72:0/1433709286 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c037070 con 0x7fdd54005120 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7fd6e2ffd640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd6d006b5d0 0x7fd6d006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6d40ac5b0 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f9aa259c640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b5d0 0x7f9a8406da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9a94006fd0 tx=0x7f9a94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240034070 con 0x7f324c0d0130 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324c15bf00 con 0x7f324c0d0130 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324c15a2b0 con 0x7f324c0d0130 2024-03-06T07:21:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3240048070 con 0x7f324c0d0130 2024-03-06T07:21:56.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.104+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3240030840 con 0x7f324c0d0130 2024-03-06T07:21:56.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd72556640 1 -- 172.21.15.72:0/1433709286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd54005120 msgr2=0x7fdd5400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd72556640 1 --2- 172.21.15.72:0/1433709286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd54005120 0x7fdd5400b9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009800 tx=0x7fdd54005830 comp rx=0 tx=0).stop 2024-03-06T07:21:56.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd70af3640 1 -- 172.21.15.72:0/1433709286 reap_dead start 2024-03-06T07:21:56.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fd6e2ffd640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd6d006b5d0 0x7fd6d006da90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd72556640 1 -- 172.21.15.72:0/1433709286 shutdown_connections 2024-03-06T07:21:56.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd72556640 1 -- 172.21.15.72:0/1433709286 >> 172.21.15.72:0/1433709286 conn(0x7fdd6c0b1c80 msgr2=0x7fdd6c0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd72556640 1 -- 172.21.15.72:0/1433709286 shutdown_connections 2024-03-06T07:21:56.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7fdd72556640 1 -- 172.21.15.72:0/1433709286 wait complete. 2024-03-06T07:21:56.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.105+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c005290 con 0x7f324c0d0130 2024-03-06T07:21:56.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.107+0000 7fdd72556640 1 Processor -- start 2024-03-06T07:21:56.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.107+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3240042070 con 0x7f324c0d0130 2024-03-06T07:21:56.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7f32497fa640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f323806b670 0x7f323806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.107+0000 7fdd72556640 1 -- start start 2024-03-06T07:21:56.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3240079f40 con 0x7f324c0d0130 2024-03-06T07:21:56.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7f32509ec640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f323806b670 0x7f323806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7fdd72556640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c13e2d0 0x7fdd6c14c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7fdd72556640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd6c13ec90 0x7fdd6c0b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7fdd72556640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd6c0b3bd0 0x7fdd6c1fbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7fdd72556640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c140790 con 0x7fdd6c13ec90 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7fdd72556640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c072b50 con 0x7fdd6c0b3bd0 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.108+0000 7fdd72556640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c06dd60 con 0x7fdd6c13e2d0 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6bfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c13e2d0 0x7fdd6c14c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd6c13ec90 0x7fdd6c0b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6bfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c13e2d0 0x7fdd6c14c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33858/0 (socket says 172.21.15.72:33858) 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6bfff640 1 -- 172.21.15.72:0/2537451963 learned_addr learned my addr 172.21.15.72:0/2537451963 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd70af3640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd6c0b3bd0 0x7fdd6c1fbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 -- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c13e2d0 msgr2=0x7fdd6c14c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c13e2d0 0x7fdd6c14c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 -- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd6c0b3bd0 msgr2=0x7fdd6c1fbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd6c0b3bd0 0x7fdd6c1fbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 -- 172.21.15.72:0/2537451963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c14cd10 con 0x7fdd6c13ec90 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7f32509ec640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f323806b670 0x7f323806db30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3244007fd0 tx=0x7f3244009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.109+0000 7fdd6b7fe640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd6c13ec90 0x7fdd6c0b3690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000caf0 tx=0x7fdd600018d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd6d407aee0 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60018070 con 0x7fdd6c13ec90 2024-03-06T07:21:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd60004cb0 con 0x7fdd6c13ec90 2024-03-06T07:21:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60004e60 con 0x7fdd6c13ec90 2024-03-06T07:21:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f324007c5a0 con 0x7f324c0d0130 2024-03-06T07:21:56.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7fdd72556640 1 -- 172.21.15.72:0/2537451963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c142ca0 con 0x7fdd6c13ec90 2024-03-06T07:21:56.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.111+0000 7fdd72556640 1 -- 172.21.15.72:0/2537451963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c143260 con 0x7fdd6c13ec90 2024-03-06T07:21:56.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.113+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fccc4058430 con 0x7fcccc03aa10 2024-03-06T07:21:56.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.113+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac404f330 con 0x7f1ac415e000 2024-03-06T07:21:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.113+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd44005290 con 0x7fdd6c13ec90 2024-03-06T07:21:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.115+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1abc012400 con 0x7f1ac415e000 2024-03-06T07:21:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.115+0000 7f1aabfff640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ab006b8d0 0x7f1ab006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.115+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1abc00aa00 con 0x7f1ac415e000 2024-03-06T07:21:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.116+0000 7f1ac37fe640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ab006b8d0 0x7f1ab006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.116+0000 7f1ac37fe640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ab006b8d0 0x7f1ab006dd90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8036000 tx=0x7f1ab8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.118+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9aa40ce430 con 0x7f9aa41431f0 2024-03-06T07:21:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.117+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd6001b070 con 0x7fdd6c13ec90 2024-03-06T07:21:56.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.117+0000 7fdd68ff9640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd5806b5d0 0x7fdd5806da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.117+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd60089840 con 0x7fdd6c13ec90 2024-03-06T07:21:56.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.118+0000 7fdd6bfff640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd5806b5d0 0x7fdd5806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.119+0000 7fdd6bfff640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd5806b5d0 0x7fdd5806da90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdd640138a0 tx=0x7fdd6400b8a0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.120+0000 7f80463b6640 1 Processor -- start 2024-03-06T07:21:56.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.120+0000 7f80463b6640 1 -- start start 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f80463b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f804014bb10 0x7f8040165fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f80463b6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80400ce8d0 0x7f80400cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f80463b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804006afb0 0x7f804006b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f80463b6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f804006b940 con 0x7f804014bb10 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f80463b6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80400cc020 con 0x7f80400ce8d0 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f80463b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804006e3c0 con 0x7f804006afb0 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f803f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80400ce8d0 0x7f80400cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80400ce8d0 0x7f80400cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42408/0 (socket says 172.21.15.72:42408) 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 -- 172.21.15.72:0/833241882 learned_addr learned my addr 172.21.15.72:0/833241882 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1abc058750 con 0x7f1ac415e000 2024-03-06T07:21:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 -- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804006afb0 msgr2=0x7f804006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.121+0000 7f8044953640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804006afb0 0x7f804006b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804006afb0 0x7f804006b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 -- 172.21.15.72:0/833241882 >> v2:172.21.15.72:3300/0 conn(0x7f804014bb10 msgr2=0x7f8040165fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 --2- 172.21.15.72:0/833241882 >> v2:172.21.15.72:3300/0 conn(0x7f804014bb10 0x7f8040165fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f803f7fe640 1 -- 172.21.15.72:0/833241882 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80401666d0 con 0x7f80400ce8d0 2024-03-06T07:21:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.122+0000 7f8044953640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804006afb0 0x7f804006b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.123+0000 7f803f7fe640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80400ce8d0 0x7f80400cbae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80400b4fa0 tx=0x7f803002b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffea4c7adee0e532 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.124+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd6c06e7b0 con 0x7fdd6c13ec90 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.127+0000 7f803effd640 1 -- 172.21.15.72:0/833241882 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80300317f0 con 0x7f80400ce8d0 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.127+0000 7f803effd640 1 -- 172.21.15.72:0/833241882 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8030037070 con 0x7f80400ce8d0 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 -- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80400ce8d0 msgr2=0x7f80400cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80400ce8d0 0x7f80400cbae0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80400b4fa0 tx=0x7f803002b890 comp rx=0 tx=0).stop 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 -- 172.21.15.72:0/833241882 shutdown_connections 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804006afb0 0x7f804006b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 --2- 172.21.15.72:0/833241882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80400ce8d0 0x7f80400cbae0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f80400b4fa0 tx=0x7f803002b890 comp rx=0 tx=0).stop 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 --2- 172.21.15.72:0/833241882 >> v2:172.21.15.72:3300/0 conn(0x7f804014bb10 0x7f8040165fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.128+0000 7f80463b6640 1 -- 172.21.15.72:0/833241882 >> 172.21.15.72:0/833241882 conn(0x7f80400b1830 msgr2=0x7f80400b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.129+0000 7f80463b6640 1 -- 172.21.15.72:0/833241882 shutdown_connections 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.129+0000 7f80463b6640 1 -- 172.21.15.72:0/833241882 wait complete. 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.131+0000 7f80463b6640 1 Processor -- start 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.131+0000 7f80463b6640 1 -- start start 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.132+0000 7f80463b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 0x7f8040167310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.132+0000 7f80463b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804014bb10 0x7f8040144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.132+0000 7f80463b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8040161670 0x7f8040164000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.132+0000 7f80463b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040071ac0 con 0x7f804006afb0 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.132+0000 7f80463b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8040166c90 con 0x7f8040161670 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.132+0000 7f80463b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8040071770 con 0x7f804014bb10 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.133+0000 7f803ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 0x7f8040167310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.133+0000 7f803ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 0x7f8040167310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54530/0 (socket says 172.21.15.72:54530) 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.133+0000 7f803ffff640 1 -- 172.21.15.72:0/1963998264 learned_addr learned my addr 172.21.15.72:0/1963998264 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f803ffff640 1 -- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804014bb10 msgr2=0x7f8040144920 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f803ffff640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804014bb10 0x7f8040144920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f803ffff640 1 -- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8040161670 msgr2=0x7f8040164000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f8044953640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8040161670 0x7f8040164000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f803ffff640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8040161670 0x7f8040164000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f803ffff640 1 -- 172.21.15.72:0/1963998264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80401623a0 con 0x7f804006afb0 2024-03-06T07:21:56.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7f8044953640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8040161670 0x7f8040164000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.133+0000 7fea5ed1e640 1 Processor -- start 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.133+0000 7fea5ed1e640 1 -- start start 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7fea5ed1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea580b4770 0x7fea58164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7fea5ed1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5815b380 con 0x7fea5806ebe0 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7fea5ed1e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea5815e8b0 con 0x7fea58157b20 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7fea5ed1e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5815e420 con 0x7fea580b4770 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.134+0000 7fea5d2bb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea580b4770 0x7fea58164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.135+0000 7fea5d2bb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea580b4770 0x7fea58164de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54544/0 (socket says 172.21.15.72:54544) 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.135+0000 7fea5d2bb640 1 -- 172.21.15.72:0/726933506 learned_addr learned my addr 172.21.15.72:0/726933506 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.135+0000 7fea5d2bb640 1 -- 172.21.15.72:0/726933506 >> v1:172.21.15.146:6789/0 conn(0x7fea58157b20 legacy=0x7fea5806e4d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.135+0000 7fea5d2bb640 1 -- 172.21.15.72:0/726933506 >> v1:172.21.15.72:6789/0 conn(0x7fea5806ebe0 legacy=0x7fea580b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.135+0000 7fea5d2bb640 1 -- 172.21.15.72:0/726933506 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea5806f090 con 0x7fea580b4770 2024-03-06T07:21:56.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.136+0000 7f803ffff640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 0x7f8040167310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f802800b780 tx=0x7f802800bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.137+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8028019070 con 0x7f804006afb0 2024-03-06T07:21:56.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.137+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802800c720 con 0x7f804006afb0 2024-03-06T07:21:56.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.137+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802800cb10 con 0x7f804006afb0 2024-03-06T07:21:56.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.137+0000 7fea5d2bb640 1 --2- 172.21.15.72:0/726933506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580b4770 0x7fea58164de0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fea4c008870 tx=0x7fea4c02df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6a1ee5827f1ff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.137+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8040167850 con 0x7f804006afb0 2024-03-06T07:21:56.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.137+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80401620c0 con 0x7f804006afb0 2024-03-06T07:21:56.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.138+0000 7fea577fe640 1 -- 172.21.15.72:0/726933506 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea4c034070 con 0x7fea580b4770 2024-03-06T07:21:56.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.138+0000 7fea577fe640 1 -- 172.21.15.72:0/726933506 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea4c02ecd0 con 0x7fea580b4770 2024-03-06T07:21:56.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.139+0000 7fea5ed1e640 1 -- 172.21.15.72:0/726933506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580b4770 msgr2=0x7fea58164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.139+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/726933506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580b4770 0x7fea58164de0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fea4c008870 tx=0x7fea4c02df20 comp rx=0 tx=0).stop 2024-03-06T07:21:56.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.140+0000 7fea5ed1e640 1 -- 172.21.15.72:0/726933506 shutdown_connections 2024-03-06T07:21:56.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.140+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/726933506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580b4770 0x7fea58164de0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.140+0000 7fea5ed1e640 1 -- 172.21.15.72:0/726933506 >> 172.21.15.72:0/726933506 conn(0x7fea580b1c50 msgr2=0x7fea580b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.140+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804004f330 con 0x7f804006afb0 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.140+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f802801c070 con 0x7f804006afb0 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.141+0000 7f803cff9640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f802006b5a0 0x7f802006da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.141+0000 7f803f7fe640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f802006b5a0 0x7f802006da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.141+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80280896e0 con 0x7f804006afb0 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.142+0000 7f803f7fe640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f802006b5a0 0x7f802006da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8030036000 tx=0x7f8030008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.143+0000 7fea5ed1e640 1 -- 172.21.15.72:0/726933506 shutdown_connections 2024-03-06T07:21:56.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.150+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f80280585d0 con 0x7f804006afb0 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.149+0000 7fea5ed1e640 1 -- 172.21.15.72:0/726933506 wait complete. 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.150+0000 7fea5ed1e640 1 Processor -- start 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.150+0000 7fea5ed1e640 1 -- start start 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea5ed1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea5806ebe0 0x7fea580a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea5ed1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 0x7fea581529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea5ed1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea58157b20 0x7fea581f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea5ed1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5815e0b0 con 0x7fea5806ebe0 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea5ed1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea5815ddd0 con 0x7fea58157b20 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea5ed1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea5806dfb0 con 0x7fea580b4770 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea57fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 0x7fea581529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea57fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 0x7fea581529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33872/0 (socket says 172.21.15.72:33872) 2024-03-06T07:21:56.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.152+0000 7fea57fff640 1 -- 172.21.15.72:0/1680307124 learned_addr learned my addr 172.21.15.72:0/1680307124 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea5caba640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea5806ebe0 0x7fea580a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea5d2bb640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea58157b20 0x7fea581f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea57fff640 1 -- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea58157b20 msgr2=0x7fea581f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea57fff640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea58157b20 0x7fea581f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea57fff640 1 -- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea5806ebe0 msgr2=0x7fea580a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea57fff640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea5806ebe0 0x7fea580a9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.153+0000 7fea57fff640 1 -- 172.21.15.72:0/1680307124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580ce100 con 0x7fea580b4770 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.154+0000 7fea57fff640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 0x7fea581529f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fea5000ba50 tx=0x7fea5000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.154+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50019070 con 0x7fea580b4770 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.154+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea580ce3e0 con 0x7fea580b4770 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.154+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea580ce9a0 con 0x7fea580b4770 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.156+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580cd270 con 0x7fea580b4770 2024-03-06T07:21:56.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.156+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5000cc20 con 0x7fea580b4770 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.156+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50014680 con 0x7fea580b4770 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.158+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea5001c070 con 0x7fea580b4770 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.159+0000 7fea557fa640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea4406b4d0 0x7fea4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.159+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea5008a6e0 con 0x7fea580b4770 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.159+0000 7fea5caba640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea4406b4d0 0x7fea4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.160+0000 7fea5caba640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea4406b4d0 0x7fea4406d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea48006fd0 tx=0x7fea480090e0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.161+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea580cd270 con 0x7fea580b4770 2024-03-06T07:21:56.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.486+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdd44005580 con 0x7fdd6c13ec90 2024-03-06T07:21:56.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.487+0000 7fdd68ff9640 1 -- 172.21.15.72:0/2537451963 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdd600580e0 con 0x7fdd6c13ec90 2024-03-06T07:21:56.489 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd5806b5d0 msgr2=0x7fdd5806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd5806b5d0 0x7fdd5806da90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdd640138a0 tx=0x7fdd6400b8a0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd6c13ec90 msgr2=0x7fdd6c0b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd6c13ec90 0x7fdd6c0b3690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000caf0 tx=0x7fdd600018d0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 shutdown_connections 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd5806b5d0 0x7fdd5806da90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd6c0b3bd0 0x7fdd6c1fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd6c13ec90 0x7fdd6c0b3690 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 --2- 172.21.15.72:0/2537451963 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd6c13e2d0 0x7fdd6c14c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 >> 172.21.15.72:0/2537451963 conn(0x7fdd6c0b1c80 msgr2=0x7fdd6c158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 shutdown_connections 2024-03-06T07:21:56.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.491+0000 7fdd527fc640 1 -- 172.21.15.72:0/2537451963 wait complete. 2024-03-06T07:21:56.509 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-03-06T07:21:56.509 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.511+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fccb4005580 con 0x7fcccc03aa10 2024-03-06T07:21:56.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.512+0000 7fccc97fa640 1 -- 172.21.15.72:0/4280117927 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fccc4057d30 con 0x7fcccc03aa10 2024-03-06T07:21:56.516 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:56.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fccc006b5a0 msgr2=0x7fccc006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fccc006b5a0 0x7fccc006da60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcccc009730 tx=0x7fccd406d030 comp rx=0 tx=0).stop 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc03aa10 msgr2=0x7fcccc1893d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc03aa10 0x7fcccc1893d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c390 tx=0x7fccc400c850 comp rx=0 tx=0).stop 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 shutdown_connections 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fccc006b5a0 0x7fccc006da60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcccc048c30 0x7fcccc189910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcccc03aa10 0x7fcccc1893d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 --2- 172.21.15.72:0/4280117927 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcccc009180 0x7fcccc007e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 >> 172.21.15.72:0/4280117927 conn(0x7fcccc01ef30 msgr2=0x7fcccc00c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 shutdown_connections 2024-03-06T07:21:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.515+0000 7fccd9e92640 1 -- 172.21.15.72:0/4280117927 wait complete. 2024-03-06T07:21:56.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.522+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd6c4005850 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.523+0000 7fd6cbfff640 1 -- 172.21.15.72:0/2161099208 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6d407ef80 con 0x7fd6dc0eacf0 2024-03-06T07:21:56.525 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:56.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd6d006b5d0 msgr2=0x7fd6d006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd6d006b5d0 0x7fd6d006da90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8006fd0 tx=0x7fd6d8008040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 msgr2=0x7fd6dc0e9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 0x7fd6dc0e9cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4009870 tx=0x7fd6d4004730 comp rx=0 tx=0).stop 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 shutdown_connections 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd6d006b5d0 0x7fd6d006da90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6dc046f10 0x7fd6dc0d7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6dc0eacf0 0x7fd6dc0e9cb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 --2- 172.21.15.72:0/2161099208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6dc006800 0x7fd6dc0393a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 >> 172.21.15.72:0/2161099208 conn(0x7fd6dc009fa0 msgr2=0x7fd6dc049f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 shutdown_connections 2024-03-06T07:21:56.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.526+0000 7fd6e3fff640 1 -- 172.21.15.72:0/2161099208 wait complete. 2024-03-06T07:21:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.533+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f322c005850 con 0x7f324c0d0130 2024-03-06T07:21:56.536 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-06T07:21:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.534+0000 7f32497fa640 1 -- 172.21.15.72:0/911278173 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f324007bea0 con 0x7f324c0d0130 2024-03-06T07:21:56.536 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:56.536 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f323806b670 msgr2=0x7f323806db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f323806b670 0x7f323806db30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3244007fd0 tx=0x7f3244009040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 msgr2=0x7f324c1f75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 0x7f324c1f75a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f324002b9c0 tx=0x7f3240004730 comp rx=0 tx=0).stop 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 shutdown_connections 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f323806b670 0x7f323806db30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f324c0d0a50 0x7f324c15a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f324c0d0130 0x7f324c1f75a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 --2- 172.21.15.72:0/911278173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f324c0af7d0 0x7f324c0cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.537+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 >> 172.21.15.72:0/911278173 conn(0x7f324c072520 msgr2=0x7f324c0ad2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.538+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 shutdown_connections 2024-03-06T07:21:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.538+0000 7f3252c50640 1 -- 172.21.15.72:0/911278173 wait complete. 2024-03-06T07:21:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.539+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9aa40ce1a0 con 0x7f9aa41431f0 2024-03-06T07:21:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.540+0000 7f9a92ffd640 1 -- 172.21.15.72:0/1958076243 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9a9c0597c0 con 0x7f9aa41431f0 2024-03-06T07:21:56.542 INFO:teuthology.orchestra.run.smithi072.stdout:55834574855 2024-03-06T07:21:56.544 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-06T07:21:56.544 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.542+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b5d0 msgr2=0x7f9a8406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b5d0 0x7f9a8406da90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9a94006fd0 tx=0x7f9a94008040 comp rx=0 tx=0).stop 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 msgr2=0x7f9aa40cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 0x7f9aa40cfdc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9aa406e740 tx=0x7f9a9c007700 comp rx=0 tx=0).stop 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 shutdown_connections 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b5d0 0x7f9a8406da90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa41431f0 0x7f9aa40cfdc0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa4142d70 0x7f9aa40cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 --2- 172.21.15.72:0/1958076243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a8c004800 0x7f9aa413f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 >> 172.21.15.72:0/1958076243 conn(0x7f9aa40b1830 msgr2=0x7f9aa40b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 shutdown_connections 2024-03-06T07:21:56.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.543+0000 7f9a90ff9640 1 -- 172.21.15.72:0/1958076243 wait complete. 2024-03-06T07:21:56.553 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-03-06T07:21:56.553 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-03-06T07:21:56.561 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.562+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8040165fd0 con 0x7f804006afb0 2024-03-06T07:21:56.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.562+0000 7f803cff9640 1 -- 172.21.15.72:0/1963998264 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8028057ed0 con 0x7f804006afb0 2024-03-06T07:21:56.564 INFO:teuthology.orchestra.run.smithi072.stdout:55834574853 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f802006b5a0 msgr2=0x7f802006da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f802006b5a0 0x7f802006da60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8030036000 tx=0x7f8030008030 comp rx=0 tx=0).stop 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 msgr2=0x7f8040167310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 0x7f8040167310 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f802800b780 tx=0x7f802800bc40 comp rx=0 tx=0).stop 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 shutdown_connections 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f802006b5a0 0x7f802006da60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8040161670 0x7f8040164000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804014bb10 0x7f8040144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.564+0000 7f80463b6640 1 --2- 172.21.15.72:0/1963998264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f804006afb0 0x7f8040167310 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.565+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 >> 172.21.15.72:0/1963998264 conn(0x7f80400b1830 msgr2=0x7f8040071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.565+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 shutdown_connections 2024-03-06T07:21:56.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.565+0000 7f80463b6640 1 -- 172.21.15.72:0/1963998264 wait complete. 2024-03-06T07:21:56.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.568+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fea580af6e0 con 0x7fea580b4770 2024-03-06T07:21:56.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.568+0000 7fea557fa640 1 -- 172.21.15.72:0/1680307124 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea580af6e0 con 0x7fea580b4770 2024-03-06T07:21:56.570 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea4406b4d0 msgr2=0x7fea4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea4406b4d0 0x7fea4406d990 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fea48006fd0 tx=0x7fea480090e0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 msgr2=0x7fea581529f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 0x7fea581529f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fea5000ba50 tx=0x7fea5000bf10 comp rx=0 tx=0).stop 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 shutdown_connections 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea4406b4d0 0x7fea4406d990 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea58157b20 0x7fea581f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea580b4770 0x7fea581529f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 --2- 172.21.15.72:0/1680307124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea5806ebe0 0x7fea580a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.570+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 >> 172.21.15.72:0/1680307124 conn(0x7fea580b1c50 msgr2=0x7fea5806e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.571+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 shutdown_connections 2024-03-06T07:21:56.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.571+0000 7fea5ed1e640 1 -- 172.21.15.72:0/1680307124 wait complete. 2024-03-06T07:21:56.580 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2024-03-06T07:21:56.580 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-03-06T07:21:56.583 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.678+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1ac406c1d0 con 0x7f1ac415e000 2024-03-06T07:21:56.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.679+0000 7f1aabfff640 1 -- 172.21.15.72:0/745022948 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1abc058050 con 0x7f1ac415e000 2024-03-06T07:21:56.681 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-03-06T07:21:56.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ab006b8d0 msgr2=0x7f1ab006dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ab006b8d0 0x7f1ab006dd90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8036000 tx=0x7f1ab8030000 comp rx=0 tx=0).stop 2024-03-06T07:21:56.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 msgr2=0x7f1ac40cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40cd9e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00caa0 tx=0x7f1abc00cf60 comp rx=0 tx=0).stop 2024-03-06T07:21:56.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 shutdown_connections 2024-03-06T07:21:56.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ab006b8d0 0x7f1ab006dd90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ac415e000 0x7f1ac40cd9e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ac40d4810 0x7f1ac4158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 --2- 172.21.15.72:0/745022948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ac40cf7d0 0x7f1ac40cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 >> 172.21.15.72:0/745022948 conn(0x7f1ac40d3cf0 msgr2=0x7f1ac415a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 shutdown_connections 2024-03-06T07:21:56.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.681+0000 7f1ac9b93640 1 -- 172.21.15.72:0/745022948 wait complete. 2024-03-06T07:21:56.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-03-06T07:21:56.694 DEBUG:teuthology.parallel:result is None 2024-03-06T07:21:56.694 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:21:56.694 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:56.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.794+0000 7f8c5dc37640 1 Processor -- start 2024-03-06T07:21:56.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.794+0000 7f8c5dc37640 1 -- start start 2024-03-06T07:21:56.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c5dc37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c580c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c5dc37640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c580cebe0 con 0x7f8c5815fa10 2024-03-06T07:21:56.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c5dc37640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c580ceec0 con 0x7f8c580d16e0 2024-03-06T07:21:56.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c5dc37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580ce980 con 0x7f8c5815bdd0 2024-03-06T07:21:56.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c56ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8c5815fa10 0x7f8c580ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49204/0 (socket says 172.21.15.72:49204) 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c56ffd640 1 -- 172.21.15.72:0/2313604789 learned_addr learned my addr 172.21.15.72:0/2313604789 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c577fe640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c580c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770669783 0 0) 0x7f8c580cebe0 con 0x7f8c5815fa10 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.795+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c40003520 con 0x7f8c5815fa10 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210527269 0 0) 0x7f8c580ceec0 con 0x7f8c580d16e0 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c580cebe0 con 0x7f8c580d16e0 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093253507 0 0) 0x7f8c40003520 con 0x7f8c5815fa10 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 msgr2=0x7f8c580c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c580c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 >> v1:172.21.15.146:6789/0 conn(0x7f8c580d16e0 legacy=0x7f8c580d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c580cf1a0 con 0x7f8c5815fa10 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f8c4c001670 con 0x7f8c5815fa10 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c577fe640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c580c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:56.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 >> v1:172.21.15.72:6789/0 conn(0x7f8c5815fa10 legacy=0x7f8c580ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c40003b70 0x7f8c40003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c40004800 0x7f8c40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c400051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580ce980 con 0x7f8c40004800 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4000bad0 con 0x7f8c40003b70 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4000bcd0 con 0x7f8c5815bdd0 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c40004800 0x7f8c40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c577fe640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c40003b70 0x7f8c40003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c57fff640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c400051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c56ffd640 1 -- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 msgr2=0x7f8c400051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c5815bdd0 0x7f8c400051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c56ffd640 1 -- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c40003b70 msgr2=0x7f8c40003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c40003b70 0x7f8c40003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.796+0000 7f8c56ffd640 1 -- 172.21.15.72:0/2313604789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c40003520 con 0x7f8c40004800 2024-03-06T07:21:56.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c40004800 0x7f8c40004c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c007c00 tx=0x7f8c40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ee620efb166196 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c007c40 con 0x7f8c40004800 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c005ad0 con 0x7f8c40004800 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c567fc640 1 -- 172.21.15.72:0/2313604789 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c007070 con 0x7f8c40004800 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c5dc37640 1 -- 172.21.15.72:0/2313604789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c40004800 msgr2=0x7f8c40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/2313604789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c40004800 0x7f8c40004c80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c007c00 tx=0x7f8c40005830 comp rx=0 tx=0).stop 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.797+0000 7f8c57fff640 1 -- 172.21.15.72:0/2313604789 reap_dead start 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.798+0000 7f8c5dc37640 1 -- 172.21.15.72:0/2313604789 shutdown_connections 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.798+0000 7f8c5dc37640 1 -- 172.21.15.72:0/2313604789 >> 172.21.15.72:0/2313604789 conn(0x7f8c5809a9f0 msgr2=0x7f8c5809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.798+0000 7f8c5dc37640 1 -- 172.21.15.72:0/2313604789 shutdown_connections 2024-03-06T07:21:56.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.798+0000 7f8c5dc37640 1 -- 172.21.15.72:0/2313604789 wait complete. 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.799+0000 7f8c5dc37640 1 Processor -- start 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.799+0000 7f8c5dc37640 1 -- start start 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c40004800 0x7f8c5815d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 0x7f8c580d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c5815e350 0x7f8c580d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5815c1b0 con 0x7f8c5815def0 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580cc5d0 con 0x7f8c5815e350 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c580c5610 con 0x7f8c40004800 2024-03-06T07:21:56.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 0x7f8c580d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 0x7f8c580d01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54586/0 (socket says 172.21.15.72:54586) 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 -- 172.21.15.72:0/3677905743 learned_addr learned my addr 172.21.15.72:0/3677905743 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c577fe640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c40004800 0x7f8c5815d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c57fff640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c5815e350 0x7f8c580d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 -- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c40004800 msgr2=0x7f8c5815d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c40004800 0x7f8c5815d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 -- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c5815e350 msgr2=0x7f8c580d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c5815e350 0x7f8c580d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 -- 172.21.15.72:0/3677905743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5815c330 con 0x7f8c5815def0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c57fff640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c5815e350 0x7f8c580d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c56ffd640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 0x7f8c580d01a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c1a0 tx=0x7f8c4c001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c041040 con 0x7f8c5815def0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4c04b070 con 0x7f8c5815def0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5815c5f0 con 0x7f8c5815def0 2024-03-06T07:21:56.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c4c001670 con 0x7f8c5815def0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.800+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5815cbb0 con 0x7f8c5815def0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.802+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8c4c0049e0 con 0x7f8c5815def0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.802+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5804f330 con 0x7f8c5815def0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.802+0000 7f8c3ffff640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8c4406b4d0 0x7f8c4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.802+0000 7f8c577fe640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8c4406b4d0 0x7f8c4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.802+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c4c0b3f50 con 0x7f8c5815def0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.802+0000 7f8c577fe640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8c4406b4d0 0x7f8c4406d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009560 tx=0x7f8c48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:56.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:56.806+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8c4c082e60 con 0x7f8c5815def0 2024-03-06T07:21:57.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.010+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c580b7a40 con 0x7f8c4406b4d0 2024-03-06T07:21:57.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.015+0000 7f8c3ffff640 1 -- 172.21.15.72:0/3677905743 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7f8c580b7a40 con 0x7f8c4406b4d0 2024-03-06T07:21:57.020 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:57.021 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8c4406b4d0 msgr2=0x7f8c4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8c4406b4d0 0x7f8c4406d990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8c48009560 tx=0x7f8c48008040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 msgr2=0x7f8c580d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 0x7f8c580d01a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c00c1a0 tx=0x7f8c4c001970 comp rx=0 tx=0).stop 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 shutdown_connections 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8c4406b4d0 0x7f8c4406d990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c5815e350 0x7f8c580d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c5815def0 0x7f8c580d01a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 --2- 172.21.15.72:0/3677905743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c40004800 0x7f8c5815d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.018+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 >> 172.21.15.72:0/3677905743 conn(0x7f8c5809a9f0 msgr2=0x7f8c580703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.019+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 shutdown_connections 2024-03-06T07:21:57.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.019+0000 7f8c5dc37640 1 -- 172.21.15.72:0/3677905743 wait complete. 2024-03-06T07:21:57.032 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:21:55.949121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219088,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755632,"statfs":{"total":773094113280,"available":772869767168,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.893745"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060871+0000","last_change":"2024-03-06T07:21:50.086544+0000","last_active":"2024-03-06T07:21:51.060871+0000","last_peered":"2024-03-06T07:21:51.060871+0000","last_clean":"2024-03-06T07:21:51.060871+0000","last_became_active":"2024-03-06T07:21:50.085415+0000","last_became_peered":"2024-03-06T07:21:50.085415+0000","last_unstale":"2024-03-06T07:21:51.060871+0000","last_undegraded":"2024-03-06T07:21:51.060871+0000","last_fullsized":"2024-03-06T07:21:51.060871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059336+0000","last_change":"2024-03-06T07:21:50.074454+0000","last_active":"2024-03-06T07:21:51.059336+0000","last_peered":"2024-03-06T07:21:51.059336+0000","last_clean":"2024-03-06T07:21:51.059336+0000","last_became_active":"2024-03-06T07:21:50.074183+0000","last_became_peered":"2024-03-06T07:21:50.074183+0000","last_unstale":"2024-03-06T07:21:51.059336+0000","last_undegraded":"2024-03-06T07:21:51.059336+0000","last_fullsized":"2024-03-06T07:21:51.059336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333626+0000","last_change":"2024-03-06T07:21:50.063938+0000","last_active":"2024-03-06T07:21:51.333626+0000","last_peered":"2024-03-06T07:21:51.333626+0000","last_clean":"2024-03-06T07:21:51.333626+0000","last_became_active":"2024-03-06T07:21:50.063674+0000","last_became_peered":"2024-03-06T07:21:50.063674+0000","last_unstale":"2024-03-06T07:21:51.333626+0000","last_undegraded":"2024-03-06T07:21:51.333626+0000","last_fullsized":"2024-03-06T07:21:51.333626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061678+0000","last_change":"2024-03-06T07:21:50.075505+0000","last_active":"2024-03-06T07:21:51.061678+0000","last_peered":"2024-03-06T07:21:51.061678+0000","last_clean":"2024-03-06T07:21:51.061678+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.061678+0000","last_undegraded":"2024-03-06T07:21:51.061678+0000","last_fullsized":"2024-03-06T07:21:51.061678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061574+0000","last_change":"2024-03-06T07:21:50.082456+0000","last_active":"2024-03-06T07:21:51.061574+0000","last_peered":"2024-03-06T07:21:51.061574+0000","last_clean":"2024-03-06T07:21:51.061574+0000","last_became_active":"2024-03-06T07:21:50.081900+0000","last_became_peered":"2024-03-06T07:21:50.081900+0000","last_unstale":"2024-03-06T07:21:51.061574+0000","last_undegraded":"2024-03-06T07:21:51.061574+0000","last_fullsized":"2024-03-06T07:21:51.061574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:30.952471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061465+0000","last_change":"2024-03-06T07:21:50.075967+0000","last_active":"2024-03-06T07:21:51.061465+0000","last_peered":"2024-03-06T07:21:51.061465+0000","last_clean":"2024-03-06T07:21:51.061465+0000","last_became_active":"2024-03-06T07:21:50.075668+0000","last_became_peered":"2024-03-06T07:21:50.075668+0000","last_unstale":"2024-03-06T07:21:51.061465+0000","last_undegraded":"2024-03-06T07:21:51.061465+0000","last_fullsized":"2024-03-06T07:21:51.061465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060628+0000","last_change":"2024-03-06T07:21:50.081328+0000","last_active":"2024-03-06T07:21:51.060628+0000","last_peered":"2024-03-06T07:21:51.060628+0000","last_clean":"2024-03-06T07:21:51.060628+0000","last_became_active":"2024-03-06T07:21:50.080941+0000","last_became_peered":"2024-03-06T07:21:50.080941+0000","last_unstale":"2024-03-06T07:21:51.060628+0000","last_undegraded":"2024-03-06T07:21:51.060628+0000","last_fullsized":"2024-03-06T07:21:51.060628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334303+0000","last_change":"2024-03-06T07:21:50.073822+0000","last_active":"2024-03-06T07:21:51.334303+0000","last_peered":"2024-03-06T07:21:51.334303+0000","last_clean":"2024-03-06T07:21:51.334303+0000","last_became_active":"2024-03-06T07:21:50.073421+0000","last_became_peered":"2024-03-06T07:21:50.073421+0000","last_unstale":"2024-03-06T07:21:51.334303+0000","last_undegraded":"2024-03-06T07:21:51.334303+0000","last_fullsized":"2024-03-06T07:21:51.334303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061065+0000","last_change":"2024-03-06T07:21:50.087355+0000","last_active":"2024-03-06T07:21:51.061065+0000","last_peered":"2024-03-06T07:21:51.061065+0000","last_clean":"2024-03-06T07:21:51.061065+0000","last_became_active":"2024-03-06T07:21:50.086854+0000","last_became_peered":"2024-03-06T07:21:50.086854+0000","last_unstale":"2024-03-06T07:21:51.061065+0000","last_undegraded":"2024-03-06T07:21:51.061065+0000","last_fullsized":"2024-03-06T07:21:51.061065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075461+0000","last_change":"2024-03-06T07:21:50.079046+0000","last_active":"2024-03-06T07:21:51.075461+0000","last_peered":"2024-03-06T07:21:51.075461+0000","last_clean":"2024-03-06T07:21:51.075461+0000","last_became_active":"2024-03-06T07:21:50.078365+0000","last_became_peered":"2024-03-06T07:21:50.078365+0000","last_unstale":"2024-03-06T07:21:51.075461+0000","last_undegraded":"2024-03-06T07:21:51.075461+0000","last_fullsized":"2024-03-06T07:21:51.075461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075358+0000","last_change":"2024-03-06T07:21:50.078291+0000","last_active":"2024-03-06T07:21:51.075358+0000","last_peered":"2024-03-06T07:21:51.075358+0000","last_clean":"2024-03-06T07:21:51.075358+0000","last_became_active":"2024-03-06T07:21:50.078024+0000","last_became_peered":"2024-03-06T07:21:50.078024+0000","last_unstale":"2024-03-06T07:21:51.075358+0000","last_undegraded":"2024-03-06T07:21:51.075358+0000","last_fullsized":"2024-03-06T07:21:51.075358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334132+0000","last_change":"2024-03-06T07:21:50.081726+0000","last_active":"2024-03-06T07:21:51.334132+0000","last_peered":"2024-03-06T07:21:51.334132+0000","last_clean":"2024-03-06T07:21:51.334132+0000","last_became_active":"2024-03-06T07:21:50.081458+0000","last_became_peered":"2024-03-06T07:21:50.081458+0000","last_unstale":"2024-03-06T07:21:51.334132+0000","last_undegraded":"2024-03-06T07:21:51.334132+0000","last_fullsized":"2024-03-06T07:21:51.334132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:41.946707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333563+0000","last_change":"2024-03-06T07:21:50.070195+0000","last_active":"2024-03-06T07:21:51.333563+0000","last_peered":"2024-03-06T07:21:51.333563+0000","last_clean":"2024-03-06T07:21:51.333563+0000","last_became_active":"2024-03-06T07:21:50.069887+0000","last_became_peered":"2024-03-06T07:21:50.069887+0000","last_unstale":"2024-03-06T07:21:51.333563+0000","last_undegraded":"2024-03-06T07:21:51.333563+0000","last_fullsized":"2024-03-06T07:21:51.333563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333869+0000","last_change":"2024-03-06T07:21:50.066290+0000","last_active":"2024-03-06T07:21:51.333869+0000","last_peered":"2024-03-06T07:21:51.333869+0000","last_clean":"2024-03-06T07:21:51.333869+0000","last_became_active":"2024-03-06T07:21:50.066008+0000","last_became_peered":"2024-03-06T07:21:50.066008+0000","last_unstale":"2024-03-06T07:21:51.333869+0000","last_undegraded":"2024-03-06T07:21:51.333869+0000","last_fullsized":"2024-03-06T07:21:51.333869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059103+0000","last_change":"2024-03-06T07:21:50.074011+0000","last_active":"2024-03-06T07:21:51.059103+0000","last_peered":"2024-03-06T07:21:51.059103+0000","last_clean":"2024-03-06T07:21:51.059103+0000","last_became_active":"2024-03-06T07:21:50.073679+0000","last_became_peered":"2024-03-06T07:21:50.073679+0000","last_unstale":"2024-03-06T07:21:51.059103+0000","last_undegraded":"2024-03-06T07:21:51.059103+0000","last_fullsized":"2024-03-06T07:21:51.059103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060633+0000","last_change":"2024-03-06T07:21:50.076532+0000","last_active":"2024-03-06T07:21:51.060633+0000","last_peered":"2024-03-06T07:21:51.060633+0000","last_clean":"2024-03-06T07:21:51.060633+0000","last_became_active":"2024-03-06T07:21:50.075857+0000","last_became_peered":"2024-03-06T07:21:50.075857+0000","last_unstale":"2024-03-06T07:21:51.060633+0000","last_undegraded":"2024-03-06T07:21:51.060633+0000","last_fullsized":"2024-03-06T07:21:51.060633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062233+0000","last_change":"2024-03-06T07:21:50.086644+0000","last_active":"2024-03-06T07:21:51.062233+0000","last_peered":"2024-03-06T07:21:51.062233+0000","last_clean":"2024-03-06T07:21:51.062233+0000","last_became_active":"2024-03-06T07:21:50.084602+0000","last_became_peered":"2024-03-06T07:21:50.084602+0000","last_unstale":"2024-03-06T07:21:51.062233+0000","last_undegraded":"2024-03-06T07:21:51.062233+0000","last_fullsized":"2024-03-06T07:21:51.062233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334765+0000","last_change":"2024-03-06T07:21:50.075838+0000","last_active":"2024-03-06T07:21:51.334765+0000","last_peered":"2024-03-06T07:21:51.334765+0000","last_clean":"2024-03-06T07:21:51.334765+0000","last_became_active":"2024-03-06T07:21:50.075531+0000","last_became_peered":"2024-03-06T07:21:50.075531+0000","last_unstale":"2024-03-06T07:21:51.334765+0000","last_undegraded":"2024-03-06T07:21:51.334765+0000","last_fullsized":"2024-03-06T07:21:51.334765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:00.903565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061683+0000","last_change":"2024-03-06T07:21:50.083575+0000","last_active":"2024-03-06T07:21:51.061683+0000","last_peered":"2024-03-06T07:21:51.061683+0000","last_clean":"2024-03-06T07:21:51.061683+0000","last_became_active":"2024-03-06T07:21:50.083016+0000","last_became_peered":"2024-03-06T07:21:50.083016+0000","last_unstale":"2024-03-06T07:21:51.061683+0000","last_undegraded":"2024-03-06T07:21:51.061683+0000","last_fullsized":"2024-03-06T07:21:51.061683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:20.854195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075317+0000","last_change":"2024-03-06T07:21:50.082493+0000","last_active":"2024-03-06T07:21:51.075317+0000","last_peered":"2024-03-06T07:21:51.075317+0000","last_clean":"2024-03-06T07:21:51.075317+0000","last_became_active":"2024-03-06T07:21:50.082217+0000","last_became_peered":"2024-03-06T07:21:50.082217+0000","last_unstale":"2024-03-06T07:21:51.075317+0000","last_undegraded":"2024-03-06T07:21:51.075317+0000","last_fullsized":"2024-03-06T07:21:51.075317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333683+0000","last_change":"2024-03-06T07:21:50.082417+0000","last_active":"2024-03-06T07:21:51.333683+0000","last_peered":"2024-03-06T07:21:51.333683+0000","last_clean":"2024-03-06T07:21:51.333683+0000","last_became_active":"2024-03-06T07:21:50.082162+0000","last_became_peered":"2024-03-06T07:21:50.082162+0000","last_unstale":"2024-03-06T07:21:51.333683+0000","last_undegraded":"2024-03-06T07:21:51.333683+0000","last_fullsized":"2024-03-06T07:21:51.333683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061987+0000","last_change":"2024-03-06T07:21:50.085327+0000","last_active":"2024-03-06T07:21:51.061987+0000","last_peered":"2024-03-06T07:21:51.061987+0000","last_clean":"2024-03-06T07:21:51.061987+0000","last_became_active":"2024-03-06T07:21:50.085019+0000","last_became_peered":"2024-03-06T07:21:50.085019+0000","last_unstale":"2024-03-06T07:21:51.061987+0000","last_undegraded":"2024-03-06T07:21:51.061987+0000","last_fullsized":"2024-03-06T07:21:51.061987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333521+0000","last_change":"2024-03-06T07:21:50.080803+0000","last_active":"2024-03-06T07:21:51.333521+0000","last_peered":"2024-03-06T07:21:51.333521+0000","last_clean":"2024-03-06T07:21:51.333521+0000","last_became_active":"2024-03-06T07:21:50.080460+0000","last_became_peered":"2024-03-06T07:21:50.080460+0000","last_unstale":"2024-03-06T07:21:51.333521+0000","last_undegraded":"2024-03-06T07:21:51.333521+0000","last_fullsized":"2024-03-06T07:21:51.333521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060865+0000","last_change":"2024-03-06T07:21:50.071650+0000","last_active":"2024-03-06T07:21:51.060865+0000","last_peered":"2024-03-06T07:21:51.060865+0000","last_clean":"2024-03-06T07:21:51.060865+0000","last_became_active":"2024-03-06T07:21:50.071383+0000","last_became_peered":"2024-03-06T07:21:50.071383+0000","last_unstale":"2024-03-06T07:21:51.060865+0000","last_undegraded":"2024-03-06T07:21:51.060865+0000","last_fullsized":"2024-03-06T07:21:51.060865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333966+0000","last_change":"2024-03-06T07:21:50.077589+0000","last_active":"2024-03-06T07:21:51.333966+0000","last_peered":"2024-03-06T07:21:51.333966+0000","last_clean":"2024-03-06T07:21:51.333966+0000","last_became_active":"2024-03-06T07:21:50.077302+0000","last_became_peered":"2024-03-06T07:21:50.077302+0000","last_unstale":"2024-03-06T07:21:51.333966+0000","last_undegraded":"2024-03-06T07:21:51.333966+0000","last_fullsized":"2024-03-06T07:21:51.333966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075478+0000","last_change":"2024-03-06T07:21:50.085031+0000","last_active":"2024-03-06T07:21:51.075478+0000","last_peered":"2024-03-06T07:21:51.075478+0000","last_clean":"2024-03-06T07:21:51.075478+0000","last_became_active":"2024-03-06T07:21:50.084758+0000","last_became_peered":"2024-03-06T07:21:50.084758+0000","last_unstale":"2024-03-06T07:21:51.075478+0000","last_undegraded":"2024-03-06T07:21:51.075478+0000","last_fullsized":"2024-03-06T07:21:51.075478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059511+0000","last_change":"2024-03-06T07:21:50.072164+0000","last_active":"2024-03-06T07:21:51.059511+0000","last_peered":"2024-03-06T07:21:51.059511+0000","last_clean":"2024-03-06T07:21:51.059511+0000","last_became_active":"2024-03-06T07:21:50.071795+0000","last_became_peered":"2024-03-06T07:21:50.071795+0000","last_unstale":"2024-03-06T07:21:51.059511+0000","last_undegraded":"2024-03-06T07:21:51.059511+0000","last_fullsized":"2024-03-06T07:21:51.059511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334595+0000","last_change":"2024-03-06T07:21:50.078217+0000","last_active":"2024-03-06T07:21:51.334595+0000","last_peered":"2024-03-06T07:21:51.334595+0000","last_clean":"2024-03-06T07:21:51.334595+0000","last_became_active":"2024-03-06T07:21:50.077809+0000","last_became_peered":"2024-03-06T07:21:50.077809+0000","last_unstale":"2024-03-06T07:21:51.334595+0000","last_undegraded":"2024-03-06T07:21:51.334595+0000","last_fullsized":"2024-03-06T07:21:51.334595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333826+0000","last_change":"2024-03-06T07:21:50.065564+0000","last_active":"2024-03-06T07:21:51.333826+0000","last_peered":"2024-03-06T07:21:51.333826+0000","last_clean":"2024-03-06T07:21:51.333826+0000","last_became_active":"2024-03-06T07:21:50.065294+0000","last_became_peered":"2024-03-06T07:21:50.065294+0000","last_unstale":"2024-03-06T07:21:51.333826+0000","last_undegraded":"2024-03-06T07:21:51.333826+0000","last_fullsized":"2024-03-06T07:21:51.333826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062062+0000","last_change":"2024-03-06T07:21:50.073038+0000","last_active":"2024-03-06T07:21:51.062062+0000","last_peered":"2024-03-06T07:21:51.062062+0000","last_clean":"2024-03-06T07:21:51.062062+0000","last_became_active":"2024-03-06T07:21:50.072603+0000","last_became_peered":"2024-03-06T07:21:50.072603+0000","last_unstale":"2024-03-06T07:21:51.062062+0000","last_undegraded":"2024-03-06T07:21:51.062062+0000","last_fullsized":"2024-03-06T07:21:51.062062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334317+0000","last_change":"2024-03-06T07:21:50.069962+0000","last_active":"2024-03-06T07:21:51.334317+0000","last_peered":"2024-03-06T07:21:51.334317+0000","last_clean":"2024-03-06T07:21:51.334317+0000","last_became_active":"2024-03-06T07:21:50.069700+0000","last_became_peered":"2024-03-06T07:21:50.069700+0000","last_unstale":"2024-03-06T07:21:51.334317+0000","last_undegraded":"2024-03-06T07:21:51.334317+0000","last_fullsized":"2024-03-06T07:21:51.334317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075081+0000","last_change":"2024-03-06T07:21:50.077457+0000","last_active":"2024-03-06T07:21:51.075081+0000","last_peered":"2024-03-06T07:21:51.075081+0000","last_clean":"2024-03-06T07:21:51.075081+0000","last_became_active":"2024-03-06T07:21:50.077173+0000","last_became_peered":"2024-03-06T07:21:50.077173+0000","last_unstale":"2024-03-06T07:21:51.075081+0000","last_undegraded":"2024-03-06T07:21:51.075081+0000","last_fullsized":"2024-03-06T07:21:51.075081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333933+0000","last_change":"2024-03-06T07:21:50.075503+0000","last_active":"2024-03-06T07:21:51.333933+0000","last_peered":"2024-03-06T07:21:51.333933+0000","last_clean":"2024-03-06T07:21:51.333933+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.333933+0000","last_undegraded":"2024-03-06T07:21:51.333933+0000","last_fullsized":"2024-03-06T07:21:51.333933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:48.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333798+0000","last_change":"2024-03-06T07:21:49.073461+0000","last_active":"2024-03-06T07:21:51.333798+0000","last_peered":"2024-03-06T07:21:51.333798+0000","last_clean":"2024-03-06T07:21:51.333798+0000","last_became_active":"2024-03-06T07:21:49.073212+0000","last_became_peered":"2024-03-06T07:21:49.073212+0000","last_unstale":"2024-03-06T07:21:51.333798+0000","last_undegraded":"2024-03-06T07:21:51.333798+0000","last_fullsized":"2024-03-06T07:21:51.333798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333964+0000","last_change":"2024-03-06T07:21:50.083293+0000","last_active":"2024-03-06T07:21:51.333964+0000","last_peered":"2024-03-06T07:21:51.333964+0000","last_clean":"2024-03-06T07:21:51.333964+0000","last_became_active":"2024-03-06T07:21:50.082931+0000","last_became_peered":"2024-03-06T07:21:50.082931+0000","last_unstale":"2024-03-06T07:21:51.333964+0000","last_undegraded":"2024-03-06T07:21:51.333964+0000","last_fullsized":"2024-03-06T07:21:51.333964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075166+0000","last_change":"2024-03-06T07:21:49.072875+0000","last_active":"2024-03-06T07:21:51.075166+0000","last_peered":"2024-03-06T07:21:51.075166+0000","last_clean":"2024-03-06T07:21:51.075166+0000","last_became_active":"2024-03-06T07:21:49.071548+0000","last_became_peered":"2024-03-06T07:21:49.071548+0000","last_unstale":"2024-03-06T07:21:51.075166+0000","last_undegraded":"2024-03-06T07:21:51.075166+0000","last_fullsized":"2024-03-06T07:21:51.075166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333606+0000","last_change":"2024-03-06T07:21:49.075539+0000","last_active":"2024-03-06T07:21:51.333606+0000","last_peered":"2024-03-06T07:21:51.333606+0000","last_clean":"2024-03-06T07:21:51.333606+0000","last_became_active":"2024-03-06T07:21:49.075192+0000","last_became_peered":"2024-03-06T07:21:49.075192+0000","last_unstale":"2024-03-06T07:21:51.333606+0000","last_undegraded":"2024-03-06T07:21:51.333606+0000","last_fullsized":"2024-03-06T07:21:51.333606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334442+0000","last_change":"2024-03-06T07:21:50.078018+0000","last_active":"2024-03-06T07:21:51.334442+0000","last_peered":"2024-03-06T07:21:51.334442+0000","last_clean":"2024-03-06T07:21:51.334442+0000","last_became_active":"2024-03-06T07:21:50.077638+0000","last_became_peered":"2024-03-06T07:21:50.077638+0000","last_unstale":"2024-03-06T07:21:51.334442+0000","last_undegraded":"2024-03-06T07:21:51.334442+0000","last_fullsized":"2024-03-06T07:21:51.334442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060973+0000","last_change":"2024-03-06T07:21:49.070076+0000","last_active":"2024-03-06T07:21:51.060973+0000","last_peered":"2024-03-06T07:21:51.060973+0000","last_clean":"2024-03-06T07:21:51.060973+0000","last_became_active":"2024-03-06T07:21:49.069491+0000","last_became_peered":"2024-03-06T07:21:49.069491+0000","last_unstale":"2024-03-06T07:21:51.060973+0000","last_undegraded":"2024-03-06T07:21:51.060973+0000","last_fullsized":"2024-03-06T07:21:51.060973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059076+0000","last_change":"2024-03-06T07:21:50.080260+0000","last_active":"2024-03-06T07:21:51.059076+0000","last_peered":"2024-03-06T07:21:51.059076+0000","last_clean":"2024-03-06T07:21:51.059076+0000","last_became_active":"2024-03-06T07:21:50.079987+0000","last_became_peered":"2024-03-06T07:21:50.079987+0000","last_unstale":"2024-03-06T07:21:51.059076+0000","last_undegraded":"2024-03-06T07:21:51.059076+0000","last_fullsized":"2024-03-06T07:21:51.059076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075328+0000","last_change":"2024-03-06T07:21:49.071143+0000","last_active":"2024-03-06T07:21:51.075328+0000","last_peered":"2024-03-06T07:21:51.075328+0000","last_clean":"2024-03-06T07:21:51.075328+0000","last_became_active":"2024-03-06T07:21:49.070872+0000","last_became_peered":"2024-03-06T07:21:49.070872+0000","last_unstale":"2024-03-06T07:21:51.075328+0000","last_undegraded":"2024-03-06T07:21:51.075328+0000","last_fullsized":"2024-03-06T07:21:51.075328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061550+0000","last_change":"2024-03-06T07:21:50.078307+0000","last_active":"2024-03-06T07:21:51.061550+0000","last_peered":"2024-03-06T07:21:51.061550+0000","last_clean":"2024-03-06T07:21:51.061550+0000","last_became_active":"2024-03-06T07:21:50.078025+0000","last_became_peered":"2024-03-06T07:21:50.078025+0000","last_unstale":"2024-03-06T07:21:51.061550+0000","last_undegraded":"2024-03-06T07:21:51.061550+0000","last_fullsized":"2024-03-06T07:21:51.061550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061672+0000","last_change":"2024-03-06T07:21:49.066235+0000","last_active":"2024-03-06T07:21:51.061672+0000","last_peered":"2024-03-06T07:21:51.061672+0000","last_clean":"2024-03-06T07:21:51.061672+0000","last_became_active":"2024-03-06T07:21:49.065968+0000","last_became_peered":"2024-03-06T07:21:49.065968+0000","last_unstale":"2024-03-06T07:21:51.061672+0000","last_undegraded":"2024-03-06T07:21:51.061672+0000","last_fullsized":"2024-03-06T07:21:51.061672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333774+0000","last_change":"2024-03-06T07:21:50.067265+0000","last_active":"2024-03-06T07:21:51.333774+0000","last_peered":"2024-03-06T07:21:51.333774+0000","last_clean":"2024-03-06T07:21:51.333774+0000","last_became_active":"2024-03-06T07:21:50.066634+0000","last_became_peered":"2024-03-06T07:21:50.066634+0000","last_unstale":"2024-03-06T07:21:51.333774+0000","last_undegraded":"2024-03-06T07:21:51.333774+0000","last_fullsized":"2024-03-06T07:21:51.333774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059350+0000","last_change":"2024-03-06T07:21:49.070187+0000","last_active":"2024-03-06T07:21:51.059350+0000","last_peered":"2024-03-06T07:21:51.059350+0000","last_clean":"2024-03-06T07:21:51.059350+0000","last_became_active":"2024-03-06T07:21:49.069894+0000","last_became_peered":"2024-03-06T07:21:49.069894+0000","last_unstale":"2024-03-06T07:21:51.059350+0000","last_undegraded":"2024-03-06T07:21:51.059350+0000","last_fullsized":"2024-03-06T07:21:51.059350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334133+0000","last_change":"2024-03-06T07:21:50.086859+0000","last_active":"2024-03-06T07:21:51.334133+0000","last_peered":"2024-03-06T07:21:51.334133+0000","last_clean":"2024-03-06T07:21:51.334133+0000","last_became_active":"2024-03-06T07:21:50.085246+0000","last_became_peered":"2024-03-06T07:21:50.085246+0000","last_unstale":"2024-03-06T07:21:51.334133+0000","last_undegraded":"2024-03-06T07:21:51.334133+0000","last_fullsized":"2024-03-06T07:21:51.334133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075355+0000","last_change":"2024-03-06T07:21:49.074802+0000","last_active":"2024-03-06T07:21:51.075355+0000","last_peered":"2024-03-06T07:21:51.075355+0000","last_clean":"2024-03-06T07:21:51.075355+0000","last_became_active":"2024-03-06T07:21:49.074143+0000","last_became_peered":"2024-03-06T07:21:49.074143+0000","last_unstale":"2024-03-06T07:21:51.075355+0000","last_undegraded":"2024-03-06T07:21:51.075355+0000","last_fullsized":"2024-03-06T07:21:51.075355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061356+0000","last_change":"2024-03-06T07:21:50.083325+0000","last_active":"2024-03-06T07:21:51.061356+0000","last_peered":"2024-03-06T07:21:51.061356+0000","last_clean":"2024-03-06T07:21:51.061356+0000","last_became_active":"2024-03-06T07:21:50.082675+0000","last_became_peered":"2024-03-06T07:21:50.082675+0000","last_unstale":"2024-03-06T07:21:51.061356+0000","last_undegraded":"2024-03-06T07:21:51.061356+0000","last_fullsized":"2024-03-06T07:21:51.061356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334058+0000","last_change":"2024-03-06T07:21:49.066349+0000","last_active":"2024-03-06T07:21:51.334058+0000","last_peered":"2024-03-06T07:21:51.334058+0000","last_clean":"2024-03-06T07:21:51.334058+0000","last_became_active":"2024-03-06T07:21:49.066024+0000","last_became_peered":"2024-03-06T07:21:49.066024+0000","last_unstale":"2024-03-06T07:21:51.334058+0000","last_undegraded":"2024-03-06T07:21:51.334058+0000","last_fullsized":"2024-03-06T07:21:51.334058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333454+0000","last_change":"2024-03-06T07:21:50.082035+0000","last_active":"2024-03-06T07:21:51.333454+0000","last_peered":"2024-03-06T07:21:51.333454+0000","last_clean":"2024-03-06T07:21:51.333454+0000","last_became_active":"2024-03-06T07:21:50.081731+0000","last_became_peered":"2024-03-06T07:21:50.081731+0000","last_unstale":"2024-03-06T07:21:51.333454+0000","last_undegraded":"2024-03-06T07:21:51.333454+0000","last_fullsized":"2024-03-06T07:21:51.333454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:57.652112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062117+0000","last_change":"2024-03-06T07:21:49.074426+0000","last_active":"2024-03-06T07:21:51.062117+0000","last_peered":"2024-03-06T07:21:51.062117+0000","last_clean":"2024-03-06T07:21:51.062117+0000","last_became_active":"2024-03-06T07:21:49.074117+0000","last_became_peered":"2024-03-06T07:21:49.074117+0000","last_unstale":"2024-03-06T07:21:51.062117+0000","last_undegraded":"2024-03-06T07:21:51.062117+0000","last_fullsized":"2024-03-06T07:21:51.062117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061727+0000","last_change":"2024-03-06T07:21:50.082545+0000","last_active":"2024-03-06T07:21:51.061727+0000","last_peered":"2024-03-06T07:21:51.061727+0000","last_clean":"2024-03-06T07:21:51.061727+0000","last_became_active":"2024-03-06T07:21:50.082197+0000","last_became_peered":"2024-03-06T07:21:50.082197+0000","last_unstale":"2024-03-06T07:21:51.061727+0000","last_undegraded":"2024-03-06T07:21:51.061727+0000","last_fullsized":"2024-03-06T07:21:51.061727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:57.476299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075070+0000","last_change":"2024-03-06T07:21:49.073223+0000","last_active":"2024-03-06T07:21:51.075070+0000","last_peered":"2024-03-06T07:21:51.075070+0000","last_clean":"2024-03-06T07:21:51.075070+0000","last_became_active":"2024-03-06T07:21:49.071609+0000","last_became_peered":"2024-03-06T07:21:49.071609+0000","last_unstale":"2024-03-06T07:21:51.075070+0000","last_undegraded":"2024-03-06T07:21:51.075070+0000","last_fullsized":"2024-03-06T07:21:51.075070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075085+0000","last_change":"2024-03-06T07:21:50.077209+0000","last_active":"2024-03-06T07:21:51.075085+0000","last_peered":"2024-03-06T07:21:51.075085+0000","last_clean":"2024-03-06T07:21:51.075085+0000","last_became_active":"2024-03-06T07:21:50.076936+0000","last_became_peered":"2024-03-06T07:21:50.076936+0000","last_unstale":"2024-03-06T07:21:51.075085+0000","last_undegraded":"2024-03-06T07:21:51.075085+0000","last_fullsized":"2024-03-06T07:21:51.075085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334123+0000","last_change":"2024-03-06T07:21:49.072833+0000","last_active":"2024-03-06T07:21:51.334123+0000","last_peered":"2024-03-06T07:21:51.334123+0000","last_clean":"2024-03-06T07:21:51.334123+0000","last_became_active":"2024-03-06T07:21:49.072543+0000","last_became_peered":"2024-03-06T07:21:49.072543+0000","last_unstale":"2024-03-06T07:21:51.334123+0000","last_undegraded":"2024-03-06T07:21:51.334123+0000","last_fullsized":"2024-03-06T07:21:51.334123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060781+0000","last_change":"2024-03-06T07:21:50.081698+0000","last_active":"2024-03-06T07:21:51.060781+0000","last_peered":"2024-03-06T07:21:51.060781+0000","last_clean":"2024-03-06T07:21:51.060781+0000","last_became_active":"2024-03-06T07:21:50.081294+0000","last_became_peered":"2024-03-06T07:21:50.081294+0000","last_unstale":"2024-03-06T07:21:51.060781+0000","last_undegraded":"2024-03-06T07:21:51.060781+0000","last_fullsized":"2024-03-06T07:21:51.060781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334204+0000","last_change":"2024-03-06T07:21:49.069462+0000","last_active":"2024-03-06T07:21:51.334204+0000","last_peered":"2024-03-06T07:21:51.334204+0000","last_clean":"2024-03-06T07:21:51.334204+0000","last_became_active":"2024-03-06T07:21:49.069161+0000","last_became_peered":"2024-03-06T07:21:49.069161+0000","last_unstale":"2024-03-06T07:21:51.334204+0000","last_undegraded":"2024-03-06T07:21:51.334204+0000","last_fullsized":"2024-03-06T07:21:51.334204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:48.201301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061018+0000","last_change":"2024-03-06T07:21:50.082841+0000","last_active":"2024-03-06T07:21:51.061018+0000","last_peered":"2024-03-06T07:21:51.061018+0000","last_clean":"2024-03-06T07:21:51.061018+0000","last_became_active":"2024-03-06T07:21:50.082543+0000","last_became_peered":"2024-03-06T07:21:50.082543+0000","last_unstale":"2024-03-06T07:21:51.061018+0000","last_undegraded":"2024-03-06T07:21:51.061018+0000","last_fullsized":"2024-03-06T07:21:51.061018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334727+0000","last_change":"2024-03-06T07:21:49.070603+0000","last_active":"2024-03-06T07:21:51.334727+0000","last_peered":"2024-03-06T07:21:51.334727+0000","last_clean":"2024-03-06T07:21:51.334727+0000","last_became_active":"2024-03-06T07:21:49.070341+0000","last_became_peered":"2024-03-06T07:21:49.070341+0000","last_unstale":"2024-03-06T07:21:51.334727+0000","last_undegraded":"2024-03-06T07:21:51.334727+0000","last_fullsized":"2024-03-06T07:21:51.334727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061230+0000","last_change":"2024-03-06T07:21:50.082050+0000","last_active":"2024-03-06T07:21:51.061230+0000","last_peered":"2024-03-06T07:21:51.061230+0000","last_clean":"2024-03-06T07:21:51.061230+0000","last_became_active":"2024-03-06T07:21:50.081723+0000","last_became_peered":"2024-03-06T07:21:50.081723+0000","last_unstale":"2024-03-06T07:21:51.061230+0000","last_undegraded":"2024-03-06T07:21:51.061230+0000","last_fullsized":"2024-03-06T07:21:51.061230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059449+0000","last_change":"2024-03-06T07:21:49.069397+0000","last_active":"2024-03-06T07:21:51.059449+0000","last_peered":"2024-03-06T07:21:51.059449+0000","last_clean":"2024-03-06T07:21:51.059449+0000","last_became_active":"2024-03-06T07:21:49.069101+0000","last_became_peered":"2024-03-06T07:21:49.069101+0000","last_unstale":"2024-03-06T07:21:51.059449+0000","last_undegraded":"2024-03-06T07:21:51.059449+0000","last_fullsized":"2024-03-06T07:21:51.059449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061985+0000","last_change":"2024-03-06T07:21:50.072534+0000","last_active":"2024-03-06T07:21:51.061985+0000","last_peered":"2024-03-06T07:21:51.061985+0000","last_clean":"2024-03-06T07:21:51.061985+0000","last_became_active":"2024-03-06T07:21:50.072252+0000","last_became_peered":"2024-03-06T07:21:50.072252+0000","last_unstale":"2024-03-06T07:21:51.061985+0000","last_undegraded":"2024-03-06T07:21:51.061985+0000","last_fullsized":"2024-03-06T07:21:51.061985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334565+0000","last_change":"2024-03-06T07:21:49.071113+0000","last_active":"2024-03-06T07:21:51.334565+0000","last_peered":"2024-03-06T07:21:51.334565+0000","last_clean":"2024-03-06T07:21:51.334565+0000","last_became_active":"2024-03-06T07:21:49.070854+0000","last_became_peered":"2024-03-06T07:21:49.070854+0000","last_unstale":"2024-03-06T07:21:51.334565+0000","last_undegraded":"2024-03-06T07:21:51.334565+0000","last_fullsized":"2024-03-06T07:21:51.334565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061519+0000","last_change":"2024-03-06T07:21:50.087871+0000","last_active":"2024-03-06T07:21:51.061519+0000","last_peered":"2024-03-06T07:21:51.061519+0000","last_clean":"2024-03-06T07:21:51.061519+0000","last_became_active":"2024-03-06T07:21:50.087545+0000","last_became_peered":"2024-03-06T07:21:50.087545+0000","last_unstale":"2024-03-06T07:21:51.061519+0000","last_undegraded":"2024-03-06T07:21:51.061519+0000","last_fullsized":"2024-03-06T07:21:51.061519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:46.267182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:21:51.061213+0000","last_peered":"2024-03-06T07:21:51.061213+0000","last_clean":"2024-03-06T07:21:51.061213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:21:51.061213+0000","last_undegraded":"2024-03-06T07:21:51.061213+0000","last_fullsized":"2024-03-06T07:21:51.061213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061289+0000","last_change":"2024-03-06T07:21:49.064484+0000","last_active":"2024-03-06T07:21:51.061289+0000","last_peered":"2024-03-06T07:21:51.061289+0000","last_clean":"2024-03-06T07:21:51.061289+0000","last_became_active":"2024-03-06T07:21:49.064202+0000","last_became_peered":"2024-03-06T07:21:49.064202+0000","last_unstale":"2024-03-06T07:21:51.061289+0000","last_undegraded":"2024-03-06T07:21:51.061289+0000","last_fullsized":"2024-03-06T07:21:51.061289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333973+0000","last_change":"2024-03-06T07:21:50.074427+0000","last_active":"2024-03-06T07:21:51.333973+0000","last_peered":"2024-03-06T07:21:51.333973+0000","last_clean":"2024-03-06T07:21:51.333973+0000","last_became_active":"2024-03-06T07:21:50.074131+0000","last_became_peered":"2024-03-06T07:21:50.074131+0000","last_unstale":"2024-03-06T07:21:51.333973+0000","last_undegraded":"2024-03-06T07:21:51.333973+0000","last_fullsized":"2024-03-06T07:21:51.333973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061151+0000","last_change":"2024-03-06T07:21:49.075192+0000","last_active":"2024-03-06T07:21:51.061151+0000","last_peered":"2024-03-06T07:21:51.061151+0000","last_clean":"2024-03-06T07:21:51.061151+0000","last_became_active":"2024-03-06T07:21:49.074904+0000","last_became_peered":"2024-03-06T07:21:49.074904+0000","last_unstale":"2024-03-06T07:21:51.061151+0000","last_undegraded":"2024-03-06T07:21:51.061151+0000","last_fullsized":"2024-03-06T07:21:51.061151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:59.897349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333341+0000","last_change":"2024-03-06T07:21:50.074328+0000","last_active":"2024-03-06T07:21:51.333341+0000","last_peered":"2024-03-06T07:21:51.333341+0000","last_clean":"2024-03-06T07:21:51.333341+0000","last_became_active":"2024-03-06T07:21:50.074021+0000","last_became_peered":"2024-03-06T07:21:50.074021+0000","last_unstale":"2024-03-06T07:21:51.333341+0000","last_undegraded":"2024-03-06T07:21:51.333341+0000","last_fullsized":"2024-03-06T07:21:51.333341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:14.753301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334402+0000","last_change":"2024-03-06T07:21:49.068516+0000","last_active":"2024-03-06T07:21:51.334402+0000","last_peered":"2024-03-06T07:21:51.334402+0000","last_clean":"2024-03-06T07:21:51.334402+0000","last_became_active":"2024-03-06T07:21:49.068184+0000","last_became_peered":"2024-03-06T07:21:49.068184+0000","last_unstale":"2024-03-06T07:21:51.334402+0000","last_undegraded":"2024-03-06T07:21:51.334402+0000","last_fullsized":"2024-03-06T07:21:51.334402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059118+0000","last_change":"2024-03-06T07:21:50.071162+0000","last_active":"2024-03-06T07:21:51.059118+0000","last_peered":"2024-03-06T07:21:51.059118+0000","last_clean":"2024-03-06T07:21:51.059118+0000","last_became_active":"2024-03-06T07:21:50.070848+0000","last_became_peered":"2024-03-06T07:21:50.070848+0000","last_unstale":"2024-03-06T07:21:51.059118+0000","last_undegraded":"2024-03-06T07:21:51.059118+0000","last_fullsized":"2024-03-06T07:21:51.059118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062173+0000","last_change":"2024-03-06T07:21:49.071017+0000","last_active":"2024-03-06T07:21:51.062173+0000","last_peered":"2024-03-06T07:21:51.062173+0000","last_clean":"2024-03-06T07:21:51.062173+0000","last_became_active":"2024-03-06T07:21:49.070588+0000","last_became_peered":"2024-03-06T07:21:49.070588+0000","last_unstale":"2024-03-06T07:21:51.062173+0000","last_undegraded":"2024-03-06T07:21:51.062173+0000","last_fullsized":"2024-03-06T07:21:51.062173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334053+0000","last_change":"2024-03-06T07:21:50.082287+0000","last_active":"2024-03-06T07:21:51.334053+0000","last_peered":"2024-03-06T07:21:51.334053+0000","last_clean":"2024-03-06T07:21:51.334053+0000","last_became_active":"2024-03-06T07:21:50.081767+0000","last_became_peered":"2024-03-06T07:21:50.081767+0000","last_unstale":"2024-03-06T07:21:51.334053+0000","last_undegraded":"2024-03-06T07:21:51.334053+0000","last_fullsized":"2024-03-06T07:21:51.334053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:09.878872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059265+0000","last_change":"2024-03-06T07:21:49.069978+0000","last_active":"2024-03-06T07:21:51.059265+0000","last_peered":"2024-03-06T07:21:51.059265+0000","last_clean":"2024-03-06T07:21:51.059265+0000","last_became_active":"2024-03-06T07:21:49.069223+0000","last_became_peered":"2024-03-06T07:21:49.069223+0000","last_unstale":"2024-03-06T07:21:51.059265+0000","last_undegraded":"2024-03-06T07:21:51.059265+0000","last_fullsized":"2024-03-06T07:21:51.059265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062012+0000","last_change":"2024-03-06T07:21:50.076147+0000","last_active":"2024-03-06T07:21:51.062012+0000","last_peered":"2024-03-06T07:21:51.062012+0000","last_clean":"2024-03-06T07:21:51.062012+0000","last_became_active":"2024-03-06T07:21:50.075840+0000","last_became_peered":"2024-03-06T07:21:50.075840+0000","last_unstale":"2024-03-06T07:21:51.062012+0000","last_undegraded":"2024-03-06T07:21:51.062012+0000","last_fullsized":"2024-03-06T07:21:51.062012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061682+0000","last_change":"2024-03-06T07:21:49.077962+0000","last_active":"2024-03-06T07:21:51.061682+0000","last_peered":"2024-03-06T07:21:51.061682+0000","last_clean":"2024-03-06T07:21:51.061682+0000","last_became_active":"2024-03-06T07:21:49.077633+0000","last_became_peered":"2024-03-06T07:21:49.077633+0000","last_unstale":"2024-03-06T07:21:51.061682+0000","last_undegraded":"2024-03-06T07:21:51.061682+0000","last_fullsized":"2024-03-06T07:21:51.061682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:13.747084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059377+0000","last_change":"2024-03-06T07:21:50.071233+0000","last_active":"2024-03-06T07:21:51.059377+0000","last_peered":"2024-03-06T07:21:51.059377+0000","last_clean":"2024-03-06T07:21:51.059377+0000","last_became_active":"2024-03-06T07:21:50.070925+0000","last_became_peered":"2024-03-06T07:21:50.070925+0000","last_unstale":"2024-03-06T07:21:51.059377+0000","last_undegraded":"2024-03-06T07:21:51.059377+0000","last_fullsized":"2024-03-06T07:21:51.059377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.074529+0000","last_change":"2024-03-06T07:21:49.075388+0000","last_active":"2024-03-06T07:21:51.074529+0000","last_peered":"2024-03-06T07:21:51.074529+0000","last_clean":"2024-03-06T07:21:51.074529+0000","last_became_active":"2024-03-06T07:21:49.075115+0000","last_became_peered":"2024-03-06T07:21:49.075115+0000","last_unstale":"2024-03-06T07:21:51.074529+0000","last_undegraded":"2024-03-06T07:21:51.074529+0000","last_fullsized":"2024-03-06T07:21:51.074529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334111+0000","last_change":"2024-03-06T07:21:50.084419+0000","last_active":"2024-03-06T07:21:51.334111+0000","last_peered":"2024-03-06T07:21:51.334111+0000","last_clean":"2024-03-06T07:21:51.334111+0000","last_became_active":"2024-03-06T07:21:50.084022+0000","last_became_peered":"2024-03-06T07:21:50.084022+0000","last_unstale":"2024-03-06T07:21:51.334111+0000","last_undegraded":"2024-03-06T07:21:51.334111+0000","last_fullsized":"2024-03-06T07:21:51.334111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075351+0000","last_change":"2024-03-06T07:21:49.072877+0000","last_active":"2024-03-06T07:21:51.075351+0000","last_peered":"2024-03-06T07:21:51.075351+0000","last_clean":"2024-03-06T07:21:51.075351+0000","last_became_active":"2024-03-06T07:21:49.072559+0000","last_became_peered":"2024-03-06T07:21:49.072559+0000","last_unstale":"2024-03-06T07:21:51.075351+0000","last_undegraded":"2024-03-06T07:21:51.075351+0000","last_fullsized":"2024-03-06T07:21:51.075351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333756+0000","last_change":"2024-03-06T07:21:50.069013+0000","last_active":"2024-03-06T07:21:51.333756+0000","last_peered":"2024-03-06T07:21:51.333756+0000","last_clean":"2024-03-06T07:21:51.333756+0000","last_became_active":"2024-03-06T07:21:50.068753+0000","last_became_peered":"2024-03-06T07:21:50.068753+0000","last_unstale":"2024-03-06T07:21:51.333756+0000","last_undegraded":"2024-03-06T07:21:51.333756+0000","last_fullsized":"2024-03-06T07:21:51.333756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060433+0000","last_change":"2024-03-06T07:21:49.077101+0000","last_active":"2024-03-06T07:21:51.060433+0000","last_peered":"2024-03-06T07:21:51.060433+0000","last_clean":"2024-03-06T07:21:51.060433+0000","last_became_active":"2024-03-06T07:21:49.076900+0000","last_became_peered":"2024-03-06T07:21:49.076900+0000","last_unstale":"2024-03-06T07:21:51.060433+0000","last_undegraded":"2024-03-06T07:21:51.060433+0000","last_fullsized":"2024-03-06T07:21:51.060433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:37.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334046+0000","last_change":"2024-03-06T07:21:50.072349+0000","last_active":"2024-03-06T07:21:51.334046+0000","last_peered":"2024-03-06T07:21:51.334046+0000","last_clean":"2024-03-06T07:21:51.334046+0000","last_became_active":"2024-03-06T07:21:50.072052+0000","last_became_peered":"2024-03-06T07:21:50.072052+0000","last_unstale":"2024-03-06T07:21:51.334046+0000","last_undegraded":"2024-03-06T07:21:51.334046+0000","last_fullsized":"2024-03-06T07:21:51.334046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:59.429356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059318+0000","last_change":"2024-03-06T07:21:49.072417+0000","last_active":"2024-03-06T07:21:51.059318+0000","last_peered":"2024-03-06T07:21:51.059318+0000","last_clean":"2024-03-06T07:21:51.059318+0000","last_became_active":"2024-03-06T07:21:49.072153+0000","last_became_peered":"2024-03-06T07:21:49.072153+0000","last_unstale":"2024-03-06T07:21:51.059318+0000","last_undegraded":"2024-03-06T07:21:51.059318+0000","last_fullsized":"2024-03-06T07:21:51.059318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334590+0000","last_change":"2024-03-06T07:21:50.068165+0000","last_active":"2024-03-06T07:21:51.334590+0000","last_peered":"2024-03-06T07:21:51.334590+0000","last_clean":"2024-03-06T07:21:51.334590+0000","last_became_active":"2024-03-06T07:21:50.067278+0000","last_became_peered":"2024-03-06T07:21:50.067278+0000","last_unstale":"2024-03-06T07:21:51.334590+0000","last_undegraded":"2024-03-06T07:21:51.334590+0000","last_fullsized":"2024-03-06T07:21:51.334590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059227+0000","last_change":"2024-03-06T07:21:49.074974+0000","last_active":"2024-03-06T07:21:51.059227+0000","last_peered":"2024-03-06T07:21:51.059227+0000","last_clean":"2024-03-06T07:21:51.059227+0000","last_became_active":"2024-03-06T07:21:49.074697+0000","last_became_peered":"2024-03-06T07:21:49.074697+0000","last_unstale":"2024-03-06T07:21:51.059227+0000","last_undegraded":"2024-03-06T07:21:51.059227+0000","last_fullsized":"2024-03-06T07:21:51.059227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333690+0000","last_change":"2024-03-06T07:21:50.078075+0000","last_active":"2024-03-06T07:21:51.333690+0000","last_peered":"2024-03-06T07:21:51.333690+0000","last_clean":"2024-03-06T07:21:51.333690+0000","last_became_active":"2024-03-06T07:21:50.077794+0000","last_became_peered":"2024-03-06T07:21:50.077794+0000","last_unstale":"2024-03-06T07:21:51.333690+0000","last_undegraded":"2024-03-06T07:21:51.333690+0000","last_fullsized":"2024-03-06T07:21:51.333690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061819+0000","last_change":"2024-03-06T07:21:49.073724+0000","last_active":"2024-03-06T07:21:51.061819+0000","last_peered":"2024-03-06T07:21:51.061819+0000","last_clean":"2024-03-06T07:21:51.061819+0000","last_became_active":"2024-03-06T07:21:49.073443+0000","last_became_peered":"2024-03-06T07:21:49.073443+0000","last_unstale":"2024-03-06T07:21:51.061819+0000","last_undegraded":"2024-03-06T07:21:51.061819+0000","last_fullsized":"2024-03-06T07:21:51.061819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:56.645896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075154+0000","last_change":"2024-03-06T07:21:50.081464+0000","last_active":"2024-03-06T07:21:51.075154+0000","last_peered":"2024-03-06T07:21:51.075154+0000","last_clean":"2024-03-06T07:21:51.075154+0000","last_became_active":"2024-03-06T07:21:50.081190+0000","last_became_peered":"2024-03-06T07:21:50.081190+0000","last_unstale":"2024-03-06T07:21:51.075154+0000","last_undegraded":"2024-03-06T07:21:51.075154+0000","last_fullsized":"2024-03-06T07:21:51.075154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061935+0000","last_change":"2024-03-06T07:21:50.074879+0000","last_active":"2024-03-06T07:21:51.061935+0000","last_peered":"2024-03-06T07:21:51.061935+0000","last_clean":"2024-03-06T07:21:51.061935+0000","last_became_active":"2024-03-06T07:21:50.074633+0000","last_became_peered":"2024-03-06T07:21:50.074633+0000","last_unstale":"2024-03-06T07:21:51.061935+0000","last_undegraded":"2024-03-06T07:21:51.061935+0000","last_fullsized":"2024-03-06T07:21:51.061935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061004+0000","last_change":"2024-03-06T07:21:49.078149+0000","last_active":"2024-03-06T07:21:51.061004+0000","last_peered":"2024-03-06T07:21:51.061004+0000","last_clean":"2024-03-06T07:21:51.061004+0000","last_became_active":"2024-03-06T07:21:49.077845+0000","last_became_peered":"2024-03-06T07:21:49.077845+0000","last_unstale":"2024-03-06T07:21:51.061004+0000","last_undegraded":"2024-03-06T07:21:51.061004+0000","last_fullsized":"2024-03-06T07:21:51.061004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061862+0000","last_change":"2024-03-06T07:21:50.073658+0000","last_active":"2024-03-06T07:21:51.061862+0000","last_peered":"2024-03-06T07:21:51.061862+0000","last_clean":"2024-03-06T07:21:51.061862+0000","last_became_active":"2024-03-06T07:21:50.073396+0000","last_became_peered":"2024-03-06T07:21:50.073396+0000","last_unstale":"2024-03-06T07:21:51.061862+0000","last_undegraded":"2024-03-06T07:21:51.061862+0000","last_fullsized":"2024-03-06T07:21:51.061862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061590+0000","last_change":"2024-03-06T07:21:49.078569+0000","last_active":"2024-03-06T07:21:51.061590+0000","last_peered":"2024-03-06T07:21:51.061590+0000","last_clean":"2024-03-06T07:21:51.061590+0000","last_became_active":"2024-03-06T07:21:49.078274+0000","last_became_peered":"2024-03-06T07:21:49.078274+0000","last_unstale":"2024-03-06T07:21:51.061590+0000","last_undegraded":"2024-03-06T07:21:51.061590+0000","last_fullsized":"2024-03-06T07:21:51.061590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:23.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333637+0000","last_change":"2024-03-06T07:21:50.067267+0000","last_active":"2024-03-06T07:21:51.333637+0000","last_peered":"2024-03-06T07:21:51.333637+0000","last_clean":"2024-03-06T07:21:51.333637+0000","last_became_active":"2024-03-06T07:21:50.067014+0000","last_became_peered":"2024-03-06T07:21:50.067014+0000","last_unstale":"2024-03-06T07:21:51.333637+0000","last_undegraded":"2024-03-06T07:21:51.333637+0000","last_fullsized":"2024-03-06T07:21:51.333637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334431+0000","last_change":"2024-03-06T07:21:49.066664+0000","last_active":"2024-03-06T07:21:51.334431+0000","last_peered":"2024-03-06T07:21:51.334431+0000","last_clean":"2024-03-06T07:21:51.334431+0000","last_became_active":"2024-03-06T07:21:49.066350+0000","last_became_peered":"2024-03-06T07:21:49.066350+0000","last_unstale":"2024-03-06T07:21:51.334431+0000","last_undegraded":"2024-03-06T07:21:51.334431+0000","last_fullsized":"2024-03-06T07:21:51.334431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334333+0000","last_change":"2024-03-06T07:21:50.079262+0000","last_active":"2024-03-06T07:21:51.334333+0000","last_peered":"2024-03-06T07:21:51.334333+0000","last_clean":"2024-03-06T07:21:51.334333+0000","last_became_active":"2024-03-06T07:21:50.078776+0000","last_became_peered":"2024-03-06T07:21:50.078776+0000","last_unstale":"2024-03-06T07:21:51.334333+0000","last_undegraded":"2024-03-06T07:21:51.334333+0000","last_fullsized":"2024-03-06T07:21:51.334333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333867+0000","last_change":"2024-03-06T07:21:49.070371+0000","last_active":"2024-03-06T07:21:51.333867+0000","last_peered":"2024-03-06T07:21:51.333867+0000","last_clean":"2024-03-06T07:21:51.333867+0000","last_became_active":"2024-03-06T07:21:49.070100+0000","last_became_peered":"2024-03-06T07:21:49.070100+0000","last_unstale":"2024-03-06T07:21:51.333867+0000","last_undegraded":"2024-03-06T07:21:51.333867+0000","last_fullsized":"2024-03-06T07:21:51.333867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:44.026321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:57.036 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:21:57.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.171+0000 7f4dec74b640 1 Processor -- start 2024-03-06T07:21:57.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.171+0000 7f4dec74b640 1 -- start start 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4dec74b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de415baf0 0x7f4de4072700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4dec74b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4de406b6c0 0x7f4de40ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4dec74b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40d0610 con 0x7f4de40cd390 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4dec74b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de4072c40 con 0x7f4de406b6c0 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4dec74b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40d08f0 con 0x7f4de415baf0 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4de9ce6640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4de40cd390 0x7f4de406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49210/0 (socket says 172.21.15.72:49210) 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4deace8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4de406b6c0 0x7f4de40ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4de9ce6640 1 -- 172.21.15.72:0/2775062665 learned_addr learned my addr 172.21.15.72:0/2775062665 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.172+0000 7f4dea4e7640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de415baf0 0x7f4de4072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.173+0000 7f4deace8640 1 -- 172.21.15.72:0/2775062665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de415baf0 msgr2=0x7f4de4072700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.173+0000 7f4deace8640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de415baf0 0x7f4de4072700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.173+0000 7f4deace8640 1 -- 172.21.15.72:0/2775062665 >> v1:172.21.15.72:6789/0 conn(0x7f4de40cd390 legacy=0x7f4de406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.173+0000 7f4deace8640 1 -- 172.21.15.72:0/2775062665 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40d0bd0 con 0x7f4de406b6c0 2024-03-06T07:21:57.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.173+0000 7f4de94e5640 1 -- 172.21.15.72:0/2775062665 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728634615 0 0) 0x7f4de40d0610 con 0x7f4de40cd390 2024-03-06T07:21:57.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.173+0000 7f4dea4e7640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de415baf0 0x7f4de4072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.174+0000 7f4deace8640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de406b6c0 0x7f4de40ced90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc009dc0 tx=0x7f4ddc02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8613130b8444b8e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.174+0000 7f4de94e5640 1 -- 172.21.15.72:0/2775062665 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc039070 con 0x7f4de406b6c0 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.174+0000 7f4de94e5640 1 -- 172.21.15.72:0/2775062665 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc02cc40 con 0x7f4de406b6c0 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.174+0000 7f4de94e5640 1 -- 172.21.15.72:0/2775062665 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc034440 con 0x7f4de406b6c0 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.174+0000 7f4dec74b640 1 -- 172.21.15.72:0/2775062665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de406b6c0 msgr2=0x7f4de40ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.174+0000 7f4dec74b640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de406b6c0 0x7f4de40ced90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc009dc0 tx=0x7f4ddc02be90 comp rx=0 tx=0).stop 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.175+0000 7f4dec74b640 1 -- 172.21.15.72:0/2775062665 shutdown_connections 2024-03-06T07:21:57.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.175+0000 7f4dec74b640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de406b6c0 0x7f4de40ced90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.175+0000 7f4dec74b640 1 --2- 172.21.15.72:0/2775062665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de415baf0 0x7f4de4072700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.175+0000 7f4dec74b640 1 -- 172.21.15.72:0/2775062665 >> 172.21.15.72:0/2775062665 conn(0x7f4de406c0c0 msgr2=0x7f4de406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.175+0000 7f4dec74b640 1 -- 172.21.15.72:0/2775062665 shutdown_connections 2024-03-06T07:21:57.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.175+0000 7f4dec74b640 1 -- 172.21.15.72:0/2775062665 wait complete. 2024-03-06T07:21:57.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 Processor -- start 2024-03-06T07:21:57.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 -- start start 2024-03-06T07:21:57.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de406b6c0 0x7f4de41414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 0x7f4de4154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de415baf0 0x7f4de4151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de415bed0 con 0x7f4de40cd390 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40d1190 con 0x7f4de415baf0 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4dec74b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40d16b0 con 0x7f4de406b6c0 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4de9ce6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 0x7f4de4154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.176+0000 7f4de9ce6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 0x7f4de4154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54598/0 (socket says 172.21.15.72:54598) 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 -- 172.21.15.72:0/1786527285 learned_addr learned my addr 172.21.15.72:0/1786527285 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4dea4e7640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de406b6c0 0x7f4de41414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4deace8640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de415baf0 0x7f4de4151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 -- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de406b6c0 msgr2=0x7f4de41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de406b6c0 0x7f4de41414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 -- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de415baf0 msgr2=0x7f4de4151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de415baf0 0x7f4de4151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 -- 172.21.15.72:0/1786527285 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4154b00 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4deace8640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de415baf0 0x7f4de4151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4de9ce6640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 0x7f4de4154040 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400c500 tx=0x7f4dd400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4018070 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd4007580 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4004a20 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de4154de0 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.177+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de4144c10 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.178+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de404f330 con 0x7f4de40cd390 2024-03-06T07:21:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.178+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dd401b070 con 0x7f4de40cd390 2024-03-06T07:21:57.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.179+0000 7f4ddaffd640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.179+0000 7f4dea4e7640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.179+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4dd40893f0 con 0x7f4de40cd390 2024-03-06T07:21:57.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.179+0000 7f4dea4e7640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4de0006e80 tx=0x7f4de0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.188+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dd4057cb0 con 0x7f4de40cd390 2024-03-06T07:21:57.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.361+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4de4071f00 con 0x7f4dc806b4d0 2024-03-06T07:21:57.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.367+0000 7f4ddaffd640 1 -- 172.21.15.72:0/1786527285 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251836 (secure 0 0 0) 0x7f4de4071f00 con 0x7f4dc806b4d0 2024-03-06T07:21:57.371 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:57.373 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 msgr2=0x7f4dc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4de0006e80 tx=0x7f4de0008040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 msgr2=0x7f4de4154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 0x7f4de4154040 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400c500 tx=0x7f4dd400c9c0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 shutdown_connections 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de415baf0 0x7f4de4151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40cd390 0x7f4de4154040 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 --2- 172.21.15.72:0/1786527285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de406b6c0 0x7f4de41414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 >> 172.21.15.72:0/1786527285 conn(0x7f4de406c0c0 msgr2=0x7f4de406ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 shutdown_connections 2024-03-06T07:21:57.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.370+0000 7f4dec74b640 1 -- 172.21.15.72:0/1786527285 wait complete. 2024-03-06T07:21:57.383 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-06T07:21:55.949121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219088,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755632,"statfs":{"total":773094113280,"available":772869767168,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.893745"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060871+0000","last_change":"2024-03-06T07:21:50.086544+0000","last_active":"2024-03-06T07:21:51.060871+0000","last_peered":"2024-03-06T07:21:51.060871+0000","last_clean":"2024-03-06T07:21:51.060871+0000","last_became_active":"2024-03-06T07:21:50.085415+0000","last_became_peered":"2024-03-06T07:21:50.085415+0000","last_unstale":"2024-03-06T07:21:51.060871+0000","last_undegraded":"2024-03-06T07:21:51.060871+0000","last_fullsized":"2024-03-06T07:21:51.060871+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059336+0000","last_change":"2024-03-06T07:21:50.074454+0000","last_active":"2024-03-06T07:21:51.059336+0000","last_peered":"2024-03-06T07:21:51.059336+0000","last_clean":"2024-03-06T07:21:51.059336+0000","last_became_active":"2024-03-06T07:21:50.074183+0000","last_became_peered":"2024-03-06T07:21:50.074183+0000","last_unstale":"2024-03-06T07:21:51.059336+0000","last_undegraded":"2024-03-06T07:21:51.059336+0000","last_fullsized":"2024-03-06T07:21:51.059336+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333626+0000","last_change":"2024-03-06T07:21:50.063938+0000","last_active":"2024-03-06T07:21:51.333626+0000","last_peered":"2024-03-06T07:21:51.333626+0000","last_clean":"2024-03-06T07:21:51.333626+0000","last_became_active":"2024-03-06T07:21:50.063674+0000","last_became_peered":"2024-03-06T07:21:50.063674+0000","last_unstale":"2024-03-06T07:21:51.333626+0000","last_undegraded":"2024-03-06T07:21:51.333626+0000","last_fullsized":"2024-03-06T07:21:51.333626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061678+0000","last_change":"2024-03-06T07:21:50.075505+0000","last_active":"2024-03-06T07:21:51.061678+0000","last_peered":"2024-03-06T07:21:51.061678+0000","last_clean":"2024-03-06T07:21:51.061678+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.061678+0000","last_undegraded":"2024-03-06T07:21:51.061678+0000","last_fullsized":"2024-03-06T07:21:51.061678+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061574+0000","last_change":"2024-03-06T07:21:50.082456+0000","last_active":"2024-03-06T07:21:51.061574+0000","last_peered":"2024-03-06T07:21:51.061574+0000","last_clean":"2024-03-06T07:21:51.061574+0000","last_became_active":"2024-03-06T07:21:50.081900+0000","last_became_peered":"2024-03-06T07:21:50.081900+0000","last_unstale":"2024-03-06T07:21:51.061574+0000","last_undegraded":"2024-03-06T07:21:51.061574+0000","last_fullsized":"2024-03-06T07:21:51.061574+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:30.952471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061465+0000","last_change":"2024-03-06T07:21:50.075967+0000","last_active":"2024-03-06T07:21:51.061465+0000","last_peered":"2024-03-06T07:21:51.061465+0000","last_clean":"2024-03-06T07:21:51.061465+0000","last_became_active":"2024-03-06T07:21:50.075668+0000","last_became_peered":"2024-03-06T07:21:50.075668+0000","last_unstale":"2024-03-06T07:21:51.061465+0000","last_undegraded":"2024-03-06T07:21:51.061465+0000","last_fullsized":"2024-03-06T07:21:51.061465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060628+0000","last_change":"2024-03-06T07:21:50.081328+0000","last_active":"2024-03-06T07:21:51.060628+0000","last_peered":"2024-03-06T07:21:51.060628+0000","last_clean":"2024-03-06T07:21:51.060628+0000","last_became_active":"2024-03-06T07:21:50.080941+0000","last_became_peered":"2024-03-06T07:21:50.080941+0000","last_unstale":"2024-03-06T07:21:51.060628+0000","last_undegraded":"2024-03-06T07:21:51.060628+0000","last_fullsized":"2024-03-06T07:21:51.060628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334303+0000","last_change":"2024-03-06T07:21:50.073822+0000","last_active":"2024-03-06T07:21:51.334303+0000","last_peered":"2024-03-06T07:21:51.334303+0000","last_clean":"2024-03-06T07:21:51.334303+0000","last_became_active":"2024-03-06T07:21:50.073421+0000","last_became_peered":"2024-03-06T07:21:50.073421+0000","last_unstale":"2024-03-06T07:21:51.334303+0000","last_undegraded":"2024-03-06T07:21:51.334303+0000","last_fullsized":"2024-03-06T07:21:51.334303+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:45.415865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061065+0000","last_change":"2024-03-06T07:21:50.087355+0000","last_active":"2024-03-06T07:21:51.061065+0000","last_peered":"2024-03-06T07:21:51.061065+0000","last_clean":"2024-03-06T07:21:51.061065+0000","last_became_active":"2024-03-06T07:21:50.086854+0000","last_became_peered":"2024-03-06T07:21:50.086854+0000","last_unstale":"2024-03-06T07:21:51.061065+0000","last_undegraded":"2024-03-06T07:21:51.061065+0000","last_fullsized":"2024-03-06T07:21:51.061065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075461+0000","last_change":"2024-03-06T07:21:50.079046+0000","last_active":"2024-03-06T07:21:51.075461+0000","last_peered":"2024-03-06T07:21:51.075461+0000","last_clean":"2024-03-06T07:21:51.075461+0000","last_became_active":"2024-03-06T07:21:50.078365+0000","last_became_peered":"2024-03-06T07:21:50.078365+0000","last_unstale":"2024-03-06T07:21:51.075461+0000","last_undegraded":"2024-03-06T07:21:51.075461+0000","last_fullsized":"2024-03-06T07:21:51.075461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075358+0000","last_change":"2024-03-06T07:21:50.078291+0000","last_active":"2024-03-06T07:21:51.075358+0000","last_peered":"2024-03-06T07:21:51.075358+0000","last_clean":"2024-03-06T07:21:51.075358+0000","last_became_active":"2024-03-06T07:21:50.078024+0000","last_became_peered":"2024-03-06T07:21:50.078024+0000","last_unstale":"2024-03-06T07:21:51.075358+0000","last_undegraded":"2024-03-06T07:21:51.075358+0000","last_fullsized":"2024-03-06T07:21:51.075358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334132+0000","last_change":"2024-03-06T07:21:50.081726+0000","last_active":"2024-03-06T07:21:51.334132+0000","last_peered":"2024-03-06T07:21:51.334132+0000","last_clean":"2024-03-06T07:21:51.334132+0000","last_became_active":"2024-03-06T07:21:50.081458+0000","last_became_peered":"2024-03-06T07:21:50.081458+0000","last_unstale":"2024-03-06T07:21:51.334132+0000","last_undegraded":"2024-03-06T07:21:51.334132+0000","last_fullsized":"2024-03-06T07:21:51.334132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:41.946707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333563+0000","last_change":"2024-03-06T07:21:50.070195+0000","last_active":"2024-03-06T07:21:51.333563+0000","last_peered":"2024-03-06T07:21:51.333563+0000","last_clean":"2024-03-06T07:21:51.333563+0000","last_became_active":"2024-03-06T07:21:50.069887+0000","last_became_peered":"2024-03-06T07:21:50.069887+0000","last_unstale":"2024-03-06T07:21:51.333563+0000","last_undegraded":"2024-03-06T07:21:51.333563+0000","last_fullsized":"2024-03-06T07:21:51.333563+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333869+0000","last_change":"2024-03-06T07:21:50.066290+0000","last_active":"2024-03-06T07:21:51.333869+0000","last_peered":"2024-03-06T07:21:51.333869+0000","last_clean":"2024-03-06T07:21:51.333869+0000","last_became_active":"2024-03-06T07:21:50.066008+0000","last_became_peered":"2024-03-06T07:21:50.066008+0000","last_unstale":"2024-03-06T07:21:51.333869+0000","last_undegraded":"2024-03-06T07:21:51.333869+0000","last_fullsized":"2024-03-06T07:21:51.333869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059103+0000","last_change":"2024-03-06T07:21:50.074011+0000","last_active":"2024-03-06T07:21:51.059103+0000","last_peered":"2024-03-06T07:21:51.059103+0000","last_clean":"2024-03-06T07:21:51.059103+0000","last_became_active":"2024-03-06T07:21:50.073679+0000","last_became_peered":"2024-03-06T07:21:50.073679+0000","last_unstale":"2024-03-06T07:21:51.059103+0000","last_undegraded":"2024-03-06T07:21:51.059103+0000","last_fullsized":"2024-03-06T07:21:51.059103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060633+0000","last_change":"2024-03-06T07:21:50.076532+0000","last_active":"2024-03-06T07:21:51.060633+0000","last_peered":"2024-03-06T07:21:51.060633+0000","last_clean":"2024-03-06T07:21:51.060633+0000","last_became_active":"2024-03-06T07:21:50.075857+0000","last_became_peered":"2024-03-06T07:21:50.075857+0000","last_unstale":"2024-03-06T07:21:51.060633+0000","last_undegraded":"2024-03-06T07:21:51.060633+0000","last_fullsized":"2024-03-06T07:21:51.060633+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062233+0000","last_change":"2024-03-06T07:21:50.086644+0000","last_active":"2024-03-06T07:21:51.062233+0000","last_peered":"2024-03-06T07:21:51.062233+0000","last_clean":"2024-03-06T07:21:51.062233+0000","last_became_active":"2024-03-06T07:21:50.084602+0000","last_became_peered":"2024-03-06T07:21:50.084602+0000","last_unstale":"2024-03-06T07:21:51.062233+0000","last_undegraded":"2024-03-06T07:21:51.062233+0000","last_fullsized":"2024-03-06T07:21:51.062233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334765+0000","last_change":"2024-03-06T07:21:50.075838+0000","last_active":"2024-03-06T07:21:51.334765+0000","last_peered":"2024-03-06T07:21:51.334765+0000","last_clean":"2024-03-06T07:21:51.334765+0000","last_became_active":"2024-03-06T07:21:50.075531+0000","last_became_peered":"2024-03-06T07:21:50.075531+0000","last_unstale":"2024-03-06T07:21:51.334765+0000","last_undegraded":"2024-03-06T07:21:51.334765+0000","last_fullsized":"2024-03-06T07:21:51.334765+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:03:00.903565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061683+0000","last_change":"2024-03-06T07:21:50.083575+0000","last_active":"2024-03-06T07:21:51.061683+0000","last_peered":"2024-03-06T07:21:51.061683+0000","last_clean":"2024-03-06T07:21:51.061683+0000","last_became_active":"2024-03-06T07:21:50.083016+0000","last_became_peered":"2024-03-06T07:21:50.083016+0000","last_unstale":"2024-03-06T07:21:51.061683+0000","last_undegraded":"2024-03-06T07:21:51.061683+0000","last_fullsized":"2024-03-06T07:21:51.061683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:20.854195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075317+0000","last_change":"2024-03-06T07:21:50.082493+0000","last_active":"2024-03-06T07:21:51.075317+0000","last_peered":"2024-03-06T07:21:51.075317+0000","last_clean":"2024-03-06T07:21:51.075317+0000","last_became_active":"2024-03-06T07:21:50.082217+0000","last_became_peered":"2024-03-06T07:21:50.082217+0000","last_unstale":"2024-03-06T07:21:51.075317+0000","last_undegraded":"2024-03-06T07:21:51.075317+0000","last_fullsized":"2024-03-06T07:21:51.075317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333683+0000","last_change":"2024-03-06T07:21:50.082417+0000","last_active":"2024-03-06T07:21:51.333683+0000","last_peered":"2024-03-06T07:21:51.333683+0000","last_clean":"2024-03-06T07:21:51.333683+0000","last_became_active":"2024-03-06T07:21:50.082162+0000","last_became_peered":"2024-03-06T07:21:50.082162+0000","last_unstale":"2024-03-06T07:21:51.333683+0000","last_undegraded":"2024-03-06T07:21:51.333683+0000","last_fullsized":"2024-03-06T07:21:51.333683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061987+0000","last_change":"2024-03-06T07:21:50.085327+0000","last_active":"2024-03-06T07:21:51.061987+0000","last_peered":"2024-03-06T07:21:51.061987+0000","last_clean":"2024-03-06T07:21:51.061987+0000","last_became_active":"2024-03-06T07:21:50.085019+0000","last_became_peered":"2024-03-06T07:21:50.085019+0000","last_unstale":"2024-03-06T07:21:51.061987+0000","last_undegraded":"2024-03-06T07:21:51.061987+0000","last_fullsized":"2024-03-06T07:21:51.061987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:53.083875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333521+0000","last_change":"2024-03-06T07:21:50.080803+0000","last_active":"2024-03-06T07:21:51.333521+0000","last_peered":"2024-03-06T07:21:51.333521+0000","last_clean":"2024-03-06T07:21:51.333521+0000","last_became_active":"2024-03-06T07:21:50.080460+0000","last_became_peered":"2024-03-06T07:21:50.080460+0000","last_unstale":"2024-03-06T07:21:51.333521+0000","last_undegraded":"2024-03-06T07:21:51.333521+0000","last_fullsized":"2024-03-06T07:21:51.333521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060865+0000","last_change":"2024-03-06T07:21:50.071650+0000","last_active":"2024-03-06T07:21:51.060865+0000","last_peered":"2024-03-06T07:21:51.060865+0000","last_clean":"2024-03-06T07:21:51.060865+0000","last_became_active":"2024-03-06T07:21:50.071383+0000","last_became_peered":"2024-03-06T07:21:50.071383+0000","last_unstale":"2024-03-06T07:21:51.060865+0000","last_undegraded":"2024-03-06T07:21:51.060865+0000","last_fullsized":"2024-03-06T07:21:51.060865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333966+0000","last_change":"2024-03-06T07:21:50.077589+0000","last_active":"2024-03-06T07:21:51.333966+0000","last_peered":"2024-03-06T07:21:51.333966+0000","last_clean":"2024-03-06T07:21:51.333966+0000","last_became_active":"2024-03-06T07:21:50.077302+0000","last_became_peered":"2024-03-06T07:21:50.077302+0000","last_unstale":"2024-03-06T07:21:51.333966+0000","last_undegraded":"2024-03-06T07:21:51.333966+0000","last_fullsized":"2024-03-06T07:21:51.333966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075478+0000","last_change":"2024-03-06T07:21:50.085031+0000","last_active":"2024-03-06T07:21:51.075478+0000","last_peered":"2024-03-06T07:21:51.075478+0000","last_clean":"2024-03-06T07:21:51.075478+0000","last_became_active":"2024-03-06T07:21:50.084758+0000","last_became_peered":"2024-03-06T07:21:50.084758+0000","last_unstale":"2024-03-06T07:21:51.075478+0000","last_undegraded":"2024-03-06T07:21:51.075478+0000","last_fullsized":"2024-03-06T07:21:51.075478+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059511+0000","last_change":"2024-03-06T07:21:50.072164+0000","last_active":"2024-03-06T07:21:51.059511+0000","last_peered":"2024-03-06T07:21:51.059511+0000","last_clean":"2024-03-06T07:21:51.059511+0000","last_became_active":"2024-03-06T07:21:50.071795+0000","last_became_peered":"2024-03-06T07:21:50.071795+0000","last_unstale":"2024-03-06T07:21:51.059511+0000","last_undegraded":"2024-03-06T07:21:51.059511+0000","last_fullsized":"2024-03-06T07:21:51.059511+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334595+0000","last_change":"2024-03-06T07:21:50.078217+0000","last_active":"2024-03-06T07:21:51.334595+0000","last_peered":"2024-03-06T07:21:51.334595+0000","last_clean":"2024-03-06T07:21:51.334595+0000","last_became_active":"2024-03-06T07:21:50.077809+0000","last_became_peered":"2024-03-06T07:21:50.077809+0000","last_unstale":"2024-03-06T07:21:51.334595+0000","last_undegraded":"2024-03-06T07:21:51.334595+0000","last_fullsized":"2024-03-06T07:21:51.334595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333826+0000","last_change":"2024-03-06T07:21:50.065564+0000","last_active":"2024-03-06T07:21:51.333826+0000","last_peered":"2024-03-06T07:21:51.333826+0000","last_clean":"2024-03-06T07:21:51.333826+0000","last_became_active":"2024-03-06T07:21:50.065294+0000","last_became_peered":"2024-03-06T07:21:50.065294+0000","last_unstale":"2024-03-06T07:21:51.333826+0000","last_undegraded":"2024-03-06T07:21:51.333826+0000","last_fullsized":"2024-03-06T07:21:51.333826+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:25:19.229992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062062+0000","last_change":"2024-03-06T07:21:50.073038+0000","last_active":"2024-03-06T07:21:51.062062+0000","last_peered":"2024-03-06T07:21:51.062062+0000","last_clean":"2024-03-06T07:21:51.062062+0000","last_became_active":"2024-03-06T07:21:50.072603+0000","last_became_peered":"2024-03-06T07:21:50.072603+0000","last_unstale":"2024-03-06T07:21:51.062062+0000","last_undegraded":"2024-03-06T07:21:51.062062+0000","last_fullsized":"2024-03-06T07:21:51.062062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334317+0000","last_change":"2024-03-06T07:21:50.069962+0000","last_active":"2024-03-06T07:21:51.334317+0000","last_peered":"2024-03-06T07:21:51.334317+0000","last_clean":"2024-03-06T07:21:51.334317+0000","last_became_active":"2024-03-06T07:21:50.069700+0000","last_became_peered":"2024-03-06T07:21:50.069700+0000","last_unstale":"2024-03-06T07:21:51.334317+0000","last_undegraded":"2024-03-06T07:21:51.334317+0000","last_fullsized":"2024-03-06T07:21:51.334317+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075081+0000","last_change":"2024-03-06T07:21:50.077457+0000","last_active":"2024-03-06T07:21:51.075081+0000","last_peered":"2024-03-06T07:21:51.075081+0000","last_clean":"2024-03-06T07:21:51.075081+0000","last_became_active":"2024-03-06T07:21:50.077173+0000","last_became_peered":"2024-03-06T07:21:50.077173+0000","last_unstale":"2024-03-06T07:21:51.075081+0000","last_undegraded":"2024-03-06T07:21:51.075081+0000","last_fullsized":"2024-03-06T07:21:51.075081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:24.760256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333933+0000","last_change":"2024-03-06T07:21:50.075503+0000","last_active":"2024-03-06T07:21:51.333933+0000","last_peered":"2024-03-06T07:21:51.333933+0000","last_clean":"2024-03-06T07:21:51.333933+0000","last_became_active":"2024-03-06T07:21:50.075247+0000","last_became_peered":"2024-03-06T07:21:50.075247+0000","last_unstale":"2024-03-06T07:21:51.333933+0000","last_undegraded":"2024-03-06T07:21:51.333933+0000","last_fullsized":"2024-03-06T07:21:51.333933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:48.693407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333798+0000","last_change":"2024-03-06T07:21:49.073461+0000","last_active":"2024-03-06T07:21:51.333798+0000","last_peered":"2024-03-06T07:21:51.333798+0000","last_clean":"2024-03-06T07:21:51.333798+0000","last_became_active":"2024-03-06T07:21:49.073212+0000","last_became_peered":"2024-03-06T07:21:49.073212+0000","last_unstale":"2024-03-06T07:21:51.333798+0000","last_undegraded":"2024-03-06T07:21:51.333798+0000","last_fullsized":"2024-03-06T07:21:51.333798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333964+0000","last_change":"2024-03-06T07:21:50.083293+0000","last_active":"2024-03-06T07:21:51.333964+0000","last_peered":"2024-03-06T07:21:51.333964+0000","last_clean":"2024-03-06T07:21:51.333964+0000","last_became_active":"2024-03-06T07:21:50.082931+0000","last_became_peered":"2024-03-06T07:21:50.082931+0000","last_unstale":"2024-03-06T07:21:51.333964+0000","last_undegraded":"2024-03-06T07:21:51.333964+0000","last_fullsized":"2024-03-06T07:21:51.333964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075166+0000","last_change":"2024-03-06T07:21:49.072875+0000","last_active":"2024-03-06T07:21:51.075166+0000","last_peered":"2024-03-06T07:21:51.075166+0000","last_clean":"2024-03-06T07:21:51.075166+0000","last_became_active":"2024-03-06T07:21:49.071548+0000","last_became_peered":"2024-03-06T07:21:49.071548+0000","last_unstale":"2024-03-06T07:21:51.075166+0000","last_undegraded":"2024-03-06T07:21:51.075166+0000","last_fullsized":"2024-03-06T07:21:51.075166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333606+0000","last_change":"2024-03-06T07:21:49.075539+0000","last_active":"2024-03-06T07:21:51.333606+0000","last_peered":"2024-03-06T07:21:51.333606+0000","last_clean":"2024-03-06T07:21:51.333606+0000","last_became_active":"2024-03-06T07:21:49.075192+0000","last_became_peered":"2024-03-06T07:21:49.075192+0000","last_unstale":"2024-03-06T07:21:51.333606+0000","last_undegraded":"2024-03-06T07:21:51.333606+0000","last_fullsized":"2024-03-06T07:21:51.333606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334442+0000","last_change":"2024-03-06T07:21:50.078018+0000","last_active":"2024-03-06T07:21:51.334442+0000","last_peered":"2024-03-06T07:21:51.334442+0000","last_clean":"2024-03-06T07:21:51.334442+0000","last_became_active":"2024-03-06T07:21:50.077638+0000","last_became_peered":"2024-03-06T07:21:50.077638+0000","last_unstale":"2024-03-06T07:21:51.334442+0000","last_undegraded":"2024-03-06T07:21:51.334442+0000","last_fullsized":"2024-03-06T07:21:51.334442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:36:50.686203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060973+0000","last_change":"2024-03-06T07:21:49.070076+0000","last_active":"2024-03-06T07:21:51.060973+0000","last_peered":"2024-03-06T07:21:51.060973+0000","last_clean":"2024-03-06T07:21:51.060973+0000","last_became_active":"2024-03-06T07:21:49.069491+0000","last_became_peered":"2024-03-06T07:21:49.069491+0000","last_unstale":"2024-03-06T07:21:51.060973+0000","last_undegraded":"2024-03-06T07:21:51.060973+0000","last_fullsized":"2024-03-06T07:21:51.060973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059076+0000","last_change":"2024-03-06T07:21:50.080260+0000","last_active":"2024-03-06T07:21:51.059076+0000","last_peered":"2024-03-06T07:21:51.059076+0000","last_clean":"2024-03-06T07:21:51.059076+0000","last_became_active":"2024-03-06T07:21:50.079987+0000","last_became_peered":"2024-03-06T07:21:50.079987+0000","last_unstale":"2024-03-06T07:21:51.059076+0000","last_undegraded":"2024-03-06T07:21:51.059076+0000","last_fullsized":"2024-03-06T07:21:51.059076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075328+0000","last_change":"2024-03-06T07:21:49.071143+0000","last_active":"2024-03-06T07:21:51.075328+0000","last_peered":"2024-03-06T07:21:51.075328+0000","last_clean":"2024-03-06T07:21:51.075328+0000","last_became_active":"2024-03-06T07:21:49.070872+0000","last_became_peered":"2024-03-06T07:21:49.070872+0000","last_unstale":"2024-03-06T07:21:51.075328+0000","last_undegraded":"2024-03-06T07:21:51.075328+0000","last_fullsized":"2024-03-06T07:21:51.075328+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061550+0000","last_change":"2024-03-06T07:21:50.078307+0000","last_active":"2024-03-06T07:21:51.061550+0000","last_peered":"2024-03-06T07:21:51.061550+0000","last_clean":"2024-03-06T07:21:51.061550+0000","last_became_active":"2024-03-06T07:21:50.078025+0000","last_became_peered":"2024-03-06T07:21:50.078025+0000","last_unstale":"2024-03-06T07:21:51.061550+0000","last_undegraded":"2024-03-06T07:21:51.061550+0000","last_fullsized":"2024-03-06T07:21:51.061550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061672+0000","last_change":"2024-03-06T07:21:49.066235+0000","last_active":"2024-03-06T07:21:51.061672+0000","last_peered":"2024-03-06T07:21:51.061672+0000","last_clean":"2024-03-06T07:21:51.061672+0000","last_became_active":"2024-03-06T07:21:49.065968+0000","last_became_peered":"2024-03-06T07:21:49.065968+0000","last_unstale":"2024-03-06T07:21:51.061672+0000","last_undegraded":"2024-03-06T07:21:51.061672+0000","last_fullsized":"2024-03-06T07:21:51.061672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333774+0000","last_change":"2024-03-06T07:21:50.067265+0000","last_active":"2024-03-06T07:21:51.333774+0000","last_peered":"2024-03-06T07:21:51.333774+0000","last_clean":"2024-03-06T07:21:51.333774+0000","last_became_active":"2024-03-06T07:21:50.066634+0000","last_became_peered":"2024-03-06T07:21:50.066634+0000","last_unstale":"2024-03-06T07:21:51.333774+0000","last_undegraded":"2024-03-06T07:21:51.333774+0000","last_fullsized":"2024-03-06T07:21:51.333774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059350+0000","last_change":"2024-03-06T07:21:49.070187+0000","last_active":"2024-03-06T07:21:51.059350+0000","last_peered":"2024-03-06T07:21:51.059350+0000","last_clean":"2024-03-06T07:21:51.059350+0000","last_became_active":"2024-03-06T07:21:49.069894+0000","last_became_peered":"2024-03-06T07:21:49.069894+0000","last_unstale":"2024-03-06T07:21:51.059350+0000","last_undegraded":"2024-03-06T07:21:51.059350+0000","last_fullsized":"2024-03-06T07:21:51.059350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334133+0000","last_change":"2024-03-06T07:21:50.086859+0000","last_active":"2024-03-06T07:21:51.334133+0000","last_peered":"2024-03-06T07:21:51.334133+0000","last_clean":"2024-03-06T07:21:51.334133+0000","last_became_active":"2024-03-06T07:21:50.085246+0000","last_became_peered":"2024-03-06T07:21:50.085246+0000","last_unstale":"2024-03-06T07:21:51.334133+0000","last_undegraded":"2024-03-06T07:21:51.334133+0000","last_fullsized":"2024-03-06T07:21:51.334133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:38.955510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075355+0000","last_change":"2024-03-06T07:21:49.074802+0000","last_active":"2024-03-06T07:21:51.075355+0000","last_peered":"2024-03-06T07:21:51.075355+0000","last_clean":"2024-03-06T07:21:51.075355+0000","last_became_active":"2024-03-06T07:21:49.074143+0000","last_became_peered":"2024-03-06T07:21:49.074143+0000","last_unstale":"2024-03-06T07:21:51.075355+0000","last_undegraded":"2024-03-06T07:21:51.075355+0000","last_fullsized":"2024-03-06T07:21:51.075355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061356+0000","last_change":"2024-03-06T07:21:50.083325+0000","last_active":"2024-03-06T07:21:51.061356+0000","last_peered":"2024-03-06T07:21:51.061356+0000","last_clean":"2024-03-06T07:21:51.061356+0000","last_became_active":"2024-03-06T07:21:50.082675+0000","last_became_peered":"2024-03-06T07:21:50.082675+0000","last_unstale":"2024-03-06T07:21:51.061356+0000","last_undegraded":"2024-03-06T07:21:51.061356+0000","last_fullsized":"2024-03-06T07:21:51.061356+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:15.487738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334058+0000","last_change":"2024-03-06T07:21:49.066349+0000","last_active":"2024-03-06T07:21:51.334058+0000","last_peered":"2024-03-06T07:21:51.334058+0000","last_clean":"2024-03-06T07:21:51.334058+0000","last_became_active":"2024-03-06T07:21:49.066024+0000","last_became_peered":"2024-03-06T07:21:49.066024+0000","last_unstale":"2024-03-06T07:21:51.334058+0000","last_undegraded":"2024-03-06T07:21:51.334058+0000","last_fullsized":"2024-03-06T07:21:51.334058+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:48.491649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333454+0000","last_change":"2024-03-06T07:21:50.082035+0000","last_active":"2024-03-06T07:21:51.333454+0000","last_peered":"2024-03-06T07:21:51.333454+0000","last_clean":"2024-03-06T07:21:51.333454+0000","last_became_active":"2024-03-06T07:21:50.081731+0000","last_became_peered":"2024-03-06T07:21:50.081731+0000","last_unstale":"2024-03-06T07:21:51.333454+0000","last_undegraded":"2024-03-06T07:21:51.333454+0000","last_fullsized":"2024-03-06T07:21:51.333454+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:57.652112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062117+0000","last_change":"2024-03-06T07:21:49.074426+0000","last_active":"2024-03-06T07:21:51.062117+0000","last_peered":"2024-03-06T07:21:51.062117+0000","last_clean":"2024-03-06T07:21:51.062117+0000","last_became_active":"2024-03-06T07:21:49.074117+0000","last_became_peered":"2024-03-06T07:21:49.074117+0000","last_unstale":"2024-03-06T07:21:51.062117+0000","last_undegraded":"2024-03-06T07:21:51.062117+0000","last_fullsized":"2024-03-06T07:21:51.062117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061727+0000","last_change":"2024-03-06T07:21:50.082545+0000","last_active":"2024-03-06T07:21:51.061727+0000","last_peered":"2024-03-06T07:21:51.061727+0000","last_clean":"2024-03-06T07:21:51.061727+0000","last_became_active":"2024-03-06T07:21:50.082197+0000","last_became_peered":"2024-03-06T07:21:50.082197+0000","last_unstale":"2024-03-06T07:21:51.061727+0000","last_undegraded":"2024-03-06T07:21:51.061727+0000","last_fullsized":"2024-03-06T07:21:51.061727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:33:57.476299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075070+0000","last_change":"2024-03-06T07:21:49.073223+0000","last_active":"2024-03-06T07:21:51.075070+0000","last_peered":"2024-03-06T07:21:51.075070+0000","last_clean":"2024-03-06T07:21:51.075070+0000","last_became_active":"2024-03-06T07:21:49.071609+0000","last_became_peered":"2024-03-06T07:21:49.071609+0000","last_unstale":"2024-03-06T07:21:51.075070+0000","last_undegraded":"2024-03-06T07:21:51.075070+0000","last_fullsized":"2024-03-06T07:21:51.075070+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075085+0000","last_change":"2024-03-06T07:21:50.077209+0000","last_active":"2024-03-06T07:21:51.075085+0000","last_peered":"2024-03-06T07:21:51.075085+0000","last_clean":"2024-03-06T07:21:51.075085+0000","last_became_active":"2024-03-06T07:21:50.076936+0000","last_became_peered":"2024-03-06T07:21:50.076936+0000","last_unstale":"2024-03-06T07:21:51.075085+0000","last_undegraded":"2024-03-06T07:21:51.075085+0000","last_fullsized":"2024-03-06T07:21:51.075085+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334123+0000","last_change":"2024-03-06T07:21:49.072833+0000","last_active":"2024-03-06T07:21:51.334123+0000","last_peered":"2024-03-06T07:21:51.334123+0000","last_clean":"2024-03-06T07:21:51.334123+0000","last_became_active":"2024-03-06T07:21:49.072543+0000","last_became_peered":"2024-03-06T07:21:49.072543+0000","last_unstale":"2024-03-06T07:21:51.334123+0000","last_undegraded":"2024-03-06T07:21:51.334123+0000","last_fullsized":"2024-03-06T07:21:51.334123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060781+0000","last_change":"2024-03-06T07:21:50.081698+0000","last_active":"2024-03-06T07:21:51.060781+0000","last_peered":"2024-03-06T07:21:51.060781+0000","last_clean":"2024-03-06T07:21:51.060781+0000","last_became_active":"2024-03-06T07:21:50.081294+0000","last_became_peered":"2024-03-06T07:21:50.081294+0000","last_unstale":"2024-03-06T07:21:51.060781+0000","last_undegraded":"2024-03-06T07:21:51.060781+0000","last_fullsized":"2024-03-06T07:21:51.060781+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334204+0000","last_change":"2024-03-06T07:21:49.069462+0000","last_active":"2024-03-06T07:21:51.334204+0000","last_peered":"2024-03-06T07:21:51.334204+0000","last_clean":"2024-03-06T07:21:51.334204+0000","last_became_active":"2024-03-06T07:21:49.069161+0000","last_became_peered":"2024-03-06T07:21:49.069161+0000","last_unstale":"2024-03-06T07:21:51.334204+0000","last_undegraded":"2024-03-06T07:21:51.334204+0000","last_fullsized":"2024-03-06T07:21:51.334204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:48.201301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061018+0000","last_change":"2024-03-06T07:21:50.082841+0000","last_active":"2024-03-06T07:21:51.061018+0000","last_peered":"2024-03-06T07:21:51.061018+0000","last_clean":"2024-03-06T07:21:51.061018+0000","last_became_active":"2024-03-06T07:21:50.082543+0000","last_became_peered":"2024-03-06T07:21:50.082543+0000","last_unstale":"2024-03-06T07:21:51.061018+0000","last_undegraded":"2024-03-06T07:21:51.061018+0000","last_fullsized":"2024-03-06T07:21:51.061018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334727+0000","last_change":"2024-03-06T07:21:49.070603+0000","last_active":"2024-03-06T07:21:51.334727+0000","last_peered":"2024-03-06T07:21:51.334727+0000","last_clean":"2024-03-06T07:21:51.334727+0000","last_became_active":"2024-03-06T07:21:49.070341+0000","last_became_peered":"2024-03-06T07:21:49.070341+0000","last_unstale":"2024-03-06T07:21:51.334727+0000","last_undegraded":"2024-03-06T07:21:51.334727+0000","last_fullsized":"2024-03-06T07:21:51.334727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061230+0000","last_change":"2024-03-06T07:21:50.082050+0000","last_active":"2024-03-06T07:21:51.061230+0000","last_peered":"2024-03-06T07:21:51.061230+0000","last_clean":"2024-03-06T07:21:51.061230+0000","last_became_active":"2024-03-06T07:21:50.081723+0000","last_became_peered":"2024-03-06T07:21:50.081723+0000","last_unstale":"2024-03-06T07:21:51.061230+0000","last_undegraded":"2024-03-06T07:21:51.061230+0000","last_fullsized":"2024-03-06T07:21:51.061230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:01:57.553650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059449+0000","last_change":"2024-03-06T07:21:49.069397+0000","last_active":"2024-03-06T07:21:51.059449+0000","last_peered":"2024-03-06T07:21:51.059449+0000","last_clean":"2024-03-06T07:21:51.059449+0000","last_became_active":"2024-03-06T07:21:49.069101+0000","last_became_peered":"2024-03-06T07:21:49.069101+0000","last_unstale":"2024-03-06T07:21:51.059449+0000","last_undegraded":"2024-03-06T07:21:51.059449+0000","last_fullsized":"2024-03-06T07:21:51.059449+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061985+0000","last_change":"2024-03-06T07:21:50.072534+0000","last_active":"2024-03-06T07:21:51.061985+0000","last_peered":"2024-03-06T07:21:51.061985+0000","last_clean":"2024-03-06T07:21:51.061985+0000","last_became_active":"2024-03-06T07:21:50.072252+0000","last_became_peered":"2024-03-06T07:21:50.072252+0000","last_unstale":"2024-03-06T07:21:51.061985+0000","last_undegraded":"2024-03-06T07:21:51.061985+0000","last_fullsized":"2024-03-06T07:21:51.061985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:15:19.505319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334565+0000","last_change":"2024-03-06T07:21:49.071113+0000","last_active":"2024-03-06T07:21:51.334565+0000","last_peered":"2024-03-06T07:21:51.334565+0000","last_clean":"2024-03-06T07:21:51.334565+0000","last_became_active":"2024-03-06T07:21:49.070854+0000","last_became_peered":"2024-03-06T07:21:49.070854+0000","last_unstale":"2024-03-06T07:21:51.334565+0000","last_undegraded":"2024-03-06T07:21:51.334565+0000","last_fullsized":"2024-03-06T07:21:51.334565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061519+0000","last_change":"2024-03-06T07:21:50.087871+0000","last_active":"2024-03-06T07:21:51.061519+0000","last_peered":"2024-03-06T07:21:51.061519+0000","last_clean":"2024-03-06T07:21:51.061519+0000","last_became_active":"2024-03-06T07:21:50.087545+0000","last_became_peered":"2024-03-06T07:21:50.087545+0000","last_unstale":"2024-03-06T07:21:51.061519+0000","last_undegraded":"2024-03-06T07:21:51.061519+0000","last_fullsized":"2024-03-06T07:21:51.061519+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:46.267182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:21:51.061213+0000","last_peered":"2024-03-06T07:21:51.061213+0000","last_clean":"2024-03-06T07:21:51.061213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:21:51.061213+0000","last_undegraded":"2024-03-06T07:21:51.061213+0000","last_fullsized":"2024-03-06T07:21:51.061213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061289+0000","last_change":"2024-03-06T07:21:49.064484+0000","last_active":"2024-03-06T07:21:51.061289+0000","last_peered":"2024-03-06T07:21:51.061289+0000","last_clean":"2024-03-06T07:21:51.061289+0000","last_became_active":"2024-03-06T07:21:49.064202+0000","last_became_peered":"2024-03-06T07:21:49.064202+0000","last_unstale":"2024-03-06T07:21:51.061289+0000","last_undegraded":"2024-03-06T07:21:51.061289+0000","last_fullsized":"2024-03-06T07:21:51.061289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333973+0000","last_change":"2024-03-06T07:21:50.074427+0000","last_active":"2024-03-06T07:21:51.333973+0000","last_peered":"2024-03-06T07:21:51.333973+0000","last_clean":"2024-03-06T07:21:51.333973+0000","last_became_active":"2024-03-06T07:21:50.074131+0000","last_became_peered":"2024-03-06T07:21:50.074131+0000","last_unstale":"2024-03-06T07:21:51.333973+0000","last_undegraded":"2024-03-06T07:21:51.333973+0000","last_fullsized":"2024-03-06T07:21:51.333973+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061151+0000","last_change":"2024-03-06T07:21:49.075192+0000","last_active":"2024-03-06T07:21:51.061151+0000","last_peered":"2024-03-06T07:21:51.061151+0000","last_clean":"2024-03-06T07:21:51.061151+0000","last_became_active":"2024-03-06T07:21:49.074904+0000","last_became_peered":"2024-03-06T07:21:49.074904+0000","last_unstale":"2024-03-06T07:21:51.061151+0000","last_undegraded":"2024-03-06T07:21:51.061151+0000","last_fullsized":"2024-03-06T07:21:51.061151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:02:59.897349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333341+0000","last_change":"2024-03-06T07:21:50.074328+0000","last_active":"2024-03-06T07:21:51.333341+0000","last_peered":"2024-03-06T07:21:51.333341+0000","last_clean":"2024-03-06T07:21:51.333341+0000","last_became_active":"2024-03-06T07:21:50.074021+0000","last_became_peered":"2024-03-06T07:21:50.074021+0000","last_unstale":"2024-03-06T07:21:51.333341+0000","last_undegraded":"2024-03-06T07:21:51.333341+0000","last_fullsized":"2024-03-06T07:21:51.333341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:14.753301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334402+0000","last_change":"2024-03-06T07:21:49.068516+0000","last_active":"2024-03-06T07:21:51.334402+0000","last_peered":"2024-03-06T07:21:51.334402+0000","last_clean":"2024-03-06T07:21:51.334402+0000","last_became_active":"2024-03-06T07:21:49.068184+0000","last_became_peered":"2024-03-06T07:21:49.068184+0000","last_unstale":"2024-03-06T07:21:51.334402+0000","last_undegraded":"2024-03-06T07:21:51.334402+0000","last_fullsized":"2024-03-06T07:21:51.334402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059118+0000","last_change":"2024-03-06T07:21:50.071162+0000","last_active":"2024-03-06T07:21:51.059118+0000","last_peered":"2024-03-06T07:21:51.059118+0000","last_clean":"2024-03-06T07:21:51.059118+0000","last_became_active":"2024-03-06T07:21:50.070848+0000","last_became_peered":"2024-03-06T07:21:50.070848+0000","last_unstale":"2024-03-06T07:21:51.059118+0000","last_undegraded":"2024-03-06T07:21:51.059118+0000","last_fullsized":"2024-03-06T07:21:51.059118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062173+0000","last_change":"2024-03-06T07:21:49.071017+0000","last_active":"2024-03-06T07:21:51.062173+0000","last_peered":"2024-03-06T07:21:51.062173+0000","last_clean":"2024-03-06T07:21:51.062173+0000","last_became_active":"2024-03-06T07:21:49.070588+0000","last_became_peered":"2024-03-06T07:21:49.070588+0000","last_unstale":"2024-03-06T07:21:51.062173+0000","last_undegraded":"2024-03-06T07:21:51.062173+0000","last_fullsized":"2024-03-06T07:21:51.062173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334053+0000","last_change":"2024-03-06T07:21:50.082287+0000","last_active":"2024-03-06T07:21:51.334053+0000","last_peered":"2024-03-06T07:21:51.334053+0000","last_clean":"2024-03-06T07:21:51.334053+0000","last_became_active":"2024-03-06T07:21:50.081767+0000","last_became_peered":"2024-03-06T07:21:50.081767+0000","last_unstale":"2024-03-06T07:21:51.334053+0000","last_undegraded":"2024-03-06T07:21:51.334053+0000","last_fullsized":"2024-03-06T07:21:51.334053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:09.878872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059265+0000","last_change":"2024-03-06T07:21:49.069978+0000","last_active":"2024-03-06T07:21:51.059265+0000","last_peered":"2024-03-06T07:21:51.059265+0000","last_clean":"2024-03-06T07:21:51.059265+0000","last_became_active":"2024-03-06T07:21:49.069223+0000","last_became_peered":"2024-03-06T07:21:49.069223+0000","last_unstale":"2024-03-06T07:21:51.059265+0000","last_undegraded":"2024-03-06T07:21:51.059265+0000","last_fullsized":"2024-03-06T07:21:51.059265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:53.410339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.062012+0000","last_change":"2024-03-06T07:21:50.076147+0000","last_active":"2024-03-06T07:21:51.062012+0000","last_peered":"2024-03-06T07:21:51.062012+0000","last_clean":"2024-03-06T07:21:51.062012+0000","last_became_active":"2024-03-06T07:21:50.075840+0000","last_became_peered":"2024-03-06T07:21:50.075840+0000","last_unstale":"2024-03-06T07:21:51.062012+0000","last_undegraded":"2024-03-06T07:21:51.062012+0000","last_fullsized":"2024-03-06T07:21:51.062012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.819435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061682+0000","last_change":"2024-03-06T07:21:49.077962+0000","last_active":"2024-03-06T07:21:51.061682+0000","last_peered":"2024-03-06T07:21:51.061682+0000","last_clean":"2024-03-06T07:21:51.061682+0000","last_became_active":"2024-03-06T07:21:49.077633+0000","last_became_peered":"2024-03-06T07:21:49.077633+0000","last_unstale":"2024-03-06T07:21:51.061682+0000","last_undegraded":"2024-03-06T07:21:51.061682+0000","last_fullsized":"2024-03-06T07:21:51.061682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:13.747084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059377+0000","last_change":"2024-03-06T07:21:50.071233+0000","last_active":"2024-03-06T07:21:51.059377+0000","last_peered":"2024-03-06T07:21:51.059377+0000","last_clean":"2024-03-06T07:21:51.059377+0000","last_became_active":"2024-03-06T07:21:50.070925+0000","last_became_peered":"2024-03-06T07:21:50.070925+0000","last_unstale":"2024-03-06T07:21:51.059377+0000","last_undegraded":"2024-03-06T07:21:51.059377+0000","last_fullsized":"2024-03-06T07:21:51.059377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:59.737425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.074529+0000","last_change":"2024-03-06T07:21:49.075388+0000","last_active":"2024-03-06T07:21:51.074529+0000","last_peered":"2024-03-06T07:21:51.074529+0000","last_clean":"2024-03-06T07:21:51.074529+0000","last_became_active":"2024-03-06T07:21:49.075115+0000","last_became_peered":"2024-03-06T07:21:49.075115+0000","last_unstale":"2024-03-06T07:21:51.074529+0000","last_undegraded":"2024-03-06T07:21:51.074529+0000","last_fullsized":"2024-03-06T07:21:51.074529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:59.252979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334111+0000","last_change":"2024-03-06T07:21:50.084419+0000","last_active":"2024-03-06T07:21:51.334111+0000","last_peered":"2024-03-06T07:21:51.334111+0000","last_clean":"2024-03-06T07:21:51.334111+0000","last_became_active":"2024-03-06T07:21:50.084022+0000","last_became_peered":"2024-03-06T07:21:50.084022+0000","last_unstale":"2024-03-06T07:21:51.334111+0000","last_undegraded":"2024-03-06T07:21:51.334111+0000","last_fullsized":"2024-03-06T07:21:51.334111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:48.078585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075351+0000","last_change":"2024-03-06T07:21:49.072877+0000","last_active":"2024-03-06T07:21:51.075351+0000","last_peered":"2024-03-06T07:21:51.075351+0000","last_clean":"2024-03-06T07:21:51.075351+0000","last_became_active":"2024-03-06T07:21:49.072559+0000","last_became_peered":"2024-03-06T07:21:49.072559+0000","last_unstale":"2024-03-06T07:21:51.075351+0000","last_undegraded":"2024-03-06T07:21:51.075351+0000","last_fullsized":"2024-03-06T07:21:51.075351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:58.423140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333756+0000","last_change":"2024-03-06T07:21:50.069013+0000","last_active":"2024-03-06T07:21:51.333756+0000","last_peered":"2024-03-06T07:21:51.333756+0000","last_clean":"2024-03-06T07:21:51.333756+0000","last_became_active":"2024-03-06T07:21:50.068753+0000","last_became_peered":"2024-03-06T07:21:50.068753+0000","last_unstale":"2024-03-06T07:21:51.333756+0000","last_undegraded":"2024-03-06T07:21:51.333756+0000","last_fullsized":"2024-03-06T07:21:51.333756+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:15.057992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.060433+0000","last_change":"2024-03-06T07:21:49.077101+0000","last_active":"2024-03-06T07:21:51.060433+0000","last_peered":"2024-03-06T07:21:51.060433+0000","last_clean":"2024-03-06T07:21:51.060433+0000","last_became_active":"2024-03-06T07:21:49.076900+0000","last_became_peered":"2024-03-06T07:21:49.076900+0000","last_unstale":"2024-03-06T07:21:51.060433+0000","last_undegraded":"2024-03-06T07:21:51.060433+0000","last_fullsized":"2024-03-06T07:21:51.060433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:26:37.949294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334046+0000","last_change":"2024-03-06T07:21:50.072349+0000","last_active":"2024-03-06T07:21:51.334046+0000","last_peered":"2024-03-06T07:21:51.334046+0000","last_clean":"2024-03-06T07:21:51.334046+0000","last_became_active":"2024-03-06T07:21:50.072052+0000","last_became_peered":"2024-03-06T07:21:50.072052+0000","last_unstale":"2024-03-06T07:21:51.334046+0000","last_undegraded":"2024-03-06T07:21:51.334046+0000","last_fullsized":"2024-03-06T07:21:51.334046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:59.429356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059318+0000","last_change":"2024-03-06T07:21:49.072417+0000","last_active":"2024-03-06T07:21:51.059318+0000","last_peered":"2024-03-06T07:21:51.059318+0000","last_clean":"2024-03-06T07:21:51.059318+0000","last_became_active":"2024-03-06T07:21:49.072153+0000","last_became_peered":"2024-03-06T07:21:49.072153+0000","last_unstale":"2024-03-06T07:21:51.059318+0000","last_undegraded":"2024-03-06T07:21:51.059318+0000","last_fullsized":"2024-03-06T07:21:51.059318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:35:08.872656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334590+0000","last_change":"2024-03-06T07:21:50.068165+0000","last_active":"2024-03-06T07:21:51.334590+0000","last_peered":"2024-03-06T07:21:51.334590+0000","last_clean":"2024-03-06T07:21:51.334590+0000","last_became_active":"2024-03-06T07:21:50.067278+0000","last_became_peered":"2024-03-06T07:21:50.067278+0000","last_unstale":"2024-03-06T07:21:51.334590+0000","last_undegraded":"2024-03-06T07:21:51.334590+0000","last_fullsized":"2024-03-06T07:21:51.334590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.059227+0000","last_change":"2024-03-06T07:21:49.074974+0000","last_active":"2024-03-06T07:21:51.059227+0000","last_peered":"2024-03-06T07:21:51.059227+0000","last_clean":"2024-03-06T07:21:51.059227+0000","last_became_active":"2024-03-06T07:21:49.074697+0000","last_became_peered":"2024-03-06T07:21:49.074697+0000","last_unstale":"2024-03-06T07:21:51.059227+0000","last_undegraded":"2024-03-06T07:21:51.059227+0000","last_fullsized":"2024-03-06T07:21:51.059227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:47.687191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333690+0000","last_change":"2024-03-06T07:21:50.078075+0000","last_active":"2024-03-06T07:21:51.333690+0000","last_peered":"2024-03-06T07:21:51.333690+0000","last_clean":"2024-03-06T07:21:51.333690+0000","last_became_active":"2024-03-06T07:21:50.077794+0000","last_became_peered":"2024-03-06T07:21:50.077794+0000","last_unstale":"2024-03-06T07:21:51.333690+0000","last_undegraded":"2024-03-06T07:21:51.333690+0000","last_fullsized":"2024-03-06T07:21:51.333690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:47:05.591617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061819+0000","last_change":"2024-03-06T07:21:49.073724+0000","last_active":"2024-03-06T07:21:51.061819+0000","last_peered":"2024-03-06T07:21:51.061819+0000","last_clean":"2024-03-06T07:21:51.061819+0000","last_became_active":"2024-03-06T07:21:49.073443+0000","last_became_peered":"2024-03-06T07:21:49.073443+0000","last_unstale":"2024-03-06T07:21:51.061819+0000","last_undegraded":"2024-03-06T07:21:51.061819+0000","last_fullsized":"2024-03-06T07:21:51.061819+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:47:56.645896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.075154+0000","last_change":"2024-03-06T07:21:50.081464+0000","last_active":"2024-03-06T07:21:51.075154+0000","last_peered":"2024-03-06T07:21:51.075154+0000","last_clean":"2024-03-06T07:21:51.075154+0000","last_became_active":"2024-03-06T07:21:50.081190+0000","last_became_peered":"2024-03-06T07:21:50.081190+0000","last_unstale":"2024-03-06T07:21:51.075154+0000","last_undegraded":"2024-03-06T07:21:51.075154+0000","last_fullsized":"2024-03-06T07:21:51.075154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:21:49.831201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061935+0000","last_change":"2024-03-06T07:21:50.074879+0000","last_active":"2024-03-06T07:21:51.061935+0000","last_peered":"2024-03-06T07:21:51.061935+0000","last_clean":"2024-03-06T07:21:51.061935+0000","last_became_active":"2024-03-06T07:21:50.074633+0000","last_became_peered":"2024-03-06T07:21:50.074633+0000","last_unstale":"2024-03-06T07:21:51.061935+0000","last_undegraded":"2024-03-06T07:21:51.061935+0000","last_fullsized":"2024-03-06T07:21:51.061935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:08:45.662017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061004+0000","last_change":"2024-03-06T07:21:49.078149+0000","last_active":"2024-03-06T07:21:51.061004+0000","last_peered":"2024-03-06T07:21:51.061004+0000","last_clean":"2024-03-06T07:21:51.061004+0000","last_became_active":"2024-03-06T07:21:49.077845+0000","last_became_peered":"2024-03-06T07:21:49.077845+0000","last_unstale":"2024-03-06T07:21:51.061004+0000","last_undegraded":"2024-03-06T07:21:51.061004+0000","last_fullsized":"2024-03-06T07:21:51.061004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:32:40.940490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061862+0000","last_change":"2024-03-06T07:21:50.073658+0000","last_active":"2024-03-06T07:21:51.061862+0000","last_peered":"2024-03-06T07:21:51.061862+0000","last_clean":"2024-03-06T07:21:51.061862+0000","last_became_active":"2024-03-06T07:21:50.073396+0000","last_became_peered":"2024-03-06T07:21:50.073396+0000","last_unstale":"2024-03-06T07:21:51.061862+0000","last_undegraded":"2024-03-06T07:21:51.061862+0000","last_fullsized":"2024-03-06T07:21:51.061862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:38:02.619077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.061590+0000","last_change":"2024-03-06T07:21:49.078569+0000","last_active":"2024-03-06T07:21:51.061590+0000","last_peered":"2024-03-06T07:21:51.061590+0000","last_clean":"2024-03-06T07:21:51.061590+0000","last_became_active":"2024-03-06T07:21:49.078274+0000","last_became_peered":"2024-03-06T07:21:49.078274+0000","last_unstale":"2024-03-06T07:21:51.061590+0000","last_undegraded":"2024-03-06T07:21:51.061590+0000","last_fullsized":"2024-03-06T07:21:51.061590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:23.662318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333637+0000","last_change":"2024-03-06T07:21:50.067267+0000","last_active":"2024-03-06T07:21:51.333637+0000","last_peered":"2024-03-06T07:21:51.333637+0000","last_clean":"2024-03-06T07:21:51.333637+0000","last_became_active":"2024-03-06T07:21:50.067014+0000","last_became_peered":"2024-03-06T07:21:50.067014+0000","last_unstale":"2024-03-06T07:21:51.333637+0000","last_undegraded":"2024-03-06T07:21:51.333637+0000","last_fullsized":"2024-03-06T07:21:51.333637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:26:19.313107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334431+0000","last_change":"2024-03-06T07:21:49.066664+0000","last_active":"2024-03-06T07:21:51.334431+0000","last_peered":"2024-03-06T07:21:51.334431+0000","last_clean":"2024-03-06T07:21:51.334431+0000","last_became_active":"2024-03-06T07:21:49.066350+0000","last_became_peered":"2024-03-06T07:21:49.066350+0000","last_unstale":"2024-03-06T07:21:51.334431+0000","last_undegraded":"2024-03-06T07:21:51.334431+0000","last_fullsized":"2024-03-06T07:21:51.334431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:11:21.442702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.334333+0000","last_change":"2024-03-06T07:21:50.079262+0000","last_active":"2024-03-06T07:21:51.334333+0000","last_peered":"2024-03-06T07:21:51.334333+0000","last_clean":"2024-03-06T07:21:51.334333+0000","last_became_active":"2024-03-06T07:21:50.078776+0000","last_became_peered":"2024-03-06T07:21:50.078776+0000","last_unstale":"2024-03-06T07:21:51.334333+0000","last_undegraded":"2024-03-06T07:21:51.334333+0000","last_fullsized":"2024-03-06T07:21:51.334333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:49.050098+0000","last_clean_scrub_stamp":"2024-03-06T07:21:49.050098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:22.333557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-06T07:21:51.333867+0000","last_change":"2024-03-06T07:21:49.070371+0000","last_active":"2024-03-06T07:21:51.333867+0000","last_peered":"2024-03-06T07:21:51.333867+0000","last_clean":"2024-03-06T07:21:51.333867+0000","last_became_active":"2024-03-06T07:21:49.070100+0000","last_became_peered":"2024-03-06T07:21:49.070100+0000","last_unstale":"2024-03-06T07:21:51.333867+0000","last_undegraded":"2024-03-06T07:21:51.333867+0000","last_fullsized":"2024-03-06T07:21:51.333867+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:48.043882+0000","last_clean_scrub_stamp":"2024-03-06T07:21:48.043882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:25:44.026321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:21:57.387 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:21:57.387 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-06T07:21:57.387 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-06T07:21:57.387 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-06T07:21:57.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.521+0000 7f51e0356640 1 Processor -- start 2024-03-06T07:21:57.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.521+0000 7f51e0356640 1 -- start start 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51e0356640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51d815bed0 0x7f51d80b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51e0356640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d80d04e0 0x7f51d80d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51e0356640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51d80d1fe0 0x7f51d815fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51e0356640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80d2470 con 0x7f51d80d1fe0 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51e0356640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d80b2d80 con 0x7f51d815bed0 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51e0356640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80ae9c0 con 0x7f51d80d04e0 2024-03-06T07:21:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51de8f3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51d80d1fe0 0x7f51d815fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51de8f3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51d80d1fe0 0x7f51d815fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54606/0 (socket says 172.21.15.72:54606) 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.522+0000 7f51de8f3640 1 -- 172.21.15.72:0/2263058970 learned_addr learned my addr 172.21.15.72:0/2263058970 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51dd8f1640 1 --2- 172.21.15.72:0/2263058970 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d80d04e0 0x7f51d80d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51de0f2640 1 --2- 172.21.15.72:0/2263058970 >> v2:172.21.15.146:3300/0 conn(0x7f51d815bed0 0x7f51d80b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51de8f3640 1 -- 172.21.15.72:0/2263058970 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d80d04e0 msgr2=0x7f51d80d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51de8f3640 1 --2- 172.21.15.72:0/2263058970 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d80d04e0 0x7f51d80d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51de8f3640 1 -- 172.21.15.72:0/2263058970 >> v2:172.21.15.146:3300/0 conn(0x7f51d815bed0 msgr2=0x7f51d80b2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51de8f3640 1 --2- 172.21.15.72:0/2263058970 >> v2:172.21.15.146:3300/0 conn(0x7f51d815bed0 0x7f51d80b2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.523+0000 7f51de8f3640 1 -- 172.21.15.72:0/2263058970 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80aeca0 con 0x7f51d80d1fe0 2024-03-06T07:21:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.524+0000 7f51de8f3640 1 --2- 172.21.15.72:0/2263058970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d1fe0 0x7f51d815fac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51d809fbe0 tx=0x7f51d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c0e43fdc9122bb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.524+0000 7f51dd0f0640 1 -- 172.21.15.72:0/2263058970 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d002e070 con 0x7f51d80d1fe0 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.524+0000 7f51dd0f0640 1 -- 172.21.15.72:0/2263058970 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0037070 con 0x7f51d80d1fe0 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.524+0000 7f51dd0f0640 1 -- 172.21.15.72:0/2263058970 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0031a00 con 0x7f51d80d1fe0 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.524+0000 7f51e0356640 1 -- 172.21.15.72:0/2263058970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d1fe0 msgr2=0x7f51d815fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.524+0000 7f51e0356640 1 --2- 172.21.15.72:0/2263058970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d1fe0 0x7f51d815fac0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51d809fbe0 tx=0x7f51d002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 -- 172.21.15.72:0/2263058970 shutdown_connections 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 --2- 172.21.15.72:0/2263058970 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d1fe0 0x7f51d815fac0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 --2- 172.21.15.72:0/2263058970 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d80d04e0 0x7f51d80d1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 --2- 172.21.15.72:0/2263058970 >> v2:172.21.15.146:3300/0 conn(0x7f51d815bed0 0x7f51d80b2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 -- 172.21.15.72:0/2263058970 >> 172.21.15.72:0/2263058970 conn(0x7f51d809a9f0 msgr2=0x7f51d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 -- 172.21.15.72:0/2263058970 shutdown_connections 2024-03-06T07:21:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.525+0000 7f51e0356640 1 -- 172.21.15.72:0/2263058970 wait complete. 2024-03-06T07:21:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 Processor -- start 2024-03-06T07:21:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 -- start start 2024-03-06T07:21:57.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 0x7f51d80c95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d80d1fe0 0x7f51d806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d815bed0 0x7f51d8150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d815c2b0 con 0x7f51d80d04e0 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80af260 con 0x7f51d80d1fe0 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51e0356640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d80c9c00 con 0x7f51d815bed0 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.526+0000 7f51de0f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 0x7f51d80c95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 0x7f51d80c95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54616/0 (socket says 172.21.15.72:54616) 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 -- 172.21.15.72:0/4093950093 learned_addr learned my addr 172.21.15.72:0/4093950093 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51dd8f1640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d80d1fe0 0x7f51d806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de8f3640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d815bed0 0x7f51d8150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 -- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d815bed0 msgr2=0x7f51d8150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d815bed0 0x7f51d8150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 -- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d80d1fe0 msgr2=0x7f51d806d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d80d1fe0 0x7f51d806d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 -- 172.21.15.72:0/4093950093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d80c9ee0 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51de0f2640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 0x7f51d80c95d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f51d4007e40 tx=0x7f51d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4014620 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d4019070 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d80ca1c0 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d4014a00 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.527+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d815db00 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.528+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d804f330 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.529+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51d4014ba0 con 0x7f51d80d04e0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.529+0000 7f51ceffd640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51bc06b4d0 0x7f51bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.530+0000 7f51dd8f1640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51bc06b4d0 0x7f51bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.530+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51d408a9a0 con 0x7f51d80d04e0 2024-03-06T07:21:57.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.530+0000 7f51dd8f1640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51bc06b4d0 0x7f51bc06d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51d806c180 tx=0x7f51c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.534+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51d40591d0 con 0x7f51d80d04e0 2024-03-06T07:21:57.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.746+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f51d80d28b0 con 0x7f51d80d04e0 2024-03-06T07:21:57.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.747+0000 7f51ceffd640 1 -- 172.21.15.72:0/4093950093 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f51d405d270 con 0x7f51d80d04e0 2024-03-06T07:21:57.749 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:57.749 INFO:teuthology.orchestra.run.smithi072.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51bc06b4d0 msgr2=0x7f51bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51bc06b4d0 0x7f51bc06d990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51d806c180 tx=0x7f51c8008040 comp rx=0 tx=0).stop 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 msgr2=0x7f51d80c95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 0x7f51d80c95d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f51d4007e40 tx=0x7f51d400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 shutdown_connections 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51bc06b4d0 0x7f51bc06d990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51d815bed0 0x7f51d8150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51d80d1fe0 0x7f51d806d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.748+0000 7f51e0356640 1 --2- 172.21.15.72:0/4093950093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51d80d04e0 0x7f51d80c95d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.749+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 >> 172.21.15.72:0/4093950093 conn(0x7f51d809a9f0 msgr2=0x7f51d80b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.749+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 shutdown_connections 2024-03-06T07:21:57.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.749+0000 7f51e0356640 1 -- 172.21.15.72:0/4093950093 wait complete. 2024-03-06T07:21:57.761 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-06T07:21:57.762 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:57.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.861+0000 7f021678c640 1 Processor -- start 2024-03-06T07:21:57.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.861+0000 7f021678c640 1 -- start start 2024-03-06T07:21:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f021678c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02100cfff0 0x7f02100d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f021678c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02100cc8b0 0x7f02100ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f021678c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0210161980 con 0x7f021015bb20 2024-03-06T07:21:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f021678c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02100d2500 con 0x7f02100cfff0 2024-03-06T07:21:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f021678c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0210161c60 con 0x7f02100cc8b0 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f020f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02100cfff0 0x7f02100d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f020f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02100cfff0 0x7f02100d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54620/0 (socket says 172.21.15.72:54620) 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f020f7fe640 1 -- 172.21.15.72:0/2131029739 learned_addr learned my addr 172.21.15.72:0/2131029739 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f020ffff640 1 --1- 172.21.15.72:0/2131029739 >> v1:172.21.15.72:6789/0 conn(0x7f021015bb20 0x7f02100d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49224/0 (socket says 172.21.15.72:49224) 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.862+0000 7f0214d29640 1 --2- 172.21.15.72:0/2131029739 >> v2:172.21.15.146:3300/0 conn(0x7f02100cc8b0 0x7f02100ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020f7fe640 1 -- 172.21.15.72:0/2131029739 >> v2:172.21.15.146:3300/0 conn(0x7f02100cc8b0 msgr2=0x7f02100ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020f7fe640 1 --2- 172.21.15.72:0/2131029739 >> v2:172.21.15.146:3300/0 conn(0x7f02100cc8b0 0x7f02100ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020f7fe640 1 -- 172.21.15.72:0/2131029739 >> v1:172.21.15.72:6789/0 conn(0x7f021015bb20 legacy=0x7f02100d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020f7fe640 1 -- 172.21.15.72:0/2131029739 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210161f40 con 0x7f02100cfff0 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020effd640 1 -- 172.21.15.72:0/2131029739 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103811891 0 0) 0x7f0210161980 con 0x7f021015bb20 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f0214d29640 1 --2- 172.21.15.72:0/2131029739 >> v2:172.21.15.146:3300/0 conn(0x7f02100cc8b0 0x7f02100ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020f7fe640 1 --2- 172.21.15.72:0/2131029739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100d04a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02101618c0 tx=0x7f020002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e409ac9ade22b034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020effd640 1 -- 172.21.15.72:0/2131029739 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200034070 con 0x7f02100cfff0 2024-03-06T07:21:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020effd640 1 -- 172.21.15.72:0/2131029739 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200030860 con 0x7f02100cfff0 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.863+0000 7f020effd640 1 -- 172.21.15.72:0/2131029739 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200030c30 con 0x7f02100cfff0 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 -- 172.21.15.72:0/2131029739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 msgr2=0x7f02100d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 --2- 172.21.15.72:0/2131029739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100d04a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02101618c0 tx=0x7f020002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 -- 172.21.15.72:0/2131029739 shutdown_connections 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 --2- 172.21.15.72:0/2131029739 >> v2:172.21.15.146:3300/0 conn(0x7f02100cc8b0 0x7f02100ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 --2- 172.21.15.72:0/2131029739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100d04a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 -- 172.21.15.72:0/2131029739 >> 172.21.15.72:0/2131029739 conn(0x7f021009a9f0 msgr2=0x7f021009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 -- 172.21.15.72:0/2131029739 shutdown_connections 2024-03-06T07:21:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.864+0000 7f021678c640 1 -- 172.21.15.72:0/2131029739 wait complete. 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.865+0000 7f021678c640 1 Processor -- start 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.865+0000 7f021678c640 1 -- start start 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f021678c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02100cc8b0 0x7f021014f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f021678c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f021678c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021015bb20 0x7f0210160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f021678c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02100d1980 con 0x7f02100cfff0 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f021678c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0210162500 con 0x7f02100cc8b0 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f021678c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02100d1b00 con 0x7f021015bb20 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54624/0 (socket says 172.21.15.72:54624) 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 -- 172.21.15.72:0/4269711491 learned_addr learned my addr 172.21.15.72:0/4269711491 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 -- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021015bb20 msgr2=0x7f0210160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f0214d29640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021015bb20 0x7f0210160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021015bb20 0x7f0210160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 -- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02100cc8b0 msgr2=0x7f021014f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02100cc8b0 0x7f021014f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.866+0000 7f020f7fe640 1 -- 172.21.15.72:0/4269711491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021015f060 con 0x7f02100cfff0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.867+0000 7f020f7fe640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100c7e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f020002b9e0 tx=0x7f02000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.867+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200034040 con 0x7f02100cfff0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.867+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200039d90 con 0x7f02100cfff0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.867+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021015f340 con 0x7f02100cfff0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.867+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200043610 con 0x7f02100cfff0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.867+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021015f900 con 0x7f02100cfff0 2024-03-06T07:21:57.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.868+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021004f330 con 0x7f02100cfff0 2024-03-06T07:21:57.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.868+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0200037070 con 0x7f02100cfff0 2024-03-06T07:21:57.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.869+0000 7f020cff9640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01fc06b4d0 0x7f01fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:57.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.869+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02000ac2f0 con 0x7f02100cfff0 2024-03-06T07:21:57.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.869+0000 7f020ffff640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01fc06b4d0 0x7f01fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:57.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.869+0000 7f020ffff640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01fc06b4d0 0x7f01fc06d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0204007fd0 tx=0x7f0204007960 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:57.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:57.873+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f020003e070 con 0x7f02100cfff0 2024-03-06T07:21:58.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.052+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f021006e240 con 0x7f02100cfff0 2024-03-06T07:21:58.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.054+0000 7f020cff9640 1 -- 172.21.15.72:0/4269711491 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f020007b200 con 0x7f02100cfff0 2024-03-06T07:21:58.055 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:58.055 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01fc06b4d0 msgr2=0x7f01fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01fc06b4d0 0x7f01fc06d990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0204007fd0 tx=0x7f0204007960 comp rx=0 tx=0).stop 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 msgr2=0x7f02100c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100c7e80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f020002b9e0 tx=0x7f02000046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 shutdown_connections 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01fc06b4d0 0x7f01fc06d990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021015bb20 0x7f0210160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02100cfff0 0x7f02100c7e80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.055+0000 7f021678c640 1 --2- 172.21.15.72:0/4269711491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02100cc8b0 0x7f021014f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.056+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 >> 172.21.15.72:0/4269711491 conn(0x7f021009a9f0 msgr2=0x7f021006fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.056+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 shutdown_connections 2024-03-06T07:21:58.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.056+0000 7f021678c640 1 -- 172.21.15.72:0/4269711491 wait complete. 2024-03-06T07:21:58.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:58.069 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:58.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.173+0000 7fd49919a640 1 Processor -- start 2024-03-06T07:21:58.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.173+0000 7fd49919a640 1 -- start start 2024-03-06T07:21:58.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49919a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 0x7fd49406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49919a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd49406fb40 0x7fd4940cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49919a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4940cd4b0 con 0x7fd4940ccca0 2024-03-06T07:21:58.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49919a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd49406ff80 con 0x7fd49406fb40 2024-03-06T07:21:58.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49919a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4940cd790 con 0x7fd49415c050 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd492d9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 0x7fd49406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49259c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd49406fb40 0x7fd4940cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49359e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd4940ccca0 0x7fd4940cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:39022/0 (socket says 172.21.15.72:39022) 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd492d9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 0x7fd49406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33970/0 (socket says 172.21.15.72:33970) 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49259c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd49406fb40 0x7fd4940cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:42488/0 (socket says 172.21.15.72:42488) 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd492d9d640 1 -- 172.21.15.72:0/40852818 learned_addr learned my addr 172.21.15.72:0/40852818 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49259c640 1 -- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 msgr2=0x7fd49406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49259c640 1 --2- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 0x7fd49406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49259c640 1 -- 172.21.15.72:0/40852818 >> v1:172.21.15.146:6789/0 conn(0x7fd4940ccca0 legacy=0x7fd4940cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd49259c640 1 -- 172.21.15.72:0/40852818 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4940cda70 con 0x7fd49406fb40 2024-03-06T07:21:58.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.175+0000 7fd492d9d640 1 --2- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 0x7fd49406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.176+0000 7fd49259c640 1 --2- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49406fb40 0x7fd4940cc760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd4880097a0 tx=0x7fd48802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac85c5cdd2e655b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.176+0000 7fd491d9b640 1 -- 172.21.15.72:0/40852818 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4880315a0 con 0x7fd49406fb40 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.176+0000 7fd491d9b640 1 -- 172.21.15.72:0/40852818 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd488037070 con 0x7fd49406fb40 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.176+0000 7fd491d9b640 1 -- 172.21.15.72:0/40852818 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd488031960 con 0x7fd49406fb40 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 -- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49406fb40 msgr2=0x7fd4940cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 --2- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49406fb40 0x7fd4940cc760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd4880097a0 tx=0x7fd48802b840 comp rx=0 tx=0).stop 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 -- 172.21.15.72:0/40852818 shutdown_connections 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 --2- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49406fb40 0x7fd4940cc760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 --2- 172.21.15.72:0/40852818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd49415c050 0x7fd49406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 -- 172.21.15.72:0/40852818 >> 172.21.15.72:0/40852818 conn(0x7fd4940b3800 msgr2=0x7fd4940b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 -- 172.21.15.72:0/40852818 shutdown_connections 2024-03-06T07:21:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.177+0000 7fd49919a640 1 -- 172.21.15.72:0/40852818 wait complete. 2024-03-06T07:21:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.179+0000 7fd49919a640 1 Processor -- start 2024-03-06T07:21:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.179+0000 7fd49919a640 1 -- start start 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.180+0000 7fd49919a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 0x7fd49406cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.180+0000 7fd49919a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4940ccca0 0x7fd4940cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.180+0000 7fd49919a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49415c050 0x7fd4940b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.180+0000 7fd49919a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4940d01f0 con 0x7fd49406fb40 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.180+0000 7fd49919a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4940cd130 con 0x7fd49415c050 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.180+0000 7fd49919a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4940d0370 con 0x7fd4940ccca0 2024-03-06T07:21:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 0x7fd49406cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 0x7fd49406cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54626/0 (socket says 172.21.15.72:54626) 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 -- 172.21.15.72:0/1995953858 learned_addr learned my addr 172.21.15.72:0/1995953858 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd49259c640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4940ccca0 0x7fd4940cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd49359e640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49415c050 0x7fd4940b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 -- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4940ccca0 msgr2=0x7fd4940cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4940ccca0 0x7fd4940cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 -- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49415c050 msgr2=0x7fd4940b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49415c050 0x7fd4940b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 -- 172.21.15.72:0/1995953858 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd49414a550 con 0x7fd49406fb40 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd492d9d640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 0x7fd49406cad0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd48400efc0 tx=0x7fd48400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484014480 con 0x7fd49406fb40 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd484019070 con 0x7fd49406fb40 2024-03-06T07:21:58.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd484014820 con 0x7fd49406fb40 2024-03-06T07:21:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49414a830 con 0x7fd49406fb40 2024-03-06T07:21:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.181+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd49414adf0 con 0x7fd49406fb40 2024-03-06T07:21:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.182+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd494068ff0 con 0x7fd49406fb40 2024-03-06T07:21:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.183+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd484005d10 con 0x7fd49406fb40 2024-03-06T07:21:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.183+0000 7fd47b7fe640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd48006b4d0 0x7fd48006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.183+0000 7fd49259c640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd48006b4d0 0x7fd48006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.183+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd48408a610 con 0x7fd49406fb40 2024-03-06T07:21:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.184+0000 7fd49259c640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd48006b4d0 0x7fd48006d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd488036000 tx=0x7fd48803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.188+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd484010040 con 0x7fd49406fb40 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.364+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4940d1500 con 0x7fd49406fb40 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.364+0000 7fd47b7fe640 1 -- 172.21.15.72:0/1995953858 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd484058e40 con 0x7fd49406fb40 2024-03-06T07:21:58.365 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:21:58.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd48006b4d0 msgr2=0x7fd48006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd48006b4d0 0x7fd48006d990 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd488036000 tx=0x7fd48803d040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 msgr2=0x7fd49406cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 0x7fd49406cad0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd48400efc0 tx=0x7fd48400c490 comp rx=0 tx=0).stop 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 shutdown_connections 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd48006b4d0 0x7fd48006d990 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd49415c050 0x7fd4940b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4940ccca0 0x7fd4940cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 --2- 172.21.15.72:0/1995953858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd49406fb40 0x7fd49406cad0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 >> 172.21.15.72:0/1995953858 conn(0x7fd4940b3800 msgr2=0x7fd4940cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 shutdown_connections 2024-03-06T07:21:58.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.366+0000 7fd49919a640 1 -- 172.21.15.72:0/1995953858 wait complete. 2024-03-06T07:21:58.378 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:21:58.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.480+0000 7f695553a640 1 Processor -- start 2024-03-06T07:21:58.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.480+0000 7f695553a640 1 -- start start 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f695553a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f695015c050 0x7f695006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f695553a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f695006fb40 0x7f69500cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f695553a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69500ccca0 0x7f69500cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f695553a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695006ffb0 con 0x7f695015c050 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f695553a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69500cd4b0 con 0x7f695006fb40 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f695553a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69500cd790 con 0x7f69500ccca0 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f695015c050 0x7f695006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f695015c050 0x7f695006f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54638/0 (socket says 172.21.15.72:54638) 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 -- 172.21.15.72:0/800614404 learned_addr learned my addr 172.21.15.72:0/800614404 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 -- 172.21.15.72:0/800614404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69500ccca0 msgr2=0x7f69500cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694e7fc640 1 --2- 172.21.15.72:0/800614404 >> v2:172.21.15.146:3300/0 conn(0x7f695006fb40 0x7f69500cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 --2- 172.21.15.72:0/800614404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69500ccca0 0x7f69500cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 -- 172.21.15.72:0/800614404 >> v2:172.21.15.146:3300/0 conn(0x7f695006fb40 msgr2=0x7f69500cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.481+0000 7f694effd640 1 --2- 172.21.15.72:0/800614404 >> v2:172.21.15.146:3300/0 conn(0x7f695006fb40 0x7f69500cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.482+0000 7f694effd640 1 -- 172.21.15.72:0/800614404 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69500cda70 con 0x7f695015c050 2024-03-06T07:21:58.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.482+0000 7f694effd640 1 --2- 172.21.15.72:0/800614404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006f600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f69500b7dc0 tx=0x7f694002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ca4ef7cae42000 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.482+0000 7f694dffb640 1 -- 172.21.15.72:0/800614404 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694002e070 con 0x7f695015c050 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.482+0000 7f694dffb640 1 -- 172.21.15.72:0/800614404 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940037070 con 0x7f695015c050 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.482+0000 7f694dffb640 1 -- 172.21.15.72:0/800614404 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940031a00 con 0x7f695015c050 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 -- 172.21.15.72:0/800614404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 msgr2=0x7f695006f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 --2- 172.21.15.72:0/800614404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006f600 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f69500b7dc0 tx=0x7f694002b880 comp rx=0 tx=0).stop 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 -- 172.21.15.72:0/800614404 shutdown_connections 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 --2- 172.21.15.72:0/800614404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69500ccca0 0x7f69500cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 --2- 172.21.15.72:0/800614404 >> v2:172.21.15.146:3300/0 conn(0x7f695006fb40 0x7f69500cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 --2- 172.21.15.72:0/800614404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006f600 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 -- 172.21.15.72:0/800614404 >> 172.21.15.72:0/800614404 conn(0x7f69500b3800 msgr2=0x7f69500b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 -- 172.21.15.72:0/800614404 shutdown_connections 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.483+0000 7f695553a640 1 -- 172.21.15.72:0/800614404 wait complete. 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.484+0000 7f695553a640 1 Processor -- start 2024-03-06T07:21:58.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 -- start start 2024-03-06T07:21:58.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006fb40 0x7f69501538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69500ccca0 0x7f69501f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69500d0020 con 0x7f695015c050 2024-03-06T07:21:58.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69500cd130 con 0x7f69500ccca0 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f695553a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69500d01a0 con 0x7f695006fb40 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f694f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f694f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54640/0 (socket says 172.21.15.72:54640) 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f694f7fe640 1 -- 172.21.15.72:0/3052838134 learned_addr learned my addr 172.21.15.72:0/3052838134 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f694effd640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006fb40 0x7f69501538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.485+0000 7f694e7fc640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69500ccca0 0x7f69501f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694f7fe640 1 -- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006fb40 msgr2=0x7f69501538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694f7fe640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006fb40 0x7f69501538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694f7fe640 1 -- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69500ccca0 msgr2=0x7f69501f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694f7fe640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69500ccca0 0x7f69501f3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694f7fe640 1 -- 172.21.15.72:0/3052838134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69501f4440 con 0x7f695015c050 2024-03-06T07:21:58.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694effd640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006fb40 0x7f69501538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f694f7fe640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006dbb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f694800b970 tx=0x7f694800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948019070 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694800cb60 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695006e0f0 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948014440 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.486+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69501f49a0 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.487+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695009fe60 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.487+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f694801c070 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.488+0000 7f69477fe640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693006b500 0x7f693006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.488+0000 7f694effd640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693006b500 0x7f693006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.488+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6948089e70 con 0x7f695015c050 2024-03-06T07:21:58.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.488+0000 7f694effd640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693006b500 0x7f693006d9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6940036000 tx=0x7f694003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.492+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f695006e0f0 con 0x7f695015c050 2024-03-06T07:21:58.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.791+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6950068ff0 con 0x7f695015c050 2024-03-06T07:21:58.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.791+0000 7f69477fe640 1 -- 172.21.15.72:0/3052838134 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69501f4440 con 0x7f695015c050 2024-03-06T07:21:58.792 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:21:58.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693006b500 msgr2=0x7f693006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693006b500 0x7f693006d9c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6940036000 tx=0x7f694003d040 comp rx=0 tx=0).stop 2024-03-06T07:21:58.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 msgr2=0x7f695006dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006dbb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f694800b970 tx=0x7f694800be30 comp rx=0 tx=0).stop 2024-03-06T07:21:58.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 shutdown_connections 2024-03-06T07:21:58.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693006b500 0x7f693006d9c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f695015c050 0x7f695006dbb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69500ccca0 0x7f69501f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 --2- 172.21.15.72:0/3052838134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006fb40 0x7f69501538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 >> 172.21.15.72:0/3052838134 conn(0x7f69500b3800 msgr2=0x7f695006ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 shutdown_connections 2024-03-06T07:21:58.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.794+0000 7f695553a640 1 -- 172.21.15.72:0/3052838134 wait complete. 2024-03-06T07:21:58.809 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:21:58.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.911+0000 7f7e2e471640 1 Processor -- start 2024-03-06T07:21:58.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.911+0000 7f7e2e471640 1 -- start start 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.912+0000 7f7e2e471640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e281614d0 0x7f7e280d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.912+0000 7f7e2e471640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e280d24b0 0x7f7e280b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.912+0000 7f7e2e471640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e281618b0 con 0x7f7e280d1910 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.912+0000 7f7e2e471640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e280cf400 con 0x7f7e280d24b0 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.912+0000 7f7e2e471640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e280cf6a0 con 0x7f7e281614d0 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e280d24b0 0x7f7e280b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e280d24b0 0x7f7e280b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54646/0 (socket says 172.21.15.72:54646) 2024-03-06T07:21:58.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4179727927 learned_addr learned my addr 172.21.15.72:0/4179727927 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e27fff640 1 --2- 172.21.15.72:0/4179727927 >> v2:172.21.15.146:3300/0 conn(0x7f7e281614d0 0x7f7e280d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4179727927 >> v2:172.21.15.146:3300/0 conn(0x7f7e281614d0 msgr2=0x7f7e280d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 --2- 172.21.15.72:0/4179727927 >> v2:172.21.15.146:3300/0 conn(0x7f7e281614d0 0x7f7e280d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4179727927 >> v1:172.21.15.146:6789/0 conn(0x7f7e280d1910 legacy=0x7f7e280d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.913+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4179727927 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280cf980 con 0x7f7e280d24b0 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2ca0e640 1 --2- 172.21.15.72:0/4179727927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280d24b0 0x7f7e280b0e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e2808c470 tx=0x7f7e2002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d068979c43d44f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e26ffd640 1 -- 172.21.15.72:0/4179727927 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2002cb20 con 0x7f7e280d24b0 2024-03-06T07:21:58.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e26ffd640 1 -- 172.21.15.72:0/4179727927 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2002ccc0 con 0x7f7e280d24b0 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e26ffd640 1 -- 172.21.15.72:0/4179727927 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20034610 con 0x7f7e280d24b0 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 -- 172.21.15.72:0/4179727927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280d24b0 msgr2=0x7f7e280b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4179727927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280d24b0 0x7f7e280b0e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e2808c470 tx=0x7f7e2002bf80 comp rx=0 tx=0).stop 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 -- 172.21.15.72:0/4179727927 shutdown_connections 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4179727927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280d24b0 0x7f7e280b0e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4179727927 >> v2:172.21.15.146:3300/0 conn(0x7f7e281614d0 0x7f7e280d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 -- 172.21.15.72:0/4179727927 >> 172.21.15.72:0/4179727927 conn(0x7f7e280d3d40 msgr2=0x7f7e280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 -- 172.21.15.72:0/4179727927 shutdown_connections 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.914+0000 7f7e2e471640 1 -- 172.21.15.72:0/4179727927 wait complete. 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.915+0000 7f7e2e471640 1 Processor -- start 2024-03-06T07:21:58.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 -- start start 2024-03-06T07:21:58.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280d1910 0x7f7e280b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280d24b0 0x7f7e280b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 0x7f7e280ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280b1370 con 0x7f7e281614d0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280b0910 con 0x7f7e280d1910 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2e471640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280b14f0 con 0x7f7e280d24b0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2ca0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 0x7f7e280ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2ca0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 0x7f7e280ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54660/0 (socket says 172.21.15.72:54660) 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.916+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4034538024 learned_addr learned my addr 172.21.15.72:0/4034538024 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e27fff640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280d1910 0x7f7e280b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e277fe640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280d24b0 0x7f7e280b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280d24b0 msgr2=0x7f7e280b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2ca0e640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280d24b0 0x7f7e280b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280d1910 msgr2=0x7f7e280b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2ca0e640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280d1910 0x7f7e280b98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2ca0e640 1 -- 172.21.15.72:0/4034538024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280cd7d0 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2ca0e640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 0x7f7e280ca060 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009f40 tx=0x7f7e20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20043020 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2003e450 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20009510 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e280b7160 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.917+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e280ccfd0 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.918+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e280ccb20 con 0x7f7e281614d0 2024-03-06T07:21:58.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.919+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e20046020 con 0x7f7e281614d0 2024-03-06T07:21:58.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.919+0000 7f7e24ff9640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:58.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.919+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e200aca10 con 0x7f7e281614d0 2024-03-06T07:21:58.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.919+0000 7f7e27fff640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:58.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.920+0000 7f7e27fff640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c006fd0 tx=0x7f7e1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:58.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:58.928+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e2004d030 con 0x7f7e281614d0 2024-03-06T07:21:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.104+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e28068ff0 con 0x7f7e281614d0 2024-03-06T07:21:59.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.104+0000 7f7e24ff9640 1 -- 172.21.15.72:0/4034538024 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e2007b890 con 0x7f7e281614d0 2024-03-06T07:21:59.106 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:21:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 msgr2=0x7f7e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c006fd0 tx=0x7f7e1c008040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 msgr2=0x7f7e280ca060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 0x7f7e280ca060 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009f40 tx=0x7f7e20004a40 comp rx=0 tx=0).stop 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 shutdown_connections 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.106+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e281614d0 0x7f7e280ca060 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.107+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280d24b0 0x7f7e280b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.107+0000 7f7e2e471640 1 --2- 172.21.15.72:0/4034538024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280d1910 0x7f7e280b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.107+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 >> 172.21.15.72:0/4034538024 conn(0x7f7e280d3d40 msgr2=0x7f7e280d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.107+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 shutdown_connections 2024-03-06T07:21:59.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.107+0000 7f7e2e471640 1 -- 172.21.15.72:0/4034538024 wait complete. 2024-03-06T07:21:59.118 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:21:59.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 Processor -- start 2024-03-06T07:21:59.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 -- start start 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f528815bdd0 0x7f52880c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f528815fa10 0x7f52880ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880cebe0 con 0x7f52880d16e0 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52880ceec0 con 0x7f528815fa10 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.218+0000 7f528f33e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52880cf1a0 con 0x7f528815bdd0 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f528815fa10 0x7f52880ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f528815fa10 0x7f52880ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54674/0 (socket says 172.21.15.72:54674) 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 -- 172.21.15.72:0/940320319 learned_addr learned my addr 172.21.15.72:0/940320319 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528d0da640 1 --2- 172.21.15.72:0/940320319 >> v2:172.21.15.146:3300/0 conn(0x7f528815bdd0 0x7f52880c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 -- 172.21.15.72:0/940320319 >> v2:172.21.15.146:3300/0 conn(0x7f528815bdd0 msgr2=0x7f52880c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 --2- 172.21.15.72:0/940320319 >> v2:172.21.15.146:3300/0 conn(0x7f528815bdd0 0x7f52880c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 -- 172.21.15.72:0/940320319 >> v1:172.21.15.146:6789/0 conn(0x7f52880d16e0 legacy=0x7f52880d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.219+0000 7f528c8d9640 1 -- 172.21.15.72:0/940320319 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52880cc8e0 con 0x7f528815fa10 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528c8d9640 1 --2- 172.21.15.72:0/940320319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815fa10 0x7f52880ce270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f528806e3c0 tx=0x7f527802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1208794d8e661b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f527ffff640 1 -- 172.21.15.72:0/940320319 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278035020 con 0x7f528815fa10 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f527ffff640 1 -- 172.21.15.72:0/940320319 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278031870 con 0x7f528815fa10 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f527ffff640 1 -- 172.21.15.72:0/940320319 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278031c40 con 0x7f528815fa10 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 -- 172.21.15.72:0/940320319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815fa10 msgr2=0x7f52880ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 --2- 172.21.15.72:0/940320319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815fa10 0x7f52880ce270 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f528806e3c0 tx=0x7f527802b910 comp rx=0 tx=0).stop 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 -- 172.21.15.72:0/940320319 shutdown_connections 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 --2- 172.21.15.72:0/940320319 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815fa10 0x7f52880ce270 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 --2- 172.21.15.72:0/940320319 >> v2:172.21.15.146:3300/0 conn(0x7f528815bdd0 0x7f52880c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 -- 172.21.15.72:0/940320319 >> 172.21.15.72:0/940320319 conn(0x7f528809a9f0 msgr2=0x7f528809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 -- 172.21.15.72:0/940320319 shutdown_connections 2024-03-06T07:21:59.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.220+0000 7f528f33e640 1 -- 172.21.15.72:0/940320319 wait complete. 2024-03-06T07:21:59.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.221+0000 7f528f33e640 1 Processor -- start 2024-03-06T07:21:59.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 -- start start 2024-03-06T07:21:59.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52880d16e0 0x7f5288149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 0x7f52880b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f528815fa10 0x7f52880b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528815c1b0 con 0x7f528815bdd0 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52880cc5d0 con 0x7f52880d16e0 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528f33e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52880ce7b0 con 0x7f528815fa10 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528c8d9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 0x7f52880b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528c8d9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 0x7f52880b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54678/0 (socket says 172.21.15.72:54678) 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.222+0000 7f528c8d9640 1 -- 172.21.15.72:0/580790384 learned_addr learned my addr 172.21.15.72:0/580790384 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528d0da640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52880d16e0 0x7f5288149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528d8db640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f528815fa10 0x7f52880b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528c8d9640 1 -- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f528815fa10 msgr2=0x7f52880b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528c8d9640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f528815fa10 0x7f52880b4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528c8d9640 1 -- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52880d16e0 msgr2=0x7f5288149db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528c8d9640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52880d16e0 0x7f5288149db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528c8d9640 1 -- 172.21.15.72:0/580790384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528815c4d0 con 0x7f528815bdd0 2024-03-06T07:21:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528d0da640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52880d16e0 0x7f5288149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528c8d9640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 0x7f52880b46d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5278000c00 tx=0x7f52780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278040070 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f527803e070 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5288155230 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278035040 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.223+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528815c770 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.224+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528804f330 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.225+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5278038020 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.225+0000 7f527dffb640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f526c06b4d0 0x7f526c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.225+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52780ac160 con 0x7f528815bdd0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.225+0000 7f528d0da640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f526c06b4d0 0x7f526c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.226+0000 7f528d0da640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f526c06b4d0 0x7f526c06d990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5284006fd0 tx=0x7f5284008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.229+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f527802f320 con 0x7f528815bdd0 2024-03-06T07:21:59.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.445+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52880b7a40 con 0x7f528815bdd0 2024-03-06T07:21:59.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.445+0000 7f527dffb640 1 -- 172.21.15.72:0/580790384 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f527807af50 con 0x7f528815bdd0 2024-03-06T07:21:59.446 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:59.447 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:21:59.447 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:21:59.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f526c06b4d0 msgr2=0x7f526c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f526c06b4d0 0x7f526c06d990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5284006fd0 tx=0x7f5284008040 comp rx=0 tx=0).stop 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 msgr2=0x7f52880b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 0x7f52880b46d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5278000c00 tx=0x7f52780046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 shutdown_connections 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f526c06b4d0 0x7f526c06d990 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f528815fa10 0x7f52880b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f528815bdd0 0x7f52880b46d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 --2- 172.21.15.72:0/580790384 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f52880d16e0 0x7f5288149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 >> 172.21.15.72:0/580790384 conn(0x7f528809a9f0 msgr2=0x7f528809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 shutdown_connections 2024-03-06T07:21:59.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.447+0000 7f528f33e640 1 -- 172.21.15.72:0/580790384 wait complete. 2024-03-06T07:21:59.459 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:21:59.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.558+0000 7ffb24972640 1 Processor -- start 2024-03-06T07:21:59.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.558+0000 7ffb24972640 1 -- start start 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb24972640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb200cfff0 0x7ffb200d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb24972640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb200cc8b0 0x7ffb200ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb24972640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb20161980 con 0x7ffb2015bb20 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb24972640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb200d2500 con 0x7ffb200cfff0 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb24972640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb20161c60 con 0x7ffb200cc8b0 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb17fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb200cfff0 0x7ffb200d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb1e59c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ffb2015bb20 0x7ffb200d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:39046/0 (socket says 172.21.15.72:39046) 2024-03-06T07:21:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb1e59c640 1 -- 172.21.15.72:0/96070152 learned_addr learned my addr 172.21.15.72:0/96070152 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.559+0000 7ffb1ed9d640 1 --2- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb200cc8b0 0x7ffb200ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.560+0000 7ffb17fff640 1 -- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb200cc8b0 msgr2=0x7ffb200ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.560+0000 7ffb17fff640 1 --2- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb200cc8b0 0x7ffb200ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.560+0000 7ffb17fff640 1 -- 172.21.15.72:0/96070152 >> v1:172.21.15.146:6789/0 conn(0x7ffb2015bb20 legacy=0x7ffb200d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.560+0000 7ffb17fff640 1 -- 172.21.15.72:0/96070152 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb20161f40 con 0x7ffb200cfff0 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.560+0000 7ffb17fff640 1 --2- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cfff0 0x7ffb200d04a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb080097a0 tx=0x7ffb0802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b04e9ea031e87c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb1dd9b640 1 -- 172.21.15.72:0/96070152 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08035020 con 0x7ffb200cfff0 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb1dd9b640 1 -- 172.21.15.72:0/96070152 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb08031830 con 0x7ffb200cfff0 2024-03-06T07:21:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb1dd9b640 1 -- 172.21.15.72:0/96070152 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08031c00 con 0x7ffb200cfff0 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 -- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cfff0 msgr2=0x7ffb200d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 --2- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cfff0 0x7ffb200d04a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb080097a0 tx=0x7ffb0802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 -- 172.21.15.72:0/96070152 shutdown_connections 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 --2- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb200cc8b0 0x7ffb200ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 --2- 172.21.15.72:0/96070152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cfff0 0x7ffb200d04a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 -- 172.21.15.72:0/96070152 >> 172.21.15.72:0/96070152 conn(0x7ffb2009a9f0 msgr2=0x7ffb2009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 -- 172.21.15.72:0/96070152 shutdown_connections 2024-03-06T07:21:59.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.561+0000 7ffb24972640 1 -- 172.21.15.72:0/96070152 wait complete. 2024-03-06T07:21:59.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.563+0000 7ffb24972640 1 Processor -- start 2024-03-06T07:21:59.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.563+0000 7ffb24972640 1 -- start start 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cc8b0 0x7ffb200c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 0x7ffb201446f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2015bb20 0x7ffb20144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200d1980 con 0x7ffb200cfff0 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb20162500 con 0x7ffb200cc8b0 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200d1b00 con 0x7ffb2015bb20 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 0x7ffb201446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 0x7ffb201446f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54690/0 (socket says 172.21.15.72:54690) 2024-03-06T07:21:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 -- 172.21.15.72:0/3386176495 learned_addr learned my addr 172.21.15.72:0/3386176495 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb1e59c640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cc8b0 0x7ffb200c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb1ed9d640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2015bb20 0x7ffb20144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 -- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2015bb20 msgr2=0x7ffb20144c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2015bb20 0x7ffb20144c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 -- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cc8b0 msgr2=0x7ffb200c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cc8b0 0x7ffb200c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 -- 172.21.15.72:0/3386176495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2006c9a0 con 0x7ffb200cfff0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb17fff640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 0x7ffb201446f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffb08034040 tx=0x7ffb080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08040070 con 0x7ffb200cfff0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0803a910 con 0x7ffb200cfff0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2006cc80 con 0x7ffb200cfff0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.565+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08035040 con 0x7ffb200cfff0 2024-03-06T07:21:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.566+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2006d240 con 0x7ffb200cfff0 2024-03-06T07:21:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.567+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2004f330 con 0x7ffb200cfff0 2024-03-06T07:21:59.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.567+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb08038020 con 0x7ffb200cfff0 2024-03-06T07:21:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.567+0000 7ffb16ffd640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0006b500 0x7ffb0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.568+0000 7ffb1e59c640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0006b500 0x7ffb0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.568+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb080abae0 con 0x7ffb200cfff0 2024-03-06T07:21:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.568+0000 7ffb1e59c640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0006b500 0x7ffb0006d9c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb10007fd0 tx=0x7ffb10007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.572+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb08031800 con 0x7ffb200cfff0 2024-03-06T07:21:59.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.752+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb200b1820 con 0x7ffb200cfff0 2024-03-06T07:21:59.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.753+0000 7ffb16ffd640 1 -- 172.21.15.72:0/3386176495 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7ffb0807a310 con 0x7ffb200cfff0 2024-03-06T07:21:59.754 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:21:59.754 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:21:59.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0006b500 msgr2=0x7ffb0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0006b500 0x7ffb0006d9c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffb10007fd0 tx=0x7ffb10007e80 comp rx=0 tx=0).stop 2024-03-06T07:21:59.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 msgr2=0x7ffb201446f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 0x7ffb201446f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffb08034040 tx=0x7ffb080046c0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 shutdown_connections 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0006b500 0x7ffb0006d9c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2015bb20 0x7ffb20144c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb200cfff0 0x7ffb201446f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 --2- 172.21.15.72:0/3386176495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb200cc8b0 0x7ffb200c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 >> 172.21.15.72:0/3386176495 conn(0x7ffb2009a9f0 msgr2=0x7ffb2006f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 shutdown_connections 2024-03-06T07:21:59.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.755+0000 7ffb24972640 1 -- 172.21.15.72:0/3386176495 wait complete. 2024-03-06T07:21:59.770 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:21:59.771 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:21:59.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.874+0000 7f65d43c7640 1 Processor -- start 2024-03-06T07:21:59.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.874+0000 7f65d43c7640 1 -- start start 2024-03-06T07:21:59.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d43c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc15fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d43c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f65cc0d0ac0 0x7f65cc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d43c7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc0d47f0 con 0x7f65cc0b4d10 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d43c7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc0ced70 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d43c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0d1ab0 con 0x7f65cc0d06e0 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d1962640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f65cc0d0ac0 0x7f65cc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d2964640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f65cc0b4d10 0x7f65cc0cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49234/0 (socket says 172.21.15.72:49234) 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d1962640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f65cc0d0ac0 0x7f65cc0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54706/0 (socket says 172.21.15.72:54706) 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d2964640 1 -- 172.21.15.72:0/2504864845 learned_addr learned my addr 172.21.15.72:0/2504864845 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d2163640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc15fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d1962640 1 -- 172.21.15.72:0/2504864845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 msgr2=0x7f65cc15fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:21:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d1962640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc15fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d1962640 1 -- 172.21.15.72:0/2504864845 >> v1:172.21.15.72:6789/0 conn(0x7f65cc0b4d10 legacy=0x7f65cc0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d1962640 1 -- 172.21.15.72:0/2504864845 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc0d1d90 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.875+0000 7f65d2163640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.876+0000 7f65d1962640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0b47d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f65c80097a0 tx=0x7f65c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2282a54b5459cf60 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.876+0000 7f65d1161640 1 -- 172.21.15.72:0/2504864845 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8035020 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.876+0000 7f65d1161640 1 -- 172.21.15.72:0/2504864845 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c8031830 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.876+0000 7f65d1161640 1 -- 172.21.15.72:0/2504864845 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8031c00 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 -- 172.21.15.72:0/2504864845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 msgr2=0x7f65cc0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0b47d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f65c80097a0 tx=0x7f65c802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 -- 172.21.15.72:0/2504864845 shutdown_connections 2024-03-06T07:21:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0b47d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 --2- 172.21.15.72:0/2504864845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc15fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 -- 172.21.15.72:0/2504864845 >> 172.21.15.72:0/2504864845 conn(0x7f65cc0d3cd0 msgr2=0x7f65cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:21:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 -- 172.21.15.72:0/2504864845 shutdown_connections 2024-03-06T07:21:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.877+0000 7f65d43c7640 1 -- 172.21.15.72:0/2504864845 wait complete. 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.878+0000 7f65d43c7640 1 Processor -- start 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.878+0000 7f65d43c7640 1 -- start start 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d43c7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0b4d10 0x7f65cc14e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d43c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc0c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d43c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d43c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0b44d0 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d43c7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0ce990 con 0x7f65cc0b4d10 2024-03-06T07:21:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d43c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc0d0f00 con 0x7f65cc0d06e0 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54722/0 (socket says 172.21.15.72:54722) 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 -- 172.21.15.72:0/1991054784 learned_addr learned my addr 172.21.15.72:0/1991054784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d1962640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc0c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2163640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0b4d10 0x7f65cc14e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 -- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 msgr2=0x7f65cc0c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc0c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 -- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0b4d10 msgr2=0x7f65cc14e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:21:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0b4d10 0x7f65cc14e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 -- 172.21.15.72:0/1991054784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc14e770 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2163640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0b4d10 0x7f65cc14e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.879+0000 7f65d2964640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0c9ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65c400cab0 tx=0x7f65c400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.880+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4017070 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.880+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c400ac30 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.880+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4012400 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.880+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc0b1370 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.880+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc0b1930 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.881+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc0a1c80 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.881+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f65c401a070 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.881+0000 7f65c2ffd640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f65b006b4d0 0x7f65b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.882+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65c4089d90 con 0x7f65cc0d0ac0 2024-03-06T07:21:59.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.882+0000 7f65d2163640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f65b006b4d0 0x7f65b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:21:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.882+0000 7f65d2163640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f65b006b4d0 0x7f65b006d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f65bc006fd0 tx=0x7f65bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:21:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:21:59.888+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f65c40585c0 con 0x7f65cc0d0ac0 2024-03-06T07:22:00.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.065+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f65cc068ff0 con 0x7f65cc0d0ac0 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.066+0000 7f65c2ffd640 1 -- 172.21.15.72:0/1991054784 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f65c405c660 con 0x7f65cc0d0ac0 2024-03-06T07:22:00.067 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.067+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f65b006b4d0 msgr2=0x7f65b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.067+0000 7f65d43c7640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f65b006b4d0 0x7f65b006d990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f65bc006fd0 tx=0x7f65bc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 msgr2=0x7f65cc0c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0c9ab0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65c400cab0 tx=0x7f65c400cf70 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 shutdown_connections 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f65b006b4d0 0x7f65b006d990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f65cc0d0ac0 0x7f65cc0c9ab0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0d06e0 0x7f65cc0c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 --2- 172.21.15.72:0/1991054784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0b4d10 0x7f65cc14e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 >> 172.21.15.72:0/1991054784 conn(0x7f65cc0d3cd0 msgr2=0x7f65cc15e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 shutdown_connections 2024-03-06T07:22:00.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.068+0000 7f65d43c7640 1 -- 172.21.15.72:0/1991054784 wait complete. 2024-03-06T07:22:00.080 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:00.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.184+0000 7f5c2426f640 1 Processor -- start 2024-03-06T07:22:00.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.184+0000 7f5c2426f640 1 -- start start 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2426f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15fc30 0x7f5c1c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2426f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c1600a0 con 0x7f5c1c0b4d10 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2426f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0d47f0 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2426f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c0ced70 con 0x7f5c1c15fc30 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2180a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c1c15fc30 0x7f5c1c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2280c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5c1c0b4d10 0x7f5c1c0cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49248/0 (socket says 172.21.15.72:49248) 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.185+0000 7f5c2280c640 1 -- 172.21.15.72:0/2617859308 learned_addr learned my addr 172.21.15.72:0/2617859308 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.186+0000 7f5c2180a640 1 -- 172.21.15.72:0/2617859308 >> v1:172.21.15.146:6789/0 conn(0x7f5c1c0d06e0 legacy=0x7f5c1c0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.186+0000 7f5c2180a640 1 -- 172.21.15.72:0/2617859308 >> v1:172.21.15.72:6789/0 conn(0x7f5c1c0b4d10 legacy=0x7f5c1c0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.186+0000 7f5c2180a640 1 -- 172.21.15.72:0/2617859308 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0d1ab0 con 0x7f5c1c15fc30 2024-03-06T07:22:00.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.186+0000 7f5c21009640 1 -- 172.21.15.72:0/2617859308 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865924109 0 0) 0x7f5c1c1600a0 con 0x7f5c1c0b4d10 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.186+0000 7f5c2180a640 1 --2- 172.21.15.72:0/2617859308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b47d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0097a0 tx=0x7f5c0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a81891d0a9ba9162 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c21009640 1 -- 172.21.15.72:0/2617859308 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c035020 con 0x7f5c1c15fc30 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c21009640 1 -- 172.21.15.72:0/2617859308 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c031830 con 0x7f5c1c15fc30 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c21009640 1 -- 172.21.15.72:0/2617859308 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c031c00 con 0x7f5c1c15fc30 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 -- 172.21.15.72:0/2617859308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15fc30 msgr2=0x7f5c1c0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 --2- 172.21.15.72:0/2617859308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b47d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0097a0 tx=0x7f5c0c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 -- 172.21.15.72:0/2617859308 shutdown_connections 2024-03-06T07:22:00.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 --2- 172.21.15.72:0/2617859308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b47d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 -- 172.21.15.72:0/2617859308 >> 172.21.15.72:0/2617859308 conn(0x7f5c1c0d3cd0 msgr2=0x7f5c1c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 -- 172.21.15.72:0/2617859308 shutdown_connections 2024-03-06T07:22:00.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.187+0000 7f5c2426f640 1 -- 172.21.15.72:0/2617859308 wait complete. 2024-03-06T07:22:00.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.189+0000 7f5c2426f640 1 Processor -- start 2024-03-06T07:22:00.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.189+0000 7f5c2426f640 1 -- start start 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2426f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c1c0b4d10 0x7f5c1c15a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2426f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 0x7f5c1c0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2426f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2426f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0ce990 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2426f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0d4ab0 con 0x7f5c1c15fc30 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2426f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0b44d0 con 0x7f5c1c0b4d10 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2180a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 0x7f5c1c0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2180a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 0x7f5c1c0b2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54740/0 (socket says 172.21.15.72:54740) 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.190+0000 7f5c2180a640 1 -- 172.21.15.72:0/1226101884 learned_addr learned my addr 172.21.15.72:0/1226101884 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2280c640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2200b640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c1c0b4d10 0x7f5c1c15a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2180a640 1 -- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c1c0b4d10 msgr2=0x7f5c1c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2180a640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c1c0b4d10 0x7f5c1c15a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2180a640 1 -- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c1c15fc30 msgr2=0x7f5c1c0b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2180a640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2180a640 1 -- 172.21.15.72:0/1226101884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0cdc10 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2280c640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2180a640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 0x7f5c1c0b2860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c000c00 tx=0x7f5c0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c004230 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0c0043d0 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c0cc7a0 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c003850 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.191+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c0ccd60 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.192+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c0a1c80 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.193+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c0c04c060 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.193+0000 7f5c12ffd640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c0006b500 0x7f5c0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.193+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c0c0abc90 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.193+0000 7f5c2200b640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c0006b500 0x7f5c0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.193+0000 7f5c2200b640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c0006b500 0x7f5c0006d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800a750 tx=0x7f5c18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.197+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c0c040070 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.374+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5c1c068230 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.374+0000 7f5c12ffd640 1 -- 172.21.15.72:0/1226101884 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5c0c07ab10 con 0x7f5c1c0d06e0 2024-03-06T07:22:00.375 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c0006b500 msgr2=0x7f5c0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c0006b500 0x7f5c0006d9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5c1800a750 tx=0x7f5c18008040 comp rx=0 tx=0).stop 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 msgr2=0x7f5c1c0b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 0x7f5c1c0b2860 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c000c00 tx=0x7f5c0c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 shutdown_connections 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c0006b500 0x7f5c0006d9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c1c15fc30 0x7f5c1c0b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c1c0d06e0 0x7f5c1c0b2860 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 --2- 172.21.15.72:0/1226101884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c1c0b4d10 0x7f5c1c15a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 >> 172.21.15.72:0/1226101884 conn(0x7f5c1c0d3cd0 msgr2=0x7f5c1c15b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 shutdown_connections 2024-03-06T07:22:00.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.376+0000 7f5c2426f640 1 -- 172.21.15.72:0/1226101884 wait complete. 2024-03-06T07:22:00.391 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.494+0000 7f44a73b6640 1 Processor -- start 2024-03-06T07:22:00.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.495+0000 7f44a73b6640 1 -- start start 2024-03-06T07:22:00.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.495+0000 7f44a73b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f44a00cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a73b6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a00cd4b0 con 0x7f44a015c050 2024-03-06T07:22:00.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a73b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a00cd790 con 0x7f44a006fd10 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a73b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00cda70 con 0x7f44a00d0af0 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a5152640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f44a015c050 0x7f44a006f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49260/0 (socket says 172.21.15.72:49260) 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a5152640 1 -- 172.21.15.72:0/3284528696 learned_addr learned my addr 172.21.15.72:0/3284528696 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a5953640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f44a00cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1937452773 0 0) 0x7f44a00cd4b0 con 0x7f44a015c050 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4488003520 con 0x7f44a015c050 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334321629 0 0) 0x7f44a00cd790 con 0x7f44a006fd10 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a00cd4b0 con 0x7f44a006fd10 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4052970297 0 0) 0x7f4488003520 con 0x7f44a015c050 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 msgr2=0x7f44a00cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f44a00cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 >> v1:172.21.15.146:6789/0 conn(0x7f44a006fd10 legacy=0x7f44a00cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a00cdd50 con 0x7f44a015c050 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f4498001670 con 0x7f44a015c050 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a5953640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f44a00cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 >> v1:172.21.15.72:6789/0 conn(0x7f44a015c050 legacy=0x7f44a006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4488003b70 0x7f4488003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f4488004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4488004d40 0x7f44880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00cda70 con 0x7f4488004d40 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448800baa0 con 0x7f4488003b70 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448800bca0 con 0x7f44a00d0af0 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a5953640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4488004d40 0x7f44880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a5152640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4488003b70 0x7f4488003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.496+0000 7f44a4951640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f4488004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f44a5953640 1 -- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 msgr2=0x7f4488004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f44a5953640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44a00d0af0 0x7f4488004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f44a5953640 1 -- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4488003b70 msgr2=0x7f4488003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f44a5953640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4488003b70 0x7f4488003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f44a5953640 1 -- 172.21.15.72:0/3284528696 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4488003520 con 0x7f4488004d40 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f44a5953640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4488004d40 0x7f44880051c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4490009c60 tx=0x7f4488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4774d854727b15d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.497+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490039070 con 0x7f4488004d40 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449002c8d0 con 0x7f4488004d40 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f449ffff640 1 -- 172.21.15.72:0/3284528696 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490009860 con 0x7f4488004d40 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a73b6640 1 -- 172.21.15.72:0/3284528696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4488004d40 msgr2=0x7f44880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a73b6640 1 --2- 172.21.15.72:0/3284528696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4488004d40 0x7f44880051c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4490009c60 tx=0x7f4488005830 comp rx=0 tx=0).stop 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a5953640 1 -- 172.21.15.72:0/3284528696 reap_dead start 2024-03-06T07:22:00.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a73b6640 1 -- 172.21.15.72:0/3284528696 shutdown_connections 2024-03-06T07:22:00.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a73b6640 1 -- 172.21.15.72:0/3284528696 >> 172.21.15.72:0/3284528696 conn(0x7f44a00b3800 msgr2=0x7f44a00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a73b6640 1 -- 172.21.15.72:0/3284528696 shutdown_connections 2024-03-06T07:22:00.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.498+0000 7f44a73b6640 1 -- 172.21.15.72:0/3284528696 wait complete. 2024-03-06T07:22:00.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.499+0000 7f44a73b6640 1 Processor -- start 2024-03-06T07:22:00.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.499+0000 7f44a73b6640 1 -- start start 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a73b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4488004d40 0x7f44a0154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a73b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44a009ccc0 0x7f44a009d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a73b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 0x7f44a00aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a73b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a00cd100 con 0x7f44a00aeae0 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a73b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0164ab0 con 0x7f44a009ccc0 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a73b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a009de20 con 0x7f4488004d40 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 0x7f44a00aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 0x7f44a00aef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54754/0 (socket says 172.21.15.72:54754) 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 -- 172.21.15.72:0/2766503973 learned_addr learned my addr 172.21.15.72:0/2766503973 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:00.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 -- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4488004d40 msgr2=0x7f44a0154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4488004d40 0x7f44a0154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a4951640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44a009ccc0 0x7f44a009d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 -- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44a009ccc0 msgr2=0x7f44a009d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44a009ccc0 0x7f44a009d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 -- 172.21.15.72:0/2766503973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a009e070 con 0x7f44a00aeae0 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a4951640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44a009ccc0 0x7f44a009d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.500+0000 7f44a5953640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 0x7f44a00aef60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4490009c60 tx=0x7f4490004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.501+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490039070 con 0x7f44a00aeae0 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.501+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449003a040 con 0x7f44a00aeae0 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.501+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a009e350 con 0x7f44a00aeae0 2024-03-06T07:22:00.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.501+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490009860 con 0x7f44a00aeae0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.501+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a01550b0 con 0x7f44a00aeae0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.502+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4490045030 con 0x7f44a00aeae0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.502+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a009c5d0 con 0x7f44a00aeae0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.503+0000 7f449dffb640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f448c06b4d0 0x7f448c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.503+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44900aaf50 con 0x7f44a00aeae0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.503+0000 7f44a5152640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f448c06b4d0 0x7f448c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.503+0000 7f44a5152640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f448c06b4d0 0x7f448c06d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4498001c80 tx=0x7f4498001b30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.507+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f449004c060 con 0x7f44a00aeae0 2024-03-06T07:22:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.696+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44a009fe60 con 0x7f44a00aeae0 2024-03-06T07:22:00.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.696+0000 7f449dffb640 1 -- 172.21.15.72:0/2766503973 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4490079dc0 con 0x7f44a00aeae0 2024-03-06T07:22:00.698 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f448c06b4d0 msgr2=0x7f448c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f448c06b4d0 0x7f448c06d990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4498001c80 tx=0x7f4498001b30 comp rx=0 tx=0).stop 2024-03-06T07:22:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 msgr2=0x7f44a00aef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 0x7f44a00aef60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4490009c60 tx=0x7f4490004730 comp rx=0 tx=0).stop 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 shutdown_connections 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f448c06b4d0 0x7f448c06d990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44a00aeae0 0x7f44a00aef60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44a009ccc0 0x7f44a009d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 --2- 172.21.15.72:0/2766503973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4488004d40 0x7f44a0154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 >> 172.21.15.72:0/2766503973 conn(0x7f44a00b3800 msgr2=0x7f44a00d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 shutdown_connections 2024-03-06T07:22:00.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.698+0000 7f44a73b6640 1 -- 172.21.15.72:0/2766503973 wait complete. 2024-03-06T07:22:00.714 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:00.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.813+0000 7f07c6dd6640 1 Processor -- start 2024-03-06T07:22:00.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.813+0000 7f07c6dd6640 1 -- start start 2024-03-06T07:22:00.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.813+0000 7f07c6dd6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07c015ab00 0x7f07c00b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.813+0000 7f07c6dd6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07c00b48f0 0x7f07c00af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07c6dd6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c00b4e30 con 0x7f07c00afc00 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07c6dd6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c00b0060 con 0x7f07c00b48f0 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07c6dd6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c00b5110 con 0x7f07c015ab00 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07c00b48f0 0x7f07c00af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07c00b48f0 0x7f07c00af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54758/0 (socket says 172.21.15.72:54758) 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 -- 172.21.15.72:0/2581719499 learned_addr learned my addr 172.21.15.72:0/2581719499 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07c4b72640 1 --2- 172.21.15.72:0/2581719499 >> v2:172.21.15.146:3300/0 conn(0x7f07c015ab00 0x7f07c00b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 -- 172.21.15.72:0/2581719499 >> v2:172.21.15.146:3300/0 conn(0x7f07c015ab00 msgr2=0x7f07c00b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 --2- 172.21.15.72:0/2581719499 >> v2:172.21.15.146:3300/0 conn(0x7f07c015ab00 0x7f07c00b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 -- 172.21.15.72:0/2581719499 >> v1:172.21.15.146:6789/0 conn(0x7f07c00afc00 legacy=0x7f07c006e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.814+0000 7f07bffff640 1 -- 172.21.15.72:0/2581719499 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c015b310 con 0x7f07c00b48f0 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07bffff640 1 --2- 172.21.15.72:0/2581719499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b48f0 0x7f07c00af6c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07c00732b0 tx=0x7f07ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f73ff0b335228 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2581719499 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac035020 con 0x7f07c00b48f0 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2581719499 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ac031870 con 0x7f07c00b48f0 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2581719499 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ac031c40 con 0x7f07c00b48f0 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 -- 172.21.15.72:0/2581719499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b48f0 msgr2=0x7f07c00af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/2581719499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b48f0 0x7f07c00af6c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f07c00732b0 tx=0x7f07ac02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 -- 172.21.15.72:0/2581719499 shutdown_connections 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/2581719499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b48f0 0x7f07c00af6c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/2581719499 >> v2:172.21.15.146:3300/0 conn(0x7f07c015ab00 0x7f07c00b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 -- 172.21.15.72:0/2581719499 >> 172.21.15.72:0/2581719499 conn(0x7f07c006c0c0 msgr2=0x7f07c006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 -- 172.21.15.72:0/2581719499 shutdown_connections 2024-03-06T07:22:00.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.815+0000 7f07c6dd6640 1 -- 172.21.15.72:0/2581719499 wait complete. 2024-03-06T07:22:00.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.816+0000 7f07c6dd6640 1 Processor -- start 2024-03-06T07:22:00.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 -- start start 2024-03-06T07:22:00.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c00afc00 0x7f07c0147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c00b48f0 0x7f07c0147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 0x7f07c00ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c0071ca0 con 0x7f07c015ab00 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c0071b20 con 0x7f07c00afc00 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c6dd6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c015aee0 con 0x7f07c00b48f0 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.817+0000 7f07c5373640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 0x7f07c00ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 0x7f07c00ce620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54768/0 (socket says 172.21.15.72:54768) 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 -- 172.21.15.72:0/1127460980 learned_addr learned my addr 172.21.15.72:0/1127460980 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c4b72640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c00afc00 0x7f07c0147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07bffff640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c00b48f0 0x7f07c0147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 -- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c00b48f0 msgr2=0x7f07c0147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c00b48f0 0x7f07c0147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 -- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c00afc00 msgr2=0x7f07c0147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c00afc00 0x7f07c0147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 -- 172.21.15.72:0/1127460980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0148170 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c5373640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 0x7f07c00ce620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c780 tx=0x7f07b800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8004b10 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b8004cb0 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c015c160 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b800aa90 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.818+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c015c720 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.819+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c004f330 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.820+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07b800ac30 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.820+0000 7f07bd7fa640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07a406b4d0 0x7f07a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.820+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07b808a1b0 con 0x7f07c015ab00 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.820+0000 7f07c4b72640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07a406b4d0 0x7f07a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:00.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.820+0000 7f07c4b72640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07a406b4d0 0x7f07a406d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07b40079a0 tx=0x7f07b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:00.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:00.824+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07b8020070 con 0x7f07c015ab00 2024-03-06T07:22:01.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.094+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07c015a170 con 0x7f07c015ab00 2024-03-06T07:22:01.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.094+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1127460980 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f07b8058a70 con 0x7f07c015ab00 2024-03-06T07:22:01.097 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:01.098 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:22:01.099 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:22:01.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07a406b4d0 msgr2=0x7f07a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07a406b4d0 0x7f07a406d990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f07b40079a0 tx=0x7f07b4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 msgr2=0x7f07c00ce620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 0x7f07c00ce620 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f07b800c780 tx=0x7f07b800cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:01.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 shutdown_connections 2024-03-06T07:22:01.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07a406b4d0 0x7f07a406d990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c015ab00 0x7f07c00ce620 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c00b48f0 0x7f07c0147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 --2- 172.21.15.72:0/1127460980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c00afc00 0x7f07c0147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.096+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 >> 172.21.15.72:0/1127460980 conn(0x7f07c006c0c0 msgr2=0x7f07c01591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.097+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 shutdown_connections 2024-03-06T07:22:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.097+0000 7f07c6dd6640 1 -- 172.21.15.72:0/1127460980 wait complete. 2024-03-06T07:22:01.109 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:01.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.208+0000 7ff1cb79a640 1 Processor -- start 2024-03-06T07:22:01.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.209+0000 7ff1cb79a640 1 -- start start 2024-03-06T07:22:01.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1cb79a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1c415bd10 0x7ff1c406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1cb79a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1c406c5c0 0x7ff1c406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1cb79a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c406ba30 0x7ff1c40acf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1cb79a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c406bee0 con 0x7ff1c415bd10 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1cb79a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40cc090 con 0x7ff1c406c5c0 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1cb79a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40ae170 con 0x7ff1c406ba30 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1c415bd10 0x7ff1c406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1c415bd10 0x7ff1c406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54770/0 (socket says 172.21.15.72:54770) 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 -- 172.21.15.72:0/2221056293 learned_addr learned my addr 172.21.15.72:0/2221056293 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c8d35640 1 --2- 172.21.15.72:0/2221056293 >> v2:172.21.15.146:3300/0 conn(0x7ff1c406c5c0 0x7ff1c406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9d37640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c406ba30 0x7ff1c40acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 -- 172.21.15.72:0/2221056293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c406ba30 msgr2=0x7ff1c40acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c406ba30 0x7ff1c40acf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 -- 172.21.15.72:0/2221056293 >> v2:172.21.15.146:3300/0 conn(0x7ff1c406c5c0 msgr2=0x7ff1c406b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 --2- 172.21.15.72:0/2221056293 >> v2:172.21.15.146:3300/0 conn(0x7ff1c406c5c0 0x7ff1c406b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9536640 1 -- 172.21.15.72:0/2221056293 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40cc330 con 0x7ff1c415bd10 2024-03-06T07:22:01.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c8d35640 1 --2- 172.21.15.72:0/2221056293 >> v2:172.21.15.146:3300/0 conn(0x7ff1c406c5c0 0x7ff1c406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:01.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.210+0000 7ff1c9d37640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c406ba30 0x7ff1c40acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.211+0000 7ff1c9536640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c415bd10 0x7ff1c406afb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff1c406cc10 tx=0x7ff1b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b315f7880229ca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.211+0000 7ff1bbfff640 1 -- 172.21.15.72:0/2221056293 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4035020 con 0x7ff1c415bd10 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.211+0000 7ff1bbfff640 1 -- 172.21.15.72:0/2221056293 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b4031870 con 0x7ff1c415bd10 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1bbfff640 1 -- 172.21.15.72:0/2221056293 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b4031c40 con 0x7ff1c415bd10 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 -- 172.21.15.72:0/2221056293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c415bd10 msgr2=0x7ff1c406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c415bd10 0x7ff1c406afb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff1c406cc10 tx=0x7ff1b402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 -- 172.21.15.72:0/2221056293 shutdown_connections 2024-03-06T07:22:01.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c406ba30 0x7ff1c40acf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/2221056293 >> v2:172.21.15.146:3300/0 conn(0x7ff1c406c5c0 0x7ff1c406b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/2221056293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c415bd10 0x7ff1c406afb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 -- 172.21.15.72:0/2221056293 >> 172.21.15.72:0/2221056293 conn(0x7ff1c40d00e0 msgr2=0x7ff1c40d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 -- 172.21.15.72:0/2221056293 shutdown_connections 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.212+0000 7ff1cb79a640 1 -- 172.21.15.72:0/2221056293 wait complete. 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.214+0000 7ff1cb79a640 1 Processor -- start 2024-03-06T07:22:01.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.214+0000 7ff1cb79a640 1 -- start start 2024-03-06T07:22:01.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1cb79a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c406ba30 0x7ff1c40c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1cb79a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 0x7ff1c40c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1cb79a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c415bd10 0x7ff1c40c5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1cb79a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c415c0f0 con 0x7ff1c406c5c0 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1cb79a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40ae450 con 0x7ff1c406ba30 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1cb79a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c415c270 con 0x7ff1c415bd10 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1c8d35640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 0x7ff1c40c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1c8d35640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 0x7ff1c40c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54772/0 (socket says 172.21.15.72:54772) 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.215+0000 7ff1c8d35640 1 -- 172.21.15.72:0/1232467879 learned_addr learned my addr 172.21.15.72:0/1232467879 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c9d37640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c415bd10 0x7ff1c40c5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c9536640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c406ba30 0x7ff1c40c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c8d35640 1 -- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c415bd10 msgr2=0x7ff1c40c5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c8d35640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c415bd10 0x7ff1c40c5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c8d35640 1 -- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c406ba30 msgr2=0x7ff1c40c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c8d35640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c406ba30 0x7ff1c40c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c8d35640 1 -- 172.21.15.72:0/1232467879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40c4060 con 0x7ff1c406c5c0 2024-03-06T07:22:01.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1c8d35640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 0x7ff1c40c5690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000c820 tx=0x7ff1c000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0004230 con 0x7ff1c406c5c0 2024-03-06T07:22:01.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c00043d0 con 0x7ff1c406c5c0 2024-03-06T07:22:01.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c409c450 con 0x7ff1c406c5c0 2024-03-06T07:22:01.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0007a70 con 0x7ff1c406c5c0 2024-03-06T07:22:01.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.216+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c409ca10 con 0x7ff1c406c5c0 2024-03-06T07:22:01.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.217+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c404f330 con 0x7ff1c406c5c0 2024-03-06T07:22:01.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.217+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1c0007c10 con 0x7ff1c406c5c0 2024-03-06T07:22:01.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.218+0000 7ff1b9ffb640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.218+0000 7ff1c9536640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.218+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1c008a1d0 con 0x7ff1c406c5c0 2024-03-06T07:22:01.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.218+0000 7ff1c9536640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4036000 tx=0x7ff1b403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.222+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1c0015020 con 0x7ff1c406c5c0 2024-03-06T07:22:01.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.400+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1c406e2f0 con 0x7ff1c406c5c0 2024-03-06T07:22:01.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.401+0000 7ff1b9ffb640 1 -- 172.21.15.72:0/1232467879 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7ff1c0058a90 con 0x7ff1c406c5c0 2024-03-06T07:22:01.404 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:01.404 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 msgr2=0x7ff1a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4036000 tx=0x7ff1b403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 msgr2=0x7ff1c40c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 0x7ff1c40c5690 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000c820 tx=0x7ff1c000cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 shutdown_connections 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c415bd10 0x7ff1c40c5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c406c5c0 0x7ff1c40c5690 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 --2- 172.21.15.72:0/1232467879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c406ba30 0x7ff1c40c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 >> 172.21.15.72:0/1232467879 conn(0x7ff1c40d00e0 msgr2=0x7ff1c406d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 shutdown_connections 2024-03-06T07:22:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.403+0000 7ff1cb79a640 1 -- 172.21.15.72:0/1232467879 wait complete. 2024-03-06T07:22:01.418 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:01.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.548+0000 7f5bcb55c640 1 Processor -- start 2024-03-06T07:22:01.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.549+0000 7f5bcb55c640 1 -- start start 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.549+0000 7f5bcb55c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bc406b220 0x7f5bc415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.549+0000 7f5bcb55c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc415ea30 con 0x7f5bc406b600 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.549+0000 7f5bcb55c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc415ed10 con 0x7f5bc40ce920 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.549+0000 7f5bcb55c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc4161b70 con 0x7f5bc406b220 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc9af9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5bc406b600 0x7f5bc4162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49276/0 (socket says 172.21.15.72:49276) 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc9af9640 1 -- 172.21.15.72:0/2998828897 learned_addr learned my addr 172.21.15.72:0/2998828897 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2998828897 >> v2:172.21.15.146:3300/0 conn(0x7f5bc406b220 0x7f5bc415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823480504 0 0) 0x7f5bc415ea30 con 0x7f5bc406b600 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb0003520 con 0x7f5bc406b600 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2839732495 0 0) 0x7f5bb0003520 con 0x7f5bc406b600 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 >> v2:172.21.15.146:3300/0 conn(0x7f5bc406b220 msgr2=0x7f5bc415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 --2- 172.21.15.72:0/2998828897 >> v2:172.21.15.146:3300/0 conn(0x7f5bc406b220 0x7f5bc415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 >> v1:172.21.15.146:6789/0 conn(0x7f5bc40ce920 legacy=0x7f5bc4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc415eff0 con 0x7f5bc406b600 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f5bbc0019f0 con 0x7f5bc406b600 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2998828897 >> v2:172.21.15.146:3300/0 conn(0x7f5bc406b220 0x7f5bc415e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 >> v1:172.21.15.72:6789/0 conn(0x7f5bc406b600 legacy=0x7f5bc4162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bb0003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb00043d0 0x7f5bb00048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb00050a0 0x7f5bb000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4161b70 con 0x7f5bb00039b0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb000bfe0 con 0x7f5bb00043d0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb000c1e0 con 0x7f5bb00050a0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bb0003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc8af7640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb00043d0 0x7f5bb00048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.550+0000 7f5bc9af9640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb00050a0 0x7f5bb000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb00050a0 msgr2=0x7f5bb000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb00050a0 0x7f5bb000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb00043d0 msgr2=0x7f5bb00048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb00043d0 0x7f5bb00048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc9af9640 1 -- 172.21.15.72:0/2998828897 reap_dead start 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2998828897 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0003520 con 0x7f5bb00039b0 2024-03-06T07:22:01.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bb0003db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009c90 tx=0x7f5bb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c70a71ade3722d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0039070 con 0x7f5bb00039b0 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc002c8f0 con 0x7f5bb00039b0 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.551+0000 7f5bb7fff640 1 -- 172.21.15.72:0/2998828897 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0009860 con 0x7f5bb00039b0 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2998828897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 msgr2=0x7f5bb0003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bb0003db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009c90 tx=0x7f5bb0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2998828897 shutdown_connections 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2998828897 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bb0003db0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2998828897 >> 172.21.15.72:0/2998828897 conn(0x7f5bc406c0c0 msgr2=0x7f5bc406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2998828897 shutdown_connections 2024-03-06T07:22:01.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.552+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2998828897 wait complete. 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.553+0000 7f5bcb55c640 1 Processor -- start 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 -- start start 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bc4145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc4152a30 0x7f5bc4157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc415a490 0x7f5bc415a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40ce380 con 0x7f5bb00039b0 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40ce200 con 0x7f5bc4152a30 2024-03-06T07:22:01.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bcb55c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc415e450 con 0x7f5bc415a490 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bc92f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bc4145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bc92f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bc4145360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54796/0 (socket says 172.21.15.72:54796) 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.554+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2940781061 learned_addr learned my addr 172.21.15.72:0/2940781061 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2940781061 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc415a490 msgr2=0x7f5bc415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc415a490 0x7f5bc415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2940781061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc4152a30 msgr2=0x7f5bc4157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc4152a30 0x7f5bc4157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bc92f8640 1 -- 172.21.15.72:0/2940781061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc4153390 con 0x7f5bb00039b0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bc92f8640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bc4145360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009c90 tx=0x7f5bc0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0039050 con 0x7f5bb00039b0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc0045070 con 0x7f5bb00039b0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc4153670 con 0x7f5bb00039b0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0009860 con 0x7f5bb00039b0 2024-03-06T07:22:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.555+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc4153c30 con 0x7f5bb00039b0 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.556+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc404f330 con 0x7f5bb00039b0 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.557+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bc0040450 con 0x7f5bb00039b0 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.557+0000 7f5bb5ffb640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba806b500 0x7f5ba806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.557+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bc00abf60 con 0x7f5bb00039b0 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.557+0000 7f5bc8af7640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba806b500 0x7f5ba806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.557+0000 7f5bc8af7640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba806b500 0x7f5ba806d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8007fd0 tx=0x7f5bb8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:01.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.565+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bc004e020 con 0x7f5bb00039b0 2024-03-06T07:22:01.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.781+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bc40cdf00 con 0x7f5bb00039b0 2024-03-06T07:22:01.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.782+0000 7f5bb5ffb640 1 -- 172.21.15.72:0/2940781061 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f5bc007ade0 con 0x7f5bb00039b0 2024-03-06T07:22:01.784 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:01.784 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:22:01.785 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:22:01.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba806b500 msgr2=0x7f5ba806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba806b500 0x7f5ba806d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8007fd0 tx=0x7f5bb8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:01.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 msgr2=0x7f5bc4145360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:01.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bc4145360 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009c90 tx=0x7f5bc0004730 comp rx=0 tx=0).stop 2024-03-06T07:22:01.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 shutdown_connections 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba806b500 0x7f5ba806d9c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc415a490 0x7f5bc415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc4152a30 0x7f5bc4157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 --2- 172.21.15.72:0/2940781061 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb00039b0 0x7f5bc4145360 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 >> 172.21.15.72:0/2940781061 conn(0x7f5bc406c0c0 msgr2=0x7f5bc40cf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 shutdown_connections 2024-03-06T07:22:01.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:01.784+0000 7f5bcb55c640 1 -- 172.21.15.72:0/2940781061 wait complete. 2024-03-06T07:22:02.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:21:49.400572+0000', 'modified': '2024-03-06T07:21:52.396816+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6836/3798106106', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3798106106}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 3798106106}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4272, 'qdb_cluster': [4272]} 2024-03-06T07:22:02.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:02.201 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:22:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.304+0000 7f6012b16640 1 Processor -- start 2024-03-06T07:22:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.304+0000 7f6012b16640 1 -- start start 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f6012b16640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f600c15e010 0x7f600c0cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f6012b16640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f600c0b3240 0x7f600c0b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f6012b16640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c0b2910 0x7f600c0d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f6012b16640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c0b1f40 con 0x7f600c0b3240 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f6012b16640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f600c15e3f0 con 0x7f600c15e010 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f6012b16640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0b1c10 con 0x7f600c0b2910 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f600c0b3240 0x7f600c0b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f600c0b3240 0x7f600c0b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54802/0 (socket says 172.21.15.72:54802) 2024-03-06T07:22:02.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 -- 172.21.15.72:0/3990518644 learned_addr learned my addr 172.21.15.72:0/3990518644 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f60108b2640 1 --2- 172.21.15.72:0/3990518644 >> v2:172.21.15.146:3300/0 conn(0x7f600c15e010 0x7f600c0cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f60110b3640 1 --2- 172.21.15.72:0/3990518644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c0b2910 0x7f600c0d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 -- 172.21.15.72:0/3990518644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c0b2910 msgr2=0x7f600c0d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 --2- 172.21.15.72:0/3990518644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c0b2910 0x7f600c0d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 -- 172.21.15.72:0/3990518644 >> v2:172.21.15.146:3300/0 conn(0x7f600c15e010 msgr2=0x7f600c0cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 --2- 172.21.15.72:0/3990518644 >> v2:172.21.15.146:3300/0 conn(0x7f600c15e010 0x7f600c0cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f600bfff640 1 -- 172.21.15.72:0/3990518644 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c06d1b0 con 0x7f600c0b3240 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.305+0000 7f60108b2640 1 --2- 172.21.15.72:0/3990518644 >> v2:172.21.15.146:3300/0 conn(0x7f600c15e010 0x7f600c0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.306+0000 7f600bfff640 1 --2- 172.21.15.72:0/3990518644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c0b3240 0x7f600c0b23d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f600c0715c0 tx=0x7f5ff402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f83aa0b3c14d4de server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.306+0000 7f600b7fe640 1 -- 172.21.15.72:0/3990518644 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff402e070 con 0x7f600c0b3240 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.306+0000 7f600b7fe640 1 -- 172.21.15.72:0/3990518644 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff4037070 con 0x7f600c0b3240 2024-03-06T07:22:02.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.306+0000 7f600b7fe640 1 -- 172.21.15.72:0/3990518644 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4031a00 con 0x7f600c0b3240 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 -- 172.21.15.72:0/3990518644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c0b3240 msgr2=0x7f600c0b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 --2- 172.21.15.72:0/3990518644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c0b3240 0x7f600c0b23d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f600c0715c0 tx=0x7f5ff402b880 comp rx=0 tx=0).stop 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 -- 172.21.15.72:0/3990518644 shutdown_connections 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 --2- 172.21.15.72:0/3990518644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c0b2910 0x7f600c0d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 --2- 172.21.15.72:0/3990518644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c0b3240 0x7f600c0b23d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 --2- 172.21.15.72:0/3990518644 >> v2:172.21.15.146:3300/0 conn(0x7f600c15e010 0x7f600c0cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 -- 172.21.15.72:0/3990518644 >> 172.21.15.72:0/3990518644 conn(0x7f600c09a9f0 msgr2=0x7f600c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 -- 172.21.15.72:0/3990518644 shutdown_connections 2024-03-06T07:22:02.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.307+0000 7f6012b16640 1 -- 172.21.15.72:0/3990518644 wait complete. 2024-03-06T07:22:02.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.308+0000 7f6012b16640 1 Processor -- start 2024-03-06T07:22:02.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.308+0000 7f6012b16640 1 -- start start 2024-03-06T07:22:02.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f6012b16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 0x7f600c1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f6012b16640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f600c157360 0x7f600c0c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f6012b16640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c1577e0 0x7f600c0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f6012b16640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0cf310 con 0x7f600c15e010 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f6012b16640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c06ced0 con 0x7f600c157360 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f6012b16640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c0d2010 con 0x7f600c1577e0 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 0x7f600c1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 0x7f600c1526d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54808/0 (socket says 172.21.15.72:54808) 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 -- 172.21.15.72:0/2461945234 learned_addr learned my addr 172.21.15.72:0/2461945234 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f600bfff640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f600c157360 0x7f600c0c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60110b3640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c1577e0 0x7f600c0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 -- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c1577e0 msgr2=0x7f600c0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c1577e0 0x7f600c0c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 -- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f600c157360 msgr2=0x7f600c0c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f600c157360 0x7f600c0c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 -- 172.21.15.72:0/2461945234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600c152c10 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60110b3640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c1577e0 0x7f600c0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.309+0000 7f60108b2640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 0x7f600c1526d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00d800 tx=0x7f5ffc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.310+0000 7f60097fa640 1 -- 172.21.15.72:0/2461945234 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc004030 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.310+0000 7f60097fa640 1 -- 172.21.15.72:0/2461945234 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc0041d0 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.310+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f600c0b48e0 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.310+0000 7f60097fa640 1 -- 172.21.15.72:0/2461945234 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0096a0 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.310+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f600c0b4ea0 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.311+0000 7f60097fa640 1 -- 172.21.15.72:0/2461945234 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ffc009840 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.311+0000 7f60097fa640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fec06b500 0x7f5fec06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.311+0000 7f600bfff640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fec06b500 0x7f5fec06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.311+0000 7f60097fa640 1 -- 172.21.15.72:0/2461945234 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ffc08a360 con 0x7f600c15e010 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.312+0000 7f600bfff640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fec06b500 0x7f5fec06d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4036000 tx=0x7f5ff403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.312+0000 7f60157d0740 1 Processor -- start 2024-03-06T07:22:02.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.312+0000 7f60157d0740 1 -- start start 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639d90870 0x560639d90c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639d948c0 0x560639d96360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60157d0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639dc0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60157d0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560639cfa6b0 con 0x560639dc0600 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560639ca3a60 con 0x560639d948c0 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560639cf6750 con 0x560639d90870 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60110b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639dc0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60110b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639dc0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54814/0 (socket says 172.21.15.72:54814) 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60110b3640 1 -- 172.21.15.72:0/2039888506 learned_addr learned my addr 172.21.15.72:0/2039888506 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f600bfff640 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639d948c0 0x560639d96360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60110b3640 1 -- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639d90870 msgr2=0x560639d90c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60108b2640 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639d90870 0x560639d90c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60110b3640 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639d90870 0x560639d90c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.313+0000 7f60110b3640 1 -- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639d948c0 msgr2=0x560639d96360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60110b3640 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639d948c0 0x560639d96360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60110b3640 1 -- 172.21.15.72:0/2039888506 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560639cc1f30 con 0x560639dc0600 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60110b3640 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639dc0a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f60040073e0 tx=0x7f60040078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87a2d9218a650be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60027fc640 1 -- 172.21.15.72:0/2039888506 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60040147e0 con 0x560639dc0600 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60027fc640 1 -- 172.21.15.72:0/2039888506 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600401a070 con 0x560639dc0600 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60027fc640 1 -- 172.21.15.72:0/2039888506 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004014bc0 con 0x560639dc0600 2024-03-06T07:22:02.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 -- 172.21.15.72:0/2039888506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 msgr2=0x560639dc0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639dc0a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f60040073e0 tx=0x7f60040078a0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 -- 172.21.15.72:0/2039888506 shutdown_connections 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639dc0a80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639d948c0 0x560639d96360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 --2- 172.21.15.72:0/2039888506 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639d90870 0x560639d90c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 -- 172.21.15.72:0/2039888506 >> 172.21.15.72:0/2039888506 conn(0x560639d928f0 msgr2=0x560639d941b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 -- 172.21.15.72:0/2039888506 shutdown_connections 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.314+0000 7f60157d0740 1 -- 172.21.15.72:0/2039888506 wait complete. 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.315+0000 7f60157d0740 1 -- start start 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.316+0000 7f60157d0740 1 Processor -- start 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639db2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639db2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639ddde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560639d94d50 con 0x560639dd9b00 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560639cb6ea0 con 0x560639db3450 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560639d95030 con 0x560639dc0600 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60157d0740 10 client.0 _ll_register_callbacks cb 0x560639cb6870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639ddde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639ddde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54824/0 (socket says 172.21.15.72:54824) 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 -- 172.21.15.72:0/2758320421 learned_addr learned my addr 172.21.15.72:0/2758320421 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f600bfff640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639db2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60108b2640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639db2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 -- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 msgr2=0x560639db2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639db2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 -- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 msgr2=0x560639db2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639db2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 -- 172.21.15.72:0/2758320421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560639cb7270 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60108b2640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639db2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f60110b3640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639ddde70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x560639dc4cf0 tx=0x7f6004004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3685aa8c2a9d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.317+0000 7f5feb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f5feb7fe640 1 -- 172.21.15.72:0/2758320421 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f600402d060 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f5feb7fe640 1 -- 172.21.15.72:0/2758320421 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6004014cf0 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f600401a070 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f5feb7fe640 1 -- 172.21.15.72:0/2758320421 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6004033050 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 10 client.4611 fetch_fsmap learned FSMap version 7 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x560639a0f2d0 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f5feb7fe640 1 -- 172.21.15.72:0/2758320421 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f600401a070 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f5feb7fe640 1 -- 172.21.15.72:0/2758320421 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6004036020 con 0x560639dd9b00 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 10 client.4611 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:22:02.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 10 client.4611 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 20 client.4611 populate_metadata read hostname 'smithi072' 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639db5be0 0x560639dbc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 4 client.4611 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 --> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560639d07bf0 con 0x560639db5be0 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60108b2640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639db5be0 0x560639dbc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f5feb7fe640 10 client.4611 ms_handle_connect on v2:172.21.15.72:6834/3798106106 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.318+0000 7f60108b2640 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639db5be0 0x560639dbc7e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6e1acb7b8a19281 server_cookie=a63977ccf4ed3ff4 in_seq=0 out_seq=0 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f5feb7fe640 1 -- 172.21.15.72:0/2758320421 <== mds.0 v2:172.21.15.72:6834/3798106106 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560639d07bf0 con 0x560639db5be0 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f5feb7fe640 10 client.4611 handle_command_reply: tid=0 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 client.4611 shutdown 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 msgr2=0x560639ddde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639ddde70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x560639dc4cf0 tx=0x7f6004004ab0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 shutdown_connections 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639db5be0 0x560639dbc7e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639ddde70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639db2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f5feb7fe640 0 client.4611 ms_handle_reset on v2:172.21.15.72:6834/3798106106 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 --2- 172.21.15.72:0/2758320421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639db2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 >> 172.21.15.72:0/2758320421 conn(0x560639d90870 msgr2=0x560639dbe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 shutdown_connections 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 1 -- 172.21.15.72:0/2758320421 wait complete. 2024-03-06T07:22:02.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.319+0000 7f60157d0740 20 client.4611 trim_cache size 0 max 16384 2024-03-06T07:22:02.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.336+0000 7f60157d0740 1 Processor -- start 2024-03-06T07:22:02.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.336+0000 7f60157d0740 1 -- start start 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639db3450 0x560639ddd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60157d0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639ddf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639dd9b00 0x560639d94080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60157d0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1ff3a0 con 0x560639dc0600 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1fe8a0 con 0x560639dd9b00 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600c1fedd0 con 0x560639db3450 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639ddf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639ddf170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54834/0 (socket says 172.21.15.72:54834) 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 -- 172.21.15.72:0/2137690794 learned_addr learned my addr 172.21.15.72:0/2137690794 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f60108b2640 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639db3450 0x560639ddd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 -- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639db3450 msgr2=0x560639ddd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639db3450 0x560639ddd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 -- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639dd9b00 msgr2=0x560639d94080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639dd9b00 0x560639d94080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 -- 172.21.15.72:0/2137690794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560639ad96b0 con 0x560639dc0600 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f600bfff640 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639ddf170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ff402b8c0 tx=0x7f5ff4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ce86d134b23c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f6001ffb640 1 -- 172.21.15.72:0/2137690794 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4055070 con 0x560639dc0600 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f6001ffb640 1 -- 172.21.15.72:0/2137690794 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff4037040 con 0x560639dc0600 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.337+0000 7f6001ffb640 1 -- 172.21.15.72:0/2137690794 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4003810 con 0x560639dc0600 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 -- 172.21.15.72:0/2137690794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 msgr2=0x560639ddf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639ddf170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ff402b8c0 tx=0x7f5ff4030000 comp rx=0 tx=0).stop 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 -- 172.21.15.72:0/2137690794 shutdown_connections 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639dd9b00 0x560639d94080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dc0600 0x560639ddf170 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 --2- 172.21.15.72:0/2137690794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639db3450 0x560639ddd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 -- 172.21.15.72:0/2137690794 >> 172.21.15.72:0/2137690794 conn(0x560639d90870 msgr2=0x560639dddc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 -- 172.21.15.72:0/2137690794 shutdown_connections 2024-03-06T07:22:02.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.338+0000 7f60157d0740 1 -- 172.21.15.72:0/2137690794 wait complete. 2024-03-06T07:22:02.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.339+0000 7f60157d0740 1 -- start start 2024-03-06T07:22:02.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.339+0000 7f60157d0740 1 Processor -- start 2024-03-06T07:22:02.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639e418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639dbf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639dbfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560639db5240 con 0x560639dd9b00 2024-03-06T07:22:02.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560639db4c90 con 0x560639db3450 2024-03-06T07:22:02.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560639db5fe0 con 0x560639dc0600 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 10 client.0 _ll_register_callbacks cb 0x7f60040146b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639dbfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639dbfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54842/0 (socket says 172.21.15.72:54842) 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 -- 172.21.15.72:0/930148518 learned_addr learned my addr 172.21.15.72:0/930148518 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60108b2640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639e418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f600bfff640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639dbf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 -- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 msgr2=0x560639dbf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639dbf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 -- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 msgr2=0x560639e418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639e418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 -- 172.21.15.72:0/930148518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560639dbd700 con 0x560639dd9b00 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60110b3640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639dbfad0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f60040078e0 tx=0x7f6004004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58b054f3f493 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f5feb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f5feb7fe640 1 -- 172.21.15.72:0/930148518 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6004015b30 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f5feb7fe640 1 -- 172.21.15.72:0/930148518 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6004015cd0 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ff4037040 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f5feb7fe640 1 -- 172.21.15.72:0/930148518 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6004028420 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 10 client.4617 fetch_fsmap learned FSMap version 7 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.340+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x560639db38f0 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f5feb7fe640 1 -- 172.21.15.72:0/930148518 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff4037040 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f5feb7fe640 1 -- 172.21.15.72:0/930148518 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 7) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6004030020 con 0x560639dd9b00 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 10 client.4617 fetch_fsmap finished waiting for FSMap version 7 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 10 client.4617 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 20 client.4617 populate_metadata read hostname 'smithi072' 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639e4b8d0 0x560639dba780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 4 client.4617 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 --> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560639d91ed0 con 0x560639e4b8d0 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60108b2640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639e4b8d0 0x560639dba780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f5feb7fe640 10 client.4617 ms_handle_connect on v2:172.21.15.72:6834/3798106106 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60108b2640 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639e4b8d0 0x560639dba780 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7863a0a34801dd7f server_cookie=47379ca48b3cc4b3 in_seq=0 out_seq=0 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f5feb7fe640 1 -- 172.21.15.72:0/930148518 <== mds.0 v2:172.21.15.72:6834/3798106106 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x560639d91ed0 con 0x560639e4b8d0 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f5feb7fe640 10 client.4617 handle_command_reply: tid=0 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 client.4617 shutdown 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 msgr2=0x560639dbfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639dbfad0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f60040078e0 tx=0x7f6004004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 shutdown_connections 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:6834/3798106106,v1:172.21.15.72:6836/3798106106] conn(0x560639e4b8d0 0x560639dba780 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560639dd9b00 0x560639dbfad0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560639dc0600 0x560639dbf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f5feb7fe640 0 client.4617 ms_handle_reset on v2:172.21.15.72:6834/3798106106 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 --2- 172.21.15.72:0/930148518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560639db3450 0x560639e418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.341+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 >> 172.21.15.72:0/930148518 conn(0x560639d90870 msgr2=0x560639db2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 shutdown_connections 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f60157d0740 1 -- 172.21.15.72:0/930148518 wait complete. 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f60157d0740 20 client.4617 trim_cache size 0 max 16384 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4272, 2024-03-06T07:22:02.351 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 12.274476907, 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 7, 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 21, 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 21, 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 18.903188579999998 2024-03-06T07:22:02.352 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fec06b500 msgr2=0x7f5fec06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fec06b500 0x7f5fec06d9c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4036000 tx=0x7f5ff403d040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 msgr2=0x7f600c1526d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 0x7f600c1526d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00d800 tx=0x7f5ffc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 shutdown_connections 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fec06b500 0x7f5fec06d9c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f600c1577e0 0x7f600c0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f600c157360 0x7f600c0c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 --2- 172.21.15.72:0/2461945234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f600c15e010 0x7f600c1526d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 >> 172.21.15.72:0/2461945234 conn(0x7f600c09a9f0 msgr2=0x7f600c0ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 shutdown_connections 2024-03-06T07:22:02.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.342+0000 7f6012b16640 1 -- 172.21.15.72:0/2461945234 wait complete. 2024-03-06T07:22:02.355 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-06T07:22:02.379 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-06T07:22:02.379 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-06T07:22:02.380 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:02.380 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:02.380 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:02.380 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-06T07:22:02.380 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:02.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:02.381 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-06T07:22:02.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:02.400 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-03-06T07:22:02.473 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:02.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:02.475 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:22:02.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:02.490 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-06T07:22:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:Cannot find device "ceph-brx" 2024-03-06T07:22:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:22:02.565 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-06T07:22:02.565 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:02.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.704+0000 7f0121237640 1 Processor -- start 2024-03-06T07:22:02.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.704+0000 7f0121237640 1 -- start start 2024-03-06T07:22:02.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f0121237640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f011c15fa10 0x7f011c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f0121237640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f011c0d16e0 0x7f011c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f0121237640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011c0cebe0 con 0x7f011c15bdd0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f0121237640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c0ceec0 con 0x7f011c0d16e0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f0121237640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011c0cf1a0 con 0x7f011c15fa10 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f011c0d16e0 0x7f011c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f011c0d16e0 0x7f011c0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38646/0 (socket says 172.21.15.72:38646) 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 -- 172.21.15.72:0/3602286485 learned_addr learned my addr 172.21.15.72:0/3602286485 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011a59c640 1 --2- 172.21.15.72:0/3602286485 >> v2:172.21.15.146:3300/0 conn(0x7f011c15fa10 0x7f011c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 -- 172.21.15.72:0/3602286485 >> v2:172.21.15.146:3300/0 conn(0x7f011c15fa10 msgr2=0x7f011c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 --2- 172.21.15.72:0/3602286485 >> v2:172.21.15.146:3300/0 conn(0x7f011c15fa10 0x7f011c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 -- 172.21.15.72:0/3602286485 >> v1:172.21.15.146:6789/0 conn(0x7f011c15bdd0 legacy=0x7f011c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.705+0000 7f011b59e640 1 -- 172.21.15.72:0/3602286485 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0cc8e0 con 0x7f011c0d16e0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.706+0000 7f011b59e640 1 --2- 172.21.15.72:0/3602286485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c0d1af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f011c0cdfd0 tx=0x7f011402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ff11c799d47dde server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.706+0000 7f0119d9b640 1 -- 172.21.15.72:0/3602286485 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011402ca80 con 0x7f011c0d16e0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.706+0000 7f0119d9b640 1 -- 172.21.15.72:0/3602286485 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011402cc20 con 0x7f011c0d16e0 2024-03-06T07:22:02.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.706+0000 7f0119d9b640 1 -- 172.21.15.72:0/3602286485 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01140343f0 con 0x7f011c0d16e0 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 -- 172.21.15.72:0/3602286485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 msgr2=0x7f011c0d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 --2- 172.21.15.72:0/3602286485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c0d1af0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f011c0cdfd0 tx=0x7f011402bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 -- 172.21.15.72:0/3602286485 shutdown_connections 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 --2- 172.21.15.72:0/3602286485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c0d1af0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 --2- 172.21.15.72:0/3602286485 >> v2:172.21.15.146:3300/0 conn(0x7f011c15fa10 0x7f011c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 -- 172.21.15.72:0/3602286485 >> 172.21.15.72:0/3602286485 conn(0x7f011c09a9f0 msgr2=0x7f011c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 -- 172.21.15.72:0/3602286485 shutdown_connections 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.707+0000 7f0121237640 1 -- 172.21.15.72:0/3602286485 wait complete. 2024-03-06T07:22:02.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.709+0000 7f0121237640 1 Processor -- start 2024-03-06T07:22:02.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.709+0000 7f0121237640 1 -- start start 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.710+0000 7f0121237640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.710+0000 7f0121237640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f011c15bdd0 0x7f011c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.710+0000 7f0121237640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f011c15fa10 0x7f011c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.710+0000 7f0121237640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0d2030 con 0x7f011c0d16e0 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.710+0000 7f0121237640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011c15c1b0 con 0x7f011c15bdd0 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.710+0000 7f0121237640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011c0ce7b0 con 0x7f011c15fa10 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c154c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38648/0 (socket says 172.21.15.72:38648) 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 -- 172.21.15.72:0/2297753521 learned_addr learned my addr 172.21.15.72:0/2297753521 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011b59e640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f011c15fa10 0x7f011c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011a59c640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f011c15bdd0 0x7f011c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 -- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f011c15fa10 msgr2=0x7f011c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f011c15fa10 0x7f011c15c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 -- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f011c15bdd0 msgr2=0x7f011c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f011c15bdd0 0x7f011c0cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 -- 172.21.15.72:0/2297753521 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c0c9db0 con 0x7f011c0d16e0 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011a59c640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f011c15bdd0 0x7f011c0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f011ad9d640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c154c40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f010c00c460 tx=0x7f010c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0052d0 con 0x7f011c0d16e0 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010c012630 con 0x7f011c0d16e0 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010c0054b0 con 0x7f011c0d16e0 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011c0ca090 con 0x7f011c0d16e0 2024-03-06T07:22:02.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.711+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011c0ca650 con 0x7f011c0d16e0 2024-03-06T07:22:02.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.712+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f011c04f330 con 0x7f011c0d16e0 2024-03-06T07:22:02.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.713+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f010c012a20 con 0x7f011c0d16e0 2024-03-06T07:22:02.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.713+0000 7f01037fe640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f010806b4d0 0x7f010806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:02.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.713+0000 7f011a59c640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f010806b4d0 0x7f010806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:02.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.713+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f010c089df0 con 0x7f011c0d16e0 2024-03-06T07:22:02.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.713+0000 7f011a59c640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f010806b4d0 0x7f010806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f011c0b5250 tx=0x7f0110008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:02.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.717+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f010c058c70 con 0x7f011c0d16e0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.941+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f011c06f9a0 con 0x7f011c0d16e0 2024-03-06T07:22:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.942+0000 7f01037fe640 1 -- 172.21.15.72:0/2297753521 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f010c058570 con 0x7f011c0d16e0 2024-03-06T07:22:02.944 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f010806b4d0 msgr2=0x7f010806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f010806b4d0 0x7f010806d990 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f011c0b5250 tx=0x7f0110008040 comp rx=0 tx=0).stop 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 msgr2=0x7f011c154c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c154c40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f010c00c460 tx=0x7f010c00c920 comp rx=0 tx=0).stop 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 shutdown_connections 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f010806b4d0 0x7f010806d990 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f011c15fa10 0x7f011c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f011c15bdd0 0x7f011c0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 --2- 172.21.15.72:0/2297753521 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f011c0d16e0 0x7f011c154c40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 >> 172.21.15.72:0/2297753521 conn(0x7f011c09a9f0 msgr2=0x7f011c0c63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 shutdown_connections 2024-03-06T07:22:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:02.944+0000 7f0121237640 1 -- 172.21.15.72:0/2297753521 wait complete. 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:02.957 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:02.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:02.958 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:22:02.972 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:02.972 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:02.972 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:02.972 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:02.972 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:02.972 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42672sec preferred_lft 42672sec 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:22:02.973 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:02.974 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:02.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:02.974 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:03.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.166 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:03.192 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:22:03.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.193 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:22:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:22:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:03.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.270 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:03.270 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:03.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:03.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:03.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:03.270 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:03.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.547 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:22:03.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.563 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-03-06T07:22:03.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.619 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:03.619 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:03.619 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:03.619 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-06T07:22:03.620 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:03.701 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-06T07:22:03.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.701 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:03.701 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:03.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-06T07:22:03.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:03.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-06T07:22:03.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-06T07:22:03.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-06T07:22:03.702 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:03.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:03.842 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:03.843 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:03.843 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:22:03.843 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:03.843 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:03.884 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:03.884 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:03.884 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:03.940 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:03.940 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:03.941 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:03.997 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:22:04.065 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:04.121 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:22:04.121 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:22:04.121 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:22:04.122 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:04.123 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:22:04.124 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:04.124 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:04.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:04.124 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:22:04.177 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:04.177 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:04.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:22:04.249 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:22:04.307 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:22:04.308 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:04.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:04.309 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:22:04.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:22:04.306+0000 7fc55ed7a480 -1 init, newargv = 0x55a97dfdb1a0 newargc=15 2024-03-06T07:22:04.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[46475]: starting ceph client 2024-03-06T07:22:04.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[46475]: starting fuse 2024-03-06T07:22:04.325 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:22:04.325 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:04.326 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:04.327 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:22:04.446 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.46475.asok 2024-03-06T07:22:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:04.455 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46475.asok 2024-03-06T07:22:04.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:04.456 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46475.asok status 2024-03-06T07:22:04.596 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "46475", 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4632, 2024-03-06T07:22:04.597 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 4632 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2144610456 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2144610456 2024-03-06T07:22:04.598 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.4632 192.168.0.1:0/2144610456", 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/2144610456", 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 7, 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 21, 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:22:04.599 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:22:04.605 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:04.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.707+0000 7fda3fe0e640 1 Processor -- start 2024-03-06T07:22:04.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.707+0000 7fda3fe0e640 1 -- start start 2024-03-06T07:22:04.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.708+0000 7fda3fe0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380d4760 0x7fda380b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.708+0000 7fda3fe0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda380d25d0 con 0x7fda380b4170 2024-03-06T07:22:04.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.708+0000 7fda3fe0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda380d2870 con 0x7fda380d0a80 2024-03-06T07:22:04.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.708+0000 7fda3fe0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda380d4bf0 con 0x7fda380d4760 2024-03-06T07:22:04.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.708+0000 7fda3dbaa640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fda380b4170 0x7fda380d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41382/0 (socket says 172.21.15.72:41382) 2024-03-06T07:22:04.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.708+0000 7fda3dbaa640 1 -- 172.21.15.72:0/1905008126 learned_addr learned my addr 172.21.15.72:0/1905008126 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3d3a9640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380d4760 0x7fda380b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582977280 0 0) 0x7fda380d25d0 con 0x7fda380b4170 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda24003520 con 0x7fda380b4170 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973649529 0 0) 0x7fda24003520 con 0x7fda380b4170 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380d4760 msgr2=0x7fda380b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380d4760 0x7fda380b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 >> v1:172.21.15.146:6789/0 conn(0x7fda380d0a80 legacy=0x7fda380cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda380d2b50 con 0x7fda380b4170 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fda28001670 con 0x7fda380b4170 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 >> v1:172.21.15.72:6789/0 conn(0x7fda380b4170 legacy=0x7fda380d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda24003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda24004390 0x7fda24004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda24004f80 0x7fda2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda380d4bf0 con 0x7fda240039e0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2400bf00 con 0x7fda24004f80 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2400c080 con 0x7fda24004390 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda24003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3d3a9640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda24004390 0x7fda24004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3e3ab640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda24004f80 0x7fda2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3dbaa640 1 -- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda24004390 msgr2=0x7fda24004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda24004390 0x7fda24004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3dbaa640 1 -- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda24004f80 msgr2=0x7fda2400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda24004f80 0x7fda2400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3dbaa640 1 -- 172.21.15.72:0/1905008126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24003520 con 0x7fda240039e0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.709+0000 7fda3e3ab640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda24004f80 0x7fda2400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.710+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda24003dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c1c0 tx=0x7fda24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8af048c5213a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.710+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28007cc0 con 0x7fda240039e0 2024-03-06T07:22:04.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.710+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28030070 con 0x7fda240039e0 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.710+0000 7fda3cba8640 1 -- 172.21.15.72:0/1905008126 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28005d10 con 0x7fda240039e0 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3fe0e640 1 -- 172.21.15.72:0/1905008126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 msgr2=0x7fda24003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/1905008126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda24003dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fda2800c1c0 tx=0x7fda24005830 comp rx=0 tx=0).stop 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3e3ab640 1 -- 172.21.15.72:0/1905008126 reap_dead start 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3fe0e640 1 -- 172.21.15.72:0/1905008126 shutdown_connections 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3fe0e640 1 -- 172.21.15.72:0/1905008126 >> 172.21.15.72:0/1905008126 conn(0x7fda380d3c40 msgr2=0x7fda380d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3fe0e640 1 -- 172.21.15.72:0/1905008126 shutdown_connections 2024-03-06T07:22:04.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.711+0000 7fda3fe0e640 1 -- 172.21.15.72:0/1905008126 wait complete. 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.712+0000 7fda3fe0e640 1 Processor -- start 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.712+0000 7fda3fe0e640 1 -- start start 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3fe0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda3809ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3fe0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3815e660 0x7fda380a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3fe0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380b7f00 0x7fda380b8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3fe0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda380b4dd0 con 0x7fda240039e0 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3fe0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda380ce9e0 con 0x7fda3815e660 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3fe0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda380b4f50 con 0x7fda380b7f00 2024-03-06T07:22:04.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda3809ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda3809ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38662/0 (socket says 172.21.15.72:38662) 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 -- 172.21.15.72:0/3599856843 learned_addr learned my addr 172.21.15.72:0/3599856843 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3d3a9640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3815e660 0x7fda380a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3e3ab640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380b7f00 0x7fda380b8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 -- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380b7f00 msgr2=0x7fda380b8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380b7f00 0x7fda380b8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 -- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3815e660 msgr2=0x7fda380a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3815e660 0x7fda380a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 -- 172.21.15.72:0/3599856843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda380a15b0 con 0x7fda240039e0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.713+0000 7fda3dbaa640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda3809ea30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fda280018f0 tx=0x7fda280019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.714+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28046070 con 0x7fda240039e0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.714+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda28000ed0 con 0x7fda240039e0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.714+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28007cc0 con 0x7fda240039e0 2024-03-06T07:22:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.714+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda380cbf50 con 0x7fda240039e0 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.714+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda380cc510 con 0x7fda240039e0 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.715+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3804f330 con 0x7fda240039e0 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.715+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda28007070 con 0x7fda240039e0 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.716+0000 7fda2e7fc640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda1c06b4d0 0x7fda1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.716+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda280ab5a0 con 0x7fda240039e0 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.716+0000 7fda3d3a9640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda1c06b4d0 0x7fda1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.716+0000 7fda3d3a9640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda1c06b4d0 0x7fda1c06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fda380d24d0 tx=0x7fda34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:04.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.720+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda2807a420 con 0x7fda240039e0 2024-03-06T07:22:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.935+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fda380733b0 con 0x7fda240039e0 2024-03-06T07:22:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.936+0000 7fda2e7fc640 1 -- 172.21.15.72:0/3599856843 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fda28079d20 con 0x7fda240039e0 2024-03-06T07:22:04.938 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda1c06b4d0 msgr2=0x7fda1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda1c06b4d0 0x7fda1c06d990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fda380d24d0 tx=0x7fda34008040 comp rx=0 tx=0).stop 2024-03-06T07:22:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 msgr2=0x7fda3809ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda3809ea30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fda280018f0 tx=0x7fda280019c0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 shutdown_connections 2024-03-06T07:22:04.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda1c06b4d0 0x7fda1c06d990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda380b7f00 0x7fda380b8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda3815e660 0x7fda380a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 --2- 172.21.15.72:0/3599856843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda240039e0 0x7fda3809ea30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:04.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 >> 172.21.15.72:0/3599856843 conn(0x7fda380d3c40 msgr2=0x7fda380b7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:04.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 shutdown_connections 2024-03-06T07:22:04.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:04.938+0000 7fda3fe0e640 1 -- 172.21.15.72:0/3599856843 wait complete. 2024-03-06T07:22:04.950 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:04.950 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:04.950 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:22:04.951 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-06T07:22:04.951 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:04.951 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:04.951 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:04.951 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:04.951 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:04.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:04.951 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:22:04.966 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:04.966 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:04.966 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:04.967 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42670sec preferred_lft 42670sec 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link tentative 2024-03-06T07:22:04.968 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:04.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:04.969 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:05.032 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:22:05.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.034 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:22:05.088 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:22:05.089 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:05.089 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:05.089 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:05.089 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:05.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.089 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:05.090 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:05.090 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:05.090 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:05.090 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:05.090 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:05.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.183 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:22:05.197 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:05.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.198 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-03-06T07:22:05.252 INFO:teuthology.orchestra.run.smithi072.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-06T07:22:05.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.252 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:05.252 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:05.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:05.252 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-06T07:22:05.252 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:05.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.333 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:05.400 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 2a:07:d1:1f:a7:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::2807:d1ff:fe1f:a7dd/64 scope link tentative 2024-03-06T07:22:05.401 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.402 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:05.469 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 2a:07:d1:1f:a7:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:05.470 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:05.470 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.470 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::2807:d1ff:fe1f:a7dd/64 scope link tentative 2024-03-06T07:22:05.470 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:05.471 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-06T07:22:05.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.471 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:05.471 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:05.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-06T07:22:05.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:05.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-06T07:22:05.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-06T07:22:05.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-06T07:22:05.472 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:05.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.597 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:05.597 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:05.597 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.1 up 2024-03-06T07:22:05.597 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:05.597 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:05.638 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-06T07:22:05.639 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:05.639 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:05.696 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-06T07:22:05.697 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:05.697 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:05.750 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:22:05.817 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:05.872 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:22:05.873 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:22:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:05.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:05.875 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:22:05.927 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:22:05.927 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-06T07:22:05.927 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-06T07:22:05.970 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:22:05.996 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:06.051 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:22:06.051 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:22:06.052 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:22:06.053 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:06.054 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:06.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.055 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:22:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:2024-03-06T07:22:06.054+0000 7efdcb160480 -1 init, newargv = 0x55fcd2969810 newargc=15 2024-03-06T07:22:06.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[46969]: starting ceph client 2024-03-06T07:22:06.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[46969]: starting fuse 2024-03-06T07:22:06.072 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:22:06.072 INFO:teuthology.orchestra.run.smithi072.stdout:44 2024-03-06T07:22:06.072 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-06T07:22:06.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:06.072 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:22:06.072 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:22:06.072 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:06.073 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.2")) 2024-03-06T07:22:06.074 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:22:06.192 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.2.46969.asok 2024-03-06T07:22:06.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:06.202 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.46969.asok 2024-03-06T07:22:06.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.202 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.46969.asok status 2024-03-06T07:22:06.339 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "2", 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "46969", 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:22:06.340 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4647, 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 4647 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1852591692 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:06.341 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1852591692 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.4647 192.168.0.2:0/1852591692", 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.2:0/1852591692", 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 7, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 21, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:22:06.342 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:22:06.343 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:22:06.348 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:06.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.449+0000 7ffaf478a640 1 Processor -- start 2024-03-06T07:22:06.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.449+0000 7ffaf478a640 1 -- start start 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf478a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffaec0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf478a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffaec0cf830 0x7ffaec0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf478a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b3430 con 0x7ffaec0d27f0 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf478a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaec0d2c50 con 0x7ffaec0cf830 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf478a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0b31e0 con 0x7ffaec15dfe0 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf2d27640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ffaec0d27f0 0x7ffaec0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41386/0 (socket says 172.21.15.72:41386) 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf2d27640 1 -- 172.21.15.72:0/1154733062 learned_addr learned my addr 172.21.15.72:0/1154733062 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:06.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1d25640 1 --2- 172.21.15.72:0/1154733062 >> v2:172.21.15.146:3300/0 conn(0x7ffaec0cf830 0x7ffaec0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffaec0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613312129 0 0) 0x7ffaec0b3430 con 0x7ffaec0d27f0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003520 con 0x7ffaec0d27f0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193912272 0 0) 0x7ffad8003520 con 0x7ffaec0d27f0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 msgr2=0x7ffaec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffaec0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 >> v2:172.21.15.146:3300/0 conn(0x7ffaec0cf830 msgr2=0x7ffaec0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.451+0000 7ffaf1524640 1 --2- 172.21.15.72:0/1154733062 >> v2:172.21.15.146:3300/0 conn(0x7ffaec0cf830 0x7ffaec0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0b3710 con 0x7ffaec0d27f0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ffae40019f0 con 0x7ffaec0d27f0 2024-03-06T07:22:06.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 >> v1:172.21.15.72:6789/0 conn(0x7ffaec0d27f0 legacy=0x7ffaec0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad80039b0 0x7ffad8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffad8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad8004d40 0x7ffad80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0b31e0 con 0x7ffad80039b0 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad800b9c0 con 0x7ffad8004d40 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad800bb40 con 0x7ffaec15dfe0 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad80039b0 0x7ffad8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1d25640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffad8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2d27640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad8004d40 0x7ffad80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2526640 1 -- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 msgr2=0x7ffad8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffad8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2526640 1 -- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad8004d40 msgr2=0x7ffad80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad8004d40 0x7ffad80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2526640 1 -- 172.21.15.72:0/1154733062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad8003520 con 0x7ffad80039b0 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf1d25640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffaec15dfe0 0x7ffad8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.452+0000 7ffaf2d27640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad8004d40 0x7ffad80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad80039b0 0x7ffad8003dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009ac0 tx=0x7ffad8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28aa174dcb1811e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8039070 con 0x7ffad80039b0 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae802c6b0 con 0x7ffad80039b0 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf1524640 1 -- 172.21.15.72:0/1154733062 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae8009690 con 0x7ffad80039b0 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf478a640 1 -- 172.21.15.72:0/1154733062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad80039b0 msgr2=0x7ffad8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1154733062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad80039b0 0x7ffad8003dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffae8009ac0 tx=0x7ffad8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf2d27640 1 -- 172.21.15.72:0/1154733062 reap_dead start 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf478a640 1 -- 172.21.15.72:0/1154733062 shutdown_connections 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf478a640 1 -- 172.21.15.72:0/1154733062 >> 172.21.15.72:0/1154733062 conn(0x7ffaec0d3cd0 msgr2=0x7ffaec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf478a640 1 -- 172.21.15.72:0/1154733062 shutdown_connections 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.453+0000 7ffaf478a640 1 -- 172.21.15.72:0/1154733062 wait complete. 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.455+0000 7ffaf478a640 1 Processor -- start 2024-03-06T07:22:06.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.455+0000 7ffaf478a640 1 -- start start 2024-03-06T07:22:06.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf478a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad80039b0 0x7ffaec15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf478a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 0x7ffaec0b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf478a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaec0b85a0 0x7ffaec09eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf478a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0d1c70 con 0x7ffaec0cada0 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf478a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0b3d60 con 0x7ffaec0b85a0 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf478a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaec0d1df0 con 0x7ffad80039b0 2024-03-06T07:22:06.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf1d25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 0x7ffaec0b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf1d25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 0x7ffaec0b8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38690/0 (socket says 172.21.15.72:38690) 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf1d25640 1 -- 172.21.15.72:0/1137727197 learned_addr learned my addr 172.21.15.72:0/1137727197 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf2d27640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaec0b85a0 0x7ffaec09eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.456+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad80039b0 0x7ffaec15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf1d25640 1 -- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad80039b0 msgr2=0x7ffaec15c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf1d25640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad80039b0 0x7ffaec15c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf1d25640 1 -- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaec0b85a0 msgr2=0x7ffaec09eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf1d25640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaec0b85a0 0x7ffaec09eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf1d25640 1 -- 172.21.15.72:0/1137727197 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0cb4f0 con 0x7ffaec0cada0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf1d25640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 0x7ffaec0b8060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffadc011660 tx=0x7ffadc011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc00f9c0 con 0x7ffaec0cada0 2024-03-06T07:22:06.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffadc00fb60 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec0cb7d0 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffadc015460 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.457+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec0cbd90 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.458+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec04f330 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.458+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffadc015600 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.459+0000 7ffae2ffd640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.459+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffadc08ac30 con 0x7ffaec0cada0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.459+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:06.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.459+0000 7ffaf2526640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffad006b4d0 0x7ffad006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffae8036000 tx=0x7ffae8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:06.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.463+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffadc019030 con 0x7ffaec0cada0 2024-03-06T07:22:06.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.678+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffaec0b8c00 con 0x7ffaec0cada0 2024-03-06T07:22:06.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.678+0000 7ffae2ffd640 1 -- 172.21.15.72:0/1137727197 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffadc059a20 con 0x7ffaec0cada0 2024-03-06T07:22:06.683 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:06.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffad006b4d0 msgr2=0x7ffad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffad006b4d0 0x7ffad006d990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffae8036000 tx=0x7ffae8030000 comp rx=0 tx=0).stop 2024-03-06T07:22:06.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 msgr2=0x7ffaec0b8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 0x7ffaec0b8060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ffadc011660 tx=0x7ffadc011b20 comp rx=0 tx=0).stop 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 shutdown_connections 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffad006b4d0 0x7ffad006d990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffaec0b85a0 0x7ffaec09eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffaec0cada0 0x7ffaec0b8060 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 --2- 172.21.15.72:0/1137727197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad80039b0 0x7ffaec15c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 >> 172.21.15.72:0/1137727197 conn(0x7ffaec0d3cd0 msgr2=0x7ffaec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 shutdown_connections 2024-03-06T07:22:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:06.681+0000 7ffaf478a640 1 -- 172.21.15.72:0/1137727197 wait complete. 2024-03-06T07:22:06.693 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:06.694 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:06.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.695 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-03-06T07:22:06.713 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:06.713 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:06.713 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:06.713 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:06.713 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42669sec preferred_lft 42669sec 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-03-06T07:22:06.714 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:06.715 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-06T07:22:06.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-06T07:22:06.715 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:06.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.868 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:06.895 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-06T07:22:06.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.897 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:06.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:06.967 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:06.967 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:06.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:06.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:06.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:06.967 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:07.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.256 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-06T07:22:07.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.271 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-03-06T07:22:07.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.326 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:07.327 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:07.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:07.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-06T07:22:07.327 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:07.409 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-06T07:22:07.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.409 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:07.409 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:07.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-06T07:22:07.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:07.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-06T07:22:07.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-06T07:22:07.410 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-06T07:22:07.410 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:07.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.546 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:07.546 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:07.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-03-06T07:22:07.546 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-03-06T07:22:07.547 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:07.588 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-06T07:22:07.588 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:07.588 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:07.643 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-06T07:22:07.644 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:07.644 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:07.698 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-03-06T07:22:07.766 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:07.822 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:07.823 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:07.824 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:07.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:07.825 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-03-06T07:22:07.877 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:07.877 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-06T07:22:07.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-03-06T07:22:07.946 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:08.002 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:08.003 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:08.004 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:08.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.005 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-03-06T07:22:08.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-03-06T07:22:08.003+0000 7fa5ff638480 -1 init, newargv = 0x55bd089c1580 newargc=15 2024-03-06T07:22:08.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44310]: starting ceph client 2024-03-06T07:22:08.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44310]: starting fuse 2024-03-06T07:22:08.020 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-03-06T07:22:08.020 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:08.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:08.020 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-03-06T07:22:08.020 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-03-06T07:22:08.020 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:08.021 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-03-06T07:22:08.022 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-03-06T07:22:08.142 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.44310.asok 2024-03-06T07:22:08.144 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:22:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:08.152 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44310.asok 2024-03-06T07:22:08.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.152 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44310.asok status 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-06T07:22:08.289 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "44310", 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4448, 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4448 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:08.290 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1795198754 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1795198754 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4448 192.168.0.1:0/1795198754", 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/1795198754", 2024-03-06T07:22:08.291 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-03-06T07:22:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 7, 2024-03-06T07:22:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 21, 2024-03-06T07:22:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-03-06T07:22:08.292 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-03-06T07:22:08.292 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-06T07:22:08.297 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:08.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.396+0000 7ff6e8e22640 1 Processor -- start 2024-03-06T07:22:08.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.396+0000 7ff6e8e22640 1 -- start start 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e8e22640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff6e40cfff0 0x7ff6e40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e8e22640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff6e40cc8b0 0x7ff6e40ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e8e22640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e4161980 con 0x7ff6e415bb20 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e8e22640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40d2500 con 0x7ff6e40cc8b0 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e8e22640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e4161c60 con 0x7ff6e40cfff0 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff6e40cc8b0 0x7ff6e40ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff6e40cc8b0 0x7ff6e40ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38698/0 (socket says 172.21.15.72:38698) 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6e2d9d640 1 -- 172.21.15.72:0/3960634293 learned_addr learned my addr 172.21.15.72:0/3960634293 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.397+0000 7ff6d9d9b640 1 --2- 172.21.15.72:0/3960634293 >> v2:172.21.15.146:3300/0 conn(0x7ff6e40cfff0 0x7ff6e40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e2d9d640 1 -- 172.21.15.72:0/3960634293 >> v2:172.21.15.146:3300/0 conn(0x7ff6e40cfff0 msgr2=0x7ff6e40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e2d9d640 1 --2- 172.21.15.72:0/3960634293 >> v2:172.21.15.146:3300/0 conn(0x7ff6e40cfff0 0x7ff6e40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e2d9d640 1 -- 172.21.15.72:0/3960634293 >> v1:172.21.15.146:6789/0 conn(0x7ff6e415bb20 legacy=0x7ff6e40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e2d9d640 1 -- 172.21.15.72:0/3960634293 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e4161f40 con 0x7ff6e40cc8b0 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e2d9d640 1 --2- 172.21.15.72:0/3960634293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cc8b0 0x7ff6e40ccce0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4071290 tx=0x7ff6dc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddaf569ace3a2b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e1d9b640 1 -- 172.21.15.72:0/3960634293 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc02cb20 con 0x7ff6e40cc8b0 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e1d9b640 1 -- 172.21.15.72:0/3960634293 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc02ccc0 con 0x7ff6e40cc8b0 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.398+0000 7ff6e1d9b640 1 -- 172.21.15.72:0/3960634293 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc034610 con 0x7ff6e40cc8b0 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3960634293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cc8b0 msgr2=0x7ff6e40ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3960634293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cc8b0 0x7ff6e40ccce0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4071290 tx=0x7ff6dc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3960634293 shutdown_connections 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3960634293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cc8b0 0x7ff6e40ccce0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3960634293 >> v2:172.21.15.146:3300/0 conn(0x7ff6e40cfff0 0x7ff6e40d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3960634293 >> 172.21.15.72:0/3960634293 conn(0x7ff6e409a9f0 msgr2=0x7ff6e409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3960634293 shutdown_connections 2024-03-06T07:22:08.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.399+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3960634293 wait complete. 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.400+0000 7ff6e8e22640 1 Processor -- start 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 -- start start 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6e40cc8b0 0x7ff6e4154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 0x7ff6e406cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6e415bb20 0x7ff6e406d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40d1980 con 0x7ff6e40cfff0 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e4162500 con 0x7ff6e415bb20 2024-03-06T07:22:08.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6e8e22640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40d1b00 con 0x7ff6e40cc8b0 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6d9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 0x7ff6e406cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6d9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 0x7ff6e406cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38714/0 (socket says 172.21.15.72:38714) 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.401+0000 7ff6d9d9b640 1 -- 172.21.15.72:0/3848605676 learned_addr learned my addr 172.21.15.72:0/3848605676 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6e2d9d640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6e415bb20 0x7ff6e406d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6e259c640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6e40cc8b0 0x7ff6e4154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6d9d9b640 1 -- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6e40cc8b0 msgr2=0x7ff6e4154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6d9d9b640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6e40cc8b0 0x7ff6e4154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6d9d9b640 1 -- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6e415bb20 msgr2=0x7ff6e406d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6d9d9b640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6e415bb20 0x7ff6e406d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6d9d9b640 1 -- 172.21.15.72:0/3848605676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e406d490 con 0x7ff6e40cfff0 2024-03-06T07:22:08.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6e259c640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6e40cc8b0 0x7ff6e4154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6d9d9b640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 0x7ff6e406cf50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00e870 tx=0x7ff6cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc017070 con 0x7ff6e40cfff0 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc00c940 con 0x7ff6e40cfff0 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e415a960 con 0x7ff6e40cfff0 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc00cd30 con 0x7ff6e40cfff0 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.402+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e40cada0 con 0x7ff6e40cfff0 2024-03-06T07:22:08.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.403+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e40ca8f0 con 0x7ff6e40cfff0 2024-03-06T07:22:08.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.404+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff6cc01a070 con 0x7ff6e40cfff0 2024-03-06T07:22:08.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.404+0000 7ff6db7fe640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff6c406b4d0 0x7ff6c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:08.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.404+0000 7ff6e259c640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff6c406b4d0 0x7ff6c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:08.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.404+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6cc08a070 con 0x7ff6e40cfff0 2024-03-06T07:22:08.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.405+0000 7ff6e259c640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff6c406b4d0 0x7ff6c406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40015e0 tx=0x7ff6d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:08.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.409+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff6cc058f00 con 0x7ff6e40cfff0 2024-03-06T07:22:08.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.618+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff6e404f330 con 0x7ff6e40cfff0 2024-03-06T07:22:08.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.618+0000 7ff6db7fe640 1 -- 172.21.15.72:0/3848605676 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff6cc058800 con 0x7ff6e40cfff0 2024-03-06T07:22:08.621 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:08.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff6c406b4d0 msgr2=0x7ff6c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff6c406b4d0 0x7ff6c406d990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6d40015e0 tx=0x7ff6d4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 msgr2=0x7ff6e406cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 0x7ff6e406cf50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc00e870 tx=0x7ff6cc00ed30 comp rx=0 tx=0).stop 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 shutdown_connections 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff6c406b4d0 0x7ff6c406d990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6e415bb20 0x7ff6e406d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6e40cfff0 0x7ff6e406cf50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 --2- 172.21.15.72:0/3848605676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6e40cc8b0 0x7ff6e4154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 >> 172.21.15.72:0/3848605676 conn(0x7ff6e409a9f0 msgr2=0x7ff6e415c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 shutdown_connections 2024-03-06T07:22:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:08.620+0000 7ff6e8e22640 1 -- 172.21.15.72:0/3848605676 wait complete. 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:08.633 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:08.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.634 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-03-06T07:22:08.650 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42667sec preferred_lft 42667sec 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 8e:0b:e6:9c:c6:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::609c:e4ff:fe12:c788/64 scope link 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.651 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 8e:0b:e6:9c:c6:f5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::8c0b:e6ff:fe9c:c6f5/64 scope link tentative 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:08.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.652 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:08.716 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-03-06T07:22:08.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.717 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-03-06T07:22:08.773 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-03-06T07:22:08.773 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:08.773 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:08.773 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:08.774 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:08.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.774 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:08.774 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:08.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:08.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:08.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:08.774 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:08.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.869 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-06T07:22:08.884 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:22:08.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.885 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-03-06T07:22:08.939 INFO:teuthology.orchestra.run.smithi146.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-06T07:22:08.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:08.940 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:08.940 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:08.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:08.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-06T07:22:08.940 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:09.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.089 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:09.090 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 9a:19:a0:62:cd:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:09.090 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:09.090 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.090 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::9819:a0ff:fe62:cdca/64 scope link tentative 2024-03-06T07:22:09.090 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.091 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 9a:19:a0:62:cd:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::9819:a0ff:fe62:cdca/64 scope link 2024-03-06T07:22:09.158 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:09.159 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-06T07:22:09.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-06T07:22:09.160 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:09.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.286 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-03-06T07:22:09.286 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-03-06T07:22:09.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.1 up 2024-03-06T07:22:09.286 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.1 master ceph-brx 2024-03-06T07:22:09.286 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-03-06T07:22:09.328 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-06T07:22:09.328 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:09.328 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:09.387 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-06T07:22:09.387 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:09.387 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:09.441 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-03-06T07:22:09.509 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:09.565 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:09.566 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:09.567 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:09.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.568 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-03-06T07:22:09.620 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-03-06T07:22:09.620 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-06T07:22:09.620 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-06T07:22:09.663 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-03-06T07:22:09.689 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:09.745 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-03-06T07:22:09.745 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-03-06T07:22:09.745 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-03-06T07:22:09.745 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-03-06T07:22:09.745 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/bpf 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-03-06T07:22:09.746 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-03-06T07:22:09.747 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-03-06T07:22:09.748 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:09.748 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:22:09.748 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:09.748 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:09.748 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:22:09.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.748 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-03-06T07:22:09.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi146.stderr:2024-03-06T07:22:09.746+0000 7fd7c5783480 -1 init, newargv = 0x55e96365e490 newargc=15 2024-03-06T07:22:09.749 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi146.stderr:ceph-fuse[44762]: starting ceph client 2024-03-06T07:22:09.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi146.stderr:ceph-fuse[44762]: starting fuse 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-03-06T07:22:09.764 INFO:teuthology.orchestra.run.smithi146.stdout:44 2024-03-06T07:22:09.764 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-06T07:22:09.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:09.765 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.3")) 2024-03-06T07:22:09.766 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-03-06T07:22:09.885 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.3.44762.asok 2024-03-06T07:22:09.888 INFO:teuthology.orchestra.run.smithi146.stderr:2024-03-06T07:22:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:09.895 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44762.asok 2024-03-06T07:22:09.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:09.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44762.asok status 2024-03-06T07:22:10.031 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "3", 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "44762", 2024-03-06T07:22:10.032 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4552, 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4552 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-03-06T07:22:10.033 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3771208465 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3771208465 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4552 192.168.0.2:0/3771208465", 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.2:0/3771208465", 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-03-06T07:22:10.034 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 7, 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 21, 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-03-06T07:22:10.035 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-03-06T07:22:10.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.040 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:22:10.040 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:10.055 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:22:10.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:10.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.056 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:10.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.125 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:22:10.125 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:10.181 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:22:10.182 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:10.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.182 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:10.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.253 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-03-06T07:22:10.254 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:10.269 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-03-06T07:22:10.269 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:10.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.269 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:10.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.337 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-03-06T07:22:10.337 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:10.393 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-03-06T07:22:10.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:10.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:10.393 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:10.461 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-06T07:22:10.477 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:10.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.575+0000 7f0473671640 1 Processor -- start 2024-03-06T07:22:10.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0473671640 1 -- start start 2024-03-06T07:22:10.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0473671640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c15dfe0 0x7f046c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0473671640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f046c0cf830 0x7f046c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0473671640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c0b3430 con 0x7f046c0d27f0 2024-03-06T07:22:10.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0473671640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046c0d2c50 con 0x7f046c0cf830 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0473671640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0b31e0 con 0x7f046c15dfe0 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0470c0c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f046c0cf830 0x7f046c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0471c0e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f046c0d27f0 0x7f046c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41392/0 (socket says 172.21.15.72:41392) 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0470c0c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f046c0cf830 0x7f046c0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38724/0 (socket says 172.21.15.72:38724) 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.576+0000 7f0471c0e640 1 -- 172.21.15.72:0/1770546716 learned_addr learned my addr 172.21.15.72:0/1770546716 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.577+0000 7f047140d640 1 --2- 172.21.15.72:0/1770546716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c15dfe0 0x7f046c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.577+0000 7f0470c0c640 1 -- 172.21.15.72:0/1770546716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c15dfe0 msgr2=0x7f046c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.577+0000 7f0470c0c640 1 --2- 172.21.15.72:0/1770546716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c15dfe0 0x7f046c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.577+0000 7f0470c0c640 1 -- 172.21.15.72:0/1770546716 >> v1:172.21.15.72:6789/0 conn(0x7f046c0d27f0 legacy=0x7f046c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.577+0000 7f0463fff640 1 -- 172.21.15.72:0/1770546716 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678734847 0 0) 0x7f046c0b3430 con 0x7f046c0d27f0 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.577+0000 7f0470c0c640 1 -- 172.21.15.72:0/1770546716 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c0b3710 con 0x7f046c0cf830 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.578+0000 7f0470c0c640 1 --2- 172.21.15.72:0/1770546716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0d22b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f045c0097a0 tx=0x7f045c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=763e258bfb2f4000 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.578+0000 7f0463fff640 1 -- 172.21.15.72:0/1770546716 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c035020 con 0x7f046c0cf830 2024-03-06T07:22:10.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.578+0000 7f0463fff640 1 -- 172.21.15.72:0/1770546716 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c031830 con 0x7f046c0cf830 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.578+0000 7f0463fff640 1 -- 172.21.15.72:0/1770546716 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c031c00 con 0x7f046c0cf830 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.578+0000 7f0473671640 1 -- 172.21.15.72:0/1770546716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 msgr2=0x7f046c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 --2- 172.21.15.72:0/1770546716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0d22b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f045c0097a0 tx=0x7f045c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 -- 172.21.15.72:0/1770546716 shutdown_connections 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 --2- 172.21.15.72:0/1770546716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0d22b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 --2- 172.21.15.72:0/1770546716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c15dfe0 0x7f046c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 -- 172.21.15.72:0/1770546716 >> 172.21.15.72:0/1770546716 conn(0x7f046c0d3cd0 msgr2=0x7f046c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 -- 172.21.15.72:0/1770546716 shutdown_connections 2024-03-06T07:22:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.579+0000 7f0473671640 1 -- 172.21.15.72:0/1770546716 wait complete. 2024-03-06T07:22:10.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.580+0000 7f0473671640 1 Processor -- start 2024-03-06T07:22:10.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.580+0000 7f0473671640 1 -- start start 2024-03-06T07:22:10.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c0d27f0 0x7f046c0b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f046c15dfe0 0x7f046c150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0d1c70 con 0x7f046c0cf830 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0b3d60 con 0x7f046c15dfe0 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0d1df0 con 0x7f046c0d27f0 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38734/0 (socket says 172.21.15.72:38734) 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 -- 172.21.15.72:0/4141792368 learned_addr learned my addr 172.21.15.72:0/4141792368 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0470c0c640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c0d27f0 0x7f046c0b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 -- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c0d27f0 msgr2=0x7f046c0b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0471c0e640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f046c15dfe0 0x7f046c150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c0d27f0 0x7f046c0b8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 -- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f046c15dfe0 msgr2=0x7f046c150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f046c15dfe0 0x7f046c150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 -- 172.21.15.72:0/4141792368 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c15a570 con 0x7f046c0cf830 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f047140d640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0b68d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f046800d5f0 tx=0x7f046800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04680125f0 con 0x7f046c0cf830 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0468019070 con 0x7f046c0cf830 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c0cdc80 con 0x7f046c0cf830 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.582+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04680129d0 con 0x7f046c0cf830 2024-03-06T07:22:10.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.583+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c0b95a0 con 0x7f046c0cf830 2024-03-06T07:22:10.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.584+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0468012b70 con 0x7f046c0cf830 2024-03-06T07:22:10.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.584+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c04f330 con 0x7f046c0cf830 2024-03-06T07:22:10.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.584+0000 7f0461ffb640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f045006b4d0 0x7f045006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.584+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f046808a630 con 0x7f046c0cf830 2024-03-06T07:22:10.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.584+0000 7f0470c0c640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f045006b4d0 0x7f045006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.585+0000 7f0470c0c640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f045006b4d0 0x7f045006d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f045c036000 tx=0x7f045c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.589+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f046c0cdc80 con 0x7f046c0cf830 2024-03-06T07:22:10.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.766+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f046c0b09c0 con 0x7f046c0cf830 2024-03-06T07:22:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.767+0000 7f0461ffb640 1 -- 172.21.15.72:0/4141792368 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f0468059490 con 0x7f046c0cf830 2024-03-06T07:22:10.769 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:10.770 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f045006b4d0 msgr2=0x7f045006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f045006b4d0 0x7f045006d990 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f045c036000 tx=0x7f045c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 msgr2=0x7f046c0b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0b68d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f046800d5f0 tx=0x7f046800dab0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 shutdown_connections 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f045006b4d0 0x7f045006d990 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f046c15dfe0 0x7f046c150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f046c0d27f0 0x7f046c0b8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 --2- 172.21.15.72:0/4141792368 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f046c0cf830 0x7f046c0b68d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 >> 172.21.15.72:0/4141792368 conn(0x7f046c0d3cd0 msgr2=0x7f046c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 shutdown_connections 2024-03-06T07:22:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.769+0000 7f0473671640 1 -- 172.21.15.72:0/4141792368 wait complete. 2024-03-06T07:22:10.785 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:10.786 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.887+0000 7f37da39a640 1 Processor -- start 2024-03-06T07:22:10.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.887+0000 7f37da39a640 1 -- start start 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.888+0000 7f37da39a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f37d415dd00 0x7f37d406e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.888+0000 7f37da39a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 0x7f37d40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.888+0000 7f37da39a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d4161bd0 con 0x7f37d415e0e0 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.888+0000 7f37da39a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d4070220 con 0x7f37d415dd00 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.888+0000 7f37da39a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d4161e90 con 0x7f37d406b6c0 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 0x7f37d40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d3fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f37d415dd00 0x7f37d406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 0x7f37d40b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:43112/0 (socket says 172.21.15.72:43112) 2024-03-06T07:22:10.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f37d415e0e0 0x7f37d406afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:46110/0 (socket says 172.21.15.72:46110) 2024-03-06T07:22:10.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 -- 172.21.15.72:0/3228494915 learned_addr learned my addr 172.21.15.72:0/3228494915 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:10.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d2ffd640 1 -- 172.21.15.72:0/3228494915 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997545079 0 0) 0x7f37d4161bd0 con 0x7f37d415e0e0 2024-03-06T07:22:10.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 -- 172.21.15.72:0/3228494915 >> v2:172.21.15.146:3300/0 conn(0x7f37d415dd00 msgr2=0x7f37d406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:10.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 --2- 172.21.15.72:0/3228494915 >> v2:172.21.15.146:3300/0 conn(0x7f37d415dd00 0x7f37d406e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 -- 172.21.15.72:0/3228494915 >> v1:172.21.15.146:6789/0 conn(0x7f37d415e0e0 legacy=0x7f37d406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.889+0000 7f37d8937640 1 -- 172.21.15.72:0/3228494915 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d4162170 con 0x7f37d406b6c0 2024-03-06T07:22:10.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37d8937640 1 --2- 172.21.15.72:0/3228494915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 0x7f37d40b42d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009e50 tx=0x7f37cc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d557109ab8ad8921 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37d2ffd640 1 -- 172.21.15.72:0/3228494915 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc02cb00 con 0x7f37d406b6c0 2024-03-06T07:22:10.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37d2ffd640 1 -- 172.21.15.72:0/3228494915 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37cc02cca0 con 0x7f37d406b6c0 2024-03-06T07:22:10.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37d2ffd640 1 -- 172.21.15.72:0/3228494915 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37cc034610 con 0x7f37d406b6c0 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37da39a640 1 -- 172.21.15.72:0/3228494915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 msgr2=0x7f37d40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37da39a640 1 --2- 172.21.15.72:0/3228494915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 0x7f37d40b42d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f37cc009e50 tx=0x7f37cc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37da39a640 1 -- 172.21.15.72:0/3228494915 shutdown_connections 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37da39a640 1 --2- 172.21.15.72:0/3228494915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d406b6c0 0x7f37d40b42d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37da39a640 1 --2- 172.21.15.72:0/3228494915 >> v2:172.21.15.146:3300/0 conn(0x7f37d415dd00 0x7f37d406e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.890+0000 7f37da39a640 1 -- 172.21.15.72:0/3228494915 >> 172.21.15.72:0/3228494915 conn(0x7f37d40b2660 msgr2=0x7f37d40b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.891+0000 7f37da39a640 1 -- 172.21.15.72:0/3228494915 shutdown_connections 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.891+0000 7f37da39a640 1 -- 172.21.15.72:0/3228494915 wait complete. 2024-03-06T07:22:10.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.892+0000 7f37da39a640 1 Processor -- start 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.892+0000 7f37da39a640 1 -- start start 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 0x7f37d41522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d415dd00 0x7f37d4154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37d41546d0 0x7f37d4154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40b0180 con 0x7f37d406b6c0 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40b0000 con 0x7f37d41546d0 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d40b0300 con 0x7f37d415dd00 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 0x7f37d41522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 0x7f37d41522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38740/0 (socket says 172.21.15.72:38740) 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 -- 172.21.15.72:0/2977137212 learned_addr learned my addr 172.21.15.72:0/2977137212 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d8937640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37d41546d0 0x7f37d4154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d37fe640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d415dd00 0x7f37d4154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 -- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d415dd00 msgr2=0x7f37d4154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d415dd00 0x7f37d4154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 -- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37d41546d0 msgr2=0x7f37d4154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37d41546d0 0x7f37d4154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 -- 172.21.15.72:0/2977137212 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d41555f0 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d3fff640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 0x7f37d41522e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00e9e0 tx=0x7f37bc00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc017070 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc00c970 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d41533d0 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.893+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc00cd60 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.894+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d4153c70 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.895+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d41535a0 con 0x7f37d406b6c0 2024-03-06T07:22:10.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.895+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37bc01a070 con 0x7f37d406b6c0 2024-03-06T07:22:10.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.895+0000 7f37d0ff9640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37b406b4d0 0x7f37b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:10.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.896+0000 7f37d37fe640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37b406b4d0 0x7f37b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:10.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.896+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37bc08a1c0 con 0x7f37d406b6c0 2024-03-06T07:22:10.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.896+0000 7f37d37fe640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37b406b4d0 0x7f37b406d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f37d406bfd0 tx=0x7f37c4007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:10.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:10.900+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37bc058970 con 0x7f37d406b6c0 2024-03-06T07:22:11.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.109+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37d4068ff0 con 0x7f37d406b6c0 2024-03-06T07:22:11.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.110+0000 7f37d0ff9640 1 -- 172.21.15.72:0/2977137212 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37bc05ca10 con 0x7f37d406b6c0 2024-03-06T07:22:11.112 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:11.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37b406b4d0 msgr2=0x7f37b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37b406b4d0 0x7f37b406d990 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f37d406bfd0 tx=0x7f37c4007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:11.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 msgr2=0x7f37d41522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 0x7f37d41522e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00e9e0 tx=0x7f37bc00eea0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 shutdown_connections 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37b406b4d0 0x7f37b406d990 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37d41546d0 0x7f37d4154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37d415dd00 0x7f37d4154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 --2- 172.21.15.72:0/2977137212 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37d406b6c0 0x7f37d41522e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 >> 172.21.15.72:0/2977137212 conn(0x7f37d40b2660 msgr2=0x7f37d40ad860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 shutdown_connections 2024-03-06T07:22:11.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.112+0000 7f37da39a640 1 -- 172.21.15.72:0/2977137212 wait complete. 2024-03-06T07:22:11.127 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:11.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.228+0000 7fd3fb70f640 1 Processor -- start 2024-03-06T07:22:11.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.228+0000 7fd3fb70f640 1 -- start start 2024-03-06T07:22:11.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3fb70f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3f406fd10 0x7fd3f40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3fb70f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3f40d0af0 0x7fd3f40ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3fb70f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f40cd1e0 con 0x7fd3f415c050 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3fb70f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f40cd4b0 con 0x7fd3f40d0af0 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3fb70f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f40cd790 con 0x7fd3f406fd10 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3f40d0af0 0x7fd3f40ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3f40d0af0 0x7fd3f40ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38748/0 (socket says 172.21.15.72:38748) 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 -- 172.21.15.72:0/2673484299 learned_addr learned my addr 172.21.15.72:0/2673484299 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f8caa640 1 --2- 172.21.15.72:0/2673484299 >> v2:172.21.15.146:3300/0 conn(0x7fd3f406fd10 0x7fd3f40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 -- 172.21.15.72:0/2673484299 >> v2:172.21.15.146:3300/0 conn(0x7fd3f406fd10 msgr2=0x7fd3f40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 --2- 172.21.15.72:0/2673484299 >> v2:172.21.15.146:3300/0 conn(0x7fd3f406fd10 0x7fd3f40cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 -- 172.21.15.72:0/2673484299 >> v1:172.21.15.146:6789/0 conn(0x7fd3f415c050 legacy=0x7fd3f406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f9cac640 1 -- 172.21.15.72:0/2673484299 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40cda70 con 0x7fd3f40d0af0 2024-03-06T07:22:11.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.229+0000 7fd3f8caa640 1 --2- 172.21.15.72:0/2673484299 >> v2:172.21.15.146:3300/0 conn(0x7fd3f406fd10 0x7fd3f40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.230+0000 7fd3f9cac640 1 --2- 172.21.15.72:0/2673484299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40ccca0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40b7dc0 tx=0x7fd3ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5de140ef678e533 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.230+0000 7fd3ebfff640 1 -- 172.21.15.72:0/2673484299 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec02ca80 con 0x7fd3f40d0af0 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.230+0000 7fd3ebfff640 1 -- 172.21.15.72:0/2673484299 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec02cc20 con 0x7fd3f40d0af0 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.230+0000 7fd3ebfff640 1 -- 172.21.15.72:0/2673484299 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec0343f0 con 0x7fd3f40d0af0 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 -- 172.21.15.72:0/2673484299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f40d0af0 msgr2=0x7fd3f40ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/2673484299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40ccca0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40b7dc0 tx=0x7fd3ec02bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 -- 172.21.15.72:0/2673484299 shutdown_connections 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/2673484299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40ccca0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/2673484299 >> v2:172.21.15.146:3300/0 conn(0x7fd3f406fd10 0x7fd3f40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 -- 172.21.15.72:0/2673484299 >> 172.21.15.72:0/2673484299 conn(0x7fd3f40b3800 msgr2=0x7fd3f40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 -- 172.21.15.72:0/2673484299 shutdown_connections 2024-03-06T07:22:11.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.231+0000 7fd3fb70f640 1 -- 172.21.15.72:0/2673484299 wait complete. 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.232+0000 7fd3fb70f640 1 Processor -- start 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.232+0000 7fd3fb70f640 1 -- start start 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.233+0000 7fd3fb70f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f406fd10 0x7fd3f4158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3fb70f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40b5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3fb70f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 0x7fd3f40b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3fb70f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4165c30 con 0x7fd3f415c050 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3fb70f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4165ab0 con 0x7fd3f40d0af0 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3fb70f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f4165db0 con 0x7fd3f406fd10 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 0x7fd3f40b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 0x7fd3f40b6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38756/0 (socket says 172.21.15.72:38756) 2024-03-06T07:22:11.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 -- 172.21.15.72:0/229429134 learned_addr learned my addr 172.21.15.72:0/229429134 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 -- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f406fd10 msgr2=0x7fd3f4158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f94ab640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f406fd10 0x7fd3f4158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f8caa640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40b5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f406fd10 0x7fd3f4158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 -- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f40d0af0 msgr2=0x7fd3f40b5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40b5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 -- 172.21.15.72:0/229429134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40b7850 con 0x7fd3f415c050 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f8caa640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3f9cac640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 0x7fd3f40b6420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec036040 tx=0x7fd3ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec043020 con 0x7fd3f415c050 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3ec03e450 con 0x7fd3f415c050 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f40b6a90 con 0x7fd3f415c050 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.234+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec009510 con 0x7fd3f415c050 2024-03-06T07:22:11.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.235+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f40b7050 con 0x7fd3f415c050 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.235+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f409fe60 con 0x7fd3f415c050 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.236+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd3ec046020 con 0x7fd3f415c050 2024-03-06T07:22:11.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.236+0000 7fd3e9ffb640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd3d806b500 0x7fd3d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.237+0000 7fd3f94ab640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd3d806b500 0x7fd3d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.237+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3ec0acac0 con 0x7fd3f415c050 2024-03-06T07:22:11.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.237+0000 7fd3f94ab640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd3d806b500 0x7fd3d806d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0001500 tx=0x7fd3f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.241+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd3ec04d030 con 0x7fd3f415c050 2024-03-06T07:22:11.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.413+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3f4068ff0 con 0x7fd3f415c050 2024-03-06T07:22:11.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.413+0000 7fd3e9ffb640 1 -- 172.21.15.72:0/229429134 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3ec07b8b0 con 0x7fd3f415c050 2024-03-06T07:22:11.415 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd3d806b500 msgr2=0x7fd3d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd3d806b500 0x7fd3d806d9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0001500 tx=0x7fd3f0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 msgr2=0x7fd3f40b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 0x7fd3f40b6420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec036040 tx=0x7fd3ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 shutdown_connections 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd3d806b500 0x7fd3d806d9c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3f415c050 0x7fd3f40b6420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f40d0af0 0x7fd3f40b5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 --2- 172.21.15.72:0/229429134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f406fd10 0x7fd3f4158d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.415+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 >> 172.21.15.72:0/229429134 conn(0x7fd3f40b3800 msgr2=0x7fd3f4154f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.416+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 shutdown_connections 2024-03-06T07:22:11.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.416+0000 7fd3fb70f640 1 -- 172.21.15.72:0/229429134 wait complete. 2024-03-06T07:22:11.429 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:11.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.541+0000 7f6c4bd1e640 1 Processor -- start 2024-03-06T07:22:11.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.542+0000 7f6c4bd1e640 1 -- start start 2024-03-06T07:22:11.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.542+0000 7f6c4bd1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 0x7f6c44166c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.542+0000 7f6c4bd1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4406b050 con 0x7f6c4415f4f0 2024-03-06T07:22:11.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.542+0000 7f6c4bd1e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4406b2c0 con 0x7f6c440b0db0 2024-03-06T07:22:11.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.542+0000 7f6c4bd1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440b0890 con 0x7f6c44066d00 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c49aba640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6c4415f4f0 0x7f6c440ce040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41402/0 (socket says 172.21.15.72:41402) 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c49aba640 1 -- 172.21.15.72:0/3291038657 learned_addr learned my addr 172.21.15.72:0/3291038657 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c4a2bb640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 0x7f6c44166c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559251981 0 0) 0x7f6c4406b050 con 0x7f6c4415f4f0 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c30003520 con 0x7f6c4415f4f0 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291506939 0 0) 0x7f6c30003520 con 0x7f6c4415f4f0 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 msgr2=0x7f6c44166c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 0x7f6c44166c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 >> v1:172.21.15.146:6789/0 conn(0x7f6c440b0db0 legacy=0x7f6c440ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c440672e0 con 0x7f6c4415f4f0 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6c34001670 con 0x7f6c4415f4f0 2024-03-06T07:22:11.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 >> v1:172.21.15.72:6789/0 conn(0x7f6c4415f4f0 legacy=0x7f6c440ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 0x7f6c300039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c30003f20 0x7f6c30004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c30004d40 0x7f6c300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440b0890 con 0x7f6c30003f20 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3000ba20 con 0x7f6c30004d40 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3000bc20 con 0x7f6c44066d00 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.543+0000 7f6c492b9640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c30003f20 0x7f6c30004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c49aba640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 0x7f6c300039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c4a2bb640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c30004d40 0x7f6c300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c492b9640 1 -- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 msgr2=0x7f6c300039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c492b9640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c44066d00 0x7f6c300039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c492b9640 1 -- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c30004d40 msgr2=0x7f6c300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c492b9640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c30004d40 0x7f6c300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c492b9640 1 -- 172.21.15.72:0/3291038657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30003520 con 0x7f6c30003f20 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c4a2bb640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c30004d40 0x7f6c300051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.544+0000 7f6c492b9640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c30003f20 0x7f6c30004800 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6c40001860 tx=0x7f6c30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289b3b96ac5de9f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40001430 con 0x7f6c30003f20 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40032620 con 0x7f6c30003f20 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c48ab8640 1 -- 172.21.15.72:0/3291038657 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40032a10 con 0x7f6c30003f20 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/3291038657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c30003f20 msgr2=0x7f6c30004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/3291038657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c30003f20 0x7f6c30004800 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6c40001860 tx=0x7f6c30005830 comp rx=0 tx=0).stop 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4a2bb640 1 -- 172.21.15.72:0/3291038657 reap_dead start 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/3291038657 shutdown_connections 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/3291038657 >> 172.21.15.72:0/3291038657 conn(0x7f6c440b1d70 msgr2=0x7f6c440b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/3291038657 shutdown_connections 2024-03-06T07:22:11.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.545+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/3291038657 wait complete. 2024-03-06T07:22:11.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.547+0000 7f6c4bd1e640 1 Processor -- start 2024-03-06T07:22:11.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.547+0000 7f6c4bd1e640 1 -- start start 2024-03-06T07:22:11.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4bd1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c30003f20 0x7f6c441534b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4bd1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 0x7f6c440c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4bd1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4415dd60 0x7f6c44146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4bd1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4415f8d0 con 0x7f6c4406d480 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4bd1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4406b5a0 con 0x7f6c4415dd60 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4bd1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4406dc00 con 0x7f6c30003f20 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 0x7f6c440c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 0x7f6c440c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38774/0 (socket says 172.21.15.72:38774) 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 -- 172.21.15.72:0/1576149634 learned_addr learned my addr 172.21.15.72:0/1576149634 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4a2bb640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4415dd60 0x7f6c44146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c49aba640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c30003f20 0x7f6c441534b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 -- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c30003f20 msgr2=0x7f6c441534b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c30003f20 0x7f6c441534b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 -- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4415dd60 msgr2=0x7f6c44146f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4415dd60 0x7f6c44146f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 -- 172.21.15.72:0/1576149634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c440cc460 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c49aba640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c30003f20 0x7f6c441534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c4a2bb640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4415dd60 0x7f6c44146f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.548+0000 7f6c492b9640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 0x7f6c440c4fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c400018c0 tx=0x7f6c40001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.549+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40040070 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.549+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c40007050 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.549+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c440cc740 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.549+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c40007400 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.549+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4406d900 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.550+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4409e3d0 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.550+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c40043070 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.551+0000 7f6c3a7fc640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c2806b4d0 0x7f6c2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.551+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c400abae0 con 0x7f6c4406d480 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.551+0000 7f6c49aba640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c2806b4d0 0x7f6c2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.551+0000 7f6c49aba640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c2806b4d0 0x7f6c2806d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6c34001050 tx=0x7f6c34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.555+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c4007a9f0 con 0x7f6c4406d480 2024-03-06T07:22:11.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.732+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c44069e80 con 0x7f6c4406d480 2024-03-06T07:22:11.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.732+0000 7f6c3a7fc640 1 -- 172.21.15.72:0/1576149634 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c4007a2f0 con 0x7f6c4406d480 2024-03-06T07:22:11.734 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:11.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c2806b4d0 msgr2=0x7f6c2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c2806b4d0 0x7f6c2806d990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6c34001050 tx=0x7f6c34009040 comp rx=0 tx=0).stop 2024-03-06T07:22:11.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 msgr2=0x7f6c440c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 0x7f6c440c4fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6c400018c0 tx=0x7f6c40001230 comp rx=0 tx=0).stop 2024-03-06T07:22:11.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 shutdown_connections 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c2806b4d0 0x7f6c2806d990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4415dd60 0x7f6c44146f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c4406d480 0x7f6c440c4fe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 --2- 172.21.15.72:0/1576149634 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c30003f20 0x7f6c441534b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 >> 172.21.15.72:0/1576149634 conn(0x7f6c440b1d70 msgr2=0x7f6c440ce2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 shutdown_connections 2024-03-06T07:22:11.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.734+0000 7f6c4bd1e640 1 -- 172.21.15.72:0/1576149634 wait complete. 2024-03-06T07:22:11.747 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:11.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.845+0000 7ff80afd7640 1 Processor -- start 2024-03-06T07:22:11.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.845+0000 7ff80afd7640 1 -- start start 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff80afd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff804158600 0x7ff804072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff80afd7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff80415e3a0 0x7ff80406d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff80afd7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff80406deb0 0x7ff804162030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff80afd7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff804161ba0 con 0x7ff80406deb0 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff80afd7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80415eb40 con 0x7ff80415e3a0 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff80afd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80415ee20 con 0x7ff804158600 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff809574640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff80406deb0 0x7ff804162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff809574640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff80406deb0 0x7ff804162030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38790/0 (socket says 172.21.15.72:38790) 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.846+0000 7ff809574640 1 -- 172.21.15.72:0/1581008232 learned_addr learned my addr 172.21.15.72:0/1581008232 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff808d73640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff804158600 0x7ff804072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff809574640 1 -- 172.21.15.72:0/1581008232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff804158600 msgr2=0x7ff804072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff803fff640 1 --2- 172.21.15.72:0/1581008232 >> v2:172.21.15.146:3300/0 conn(0x7ff80415e3a0 0x7ff80406d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff809574640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff804158600 0x7ff804072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff809574640 1 -- 172.21.15.72:0/1581008232 >> v2:172.21.15.146:3300/0 conn(0x7ff80415e3a0 msgr2=0x7ff80406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff809574640 1 --2- 172.21.15.72:0/1581008232 >> v2:172.21.15.146:3300/0 conn(0x7ff80415e3a0 0x7ff80406d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff809574640 1 -- 172.21.15.72:0/1581008232 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80415f100 con 0x7ff80406deb0 2024-03-06T07:22:11.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff803fff640 1 --2- 172.21.15.72:0/1581008232 >> v2:172.21.15.146:3300/0 conn(0x7ff80415e3a0 0x7ff80406d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff808d73640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff804158600 0x7ff804072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff809574640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80406deb0 0x7ff804162030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff8040d1a80 tx=0x7ff7fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16933f555ff96806 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff8037fe640 1 -- 172.21.15.72:0/1581008232 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc02c9e0 con 0x7ff80406deb0 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff8037fe640 1 -- 172.21.15.72:0/1581008232 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc02cb80 con 0x7ff80406deb0 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.847+0000 7ff8037fe640 1 -- 172.21.15.72:0/1581008232 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0343f0 con 0x7ff80406deb0 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 -- 172.21.15.72:0/1581008232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80406deb0 msgr2=0x7ff804162030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80406deb0 0x7ff804162030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ff8040d1a80 tx=0x7ff7fc02be40 comp rx=0 tx=0).stop 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 -- 172.21.15.72:0/1581008232 shutdown_connections 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80406deb0 0x7ff804162030 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 --2- 172.21.15.72:0/1581008232 >> v2:172.21.15.146:3300/0 conn(0x7ff80415e3a0 0x7ff80406d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 --2- 172.21.15.72:0/1581008232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff804158600 0x7ff804072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 -- 172.21.15.72:0/1581008232 >> 172.21.15.72:0/1581008232 conn(0x7ff80406c0c0 msgr2=0x7ff80406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 -- 172.21.15.72:0/1581008232 shutdown_connections 2024-03-06T07:22:11.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.848+0000 7ff80afd7640 1 -- 172.21.15.72:0/1581008232 wait complete. 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.849+0000 7ff80afd7640 1 Processor -- start 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.849+0000 7ff80afd7640 1 -- start start 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff80afd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80406deb0 0x7ff8041549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff80afd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 0x7ff804160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff80afd7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80415e3a0 0x7ff804140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff80afd7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8041626f0 con 0x7ff804158600 2024-03-06T07:22:11.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff80afd7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff804162570 con 0x7ff80415e3a0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff80afd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff804162870 con 0x7ff80406deb0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 0x7ff804160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 0x7ff804160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38800/0 (socket says 172.21.15.72:38800) 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 -- 172.21.15.72:0/2098749995 learned_addr learned my addr 172.21.15.72:0/2098749995 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff808d73640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80406deb0 0x7ff8041549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff809574640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80415e3a0 0x7ff804140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 -- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80406deb0 msgr2=0x7ff8041549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80406deb0 0x7ff8041549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 -- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80415e3a0 msgr2=0x7ff804140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80415e3a0 0x7ff804140ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.850+0000 7ff803fff640 1 -- 172.21.15.72:0/2098749995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8041612c0 con 0x7ff804158600 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.851+0000 7ff803fff640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 0x7ff804160d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000c7f0 tx=0x7ff7f000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.851+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0015070 con 0x7ff804158600 2024-03-06T07:22:11.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.851+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f0007a30 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.851+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0007e00 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.851+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff804141570 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.851+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff804141b30 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.852+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80404f330 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.853+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7f0018070 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.853+0000 7ff8017fa640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e806b500 0x7ff7e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.853+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7f008a110 con 0x7ff804158600 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.853+0000 7ff808d73640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e806b500 0x7ff7e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:11.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.854+0000 7ff808d73640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e806b500 0x7ff7e806d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800a950 tx=0x7ff7f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:11.858+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7f0058f90 con 0x7ff804158600 2024-03-06T07:22:12.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.038+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8040d06f0 con 0x7ff804158600 2024-03-06T07:22:12.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.039+0000 7ff8017fa640 1 -- 172.21.15.72:0/2098749995 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7ff7f0058890 con 0x7ff804158600 2024-03-06T07:22:12.042 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:12.042 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.041+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e806b500 msgr2=0x7ff7e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.041+0000 7ff80afd7640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e806b500 0x7ff7e806d9c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800a950 tx=0x7ff7f8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.041+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 msgr2=0x7ff804160d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.041+0000 7ff80afd7640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 0x7ff804160d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000c7f0 tx=0x7ff7f000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 shutdown_connections 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e806b500 0x7ff7e806d9c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80415e3a0 0x7ff804140ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff804158600 0x7ff804160d80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 --2- 172.21.15.72:0/2098749995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80406deb0 0x7ff8041549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 >> 172.21.15.72:0/2098749995 conn(0x7ff80406c0c0 msgr2=0x7ff8040b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 shutdown_connections 2024-03-06T07:22:12.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.042+0000 7ff80afd7640 1 -- 172.21.15.72:0/2098749995 wait complete. 2024-03-06T07:22:12.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:12.055 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:12.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.158+0000 7fea3a59a640 1 Processor -- start 2024-03-06T07:22:12.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.158+0000 7fea3a59a640 1 -- start start 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.159+0000 7fea3a59a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.159+0000 7fea3a59a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea340d0af0 0x7fea340cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.159+0000 7fea3a59a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea340cd4b0 con 0x7fea3406fb40 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.159+0000 7fea3a59a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3406ff80 con 0x7fea340d0af0 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.159+0000 7fea3a59a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea340cd790 con 0x7fea3415c050 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea33fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea38b37640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea340d0af0 0x7fea340cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea337fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fea3406fb40 0x7fea340cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:46134/0 (socket says 172.21.15.72:46134) 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea33fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:43174/0 (socket says 172.21.15.72:43174) 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea337fe640 1 -- 172.21.15.72:0/305864223 learned_addr learned my addr 172.21.15.72:0/305864223 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea38b37640 1 -- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 msgr2=0x7fea3406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea38b37640 1 --2- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea38b37640 1 -- 172.21.15.72:0/305864223 >> v1:172.21.15.146:6789/0 conn(0x7fea3406fb40 legacy=0x7fea340cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea38b37640 1 -- 172.21.15.72:0/305864223 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea340cda70 con 0x7fea340d0af0 2024-03-06T07:22:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.160+0000 7fea33fff640 1 --2- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.161+0000 7fea38b37640 1 --2- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea340cce70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009d90 tx=0x7fea2c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82d39fb61be89e59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.161+0000 7fea32ffd640 1 -- 172.21.15.72:0/305864223 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c039070 con 0x7fea340d0af0 2024-03-06T07:22:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.161+0000 7fea32ffd640 1 -- 172.21.15.72:0/305864223 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c02cbc0 con 0x7fea340d0af0 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.161+0000 7fea32ffd640 1 -- 172.21.15.72:0/305864223 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c034440 con 0x7fea340d0af0 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 -- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 msgr2=0x7fea340cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 --2- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea340cce70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fea2c009d90 tx=0x7fea2c02be10 comp rx=0 tx=0).stop 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 -- 172.21.15.72:0/305864223 shutdown_connections 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 --2- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea340cce70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 --2- 172.21.15.72:0/305864223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 -- 172.21.15.72:0/305864223 >> 172.21.15.72:0/305864223 conn(0x7fea340b3800 msgr2=0x7fea340b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 -- 172.21.15.72:0/305864223 shutdown_connections 2024-03-06T07:22:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.162+0000 7fea3a59a640 1 -- 172.21.15.72:0/305864223 wait complete. 2024-03-06T07:22:12.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.164+0000 7fea3a59a640 1 Processor -- start 2024-03-06T07:22:12.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.164+0000 7fea3a59a640 1 -- start start 2024-03-06T07:22:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea3a59a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 0x7fea341509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea3a59a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea341598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea3a59a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3415a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea3a59a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea34165c30 con 0x7fea3406fb40 2024-03-06T07:22:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea3a59a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea34165ab0 con 0x7fea340d0af0 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea3a59a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea34165db0 con 0x7fea3415c050 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea33fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 0x7fea341509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea33fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 0x7fea341509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38802/0 (socket says 172.21.15.72:38802) 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea33fff640 1 -- 172.21.15.72:0/2173775662 learned_addr learned my addr 172.21.15.72:0/2173775662 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea337fe640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea341598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.165+0000 7fea38b37640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3415a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea33fff640 1 -- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 msgr2=0x7fea3415a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea33fff640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3415a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea33fff640 1 -- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 msgr2=0x7fea341598c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea33fff640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea341598c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea33fff640 1 -- 172.21.15.72:0/2173775662 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3414c7a0 con 0x7fea3406fb40 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea33fff640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 0x7fea341509e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fea2400ebe0 tx=0x7fea2400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24017070 con 0x7fea3406fb40 2024-03-06T07:22:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2400cc30 con 0x7fea3406fb40 2024-03-06T07:22:12.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea340af8e0 con 0x7fea3406fb40 2024-03-06T07:22:12.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24013470 con 0x7fea3406fb40 2024-03-06T07:22:12.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.166+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea340afea0 con 0x7fea3406fb40 2024-03-06T07:22:12.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.167+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3409fe60 con 0x7fea3406fb40 2024-03-06T07:22:12.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.167+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fea2401a070 con 0x7fea3406fb40 2024-03-06T07:22:12.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.168+0000 7fea30ff9640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea2006b4d0 0x7fea2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.168+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea2408a000 con 0x7fea3406fb40 2024-03-06T07:22:12.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.168+0000 7fea337fe640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea2006b4d0 0x7fea2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.168+0000 7fea337fe640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea2006b4d0 0x7fea2006d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea34152960 tx=0x7fea28007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.172+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea240588c0 con 0x7fea3406fb40 2024-03-06T07:22:12.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.345+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fea34068ff0 con 0x7fea3406fb40 2024-03-06T07:22:12.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.345+0000 7fea30ff9640 1 -- 172.21.15.72:0/2173775662 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fea2405c960 con 0x7fea3406fb40 2024-03-06T07:22:12.347 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea2006b4d0 msgr2=0x7fea2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea2006b4d0 0x7fea2006d990 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea34152960 tx=0x7fea28007e80 comp rx=0 tx=0).stop 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 msgr2=0x7fea341509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 0x7fea341509e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fea2400ebe0 tx=0x7fea2400c520 comp rx=0 tx=0).stop 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 shutdown_connections 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fea2006b4d0 0x7fea2006d990 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea3415c050 0x7fea3415a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea340d0af0 0x7fea341598c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 --2- 172.21.15.72:0/2173775662 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea3406fb40 0x7fea341509e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.347+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 >> 172.21.15.72:0/2173775662 conn(0x7fea340b3800 msgr2=0x7fea34158d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.348+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 shutdown_connections 2024-03-06T07:22:12.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.348+0000 7fea3a59a640 1 -- 172.21.15.72:0/2173775662 wait complete. 2024-03-06T07:22:12.360 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.462+0000 7f43dec0e640 1 Processor -- start 2024-03-06T07:22:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.462+0000 7f43dec0e640 1 -- start start 2024-03-06T07:22:12.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dec0e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f43d81614d0 0x7f43d80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dec0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43d80d24b0 0x7f43d80b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dec0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d81618b0 con 0x7f43d80d1910 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dec0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d80cf400 con 0x7f43d80d24b0 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dec0e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43d80cf6a0 con 0x7f43d81614d0 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dd1ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43d80d24b0 0x7f43d80b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dd1ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43d80d24b0 0x7f43d80b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41842/0 (socket says 172.21.15.72:41842) 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dd1ab640 1 -- 172.21.15.72:0/3566436338 learned_addr learned my addr 172.21.15.72:0/3566436338 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43d7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f43d80d1910 0x7f43d80d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38328/0 (socket says 172.21.15.72:38328) 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.463+0000 7f43dc9aa640 1 --2- 172.21.15.72:0/3566436338 >> v2:172.21.15.146:3300/0 conn(0x7f43d81614d0 0x7f43d80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.464+0000 7f43dd1ab640 1 -- 172.21.15.72:0/3566436338 >> v2:172.21.15.146:3300/0 conn(0x7f43d81614d0 msgr2=0x7f43d80d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.464+0000 7f43dd1ab640 1 --2- 172.21.15.72:0/3566436338 >> v2:172.21.15.146:3300/0 conn(0x7f43d81614d0 0x7f43d80d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.464+0000 7f43dd1ab640 1 -- 172.21.15.72:0/3566436338 >> v1:172.21.15.72:6789/0 conn(0x7f43d80d1910 legacy=0x7f43d80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.464+0000 7f43dd1ab640 1 -- 172.21.15.72:0/3566436338 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d80cf980 con 0x7f43d80d24b0 2024-03-06T07:22:12.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.464+0000 7f43dd1ab640 1 --2- 172.21.15.72:0/3566436338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80b0e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f43d808c470 tx=0x7f43d002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9ab17c6d8d7643 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43d77fe640 1 -- 172.21.15.72:0/3566436338 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d002c8d0 con 0x7f43d80d24b0 2024-03-06T07:22:12.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43d77fe640 1 -- 172.21.15.72:0/3566436338 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d002ca70 con 0x7f43d80d24b0 2024-03-06T07:22:12.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43d77fe640 1 -- 172.21.15.72:0/3566436338 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d002ce40 con 0x7f43d80d24b0 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 -- 172.21.15.72:0/3566436338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 msgr2=0x7f43d80b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 --2- 172.21.15.72:0/3566436338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80b0e30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f43d808c470 tx=0x7f43d002bdb0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 -- 172.21.15.72:0/3566436338 shutdown_connections 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 --2- 172.21.15.72:0/3566436338 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80b0e30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 --2- 172.21.15.72:0/3566436338 >> v2:172.21.15.146:3300/0 conn(0x7f43d81614d0 0x7f43d80d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 -- 172.21.15.72:0/3566436338 >> 172.21.15.72:0/3566436338 conn(0x7f43d80d3d40 msgr2=0x7f43d80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 -- 172.21.15.72:0/3566436338 shutdown_connections 2024-03-06T07:22:12.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.465+0000 7f43dec0e640 1 -- 172.21.15.72:0/3566436338 wait complete. 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 Processor -- start 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 -- start start 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43d80d1910 0x7f43d814aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43d81614d0 0x7f43d814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80b1370 con 0x7f43d80d24b0 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80b0910 con 0x7f43d80d1910 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43dec0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43d80b14f0 con 0x7f43d81614d0 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.467+0000 7f43d7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41848/0 (socket says 172.21.15.72:41848) 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 -- 172.21.15.72:0/419404761 learned_addr learned my addr 172.21.15.72:0/419404761 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:12.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43dc9aa640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43d80d1910 0x7f43d814aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 -- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43d81614d0 msgr2=0x7f43d814b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43d81614d0 0x7f43d814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 -- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43d80d1910 msgr2=0x7f43d814aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43d80d1910 0x7f43d814aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 -- 172.21.15.72:0/419404761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d80c7db0 con 0x7f43d80d24b0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d7fff640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80c7870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f43c400c7b0 tx=0x7f43c400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c4018070 con 0x7f43d80d24b0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43c40072d0 con 0x7f43d80d24b0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d814ddb0 con 0x7f43d80d24b0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c40047e0 con 0x7f43d80d24b0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.468+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d80c96c0 con 0x7f43d80d24b0 2024-03-06T07:22:12.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.469+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d8068ff0 con 0x7f43d80d24b0 2024-03-06T07:22:12.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.470+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43c401b070 con 0x7f43d80d24b0 2024-03-06T07:22:12.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.470+0000 7f43d57fa640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f43bc06b4d0 0x7f43bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.470+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43c40896a0 con 0x7f43d80d24b0 2024-03-06T07:22:12.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.470+0000 7f43dc9aa640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f43bc06b4d0 0x7f43bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.470+0000 7f43dc9aa640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f43bc06b4d0 0x7f43bc06d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009970 tx=0x7f43cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.474+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f43c4058520 con 0x7f43d80d24b0 2024-03-06T07:22:12.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.708+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f43d814b6b0 con 0x7f43d80d24b0 2024-03-06T07:22:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.709+0000 7f43d57fa640 1 -- 172.21.15.72:0/419404761 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f43c4057e20 con 0x7f43d80d24b0 2024-03-06T07:22:12.711 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f43bc06b4d0 msgr2=0x7f43bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f43bc06b4d0 0x7f43bc06d990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009970 tx=0x7f43cc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 msgr2=0x7f43d80c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80c7870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f43c400c7b0 tx=0x7f43c400cc70 comp rx=0 tx=0).stop 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 shutdown_connections 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f43bc06b4d0 0x7f43bc06d990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43d81614d0 0x7f43d814b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d80d24b0 0x7f43d80c7870 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 --2- 172.21.15.72:0/419404761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43d80d1910 0x7f43d814aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 >> 172.21.15.72:0/419404761 conn(0x7f43d80d3d40 msgr2=0x7f43d80d2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 shutdown_connections 2024-03-06T07:22:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.711+0000 7f43dec0e640 1 -- 172.21.15.72:0/419404761 wait complete. 2024-03-06T07:22:12.728 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:12.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.830+0000 7f2404a39640 1 Processor -- start 2024-03-06T07:22:12.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.830+0000 7f2404a39640 1 -- start start 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f2404a39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f24000d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f2404a39640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000cfa50 con 0x7f240015ecd0 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f2404a39640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000d0a30 con 0x7f24000d14a0 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f2404a39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240015c050 con 0x7f24000d10c0 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f23fed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f240015ecd0 0x7f24000cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38342/0 (socket says 172.21.15.72:38342) 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f23fed9d640 1 -- 172.21.15.72:0/3984200847 learned_addr learned my addr 172.21.15.72:0/3984200847 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f24000d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991774908 0 0) 0x7f24000cfa50 con 0x7f240015ecd0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.831+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ec003520 con 0x7f240015ecd0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200022833 0 0) 0x7f24000d0a30 con 0x7f24000d14a0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24000cfa50 con 0x7f24000d14a0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1140363606 0 0) 0x7f23ec003520 con 0x7f240015ecd0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 msgr2=0x7f24000d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f24000d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 >> v1:172.21.15.146:6789/0 conn(0x7f24000d14a0 legacy=0x7f240015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000b45c0 con 0x7f240015ecd0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f23f80019f0 con 0x7f240015ecd0 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f24000d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 >> v1:172.21.15.72:6789/0 conn(0x7f240015ecd0 legacy=0x7f24000cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23ec003b70 0x7f23ec003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec004800 0x7f23ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f23ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec00b9c0 con 0x7f23ec003b70 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec00bb70 con 0x7f23ec004800 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23ec00bd70 con 0x7f24000d10c0 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23ec003b70 0x7f23ec003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23f7fff640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec004800 0x7f23ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fed9d640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f23ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 -- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 msgr2=0x7f23ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000d10c0 0x7f23ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 -- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec004800 msgr2=0x7f23ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec004800 0x7f23ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23fe59c640 1 -- 172.21.15.72:0/3984200847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23ec003520 con 0x7f23ec003b70 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.832+0000 7f23f7fff640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec004800 0x7f23ec004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23ec003b70 0x7f23ec003f70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009c60 tx=0x7f23ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3fcfcf7dcdd85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0039070 con 0x7f23ec003b70 2024-03-06T07:22:12.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f002c8a0 con 0x7f23ec003b70 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f23fdd9b640 1 -- 172.21.15.72:0/3984200847 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0009860 con 0x7f23ec003b70 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f2404a39640 1 -- 172.21.15.72:0/3984200847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23ec003b70 msgr2=0x7f23ec003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f2404a39640 1 --2- 172.21.15.72:0/3984200847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23ec003b70 0x7f23ec003f70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009c60 tx=0x7f23ec005830 comp rx=0 tx=0).stop 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.833+0000 7f23fed9d640 1 -- 172.21.15.72:0/3984200847 reap_dead start 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.834+0000 7f2404a39640 1 -- 172.21.15.72:0/3984200847 shutdown_connections 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.834+0000 7f2404a39640 1 -- 172.21.15.72:0/3984200847 >> 172.21.15.72:0/3984200847 conn(0x7f24000b3800 msgr2=0x7f24000b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.834+0000 7f2404a39640 1 -- 172.21.15.72:0/3984200847 shutdown_connections 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.834+0000 7f2404a39640 1 -- 172.21.15.72:0/3984200847 wait complete. 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.835+0000 7f2404a39640 1 Processor -- start 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.835+0000 7f2404a39640 1 -- start start 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.836+0000 7f2404a39640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec003b70 0x7f24001495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.836+0000 7f2404a39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000ccb90 0x7f24000b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.836+0000 7f2404a39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 0x7f24000b6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.836+0000 7f2404a39640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240015f440 con 0x7f24000cd010 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.836+0000 7f2404a39640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000d0060 con 0x7f23ec003b70 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.836+0000 7f2404a39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000cfcf0 con 0x7f24000ccb90 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 0x7f24000b6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 0x7f24000b6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41862/0 (socket says 172.21.15.72:41862) 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 -- 172.21.15.72:0/3526848275 learned_addr learned my addr 172.21.15.72:0/3526848275 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec003b70 0x7f24001495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23f7fff640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000ccb90 0x7f24000b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 -- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000ccb90 msgr2=0x7f24000b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000ccb90 0x7f24000b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 -- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec003b70 msgr2=0x7f24001495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec003b70 0x7f24001495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 -- 172.21.15.72:0/3526848275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2400149b20 con 0x7f24000cd010 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23fed9d640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 0x7f24000b6fa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2400071930 tx=0x7f23f800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f800ec10 con 0x7f24000cd010 2024-03-06T07:22:12.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f800edd0 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24000cd570 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8014670 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.837+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24000b4f60 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.838+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24000cda90 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.839+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f23f8014810 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.839+0000 7f23f6ffd640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f23e006b500 0x7f23e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.839+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f23e006b500 0x7f23e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.839+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23f8089e40 con 0x7f24000cd010 2024-03-06T07:22:12.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.840+0000 7f23fe59c640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f23e006b500 0x7f23e006d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23f0036000 tx=0x7f23f0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:12.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:12.843+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24000cda90 con 0x7f24000cd010 2024-03-06T07:22:13.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.020+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f240009fe60 con 0x7f24000cd010 2024-03-06T07:22:13.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.021+0000 7f23f6ffd640 1 -- 172.21.15.72:0/3526848275 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2400149b20 con 0x7f24000cd010 2024-03-06T07:22:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:13.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f23e006b500 msgr2=0x7f23e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f23e006b500 0x7f23e006d9c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f23f0036000 tx=0x7f23f0030000 comp rx=0 tx=0).stop 2024-03-06T07:22:13.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 msgr2=0x7f24000b6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 0x7f24000b6fa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2400071930 tx=0x7f23f800e530 comp rx=0 tx=0).stop 2024-03-06T07:22:13.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 shutdown_connections 2024-03-06T07:22:13.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f23e006b500 0x7f23e006d9c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24000cd010 0x7f24000b6fa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000ccb90 0x7f24000b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 --2- 172.21.15.72:0/3526848275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23ec003b70 0x7f24001495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 >> 172.21.15.72:0/3526848275 conn(0x7f24000b3800 msgr2=0x7f24000b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 shutdown_connections 2024-03-06T07:22:13.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.023+0000 7f2404a39640 1 -- 172.21.15.72:0/3526848275 wait complete. 2024-03-06T07:22:13.040 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:13.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.157+0000 7f81bbd9a640 1 Processor -- start 2024-03-06T07:22:13.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.158+0000 7f81bbd9a640 1 -- start start 2024-03-06T07:22:13.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.158+0000 7f81bbd9a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81b415bdd0 0x7f81b40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.158+0000 7f81bbd9a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 0x7f81b40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.158+0000 7f81bbd9a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81b40cebe0 con 0x7f81b40d16e0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.158+0000 7f81bbd9a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b40ceec0 con 0x7f81b415bdd0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.158+0000 7f81bbd9a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40cf1a0 con 0x7f81b415fa10 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81ba337640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f81b40d16e0 0x7f81b40d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38356/0 (socket says 172.21.15.72:38356) 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81ba337640 1 -- 172.21.15.72:0/387485765 learned_addr learned my addr 172.21.15.72:0/387485765 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b9335640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 0x7f81b40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b9b36640 1 --2- 172.21.15.72:0/387485765 >> v2:172.21.15.146:3300/0 conn(0x7f81b415bdd0 0x7f81b40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181019505 0 0) 0x7f81b40cebe0 con 0x7f81b40d16e0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a0003520 con 0x7f81b40d16e0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2902578323 0 0) 0x7f81a0003520 con 0x7f81b40d16e0 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 msgr2=0x7f81b40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 0x7f81b40ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 >> v2:172.21.15.146:3300/0 conn(0x7f81b415bdd0 msgr2=0x7f81b40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 --2- 172.21.15.72:0/387485765 >> v2:172.21.15.146:3300/0 conn(0x7f81b415bdd0 0x7f81b40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40cc8e0 con 0x7f81b40d16e0 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f81ac0019f0 con 0x7f81b40d16e0 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 >> v1:172.21.15.72:6789/0 conn(0x7f81b40d16e0 legacy=0x7f81b40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a00039b0 0x7f81a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 0x7f81a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40ceec0 con 0x7f81a0004800 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40cf1a0 con 0x7f81a00039b0 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a000b9f0 con 0x7f81b415fa10 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b9335640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81b9b36640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a00039b0 0x7f81a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.159+0000 7f81ba337640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 0x7f81a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b9335640 1 -- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 msgr2=0x7f81a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b9335640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415fa10 0x7f81a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b9335640 1 -- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a00039b0 msgr2=0x7f81a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b9335640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81a00039b0 0x7f81a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b9335640 1 -- 172.21.15.72:0/387485765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a0003520 con 0x7f81a0004800 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b9335640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81a0004c80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009ac0 tx=0x7f81a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceaa93b986ea0499 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0039070 con 0x7f81a0004800 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b002c6b0 con 0x7f81a0004800 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.160+0000 7f81b8b34640 1 -- 172.21.15.72:0/387485765 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0009690 con 0x7f81a0004800 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81bbd9a640 1 -- 172.21.15.72:0/387485765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 msgr2=0x7f81a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/387485765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81a0004c80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f81b0009ac0 tx=0x7f81a0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:13.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81ba337640 1 -- 172.21.15.72:0/387485765 reap_dead start 2024-03-06T07:22:13.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81bbd9a640 1 -- 172.21.15.72:0/387485765 shutdown_connections 2024-03-06T07:22:13.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81bbd9a640 1 -- 172.21.15.72:0/387485765 >> 172.21.15.72:0/387485765 conn(0x7f81b409a9f0 msgr2=0x7f81b409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81bbd9a640 1 -- 172.21.15.72:0/387485765 shutdown_connections 2024-03-06T07:22:13.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.161+0000 7f81bbd9a640 1 -- 172.21.15.72:0/387485765 wait complete. 2024-03-06T07:22:13.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.162+0000 7f81bbd9a640 1 Processor -- start 2024-03-06T07:22:13.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.162+0000 7f81bbd9a640 1 -- start start 2024-03-06T07:22:13.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81bbd9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81b41f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81bbd9a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415c330 0x7f81b415c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81bbd9a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81b40b5720 0x7f81b40b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81bbd9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b415c1b0 con 0x7f81a0004800 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81bbd9a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40cc5d0 con 0x7f81b40b5720 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81bbd9a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b40ce270 con 0x7f81b415c330 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81b41f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81b41f12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41878/0 (socket says 172.21.15.72:41878) 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 -- 172.21.15.72:0/3764024964 learned_addr learned my addr 172.21.15.72:0/3764024964 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81ba337640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81b40b5720 0x7f81b40b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9335640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415c330 0x7f81b415c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 -- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415c330 msgr2=0x7f81b415c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415c330 0x7f81b415c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 -- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81b40b5720 msgr2=0x7f81b40b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81b40b5720 0x7f81b40b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 -- 172.21.15.72:0/3764024964 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b40b6300 con 0x7f81a0004800 2024-03-06T07:22:13.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9335640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415c330 0x7f81b415c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.163+0000 7f81b9b36640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81b41f12e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f81a4011a70 tx=0x7f81a4011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.164+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a400fb50 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.164+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a400fcf0 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.164+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b40b65e0 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.164+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a400fea0 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.164+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b40b6ba0 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.165+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b404f330 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.165+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f81a4012040 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.165+0000 7f81aa7fc640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f819806b500 0x7f819806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.166+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81a408a8a0 con 0x7f81a0004800 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.166+0000 7f81b9335640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f819806b500 0x7f819806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.166+0000 7f81b9335640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f819806b500 0x7f819806d9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f81b0036000 tx=0x7f81b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.170+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f81a4019030 con 0x7f81a0004800 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.349+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81b40b7a40 con 0x7f81a0004800 2024-03-06T07:22:13.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.350+0000 7f81aa7fc640 1 -- 172.21.15.72:0/3764024964 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f81a4059160 con 0x7f81a0004800 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:13.352 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f819806b500 msgr2=0x7f819806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f819806b500 0x7f819806d9c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f81b0036000 tx=0x7f81b0030000 comp rx=0 tx=0).stop 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 msgr2=0x7f81b41f12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81b41f12e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f81a4011a70 tx=0x7f81a4011f30 comp rx=0 tx=0).stop 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 shutdown_connections 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f819806b500 0x7f819806d9c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81b40b5720 0x7f81b40b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81b415c330 0x7f81b415c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 --2- 172.21.15.72:0/3764024964 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81a0004800 0x7f81b41f12e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.352+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 >> 172.21.15.72:0/3764024964 conn(0x7f81b409a9f0 msgr2=0x7f81b40cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.353+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 shutdown_connections 2024-03-06T07:22:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.353+0000 7f81bbd9a640 1 -- 172.21.15.72:0/3764024964 wait complete. 2024-03-06T07:22:13.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:13.366 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:13.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.486+0000 7fba4bea4640 1 Processor -- start 2024-03-06T07:22:13.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.486+0000 7fba4bea4640 1 -- start start 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba4bea4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba440b4970 0x7fba4406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba4bea4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba4406d5c0 0x7fba440cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba4bea4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba4415ff90 con 0x7fba4415bb40 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba4bea4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440b4dd0 con 0x7fba440b4970 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba4bea4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba4406c910 con 0x7fba4406d5c0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba4943f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba440b4970 0x7fba4406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fba440b4970 0x7fba4406d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41890/0 (socket says 172.21.15.72:41890) 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 -- 172.21.15.72:0/2551953763 learned_addr learned my addr 172.21.15.72:0/2551953763 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:13.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.487+0000 7fba49c40640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fba4415bb40 0x7fba4415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38360/0 (socket says 172.21.15.72:38360) 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 -- 172.21.15.72:0/2551953763 >> v2:172.21.15.146:3300/0 conn(0x7fba4406d5c0 msgr2=0x7fba440cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 --2- 172.21.15.72:0/2551953763 >> v2:172.21.15.146:3300/0 conn(0x7fba4406d5c0 0x7fba440cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 -- 172.21.15.72:0/2551953763 >> v1:172.21.15.72:6789/0 conn(0x7fba4415bb40 legacy=0x7fba4415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 -- 172.21.15.72:0/2551953763 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44160270 con 0x7fba440b4970 2024-03-06T07:22:13.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba48c3e640 1 -- 172.21.15.72:0/2551953763 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506267272 0 0) 0x7fba4415ff90 con 0x7fba4415bb40 2024-03-06T07:22:13.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.488+0000 7fba4943f640 1 --2- 172.21.15.72:0/2551953763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4406d080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fba44072c60 tx=0x7fba4002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5d0d9e70debc1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba48c3e640 1 -- 172.21.15.72:0/2551953763 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40034070 con 0x7fba440b4970 2024-03-06T07:22:13.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba48c3e640 1 -- 172.21.15.72:0/2551953763 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba40030860 con 0x7fba440b4970 2024-03-06T07:22:13.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba48c3e640 1 -- 172.21.15.72:0/2551953763 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40030c30 con 0x7fba440b4970 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 -- 172.21.15.72:0/2551953763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 msgr2=0x7fba4406d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 --2- 172.21.15.72:0/2551953763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4406d080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fba44072c60 tx=0x7fba4002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 -- 172.21.15.72:0/2551953763 shutdown_connections 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 --2- 172.21.15.72:0/2551953763 >> v2:172.21.15.146:3300/0 conn(0x7fba4406d5c0 0x7fba440cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 --2- 172.21.15.72:0/2551953763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4406d080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 -- 172.21.15.72:0/2551953763 >> 172.21.15.72:0/2551953763 conn(0x7fba4406b190 msgr2=0x7fba4406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 -- 172.21.15.72:0/2551953763 shutdown_connections 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.489+0000 7fba4bea4640 1 -- 172.21.15.72:0/2551953763 wait complete. 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.490+0000 7fba4bea4640 1 Processor -- start 2024-03-06T07:22:13.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.490+0000 7fba4bea4640 1 -- start start 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4bea4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba4406d5c0 0x7fba44140160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4bea4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4415c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4bea4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba4415bb40 0x7fba4415c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4bea4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4415bf20 con 0x7fba440b4970 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4bea4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4406cb60 con 0x7fba4406d5c0 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4bea4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4415c0a0 con 0x7fba4415bb40 2024-03-06T07:22:13.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4943f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4943f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4415c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41904/0 (socket says 172.21.15.72:41904) 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4943f640 1 -- 172.21.15.72:0/1685367784 learned_addr learned my addr 172.21.15.72:0/1685367784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba49c40640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba4406d5c0 0x7fba44140160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.491+0000 7fba4a441640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba4415bb40 0x7fba4415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4943f640 1 -- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba4415bb40 msgr2=0x7fba4415c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4943f640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba4415bb40 0x7fba4415c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4943f640 1 -- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba4406d5c0 msgr2=0x7fba44140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4943f640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba4406d5c0 0x7fba44140160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4943f640 1 -- 172.21.15.72:0/1685367784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4415cdc0 con 0x7fba440b4970 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba49c40640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba4406d5c0 0x7fba44140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4943f640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4415c250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba40009960 tx=0x7fba400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40034040 con 0x7fba440b4970 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba40039b00 con 0x7fba440b4970 2024-03-06T07:22:13.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4415d0a0 con 0x7fba440b4970 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba40004030 con 0x7fba440b4970 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.492+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba4415d660 con 0x7fba440b4970 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.493+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba440732b0 con 0x7fba440b4970 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.493+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba40037070 con 0x7fba440b4970 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.494+0000 7fba3a7fc640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba2806b4d0 0x7fba2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.494+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba400ac1f0 con 0x7fba440b4970 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.494+0000 7fba49c40640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba2806b4d0 0x7fba2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.494+0000 7fba49c40640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba2806b4d0 0x7fba2806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba34000f60 tx=0x7fba34007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.502+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba4007afe0 con 0x7fba440b4970 2024-03-06T07:22:13.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.678+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba440ce860 con 0x7fba440b4970 2024-03-06T07:22:13.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.678+0000 7fba3a7fc640 1 -- 172.21.15.72:0/1685367784 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba4007a8e0 con 0x7fba440b4970 2024-03-06T07:22:13.681 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.680+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba2806b4d0 msgr2=0x7fba2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.680+0000 7fba4bea4640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba2806b4d0 0x7fba2806d990 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fba34000f60 tx=0x7fba34007ca0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.680+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 msgr2=0x7fba4415c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.680+0000 7fba4bea4640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4415c250 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fba40009960 tx=0x7fba400046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 shutdown_connections 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba2806b4d0 0x7fba2806d990 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba4415bb40 0x7fba4415c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba440b4970 0x7fba4415c250 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 --2- 172.21.15.72:0/1685367784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba4406d5c0 0x7fba44140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 >> 172.21.15.72:0/1685367784 conn(0x7fba4406b190 msgr2=0x7fba440cdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 shutdown_connections 2024-03-06T07:22:13.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.681+0000 7fba4bea4640 1 -- 172.21.15.72:0/1685367784 wait complete. 2024-03-06T07:22:13.694 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:13.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 Processor -- start 2024-03-06T07:22:13.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 -- start start 2024-03-06T07:22:13.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a500d10c0 0x7f4a500d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015ecd0 0x7f4a500cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a500cfa50 con 0x7f4a500d14a0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5015c050 con 0x7f4a500d10c0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.795+0000 7f4a55df7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a500d0060 con 0x7f4a5015ecd0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a500d10c0 0x7f4a500d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4a500d14a0 0x7f4a5015e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38376/0 (socket says 172.21.15.72:38376) 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4effd640 1 -- 172.21.15.72:0/3817957194 learned_addr learned my addr 172.21.15.72:0/3817957194 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4ffff640 1 --2- 172.21.15.72:0/3817957194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015ecd0 0x7f4a500cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4f7fe640 1 -- 172.21.15.72:0/3817957194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015ecd0 msgr2=0x7f4a500cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/3817957194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015ecd0 0x7f4a500cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4f7fe640 1 -- 172.21.15.72:0/3817957194 >> v1:172.21.15.72:6789/0 conn(0x7f4a500d14a0 legacy=0x7f4a5015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4f7fe640 1 -- 172.21.15.72:0/3817957194 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500d0a30 con 0x7f4a500d10c0 2024-03-06T07:22:13.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.796+0000 7f4a4e7fc640 1 -- 172.21.15.72:0/3817957194 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852094348 0 0) 0x7f4a500cfa50 con 0x7f4a500d14a0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.797+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/3817957194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500d10c0 0x7f4a500d04f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4a400097a0 tx=0x7f4a4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=755bf8b9fcded464 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.797+0000 7f4a4e7fc640 1 -- 172.21.15.72:0/3817957194 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40035020 con 0x7f4a500d10c0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.797+0000 7f4a4e7fc640 1 -- 172.21.15.72:0/3817957194 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a40031830 con 0x7f4a500d10c0 2024-03-06T07:22:13.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.797+0000 7f4a4e7fc640 1 -- 172.21.15.72:0/3817957194 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a40031c00 con 0x7f4a500d10c0 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 -- 172.21.15.72:0/3817957194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500d10c0 msgr2=0x7f4a500d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 --2- 172.21.15.72:0/3817957194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500d10c0 0x7f4a500d04f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4a400097a0 tx=0x7f4a4002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 -- 172.21.15.72:0/3817957194 shutdown_connections 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 --2- 172.21.15.72:0/3817957194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015ecd0 0x7f4a500cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 --2- 172.21.15.72:0/3817957194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500d10c0 0x7f4a500d04f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 -- 172.21.15.72:0/3817957194 >> 172.21.15.72:0/3817957194 conn(0x7f4a500b3800 msgr2=0x7f4a500b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 -- 172.21.15.72:0/3817957194 shutdown_connections 2024-03-06T07:22:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.798+0000 7f4a55df7640 1 -- 172.21.15.72:0/3817957194 wait complete. 2024-03-06T07:22:13.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.799+0000 7f4a55df7640 1 Processor -- start 2024-03-06T07:22:13.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.800+0000 7f4a55df7640 1 -- start start 2024-03-06T07:22:13.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a55df7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d10c0 0x7f4a50152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a55df7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a500d14a0 0x7f4a500b6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a55df7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 0x7f4a5006cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a55df7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5015f440 con 0x7f4a500b6980 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a55df7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5015f160 con 0x7f4a500d14a0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a55df7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a500cfcf0 con 0x7f4a500d10c0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 0x7f4a5006cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 0x7f4a5006cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41918/0 (socket says 172.21.15.72:41918) 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 -- 172.21.15.72:0/2592003549 learned_addr learned my addr 172.21.15.72:0/2592003549 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4effd640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a500d14a0 0x7f4a500b6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d10c0 0x7f4a50152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 -- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d10c0 msgr2=0x7f4a50152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d10c0 0x7f4a50152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 -- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a500d14a0 msgr2=0x7f4a500b6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a500d14a0 0x7f4a500b6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 -- 172.21.15.72:0/2592003549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500b53a0 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d10c0 0x7f4a50152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a4ffff640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 0x7f4a5006cab0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800efc0 tx=0x7f4a4800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.801+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48018020 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.802+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48014480 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.802+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a500b5680 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.802+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48014870 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.802+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a500b5c40 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.802+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a50068ff0 con 0x7f4a500b6980 2024-03-06T07:22:13.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.803+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a48005d10 con 0x7f4a500b6980 2024-03-06T07:22:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.803+0000 7f4a37fff640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3c06b4d0 0x7f4a3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.803+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3c06b4d0 0x7f4a3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.803+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a4808ac80 con 0x7f4a500b6980 2024-03-06T07:22:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.804+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3c06b4d0 0x7f4a3c06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4a40036000 tx=0x7f4a40008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:13.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.808+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a48022070 con 0x7f4a500b6980 2024-03-06T07:22:13.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.981+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a5015dc70 con 0x7f4a500b6980 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.982+0000 7f4a37fff640 1 -- 172.21.15.72:0/2592003549 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a48010040 con 0x7f4a500b6980 2024-03-06T07:22:13.984 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3c06b4d0 msgr2=0x7f4a3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3c06b4d0 0x7f4a3c06d990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4a40036000 tx=0x7f4a40008030 comp rx=0 tx=0).stop 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 msgr2=0x7f4a5006cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 0x7f4a5006cab0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800efc0 tx=0x7f4a4800c490 comp rx=0 tx=0).stop 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 shutdown_connections 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3c06b4d0 0x7f4a3c06d990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500b6980 0x7f4a5006cab0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a500d14a0 0x7f4a500b6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 --2- 172.21.15.72:0/2592003549 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d10c0 0x7f4a50152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 >> 172.21.15.72:0/2592003549 conn(0x7f4a500b3800 msgr2=0x7f4a50160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 shutdown_connections 2024-03-06T07:22:13.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:13.984+0000 7f4a55df7640 1 -- 172.21.15.72:0/2592003549 wait complete. 2024-03-06T07:22:13.997 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:14.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.102+0000 7fcc22e8d640 1 Processor -- start 2024-03-06T07:22:14.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.102+0000 7fcc22e8d640 1 -- start start 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.102+0000 7fcc22e8d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.102+0000 7fcc22e8d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c15e4d0 con 0x7fcc1c15e0f0 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.102+0000 7fcc22e8d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c0d0460 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.102+0000 7fcc22e8d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0b62f0 con 0x7fcc1c072ca0 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc20c29640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcc1c15e0f0 0x7fcc1c072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38384/0 (socket says 172.21.15.72:38384) 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc20c29640 1 -- 172.21.15.72:0/213568879 learned_addr learned my addr 172.21.15.72:0/213568879 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1bfff640 1 --2- 172.21.15.72:0/213568879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2104999602 0 0) 0x7fcc1c15e4d0 con 0x7fcc1c15e0f0 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c003520 con 0x7fcc1c15e0f0 2024-03-06T07:22:14.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261072375 0 0) 0x7fcc1c0d0460 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c15e4d0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1bfff640 1 -- 172.21.15.72:0/213568879 >> v1:172.21.15.146:6789/0 conn(0x7fcc1c0ad3a0 legacy=0x7fcc1c0ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1bfff640 1 -- 172.21.15.72:0/213568879 >> v1:172.21.15.72:6789/0 conn(0x7fcc1c15e0f0 legacy=0x7fcc1c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.103+0000 7fcc1bfff640 1 -- 172.21.15.72:0/213568879 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0d0740 con 0x7fcc1c072ca0 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.104+0000 7fcc1bfff640 1 --2- 172.21.15.72:0/213568879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c0b5db0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc080097d0 tx=0x7fcc0802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1f4834c2fe12f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.104+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08035020 con 0x7fcc1c072ca0 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.104+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08031800 con 0x7fcc1c072ca0 2024-03-06T07:22:14.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.104+0000 7fcc1b7fe640 1 -- 172.21.15.72:0/213568879 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08031bb0 con 0x7fcc1c072ca0 2024-03-06T07:22:14.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 -- 172.21.15.72:0/213568879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 msgr2=0x7fcc1c0b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/213568879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c0b5db0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcc080097d0 tx=0x7fcc0802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 -- 172.21.15.72:0/213568879 shutdown_connections 2024-03-06T07:22:14.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/213568879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c0b5db0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 -- 172.21.15.72:0/213568879 >> 172.21.15.72:0/213568879 conn(0x7fcc1c0b4fe0 msgr2=0x7fcc1c0b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 -- 172.21.15.72:0/213568879 shutdown_connections 2024-03-06T07:22:14.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.105+0000 7fcc22e8d640 1 -- 172.21.15.72:0/213568879 wait complete. 2024-03-06T07:22:14.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.106+0000 7fcc22e8d640 1 Processor -- start 2024-03-06T07:22:14.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.106+0000 7fcc22e8d640 1 -- start start 2024-03-06T07:22:14.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc22e8d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c1605f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc22e8d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 0x7fcc1c15bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc22e8d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc1c15e0f0 0x7fcc1c143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc22e8d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0d0fe0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc22e8d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0d0d00 con 0x7fcc1c15e0f0 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc22e8d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c06b550 con 0x7fcc1c072ca0 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 0x7fcc1c15bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 0x7fcc1c15bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41934/0 (socket says 172.21.15.72:41934) 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 -- 172.21.15.72:0/630387891 learned_addr learned my addr 172.21.15.72:0/630387891 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc2142a640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc1c15e0f0 0x7fcc1c143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc20c29640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c1605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 -- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 msgr2=0x7fcc1c1605f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c1605f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 -- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc1c15e0f0 msgr2=0x7fcc1c143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc1c15e0f0 0x7fcc1c143370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 -- 172.21.15.72:0/630387891 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c15f550 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.107+0000 7fcc1bfff640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 0x7fcc1c15bba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcc080098f0 tx=0x7fcc080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.108+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08035040 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.108+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08003820 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.108+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c15f830 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.108+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08003bd0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.108+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c15fdf0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.109+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c09e4e0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.110+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc08038020 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.110+0000 7fcc197fa640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc0006b500 0x7fcc0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.110+0000 7fcc20c29640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc0006b500 0x7fcc0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.110+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc080abfe0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.111+0000 7fcc20c29640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc0006b500 0x7fcc0006d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000f60 tx=0x7fcc10007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.115+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc0807a820 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.287+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcc1c068ff0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.288+0000 7fcc197fa640 1 -- 172.21.15.72:0/630387891 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcc0807e8c0 con 0x7fcc1c0ad3a0 2024-03-06T07:22:14.290 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.289+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc0006b500 msgr2=0x7fcc0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.289+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc0006b500 0x7fcc0006d9c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcc10000f60 tx=0x7fcc10007d50 comp rx=0 tx=0).stop 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.289+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 msgr2=0x7fcc1c15bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.289+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 0x7fcc1c15bba0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcc080098f0 tx=0x7fcc080046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.289+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 shutdown_connections 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.289+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc0006b500 0x7fcc0006d9c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.290+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc1c15e0f0 0x7fcc1c143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.290+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc1c0ad3a0 0x7fcc1c15bba0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.290+0000 7fcc22e8d640 1 --2- 172.21.15.72:0/630387891 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc1c072ca0 0x7fcc1c1605f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.290+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 >> 172.21.15.72:0/630387891 conn(0x7fcc1c0b4fe0 msgr2=0x7fcc1c15d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.290+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 shutdown_connections 2024-03-06T07:22:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.290+0000 7fcc22e8d640 1 -- 172.21.15.72:0/630387891 wait complete. 2024-03-06T07:22:14.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.401+0000 7f06053ac640 1 Processor -- start 2024-03-06T07:22:14.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.401+0000 7f06053ac640 1 -- start start 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f06053ac640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f06000d00c0 0x7f06000d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f06053ac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f060006cf20 0x7f060006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f06053ac640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06000d09e0 con 0x7f06000cf850 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f06053ac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060006ca90 con 0x7f060006cf20 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f06053ac640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f060015e380 con 0x7f06000d00c0 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f060006cf20 0x7f060006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f060006cf20 0x7f060006d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41948/0 (socket says 172.21.15.72:41948) 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 -- 172.21.15.72:0/3210559272 learned_addr learned my addr 172.21.15.72:0/3210559272 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:14.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05feffd640 1 --2- 172.21.15.72:0/3210559272 >> v2:172.21.15.146:3300/0 conn(0x7f06000d00c0 0x7f06000d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 -- 172.21.15.72:0/3210559272 >> v2:172.21.15.146:3300/0 conn(0x7f06000d00c0 msgr2=0x7f06000d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 --2- 172.21.15.72:0/3210559272 >> v2:172.21.15.146:3300/0 conn(0x7f06000d00c0 0x7f06000d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 -- 172.21.15.72:0/3210559272 >> v1:172.21.15.146:6789/0 conn(0x7f06000cf850 legacy=0x7f06000cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05ff7fe640 1 -- 172.21.15.72:0/3210559272 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000cdc10 con 0x7f060006cf20 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.402+0000 7f05feffd640 1 --2- 172.21.15.72:0/3210559272 >> v2:172.21.15.146:3300/0 conn(0x7f06000d00c0 0x7f06000d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.403+0000 7f05ff7fe640 1 --2- 172.21.15.72:0/3210559272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f060006cf20 0x7f060006d3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06000d1a40 tx=0x7f05f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29f53110994358 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.403+0000 7f05fdffb640 1 -- 172.21.15.72:0/3210559272 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f802ca80 con 0x7f060006cf20 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.403+0000 7f05fdffb640 1 -- 172.21.15.72:0/3210559272 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f802cc20 con 0x7f060006cf20 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.403+0000 7f05fdffb640 1 -- 172.21.15.72:0/3210559272 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f80343f0 con 0x7f060006cf20 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 -- 172.21.15.72:0/3210559272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f060006cf20 msgr2=0x7f060006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 --2- 172.21.15.72:0/3210559272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f060006cf20 0x7f060006d3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06000d1a40 tx=0x7f05f802bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 -- 172.21.15.72:0/3210559272 shutdown_connections 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 --2- 172.21.15.72:0/3210559272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f060006cf20 0x7f060006d3b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 --2- 172.21.15.72:0/3210559272 >> v2:172.21.15.146:3300/0 conn(0x7f06000d00c0 0x7f06000d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 -- 172.21.15.72:0/3210559272 >> 172.21.15.72:0/3210559272 conn(0x7f06000b35a0 msgr2=0x7f06000b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 -- 172.21.15.72:0/3210559272 shutdown_connections 2024-03-06T07:22:14.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.404+0000 7f06053ac640 1 -- 172.21.15.72:0/3210559272 wait complete. 2024-03-06T07:22:14.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.406+0000 7f06053ac640 1 Processor -- start 2024-03-06T07:22:14.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.406+0000 7f06053ac640 1 -- start start 2024-03-06T07:22:14.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f06053ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060006cf20 0x7f06000b4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f06053ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06000cf850 0x7f06001f7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f06053ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 0x7f06001f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f06053ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060006c720 con 0x7f06000d00c0 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f06053ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060006c440 con 0x7f060006cf20 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f06053ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000b1030 con 0x7f06000cf850 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 0x7f06001f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 0x7f06001f76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41962/0 (socket says 172.21.15.72:41962) 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 -- 172.21.15.72:0/66626896 learned_addr learned my addr 172.21.15.72:0/66626896 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05feffd640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060006cf20 0x7f06000b4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05fe7fc640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06000cf850 0x7f06001f7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 -- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06000cf850 msgr2=0x7f06001f7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06000cf850 0x7f06001f7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 -- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060006cf20 msgr2=0x7f06000b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060006cf20 0x7f06000b4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 -- 172.21.15.72:0/66626896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000b5760 con 0x7f06000d00c0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.407+0000 7f05ff7fe640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 0x7f06001f76c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009f40 tx=0x7f05f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.408+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f8043020 con 0x7f06000d00c0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.408+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f803e450 con 0x7f06000d00c0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.408+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f803e820 con 0x7f06000d00c0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.408+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06000b5a40 con 0x7f06000d00c0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.408+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060014c060 con 0x7f06000d00c0 2024-03-06T07:22:14.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.409+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060004f330 con 0x7f06000d00c0 2024-03-06T07:22:14.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.409+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05f8046020 con 0x7f06000d00c0 2024-03-06T07:22:14.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.409+0000 7f05f77fe640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.410+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05f80aca40 con 0x7f06000d00c0 2024-03-06T07:22:14.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.410+0000 7f05feffd640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.410+0000 7f05feffd640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05e006b4d0 0x7f05e006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f05e800ab60 tx=0x7f05e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.414+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05f804d030 con 0x7f06000d00c0 2024-03-06T07:22:14.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.592+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06000ceb00 con 0x7f06000d00c0 2024-03-06T07:22:14.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.593+0000 7f05f77fe640 1 -- 172.21.15.72:0/66626896 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f05f807b950 con 0x7f06000d00c0 2024-03-06T07:22:14.596 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:14.596 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05e006b4d0 msgr2=0x7f05e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05e006b4d0 0x7f05e006d990 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f05e800ab60 tx=0x7f05e8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 msgr2=0x7f06001f76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 0x7f06001f76c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009f40 tx=0x7f05f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 shutdown_connections 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05e006b4d0 0x7f05e006d990 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06000d00c0 0x7f06001f76c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06000cf850 0x7f06001f7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 --2- 172.21.15.72:0/66626896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f060006cf20 0x7f06000b4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.595+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 >> 172.21.15.72:0/66626896 conn(0x7f06000b35a0 msgr2=0x7f06000cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.596+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 shutdown_connections 2024-03-06T07:22:14.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.596+0000 7f06053ac640 1 -- 172.21.15.72:0/66626896 wait complete. 2024-03-06T07:22:14.609 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:14.609 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 Processor -- start 2024-03-06T07:22:14.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 -- start start 2024-03-06T07:22:14.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7a815c050 0x7fd7a806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a80d0af0 0x7fd7a80cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80cd4b0 con 0x7fd7a806fb40 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a806ffb0 con 0x7fd7a815c050 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.711+0000 7fd7ad215640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a80cd790 con 0x7fd7a80d0af0 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd7a6d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7a815c050 0x7fd7a806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd79e59c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd7a806fb40 0x7fd7a80cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:35986/0 (socket says 172.21.15.72:35986) 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd79e59c640 1 -- 172.21.15.72:0/4025795409 learned_addr learned my addr 172.21.15.72:0/4025795409 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd7a759e640 1 --2- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a80d0af0 0x7fd7a80cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd7a6d9d640 1 -- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a80d0af0 msgr2=0x7fd7a80cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd7a6d9d640 1 --2- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a80d0af0 0x7fd7a80cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd7a6d9d640 1 -- 172.21.15.72:0/4025795409 >> v1:172.21.15.146:6789/0 conn(0x7fd7a806fb40 legacy=0x7fd7a80cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.712+0000 7fd7a6d9d640 1 -- 172.21.15.72:0/4025795409 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80cda70 con 0x7fd7a815c050 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.713+0000 7fd7a6d9d640 1 --2- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815c050 0x7fd7a806f600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7900097a0 tx=0x7fd79002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e07e258f71bbba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.713+0000 7fd7a659c640 1 -- 172.21.15.72:0/4025795409 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900315a0 con 0x7fd7a815c050 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.713+0000 7fd7a659c640 1 -- 172.21.15.72:0/4025795409 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd790037070 con 0x7fd7a815c050 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.713+0000 7fd7a659c640 1 -- 172.21.15.72:0/4025795409 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd790031960 con 0x7fd7a815c050 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 -- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815c050 msgr2=0x7fd7a806f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 --2- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815c050 0x7fd7a806f600 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7900097a0 tx=0x7fd79002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 -- 172.21.15.72:0/4025795409 shutdown_connections 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 --2- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a80d0af0 0x7fd7a80cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 --2- 172.21.15.72:0/4025795409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a815c050 0x7fd7a806f600 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 -- 172.21.15.72:0/4025795409 >> 172.21.15.72:0/4025795409 conn(0x7fd7a80b3800 msgr2=0x7fd7a80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 -- 172.21.15.72:0/4025795409 shutdown_connections 2024-03-06T07:22:14.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.714+0000 7fd7ad215640 1 -- 172.21.15.72:0/4025795409 wait complete. 2024-03-06T07:22:14.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.716+0000 7fd7ad215640 1 Processor -- start 2024-03-06T07:22:14.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.716+0000 7fd7ad215640 1 -- start start 2024-03-06T07:22:14.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.717+0000 7fd7ad215640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a806fb40 0x7fd7a81453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.717+0000 7fd7ad215640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a80d0af0 0x7fd7a80b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.717+0000 7fd7ad215640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 0x7fd7a80b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.717+0000 7fd7ad215640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8165c30 con 0x7fd7a815c050 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.717+0000 7fd7ad215640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8165ab0 con 0x7fd7a80d0af0 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.717+0000 7fd7ad215640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a8165db0 con 0x7fd7a806fb40 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 0x7fd7a80b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 0x7fd7a80b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41976/0 (socket says 172.21.15.72:41976) 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 -- 172.21.15.72:0/185347773 learned_addr learned my addr 172.21.15.72:0/185347773 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd79e59c640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a80d0af0 0x7fd7a80b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a6d9d640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a806fb40 0x7fd7a81453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 -- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a806fb40 msgr2=0x7fd7a81453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a806fb40 0x7fd7a81453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 -- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a80d0af0 msgr2=0x7fd7a80b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a80d0af0 0x7fd7a80b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 -- 172.21.15.72:0/185347773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a8156e60 con 0x7fd7a815c050 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7a759e640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 0x7fd7a80b0db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000efc0 tx=0x7fd7a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0018020 con 0x7fd7a815c050 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a0014480 con 0x7fd7a815c050 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0009e60 con 0x7fd7a815c050 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a8157140 con 0x7fd7a815c050 2024-03-06T07:22:14.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.718+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a8157700 con 0x7fd7a815c050 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.720+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a80707e0 con 0x7fd7a815c050 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.720+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7a0005d10 con 0x7fd7a815c050 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.720+0000 7fd79ffff640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd78c06b4d0 0x7fd78c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.720+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7a008aee0 con 0x7fd7a815c050 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.720+0000 7fd7a6d9d640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd78c06b4d0 0x7fd78c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:14.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.721+0000 7fd7a6d9d640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd78c06b4d0 0x7fd78c06d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd790036000 tx=0x7fd79003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:14.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.725+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7a80707e0 con 0x7fd7a815c050 2024-03-06T07:22:14.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.902+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7a8068ff0 con 0x7fd7a815c050 2024-03-06T07:22:14.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.902+0000 7fd79ffff640 1 -- 172.21.15.72:0/185347773 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7a0022030 con 0x7fd7a815c050 2024-03-06T07:22:14.905 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd78c06b4d0 msgr2=0x7fd78c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd78c06b4d0 0x7fd78c06d990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd790036000 tx=0x7fd79003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 msgr2=0x7fd7a80b0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 0x7fd7a80b0db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000efc0 tx=0x7fd7a000c490 comp rx=0 tx=0).stop 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 shutdown_connections 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd78c06b4d0 0x7fd78c06d990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7a815c050 0x7fd7a80b0db0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7a80d0af0 0x7fd7a80b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 --2- 172.21.15.72:0/185347773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7a806fb40 0x7fd7a81453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 >> 172.21.15.72:0/185347773 conn(0x7fd7a80b3800 msgr2=0x7fd7a80b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 shutdown_connections 2024-03-06T07:22:14.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:14.904+0000 7fd7ad215640 1 -- 172.21.15.72:0/185347773 wait complete. 2024-03-06T07:22:14.917 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:15.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.019+0000 7fa991b65640 1 Processor -- start 2024-03-06T07:22:15.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.019+0000 7fa991b65640 1 -- start start 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa991b65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa98c1614d0 0x7fa98c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa991b65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d24b0 0x7fa98c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa991b65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98c1618b0 con 0x7fa98c0d1910 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa991b65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c0cf400 con 0x7fa98c1614d0 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa991b65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0cf6a0 con 0x7fa98c0d24b0 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa98c1614d0 0x7fa98c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa98c0d1910 0x7fa98c0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38388/0 (socket says 172.21.15.72:38388) 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98affd640 1 -- 172.21.15.72:0/2294403393 learned_addr learned my addr 172.21.15.72:0/2294403393 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98b7fe640 1 -- 172.21.15.72:0/2294403393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d24b0 msgr2=0x7fa98c0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98b7fe640 1 --2- 172.21.15.72:0/2294403393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d24b0 0x7fa98c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98b7fe640 1 -- 172.21.15.72:0/2294403393 >> v1:172.21.15.72:6789/0 conn(0x7fa98c0d1910 legacy=0x7fa98c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98b7fe640 1 -- 172.21.15.72:0/2294403393 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c0cf980 con 0x7fa98c1614d0 2024-03-06T07:22:15.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.020+0000 7fa98a7fc640 1 -- 172.21.15.72:0/2294403393 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345717500 0 0) 0x7fa98c1618b0 con 0x7fa98c0d1910 2024-03-06T07:22:15.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa98b7fe640 1 --2- 172.21.15.72:0/2294403393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c1614d0 0x7fa98c0d4860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0097a0 tx=0x7fa97c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88be3070d9c7f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa98a7fc640 1 -- 172.21.15.72:0/2294403393 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c035020 con 0x7fa98c1614d0 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa98a7fc640 1 -- 172.21.15.72:0/2294403393 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97c031830 con 0x7fa98c1614d0 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa98a7fc640 1 -- 172.21.15.72:0/2294403393 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c031c00 con 0x7fa98c1614d0 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 -- 172.21.15.72:0/2294403393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c1614d0 msgr2=0x7fa98c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 --2- 172.21.15.72:0/2294403393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c1614d0 0x7fa98c0d4860 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0097a0 tx=0x7fa97c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 -- 172.21.15.72:0/2294403393 shutdown_connections 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 --2- 172.21.15.72:0/2294403393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d24b0 0x7fa98c0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 --2- 172.21.15.72:0/2294403393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c1614d0 0x7fa98c0d4860 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 -- 172.21.15.72:0/2294403393 >> 172.21.15.72:0/2294403393 conn(0x7fa98c0d3d40 msgr2=0x7fa98c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 -- 172.21.15.72:0/2294403393 shutdown_connections 2024-03-06T07:22:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.022+0000 7fa991b65640 1 -- 172.21.15.72:0/2294403393 wait complete. 2024-03-06T07:22:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.023+0000 7fa991b65640 1 Processor -- start 2024-03-06T07:22:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 -- start start 2024-03-06T07:22:15.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d1910 0x7fa98c1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 0x7fa98c1f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98c1614d0 0x7fa98c0b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b1370 con 0x7fa98c0d24b0 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b0910 con 0x7fa98c1614d0 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa991b65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0b14f0 con 0x7fa98c0d1910 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa98affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 0x7fa98c1f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa98affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 0x7fa98c1f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41986/0 (socket says 172.21.15.72:41986) 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.024+0000 7fa98affd640 1 -- 172.21.15.72:0/3080925937 learned_addr learned my addr 172.21.15.72:0/3080925937 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98bfff640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98c1614d0 0x7fa98c0b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98b7fe640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d1910 0x7fa98c1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98affd640 1 -- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d1910 msgr2=0x7fa98c1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98affd640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d1910 0x7fa98c1551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98affd640 1 -- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98c1614d0 msgr2=0x7fa98c0b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98affd640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98c1614d0 0x7fa98c0b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98affd640 1 -- 172.21.15.72:0/3080925937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c1f7bf0 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98bfff640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98c1614d0 0x7fa98c0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa98affd640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 0x7fa98c1f71a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c4d0 tx=0x7fa98000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980018070 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa980007500 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c1f7ed0 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa980004a20 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.025+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c155730 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.026+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c068ff0 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.027+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa98001b070 con 0x7fa98c0d24b0 2024-03-06T07:22:15.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.027+0000 7fa973fff640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.027+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa980089510 con 0x7fa98c0d24b0 2024-03-06T07:22:15.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.027+0000 7fa98b7fe640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.028+0000 7fa98b7fe640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa97806b4d0 0x7fa97806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa97c036000 tx=0x7fa97c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.036+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa980058390 con 0x7fa98c0d24b0 2024-03-06T07:22:15.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.216+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa98c1f80a0 con 0x7fa98c0d24b0 2024-03-06T07:22:15.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.216+0000 7fa973fff640 1 -- 172.21.15.72:0/3080925937 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa980057c90 con 0x7fa98c0d24b0 2024-03-06T07:22:15.218 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:15.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa97806b4d0 msgr2=0x7fa97806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa97806b4d0 0x7fa97806d990 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa97c036000 tx=0x7fa97c03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 msgr2=0x7fa98c1f71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 0x7fa98c1f71a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c4d0 tx=0x7fa98000c990 comp rx=0 tx=0).stop 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 shutdown_connections 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa97806b4d0 0x7fa97806d990 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98c1614d0 0x7fa98c0b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98c0d24b0 0x7fa98c1f71a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 --2- 172.21.15.72:0/3080925937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98c0d1910 0x7fa98c1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 >> 172.21.15.72:0/3080925937 conn(0x7fa98c0d3d40 msgr2=0x7fa98c0cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 shutdown_connections 2024-03-06T07:22:15.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.218+0000 7fa991b65640 1 -- 172.21.15.72:0/3080925937 wait complete. 2024-03-06T07:22:15.235 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:15.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.337+0000 7fc7582a4640 1 Processor -- start 2024-03-06T07:22:15.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.337+0000 7fc7582a4640 1 -- start start 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc7582a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc750144560 0x7fc7500cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc7582a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc75015e680 0x7fc7500d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc7582a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7500ce190 con 0x7fc75006f2c0 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc7582a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc750144940 con 0x7fc750144560 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc7582a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75015eb10 con 0x7fc75015e680 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc756040640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc750144560 0x7fc7500cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc756841640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc75006f2c0 0x7fc7500b44f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38400/0 (socket says 172.21.15.72:38400) 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc756040640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc750144560 0x7fc7500cdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41998/0 (socket says 172.21.15.72:41998) 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.338+0000 7fc756841640 1 -- 172.21.15.72:0/1911378096 learned_addr learned my addr 172.21.15.72:0/1911378096 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.339+0000 7fc756040640 1 -- 172.21.15.72:0/1911378096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc75015e680 msgr2=0x7fc7500d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.339+0000 7fc756040640 1 --2- 172.21.15.72:0/1911378096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc75015e680 0x7fc7500d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.339+0000 7fc756040640 1 -- 172.21.15.72:0/1911378096 >> v1:172.21.15.72:6789/0 conn(0x7fc75006f2c0 legacy=0x7fc7500b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.339+0000 7fc75503e640 1 -- 172.21.15.72:0/1911378096 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722017772 0 0) 0x7fc748001490 con 0x7fc75006f2c0 2024-03-06T07:22:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.339+0000 7fc756040640 1 -- 172.21.15.72:0/1911378096 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75015edf0 con 0x7fc750144560 2024-03-06T07:22:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.339+0000 7fc756040640 1 --2- 172.21.15.72:0/1911378096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cdc50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097a0 tx=0x7fc74c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da52434b343a86c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc75503e640 1 -- 172.21.15.72:0/1911378096 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c035020 con 0x7fc750144560 2024-03-06T07:22:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc75503e640 1 -- 172.21.15.72:0/1911378096 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c031830 con 0x7fc750144560 2024-03-06T07:22:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc75503e640 1 -- 172.21.15.72:0/1911378096 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c031c00 con 0x7fc750144560 2024-03-06T07:22:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 -- 172.21.15.72:0/1911378096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 msgr2=0x7fc7500cdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1911378096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cdc50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc74c0097a0 tx=0x7fc74c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 -- 172.21.15.72:0/1911378096 shutdown_connections 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1911378096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc75015e680 0x7fc7500d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1911378096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cdc50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 -- 172.21.15.72:0/1911378096 >> 172.21.15.72:0/1911378096 conn(0x7fc75006c0c0 msgr2=0x7fc75006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 -- 172.21.15.72:0/1911378096 shutdown_connections 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.340+0000 7fc7582a4640 1 -- 172.21.15.72:0/1911378096 wait complete. 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.341+0000 7fc7582a4640 1 Processor -- start 2024-03-06T07:22:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.341+0000 7fc7582a4640 1 -- start start 2024-03-06T07:22:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc7582a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc75006f2c0 0x7fc75015cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc7582a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc7582a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7500cc230 0x7fc7501372f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc7582a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75006e100 con 0x7fc750144560 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc7582a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75015f3b0 con 0x7fc75006f2c0 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc7582a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75006e280 con 0x7fc7500cc230 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc75583f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc75583f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42000/0 (socket says 172.21.15.72:42000) 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc75583f640 1 -- 172.21.15.72:0/1092115546 learned_addr learned my addr 172.21.15.72:0/1092115546 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc756040640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc75006f2c0 0x7fc75015cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.342+0000 7fc756841640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7500cc230 0x7fc7501372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc75583f640 1 -- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7500cc230 msgr2=0x7fc7501372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc75583f640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7500cc230 0x7fc7501372f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc75583f640 1 -- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc75006f2c0 msgr2=0x7fc75015cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc75583f640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc75006f2c0 0x7fc75015cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc75583f640 1 -- 172.21.15.72:0/1092115546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750138370 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc756040640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc75006f2c0 0x7fc75015cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc75583f640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cbcf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b410 tx=0x7fc74000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740015020 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc740011850 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc750138650 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740011c40 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.343+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7500a9660 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.344+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7500a9450 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.344+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc740018020 con 0x7fc750144560 2024-03-06T07:22:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.345+0000 7fc746ffd640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73406b500 0x7fc73406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.345+0000 7fc756040640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73406b500 0x7fc73406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.345+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc74008a160 con 0x7fc750144560 2024-03-06T07:22:15.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.345+0000 7fc756040640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73406b500 0x7fc73406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036000 tx=0x7fc74c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.349+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc74001f020 con 0x7fc750144560 2024-03-06T07:22:15.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.521+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7500733b0 con 0x7fc750144560 2024-03-06T07:22:15.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.522+0000 7fc746ffd640 1 -- 172.21.15.72:0/1092115546 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc7400172c0 con 0x7fc750144560 2024-03-06T07:22:15.524 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73406b500 msgr2=0x7fc73406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73406b500 0x7fc73406d9c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc74c036000 tx=0x7fc74c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 msgr2=0x7fc7500cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cbcf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b410 tx=0x7fc74000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 shutdown_connections 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73406b500 0x7fc73406d9c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7500cc230 0x7fc7501372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc750144560 0x7fc7500cbcf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 --2- 172.21.15.72:0/1092115546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc75006f2c0 0x7fc75015cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 >> 172.21.15.72:0/1092115546 conn(0x7fc75006c0c0 msgr2=0x7fc7500b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 shutdown_connections 2024-03-06T07:22:15.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.524+0000 7fc7582a4640 1 -- 172.21.15.72:0/1092115546 wait complete. 2024-03-06T07:22:15.547 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-03-06T07:22:15.547 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:22:15.548 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:15.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.649+0000 7f54f6437640 1 Processor -- start 2024-03-06T07:22:15.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.649+0000 7f54f6437640 1 -- start start 2024-03-06T07:22:15.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54f6437640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54f00d2710 0x7f54f00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54f6437640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f00b3430 con 0x7f54f00d22b0 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54f6437640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f00b3710 con 0x7f54f015dfe0 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54f6437640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f00b31e0 con 0x7f54f00d2710 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54ef7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f54f00d22b0 0x7f54f00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38408/0 (socket says 172.21.15.72:38408) 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54ef7fe640 1 -- 172.21.15.72:0/1780554068 learned_addr learned my addr 172.21.15.72:0/1780554068 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54f49d4640 1 --2- 172.21.15.72:0/1780554068 >> v2:172.21.15.146:3300/0 conn(0x7f54f00d2710 0x7f54f00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640616838 0 0) 0x7f54f00b3430 con 0x7f54f00d22b0 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d8003520 con 0x7f54f00d22b0 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2303652379 0 0) 0x7f54d8003520 con 0x7f54f00d22b0 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 >> v2:172.21.15.146:3300/0 conn(0x7f54f00d2710 msgr2=0x7f54f00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 --2- 172.21.15.72:0/1780554068 >> v2:172.21.15.146:3300/0 conn(0x7f54f00d2710 0x7f54f00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 >> v1:172.21.15.146:6789/0 conn(0x7f54f015dfe0 legacy=0x7f54f00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f00b39f0 con 0x7f54f00d22b0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f54e4001670 con 0x7f54f00d22b0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 >> v1:172.21.15.72:6789/0 conn(0x7f54f00d22b0 legacy=0x7f54f00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d80039e0 0x7f54d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54d8004800 0x7f54d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d80051c0 0x7f54d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00b31e0 con 0x7f54d8004800 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d800c090 con 0x7f54d80039e0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.650+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d800c290 con 0x7f54d80051c0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54d8004800 0x7f54d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54effff640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d80039e0 0x7f54d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54f49d4640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d80051c0 0x7f54d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54ef7fe640 1 -- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d80051c0 msgr2=0x7f54d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d80051c0 0x7f54d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54ef7fe640 1 -- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d80039e0 msgr2=0x7f54d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54d80039e0 0x7f54d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54ef7fe640 1 -- 172.21.15.72:0/1780554068 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8003520 con 0x7f54d8004800 2024-03-06T07:22:15.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54f49d4640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d80051c0 0x7f54d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:15.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.651+0000 7f54f49d4640 1 -- 172.21.15.72:0/1780554068 reap_dead start 2024-03-06T07:22:15.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.652+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54d8004800 0x7f54d8004c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c240 tx=0x7f54d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66781f6317290e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.652+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4007d10 con 0x7f54d8004800 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.652+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e4030070 con 0x7f54d8004800 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.652+0000 7f54eeffd640 1 -- 172.21.15.72:0/1780554068 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4005d90 con 0x7f54d8004800 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 -- 172.21.15.72:0/1780554068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54d8004800 msgr2=0x7f54d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54d8004800 0x7f54d8004c80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c240 tx=0x7f54d8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 -- 172.21.15.72:0/1780554068 shutdown_connections 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 --2- 172.21.15.72:0/1780554068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54d8004800 0x7f54d8004c80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 -- 172.21.15.72:0/1780554068 >> 172.21.15.72:0/1780554068 conn(0x7f54f00d3cd0 msgr2=0x7f54f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 -- 172.21.15.72:0/1780554068 shutdown_connections 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.653+0000 7f54f6437640 1 -- 172.21.15.72:0/1780554068 wait complete. 2024-03-06T07:22:15.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.654+0000 7f54f6437640 1 Processor -- start 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.654+0000 7f54f6437640 1 -- start start 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f6437640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d8004800 0x7f54f00b9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f6437640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 0x7f54f014ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f6437640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54f014bf70 0x7f54f015f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f6437640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00d1c70 con 0x7f54f00b8d40 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f6437640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00b3d60 con 0x7f54f014bf70 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f6437640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f00d1df0 con 0x7f54d8004800 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 0x7f54f014ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 0x7f54f014ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42022/0 (socket says 172.21.15.72:42022) 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 -- 172.21.15.72:0/2404274183 learned_addr learned my addr 172.21.15.72:0/2404274183 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54f49d4640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54f014bf70 0x7f54f015f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54effff640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d8004800 0x7f54f00b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 -- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d8004800 msgr2=0x7f54f00b9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d8004800 0x7f54f00b9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 -- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54f014bf70 msgr2=0x7f54f015f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54f014bf70 0x7f54f015f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 -- 172.21.15.72:0/2404274183 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f00ca200 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ef7fe640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 0x7f54f014ba30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c240 tx=0x7f54f00b3600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.655+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4045070 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.656+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e40049f0 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.656+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f00ca4e0 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.656+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4007d10 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.656+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f0153940 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.657+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f004f330 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.657+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f54e4007070 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.657+0000 7f54ecff9640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f54dc06b4d0 0x7f54dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.657+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54e40aaec0 con 0x7f54f00b8d40 2024-03-06T07:22:15.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.657+0000 7f54effff640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f54dc06b4d0 0x7f54dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.658+0000 7f54effff640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f54dc06b4d0 0x7f54dc06d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f54e0000c00 tx=0x7f54e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.662+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f54e4079dd0 con 0x7f54f00b8d40 2024-03-06T07:22:15.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.834+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54f00b09c0 con 0x7f54f00b8d40 2024-03-06T07:22:15.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.835+0000 7f54ecff9640 1 -- 172.21.15.72:0/2404274183 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f54e40796d0 con 0x7f54f00b8d40 2024-03-06T07:22:15.837 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:15.837 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f54dc06b4d0 msgr2=0x7f54dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f54dc06b4d0 0x7f54dc06d990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f54e0000c00 tx=0x7f54e0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 msgr2=0x7f54f014ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 0x7f54f014ba30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f54e400c240 tx=0x7f54f00b3600 comp rx=0 tx=0).stop 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 shutdown_connections 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f54dc06b4d0 0x7f54dc06d990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54f014bf70 0x7f54f015f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54f00b8d40 0x7f54f014ba30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 --2- 172.21.15.72:0/2404274183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54d8004800 0x7f54f00b9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 >> 172.21.15.72:0/2404274183 conn(0x7f54f00d3cd0 msgr2=0x7f54f00b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 shutdown_connections 2024-03-06T07:22:15.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.837+0000 7f54f6437640 1 -- 172.21.15.72:0/2404274183 wait complete. 2024-03-06T07:22:15.850 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:15.850 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:15.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.953+0000 7ff795288640 1 Processor -- start 2024-03-06T07:22:15.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.953+0000 7ff795288640 1 -- start start 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff795288640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7901614d0 0x7ff7900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff795288640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff7900d1910 0x7ff7900d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff795288640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7901618b0 con 0x7ff7900d22e0 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff795288640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7900cf400 con 0x7ff7900d1910 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff795288640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7900d2710 con 0x7ff7901614d0 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff7900d1910 0x7ff7900d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff7900d1910 0x7ff7900d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42036/0 (socket says 172.21.15.72:42036) 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff7900d22e0 0x7ff7900d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38418/0 (socket says 172.21.15.72:38418) 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 -- 172.21.15.72:0/1443644754 learned_addr learned my addr 172.21.15.72:0/1443644754 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 -- 172.21.15.72:0/1443644754 >> v2:172.21.15.146:3300/0 conn(0x7ff7901614d0 msgr2=0x7ff7900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78effd640 1 --2- 172.21.15.72:0/1443644754 >> v2:172.21.15.146:3300/0 conn(0x7ff7901614d0 0x7ff7900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 --2- 172.21.15.72:0/1443644754 >> v2:172.21.15.146:3300/0 conn(0x7ff7901614d0 0x7ff7900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78dffb640 1 -- 172.21.15.72:0/1443644754 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334943597 0 0) 0x7ff7901618b0 con 0x7ff7900d22e0 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 -- 172.21.15.72:0/1443644754 >> v1:172.21.15.72:6789/0 conn(0x7ff7900d22e0 legacy=0x7ff7900d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78e7fc640 1 -- 172.21.15.72:0/1443644754 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900cf6a0 con 0x7ff7900d1910 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.954+0000 7ff78effd640 1 --2- 172.21.15.72:0/1443644754 >> v2:172.21.15.146:3300/0 conn(0x7ff7901614d0 0x7ff7900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.955+0000 7ff78e7fc640 1 --2- 172.21.15.72:0/1443644754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7900d1910 0x7ff7900d1da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff79008c470 tx=0x7ff78402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b96b4b5380ad3a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.955+0000 7ff78dffb640 1 -- 172.21.15.72:0/1443644754 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784034070 con 0x7ff7900d1910 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.955+0000 7ff78dffb640 1 -- 172.21.15.72:0/1443644754 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff784030860 con 0x7ff7900d1910 2024-03-06T07:22:15.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.955+0000 7ff78dffb640 1 -- 172.21.15.72:0/1443644754 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784030c30 con 0x7ff7900d1910 2024-03-06T07:22:15.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 -- 172.21.15.72:0/1443644754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7900d1910 msgr2=0x7ff7900d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 --2- 172.21.15.72:0/1443644754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7900d1910 0x7ff7900d1da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff79008c470 tx=0x7ff78402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 -- 172.21.15.72:0/1443644754 shutdown_connections 2024-03-06T07:22:15.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 --2- 172.21.15.72:0/1443644754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7900d1910 0x7ff7900d1da0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 --2- 172.21.15.72:0/1443644754 >> v2:172.21.15.146:3300/0 conn(0x7ff7901614d0 0x7ff7900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 -- 172.21.15.72:0/1443644754 >> 172.21.15.72:0/1443644754 conn(0x7ff7900d3d40 msgr2=0x7ff7900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 -- 172.21.15.72:0/1443644754 shutdown_connections 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.956+0000 7ff795288640 1 -- 172.21.15.72:0/1443644754 wait complete. 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.957+0000 7ff795288640 1 Processor -- start 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.957+0000 7ff795288640 1 -- start start 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff795288640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7900d1910 0x7ff7900ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff795288640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7900d22e0 0x7ff79015da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff795288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 0x7ff790147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff795288640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900b0910 con 0x7ff7901614d0 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff795288640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900cfc60 con 0x7ff7900d22e0 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff795288640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7900b1e40 con 0x7ff7900d1910 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 0x7ff790147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 0x7ff790147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42044/0 (socket says 172.21.15.72:42044) 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 -- 172.21.15.72:0/4086371426 learned_addr learned my addr 172.21.15.72:0/4086371426 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:15.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78e7fc640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7900d22e0 0x7ff79015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78effd640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7900d1910 0x7ff7900ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 -- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7900d1910 msgr2=0x7ff7900ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7900d1910 0x7ff7900ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 -- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7900d22e0 msgr2=0x7ff79015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7900d22e0 0x7ff79015da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 -- 172.21.15.72:0/4086371426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79015e360 con 0x7ff7901614d0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.958+0000 7ff78f7fe640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 0x7ff790147da0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c7e0 tx=0x7ff78800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.959+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff788017070 con 0x7ff7901614d0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.959+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff78800a640 con 0x7ff7901614d0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.959+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79015e640 con 0x7ff7901614d0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.959+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78800aa10 con 0x7ff7901614d0 2024-03-06T07:22:15.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.959+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7900ccc90 con 0x7ff7901614d0 2024-03-06T07:22:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.960+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff790068ff0 con 0x7ff7901614d0 2024-03-06T07:22:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.960+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff78801a070 con 0x7ff7901614d0 2024-03-06T07:22:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.960+0000 7ff7777fe640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.961+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff788089860 con 0x7ff7901614d0 2024-03-06T07:22:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.961+0000 7ff78effd640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.961+0000 7ff78effd640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff78000b840 tx=0x7ff780008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:15.965+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff788058770 con 0x7ff7901614d0 2024-03-06T07:22:16.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.141+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff79015f720 con 0x7ff7901614d0 2024-03-06T07:22:16.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.142+0000 7ff7777fe640 1 -- 172.21.15.72:0/4086371426 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff788089030 con 0x7ff7901614d0 2024-03-06T07:22:16.144 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff77c06b4d0 msgr2=0x7ff77c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff77c06b4d0 0x7ff77c06d990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff78000b840 tx=0x7ff780008040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 msgr2=0x7ff790147da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 0x7ff790147da0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c7e0 tx=0x7ff78800cca0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 shutdown_connections 2024-03-06T07:22:16.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff77c06b4d0 0x7ff77c06d990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7901614d0 0x7ff790147da0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7900d22e0 0x7ff79015da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 --2- 172.21.15.72:0/4086371426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7900d1910 0x7ff7900ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 >> 172.21.15.72:0/4086371426 conn(0x7ff7900d3d40 msgr2=0x7ff7900d0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 shutdown_connections 2024-03-06T07:22:16.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.144+0000 7ff795288640 1 -- 172.21.15.72:0/4086371426 wait complete. 2024-03-06T07:22:16.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:16.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.258+0000 7f6690e39640 1 Processor -- start 2024-03-06T07:22:16.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.258+0000 7f6690e39640 1 -- start start 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6690e39640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f668c0cfa90 0x7f668c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6690e39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0b33a0 0x7f668c0b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6690e39640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c158ac0 con 0x7f668c1586e0 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6690e39640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c15b2a0 con 0x7f668c0cfa90 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6690e39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15b540 con 0x7f668c0b33a0 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f668c0cfa90 0x7f668c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f668c0cfa90 0x7f668c0cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42052/0 (socket says 172.21.15.72:42052) 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 -- 172.21.15.72:0/3495910065 learned_addr learned my addr 172.21.15.72:0/3495910065 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 -- 172.21.15.72:0/3495910065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0b33a0 msgr2=0x7f668c0b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f668ad9d640 1 --2- 172.21.15.72:0/3495910065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0b33a0 0x7f668c0b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 --2- 172.21.15.72:0/3495910065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0b33a0 0x7f668c0b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 -- 172.21.15.72:0/3495910065 >> v1:172.21.15.146:6789/0 conn(0x7f668c1586e0 legacy=0x7f668c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.259+0000 7f6689d9b640 1 -- 172.21.15.72:0/3495910065 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c15b820 con 0x7f668c0cfa90 2024-03-06T07:22:16.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.260+0000 7f6689d9b640 1 --2- 172.21.15.72:0/3495910065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c0cfa90 0x7f668c0cff40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f667c009870 tx=0x7f667c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c03cb71e23449e59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.260+0000 7f668959a640 1 -- 172.21.15.72:0/3495910065 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c035020 con 0x7f668c0cfa90 2024-03-06T07:22:16.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.260+0000 7f668959a640 1 -- 172.21.15.72:0/3495910065 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f667c031640 con 0x7f668c0cfa90 2024-03-06T07:22:16.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.260+0000 7f668959a640 1 -- 172.21.15.72:0/3495910065 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f667c02d040 con 0x7f668c0cfa90 2024-03-06T07:22:16.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 -- 172.21.15.72:0/3495910065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c0cfa90 msgr2=0x7f668c0cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 --2- 172.21.15.72:0/3495910065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c0cfa90 0x7f668c0cff40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f667c009870 tx=0x7f667c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 -- 172.21.15.72:0/3495910065 shutdown_connections 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 --2- 172.21.15.72:0/3495910065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0b33a0 0x7f668c0b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 --2- 172.21.15.72:0/3495910065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c0cfa90 0x7f668c0cff40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 -- 172.21.15.72:0/3495910065 >> 172.21.15.72:0/3495910065 conn(0x7f668c06b190 msgr2=0x7f668c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 -- 172.21.15.72:0/3495910065 shutdown_connections 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.261+0000 7f6690e39640 1 -- 172.21.15.72:0/3495910065 wait complete. 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.262+0000 7f6690e39640 1 Processor -- start 2024-03-06T07:22:16.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.262+0000 7f6690e39640 1 -- start start 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6690e39640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f668c0b33a0 0x7f668c0cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6690e39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0cfa90 0x7f668c0acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6690e39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 0x7f668c06d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6690e39640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c0af150 con 0x7f668c1586e0 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6690e39640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15bde0 con 0x7f668c0b33a0 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6690e39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c071350 con 0x7f668c0cfa90 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f668ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 0x7f668c06d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f668ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 0x7f668c06d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42060/0 (socket says 172.21.15.72:42060) 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f668ad9d640 1 -- 172.21.15.72:0/3211674798 learned_addr learned my addr 172.21.15.72:0/3211674798 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f668a59c640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f668c0b33a0 0x7f668c0cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.263+0000 7f6689d9b640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0cfa90 0x7f668c0acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668ad9d640 1 -- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0cfa90 msgr2=0x7f668c0acc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668ad9d640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0cfa90 0x7f668c0acc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668ad9d640 1 -- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f668c0b33a0 msgr2=0x7f668c0cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668ad9d640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f668c0b33a0 0x7f668c0cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668ad9d640 1 -- 172.21.15.72:0/3211674798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c0ad150 con 0x7f668c1586e0 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668a59c640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f668c0b33a0 0x7f668c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f668ad9d640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 0x7f668c06d280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f668400cc30 tx=0x7f668400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f668400ce40 con 0x7f668c1586e0 2024-03-06T07:22:16.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6684003990 con 0x7f668c1586e0 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684003d40 con 0x7f668c1586e0 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c1509d0 con 0x7f668c1586e0 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.264+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c150f90 con 0x7f668c1586e0 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.265+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c157f40 con 0x7f668c1586e0 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.265+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6684003ee0 con 0x7f668c1586e0 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.266+0000 7f667affd640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f666c06b4d0 0x7f666c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.266+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f668408a6e0 con 0x7f668c1586e0 2024-03-06T07:22:16.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.266+0000 7f668a59c640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f666c06b4d0 0x7f666c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.266+0000 7f668a59c640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f666c06b4d0 0x7f666c06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6680003eb0 tx=0x7f6680009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.270+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f668c157f40 con 0x7f668c1586e0 2024-03-06T07:22:16.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.500+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f668c0732b0 con 0x7f668c1586e0 2024-03-06T07:22:16.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.500+0000 7f667affd640 1 -- 172.21.15.72:0/3211674798 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6684014030 con 0x7f668c1586e0 2024-03-06T07:22:16.502 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:16.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f666c06b4d0 msgr2=0x7f666c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f666c06b4d0 0x7f666c06d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6680003eb0 tx=0x7f6680009040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 msgr2=0x7f668c06d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 0x7f668c06d280 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f668400cc30 tx=0x7f668400a5c0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 shutdown_connections 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f666c06b4d0 0x7f666c06d990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f668c1586e0 0x7f668c06d280 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f668c0cfa90 0x7f668c0acc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 --2- 172.21.15.72:0/3211674798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f668c0b33a0 0x7f668c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 >> 172.21.15.72:0/3211674798 conn(0x7f668c06b190 msgr2=0x7f668c0aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 shutdown_connections 2024-03-06T07:22:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.502+0000 7f6690e39640 1 -- 172.21.15.72:0/3211674798 wait complete. 2024-03-06T07:22:16.518 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:16.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.622+0000 7fbe8ff3e640 1 Processor -- start 2024-03-06T07:22:16.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.622+0000 7fbe8ff3e640 1 -- start start 2024-03-06T07:22:16.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8ff3e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbe8806fd10 0x7fbe880cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8ff3e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe880d0af0 0x7fbe880ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8ff3e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe880cd1e0 con 0x7fbe8815c050 2024-03-06T07:22:16.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8ff3e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe880cd4b0 con 0x7fbe8806fd10 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8ff3e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe880cd790 con 0x7fbe880d0af0 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8d4d9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbe8806fd10 0x7fbe880cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8dcda640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbe8815c050 0x7fbe8806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38430/0 (socket says 172.21.15.72:38430) 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8dcda640 1 -- 172.21.15.72:0/5202000 learned_addr learned my addr 172.21.15.72:0/5202000 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8d4d9640 1 -- 172.21.15.72:0/5202000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe880d0af0 msgr2=0x7fbe880ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8e4db640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe880d0af0 0x7fbe880ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8d4d9640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe880d0af0 0x7fbe880ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8ccd8640 1 -- 172.21.15.72:0/5202000 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418110293 0 0) 0x7fbe880cd1e0 con 0x7fbe8815c050 2024-03-06T07:22:16.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8d4d9640 1 -- 172.21.15.72:0/5202000 >> v1:172.21.15.72:6789/0 conn(0x7fbe8815c050 legacy=0x7fbe8806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8d4d9640 1 -- 172.21.15.72:0/5202000 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe880cda70 con 0x7fbe8806fd10 2024-03-06T07:22:16.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.624+0000 7fbe8e4db640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe880d0af0 0x7fbe880ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:16.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.625+0000 7fbe8d4d9640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe880cc760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0097a0 tx=0x7fbe7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4d032d22e5ade2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.625+0000 7fbe8ccd8640 1 -- 172.21.15.72:0/5202000 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0315a0 con 0x7fbe8806fd10 2024-03-06T07:22:16.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.625+0000 7fbe8ccd8640 1 -- 172.21.15.72:0/5202000 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c037070 con 0x7fbe8806fd10 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.625+0000 7fbe8ccd8640 1 -- 172.21.15.72:0/5202000 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c031960 con 0x7fbe8806fd10 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/5202000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 msgr2=0x7fbe880cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe880cc760 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0097a0 tx=0x7fbe7c02b840 comp rx=0 tx=0).stop 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/5202000 shutdown_connections 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe880d0af0 0x7fbe880ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/5202000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe880cc760 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/5202000 >> 172.21.15.72:0/5202000 conn(0x7fbe880b3800 msgr2=0x7fbe880b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/5202000 shutdown_connections 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.626+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/5202000 wait complete. 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 Processor -- start 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 -- start start 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe88153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe880d0af0 0x7fbe8815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8815c050 0x7fbe8815ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe88165c30 con 0x7fbe8806fd10 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe88165ab0 con 0x7fbe880d0af0 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.627+0000 7fbe8ff3e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe88165db0 con 0x7fbe8815c050 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe88153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe88153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42070/0 (socket says 172.21.15.72:42070) 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 -- 172.21.15.72:0/2808015371 learned_addr learned my addr 172.21.15.72:0/2808015371 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8d4d9640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe880d0af0 0x7fbe8815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8e4db640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8815c050 0x7fbe8815ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 -- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8815c050 msgr2=0x7fbe8815ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8815c050 0x7fbe8815ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 -- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe880d0af0 msgr2=0x7fbe8815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe880d0af0 0x7fbe8815b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 -- 172.21.15.72:0/2808015371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8815a5e0 con 0x7fbe8806fd10 2024-03-06T07:22:16.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8d4d9640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe880d0af0 0x7fbe8815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8dcda640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe88153bd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000c4d0 tx=0x7fbe8000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80018070 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe80007500 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8806d9f0 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe80004a20 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.628+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8815a880 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.629+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8809fe60 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.630+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe8001b070 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.630+0000 7fbe867fc640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe7406b500 0x7fbe7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.630+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe80089430 con 0x7fbe8806fd10 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.630+0000 7fbe8d4d9640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe7406b500 0x7fbe7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.631+0000 7fbe8d4d9640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe7406b500 0x7fbe7406d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c036000 tx=0x7fbe7c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.635+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe80058340 con 0x7fbe8806fd10 2024-03-06T07:22:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.807+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbe88068ff0 con 0x7fbe8806fd10 2024-03-06T07:22:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.807+0000 7fbe867fc640 1 -- 172.21.15.72:0/2808015371 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbe80057c40 con 0x7fbe8806fd10 2024-03-06T07:22:16.809 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:16.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe7406b500 msgr2=0x7fbe7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe7406b500 0x7fbe7406d9c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c036000 tx=0x7fbe7c03d040 comp rx=0 tx=0).stop 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 msgr2=0x7fbe88153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe88153bd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbe8000c4d0 tx=0x7fbe8000c990 comp rx=0 tx=0).stop 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 shutdown_connections 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe7406b500 0x7fbe7406d9c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe8815c050 0x7fbe8815ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe880d0af0 0x7fbe8815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 --2- 172.21.15.72:0/2808015371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe8806fd10 0x7fbe88153bd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 >> 172.21.15.72:0/2808015371 conn(0x7fbe880b3800 msgr2=0x7fbe88158a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 shutdown_connections 2024-03-06T07:22:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.809+0000 7fbe8ff3e640 1 -- 172.21.15.72:0/2808015371 wait complete. 2024-03-06T07:22:16.825 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:22:16.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.924+0000 7fabef5d0640 1 Processor -- start 2024-03-06T07:22:16.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabef5d0640 1 -- start start 2024-03-06T07:22:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabef5d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 0x7fabe80b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabef5d0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80b4290 con 0x7fabe80b3df0 2024-03-06T07:22:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabef5d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe80b4570 con 0x7fabe80d2a90 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabef5d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80ce990 con 0x7fabe815e0b0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabecb6b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fabe80b3df0 0x7fabe80d2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38434/0 (socket says 172.21.15.72:38434) 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.925+0000 7fabecb6b640 1 -- 172.21.15.72:0/4252071905 learned_addr learned my addr 172.21.15.72:0/4252071905 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabed36c640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 0x7fabe80b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367670221 0 0) 0x7fabe80b4290 con 0x7fabe80b3df0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd4003520 con 0x7fabe80b3df0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078792819 0 0) 0x7fabe80b4570 con 0x7fabe80d2a90 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe80b4290 con 0x7fabe80d2a90 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070581468 0 0) 0x7fabd4003520 con 0x7fabe80b3df0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 msgr2=0x7fabe80b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 0x7fabe80b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 >> v1:172.21.15.146:6789/0 conn(0x7fabe80d2a90 legacy=0x7fabe80cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80b4850 con 0x7fabe80b3df0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fabe4001670 con 0x7fabe80b3df0 2024-03-06T07:22:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 >> v1:172.21.15.72:6789/0 conn(0x7fabe80b3df0 legacy=0x7fabe80d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 0x7fabd4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd40040d0 0x7fabd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabd4004d40 0x7fabd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd400ba70 con 0x7fabd4004d40 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd400bc20 con 0x7fabd40040d0 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd400be20 con 0x7fabe815e0b0 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabedb6d640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabd4004d40 0x7fabd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabed36c640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 0x7fabd4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabecb6b640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd40040d0 0x7fabd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabedb6d640 1 -- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 msgr2=0x7fabd4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabedb6d640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815e0b0 0x7fabd4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabedb6d640 1 -- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd40040d0 msgr2=0x7fabd4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabedb6d640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabd40040d0 0x7fabd4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.926+0000 7fabedb6d640 1 -- 172.21.15.72:0/4252071905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd4003520 con 0x7fabd4004d40 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.927+0000 7fabedb6d640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabd4004d40 0x7fabd40051c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fabe0001fa0 tx=0x7fabd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7828f96cb37ca402 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.927+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0001bf0 con 0x7fabd4004d40 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.927+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe0003b70 con 0x7fabd4004d40 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.927+0000 7fabdffff640 1 -- 172.21.15.72:0/4252071905 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe00050d0 con 0x7fabd4004d40 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabef5d0640 1 -- 172.21.15.72:0/4252071905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabd4004d40 msgr2=0x7fabd40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabef5d0640 1 --2- 172.21.15.72:0/4252071905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabd4004d40 0x7fabd40051c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fabe0001fa0 tx=0x7fabd4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabedb6d640 1 -- 172.21.15.72:0/4252071905 reap_dead start 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabef5d0640 1 -- 172.21.15.72:0/4252071905 shutdown_connections 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabef5d0640 1 -- 172.21.15.72:0/4252071905 >> 172.21.15.72:0/4252071905 conn(0x7fabe808d6f0 msgr2=0x7fabe809d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabef5d0640 1 -- 172.21.15.72:0/4252071905 shutdown_connections 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.928+0000 7fabef5d0640 1 -- 172.21.15.72:0/4252071905 wait complete. 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.930+0000 7fabef5d0640 1 Processor -- start 2024-03-06T07:22:16.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.930+0000 7fabef5d0640 1 -- start start 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabef5d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 0x7fabe80b8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabef5d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815c470 0x7fabe80b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabef5d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe80b91e0 0x7fabe815af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabef5d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80d1cd0 con 0x7fabe80a1430 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabef5d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe815e490 con 0x7fabe80b91e0 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabef5d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe80d1e50 con 0x7fabe815c470 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 0x7fabe80b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 0x7fabe80b8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42082/0 (socket says 172.21.15.72:42082) 2024-03-06T07:22:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 -- 172.21.15.72:0/3761430206 learned_addr learned my addr 172.21.15.72:0/3761430206 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabedb6d640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe80b91e0 0x7fabe815af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabecb6b640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815c470 0x7fabe80b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 -- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815c470 msgr2=0x7fabe80b8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815c470 0x7fabe80b8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 -- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe80b91e0 msgr2=0x7fabe815af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe80b91e0 0x7fabe815af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.931+0000 7fabed36c640 1 -- 172.21.15.72:0/3761430206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe80a1810 con 0x7fabe80a1430 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.932+0000 7fabed36c640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 0x7fabe80b8760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fabd8011a00 tx=0x7fabd8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.932+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd800fb50 con 0x7fabe80a1430 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.932+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd800fcf0 con 0x7fabe80a1430 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.932+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe80b9800 con 0x7fabe80a1430 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.932+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd800fea0 con 0x7fabe80a1430 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.932+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe80b76b0 con 0x7fabe80a1430 2024-03-06T07:22:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.933+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe804f330 con 0x7fabe80a1430 2024-03-06T07:22:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.934+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fabd8012040 con 0x7fabe80a1430 2024-03-06T07:22:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.934+0000 7fabddffb640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.934+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabd808a6c0 con 0x7fabe80a1430 2024-03-06T07:22:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.934+0000 7fabecb6b640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.935+0000 7fabecb6b640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fabcc06b500 0x7fabcc06d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fabe81f7530 tx=0x7fabe4001e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:16.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:16.943+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fabd8019030 con 0x7fabe80a1430 2024-03-06T07:22:17.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:17.314+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7fabe80d2160 con 0x7fabe80a1430 2024-03-06T07:22:18.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.141+0000 7fabddffb640 1 -- 172.21.15.72:0/3761430206 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v47) v1 ==== 135+0+0 (secure 0 0 0) 0x7fabd8058f80 con 0x7fabe80a1430 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fabcc06b500 msgr2=0x7fabcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fabcc06b500 0x7fabcc06d9c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fabe81f7530 tx=0x7fabe4001e40 comp rx=0 tx=0).stop 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 msgr2=0x7fabe80b8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 0x7fabe80b8760 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fabd8011a00 tx=0x7fabd8011ec0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 shutdown_connections 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fabcc06b500 0x7fabcc06d9c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe80b91e0 0x7fabe815af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe815c470 0x7fabe80b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 --2- 172.21.15.72:0/3761430206 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fabe80a1430 0x7fabe80b8760 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 >> 172.21.15.72:0/3761430206 conn(0x7fabe808d6f0 msgr2=0x7fabe80d2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 shutdown_connections 2024-03-06T07:22:18.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.144+0000 7fabef5d0640 1 -- 172.21.15.72:0/3761430206 wait complete. 2024-03-06T07:22:18.157 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:22:18.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.259+0000 7f0a55193640 1 Processor -- start 2024-03-06T07:22:18.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.259+0000 7f0a55193640 1 -- start start 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a55193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5015c050 0x7f0a5006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a55193640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0a500d0af0 0x7f0a500cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a55193640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a500cd4b0 con 0x7f0a5006fb40 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a55193640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5006ff80 con 0x7f0a500d0af0 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a55193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a500cd790 con 0x7f0a5015c050 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4f59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0a500d0af0 0x7f0a500cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0a5006fb40 0x7f0a500cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38442/0 (socket says 172.21.15.72:38442) 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4e59c640 1 -- 172.21.15.72:0/1493873827 learned_addr learned my addr 172.21.15.72:0/1493873827 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4ed9d640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5015c050 0x7f0a5006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4f59e640 1 -- 172.21.15.72:0/1493873827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5015c050 msgr2=0x7f0a5006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4f59e640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5015c050 0x7f0a5006f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4f59e640 1 -- 172.21.15.72:0/1493873827 >> v1:172.21.15.72:6789/0 conn(0x7f0a5006fb40 legacy=0x7f0a500cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4f59e640 1 -- 172.21.15.72:0/1493873827 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a500cda70 con 0x7f0a500d0af0 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4dd9b640 1 -- 172.21.15.72:0/1493873827 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413459361 0 0) 0x7f0a500cd4b0 con 0x7f0a5006fb40 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.260+0000 7f0a4ed9d640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5015c050 0x7f0a5006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.261+0000 7f0a4f59e640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a500cce70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009d90 tx=0x7f0a4802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8779e29a8833ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.261+0000 7f0a4dd9b640 1 -- 172.21.15.72:0/1493873827 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48039070 con 0x7f0a500d0af0 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.261+0000 7f0a4dd9b640 1 -- 172.21.15.72:0/1493873827 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4802cbc0 con 0x7f0a500d0af0 2024-03-06T07:22:18.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.261+0000 7f0a4dd9b640 1 -- 172.21.15.72:0/1493873827 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48034440 con 0x7f0a500d0af0 2024-03-06T07:22:18.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 -- 172.21.15.72:0/1493873827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 msgr2=0x7f0a500cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a500cce70 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009d90 tx=0x7f0a4802be10 comp rx=0 tx=0).stop 2024-03-06T07:22:18.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 -- 172.21.15.72:0/1493873827 shutdown_connections 2024-03-06T07:22:18.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a500cce70 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 --2- 172.21.15.72:0/1493873827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5015c050 0x7f0a5006f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 -- 172.21.15.72:0/1493873827 >> 172.21.15.72:0/1493873827 conn(0x7f0a500b3800 msgr2=0x7f0a500b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 -- 172.21.15.72:0/1493873827 shutdown_connections 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.262+0000 7f0a55193640 1 -- 172.21.15.72:0/1493873827 wait complete. 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.263+0000 7f0a55193640 1 Processor -- start 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.263+0000 7f0a55193640 1 -- start start 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a55193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5006fb40 0x7f0a50153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a55193640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a5015ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a55193640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5015c050 0x7f0a500cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a55193640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50165c30 con 0x7f0a500d0af0 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a55193640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50165ab0 con 0x7f0a5015c050 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a55193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a50165db0 con 0x7f0a5006fb40 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a5015ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a5015ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42096/0 (socket says 172.21.15.72:42096) 2024-03-06T07:22:18.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 -- 172.21.15.72:0/1794073117 learned_addr learned my addr 172.21.15.72:0/1794073117 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4f59e640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5015c050 0x7f0a500cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4ed9d640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5006fb40 0x7f0a50153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 -- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5006fb40 msgr2=0x7f0a50153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5006fb40 0x7f0a50153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 -- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5015c050 msgr2=0x7f0a500cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5015c050 0x7f0a500cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 -- 172.21.15.72:0/1794073117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a500b69f0 con 0x7f0a500d0af0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4ed9d640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5006fb40 0x7f0a50153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.264+0000 7f0a4e59c640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a5015ade0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000c430 tx=0x7f0a4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.265+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a400074e0 con 0x7f0a500d0af0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.265+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a400047e0 con 0x7f0a500d0af0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.265+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a500b6cd0 con 0x7f0a500d0af0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.265+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40004bd0 con 0x7f0a500d0af0 2024-03-06T07:22:18.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.265+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a500b7290 con 0x7f0a500d0af0 2024-03-06T07:22:18.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.266+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a5009fe60 con 0x7f0a500d0af0 2024-03-06T07:22:18.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.266+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0a40011070 con 0x7f0a500d0af0 2024-03-06T07:22:18.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.266+0000 7f0a477fe640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0a3006b500 0x7f0a3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.267+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a40089e70 con 0x7f0a500d0af0 2024-03-06T07:22:18.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.267+0000 7f0a4ed9d640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0a3006b500 0x7f0a3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.267+0000 7f0a4ed9d640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0a3006b500 0x7f0a3006d9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0a38006fd0 tx=0x7f0a38008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.271+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0a40058cf0 con 0x7f0a500d0af0 2024-03-06T07:22:18.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.447+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0a50068ff0 con 0x7f0a500d0af0 2024-03-06T07:22:18.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.453+0000 7f0a477fe640 1 -- 172.21.15.72:0/1794073117 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0a400585f0 con 0x7f0a500d0af0 2024-03-06T07:22:18.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0a3006b500 msgr2=0x7f0a3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0a3006b500 0x7f0a3006d9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0a38006fd0 tx=0x7f0a38008040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 msgr2=0x7f0a5015ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a5015ade0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000c430 tx=0x7f0a4000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 shutdown_connections 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0a3006b500 0x7f0a3006d9c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5015c050 0x7f0a500cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0a500d0af0 0x7f0a5015ade0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 --2- 172.21.15.72:0/1794073117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5006fb40 0x7f0a50153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 >> 172.21.15.72:0/1794073117 conn(0x7f0a500b3800 msgr2=0x7f0a50158d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 shutdown_connections 2024-03-06T07:22:18.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.455+0000 7f0a55193640 1 -- 172.21.15.72:0/1794073117 wait complete. 2024-03-06T07:22:18.468 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:22:18.468 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:18.509 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:22:18.558 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:22:18.558 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:22:18.558 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:18.573 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:22:18.573 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:18.628 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:18.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.769+0000 7fc3a6dd6640 1 Processor -- start 2024-03-06T07:22:18.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.769+0000 7fc3a6dd6640 1 -- start start 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc3a6dd6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 0x7fc3a00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc3a6dd6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a0161980 con 0x7fc3a00cfff0 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc3a6dd6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a0161c60 con 0x7fc3a00cd8c0 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc3a6dd6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0161f40 con 0x7fc3a015bb20 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc39ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc3a00cfff0 0x7fc3a00ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38450/0 (socket says 172.21.15.72:38450) 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc39ffff640 1 -- 172.21.15.72:0/3905860740 learned_addr learned my addr 172.21.15.72:0/3905860740 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625583548 0 0) 0x7fc3a0161980 con 0x7fc3a00cfff0 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc3a4b72640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 0x7fc3a00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.770+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc388003520 con 0x7fc3a00cfff0 2024-03-06T07:22:18.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494943914 0 0) 0x7fc388003520 con 0x7fc3a00cfff0 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 msgr2=0x7fc3a00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 0x7fc3a00d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 >> v1:172.21.15.146:6789/0 conn(0x7fc3a00cd8c0 legacy=0x7fc3a00cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a0162220 con 0x7fc3a00cfff0 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fc394001670 con 0x7fc3a00cfff0 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 >> v1:172.21.15.72:6789/0 conn(0x7fc3a00cfff0 legacy=0x7fc3a00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 0x7fc3880039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc388003f20 0x7fc388004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a0161f40 con 0x7fc388003f20 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800bad0 con 0x7fc388004d40 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38800bcd0 con 0x7fc3a015bb20 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39ffff640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc388003f20 0x7fc388004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc3a4b72640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 0x7fc3880039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc3a5373640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39ffff640 1 -- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 msgr2=0x7fc3880039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39ffff640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a015bb20 0x7fc3880039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39ffff640 1 -- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388004d40 msgr2=0x7fc3880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39ffff640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc39ffff640 1 -- 172.21.15.72:0/3905860740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc388003520 con 0x7fc388003f20 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.771+0000 7fc3a5373640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388004d40 0x7fc3880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:18.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc39ffff640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc388003f20 0x7fc388004800 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c290 tx=0x7fc388005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adacf0ad3328d858 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394007d50 con 0x7fc388003f20 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394030070 con 0x7fc388003f20 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3905860740 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39402cda0 con 0x7fc388003f20 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/3905860740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc388003f20 msgr2=0x7fc388004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/3905860740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc388003f20 0x7fc388004800 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c290 tx=0x7fc388005830 comp rx=0 tx=0).stop 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a5373640 1 -- 172.21.15.72:0/3905860740 reap_dead start 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/3905860740 shutdown_connections 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/3905860740 >> 172.21.15.72:0/3905860740 conn(0x7fc3a009a9f0 msgr2=0x7fc3a009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/3905860740 shutdown_connections 2024-03-06T07:22:18.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.772+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/3905860740 wait complete. 2024-03-06T07:22:18.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.774+0000 7fc3a6dd6640 1 Processor -- start 2024-03-06T07:22:18.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.774+0000 7fc3a6dd6640 1 -- start start 2024-03-06T07:22:18.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.774+0000 7fc3a6dd6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388003f20 0x7fc3a0150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.774+0000 7fc3a6dd6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a00b4650 0x7fc3a0160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a6dd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 0x7fc3a00afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a6dd6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00d1980 con 0x7fc3a00af940 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a6dd6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00cf2d0 con 0x7fc388003f20 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a6dd6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a00d1b00 con 0x7fc3a00b4650 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 0x7fc3a00afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 0x7fc3a00afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42100/0 (socket says 172.21.15.72:42100) 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 -- 172.21.15.72:0/1304367945 learned_addr learned my addr 172.21.15.72:0/1304367945 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a4b72640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388003f20 0x7fc3a0150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc39ffff640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a00b4650 0x7fc3a0160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 -- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a00b4650 msgr2=0x7fc3a0160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a00b4650 0x7fc3a0160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 -- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388003f20 msgr2=0x7fc3a0150fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388003f20 0x7fc3a0150fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 -- 172.21.15.72:0/1304367945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a015b340 con 0x7fc3a00af940 2024-03-06T07:22:18.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc39ffff640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a00b4650 0x7fc3a0160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a5373640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 0x7fc3a00afdc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0071420 tx=0x7fc39000e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc390019070 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3900052e0 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a0161200 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc390003850 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.775+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a00b4ea0 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.776+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a004f330 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.777+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc39001c070 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.777+0000 7fc39d7fa640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc38c06b4d0 0x7fc38c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.777+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3900895f0 con 0x7fc3a00af940 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.777+0000 7fc3a4b72640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc38c06b4d0 0x7fc38c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:18.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.778+0000 7fc3a4b72640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc38c06b4d0 0x7fc38c06d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0160ab0 tx=0x7fc39800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:18.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.781+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc390057eb0 con 0x7fc3a00af940 2024-03-06T07:22:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.960+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a00b1820 con 0x7fc3a00af940 2024-03-06T07:22:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.962+0000 7fc39d7fa640 1 -- 172.21.15.72:0/1304367945 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fc39005bf50 con 0x7fc3a00af940 2024-03-06T07:22:18.964 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:18.964 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc38c06b4d0 msgr2=0x7fc38c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc38c06b4d0 0x7fc38c06d990 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0160ab0 tx=0x7fc39800b040 comp rx=0 tx=0).stop 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 msgr2=0x7fc3a00afdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 0x7fc3a00afdc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0071420 tx=0x7fc39000e890 comp rx=0 tx=0).stop 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 shutdown_connections 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc38c06b4d0 0x7fc38c06d990 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00af940 0x7fc3a00afdc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3a00b4650 0x7fc3a0160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 --2- 172.21.15.72:0/1304367945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc388003f20 0x7fc3a0150fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 >> 172.21.15.72:0/1304367945 conn(0x7fc3a009a9f0 msgr2=0x7fc3a006efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 shutdown_connections 2024-03-06T07:22:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:18.964+0000 7fc3a6dd6640 1 -- 172.21.15.72:0/1304367945 wait complete. 2024-03-06T07:22:18.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:18.978 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:19.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.080+0000 7f482451e640 1 Processor -- start 2024-03-06T07:22:19.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.080+0000 7f482451e640 1 -- start start 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f482451e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f481c15def0 0x7f481c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f482451e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c0cea50 0x7f481c0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f482451e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c06d030 con 0x7f481c06cab0 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f482451e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c0cf1c0 con 0x7f481c15def0 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f482451e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0cc760 con 0x7f481c0cea50 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f48222ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f481c15def0 0x7f481c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.081+0000 7f48222ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f481c15def0 0x7f481c0b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42104/0 (socket says 172.21.15.72:42104) 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f48222ba640 1 -- 172.21.15.72:0/3623360144 learned_addr learned my addr 172.21.15.72:0/3623360144 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:19.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f4822abb640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c0cea50 0x7f481c0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f48222ba640 1 -- 172.21.15.72:0/3623360144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c0cea50 msgr2=0x7f481c0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f48222ba640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c0cea50 0x7f481c0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f48222ba640 1 -- 172.21.15.72:0/3623360144 >> v1:172.21.15.146:6789/0 conn(0x7f481c06cab0 legacy=0x7f481c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f48222ba640 1 -- 172.21.15.72:0/3623360144 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c06d310 con 0x7f481c15def0 2024-03-06T07:22:19.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.082+0000 7f4822abb640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c0cea50 0x7f481c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f48222ba640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c15def0 0x7f481c0b41b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f480c009870 tx=0x7f480c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ff89ba0d5a4da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f48212b8640 1 -- 172.21.15.72:0/3623360144 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c034070 con 0x7f481c15def0 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f48212b8640 1 -- 172.21.15.72:0/3623360144 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480c030840 con 0x7f481c15def0 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f48212b8640 1 -- 172.21.15.72:0/3623360144 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c030c10 con 0x7f481c15def0 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f482451e640 1 -- 172.21.15.72:0/3623360144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c15def0 msgr2=0x7f481c0b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f482451e640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c15def0 0x7f481c0b41b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f480c009870 tx=0x7f480c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:19.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f482451e640 1 -- 172.21.15.72:0/3623360144 shutdown_connections 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f482451e640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c0cea50 0x7f481c0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f482451e640 1 --2- 172.21.15.72:0/3623360144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c15def0 0x7f481c0b41b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.083+0000 7f482451e640 1 -- 172.21.15.72:0/3623360144 >> 172.21.15.72:0/3623360144 conn(0x7f481c0b3690 msgr2=0x7f481c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.084+0000 7f482451e640 1 -- 172.21.15.72:0/3623360144 shutdown_connections 2024-03-06T07:22:19.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.084+0000 7f482451e640 1 -- 172.21.15.72:0/3623360144 wait complete. 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 Processor -- start 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 -- start start 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c06cab0 0x7f481c14f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 0x7f481c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c15def0 0x7f481c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c15e2d0 con 0x7f481c0cea50 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0cc3f0 con 0x7f481c15def0 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f482451e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c15e450 con 0x7f481c06cab0 2024-03-06T07:22:19.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.085+0000 7f4821ab9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 0x7f481c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 0x7f481c0c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42120/0 (socket says 172.21.15.72:42120) 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 -- 172.21.15.72:0/1603149671 learned_addr learned my addr 172.21.15.72:0/1603149671 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f48222ba640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c06cab0 0x7f481c14f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4822abb640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c15def0 0x7f481c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 -- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c06cab0 msgr2=0x7f481c14f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c06cab0 0x7f481c14f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 -- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c15def0 msgr2=0x7f481c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c15def0 0x7f481c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 -- 172.21.15.72:0/1603149671 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0b5d80 con 0x7f481c0cea50 2024-03-06T07:22:19.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4821ab9640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 0x7f481c0c8dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f481800c430 tx=0x7f481800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818011790 con 0x7f481c0cea50 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4818017070 con 0x7f481c0cea50 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4818011bb0 con 0x7f481c0cea50 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c1f4240 con 0x7f481c0cea50 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.086+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c1f4800 con 0x7f481c0cea50 2024-03-06T07:22:19.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.087+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c06dee0 con 0x7f481c0cea50 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.088+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f481801c460 con 0x7f481c0cea50 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.089+0000 7f4812ffd640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f480006b500 0x7f480006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.089+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f481808a990 con 0x7f481c0cea50 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.089+0000 7f48222ba640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f480006b500 0x7f480006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.089+0000 7f48222ba640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f480006b500 0x7f480006d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f480c036000 tx=0x7f480c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.092+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f481c06dee0 con 0x7f481c0cea50 2024-03-06T07:22:19.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.265+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f481c09fcf0 con 0x7f481c0cea50 2024-03-06T07:22:19.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.265+0000 7f4812ffd640 1 -- 172.21.15.72:0/1603149671 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f481c0b5d80 con 0x7f481c0cea50 2024-03-06T07:22:19.267 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:19.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f480006b500 msgr2=0x7f480006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f480006b500 0x7f480006d9c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f480c036000 tx=0x7f480c03b040 comp rx=0 tx=0).stop 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 msgr2=0x7f481c0c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 0x7f481c0c8dc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f481800c430 tx=0x7f481800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 shutdown_connections 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f480006b500 0x7f480006d9c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c15def0 0x7f481c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f481c0cea50 0x7f481c0c8dc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 --2- 172.21.15.72:0/1603149671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c06cab0 0x7f481c14f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 >> 172.21.15.72:0/1603149671 conn(0x7f481c0b3690 msgr2=0x7f481c0ccbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 shutdown_connections 2024-03-06T07:22:19.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.267+0000 7f482451e640 1 -- 172.21.15.72:0/1603149671 wait complete. 2024-03-06T07:22:19.283 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:19.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.384+0000 7f2d6f3ac640 1 Processor -- start 2024-03-06T07:22:19.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.384+0000 7f2d6f3ac640 1 -- start start 2024-03-06T07:22:19.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6f3ac640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2d681614d0 0x7f2d680d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6f3ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6f3ac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d681618b0 con 0x7f2d680d1910 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6f3ac640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d680cf400 con 0x7f2d681614d0 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6f3ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680cf6a0 con 0x7f2d680d24b0 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6c947640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2d680d1910 0x7f2d680d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38460/0 (socket says 172.21.15.72:38460) 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6c947640 1 -- 172.21.15.72:0/33991140 learned_addr learned my addr 172.21.15.72:0/33991140 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6d148640 1 --2- 172.21.15.72:0/33991140 >> v2:172.21.15.146:3300/0 conn(0x7f2d681614d0 0x7f2d680d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6d949640 1 --2- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d5ffff640 1 -- 172.21.15.72:0/33991140 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857497628 0 0) 0x7f2d681618b0 con 0x7f2d680d1910 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d5ffff640 1 -- 172.21.15.72:0/33991140 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d54003520 con 0x7f2d680d1910 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6d148640 1 -- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 msgr2=0x7f2d680b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6d148640 1 --2- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d5ffff640 1 -- 172.21.15.72:0/33991140 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699224059 0 0) 0x7f2d54003520 con 0x7f2d680d1910 2024-03-06T07:22:19.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6d148640 1 -- 172.21.15.72:0/33991140 >> v1:172.21.15.72:6789/0 conn(0x7f2d680d1910 legacy=0x7f2d680d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.385+0000 7f2d6d148640 1 -- 172.21.15.72:0/33991140 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680cf980 con 0x7f2d681614d0 2024-03-06T07:22:19.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.386+0000 7f2d6d148640 1 --2- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d680d4860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d640097a0 tx=0x7f2d6402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=423319a3c8e55e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.386+0000 7f2d5ffff640 1 -- 172.21.15.72:0/33991140 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64037070 con 0x7f2d681614d0 2024-03-06T07:22:19.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.386+0000 7f2d5ffff640 1 -- 172.21.15.72:0/33991140 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d64030510 con 0x7f2d681614d0 2024-03-06T07:22:19.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.386+0000 7f2d5ffff640 1 -- 172.21.15.72:0/33991140 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58001670 con 0x7f2d681614d0 2024-03-06T07:22:19.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 msgr2=0x7f2d680d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d680d4860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2d640097a0 tx=0x7f2d6402b790 comp rx=0 tx=0).stop 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/33991140 shutdown_connections 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/33991140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d680d4860 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/33991140 >> 172.21.15.72:0/33991140 conn(0x7f2d680d3d40 msgr2=0x7f2d680d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/33991140 shutdown_connections 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.387+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/33991140 wait complete. 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.388+0000 7f2d6f3ac640 1 Processor -- start 2024-03-06T07:22:19.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.388+0000 7f2d6f3ac640 1 -- start start 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6f3ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 0x7f2d680a1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6f3ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6f3ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d6806b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6f3ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680b1370 con 0x7f2d680d1910 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6f3ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680b0910 con 0x7f2d681614d0 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6f3ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d680b14f0 con 0x7f2d680d24b0 2024-03-06T07:22:19.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 0x7f2d680a1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 0x7f2d680a1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42136/0 (socket says 172.21.15.72:42136) 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 -- 172.21.15.72:0/1150774554 learned_addr learned my addr 172.21.15.72:0/1150774554 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 -- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 msgr2=0x7f2d680ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d949640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d6806b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 -- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 msgr2=0x7f2d6806b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d6806b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 -- 172.21.15.72:0/1150774554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d680c7be0 con 0x7f2d680d1910 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.389+0000 7f2d6d148640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 0x7f2d680a1340 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2d64000c00 tx=0x7f2d640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.390+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64041020 con 0x7f2d680d1910 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.390+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6403c870 con 0x7f2d680d1910 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.390+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d680c7ec0 con 0x7f2d680d1910 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.390+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d64037040 con 0x7f2d680d1910 2024-03-06T07:22:19.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.390+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d680c8480 con 0x7f2d680d1910 2024-03-06T07:22:19.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.391+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d68068ff0 con 0x7f2d680d1910 2024-03-06T07:22:19.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.391+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d64044020 con 0x7f2d680d1910 2024-03-06T07:22:19.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.392+0000 7f2d5dffb640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d4c06b4d0 0x7f2d4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.392+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d640abad0 con 0x7f2d680d1910 2024-03-06T07:22:19.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.392+0000 7f2d6c947640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d4c06b4d0 0x7f2d4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.392+0000 7f2d6c947640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d4c06b4d0 0x7f2d4c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2d58001eb0 tx=0x7f2d58001a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.396+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d6404b030 con 0x7f2d680d1910 2024-03-06T07:22:19.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:19.475 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.730+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2d680d00f0 con 0x7f2d680d1910 2024-03-06T07:22:19.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.730+0000 7f2d5dffb640 1 -- 172.21.15.72:0/1150774554 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2d6407a9e0 con 0x7f2d680d1910 2024-03-06T07:22:19.733 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d4c06b4d0 msgr2=0x7f2d4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d4c06b4d0 0x7f2d4c06d990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2d58001eb0 tx=0x7f2d58001a40 comp rx=0 tx=0).stop 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 msgr2=0x7f2d680a1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 0x7f2d680a1340 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2d64000c00 tx=0x7f2d640046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 shutdown_connections 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d4c06b4d0 0x7f2d4c06d990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d681614d0 0x7f2d6806b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d680d24b0 0x7f2d680ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 --2- 172.21.15.72:0/1150774554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d680d1910 0x7f2d680a1340 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 >> 172.21.15.72:0/1150774554 conn(0x7f2d680d3d40 msgr2=0x7f2d6806ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 shutdown_connections 2024-03-06T07:22:19.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.732+0000 7f2d6f3ac640 1 -- 172.21.15.72:0/1150774554 wait complete. 2024-03-06T07:22:19.749 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:19.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.853+0000 7f69ad3a0640 1 Processor -- start 2024-03-06T07:22:19.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.853+0000 7f69ad3a0640 1 -- start start 2024-03-06T07:22:19.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69ad3a0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69a80b3fe0 0x7f69a80cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69ad3a0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69a815f7e0 0x7f69a815fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69ad3a0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a8160130 con 0x7f69a815bd40 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69ad3a0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a80ccdb0 con 0x7f69a80b3fe0 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69ad3a0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a815c120 con 0x7f69a815f7e0 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69a80b3fe0 0x7f69a80cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69a80b3fe0 0x7f69a80cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42146/0 (socket says 172.21.15.72:42146) 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 -- 172.21.15.72:0/676038613 learned_addr learned my addr 172.21.15.72:0/676038613 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69acb9f640 1 --2- 172.21.15.72:0/676038613 >> v2:172.21.15.146:3300/0 conn(0x7f69a815f7e0 0x7f69a815fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 -- 172.21.15.72:0/676038613 >> v2:172.21.15.146:3300/0 conn(0x7f69a815f7e0 msgr2=0x7f69a815fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 --2- 172.21.15.72:0/676038613 >> v2:172.21.15.146:3300/0 conn(0x7f69a815f7e0 0x7f69a815fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 -- 172.21.15.72:0/676038613 >> v1:172.21.15.146:6789/0 conn(0x7f69a815bd40 legacy=0x7f69a80cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.854+0000 7f69a77fe640 1 -- 172.21.15.72:0/676038613 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a80d06c0 con 0x7f69a80b3fe0 2024-03-06T07:22:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.855+0000 7f69a77fe640 1 --2- 172.21.15.72:0/676038613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b3fe0 0x7f69a80cd240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f69a80b7830 tx=0x7f699802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce42f437e2896ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69a6ffd640 1 -- 172.21.15.72:0/676038613 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998035020 con 0x7f69a80b3fe0 2024-03-06T07:22:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69a6ffd640 1 -- 172.21.15.72:0/676038613 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6998031870 con 0x7f69a80b3fe0 2024-03-06T07:22:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69a6ffd640 1 -- 172.21.15.72:0/676038613 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998031c40 con 0x7f69a80b3fe0 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 -- 172.21.15.72:0/676038613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b3fe0 msgr2=0x7f69a80cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/676038613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b3fe0 0x7f69a80cd240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f69a80b7830 tx=0x7f699802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 -- 172.21.15.72:0/676038613 shutdown_connections 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/676038613 >> v2:172.21.15.146:3300/0 conn(0x7f69a815f7e0 0x7f69a815fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/676038613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b3fe0 0x7f69a80cd240 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 -- 172.21.15.72:0/676038613 >> 172.21.15.72:0/676038613 conn(0x7f69a80b3450 msgr2=0x7f69a80b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 -- 172.21.15.72:0/676038613 shutdown_connections 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.856+0000 7f69ad3a0640 1 -- 172.21.15.72:0/676038613 wait complete. 2024-03-06T07:22:19.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.858+0000 7f69ad3a0640 1 Processor -- start 2024-03-06T07:22:19.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.858+0000 7f69ad3a0640 1 -- start start 2024-03-06T07:22:19.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.858+0000 7f69ad3a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a80b3fe0 0x7f69a81573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 0x7f69a80c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a815f7e0 0x7f69a80c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cd780 con 0x7f69a815bd40 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cca40 con 0x7f69a80b3fe0 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cc3a0 con 0x7f69a815f7e0 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 0x7f69a80c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 0x7f69a80c5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42150/0 (socket says 172.21.15.72:42150) 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 -- 172.21.15.72:0/507972348 learned_addr learned my addr 172.21.15.72:0/507972348 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a7fff640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a80b3fe0 0x7f69a81573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69acb9f640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a815f7e0 0x7f69a80c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 -- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a815f7e0 msgr2=0x7f69a80c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a815f7e0 0x7f69a80c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 -- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a80b3fe0 msgr2=0x7f69a81573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a80b3fe0 0x7f69a81573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 -- 172.21.15.72:0/507972348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a80ca800 con 0x7f69a815bd40 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69acb9f640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a815f7e0 0x7f69a80c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a77fe640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 0x7f69a80c5430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6998000c00 tx=0x7f69980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998040070 con 0x7f69a815bd40 2024-03-06T07:22:19.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6998030d00 con 0x7f69a815bd40 2024-03-06T07:22:19.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998035040 con 0x7f69a815bd40 2024-03-06T07:22:19.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a80caae0 con 0x7f69a815bd40 2024-03-06T07:22:19.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.859+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a80cb0a0 con 0x7f69a815bd40 2024-03-06T07:22:19.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.860+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a8068f10 con 0x7f69a815bd40 2024-03-06T07:22:19.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.861+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f699804b060 con 0x7f69a815bd40 2024-03-06T07:22:19.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.861+0000 7f69a4ff9640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f699406b4d0 0x7f699406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:19.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.861+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69980abe20 con 0x7f69a815bd40 2024-03-06T07:22:19.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.861+0000 7f69a7fff640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f699406b4d0 0x7f699406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:19.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.862+0000 7f69a7fff640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f699406b4d0 0x7f699406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f699c006fd0 tx=0x7f699c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:19.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:19.865+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69a8160130 con 0x7f69a815bd40 2024-03-06T07:22:20.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.054+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69a80b7880 con 0x7f69a815bd40 2024-03-06T07:22:20.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.054+0000 7f69a4ff9640 1 -- 172.21.15.72:0/507972348 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f699807ad30 con 0x7f69a815bd40 2024-03-06T07:22:20.056 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:20.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f699406b4d0 msgr2=0x7f699406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f699406b4d0 0x7f699406d990 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f699c006fd0 tx=0x7f699c009060 comp rx=0 tx=0).stop 2024-03-06T07:22:20.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 msgr2=0x7f69a80c5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 0x7f69a80c5430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6998000c00 tx=0x7f69980046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 shutdown_connections 2024-03-06T07:22:20.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f699406b4d0 0x7f699406d990 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a815f7e0 0x7f69a80c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a815bd40 0x7f69a80c5430 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 --2- 172.21.15.72:0/507972348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a80b3fe0 0x7f69a81573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 >> 172.21.15.72:0/507972348 conn(0x7f69a80b3450 msgr2=0x7f69a806f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 shutdown_connections 2024-03-06T07:22:20.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.057+0000 7f69ad3a0640 1 -- 172.21.15.72:0/507972348 wait complete. 2024-03-06T07:22:20.071 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:20.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.171+0000 7fbd0779a640 1 Processor -- start 2024-03-06T07:22:20.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.172+0000 7fbd0779a640 1 -- start start 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd0779a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbd0006ee10 0x7fbd0006f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd0779a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd000b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd0779a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd000cdde0 0x7fbd000ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd0779a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd000b4a60 con 0x7fbd0006ee10 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd0779a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd000b4450 con 0x7fbd000cdde0 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd0779a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0006f730 con 0x7fbd00144030 2024-03-06T07:22:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbd0006ee10 0x7fbd0006f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbd0006ee10 0x7fbd0006f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42160/0 (socket says 172.21.15.72:42160) 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 -- 172.21.15.72:0/4100389458 learned_addr learned my addr 172.21.15.72:0/4100389458 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05d37640 1 --2- 172.21.15.72:0/4100389458 >> v2:172.21.15.146:3300/0 conn(0x7fbd000cdde0 0x7fbd000ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd04d35640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd000b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 -- 172.21.15.72:0/4100389458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 msgr2=0x7fbd000b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd000b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 -- 172.21.15.72:0/4100389458 >> v2:172.21.15.146:3300/0 conn(0x7fbd000cdde0 msgr2=0x7fbd000ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 --2- 172.21.15.72:0/4100389458 >> v2:172.21.15.146:3300/0 conn(0x7fbd000cdde0 0x7fbd000ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd05536640 1 -- 172.21.15.72:0/4100389458 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd000b5b60 con 0x7fbd0006ee10 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.173+0000 7fbd04d35640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd000b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.174+0000 7fbd05536640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd0006ee10 0x7fbd0006f1f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd000736c0 tx=0x7fbcfc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59627b343b84dcf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.174+0000 7fbcf3fff640 1 -- 172.21.15.72:0/4100389458 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc035020 con 0x7fbd0006ee10 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.174+0000 7fbcf3fff640 1 -- 172.21.15.72:0/4100389458 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcfc031870 con 0x7fbd0006ee10 2024-03-06T07:22:20.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.174+0000 7fbcf3fff640 1 -- 172.21.15.72:0/4100389458 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc031c40 con 0x7fbd0006ee10 2024-03-06T07:22:20.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 -- 172.21.15.72:0/4100389458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd0006ee10 msgr2=0x7fbd0006f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd0006ee10 0x7fbd0006f1f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd000736c0 tx=0x7fbcfc02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:20.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 -- 172.21.15.72:0/4100389458 shutdown_connections 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 --2- 172.21.15.72:0/4100389458 >> v2:172.21.15.146:3300/0 conn(0x7fbd000cdde0 0x7fbd000ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd000b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 --2- 172.21.15.72:0/4100389458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd0006ee10 0x7fbd0006f1f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 -- 172.21.15.72:0/4100389458 >> 172.21.15.72:0/4100389458 conn(0x7fbd000d0a10 msgr2=0x7fbd000d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 -- 172.21.15.72:0/4100389458 shutdown_connections 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.175+0000 7fbd0779a640 1 -- 172.21.15.72:0/4100389458 wait complete. 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.176+0000 7fbd0779a640 1 Processor -- start 2024-03-06T07:22:20.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.176+0000 7fbd0779a640 1 -- start start 2024-03-06T07:22:20.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd0779a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd0006ee10 0x7fbd000ae380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd0779a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 0x7fbd000aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd0779a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd001f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd0779a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd00144840 con 0x7fbd000cdde0 2024-03-06T07:22:20.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd0779a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd000b46f0 con 0x7fbd0006ee10 2024-03-06T07:22:20.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd0779a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd000b5430 con 0x7fbd00144030 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 0x7fbd000aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 0x7fbd000aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42168/0 (socket says 172.21.15.72:42168) 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 -- 172.21.15.72:0/3252978974 learned_addr learned my addr 172.21.15.72:0/3252978974 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd05536640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd0006ee10 0x7fbd000ae380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd05d37640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd001f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 -- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 msgr2=0x7fbd001f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd001f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 -- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd0006ee10 msgr2=0x7fbd000ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd0006ee10 0x7fbd000ae380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd04d35640 1 -- 172.21.15.72:0/3252978974 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd000af590 con 0x7fbd000cdde0 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.177+0000 7fbd05536640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd0006ee10 0x7fbd000ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.178+0000 7fbd04d35640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 0x7fbd000aee60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400c820 tx=0x7fbcf400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.178+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4015070 con 0x7fbd000cdde0 2024-03-06T07:22:20.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.178+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf40079d0 con 0x7fbd000cdde0 2024-03-06T07:22:20.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.178+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd000af870 con 0x7fbd000cdde0 2024-03-06T07:22:20.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.178+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4007d80 con 0x7fbd000cdde0 2024-03-06T07:22:20.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.178+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd000afe30 con 0x7fbd000cdde0 2024-03-06T07:22:20.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.179+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0004f330 con 0x7fbd000cdde0 2024-03-06T07:22:20.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.180+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcf4018070 con 0x7fbd000cdde0 2024-03-06T07:22:20.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.180+0000 7fbcf1ffb640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce406b4d0 0x7fbce406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.180+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcf408a0e0 con 0x7fbd000cdde0 2024-03-06T07:22:20.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.180+0000 7fbd05536640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce406b4d0 0x7fbce406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.181+0000 7fbd05536640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce406b4d0 0x7fbce406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc036000 tx=0x7fbcfc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.185+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcf4058f60 con 0x7fbd000cdde0 2024-03-06T07:22:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.375+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd000b3f50 con 0x7fbd000cdde0 2024-03-06T07:22:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.375+0000 7fbcf1ffb640 1 -- 172.21.15.72:0/3252978974 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbcf4058860 con 0x7fbd000cdde0 2024-03-06T07:22:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce406b4d0 msgr2=0x7fbce406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce406b4d0 0x7fbce406d990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc036000 tx=0x7fbcfc03c040 comp rx=0 tx=0).stop 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 msgr2=0x7fbd000aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 0x7fbd000aee60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400c820 tx=0x7fbcf400cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 shutdown_connections 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce406b4d0 0x7fbce406d990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd00144030 0x7fbd001f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd000cdde0 0x7fbd000aee60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 --2- 172.21.15.72:0/3252978974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd0006ee10 0x7fbd000ae380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 >> 172.21.15.72:0/3252978974 conn(0x7fbd000d0a10 msgr2=0x7fbd00142570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.377+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 shutdown_connections 2024-03-06T07:22:20.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.378+0000 7fbd0779a640 1 -- 172.21.15.72:0/3252978974 wait complete. 2024-03-06T07:22:20.390 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:22:20.390 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:20.405 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.543+0000 7fdcd77f6640 1 Processor -- start 2024-03-06T07:22:20.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd77f6640 1 -- start start 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd77f6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcd015dfe0 0x7fdcd00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd77f6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdcd00cf830 0x7fdcd00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd77f6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd00b3430 con 0x7fdcd00d27f0 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd77f6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd00b31e0 con 0x7fdcd015dfe0 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd77f6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd00b3d60 con 0x7fdcd00cf830 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd5592640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcd015dfe0 0x7fdcd00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd5d93640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdcd00d27f0 0x7fdcd00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38470/0 (socket says 172.21.15.72:38470) 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.544+0000 7fdcd5d93640 1 -- 172.21.15.72:0/494783267 learned_addr learned my addr 172.21.15.72:0/494783267 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.545+0000 7fdcd4d91640 1 --2- 172.21.15.72:0/494783267 >> v2:172.21.15.146:3300/0 conn(0x7fdcd00cf830 0x7fdcd00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.545+0000 7fdcd5592640 1 -- 172.21.15.72:0/494783267 >> v2:172.21.15.146:3300/0 conn(0x7fdcd00cf830 msgr2=0x7fdcd00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.545+0000 7fdcd5592640 1 --2- 172.21.15.72:0/494783267 >> v2:172.21.15.146:3300/0 conn(0x7fdcd00cf830 0x7fdcd00d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.545+0000 7fdcd5592640 1 -- 172.21.15.72:0/494783267 >> v1:172.21.15.72:6789/0 conn(0x7fdcd00d27f0 legacy=0x7fdcd00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.545+0000 7fdcd5592640 1 -- 172.21.15.72:0/494783267 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd00b3710 con 0x7fdcd015dfe0 2024-03-06T07:22:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.545+0000 7fdcc7fff640 1 -- 172.21.15.72:0/494783267 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015316694 0 0) 0x7fdcd00b3430 con 0x7fdcd00d27f0 2024-03-06T07:22:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd5592640 1 --2- 172.21.15.72:0/494783267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd015dfe0 0x7fdcd00cf2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00097a0 tx=0x7fdcc002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df5a8aebe566b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcc7fff640 1 -- 172.21.15.72:0/494783267 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0035020 con 0x7fdcd015dfe0 2024-03-06T07:22:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcc7fff640 1 -- 172.21.15.72:0/494783267 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0031830 con 0x7fdcd015dfe0 2024-03-06T07:22:20.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcc7fff640 1 -- 172.21.15.72:0/494783267 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0031c00 con 0x7fdcd015dfe0 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 -- 172.21.15.72:0/494783267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd015dfe0 msgr2=0x7fdcd00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/494783267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd015dfe0 0x7fdcd00cf2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fdcc00097a0 tx=0x7fdcc002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 -- 172.21.15.72:0/494783267 shutdown_connections 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/494783267 >> v2:172.21.15.146:3300/0 conn(0x7fdcd00cf830 0x7fdcd00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/494783267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd015dfe0 0x7fdcd00cf2f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 -- 172.21.15.72:0/494783267 >> 172.21.15.72:0/494783267 conn(0x7fdcd00d3cd0 msgr2=0x7fdcd00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 -- 172.21.15.72:0/494783267 shutdown_connections 2024-03-06T07:22:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.546+0000 7fdcd77f6640 1 -- 172.21.15.72:0/494783267 wait complete. 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.547+0000 7fdcd77f6640 1 Processor -- start 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 -- start start 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 0x7fdcd00b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd00d27f0 0x7fdcd01f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd015dfe0 0x7fdcd015b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00d1df0 con 0x7fdcd00cf830 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00d1c70 con 0x7fdcd00d27f0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd77f6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd015e3c0 con 0x7fdcd015dfe0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd5592640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 0x7fdcd00b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd5592640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 0x7fdcd00b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42184/0 (socket says 172.21.15.72:42184) 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd5592640 1 -- 172.21.15.72:0/2839387702 learned_addr learned my addr 172.21.15.72:0/2839387702 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd4d91640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd00d27f0 0x7fdcd01f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.548+0000 7fdcd5d93640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd015dfe0 0x7fdcd015b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5592640 1 -- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd015dfe0 msgr2=0x7fdcd015b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5592640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd015dfe0 0x7fdcd015b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5592640 1 -- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd00d27f0 msgr2=0x7fdcd01f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5592640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd00d27f0 0x7fdcd01f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5592640 1 -- 172.21.15.72:0/2839387702 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd00b95a0 con 0x7fdcd00cf830 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5d93640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd015dfe0 0x7fdcd015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd5592640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 0x7fdcd00b8bc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0000c00 tx=0x7fdcc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc0035070 con 0x7fdcd00cf830 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0004230 con 0x7fdcd00cf830 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc003a6f0 con 0x7fdcd00cf830 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd01f6d60 con 0x7fdcd00cf830 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.549+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd01f7320 con 0x7fdcd00cf830 2024-03-06T07:22:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.550+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd004f330 con 0x7fdcd00cf830 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.551+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcc0038020 con 0x7fdcd00cf830 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.551+0000 7fdcc5ffb640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcb406b4d0 0x7fdcb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.551+0000 7fdcd4d91640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcb406b4d0 0x7fdcb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.551+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdcc00abe60 con 0x7fdcd00cf830 2024-03-06T07:22:20.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.552+0000 7fdcd4d91640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcb406b4d0 0x7fdcb406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdccc006fd0 tx=0x7fdccc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.556+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdcc0040070 con 0x7fdcd00cf830 2024-03-06T07:22:20.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.755+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcd01f7560 con 0x7fdcd00cf830 2024-03-06T07:22:20.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.756+0000 7fdcc5ffb640 1 -- 172.21.15.72:0/2839387702 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fdcc0031800 con 0x7fdcd00cf830 2024-03-06T07:22:20.758 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:20.759 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcb406b4d0 msgr2=0x7fdcb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcb406b4d0 0x7fdcb406d990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdccc006fd0 tx=0x7fdccc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 msgr2=0x7fdcd00b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 0x7fdcd00b8bc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fdcc0000c00 tx=0x7fdcc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 shutdown_connections 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcb406b4d0 0x7fdcb406d990 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd015dfe0 0x7fdcd015b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd00d27f0 0x7fdcd01f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 --2- 172.21.15.72:0/2839387702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd00cf830 0x7fdcd00b8bc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 >> 172.21.15.72:0/2839387702 conn(0x7fdcd00d3cd0 msgr2=0x7fdcd00d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 shutdown_connections 2024-03-06T07:22:20.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.758+0000 7fdcd77f6640 1 -- 172.21.15.72:0/2839387702 wait complete. 2024-03-06T07:22:20.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:20.772 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:20.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.874+0000 7f281a183640 1 Processor -- start 2024-03-06T07:22:20.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.874+0000 7f281a183640 1 -- start start 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f281a183640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 0x7f281406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f281a183640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 0x7f28140cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f281a183640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28140cd4b0 con 0x7f28140ccca0 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f281a183640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f281406ff80 con 0x7f281415c050 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f281a183640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28140cd790 con 0x7f281406fb40 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f2812ffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 0x7f28140cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f28137fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 0x7f281406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f2812ffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 0x7f28140cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:41344/0 (socket says 172.21.15.72:41344) 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f28137fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 0x7f281406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:59332/0 (socket says 172.21.15.72:59332) 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f2812ffd640 1 -- 172.21.15.72:0/15989790 learned_addr learned my addr 172.21.15.72:0/15989790 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.875+0000 7f2813fff640 1 --1- 172.21.15.72:0/15989790 >> v1:172.21.15.146:6789/0 conn(0x7f28140ccca0 0x7f28140cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:36056/0 (socket says 172.21.15.72:36056) 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.876+0000 7f2812ffd640 1 -- 172.21.15.72:0/15989790 >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 msgr2=0x7f281406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:20.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.876+0000 7f2812ffd640 1 --2- 172.21.15.72:0/15989790 >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 0x7f281406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.876+0000 7f2812ffd640 1 -- 172.21.15.72:0/15989790 >> v1:172.21.15.146:6789/0 conn(0x7f28140ccca0 legacy=0x7f28140cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.876+0000 7f2812ffd640 1 -- 172.21.15.72:0/15989790 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28140cda70 con 0x7f281406fb40 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.876+0000 7f28137fe640 1 --2- 172.21.15.72:0/15989790 >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 0x7f281406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f2812ffd640 1 --2- 172.21.15.72:0/15989790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 0x7f28140cc760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2804009870 tx=0x7f280402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e00d85a2af4e806e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f28127fc640 1 -- 172.21.15.72:0/15989790 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2804035020 con 0x7f281406fb40 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f28127fc640 1 -- 172.21.15.72:0/15989790 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2804031640 con 0x7f281406fb40 2024-03-06T07:22:20.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f28127fc640 1 -- 172.21.15.72:0/15989790 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280402d040 con 0x7f281406fb40 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 -- 172.21.15.72:0/15989790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 msgr2=0x7f28140cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 --2- 172.21.15.72:0/15989790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 0x7f28140cc760 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2804009870 tx=0x7f280402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 -- 172.21.15.72:0/15989790 shutdown_connections 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 --2- 172.21.15.72:0/15989790 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f281406fb40 0x7f28140cc760 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 --2- 172.21.15.72:0/15989790 >> v2:172.21.15.146:3300/0 conn(0x7f281415c050 0x7f281406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 -- 172.21.15.72:0/15989790 >> 172.21.15.72:0/15989790 conn(0x7f28140b3800 msgr2=0x7f28140b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 -- 172.21.15.72:0/15989790 shutdown_connections 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.877+0000 7f281a183640 1 -- 172.21.15.72:0/15989790 wait complete. 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.878+0000 7f281a183640 1 Processor -- start 2024-03-06T07:22:20.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 -- start start 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 0x7f28140b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28140ccca0 0x7f28141469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f281415c050 0x7f28141566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28140d01f0 con 0x7f281406fb40 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28140cd130 con 0x7f281415c050 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f281a183640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28140d0370 con 0x7f28140ccca0 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.879+0000 7f28137fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 0x7f28140b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 0x7f28140b74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42188/0 (socket says 172.21.15.72:42188) 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 -- 172.21.15.72:0/1292595602 learned_addr learned my addr 172.21.15.72:0/1292595602 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f2812ffd640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28140ccca0 0x7f28141469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f2813fff640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f281415c050 0x7f28141566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 -- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28140ccca0 msgr2=0x7f28141469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28140ccca0 0x7f28141469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 -- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f281415c050 msgr2=0x7f28141566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f281415c050 0x7f28141566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 -- 172.21.15.72:0/1292595602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2814156c30 con 0x7f281406fb40 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f2812ffd640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28140ccca0 0x7f28141469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f28137fe640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 0x7f28140b74d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f280800ebe0 tx=0x7f280800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808017070 con 0x7f281406fb40 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280800cc30 con 0x7f281406fb40 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2814156ed0 con 0x7f281406fb40 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2808013470 con 0x7f281406fb40 2024-03-06T07:22:20.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.880+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2814157490 con 0x7f281406fb40 2024-03-06T07:22:20.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.882+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2814157bb0 con 0x7f281406fb40 2024-03-06T07:22:20.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.882+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f280801a070 con 0x7f281406fb40 2024-03-06T07:22:20.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.882+0000 7f27fbfff640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f280006b500 0x7f280006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:20.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.882+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f280808a440 con 0x7f281406fb40 2024-03-06T07:22:20.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.882+0000 7f2812ffd640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f280006b500 0x7f280006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:20.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.883+0000 7f2812ffd640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f280006b500 0x7f280006d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2804036000 tx=0x7f280403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:20.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:20.887+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2808058c70 con 0x7f281406fb40 2024-03-06T07:22:21.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.093+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f281409fe60 con 0x7f281406fb40 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.094+0000 7f27fbfff640 1 -- 172.21.15.72:0/1292595602 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f280805cd10 con 0x7f281406fb40 2024-03-06T07:22:21.096 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f280006b500 msgr2=0x7f280006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f280006b500 0x7f280006d9c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2804036000 tx=0x7f280403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 msgr2=0x7f28140b74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 0x7f28140b74d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f280800ebe0 tx=0x7f280800c520 comp rx=0 tx=0).stop 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 shutdown_connections 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f280006b500 0x7f280006d9c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f281415c050 0x7f28141566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28140ccca0 0x7f28141469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 --2- 172.21.15.72:0/1292595602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f281406fb40 0x7f28140b74d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 >> 172.21.15.72:0/1292595602 conn(0x7f28140b3800 msgr2=0x7f28140d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 shutdown_connections 2024-03-06T07:22:21.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.096+0000 7f281a183640 1 -- 172.21.15.72:0/1292595602 wait complete. 2024-03-06T07:22:21.109 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:21.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.210+0000 7f9dc0156640 1 Processor -- start 2024-03-06T07:22:21.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.211+0000 7f9dc0156640 1 -- start start 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.211+0000 7f9dc0156640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9db80d14a0 0x7f9db815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.211+0000 7f9dc0156640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9db815eb00 0x7f9db80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.211+0000 7f9dc0156640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9db80cfa50 con 0x7f9db80d10c0 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.211+0000 7f9dc0156640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db815c050 con 0x7f9db80d14a0 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.211+0000 7f9dc0156640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9db80d0060 con 0x7f9db815eb00 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbd6f1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9db80d14a0 0x7f9db815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbdef2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9db80d10c0 0x7f9db80d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38486/0 (socket says 172.21.15.72:38486) 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbdef2640 1 -- 172.21.15.72:0/488512408 learned_addr learned my addr 172.21.15.72:0/488512408 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbe6f3640 1 --2- 172.21.15.72:0/488512408 >> v2:172.21.15.146:3300/0 conn(0x7f9db815eb00 0x7f9db80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/488512408 >> v2:172.21.15.146:3300/0 conn(0x7f9db815eb00 msgr2=0x7f9db80cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbd6f1640 1 --2- 172.21.15.72:0/488512408 >> v2:172.21.15.146:3300/0 conn(0x7f9db815eb00 0x7f9db80cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/488512408 >> v1:172.21.15.72:6789/0 conn(0x7f9db80d10c0 legacy=0x7f9db80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.212+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/488512408 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db80b45c0 con 0x7f9db80d14a0 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.213+0000 7f9dbd6f1640 1 --2- 172.21.15.72:0/488512408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db815e5c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9db8071930 tx=0x7f9da802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb6d4c151ba4b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.213+0000 7f9dbcef0640 1 -- 172.21.15.72:0/488512408 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8035020 con 0x7f9db80d14a0 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.213+0000 7f9dbcef0640 1 -- 172.21.15.72:0/488512408 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da8031830 con 0x7f9db80d14a0 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.213+0000 7f9dbcef0640 1 -- 172.21.15.72:0/488512408 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8031c00 con 0x7f9db80d14a0 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 -- 172.21.15.72:0/488512408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 msgr2=0x7f9db815e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 --2- 172.21.15.72:0/488512408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db815e5c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9db8071930 tx=0x7f9da802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 -- 172.21.15.72:0/488512408 shutdown_connections 2024-03-06T07:22:21.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 --2- 172.21.15.72:0/488512408 >> v2:172.21.15.146:3300/0 conn(0x7f9db815eb00 0x7f9db80cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 --2- 172.21.15.72:0/488512408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db815e5c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 -- 172.21.15.72:0/488512408 >> 172.21.15.72:0/488512408 conn(0x7f9db80b3800 msgr2=0x7f9db80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 -- 172.21.15.72:0/488512408 shutdown_connections 2024-03-06T07:22:21.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.214+0000 7f9dc0156640 1 -- 172.21.15.72:0/488512408 wait complete. 2024-03-06T07:22:21.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.215+0000 7f9dc0156640 1 Processor -- start 2024-03-06T07:22:21.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.215+0000 7f9dc0156640 1 -- start start 2024-03-06T07:22:21.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9db80d10c0 0x7f9db80b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db80cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9db815eb00 0x7f9db80cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80cfcf0 con 0x7f9db80d14a0 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db815f440 con 0x7f9db815eb00 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9db80cb3c0 con 0x7f9db80d10c0 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db80cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db80cc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42202/0 (socket says 172.21.15.72:42202) 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/1953092570 learned_addr learned my addr 172.21.15.72:0/1953092570 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbe6f3640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9db815eb00 0x7f9db80cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbdef2640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9db80d10c0 0x7f9db80b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9db80d10c0 msgr2=0x7f9db80b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9db80d10c0 0x7f9db80b64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9db815eb00 msgr2=0x7f9db80cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9db815eb00 0x7f9db80cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 -- 172.21.15.72:0/1953092570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db81f7b10 con 0x7f9db80d14a0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbdef2640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9db80d10c0 0x7f9db80b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dbd6f1640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db80cc180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009890 tx=0x7f9da80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8041070 con 0x7f9db80d14a0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da803aa30 con 0x7f9db80d14a0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db80cccf0 con 0x7f9db80d14a0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8035040 con 0x7f9db80d14a0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.216+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db80cd2b0 con 0x7f9db80d14a0 2024-03-06T07:22:21.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.217+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db809fe60 con 0x7f9db80d14a0 2024-03-06T07:22:21.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.218+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9da8038020 con 0x7f9db80d14a0 2024-03-06T07:22:21.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.218+0000 7f9dae7fc640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d9c06b4d0 0x7f9d9c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.218+0000 7f9dbdef2640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d9c06b4d0 0x7f9d9c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.219+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9da80abc90 con 0x7f9db80d14a0 2024-03-06T07:22:21.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.219+0000 7f9dbdef2640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d9c06b4d0 0x7f9d9c06d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9db4000f60 tx=0x7f9db4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.223+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9da807ab10 con 0x7f9db80d14a0 2024-03-06T07:22:21.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.398+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9db80d0c30 con 0x7f9db80d14a0 2024-03-06T07:22:21.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.398+0000 7f9dae7fc640 1 -- 172.21.15.72:0/1953092570 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9da807a410 con 0x7f9db80d14a0 2024-03-06T07:22:21.400 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:21.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d9c06b4d0 msgr2=0x7f9d9c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d9c06b4d0 0x7f9d9c06d990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9db4000f60 tx=0x7f9db4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 msgr2=0x7f9db80cc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db80cc180 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9da8009890 tx=0x7f9da80046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 shutdown_connections 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d9c06b4d0 0x7f9d9c06d990 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9db815eb00 0x7f9db80cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9db80d14a0 0x7f9db80cc180 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 --2- 172.21.15.72:0/1953092570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9db80d10c0 0x7f9db80b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 >> 172.21.15.72:0/1953092570 conn(0x7f9db80b3800 msgr2=0x7f9db815d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 shutdown_connections 2024-03-06T07:22:21.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.400+0000 7f9dc0156640 1 -- 172.21.15.72:0/1953092570 wait complete. 2024-03-06T07:22:21.413 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:21.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.531+0000 7f179b2c9640 1 Processor -- start 2024-03-06T07:22:21.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.531+0000 7f179b2c9640 1 -- start start 2024-03-06T07:22:21.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f179b2c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f179415c510 0x7f17940b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f179b2c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179406be30 con 0x7f179415b240 2024-03-06T07:22:21.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f179b2c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179415b620 con 0x7f17940af870 2024-03-06T07:22:21.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f179b2c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17940cf980 con 0x7f179415c510 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f1799866640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f179415c510 0x7f17940b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f1799065640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f179415b240 0x7f179406b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38496/0 (socket says 172.21.15.72:38496) 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f1799065640 1 -- 172.21.15.72:0/3765587912 learned_addr learned my addr 172.21.15.72:0/3765587912 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f178bfff640 1 -- 172.21.15.72:0/3765587912 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285339145 0 0) 0x7f179406be30 con 0x7f179415b240 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f178bfff640 1 -- 172.21.15.72:0/3765587912 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1780003520 con 0x7f179415b240 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f1799866640 1 -- 172.21.15.72:0/3765587912 >> v1:172.21.15.146:6789/0 conn(0x7f17940af870 legacy=0x7f179415be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f178bfff640 1 -- 172.21.15.72:0/3765587912 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124477343 0 0) 0x7f179415b620 con 0x7f17940af870 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f1799866640 1 -- 172.21.15.72:0/3765587912 >> v1:172.21.15.72:6789/0 conn(0x7f179415b240 legacy=0x7f179406b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.533+0000 7f1799866640 1 -- 172.21.15.72:0/3765587912 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1794073170 con 0x7f179415c510 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.534+0000 7f1799866640 1 --2- 172.21.15.72:0/3765587912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f179415c510 0x7f17940b2890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1790009d90 tx=0x7f179002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a5ec4489dfadef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.534+0000 7f178bfff640 1 -- 172.21.15.72:0/3765587912 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790039070 con 0x7f179415c510 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.534+0000 7f178bfff640 1 -- 172.21.15.72:0/3765587912 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179002cd00 con 0x7f179415c510 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.534+0000 7f178bfff640 1 -- 172.21.15.72:0/3765587912 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790034660 con 0x7f179415c510 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.534+0000 7f179b2c9640 1 -- 172.21.15.72:0/3765587912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f179415c510 msgr2=0x7f17940b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.534+0000 7f179b2c9640 1 --2- 172.21.15.72:0/3765587912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f179415c510 0x7f17940b2890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1790009d90 tx=0x7f179002bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.535+0000 7f179b2c9640 1 -- 172.21.15.72:0/3765587912 shutdown_connections 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.535+0000 7f179b2c9640 1 --2- 172.21.15.72:0/3765587912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f179415c510 0x7f17940b2890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.535+0000 7f179b2c9640 1 -- 172.21.15.72:0/3765587912 >> 172.21.15.72:0/3765587912 conn(0x7f17940b1d70 msgr2=0x7f17940b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.535+0000 7f179b2c9640 1 -- 172.21.15.72:0/3765587912 shutdown_connections 2024-03-06T07:22:21.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.535+0000 7f179b2c9640 1 -- 172.21.15.72:0/3765587912 wait complete. 2024-03-06T07:22:21.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 Processor -- start 2024-03-06T07:22:21.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 -- start start 2024-03-06T07:22:21.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 0x7f179414c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f179415b240 0x7f17941f3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179415c510 0x7f17940c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179406b460 con 0x7f17940af870 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179415b8e0 con 0x7f179415c510 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.536+0000 7f179b2c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17940b0ed0 con 0x7f179415b240 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 0x7f179414c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 0x7f179414c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42218/0 (socket says 172.21.15.72:42218) 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 -- 172.21.15.72:0/501794912 learned_addr learned my addr 172.21.15.72:0/501794912 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799866640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179415c510 0x7f17940c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1798864640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f179415b240 0x7f17941f3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 -- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f179415b240 msgr2=0x7f17941f3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f179415b240 0x7f17941f3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 -- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179415c510 msgr2=0x7f17940c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179415c510 0x7f17940c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 -- 172.21.15.72:0/501794912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179414ce00 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1798864640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f179415b240 0x7f17941f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1799065640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 0x7f179414c8c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f178c00c430 tx=0x7f178c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c017070 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178c005ce0 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178c0072f0 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17941f4750 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.537+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17940c73b0 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.538+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17941f4c70 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.539+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f178c01a070 con 0x7f17940af870 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.540+0000 7f1789ffb640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f177806b500 0x7f177806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.540+0000 7f1798864640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f177806b500 0x7f177806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.540+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f178c089680 con 0x7f17940af870 2024-03-06T07:22:21.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.540+0000 7f1798864640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f177806b500 0x7f177806d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f179406dc80 tx=0x7f1784009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.544+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f17941f4c70 con 0x7f17940af870 2024-03-06T07:22:21.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.723+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f179409e3d0 con 0x7f17940af870 2024-03-06T07:22:21.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.723+0000 7f1789ffb640 1 -- 172.21.15.72:0/501794912 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f178c058500 con 0x7f17940af870 2024-03-06T07:22:21.725 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:21.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f177806b500 msgr2=0x7f177806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f177806b500 0x7f177806d9c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f179406dc80 tx=0x7f1784009040 comp rx=0 tx=0).stop 2024-03-06T07:22:21.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 msgr2=0x7f179414c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 0x7f179414c8c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f178c00c430 tx=0x7f178c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 shutdown_connections 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f177806b500 0x7f177806d9c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f179415c510 0x7f17940c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f179415b240 0x7f17941f3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 --2- 172.21.15.72:0/501794912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17940af870 0x7f179414c8c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 >> 172.21.15.72:0/501794912 conn(0x7f17940b1d70 msgr2=0x7f1794072550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 shutdown_connections 2024-03-06T07:22:21.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.726+0000 7f179b2c9640 1 -- 172.21.15.72:0/501794912 wait complete. 2024-03-06T07:22:21.742 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.842+0000 7fbc9f4b5640 1 Processor -- start 2024-03-06T07:22:21.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.842+0000 7fbc9f4b5640 1 -- start start 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9f4b5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbc98144030 0x7fbc9806e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9f4b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc980b4ef0 0x7fbc980cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9f4b5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc980b4450 con 0x7fbc9806ee10 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9f4b5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc980b4a60 con 0x7fbc98144030 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9f4b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980b5b60 con 0x7fbc980b4ef0 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9da52640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc980b4ef0 0x7fbc980cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9ca50640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbc98144030 0x7fbc9806e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9d251640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbc9806ee10 0x7fbc9806f1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:36084/0 (socket says 172.21.15.72:36084) 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.843+0000 7fbc9d251640 1 -- 172.21.15.72:0/78223941 learned_addr learned my addr 172.21.15.72:0/78223941 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc8ffff640 1 -- 172.21.15.72:0/78223941 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091174614 0 0) 0x7fbc980b4450 con 0x7fbc9806ee10 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc8ffff640 1 -- 172.21.15.72:0/78223941 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84003520 con 0x7fbc9806ee10 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc9da52640 1 -- 172.21.15.72:0/78223941 >> v2:172.21.15.146:3300/0 conn(0x7fbc98144030 msgr2=0x7fbc9806e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc9da52640 1 --2- 172.21.15.72:0/78223941 >> v2:172.21.15.146:3300/0 conn(0x7fbc98144030 0x7fbc9806e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc9da52640 1 -- 172.21.15.72:0/78223941 >> v1:172.21.15.146:6789/0 conn(0x7fbc9806ee10 legacy=0x7fbc9806f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc9da52640 1 -- 172.21.15.72:0/78223941 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980b5e40 con 0x7fbc980b4ef0 2024-03-06T07:22:21.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.844+0000 7fbc9ca50640 1 --2- 172.21.15.72:0/78223941 >> v2:172.21.15.146:3300/0 conn(0x7fbc98144030 0x7fbc9806e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9da52640 1 --2- 172.21.15.72:0/78223941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc980b4ef0 0x7fbc980cdde0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000bfa0 tx=0x7fbc9002e0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=174e8242e95a49ed server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc8ffff640 1 -- 172.21.15.72:0/78223941 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9002ea40 con 0x7fbc980b4ef0 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc8ffff640 1 -- 172.21.15.72:0/78223941 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9002ebe0 con 0x7fbc980b4ef0 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc8ffff640 1 -- 172.21.15.72:0/78223941 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90009540 con 0x7fbc980b4ef0 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/78223941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc980b4ef0 msgr2=0x7fbc980cdde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/78223941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc980b4ef0 0x7fbc980cdde0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000bfa0 tx=0x7fbc9002e0a0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/78223941 shutdown_connections 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/78223941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc980b4ef0 0x7fbc980cdde0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/78223941 >> v2:172.21.15.146:3300/0 conn(0x7fbc98144030 0x7fbc9806e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/78223941 >> 172.21.15.72:0/78223941 conn(0x7fbc980d0a10 msgr2=0x7fbc980d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/78223941 shutdown_connections 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.845+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/78223941 wait complete. 2024-03-06T07:22:21.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 Processor -- start 2024-03-06T07:22:21.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 -- start start 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9806ee10 0x7fbc980ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 0x7fbc9813b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc98144030 0x7fbc980cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980b5380 con 0x7fbc980b4ef0 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98144840 con 0x7fbc9806ee10 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9f4b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc980b5500 con 0x7fbc98144030 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9ca50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 0x7fbc9813b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9ca50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 0x7fbc9813b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42222/0 (socket says 172.21.15.72:42222) 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.847+0000 7fbc9ca50640 1 -- 172.21.15.72:0/3193997957 learned_addr learned my addr 172.21.15.72:0/3193997957 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9d251640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9806ee10 0x7fbc980ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9da52640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc98144030 0x7fbc980cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9ca50640 1 -- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc98144030 msgr2=0x7fbc980cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9ca50640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc98144030 0x7fbc980cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9ca50640 1 -- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9806ee10 msgr2=0x7fbc980ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9ca50640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9806ee10 0x7fbc980ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9ca50640 1 -- 172.21.15.72:0/3193997957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980cdfd0 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9ca50640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 0x7fbc9813b7e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400e960 tx=0x7fbc9400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94017070 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9400ca00 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc980ce2b0 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9400cdd0 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.848+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc980ce870 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.849+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9804f330 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.850+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc9401a070 con 0x7fbc980b4ef0 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.851+0000 7fbc8dffb640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:21.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.851+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc9408a3c0 con 0x7fbc980b4ef0 2024-03-06T07:22:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.851+0000 7fbc9d251640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.851+0000 7fbc9d251640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc88005880 tx=0x7fbc88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:21.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:21.855+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc94058bf0 con 0x7fbc980b4ef0 2024-03-06T07:22:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.045+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbc9806b200 con 0x7fbc980b4ef0 2024-03-06T07:22:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.046+0000 7fbc8dffb640 1 -- 172.21.15.72:0/3193997957 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbc9405cc90 con 0x7fbc980b4ef0 2024-03-06T07:22:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc7c06b4d0 msgr2=0x7fbc7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbc88005880 tx=0x7fbc88009040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 msgr2=0x7fbc9813b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 0x7fbc9813b7e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbc9400e960 tx=0x7fbc9400ee20 comp rx=0 tx=0).stop 2024-03-06T07:22:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 shutdown_connections 2024-03-06T07:22:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc7c06b4d0 0x7fbc7c06d990 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc98144030 0x7fbc980cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc980b4ef0 0x7fbc9813b7e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 --2- 172.21.15.72:0/3193997957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc9806ee10 0x7fbc980ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 >> 172.21.15.72:0/3193997957 conn(0x7fbc980d0a10 msgr2=0x7fbc98140fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 shutdown_connections 2024-03-06T07:22:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.049+0000 7fbc9f4b5640 1 -- 172.21.15.72:0/3193997957 wait complete. 2024-03-06T07:22:22.062 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:22:22.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.063 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:22.091 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:22.092 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:22:22.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:22.092 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:22:22.152 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:22:22.153 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:22.171 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:22:22.171 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:22.229 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:22.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.332+0000 7fb4a0a5c640 1 Processor -- start 2024-03-06T07:22:22.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb4a0a5c640 1 -- start start 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb4a0a5c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb49c15dfe0 0x7fb49c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb4a0a5c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb49c0d29c0 0x7fb49c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb4a0a5c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49c0b3430 con 0x7fb49c0cf830 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb4a0a5c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c0b31e0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb4a0a5c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49c0b3d60 con 0x7fb49c15dfe0 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb493fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb49c0cf830 0x7fb49c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38498/0 (socket says 172.21.15.72:38498) 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb49ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb49c0d29c0 0x7fb49c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.333+0000 7fb493fff640 1 -- 172.21.15.72:0/1308297552 learned_addr learned my addr 172.21.15.72:0/1308297552 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.334+0000 7fb49a59c640 1 --2- 172.21.15.72:0/1308297552 >> v2:172.21.15.146:3300/0 conn(0x7fb49c15dfe0 0x7fb49c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.334+0000 7fb49ad9d640 1 -- 172.21.15.72:0/1308297552 >> v2:172.21.15.146:3300/0 conn(0x7fb49c15dfe0 msgr2=0x7fb49c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.334+0000 7fb49ad9d640 1 --2- 172.21.15.72:0/1308297552 >> v2:172.21.15.146:3300/0 conn(0x7fb49c15dfe0 0x7fb49c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.334+0000 7fb49ad9d640 1 -- 172.21.15.72:0/1308297552 >> v1:172.21.15.72:6789/0 conn(0x7fb49c0cf830 legacy=0x7fb49c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.334+0000 7fb49ad9d640 1 -- 172.21.15.72:0/1308297552 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c0b3710 con 0x7fb49c0d29c0 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.334+0000 7fb49ad9d640 1 --2- 172.21.15.72:0/1308297552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b41f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb49c067990 tx=0x7fb49402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce34c2061a8c8bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb499d9b640 1 -- 172.21.15.72:0/1308297552 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494039070 con 0x7fb49c0d29c0 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb499d9b640 1 -- 172.21.15.72:0/1308297552 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49402cc80 con 0x7fb49c0d29c0 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb499d9b640 1 -- 172.21.15.72:0/1308297552 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb494034660 con 0x7fb49c0d29c0 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1308297552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 msgr2=0x7fb49c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1308297552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b41f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb49c067990 tx=0x7fb49402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1308297552 shutdown_connections 2024-03-06T07:22:22.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1308297552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b41f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1308297552 >> v2:172.21.15.146:3300/0 conn(0x7fb49c15dfe0 0x7fb49c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1308297552 >> 172.21.15.72:0/1308297552 conn(0x7fb49c0d3cd0 msgr2=0x7fb49c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1308297552 shutdown_connections 2024-03-06T07:22:22.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.335+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1308297552 wait complete. 2024-03-06T07:22:22.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.337+0000 7fb4a0a5c640 1 Processor -- start 2024-03-06T07:22:22.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.337+0000 7fb4a0a5c640 1 -- start start 2024-03-06T07:22:22.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb49c0cf830 0x7fb49c161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb49c15dfe0 0x7fb49c0b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0d1df0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c0d1c70 con 0x7fb49c15dfe0 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49c15e3c0 con 0x7fb49c0cf830 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42242/0 (socket says 172.21.15.72:42242) 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 -- 172.21.15.72:0/1562205735 learned_addr learned my addr 172.21.15.72:0/1562205735 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb49ad9d640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb49c15dfe0 0x7fb49c0b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb49a59c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb49c0cf830 0x7fb49c161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 -- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb49c0cf830 msgr2=0x7fb49c161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb49c0cf830 0x7fb49c161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 -- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb49c15dfe0 msgr2=0x7fb49c0b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb49c15dfe0 0x7fb49c0b8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 -- 172.21.15.72:0/1562205735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c161580 con 0x7fb49c0d29c0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb49a59c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb49c0cf830 0x7fb49c161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb493fff640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b88e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb48400c440 tx=0x7fb484001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484017070 con 0x7fb49c0d29c0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb484005ce0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4840072f0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49c14fc00 con 0x7fb49c0d29c0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.338+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49c1501c0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.340+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49c04f330 con 0x7fb49c0d29c0 2024-03-06T07:22:22.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.340+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb48401a070 con 0x7fb49c0d29c0 2024-03-06T07:22:22.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.340+0000 7fb492ffd640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb47c06b4d0 0x7fb47c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.340+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4840894b0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.340+0000 7fb49a59c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb47c06b4d0 0x7fb47c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.341+0000 7fb49a59c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb47c06b4d0 0x7fb47c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.345+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb484057d70 con 0x7fb49c0d29c0 2024-03-06T07:22:22.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.634+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb49c0ca9b0 con 0x7fb49c0d29c0 2024-03-06T07:22:22.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.635+0000 7fb492ffd640 1 -- 172.21.15.72:0/1562205735 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fb48405be10 con 0x7fb49c0d29c0 2024-03-06T07:22:22.637 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:22.637 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb47c06b4d0 msgr2=0x7fb47c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb47c06b4d0 0x7fb47c06d990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb48c006fd0 tx=0x7fb48c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 msgr2=0x7fb49c0b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b88e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb48400c440 tx=0x7fb484001e70 comp rx=0 tx=0).stop 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 shutdown_connections 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb47c06b4d0 0x7fb47c06d990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb49c15dfe0 0x7fb49c0b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb49c0d29c0 0x7fb49c0b88e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 --2- 172.21.15.72:0/1562205735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb49c0cf830 0x7fb49c161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 >> 172.21.15.72:0/1562205735 conn(0x7fb49c0d3cd0 msgr2=0x7fb49c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 shutdown_connections 2024-03-06T07:22:22.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.637+0000 7fb4a0a5c640 1 -- 172.21.15.72:0/1562205735 wait complete. 2024-03-06T07:22:22.651 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:22.652 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:22.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.754+0000 7f32fed65640 1 Processor -- start 2024-03-06T07:22:22.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.754+0000 7f32fed65640 1 -- start start 2024-03-06T07:22:22.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32fed65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32f80d4860 0x7f32f80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32fed65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f81618b0 con 0x7f32f81614d0 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32fed65640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f80cf400 con 0x7f32f80b0e30 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32fed65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80d4cf0 con 0x7f32f80d4860 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32f7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32f80d4860 0x7f32f80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32fcb01640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f32f81614d0 0x7f32f80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39242/0 (socket says 172.21.15.72:39242) 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32fcb01640 1 -- 172.21.15.72:0/2030964076 learned_addr learned my addr 172.21.15.72:0/2030964076 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32f7fff640 1 -- 172.21.15.72:0/2030964076 >> v1:172.21.15.146:6789/0 conn(0x7f32f80b0e30 legacy=0x7f32f80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32f7fff640 1 -- 172.21.15.72:0/2030964076 >> v1:172.21.15.72:6789/0 conn(0x7f32f81614d0 legacy=0x7f32f80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32f77fe640 1 -- 172.21.15.72:0/2030964076 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989926832 0 0) 0x7f32f81618b0 con 0x7f32f81614d0 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.755+0000 7f32f7fff640 1 -- 172.21.15.72:0/2030964076 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80cf6a0 con 0x7f32f80d4860 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.756+0000 7f32f7fff640 1 --2- 172.21.15.72:0/2030964076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80d2020 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f32e40097a0 tx=0x7f32e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a1a8904dff74d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.756+0000 7f32f77fe640 1 -- 172.21.15.72:0/2030964076 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4035020 con 0x7f32f80d4860 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.756+0000 7f32f77fe640 1 -- 172.21.15.72:0/2030964076 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e4031830 con 0x7f32f80d4860 2024-03-06T07:22:22.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.756+0000 7f32f77fe640 1 -- 172.21.15.72:0/2030964076 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e4031c00 con 0x7f32f80d4860 2024-03-06T07:22:22.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 -- 172.21.15.72:0/2030964076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 msgr2=0x7f32f80d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 --2- 172.21.15.72:0/2030964076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80d2020 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f32e40097a0 tx=0x7f32e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 -- 172.21.15.72:0/2030964076 shutdown_connections 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 --2- 172.21.15.72:0/2030964076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80d2020 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 -- 172.21.15.72:0/2030964076 >> 172.21.15.72:0/2030964076 conn(0x7f32f80d3d40 msgr2=0x7f32f80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 -- 172.21.15.72:0/2030964076 shutdown_connections 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.757+0000 7f32fed65640 1 -- 172.21.15.72:0/2030964076 wait complete. 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.758+0000 7f32fed65640 1 Processor -- start 2024-03-06T07:22:22.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.758+0000 7f32fed65640 1 -- start start 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32fed65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f80b0e30 0x7f32f80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32fed65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80b7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32fed65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f81614d0 0x7f32f806cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32fed65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80b0910 con 0x7f32f80d4860 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32fed65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80cfc60 con 0x7f32f81614d0 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32fed65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80d2560 con 0x7f32f80b0e30 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80b7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80b7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38412/0 (socket says 172.21.15.72:38412) 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.759+0000 7f32f7fff640 1 -- 172.21.15.72:0/1076549250 learned_addr learned my addr 172.21.15.72:0/1076549250 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32fd302640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f81614d0 0x7f32f806cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32fcb01640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f80b0e30 0x7f32f80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f7fff640 1 -- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f80b0e30 msgr2=0x7f32f80b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f7fff640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f80b0e30 0x7f32f80b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f7fff640 1 -- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f81614d0 msgr2=0x7f32f806cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f7fff640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f81614d0 0x7f32f806cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f7fff640 1 -- 172.21.15.72:0/1076549250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80b82b0 con 0x7f32f80d4860 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32fcb01640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f80b0e30 0x7f32f80b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f7fff640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80b7d70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32e40098c0 tx=0x7f32e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e403ba80 con 0x7f32f80d4860 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e403bc20 con 0x7f32f80d4860 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80b95c0 con 0x7f32f80d4860 2024-03-06T07:22:22.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e40453f0 con 0x7f32f80d4860 2024-03-06T07:22:22.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.760+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f80b7570 con 0x7f32f80d4860 2024-03-06T07:22:22.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.761+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f80b9270 con 0x7f32f80d4860 2024-03-06T07:22:22.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.762+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f32e404c060 con 0x7f32f80d4860 2024-03-06T07:22:22.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.762+0000 7f32f57fa640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f32dc06b4d0 0x7f32dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:22.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.762+0000 7f32fcb01640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f32dc06b4d0 0x7f32dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:22.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.762+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32e40abed0 con 0x7f32f80d4860 2024-03-06T07:22:22.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.763+0000 7f32fcb01640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f32dc06b4d0 0x7f32dc06d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32ec000f60 tx=0x7f32ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:22.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.766+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f32e4040070 con 0x7f32f80d4860 2024-03-06T07:22:22.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.986+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f32f8068ff0 con 0x7f32f80d4860 2024-03-06T07:22:22.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.986+0000 7f32f57fa640 1 -- 172.21.15.72:0/1076549250 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32e407ade0 con 0x7f32f80d4860 2024-03-06T07:22:22.989 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:22.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f32dc06b4d0 msgr2=0x7f32dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f32dc06b4d0 0x7f32dc06d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32ec000f60 tx=0x7f32ec009040 comp rx=0 tx=0).stop 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 msgr2=0x7f32f80b7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80b7d70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f32e40098c0 tx=0x7f32e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 shutdown_connections 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f32dc06b4d0 0x7f32dc06d990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f81614d0 0x7f32f806cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32f80d4860 0x7f32f80b7d70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 --2- 172.21.15.72:0/1076549250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f80b0e30 0x7f32f80b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 >> 172.21.15.72:0/1076549250 conn(0x7f32f80d3d40 msgr2=0x7f32f8160360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 shutdown_connections 2024-03-06T07:22:22.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:22.989+0000 7f32fed65640 1 -- 172.21.15.72:0/1076549250 wait complete. 2024-03-06T07:22:23.006 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:23.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.107+0000 7f5bc4593640 1 Processor -- start 2024-03-06T07:22:23.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.107+0000 7f5bc4593640 1 -- start start 2024-03-06T07:22:23.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.108+0000 7f5bc4593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 0x7f5bbc0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.108+0000 7f5bc4593640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bbc0d1910 0x7f5bbc0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.108+0000 7f5bc4593640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc0d2740 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.108+0000 7f5bc4593640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bbc1618b0 con 0x7f5bbc0d1910 2024-03-06T07:22:23.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.108+0000 7f5bc4593640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0cf400 con 0x7f5bbc1614d0 2024-03-06T07:22:23.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 0x7f5bbc0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc1b2e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bbc0d1910 0x7f5bbc0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 0x7f5bbc0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:51374/0 (socket says 172.21.15.72:51374) 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc2b30640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5bbc0d22b0 0x7f5bbc0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:36408/0 (socket says 172.21.15.72:36408) 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 -- 172.21.15.72:0/1681495425 learned_addr learned my addr 172.21.15.72:0/1681495425 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 -- 172.21.15.72:0/1681495425 >> v2:172.21.15.146:3300/0 conn(0x7f5bbc0d1910 msgr2=0x7f5bbc0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc132d640 1 -- 172.21.15.72:0/1681495425 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637273769 0 0) 0x7f5bb4001490 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 --2- 172.21.15.72:0/1681495425 >> v2:172.21.15.146:3300/0 conn(0x7f5bbc0d1910 0x7f5bbc0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 -- 172.21.15.72:0/1681495425 >> v1:172.21.15.146:6789/0 conn(0x7f5bbc0d22b0 legacy=0x7f5bbc0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc232f640 1 -- 172.21.15.72:0/1681495425 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0cf6a0 con 0x7f5bbc1614d0 2024-03-06T07:22:23.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.109+0000 7f5bc1b2e640 1 --2- 172.21.15.72:0/1681495425 >> v2:172.21.15.146:3300/0 conn(0x7f5bbc0d1910 0x7f5bbc0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:23.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.110+0000 7f5bc232f640 1 --2- 172.21.15.72:0/1681495425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 0x7f5bbc0d4860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009870 tx=0x7f5bb802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c86a5d4f5c81c3e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.110+0000 7f5bc132d640 1 -- 172.21.15.72:0/1681495425 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8034070 con 0x7f5bbc1614d0 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.110+0000 7f5bc132d640 1 -- 172.21.15.72:0/1681495425 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb8030840 con 0x7f5bbc1614d0 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.110+0000 7f5bc132d640 1 -- 172.21.15.72:0/1681495425 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8030c10 con 0x7f5bbc1614d0 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 -- 172.21.15.72:0/1681495425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 msgr2=0x7f5bbc0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1681495425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 0x7f5bbc0d4860 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009870 tx=0x7f5bb802b980 comp rx=0 tx=0).stop 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 -- 172.21.15.72:0/1681495425 shutdown_connections 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1681495425 >> v2:172.21.15.146:3300/0 conn(0x7f5bbc0d1910 0x7f5bbc0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1681495425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc1614d0 0x7f5bbc0d4860 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 -- 172.21.15.72:0/1681495425 >> 172.21.15.72:0/1681495425 conn(0x7f5bbc0d3d40 msgr2=0x7f5bbc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 -- 172.21.15.72:0/1681495425 shutdown_connections 2024-03-06T07:22:23.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.111+0000 7f5bc4593640 1 -- 172.21.15.72:0/1681495425 wait complete. 2024-03-06T07:22:23.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.113+0000 7f5bc4593640 1 Processor -- start 2024-03-06T07:22:23.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.113+0000 7f5bc4593640 1 -- start start 2024-03-06T07:22:23.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc4593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0d1910 0x7f5bbc155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc4593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 0x7f5bbc0c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc4593640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc1614d0 0x7f5bbc0cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc4593640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0b0910 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc4593640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0cfc60 con 0x7f5bbc1614d0 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc4593640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc0b1e40 con 0x7f5bbc0d1910 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 0x7f5bbc0c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 0x7f5bbc0c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38420/0 (socket says 172.21.15.72:38420) 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 -- 172.21.15.72:0/1333574489 learned_addr learned my addr 172.21.15.72:0/1333574489 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc2b30640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc1614d0 0x7f5bbc0cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc232f640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0d1910 0x7f5bbc155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 -- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0d1910 msgr2=0x7f5bbc155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0d1910 0x7f5bbc155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 -- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc1614d0 msgr2=0x7f5bbc0cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc1614d0 0x7f5bbc0cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.114+0000 7f5bc1b2e640 1 -- 172.21.15.72:0/1333574489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0caeb0 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.115+0000 7f5bc1b2e640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 0x7f5bbc0c7dc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00ebe0 tx=0x7f5bac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.115+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac017070 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.115+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac00cc30 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.115+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac013470 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.115+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bbc0cb190 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.115+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bbc0cb750 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.116+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bbc068ff0 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.116+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5bac01a070 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.116+0000 7f5bb2ffd640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba006b4d0 0x7f5ba006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.117+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bac08a050 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.117+0000 7f5bc232f640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba006b4d0 0x7f5ba006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.117+0000 7f5bc232f640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba006b4d0 0x7f5ba006d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8036000 tx=0x7f5bb803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.121+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5bac058910 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.294+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5bbc0d07e0 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.294+0000 7f5bb2ffd640 1 -- 172.21.15.72:0/1333574489 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5bac05c9b0 con 0x7f5bbc0d22b0 2024-03-06T07:22:23.296 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:23.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba006b4d0 msgr2=0x7f5ba006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba006b4d0 0x7f5ba006d990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8036000 tx=0x7f5bb803c040 comp rx=0 tx=0).stop 2024-03-06T07:22:23.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 msgr2=0x7f5bbc0c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 0x7f5bbc0c7dc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00ebe0 tx=0x7f5bac00c520 comp rx=0 tx=0).stop 2024-03-06T07:22:23.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 shutdown_connections 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ba006b4d0 0x7f5ba006d990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc1614d0 0x7f5bbc0cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bbc0d22b0 0x7f5bbc0c7dc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 --2- 172.21.15.72:0/1333574489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0d1910 0x7f5bbc155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 >> 172.21.15.72:0/1333574489 conn(0x7f5bbc0d3d40 msgr2=0x7f5bbc0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 shutdown_connections 2024-03-06T07:22:23.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.296+0000 7f5bc4593640 1 -- 172.21.15.72:0/1333574489 wait complete. 2024-03-06T07:22:23.309 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:23.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.411+0000 7fe730c15640 1 Processor -- start 2024-03-06T07:22:23.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.411+0000 7fe730c15640 1 -- start start 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe730c15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe72c15def0 0x7fe72c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe730c15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 0x7fe72c0ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe730c15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe72c06d030 con 0x7fe72c0ce880 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe730c15640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c0cecf0 con 0x7fe72c15def0 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe730c15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0cf1c0 con 0x7fe72c06cab0 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe72ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe72c0ce880 0x7fe72c0cfd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39254/0 (socket says 172.21.15.72:39254) 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe72ad9d640 1 -- 172.21.15.72:0/4033976633 learned_addr learned my addr 172.21.15.72:0/4033976633 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe723fff640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 0x7fe72c0ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe72a59c640 1 --2- 172.21.15.72:0/4033976633 >> v2:172.21.15.146:3300/0 conn(0x7fe72c15def0 0x7fe72c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377902573 0 0) 0x7fe72c06d030 con 0x7fe72c0ce880 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe718003520 con 0x7fe72c0ce880 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690380713 0 0) 0x7fe718003520 con 0x7fe72c0ce880 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 msgr2=0x7fe72c0ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 0x7fe72c0ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 >> v2:172.21.15.146:3300/0 conn(0x7fe72c15def0 msgr2=0x7fe72c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 --2- 172.21.15.72:0/4033976633 >> v2:172.21.15.146:3300/0 conn(0x7fe72c15def0 0x7fe72c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c06d310 con 0x7fe72c0ce880 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe7240019f0 con 0x7fe72c0ce880 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 >> v1:172.21.15.72:6789/0 conn(0x7fe72c0ce880 legacy=0x7fe72c0cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 0x7fe7180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe718003f30 0x7fe718004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe7180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0cecf0 con 0x7fe718004d40 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71800b9c0 con 0x7fe718003f30 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71800bb70 con 0x7fe72c06cab0 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.412+0000 7fe72a59c640 1 --2- 172.21.15.72:0/4033976633 >> v2:172.21.15.146:3300/0 conn(0x7fe72c15def0 0x7fe72c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72ad9d640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe7180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe723fff640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe718003f30 0x7fe718004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72a59c640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 0x7fe7180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72ad9d640 1 -- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 msgr2=0x7fe7180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72ad9d640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c06cab0 0x7fe7180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72ad9d640 1 -- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe718003f30 msgr2=0x7fe718004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72ad9d640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe718003f30 0x7fe718004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.413+0000 7fe72ad9d640 1 -- 172.21.15.72:0/4033976633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe718003520 con 0x7fe718004d40 2024-03-06T07:22:23.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe72ad9d640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe7180051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe72400a8a0 tx=0x7fe718005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d048e925846df53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72400a4c0 con 0x7fe718004d40 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe724034630 con 0x7fe718004d40 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe729d9b640 1 -- 172.21.15.72:0/4033976633 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe724034aa0 con 0x7fe718004d40 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe730c15640 1 -- 172.21.15.72:0/4033976633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 msgr2=0x7fe7180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe730c15640 1 --2- 172.21.15.72:0/4033976633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe7180051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe72400a8a0 tx=0x7fe718005830 comp rx=0 tx=0).stop 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe72ad9d640 1 -- 172.21.15.72:0/4033976633 reap_dead start 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe730c15640 1 -- 172.21.15.72:0/4033976633 shutdown_connections 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe730c15640 1 -- 172.21.15.72:0/4033976633 >> 172.21.15.72:0/4033976633 conn(0x7fe72c0b3690 msgr2=0x7fe72c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe730c15640 1 -- 172.21.15.72:0/4033976633 shutdown_connections 2024-03-06T07:22:23.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.414+0000 7fe730c15640 1 -- 172.21.15.72:0/4033976633 wait complete. 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.416+0000 7fe730c15640 1 Processor -- start 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.416+0000 7fe730c15640 1 -- start start 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.416+0000 7fe730c15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe72c1f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.416+0000 7fe730c15640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c0caf40 0x7fe72c0cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.416+0000 7fe730c15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c1f3a50 0x7fe72c1f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.416+0000 7fe730c15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0cc3f0 con 0x7fe718004d40 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe730c15640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c0cc760 con 0x7fe72c0caf40 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe730c15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c15e2d0 con 0x7fe72c1f3a50 2024-03-06T07:22:23.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe72c1f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe72c1f0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38448/0 (socket says 172.21.15.72:38448) 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 -- 172.21.15.72:0/1036760062 learned_addr learned my addr 172.21.15.72:0/1036760062 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe723fff640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c0caf40 0x7fe72c0cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72ad9d640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c1f3a50 0x7fe72c1f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 -- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c1f3a50 msgr2=0x7fe72c1f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c1f3a50 0x7fe72c1f3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 -- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c0caf40 msgr2=0x7fe72c0cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c0caf40 0x7fe72c0cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 -- 172.21.15.72:0/1036760062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c15e590 con 0x7fe718004d40 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe723fff640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c0caf40 0x7fe72c0cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72ad9d640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c1f3a50 0x7fe72c1f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe72a59c640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe72c1f0aa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe71c011960 tx=0x7fe71c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c00fb70 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c00fd10 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c15e870 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c015660 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.417+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c15ee30 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.419+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c09fcf0 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.419+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe71c015800 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.420+0000 7fe722ffd640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe70c06b500 0x7fe70c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.420+0000 7fe723fff640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe70c06b500 0x7fe70c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.420+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe71c08ad70 con 0x7fe718004d40 2024-03-06T07:22:23.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.420+0000 7fe723fff640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe70c06b500 0x7fe70c06d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe72c070210 tx=0x7fe714008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.424+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe71c019030 con 0x7fe718004d40 2024-03-06T07:22:23.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.737+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe72c068ff0 con 0x7fe718004d40 2024-03-06T07:22:23.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.737+0000 7fe722ffd640 1 -- 172.21.15.72:0/1036760062 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe71c059bf0 con 0x7fe718004d40 2024-03-06T07:22:23.739 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:23.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe70c06b500 msgr2=0x7fe70c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe70c06b500 0x7fe70c06d9c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe72c070210 tx=0x7fe714008040 comp rx=0 tx=0).stop 2024-03-06T07:22:23.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 msgr2=0x7fe72c1f0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe72c1f0aa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe71c011960 tx=0x7fe71c011e20 comp rx=0 tx=0).stop 2024-03-06T07:22:23.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 shutdown_connections 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe70c06b500 0x7fe70c06d9c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe72c1f3a50 0x7fe72c1f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe72c0caf40 0x7fe72c0cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 --2- 172.21.15.72:0/1036760062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe718004d40 0x7fe72c1f0aa0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 >> 172.21.15.72:0/1036760062 conn(0x7fe72c0b3690 msgr2=0x7fe72c06ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 shutdown_connections 2024-03-06T07:22:23.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.739+0000 7fe730c15640 1 -- 172.21.15.72:0/1036760062 wait complete. 2024-03-06T07:22:23.752 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:23.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.853+0000 7feb7c713640 1 Processor -- start 2024-03-06T07:22:23.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.853+0000 7feb7c713640 1 -- start start 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7c713640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7c713640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feb74070e10 0x7feb740cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7c713640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7406d370 con 0x7feb741444c0 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7c713640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb741448a0 con 0x7feb74070e10 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7c713640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb74071260 con 0x7feb7415b2d0 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7acb0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feb74070e10 0x7feb740cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7acb0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feb74070e10 0x7feb740cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38452/0 (socket says 172.21.15.72:38452) 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb7acb0640 1 -- 172.21.15.72:0/2224110978 learned_addr learned my addr 172.21.15.72:0/2224110978 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.854+0000 7feb79cae640 1 --2- 172.21.15.72:0/2224110978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.855+0000 7feb7acb0640 1 -- 172.21.15.72:0/2224110978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 msgr2=0x7feb740708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:23.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.855+0000 7feb7acb0640 1 --2- 172.21.15.72:0/2224110978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.855+0000 7feb7acb0640 1 -- 172.21.15.72:0/2224110978 >> v1:172.21.15.146:6789/0 conn(0x7feb741444c0 legacy=0x7feb7406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.855+0000 7feb7acb0640 1 -- 172.21.15.72:0/2224110978 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7406d650 con 0x7feb74070e10 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7acb0640 1 --2- 172.21.15.72:0/2224110978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb740cda60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00d150 tx=0x7feb6c02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3063659ad9ba64bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb794ad640 1 -- 172.21.15.72:0/2224110978 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c00b0d0 con 0x7feb74070e10 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb794ad640 1 -- 172.21.15.72:0/2224110978 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c00b270 con 0x7feb74070e10 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb794ad640 1 -- 172.21.15.72:0/2224110978 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c009600 con 0x7feb74070e10 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 -- 172.21.15.72:0/2224110978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 msgr2=0x7feb740cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 --2- 172.21.15.72:0/2224110978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb740cda60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00d150 tx=0x7feb6c02e160 comp rx=0 tx=0).stop 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 -- 172.21.15.72:0/2224110978 shutdown_connections 2024-03-06T07:22:23.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 --2- 172.21.15.72:0/2224110978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb740cda60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 --2- 172.21.15.72:0/2224110978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 -- 172.21.15.72:0/2224110978 >> 172.21.15.72:0/2224110978 conn(0x7feb740b1c80 msgr2=0x7feb740cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:23.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 -- 172.21.15.72:0/2224110978 shutdown_connections 2024-03-06T07:22:23.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.856+0000 7feb7c713640 1 -- 172.21.15.72:0/2224110978 wait complete. 2024-03-06T07:22:23.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 Processor -- start 2024-03-06T07:22:23.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 -- start start 2024-03-06T07:22:23.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb7413d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb741444c0 0x7feb740b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7406dc10 con 0x7feb74070e10 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7406d110 con 0x7feb741444c0 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7c713640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7406def0 con 0x7feb7415b2d0 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7a4af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb7413d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7a4af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb7413d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38456/0 (socket says 172.21.15.72:38456) 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.858+0000 7feb7a4af640 1 -- 172.21.15.72:0/1755657892 learned_addr learned my addr 172.21.15.72:0/1755657892 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb79cae640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb741444c0 0x7feb740b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7acb0640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7a4af640 1 -- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 msgr2=0x7feb740b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7a4af640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7a4af640 1 -- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb741444c0 msgr2=0x7feb740b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7a4af640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb741444c0 0x7feb740b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7a4af640 1 -- 172.21.15.72:0/1755657892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740cea00 con 0x7feb74070e10 2024-03-06T07:22:23.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7a4af640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb7413d450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7feb6400c400 tx=0x7feb6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb640052d0 con 0x7feb74070e10 2024-03-06T07:22:23.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb64005490 con 0x7feb74070e10 2024-03-06T07:22:23.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb740cece0 con 0x7feb74070e10 2024-03-06T07:22:23.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb64007260 con 0x7feb74070e10 2024-03-06T07:22:23.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.859+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb740cefc0 con 0x7feb74070e10 2024-03-06T07:22:23.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.860+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb7404f330 con 0x7feb74070e10 2024-03-06T07:22:23.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.861+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb64007400 con 0x7feb74070e10 2024-03-06T07:22:23.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.861+0000 7feb6affd640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb5806b4d0 0x7feb5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:23.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.861+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb64089610 con 0x7feb74070e10 2024-03-06T07:22:23.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.861+0000 7feb79cae640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb5806b4d0 0x7feb5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:23.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.862+0000 7feb79cae640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb5806b4d0 0x7feb5806d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb740b15b0 tx=0x7feb70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:23.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:23.865+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb64057ed0 con 0x7feb74070e10 2024-03-06T07:22:24.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.097+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7feb7415af40 con 0x7feb74070e10 2024-03-06T07:22:24.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.098+0000 7feb6affd640 1 -- 172.21.15.72:0/1755657892 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7feb6405bf70 con 0x7feb74070e10 2024-03-06T07:22:24.100 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb5806b4d0 msgr2=0x7feb5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb5806b4d0 0x7feb5806d990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7feb740b15b0 tx=0x7feb70008040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 msgr2=0x7feb7413d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb7413d450 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7feb6400c400 tx=0x7feb6400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 shutdown_connections 2024-03-06T07:22:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb5806b4d0 0x7feb5806d990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7415b2d0 0x7feb740b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb741444c0 0x7feb740b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 --2- 172.21.15.72:0/1755657892 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb74070e10 0x7feb7413d450 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 >> 172.21.15.72:0/1755657892 conn(0x7feb740b1c80 msgr2=0x7feb74142b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 shutdown_connections 2024-03-06T07:22:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.100+0000 7feb7c713640 1 -- 172.21.15.72:0/1755657892 wait complete. 2024-03-06T07:22:24.113 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:22:24.113 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:24.132 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:24.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.231+0000 7fe3cc7f7640 1 Processor -- start 2024-03-06T07:22:24.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.231+0000 7fe3cc7f7640 1 -- start start 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.231+0000 7fe3cc7f7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3c40d4760 0x7fe3c40b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.231+0000 7fe3cc7f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe3c40d0a80 0x7fe3c40cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3cc7f7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40d25d0 con 0x7fe3c40b4170 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3cc7f7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c40d4bf0 con 0x7fe3c40d4760 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3cc7f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c40ce9e0 con 0x7fe3c40d0a80 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3c40d4760 0x7fe3c40b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3c40d4760 0x7fe3c40b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38470/0 (socket says 172.21.15.72:38470) 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 -- 172.21.15.72:0/1317372462 learned_addr learned my addr 172.21.15.72:0/1317372462 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3cad94640 1 --2- 172.21.15.72:0/1317372462 >> v2:172.21.15.146:3300/0 conn(0x7fe3c40d0a80 0x7fe3c40cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 -- 172.21.15.72:0/1317372462 >> v2:172.21.15.146:3300/0 conn(0x7fe3c40d0a80 msgr2=0x7fe3c40cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 --2- 172.21.15.72:0/1317372462 >> v2:172.21.15.146:3300/0 conn(0x7fe3c40d0a80 0x7fe3c40cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 -- 172.21.15.72:0/1317372462 >> v1:172.21.15.146:6789/0 conn(0x7fe3c40b4170 legacy=0x7fe3c40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.232+0000 7fe3c9d92640 1 -- 172.21.15.72:0/1317372462 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40d2870 con 0x7fe3c40d4760 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.233+0000 7fe3c9d92640 1 --2- 172.21.15.72:0/1317372462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40d4760 0x7fe3c40b6a90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe3c408df30 tx=0x7fe3b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27d1362b91329737 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.233+0000 7fe3c9591640 1 -- 172.21.15.72:0/1317372462 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8034070 con 0x7fe3c40d4760 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.233+0000 7fe3c9591640 1 -- 172.21.15.72:0/1317372462 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b8030860 con 0x7fe3c40d4760 2024-03-06T07:22:24.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.233+0000 7fe3c9591640 1 -- 172.21.15.72:0/1317372462 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8030c30 con 0x7fe3c40d4760 2024-03-06T07:22:24.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.233+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1317372462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40d4760 msgr2=0x7fe3c40b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.233+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1317372462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40d4760 0x7fe3c40b6a90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe3c408df30 tx=0x7fe3b802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.234+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1317372462 shutdown_connections 2024-03-06T07:22:24.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.234+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1317372462 >> v2:172.21.15.146:3300/0 conn(0x7fe3c40d0a80 0x7fe3c40cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.234+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1317372462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40d4760 0x7fe3c40b6a90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.234+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1317372462 >> 172.21.15.72:0/1317372462 conn(0x7fe3c40d3c40 msgr2=0x7fe3c40d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.234+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1317372462 shutdown_connections 2024-03-06T07:22:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.234+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1317372462 wait complete. 2024-03-06T07:22:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.235+0000 7fe3cc7f7640 1 Processor -- start 2024-03-06T07:22:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.235+0000 7fe3cc7f7640 1 -- start start 2024-03-06T07:22:24.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 0x7fe3c4156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3c40d0a80 0x7fe3c4158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3c40cb820 0x7fe3c41f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40b4dd0 con 0x7fe3c40b4170 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40b4550 con 0x7fe3c40cb820 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40b4f50 con 0x7fe3c40d0a80 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 0x7fe3c4156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 0x7fe3c4156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38472/0 (socket says 172.21.15.72:38472) 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 -- 172.21.15.72:0/1717828580 learned_addr learned my addr 172.21.15.72:0/1717828580 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3c9d92640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3c40d0a80 0x7fe3c4158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cad94640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3c40cb820 0x7fe3c41f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 -- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3c40d0a80 msgr2=0x7fe3c4158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3c40d0a80 0x7fe3c4158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 -- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3c40cb820 msgr2=0x7fe3c41f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3c40cb820 0x7fe3c41f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 -- 172.21.15.72:0/1717828580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4158fe0 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cad94640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3c40cb820 0x7fe3c41f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3c9d92640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3c40d0a80 0x7fe3c4158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3ca593640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 0x7fe3c4156cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400c7a0 tx=0x7fe3b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4003b90 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b4003d50 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c415c510 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400aa90 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.237+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c415ca90 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.239+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c404f330 con 0x7fe3c40b4170 2024-03-06T07:22:24.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.239+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe3b400ac30 con 0x7fe3c40b4170 2024-03-06T07:22:24.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.239+0000 7fe3c2ffd640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe3b006b500 0x7fe3b006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.239+0000 7fe3c9d92640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe3b006b500 0x7fe3b006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.239+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b408a090 con 0x7fe3c40b4170 2024-03-06T07:22:24.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.240+0000 7fe3c9d92640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe3b006b500 0x7fe3b006d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8036000 tx=0x7fe3b803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.244+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe3b4058fa0 con 0x7fe3c40b4170 2024-03-06T07:22:24.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.440+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c40733b0 con 0x7fe3c40b4170 2024-03-06T07:22:24.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.441+0000 7fe3c2ffd640 1 -- 172.21.15.72:0/1717828580 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fe3b40588a0 con 0x7fe3c40b4170 2024-03-06T07:22:24.444 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:24.444 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe3b006b500 msgr2=0x7fe3b006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe3b006b500 0x7fe3b006d9c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8036000 tx=0x7fe3b803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 msgr2=0x7fe3c4156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 0x7fe3c4156cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400c7a0 tx=0x7fe3b400cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 shutdown_connections 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe3b006b500 0x7fe3b006d9c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3c40cb820 0x7fe3c41f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3c40d0a80 0x7fe3c4158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 --2- 172.21.15.72:0/1717828580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c40b4170 0x7fe3c4156cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.443+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 >> 172.21.15.72:0/1717828580 conn(0x7fe3c40d3c40 msgr2=0x7fe3c40b7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.444+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 shutdown_connections 2024-03-06T07:22:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.444+0000 7fe3cc7f7640 1 -- 172.21.15.72:0/1717828580 wait complete. 2024-03-06T07:22:24.460 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:24.460 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:24.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.562+0000 7f2c82037640 1 Processor -- start 2024-03-06T07:22:24.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.562+0000 7f2c82037640 1 -- start start 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c82037640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15c050 0x7f2c7c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c82037640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c7c06fb40 0x7f2c7c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c82037640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0cd4b0 con 0x7f2c7c0ccca0 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c82037640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c06ffb0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c82037640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c0cd790 con 0x7f2c7c15c050 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c7affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c7c06fb40 0x7f2c7c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c7affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c7c06fb40 0x7f2c7c0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38486/0 (socket says 172.21.15.72:38486) 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c7affd640 1 -- 172.21.15.72:0/2197238489 learned_addr learned my addr 172.21.15.72:0/2197238489 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.563+0000 7f2c7b7fe640 1 --2- 172.21.15.72:0/2197238489 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15c050 0x7f2c7c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7affd640 1 -- 172.21.15.72:0/2197238489 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15c050 msgr2=0x7f2c7c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7affd640 1 --2- 172.21.15.72:0/2197238489 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15c050 0x7f2c7c06f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7affd640 1 -- 172.21.15.72:0/2197238489 >> v1:172.21.15.146:6789/0 conn(0x7f2c7c0ccca0 legacy=0x7f2c7c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7affd640 1 -- 172.21.15.72:0/2197238489 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c0cda70 con 0x7f2c7c06fb40 2024-03-06T07:22:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7affd640 1 --2- 172.21.15.72:0/2197238489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0cc760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c070ef0 tx=0x7f2c6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9efb9fd037865264 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7a7fc640 1 -- 172.21.15.72:0/2197238489 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c035020 con 0x7f2c7c06fb40 2024-03-06T07:22:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7a7fc640 1 -- 172.21.15.72:0/2197238489 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c031870 con 0x7f2c7c06fb40 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.564+0000 7f2c7a7fc640 1 -- 172.21.15.72:0/2197238489 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c031c40 con 0x7f2c7c06fb40 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 -- 172.21.15.72:0/2197238489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 msgr2=0x7f2c7c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 --2- 172.21.15.72:0/2197238489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0cc760 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c070ef0 tx=0x7f2c6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 -- 172.21.15.72:0/2197238489 shutdown_connections 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 --2- 172.21.15.72:0/2197238489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0cc760 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 --2- 172.21.15.72:0/2197238489 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15c050 0x7f2c7c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 -- 172.21.15.72:0/2197238489 >> 172.21.15.72:0/2197238489 conn(0x7f2c7c0b3800 msgr2=0x7f2c7c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 -- 172.21.15.72:0/2197238489 shutdown_connections 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.565+0000 7f2c82037640 1 -- 172.21.15.72:0/2197238489 wait complete. 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.566+0000 7f2c82037640 1 Processor -- start 2024-03-06T07:22:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.566+0000 7f2c82037640 1 -- start start 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c82037640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c82037640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0ccca0 0x7f2c7c0b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c82037640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c15c050 0x7f2c7c1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c82037640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0d01f0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c82037640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0cd130 con 0x7f2c7c0ccca0 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c82037640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0d0370 con 0x7f2c7c15c050 2024-03-06T07:22:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0b7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38494/0 (socket says 172.21.15.72:38494) 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 -- 172.21.15.72:0/4099465392 learned_addr learned my addr 172.21.15.72:0/4099465392 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7affd640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0ccca0 0x7f2c7c0b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7bfff640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c15c050 0x7f2c7c1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 -- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c15c050 msgr2=0x7f2c7c1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c15c050 0x7f2c7c1592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 -- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0ccca0 msgr2=0x7f2c7c0b5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.567+0000 7f2c7b7fe640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0ccca0 0x7f2c7c0b5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c7b7fe640 1 -- 172.21.15.72:0/4099465392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c0b4cd0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c7affd640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0ccca0 0x7f2c7c0b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c7b7fe640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0b7410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000d800 tx=0x7f2c7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000b6c0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c70019070 con 0x7f2c7c06fb40 2024-03-06T07:22:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c0b4fb0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000ba80 con 0x7f2c7c06fb40 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.568+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c0b5570 con 0x7f2c7c06fb40 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.569+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c09fe60 con 0x7f2c7c06fb40 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.569+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c7000bc20 con 0x7f2c7c06fb40 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.570+0000 7f2c63fff640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.570+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c7008a660 con 0x7f2c7c06fb40 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.570+0000 7f2c7affd640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.570+0000 7f2c7affd640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c036000 tx=0x7f2c6c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.574+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c700594e0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.748+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c7c068ff0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.749+0000 7f2c63fff640 1 -- 172.21.15.72:0/4099465392 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2c70058de0 con 0x7f2c7c06fb40 2024-03-06T07:22:24.751 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 msgr2=0x7f2c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c036000 tx=0x7f2c6c008030 comp rx=0 tx=0).stop 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 msgr2=0x7f2c7c0b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0b7410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000d800 tx=0x7f2c7000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 shutdown_connections 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c15c050 0x7f2c7c1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0ccca0 0x7f2c7c0b5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 --2- 172.21.15.72:0/4099465392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c06fb40 0x7f2c7c0b7410 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.750+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 >> 172.21.15.72:0/4099465392 conn(0x7f2c7c0b3800 msgr2=0x7f2c7c0d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.751+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 shutdown_connections 2024-03-06T07:22:24.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.751+0000 7f2c82037640 1 -- 172.21.15.72:0/4099465392 wait complete. 2024-03-06T07:22:24.764 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:24.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.868+0000 7f2a0304e640 1 Processor -- start 2024-03-06T07:22:24.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.868+0000 7f2a0304e640 1 -- start start 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f2a0304e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f29fc15fc30 0x7f29fc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f2a0304e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc1600a0 con 0x7f29fc0d06e0 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f2a0304e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29fc0d47f0 con 0x7f29fc0b4d10 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f2a0304e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29fc0ced70 con 0x7f29fc15fc30 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f2a00dea640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f29fc0d06e0 0x7f29fc0d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39268/0 (socket says 172.21.15.72:39268) 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f2a00dea640 1 -- 172.21.15.72:0/3158416108 learned_addr learned my addr 172.21.15.72:0/3158416108 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977139039 0 0) 0x7f29fc1600a0 con 0x7f29fc0d06e0 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003520 con 0x7f29fc0d06e0 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.869+0000 7f29fbfff640 1 --2- 172.21.15.72:0/3158416108 >> v2:172.21.15.146:3300/0 conn(0x7f29fc15fc30 0x7f29fc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 446303675 0 0) 0x7f29e8003520 con 0x7f29fc0d06e0 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 >> v2:172.21.15.146:3300/0 conn(0x7f29fc15fc30 msgr2=0x7f29fc0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 --2- 172.21.15.72:0/3158416108 >> v2:172.21.15.146:3300/0 conn(0x7f29fc15fc30 0x7f29fc0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 >> v1:172.21.15.146:6789/0 conn(0x7f29fc0b4d10 legacy=0x7f29fc0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc0d1ab0 con 0x7f29fc0d06e0 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f29f40019f0 con 0x7f29fc0d06e0 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 >> v1:172.21.15.72:6789/0 conn(0x7f29fc0d06e0 legacy=0x7f29fc0d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29e80051c0 0x7f29e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0ced70 con 0x7f29e80051c0 2024-03-06T07:22:24.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800c110 con 0x7f29e8004800 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800c310 con 0x7f29e80039b0 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29e80051c0 0x7f29e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f29fbfff640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a00dea640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a015eb640 1 -- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80039b0 msgr2=0x7f29e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80039b0 0x7f29e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a015eb640 1 -- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29e8004800 msgr2=0x7f29e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.870+0000 7f2a015eb640 1 -- 172.21.15.72:0/3158416108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8003520 con 0x7f29e80051c0 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.871+0000 7f2a015eb640 1 -- 172.21.15.72:0/3158416108 reap_dead start 2024-03-06T07:22:24.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.871+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29e80051c0 0x7f29e800b9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f29ec001750 tx=0x7f29e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de670080ff0b9ce8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.871+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec001370 con 0x7f29e80051c0 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.871+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec005310 con 0x7f29e80051c0 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.871+0000 7f29fb7fe640 1 -- 172.21.15.72:0/3158416108 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec003830 con 0x7f29e80051c0 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 -- 172.21.15.72:0/3158416108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29e80051c0 msgr2=0x7f29e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29e80051c0 0x7f29e800b9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f29ec001750 tx=0x7f29e8005830 comp rx=0 tx=0).stop 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 -- 172.21.15.72:0/3158416108 shutdown_connections 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3158416108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29e80051c0 0x7f29e800b9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 -- 172.21.15.72:0/3158416108 >> 172.21.15.72:0/3158416108 conn(0x7f29fc0d3cd0 msgr2=0x7f29fc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 -- 172.21.15.72:0/3158416108 shutdown_connections 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.872+0000 7f2a0304e640 1 -- 172.21.15.72:0/3158416108 wait complete. 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.873+0000 7f2a0304e640 1 Processor -- start 2024-03-06T07:22:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.873+0000 7f2a0304e640 1 -- start start 2024-03-06T07:22:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80051c0 0x7f29fc0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29fc0c93b0 0x7f29fc14e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 0x7f29fc0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0ce990 con 0x7f29fc0b6850 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0d4ab0 con 0x7f29fc0c93b0 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29fc0b44d0 con 0x7f29e80051c0 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 0x7f29fc0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 0x7f29fc0b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38518/0 (socket says 172.21.15.72:38518) 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 -- 172.21.15.72:0/3435252975 learned_addr learned my addr 172.21.15.72:0/3435252975 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f29fbfff640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29fc0c93b0 0x7f29fc14e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a00dea640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80051c0 0x7f29fc0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 -- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80051c0 msgr2=0x7f29fc0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80051c0 0x7f29fc0cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 -- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29fc0c93b0 msgr2=0x7f29fc14e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29fc0c93b0 0x7f29fc14e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 -- 172.21.15.72:0/3435252975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29fc0cb9c0 con 0x7f29fc0b6850 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a00dea640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80051c0 0x7f29fc0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f29fbfff640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29fc0c93b0 0x7f29fc14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a015eb640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 0x7f29fc0b6cd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29ec02bd00 tx=0x7f29ec001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec001370 con 0x7f29fc0b6850 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ec007240 con 0x7f29fc0b6850 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec007a30 con 0x7f29fc0b6850 2024-03-06T07:22:24.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29fc0cbca0 con 0x7f29fc0b6850 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.874+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29fc0cc260 con 0x7f29fc0b6850 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.876+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29fc0a1c80 con 0x7f29fc0b6850 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.876+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29ec0067d0 con 0x7f29fc0b6850 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.876+0000 7f29f97fa640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f29dc06b4d0 0x7f29dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.876+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29ec03a020 con 0x7f29fc0b6850 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.876+0000 7f2a00dea640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f29dc06b4d0 0x7f29dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.877+0000 7f2a00dea640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f29dc06b4d0 0x7f29dc06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f29f4002230 tx=0x7f29f4006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:24.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:24.880+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29ec079a00 con 0x7f29fc0b6850 2024-03-06T07:22:25.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.057+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f29fc068ff0 con 0x7f29fc0b6850 2024-03-06T07:22:25.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.058+0000 7f29f97fa640 1 -- 172.21.15.72:0/3435252975 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29ec079300 con 0x7f29fc0b6850 2024-03-06T07:22:25.060 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.059+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f29dc06b4d0 msgr2=0x7f29dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.059+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f29dc06b4d0 0x7f29dc06d990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f29f4002230 tx=0x7f29f4006040 comp rx=0 tx=0).stop 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.059+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 msgr2=0x7f29fc0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.059+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 0x7f29fc0b6cd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f29ec02bd00 tx=0x7f29ec001120 comp rx=0 tx=0).stop 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 shutdown_connections 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f29dc06b4d0 0x7f29dc06d990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29fc0b6850 0x7f29fc0b6cd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29fc0c93b0 0x7f29fc14e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 --2- 172.21.15.72:0/3435252975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29e80051c0 0x7f29fc0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 >> 172.21.15.72:0/3435252975 conn(0x7f29fc0d3cd0 msgr2=0x7f29fc0cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 shutdown_connections 2024-03-06T07:22:25.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.060+0000 7f2a0304e640 1 -- 172.21.15.72:0/3435252975 wait complete. 2024-03-06T07:22:25.073 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:25.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.175+0000 7f9cccd1e640 1 Processor -- start 2024-03-06T07:22:25.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.175+0000 7f9cccd1e640 1 -- start start 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cccd1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9cc815bb40 0x7f9cc80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cccd1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9cc80cfa90 0x7f9cc80cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cccd1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc815ff90 con 0x7f9cc815f7d0 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cccd1e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc80d0440 con 0x7f9cc815bb40 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cccd1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc806c910 con 0x7f9cc80cfa90 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc659c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9cc815bb40 0x7f9cc80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cbdd9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9cc815f7d0 0x7f9cc806d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39274/0 (socket says 172.21.15.72:39274) 2024-03-06T07:22:25.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cbdd9b640 1 -- 172.21.15.72:0/2319065948 learned_addr learned my addr 172.21.15.72:0/2319065948 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc6d9d640 1 --2- 172.21.15.72:0/2319065948 >> v2:172.21.15.146:3300/0 conn(0x7f9cc80cfa90 0x7f9cc80cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc659c640 1 -- 172.21.15.72:0/2319065948 >> v2:172.21.15.146:3300/0 conn(0x7f9cc80cfa90 msgr2=0x7f9cc80cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc659c640 1 --2- 172.21.15.72:0/2319065948 >> v2:172.21.15.146:3300/0 conn(0x7f9cc80cfa90 0x7f9cc80cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc659c640 1 -- 172.21.15.72:0/2319065948 >> v1:172.21.15.72:6789/0 conn(0x7f9cc815f7d0 legacy=0x7f9cc806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc659c640 1 -- 172.21.15.72:0/2319065948 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc8160270 con 0x7f9cc815bb40 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.176+0000 7f9cc5d9b640 1 -- 172.21.15.72:0/2319065948 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043339060 0 0) 0x7f9cc815ff90 con 0x7f9cc815f7d0 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.177+0000 7f9cc659c640 1 --2- 172.21.15.72:0/2319065948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80b4970 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8072c60 tx=0x7f9cb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08d8b78f74d04dc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.177+0000 7f9cc5d9b640 1 -- 172.21.15.72:0/2319065948 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8035020 con 0x7f9cc815bb40 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.177+0000 7f9cc5d9b640 1 -- 172.21.15.72:0/2319065948 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb8031830 con 0x7f9cc815bb40 2024-03-06T07:22:25.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.177+0000 7f9cc5d9b640 1 -- 172.21.15.72:0/2319065948 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8031c00 con 0x7f9cc815bb40 2024-03-06T07:22:25.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 -- 172.21.15.72:0/2319065948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 msgr2=0x7f9cc80b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/2319065948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80b4970 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8072c60 tx=0x7f9cb802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 -- 172.21.15.72:0/2319065948 shutdown_connections 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/2319065948 >> v2:172.21.15.146:3300/0 conn(0x7f9cc80cfa90 0x7f9cc80cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/2319065948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80b4970 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 -- 172.21.15.72:0/2319065948 >> 172.21.15.72:0/2319065948 conn(0x7f9cc806b190 msgr2=0x7f9cc806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 -- 172.21.15.72:0/2319065948 shutdown_connections 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.178+0000 7f9cccd1e640 1 -- 172.21.15.72:0/2319065948 wait complete. 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.179+0000 7f9cccd1e640 1 Processor -- start 2024-03-06T07:22:25.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.179+0000 7f9cccd1e640 1 -- start start 2024-03-06T07:22:25.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cccd1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80cfa90 0x7f9cc81493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cccd1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cccd1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cc815f7d0 0x7f9cc81550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cccd1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc815fc30 con 0x7f9cc815bb40 2024-03-06T07:22:25.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cccd1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc806cb60 con 0x7f9cc80cfa90 2024-03-06T07:22:25.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cccd1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc806d790 con 0x7f9cc815f7d0 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80a9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38532/0 (socket says 172.21.15.72:38532) 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 -- 172.21.15.72:0/636596275 learned_addr learned my addr 172.21.15.72:0/636596275 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cc659c640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80cfa90 0x7f9cc81493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cc6d9d640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cc815f7d0 0x7f9cc81550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 -- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cc815f7d0 msgr2=0x7f9cc81550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cc815f7d0 0x7f9cc81550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 -- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80cfa90 msgr2=0x7f9cc81493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80cfa90 0x7f9cc81493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cbdd9b640 1 -- 172.21.15.72:0/636596275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc81498f0 con 0x7f9cc815bb40 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.180+0000 7f9cc659c640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80cfa90 0x7f9cc81493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.181+0000 7f9cbdd9b640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80a9f10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000c4d0 tx=0x7f9cb000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.181+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0018070 con 0x7f9cc815bb40 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.181+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb0007500 con 0x7f9cc815bb40 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.181+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc80aa880 con 0x7f9cc815bb40 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.181+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0004a20 con 0x7f9cc815bb40 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.181+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc8159bf0 con 0x7f9cc815bb40 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.182+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc8068ff0 con 0x7f9cc815bb40 2024-03-06T07:22:25.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.183+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9cb001b070 con 0x7f9cc815bb40 2024-03-06T07:22:25.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.184+0000 7f9cbf7fe640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ca806b4d0 0x7f9ca806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.184+0000 7f9cc659c640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ca806b4d0 0x7f9ca806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.184+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cb0089830 con 0x7f9cc815bb40 2024-03-06T07:22:25.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.184+0000 7f9cc659c640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ca806b4d0 0x7f9ca806d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8036000 tx=0x7f9cb8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.187+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9cb0058790 con 0x7f9cc815bb40 2024-03-06T07:22:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.364+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9cc806e130 con 0x7f9cc815bb40 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.365+0000 7f9cbf7fe640 1 -- 172.21.15.72:0/636596275 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cb0058090 con 0x7f9cc815bb40 2024-03-06T07:22:25.367 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ca806b4d0 msgr2=0x7f9ca806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ca806b4d0 0x7f9ca806d990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8036000 tx=0x7f9cb8008030 comp rx=0 tx=0).stop 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 msgr2=0x7f9cc80a9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80a9f10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000c4d0 tx=0x7f9cb000c990 comp rx=0 tx=0).stop 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 shutdown_connections 2024-03-06T07:22:25.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ca806b4d0 0x7f9ca806d990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cc815f7d0 0x7f9cc81550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9cc815bb40 0x7f9cc80a9f10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 --2- 172.21.15.72:0/636596275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cc80cfa90 0x7f9cc81493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 >> 172.21.15.72:0/636596275 conn(0x7f9cc806b190 msgr2=0x7f9cc80af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 shutdown_connections 2024-03-06T07:22:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.367+0000 7f9cccd1e640 1 -- 172.21.15.72:0/636596275 wait complete. 2024-03-06T07:22:25.380 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:25.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 Processor -- start 2024-03-06T07:22:25.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 -- start start 2024-03-06T07:22:25.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ec80b4c00 0x7f4ec80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec80b28a0 0x7f4ec80b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec815d510 con 0x7f4ec815e540 2024-03-06T07:22:25.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec815e0b0 con 0x7f4ec80b4c00 2024-03-06T07:22:25.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.480+0000 7f4ecf21c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec815d780 con 0x7f4ec80b28a0 2024-03-06T07:22:25.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ecd7b9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4ec815e540 0x7f4ec815e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39286/0 (socket says 172.21.15.72:39286) 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ecd7b9640 1 -- 172.21.15.72:0/1397503914 learned_addr learned my addr 172.21.15.72:0/1397503914 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1397503914 >> v2:172.21.15.146:3300/0 conn(0x7f4ec80b4c00 0x7f4ec80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ebffff640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec80b28a0 0x7f4ec80b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ebf7fe640 1 -- 172.21.15.72:0/1397503914 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768658781 0 0) 0x7f4ec815d510 con 0x7f4ec815e540 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ebf7fe640 1 -- 172.21.15.72:0/1397503914 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0003520 con 0x7f4ec815e540 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4eccfb8640 1 -- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec80b28a0 msgr2=0x7f4ec80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec80b28a0 0x7f4ec80b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4eccfb8640 1 -- 172.21.15.72:0/1397503914 >> v1:172.21.15.72:6789/0 conn(0x7f4ec815e540 legacy=0x7f4ec815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ebf7fe640 1 -- 172.21.15.72:0/1397503914 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 427547833 0 0) 0x7f4eb0003520 con 0x7f4ec815e540 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4eccfb8640 1 -- 172.21.15.72:0/1397503914 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec815da60 con 0x7f4ec80b4c00 2024-03-06T07:22:25.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.481+0000 7f4ebffff640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec80b28a0 0x7f4ec80b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:25.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.482+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80b4fe0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40097a0 tx=0x7f4ec402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4700f4a78379be59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.482+0000 7f4ebf7fe640 1 -- 172.21.15.72:0/1397503914 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4038070 con 0x7f4ec80b4c00 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.482+0000 7f4ebf7fe640 1 -- 172.21.15.72:0/1397503914 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec4031500 con 0x7f4ec80b4c00 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.482+0000 7f4ebf7fe640 1 -- 172.21.15.72:0/1397503914 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00019f0 con 0x7f4ec80b4c00 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b4c00 msgr2=0x7f4ec80b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80b4fe0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40097a0 tx=0x7f4ec402b820 comp rx=0 tx=0).stop 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1397503914 shutdown_connections 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec80b28a0 0x7f4ec80b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1397503914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80b4fe0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1397503914 >> 172.21.15.72:0/1397503914 conn(0x7f4ec809a9f0 msgr2=0x7f4ec809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1397503914 shutdown_connections 2024-03-06T07:22:25.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.483+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1397503914 wait complete. 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.484+0000 7f4ecf21c640 1 Processor -- start 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.484+0000 7f4ecf21c640 1 -- start start 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecf21c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b28a0 0x7f4ec81eda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecf21c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80d1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecf21c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec815e540 0x7f4ec80d1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecf21c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80b33a0 con 0x7f4ec80b4c00 2024-03-06T07:22:25.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecf21c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80b3220 con 0x7f4ec80b28a0 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecf21c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec815f090 con 0x7f4ec815e540 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38542/0 (socket says 172.21.15.72:38542) 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 -- 172.21.15.72:0/1445595955 learned_addr learned my addr 172.21.15.72:0/1445595955 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecd7b9640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec815e540 0x7f4ec80d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b28a0 0x7f4ec81eda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 -- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec815e540 msgr2=0x7f4ec80d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec815e540 0x7f4ec80d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 -- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b28a0 msgr2=0x7f4ec81eda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b28a0 0x7f4ec81eda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ebffff640 1 -- 172.21.15.72:0/1445595955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec81f4b20 con 0x7f4ec80b4c00 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b28a0 0x7f4ec81eda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.485+0000 7f4ecd7b9640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec815e540 0x7f4ec80d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:25.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4ebffff640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80d1110 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800ebe0 tx=0x7f4eb800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8017070 con 0x7f4ec80b4c00 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb800cc30 con 0x7f4ec80b4c00 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8013470 con 0x7f4ec80b4c00 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec81f3d60 con 0x7f4ec80b4c00 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.486+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec81f4320 con 0x7f4ec80b4c00 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.487+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec804f330 con 0x7f4ec80b4c00 2024-03-06T07:22:25.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.487+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4eb801a070 con 0x7f4ec80b4c00 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.488+0000 7f4ebd7fa640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eb406b4d0 0x7f4eb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.488+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4eb808a100 con 0x7f4ec80b4c00 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.488+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eb406b4d0 0x7f4eb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:25.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.488+0000 7f4eccfb8640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eb406b4d0 0x7f4eb406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4035000 tx=0x7f4ec4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:25.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.492+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4eb8059010 con 0x7f4ec80b4c00 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.675+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4ec815c260 con 0x7f4ec80b4c00 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.675+0000 7f4ebd7fa640 1 -- 172.21.15.72:0/1445595955 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4eb8058910 con 0x7f4ec80b4c00 2024-03-06T07:22:25.677 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:25.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eb406b4d0 msgr2=0x7f4eb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eb406b4d0 0x7f4eb406d990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4035000 tx=0x7f4ec4008030 comp rx=0 tx=0).stop 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 msgr2=0x7f4ec80d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80d1110 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800ebe0 tx=0x7f4eb800c520 comp rx=0 tx=0).stop 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 shutdown_connections 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eb406b4d0 0x7f4eb406d990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ec815e540 0x7f4ec80d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ec80b4c00 0x7f4ec80d1110 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 --2- 172.21.15.72:0/1445595955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ec80b28a0 0x7f4ec81eda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.677+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 >> 172.21.15.72:0/1445595955 conn(0x7f4ec809a9f0 msgr2=0x7f4ec809afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.678+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 shutdown_connections 2024-03-06T07:22:25.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:25.678+0000 7f4ecf21c640 1 -- 172.21.15.72:0/1445595955 wait complete. 2024-03-06T07:22:25.690 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-03-06T07:22:25.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:25.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-06T07:22:25.723 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:25.724 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:22:25.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:25.724 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:22:25.789 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:22:25.789 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:25.805 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:22:25.805 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:25.860 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.001+0000 7fa5b884e640 1 Processor -- start 2024-03-06T07:22:26.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.001+0000 7fa5b884e640 1 -- start start 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b884e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5b415dfe0 0x7fa5b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b884e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b40d29c0 0x7fa5b40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b884e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b40b3430 con 0x7fa5b40cf830 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b884e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b40b31e0 con 0x7fa5b415dfe0 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b884e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40b3d60 con 0x7fa5b40d29c0 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5b415dfe0 0x7fa5b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5b415dfe0 0x7fa5b40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38548/0 (socket says 172.21.15.72:38548) 2024-03-06T07:22:26.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.002+0000 7fa5b259c640 1 -- 172.21.15.72:0/1936148502 learned_addr learned my addr 172.21.15.72:0/1936148502 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.003+0000 7fa5b259c640 1 -- 172.21.15.72:0/1936148502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b40d29c0 msgr2=0x7fa5b40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.003+0000 7fa5b259c640 1 --2- 172.21.15.72:0/1936148502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b40d29c0 0x7fa5b40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.003+0000 7fa5b259c640 1 -- 172.21.15.72:0/1936148502 >> v1:172.21.15.146:6789/0 conn(0x7fa5b40cf830 legacy=0x7fa5b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.003+0000 7fa5b259c640 1 -- 172.21.15.72:0/1936148502 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b40b3710 con 0x7fa5b415dfe0 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.003+0000 7fa5b259c640 1 --2- 172.21.15.72:0/1936148502 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b415dfe0 0x7fa5b40cf2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009870 tx=0x7fa5a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e930324711bed3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.003+0000 7fa5b159a640 1 -- 172.21.15.72:0/1936148502 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4035020 con 0x7fa5b415dfe0 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b159a640 1 -- 172.21.15.72:0/1936148502 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a4031640 con 0x7fa5b415dfe0 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b159a640 1 -- 172.21.15.72:0/1936148502 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a402d040 con 0x7fa5b415dfe0 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 -- 172.21.15.72:0/1936148502 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b415dfe0 msgr2=0x7fa5b40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 --2- 172.21.15.72:0/1936148502 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b415dfe0 0x7fa5b40cf2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009870 tx=0x7fa5a402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 -- 172.21.15.72:0/1936148502 shutdown_connections 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 --2- 172.21.15.72:0/1936148502 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b40d29c0 0x7fa5b40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 --2- 172.21.15.72:0/1936148502 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b415dfe0 0x7fa5b40cf2f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 -- 172.21.15.72:0/1936148502 >> 172.21.15.72:0/1936148502 conn(0x7fa5b40d3cd0 msgr2=0x7fa5b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 -- 172.21.15.72:0/1936148502 shutdown_connections 2024-03-06T07:22:26.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.004+0000 7fa5b884e640 1 -- 172.21.15.72:0/1936148502 wait complete. 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.005+0000 7fa5b884e640 1 Processor -- start 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.005+0000 7fa5b884e640 1 -- start start 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b884e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 0x7fa5b41f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b884e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b40d29c0 0x7fa5b40cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b884e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b415dfe0 0x7fa5b4160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b884e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40d1df0 con 0x7fa5b40cf830 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b884e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b40d1c70 con 0x7fa5b40d29c0 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b884e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b415e3c0 con 0x7fa5b415dfe0 2024-03-06T07:22:26.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 0x7fa5b41f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 0x7fa5b41f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38552/0 (socket says 172.21.15.72:38552) 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b259c640 1 -- 172.21.15.72:0/771982556 learned_addr learned my addr 172.21.15.72:0/771982556 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b1d9b640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b40d29c0 0x7fa5b40cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b2d9d640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b415dfe0 0x7fa5b4160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.006+0000 7fa5b259c640 1 -- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b415dfe0 msgr2=0x7fa5b4160ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b259c640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b415dfe0 0x7fa5b4160ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b259c640 1 -- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b40d29c0 msgr2=0x7fa5b40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b259c640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b40d29c0 0x7fa5b40cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b259c640 1 -- 172.21.15.72:0/771982556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b415fc80 con 0x7fa5b40cf830 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b1d9b640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b40d29c0 0x7fa5b40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b2d9d640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b415dfe0 0x7fa5b4160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b259c640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 0x7fa5b41f3040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa5a402b930 tx=0x7fa5a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a403ac60 con 0x7fa5b40cf830 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a403ae20 con 0x7fa5b40cf830 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b40cb780 con 0x7fa5b40cf830 2024-03-06T07:22:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4044630 con 0x7fa5b40cf830 2024-03-06T07:22:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.007+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b40cbd40 con 0x7fa5b40cf830 2024-03-06T07:22:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.008+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b404f330 con 0x7fa5b40cf830 2024-03-06T07:22:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.008+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa5a404b060 con 0x7fa5b40cf830 2024-03-06T07:22:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.009+0000 7fa5a2ffd640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa59406b4d0 0x7fa59406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.009+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5a40abf40 con 0x7fa5b40cf830 2024-03-06T07:22:26.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.009+0000 7fa5b1d9b640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa59406b4d0 0x7fa59406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.009+0000 7fa5b1d9b640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa59406b4d0 0x7fa59406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa5b406d510 tx=0x7fa5a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.013+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa5a403f070 con 0x7fa5b40cf830 2024-03-06T07:22:26.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.189+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5b40747a0 con 0x7fa5b40cf830 2024-03-06T07:22:26.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.190+0000 7fa5a2ffd640 1 -- 172.21.15.72:0/771982556 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fa5a402f300 con 0x7fa5b40cf830 2024-03-06T07:22:26.192 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:26.193 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:26.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa59406b4d0 msgr2=0x7fa59406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa59406b4d0 0x7fa59406d990 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa5b406d510 tx=0x7fa5a8009040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 msgr2=0x7fa5b41f3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 0x7fa5b41f3040 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa5a402b930 tx=0x7fa5a40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 shutdown_connections 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa59406b4d0 0x7fa59406d990 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5b415dfe0 0x7fa5b4160ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5b40d29c0 0x7fa5b40cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 --2- 172.21.15.72:0/771982556 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5b40cf830 0x7fa5b41f3040 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 >> 172.21.15.72:0/771982556 conn(0x7fa5b40d3cd0 msgr2=0x7fa5b40d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 shutdown_connections 2024-03-06T07:22:26.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.192+0000 7fa5b884e640 1 -- 172.21.15.72:0/771982556 wait complete. 2024-03-06T07:22:26.205 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:26.206 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:26.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 Processor -- start 2024-03-06T07:22:26.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 -- start start 2024-03-06T07:22:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f11681586e0 0x7f11680aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11680b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1168158ac0 con 0x7f11680aef80 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116815b2a0 con 0x7f11681586e0 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f116d37a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116815b540 con 0x7f11680d01a0 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f11667fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f11680aef80 0x7f11680cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39298/0 (socket says 172.21.15.72:39298) 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.308+0000 7f11667fc640 1 -- 172.21.15.72:0/299268795 learned_addr learned my addr 172.21.15.72:0/299268795 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782446525 0 0) 0x7f1168158ac0 con 0x7f11680aef80 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1166ffd640 1 --2- 172.21.15.72:0/299268795 >> v2:172.21.15.146:3300/0 conn(0x7f11681586e0 0x7f11680aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150003520 con 0x7f11680aef80 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11677fe640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11680b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587131026 0 0) 0x7f1150003520 con 0x7f11680aef80 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 msgr2=0x7f11680b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11680b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 >> v2:172.21.15.146:3300/0 conn(0x7f11681586e0 msgr2=0x7f11680aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 --2- 172.21.15.72:0/299268795 >> v2:172.21.15.146:3300/0 conn(0x7f11681586e0 0x7f11680aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116815b820 con 0x7f11680aef80 2024-03-06T07:22:26.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f115c001670 con 0x7f11680aef80 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 >> v1:172.21.15.72:6789/0 conn(0x7f11680aef80 legacy=0x7f11680cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11500039b0 0x7f1150003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1150004800 0x7f1150004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815b2a0 con 0x7f1150004800 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815b540 con 0x7f11500039b0 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115000b9c0 con 0x7f11680d01a0 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11667fc640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1150004800 0x7f1150004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1166ffd640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11500039b0 0x7f1150003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11677fe640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11667fc640 1 -- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 msgr2=0x7f11500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11667fc640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11667fc640 1 -- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11500039b0 msgr2=0x7f1150003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11667fc640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11500039b0 0x7f1150003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11667fc640 1 -- 172.21.15.72:0/299268795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150003520 con 0x7f1150004800 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f1166ffd640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11500039b0 0x7f1150003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.309+0000 7f11677fe640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680d01a0 0x7f11500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.310+0000 7f11667fc640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1150004800 0x7f1150004c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f11680af390 tx=0x7f1150005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b4a33b6bc105c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.310+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c007d50 con 0x7f1150004800 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.310+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c030070 con 0x7f1150004800 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.310+0000 7f1165ffb640 1 -- 172.21.15.72:0/299268795 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c005d10 con 0x7f1150004800 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f116d37a640 1 -- 172.21.15.72:0/299268795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1150004800 msgr2=0x7f1150004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f116d37a640 1 --2- 172.21.15.72:0/299268795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1150004800 0x7f1150004c80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f11680af390 tx=0x7f1150005830 comp rx=0 tx=0).stop 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f11677fe640 1 -- 172.21.15.72:0/299268795 reap_dead start 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f116d37a640 1 -- 172.21.15.72:0/299268795 shutdown_connections 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f116d37a640 1 -- 172.21.15.72:0/299268795 >> 172.21.15.72:0/299268795 conn(0x7f116806b190 msgr2=0x7f116806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f116d37a640 1 -- 172.21.15.72:0/299268795 shutdown_connections 2024-03-06T07:22:26.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.311+0000 7f116d37a640 1 -- 172.21.15.72:0/299268795 wait complete. 2024-03-06T07:22:26.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.313+0000 7f116d37a640 1 Processor -- start 2024-03-06T07:22:26.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.313+0000 7f116d37a640 1 -- start start 2024-03-06T07:22:26.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f116d37a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1150004800 0x7f1168140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f116d37a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 0x7f11681599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f116d37a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680ce510 0x7f11680ce990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f116d37a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168071350 con 0x7f1168159520 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f116d37a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815bde0 con 0x7f1150004800 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f116d37a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11680714d0 con 0x7f11680ce510 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f11667fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 0x7f11681599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f11667fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 0x7f11681599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38570/0 (socket says 172.21.15.72:38570) 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f11667fc640 1 -- 172.21.15.72:0/2341367768 learned_addr learned my addr 172.21.15.72:0/2341367768 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f11677fe640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680ce510 0x7f11680ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.314+0000 7f1166ffd640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1150004800 0x7f1168140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11667fc640 1 -- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680ce510 msgr2=0x7f11680ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11667fc640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680ce510 0x7f11680ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11667fc640 1 -- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1150004800 msgr2=0x7f1168140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11667fc640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1150004800 0x7f1168140180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11667fc640 1 -- 172.21.15.72:0/2341367768 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11681463a0 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11677fe640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680ce510 0x7f11680ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f11667fc640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 0x7f11681599a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f115c00c2d0 tx=0x7f116815b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c004bf0 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c004d90 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11680cf0f0 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c0072b0 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.315+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11681e16a0 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.316+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11680732b0 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.316+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f115c047020 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.317+0000 7f114f7fe640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f115406b500 0x7f115406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.317+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f115c0ab390 con 0x7f1168159520 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.317+0000 7f1166ffd640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f115406b500 0x7f115406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.317+0000 7f1166ffd640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f115406b500 0x7f115406d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1158009a10 tx=0x7f1158008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.321+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f11681463a0 con 0x7f1168159520 2024-03-06T07:22:26.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.499+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1168068ff0 con 0x7f1168159520 2024-03-06T07:22:26.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.499+0000 7f114f7fe640 1 -- 172.21.15.72:0/2341367768 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f115c07a2a0 con 0x7f1168159520 2024-03-06T07:22:26.555 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:26.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f115406b500 msgr2=0x7f115406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f115406b500 0x7f115406d9c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1158009a10 tx=0x7f1158008040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 msgr2=0x7f11681599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 0x7f11681599a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f115c00c2d0 tx=0x7f116815b9f0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 shutdown_connections 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f115406b500 0x7f115406d9c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f11680ce510 0x7f11680ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1168159520 0x7f11681599a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 --2- 172.21.15.72:0/2341367768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1150004800 0x7f1168140180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 >> 172.21.15.72:0/2341367768 conn(0x7f116806b190 msgr2=0x7f1168153c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 shutdown_connections 2024-03-06T07:22:26.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.501+0000 7f116d37a640 1 -- 172.21.15.72:0/2341367768 wait complete. 2024-03-06T07:22:26.562 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:26.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.663+0000 7fafe7f0c640 1 Processor -- start 2024-03-06T07:22:26.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe7f0c640 1 -- start start 2024-03-06T07:22:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe7f0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 0x7fafe00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe7f0c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe01618b0 con 0x7fafe00d2480 2024-03-06T07:22:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe7f0c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00cf400 con 0x7fafe00d1910 2024-03-06T07:22:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe7f0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00cf6a0 con 0x7fafe01614d0 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe64a9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fafe00d2480 0x7fafe00b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39310/0 (socket says 172.21.15.72:39310) 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.664+0000 7fafe64a9640 1 -- 172.21.15.72:0/3387502730 learned_addr learned my addr 172.21.15.72:0/3387502730 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe5ca8640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 0x7fafe00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404123190 0 0) 0x7fafe01618b0 con 0x7fafe00d2480 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc003520 con 0x7fafe00d2480 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3268864736 0 0) 0x7fafcc003520 con 0x7fafe00d2480 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 msgr2=0x7fafe00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 0x7fafe00d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 >> v1:172.21.15.146:6789/0 conn(0x7fafe00d1910 legacy=0x7fafe00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00cf980 con 0x7fafe00d2480 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fafd80019f0 con 0x7fafe00d2480 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 >> v1:172.21.15.72:6789/0 conn(0x7fafe00d2480 legacy=0x7fafe00b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc0039e0 0x7fafcc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafcc004390 0x7fafcc004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 0x7fafcc0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00cf6a0 con 0x7fafcc004390 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc00b9c0 con 0x7fafcc0039e0 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc00bbc0 con 0x7fafe01614d0 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe54a7640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafcc004390 0x7fafcc004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe5ca8640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc0039e0 0x7fafcc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe64a9640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 0x7fafcc0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe54a7640 1 -- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 msgr2=0x7fafcc0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe54a7640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe01614d0 0x7fafcc0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe54a7640 1 -- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc0039e0 msgr2=0x7fafcc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe54a7640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc0039e0 0x7fafcc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe54a7640 1 -- 172.21.15.72:0/3387502730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc003520 con 0x7fafcc004390 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.665+0000 7fafe5ca8640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc0039e0 0x7fafcc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.666+0000 7fafe54a7640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafcc004390 0x7fafcc004860 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fafd0001860 tx=0x7fafcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9447fa6531f2b57c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.666+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0038070 con 0x7fafcc004390 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.666+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd0033410 con 0x7fafcc004390 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.666+0000 7fafe4ca6640 1 -- 172.21.15.72:0/3387502730 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0001430 con 0x7fafcc004390 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe7f0c640 1 -- 172.21.15.72:0/3387502730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafcc004390 msgr2=0x7fafcc004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/3387502730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafcc004390 0x7fafcc004860 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fafd0001860 tx=0x7fafcc005830 comp rx=0 tx=0).stop 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe64a9640 1 -- 172.21.15.72:0/3387502730 reap_dead start 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe7f0c640 1 -- 172.21.15.72:0/3387502730 shutdown_connections 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe7f0c640 1 -- 172.21.15.72:0/3387502730 >> 172.21.15.72:0/3387502730 conn(0x7fafe00d3d40 msgr2=0x7fafe00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe7f0c640 1 -- 172.21.15.72:0/3387502730 shutdown_connections 2024-03-06T07:22:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.667+0000 7fafe7f0c640 1 -- 172.21.15.72:0/3387502730 wait complete. 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.668+0000 7fafe7f0c640 1 Processor -- start 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.668+0000 7fafe7f0c640 1 -- start start 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe7f0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc004390 0x7fafe00cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe7f0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe00cbe60 0x7fafe015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe7f0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 0x7fafe015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe7f0c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00d4860 con 0x7fafe0146a50 2024-03-06T07:22:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe7f0c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00b0910 con 0x7fafcc004390 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe7f0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00d49e0 con 0x7fafe00cbe60 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 0x7fafe015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 0x7fafe015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38578/0 (socket says 172.21.15.72:38578) 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 -- 172.21.15.72:0/704034816 learned_addr learned my addr 172.21.15.72:0/704034816 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe5ca8640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc004390 0x7fafe00cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe54a7640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe00cbe60 0x7fafe015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 -- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe00cbe60 msgr2=0x7fafe015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe00cbe60 0x7fafe015d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 -- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc004390 msgr2=0x7fafe00cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc004390 0x7fafe00cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 -- 172.21.15.72:0/704034816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe015daf0 con 0x7fafe0146a50 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe54a7640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe00cbe60 0x7fafe015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.669+0000 7fafe64a9640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 0x7fafe015a7d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fafd8000f80 tx=0x7fafd8007ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.670+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd8017020 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.670+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd800ab80 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.670+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe00cc3d0 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.670+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd80133f0 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.670+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe00cc990 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.671+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe0068ff0 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.671+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafd801a020 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.672+0000 7fafd67fc640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafc406b500 0x7fafc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.672+0000 7fafe5ca8640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafc406b500 0x7fafc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.672+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafd808a9f0 con 0x7fafe0146a50 2024-03-06T07:22:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.672+0000 7fafe5ca8640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafc406b500 0x7fafc406d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009a10 tx=0x7fafdc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.676+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafd8021020 con 0x7fafe0146a50 2024-03-06T07:22:26.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.854+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fafe00d2050 con 0x7fafe0146a50 2024-03-06T07:22:26.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.854+0000 7fafd67fc640 1 -- 172.21.15.72:0/704034816 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fafd8059900 con 0x7fafe0146a50 2024-03-06T07:22:26.856 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:26.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafc406b500 msgr2=0x7fafc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafc406b500 0x7fafc406d9c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009a10 tx=0x7fafdc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:26.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 msgr2=0x7fafe015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 0x7fafe015a7d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fafd8000f80 tx=0x7fafd8007ae0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 shutdown_connections 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafc406b500 0x7fafc406d9c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafe0146a50 0x7fafe015a7d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafe00cbe60 0x7fafe015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 --2- 172.21.15.72:0/704034816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafcc004390 0x7fafe00cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 >> 172.21.15.72:0/704034816 conn(0x7fafe00d3d40 msgr2=0x7fafe0160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 shutdown_connections 2024-03-06T07:22:26.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.856+0000 7fafe7f0c640 1 -- 172.21.15.72:0/704034816 wait complete. 2024-03-06T07:22:26.869 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:26.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.971+0000 7f2181602640 1 Processor -- start 2024-03-06T07:22:26.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.971+0000 7f2181602640 1 -- start start 2024-03-06T07:22:26.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.972+0000 7f2181602640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f217c1614e0 0x7f217c0b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.972+0000 7f2181602640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f217c0b7d80 0x7f217c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.972+0000 7f2181602640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f217c0728f0 con 0x7f217c09ee00 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.972+0000 7f2181602640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217c0b78f0 con 0x7f217c0b7d80 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.972+0000 7f2181602640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217c0b4670 con 0x7f217c1614e0 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.972+0000 7f217a7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f217c0b7d80 0x7f217c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217a7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f217c0b7d80 0x7f217c06afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38586/0 (socket says 172.21.15.72:38586) 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217a7fc640 1 -- 172.21.15.72:0/62120180 learned_addr learned my addr 172.21.15.72:0/62120180 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217affd640 1 --2- 172.21.15.72:0/62120180 >> v2:172.21.15.146:3300/0 conn(0x7f217c1614e0 0x7f217c0b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217a7fc640 1 -- 172.21.15.72:0/62120180 >> v2:172.21.15.146:3300/0 conn(0x7f217c1614e0 msgr2=0x7f217c0b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217a7fc640 1 --2- 172.21.15.72:0/62120180 >> v2:172.21.15.146:3300/0 conn(0x7f217c1614e0 0x7f217c0b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217a7fc640 1 -- 172.21.15.72:0/62120180 >> v1:172.21.15.146:6789/0 conn(0x7f217c09ee00 legacy=0x7f217c166bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.973+0000 7f217a7fc640 1 -- 172.21.15.72:0/62120180 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c0b4950 con 0x7f217c0b7d80 2024-03-06T07:22:26.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f217a7fc640 1 --2- 172.21.15.72:0/62120180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06afb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f217c06d860 tx=0x7f216c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210207151436703a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2179ffb640 1 -- 172.21.15.72:0/62120180 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c034070 con 0x7f217c0b7d80 2024-03-06T07:22:26.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2179ffb640 1 -- 172.21.15.72:0/62120180 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c030860 con 0x7f217c0b7d80 2024-03-06T07:22:26.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2179ffb640 1 -- 172.21.15.72:0/62120180 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c030c30 con 0x7f217c0b7d80 2024-03-06T07:22:26.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 -- 172.21.15.72:0/62120180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 msgr2=0x7f217c06afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 --2- 172.21.15.72:0/62120180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06afb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f217c06d860 tx=0x7f216c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 -- 172.21.15.72:0/62120180 shutdown_connections 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 --2- 172.21.15.72:0/62120180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06afb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 --2- 172.21.15.72:0/62120180 >> v2:172.21.15.146:3300/0 conn(0x7f217c1614e0 0x7f217c0b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 -- 172.21.15.72:0/62120180 >> 172.21.15.72:0/62120180 conn(0x7f217c06d3b0 msgr2=0x7f217c0b70b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 -- 172.21.15.72:0/62120180 shutdown_connections 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.974+0000 7f2181602640 1 -- 172.21.15.72:0/62120180 wait complete. 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 Processor -- start 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 -- start start 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f217c09ee00 0x7f217c155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f217c1614e0 0x7f217c14a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0b4c30 con 0x7f217c0b7d80 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c1618c0 con 0x7f217c1614e0 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f2181602640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217c06b4f0 con 0x7f217c09ee00 2024-03-06T07:22:26.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f217a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f217a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38590/0 (socket says 172.21.15.72:38590) 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.976+0000 7f217a7fc640 1 -- 172.21.15.72:0/3867283629 learned_addr learned my addr 172.21.15.72:0/3867283629 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217affd640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f217c09ee00 0x7f217c155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217bfff640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f217c1614e0 0x7f217c14a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217a7fc640 1 -- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f217c09ee00 msgr2=0x7f217c155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217a7fc640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f217c09ee00 0x7f217c155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217a7fc640 1 -- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f217c1614e0 msgr2=0x7f217c14a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217a7fc640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f217c1614e0 0x7f217c14a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217a7fc640 1 -- 172.21.15.72:0/3867283629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c154db0 con 0x7f217c0b7d80 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f217a7fc640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06bb20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f216c02b9e0 tx=0x7f216c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c048070 con 0x7f217c0b7d80 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c030660 con 0x7f217c0b7d80 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c155090 con 0x7f217c0b7d80 2024-03-06T07:22:26.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c034040 con 0x7f217c0b7d80 2024-03-06T07:22:26.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.977+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c1560b0 con 0x7f217c0b7d80 2024-03-06T07:22:26.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.978+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217c0b7680 con 0x7f217c0b7d80 2024-03-06T07:22:26.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.979+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f216c04b070 con 0x7f217c0b7d80 2024-03-06T07:22:26.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.979+0000 7f21737fe640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f215c06b4d0 0x7f215c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:26.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.979+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f216c0ac090 con 0x7f217c0b7d80 2024-03-06T07:22:26.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.979+0000 7f217affd640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f215c06b4d0 0x7f215c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:26.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.979+0000 7f217affd640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f215c06b4d0 0x7f215c06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2164009aa0 tx=0x7f2164009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:26.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:26.984+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f217c0b7680 con 0x7f217c0b7d80 2024-03-06T07:22:27.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.161+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f217c04f330 con 0x7f217c0b7d80 2024-03-06T07:22:27.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.161+0000 7f21737fe640 1 -- 172.21.15.72:0/3867283629 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f216c07a8c0 con 0x7f217c0b7d80 2024-03-06T07:22:27.164 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:27.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f215c06b4d0 msgr2=0x7f215c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f215c06b4d0 0x7f215c06d990 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2164009aa0 tx=0x7f2164009190 comp rx=0 tx=0).stop 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 msgr2=0x7f217c06bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06bb20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f216c02b9e0 tx=0x7f216c004730 comp rx=0 tx=0).stop 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 shutdown_connections 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f215c06b4d0 0x7f215c06d990 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f217c1614e0 0x7f217c14a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f217c0b7d80 0x7f217c06bb20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 --2- 172.21.15.72:0/3867283629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f217c09ee00 0x7f217c155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 >> 172.21.15.72:0/3867283629 conn(0x7f217c06d3b0 msgr2=0x7f217c0b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 shutdown_connections 2024-03-06T07:22:27.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.164+0000 7f2181602640 1 -- 172.21.15.72:0/3867283629 wait complete. 2024-03-06T07:22:27.178 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:27.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 Processor -- start 2024-03-06T07:22:27.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 -- start start 2024-03-06T07:22:27.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f242806d3c0 0x7f24280b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24280b1850 0x7f24280b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280af4b0 con 0x7f24281614a0 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24280af790 con 0x7f242806d3c0 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f242d381640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24280afa70 con 0x7f24280b1850 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f241ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f242806d3c0 0x7f24280b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f2426ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f24281614a0 0x7f24280b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39318/0 (socket says 172.21.15.72:39318) 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.279+0000 7f2426ffd640 1 -- 172.21.15.72:0/38836985 learned_addr learned my addr 172.21.15.72:0/38836985 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.280+0000 7f24277fe640 1 --2- 172.21.15.72:0/38836985 >> v2:172.21.15.146:3300/0 conn(0x7f24280b1850 0x7f24280b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.280+0000 7f241ffff640 1 -- 172.21.15.72:0/38836985 >> v2:172.21.15.146:3300/0 conn(0x7f24280b1850 msgr2=0x7f24280b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.280+0000 7f241ffff640 1 --2- 172.21.15.72:0/38836985 >> v2:172.21.15.146:3300/0 conn(0x7f24280b1850 0x7f24280b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.280+0000 7f241ffff640 1 -- 172.21.15.72:0/38836985 >> v1:172.21.15.72:6789/0 conn(0x7f24281614a0 legacy=0x7f24280b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.280+0000 7f241ffff640 1 -- 172.21.15.72:0/38836985 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280afd50 con 0x7f242806d3c0 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.280+0000 7f24267fc640 1 -- 172.21.15.72:0/38836985 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859697411 0 0) 0x7f24280af4b0 con 0x7f24281614a0 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f241ffff640 1 --2- 172.21.15.72:0/38836985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f24280b1310 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f24280b34f0 tx=0x7f241002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97250445a20ebf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f24267fc640 1 -- 172.21.15.72:0/38836985 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2410035020 con 0x7f242806d3c0 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f24267fc640 1 -- 172.21.15.72:0/38836985 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2410031830 con 0x7f242806d3c0 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f24267fc640 1 -- 172.21.15.72:0/38836985 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2410031c00 con 0x7f242806d3c0 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 -- 172.21.15.72:0/38836985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 msgr2=0x7f24280b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 --2- 172.21.15.72:0/38836985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f24280b1310 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f24280b34f0 tx=0x7f241002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 -- 172.21.15.72:0/38836985 shutdown_connections 2024-03-06T07:22:27.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 --2- 172.21.15.72:0/38836985 >> v2:172.21.15.146:3300/0 conn(0x7f24280b1850 0x7f24280b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 --2- 172.21.15.72:0/38836985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f24280b1310 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 -- 172.21.15.72:0/38836985 >> 172.21.15.72:0/38836985 conn(0x7f242809a9f0 msgr2=0x7f242809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 -- 172.21.15.72:0/38836985 shutdown_connections 2024-03-06T07:22:27.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.281+0000 7f242d381640 1 -- 172.21.15.72:0/38836985 wait complete. 2024-03-06T07:22:27.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.282+0000 7f242d381640 1 Processor -- start 2024-03-06T07:22:27.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.282+0000 7f242d381640 1 -- start start 2024-03-06T07:22:27.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f242d381640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f2428158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f242d381640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280b1850 0x7f242815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f242d381640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24281614a0 0x7f24280ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f242d381640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b1db0 con 0x7f242806d3c0 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f242d381640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b1c30 con 0x7f24280b1850 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f242d381640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b1f30 con 0x7f24281614a0 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f2428158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f2428158080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38598/0 (socket says 172.21.15.72:38598) 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 -- 172.21.15.72:0/2612111698 learned_addr learned my addr 172.21.15.72:0/2612111698 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f241ffff640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280b1850 0x7f242815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f24277fe640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24281614a0 0x7f24280ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 -- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24281614a0 msgr2=0x7f24280ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24281614a0 0x7f24280ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 -- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280b1850 msgr2=0x7f242815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.283+0000 7f2426ffd640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280b1850 0x7f242815ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f2426ffd640 1 -- 172.21.15.72:0/2612111698 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280cad10 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f241ffff640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280b1850 0x7f242815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f2426ffd640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f2428158080 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f241800c4d0 tx=0x7f241800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418018070 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2418007500 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242815d4a0 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418004a20 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.284+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242815da60 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.285+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24280caee0 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.285+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f241801b070 con 0x7f242806d3c0 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.286+0000 7f241f7fe640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f240806b4d0 0x7f240806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.286+0000 7f241ffff640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f240806b4d0 0x7f240806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.286+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2418089420 con 0x7f242806d3c0 2024-03-06T07:22:27.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.286+0000 7f241ffff640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f240806b4d0 0x7f240806d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2410036000 tx=0x7f2410008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.290+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2418058330 con 0x7f242806d3c0 2024-03-06T07:22:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.572+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f242806e8e0 con 0x7f242806d3c0 2024-03-06T07:22:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.573+0000 7f241f7fe640 1 -- 172.21.15.72:0/2612111698 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2418057c30 con 0x7f242806d3c0 2024-03-06T07:22:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f240806b4d0 msgr2=0x7f240806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f240806b4d0 0x7f240806d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2410036000 tx=0x7f2410008030 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 msgr2=0x7f2428158080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f2428158080 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f241800c4d0 tx=0x7f241800c990 comp rx=0 tx=0).stop 2024-03-06T07:22:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 shutdown_connections 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f240806b4d0 0x7f240806d990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24281614a0 0x7f24280ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280b1850 0x7f242815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 --2- 172.21.15.72:0/2612111698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242806d3c0 0x7f2428158080 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 >> 172.21.15.72:0/2612111698 conn(0x7f242809a9f0 msgr2=0x7f24281600c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 shutdown_connections 2024-03-06T07:22:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.575+0000 7f242d381640 1 -- 172.21.15.72:0/2612111698 wait complete. 2024-03-06T07:22:27.588 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:22:27.588 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:27.604 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:27.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.746+0000 7ff3b2d7a640 1 Processor -- start 2024-03-06T07:22:27.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.746+0000 7ff3b2d7a640 1 -- start start 2024-03-06T07:22:27.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3b2d7a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff3ac0cfff0 0x7ff3ac0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3b2d7a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac161980 con 0x7ff3ac15bb20 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3b2d7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac161c60 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3b2d7a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ac0d2500 con 0x7ff3ac0cfff0 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3abfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff3ac0cfff0 0x7ff3ac0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3abfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff3ac0cfff0 0x7ff3ac0d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38604/0 (socket says 172.21.15.72:38604) 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3b0b16640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff3ac15bb20 0x7ff3ac0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39332/0 (socket says 172.21.15.72:39332) 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3abfff640 1 -- 172.21.15.72:0/3865791491 learned_addr learned my addr 172.21.15.72:0/3865791491 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3abfff640 1 -- 172.21.15.72:0/3865791491 >> v1:172.21.15.146:6789/0 conn(0x7ff3ac0cc8b0 legacy=0x7ff3ac0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3abfff640 1 -- 172.21.15.72:0/3865791491 >> v1:172.21.15.72:6789/0 conn(0x7ff3ac15bb20 legacy=0x7ff3ac0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3abfff640 1 -- 172.21.15.72:0/3865791491 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac161f40 con 0x7ff3ac0cfff0 2024-03-06T07:22:27.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.747+0000 7ff3ab7fe640 1 -- 172.21.15.72:0/3865791491 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363608651 0 0) 0x7ff3ac161980 con 0x7ff3ac15bb20 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3abfff640 1 --2- 172.21.15.72:0/3865791491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cfff0 0x7ff3ac0d04a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff398009870 tx=0x7ff39802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4752bb5ff59610 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3ab7fe640 1 -- 172.21.15.72:0/3865791491 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398034070 con 0x7ff3ac0cfff0 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3ab7fe640 1 -- 172.21.15.72:0/3865791491 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff398030860 con 0x7ff3ac0cfff0 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3ab7fe640 1 -- 172.21.15.72:0/3865791491 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398030c30 con 0x7ff3ac0cfff0 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3865791491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cfff0 msgr2=0x7ff3ac0d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3865791491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cfff0 0x7ff3ac0d04a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff398009870 tx=0x7ff39802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3865791491 shutdown_connections 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3865791491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cfff0 0x7ff3ac0d04a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3865791491 >> 172.21.15.72:0/3865791491 conn(0x7ff3ac09a9f0 msgr2=0x7ff3ac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3865791491 shutdown_connections 2024-03-06T07:22:27.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.749+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3865791491 wait complete. 2024-03-06T07:22:27.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.750+0000 7ff3b2d7a640 1 Processor -- start 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.750+0000 7ff3b2d7a640 1 -- start start 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b2d7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 0x7ff3ac154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b2d7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3ac0cfff0 0x7ff3ac1e77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b2d7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ac15bb20 0x7ff3ac1e96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b2d7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0d1980 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b2d7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac162500 con 0x7ff3ac15bb20 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b2d7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0d1b00 con 0x7ff3ac0cfff0 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b0b16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 0x7ff3ac154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b0b16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 0x7ff3ac154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38610/0 (socket says 172.21.15.72:38610) 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.751+0000 7ff3b0b16640 1 -- 172.21.15.72:0/3879720496 learned_addr learned my addr 172.21.15.72:0/3879720496 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b1317640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ac15bb20 0x7ff3ac1e96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3abfff640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3ac0cfff0 0x7ff3ac1e77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b0b16640 1 -- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3ac0cfff0 msgr2=0x7ff3ac1e77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b0b16640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3ac0cfff0 0x7ff3ac1e77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b0b16640 1 -- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ac15bb20 msgr2=0x7ff3ac1e96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b0b16640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ac15bb20 0x7ff3ac1e96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b0b16640 1 -- 172.21.15.72:0/3879720496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac0c9190 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b0b16640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 0x7ff3ac154d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c500 tx=0x7ff3a000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0018070 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0007580 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0004a20 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac1e9c20 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.752+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac1e8150 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.753+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac04f330 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.754+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3a001b070 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.754+0000 7ff3a97fa640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff39006b4d0 0x7ff39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:27.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.754+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3a0089430 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.754+0000 7ff3abfff640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff39006b4d0 0x7ff39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:27.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.755+0000 7ff3abfff640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff39006b4d0 0x7ff39006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff398036000 tx=0x7ff39803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:27.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.758+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3a0057c60 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.937+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3ac0b1fe0 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.939+0000 7ff3a97fa640 1 -- 172.21.15.72:0/3879720496 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7ff3a005bd00 con 0x7ff3ac0cc8b0 2024-03-06T07:22:27.941 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:27.941 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.940+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff39006b4d0 msgr2=0x7ff39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.940+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff39006b4d0 0x7ff39006d990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff398036000 tx=0x7ff39803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.940+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 msgr2=0x7ff3ac154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 0x7ff3ac154d60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000c500 tx=0x7ff3a000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 shutdown_connections 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff39006b4d0 0x7ff39006d990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ac15bb20 0x7ff3ac1e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3ac0cfff0 0x7ff3ac1e77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 --2- 172.21.15.72:0/3879720496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ac0cc8b0 0x7ff3ac154d60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:27.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 >> 172.21.15.72:0/3879720496 conn(0x7ff3ac09a9f0 msgr2=0x7ff3ac06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:27.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 shutdown_connections 2024-03-06T07:22:27.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:27.941+0000 7ff3b2d7a640 1 -- 172.21.15.72:0/3879720496 wait complete. 2024-03-06T07:22:27.954 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:27.955 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.057+0000 7ff1cc50c640 1 Processor -- start 2024-03-06T07:22:28.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.057+0000 7ff1cc50c640 1 -- start start 2024-03-06T07:22:28.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1cc50c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1c41614d0 0x7ff1c40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1cc50c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c41618b0 con 0x7ff1c40d24b0 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1cc50c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c40cf400 con 0x7ff1c40d1910 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1cc50c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c40cf6a0 con 0x7ff1c41614d0 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1ca2a8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1c41614d0 0x7ff1c40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1caaa9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff1c40d24b0 0x7ff1c40b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39342/0 (socket says 172.21.15.72:39342) 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1caaa9640 1 -- 172.21.15.72:0/3284691787 learned_addr learned my addr 172.21.15.72:0/3284691787 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/3284691787 >> v1:172.21.15.146:6789/0 conn(0x7ff1c40d1910 legacy=0x7ff1c40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/3284691787 >> v1:172.21.15.72:6789/0 conn(0x7ff1c40d24b0 legacy=0x7ff1c40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/3284691787 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40cf980 con 0x7ff1c41614d0 2024-03-06T07:22:28.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.058+0000 7ff1c92a6640 1 -- 172.21.15.72:0/3284691787 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183008556 0 0) 0x7ff1c41618b0 con 0x7ff1c40d24b0 2024-03-06T07:22:28.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.059+0000 7ff1ca2a8640 1 --2- 172.21.15.72:0/3284691787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c41614d0 0x7ff1c40d4860 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff1c408c470 tx=0x7ff1bc02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a7a68fe2b550a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.059+0000 7ff1c92a6640 1 -- 172.21.15.72:0/3284691787 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc039070 con 0x7ff1c41614d0 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.059+0000 7ff1c92a6640 1 -- 172.21.15.72:0/3284691787 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc02cc40 con 0x7ff1c41614d0 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.059+0000 7ff1c92a6640 1 -- 172.21.15.72:0/3284691787 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc034440 con 0x7ff1c41614d0 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 -- 172.21.15.72:0/3284691787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c41614d0 msgr2=0x7ff1c40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/3284691787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c41614d0 0x7ff1c40d4860 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff1c408c470 tx=0x7ff1bc02be90 comp rx=0 tx=0).stop 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 -- 172.21.15.72:0/3284691787 shutdown_connections 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/3284691787 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c41614d0 0x7ff1c40d4860 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 -- 172.21.15.72:0/3284691787 >> 172.21.15.72:0/3284691787 conn(0x7ff1c40d3d40 msgr2=0x7ff1c40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 -- 172.21.15.72:0/3284691787 shutdown_connections 2024-03-06T07:22:28.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.060+0000 7ff1cc50c640 1 -- 172.21.15.72:0/3284691787 wait complete. 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.061+0000 7ff1cc50c640 1 Processor -- start 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.061+0000 7ff1cc50c640 1 -- start start 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 0x7ff1c415f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40d24b0 0x7ff1c41f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c41614d0 0x7ff1c41f7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b1540 con 0x7ff1c40d1910 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b0910 con 0x7ff1c41614d0 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c40b16c0 con 0x7ff1c40d24b0 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 0x7ff1c415f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 0x7ff1c415f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38636/0 (socket says 172.21.15.72:38636) 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/1537288866 learned_addr learned my addr 172.21.15.72:0/1537288866 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40d24b0 msgr2=0x7ff1c41f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1caaa9640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c41614d0 0x7ff1c41f7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40d24b0 0x7ff1c41f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c41614d0 msgr2=0x7ff1c41f7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c41614d0 0x7ff1c41f7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 -- 172.21.15.72:0/1537288866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c40cbc20 con 0x7ff1c40d1910 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1caaa9640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c41614d0 0x7ff1c41f7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1ca2a8640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 0x7ff1c415f1d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc037040 tx=0x7ff1bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc044020 con 0x7ff1c40d1910 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc03f470 con 0x7ff1c40d1910 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.062+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c4153b70 con 0x7ff1c40d1910 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.063+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc039040 con 0x7ff1c40d1910 2024-03-06T07:22:28.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.063+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c415f710 con 0x7ff1c40d1910 2024-03-06T07:22:28.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.063+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c40d3770 con 0x7ff1c40d1910 2024-03-06T07:22:28.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.064+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1bc047020 con 0x7ff1c40d1910 2024-03-06T07:22:28.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.064+0000 7ff1baffd640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.065+0000 7ff1c9aa7640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.065+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1bc0acc00 con 0x7ff1c40d1910 2024-03-06T07:22:28.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.065+0000 7ff1c9aa7640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40b6050 tx=0x7ff1b40079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.069+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1bc0b3080 con 0x7ff1c40d1910 2024-03-06T07:22:28.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.308+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff1c4068ff0 con 0x7ff1c40d1910 2024-03-06T07:22:28.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.309+0000 7ff1baffd640 1 -- 172.21.15.72:0/1537288866 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1bc04e030 con 0x7ff1c40d1910 2024-03-06T07:22:28.311 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.310+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 msgr2=0x7ff1a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.310+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40b6050 tx=0x7ff1b40079b0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.310+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 msgr2=0x7ff1c415f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.310+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 0x7ff1c415f1d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc037040 tx=0x7ff1bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 shutdown_connections 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1a806b4d0 0x7ff1a806d990 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1c41614d0 0x7ff1c41f7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1c40d24b0 0x7ff1c41f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 --2- 172.21.15.72:0/1537288866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1c40d1910 0x7ff1c415f1d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 >> 172.21.15.72:0/1537288866 conn(0x7ff1c40d3d40 msgr2=0x7ff1c40b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 shutdown_connections 2024-03-06T07:22:28.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.311+0000 7ff1cc50c640 1 -- 172.21.15.72:0/1537288866 wait complete. 2024-03-06T07:22:28.327 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:28.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.428+0000 7f12b8ce1640 1 Processor -- start 2024-03-06T07:22:28.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.428+0000 7f12b8ce1640 1 -- start start 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b8ce1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12b40d14a0 0x7f12b415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b8ce1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 0x7f12b40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b8ce1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40cfa50 con 0x7f12b40d10c0 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b8ce1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b415c050 con 0x7f12b40d14a0 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b8ce1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40d0060 con 0x7f12b415eb00 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f12b40d10c0 0x7f12b40d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39354/0 (socket says 172.21.15.72:39354) 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b259c640 1 -- 172.21.15.72:0/114420491 learned_addr learned my addr 172.21.15.72:0/114420491 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12abfff640 1 --2- 172.21.15.72:0/114420491 >> v2:172.21.15.146:3300/0 conn(0x7f12b40d14a0 0x7f12b415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b2d9d640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 0x7f12b40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535575438 0 0) 0x7f12b40cfa50 con 0x7f12b40d10c0 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a0003520 con 0x7f12b40d10c0 2024-03-06T07:22:28.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1167700335 0 0) 0x7f12a0003520 con 0x7f12b40d10c0 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 msgr2=0x7f12b40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 0x7f12b40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 >> v2:172.21.15.146:3300/0 conn(0x7f12b40d14a0 msgr2=0x7f12b415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 --2- 172.21.15.72:0/114420491 >> v2:172.21.15.146:3300/0 conn(0x7f12b40d14a0 0x7f12b415e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40b45c0 con 0x7f12b40d10c0 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f12a4001670 con 0x7f12b40d10c0 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.430+0000 7f12abfff640 1 --2- 172.21.15.72:0/114420491 >> v2:172.21.15.146:3300/0 conn(0x7f12b40d14a0 0x7f12b415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 >> v1:172.21.15.72:6789/0 conn(0x7f12b40d10c0 legacy=0x7f12b40d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 0x7f12a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40d0060 con 0x7f12a00039b0 2024-03-06T07:22:28.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000b9c0 con 0x7f12a0004d40 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a000bb40 con 0x7f12b415eb00 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b259c640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12abfff640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 0x7f12a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b2d9d640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b259c640 1 -- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 msgr2=0x7f12a0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b259c640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b415eb00 0x7f12a0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b259c640 1 -- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12a0004d40 msgr2=0x7f12a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b259c640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12a0004d40 0x7f12a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.431+0000 7f12b259c640 1 -- 172.21.15.72:0/114420491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a0003520 con 0x7f12a00039b0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b259c640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c210 tx=0x7f12a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d094ad928aa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4007d70 con 0x7f12a00039b0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4030070 con 0x7f12a00039b0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b1d9b640 1 -- 172.21.15.72:0/114420491 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a402cd60 con 0x7f12a00039b0 2024-03-06T07:22:28.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b8ce1640 1 -- 172.21.15.72:0/114420491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 msgr2=0x7f12a0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/114420491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12a0003dd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f12a400c210 tx=0x7f12a0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:28.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b2d9d640 1 -- 172.21.15.72:0/114420491 reap_dead start 2024-03-06T07:22:28.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b8ce1640 1 -- 172.21.15.72:0/114420491 shutdown_connections 2024-03-06T07:22:28.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b8ce1640 1 -- 172.21.15.72:0/114420491 >> 172.21.15.72:0/114420491 conn(0x7f12b40b3800 msgr2=0x7f12b40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b8ce1640 1 -- 172.21.15.72:0/114420491 shutdown_connections 2024-03-06T07:22:28.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.432+0000 7f12b8ce1640 1 -- 172.21.15.72:0/114420491 wait complete. 2024-03-06T07:22:28.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 Processor -- start 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 -- start start 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12b40ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b40b48f0 0x7f12b40b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b40b6fe0 0x7f12b414f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40cfcf0 con 0x7f12a00039b0 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b415f440 con 0x7f12b40b6fe0 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b8ce1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40cae80 con 0x7f12b40b48f0 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12b40ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12b40ce540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38654/0 (socket says 172.21.15.72:38654) 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.434+0000 7f12b259c640 1 -- 172.21.15.72:0/3873846546 learned_addr learned my addr 172.21.15.72:0/3873846546 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12abfff640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b40b48f0 0x7f12b40b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b2d9d640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b40b6fe0 0x7f12b414f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b259c640 1 -- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b40b48f0 msgr2=0x7f12b40b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b259c640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b40b48f0 0x7f12b40b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b259c640 1 -- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b40b6fe0 msgr2=0x7f12b414f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b259c640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b40b6fe0 0x7f12b414f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b259c640 1 -- 172.21.15.72:0/3873846546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40b6470 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b259c640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12b40ce540 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f12a4000c00 tx=0x7f12a4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4004bf0 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a4004d90 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4007270 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4158920 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.435+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b40b5170 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.436+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b409fe60 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.437+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12a4047020 con 0x7f12a00039b0 2024-03-06T07:22:28.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.437+0000 7f12aaffd640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.437+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12a40ab310 con 0x7f12a00039b0 2024-03-06T07:22:28.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.437+0000 7f12abfff640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.438+0000 7f12abfff640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12b414fd30 tx=0x7f129c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.441+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12a4079ab0 con 0x7f12a00039b0 2024-03-06T07:22:28.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.757+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f12b40b3570 con 0x7f12a00039b0 2024-03-06T07:22:28.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.758+0000 7f12aaffd640 1 -- 172.21.15.72:0/3873846546 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f12a407db50 con 0x7f12a00039b0 2024-03-06T07:22:28.760 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 msgr2=0x7f129406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12b414fd30 tx=0x7f129c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 msgr2=0x7f12b40ce540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12b40ce540 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f12a4000c00 tx=0x7f12a4000fe0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 shutdown_connections 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b40b6fe0 0x7f12b414f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b40b48f0 0x7f12b40b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 --2- 172.21.15.72:0/3873846546 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12a00039b0 0x7f12b40ce540 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 >> 172.21.15.72:0/3873846546 conn(0x7f12b40b3800 msgr2=0x7f12b40d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 shutdown_connections 2024-03-06T07:22:28.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.760+0000 7f12b8ce1640 1 -- 172.21.15.72:0/3873846546 wait complete. 2024-03-06T07:22:28.773 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:28.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 Processor -- start 2024-03-06T07:22:28.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 -- start start 2024-03-06T07:22:28.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 0x7f78780cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78780d0af0 0x7f78780ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78780cd1e0 con 0x7f787815c050 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78780cd4b0 con 0x7f78780d0af0 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.876+0000 7f787f13e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780cd790 con 0x7f787806fd10 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f787ceda640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f787815c050 0x7f787806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39364/0 (socket says 172.21.15.72:39364) 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f787ceda640 1 -- 172.21.15.72:0/3292772608 learned_addr learned my addr 172.21.15.72:0/3292772608 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f787d6db640 1 --2- 172.21.15.72:0/3292772608 >> v2:172.21.15.146:3300/0 conn(0x7f78780d0af0 0x7f78780ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061487848 0 0) 0x7f78780cd1e0 con 0x7f787815c050 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f7877fff640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 0x7f78780cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7868003520 con 0x7f787815c050 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4174318820 0 0) 0x7f7868003520 con 0x7f787815c050 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 msgr2=0x7f78780cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 0x7f78780cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 >> v2:172.21.15.146:3300/0 conn(0x7f78780d0af0 msgr2=0x7f78780ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 --2- 172.21.15.72:0/3292772608 >> v2:172.21.15.146:3300/0 conn(0x7f78780d0af0 0x7f78780ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78780cda70 con 0x7f787815c050 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f786c001670 con 0x7f787815c050 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 >> v1:172.21.15.72:6789/0 conn(0x7f787815c050 legacy=0x7f787806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78680039b0 0x7f7868003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f7868004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 0x7f78680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780cd4b0 con 0x7f7868004800 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780cd790 con 0x7f78680039b0 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786800b9c0 con 0x7f787806fd10 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f7877fff640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f7868004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f787ceda640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78680039b0 0x7f7868003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.877+0000 7f787d6db640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 0x7f78680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f7877fff640 1 -- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 msgr2=0x7f78680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f7877fff640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f787806fd10 0x7f78680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f7877fff640 1 -- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78680039b0 msgr2=0x7f7868003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f7877fff640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78680039b0 0x7f7868003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f7877fff640 1 -- 172.21.15.72:0/3292772608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868003520 con 0x7f7868004800 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f7877fff640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f7868004c80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7864009af0 tx=0x7f7868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f526747b8abcabcc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.878+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864039070 con 0x7f7868004800 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786402c6b0 con 0x7f7868004800 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f78777fe640 1 -- 172.21.15.72:0/3292772608 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864009690 con 0x7f7868004800 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787f13e640 1 -- 172.21.15.72:0/3292772608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 msgr2=0x7f7868004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787f13e640 1 --2- 172.21.15.72:0/3292772608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f7868004c80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7864009af0 tx=0x7f7868005830 comp rx=0 tx=0).stop 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787d6db640 1 -- 172.21.15.72:0/3292772608 reap_dead start 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787f13e640 1 -- 172.21.15.72:0/3292772608 shutdown_connections 2024-03-06T07:22:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787f13e640 1 -- 172.21.15.72:0/3292772608 >> 172.21.15.72:0/3292772608 conn(0x7f78780b3800 msgr2=0x7f78780b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787f13e640 1 -- 172.21.15.72:0/3292772608 shutdown_connections 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.879+0000 7f787f13e640 1 -- 172.21.15.72:0/3292772608 wait complete. 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.880+0000 7f787f13e640 1 Processor -- start 2024-03-06T07:22:28.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.880+0000 7f787f13e640 1 -- start start 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787f13e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f78781507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787f13e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b48f0 0x7f78780b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787f13e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78780cae80 0x7f78780cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787f13e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780cc8e0 con 0x7f7868004800 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787f13e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78780cc760 con 0x7f78780cae80 2024-03-06T07:22:28.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787f13e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78780cca60 con 0x7f78780b48f0 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f78781507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f78781507e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38670/0 (socket says 172.21.15.72:38670) 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 -- 172.21.15.72:0/1134914447 learned_addr learned my addr 172.21.15.72:0/1134914447 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 -- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b48f0 msgr2=0x7f78780b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f7877fff640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b48f0 0x7f78780b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787d6db640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78780cae80 0x7f78780cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b48f0 0x7f78780b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 -- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78780cae80 msgr2=0x7f78780cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.881+0000 7f787ceda640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78780cae80 0x7f78780cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f787ceda640 1 -- 172.21.15.72:0/1134914447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78780b79c0 con 0x7f7868004800 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f787d6db640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78780cae80 0x7f78780cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f7877fff640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b48f0 0x7f78780b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:28.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f787ceda640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f78781507e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f786c007310 tx=0x7f786c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c00f960 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c018070 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78780b6e40 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c00fd60 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.882+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78780b7400 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.883+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78780cb840 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.883+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f786c01d460 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.884+0000 7f78757fa640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f785c06b500 0x7f785c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.884+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f786c08a280 con 0x7f7868004800 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.884+0000 7f7877fff640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f785c06b500 0x7f785c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:28.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.884+0000 7f7877fff640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f785c06b500 0x7f785c06d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7864036000 tx=0x7f7864030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:28.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:28.888+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78780b79c0 con 0x7f7868004800 2024-03-06T07:22:29.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.064+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f787809fe60 con 0x7f7868004800 2024-03-06T07:22:29.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.065+0000 7f78757fa640 1 -- 172.21.15.72:0/1134914447 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f786c059190 con 0x7f7868004800 2024-03-06T07:22:29.067 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f785c06b500 msgr2=0x7f785c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f785c06b500 0x7f785c06d9c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7864036000 tx=0x7f7864030000 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 msgr2=0x7f78781507e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f78781507e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f786c007310 tx=0x7f786c0077d0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 shutdown_connections 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f785c06b500 0x7f785c06d9c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78780cae80 0x7f78780cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78780b48f0 0x7f78780b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 --2- 172.21.15.72:0/1134914447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7868004800 0x7f78781507e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 >> 172.21.15.72:0/1134914447 conn(0x7f78780b3800 msgr2=0x7f78780aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 shutdown_connections 2024-03-06T07:22:29.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.067+0000 7f787f13e640 1 -- 172.21.15.72:0/1134914447 wait complete. 2024-03-06T07:22:29.081 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:29.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.180+0000 7fb2d8313640 1 Processor -- start 2024-03-06T07:22:29.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.180+0000 7fb2d8313640 1 -- start start 2024-03-06T07:22:29.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d8313640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb2d006b6c0 0x7fb2d0168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d8313640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00ce510 con 0x7fb2d01586c0 2024-03-06T07:22:29.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d8313640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d0161cf0 con 0x7fb2d00b0240 2024-03-06T07:22:29.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d8313640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d0158aa0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d60af640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb2d01586c0 0x7fb2d015efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39368/0 (socket says 172.21.15.72:39368) 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d60af640 1 -- 172.21.15.72:0/1646120942 learned_addr learned my addr 172.21.15.72:0/1646120942 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.181+0000 7fb2d68b0640 1 --2- 172.21.15.72:0/1646120942 >> v2:172.21.15.146:3300/0 conn(0x7fb2d006b6c0 0x7fb2d0168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.182+0000 7fb2d68b0640 1 -- 172.21.15.72:0/1646120942 >> v1:172.21.15.146:6789/0 conn(0x7fb2d00b0240 legacy=0x7fb2d006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.182+0000 7fb2d68b0640 1 -- 172.21.15.72:0/1646120942 >> v1:172.21.15.72:6789/0 conn(0x7fb2d01586c0 legacy=0x7fb2d015efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.182+0000 7fb2d68b0640 1 -- 172.21.15.72:0/1646120942 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0161fd0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.183+0000 7fb2d68b0640 1 --2- 172.21.15.72:0/1646120942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0168ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009d90 tx=0x7fb2c802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6116d2d4beb43b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.183+0000 7fb2d50ad640 1 -- 172.21.15.72:0/1646120942 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8039070 con 0x7fb2d006b6c0 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d50ad640 1 -- 172.21.15.72:0/1646120942 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c802cd00 con 0x7fb2d006b6c0 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d50ad640 1 -- 172.21.15.72:0/1646120942 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8034660 con 0x7fb2d006b6c0 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 -- 172.21.15.72:0/1646120942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006b6c0 msgr2=0x7fb2d0168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 --2- 172.21.15.72:0/1646120942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0168ee0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009d90 tx=0x7fb2c802bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 -- 172.21.15.72:0/1646120942 shutdown_connections 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 --2- 172.21.15.72:0/1646120942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0168ee0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 -- 172.21.15.72:0/1646120942 >> 172.21.15.72:0/1646120942 conn(0x7fb2d00d09d0 msgr2=0x7fb2d00d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 -- 172.21.15.72:0/1646120942 shutdown_connections 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.184+0000 7fb2d8313640 1 -- 172.21.15.72:0/1646120942 wait complete. 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.185+0000 7fb2d8313640 1 Processor -- start 2024-03-06T07:22:29.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.185+0000 7fb2d8313640 1 -- start start 2024-03-06T07:22:29.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d8313640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d8313640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d00b0240 0x7fb2d01e8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d8313640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d01586c0 0x7fb2d01f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d8313640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0162870 con 0x7fb2d006b6c0 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d8313640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d0162590 con 0x7fb2d01586c0 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d8313640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00719e0 con 0x7fb2d00b0240 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d60af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d60af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38686/0 (socket says 172.21.15.72:38686) 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d60af640 1 -- 172.21.15.72:0/2043231273 learned_addr learned my addr 172.21.15.72:0/2043231273 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d68b0640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d01586c0 0x7fb2d01f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.186+0000 7fb2d58ae640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d00b0240 0x7fb2d01e8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d60af640 1 -- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d00b0240 msgr2=0x7fb2d01e8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d60af640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d00b0240 0x7fb2d01e8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d60af640 1 -- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d01586c0 msgr2=0x7fb2d01f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d60af640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d01586c0 0x7fb2d01f78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d60af640 1 -- 172.21.15.72:0/2043231273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d01e8b10 con 0x7fb2d006b6c0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d60af640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0160530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c400 tx=0x7fb2cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc017070 con 0x7fb2d006b6c0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc005ce0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d01e8df0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc007310 con 0x7fb2d006b6c0 2024-03-06T07:22:29.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.187+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d01e93b0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.188+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d004f330 con 0x7fb2d006b6c0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.188+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2cc01a070 con 0x7fb2d006b6c0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.189+0000 7fb2c6ffd640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b500 0x7fb2b406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.189+0000 7fb2d58ae640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b500 0x7fb2b406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.189+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2cc089460 con 0x7fb2d006b6c0 2024-03-06T07:22:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.189+0000 7fb2d58ae640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b500 0x7fb2b406d9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00af7d0 tx=0x7fb2c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.193+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2cc057d20 con 0x7fb2d006b6c0 2024-03-06T07:22:29.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.383+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb2d00d17a0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.384+0000 7fb2c6ffd640 1 -- 172.21.15.72:0/2043231273 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb2cc05bdc0 con 0x7fb2d006b6c0 2024-03-06T07:22:29.386 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b500 msgr2=0x7fb2b406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b500 0x7fb2b406d9c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00af7d0 tx=0x7fb2c0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 msgr2=0x7fb2d0160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0160530 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00c400 tx=0x7fb2cc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 shutdown_connections 2024-03-06T07:22:29.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b500 0x7fb2b406d9c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d01586c0 0x7fb2d01f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d00b0240 0x7fb2d01e8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 --2- 172.21.15.72:0/2043231273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d006b6c0 0x7fb2d0160530 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 >> 172.21.15.72:0/2043231273 conn(0x7fb2d00d09d0 msgr2=0x7fb2d00b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 shutdown_connections 2024-03-06T07:22:29.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.386+0000 7fb2d8313640 1 -- 172.21.15.72:0/2043231273 wait complete. 2024-03-06T07:22:29.402 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:22:29.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:29.402 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-06T07:22:29.429 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:29.429 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:22:29.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:29.429 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:22:29.485 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:22:29.485 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:29.503 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:22:29.503 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:29.562 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:29.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.663+0000 7f246a6fa640 1 Processor -- start 2024-03-06T07:22:29.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.663+0000 7f246a6fa640 1 -- start start 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f246a6fa640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f246413dc10 0x7f24640b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f246a6fa640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f246415bf30 0x7f24640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f246a6fa640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246415f580 con 0x7f246415e850 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f246a6fa640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640cf3f0 con 0x7f246413dc10 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f246a6fa640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246415b210 con 0x7f246415bf30 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f2463fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f246413dc10 0x7f24640b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f2468c97640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f246415e850 0x7f246415ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39382/0 (socket says 172.21.15.72:39382) 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.664+0000 7f2468c97640 1 -- 172.21.15.72:0/2597774703 learned_addr learned my addr 172.21.15.72:0/2597774703 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:29.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f24637fe640 1 --2- 172.21.15.72:0/2597774703 >> v2:172.21.15.146:3300/0 conn(0x7f246415bf30 0x7f24640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f2463fff640 1 -- 172.21.15.72:0/2597774703 >> v2:172.21.15.146:3300/0 conn(0x7f246415bf30 msgr2=0x7f24640b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f2463fff640 1 --2- 172.21.15.72:0/2597774703 >> v2:172.21.15.146:3300/0 conn(0x7f246415bf30 0x7f24640b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f2463fff640 1 -- 172.21.15.72:0/2597774703 >> v1:172.21.15.72:6789/0 conn(0x7f246415e850 legacy=0x7f246415ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f2463fff640 1 -- 172.21.15.72:0/2597774703 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246415b450 con 0x7f246413dc10 2024-03-06T07:22:29.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f2462ffd640 1 -- 172.21.15.72:0/2597774703 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985352263 0 0) 0x7f246415f580 con 0x7f246415e850 2024-03-06T07:22:29.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f24637fe640 1 --2- 172.21.15.72:0/2597774703 >> v2:172.21.15.146:3300/0 conn(0x7f246415bf30 0x7f24640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.665+0000 7f2463fff640 1 --2- 172.21.15.72:0/2597774703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246413dc10 0x7f24640b2950 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f246406c790 tx=0x7f245802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0954398acd1abd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f2462ffd640 1 -- 172.21.15.72:0/2597774703 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24580315a0 con 0x7f246413dc10 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f2462ffd640 1 -- 172.21.15.72:0/2597774703 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458037070 con 0x7f246413dc10 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f2462ffd640 1 -- 172.21.15.72:0/2597774703 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458031960 con 0x7f246413dc10 2024-03-06T07:22:29.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 -- 172.21.15.72:0/2597774703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246413dc10 msgr2=0x7f24640b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 --2- 172.21.15.72:0/2597774703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246413dc10 0x7f24640b2950 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f246406c790 tx=0x7f245802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 -- 172.21.15.72:0/2597774703 shutdown_connections 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 --2- 172.21.15.72:0/2597774703 >> v2:172.21.15.146:3300/0 conn(0x7f246415bf30 0x7f24640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 --2- 172.21.15.72:0/2597774703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246413dc10 0x7f24640b2950 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 -- 172.21.15.72:0/2597774703 >> 172.21.15.72:0/2597774703 conn(0x7f24640cf8a0 msgr2=0x7f24640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 -- 172.21.15.72:0/2597774703 shutdown_connections 2024-03-06T07:22:29.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.666+0000 7f246a6fa640 1 -- 172.21.15.72:0/2597774703 wait complete. 2024-03-06T07:22:29.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.668+0000 7f246a6fa640 1 Processor -- start 2024-03-06T07:22:29.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.668+0000 7f246a6fa640 1 -- start start 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.669+0000 7f246a6fa640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246413dc10 0x7f2464141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.669+0000 7f246a6fa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 0x7f2464140c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.669+0000 7f246a6fa640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246415e850 0x7f24641542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.669+0000 7f246a6fa640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24640708d0 con 0x7f246415bf30 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.669+0000 7f246a6fa640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246413dff0 con 0x7f246413dc10 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.669+0000 7f246a6fa640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2464070a50 con 0x7f246415e850 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 0x7f2464140c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 0x7f2464140c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38700/0 (socket says 172.21.15.72:38700) 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 -- 172.21.15.72:0/394290426 learned_addr learned my addr 172.21.15.72:0/394290426 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f2463fff640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246413dc10 0x7f2464141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f2468c97640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246415e850 0x7f24641542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 -- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246415e850 msgr2=0x7f24641542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246415e850 0x7f24641542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 -- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246413dc10 msgr2=0x7f2464141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246413dc10 0x7f2464141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 -- 172.21.15.72:0/394290426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2464140010 con 0x7f246415bf30 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f24637fe640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 0x7f2464140c30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f245400c7f0 tx=0x7f245400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454015070 con 0x7f246415bf30 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454007a30 con 0x7f246415bf30 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24641f1b30 con 0x7f246415bf30 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454007e00 con 0x7f246415bf30 2024-03-06T07:22:29.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.670+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24641f2370 con 0x7f246415bf30 2024-03-06T07:22:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.671+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246404f330 con 0x7f246415bf30 2024-03-06T07:22:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.672+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2454018070 con 0x7f246415bf30 2024-03-06T07:22:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.672+0000 7f2460ff9640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f245006b4d0 0x7f245006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.672+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f245408a0c0 con 0x7f246415bf30 2024-03-06T07:22:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.672+0000 7f2463fff640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f245006b4d0 0x7f245006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.673+0000 7f2463fff640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f245006b4d0 0x7f245006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2458036000 tx=0x7f245803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.680+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24641f1b30 con 0x7f246415bf30 2024-03-06T07:22:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.859+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24640726c0 con 0x7f246415bf30 2024-03-06T07:22:29.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.860+0000 7f2460ff9640 1 -- 172.21.15.72:0/394290426 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f2464140010 con 0x7f246415bf30 2024-03-06T07:22:29.863 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:29.863 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:21:51.055888+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:29.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f245006b4d0 msgr2=0x7f245006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f245006b4d0 0x7f245006d990 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2458036000 tx=0x7f245803d040 comp rx=0 tx=0).stop 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 msgr2=0x7f2464140c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 0x7f2464140c30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f245400c7f0 tx=0x7f245400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 shutdown_connections 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f245006b4d0 0x7f245006d990 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246415e850 0x7f24641542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f246415bf30 0x7f2464140c30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 --2- 172.21.15.72:0/394290426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246413dc10 0x7f2464141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.862+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 >> 172.21.15.72:0/394290426 conn(0x7f24640cf8a0 msgr2=0x7f24640b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.863+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 shutdown_connections 2024-03-06T07:22:29.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.863+0000 7f246a6fa640 1 -- 172.21.15.72:0/394290426 wait complete. 2024-03-06T07:22:29.875 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:29.876 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:29.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.977+0000 7fc244833640 1 Processor -- start 2024-03-06T07:22:29.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.977+0000 7fc244833640 1 -- start start 2024-03-06T07:22:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc244833640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2401614f0 0x7fc2400c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc244833640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc2401619d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc244833640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24006cd20 con 0x7fc240161f10 2024-03-06T07:22:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc244833640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24006d000 con 0x7fc2401614f0 2024-03-06T07:22:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc244833640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24006d2e0 con 0x7fc24006c860 2024-03-06T07:22:29.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc23ed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc240161f10 0x7fc24015e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39390/0 (socket says 172.21.15.72:39390) 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.978+0000 7fc23ed9d640 1 -- 172.21.15.72:0/616464049 learned_addr learned my addr 172.21.15.72:0/616464049 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449308386 0 0) 0x7fc24006cd20 con 0x7fc240161f10 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23e59c640 1 --2- 172.21.15.72:0/616464049 >> v2:172.21.15.146:3300/0 conn(0x7fc2401614f0 0x7fc2400c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc22c003520 con 0x7fc240161f10 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23dd9b640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc2401619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1313764455 0 0) 0x7fc22c003520 con 0x7fc240161f10 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 msgr2=0x7fc2401619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc2401619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 >> v2:172.21.15.146:3300/0 conn(0x7fc2401614f0 msgr2=0x7fc2400c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 --2- 172.21.15.72:0/616464049 >> v2:172.21.15.146:3300/0 conn(0x7fc2401614f0 0x7fc2400c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24006d5c0 con 0x7fc240161f10 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fc230001670 con 0x7fc240161f10 2024-03-06T07:22:29.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 >> v1:172.21.15.72:6789/0 conn(0x7fc240161f10 legacy=0x7fc24015e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc22c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc22c003f30 0x7fc22c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc22c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24006d000 con 0x7fc22c004d40 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24006d2e0 con 0x7fc22c003f30 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c00bac0 con 0x7fc24006c860 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23ed9d640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc22c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23e59c640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc22c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23dd9b640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc22c003f30 0x7fc22c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23ed9d640 1 -- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 msgr2=0x7fc22c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23ed9d640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc22c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23ed9d640 1 -- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc22c003f30 msgr2=0x7fc22c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:29.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23ed9d640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc22c003f30 0x7fc22c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23ed9d640 1 -- 172.21.15.72:0/616464049 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c003520 con 0x7fc22c004d40 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23e59c640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc24006c860 0x7fc22c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.979+0000 7fc23dd9b640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc22c003f30 0x7fc22c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.980+0000 7fc23ed9d640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc22c0051c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc23000c240 tx=0x7fc22c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c22da9b180f334 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.980+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc230007d50 con 0x7fc22c004d40 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.980+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc230030070 con 0x7fc22c004d40 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.980+0000 7fc23d59a640 1 -- 172.21.15.72:0/616464049 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23002cde0 con 0x7fc22c004d40 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc244833640 1 -- 172.21.15.72:0/616464049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 msgr2=0x7fc22c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc244833640 1 --2- 172.21.15.72:0/616464049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc22c0051c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc23000c240 tx=0x7fc22c005830 comp rx=0 tx=0).stop 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc23ed9d640 1 -- 172.21.15.72:0/616464049 reap_dead start 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc244833640 1 -- 172.21.15.72:0/616464049 shutdown_connections 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc244833640 1 -- 172.21.15.72:0/616464049 >> 172.21.15.72:0/616464049 conn(0x7fc2400d1b40 msgr2=0x7fc2400d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc244833640 1 -- 172.21.15.72:0/616464049 shutdown_connections 2024-03-06T07:22:29.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.981+0000 7fc244833640 1 -- 172.21.15.72:0/616464049 wait complete. 2024-03-06T07:22:29.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.982+0000 7fc244833640 1 Processor -- start 2024-03-06T07:22:29.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.982+0000 7fc244833640 1 -- start start 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc244833640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc24015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc244833640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2400cb270 0x7fc2400b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc244833640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2400b7700 0x7fc24006f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc244833640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2400c7cf0 con 0x7fc22c004d40 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc244833640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2400d28f0 con 0x7fc2400b7700 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc244833640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24015c050 con 0x7fc2400cb270 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc24015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc24015bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38726/0 (socket says 172.21.15.72:38726) 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 -- 172.21.15.72:0/2264824760 learned_addr learned my addr 172.21.15.72:0/2264824760 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23ed9d640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2400b7700 0x7fc24006f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23dd9b640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2400cb270 0x7fc2400b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 -- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2400cb270 msgr2=0x7fc2400b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2400cb270 0x7fc2400b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 -- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2400b7700 msgr2=0x7fc24006f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2400b7700 0x7fc24006f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 -- 172.21.15.72:0/2264824760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2400ca1d0 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.983+0000 7fc23e59c640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc24015bb10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc2380101c0 tx=0x7fc238010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.984+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23801a030 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.984+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc238003990 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.984+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2400ca4b0 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.984+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc238003d60 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.984+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2400caa70 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.985+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24009fdf0 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.985+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc238005d10 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.986+0000 7fc236ffd640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc22006b500 0x7fc22006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.986+0000 7fc23dd9b640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc22006b500 0x7fc22006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.986+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc23808b020 con 0x7fc22c004d40 2024-03-06T07:22:29.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.986+0000 7fc23dd9b640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc22006b500 0x7fc22006d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2400b7bd0 tx=0x7fc228008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:29.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:29.990+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc238024070 con 0x7fc22c004d40 2024-03-06T07:22:30.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.207+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc240068ff0 con 0x7fc22c004d40 2024-03-06T07:22:30.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.207+0000 7fc236ffd640 1 -- 172.21.15.72:0/2264824760 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc23801de40 con 0x7fc22c004d40 2024-03-06T07:22:30.209 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:30.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc22006b500 msgr2=0x7fc22006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc22006b500 0x7fc22006d9c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc2400b7bd0 tx=0x7fc228008040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 msgr2=0x7fc24015bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc24015bb10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc2380101c0 tx=0x7fc238010680 comp rx=0 tx=0).stop 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 shutdown_connections 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc22006b500 0x7fc22006d9c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2400b7700 0x7fc24006f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2400cb270 0x7fc2400b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 --2- 172.21.15.72:0/2264824760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc22c004d40 0x7fc24015bb10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 >> 172.21.15.72:0/2264824760 conn(0x7fc2400d1b40 msgr2=0x7fc2400cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 shutdown_connections 2024-03-06T07:22:30.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.209+0000 7fc244833640 1 -- 172.21.15.72:0/2264824760 wait complete. 2024-03-06T07:22:30.222 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:30.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.324+0000 7f9d9e246640 1 Processor -- start 2024-03-06T07:22:30.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.324+0000 7f9d9e246640 1 -- start start 2024-03-06T07:22:30.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d9e246640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d98146f60 0x7f9d98147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d9e246640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d98070e20 0x7f9d9806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d9e246640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d980712b0 con 0x7f9d98146b80 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d9e246640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d980b5080 con 0x7f9d98070e20 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d9e246640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d980ad900 con 0x7f9d98146f60 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d98070e20 0x7f9d9806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d98070e20 0x7f9d9806e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38730/0 (socket says 172.21.15.72:38730) 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 -- 172.21.15.72:0/342427704 learned_addr learned my addr 172.21.15.72:0/342427704 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d8ffff640 1 --2- 172.21.15.72:0/342427704 >> v2:172.21.15.146:3300/0 conn(0x7f9d98146f60 0x7f9d98147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 -- 172.21.15.72:0/342427704 >> v2:172.21.15.146:3300/0 conn(0x7f9d98146f60 msgr2=0x7f9d98147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 --2- 172.21.15.72:0/342427704 >> v2:172.21.15.146:3300/0 conn(0x7f9d98146f60 0x7f9d98147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 -- 172.21.15.72:0/342427704 >> v1:172.21.15.146:6789/0 conn(0x7f9d98146b80 legacy=0x7f9d980b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.325+0000 7f9d97fff640 1 -- 172.21.15.72:0/342427704 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d980adbe0 con 0x7f9d98070e20 2024-03-06T07:22:30.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.326+0000 7f9d97fff640 1 --2- 172.21.15.72:0/342427704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d9806e2c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9d980b4f80 tx=0x7f9d9002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39682560633bed94 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.326+0000 7f9d96ffd640 1 -- 172.21.15.72:0/342427704 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d9002cb20 con 0x7f9d98070e20 2024-03-06T07:22:30.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.326+0000 7f9d96ffd640 1 -- 172.21.15.72:0/342427704 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9002ccc0 con 0x7f9d98070e20 2024-03-06T07:22:30.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.326+0000 7f9d96ffd640 1 -- 172.21.15.72:0/342427704 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d90034610 con 0x7f9d98070e20 2024-03-06T07:22:30.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 -- 172.21.15.72:0/342427704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 msgr2=0x7f9d9806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 --2- 172.21.15.72:0/342427704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d9806e2c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9d980b4f80 tx=0x7f9d9002bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 -- 172.21.15.72:0/342427704 shutdown_connections 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 --2- 172.21.15.72:0/342427704 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d9806e2c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 --2- 172.21.15.72:0/342427704 >> v2:172.21.15.146:3300/0 conn(0x7f9d98146f60 0x7f9d98147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 -- 172.21.15.72:0/342427704 >> 172.21.15.72:0/342427704 conn(0x7f9d9806b830 msgr2=0x7f9d980b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 -- 172.21.15.72:0/342427704 shutdown_connections 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.327+0000 7f9d9e246640 1 -- 172.21.15.72:0/342427704 wait complete. 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.328+0000 7f9d9e246640 1 Processor -- start 2024-03-06T07:22:30.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.328+0000 7f9d9e246640 1 -- start start 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d9e246640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d981f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d9e246640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d98146b80 0x7f9d9815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d9e246640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9813f9d0 0x7f9d98160fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d9e246640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9806e800 con 0x7f9d98070e20 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d9e246640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d980b5360 con 0x7f9d98146b80 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d9e246640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9806e980 con 0x7f9d9813f9d0 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d981f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d981f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38736/0 (socket says 172.21.15.72:38736) 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 -- 172.21.15.72:0/2262099683 learned_addr learned my addr 172.21.15.72:0/2262099683 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d8ffff640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d98146b80 0x7f9d9815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d97fff640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9813f9d0 0x7f9d98160fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 -- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9813f9d0 msgr2=0x7f9d98160fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9813f9d0 0x7f9d98160fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 -- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d98146b80 msgr2=0x7f9d9815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d98146b80 0x7f9d9815d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d977fe640 1 -- 172.21.15.72:0/2262099683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d98140880 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.329+0000 7f9d8ffff640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d98146b80 0x7f9d9815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.330+0000 7f9d977fe640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d981f4190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800c400 tx=0x7f9d8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.330+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d880052d0 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.330+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d88005490 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.330+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d98140b40 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.330+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d88007260 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.330+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d98141100 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.331+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9806e0b0 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.331+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d88007400 con 0x7f9d98070e20 2024-03-06T07:22:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.331+0000 7f9d94ff9640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d7806b4d0 0x7f9d7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.332+0000 7f9d8ffff640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d7806b4d0 0x7f9d7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.332+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9d880897b0 con 0x7f9d98070e20 2024-03-06T07:22:30.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.332+0000 7f9d8ffff640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d7806b4d0 0x7f9d7806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d9815ec30 tx=0x7f9d80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.336+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d880585b0 con 0x7f9d98070e20 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.512+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9d98142db0 con 0x7f9d98070e20 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.512+0000 7f9d94ff9640 1 -- 172.21.15.72:0/2262099683 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9d88057eb0 con 0x7f9d98070e20 2024-03-06T07:22:30.514 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:30.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d7806b4d0 msgr2=0x7f9d7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d7806b4d0 0x7f9d7806d990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9d9815ec30 tx=0x7f9d80008040 comp rx=0 tx=0).stop 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 msgr2=0x7f9d981f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d981f4190 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f9d8800c400 tx=0x7f9d8800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 shutdown_connections 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d7806b4d0 0x7f9d7806d990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9813f9d0 0x7f9d98160fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d98146b80 0x7f9d9815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 --2- 172.21.15.72:0/2262099683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d98070e20 0x7f9d981f4190 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 >> 172.21.15.72:0/2262099683 conn(0x7f9d9806b830 msgr2=0x7f9d98144170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 shutdown_connections 2024-03-06T07:22:30.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.514+0000 7f9d9e246640 1 -- 172.21.15.72:0/2262099683 wait complete. 2024-03-06T07:22:30.527 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:30.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.628+0000 7f4284284640 1 Processor -- start 2024-03-06T07:22:30.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.629+0000 7f4284284640 1 -- start start 2024-03-06T07:22:30.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.629+0000 7f4284284640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 0x7f427c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.629+0000 7f4284284640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0cd4b0 con 0x7f427c0d0af0 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.629+0000 7f4284284640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0cd790 con 0x7f427c15c050 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.629+0000 7f4284284640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c0cda70 con 0x7f427c06fd10 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f4282821640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f427c0d0af0 0x7f427c0ccca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39406/0 (socket says 172.21.15.72:39406) 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f4282821640 1 -- 172.21.15.72:0/2680762867 learned_addr learned my addr 172.21.15.72:0/2680762867 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428181f640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 0x7f427c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152379627 0 0) 0x7f427c0cd4b0 con 0x7f427c0d0af0 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4268003520 con 0x7f427c0d0af0 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441576939 0 0) 0x7f427c0cd790 con 0x7f427c15c050 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c0cd4b0 con 0x7f427c15c050 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1349098128 0 0) 0x7f4268003520 con 0x7f427c0d0af0 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 msgr2=0x7f427c0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 0x7f427c0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 >> v1:172.21.15.146:6789/0 conn(0x7f427c15c050 legacy=0x7f427c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0cdd50 con 0x7f427c0d0af0 2024-03-06T07:22:30.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f42740019f0 con 0x7f427c0d0af0 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 >> v1:172.21.15.72:6789/0 conn(0x7f427c0d0af0 legacy=0x7f427c0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4268003b70 0x7f4268003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 0x7f4268004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f42680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c0cda70 con 0x7f4268004d40 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426800bac0 con 0x7f4268003b70 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426800bcc0 con 0x7f427c06fd10 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f4282821640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f42680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f4282020640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4268003b70 0x7f4268003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.630+0000 7f428181f640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 0x7f4268004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.631+0000 7f4282821640 1 -- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 msgr2=0x7f4268004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.631+0000 7f4282821640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c06fd10 0x7f4268004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.631+0000 7f4282821640 1 -- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4268003b70 msgr2=0x7f4268003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.631+0000 7f4282821640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4268003b70 0x7f4268003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.631+0000 7f4282821640 1 -- 172.21.15.72:0/2680762867 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268003520 con 0x7f4268004d40 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f4282821640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f42680051c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f427400a7d0 tx=0x7f4268005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6418843c7ac2dde8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427400a430 con 0x7f4268004d40 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4274034630 con 0x7f4268004d40 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f428101e640 1 -- 172.21.15.72:0/2680762867 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274034a20 con 0x7f4268004d40 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f4284284640 1 -- 172.21.15.72:0/2680762867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 msgr2=0x7f42680051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f4284284640 1 --2- 172.21.15.72:0/2680762867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f42680051c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f427400a7d0 tx=0x7f4268005830 comp rx=0 tx=0).stop 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f4282821640 1 -- 172.21.15.72:0/2680762867 reap_dead start 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f4284284640 1 -- 172.21.15.72:0/2680762867 shutdown_connections 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.632+0000 7f4284284640 1 -- 172.21.15.72:0/2680762867 >> 172.21.15.72:0/2680762867 conn(0x7f427c0b3800 msgr2=0x7f427c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.633+0000 7f4284284640 1 -- 172.21.15.72:0/2680762867 shutdown_connections 2024-03-06T07:22:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.633+0000 7f4284284640 1 -- 172.21.15.72:0/2680762867 wait complete. 2024-03-06T07:22:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 Processor -- start 2024-03-06T07:22:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 -- start start 2024-03-06T07:22:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f427c1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0b22a0 0x7f427c0b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c09cc90 0x7f427c09d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c0ccaf0 con 0x7f4268004d40 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c0cc970 con 0x7f427c0b22a0 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4284284640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c161a90 con 0x7f427c09cc90 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4282020640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f427c1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4282020640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f427c1587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38742/0 (socket says 172.21.15.72:38742) 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.634+0000 7f4282020640 1 -- 172.21.15.72:0/3299302535 learned_addr learned my addr 172.21.15.72:0/3299302535 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f428181f640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0b22a0 0x7f427c0b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282821640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c09cc90 0x7f427c09d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282020640 1 -- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c09cc90 msgr2=0x7f427c09d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282020640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c09cc90 0x7f427c09d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282020640 1 -- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0b22a0 msgr2=0x7f427c0b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282020640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0b22a0 0x7f427c0b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282020640 1 -- 172.21.15.72:0/3299302535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c09da30 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4282020640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f427c1587d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f427800f7a0 tx=0x7f427800fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42780050d0 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4278018070 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427c09dd10 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427800d730 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.635+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c09d610 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.636+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427c0ceb60 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.637+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f427800d8d0 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.637+0000 7f4272ffd640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f426006b4d0 0x7f426006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.637+0000 7f428181f640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f426006b4d0 0x7f426006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.637+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f427808a0d0 con 0x7f4268004d40 2024-03-06T07:22:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.638+0000 7f428181f640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f426006b4d0 0x7f426006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f427c15a5e0 tx=0x7f426c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.642+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f427c0ceb60 con 0x7f4268004d40 2024-03-06T07:22:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.821+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f427c068ff0 con 0x7f4268004d40 2024-03-06T07:22:30.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.821+0000 7f4272ffd640 1 -- 172.21.15.72:0/3299302535 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4278058ed0 con 0x7f4268004d40 2024-03-06T07:22:30.823 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:30.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f426006b4d0 msgr2=0x7f426006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f426006b4d0 0x7f426006d990 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f427c15a5e0 tx=0x7f426c009190 comp rx=0 tx=0).stop 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 msgr2=0x7f427c1587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f427c1587d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f427800f7a0 tx=0x7f427800fba0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 shutdown_connections 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f426006b4d0 0x7f426006d990 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c09cc90 0x7f427c09d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c0b22a0 0x7f427c0b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 --2- 172.21.15.72:0/3299302535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4268004d40 0x7f427c1587d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 >> 172.21.15.72:0/3299302535 conn(0x7f427c0b3800 msgr2=0x7f427c0aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 shutdown_connections 2024-03-06T07:22:30.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.823+0000 7f4284284640 1 -- 172.21.15.72:0/3299302535 wait complete. 2024-03-06T07:22:30.836 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:30.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.937+0000 7f34a0cf4640 1 Processor -- start 2024-03-06T07:22:30.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.937+0000 7f34a0cf4640 1 -- start start 2024-03-06T07:22:30.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f34a0cf4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f349c15e000 0x7f349c0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f34a0cf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f349c0d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f34a0cf4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349c0d25e0 con 0x7f349c0d4810 2024-03-06T07:22:30.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f34a0cf4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c0c7c50 con 0x7f349c15e000 2024-03-06T07:22:30.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f34a0cf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0d2880 con 0x7f349c0cf7d0 2024-03-06T07:22:30.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f3493fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f349c0d4810 0x7f349c0c80e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39422/0 (socket says 172.21.15.72:39422) 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f3493fff640 1 -- 172.21.15.72:0/3600407003 learned_addr learned my addr 172.21.15.72:0/3600407003 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292763776 0 0) 0x7f349c0d25e0 con 0x7f349c0d4810 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3488003520 con 0x7f349c0d4810 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f349ad9d640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f349c0d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.938+0000 7f349a59c640 1 --2- 172.21.15.72:0/3600407003 >> v2:172.21.15.146:3300/0 conn(0x7f349c15e000 0x7f349c0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 349231728 0 0) 0x7f3488003520 con 0x7f349c0d4810 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 msgr2=0x7f349c0d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f349c0d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 >> v2:172.21.15.146:3300/0 conn(0x7f349c15e000 msgr2=0x7f349c0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 --2- 172.21.15.72:0/3600407003 >> v2:172.21.15.146:3300/0 conn(0x7f349c15e000 0x7f349c0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c0d2b40 con 0x7f349c0d4810 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3484001670 con 0x7f349c0d4810 2024-03-06T07:22:30.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 >> v1:172.21.15.72:6789/0 conn(0x7f349c0d4810 legacy=0x7f349c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34880039b0 0x7f3488003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3488004800 0x7f3488004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f34880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0c7c50 con 0x7f3488004800 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0d2880 con 0x7f34880039b0 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348800ba40 con 0x7f349c0cf7d0 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3493fff640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3488004800 0x7f3488004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f349a59c640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34880039b0 0x7f3488003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f349ad9d640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f34880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3493fff640 1 -- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 msgr2=0x7f34880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3493fff640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f34880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3493fff640 1 -- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34880039b0 msgr2=0x7f3488003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3493fff640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34880039b0 0x7f3488003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f3493fff640 1 -- 172.21.15.72:0/3600407003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3488003520 con 0x7f3488004800 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.939+0000 7f349ad9d640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f349c0cf7d0 0x7f34880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.940+0000 7f3493fff640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3488004800 0x7f3488004c80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3484001050 tx=0x7f3488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c453567f67b256 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.940+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484007ca0 con 0x7f3488004800 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.940+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3484030070 con 0x7f3488004800 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.940+0000 7f3499d9b640 1 -- 172.21.15.72:0/3600407003 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484005d10 con 0x7f3488004800 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3600407003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3488004800 msgr2=0x7f3488004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3600407003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3488004800 0x7f3488004c80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3484001050 tx=0x7f3488005830 comp rx=0 tx=0).stop 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f349ad9d640 1 -- 172.21.15.72:0/3600407003 reap_dead start 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3600407003 shutdown_connections 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3600407003 >> 172.21.15.72:0/3600407003 conn(0x7f349c0d3cf0 msgr2=0x7f349c0d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3600407003 shutdown_connections 2024-03-06T07:22:30.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.941+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3600407003 wait complete. 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.942+0000 7f34a0cf4640 1 Processor -- start 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.942+0000 7f34a0cf4640 1 -- start start 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f34a0cf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3488004800 0x7f349c150470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f34a0cf4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f349c1f7860 0x7f349c1f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f34a0cf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 0x7f349c0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f34a0cf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c15e3e0 con 0x7f349c0cbf40 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f34a0cf4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0d22d0 con 0x7f349c1f7860 2024-03-06T07:22:30.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f34a0cf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c0d0bc0 con 0x7f3488004800 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 0x7f349c0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 0x7f349c0cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38762/0 (socket says 172.21.15.72:38762) 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 -- 172.21.15.72:0/3103938102 learned_addr learned my addr 172.21.15.72:0/3103938102 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349a59c640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3488004800 0x7f349c150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f3493fff640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f349c1f7860 0x7f349c1f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 -- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3488004800 msgr2=0x7f349c150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3488004800 0x7f349c150470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 -- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f349c1f7860 msgr2=0x7f349c1f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f349c1f7860 0x7f349c1f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349ad9d640 1 -- 172.21.15.72:0/3103938102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c1509b0 con 0x7f349c0cbf40 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f349a59c640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3488004800 0x7f349c150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.943+0000 7f3493fff640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f349c1f7860 0x7f349c1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.944+0000 7f349ad9d640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 0x7f349c0cc3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f349c08dcb0 tx=0x7f3494010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.944+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494010da0 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.944+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3494003990 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.944+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c0cc900 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.944+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494003d80 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.944+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c0ccea0 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.945+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f349401e030 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.945+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c067ff0 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.945+0000 7f3492ffd640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f347c06b4d0 0x7f347c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.946+0000 7f349a59c640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f347c06b4d0 0x7f347c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.946+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f349408a8d0 con 0x7f349c0cbf40 2024-03-06T07:22:30.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.946+0000 7f349a59c640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f347c06b4d0 0x7f347c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f348c009bc0 tx=0x7f348c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:30.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:30.950+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f349c067ff0 con 0x7f349c0cbf40 2024-03-06T07:22:31.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.141+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f349c04f330 con 0x7f349c0cbf40 2024-03-06T07:22:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.142+0000 7f3492ffd640 1 -- 172.21.15.72:0/3103938102 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3494025020 con 0x7f349c0cbf40 2024-03-06T07:22:31.144 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f347c06b4d0 msgr2=0x7f347c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f347c06b4d0 0x7f347c06d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f348c009bc0 tx=0x7f348c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 msgr2=0x7f349c0cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 0x7f349c0cc3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f349c08dcb0 tx=0x7f3494010590 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 shutdown_connections 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f347c06b4d0 0x7f347c06d990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f349c0cbf40 0x7f349c0cc3c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f349c1f7860 0x7f349c1f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 --2- 172.21.15.72:0/3103938102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3488004800 0x7f349c150470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 >> 172.21.15.72:0/3103938102 conn(0x7f349c0d3cf0 msgr2=0x7f349c0c7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 shutdown_connections 2024-03-06T07:22:31.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.144+0000 7f34a0cf4640 1 -- 172.21.15.72:0/3103938102 wait complete. 2024-03-06T07:22:31.158 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:22:31.158 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:22:31.177 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:31.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.278+0000 7effc580e640 1 Processor -- start 2024-03-06T07:22:31.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.278+0000 7effc580e640 1 -- start start 2024-03-06T07:22:31.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effc580e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 0x7effc015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effc580e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7effc0162030 0x7effc01624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effc580e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc0072bb0 con 0x7effc0158600 2024-03-06T07:22:31.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effc580e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc0161ba0 con 0x7effc0162030 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effc580e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc015e8e0 con 0x7effc0072730 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effbeffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7effc0158600 0x7effc006d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39424/0 (socket says 172.21.15.72:39424) 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.279+0000 7effbeffd640 1 -- 172.21.15.72:0/1727108300 learned_addr learned my addr 172.21.15.72:0/1727108300 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903907174 0 0) 0x7effc0072bb0 con 0x7effc0158600 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effac003520 con 0x7effc0158600 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbe7fc640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 0x7effc015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbf7fe640 1 --2- 172.21.15.72:0/1727108300 >> v2:172.21.15.146:3300/0 conn(0x7effc0162030 0x7effc01624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1637680494 0 0) 0x7effac003520 con 0x7effc0158600 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 msgr2=0x7effc015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 0x7effc015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 >> v2:172.21.15.146:3300/0 conn(0x7effc0162030 msgr2=0x7effc01624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 --2- 172.21.15.72:0/1727108300 >> v2:172.21.15.146:3300/0 conn(0x7effc0162030 0x7effc01624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc015eb40 con 0x7effc0158600 2024-03-06T07:22:31.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7effb0001670 con 0x7effc0158600 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 >> v1:172.21.15.72:6789/0 conn(0x7effc0158600 legacy=0x7effc006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 0x7effac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effac004d40 0x7effac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc0161ba0 con 0x7effac003f80 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc015e8e0 con 0x7effac004d40 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac00ba70 con 0x7effc0072730 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbe7fc640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbeffd640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 0x7effac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbf7fe640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effac004d40 0x7effac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbe7fc640 1 -- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 msgr2=0x7effac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.280+0000 7effbe7fc640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc0072730 0x7effac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbe7fc640 1 -- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effac004d40 msgr2=0x7effac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbe7fc640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effac004d40 0x7effac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbe7fc640 1 -- 172.21.15.72:0/1727108300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac003520 con 0x7effac003f80 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbe7fc640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effac004800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7effa8009ac0 tx=0x7effac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13da63a67e594d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8039070 con 0x7effac003f80 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa802c6b0 con 0x7effac003f80 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.281+0000 7effbdffb640 1 -- 172.21.15.72:0/1727108300 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8009690 con 0x7effac003f80 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effc580e640 1 -- 172.21.15.72:0/1727108300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 msgr2=0x7effac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effc580e640 1 --2- 172.21.15.72:0/1727108300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effac004800 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7effa8009ac0 tx=0x7effac005830 comp rx=0 tx=0).stop 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effbf7fe640 1 -- 172.21.15.72:0/1727108300 reap_dead start 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effc580e640 1 -- 172.21.15.72:0/1727108300 shutdown_connections 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effc580e640 1 -- 172.21.15.72:0/1727108300 >> 172.21.15.72:0/1727108300 conn(0x7effc006c0c0 msgr2=0x7effc006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effc580e640 1 -- 172.21.15.72:0/1727108300 shutdown_connections 2024-03-06T07:22:31.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.282+0000 7effc580e640 1 -- 172.21.15.72:0/1727108300 wait complete. 2024-03-06T07:22:31.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.283+0000 7effc580e640 1 Processor -- start 2024-03-06T07:22:31.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.283+0000 7effc580e640 1 -- start start 2024-03-06T07:22:31.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effc580e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effc0147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effc580e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc015d610 0x7effc00b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effc580e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b11f0 0x7effc015af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effc580e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc006e080 con 0x7effac003f80 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effc580e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc015f100 con 0x7effc015d610 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effc580e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc01597f0 con 0x7effc00b11f0 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effc0147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effc0147110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38786/0 (socket says 172.21.15.72:38786) 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 -- 172.21.15.72:0/4098991373 learned_addr learned my addr 172.21.15.72:0/4098991373 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbe7fc640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc015d610 0x7effc00b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbf7fe640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b11f0 0x7effc015af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 -- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b11f0 msgr2=0x7effc015af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b11f0 0x7effc015af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 -- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc015d610 msgr2=0x7effc00b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc015d610 0x7effc00b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbeffd640 1 -- 172.21.15.72:0/4098991373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc0158be0 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.284+0000 7effbf7fe640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b11f0 0x7effc015af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.285+0000 7effbeffd640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effc0147110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7effb0007310 tx=0x7effb00077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.285+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000f8e0 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.285+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0018070 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.285+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc0158ec0 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.285+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb000fd00 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.285+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc0159480 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.286+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc004f330 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.286+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7effb001d460 con 0x7effac003f80 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.287+0000 7effb7fff640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7effa006b4d0 0x7effa006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.287+0000 7effbe7fc640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7effa006b4d0 0x7effa006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.287+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7effb008a1f0 con 0x7effac003f80 2024-03-06T07:22:31.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.287+0000 7effbe7fc640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7effa006b4d0 0x7effa006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7effa8036000 tx=0x7effa8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.291+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7effb0058ff0 con 0x7effac003f80 2024-03-06T07:22:31.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.485+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effc015d9f0 con 0x7effac003f80 2024-03-06T07:22:31.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.486+0000 7effb7fff640 1 -- 172.21.15.72:0/4098991373 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16810 (secure 0 0 0) 0x7effb00588f0 con 0x7effac003f80 2024-03-06T07:22:31.488 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:31.488 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:31.188443+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:31.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7effa006b4d0 msgr2=0x7effa006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7effa006b4d0 0x7effa006d990 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7effa8036000 tx=0x7effa8030000 comp rx=0 tx=0).stop 2024-03-06T07:22:31.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 msgr2=0x7effc0147110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effc0147110 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7effb0007310 tx=0x7effb00077d0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 shutdown_connections 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7effa006b4d0 0x7effa006d990 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effc00b11f0 0x7effc015af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effc015d610 0x7effc00b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 --2- 172.21.15.72:0/4098991373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7effac003f80 0x7effc0147110 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 >> 172.21.15.72:0/4098991373 conn(0x7effc006c0c0 msgr2=0x7effc0157740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 shutdown_connections 2024-03-06T07:22:31.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.488+0000 7effc580e640 1 -- 172.21.15.72:0/4098991373 wait complete. 2024-03-06T07:22:31.501 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:31.502 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.605+0000 7fa8df1d0640 1 Processor -- start 2024-03-06T07:22:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.605+0000 7fa8df1d0640 1 -- start start 2024-03-06T07:22:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8df1d0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa8d80d06e0 0x7fa8d815fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8df1d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa8d80d0ac0 0x7fa8d80b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8df1d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80b4d10 0x7fa8d80cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8df1d0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d80ced70 con 0x7fa8d80d06e0 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8df1d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d80d47f0 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8df1d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80d1ab0 con 0x7fa8d80b4d10 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8dcf6c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa8d80d06e0 0x7fa8d815fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8dcf6c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa8d80d06e0 0x7fa8d815fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38788/0 (socket says 172.21.15.72:38788) 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.606+0000 7fa8dcf6c640 1 -- 172.21.15.72:0/2742298937 learned_addr learned my addr 172.21.15.72:0/2742298937 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dd76d640 1 --2- 172.21.15.72:0/2742298937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80b4d10 0x7fa8d80cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8d7fff640 1 --2- 172.21.15.72:0/2742298937 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80d0ac0 0x7fa8d80b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dcf6c640 1 -- 172.21.15.72:0/2742298937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80b4d10 msgr2=0x7fa8d80cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/2742298937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80b4d10 0x7fa8d80cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dcf6c640 1 -- 172.21.15.72:0/2742298937 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80d0ac0 msgr2=0x7fa8d80b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/2742298937 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80d0ac0 0x7fa8d80b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dcf6c640 1 -- 172.21.15.72:0/2742298937 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80d1d90 con 0x7fa8d80d06e0 2024-03-06T07:22:31.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/2742298937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d06e0 0x7fa8d815fc30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80c7600 tx=0x7fa8cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7cef0988709f70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.607+0000 7fa8d77fe640 1 -- 172.21.15.72:0/2742298937 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc035020 con 0x7fa8d80d06e0 2024-03-06T07:22:31.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8d77fe640 1 -- 172.21.15.72:0/2742298937 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8cc031870 con 0x7fa8d80d06e0 2024-03-06T07:22:31.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8d77fe640 1 -- 172.21.15.72:0/2742298937 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc031c40 con 0x7fa8d80d06e0 2024-03-06T07:22:31.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 -- 172.21.15.72:0/2742298937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d06e0 msgr2=0x7fa8d815fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/2742298937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d06e0 0x7fa8d815fc30 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa8d80c7600 tx=0x7fa8cc02b910 comp rx=0 tx=0).stop 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 -- 172.21.15.72:0/2742298937 shutdown_connections 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/2742298937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80b4d10 0x7fa8d80cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/2742298937 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80d0ac0 0x7fa8d80b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/2742298937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d06e0 0x7fa8d815fc30 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 -- 172.21.15.72:0/2742298937 >> 172.21.15.72:0/2742298937 conn(0x7fa8d80d3cd0 msgr2=0x7fa8d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 -- 172.21.15.72:0/2742298937 shutdown_connections 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.608+0000 7fa8df1d0640 1 -- 172.21.15.72:0/2742298937 wait complete. 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.609+0000 7fa8df1d0640 1 Processor -- start 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.609+0000 7fa8df1d0640 1 -- start start 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8df1d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d80b4d10 0x7fa8d80cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8df1d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80d06e0 0x7fa8d814a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8df1d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 0x7fa8d814ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8df1d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80b44d0 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8df1d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80ce990 con 0x7fa8d80b4d10 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8df1d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80d0f30 con 0x7fa8d80d06e0 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 0x7fa8d814ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 0x7fa8d814ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38796/0 (socket says 172.21.15.72:38796) 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 -- 172.21.15.72:0/1149598651 learned_addr learned my addr 172.21.15.72:0/1149598651 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8d7fff640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80d06e0 0x7fa8d814a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d80b4d10 0x7fa8d80cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 -- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80d06e0 msgr2=0x7fa8d814a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80d06e0 0x7fa8d814a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 -- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d80b4d10 msgr2=0x7fa8d80cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d80b4d10 0x7fa8d80cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dd76d640 1 -- 172.21.15.72:0/1149598651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d80ccc70 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.610+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d80b4d10 0x7fa8d80cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:31.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.611+0000 7fa8dd76d640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 0x7fa8d814ad30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0007d20 tx=0x7fa8d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.611+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0014420 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.611+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d0019070 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.611+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0014820 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.611+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d814e130 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.611+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d80cd1f0 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.612+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d80a1c80 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.612+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa8d00149c0 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.613+0000 7fa8d57fa640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.613+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa8d008aa40 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.613+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.613+0000 7fa8dcf6c640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc036000 tx=0x7fa8cc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.617+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8d0059840 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.795+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8d8068ff0 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.795+0000 7fa8d57fa640 1 -- 172.21.15.72:0/1149598651 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8d005f070 con 0x7fa8d80d0ac0 2024-03-06T07:22:31.797 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:31.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa8bc06b4d0 msgr2=0x7fa8bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc036000 tx=0x7fa8cc008030 comp rx=0 tx=0).stop 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 msgr2=0x7fa8d814ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 0x7fa8d814ad30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0007d20 tx=0x7fa8d000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 shutdown_connections 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa8bc06b4d0 0x7fa8bc06d990 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8d80d0ac0 0x7fa8d814ad30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80d06e0 0x7fa8d814a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 --2- 172.21.15.72:0/1149598651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d80b4d10 0x7fa8d80cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 >> 172.21.15.72:0/1149598651 conn(0x7fa8d80d3cd0 msgr2=0x7fa8d80cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 shutdown_connections 2024-03-06T07:22:31.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.797+0000 7fa8df1d0640 1 -- 172.21.15.72:0/1149598651 wait complete. 2024-03-06T07:22:31.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:31.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.916+0000 7f74935bf640 1 Processor -- start 2024-03-06T07:22:31.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.916+0000 7f74935bf640 1 -- start start 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f74935bf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f748c1455d0 0x7f748c072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f74935bf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c06afb0 0x7f748c06b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f74935bf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c1459b0 con 0x7f748c06ded0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f74935bf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c0cfdf0 con 0x7f748c1455d0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f74935bf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c06b930 con 0x7f748c06afb0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f7491b5c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f748c06ded0 0x7f748c06e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39440/0 (socket says 172.21.15.72:39440) 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f7491b5c640 1 -- 172.21.15.72:0/795718883 learned_addr learned my addr 172.21.15.72:0/795718883 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f749135b640 1 --2- 172.21.15.72:0/795718883 >> v2:172.21.15.146:3300/0 conn(0x7f748c1455d0 0x7f748c072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f7490b5a640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c06afb0 0x7f748c06b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676936841 0 0) 0x7f748c1459b0 con 0x7f748c06ded0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.917+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7478003520 con 0x7f748c06ded0 2024-03-06T07:22:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4181627130 0 0) 0x7f7478003520 con 0x7f748c06ded0 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c06afb0 msgr2=0x7f748c06b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c06afb0 0x7f748c06b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 >> v2:172.21.15.146:3300/0 conn(0x7f748c1455d0 msgr2=0x7f748c072480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 --2- 172.21.15.72:0/795718883 >> v2:172.21.15.146:3300/0 conn(0x7f748c1455d0 0x7f748c072480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0cf1e0 con 0x7f748c06ded0 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f74840019f0 con 0x7f748c06ded0 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f749135b640 1 --2- 172.21.15.72:0/795718883 >> v2:172.21.15.146:3300/0 conn(0x7f748c1455d0 0x7f748c072480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 >> v1:172.21.15.72:6789/0 conn(0x7f748c06ded0 legacy=0x7f748c06e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74780039b0 0x7f7478003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f7478004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7478004ef0 0x7f747800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0cfdf0 con 0x7f74780043a0 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c06b930 con 0x7f7478004ef0 2024-03-06T07:22:31.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747800bf00 con 0x7f74780039b0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c06afb0 0x7f748c06b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f7478004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7491b5c640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7478004ef0 0x7f747800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f749135b640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74780039b0 0x7f7478003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 -- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74780039b0 msgr2=0x7f7478003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f74780039b0 0x7f7478003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 -- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7478004ef0 msgr2=0x7f747800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7478004ef0 0x7f747800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.918+0000 7f7490b5a640 1 -- 172.21.15.72:0/795718883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7478003520 con 0x7f74780043a0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.919+0000 7f7490b5a640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f7478004880 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f748c06f700 tx=0x7f7478005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67495b4064e81456 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.919+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c0097d0 con 0x7f74780043a0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.919+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c004030 con 0x7f74780043a0 2024-03-06T07:22:31.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.919+0000 7f7483fff640 1 -- 172.21.15.72:0/795718883 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c004400 con 0x7f74780043a0 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f74935bf640 1 -- 172.21.15.72:0/795718883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 msgr2=0x7f7478004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f74935bf640 1 --2- 172.21.15.72:0/795718883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f7478004880 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f748c06f700 tx=0x7f7478005830 comp rx=0 tx=0).stop 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f7491b5c640 1 -- 172.21.15.72:0/795718883 reap_dead start 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f74935bf640 1 -- 172.21.15.72:0/795718883 shutdown_connections 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f74935bf640 1 -- 172.21.15.72:0/795718883 >> 172.21.15.72:0/795718883 conn(0x7f748c0b2660 msgr2=0x7f748c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f74935bf640 1 -- 172.21.15.72:0/795718883 shutdown_connections 2024-03-06T07:22:31.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.920+0000 7f74935bf640 1 -- 172.21.15.72:0/795718883 wait complete. 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.921+0000 7f74935bf640 1 Processor -- start 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.921+0000 7f74935bf640 1 -- start start 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f74935bf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f748c13e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f74935bf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c139820 0x7f748c1f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f74935bf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f748c1f1cb0 0x7f748c1f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f74935bf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0cfa80 con 0x7f74780043a0 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f74935bf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0cf7a0 con 0x7f748c1f1cb0 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f74935bf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0729c0 con 0x7f748c139820 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f749135b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f748c13e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f749135b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f748c13e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38810/0 (socket says 172.21.15.72:38810) 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.922+0000 7f749135b640 1 -- 172.21.15.72:0/831961874 learned_addr learned my addr 172.21.15.72:0/831961874 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f7490b5a640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c139820 0x7f748c1f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f7491b5c640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f748c1f1cb0 0x7f748c1f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f749135b640 1 -- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c139820 msgr2=0x7f748c1f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f749135b640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c139820 0x7f748c1f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f749135b640 1 -- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f748c1f1cb0 msgr2=0x7f748c1f2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f749135b640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f748c1f1cb0 0x7f748c1f2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f749135b640 1 -- 172.21.15.72:0/831961874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c1f2670 con 0x7f74780043a0 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f7490b5a640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c139820 0x7f748c1f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f749135b640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f748c13e2c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7488011cd0 tx=0x7f748800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f748800fd20 con 0x7f74780043a0 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7488004030 con 0x7f74780043a0 2024-03-06T07:22:31.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7488004420 con 0x7f74780043a0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c13a000 con 0x7f74780043a0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.923+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c13a5c0 con 0x7f74780043a0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.924+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c13a7c0 con 0x7f74780043a0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.925+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7488012040 con 0x7f74780043a0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.925+0000 7f7481ffb640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f747006b500 0x7f747006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.925+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f748808a780 con 0x7f74780043a0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.925+0000 7f7490b5a640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f747006b500 0x7f747006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:31.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.926+0000 7f7490b5a640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f747006b500 0x7f747006d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f747c035000 tx=0x7f747c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:31.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:31.929+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f748c13a7c0 con 0x7f74780043a0 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.125+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f748c068ff0 con 0x7f74780043a0 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.125+0000 7f7481ffb640 1 -- 172.21.15.72:0/831961874 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7488019070 con 0x7f74780043a0 2024-03-06T07:22:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.127+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f747006b500 msgr2=0x7f747006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.127+0000 7f74935bf640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f747006b500 0x7f747006d9c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f747c035000 tx=0x7f747c030000 comp rx=0 tx=0).stop 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.127+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 msgr2=0x7f748c13e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.127+0000 7f74935bf640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f748c13e2c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7488011cd0 tx=0x7f748800f610 comp rx=0 tx=0).stop 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.127+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 shutdown_connections 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f747006b500 0x7f747006d9c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f748c1f1cb0 0x7f748c1f2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f748c139820 0x7f748c1f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 --2- 172.21.15.72:0/831961874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74780043a0 0x7f748c13e2c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 >> 172.21.15.72:0/831961874 conn(0x7f748c0b2660 msgr2=0x7f748c06f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 shutdown_connections 2024-03-06T07:22:32.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.128+0000 7f74935bf640 1 -- 172.21.15.72:0/831961874 wait complete. 2024-03-06T07:22:32.140 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:32.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.243+0000 7f1b27747640 1 Processor -- start 2024-03-06T07:22:32.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.243+0000 7f1b27747640 1 -- start start 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b27747640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b200cea50 0x7f1b200b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b27747640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b2006cab0 0x7f1b200ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b27747640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2006d030 con 0x7f1b2015def0 2024-03-06T07:22:32.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b27747640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b200cf1c0 con 0x7f1b2006cab0 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b27747640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b200cc760 con 0x7f1b200cea50 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b2006cab0 0x7f1b200ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b2006cab0 0x7f1b200ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38816/0 (socket says 172.21.15.72:38816) 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 -- 172.21.15.72:0/3349533894 learned_addr learned my addr 172.21.15.72:0/3349533894 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 -- 172.21.15.72:0/3349533894 >> v2:172.21.15.146:3300/0 conn(0x7f1b200cea50 msgr2=0x7f1b200b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 --2- 172.21.15.72:0/3349533894 >> v2:172.21.15.146:3300/0 conn(0x7f1b200cea50 0x7f1b200b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 -- 172.21.15.72:0/3349533894 >> v1:172.21.15.146:6789/0 conn(0x7f1b2015def0 legacy=0x7f1b200ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.244+0000 7f1b25ce4640 1 -- 172.21.15.72:0/3349533894 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2006d310 con 0x7f1b2006cab0 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.245+0000 7f1b25ce4640 1 --2- 172.21.15.72:0/3349533894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2006cab0 0x7f1b200ccbf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1b200b3660 tx=0x7f1b1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14417fcbdf54ab66 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.245+0000 7f1b17fff640 1 -- 172.21.15.72:0/3349533894 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1802ca80 con 0x7f1b2006cab0 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.245+0000 7f1b17fff640 1 -- 172.21.15.72:0/3349533894 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1802cc20 con 0x7f1b2006cab0 2024-03-06T07:22:32.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.245+0000 7f1b17fff640 1 -- 172.21.15.72:0/3349533894 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b180343f0 con 0x7f1b2006cab0 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 -- 172.21.15.72:0/3349533894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2006cab0 msgr2=0x7f1b200ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 --2- 172.21.15.72:0/3349533894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2006cab0 0x7f1b200ccbf0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1b200b3660 tx=0x7f1b1802bee0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 -- 172.21.15.72:0/3349533894 shutdown_connections 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 --2- 172.21.15.72:0/3349533894 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b2006cab0 0x7f1b200ccbf0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 --2- 172.21.15.72:0/3349533894 >> v2:172.21.15.146:3300/0 conn(0x7f1b200cea50 0x7f1b200b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 -- 172.21.15.72:0/3349533894 >> 172.21.15.72:0/3349533894 conn(0x7f1b200b3690 msgr2=0x7f1b200b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 -- 172.21.15.72:0/3349533894 shutdown_connections 2024-03-06T07:22:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.246+0000 7f1b27747640 1 -- 172.21.15.72:0/3349533894 wait complete. 2024-03-06T07:22:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.247+0000 7f1b27747640 1 Processor -- start 2024-03-06T07:22:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.247+0000 7f1b27747640 1 -- start start 2024-03-06T07:22:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b27747640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2006cab0 0x7f1b2015cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b27747640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 0x7f1b200c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b27747640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2015def0 0x7f1b200c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b27747640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2015e2d0 con 0x7f1b200cea50 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b27747640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b200cc3f0 con 0x7f1b2006cab0 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b27747640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2015e450 con 0x7f1b2015def0 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 0x7f1b200c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 0x7f1b200c5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38820/0 (socket says 172.21.15.72:38820) 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 -- 172.21.15.72:0/288217422 learned_addr learned my addr 172.21.15.72:0/288217422 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b254e3640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2006cab0 0x7f1b2015cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b25ce4640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2015def0 0x7f1b200c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 -- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2015def0 msgr2=0x7f1b200c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2015def0 0x7f1b200c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 -- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2006cab0 msgr2=0x7f1b2015cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2006cab0 0x7f1b2015cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 -- 172.21.15.72:0/288217422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200c70a0 con 0x7f1b200cea50 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b25ce4640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2015def0 0x7f1b200c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:32.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.248+0000 7f1b24ce2640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 0x7f1b200c5540 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b4a0 tx=0x7f1b1c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.249+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c014070 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.249+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c010610 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.249+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c010a00 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.249+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b200c6000 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.249+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b200c68a0 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.250+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b1c017070 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.250+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2009fcf0 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.250+0000 7f1b15ffb640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0406b4d0 0x7f1b0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.250+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1b1c089cc0 con 0x7f1b200cea50 2024-03-06T07:22:32.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.250+0000 7f1b254e3640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0406b4d0 0x7f1b0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.251+0000 7f1b254e3640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0406b4d0 0x7f1b0406d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b10006fd0 tx=0x7f1b10008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.255+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b1c058ac0 con 0x7f1b200cea50 2024-03-06T07:22:32.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.432+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1b20068ff0 con 0x7f1b200cea50 2024-03-06T07:22:32.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.433+0000 7f1b15ffb640 1 -- 172.21.15.72:0/288217422 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1b1c0583c0 con 0x7f1b200cea50 2024-03-06T07:22:32.435 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:32.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0406b4d0 msgr2=0x7f1b0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0406b4d0 0x7f1b0406d990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1b10006fd0 tx=0x7f1b10008040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 msgr2=0x7f1b200c5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 0x7f1b200c5540 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b4a0 tx=0x7f1b1c00b960 comp rx=0 tx=0).stop 2024-03-06T07:22:32.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 shutdown_connections 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0406b4d0 0x7f1b0406d990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2015def0 0x7f1b200c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b200cea50 0x7f1b200c5540 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 --2- 172.21.15.72:0/288217422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2006cab0 0x7f1b2015cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 >> 172.21.15.72:0/288217422 conn(0x7f1b200b3690 msgr2=0x7f1b20160620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 shutdown_connections 2024-03-06T07:22:32.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.435+0000 7f1b27747640 1 -- 172.21.15.72:0/288217422 wait complete. 2024-03-06T07:22:32.448 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:22:32.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.548+0000 7faa733bc640 1 Processor -- start 2024-03-06T07:22:32.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.549+0000 7faa733bc640 1 -- start start 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.549+0000 7faa733bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa6c0b4c00 0x7faa6c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.549+0000 7faa733bc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa6c15ba50 0x7faa6c15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.549+0000 7faa733bc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c15d510 con 0x7faa6c0b28a0 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.549+0000 7faa733bc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6c0b3420 con 0x7faa6c15ba50 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.549+0000 7faa733bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6c15e0b0 con 0x7faa6c0b4c00 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa71959640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa6c15ba50 0x7faa6c15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa70957640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7faa6c0b28a0 0x7faa6c0b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35324/0 (socket says 172.21.15.72:35324) 2024-03-06T07:22:32.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa70957640 1 -- 172.21.15.72:0/2340199015 learned_addr learned my addr 172.21.15.72:0/2340199015 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:32.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa71158640 1 --2- 172.21.15.72:0/2340199015 >> v2:172.21.15.146:3300/0 conn(0x7faa6c0b4c00 0x7faa6c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa71959640 1 -- 172.21.15.72:0/2340199015 >> v2:172.21.15.146:3300/0 conn(0x7faa6c0b4c00 msgr2=0x7faa6c0b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa71959640 1 --2- 172.21.15.72:0/2340199015 >> v2:172.21.15.146:3300/0 conn(0x7faa6c0b4c00 0x7faa6c0b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa71959640 1 -- 172.21.15.72:0/2340199015 >> v1:172.21.15.72:6789/0 conn(0x7faa6c0b28a0 legacy=0x7faa6c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa71959640 1 -- 172.21.15.72:0/2340199015 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c15d780 con 0x7faa6c15ba50 2024-03-06T07:22:32.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.550+0000 7faa63fff640 1 -- 172.21.15.72:0/2340199015 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839507509 0 0) 0x7faa6c15d510 con 0x7faa6c0b28a0 2024-03-06T07:22:32.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa71959640 1 --2- 172.21.15.72:0/2340199015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c15bec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0721d0 tx=0x7faa6402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d773cc2b88f6b490 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa63fff640 1 -- 172.21.15.72:0/2340199015 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64039070 con 0x7faa6c15ba50 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa63fff640 1 -- 172.21.15.72:0/2340199015 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6402cd00 con 0x7faa6c15ba50 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa63fff640 1 -- 172.21.15.72:0/2340199015 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64034660 con 0x7faa6c15ba50 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 -- 172.21.15.72:0/2340199015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 msgr2=0x7faa6c15bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 --2- 172.21.15.72:0/2340199015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c15bec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0721d0 tx=0x7faa6402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 -- 172.21.15.72:0/2340199015 shutdown_connections 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 --2- 172.21.15.72:0/2340199015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c15bec0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 --2- 172.21.15.72:0/2340199015 >> v2:172.21.15.146:3300/0 conn(0x7faa6c0b4c00 0x7faa6c0b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 -- 172.21.15.72:0/2340199015 >> 172.21.15.72:0/2340199015 conn(0x7faa6c09a9f0 msgr2=0x7faa6c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 -- 172.21.15.72:0/2340199015 shutdown_connections 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.551+0000 7faa733bc640 1 -- 172.21.15.72:0/2340199015 wait complete. 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.552+0000 7faa733bc640 1 Processor -- start 2024-03-06T07:22:32.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.552+0000 7faa733bc640 1 -- start start 2024-03-06T07:22:32.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa733bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa6c0b28a0 0x7faa6c14a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa733bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa6c0b4c00 0x7faa6c154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa733bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa733bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c15f550 con 0x7faa6c15ba50 2024-03-06T07:22:32.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa733bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c15dd40 con 0x7faa6c0b4c00 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa733bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c15f6d0 con 0x7faa6c0b28a0 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa71959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa71959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60572/0 (socket says 172.21.15.72:60572) 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa71959640 1 -- 172.21.15.72:0/42146812 learned_addr learned my addr 172.21.15.72:0/42146812 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.553+0000 7faa71158640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa6c0b28a0 0x7faa6c14a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa71959640 1 -- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa6c0b28a0 msgr2=0x7faa6c14a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa70957640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa6c0b4c00 0x7faa6c154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa71959640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa6c0b28a0 0x7faa6c14a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa71959640 1 -- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa6c0b4c00 msgr2=0x7faa6c154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa71959640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa6c0b4c00 0x7faa6c154730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa71959640 1 -- 172.21.15.72:0/42146812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c1528d0 con 0x7faa6c15ba50 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa71959640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c154c70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7faa6402c560 tx=0x7faa64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64039050 con 0x7faa6c15ba50 2024-03-06T07:22:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa64045070 con 0x7faa6c15ba50 2024-03-06T07:22:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c1525f0 con 0x7faa6c15ba50 2024-03-06T07:22:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6403e830 con 0x7faa6c15ba50 2024-03-06T07:22:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.554+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c0af6b0 con 0x7faa6c15ba50 2024-03-06T07:22:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.555+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c04f330 con 0x7faa6c15ba50 2024-03-06T07:22:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.556+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa6403e9d0 con 0x7faa6c15ba50 2024-03-06T07:22:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.556+0000 7faa61ffb640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa5006b4d0 0x7faa5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.556+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faa640ad000 con 0x7faa6c15ba50 2024-03-06T07:22:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.556+0000 7faa71158640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa5006b4d0 0x7faa5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.557+0000 7faa71158640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa5006b4d0 0x7faa5006d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faa68006fd0 tx=0x7faa68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.560+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa64032e20 con 0x7faa6c15ba50 2024-03-06T07:22:32.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.750+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faa6c15cf10 con 0x7faa6c15ba50 2024-03-06T07:22:32.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.751+0000 7faa61ffb640 1 -- 172.21.15.72:0/42146812 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7faa6407be00 con 0x7faa6c15ba50 2024-03-06T07:22:32.753 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:22:32.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa5006b4d0 msgr2=0x7faa5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa5006b4d0 0x7faa5006d990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7faa68006fd0 tx=0x7faa68008040 comp rx=0 tx=0).stop 2024-03-06T07:22:32.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 msgr2=0x7faa6c154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c154c70 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7faa6402c560 tx=0x7faa64004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 shutdown_connections 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa5006b4d0 0x7faa5006d990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa6c15ba50 0x7faa6c154c70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.752+0000 7faa733bc640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa6c0b4c00 0x7faa6c154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.753+0000 7faa733bc640 1 --2- 172.21.15.72:0/42146812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa6c0b28a0 0x7faa6c14a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.753+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 >> 172.21.15.72:0/42146812 conn(0x7faa6c09a9f0 msgr2=0x7faa6c15be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.753+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 shutdown_connections 2024-03-06T07:22:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.753+0000 7faa733bc640 1 -- 172.21.15.72:0/42146812 wait complete. 2024-03-06T07:22:32.765 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-03-06T07:22:32.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:32.766 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-06T07:22:32.800 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:32.800 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:22:32.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:32.800 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:22:32.857 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:22:32.858 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:32.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.958+0000 7efdceac2640 1 Processor -- start 2024-03-06T07:22:32.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.958+0000 7efdceac2640 1 -- start start 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.958+0000 7efdceac2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efdc815fa10 0x7efdc80ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.958+0000 7efdceac2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80cebe0 con 0x7efdc815bdd0 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.958+0000 7efdceac2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc80ceec0 con 0x7efdc80d16e0 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.958+0000 7efdceac2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdc80cf1a0 con 0x7efdc815fa10 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdc7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efdc815fa10 0x7efdc80ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdcc85e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efdc815bdd0 0x7efdc80c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35338/0 (socket says 172.21.15.72:35338) 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdcc85e640 1 -- 172.21.15.72:0/675077636 learned_addr learned my addr 172.21.15.72:0/675077636 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdc7fff640 1 -- 172.21.15.72:0/675077636 >> v1:172.21.15.146:6789/0 conn(0x7efdc80d16e0 legacy=0x7efdc80d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdc7fff640 1 -- 172.21.15.72:0/675077636 >> v1:172.21.15.72:6789/0 conn(0x7efdc815bdd0 legacy=0x7efdc80c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdc77fe640 1 -- 172.21.15.72:0/675077636 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570076158 0 0) 0x7efdc80cebe0 con 0x7efdc815bdd0 2024-03-06T07:22:32.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.959+0000 7efdc7fff640 1 -- 172.21.15.72:0/675077636 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc80cc8e0 con 0x7efdc815fa10 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdc7fff640 1 --2- 172.21.15.72:0/675077636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815fa10 0x7efdc80ce270 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efdb40097a0 tx=0x7efdb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21dbd5c747f1468d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdc77fe640 1 -- 172.21.15.72:0/675077636 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40315a0 con 0x7efdc815fa10 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdc77fe640 1 -- 172.21.15.72:0/675077636 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb4037070 con 0x7efdc815fa10 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdc77fe640 1 -- 172.21.15.72:0/675077636 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4031960 con 0x7efdc815fa10 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdceac2640 1 -- 172.21.15.72:0/675077636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815fa10 msgr2=0x7efdc80ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdceac2640 1 --2- 172.21.15.72:0/675077636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815fa10 0x7efdc80ce270 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7efdb40097a0 tx=0x7efdb402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdceac2640 1 -- 172.21.15.72:0/675077636 shutdown_connections 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdceac2640 1 --2- 172.21.15.72:0/675077636 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815fa10 0x7efdc80ce270 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.960+0000 7efdceac2640 1 -- 172.21.15.72:0/675077636 >> 172.21.15.72:0/675077636 conn(0x7efdc809a9f0 msgr2=0x7efdc809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:32.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.961+0000 7efdceac2640 1 -- 172.21.15.72:0/675077636 shutdown_connections 2024-03-06T07:22:32.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.961+0000 7efdceac2640 1 -- 172.21.15.72:0/675077636 wait complete. 2024-03-06T07:22:32.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.962+0000 7efdceac2640 1 Processor -- start 2024-03-06T07:22:32.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.962+0000 7efdceac2640 1 -- start start 2024-03-06T07:22:32.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efdc80d16e0 0x7efdc814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 0x7efdc80afd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efdc815fa10 0x7efdc814f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80ce7b0 con 0x7efdc815bdd0 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc815c1b0 con 0x7efdc815fa10 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc80ce930 con 0x7efdc80d16e0 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 0x7efdc80afd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 0x7efdc80afd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60586/0 (socket says 172.21.15.72:60586) 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 -- 172.21.15.72:0/2314916195 learned_addr learned my addr 172.21.15.72:0/2314916195 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdcd05f640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efdc815fa10 0x7efdc814f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdcc85e640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efdc80d16e0 0x7efdc814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 -- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efdc80d16e0 msgr2=0x7efdc814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efdc80d16e0 0x7efdc814ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 -- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efdc815fa10 msgr2=0x7efdc814f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efdc815fa10 0x7efdc814f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 -- 172.21.15.72:0/2314916195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc814e720 con 0x7efdc815bdd0 2024-03-06T07:22:32.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdcc85e640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efdc80d16e0 0x7efdc814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc7fff640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 0x7efdc80afd80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efdb40097a0 tx=0x7efdb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4004230 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb40043f0 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc814ea00 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.964+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4032880 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.965+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc814efc0 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.966+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc804f330 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.966+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efdb4032a20 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.966+0000 7efdc57fa640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efdac06b4d0 0x7efdac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.966+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efdb402f030 con 0x7efdc815bdd0 2024-03-06T07:22:32.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.966+0000 7efdcc85e640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efdac06b4d0 0x7efdac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:32.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.967+0000 7efdcc85e640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efdac06b4d0 0x7efdac06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:32.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:32.971+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efdb4041070 con 0x7efdc815bdd0 2024-03-06T07:22:33.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.208+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdc80d06e0 con 0x7efdc815bdd0 2024-03-06T07:22:33.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.208+0000 7efdc57fa640 1 -- 172.21.15.72:0/2314916195 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7efdb4044340 con 0x7efdc815bdd0 2024-03-06T07:22:33.210 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:33.210 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:22:33.211 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:22:33.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efdac06b4d0 msgr2=0x7efdac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efdac06b4d0 0x7efdac06d990 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efdbc007fd0 tx=0x7efdbc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 msgr2=0x7efdc80afd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 0x7efdc80afd80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7efdb40097a0 tx=0x7efdb40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 shutdown_connections 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efdac06b4d0 0x7efdac06d990 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efdc815fa10 0x7efdc814f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdc815bdd0 0x7efdc80afd80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 --2- 172.21.15.72:0/2314916195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efdc80d16e0 0x7efdc814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 >> 172.21.15.72:0/2314916195 conn(0x7efdc809a9f0 msgr2=0x7efdc809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 shutdown_connections 2024-03-06T07:22:33.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.210+0000 7efdceac2640 1 -- 172.21.15.72:0/2314916195 wait complete. 2024-03-06T07:22:33.227 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.326+0000 7ff0bb864640 1 Processor -- start 2024-03-06T07:22:33.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.326+0000 7ff0bb864640 1 -- start start 2024-03-06T07:22:33.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0bb864640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b415dfe0 0x7ff0b40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0bb864640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0b40d29c0 0x7ff0b40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0bb864640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b40b3430 con 0x7ff0b40cf830 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0bb864640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b40b31e0 con 0x7ff0b40d29c0 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0bb864640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40b3d60 con 0x7ff0b415dfe0 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b8dff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff0b40cf830 0x7ff0b40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:41922/0 (socket says 172.21.15.72:41922) 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b415dfe0 0x7ff0b40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b8dff640 1 -- 172.21.15.72:0/561858307 learned_addr learned my addr 172.21.15.72:0/561858307 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b9e01640 1 --2- 172.21.15.72:0/561858307 >> v2:172.21.15.146:3300/0 conn(0x7ff0b40d29c0 0x7ff0b40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0abfff640 1 -- 172.21.15.72:0/561858307 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967206806 0 0) 0x7ff0b40b3430 con 0x7ff0b40cf830 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0abfff640 1 -- 172.21.15.72:0/561858307 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a0003520 con 0x7ff0b40cf830 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b9600640 1 -- 172.21.15.72:0/561858307 >> v2:172.21.15.146:3300/0 conn(0x7ff0b40d29c0 msgr2=0x7ff0b40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b9600640 1 --2- 172.21.15.72:0/561858307 >> v2:172.21.15.146:3300/0 conn(0x7ff0b40d29c0 0x7ff0b40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b9600640 1 -- 172.21.15.72:0/561858307 >> v1:172.21.15.146:6789/0 conn(0x7ff0b40cf830 legacy=0x7ff0b40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.327+0000 7ff0b9600640 1 -- 172.21.15.72:0/561858307 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b40b3710 con 0x7ff0b415dfe0 2024-03-06T07:22:33.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.328+0000 7ff0b9600640 1 --2- 172.21.15.72:0/561858307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b415dfe0 0x7ff0b40cf2f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00097a0 tx=0x7ff0b002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1949f89ca99a8ffa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.328+0000 7ff0abfff640 1 -- 172.21.15.72:0/561858307 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0035020 con 0x7ff0b415dfe0 2024-03-06T07:22:33.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.328+0000 7ff0abfff640 1 -- 172.21.15.72:0/561858307 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b0031830 con 0x7ff0b415dfe0 2024-03-06T07:22:33.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0abfff640 1 -- 172.21.15.72:0/561858307 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0031be0 con 0x7ff0b415dfe0 2024-03-06T07:22:33.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 -- 172.21.15.72:0/561858307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b415dfe0 msgr2=0x7ff0b40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 --2- 172.21.15.72:0/561858307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b415dfe0 0x7ff0b40cf2f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00097a0 tx=0x7ff0b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 -- 172.21.15.72:0/561858307 shutdown_connections 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 --2- 172.21.15.72:0/561858307 >> v2:172.21.15.146:3300/0 conn(0x7ff0b40d29c0 0x7ff0b40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 --2- 172.21.15.72:0/561858307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b415dfe0 0x7ff0b40cf2f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 -- 172.21.15.72:0/561858307 >> 172.21.15.72:0/561858307 conn(0x7ff0b40d3cd0 msgr2=0x7ff0b40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 -- 172.21.15.72:0/561858307 shutdown_connections 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.329+0000 7ff0bb864640 1 -- 172.21.15.72:0/561858307 wait complete. 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.330+0000 7ff0bb864640 1 Processor -- start 2024-03-06T07:22:33.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.330+0000 7ff0bb864640 1 -- start start 2024-03-06T07:22:33.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0bb864640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b40cf830 0x7ff0b41f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0bb864640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 0x7ff0b40b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0bb864640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b415dfe0 0x7ff0b40b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0bb864640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40d1df0 con 0x7ff0b40d29c0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0bb864640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b40d1c70 con 0x7ff0b415dfe0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0bb864640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b415e3c0 con 0x7ff0b40cf830 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0b8dff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 0x7ff0b40b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0b8dff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 0x7ff0b40b10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60588/0 (socket says 172.21.15.72:60588) 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0b8dff640 1 -- 172.21.15.72:0/2718331417 learned_addr learned my addr 172.21.15.72:0/2718331417 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0b9e01640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b415dfe0 0x7ff0b40b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.331+0000 7ff0b9600640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b40cf830 0x7ff0b41f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0b8dff640 1 -- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b40cf830 msgr2=0x7ff0b41f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0b8dff640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b40cf830 0x7ff0b41f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0b8dff640 1 -- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b415dfe0 msgr2=0x7ff0b40b1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0b8dff640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b415dfe0 0x7ff0b40b1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0b8dff640 1 -- 172.21.15.72:0/2718331417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4160030 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0b8dff640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 0x7ff0b40b10f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400c400 tx=0x7ff0a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4017070 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a4005d00 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b4160310 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4007310 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.332+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b41608d0 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.333+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b404f330 con 0x7ff0b40d29c0 2024-03-06T07:22:33.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.334+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0a401a070 con 0x7ff0b40d29c0 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.334+0000 7ff0a9ffb640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09806b4d0 0x7ff09806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.334+0000 7ff0b9600640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09806b4d0 0x7ff09806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.334+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff0a4089660 con 0x7ff0b40d29c0 2024-03-06T07:22:33.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.335+0000 7ff0b9600640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09806b4d0 0x7ff09806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0036000 tx=0x7ff0b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.339+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0a4057e10 con 0x7ff0b40d29c0 2024-03-06T07:22:33.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.512+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0b40b2cd0 con 0x7ff0b40d29c0 2024-03-06T07:22:33.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.513+0000 7ff0a9ffb640 1 -- 172.21.15.72:0/2718331417 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16810 (secure 0 0 0) 0x7ff0a405beb0 con 0x7ff0b40d29c0 2024-03-06T07:22:33.517 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:33.517 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:31.188443+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09806b4d0 msgr2=0x7ff09806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09806b4d0 0x7ff09806d990 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0036000 tx=0x7ff0b003c040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 msgr2=0x7ff0b40b10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 0x7ff0b40b10f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400c400 tx=0x7ff0a400c8c0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 shutdown_connections 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09806b4d0 0x7ff09806d990 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b415dfe0 0x7ff0b40b1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0b40d29c0 0x7ff0b40b10f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 --2- 172.21.15.72:0/2718331417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b40cf830 0x7ff0b41f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 >> 172.21.15.72:0/2718331417 conn(0x7ff0b40d3cd0 msgr2=0x7ff0b406cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 shutdown_connections 2024-03-06T07:22:33.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.516+0000 7ff0bb864640 1 -- 172.21.15.72:0/2718331417 wait complete. 2024-03-06T07:22:33.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:21:47.857029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:21:48.376009+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:33.534 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:33.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.636+0000 7feb329f7640 1 Processor -- start 2024-03-06T07:22:33.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.636+0000 7feb329f7640 1 -- start start 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb329f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c1614d0 0x7feb2c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb329f7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feb2c0d1910 0x7feb2c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb329f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feb2c0d22b0 0x7feb2c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb329f7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c0d2740 con 0x7feb2c0d1910 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb329f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb2c1618b0 con 0x7feb2c0d22b0 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb329f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0cf400 con 0x7feb2c1614d0 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feb2c0d1910 0x7feb2c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feb2c0d1910 0x7feb2c0d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60592/0 (socket says 172.21.15.72:60592) 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 -- 172.21.15.72:0/3296569355 learned_addr learned my addr 172.21.15.72:0/3296569355 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:33.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2bfff640 1 --2- 172.21.15.72:0/3296569355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c1614d0 0x7feb2c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb30f94640 1 --2- 172.21.15.72:0/3296569355 >> v2:172.21.15.146:3300/0 conn(0x7feb2c0d22b0 0x7feb2c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 -- 172.21.15.72:0/3296569355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c1614d0 msgr2=0x7feb2c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 --2- 172.21.15.72:0/3296569355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c1614d0 0x7feb2c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 -- 172.21.15.72:0/3296569355 >> v2:172.21.15.146:3300/0 conn(0x7feb2c0d22b0 msgr2=0x7feb2c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 --2- 172.21.15.72:0/3296569355 >> v2:172.21.15.146:3300/0 conn(0x7feb2c0d22b0 0x7feb2c0d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb2b7fe640 1 -- 172.21.15.72:0/3296569355 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0cf6a0 con 0x7feb2c0d1910 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.637+0000 7feb30f94640 1 --2- 172.21.15.72:0/3296569355 >> v2:172.21.15.146:3300/0 conn(0x7feb2c0d22b0 0x7feb2c0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.638+0000 7feb2b7fe640 1 --2- 172.21.15.72:0/3296569355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c0d1910 0x7feb2c0d1d70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0b57d0 tx=0x7feb2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce7e842c6e534bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.638+0000 7feb2affd640 1 -- 172.21.15.72:0/3296569355 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20035020 con 0x7feb2c0d1910 2024-03-06T07:22:33.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.638+0000 7feb2affd640 1 -- 172.21.15.72:0/3296569355 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb20031870 con 0x7feb2c0d1910 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.638+0000 7feb2affd640 1 -- 172.21.15.72:0/3296569355 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb20031c40 con 0x7feb2c0d1910 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 -- 172.21.15.72:0/3296569355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c0d1910 msgr2=0x7feb2c0d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 --2- 172.21.15.72:0/3296569355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c0d1910 0x7feb2c0d1d70 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7feb2c0b57d0 tx=0x7feb2002b910 comp rx=0 tx=0).stop 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 -- 172.21.15.72:0/3296569355 shutdown_connections 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 --2- 172.21.15.72:0/3296569355 >> v2:172.21.15.146:3300/0 conn(0x7feb2c0d22b0 0x7feb2c0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 --2- 172.21.15.72:0/3296569355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c0d1910 0x7feb2c0d1d70 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 --2- 172.21.15.72:0/3296569355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c1614d0 0x7feb2c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 -- 172.21.15.72:0/3296569355 >> 172.21.15.72:0/3296569355 conn(0x7feb2c0d3d40 msgr2=0x7feb2c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 -- 172.21.15.72:0/3296569355 shutdown_connections 2024-03-06T07:22:33.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.639+0000 7feb329f7640 1 -- 172.21.15.72:0/3296569355 wait complete. 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.640+0000 7feb329f7640 1 Processor -- start 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.640+0000 7feb329f7640 1 -- start start 2024-03-06T07:22:33.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c0d1910 0x7feb2c15d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb2c0d22b0 0x7feb2c0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 0x7feb2c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0b0910 con 0x7feb2c1614d0 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0cfc60 con 0x7feb2c0d22b0 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c0d0be0 con 0x7feb2c0d1910 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 0x7feb2c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 0x7feb2c0cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60604/0 (socket says 172.21.15.72:60604) 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 -- 172.21.15.72:0/143869925 learned_addr learned my addr 172.21.15.72:0/143869925 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb2bfff640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c0d1910 0x7feb2c15d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb2b7fe640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb2c0d22b0 0x7feb2c0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 -- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c0d1910 msgr2=0x7feb2c15d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c0d1910 0x7feb2c15d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 -- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb2c0d22b0 msgr2=0x7feb2c0b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb2c0d22b0 0x7feb2c0b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 -- 172.21.15.72:0/143869925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c0a1260 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb30f94640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 0x7feb2c0cae10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7feb2400cef0 tx=0x7feb2400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2401a070 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb2400d860 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2400dc50 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c0a1540 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.641+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c0cd1c0 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.642+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c0ccfe0 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.643+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb2401d070 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.643+0000 7feb28ff9640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb1806b4d0 0x7feb1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.644+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feb2408aa10 con 0x7feb2c1614d0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.644+0000 7feb2bfff640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb1806b4d0 0x7feb1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.644+0000 7feb2bfff640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb1806b4d0 0x7feb1806d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feb1c007920 tx=0x7feb1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.648+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb2c0ccfe0 con 0x7feb2c1614d0 2024-03-06T07:22:33.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.826+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feb2c068ff0 con 0x7feb2c1614d0 2024-03-06T07:22:33.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.826+0000 7feb28ff9640 1 -- 172.21.15.72:0/143869925 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7feb24059310 con 0x7feb2c1614d0 2024-03-06T07:22:33.828 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.828+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb1806b4d0 msgr2=0x7feb1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.828+0000 7feb329f7640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb1806b4d0 0x7feb1806d990 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feb1c007920 tx=0x7feb1c008040 comp rx=0 tx=0).stop 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.828+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 msgr2=0x7feb2c0cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.828+0000 7feb329f7640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 0x7feb2c0cae10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7feb2400cef0 tx=0x7feb2400a710 comp rx=0 tx=0).stop 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 shutdown_connections 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb1806b4d0 0x7feb1806d990 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb2c1614d0 0x7feb2c0cae10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb2c0d22b0 0x7feb2c0b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 --2- 172.21.15.72:0/143869925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb2c0d1910 0x7feb2c15d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 >> 172.21.15.72:0/143869925 conn(0x7feb2c0d3d40 msgr2=0x7feb2c0b3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 shutdown_connections 2024-03-06T07:22:33.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.829+0000 7feb329f7640 1 -- 172.21.15.72:0/143869925 wait complete. 2024-03-06T07:22:33.842 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:22:33.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 Processor -- start 2024-03-06T07:22:33.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 -- start start 2024-03-06T07:22:33.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e2815bdd0 0x7f7e280b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280b15c0 0x7f7e280b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e280d0240 con 0x7f7e280ce300 2024-03-06T07:22:33.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e280cf4f0 con 0x7f7e2815bdd0 2024-03-06T07:22:33.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.944+0000 7f7e2e6af640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280d0850 con 0x7f7e280b15c0 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e27fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e2815bdd0 0x7f7e280b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e277fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e280ce300 0x7f7e280ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35346/0 (socket says 172.21.15.72:35346) 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e277fe640 1 -- 172.21.15.72:0/914322536 learned_addr learned my addr 172.21.15.72:0/914322536 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e2cc4c640 1 --2- 172.21.15.72:0/914322536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280b15c0 0x7f7e280b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e27fff640 1 -- 172.21.15.72:0/914322536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280b15c0 msgr2=0x7f7e280b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e27fff640 1 --2- 172.21.15.72:0/914322536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280b15c0 0x7f7e280b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e27fff640 1 -- 172.21.15.72:0/914322536 >> v1:172.21.15.72:6789/0 conn(0x7f7e280ce300 legacy=0x7f7e280ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.945+0000 7f7e27fff640 1 -- 172.21.15.72:0/914322536 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280cf210 con 0x7f7e2815bdd0 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.946+0000 7f7e27fff640 1 --2- 172.21.15.72:0/914322536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2815bdd0 0x7f7e280b41b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097a0 tx=0x7f7e1802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a504c3ff23f20c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.946+0000 7f7e26ffd640 1 -- 172.21.15.72:0/914322536 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18037070 con 0x7f7e2815bdd0 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.946+0000 7f7e26ffd640 1 -- 172.21.15.72:0/914322536 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e18030710 con 0x7f7e2815bdd0 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.946+0000 7f7e26ffd640 1 -- 172.21.15.72:0/914322536 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18030ae0 con 0x7f7e2815bdd0 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 -- 172.21.15.72:0/914322536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2815bdd0 msgr2=0x7f7e280b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/914322536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2815bdd0 0x7f7e280b41b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097a0 tx=0x7f7e1802b710 comp rx=0 tx=0).stop 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 -- 172.21.15.72:0/914322536 shutdown_connections 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/914322536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e280b15c0 0x7f7e280b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/914322536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2815bdd0 0x7f7e280b41b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 -- 172.21.15.72:0/914322536 >> 172.21.15.72:0/914322536 conn(0x7f7e280b3690 msgr2=0x7f7e280b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 -- 172.21.15.72:0/914322536 shutdown_connections 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.947+0000 7f7e2e6af640 1 -- 172.21.15.72:0/914322536 wait complete. 2024-03-06T07:22:33.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.948+0000 7f7e2e6af640 1 Processor -- start 2024-03-06T07:22:33.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.948+0000 7f7e2e6af640 1 -- start start 2024-03-06T07:22:33.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280b15c0 0x7f7e2814df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 0x7f7e280b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2815bdd0 0x7f7e280c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2815c1b0 con 0x7f7e280ce300 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280ceea0 con 0x7f7e280b15c0 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e280b10c0 con 0x7f7e2815bdd0 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 0x7f7e280b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 0x7f7e280b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60618/0 (socket says 172.21.15.72:60618) 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 -- 172.21.15.72:0/3969423050 learned_addr learned my addr 172.21.15.72:0/3969423050 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 -- 172.21.15.72:0/3969423050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2815bdd0 msgr2=0x7f7e280c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2815bdd0 0x7f7e280c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 -- 172.21.15.72:0/3969423050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280b15c0 msgr2=0x7f7e2814df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280b15c0 0x7f7e2814df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 -- 172.21.15.72:0/3969423050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e280cb990 con 0x7f7e280ce300 2024-03-06T07:22:33.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e277fe640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 0x7f7e280b5300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00c7b0 tx=0x7f7e1c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c018070 con 0x7f7e280ce300 2024-03-06T07:22:33.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c0072d0 con 0x7f7e280ce300 2024-03-06T07:22:33.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c0047e0 con 0x7f7e280ce300 2024-03-06T07:22:33.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e280cb6b0 con 0x7f7e280ce300 2024-03-06T07:22:33.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.949+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e280b4b00 con 0x7f7e280ce300 2024-03-06T07:22:33.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.951+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e1c004980 con 0x7f7e280ce300 2024-03-06T07:22:33.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.951+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2809fcf0 con 0x7f7e280ce300 2024-03-06T07:22:33.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.951+0000 7f7e24ff9640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:33.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.951+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7e1c089750 con 0x7f7e280ce300 2024-03-06T07:22:33.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.951+0000 7f7e27fff640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:33.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.952+0000 7f7e27fff640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7e18034000 tx=0x7f7e1802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:33.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:33.955+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e1c058550 con 0x7f7e280ce300 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.199+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e28068ff0 con 0x7f7e280ce300 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.200+0000 7f7e24ff9640 1 -- 172.21.15.72:0/3969423050 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e1c057e50 con 0x7f7e280ce300 2024-03-06T07:22:34.202 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 msgr2=0x7f7e1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7e18034000 tx=0x7f7e1802f000 comp rx=0 tx=0).stop 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 msgr2=0x7f7e280b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 0x7f7e280b5300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00c7b0 tx=0x7f7e1c00cc70 comp rx=0 tx=0).stop 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 shutdown_connections 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e1406b4d0 0x7f7e1406d990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2815bdd0 0x7f7e280c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e280ce300 0x7f7e280b5300 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 --2- 172.21.15.72:0/3969423050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e280b15c0 0x7f7e2814df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 >> 172.21.15.72:0/3969423050 conn(0x7f7e280b3690 msgr2=0x7f7e2806f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 shutdown_connections 2024-03-06T07:22:34.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.202+0000 7f7e2e6af640 1 -- 172.21.15.72:0/3969423050 wait complete. 2024-03-06T07:22:34.215 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:22:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.316+0000 7f87ba908640 1 Processor -- start 2024-03-06T07:22:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.316+0000 7f87ba908640 1 -- start start 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87ba908640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87b415c050 0x7f87b406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87ba908640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f87b406fb40 0x7f87b40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87ba908640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b40cd4b0 con 0x7f87b40ccca0 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87ba908640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b406ffb0 con 0x7f87b415c050 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87ba908640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b40cd790 con 0x7f87b406fb40 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87b3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87b415c050 0x7f87b406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87b3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87b415c050 0x7f87b406f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60620/0 (socket says 172.21.15.72:60620) 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87b8ea5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f87b40ccca0 0x7f87b40cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35354/0 (socket says 172.21.15.72:35354) 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.317+0000 7f87b3fff640 1 -- 172.21.15.72:0/626873616 learned_addr learned my addr 172.21.15.72:0/626873616 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b37fe640 1 --2- 172.21.15.72:0/626873616 >> v2:172.21.15.146:3300/0 conn(0x7f87b406fb40 0x7f87b40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b3fff640 1 -- 172.21.15.72:0/626873616 >> v2:172.21.15.146:3300/0 conn(0x7f87b406fb40 msgr2=0x7f87b40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b3fff640 1 --2- 172.21.15.72:0/626873616 >> v2:172.21.15.146:3300/0 conn(0x7f87b406fb40 0x7f87b40cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b3fff640 1 -- 172.21.15.72:0/626873616 >> v1:172.21.15.72:6789/0 conn(0x7f87b40ccca0 legacy=0x7f87b40cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b3fff640 1 -- 172.21.15.72:0/626873616 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b40cda70 con 0x7f87b415c050 2024-03-06T07:22:34.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b2ffd640 1 -- 172.21.15.72:0/626873616 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289244903 0 0) 0x7f87b40cd4b0 con 0x7f87b40ccca0 2024-03-06T07:22:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.318+0000 7f87b3fff640 1 --2- 172.21.15.72:0/626873616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b415c050 0x7f87b406f600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f87b40b7dc0 tx=0x7f87a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4cb85f7a8b8a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87b2ffd640 1 -- 172.21.15.72:0/626873616 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4035020 con 0x7f87b415c050 2024-03-06T07:22:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87b2ffd640 1 -- 172.21.15.72:0/626873616 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a4031870 con 0x7f87b415c050 2024-03-06T07:22:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87b2ffd640 1 -- 172.21.15.72:0/626873616 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4031c40 con 0x7f87b415c050 2024-03-06T07:22:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 -- 172.21.15.72:0/626873616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b415c050 msgr2=0x7f87b406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 --2- 172.21.15.72:0/626873616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b415c050 0x7f87b406f600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f87b40b7dc0 tx=0x7f87a402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 -- 172.21.15.72:0/626873616 shutdown_connections 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 --2- 172.21.15.72:0/626873616 >> v2:172.21.15.146:3300/0 conn(0x7f87b406fb40 0x7f87b40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 --2- 172.21.15.72:0/626873616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b415c050 0x7f87b406f600 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 -- 172.21.15.72:0/626873616 >> 172.21.15.72:0/626873616 conn(0x7f87b40b3800 msgr2=0x7f87b40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 -- 172.21.15.72:0/626873616 shutdown_connections 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.319+0000 7f87ba908640 1 -- 172.21.15.72:0/626873616 wait complete. 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.320+0000 7f87ba908640 1 Processor -- start 2024-03-06T07:22:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.321+0000 7f87ba908640 1 -- start start 2024-03-06T07:22:34.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87b406fb40 0x7f87b414c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 0x7f87b409caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87b415c050 0x7f87b414ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40d01f0 con 0x7f87b40ccca0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40cd130 con 0x7f87b415c050 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40d0370 con 0x7f87b406fb40 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 0x7f87b409caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 0x7f87b409caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60628/0 (socket says 172.21.15.72:60628) 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 -- 172.21.15.72:0/3850003611 learned_addr learned my addr 172.21.15.72:0/3850003611 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b8ea5640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87b415c050 0x7f87b414ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b3fff640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87b406fb40 0x7f87b414c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 -- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87b406fb40 msgr2=0x7f87b414c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87b406fb40 0x7f87b414c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 -- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87b415c050 msgr2=0x7f87b414ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87b415c050 0x7f87b414ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 -- 172.21.15.72:0/3850003611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b40b77a0 con 0x7f87b40ccca0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b37fe640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 0x7f87b409caf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f87a800d620 tx=0x7f87a800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800b620 con 0x7f87b40ccca0 2024-03-06T07:22:34.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a8019070 con 0x7f87b40ccca0 2024-03-06T07:22:34.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b414c700 con 0x7f87b40ccca0 2024-03-06T07:22:34.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.322+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a800ba00 con 0x7f87b40ccca0 2024-03-06T07:22:34.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.323+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b414cad0 con 0x7f87b40ccca0 2024-03-06T07:22:34.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.323+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b409fe60 con 0x7f87b40ccca0 2024-03-06T07:22:34.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.324+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87a800bba0 con 0x7f87b40ccca0 2024-03-06T07:22:34.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.325+0000 7f87b0ff9640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87a006b500 0x7f87a006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.325+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f87a808a660 con 0x7f87b40ccca0 2024-03-06T07:22:34.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.325+0000 7f87b3fff640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87a006b500 0x7f87a006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.326+0000 7f87b3fff640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87a006b500 0x7f87a006d9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f87a4036000 tx=0x7f87a403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.329+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87a8059460 con 0x7f87b40ccca0 2024-03-06T07:22:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.505+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f87b4068ff0 con 0x7f87b40ccca0 2024-03-06T07:22:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.506+0000 7f87b0ff9640 1 -- 172.21.15.72:0/3850003611 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f87a8058d60 con 0x7f87b40ccca0 2024-03-06T07:22:34.508 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87a006b500 msgr2=0x7f87a006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87a006b500 0x7f87a006d9c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f87a4036000 tx=0x7f87a403c040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 msgr2=0x7f87b409caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 0x7f87b409caf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f87a800d620 tx=0x7f87a800dae0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 shutdown_connections 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87a006b500 0x7f87a006d9c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87b415c050 0x7f87b414ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87b40ccca0 0x7f87b409caf0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 --2- 172.21.15.72:0/3850003611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87b406fb40 0x7f87b414c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 >> 172.21.15.72:0/3850003611 conn(0x7f87b40b3800 msgr2=0x7f87b4070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 shutdown_connections 2024-03-06T07:22:34.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.508+0000 7f87ba908640 1 -- 172.21.15.72:0/3850003611 wait complete. 2024-03-06T07:22:34.521 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:34.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.618+0000 7f0dd6381640 1 Processor -- start 2024-03-06T07:22:34.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.618+0000 7f0dd6381640 1 -- start start 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd6381640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd015bdd0 0x7f0dd00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd6381640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0dd015fa10 0x7f0dd00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd6381640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0dd00d16e0 0x7f0dd00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd6381640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd00cebe0 con 0x7f0dd00d16e0 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd6381640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd00ceec0 con 0x7f0dd015fa10 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd6381640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00cf1a0 con 0x7f0dd015bdd0 2024-03-06T07:22:34.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0dd00d16e0 0x7f0dd00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0dd00d16e0 0x7f0dd00d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60632/0 (socket says 172.21.15.72:60632) 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 -- 172.21.15.72:0/2242740994 learned_addr learned my addr 172.21.15.72:0/2242740994 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dcffff640 1 --2- 172.21.15.72:0/2242740994 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd015bdd0 0x7f0dd00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dcf7fe640 1 --2- 172.21.15.72:0/2242740994 >> v2:172.21.15.146:3300/0 conn(0x7f0dd015fa10 0x7f0dd00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 -- 172.21.15.72:0/2242740994 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd015bdd0 msgr2=0x7f0dd00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 --2- 172.21.15.72:0/2242740994 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd015bdd0 0x7f0dd00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 -- 172.21.15.72:0/2242740994 >> v2:172.21.15.146:3300/0 conn(0x7f0dd015fa10 msgr2=0x7f0dd00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 --2- 172.21.15.72:0/2242740994 >> v2:172.21.15.146:3300/0 conn(0x7f0dd015fa10 0x7f0dd00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dd491e640 1 -- 172.21.15.72:0/2242740994 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd00cc8e0 con 0x7f0dd00d16e0 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.619+0000 7f0dcf7fe640 1 --2- 172.21.15.72:0/2242740994 >> v2:172.21.15.146:3300/0 conn(0x7f0dd015fa10 0x7f0dd00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.620+0000 7f0dd491e640 1 --2- 172.21.15.72:0/2242740994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd00d16e0 0x7f0dd00d1b30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0dd006e2d0 tx=0x7f0dc802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f49980bf415e247 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.620+0000 7f0dceffd640 1 -- 172.21.15.72:0/2242740994 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc802c9e0 con 0x7f0dd00d16e0 2024-03-06T07:22:34.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.620+0000 7f0dceffd640 1 -- 172.21.15.72:0/2242740994 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc802cb80 con 0x7f0dd00d16e0 2024-03-06T07:22:34.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.620+0000 7f0dceffd640 1 -- 172.21.15.72:0/2242740994 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc80343f0 con 0x7f0dd00d16e0 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 -- 172.21.15.72:0/2242740994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd00d16e0 msgr2=0x7f0dd00d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 --2- 172.21.15.72:0/2242740994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd00d16e0 0x7f0dd00d1b30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0dd006e2d0 tx=0x7f0dc802be40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 -- 172.21.15.72:0/2242740994 shutdown_connections 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 --2- 172.21.15.72:0/2242740994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd00d16e0 0x7f0dd00d1b30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 --2- 172.21.15.72:0/2242740994 >> v2:172.21.15.146:3300/0 conn(0x7f0dd015fa10 0x7f0dd00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 --2- 172.21.15.72:0/2242740994 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd015bdd0 0x7f0dd00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 -- 172.21.15.72:0/2242740994 >> 172.21.15.72:0/2242740994 conn(0x7f0dd009a9f0 msgr2=0x7f0dd009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 -- 172.21.15.72:0/2242740994 shutdown_connections 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.621+0000 7f0dd6381640 1 -- 172.21.15.72:0/2242740994 wait complete. 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 Processor -- start 2024-03-06T07:22:34.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 -- start start 2024-03-06T07:22:34.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd00d16e0 0x7f0dd00cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dd015bdd0 0x7f0dd006d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 0x7f0dd006c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00cc5d0 con 0x7f0dd015fa10 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd00d2070 con 0x7f0dd015bdd0 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.623+0000 7f0dd6381640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd015c1b0 con 0x7f0dd00d16e0 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 0x7f0dd006c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 0x7f0dd006c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60648/0 (socket says 172.21.15.72:60648) 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 -- 172.21.15.72:0/1702749606 learned_addr learned my addr 172.21.15.72:0/1702749606 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dcffff640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd00d16e0 0x7f0dd00cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dcf7fe640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dd015bdd0 0x7f0dd006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 -- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd00d16e0 msgr2=0x7f0dd00cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd00d16e0 0x7f0dd00cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 -- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dd015bdd0 msgr2=0x7f0dd006d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dd015bdd0 0x7f0dd006d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 -- 172.21.15.72:0/1702749606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd006c7c0 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd491e640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 0x7f0dd006c0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0dc802c450 tx=0x7f0dc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8043020 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc803e450 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd006caa0 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8009510 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.624+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd006d060 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.626+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd004f330 con 0x7f0dd015fa10 2024-03-06T07:22:34.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.626+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0dc8046020 con 0x7f0dd015fa10 2024-03-06T07:22:34.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.626+0000 7f0dccff9640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0dbc06b4d0 0x7f0dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.626+0000 7f0dcffff640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0dbc06b4d0 0x7f0dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.626+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0dc80ac8d0 con 0x7f0dd015fa10 2024-03-06T07:22:34.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.627+0000 7f0dcffff640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0dbc06b4d0 0x7f0dbc06d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00079a0 tx=0x7f0dc0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.630+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0dc804d030 con 0x7f0dd015fa10 2024-03-06T07:22:34.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.838+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0dd00cbed0 con 0x7f0dd015fa10 2024-03-06T07:22:34.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.839+0000 7f0dccff9640 1 -- 172.21.15.72:0/1702749606 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f0dc807b110 con 0x7f0dd015fa10 2024-03-06T07:22:34.841 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:34.841 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:22:34.841 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.841+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0dbc06b4d0 msgr2=0x7f0dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.841+0000 7f0dd6381640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0dbc06b4d0 0x7f0dbc06d990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0dc00079a0 tx=0x7f0dc0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.841+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 msgr2=0x7f0dd006c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.841+0000 7f0dd6381640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 0x7f0dd006c0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0dc802c450 tx=0x7f0dc8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 shutdown_connections 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0dbc06b4d0 0x7f0dbc06d990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0dd015fa10 0x7f0dd006c0c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0dd015bdd0 0x7f0dd006d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 --2- 172.21.15.72:0/1702749606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dd00d16e0 0x7f0dd00cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 >> 172.21.15.72:0/1702749606 conn(0x7f0dd009a9f0 msgr2=0x7f0dd006eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 shutdown_connections 2024-03-06T07:22:34.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.842+0000 7f0dd6381640 1 -- 172.21.15.72:0/1702749606 wait complete. 2024-03-06T07:22:34.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:34.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.954+0000 7f99a5815640 1 Processor -- start 2024-03-06T07:22:34.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.954+0000 7f99a5815640 1 -- start start 2024-03-06T07:22:34.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f99a5815640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99a00d00c0 0x7f99a00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f99a5815640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f99a5815640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a00d09e0 con 0x7f99a00ce1b0 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f99a5815640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a006ca90 con 0x7f99a00d00c0 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f99a5815640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a006df30 con 0x7f99a00cf850 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99a00d00c0 0x7f99a00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99a00d00c0 0x7f99a00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60658/0 (socket says 172.21.15.72:60658) 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 -- 172.21.15.72:0/1930794518 learned_addr learned my addr 172.21.15.72:0/1930794518 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f9997fff640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 -- 172.21.15.72:0/1930794518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 msgr2=0x7f99a00cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 -- 172.21.15.72:0/1930794518 >> v1:172.21.15.146:6789/0 conn(0x7f99a00ce1b0 legacy=0x7f99a00ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f999effd640 1 -- 172.21.15.72:0/1930794518 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a015e380 con 0x7f99a00d00c0 2024-03-06T07:22:34.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.955+0000 7f9997fff640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.956+0000 7f999effd640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a00d04a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9990009870 tx=0x7f999002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e840f0e65e22d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.956+0000 7f999e7fc640 1 -- 172.21.15.72:0/1930794518 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990034070 con 0x7f99a00d00c0 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.956+0000 7f999e7fc640 1 -- 172.21.15.72:0/1930794518 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9990030840 con 0x7f99a00d00c0 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.956+0000 7f999e7fc640 1 -- 172.21.15.72:0/1930794518 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990030c10 con 0x7f99a00d00c0 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 -- 172.21.15.72:0/1930794518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 msgr2=0x7f99a00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a00d04a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9990009870 tx=0x7f999002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 -- 172.21.15.72:0/1930794518 shutdown_connections 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 --2- 172.21.15.72:0/1930794518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a00d04a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 -- 172.21.15.72:0/1930794518 >> 172.21.15.72:0/1930794518 conn(0x7f99a00b35a0 msgr2=0x7f99a00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:34.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 -- 172.21.15.72:0/1930794518 shutdown_connections 2024-03-06T07:22:34.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.957+0000 7f99a5815640 1 -- 172.21.15.72:0/1930794518 wait complete. 2024-03-06T07:22:34.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.958+0000 7f99a5815640 1 Processor -- start 2024-03-06T07:22:34.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.958+0000 7f99a5815640 1 -- start start 2024-03-06T07:22:34.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f99a5815640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00ce1b0 0x7f99a00cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f99a5815640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f99a5815640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a0153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f99a5815640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a006c440 con 0x7f99a00d00c0 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f99a5815640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a006c160 con 0x7f99a00ce1b0 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f99a5815640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a006c720 con 0x7f99a00cf850 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a0153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a0153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60668/0 (socket says 172.21.15.72:60668) 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 -- 172.21.15.72:0/3513391005 learned_addr learned my addr 172.21.15.72:0/3513391005 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f9997fff640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999effd640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00ce1b0 0x7f99a00cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 -- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 msgr2=0x7f99a00c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 -- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00ce1b0 msgr2=0x7f99a00cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:34.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00ce1b0 0x7f99a00cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 -- 172.21.15.72:0/3513391005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a00c9e00 con 0x7f99a00d00c0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999effd640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00ce1b0 0x7f99a00cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.959+0000 7f999f7fe640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a0153f30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f999800c7d0 tx=0x7f999800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.960+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998003b70 con 0x7f99a00d00c0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.960+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998003d30 con 0x7f99a00d00c0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.960+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a00ca0a0 con 0x7f99a00d00c0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.960+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999800aa90 con 0x7f99a00d00c0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.960+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a00b5e10 con 0x7f99a00d00c0 2024-03-06T07:22:34.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.961+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a004f330 con 0x7f99a00d00c0 2024-03-06T07:22:34.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.961+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f999800ac30 con 0x7f99a00d00c0 2024-03-06T07:22:34.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.962+0000 7f999cff9640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f998006b4d0 0x7f998006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:34.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.962+0000 7f999effd640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f998006b4d0 0x7f998006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:34.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.962+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f999808a060 con 0x7f99a00d00c0 2024-03-06T07:22:34.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.962+0000 7f999effd640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f998006b4d0 0x7f998006d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9990036000 tx=0x7f999003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:34.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:34.970+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9998058ef0 con 0x7f99a00d00c0 2024-03-06T07:22:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.148+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99a015daf0 con 0x7f99a00d00c0 2024-03-06T07:22:35.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.150+0000 7f999cff9640 1 -- 172.21.15.72:0/3513391005 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16810 (secure 0 0 0) 0x7f99980587f0 con 0x7f99a00d00c0 2024-03-06T07:22:35.152 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:35.152 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:31.188443+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.151+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f998006b4d0 msgr2=0x7f998006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f998006b4d0 0x7f998006d990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9990036000 tx=0x7f999003b040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 msgr2=0x7f99a0153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a0153f30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f999800c7d0 tx=0x7f999800cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 shutdown_connections 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f998006b4d0 0x7f998006d990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a00d00c0 0x7f99a0153f30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00cf850 0x7f99a00c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 --2- 172.21.15.72:0/3513391005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00ce1b0 0x7f99a00cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 >> 172.21.15.72:0/3513391005 conn(0x7f99a00b35a0 msgr2=0x7f99a00b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 shutdown_connections 2024-03-06T07:22:35.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.152+0000 7f99a5815640 1 -- 172.21.15.72:0/3513391005 wait complete. 2024-03-06T07:22:35.165 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:22:35.165 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:35.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.268+0000 7f90701bc640 1 Processor -- start 2024-03-06T07:22:35.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.268+0000 7f90701bc640 1 -- start start 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f90701bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f90680d4760 0x7f90680b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f90701bc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90680d0a80 0x7f90680cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f90701bc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90680d25d0 con 0x7f90680b4170 2024-03-06T07:22:35.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f90701bc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90680d4bf0 con 0x7f90680d0a80 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f90701bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90680ce9e0 con 0x7f90680d4760 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906e759640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90680d0a80 0x7f90680cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906df58640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f90680b4170 0x7f90680d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35356/0 (socket says 172.21.15.72:35356) 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906df58640 1 -- 172.21.15.72:0/3040900102 learned_addr learned my addr 172.21.15.72:0/3040900102 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906d757640 1 --2- 172.21.15.72:0/3040900102 >> v2:172.21.15.146:3300/0 conn(0x7f90680d4760 0x7f90680b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906e759640 1 -- 172.21.15.72:0/3040900102 >> v2:172.21.15.146:3300/0 conn(0x7f90680d4760 msgr2=0x7f90680b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906e759640 1 --2- 172.21.15.72:0/3040900102 >> v2:172.21.15.146:3300/0 conn(0x7f90680d4760 0x7f90680b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906e759640 1 -- 172.21.15.72:0/3040900102 >> v1:172.21.15.72:6789/0 conn(0x7f90680b4170 legacy=0x7f90680d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906e759640 1 -- 172.21.15.72:0/3040900102 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680d2870 con 0x7f90680d0a80 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.269+0000 7f906d757640 1 --2- 172.21.15.72:0/3040900102 >> v2:172.21.15.146:3300/0 conn(0x7f90680d4760 0x7f90680b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.270+0000 7f906e759640 1 --2- 172.21.15.72:0/3040900102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680cee70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9068067920 tx=0x7f906402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d808b3b2e67bf8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.270+0000 7f906cf56640 1 -- 172.21.15.72:0/3040900102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064039070 con 0x7f90680d0a80 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.270+0000 7f906cf56640 1 -- 172.21.15.72:0/3040900102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906402cc80 con 0x7f90680d0a80 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.270+0000 7f906cf56640 1 -- 172.21.15.72:0/3040900102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064034660 con 0x7f90680d0a80 2024-03-06T07:22:35.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 -- 172.21.15.72:0/3040900102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 msgr2=0x7f90680cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 --2- 172.21.15.72:0/3040900102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680cee70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f9068067920 tx=0x7f906402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 -- 172.21.15.72:0/3040900102 shutdown_connections 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 --2- 172.21.15.72:0/3040900102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680cee70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 --2- 172.21.15.72:0/3040900102 >> v2:172.21.15.146:3300/0 conn(0x7f90680d4760 0x7f90680b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 -- 172.21.15.72:0/3040900102 >> 172.21.15.72:0/3040900102 conn(0x7f90680d3c40 msgr2=0x7f90680d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 -- 172.21.15.72:0/3040900102 shutdown_connections 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.271+0000 7f90701bc640 1 -- 172.21.15.72:0/3040900102 wait complete. 2024-03-06T07:22:35.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.272+0000 7f90701bc640 1 Processor -- start 2024-03-06T07:22:35.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.272+0000 7f90701bc640 1 -- start start 2024-03-06T07:22:35.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f90701bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680b4170 0x7f90681f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f90701bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f90701bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90680b3540 0x7f90680b1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f90701bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90680b4dd0 con 0x7f90680d0a80 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f90701bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90680b4550 con 0x7f90680b4170 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f90701bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90680b4f50 con 0x7f90680b3540 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680b3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60682/0 (socket says 172.21.15.72:60682) 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 -- 172.21.15.72:0/3130719029 learned_addr learned my addr 172.21.15.72:0/3130719029 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:35.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906df58640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680b4170 0x7f90681f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 -- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90680b3540 msgr2=0x7f90680b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90680b3540 0x7f90680b1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 -- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680b4170 msgr2=0x7f90681f3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680b4170 0x7f90681f3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 -- 172.21.15.72:0/3130719029 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90680c9690 con 0x7f90680d0a80 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906df58640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680b4170 0x7f90681f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f906d757640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680b3000 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f905800c7c0 tx=0x7f905800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.273+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058015070 con 0x7f90680d0a80 2024-03-06T07:22:35.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.274+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90580077c0 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.274+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058007b90 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.274+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90680c9970 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.274+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90680c9f30 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.275+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f906804f330 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.275+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9058018070 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.275+0000 7f905e7fc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f904c06b4d0 0x7f904c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.275+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f905808a100 con 0x7f90680d0a80 2024-03-06T07:22:35.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.275+0000 7f906df58640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f904c06b4d0 0x7f904c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.276+0000 7f906df58640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f904c06b4d0 0x7f904c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9060000f60 tx=0x7f9060009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.280+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9058058f00 con 0x7f90680d0a80 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.495+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90680ca7d0 con 0x7f90680d0a80 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.496+0000 7f905e7fc640 1 -- 172.21.15.72:0/3130719029 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f9058058800 con 0x7f90680d0a80 2024-03-06T07:22:35.498 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:22:35.499 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:22:35.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f904c06b4d0 msgr2=0x7f904c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f904c06b4d0 0x7f904c06d990 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9060000f60 tx=0x7f9060009040 comp rx=0 tx=0).stop 2024-03-06T07:22:35.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 msgr2=0x7f90680b3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680b3000 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f905800c7c0 tx=0x7f905800cc80 comp rx=0 tx=0).stop 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 shutdown_connections 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f904c06b4d0 0x7f904c06d990 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90680b3540 0x7f90680b1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90680d0a80 0x7f90680b3000 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 --2- 172.21.15.72:0/3130719029 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90680b4170 0x7f90681f3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 >> 172.21.15.72:0/3130719029 conn(0x7f90680d3c40 msgr2=0x7f906806b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 shutdown_connections 2024-03-06T07:22:35.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.498+0000 7f90701bc640 1 -- 172.21.15.72:0/3130719029 wait complete. 2024-03-06T07:22:35.511 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:35.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.611+0000 7f3940732640 1 Processor -- start 2024-03-06T07:22:35.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.611+0000 7f3940732640 1 -- start start 2024-03-06T07:22:35.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.612+0000 7f3940732640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 0x7f393815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.612+0000 7f3940732640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b600 0x7f3938162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.612+0000 7f3940732640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393815ea30 con 0x7f39380ce920 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.612+0000 7f3940732640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3938161b70 con 0x7f393806b220 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.612+0000 7f3940732640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393815ed10 con 0x7f393806b600 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393e4ce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 0x7f393815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393eccf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b600 0x7f3938162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393e4ce640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 0x7f393815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:48756/0 (socket says 172.21.15.72:48756) 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393e4ce640 1 -- 172.21.15.72:0/1714972737 learned_addr learned my addr 172.21.15.72:0/1714972737 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393dccd640 1 --1- 172.21.15.72:0/1714972737 >> v1:172.21.15.146:6789/0 conn(0x7f39380ce920 0x7f3938071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:41944/0 (socket says 172.21.15.72:41944) 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393eccf640 1 -- 172.21.15.72:0/1714972737 >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 msgr2=0x7f393815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393eccf640 1 --2- 172.21.15.72:0/1714972737 >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 0x7f393815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393eccf640 1 -- 172.21.15.72:0/1714972737 >> v1:172.21.15.146:6789/0 conn(0x7f39380ce920 legacy=0x7f3938071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393d4cc640 1 -- 172.21.15.72:0/1714972737 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353124501 0 0) 0x7f393815ea30 con 0x7f39380ce920 2024-03-06T07:22:35.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393eccf640 1 -- 172.21.15.72:0/1714972737 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393815eff0 con 0x7f393806b600 2024-03-06T07:22:35.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.613+0000 7f393e4ce640 1 --2- 172.21.15.72:0/1714972737 >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 0x7f393815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:35.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.614+0000 7f393eccf640 1 --2- 172.21.15.72:0/1714972737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b600 0x7f3938162000 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f39280097a0 tx=0x7f392802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=332aa9291ee598a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.614+0000 7f393d4cc640 1 -- 172.21.15.72:0/1714972737 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280315a0 con 0x7f393806b600 2024-03-06T07:22:35.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.614+0000 7f393d4cc640 1 -- 172.21.15.72:0/1714972737 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3928037070 con 0x7f393806b600 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.614+0000 7f393d4cc640 1 -- 172.21.15.72:0/1714972737 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928031960 con 0x7f393806b600 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 -- 172.21.15.72:0/1714972737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b600 msgr2=0x7f3938162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 --2- 172.21.15.72:0/1714972737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b600 0x7f3938162000 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f39280097a0 tx=0x7f392802b840 comp rx=0 tx=0).stop 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 -- 172.21.15.72:0/1714972737 shutdown_connections 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 --2- 172.21.15.72:0/1714972737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b600 0x7f3938162000 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 --2- 172.21.15.72:0/1714972737 >> v2:172.21.15.146:3300/0 conn(0x7f393806b220 0x7f393815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 -- 172.21.15.72:0/1714972737 >> 172.21.15.72:0/1714972737 conn(0x7f393806c0c0 msgr2=0x7f393806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 -- 172.21.15.72:0/1714972737 shutdown_connections 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.615+0000 7f3940732640 1 -- 172.21.15.72:0/1714972737 wait complete. 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.616+0000 7f3940732640 1 Processor -- start 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.616+0000 7f3940732640 1 -- start start 2024-03-06T07:22:35.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3940732640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b220 0x7f39380c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3940732640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f393806b600 0x7f39380b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3940732640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 0x7f3938160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3940732640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3938072090 con 0x7f39380ce920 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3940732640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39380ce200 con 0x7f393806b600 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3940732640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39380c68d0 con 0x7f393806b220 2024-03-06T07:22:35.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 0x7f3938160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 0x7f3938160840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60692/0 (socket says 172.21.15.72:60692) 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 -- 172.21.15.72:0/2496456512 learned_addr learned my addr 172.21.15.72:0/2496456512 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393e4ce640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b220 0x7f39380c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393dccd640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f393806b600 0x7f39380b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 -- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b220 msgr2=0x7f39380c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b220 0x7f39380c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 -- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f393806b600 msgr2=0x7f39380b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f393806b600 0x7f39380b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 -- 172.21.15.72:0/2496456512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39380b1d80 con 0x7f39380ce920 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f393eccf640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 0x7f3938160840 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3928034040 tx=0x7f39280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.617+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928004230 con 0x7f39380ce920 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.618+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39380b2060 con 0x7f39380ce920 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.618+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39280043d0 con 0x7f39380ce920 2024-03-06T07:22:35.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.618+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928003850 con 0x7f39380ce920 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.618+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39380c5e70 con 0x7f39380ce920 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.618+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393804f330 con 0x7f39380ce920 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.619+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39280039f0 con 0x7f39380ce920 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.619+0000 7f3936ffd640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f392406b4d0 0x7f392406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.619+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f392802f030 con 0x7f39380ce920 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.619+0000 7f393e4ce640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f392406b4d0 0x7f392406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.620+0000 7f393e4ce640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f392406b4d0 0x7f392406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3930006ca0 tx=0x7f3930009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.624+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3928041070 con 0x7f39380ce920 2024-03-06T07:22:35.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.839+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39380cdf00 con 0x7f39380ce920 2024-03-06T07:22:35.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.839+0000 7f3936ffd640 1 -- 172.21.15.72:0/2496456512 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4977 (secure 0 0 0) 0x7f392807aad0 con 0x7f39380ce920 2024-03-06T07:22:35.841 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:35.842 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1687951010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1687951010},{"type":"v1","addr":"172.21.15.146:6834","nonce":1687951010}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4232,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3035164188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3035164188},{"type":"v1","addr":"172.21.15.146:6835","nonce":3035164188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3276015872","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3276015872},{"type":"v1","addr":"172.21.15.72:6837","nonce":3276015872}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:21:49.400572+0000","modified":"2024-03-06T07:21:52.396816+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/3798106106","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3798106106},{"type":"v1","addr":"172.21.15.72:6836","nonce":3798106106}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4272,"qdb_cluster":[4272]},"id":1}]} 2024-03-06T07:22:35.842 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-03-06T07:22:35.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f392406b4d0 msgr2=0x7f392406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f392406b4d0 0x7f392406d990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3930006ca0 tx=0x7f3930009240 comp rx=0 tx=0).stop 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 msgr2=0x7f3938160840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 0x7f3938160840 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3928034040 tx=0x7f39280046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 shutdown_connections 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f392406b4d0 0x7f392406d990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39380ce920 0x7f3938160840 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f393806b600 0x7f39380b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 --2- 172.21.15.72:0/2496456512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f393806b220 0x7f39380c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 >> 172.21.15.72:0/2496456512 conn(0x7f393806c0c0 msgr2=0x7f3938071980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 shutdown_connections 2024-03-06T07:22:35.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.841+0000 7f3940732640 1 -- 172.21.15.72:0/2496456512 wait complete. 2024-03-06T07:22:35.854 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:35.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.952+0000 7f4ddb9b6640 1 Processor -- start 2024-03-06T07:22:35.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 -- start start 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd415dfe0 0x7f4dd40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4dd40cf830 0x7f4dd40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4dd40d27f0 0x7f4dd40d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40d2c50 con 0x7f4dd40d27f0 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40b31e0 con 0x7f4dd40cf830 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.953+0000 7f4ddb9b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b3d60 con 0x7f4dd415dfe0 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4dd40d27f0 0x7f4dd40d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4dd40d27f0 0x7f4dd40d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60698/0 (socket says 172.21.15.72:60698) 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 -- 172.21.15.72:0/995187505 learned_addr learned my addr 172.21.15.72:0/995187505 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:35.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9752640 1 --2- 172.21.15.72:0/995187505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd415dfe0 0x7f4dd40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd8f51640 1 --2- 172.21.15.72:0/995187505 >> v2:172.21.15.146:3300/0 conn(0x7f4dd40cf830 0x7f4dd40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 -- 172.21.15.72:0/995187505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd415dfe0 msgr2=0x7f4dd40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 --2- 172.21.15.72:0/995187505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd415dfe0 0x7f4dd40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 -- 172.21.15.72:0/995187505 >> v2:172.21.15.146:3300/0 conn(0x7f4dd40cf830 msgr2=0x7f4dd40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 --2- 172.21.15.72:0/995187505 >> v2:172.21.15.146:3300/0 conn(0x7f4dd40cf830 0x7f4dd40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.954+0000 7f4dd9f53640 1 -- 172.21.15.72:0/995187505 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40b3430 con 0x7f4dd40d27f0 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.955+0000 7f4dd9f53640 1 --2- 172.21.15.72:0/995187505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40d0300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4dd408cba0 tx=0x7f4dcc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8260d51f3d0ec93d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.955+0000 7f4dcbfff640 1 -- 172.21.15.72:0/995187505 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc02ca70 con 0x7f4dd40d27f0 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.955+0000 7f4dcbfff640 1 -- 172.21.15.72:0/995187505 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc02cc10 con 0x7f4dd40d27f0 2024-03-06T07:22:35.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.955+0000 7f4dcbfff640 1 -- 172.21.15.72:0/995187505 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0343f0 con 0x7f4dd40d27f0 2024-03-06T07:22:35.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/995187505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 msgr2=0x7f4dd40d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/995187505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40d0300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4dd408cba0 tx=0x7f4dcc02bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/995187505 shutdown_connections 2024-03-06T07:22:35.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/995187505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40d0300 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/995187505 >> v2:172.21.15.146:3300/0 conn(0x7f4dd40cf830 0x7f4dd40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/995187505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd415dfe0 0x7f4dd40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/995187505 >> 172.21.15.72:0/995187505 conn(0x7f4dd40d3cd0 msgr2=0x7f4dd40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/995187505 shutdown_connections 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.956+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/995187505 wait complete. 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.957+0000 7f4ddb9b6640 1 Processor -- start 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.957+0000 7f4ddb9b6640 1 -- start start 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd40cf830 0x7f4dd41f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd415dfe0 0x7f4dd40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40d1c70 con 0x7f4dd40d27f0 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b39f0 con 0x7f4dd415dfe0 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40d1df0 con 0x7f4dd40cf830 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60712/0 (socket says 172.21.15.72:60712) 2024-03-06T07:22:35.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 -- 172.21.15.72:0/1726454500 learned_addr learned my addr 172.21.15.72:0/1726454500 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd9752640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd40cf830 0x7f4dd41f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd9f53640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd415dfe0 0x7f4dd40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 -- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd40cf830 msgr2=0x7f4dd41f71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd40cf830 0x7f4dd41f71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 -- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd415dfe0 msgr2=0x7f4dd40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd415dfe0 0x7f4dd40b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 -- 172.21.15.72:0/1726454500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40b8970 con 0x7f4dd40d27f0 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd9752640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd40cf830 0x7f4dd41f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dd8f51640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40cbdf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400e860 tx=0x7f4dc400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4017070 con 0x7f4dd40d27f0 2024-03-06T07:22:35.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc400c940 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd41f7910 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.958+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc400cd10 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.959+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd41f7ed0 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.960+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd404f330 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.960+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dc401a070 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.960+0000 7f4dc9ffb640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4db806b500 0x7f4db806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.960+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4dc408a1c0 con 0x7f4dd40d27f0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.960+0000 7f4dd9752640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4db806b500 0x7f4db806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.961+0000 7f4dd9752640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4db806b500 0x7f4db806d9c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4157b90 tx=0x7f4dd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:35.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:35.965+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dd40b8970 con 0x7f4dd40d27f0 2024-03-06T07:22:36.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.169+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dd40ce870 con 0x7f4dd40d27f0 2024-03-06T07:22:36.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.170+0000 7f4dc9ffb640 1 -- 172.21.15.72:0/1726454500 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16810 (secure 0 0 0) 0x7f4dc4059110 con 0x7f4dd40d27f0 2024-03-06T07:22:36.172 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:36.172 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":23,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:31.188443+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:21:47.857029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-06T07:21:48.376009+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]},{"pgid":"3.33","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:36.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4db806b500 msgr2=0x7f4db806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4db806b500 0x7f4db806d9c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4157b90 tx=0x7f4dd0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 msgr2=0x7f4dd40cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40cbdf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400e860 tx=0x7f4dc400ed20 comp rx=0 tx=0).stop 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 shutdown_connections 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4db806b500 0x7f4db806d9c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd415dfe0 0x7f4dd40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dd40d27f0 0x7f4dd40cbdf0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 --2- 172.21.15.72:0/1726454500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd40cf830 0x7f4dd41f71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 >> 172.21.15.72:0/1726454500 conn(0x7f4dd40d3cd0 msgr2=0x7f4dd40cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 shutdown_connections 2024-03-06T07:22:36.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.172+0000 7f4ddb9b6640 1 -- 172.21.15.72:0/1726454500 wait complete. 2024-03-06T07:22:36.185 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:22:36.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.287+0000 7f410cf1e640 1 Processor -- start 2024-03-06T07:22:36.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.287+0000 7f410cf1e640 1 -- start start 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f410cf1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f41080d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f410cf1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4108162970 0x7f41080b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f410cf1e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41080d22c0 con 0x7f4108161cf0 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f410cf1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4108161b00 con 0x7f4108162970 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f410cf1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41080cecc0 con 0x7f41080b4460 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f410659c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f41080d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f40fffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4108162970 0x7f41080b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f4106d9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4108161cf0 0x7f41080cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:41958/0 (socket says 172.21.15.72:41958) 2024-03-06T07:22:36.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f4106d9d640 1 -- 172.21.15.72:0/2665358585 learned_addr learned my addr 172.21.15.72:0/2665358585 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.288+0000 7f4105d9b640 1 -- 172.21.15.72:0/2665358585 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011466886 0 0) 0x7f41080d22c0 con 0x7f4108161cf0 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f4105d9b640 1 -- 172.21.15.72:0/2665358585 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4003520 con 0x7f4108161cf0 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f40fffff640 1 -- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 msgr2=0x7f41080d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f40fffff640 1 --2- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f41080d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f40fffff640 1 -- 172.21.15.72:0/2665358585 >> v1:172.21.15.146:6789/0 conn(0x7f4108161cf0 legacy=0x7f41080cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f40fffff640 1 -- 172.21.15.72:0/2665358585 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41080d25a0 con 0x7f4108162970 2024-03-06T07:22:36.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f410659c640 1 --2- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f41080d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.289+0000 7f40fffff640 1 --2- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b3580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f40f00097a0 tx=0x7f40f002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb07e36ef9e7848 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f4105d9b640 1 -- 172.21.15.72:0/2665358585 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0035020 con 0x7f4108162970 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f4105d9b640 1 -- 172.21.15.72:0/2665358585 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0031810 con 0x7f4108162970 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f4105d9b640 1 -- 172.21.15.72:0/2665358585 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0031bc0 con 0x7f4108162970 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 -- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 msgr2=0x7f41080b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b3580 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f40f00097a0 tx=0x7f40f002b8b0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 -- 172.21.15.72:0/2665358585 shutdown_connections 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b3580 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2665358585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f41080d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 -- 172.21.15.72:0/2665358585 >> 172.21.15.72:0/2665358585 conn(0x7f41080d3ce0 msgr2=0x7f41080d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 -- 172.21.15.72:0/2665358585 shutdown_connections 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.290+0000 7f410cf1e640 1 -- 172.21.15.72:0/2665358585 wait complete. 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.291+0000 7f410cf1e640 1 Processor -- start 2024-03-06T07:22:36.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.291+0000 7f410cf1e640 1 -- start start 2024-03-06T07:22:36.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410cf1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f4108155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410cf1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 0x7f41080cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410cf1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410cf1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080ce950 con 0x7f4108161cf0 2024-03-06T07:22:36.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410cf1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080d2b60 con 0x7f4108162970 2024-03-06T07:22:36.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410cf1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41080b3ac0 con 0x7f41080b4460 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f40fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 0x7f41080cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f40fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 0x7f41080cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60728/0 (socket says 172.21.15.72:60728) 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f40fffff640 1 -- 172.21.15.72:0/2255164977 learned_addr learned my addr 172.21.15.72:0/2255164977 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f410659c640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f4108155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.292+0000 7f4106d9d640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40fffff640 1 -- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 msgr2=0x7f4108155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40fffff640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f4108155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40fffff640 1 -- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 msgr2=0x7f41080b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40fffff640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40fffff640 1 -- 172.21.15.72:0/2255164977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41080b6b50 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40fffff640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 0x7f41080cc540 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098c0 tx=0x7f40f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0041070 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f0031e40 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0035040 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41080b6e30 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.293+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41080b73f0 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.294+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410804f330 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.294+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f40f004c060 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.295+0000 7f40feffd640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f40e806b4d0 0x7f40e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.295+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f40f00abd80 con 0x7f4108161cf0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.295+0000 7f410659c640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f40e806b4d0 0x7f40e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.295+0000 7f410659c640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f40e806b4d0 0x7f40e806d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009a30 tx=0x7f40f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.299+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f40f00372c0 con 0x7f4108161cf0 2024-03-06T07:22:36.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.515+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f410806b180 con 0x7f4108161cf0 2024-03-06T07:22:36.517 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:22:36.515+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:22:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.517+0000 7f40feffd640 1 -- 172.21.15.72:0/2255164977 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f40f007ab80 con 0x7f4108161cf0 2024-03-06T07:22:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.520+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f40e806b4d0 msgr2=0x7f40e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.520+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f40e806b4d0 0x7f40e806d990 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009a30 tx=0x7f40f8008040 comp rx=0 tx=0).stop 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.520+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 msgr2=0x7f41080cc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.520+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 0x7f41080cc540 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f40f00098c0 tx=0x7f40f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 shutdown_connections 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f40e806b4d0 0x7f40e806d990 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108162970 0x7f41080b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4108161cf0 0x7f41080cc540 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 --2- 172.21.15.72:0/2255164977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41080b4460 0x7f4108155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 >> 172.21.15.72:0/2255164977 conn(0x7f41080d3ce0 msgr2=0x7f4108161270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 shutdown_connections 2024-03-06T07:22:36.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.521+0000 7f410cf1e640 1 -- 172.21.15.72:0/2255164977 wait complete. 2024-03-06T07:22:36.524 INFO:tasks.ceph.mds.c.smithi072.stderr: -3031> 2024-03-06T07:21:53.461+0000 7f6d9d8fb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:36.534 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:22:36.552 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:22:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.635+0000 7f8d92cb5640 1 Processor -- start 2024-03-06T07:22:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.635+0000 7f8d92cb5640 1 -- start start 2024-03-06T07:22:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d92cb5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d8c0b25d0 0x7f8d8c1669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d92cb5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8c0d0240 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d92cb5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8c0cf210 con 0x7f8d8c15bdd0 2024-03-06T07:22:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d92cb5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c0cf4f0 con 0x7f8d8c0b25d0 2024-03-06T07:22:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d91252640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d8c0b25d0 0x7f8d8c1669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d8bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8d8c0b41b0 0x7f8d8c0cea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35370/0 (socket says 172.21.15.72:35370) 2024-03-06T07:22:36.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d8bfff640 1 -- 172.21.15.72:0/4027859624 learned_addr learned my addr 172.21.15.72:0/4027859624 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:36.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d91252640 1 -- 172.21.15.72:0/4027859624 >> v1:172.21.15.146:6789/0 conn(0x7f8d8c15bdd0 legacy=0x7f8d8c0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:36.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d91252640 1 -- 172.21.15.72:0/4027859624 >> v1:172.21.15.72:6789/0 conn(0x7f8d8c0b41b0 legacy=0x7f8d8c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d8b7fe640 1 -- 172.21.15.72:0/4027859624 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340716047 0 0) 0x7f8d8c0d0240 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.636+0000 7f8d91252640 1 -- 172.21.15.72:0/4027859624 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0d04e0 con 0x7f8d8c0b25d0 2024-03-06T07:22:36.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d91252640 1 --2- 172.21.15.72:0/4027859624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b25d0 0x7f8d8c1669e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009d90 tx=0x7f8d8402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7799340a77ec2d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d8b7fe640 1 -- 172.21.15.72:0/4027859624 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84039070 con 0x7f8d8c0b25d0 2024-03-06T07:22:36.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d8b7fe640 1 -- 172.21.15.72:0/4027859624 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8402cd00 con 0x7f8d8c0b25d0 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d8b7fe640 1 -- 172.21.15.72:0/4027859624 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84034660 con 0x7f8d8c0b25d0 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 -- 172.21.15.72:0/4027859624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b25d0 msgr2=0x7f8d8c1669e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/4027859624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b25d0 0x7f8d8c1669e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009d90 tx=0x7f8d8402bf50 comp rx=0 tx=0).stop 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 -- 172.21.15.72:0/4027859624 shutdown_connections 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/4027859624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b25d0 0x7f8d8c1669e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 -- 172.21.15.72:0/4027859624 >> 172.21.15.72:0/4027859624 conn(0x7f8d8c0b3690 msgr2=0x7f8d8c0b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 -- 172.21.15.72:0/4027859624 shutdown_connections 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.637+0000 7f8d92cb5640 1 -- 172.21.15.72:0/4027859624 wait complete. 2024-03-06T07:22:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 Processor -- start 2024-03-06T07:22:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 -- start start 2024-03-06T07:22:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d8c0b25d0 0x7f8d8c0c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 0x7f8d8c1475d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d8c15bdd0 0x7f8d8c147b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0b10c0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0d0850 con 0x7f8d8c15bdd0 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d92cb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0b1240 con 0x7f8d8c0b25d0 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.639+0000 7f8d8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 0x7f8d8c1475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 0x7f8d8c1475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60760/0 (socket says 172.21.15.72:60760) 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3655143955 learned_addr learned my addr 172.21.15.72:0/3655143955 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d91252640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d8c15bdd0 0x7f8d8c147b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d90a51640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d8c0b25d0 0x7f8d8c0c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d8c0b25d0 msgr2=0x7f8d8c0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d8c0b25d0 0x7f8d8c0c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d8c15bdd0 msgr2=0x7f8d8c147b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d8c15bdd0 0x7f8d8c147b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:36.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3655143955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0c72d0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d90a51640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d8c0b25d0 0x7f8d8c0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 0x7f8d8c1475d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800c420 tx=0x7f8d7800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d780074e0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d780047e0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d78004c10 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c0c75b0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.640+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c0c7b70 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.641+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c09fcf0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.641+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d78011070 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.642+0000 7f8d897fa640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7006b500 0x7f8d7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.642+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f8d78089e80 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.642+0000 7f8d90a51640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7006b500 0x7f8d7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:36.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.642+0000 7f8d90a51640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7006b500 0x7f8d7006d9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8d80007fd0 tx=0x7f8d80009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:36.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.646+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d78058ba0 con 0x7f8d8c0b41b0 2024-03-06T07:22:36.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:36.851+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8d8c06f000 con 0x7f8d8c0b41b0 2024-03-06T07:22:37.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.518+0000 7f8d897fa640 1 -- 172.21.15.72:0/3655143955 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8d780584a0 con 0x7f8d8c0b41b0 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.520+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7006b500 msgr2=0x7f8d7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.520+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7006b500 0x7f8d7006d9c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8d80007fd0 tx=0x7f8d80009040 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.520+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 msgr2=0x7f8d8c1475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.520+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 0x7f8d8c1475d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800c420 tx=0x7f8d7800c8e0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 shutdown_connections 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7006b500 0x7f8d7006d9c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d8c15bdd0 0x7f8d8c147b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d8c0b41b0 0x7f8d8c1475d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 --2- 172.21.15.72:0/3655143955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d8c0b25d0 0x7f8d8c0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 >> 172.21.15.72:0/3655143955 conn(0x7f8d8c0b3690 msgr2=0x7f8d8c15b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 shutdown_connections 2024-03-06T07:22:37.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.521+0000 7f8d92cb5640 1 -- 172.21.15.72:0/3655143955 wait complete. 2024-03-06T07:22:37.535 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:22:37.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.677+0000 7f6848516640 1 Processor -- start 2024-03-06T07:22:37.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.677+0000 7f6848516640 1 -- start start 2024-03-06T07:22:37.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6848516640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f684015c5a0 0x7f684006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6848516640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68400ce8c0 0x7f68400cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6848516640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f684015c980 0x7f6840071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6848516640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6840072a20 con 0x7f684015c980 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6848516640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68400708b0 con 0x7f684015c5a0 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6848516640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6840070e70 con 0x7f68400ce8c0 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.678+0000 7f6846ab3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f684015c980 0x7f6840071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f684015c980 0x7f6840071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60762/0 (socket says 172.21.15.72:60762) 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 -- 172.21.15.72:0/3960190761 learned_addr learned my addr 172.21.15.72:0/3960190761 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f68462b2640 1 --2- 172.21.15.72:0/3960190761 >> v2:172.21.15.146:3300/0 conn(0x7f684015c5a0 0x7f684006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6845ab1640 1 --2- 172.21.15.72:0/3960190761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68400ce8c0 0x7f68400cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 -- 172.21.15.72:0/3960190761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68400ce8c0 msgr2=0x7f68400cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 --2- 172.21.15.72:0/3960190761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68400ce8c0 0x7f68400cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 -- 172.21.15.72:0/3960190761 >> v2:172.21.15.146:3300/0 conn(0x7f684015c5a0 msgr2=0x7f684006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 --2- 172.21.15.72:0/3960190761 >> v2:172.21.15.146:3300/0 conn(0x7f684015c5a0 0x7f684006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f6846ab3640 1 -- 172.21.15.72:0/3960190761 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684006ce50 con 0x7f684015c980 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.679+0000 7f68462b2640 1 --2- 172.21.15.72:0/3960190761 >> v2:172.21.15.146:3300/0 conn(0x7f684015c5a0 0x7f684006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6846ab3640 1 --2- 172.21.15.72:0/3960190761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684015c980 0x7f6840071300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f684006bf40 tx=0x7f683002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88aec674721552ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f68452b0640 1 -- 172.21.15.72:0/3960190761 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683002e070 con 0x7f684015c980 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f68452b0640 1 -- 172.21.15.72:0/3960190761 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6830037070 con 0x7f684015c980 2024-03-06T07:22:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f68452b0640 1 -- 172.21.15.72:0/3960190761 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830031a00 con 0x7f684015c980 2024-03-06T07:22:37.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 -- 172.21.15.72:0/3960190761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684015c980 msgr2=0x7f6840071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 --2- 172.21.15.72:0/3960190761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684015c980 0x7f6840071300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f684006bf40 tx=0x7f683002b880 comp rx=0 tx=0).stop 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 -- 172.21.15.72:0/3960190761 shutdown_connections 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 --2- 172.21.15.72:0/3960190761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684015c980 0x7f6840071300 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 --2- 172.21.15.72:0/3960190761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68400ce8c0 0x7f68400cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 --2- 172.21.15.72:0/3960190761 >> v2:172.21.15.146:3300/0 conn(0x7f684015c5a0 0x7f684006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 -- 172.21.15.72:0/3960190761 >> 172.21.15.72:0/3960190761 conn(0x7f68400b1650 msgr2=0x7f68400b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 -- 172.21.15.72:0/3960190761 shutdown_connections 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.680+0000 7f6848516640 1 -- 172.21.15.72:0/3960190761 wait complete. 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.681+0000 7f6848516640 1 Processor -- start 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 -- start start 2024-03-06T07:22:37.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684015c5a0 0x7f684013b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684015c980 0x7f68401f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 0x7f68400ac690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6840070b00 con 0x7f68400ac1e0 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400cf030 con 0x7f684015c5a0 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6848516640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68400cc870 con 0x7f684015c980 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6846ab3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 0x7f68400ac690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6846ab3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 0x7f68400ac690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60766/0 (socket says 172.21.15.72:60766) 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6846ab3640 1 -- 172.21.15.72:0/4025149396 learned_addr learned my addr 172.21.15.72:0/4025149396 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f68462b2640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684015c5a0 0x7f684013b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.682+0000 7f6845ab1640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684015c980 0x7f68401f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6846ab3640 1 -- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684015c980 msgr2=0x7f68401f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6846ab3640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684015c980 0x7f68401f2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6846ab3640 1 -- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684015c5a0 msgr2=0x7f684013b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6846ab3640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684015c5a0 0x7f684013b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6846ab3640 1 -- 172.21.15.72:0/4025149396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684013b7e0 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6845ab1640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684015c980 0x7f68401f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6846ab3640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 0x7f68400ac690 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f68300090b0 tx=0x7f68300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830032670 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6830032810 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68401f1880 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830032c00 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.683+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68400af140 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.684+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68400687d0 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.684+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f683004c060 con 0x7f68400ac1e0 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.685+0000 7f683effd640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f682c06b4d0 0x7f682c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:37.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.685+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f68300abd40 con 0x7f68400ac1e0 2024-03-06T07:22:37.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.685+0000 7f68462b2640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f682c06b4d0 0x7f682c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:37.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.685+0000 7f68462b2640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f682c06b4d0 0x7f682c06d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6838003b80 tx=0x7f6838009890 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:37.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.689+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6830040070 con 0x7f68400ac1e0 2024-03-06T07:22:37.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:37.861+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f684004f330 con 0x7f68400ac1e0 2024-03-06T07:22:38.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.523+0000 7f683effd640 1 -- 172.21.15.72:0/4025149396 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f683007ac10 con 0x7f68400ac1e0 2024-03-06T07:22:38.525 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:22:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.527+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f682c06b4d0 msgr2=0x7f682c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.527+0000 7f6848516640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f682c06b4d0 0x7f682c06d990 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6838003b80 tx=0x7f6838009890 comp rx=0 tx=0).stop 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 msgr2=0x7f68400ac690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 0x7f68400ac690 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f68300090b0 tx=0x7f68300046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 shutdown_connections 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f682c06b4d0 0x7f682c06d990 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68400ac1e0 0x7f68400ac690 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684015c980 0x7f68401f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 --2- 172.21.15.72:0/4025149396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684015c5a0 0x7f684013b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 >> 172.21.15.72:0/4025149396 conn(0x7f68400b1650 msgr2=0x7f68400b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 shutdown_connections 2024-03-06T07:22:38.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.528+0000 7f6848516640 1 -- 172.21.15.72:0/4025149396 wait complete. 2024-03-06T07:22:38.544 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:22:38.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff9a600f640 1 Processor -- start 2024-03-06T07:22:38.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff9a600f640 1 -- start start 2024-03-06T07:22:38.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff9a600f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff9a00d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff9a600f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a015e380 con 0x7ff9a00d29f0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff9a600f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a00d1e30 con 0x7ff9a00d4630 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff9a600f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00d2100 con 0x7ff9a015dfa0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff99ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff9a00d29f0 0x7ff9a00cb190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35380/0 (socket says 172.21.15.72:35380) 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.648+0000 7ff99ffff640 1 -- 172.21.15.72:0/712382750 learned_addr learned my addr 172.21.15.72:0/712382750 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff9a00d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946044504 0 0) 0x7ff9a015e380 con 0x7ff9a00d29f0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff988003520 con 0x7ff9a00d29f0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200774651 0 0) 0x7ff988003520 con 0x7ff9a00d29f0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 msgr2=0x7ff9a00d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff9a00d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 >> v1:172.21.15.146:6789/0 conn(0x7ff9a00d4630 legacy=0x7ff9a00d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00cf470 con 0x7ff9a00d29f0 2024-03-06T07:22:38.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff9980019f0 con 0x7ff9a00d29f0 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 >> v1:172.21.15.72:6789/0 conn(0x7ff9a00d29f0 legacy=0x7ff9a00cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9880039e0 0x7ff988003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff988004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff988004d40 0x7ff9880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00d2100 con 0x7ff9880039e0 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800baf0 con 0x7ff988004d40 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800bcf0 con 0x7ff9a015dfa0 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9880039e0 0x7ff988003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99effd640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff988004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99ffff640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff988004d40 0x7ff9880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 -- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 msgr2=0x7ff988004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff988004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 -- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff988004d40 msgr2=0x7ff9880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff988004d40 0x7ff9880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99f7fe640 1 -- 172.21.15.72:0/712382750 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988003520 con 0x7ff9880039e0 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99ffff640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff988004d40 0x7ff9880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.649+0000 7ff99effd640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9a015dfa0 0x7ff988004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.650+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9880039e0 0x7ff988003e00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff990009ac0 tx=0x7ff988005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da17ec23c8c41e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.650+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff990039070 con 0x7ff9880039e0 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.650+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99002c600 con 0x7ff9880039e0 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.650+0000 7ff99e7fc640 1 -- 172.21.15.72:0/712382750 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff990009690 con 0x7ff9880039e0 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff9a600f640 1 -- 172.21.15.72:0/712382750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9880039e0 msgr2=0x7ff988003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff9a600f640 1 --2- 172.21.15.72:0/712382750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9880039e0 0x7ff988003e00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff990009ac0 tx=0x7ff988005830 comp rx=0 tx=0).stop 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff99ffff640 1 -- 172.21.15.72:0/712382750 reap_dead start 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff9a600f640 1 -- 172.21.15.72:0/712382750 shutdown_connections 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff9a600f640 1 -- 172.21.15.72:0/712382750 >> 172.21.15.72:0/712382750 conn(0x7ff9a009d000 msgr2=0x7ff9a00d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff9a600f640 1 -- 172.21.15.72:0/712382750 shutdown_connections 2024-03-06T07:22:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.651+0000 7ff9a600f640 1 -- 172.21.15.72:0/712382750 wait complete. 2024-03-06T07:22:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.652+0000 7ff9a600f640 1 Processor -- start 2024-03-06T07:22:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.652+0000 7ff9a600f640 1 -- start start 2024-03-06T07:22:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9880039e0 0x7ff9a01f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a0154630 0x7ff9a01f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 0x7ff9a0156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00d18f0 con 0x7ff9a0156470 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00cfa30 con 0x7ff9a0154630 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a00d1a70 con 0x7ff9880039e0 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 0x7ff9a0156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 0x7ff9a0156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60788/0 (socket says 172.21.15.72:60788) 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 -- 172.21.15.72:0/3354170167 learned_addr learned my addr 172.21.15.72:0/3354170167 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99effd640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a0154630 0x7ff9a01f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9880039e0 0x7ff9a01f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 -- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9880039e0 msgr2=0x7ff9a01f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9880039e0 0x7ff9a01f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 -- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a0154630 msgr2=0x7ff9a01f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a0154630 0x7ff9a01f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:38.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 -- 172.21.15.72:0/3354170167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00b77a0 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99effd640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a0154630 0x7ff9a01f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9880039e0 0x7ff9a01f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff99ffff640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 0x7ff9a0156920 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff998000f80 tx=0x7ff99800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99800bb80 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99800bd40 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a00b7a10 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.653+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980072b0 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.654+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a00b7fd0 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.654+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a00b81d0 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.655+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff99801b020 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.655+0000 7ff987fff640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff98c06b500 0x7ff98c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.655+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff98c06b500 0x7ff98c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.655+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5473+0+0 (secure 0 0 0) 0x7ff99808a370 con 0x7ff9a0156470 2024-03-06T07:22:38.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.656+0000 7ff99f7fe640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff98c06b500 0x7ff98c06d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff990036000 tx=0x7ff990030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:38.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.659+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff9a00b81d0 con 0x7ff9a0156470 2024-03-06T07:22:38.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:38.830+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff9a00b5390 con 0x7ff9a0156470 2024-03-06T07:22:39.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.528+0000 7ff987fff640 1 -- 172.21.15.72:0/3354170167 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff9a00b5390 con 0x7ff9a0156470 2024-03-06T07:22:39.530 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.530+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff98c06b500 msgr2=0x7ff98c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.530+0000 7ff9a600f640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff98c06b500 0x7ff98c06d9c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff990036000 tx=0x7ff990030000 comp rx=0 tx=0).stop 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.530+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 msgr2=0x7ff9a0156920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.530+0000 7ff9a600f640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 0x7ff9a0156920 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff998000f80 tx=0x7ff99800b4a0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 shutdown_connections 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff98c06b500 0x7ff98c06d9c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9a0156470 0x7ff9a0156920 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9a0154630 0x7ff9a01f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 --2- 172.21.15.72:0/3354170167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9880039e0 0x7ff9a01f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 >> 172.21.15.72:0/3354170167 conn(0x7ff9a009d000 msgr2=0x7ff9a00d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 shutdown_connections 2024-03-06T07:22:39.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.531+0000 7ff9a600f640 1 -- 172.21.15.72:0/3354170167 wait complete. 2024-03-06T07:22:39.548 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:22:39.548 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:22:39.548 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:22:39.548 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:39.549 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:22:39.549 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:22:39.549 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:22:39.549 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:39.549 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:22:39.549 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:22:39.550 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:22:39.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:39.550 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:22:39.550 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:22:39.550 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:22:39.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:22:39.558 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:22:39.558 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:22:39.559 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:22:39.559 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:22:39.561 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:22:39.561 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:22:39.562 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:22:39.562 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:22:39.601 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:22:39.601 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:39.602 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:22:39.602 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:39.604 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:22:39.604 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:39.605 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:22:39.605 DEBUG:teuthology.parallel:result is None 2024-03-06T07:22:39.605 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:22:39.679 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:22:39.679 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:22:39.685 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:22:39.685 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:22:39.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.711+0000 7f4de9943640 1 Processor -- start 2024-03-06T07:22:39.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de9943640 1 -- start start 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de9943640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4de4162970 0x7f4de40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de9943640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4de4161cf0 0x7f4de4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de9943640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40d22c0 con 0x7f4de40b4460 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de9943640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de4161b00 con 0x7f4de4161cf0 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de9943640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de40cecc0 con 0x7f4de4162970 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4de4161cf0 0x7f4de4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4de4161cf0 0x7f4de4162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60828/0 (socket says 172.21.15.72:60828) 2024-03-06T07:22:39.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.712+0000 7f4de37fe640 1 -- 172.21.15.72:0/680733685 learned_addr learned my addr 172.21.15.72:0/680733685 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:39.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.713+0000 7f4de27fc640 1 --2- 172.21.15.72:0/680733685 >> v2:172.21.15.146:3300/0 conn(0x7f4de4162970 0x7f4de40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.713+0000 7f4de37fe640 1 -- 172.21.15.72:0/680733685 >> v2:172.21.15.146:3300/0 conn(0x7f4de4162970 msgr2=0x7f4de40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:39.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.713+0000 7f4de37fe640 1 --2- 172.21.15.72:0/680733685 >> v2:172.21.15.146:3300/0 conn(0x7f4de4162970 0x7f4de40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.713+0000 7f4de37fe640 1 -- 172.21.15.72:0/680733685 >> v1:172.21.15.146:6789/0 conn(0x7f4de40b4460 legacy=0x7f4de40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.713+0000 7f4de37fe640 1 -- 172.21.15.72:0/680733685 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40d25a0 con 0x7f4de4161cf0 2024-03-06T07:22:39.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de37fe640 1 --2- 172.21.15.72:0/680733685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de4161cf0 0x7f4de4162180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4de40d17e0 tx=0x7f4ddc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e017b7af69dd5d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de1ffb640 1 -- 172.21.15.72:0/680733685 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc02cb20 con 0x7f4de4161cf0 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de1ffb640 1 -- 172.21.15.72:0/680733685 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc02ccc0 con 0x7f4de4161cf0 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de1ffb640 1 -- 172.21.15.72:0/680733685 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc034610 con 0x7f4de4161cf0 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 -- 172.21.15.72:0/680733685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de4161cf0 msgr2=0x7f4de4162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 --2- 172.21.15.72:0/680733685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de4161cf0 0x7f4de4162180 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4de40d17e0 tx=0x7f4ddc02bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 -- 172.21.15.72:0/680733685 shutdown_connections 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 --2- 172.21.15.72:0/680733685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de4161cf0 0x7f4de4162180 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 --2- 172.21.15.72:0/680733685 >> v2:172.21.15.146:3300/0 conn(0x7f4de4162970 0x7f4de40d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 -- 172.21.15.72:0/680733685 >> 172.21.15.72:0/680733685 conn(0x7f4de40d3ce0 msgr2=0x7f4de40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 -- 172.21.15.72:0/680733685 shutdown_connections 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.714+0000 7f4de9943640 1 -- 172.21.15.72:0/680733685 wait complete. 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.716+0000 7f4de9943640 1 Processor -- start 2024-03-06T07:22:39.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.716+0000 7f4de9943640 1 -- start start 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 0x7f4de40b9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de4161cf0 0x7f4de41f4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de4162970 0x7f4de41f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40ce950 con 0x7f4de40b4460 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 0x7f4de40b9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40d2b60 con 0x7f4de4161cf0 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40b4840 con 0x7f4de4162970 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 0x7f4de40b9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60834/0 (socket says 172.21.15.72:60834) 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 -- 172.21.15.72:0/1955103917 learned_addr learned my addr 172.21.15.72:0/1955103917 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:39.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de27fc640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de4161cf0 0x7f4de41f4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de37fe640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de4162970 0x7f4de41f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 -- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de4162970 msgr2=0x7f4de41f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de4162970 0x7f4de41f3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 -- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de4161cf0 msgr2=0x7f4de41f4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de4161cf0 0x7f4de41f4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 -- 172.21.15.72:0/1955103917 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de41f4620 con 0x7f4de40b4460 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de2ffd640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 0x7f4de40b9070 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400c3c0 tx=0x7f4dd400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd40052d0 con 0x7f4de40b4460 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd4005490 con 0x7f4de40b4460 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de41f4900 con 0x7f4de40b4460 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.717+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4007260 con 0x7f4de40b4460 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.718+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de40cd7a0 con 0x7f4de40b4460 2024-03-06T07:22:39.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.719+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de40b6900 con 0x7f4de40b4460 2024-03-06T07:22:39.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.719+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dd4007400 con 0x7f4de40b4460 2024-03-06T07:22:39.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.720+0000 7f4ddbfff640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc406b4d0 0x7f4dc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:39.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.720+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4dd40892f0 con 0x7f4de40b4460 2024-03-06T07:22:39.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.720+0000 7f4de27fc640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc406b4d0 0x7f4dc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:39.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.720+0000 7f4de27fc640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc406b4d0 0x7f4dc406d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4de4160d60 tx=0x7f4dcc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:39.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.724+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dd4057e20 con 0x7f4de40b4460 2024-03-06T07:22:39.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:39.907+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4de404f330 con 0x7f4de40b4460 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.547+0000 7f4ddbfff640 1 -- 172.21.15.72:0/1955103917 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4dd405bec0 con 0x7f4de40b4460 2024-03-06T07:22:40.549 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc406b4d0 msgr2=0x7f4dc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc406b4d0 0x7f4dc406d990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4de4160d60 tx=0x7f4dcc009190 comp rx=0 tx=0).stop 2024-03-06T07:22:40.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 msgr2=0x7f4de40b9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 0x7f4de40b9070 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400c3c0 tx=0x7f4dd400c880 comp rx=0 tx=0).stop 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 shutdown_connections 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc406b4d0 0x7f4dc406d990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4de4162970 0x7f4de41f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4de4161cf0 0x7f4de41f4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 --2- 172.21.15.72:0/1955103917 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4de40b4460 0x7f4de40b9070 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 >> 172.21.15.72:0/1955103917 conn(0x7f4de40d3ce0 msgr2=0x7f4de40cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 shutdown_connections 2024-03-06T07:22:40.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.549+0000 7f4de9943640 1 -- 172.21.15.72:0/1955103917 wait complete. 2024-03-06T07:22:40.568 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:22:40.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.668+0000 7fb241c83640 1 Processor -- start 2024-03-06T07:22:40.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.668+0000 7fb241c83640 1 -- start start 2024-03-06T07:22:40.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb241c83640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb23c0cf790 0x7fb23c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb241c83640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb23c0adda0 0x7fb23c071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb241c83640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c1619b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb241c83640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c06cf70 con 0x7fb23c0adda0 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb241c83640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23c0ae240 con 0x7fb23c0cf790 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb241c83640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c0ae520 con 0x7fb23c072220 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb23c0adda0 0x7fb23c071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb23c0adda0 0x7fb23c071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60848/0 (socket says 172.21.15.72:60848) 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 -- 172.21.15.72:0/4122107668 learned_addr learned my addr 172.21.15.72:0/4122107668 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23b7fe640 1 --2- 172.21.15.72:0/4122107668 >> v2:172.21.15.146:3300/0 conn(0x7fb23c0cf790 0x7fb23c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23bfff640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c1619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 -- 172.21.15.72:0/4122107668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 msgr2=0x7fb23c1619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c1619b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 -- 172.21.15.72:0/4122107668 >> v2:172.21.15.146:3300/0 conn(0x7fb23c0cf790 msgr2=0x7fb23c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 --2- 172.21.15.72:0/4122107668 >> v2:172.21.15.146:3300/0 conn(0x7fb23c0cf790 0x7fb23c0ad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23affd640 1 -- 172.21.15.72:0/4122107668 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c0ae800 con 0x7fb23c0adda0 2024-03-06T07:22:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23b7fe640 1 --2- 172.21.15.72:0/4122107668 >> v2:172.21.15.146:3300/0 conn(0x7fb23c0cf790 0x7fb23c0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.669+0000 7fb23bfff640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c1619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.670+0000 7fb23affd640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c071ce0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0b27b0 tx=0x7fb22c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d2204b33455a08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.670+0000 7fb23a7fc640 1 -- 172.21.15.72:0/4122107668 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c035020 con 0x7fb23c0adda0 2024-03-06T07:22:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.670+0000 7fb23a7fc640 1 -- 172.21.15.72:0/4122107668 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c02d040 con 0x7fb23c0adda0 2024-03-06T07:22:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.670+0000 7fb23a7fc640 1 -- 172.21.15.72:0/4122107668 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c031a80 con 0x7fb23c0adda0 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 -- 172.21.15.72:0/4122107668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 msgr2=0x7fb23c071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c071ce0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb23c0b27b0 tx=0x7fb22c02b900 comp rx=0 tx=0).stop 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 -- 172.21.15.72:0/4122107668 shutdown_connections 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c1619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 --2- 172.21.15.72:0/4122107668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c071ce0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 --2- 172.21.15.72:0/4122107668 >> v2:172.21.15.146:3300/0 conn(0x7fb23c0cf790 0x7fb23c0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 -- 172.21.15.72:0/4122107668 >> 172.21.15.72:0/4122107668 conn(0x7fb23c0b1a90 msgr2=0x7fb23c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 -- 172.21.15.72:0/4122107668 shutdown_connections 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.671+0000 7fb241c83640 1 -- 172.21.15.72:0/4122107668 wait complete. 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.672+0000 7fb241c83640 1 Processor -- start 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.672+0000 7fb241c83640 1 -- start start 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb241c83640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb241c83640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c0c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb241c83640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb23c0cf790 0x7fb23c0c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb241c83640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c15e700 con 0x7fb23c0adda0 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb241c83640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c15e580 con 0x7fb23c0cf790 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb241c83640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c161ef0 con 0x7fb23c072220 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c0c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c0c4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60862/0 (socket says 172.21.15.72:60862) 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 -- 172.21.15.72:0/719280518 learned_addr learned my addr 172.21.15.72:0/719280518 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 -- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 msgr2=0x7fb23c146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 -- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb23c0cf790 msgr2=0x7fb23c0c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23bfff640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb23c0cf790 0x7fb23c0c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb23c0cf790 0x7fb23c0c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.673+0000 7fb23affd640 1 -- 172.21.15.72:0/719280518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb23c15c270 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.674+0000 7fb23affd640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c0c4d00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb22c000c00 tx=0x7fb22c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.674+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c040070 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.674+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c039bf0 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.674+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c15d1e0 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.674+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c035040 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.674+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c15d7a0 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.675+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c0aedb0 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.675+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb22c038020 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.675+0000 7fb223fff640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb22806b4d0 0x7fb22806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.676+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb22c0abc20 con 0x7fb23c0adda0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.676+0000 7fb23b7fe640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb22806b4d0 0x7fb22806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.676+0000 7fb23b7fe640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb22806b4d0 0x7fb22806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb230006fd0 tx=0x7fb230008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:40.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.680+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb22c07adf0 con 0x7fb23c0adda0 2024-03-06T07:22:40.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.915+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb23c15c5b0 con 0x7fb23c0adda0 2024-03-06T07:22:40.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.915+0000 7fb223fff640 1 -- 172.21.15.72:0/719280518 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fb22c07a6f0 con 0x7fb23c0adda0 2024-03-06T07:22:40.917 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:40.917 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:22:40.917 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:22:40.918 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:22:40.919 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.920 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.921 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.922 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.923 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.924 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:22:40.925 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.926 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.927 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:22:40.928 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-mds", 2024-03-06T07:22:40.929 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQB1GehlzOfjKRAAPA4hdYlz2MMfKo1TEui4mg==", 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-mds" 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-mgr", 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQB1Gehl5vHjKRAAlnqvuaH+MZJb3xNbfad8Yg==", 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.930 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-osd", 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQB1GehlhvjjKRAA+qYIKwoh7YxL/Mna3DuBZw==", 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-osd" 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-rbd", 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQB1Gehl7/7jKRAAVr00BW5x9rpbppewiZ2KKA==", 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-06T07:22:40.931 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQB1GehljQXkKRAAcQw5wMnNKsiVxYQpBJxaQw==", 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-rgw", 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQB1GehlYAzkKRAAZe8WJq+C5ZwPE31RKSi27g==", 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:22:40.932 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:22:40.933 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.934 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:40.934 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:22:40.934 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:22:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb22806b4d0 msgr2=0x7fb22806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb22806b4d0 0x7fb22806d990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb230006fd0 tx=0x7fb230008040 comp rx=0 tx=0).stop 2024-03-06T07:22:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 msgr2=0x7fb23c0c4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c0c4d00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb22c000c00 tx=0x7fb22c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 shutdown_connections 2024-03-06T07:22:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb22806b4d0 0x7fb22806d990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb23c0cf790 0x7fb23c0c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb23c0adda0 0x7fb23c0c4d00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 --2- 172.21.15.72:0/719280518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23c072220 0x7fb23c146830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:40.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 >> 172.21.15.72:0/719280518 conn(0x7fb23c0b1a90 msgr2=0x7fb23c160640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:40.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 shutdown_connections 2024-03-06T07:22:40.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:40.917+0000 7fb241c83640 1 -- 172.21.15.72:0/719280518 wait complete. 2024-03-06T07:22:40.936 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-06T07:22:41.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.035+0000 7fafc4e8b640 1 Processor -- start 2024-03-06T07:22:41.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.035+0000 7fafc4e8b640 1 -- start start 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafc4e8b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc015de40 0x7fafc00cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafc4e8b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fafc00d2090 0x7fafc00ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafc4e8b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fafc006c570 0x7fafc006ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafc4e8b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc00cc4e0 con 0x7fafc00d2090 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafc4e8b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc00cf9e0 con 0x7fafc006c570 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafc4e8b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00ce710 con 0x7fafc015de40 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafbeffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fafc00d2090 0x7fafc00ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafbeffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fafc00d2090 0x7fafc00ce1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60866/0 (socket says 172.21.15.72:60866) 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafbeffd640 1 -- 172.21.15.72:0/3929052200 learned_addr learned my addr 172.21.15.72:0/3929052200 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:41.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafbffff640 1 --2- 172.21.15.72:0/3929052200 >> v2:172.21.15.146:3300/0 conn(0x7fafc006c570 0x7fafc006ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.036+0000 7fafbf7fe640 1 --2- 172.21.15.72:0/3929052200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc015de40 0x7fafc00cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbeffd640 1 -- 172.21.15.72:0/3929052200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc015de40 msgr2=0x7fafc00cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbeffd640 1 --2- 172.21.15.72:0/3929052200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc015de40 0x7fafc00cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbeffd640 1 -- 172.21.15.72:0/3929052200 >> v2:172.21.15.146:3300/0 conn(0x7fafc006c570 msgr2=0x7fafc006ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbeffd640 1 --2- 172.21.15.72:0/3929052200 >> v2:172.21.15.146:3300/0 conn(0x7fafc006c570 0x7fafc006ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbeffd640 1 -- 172.21.15.72:0/3929052200 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc00ce9b0 con 0x7fafc00d2090 2024-03-06T07:22:41.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbeffd640 1 --2- 172.21.15.72:0/3929052200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc00d2090 0x7fafc00ce1d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fafc00b2e10 tx=0x7fafb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a87e1a253bdcdc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbe7fc640 1 -- 172.21.15.72:0/3929052200 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4035020 con 0x7fafc00d2090 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbe7fc640 1 -- 172.21.15.72:0/3929052200 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb4031870 con 0x7fafc00d2090 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.037+0000 7fafbe7fc640 1 -- 172.21.15.72:0/3929052200 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4031c40 con 0x7fafc00d2090 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 -- 172.21.15.72:0/3929052200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc00d2090 msgr2=0x7fafc00ce1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/3929052200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc00d2090 0x7fafc00ce1d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fafc00b2e10 tx=0x7fafb402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 -- 172.21.15.72:0/3929052200 shutdown_connections 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/3929052200 >> v2:172.21.15.146:3300/0 conn(0x7fafc006c570 0x7fafc006ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/3929052200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc00d2090 0x7fafc00ce1d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/3929052200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc015de40 0x7fafc00cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 -- 172.21.15.72:0/3929052200 >> 172.21.15.72:0/3929052200 conn(0x7fafc009a9f0 msgr2=0x7fafc009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 -- 172.21.15.72:0/3929052200 shutdown_connections 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.038+0000 7fafc4e8b640 1 -- 172.21.15.72:0/3929052200 wait complete. 2024-03-06T07:22:41.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.039+0000 7fafc4e8b640 1 Processor -- start 2024-03-06T07:22:41.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.039+0000 7fafc4e8b640 1 -- start start 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafc4e8b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc006c570 0x7fafc0160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafc4e8b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafc00d2090 0x7fafc01572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafc4e8b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 0x7fafc015e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafc4e8b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00cfc80 con 0x7fafc015de40 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafc4e8b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00cef70 con 0x7fafc00d2090 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafc4e8b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc00d01a0 con 0x7fafc006c570 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafbffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 0x7fafc015e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafbffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 0x7fafc015e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60878/0 (socket says 172.21.15.72:60878) 2024-03-06T07:22:41.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafbffff640 1 -- 172.21.15.72:0/360829445 learned_addr learned my addr 172.21.15.72:0/360829445 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafbf7fe640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc006c570 0x7fafc0160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.040+0000 7fafbeffd640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafc00d2090 0x7fafc01572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbffff640 1 -- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc006c570 msgr2=0x7fafc0160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbffff640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc006c570 0x7fafc0160a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbffff640 1 -- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafc00d2090 msgr2=0x7fafc01572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbffff640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafc00d2090 0x7fafc01572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbffff640 1 -- 172.21.15.72:0/360829445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc0157d60 con 0x7fafc015de40 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbeffd640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafc00d2090 0x7fafc01572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafbffff640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 0x7fafc015e2e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fafb8007e40 tx=0x7fafb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8014620 con 0x7fafc015de40 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb8019070 con 0x7fafc015de40 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc0158040 con 0x7fafc015de40 2024-03-06T07:22:41.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8014a20 con 0x7fafc015de40 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.041+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc0158600 con 0x7fafc015de40 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.042+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc0068f80 con 0x7fafc015de40 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.042+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fafb8014bc0 con 0x7fafc015de40 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.043+0000 7fafa7fff640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafac06b4d0 0x7fafac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.043+0000 7fafbf7fe640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafac06b4d0 0x7fafac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.043+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fafb808a760 con 0x7fafc015de40 2024-03-06T07:22:41.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.043+0000 7fafbf7fe640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafac06b4d0 0x7fafac06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fafb0007920 tx=0x7fafb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.047+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fafb8059930 con 0x7fafc015de40 2024-03-06T07:22:41.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.283+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fafc00cb830 con 0x7fafc015de40 2024-03-06T07:22:41.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.285+0000 7fafa7fff640 1 -- 172.21.15.72:0/360829445 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fafb8059230 con 0x7fafc015de40 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafac06b4d0 msgr2=0x7fafac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafac06b4d0 0x7fafac06d990 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fafb0007920 tx=0x7fafb0008040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 msgr2=0x7fafc015e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 0x7fafc015e2e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fafb8007e40 tx=0x7fafb800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 shutdown_connections 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fafac06b4d0 0x7fafac06d990 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc015de40 0x7fafc015e2e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafc00d2090 0x7fafc01572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 --2- 172.21.15.72:0/360829445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafc006c570 0x7fafc0160a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 >> 172.21.15.72:0/360829445 conn(0x7fafc009a9f0 msgr2=0x7fafc015bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 shutdown_connections 2024-03-06T07:22:41.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.287+0000 7fafc4e8b640 1 -- 172.21.15.72:0/360829445 wait complete. 2024-03-06T07:22:41.300 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-06T07:22:41.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.398+0000 7f0f3d29d640 1 Processor -- start 2024-03-06T07:22:41.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.398+0000 7f0f3d29d640 1 -- start start 2024-03-06T07:22:41.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f3d29d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f380ce060 0x7f0f3806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f3d29d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f3806b4f0 0x7f0f380719e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f3d29d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380cfff0 con 0x7f0f38145760 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f3d29d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3806b920 con 0x7f0f3806b4f0 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f3d29d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380cfd10 con 0x7f0f380ce060 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f377fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f3806b4f0 0x7f0f380719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f377fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f3806b4f0 0x7f0f380719e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60890/0 (socket says 172.21.15.72:60890) 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.399+0000 7f0f377fe640 1 -- 172.21.15.72:0/4089919828 learned_addr learned my addr 172.21.15.72:0/4089919828 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.400+0000 7f0f367fc640 1 --2- 172.21.15.72:0/4089919828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f380ce060 0x7f0f3806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.400+0000 7f0f377fe640 1 -- 172.21.15.72:0/4089919828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f380ce060 msgr2=0x7f0f3806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.400+0000 7f0f377fe640 1 --2- 172.21.15.72:0/4089919828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f380ce060 0x7f0f3806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.400+0000 7f0f377fe640 1 -- 172.21.15.72:0/4089919828 >> v1:172.21.15.146:6789/0 conn(0x7f0f38145760 legacy=0x7f0f380cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.400+0000 7f0f377fe640 1 -- 172.21.15.72:0/4089919828 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380cd580 con 0x7f0f3806b4f0 2024-03-06T07:22:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f377fe640 1 --2- 172.21.15.72:0/4089919828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f3806b4f0 0x7f0f380719e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009870 tx=0x7f0f3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de11bb64c4084e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f35ffb640 1 -- 172.21.15.72:0/4089919828 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30034070 con 0x7f0f3806b4f0 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f35ffb640 1 -- 172.21.15.72:0/4089919828 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f30030840 con 0x7f0f3806b4f0 2024-03-06T07:22:41.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f35ffb640 1 -- 172.21.15.72:0/4089919828 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f30030c10 con 0x7f0f3806b4f0 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 -- 172.21.15.72:0/4089919828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f3806b4f0 msgr2=0x7f0f380719e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/4089919828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f3806b4f0 0x7f0f380719e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009870 tx=0x7f0f3002b980 comp rx=0 tx=0).stop 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 -- 172.21.15.72:0/4089919828 shutdown_connections 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/4089919828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f3806b4f0 0x7f0f380719e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/4089919828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f380ce060 0x7f0f3806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 -- 172.21.15.72:0/4089919828 >> 172.21.15.72:0/4089919828 conn(0x7f0f380d09d0 msgr2=0x7f0f380d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 -- 172.21.15.72:0/4089919828 shutdown_connections 2024-03-06T07:22:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.401+0000 7f0f3d29d640 1 -- 172.21.15.72:0/4089919828 wait complete. 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 Processor -- start 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 -- start start 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f3806b4f0 0x7f0f380c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 0x7f0f380c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f38145760 0x7f0f380c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3815d1e0 con 0x7f0f380ce060 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380cdb40 con 0x7f0f38145760 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f3d29d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f380cf980 con 0x7f0f3806b4f0 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f367fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 0x7f0f380c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f367fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 0x7f0f380c5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60892/0 (socket says 172.21.15.72:60892) 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.403+0000 7f0f367fc640 1 -- 172.21.15.72:0/688335114 learned_addr learned my addr 172.21.15.72:0/688335114 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f377fe640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f38145760 0x7f0f380c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f36ffd640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f3806b4f0 0x7f0f380c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f367fc640 1 -- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f3806b4f0 msgr2=0x7f0f380c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f367fc640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f3806b4f0 0x7f0f380c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f367fc640 1 -- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f38145760 msgr2=0x7f0f380c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f367fc640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f38145760 0x7f0f380c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f367fc640 1 -- 172.21.15.72:0/688335114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380c64a0 con 0x7f0f380ce060 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f377fe640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f38145760 0x7f0f380c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f367fc640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 0x7f0f380c5a20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00d830 tx=0x7f0f2c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00b660 con 0x7f0f380ce060 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c019070 con 0x7f0f380ce060 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00ba60 con 0x7f0f380ce060 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f380c6750 con 0x7f0f380ce060 2024-03-06T07:22:41.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.404+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f380c6ff0 con 0x7f0f380ce060 2024-03-06T07:22:41.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.405+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f380c6b40 con 0x7f0f380ce060 2024-03-06T07:22:41.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.405+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f2c01e460 con 0x7f0f380ce060 2024-03-06T07:22:41.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.406+0000 7f0f1f7fe640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f2406b4d0 0x7f0f2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.406+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0f2c08a540 con 0x7f0f380ce060 2024-03-06T07:22:41.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.406+0000 7f0f36ffd640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f2406b4d0 0x7f0f2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.406+0000 7f0f36ffd640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f2406b4d0 0x7f0f2406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0f28001ca0 tx=0x7f0f28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.410+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f2c059710 con 0x7f0f380ce060 2024-03-06T07:22:41.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.648+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f0f3804f330 con 0x7f0f380ce060 2024-03-06T07:22:41.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.650+0000 7f0f1f7fe640 1 -- 172.21.15.72:0/688335114 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0f2c059010 con 0x7f0f380ce060 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f2406b4d0 msgr2=0x7f0f2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f2406b4d0 0x7f0f2406d990 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0f28001ca0 tx=0x7f0f28009040 comp rx=0 tx=0).stop 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 msgr2=0x7f0f380c5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 0x7f0f380c5a20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00d830 tx=0x7f0f2c00dcf0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 shutdown_connections 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f2406b4d0 0x7f0f2406d990 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f38145760 0x7f0f380c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f380ce060 0x7f0f380c5a20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 --2- 172.21.15.72:0/688335114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f3806b4f0 0x7f0f380c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.652+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 >> 172.21.15.72:0/688335114 conn(0x7f0f380d09d0 msgr2=0x7f0f380b53e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.653+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 shutdown_connections 2024-03-06T07:22:41.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.653+0000 7f0f3d29d640 1 -- 172.21.15.72:0/688335114 wait complete. 2024-03-06T07:22:41.665 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-06T07:22:41.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.764+0000 7f8d0d80e640 1 Processor -- start 2024-03-06T07:22:41.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.764+0000 7f8d0d80e640 1 -- start start 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d0d80e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8d080b3240 0x7f8d080b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d0d80e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d080b2910 0x7f8d080d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d0d80e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0815e3f0 con 0x7f8d0815e010 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d0d80e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d080b1f40 con 0x7f8d080b2910 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d0d80e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d080b1c10 con 0x7f8d080b3240 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d080b2910 0x7f8d080d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d080b2910 0x7f8d080d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60894/0 (socket says 172.21.15.72:60894) 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 -- 172.21.15.72:0/3258468253 learned_addr learned my addr 172.21.15.72:0/3258468253 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d067fc640 1 --2- 172.21.15.72:0/3258468253 >> v2:172.21.15.146:3300/0 conn(0x7f8d080b3240 0x7f8d080b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 -- 172.21.15.72:0/3258468253 >> v2:172.21.15.146:3300/0 conn(0x7f8d080b3240 msgr2=0x7f8d080b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 --2- 172.21.15.72:0/3258468253 >> v2:172.21.15.146:3300/0 conn(0x7f8d080b3240 0x7f8d080b23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 -- 172.21.15.72:0/3258468253 >> v1:172.21.15.146:6789/0 conn(0x7f8d0815e010 legacy=0x7f8d080cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:41.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.765+0000 7f8d077fe640 1 -- 172.21.15.72:0/3258468253 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d0806d1b0 con 0x7f8d080b2910 2024-03-06T07:22:41.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d077fe640 1 --2- 172.21.15.72:0/3258468253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d080b2910 0x7f8d080d1ad0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8d0806e2a0 tx=0x7f8d0002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3c07fe7574ff83 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d05ffb640 1 -- 172.21.15.72:0/3258468253 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d0002c9f0 con 0x7f8d080b2910 2024-03-06T07:22:41.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d05ffb640 1 -- 172.21.15.72:0/3258468253 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0002cb90 con 0x7f8d080b2910 2024-03-06T07:22:41.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d05ffb640 1 -- 172.21.15.72:0/3258468253 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d000343f0 con 0x7f8d080b2910 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 -- 172.21.15.72:0/3258468253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d080b2910 msgr2=0x7f8d080d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/3258468253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d080b2910 0x7f8d080d1ad0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8d0806e2a0 tx=0x7f8d0002be50 comp rx=0 tx=0).stop 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 -- 172.21.15.72:0/3258468253 shutdown_connections 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/3258468253 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d080b2910 0x7f8d080d1ad0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/3258468253 >> v2:172.21.15.146:3300/0 conn(0x7f8d080b3240 0x7f8d080b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 -- 172.21.15.72:0/3258468253 >> 172.21.15.72:0/3258468253 conn(0x7f8d0809a9f0 msgr2=0x7f8d0809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.766+0000 7f8d0d80e640 1 -- 172.21.15.72:0/3258468253 shutdown_connections 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.767+0000 7f8d0d80e640 1 -- 172.21.15.72:0/3258468253 wait complete. 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 Processor -- start 2024-03-06T07:22:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 -- start start 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0815e010 0x7f8d08156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d081f49c0 0x7f8d0815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 0x7f8d0815aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d080d2010 con 0x7f8d08157150 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0806ced0 con 0x7f8d081f49c0 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.768+0000 7f8d0d80e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d080d2190 con 0x7f8d0815e010 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 0x7f8d0815aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 0x7f8d0815aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60910/0 (socket says 172.21.15.72:60910) 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 -- 172.21.15.72:0/1940328955 learned_addr learned my addr 172.21.15.72:0/1940328955 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d067fc640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d081f49c0 0x7f8d0815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d06ffd640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0815e010 0x7f8d08156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 -- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0815e010 msgr2=0x7f8d08156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0815e010 0x7f8d08156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 -- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d081f49c0 msgr2=0x7f8d0815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d081f49c0 0x7f8d0815a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 -- 172.21.15.72:0/1940328955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08158960 con 0x7f8d08157150 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d077fe640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 0x7f8d0815aa50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8d00036040 tx=0x7f8d00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00043020 con 0x7f8d08157150 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d0003e450 con 0x7f8d08157150 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d08158c40 con 0x7f8d08157150 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d00009510 con 0x7f8d08157150 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.769+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d080b5aa0 con 0x7f8d08157150 2024-03-06T07:22:41.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.770+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d080b5ca0 con 0x7f8d08157150 2024-03-06T07:22:41.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.771+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d00005d10 con 0x7f8d08157150 2024-03-06T07:22:41.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.771+0000 7f8ceffff640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8cf406b500 0x7f8cf406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:41.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.771+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8d000ac0b0 con 0x7f8d08157150 2024-03-06T07:22:41.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.771+0000 7f8d06ffd640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8cf406b500 0x7f8cf406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:41.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.772+0000 7f8d06ffd640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8cf406b500 0x7f8cf406d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8006fd0 tx=0x7f8cf8009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:41.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:41.775+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d080b5ca0 con 0x7f8d08157150 2024-03-06T07:22:42.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.017+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f8d0804f330 con 0x7f8d08157150 2024-03-06T07:22:42.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.018+0000 7f8ceffff640 1 -- 172.21.15.72:0/1940328955 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8d0004d020 con 0x7f8d08157150 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8cf406b500 msgr2=0x7f8cf406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8cf406b500 0x7f8cf406d9c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8006fd0 tx=0x7f8cf8009060 comp rx=0 tx=0).stop 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 msgr2=0x7f8d0815aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 0x7f8d0815aa50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8d00036040 tx=0x7f8d00004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 shutdown_connections 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8cf406b500 0x7f8cf406d9c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d08157150 0x7f8d0815aa50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d081f49c0 0x7f8d0815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 --2- 172.21.15.72:0/1940328955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0815e010 0x7f8d08156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 >> 172.21.15.72:0/1940328955 conn(0x7f8d0809a9f0 msgr2=0x7f8d080ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 shutdown_connections 2024-03-06T07:22:42.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.021+0000 7f8d0d80e640 1 -- 172.21.15.72:0/1940328955 wait complete. 2024-03-06T07:22:42.033 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-06T07:22:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.132+0000 7fc75c172640 1 Processor -- start 2024-03-06T07:22:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.132+0000 7fc75c172640 1 -- start start 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75c172640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7541444c0 0x7fc75406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75c172640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc754070fe0 0x7fc7540cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75c172640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc75406d370 con 0x7fc75415b2d0 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75c172640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7541448a0 con 0x7fc7541444c0 2024-03-06T07:22:42.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75c172640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc75406d650 con 0x7fc754070fe0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc759f0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7541444c0 0x7fc75406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75970d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc75415b2d0 0x7fc7540708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35410/0 (socket says 172.21.15.72:35410) 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc75970d640 1 -- 172.21.15.72:0/940930154 learned_addr learned my addr 172.21.15.72:0/940930154 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc759f0e640 1 -- 172.21.15.72:0/940930154 >> v2:172.21.15.146:3300/0 conn(0x7fc754070fe0 msgr2=0x7fc7540cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc759f0e640 1 --2- 172.21.15.72:0/940930154 >> v2:172.21.15.146:3300/0 conn(0x7fc754070fe0 0x7fc7540cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc759f0e640 1 -- 172.21.15.72:0/940930154 >> v1:172.21.15.72:6789/0 conn(0x7fc75415b2d0 legacy=0x7fc7540708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc758f0c640 1 -- 172.21.15.72:0/940930154 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908890488 0 0) 0x7fc75406d370 con 0x7fc75415b2d0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.133+0000 7fc759f0e640 1 -- 172.21.15.72:0/940930154 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc75406d930 con 0x7fc7541444c0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.134+0000 7fc759f0e640 1 --2- 172.21.15.72:0/940930154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7541444c0 0x7fc75406ca00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc75415b180 tx=0x7fc75002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa722a33d09ca210 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.134+0000 7fc758f0c640 1 -- 172.21.15.72:0/940930154 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7500315a0 con 0x7fc7541444c0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.134+0000 7fc758f0c640 1 -- 172.21.15.72:0/940930154 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750037070 con 0x7fc7541444c0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.134+0000 7fc758f0c640 1 -- 172.21.15.72:0/940930154 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750031960 con 0x7fc7541444c0 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 -- 172.21.15.72:0/940930154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7541444c0 msgr2=0x7fc75406ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 --2- 172.21.15.72:0/940930154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7541444c0 0x7fc75406ca00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc75415b180 tx=0x7fc75002b840 comp rx=0 tx=0).stop 2024-03-06T07:22:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 -- 172.21.15.72:0/940930154 shutdown_connections 2024-03-06T07:22:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 --2- 172.21.15.72:0/940930154 >> v2:172.21.15.146:3300/0 conn(0x7fc754070fe0 0x7fc7540cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 --2- 172.21.15.72:0/940930154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7541444c0 0x7fc75406ca00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 -- 172.21.15.72:0/940930154 >> 172.21.15.72:0/940930154 conn(0x7fc7540b1c80 msgr2=0x7fc7540cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 -- 172.21.15.72:0/940930154 shutdown_connections 2024-03-06T07:22:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.135+0000 7fc75c172640 1 -- 172.21.15.72:0/940930154 wait complete. 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.136+0000 7fc75c172640 1 Processor -- start 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.136+0000 7fc75c172640 1 -- start start 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc754070fe0 0x7fc7540ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7541444c0 0x7fc7540ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 0x7fc754132e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75406cf40 con 0x7fc75415b2d0 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc75406def0 con 0x7fc754070fe0 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75406d0c0 con 0x7fc7541444c0 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 0x7fc754132e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 0x7fc754132e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60916/0 (socket says 172.21.15.72:60916) 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 -- 172.21.15.72:0/2601933515 learned_addr learned my addr 172.21.15.72:0/2601933515 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc759f0e640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc754070fe0 0x7fc7540ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75970d640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7541444c0 0x7fc7540ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 -- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7541444c0 msgr2=0x7fc7540ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7541444c0 0x7fc7540ab5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 -- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc754070fe0 msgr2=0x7fc7540ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc754070fe0 0x7fc7540ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 -- 172.21.15.72:0/2601933515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754131d90 con 0x7fc75415b2d0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc759f0e640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc754070fe0 0x7fc7540ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75a70f640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 0x7fc754132e30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00b970 tx=0x7fc74c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c019070 con 0x7fc75415b2d0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc74c00cb60 con 0x7fc75415b2d0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc74c014440 con 0x7fc75415b2d0 2024-03-06T07:22:42.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc754132070 con 0x7fc75415b2d0 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.137+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc754132630 con 0x7fc75415b2d0 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.138+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc75404f330 con 0x7fc75415b2d0 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.139+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc74c01c070 con 0x7fc75415b2d0 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.139+0000 7fc74a7fc640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73806b500 0x7fc73806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.139+0000 7fc759f0e640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73806b500 0x7fc73806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.139+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc74c089c10 con 0x7fc75415b2d0 2024-03-06T07:22:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.140+0000 7fc759f0e640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73806b500 0x7fc73806d9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc750036000 tx=0x7fc750008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.143+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc74c058d50 con 0x7fc75415b2d0 2024-03-06T07:22:42.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.379+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fc7540748c0 con 0x7fc75415b2d0 2024-03-06T07:22:42.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.381+0000 7fc74a7fc640 1 -- 172.21.15.72:0/2601933515 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc74c058650 con 0x7fc75415b2d0 2024-03-06T07:22:42.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73806b500 msgr2=0x7fc73806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73806b500 0x7fc73806d9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc750036000 tx=0x7fc750008030 comp rx=0 tx=0).stop 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 msgr2=0x7fc754132e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 0x7fc754132e30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00b970 tx=0x7fc74c00be30 comp rx=0 tx=0).stop 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 shutdown_connections 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc73806b500 0x7fc73806d9c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc75415b2d0 0x7fc754132e30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7541444c0 0x7fc7540ab5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 --2- 172.21.15.72:0/2601933515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc754070fe0 0x7fc7540ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 >> 172.21.15.72:0/2601933515 conn(0x7fc7540b1c80 msgr2=0x7fc7540cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 shutdown_connections 2024-03-06T07:22:42.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.383+0000 7fc75c172640 1 -- 172.21.15.72:0/2601933515 wait complete. 2024-03-06T07:22:42.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-06T07:22:42.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.494+0000 7fdf4e2a4640 1 Processor -- start 2024-03-06T07:22:42.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.494+0000 7fdf4e2a4640 1 -- start start 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf4e2a4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdf480b4a50 0x7fdf4815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf4e2a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4806e770 con 0x7fdf4806e000 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf4e2a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4806e490 con 0x7fdf480cc920 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf4e2a4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4815e660 con 0x7fdf480b4a50 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf477fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdf4806e000 0x7fdf48150b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41022/0 (socket says 172.21.15.72:41022) 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf477fe640 1 -- 172.21.15.72:0/652813198 learned_addr learned my addr 172.21.15.72:0/652813198 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488167491 0 0) 0x7fdf4806e770 con 0x7fdf4806e000 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf30004630 con 0x7fdf4806e000 2024-03-06T07:22:42.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf4c841640 1 --2- 172.21.15.72:0/652813198 >> v2:172.21.15.146:3300/0 conn(0x7fdf480b4a50 0x7fdf4815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512554032 0 0) 0x7fdf30004630 con 0x7fdf4806e000 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 >> v2:172.21.15.146:3300/0 conn(0x7fdf480b4a50 msgr2=0x7fdf4815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 --2- 172.21.15.72:0/652813198 >> v2:172.21.15.146:3300/0 conn(0x7fdf480b4a50 0x7fdf4815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 >> v1:172.21.15.146:6789/0 conn(0x7fdf480cc920 legacy=0x7fdf48072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4815e010 con 0x7fdf4806e000 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fdf3c001670 con 0x7fdf4806e000 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 >> v1:172.21.15.72:6789/0 conn(0x7fdf4806e000 legacy=0x7fdf48150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004ac0 0x7fdf30004f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf30005910 0x7fdf30005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf30006320 0x7fdf3000cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4815e660 con 0x7fdf30005910 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3000d250 con 0x7fdf30004ac0 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.495+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3000d450 con 0x7fdf30006320 2024-03-06T07:22:42.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf477fe640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf30005910 0x7fdf30005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf47fff640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004ac0 0x7fdf30004f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf4c841640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf30006320 0x7fdf3000cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf477fe640 1 -- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf30006320 msgr2=0x7fdf3000cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf477fe640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf30006320 0x7fdf3000cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf477fe640 1 -- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004ac0 msgr2=0x7fdf30004f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf477fe640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004ac0 0x7fdf30004f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf477fe640 1 -- 172.21.15.72:0/652813198 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf30004630 con 0x7fdf30005910 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf4c841640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf30006320 0x7fdf3000cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.496+0000 7fdf4c841640 1 -- 172.21.15.72:0/652813198 reap_dead start 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf477fe640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf30005910 0x7fdf30005d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c210 tx=0x7fdf30006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ebed05b07f0b79 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c007d10 con 0x7fdf30005910 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c030070 con 0x7fdf30005910 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf46ffd640 1 -- 172.21.15.72:0/652813198 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c005d90 con 0x7fdf30005910 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/652813198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf30005910 msgr2=0x7fdf30005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf30005910 0x7fdf30005d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c210 tx=0x7fdf30006940 comp rx=0 tx=0).stop 2024-03-06T07:22:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/652813198 shutdown_connections 2024-03-06T07:22:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/652813198 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf30005910 0x7fdf30005d90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/652813198 >> 172.21.15.72:0/652813198 conn(0x7fdf480b1c80 msgr2=0x7fdf480cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/652813198 shutdown_connections 2024-03-06T07:22:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.497+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/652813198 wait complete. 2024-03-06T07:22:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.498+0000 7fdf4e2a4640 1 Processor -- start 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 -- start start 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30005910 0x7fdf4815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 0x7fdf48146350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf4815d270 0x7fdf4815d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480cce80 con 0x7fdf480aee90 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf480ccd00 con 0x7fdf30005910 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf4e2a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf48070940 con 0x7fdf4815d270 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 0x7fdf48146350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 0x7fdf48146350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33616/0 (socket says 172.21.15.72:33616) 2024-03-06T07:22:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.499+0000 7fdf477fe640 1 -- 172.21.15.72:0/2736598739 learned_addr learned my addr 172.21.15.72:0/2736598739 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf47fff640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30005910 0x7fdf4815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf4c841640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf4815d270 0x7fdf4815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf477fe640 1 -- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf4815d270 msgr2=0x7fdf4815d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf477fe640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf4815d270 0x7fdf4815d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf477fe640 1 -- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30005910 msgr2=0x7fdf4815bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf477fe640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30005910 0x7fdf4815bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf477fe640 1 -- 172.21.15.72:0/2736598739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf480af310 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf477fe640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 0x7fdf48146350 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c001970 tx=0x7fdf4806e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c045070 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c0047e0 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4815c1d0 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c004bb0 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.500+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4815c790 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.501+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4804f330 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.501+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf3c007070 con 0x7fdf480aee90 2024-03-06T07:22:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.502+0000 7fdf44ff9640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf3406b500 0x7fdf3406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.502+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdf3c0aad80 con 0x7fdf480aee90 2024-03-06T07:22:42.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.502+0000 7fdf47fff640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf3406b500 0x7fdf3406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.502+0000 7fdf47fff640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf3406b500 0x7fdf3406d9c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf38000c00 tx=0x7fdf38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.506+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf3c079fe0 con 0x7fdf480aee90 2024-03-06T07:22:42.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.745+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fdf480b40d0 con 0x7fdf480aee90 2024-03-06T07:22:42.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.747+0000 7fdf44ff9640 1 -- 172.21.15.72:0/2736598739 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdf3c0798e0 con 0x7fdf480aee90 2024-03-06T07:22:42.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf3406b500 msgr2=0x7fdf3406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf3406b500 0x7fdf3406d9c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdf38000c00 tx=0x7fdf38009040 comp rx=0 tx=0).stop 2024-03-06T07:22:42.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 msgr2=0x7fdf48146350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 0x7fdf48146350 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c001970 tx=0x7fdf4806e660 comp rx=0 tx=0).stop 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 shutdown_connections 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf3406b500 0x7fdf3406d9c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf4815d270 0x7fdf4815d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf480aee90 0x7fdf48146350 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 --2- 172.21.15.72:0/2736598739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30005910 0x7fdf4815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 >> 172.21.15.72:0/2736598739 conn(0x7fdf480b1c80 msgr2=0x7fdf480d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 shutdown_connections 2024-03-06T07:22:42.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.750+0000 7fdf4e2a4640 1 -- 172.21.15.72:0/2736598739 wait complete. 2024-03-06T07:22:42.762 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-06T07:22:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.860+0000 7fa44aeb5640 1 Processor -- start 2024-03-06T07:22:42.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.860+0000 7fa44aeb5640 1 -- start start 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.861+0000 7fa44aeb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa4441658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.861+0000 7fa44aeb5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44406d370 con 0x7fa44415b2d0 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.861+0000 7fa44aeb5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44406d650 con 0x7fa4441444c0 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.861+0000 7fa44aeb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4441448a0 con 0x7fa444070fe0 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.861+0000 7fa443fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa44415b2d0 0x7fa4440708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41030/0 (socket says 172.21.15.72:41030) 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.861+0000 7fa443fff640 1 -- 172.21.15.72:0/233376873 learned_addr learned my addr 172.21.15.72:0/233376873 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa4441658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871131887 0 0) 0x7fa44406d370 con 0x7fa44415b2d0 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa434003520 con 0x7fa44415b2d0 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434411392 0 0) 0x7fa434003520 con 0x7fa44415b2d0 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 msgr2=0x7fa4441658d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa4441658d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 >> v1:172.21.15.146:6789/0 conn(0x7fa4441444c0 legacy=0x7fa44406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44406d930 con 0x7fa44415b2d0 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fa430001670 con 0x7fa44415b2d0 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 >> v1:172.21.15.72:6789/0 conn(0x7fa44415b2d0 legacy=0x7fa4440708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4340039e0 0x7fa434003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa434004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434004d40 0x7fa4340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4441448a0 con 0x7fa434004d40 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43400ffd0 con 0x7fa4340039e0 2024-03-06T07:22:42.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa434010150 con 0x7fa444070fe0 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434004d40 0x7fa4340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa448c51640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4340039e0 0x7fa434003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa443fff640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa434004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 -- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 msgr2=0x7fa434004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa434004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 -- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4340039e0 msgr2=0x7fa434003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4340039e0 0x7fa434003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa449452640 1 -- 172.21.15.72:0/233376873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434003520 con 0x7fa434004d40 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.862+0000 7fa443fff640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa444070fe0 0x7fa434004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.863+0000 7fa449452640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434004d40 0x7fa4340051c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009fc0 tx=0x7fa434005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b83fa44dd38fb01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.863+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c039070 con 0x7fa434004d40 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.863+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43c02cc20 con 0x7fa434004d40 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.863+0000 7fa4437fe640 1 -- 172.21.15.72:0/233376873 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c009c10 con 0x7fa434004d40 2024-03-06T07:22:42.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa44aeb5640 1 -- 172.21.15.72:0/233376873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434004d40 msgr2=0x7fa4340051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/233376873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa434004d40 0x7fa4340051c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009fc0 tx=0x7fa434005830 comp rx=0 tx=0).stop 2024-03-06T07:22:42.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa449452640 1 -- 172.21.15.72:0/233376873 reap_dead start 2024-03-06T07:22:42.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa44aeb5640 1 -- 172.21.15.72:0/233376873 shutdown_connections 2024-03-06T07:22:42.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa44aeb5640 1 -- 172.21.15.72:0/233376873 >> 172.21.15.72:0/233376873 conn(0x7fa4440b1c80 msgr2=0x7fa4440cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:42.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa44aeb5640 1 -- 172.21.15.72:0/233376873 shutdown_connections 2024-03-06T07:22:42.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.864+0000 7fa44aeb5640 1 -- 172.21.15.72:0/233376873 wait complete. 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.865+0000 7fa44aeb5640 1 Processor -- start 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.865+0000 7fa44aeb5640 1 -- start start 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa44aeb5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4440b4710 0x7fa44413ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa44aeb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 0x7fa44413f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa44aeb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4440a9450 0x7fa4440a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa44aeb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44406def0 con 0x7fa4440b4af0 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa44aeb5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44406d110 con 0x7fa4440b4710 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa44aeb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4440cea70 con 0x7fa4440a9450 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 0x7fa44413f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 0x7fa44413f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33636/0 (socket says 172.21.15.72:33636) 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 -- 172.21.15.72:0/3666822405 learned_addr learned my addr 172.21.15.72:0/3666822405 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa448c51640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4440b4710 0x7fa44413ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa449452640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4440a9450 0x7fa4440a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 -- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4440a9450 msgr2=0x7fa4440a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4440a9450 0x7fa4440a9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 -- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4440b4710 msgr2=0x7fa44413ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4440b4710 0x7fa44413ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 -- 172.21.15.72:0/3666822405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4440aa180 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa443fff640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 0x7fa44413f380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa43000eb20 tx=0x7fa430001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.866+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000c910 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.867+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa430018070 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.867+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43000ccb0 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.867+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4441f32f0 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.867+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4441f38b0 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.868+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4441f3ab0 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.868+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa43001d460 con 0x7fa4440b4af0 2024-03-06T07:22:42.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.868+0000 7fa4417fa640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa42806b4d0 0x7fa42806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:42.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.868+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa430089c50 con 0x7fa4440b4af0 2024-03-06T07:22:42.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.868+0000 7fa448c51640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa42806b4d0 0x7fa42806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:42.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.869+0000 7fa448c51640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa42806b4d0 0x7fa42806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa438007fd0 tx=0x7fa438009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:42.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:42.873+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4441f3ab0 con 0x7fa4440b4af0 2024-03-06T07:22:43.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.108+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fa44404f330 con 0x7fa4440b4af0 2024-03-06T07:22:43.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.109+0000 7fa4417fa640 1 -- 172.21.15.72:0/3666822405 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fa43008f020 con 0x7fa4440b4af0 2024-03-06T07:22:43.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa42806b4d0 msgr2=0x7fa42806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa42806b4d0 0x7fa42806d990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa438007fd0 tx=0x7fa438009040 comp rx=0 tx=0).stop 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 msgr2=0x7fa44413f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 0x7fa44413f380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa43000eb20 tx=0x7fa430001720 comp rx=0 tx=0).stop 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 shutdown_connections 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa42806b4d0 0x7fa42806d990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4440a9450 0x7fa4440a9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4440b4af0 0x7fa44413f380 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 --2- 172.21.15.72:0/3666822405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4440b4710 0x7fa44413ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 >> 172.21.15.72:0/3666822405 conn(0x7fa4440b1c80 msgr2=0x7fa4440708d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 shutdown_connections 2024-03-06T07:22:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.111+0000 7fa44aeb5640 1 -- 172.21.15.72:0/3666822405 wait complete. 2024-03-06T07:22:43.124 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:43.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.222+0000 7f07c00e0640 1 Processor -- start 2024-03-06T07:22:43.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.222+0000 7f07c00e0640 1 -- start start 2024-03-06T07:22:43.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07c00e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07b813dc10 0x7f07b80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07c00e0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b815f580 con 0x7f07b815bf30 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07c00e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b815b450 con 0x7f07b815ef60 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07c00e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b80cf3f0 con 0x7f07b813dc10 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07bd67b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f07b815bf30 0x7f07b815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41034/0 (socket says 172.21.15.72:41034) 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07bd67b640 1 -- 172.21.15.72:0/2171423981 learned_addr learned my addr 172.21.15.72:0/2171423981 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07bde7c640 1 --2- 172.21.15.72:0/2171423981 >> v2:172.21.15.146:3300/0 conn(0x7f07b813dc10 0x7f07b80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578444285 0 0) 0x7f07b815f580 con 0x7f07b815bf30 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.223+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a4003520 con 0x7f07b815bf30 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1540592978 0 0) 0x7f07a4003520 con 0x7f07b815bf30 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 >> v2:172.21.15.146:3300/0 conn(0x7f07b813dc10 msgr2=0x7f07b80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 --2- 172.21.15.72:0/2171423981 >> v2:172.21.15.146:3300/0 conn(0x7f07b813dc10 0x7f07b80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 >> v1:172.21.15.146:6789/0 conn(0x7f07b815ef60 legacy=0x7f07b80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b815b730 con 0x7f07b815bf30 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f07a8001670 con 0x7f07b815bf30 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 >> v1:172.21.15.72:6789/0 conn(0x7f07b815bf30 legacy=0x7f07b815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40039b0 0x7f07a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07a4004800 0x7f07a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40051c0 0x7f07a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b80cf3f0 con 0x7f07a40039b0 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a400c160 con 0x7f07a40051c0 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a400c360 con 0x7f07a4004800 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bde7c640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40039b0 0x7f07a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07be67d640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40051c0 0x7f07a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bd67b640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07a4004800 0x7f07a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bde7c640 1 -- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07a4004800 msgr2=0x7f07a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bde7c640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07a4004800 0x7f07a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bde7c640 1 -- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40051c0 msgr2=0x7f07a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bde7c640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40051c0 0x7f07a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bde7c640 1 -- 172.21.15.72:0/2171423981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4003520 con 0x7f07a40039b0 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07be67d640 1 -- 172.21.15.72:0/2171423981 reap_dead start 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.224+0000 7f07bd67b640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07a4004800 0x7f07a4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07bde7c640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40039b0 0x7f07a4003dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009fc0 tx=0x7f07a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b6dbd6775b0cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4039070 con 0x7f07a40039b0 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b402cc20 con 0x7f07a40039b0 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07bce7a640 1 -- 172.21.15.72:0/2171423981 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4009c10 con 0x7f07a40039b0 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07c00e0640 1 -- 172.21.15.72:0/2171423981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40039b0 msgr2=0x7f07a4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07c00e0640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40039b0 0x7f07a4003dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f07b4009fc0 tx=0x7f07a4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07c00e0640 1 -- 172.21.15.72:0/2171423981 shutdown_connections 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07c00e0640 1 --2- 172.21.15.72:0/2171423981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07a40039b0 0x7f07a4003dd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07c00e0640 1 -- 172.21.15.72:0/2171423981 >> 172.21.15.72:0/2171423981 conn(0x7f07b80cf8a0 msgr2=0x7f07b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.225+0000 7f07c00e0640 1 -- 172.21.15.72:0/2171423981 shutdown_connections 2024-03-06T07:22:43.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.226+0000 7f07c00e0640 1 -- 172.21.15.72:0/2171423981 wait complete. 2024-03-06T07:22:43.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 Processor -- start 2024-03-06T07:22:43.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 -- start start 2024-03-06T07:22:43.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40039b0 0x7f07b81f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 0x7f07b8159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07b815a270 0x7f07b8132960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b815e850 con 0x7f07b8154b30 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b815b210 con 0x7f07a40039b0 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07c00e0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b815e9d0 con 0x7f07b815a270 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07bd67b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 0x7f07b8159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07bd67b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 0x7f07b8159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33648/0 (socket says 172.21.15.72:33648) 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.227+0000 7f07bd67b640 1 -- 172.21.15.72:0/3312214690 learned_addr learned my addr 172.21.15.72:0/3312214690 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07be67d640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07b815a270 0x7f07b8132960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bde7c640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40039b0 0x7f07b81f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bd67b640 1 -- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07b815a270 msgr2=0x7f07b8132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bd67b640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07b815a270 0x7f07b8132960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bd67b640 1 -- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40039b0 msgr2=0x7f07b81f8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bd67b640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40039b0 0x7f07b81f8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bd67b640 1 -- 172.21.15.72:0/3312214690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b81f8840 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07bd67b640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 0x7f07b8159d30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d770 tx=0x7f07a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8019070 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a80074e0 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8004a00 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b813fd70 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.228+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b8140330 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.229+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b804f330 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.229+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07a801c070 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.230+0000 7f07ae7fc640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f079c06b4d0 0x7f079c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.230+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f07a80897e0 con 0x7f07b8154b30 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.230+0000 7f07bde7c640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f079c06b4d0 0x7f079c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.230+0000 7f07bde7c640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f079c06b4d0 0x7f079c06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f07b4036000 tx=0x7f07b402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.234+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07b81f8840 con 0x7f07b8154b30 2024-03-06T07:22:43.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.409+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07b806eb00 con 0x7f07b8154b30 2024-03-06T07:22:43.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.410+0000 7f07ae7fc640 1 -- 172.21.15.72:0/3312214690 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12903 (secure 0 0 0) 0x7f07a800de20 con 0x7f07b8154b30 2024-03-06T07:22:43.412 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:43.412 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":30,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:40.561258+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f079c06b4d0 msgr2=0x7f079c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f079c06b4d0 0x7f079c06d990 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f07b4036000 tx=0x7f07b402e000 comp rx=0 tx=0).stop 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 msgr2=0x7f07b8159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 0x7f07b8159d30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d770 tx=0x7f07a800dc30 comp rx=0 tx=0).stop 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 shutdown_connections 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f079c06b4d0 0x7f079c06d990 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07b815a270 0x7f07b8132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07b8154b30 0x7f07b8159d30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 --2- 172.21.15.72:0/3312214690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07a40039b0 0x7f07b81f8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 >> 172.21.15.72:0/3312214690 conn(0x7f07b80cf8a0 msgr2=0x7f07b80cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 shutdown_connections 2024-03-06T07:22:43.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.412+0000 7f07c00e0640 1 -- 172.21.15.72:0/3312214690 wait complete. 2024-03-06T07:22:43.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:22:43.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:22:43.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.525+0000 7f24df49a640 1 Processor -- start 2024-03-06T07:22:43.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.525+0000 7f24df49a640 1 -- start start 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24df49a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24d815bb40 0x7f24d80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24df49a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f24d815f7d0 0x7f24d806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24df49a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806d5c0 0x7f24d80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24df49a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d806c910 con 0x7f24d815f7d0 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24df49a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d815ff90 con 0x7f24d815bb40 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24df49a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d8160270 con 0x7f24d806d5c0 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f24d815f7d0 0x7f24d806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f24d815f7d0 0x7f24d806d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33654/0 (socket says 172.21.15.72:33654) 2024-03-06T07:22:43.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 -- 172.21.15.72:0/2201561977 learned_addr learned my addr 172.21.15.72:0/2201561977 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dd236640 1 --2- 172.21.15.72:0/2201561977 >> v2:172.21.15.146:3300/0 conn(0x7f24d815bb40 0x7f24d80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dda37640 1 --2- 172.21.15.72:0/2201561977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806d5c0 0x7f24d80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 -- 172.21.15.72:0/2201561977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806d5c0 msgr2=0x7f24d80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 --2- 172.21.15.72:0/2201561977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806d5c0 0x7f24d80cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 -- 172.21.15.72:0/2201561977 >> v2:172.21.15.146:3300/0 conn(0x7f24d815bb40 msgr2=0x7f24d80b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 --2- 172.21.15.72:0/2201561977 >> v2:172.21.15.146:3300/0 conn(0x7f24d815bb40 0x7f24d80b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 -- 172.21.15.72:0/2201561977 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d8160550 con 0x7f24d815f7d0 2024-03-06T07:22:43.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24dca35640 1 --2- 172.21.15.72:0/2201561977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d806d080 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24d80b4f60 tx=0x7f24c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e869eee263e0b582 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24cffff640 1 -- 172.21.15.72:0/2201561977 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8035020 con 0x7f24d815f7d0 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24cffff640 1 -- 172.21.15.72:0/2201561977 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c8031870 con 0x7f24d815f7d0 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.527+0000 7f24cffff640 1 -- 172.21.15.72:0/2201561977 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c8031c40 con 0x7f24d815f7d0 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 -- 172.21.15.72:0/2201561977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 msgr2=0x7f24d806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 --2- 172.21.15.72:0/2201561977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d806d080 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f24d80b4f60 tx=0x7f24c802b910 comp rx=0 tx=0).stop 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 -- 172.21.15.72:0/2201561977 shutdown_connections 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 --2- 172.21.15.72:0/2201561977 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d806d5c0 0x7f24d80cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 --2- 172.21.15.72:0/2201561977 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d806d080 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 --2- 172.21.15.72:0/2201561977 >> v2:172.21.15.146:3300/0 conn(0x7f24d815bb40 0x7f24d80b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 -- 172.21.15.72:0/2201561977 >> 172.21.15.72:0/2201561977 conn(0x7f24d806b190 msgr2=0x7f24d806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 -- 172.21.15.72:0/2201561977 shutdown_connections 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.528+0000 7f24df49a640 1 -- 172.21.15.72:0/2201561977 wait complete. 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.529+0000 7f24df49a640 1 Processor -- start 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.529+0000 7f24df49a640 1 -- start start 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24df49a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d806d5c0 0x7f24d81f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24df49a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d815bb40 0x7f24d8159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24df49a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d8159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24df49a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d815bf20 con 0x7f24d815f7d0 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24df49a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d815fc30 con 0x7f24d806d5c0 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24df49a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d815c0a0 con 0x7f24d815bb40 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d8159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d8159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33658/0 (socket says 172.21.15.72:33658) 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 -- 172.21.15.72:0/3482672388 learned_addr learned my addr 172.21.15.72:0/3482672388 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dd236640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d806d5c0 0x7f24d81f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dca35640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d815bb40 0x7f24d8159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 -- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d815bb40 msgr2=0x7f24d8159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d815bb40 0x7f24d8159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 -- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d806d5c0 msgr2=0x7f24d81f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d806d5c0 0x7f24d81f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 -- 172.21.15.72:0/3482672388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d81f1b10 con 0x7f24d815f7d0 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.530+0000 7f24dda37640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d8159ed0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f24d000d5d0 tx=0x7f24d000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.531+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d00125f0 con 0x7f24d815f7d0 2024-03-06T07:22:43.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.531+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d0019070 con 0x7f24d815f7d0 2024-03-06T07:22:43.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.531+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d81f1df0 con 0x7f24d815f7d0 2024-03-06T07:22:43.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.531+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d00129f0 con 0x7f24d815f7d0 2024-03-06T07:22:43.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.531+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d81f23b0 con 0x7f24d815f7d0 2024-03-06T07:22:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.532+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d80732b0 con 0x7f24d815f7d0 2024-03-06T07:22:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.532+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f24d0012b90 con 0x7f24d815f7d0 2024-03-06T07:22:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.533+0000 7f24cdffb640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f24bc06b4d0 0x7f24bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.533+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f24d008a270 con 0x7f24d815f7d0 2024-03-06T07:22:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.533+0000 7f24dd236640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f24bc06b4d0 0x7f24bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.533+0000 7f24dd236640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f24bc06b4d0 0x7f24bc06d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f24d40099e0 tx=0x7f24d4003cd0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.537+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f24d0059450 con 0x7f24d815f7d0 2024-03-06T07:22:43.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.709+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24d8068ff0 con 0x7f24d815f7d0 2024-03-06T07:22:43.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.710+0000 7f24cdffb640 1 -- 172.21.15.72:0/3482672388 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24d0058d50 con 0x7f24d815f7d0 2024-03-06T07:22:43.712 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f24bc06b4d0 msgr2=0x7f24bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f24bc06b4d0 0x7f24bc06d990 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f24d40099e0 tx=0x7f24d4003cd0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 msgr2=0x7f24d8159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d8159ed0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f24d000d5d0 tx=0x7f24d000da90 comp rx=0 tx=0).stop 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 shutdown_connections 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f24bc06b4d0 0x7f24bc06d990 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24d815f7d0 0x7f24d8159ed0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24d815bb40 0x7f24d8159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 --2- 172.21.15.72:0/3482672388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24d806d5c0 0x7f24d81f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.711+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 >> 172.21.15.72:0/3482672388 conn(0x7f24d806b190 msgr2=0x7f24d80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.712+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 shutdown_connections 2024-03-06T07:22:43.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.712+0000 7f24df49a640 1 -- 172.21.15.72:0/3482672388 wait complete. 2024-03-06T07:22:43.724 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:22:43.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.822+0000 7f6d23678640 1 Processor -- start 2024-03-06T07:22:43.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.822+0000 7f6d23678640 1 -- start start 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d23678640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d1c0d0320 0x7f6d1c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d23678640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 0x7f6d1c0b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d23678640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1c15c270 con 0x7f6d1c0ccae0 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d23678640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1c0b04e0 con 0x7f6d1c0d0320 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d23678640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0d0e90 con 0x7f6d1c0d0700 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d21414640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d1c0d0320 0x7f6d1c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d20c13640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 0x7f6d1c0b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d20c13640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 0x7f6d1c0b54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:57664/0 (socket says 172.21.15.72:57664) 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d20c13640 1 -- 172.21.15.72:0/809428885 learned_addr learned my addr 172.21.15.72:0/809428885 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.823+0000 7f6d21c15640 1 --1- 172.21.15.72:0/809428885 >> v1:172.21.15.146:6789/0 conn(0x7f6d1c0ccae0 0x7f6d1c0ccf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:59588/0 (socket says 172.21.15.72:59588) 2024-03-06T07:22:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.824+0000 7f6d20c13640 1 -- 172.21.15.72:0/809428885 >> v2:172.21.15.146:3300/0 conn(0x7f6d1c0d0320 msgr2=0x7f6d1c0b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.824+0000 7f6d20c13640 1 --2- 172.21.15.72:0/809428885 >> v2:172.21.15.146:3300/0 conn(0x7f6d1c0d0320 0x7f6d1c0b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.824+0000 7f6d20c13640 1 -- 172.21.15.72:0/809428885 >> v1:172.21.15.146:6789/0 conn(0x7f6d1c0ccae0 legacy=0x7f6d1c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.824+0000 7f6d20c13640 1 -- 172.21.15.72:0/809428885 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c0b5ae0 con 0x7f6d1c0d0700 2024-03-06T07:22:43.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.824+0000 7f6d21414640 1 --2- 172.21.15.72:0/809428885 >> v2:172.21.15.146:3300/0 conn(0x7f6d1c0d0320 0x7f6d1c0b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d20c13640 1 --2- 172.21.15.72:0/809428885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 0x7f6d1c0b54b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009870 tx=0x7f6d0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4914ea9d541b79f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d13fff640 1 -- 172.21.15.72:0/809428885 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c034070 con 0x7f6d1c0d0700 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d13fff640 1 -- 172.21.15.72:0/809428885 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c030840 con 0x7f6d1c0d0700 2024-03-06T07:22:43.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d13fff640 1 -- 172.21.15.72:0/809428885 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c030c10 con 0x7f6d1c0d0700 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 -- 172.21.15.72:0/809428885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 msgr2=0x7f6d1c0b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 --2- 172.21.15.72:0/809428885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 0x7f6d1c0b54b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009870 tx=0x7f6d0c02b980 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 -- 172.21.15.72:0/809428885 shutdown_connections 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 --2- 172.21.15.72:0/809428885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0700 0x7f6d1c0b54b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 --2- 172.21.15.72:0/809428885 >> v2:172.21.15.146:3300/0 conn(0x7f6d1c0d0320 0x7f6d1c0b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 -- 172.21.15.72:0/809428885 >> 172.21.15.72:0/809428885 conn(0x7f6d1c0724d0 msgr2=0x7f6d1c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 -- 172.21.15.72:0/809428885 shutdown_connections 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.825+0000 7f6d23678640 1 -- 172.21.15.72:0/809428885 wait complete. 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.826+0000 7f6d23678640 1 Processor -- start 2024-03-06T07:22:43.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 -- start start 2024-03-06T07:22:43.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d1c0ccae0 0x7f6d1c144b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0320 0x7f6d1c143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 0x7f6d1c1459a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0b60a0 con 0x7f6d1c0d0700 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0b0170 con 0x7f6d1c0ccae0 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d23678640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1c0ca790 con 0x7f6d1c0d0320 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d21c15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 0x7f6d1c1459a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d21c15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 0x7f6d1c1459a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33668/0 (socket says 172.21.15.72:33668) 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.827+0000 7f6d21c15640 1 -- 172.21.15.72:0/2432525500 learned_addr learned my addr 172.21.15.72:0/2432525500 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d20c13640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0320 0x7f6d1c143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21414640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d1c0ccae0 0x7f6d1c144b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21c15640 1 -- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0320 msgr2=0x7f6d1c143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21c15640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0320 0x7f6d1c143290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21c15640 1 -- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d1c0ccae0 msgr2=0x7f6d1c144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:43.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21c15640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d1c0ccae0 0x7f6d1c144b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21c15640 1 -- 172.21.15.72:0/2432525500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1c145060 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21414640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d1c0ccae0 0x7f6d1c144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d20c13640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0320 0x7f6d1c143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d21c15640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 0x7f6d1c1459a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c780 tx=0x7f6d1400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14017070 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1c1437d0 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1400a5e0 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1400a9d0 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.828+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1c143d60 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.829+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1c04f330 con 0x7f6d1c0d0700 2024-03-06T07:22:43.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.829+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d1401a070 con 0x7f6d1c0d0700 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.830+0000 7f6d11ffb640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d0006b4d0 0x7f6d0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.830+0000 7f6d21414640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d0006b4d0 0x7f6d0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.830+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6d14089430 con 0x7f6d1c0d0700 2024-03-06T07:22:43.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.830+0000 7f6d21414640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d0006b4d0 0x7f6d0006d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009ac0 tx=0x7f6d18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:43.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:43.834+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d14057f30 con 0x7f6d1c0d0700 2024-03-06T07:22:44.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.007+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6d1c0af330 con 0x7f6d1c0d0700 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.008+0000 7f6d11ffb640 1 -- 172.21.15.72:0/2432525500 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6d1405bfd0 con 0x7f6d1c0d0700 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:44.010 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.009+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d0006b4d0 msgr2=0x7f6d0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d0006b4d0 0x7f6d0006d990 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009ac0 tx=0x7f6d18008040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 msgr2=0x7f6d1c1459a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 0x7f6d1c1459a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6d1400c780 tx=0x7f6d1400cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 shutdown_connections 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d0006b4d0 0x7f6d0006d990 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d1c0d0700 0x7f6d1c1459a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d1c0d0320 0x7f6d1c143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 --2- 172.21.15.72:0/2432525500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d1c0ccae0 0x7f6d1c144b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 >> 172.21.15.72:0/2432525500 conn(0x7f6d1c0724d0 msgr2=0x7f6d1c0cabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 shutdown_connections 2024-03-06T07:22:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.010+0000 7f6d23678640 1 -- 172.21.15.72:0/2432525500 wait complete. 2024-03-06T07:22:44.022 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:22:44.023 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:22:44.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.125+0000 7f191fd8a640 1 Processor -- start 2024-03-06T07:22:44.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.125+0000 7f191fd8a640 1 -- start start 2024-03-06T07:22:44.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191fd8a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19180708b0 0x7f1918070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191fd8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19180ce0a0 0x7f19180ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191fd8a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191815afe0 con 0x7f191815ac00 2024-03-06T07:22:44.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191fd8a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1918071250 con 0x7f19180ce0a0 2024-03-06T07:22:44.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191fd8a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19180cdc10 con 0x7f19180708b0 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191e327640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19180ce0a0 0x7f19180ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191db26640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f191815ac00 0x7f191806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41042/0 (socket says 172.21.15.72:41042) 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191db26640 1 -- 172.21.15.72:0/2541398048 learned_addr learned my addr 172.21.15.72:0/2541398048 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191d325640 1 --2- 172.21.15.72:0/2541398048 >> v2:172.21.15.146:3300/0 conn(0x7f19180708b0 0x7f1918070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191e327640 1 -- 172.21.15.72:0/2541398048 >> v2:172.21.15.146:3300/0 conn(0x7f19180708b0 msgr2=0x7f1918070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191e327640 1 --2- 172.21.15.72:0/2541398048 >> v2:172.21.15.146:3300/0 conn(0x7f19180708b0 0x7f1918070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191e327640 1 -- 172.21.15.72:0/2541398048 >> v1:172.21.15.72:6789/0 conn(0x7f191815ac00 legacy=0x7f191806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191cb24640 1 -- 172.21.15.72:0/2541398048 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512096457 0 0) 0x7f191815afe0 con 0x7f191815ac00 2024-03-06T07:22:44.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.126+0000 7f191e327640 1 -- 172.21.15.72:0/2541398048 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19180cd000 con 0x7f19180ce0a0 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191e327640 1 --2- 172.21.15.72:0/2541398048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180ce0a0 0x7f19180ce550 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1918074f80 tx=0x7f191002e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fee29837a49e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191cb24640 1 -- 172.21.15.72:0/2541398048 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191002e9d0 con 0x7f19180ce0a0 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191cb24640 1 -- 172.21.15.72:0/2541398048 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191002eb70 con 0x7f19180ce0a0 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191cb24640 1 -- 172.21.15.72:0/2541398048 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910009540 con 0x7f19180ce0a0 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191fd8a640 1 -- 172.21.15.72:0/2541398048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180ce0a0 msgr2=0x7f19180ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2541398048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180ce0a0 0x7f19180ce550 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1918074f80 tx=0x7f191002e030 comp rx=0 tx=0).stop 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191fd8a640 1 -- 172.21.15.72:0/2541398048 shutdown_connections 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2541398048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180ce0a0 0x7f19180ce550 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2541398048 >> v2:172.21.15.146:3300/0 conn(0x7f19180708b0 0x7f1918070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.127+0000 7f191fd8a640 1 -- 172.21.15.72:0/2541398048 >> 172.21.15.72:0/2541398048 conn(0x7f19180b1650 msgr2=0x7f19180b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.128+0000 7f191fd8a640 1 -- 172.21.15.72:0/2541398048 shutdown_connections 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.128+0000 7f191fd8a640 1 -- 172.21.15.72:0/2541398048 wait complete. 2024-03-06T07:22:44.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 Processor -- start 2024-03-06T07:22:44.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 -- start start 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 0x7f19181eec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19180ce0a0 0x7f1918157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f191815ac00 0x7f19180af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19180cd8a0 con 0x7f19180708b0 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19180cd5c0 con 0x7f19180ce0a0 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191fd8a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19180cf5d0 con 0x7f191815ac00 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191db26640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 0x7f19181eec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191db26640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 0x7f19181eec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33686/0 (socket says 172.21.15.72:33686) 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.129+0000 7f191db26640 1 -- 172.21.15.72:0/2901758474 learned_addr learned my addr 172.21.15.72:0/2901758474 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:44.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191d325640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19180ce0a0 0x7f1918157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191e327640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f191815ac00 0x7f19180af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191db26640 1 -- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f191815ac00 msgr2=0x7f19180af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191db26640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f191815ac00 0x7f19180af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191db26640 1 -- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19180ce0a0 msgr2=0x7f1918157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191db26640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19180ce0a0 0x7f1918157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191db26640 1 -- 172.21.15.72:0/2901758474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19181579e0 con 0x7f19180708b0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191db26640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 0x7f19181eec20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f191400c430 tx=0x7f191400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19140074e0 con 0x7f19180708b0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19140047e0 con 0x7f19180708b0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1918141ef0 con 0x7f19180708b0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1914004bd0 con 0x7f19180708b0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.130+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19181424b0 con 0x7f19180708b0 2024-03-06T07:22:44.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.131+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191804f330 con 0x7f19180708b0 2024-03-06T07:22:44.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.132+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1914011070 con 0x7f19180708b0 2024-03-06T07:22:44.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.132+0000 7f190e7fc640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18fc06b4d0 0x7f18fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.132+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19140899b0 con 0x7f19180708b0 2024-03-06T07:22:44.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.132+0000 7f191d325640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18fc06b4d0 0x7f18fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.132+0000 7f191d325640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18fc06b4d0 0x7f18fc06d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f19180b3e50 tx=0x7f1908008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.136+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1914058530 con 0x7f19180708b0 2024-03-06T07:22:44.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.305+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1918154ab0 con 0x7f19180708b0 2024-03-06T07:22:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.659+0000 7f190e7fc640 1 -- 172.21.15.72:0/2901758474 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f191405c5d0 con 0x7f19180708b0 2024-03-06T07:22:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:22:44.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18fc06b4d0 msgr2=0x7f18fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18fc06b4d0 0x7f18fc06d990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f19180b3e50 tx=0x7f1908008040 comp rx=0 tx=0).stop 2024-03-06T07:22:44.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 msgr2=0x7f19181eec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 0x7f19181eec20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f191400c430 tx=0x7f191400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 shutdown_connections 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18fc06b4d0 0x7f18fc06d990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f191815ac00 0x7f19180af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19180ce0a0 0x7f1918157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 --2- 172.21.15.72:0/2901758474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19180708b0 0x7f19181eec20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 >> 172.21.15.72:0/2901758474 conn(0x7f19180b1650 msgr2=0x7f1918153480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.662+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 shutdown_connections 2024-03-06T07:22:44.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.663+0000 7f191fd8a640 1 -- 172.21.15.72:0/2901758474 wait complete. 2024-03-06T07:22:44.676 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:22:44.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.781+0000 7f0c0844e640 1 Processor -- start 2024-03-06T07:22:44.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.781+0000 7f0c0844e640 1 -- start start 2024-03-06T07:22:44.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.782+0000 7f0c0844e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c001586e0 0x7f0c000aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.782+0000 7f0c0844e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c000d01a0 0x7f0c000b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.782+0000 7f0c0844e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c00158ac0 con 0x7f0c000aef80 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.782+0000 7f0c0844e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0015b2a0 con 0x7f0c000d01a0 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.782+0000 7f0c0844e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0015b540 con 0x7f0c001586e0 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c069eb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c000d01a0 0x7f0c000b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c059e9640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0c000aef80 0x7f0c000cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:59604/0 (socket says 172.21.15.72:59604) 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c061ea640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c001586e0 0x7f0c000aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c059e9640 1 -- 172.21.15.72:0/2474749957 learned_addr learned my addr 172.21.15.72:0/2474749957 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c069eb640 1 -- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c001586e0 msgr2=0x7f0c000aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c069eb640 1 --2- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c001586e0 0x7f0c000aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c069eb640 1 -- 172.21.15.72:0/2474749957 >> v1:172.21.15.146:6789/0 conn(0x7f0c000aef80 legacy=0x7f0c000cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c069eb640 1 -- 172.21.15.72:0/2474749957 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0015b820 con 0x7f0c000d01a0 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c051e8640 1 -- 172.21.15.72:0/2474749957 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968252712 0 0) 0x7f0c00158ac0 con 0x7f0c000aef80 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.783+0000 7f0c061ea640 1 --2- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c001586e0 0x7f0c000aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.784+0000 7f0c069eb640 1 --2- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000d01a0 0x7f0c000b33a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009d90 tx=0x7f0bf802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8c4650a287d0360 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.784+0000 7f0c051e8640 1 -- 172.21.15.72:0/2474749957 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8039070 con 0x7f0c000d01a0 2024-03-06T07:22:44.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.784+0000 7f0c051e8640 1 -- 172.21.15.72:0/2474749957 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf802cc50 con 0x7f0c000d01a0 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.784+0000 7f0c051e8640 1 -- 172.21.15.72:0/2474749957 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8034440 con 0x7f0c000d01a0 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 -- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000d01a0 msgr2=0x7f0c000b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 --2- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000d01a0 0x7f0c000b33a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009d90 tx=0x7f0bf802bea0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 -- 172.21.15.72:0/2474749957 shutdown_connections 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 --2- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000d01a0 0x7f0c000b33a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 --2- 172.21.15.72:0/2474749957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c001586e0 0x7f0c000aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 -- 172.21.15.72:0/2474749957 >> 172.21.15.72:0/2474749957 conn(0x7f0c0006b190 msgr2=0x7f0c0006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 -- 172.21.15.72:0/2474749957 shutdown_connections 2024-03-06T07:22:44.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.785+0000 7f0c0844e640 1 -- 172.21.15.72:0/2474749957 wait complete. 2024-03-06T07:22:44.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.786+0000 7f0c0844e640 1 Processor -- start 2024-03-06T07:22:44.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.786+0000 7f0c0844e640 1 -- start start 2024-03-06T07:22:44.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000aef80 0x7f0c00147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c000d01a0 0x7f0c000b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 0x7f0c000ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00071350 con 0x7f0c001586e0 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0015bde0 con 0x7f0c000aef80 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000714d0 con 0x7f0c000d01a0 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 0x7f0c000ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 0x7f0c000ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33702/0 (socket says 172.21.15.72:33702) 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 -- 172.21.15.72:0/3267277129 learned_addr learned my addr 172.21.15.72:0/3267277129 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c059e9640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c000d01a0 0x7f0c000b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c061ea640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000aef80 0x7f0c00147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 -- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c000d01a0 msgr2=0x7f0c000b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c000d01a0 0x7f0c000b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 -- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000aef80 msgr2=0x7f0c00147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000aef80 0x7f0c00147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 -- 172.21.15.72:0/3267277129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000cdf40 con 0x7f0c001586e0 2024-03-06T07:22:44.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c061ea640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000aef80 0x7f0c00147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c069eb640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 0x7f0c000ad860 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009d90 tx=0x7f0bf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8044020 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf803a040 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000ce220 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8039050 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.787+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000ce7e0 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.788+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0014fc10 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.789+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bf8003990 con 0x7f0c001586e0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.789+0000 7f0bf6ffd640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.789+0000 7f0c061ea640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0be406b4d0 0x7f0be406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:44.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.789+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f0bf80ac320 con 0x7f0c001586e0 2024-03-06T07:22:44.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.790+0000 7f0c061ea640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009af0 tx=0x7f0bfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:44.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.794+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bf804e060 con 0x7f0c001586e0 2024-03-06T07:22:44.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:44.965+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0c0006e930 con 0x7f0c001586e0 2024-03-06T07:22:45.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.665+0000 7f0bf6ffd640 1 -- 172.21.15.72:0/3267277129 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0bf807b410 con 0x7f0c001586e0 2024-03-06T07:22:45.666 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:22:45.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.669+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0be406b4d0 msgr2=0x7f0be406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.669+0000 7f0c0844e640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009af0 tx=0x7f0bfc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.669+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 msgr2=0x7f0c000ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.669+0000 7f0c0844e640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 0x7f0c000ad860 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0bf8009d90 tx=0x7f0bf8004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.670+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 shutdown_connections 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.670+0000 7f0c0844e640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0be406b4d0 0x7f0be406d990 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009af0 tx=0x7f0bfc008040 comp rx=0 tx=0).stop 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.670+0000 7f0c0844e640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c001586e0 0x7f0c000ad860 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.670+0000 7f0c0844e640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c000d01a0 0x7f0c000b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.670+0000 7f0c0844e640 1 --2- 172.21.15.72:0/3267277129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c000aef80 0x7f0c00147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.670+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 >> 172.21.15.72:0/3267277129 conn(0x7f0c0006b190 msgr2=0x7f0c000cfa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.673+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 shutdown_connections 2024-03-06T07:22:45.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.673+0000 7f0c0844e640 1 -- 172.21.15.72:0/3267277129 wait complete. 2024-03-06T07:22:45.695 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:22:45.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.798+0000 7f87f908f640 1 Processor -- start 2024-03-06T07:22:45.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.798+0000 7f87f908f640 1 -- start start 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f908f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f87f40712d0 0x7f87f4071740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f908f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f4072c40 con 0x7f87f406d260 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f908f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f406caa0 con 0x7f87f415a490 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f908f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f4071c80 con 0x7f87f40712d0 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f359e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f87f406d260 0x7f87f406d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41058/0 (socket says 172.21.15.72:41058) 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f359e640 1 -- 172.21.15.72:0/3623131427 learned_addr learned my addr 172.21.15.72:0/3623131427 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870834288 0 0) 0x7f87f4072c40 con 0x7f87f406d260 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e0003520 con 0x7f87f406d260 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87ebfff640 1 --2- 172.21.15.72:0/3623131427 >> v2:172.21.15.146:3300/0 conn(0x7f87f40712d0 0x7f87f4071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577647811 0 0) 0x7f87e0003520 con 0x7f87f406d260 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 >> v2:172.21.15.146:3300/0 conn(0x7f87f40712d0 msgr2=0x7f87f4071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 --2- 172.21.15.72:0/3623131427 >> v2:172.21.15.146:3300/0 conn(0x7f87f40712d0 0x7f87f4071740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 >> v1:172.21.15.146:6789/0 conn(0x7f87f415a490 legacy=0x7f87f4072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40b3340 con 0x7f87f406d260 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f87ec0019f0 con 0x7f87f406d260 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 >> v1:172.21.15.72:6789/0 conn(0x7f87f406d260 legacy=0x7f87f406d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00039b0 0x7f87e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e0004800 0x7f87e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.799+0000 7f87f259c640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e00051c0 0x7f87e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4071c80 con 0x7f87e00051c0 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e000c070 con 0x7f87e0004800 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e000c270 con 0x7f87e00039b0 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f359e640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e00051c0 0x7f87e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f2d9d640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00039b0 0x7f87e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87ebfff640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e0004800 0x7f87e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f359e640 1 -- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00039b0 msgr2=0x7f87e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f359e640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00039b0 0x7f87e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f359e640 1 -- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e0004800 msgr2=0x7f87e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f359e640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e0004800 0x7f87e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.800+0000 7f87f359e640 1 -- 172.21.15.72:0/3623131427 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e0003520 con 0x7f87e00051c0 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f359e640 1 -- 172.21.15.72:0/3623131427 reap_dead start 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f359e640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e00051c0 0x7f87e000b9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00a830 tx=0x7f87e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7508f0b3e7c37053 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00a480 con 0x7f87e00051c0 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec034650 con 0x7f87e00051c0 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f259c640 1 -- 172.21.15.72:0/3623131427 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec034a40 con 0x7f87e00051c0 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 -- 172.21.15.72:0/3623131427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e00051c0 msgr2=0x7f87e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e00051c0 0x7f87e000b9c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00a830 tx=0x7f87e0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 -- 172.21.15.72:0/3623131427 shutdown_connections 2024-03-06T07:22:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 --2- 172.21.15.72:0/3623131427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e00051c0 0x7f87e000b9c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 -- 172.21.15.72:0/3623131427 >> 172.21.15.72:0/3623131427 conn(0x7f87f406afb0 msgr2=0x7f87f406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 -- 172.21.15.72:0/3623131427 shutdown_connections 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.801+0000 7f87f908f640 1 -- 172.21.15.72:0/3623131427 wait complete. 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 Processor -- start 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 -- start start 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00051c0 0x7f87f4151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 0x7f87f40ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f4145d80 0x7f87f4146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40b3900 con 0x7f87f40ae9f0 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f406cd40 con 0x7f87f4145d80 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87f908f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40b3a80 con 0x7f87e00051c0 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87ebfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 0x7f87f40ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87ebfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 0x7f87f40ce3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33722/0 (socket says 172.21.15.72:33722) 2024-03-06T07:22:45.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.803+0000 7f87ebfff640 1 -- 172.21.15.72:0/2022128523 learned_addr learned my addr 172.21.15.72:0/2022128523 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87f2d9d640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00051c0 0x7f87f4151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87f359e640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f4145d80 0x7f87f4146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87ebfff640 1 -- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00051c0 msgr2=0x7f87f4151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87ebfff640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00051c0 0x7f87f4151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87ebfff640 1 -- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f4145d80 msgr2=0x7f87f4146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87ebfff640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f4145d80 0x7f87f4146230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87ebfff640 1 -- 172.21.15.72:0/2022128523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4151d80 con 0x7f87f40ae9f0 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87f2d9d640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00051c0 0x7f87f4151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87f359e640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f4145d80 0x7f87f4146230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87ebfff640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 0x7f87f40ce3c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0118e0 tx=0x7f87dc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00fa90 con 0x7f87f40ae9f0 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc00fc30 con 0x7f87f40ae9f0 2024-03-06T07:22:45.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc014440 con 0x7f87f40ae9f0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f40ce900 con 0x7f87f40ae9f0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.804+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f40af240 con 0x7f87f40ae9f0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.805+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f404f330 con 0x7f87f40ae9f0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.806+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87dc012040 con 0x7f87f40ae9f0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.806+0000 7f87eb7fe640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87d406b500 0x7f87d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.806+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f87dc08a770 con 0x7f87f40ae9f0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.806+0000 7f87f2d9d640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87d406b500 0x7f87d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.807+0000 7f87f2d9d640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87d406b500 0x7f87d406d9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87e4000c00 tx=0x7f87e4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:45.811+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87dc019030 con 0x7f87f40ae9f0 2024-03-06T07:22:46.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.019+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f87f40725c0 con 0x7f87f40ae9f0 2024-03-06T07:22:46.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.676+0000 7f87eb7fe640 1 -- 172.21.15.72:0/2022128523 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f87dc059650 con 0x7f87f40ae9f0 2024-03-06T07:22:46.677 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 4 and data pool 5 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87d406b500 msgr2=0x7f87d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87d406b500 0x7f87d406d9c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87e4000c00 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 msgr2=0x7f87f40ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 0x7f87f40ce3c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f87dc0118e0 tx=0x7f87dc011da0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 shutdown_connections 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87d406b500 0x7f87d406d9c0 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f87e4000c00 tx=0x7f87e4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f4145d80 0x7f87f4146230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87f40ae9f0 0x7f87f40ce3c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 --2- 172.21.15.72:0/2022128523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87e00051c0 0x7f87f4151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.690+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 >> 172.21.15.72:0/2022128523 conn(0x7f87f406afb0 msgr2=0x7f87f406d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.691+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 shutdown_connections 2024-03-06T07:22:46.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.692+0000 7f87f908f640 1 -- 172.21.15.72:0/2022128523 wait complete. 2024-03-06T07:22:46.708 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:46.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.806+0000 7efd94f8a640 1 Processor -- start 2024-03-06T07:22:46.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.806+0000 7efd94f8a640 1 -- start start 2024-03-06T07:22:46.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd94f8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd900d2710 0x7efd900b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd94f8a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd900b3430 con 0x7efd900d22b0 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd94f8a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd900b3710 con 0x7efd9015dfe0 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd94f8a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd900b31e0 con 0x7efd900d2710 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd8ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd900d2710 0x7efd900b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd8dd9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efd900d22b0 0x7efd900b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41068/0 (socket says 172.21.15.72:41068) 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.807+0000 7efd8dd9b640 1 -- 172.21.15.72:0/2144337166 learned_addr learned my addr 172.21.15.72:0/2144337166 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.808+0000 7efd8d59a640 1 -- 172.21.15.72:0/2144337166 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880818981 0 0) 0x7efd900b3430 con 0x7efd900d22b0 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.808+0000 7efd8d59a640 1 -- 172.21.15.72:0/2144337166 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd7c003520 con 0x7efd900d22b0 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.808+0000 7efd8ed9d640 1 -- 172.21.15.72:0/2144337166 >> v1:172.21.15.146:6789/0 conn(0x7efd9015dfe0 legacy=0x7efd900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.808+0000 7efd8ed9d640 1 -- 172.21.15.72:0/2144337166 >> v1:172.21.15.72:6789/0 conn(0x7efd900d22b0 legacy=0x7efd900b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.808+0000 7efd8ed9d640 1 -- 172.21.15.72:0/2144337166 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd900b39f0 con 0x7efd900d2710 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.808+0000 7efd8ed9d640 1 --2- 172.21.15.72:0/2144337166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd900d2710 0x7efd900b4900 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd880097d0 tx=0x7efd8802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2bda909387e024 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd8d59a640 1 -- 172.21.15.72:0/2144337166 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88035020 con 0x7efd900d2710 2024-03-06T07:22:46.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd8d59a640 1 -- 172.21.15.72:0/2144337166 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd88031830 con 0x7efd900d2710 2024-03-06T07:22:46.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd8d59a640 1 -- 172.21.15.72:0/2144337166 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88031c00 con 0x7efd900d2710 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 -- 172.21.15.72:0/2144337166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd900d2710 msgr2=0x7efd900b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 --2- 172.21.15.72:0/2144337166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd900d2710 0x7efd900b4900 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd880097d0 tx=0x7efd8802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 -- 172.21.15.72:0/2144337166 shutdown_connections 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 --2- 172.21.15.72:0/2144337166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd900d2710 0x7efd900b4900 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 -- 172.21.15.72:0/2144337166 >> 172.21.15.72:0/2144337166 conn(0x7efd900d3cd0 msgr2=0x7efd900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 -- 172.21.15.72:0/2144337166 shutdown_connections 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.809+0000 7efd94f8a640 1 -- 172.21.15.72:0/2144337166 wait complete. 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 Processor -- start 2024-03-06T07:22:46.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 -- start start 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd900d22b0 0x7efd900b7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd900d2710 0x7efd900b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 0x7efd90160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900cfa00 con 0x7efd9015dfe0 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd900b3d60 con 0x7efd900d22b0 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd94f8a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd900d1c70 con 0x7efd900d2710 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.811+0000 7efd8ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 0x7efd90160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 0x7efd90160930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33740/0 (socket says 172.21.15.72:33740) 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 -- 172.21.15.72:0/3648582904 learned_addr learned my addr 172.21.15.72:0/3648582904 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8e59c640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd900d22b0 0x7efd900b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8dd9b640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd900d2710 0x7efd900b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 -- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd900d2710 msgr2=0x7efd900b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd900d2710 0x7efd900b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 -- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd900d22b0 msgr2=0x7efd900b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd900d22b0 0x7efd900b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 -- 172.21.15.72:0/3648582904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9015f240 con 0x7efd9015dfe0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8dd9b640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd900d2710 0x7efd900b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd8ed9d640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 0x7efd90160930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efd880097d0 tx=0x7efd880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88035040 con 0x7efd9015dfe0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8803b840 con 0x7efd9015dfe0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9015a770 con 0x7efd9015dfe0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8803bc10 con 0x7efd9015dfe0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.812+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd90156d40 con 0x7efd9015dfe0 2024-03-06T07:22:46.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.813+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd90068090 con 0x7efd9015dfe0 2024-03-06T07:22:46.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.814+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd88038020 con 0x7efd9015dfe0 2024-03-06T07:22:46.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.814+0000 7efd86ffd640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:46.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.814+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd880ac290 con 0x7efd9015dfe0 2024-03-06T07:22:46.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.814+0000 7efd8e59c640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:46.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.814+0000 7efd8e59c640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efd80009f20 tx=0x7efd800094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.818+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd90068090 con 0x7efd9015dfe0 2024-03-06T07:22:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.993+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd9004f330 con 0x7efd9015dfe0 2024-03-06T07:22:46.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.994+0000 7efd86ffd640 1 -- 172.21.15.72:0/3648582904 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16694 (secure 0 0 0) 0x7efd880a9150 con 0x7efd9015dfe0 2024-03-06T07:22:46.995 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:46.995 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:46.019503+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:46.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 msgr2=0x7efd7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efd80009f20 tx=0x7efd800094d0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 msgr2=0x7efd90160930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 0x7efd90160930 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efd880097d0 tx=0x7efd880046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 shutdown_connections 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd9015dfe0 0x7efd90160930 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd900d2710 0x7efd900b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 --2- 172.21.15.72:0/3648582904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd900d22b0 0x7efd900b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 >> 172.21.15.72:0/3648582904 conn(0x7efd900d3cd0 msgr2=0x7efd9006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 shutdown_connections 2024-03-06T07:22:46.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:46.996+0000 7efd94f8a640 1 -- 172.21.15.72:0/3648582904 wait complete. 2024-03-06T07:22:47.008 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:47.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.106+0000 7f6d64c0e640 1 Processor -- start 2024-03-06T07:22:47.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.106+0000 7f6d64c0e640 1 -- start start 2024-03-06T07:22:47.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d64c0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d600cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d64c0e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d600cd7a0 0x7f6d6015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d64c0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d600cc980 con 0x7f6d601424f0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d64c0e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d600cd220 con 0x7f6d600cd7a0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d64c0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600ccc60 con 0x7f6d6006da40 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d5e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6d601424f0 0x7f6d6006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41076/0 (socket says 172.21.15.72:41076) 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d5e59c640 1 -- 172.21.15.72:0/4218987436 learned_addr learned my addr 172.21.15.72:0/4218987436 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191779284 0 0) 0x7f6d600cc980 con 0x7f6d601424f0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d48003520 con 0x7f6d601424f0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d5dd9b640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d600cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.107+0000 7f6d5ed9d640 1 --2- 172.21.15.72:0/4218987436 >> v2:172.21.15.146:3300/0 conn(0x7f6d600cd7a0 0x7f6d6015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4066961914 0 0) 0x7f6d48003520 con 0x7f6d601424f0 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 msgr2=0x7f6d600cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d600cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 >> v2:172.21.15.146:3300/0 conn(0x7f6d600cd7a0 msgr2=0x7f6d6015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 --2- 172.21.15.72:0/4218987436 >> v2:172.21.15.146:3300/0 conn(0x7f6d600cd7a0 0x7f6d6015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d600ccf40 con 0x7f6d601424f0 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6d500019f0 con 0x7f6d601424f0 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 >> v1:172.21.15.72:6789/0 conn(0x7f6d601424f0 legacy=0x7f6d6006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d480039b0 0x7f6d48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d48004800 0x7f6d48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600cd220 con 0x7f6d480039b0 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600ccc60 con 0x7f6d48004800 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4800b9c0 con 0x7f6d6006da40 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d480039b0 0x7f6d48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5dd9b640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d48004800 0x7f6d48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5ed9d640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5e59c640 1 -- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 msgr2=0x7f6d480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5e59c640 1 -- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d48004800 msgr2=0x7f6d48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d48004800 0x7f6d48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5e59c640 1 -- 172.21.15.72:0/4218987436 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48003520 con 0x7f6d480039b0 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5dd9b640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d48004800 0x7f6d48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.108+0000 7f6d5ed9d640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d6006da40 0x7f6d480051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d480039b0 0x7f6d48003dd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6d500013d0 tx=0x7f6d48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804b81a1392b2aae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5000a4e0 con 0x7f6d480039b0 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d50034650 con 0x7f6d480039b0 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d5d59a640 1 -- 172.21.15.72:0/4218987436 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50034a40 con 0x7f6d480039b0 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d64c0e640 1 -- 172.21.15.72:0/4218987436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d480039b0 msgr2=0x7f6d48003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/4218987436 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d480039b0 0x7f6d48003dd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6d500013d0 tx=0x7f6d48005830 comp rx=0 tx=0).stop 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d5ed9d640 1 -- 172.21.15.72:0/4218987436 reap_dead start 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d64c0e640 1 -- 172.21.15.72:0/4218987436 shutdown_connections 2024-03-06T07:22:47.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.109+0000 7f6d64c0e640 1 -- 172.21.15.72:0/4218987436 >> 172.21.15.72:0/4218987436 conn(0x7f6d600cf8a0 msgr2=0x7f6d600cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.110+0000 7f6d64c0e640 1 -- 172.21.15.72:0/4218987436 shutdown_connections 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.110+0000 7f6d64c0e640 1 -- 172.21.15.72:0/4218987436 wait complete. 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.111+0000 7f6d64c0e640 1 Processor -- start 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.111+0000 7f6d64c0e640 1 -- start start 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.111+0000 7f6d64c0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d480039b0 0x7f6d6013d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.111+0000 7f6d64c0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d601428d0 0x7f6d60142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d64c0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 0x7f6d60144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d64c0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600cc470 con 0x7f6d60144880 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d64c0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600cc2f0 con 0x7f6d480039b0 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d64c0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600cc5f0 con 0x7f6d601428d0 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 0x7f6d60144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 0x7f6d60144d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33756/0 (socket says 172.21.15.72:33756) 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 -- 172.21.15.72:0/3492538295 learned_addr learned my addr 172.21.15.72:0/3492538295 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:47.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d480039b0 0x7f6d6013d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5dd9b640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d601428d0 0x7f6d60142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 -- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d601428d0 msgr2=0x7f6d60142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d601428d0 0x7f6d60142d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 -- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d480039b0 msgr2=0x7f6d6013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d480039b0 0x7f6d6013d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 -- 172.21.15.72:0/3492538295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d60145b20 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d5ed9d640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 0x7f6d60144d00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6d600d08d0 tx=0x7f6d58010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800f040 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5800e8d0 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d60145560 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5800eca0 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.112+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d601e6970 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.113+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6004f330 con 0x7f6d60144880 2024-03-06T07:22:47.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.114+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6d5801c400 con 0x7f6d60144880 2024-03-06T07:22:47.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.114+0000 7f6d4effd640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d4006b500 0x7f6d4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.114+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d5808a3b0 con 0x7f6d60144880 2024-03-06T07:22:47.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.114+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d4006b500 0x7f6d4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.115+0000 7f6d5e59c640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d4006b500 0x7f6d4006d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6d50034000 tx=0x7f6d50009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.119+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6d58018030 con 0x7f6d60144880 2024-03-06T07:22:47.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.292+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d600ce4a0 con 0x7f6d60144880 2024-03-06T07:22:47.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.293+0000 7f6d4effd640 1 -- 172.21.15.72:0/3492538295 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f6d580592c0 con 0x7f6d60144880 2024-03-06T07:22:47.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:47.295 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:46.019503+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:47.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d4006b500 msgr2=0x7f6d4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d4006b500 0x7f6d4006d9c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6d50034000 tx=0x7f6d50009030 comp rx=0 tx=0).stop 2024-03-06T07:22:47.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 msgr2=0x7f6d60144d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 0x7f6d60144d00 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6d600d08d0 tx=0x7f6d58010c20 comp rx=0 tx=0).stop 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 shutdown_connections 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6d4006b500 0x7f6d4006d9c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d60144880 0x7f6d60144d00 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d601428d0 0x7f6d60142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 --2- 172.21.15.72:0/3492538295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d480039b0 0x7f6d6013d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 >> 172.21.15.72:0/3492538295 conn(0x7f6d600cf8a0 msgr2=0x7f6d600af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 shutdown_connections 2024-03-06T07:22:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.295+0000 7f6d64c0e640 1 -- 172.21.15.72:0/3492538295 wait complete. 2024-03-06T07:22:47.308 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:22:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.408+0000 7f1df1e81640 1 Processor -- start 2024-03-06T07:22:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1df1e81640 1 -- start start 2024-03-06T07:22:47.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1df1e81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 0x7f1dec066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1df1e81640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1dec0b0db0 0x7f1dec0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1df1e81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dec06b050 con 0x7f1dec0ce580 2024-03-06T07:22:47.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1df1e81640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dec06b2c0 con 0x7f1dec0b0db0 2024-03-06T07:22:47.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1df1e81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0b0890 con 0x7f1dec15f4f0 2024-03-06T07:22:47.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1debfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1dec0ce580 0x7f1dec072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41090/0 (socket says 172.21.15.72:41090) 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.409+0000 7f1debfff640 1 -- 172.21.15.72:0/1382676848 learned_addr learned my addr 172.21.15.72:0/1382676848 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981041215 0 0) 0x7f1dec06b050 con 0x7f1dec0ce580 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 --2- 172.21.15.72:0/1382676848 >> v2:172.21.15.146:3300/0 conn(0x7f1dec0b0db0 0x7f1dec0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dd4003520 con 0x7f1dec0ce580 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deb7fe640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 0x7f1dec066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3854404319 0 0) 0x7f1dd4003520 con 0x7f1dec0ce580 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 msgr2=0x7f1dec066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 0x7f1dec066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 >> v2:172.21.15.146:3300/0 conn(0x7f1dec0b0db0 msgr2=0x7f1dec0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 --2- 172.21.15.72:0/1382676848 >> v2:172.21.15.146:3300/0 conn(0x7f1dec0b0db0 0x7f1dec0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec0672e0 con 0x7f1dec0ce580 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1de40019f0 con 0x7f1dec0ce580 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 >> v1:172.21.15.72:6789/0 conn(0x7f1dec0ce580 legacy=0x7f1dec072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 0x7f1dd40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4004d40 0x7f1dd40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec06b2c0 con 0x7f1dd4003f30 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec0b0890 con 0x7f1dd4004d40 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd400b9c0 con 0x7f1dec15f4f0 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deb7fe640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 0x7f1dd40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1debfff640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4004d40 0x7f1dd40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 -- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 msgr2=0x7f1dd40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec15f4f0 0x7f1dd40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 -- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4004d40 msgr2=0x7f1dd40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4004d40 0x7f1dd40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.410+0000 7f1deaffd640 1 -- 172.21.15.72:0/1382676848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4003520 con 0x7f1dd4003f30 2024-03-06T07:22:47.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.411+0000 7f1deaffd640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009ac0 tx=0x7f1dd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdea0185c46f24cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.411+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0039070 con 0x7f1dd4003f30 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.411+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de002c6d0 con 0x7f1dd4003f30 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.411+0000 7f1dea7fc640 1 -- 172.21.15.72:0/1382676848 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0009690 con 0x7f1dd4003f30 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1df1e81640 1 -- 172.21.15.72:0/1382676848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dd4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1df1e81640 1 --2- 172.21.15.72:0/1382676848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dd4003f30 0x7f1dd4004800 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009ac0 tx=0x7f1dd4005830 comp rx=0 tx=0).stop 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1debfff640 1 -- 172.21.15.72:0/1382676848 reap_dead start 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1df1e81640 1 -- 172.21.15.72:0/1382676848 shutdown_connections 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1df1e81640 1 -- 172.21.15.72:0/1382676848 >> 172.21.15.72:0/1382676848 conn(0x7f1dec0b1d70 msgr2=0x7f1dec0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1df1e81640 1 -- 172.21.15.72:0/1382676848 shutdown_connections 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.412+0000 7f1df1e81640 1 -- 172.21.15.72:0/1382676848 wait complete. 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.413+0000 7f1df1e81640 1 Processor -- start 2024-03-06T07:22:47.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.413+0000 7f1df1e81640 1 -- start start 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1df1e81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4003f30 0x7f1dec0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1df1e81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec0cb4d0 0x7f1dec14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1df1e81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 0x7f1dec072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1df1e81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec15f8d0 con 0x7f1dec072a50 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1df1e81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec06b5a0 con 0x7f1dd4003f30 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1df1e81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec159ca0 con 0x7f1dec0cb4d0 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 0x7f1dec072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 0x7f1dec072ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33770/0 (socket says 172.21.15.72:33770) 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 -- 172.21.15.72:0/2008789995 learned_addr learned my addr 172.21.15.72:0/2008789995 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:47.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1deaffd640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec0cb4d0 0x7f1dec14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 -- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec0cb4d0 msgr2=0x7f1dec14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec0cb4d0 0x7f1dec14d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 -- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4003f30 msgr2=0x7f1dec0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4003f30 0x7f1dec0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 -- 172.21.15.72:0/2008789995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec0c8330 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.414+0000 7f1debfff640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 0x7f1dec072ed0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1de40013d0 tx=0x7f1de4007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.415+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4019070 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.415+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400ac80 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.415+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec0c8610 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.415+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4013630 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.415+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec0c8bd0 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.416+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dec0cb8b0 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.416+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1de401c070 con 0x7f1dec072a50 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.416+0000 7f1dd3fff640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dd806b4d0 0x7f1dd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.416+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1de4089ec0 con 0x7f1dec072a50 2024-03-06T07:22:47.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.416+0000 7f1deb7fe640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dd806b4d0 0x7f1dd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.417+0000 7f1deb7fe640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dd806b4d0 0x7f1dd806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0099e0 tx=0x7f1ddc006c30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.421+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1de4058d40 con 0x7f1dec072a50 2024-03-06T07:22:47.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.633+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1dec09e3d0 con 0x7f1dec072a50 2024-03-06T07:22:47.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.679+0000 7f1dd3fff640 1 -- 172.21.15.72:0/2008789995 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1de4058640 con 0x7f1dec072a50 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dd806b4d0 msgr2=0x7f1dd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dd806b4d0 0x7f1dd806d990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0099e0 tx=0x7f1ddc006c30 comp rx=0 tx=0).stop 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 msgr2=0x7f1dec072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 0x7f1dec072ed0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1de40013d0 tx=0x7f1de4007bd0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 shutdown_connections 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dd806b4d0 0x7f1dd806d990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1dec072a50 0x7f1dec072ed0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.681+0000 7f1df1e81640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dec0cb4d0 0x7f1dec14d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.682+0000 7f1df1e81640 1 --2- 172.21.15.72:0/2008789995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd4003f30 0x7f1dec0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.682+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 >> 172.21.15.72:0/2008789995 conn(0x7f1dec0b1d70 msgr2=0x7f1dec06d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.682+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 shutdown_connections 2024-03-06T07:22:47.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.682+0000 7f1df1e81640 1 -- 172.21.15.72:0/2008789995 wait complete. 2024-03-06T07:22:47.695 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:22:47.695 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:22:47.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.795+0000 7f318473e640 1 Processor -- start 2024-03-06T07:22:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.795+0000 7f318473e640 1 -- start start 2024-03-06T07:22:47.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f318473e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f317c1614d0 0x7f317c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f318473e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f317c0d1910 0x7f317c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f318473e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f317c1618b0 con 0x7f317c0d22e0 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f318473e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317c0cf400 con 0x7f317c1614d0 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f318473e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317c0d2710 con 0x7f317c0d1910 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f31824da640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f317c1614d0 0x7f317c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f3182cdb640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f317c0d22e0 0x7f317c0d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41094/0 (socket says 172.21.15.72:41094) 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.796+0000 7f3182cdb640 1 -- 172.21.15.72:0/1817030027 learned_addr learned my addr 172.21.15.72:0/1817030027 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f3181cd9640 1 --2- 172.21.15.72:0/1817030027 >> v2:172.21.15.146:3300/0 conn(0x7f317c0d1910 0x7f317c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f31814d8640 1 -- 172.21.15.72:0/1817030027 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784771755 0 0) 0x7f317c1618b0 con 0x7f317c0d22e0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f31814d8640 1 -- 172.21.15.72:0/1817030027 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3168003520 con 0x7f317c0d22e0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f31824da640 1 -- 172.21.15.72:0/1817030027 >> v2:172.21.15.146:3300/0 conn(0x7f317c0d1910 msgr2=0x7f317c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f31824da640 1 --2- 172.21.15.72:0/1817030027 >> v2:172.21.15.146:3300/0 conn(0x7f317c0d1910 0x7f317c0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f31824da640 1 -- 172.21.15.72:0/1817030027 >> v1:172.21.15.72:6789/0 conn(0x7f317c0d22e0 legacy=0x7f317c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f31824da640 1 -- 172.21.15.72:0/1817030027 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c0cf6a0 con 0x7f317c1614d0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.797+0000 7f3181cd9640 1 --2- 172.21.15.72:0/1817030027 >> v2:172.21.15.146:3300/0 conn(0x7f317c0d1910 0x7f317c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f31824da640 1 --2- 172.21.15.72:0/1817030027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c1614d0 0x7f317c0d4860 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f317c0ce550 tx=0x7f317402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0375e130528508f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f31814d8640 1 -- 172.21.15.72:0/1817030027 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174039070 con 0x7f317c1614d0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f31814d8640 1 -- 172.21.15.72:0/1817030027 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317402cc80 con 0x7f317c1614d0 2024-03-06T07:22:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f31814d8640 1 -- 172.21.15.72:0/1817030027 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174034660 con 0x7f317c1614d0 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 -- 172.21.15.72:0/1817030027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c1614d0 msgr2=0x7f317c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 --2- 172.21.15.72:0/1817030027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c1614d0 0x7f317c0d4860 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f317c0ce550 tx=0x7f317402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 -- 172.21.15.72:0/1817030027 shutdown_connections 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 --2- 172.21.15.72:0/1817030027 >> v2:172.21.15.146:3300/0 conn(0x7f317c0d1910 0x7f317c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 --2- 172.21.15.72:0/1817030027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c1614d0 0x7f317c0d4860 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 -- 172.21.15.72:0/1817030027 >> 172.21.15.72:0/1817030027 conn(0x7f317c0d3d40 msgr2=0x7f317c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 -- 172.21.15.72:0/1817030027 shutdown_connections 2024-03-06T07:22:47.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.798+0000 7f318473e640 1 -- 172.21.15.72:0/1817030027 wait complete. 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.799+0000 7f318473e640 1 Processor -- start 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.799+0000 7f318473e640 1 -- start start 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f318473e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f317c0d1910 0x7f317c0ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f318473e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 0x7f317c0c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f318473e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f317c1614d0 0x7f317c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f318473e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0b0910 con 0x7f317c0d22e0 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f318473e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0cfc60 con 0x7f317c1614d0 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f318473e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317c0b1e40 con 0x7f317c0d1910 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f3181cd9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 0x7f317c0c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f3181cd9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 0x7f317c0c9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33784/0 (socket says 172.21.15.72:33784) 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f3181cd9640 1 -- 172.21.15.72:0/1503555675 learned_addr learned my addr 172.21.15.72:0/1503555675 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f31824da640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f317c0d1910 0x7f317c0ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.800+0000 7f3182cdb640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f317c1614d0 0x7f317c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3181cd9640 1 -- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f317c0d1910 msgr2=0x7f317c0ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3181cd9640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f317c0d1910 0x7f317c0ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3181cd9640 1 -- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f317c1614d0 msgr2=0x7f317c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3181cd9640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f317c1614d0 0x7f317c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3181cd9640 1 -- 172.21.15.72:0/1503555675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317c157a20 con 0x7f317c0d22e0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f31824da640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f317c0d1910 0x7f317c0ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3181cd9640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 0x7f317c0c9200 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f316c00e760 tx=0x7f316c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c00f040 con 0x7f317c0d22e0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316c004230 con 0x7f317c0d22e0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317c158280 con 0x7f317c0d22e0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316c00ee70 con 0x7f317c0d22e0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.801+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317c157fa0 con 0x7f317c0d22e0 2024-03-06T07:22:47.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.802+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317c068ff0 con 0x7f317c0d22e0 2024-03-06T07:22:47.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.803+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f316c00c6d0 con 0x7f317c0d22e0 2024-03-06T07:22:47.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.803+0000 7f3172ffd640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f316006b4d0 0x7f316006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:47.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.803+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f316c089f40 con 0x7f317c0d22e0 2024-03-06T07:22:47.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.803+0000 7f31824da640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f316006b4d0 0x7f316006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:47.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.803+0000 7f31824da640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f316006b4d0 0x7f316006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3174036000 tx=0x7f3174030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:47.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:47.811+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f316c017020 con 0x7f317c0d22e0 2024-03-06T07:22:48.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.023+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f317c0c71c0 con 0x7f317c0d22e0 2024-03-06T07:22:48.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.683+0000 7f3172ffd640 1 -- 172.21.15.72:0/1503555675 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f316c058dc0 con 0x7f317c0d22e0 2024-03-06T07:22:48.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f316006b4d0 msgr2=0x7f316006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f316006b4d0 0x7f316006d990 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3174036000 tx=0x7f3174030000 comp rx=0 tx=0).stop 2024-03-06T07:22:48.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 msgr2=0x7f317c0c9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 0x7f317c0c9200 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f316c00e760 tx=0x7f316c00ec20 comp rx=0 tx=0).stop 2024-03-06T07:22:48.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 shutdown_connections 2024-03-06T07:22:48.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f316006b4d0 0x7f316006d990 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f317c1614d0 0x7f317c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f317c0d22e0 0x7f317c0c9200 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 --2- 172.21.15.72:0/1503555675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f317c0d1910 0x7f317c0ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 >> 172.21.15.72:0/1503555675 conn(0x7f317c0d3d40 msgr2=0x7f317c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 shutdown_connections 2024-03-06T07:22:48.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.686+0000 7f318473e640 1 -- 172.21.15.72:0/1503555675 wait complete. 2024-03-06T07:22:48.702 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:22:48.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:48.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 Processor -- start 2024-03-06T07:22:48.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 -- start start 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8cc0cb3e0 0x7ff8cc0ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8cc15f370 0x7ff8cc071430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8cc0ade70 con 0x7ff8cc0cb000 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc073340 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8d0fa2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8cc067090 con 0x7ff8cc15f370 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8caffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8cc0cb3e0 0x7ff8cc0ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8cb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff8cc0cb000 0x7ff8cc15ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41102/0 (socket says 172.21.15.72:41102) 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.801+0000 7ff8cb7fe640 1 -- 172.21.15.72:0/3099661730 learned_addr learned my addr 172.21.15.72:0/3099661730 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.802+0000 7ff8caffd640 1 -- 172.21.15.72:0/3099661730 >> v2:172.21.15.146:3300/0 conn(0x7ff8cc15f370 msgr2=0x7ff8cc071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.802+0000 7ff8caffd640 1 --2- 172.21.15.72:0/3099661730 >> v2:172.21.15.146:3300/0 conn(0x7ff8cc15f370 0x7ff8cc071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.802+0000 7ff8caffd640 1 -- 172.21.15.72:0/3099661730 >> v1:172.21.15.72:6789/0 conn(0x7ff8cc0cb000 legacy=0x7ff8cc15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.802+0000 7ff8ca7fc640 1 -- 172.21.15.72:0/3099661730 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404648260 0 0) 0x7ff8cc0ade70 con 0x7ff8cc0cb000 2024-03-06T07:22:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.802+0000 7ff8caffd640 1 -- 172.21.15.72:0/3099661730 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc072e20 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.802+0000 7ff8caffd640 1 --2- 172.21.15.72:0/3099661730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc0ad880 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0cb870 tx=0x7ff8b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40231fec037e469b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8ca7fc640 1 -- 172.21.15.72:0/3099661730 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b40315a0 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8ca7fc640 1 -- 172.21.15.72:0/3099661730 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b4037070 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8ca7fc640 1 -- 172.21.15.72:0/3099661730 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4031960 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/3099661730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 msgr2=0x7ff8cc0ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/3099661730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc0ad880 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc0cb870 tx=0x7ff8b402b840 comp rx=0 tx=0).stop 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/3099661730 shutdown_connections 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/3099661730 >> v2:172.21.15.146:3300/0 conn(0x7ff8cc15f370 0x7ff8cc071430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/3099661730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc0ad880 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/3099661730 >> 172.21.15.72:0/3099661730 conn(0x7ff8cc0cffc0 msgr2=0x7ff8cc0d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:48.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/3099661730 shutdown_connections 2024-03-06T07:22:48.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.803+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/3099661730 wait complete. 2024-03-06T07:22:48.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.805+0000 7ff8d0fa2640 1 Processor -- start 2024-03-06T07:22:48.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.805+0000 7ff8d0fa2640 1 -- start start 2024-03-06T07:22:48.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8d0fa2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8cc0cb000 0x7ff8cc09b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8d0fa2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8d0fa2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8cc15f370 0x7ff8cc1f1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8d0fa2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc06c140 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8d0fa2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc06bfc0 con 0x7ff8cc0cb000 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8d0fa2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8cc15f780 con 0x7ff8cc15f370 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33808/0 (socket says 172.21.15.72:33808) 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 -- 172.21.15.72:0/4134868366 learned_addr learned my addr 172.21.15.72:0/4134868366 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8cb7fe640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8cc0cb000 0x7ff8cc09b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 -- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8cc15f370 msgr2=0x7ff8cc1f1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8cbfff640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8cc15f370 0x7ff8cc1f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8cc15f370 0x7ff8cc1f1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 -- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8cc0cb000 msgr2=0x7ff8cc09b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8cc0cb000 0x7ff8cc09b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8caffd640 1 -- 172.21.15.72:0/4134868366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8cc0c7090 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8cb7fe640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8cc0cb000 0x7ff8cc09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.806+0000 7ff8cbfff640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8cc15f370 0x7ff8cc1f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.807+0000 7ff8caffd640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc15cc00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4034040 tx=0x7ff8b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.807+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4004230 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.807+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8b40043d0 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.807+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8cc09bed0 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.807+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4032880 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.807+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8cc0c7630 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.808+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8cc0ca7c0 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.808+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff8b4032a20 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.809+0000 7ff8c3fff640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8ac06b4d0 0x7ff8ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:48.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.809+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8b402f030 con 0x7ff8cc0cb3e0 2024-03-06T07:22:48.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.809+0000 7ff8cb7fe640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8ac06b4d0 0x7ff8ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:48.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.809+0000 7ff8cb7fe640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8ac06b4d0 0x7ff8ac06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc000f60 tx=0x7ff8bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:48.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:48.813+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff8b4041070 con 0x7ff8cc0cb3e0 2024-03-06T07:22:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.024+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8cc068f80 con 0x7ff8cc0cb3e0 2024-03-06T07:22:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.024+0000 7ff8c3fff640 1 -- 172.21.15.72:0/4134868366 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7ff8b4044340 con 0x7ff8cc0cb3e0 2024-03-06T07:22:49.026 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:49.027 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:22:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:22:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8ac06b4d0 msgr2=0x7ff8ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8ac06b4d0 0x7ff8ac06d990 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc000f60 tx=0x7ff8bc009040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 msgr2=0x7ff8cc15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc15cc00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4034040 tx=0x7ff8b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 shutdown_connections 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8ac06b4d0 0x7ff8ac06d990 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8cc15f370 0x7ff8cc1f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8cc0cb3e0 0x7ff8cc15cc00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 --2- 172.21.15.72:0/4134868366 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8cc0cb000 0x7ff8cc09b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 >> 172.21.15.72:0/4134868366 conn(0x7ff8cc0cffc0 msgr2=0x7ff8cc15d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 shutdown_connections 2024-03-06T07:22:49.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.026+0000 7ff8d0fa2640 1 -- 172.21.15.72:0/4134868366 wait complete. 2024-03-06T07:22:49.039 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:22:49.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.138+0000 7fca197f4640 1 Processor -- start 2024-03-06T07:22:49.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.138+0000 7fca197f4640 1 -- start start 2024-03-06T07:22:49.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca197f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fca140cf790 0x7fca140ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca197f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca140adda0 0x7fca14071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca197f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca140ae240 con 0x7fca14072220 2024-03-06T07:22:49.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca197f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1406cf70 con 0x7fca140cf790 2024-03-06T07:22:49.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca197f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca140ae520 con 0x7fca140adda0 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca12ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fca140cf790 0x7fca140ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca12ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fca140cf790 0x7fca140ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33814/0 (socket says 172.21.15.72:33814) 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.139+0000 7fca12ffd640 1 -- 172.21.15.72:0/382027989 learned_addr learned my addr 172.21.15.72:0/382027989 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.140+0000 7fca127fc640 1 --2- 172.21.15.72:0/382027989 >> v2:172.21.15.146:3300/0 conn(0x7fca140adda0 0x7fca14071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.140+0000 7fca12ffd640 1 -- 172.21.15.72:0/382027989 >> v2:172.21.15.146:3300/0 conn(0x7fca140adda0 msgr2=0x7fca14071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.140+0000 7fca12ffd640 1 --2- 172.21.15.72:0/382027989 >> v2:172.21.15.146:3300/0 conn(0x7fca140adda0 0x7fca14071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.140+0000 7fca12ffd640 1 -- 172.21.15.72:0/382027989 >> v1:172.21.15.146:6789/0 conn(0x7fca14072220 legacy=0x7fca141619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.140+0000 7fca12ffd640 1 -- 172.21.15.72:0/382027989 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca140ae800 con 0x7fca140cf790 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.140+0000 7fca12ffd640 1 --2- 172.21.15.72:0/382027989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca140ad860 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fca140737d0 tx=0x7fca0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f114e92b635fef4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca11ffb640 1 -- 172.21.15.72:0/382027989 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08034070 con 0x7fca140cf790 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca11ffb640 1 -- 172.21.15.72:0/382027989 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca08030860 con 0x7fca140cf790 2024-03-06T07:22:49.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca11ffb640 1 -- 172.21.15.72:0/382027989 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08030c30 con 0x7fca140cf790 2024-03-06T07:22:49.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 -- 172.21.15.72:0/382027989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 msgr2=0x7fca140ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 --2- 172.21.15.72:0/382027989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca140ad860 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fca140737d0 tx=0x7fca0802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 -- 172.21.15.72:0/382027989 shutdown_connections 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 --2- 172.21.15.72:0/382027989 >> v2:172.21.15.146:3300/0 conn(0x7fca140adda0 0x7fca14071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 --2- 172.21.15.72:0/382027989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca140ad860 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 -- 172.21.15.72:0/382027989 >> 172.21.15.72:0/382027989 conn(0x7fca140b1a90 msgr2=0x7fca140b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 -- 172.21.15.72:0/382027989 shutdown_connections 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.141+0000 7fca197f4640 1 -- 172.21.15.72:0/382027989 wait complete. 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.142+0000 7fca197f4640 1 Processor -- start 2024-03-06T07:22:49.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.142+0000 7fca197f4640 1 -- start start 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca197f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca14072220 0x7fca141502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca197f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca140adda0 0x7fca141597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca197f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca1414fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca197f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1415e700 con 0x7fca140cf790 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca197f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1415e580 con 0x7fca140adda0 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca197f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1406e410 con 0x7fca14072220 2024-03-06T07:22:49.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca137fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca1414fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca137fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca1414fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33826/0 (socket says 172.21.15.72:33826) 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca137fe640 1 -- 172.21.15.72:0/1965844572 learned_addr learned my addr 172.21.15.72:0/1965844572 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca12ffd640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca14072220 0x7fca141502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.143+0000 7fca127fc640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca140adda0 0x7fca141597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca137fe640 1 -- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca14072220 msgr2=0x7fca141502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca137fe640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca14072220 0x7fca141502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca137fe640 1 -- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca140adda0 msgr2=0x7fca141597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca137fe640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca140adda0 0x7fca141597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca137fe640 1 -- 172.21.15.72:0/1965844572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca14159cf0 con 0x7fca140cf790 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca127fc640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca140adda0 0x7fca141597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca137fe640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca1414fb50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00c7d0 tx=0x7fca0c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c003b70 con 0x7fca140cf790 2024-03-06T07:22:49.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0c003d30 con 0x7fca140cf790 2024-03-06T07:22:49.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0c00aa90 con 0x7fca140cf790 2024-03-06T07:22:49.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca14150090 con 0x7fca140cf790 2024-03-06T07:22:49.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.144+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca141589f0 con 0x7fca140cf790 2024-03-06T07:22:49.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.145+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca1404f330 con 0x7fca140cf790 2024-03-06T07:22:49.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.146+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca0c00ac30 con 0x7fca140cf790 2024-03-06T07:22:49.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.146+0000 7fc9fb7fe640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca0006b4d0 0x7fca0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.146+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca0c08a080 con 0x7fca140cf790 2024-03-06T07:22:49.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.146+0000 7fca12ffd640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca0006b4d0 0x7fca0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.147+0000 7fca12ffd640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca0006b4d0 0x7fca0006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca08036000 tx=0x7fca0803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.151+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca14159cf0 con 0x7fca140cf790 2024-03-06T07:22:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.329+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca1406e6b0 con 0x7fca140cf790 2024-03-06T07:22:49.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.330+0000 7fc9fb7fe640 1 -- 172.21.15.72:0/1965844572 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fca0c058f00 con 0x7fca140cf790 2024-03-06T07:22:49.332 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:49.333 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":34,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:47.667116+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca0006b4d0 msgr2=0x7fca0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca0006b4d0 0x7fca0006d990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca08036000 tx=0x7fca0803b040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 msgr2=0x7fca1414fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca1414fb50 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fca0c00c7d0 tx=0x7fca0c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 shutdown_connections 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca0006b4d0 0x7fca0006d990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca140cf790 0x7fca1414fb50 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca140adda0 0x7fca141597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 --2- 172.21.15.72:0/1965844572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca14072220 0x7fca141502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.332+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 >> 172.21.15.72:0/1965844572 conn(0x7fca140b1a90 msgr2=0x7fca1415e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.333+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 shutdown_connections 2024-03-06T07:22:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.333+0000 7fca197f4640 1 -- 172.21.15.72:0/1965844572 wait complete. 2024-03-06T07:22:49.349 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:22:49.349 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:49.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.449+0000 7f3e9a577640 1 Processor -- start 2024-03-06T07:22:49.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.449+0000 7f3e9a577640 1 -- start start 2024-03-06T07:22:49.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e9a577640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e9415e090 0x7f3e940d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e9a577640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3e940d0650 0x7f3e940d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e9a577640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e9415e470 con 0x7f3e940d2920 2024-03-06T07:22:49.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e9a577640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e940d0fe0 con 0x7f3e9415e090 2024-03-06T07:22:49.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e9a577640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e940d18f0 con 0x7f3e940d0650 2024-03-06T07:22:49.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e93fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e9415e090 0x7f3e940d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e937fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3e940d2920 0x7f3e940b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41104/0 (socket says 172.21.15.72:41104) 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e937fe640 1 -- 172.21.15.72:0/3301363788 learned_addr learned my addr 172.21.15.72:0/3301363788 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e98b14640 1 --2- 172.21.15.72:0/3301363788 >> v2:172.21.15.146:3300/0 conn(0x7f3e940d0650 0x7f3e940d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e92ffd640 1 -- 172.21.15.72:0/3301363788 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4071782407 0 0) 0x7f3e9415e470 con 0x7f3e940d2920 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e92ffd640 1 -- 172.21.15.72:0/3301363788 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e7c003520 con 0x7f3e940d2920 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e93fff640 1 -- 172.21.15.72:0/3301363788 >> v2:172.21.15.146:3300/0 conn(0x7f3e940d0650 msgr2=0x7f3e940d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.450+0000 7f3e93fff640 1 --2- 172.21.15.72:0/3301363788 >> v2:172.21.15.146:3300/0 conn(0x7f3e940d0650 0x7f3e940d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.451+0000 7f3e93fff640 1 -- 172.21.15.72:0/3301363788 >> v1:172.21.15.72:6789/0 conn(0x7f3e940d2920 legacy=0x7f3e940b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.451+0000 7f3e93fff640 1 -- 172.21.15.72:0/3301363788 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940b3cf0 con 0x7f3e9415e090 2024-03-06T07:22:49.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.451+0000 7f3e98b14640 1 --2- 172.21.15.72:0/3301363788 >> v2:172.21.15.146:3300/0 conn(0x7f3e940d0650 0x7f3e940d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e93fff640 1 --2- 172.21.15.72:0/3301363788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e9415e090 0x7f3e940d47a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e9408c550 tx=0x7f3e8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2751dd84c205816e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e92ffd640 1 -- 172.21.15.72:0/3301363788 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84035020 con 0x7f3e9415e090 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e92ffd640 1 -- 172.21.15.72:0/3301363788 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e84031830 con 0x7f3e9415e090 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e92ffd640 1 -- 172.21.15.72:0/3301363788 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84031be0 con 0x7f3e9415e090 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 -- 172.21.15.72:0/3301363788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e9415e090 msgr2=0x7f3e940d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 --2- 172.21.15.72:0/3301363788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e9415e090 0x7f3e940d47a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e9408c550 tx=0x7f3e8402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 -- 172.21.15.72:0/3301363788 shutdown_connections 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 --2- 172.21.15.72:0/3301363788 >> v2:172.21.15.146:3300/0 conn(0x7f3e940d0650 0x7f3e940d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 --2- 172.21.15.72:0/3301363788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e9415e090 0x7f3e940d47a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 -- 172.21.15.72:0/3301363788 >> 172.21.15.72:0/3301363788 conn(0x7f3e940d3c80 msgr2=0x7f3e940d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 -- 172.21.15.72:0/3301363788 shutdown_connections 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.452+0000 7f3e9a577640 1 -- 172.21.15.72:0/3301363788 wait complete. 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.453+0000 7f3e9a577640 1 Processor -- start 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 -- start start 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 0x7f3e940b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e940d2920 0x7f3e940b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e9415e090 0x7f3e9414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e940cee30 con 0x7f3e940d0650 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e940ceb50 con 0x7f3e940d2920 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e9a577640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e940b2df0 con 0x7f3e9415e090 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 0x7f3e940b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 0x7f3e940b7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33840/0 (socket says 172.21.15.72:33840) 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.454+0000 7f3e93fff640 1 -- 172.21.15.72:0/2440144574 learned_addr learned my addr 172.21.15.72:0/2440144574 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e937fe640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e940d2920 0x7f3e940b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e98b14640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e9415e090 0x7f3e9414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e93fff640 1 -- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e9415e090 msgr2=0x7f3e9414cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e93fff640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e9415e090 0x7f3e9414cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e93fff640 1 -- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e940d2920 msgr2=0x7f3e940b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e93fff640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e940d2920 0x7f3e940b8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e93fff640 1 -- 172.21.15.72:0/2440144574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e940b6e10 con 0x7f3e940d0650 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e937fe640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e940d2920 0x7f3e940b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e98b14640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e9415e090 0x7f3e9414cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e93fff640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 0x7f3e940b7eb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000c00 tx=0x7f3e840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84041070 con 0x7f3e940d0650 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8403b860 con 0x7f3e940d0650 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e940b70f0 con 0x7f3e940d0650 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84035040 con 0x7f3e940d0650 2024-03-06T07:22:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.455+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e940b76b0 con 0x7f3e940d0650 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.456+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9404f330 con 0x7f3e940d0650 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.457+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e84038020 con 0x7f3e940d0650 2024-03-06T07:22:49.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.457+0000 7f3e90ff9640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8006b4d0 0x7f3e8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.457+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e840abaa0 con 0x7f3e940d0650 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.458+0000 7f3e937fe640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8006b4d0 0x7f3e8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.458+0000 7f3e937fe640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8006b4d0 0x7f3e8006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3e88007fd0 tx=0x7f3e88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.462+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e84031800 con 0x7f3e940d0650 2024-03-06T07:22:49.540 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:49.543 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:22:49.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.637+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e940d0430 con 0x7f3e8006b4d0 2024-03-06T07:22:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.643+0000 7f3e90ff9640 1 -- 172.21.15.72:0/2440144574 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251885 (secure 0 0 0) 0x7f3e940d0430 con 0x7f3e8006b4d0 2024-03-06T07:22:49.648 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:49.649 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8006b4d0 msgr2=0x7f3e8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8006b4d0 0x7f3e8006d990 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3e88007fd0 tx=0x7f3e88009040 comp rx=0 tx=0).stop 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 msgr2=0x7f3e940b7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 0x7f3e940b7eb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000c00 tx=0x7f3e840046c0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 shutdown_connections 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8006b4d0 0x7f3e8006d990 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e9415e090 0x7f3e9414cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e940d2920 0x7f3e940b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 --2- 172.21.15.72:0/2440144574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e940d0650 0x7f3e940b7eb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 >> 172.21.15.72:0/2440144574 conn(0x7f3e940d3c80 msgr2=0x7f3e9406ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 shutdown_connections 2024-03-06T07:22:49.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.646+0000 7f3e9a577640 1 -- 172.21.15.72:0/2440144574 wait complete. 2024-03-06T07:22:49.659 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-06T07:22:47.958523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220216,"kb_used_data":4016,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754504,"statfs":{"total":773094113280,"available":772868612096,"internally_reserved":0,"allocated":4112384,"data_stored":2300694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000578"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.690568+0000","last_change":"2024-03-06T07:22:46.687032+0000","last_active":"2024-03-06T07:22:46.690568+0000","last_peered":"2024-03-06T07:22:46.690568+0000","last_clean":"2024-03-06T07:22:46.690568+0000","last_became_active":"2024-03-06T07:22:46.686520+0000","last_became_peered":"2024-03-06T07:22:46.686520+0000","last_unstale":"2024-03-06T07:22:46.690568+0000","last_undegraded":"2024-03-06T07:22:46.690568+0000","last_fullsized":"2024-03-06T07:22:46.690568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:57.438922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687912+0000","last_change":"2024-03-06T07:22:46.682361+0000","last_active":"2024-03-06T07:22:46.687912+0000","last_peered":"2024-03-06T07:22:46.687912+0000","last_clean":"2024-03-06T07:22:46.687912+0000","last_became_active":"2024-03-06T07:22:46.682082+0000","last_became_peered":"2024-03-06T07:22:46.682082+0000","last_unstale":"2024-03-06T07:22:46.687912+0000","last_undegraded":"2024-03-06T07:22:46.687912+0000","last_fullsized":"2024-03-06T07:22:46.687912+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:13.159164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.690607+0000","last_change":"2024-03-06T07:22:46.684641+0000","last_active":"2024-03-06T07:22:46.690607+0000","last_peered":"2024-03-06T07:22:46.690607+0000","last_clean":"2024-03-06T07:22:46.690607+0000","last_became_active":"2024-03-06T07:22:46.684341+0000","last_became_peered":"2024-03-06T07:22:46.684341+0000","last_unstale":"2024-03-06T07:22:46.690607+0000","last_undegraded":"2024-03-06T07:22:46.690607+0000","last_fullsized":"2024-03-06T07:22:46.690607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.458500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699355+0000","last_change":"2024-03-06T07:22:46.697560+0000","last_active":"2024-03-06T07:22:46.699355+0000","last_peered":"2024-03-06T07:22:46.699355+0000","last_clean":"2024-03-06T07:22:46.699355+0000","last_became_active":"2024-03-06T07:22:46.695190+0000","last_became_peered":"2024-03-06T07:22:46.695190+0000","last_unstale":"2024-03-06T07:22:46.699355+0000","last_undegraded":"2024-03-06T07:22:46.699355+0000","last_fullsized":"2024-03-06T07:22:46.699355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.954968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685036+0000","last_change":"2024-03-06T07:22:46.678323+0000","last_active":"2024-03-06T07:22:46.685036+0000","last_peered":"2024-03-06T07:22:46.685036+0000","last_clean":"2024-03-06T07:22:46.685036+0000","last_became_active":"2024-03-06T07:22:46.678080+0000","last_became_peered":"2024-03-06T07:22:46.678080+0000","last_unstale":"2024-03-06T07:22:46.685036+0000","last_undegraded":"2024-03-06T07:22:46.685036+0000","last_fullsized":"2024-03-06T07:22:46.685036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:54.163719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689872+0000","last_change":"2024-03-06T07:22:46.684393+0000","last_active":"2024-03-06T07:22:46.689872+0000","last_peered":"2024-03-06T07:22:46.689872+0000","last_clean":"2024-03-06T07:22:46.689872+0000","last_became_active":"2024-03-06T07:22:46.684120+0000","last_became_peered":"2024-03-06T07:22:46.684120+0000","last_unstale":"2024-03-06T07:22:46.689872+0000","last_undegraded":"2024-03-06T07:22:46.689872+0000","last_fullsized":"2024-03-06T07:22:46.689872+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:44.963260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.693921+0000","last_change":"2024-03-06T07:22:46.688586+0000","last_active":"2024-03-06T07:22:46.693921+0000","last_peered":"2024-03-06T07:22:46.693921+0000","last_clean":"2024-03-06T07:22:46.693921+0000","last_became_active":"2024-03-06T07:22:46.688280+0000","last_became_peered":"2024-03-06T07:22:46.688280+0000","last_unstale":"2024-03-06T07:22:46.693921+0000","last_undegraded":"2024-03-06T07:22:46.693921+0000","last_fullsized":"2024-03-06T07:22:46.693921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:13.159164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.692672+0000","last_change":"2024-03-06T07:22:46.686720+0000","last_active":"2024-03-06T07:22:46.692672+0000","last_peered":"2024-03-06T07:22:46.692672+0000","last_clean":"2024-03-06T07:22:46.692672+0000","last_became_active":"2024-03-06T07:22:46.686418+0000","last_became_peered":"2024-03-06T07:22:46.686418+0000","last_unstale":"2024-03-06T07:22:46.692672+0000","last_undegraded":"2024-03-06T07:22:46.692672+0000","last_fullsized":"2024-03-06T07:22:46.692672+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.790983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699076+0000","last_change":"2024-03-06T07:22:46.694110+0000","last_active":"2024-03-06T07:22:46.699076+0000","last_peered":"2024-03-06T07:22:46.699076+0000","last_clean":"2024-03-06T07:22:46.699076+0000","last_became_active":"2024-03-06T07:22:46.693050+0000","last_became_peered":"2024-03-06T07:22:46.693050+0000","last_unstale":"2024-03-06T07:22:46.699076+0000","last_undegraded":"2024-03-06T07:22:46.699076+0000","last_fullsized":"2024-03-06T07:22:46.699076+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:57.438922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.692677+0000","last_change":"2024-03-06T07:22:46.688246+0000","last_active":"2024-03-06T07:22:46.692677+0000","last_peered":"2024-03-06T07:22:46.692677+0000","last_clean":"2024-03-06T07:22:46.692677+0000","last_became_active":"2024-03-06T07:22:46.687953+0000","last_became_peered":"2024-03-06T07:22:46.687953+0000","last_unstale":"2024-03-06T07:22:46.692677+0000","last_undegraded":"2024-03-06T07:22:46.692677+0000","last_fullsized":"2024-03-06T07:22:46.692677+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.409708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689500+0000","last_change":"2024-03-06T07:22:46.684614+0000","last_active":"2024-03-06T07:22:46.689500+0000","last_peered":"2024-03-06T07:22:46.689500+0000","last_clean":"2024-03-06T07:22:46.689500+0000","last_became_active":"2024-03-06T07:22:46.683962+0000","last_became_peered":"2024-03-06T07:22:46.683962+0000","last_unstale":"2024-03-06T07:22:46.689500+0000","last_undegraded":"2024-03-06T07:22:46.689500+0000","last_fullsized":"2024-03-06T07:22:46.689500+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:38.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689921+0000","last_change":"2024-03-06T07:22:46.683785+0000","last_active":"2024-03-06T07:22:46.689921+0000","last_peered":"2024-03-06T07:22:46.689921+0000","last_clean":"2024-03-06T07:22:46.689921+0000","last_became_active":"2024-03-06T07:22:46.683495+0000","last_became_peered":"2024-03-06T07:22:46.683495+0000","last_unstale":"2024-03-06T07:22:46.689921+0000","last_undegraded":"2024-03-06T07:22:46.689921+0000","last_fullsized":"2024-03-06T07:22:46.689921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:46.721544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.679787+0000","last_change":"2024-03-06T07:22:46.674124+0000","last_active":"2024-03-06T07:22:46.679787+0000","last_peered":"2024-03-06T07:22:46.679787+0000","last_clean":"2024-03-06T07:22:46.679787+0000","last_became_active":"2024-03-06T07:22:46.673799+0000","last_became_peered":"2024-03-06T07:22:46.673799+0000","last_unstale":"2024-03-06T07:22:46.679787+0000","last_undegraded":"2024-03-06T07:22:46.679787+0000","last_fullsized":"2024-03-06T07:22:46.679787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:42.429504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689854+0000","last_change":"2024-03-06T07:22:46.681198+0000","last_active":"2024-03-06T07:22:46.689854+0000","last_peered":"2024-03-06T07:22:46.689854+0000","last_clean":"2024-03-06T07:22:46.689854+0000","last_became_active":"2024-03-06T07:22:46.680952+0000","last_became_peered":"2024-03-06T07:22:46.680952+0000","last_unstale":"2024-03-06T07:22:46.689854+0000","last_undegraded":"2024-03-06T07:22:46.689854+0000","last_fullsized":"2024-03-06T07:22:46.689854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.790983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689778+0000","last_change":"2024-03-06T07:22:46.683267+0000","last_active":"2024-03-06T07:22:46.689778+0000","last_peered":"2024-03-06T07:22:46.689778+0000","last_clean":"2024-03-06T07:22:46.689778+0000","last_became_active":"2024-03-06T07:22:46.682975+0000","last_became_peered":"2024-03-06T07:22:46.682975+0000","last_unstale":"2024-03-06T07:22:46.689778+0000","last_undegraded":"2024-03-06T07:22:46.689778+0000","last_fullsized":"2024-03-06T07:22:46.689778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:16.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.695748+0000","last_change":"2024-03-06T07:22:46.689884+0000","last_active":"2024-03-06T07:22:46.695748+0000","last_peered":"2024-03-06T07:22:46.695748+0000","last_clean":"2024-03-06T07:22:46.695748+0000","last_became_active":"2024-03-06T07:22:46.689602+0000","last_became_peered":"2024-03-06T07:22:46.689602+0000","last_unstale":"2024-03-06T07:22:46.695748+0000","last_undegraded":"2024-03-06T07:22:46.695748+0000","last_fullsized":"2024-03-06T07:22:46.695748+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:32.008067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.691837+0000","last_change":"2024-03-06T07:22:46.685250+0000","last_active":"2024-03-06T07:22:46.691837+0000","last_peered":"2024-03-06T07:22:46.691837+0000","last_clean":"2024-03-06T07:22:46.691837+0000","last_became_active":"2024-03-06T07:22:46.683498+0000","last_became_peered":"2024-03-06T07:22:46.683498+0000","last_unstale":"2024-03-06T07:22:46.691837+0000","last_undegraded":"2024-03-06T07:22:46.691837+0000","last_fullsized":"2024-03-06T07:22:46.691837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.954968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696841+0000","last_change":"2024-03-06T07:22:46.691530+0000","last_active":"2024-03-06T07:22:46.696841+0000","last_peered":"2024-03-06T07:22:46.696841+0000","last_clean":"2024-03-06T07:22:46.696841+0000","last_became_active":"2024-03-06T07:22:46.691226+0000","last_became_peered":"2024-03-06T07:22:46.691226+0000","last_unstale":"2024-03-06T07:22:46.696841+0000","last_undegraded":"2024-03-06T07:22:46.696841+0000","last_fullsized":"2024-03-06T07:22:46.696841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:27.146574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687841+0000","last_change":"2024-03-06T07:22:46.682935+0000","last_active":"2024-03-06T07:22:46.687841+0000","last_peered":"2024-03-06T07:22:46.687841+0000","last_clean":"2024-03-06T07:22:46.687841+0000","last_became_active":"2024-03-06T07:22:46.682679+0000","last_became_peered":"2024-03-06T07:22:46.682679+0000","last_unstale":"2024-03-06T07:22:46.687841+0000","last_undegraded":"2024-03-06T07:22:46.687841+0000","last_fullsized":"2024-03-06T07:22:46.687841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:32.008067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.684977+0000","last_change":"2024-03-06T07:22:46.679651+0000","last_active":"2024-03-06T07:22:46.684977+0000","last_peered":"2024-03-06T07:22:46.684977+0000","last_clean":"2024-03-06T07:22:46.684977+0000","last_became_active":"2024-03-06T07:22:46.679417+0000","last_became_peered":"2024-03-06T07:22:46.679417+0000","last_unstale":"2024-03-06T07:22:46.684977+0000","last_undegraded":"2024-03-06T07:22:46.684977+0000","last_fullsized":"2024-03-06T07:22:46.684977+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:42.877251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696465+0000","last_change":"2024-03-06T07:22:46.692363+0000","last_active":"2024-03-06T07:22:46.696465+0000","last_peered":"2024-03-06T07:22:46.696465+0000","last_clean":"2024-03-06T07:22:46.696465+0000","last_became_active":"2024-03-06T07:22:46.692086+0000","last_became_peered":"2024-03-06T07:22:46.692086+0000","last_unstale":"2024-03-06T07:22:46.696465+0000","last_undegraded":"2024-03-06T07:22:46.696465+0000","last_fullsized":"2024-03-06T07:22:46.696465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:31.431049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.701850+0000","last_change":"2024-03-06T07:22:46.697671+0000","last_active":"2024-03-06T07:22:46.701850+0000","last_peered":"2024-03-06T07:22:46.701850+0000","last_clean":"2024-03-06T07:22:46.701850+0000","last_became_active":"2024-03-06T07:22:46.697461+0000","last_became_peered":"2024-03-06T07:22:46.697461+0000","last_unstale":"2024-03-06T07:22:46.701850+0000","last_undegraded":"2024-03-06T07:22:46.701850+0000","last_fullsized":"2024-03-06T07:22:46.701850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:06.331598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685041+0000","last_change":"2024-03-06T07:22:46.681333+0000","last_active":"2024-03-06T07:22:46.685041+0000","last_peered":"2024-03-06T07:22:46.685041+0000","last_clean":"2024-03-06T07:22:46.685041+0000","last_became_active":"2024-03-06T07:22:46.681022+0000","last_became_peered":"2024-03-06T07:22:46.681022+0000","last_unstale":"2024-03-06T07:22:46.685041+0000","last_undegraded":"2024-03-06T07:22:46.685041+0000","last_fullsized":"2024-03-06T07:22:46.685041+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:02.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689767+0000","last_change":"2024-03-06T07:22:46.683613+0000","last_active":"2024-03-06T07:22:46.689767+0000","last_peered":"2024-03-06T07:22:46.689767+0000","last_clean":"2024-03-06T07:22:46.689767+0000","last_became_active":"2024-03-06T07:22:46.683123+0000","last_became_peered":"2024-03-06T07:22:46.683123+0000","last_unstale":"2024-03-06T07:22:46.689767+0000","last_undegraded":"2024-03-06T07:22:46.689767+0000","last_fullsized":"2024-03-06T07:22:46.689767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.954968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689163+0000","last_change":"2024-03-06T07:22:46.684110+0000","last_active":"2024-03-06T07:22:46.689163+0000","last_peered":"2024-03-06T07:22:46.689163+0000","last_clean":"2024-03-06T07:22:46.689163+0000","last_became_active":"2024-03-06T07:22:46.683826+0000","last_became_peered":"2024-03-06T07:22:46.683826+0000","last_unstale":"2024-03-06T07:22:46.689163+0000","last_undegraded":"2024-03-06T07:22:46.689163+0000","last_fullsized":"2024-03-06T07:22:46.689163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:02.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687936+0000","last_change":"2024-03-06T07:22:46.683483+0000","last_active":"2024-03-06T07:22:46.687936+0000","last_peered":"2024-03-06T07:22:46.687936+0000","last_clean":"2024-03-06T07:22:46.687936+0000","last_became_active":"2024-03-06T07:22:46.682294+0000","last_became_peered":"2024-03-06T07:22:46.682294+0000","last_unstale":"2024-03-06T07:22:46.687936+0000","last_undegraded":"2024-03-06T07:22:46.687936+0000","last_fullsized":"2024-03-06T07:22:46.687936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:25.360873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699110+0000","last_change":"2024-03-06T07:22:46.696994+0000","last_active":"2024-03-06T07:22:46.699110+0000","last_peered":"2024-03-06T07:22:46.699110+0000","last_clean":"2024-03-06T07:22:46.699110+0000","last_became_active":"2024-03-06T07:22:46.696539+0000","last_became_peered":"2024-03-06T07:22:46.696539+0000","last_unstale":"2024-03-06T07:22:46.699110+0000","last_undegraded":"2024-03-06T07:22:46.699110+0000","last_fullsized":"2024-03-06T07:22:46.699110+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:38.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685266+0000","last_change":"2024-03-06T07:22:46.681842+0000","last_active":"2024-03-06T07:22:46.685266+0000","last_peered":"2024-03-06T07:22:46.685266+0000","last_clean":"2024-03-06T07:22:46.685266+0000","last_became_active":"2024-03-06T07:22:46.680146+0000","last_became_peered":"2024-03-06T07:22:46.680146+0000","last_unstale":"2024-03-06T07:22:46.685266+0000","last_undegraded":"2024-03-06T07:22:46.685266+0000","last_fullsized":"2024-03-06T07:22:46.685266+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:30.921192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696487+0000","last_change":"2024-03-06T07:22:46.692899+0000","last_active":"2024-03-06T07:22:46.696487+0000","last_peered":"2024-03-06T07:22:46.696487+0000","last_clean":"2024-03-06T07:22:46.696487+0000","last_became_active":"2024-03-06T07:22:46.692634+0000","last_became_peered":"2024-03-06T07:22:46.692634+0000","last_unstale":"2024-03-06T07:22:46.696487+0000","last_undegraded":"2024-03-06T07:22:46.696487+0000","last_fullsized":"2024-03-06T07:22:46.696487+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:13.715103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689356+0000","last_change":"2024-03-06T07:22:46.683599+0000","last_active":"2024-03-06T07:22:46.689356+0000","last_peered":"2024-03-06T07:22:46.689356+0000","last_clean":"2024-03-06T07:22:46.689356+0000","last_became_active":"2024-03-06T07:22:46.683183+0000","last_became_peered":"2024-03-06T07:22:46.683183+0000","last_unstale":"2024-03-06T07:22:46.689356+0000","last_undegraded":"2024-03-06T07:22:46.689356+0000","last_fullsized":"2024-03-06T07:22:46.689356+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:16.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687511+0000","last_change":"2024-03-06T07:22:46.680810+0000","last_active":"2024-03-06T07:22:46.687511+0000","last_peered":"2024-03-06T07:22:46.687511+0000","last_clean":"2024-03-06T07:22:46.687511+0000","last_became_active":"2024-03-06T07:22:46.680513+0000","last_became_peered":"2024-03-06T07:22:46.680513+0000","last_unstale":"2024-03-06T07:22:46.687511+0000","last_undegraded":"2024-03-06T07:22:46.687511+0000","last_fullsized":"2024-03-06T07:22:46.687511+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:13.715103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698696+0000","last_change":"2024-03-06T07:22:46.695418+0000","last_active":"2024-03-06T07:22:46.698696+0000","last_peered":"2024-03-06T07:22:46.698696+0000","last_clean":"2024-03-06T07:22:46.698696+0000","last_became_active":"2024-03-06T07:22:46.695161+0000","last_became_peered":"2024-03-06T07:22:46.695161+0000","last_unstale":"2024-03-06T07:22:46.698696+0000","last_undegraded":"2024-03-06T07:22:46.698696+0000","last_fullsized":"2024-03-06T07:22:46.698696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:44.963260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685375+0000","last_change":"2024-03-06T07:22:46.681692+0000","last_active":"2024-03-06T07:22:46.685375+0000","last_peered":"2024-03-06T07:22:46.685375+0000","last_clean":"2024-03-06T07:22:46.685375+0000","last_became_active":"2024-03-06T07:22:46.681454+0000","last_became_peered":"2024-03-06T07:22:46.681454+0000","last_unstale":"2024-03-06T07:22:46.685375+0000","last_undegraded":"2024-03-06T07:22:46.685375+0000","last_fullsized":"2024-03-06T07:22:46.685375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.285087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699391+0000","last_change":"2024-03-06T07:22:45.683890+0000","last_active":"2024-03-06T07:22:46.699391+0000","last_peered":"2024-03-06T07:22:46.699391+0000","last_clean":"2024-03-06T07:22:46.699391+0000","last_became_active":"2024-03-06T07:22:45.683576+0000","last_became_peered":"2024-03-06T07:22:45.683576+0000","last_unstale":"2024-03-06T07:22:46.699391+0000","last_undegraded":"2024-03-06T07:22:46.699391+0000","last_fullsized":"2024-03-06T07:22:46.699391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:30.425903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.697014+0000","last_change":"2024-03-06T07:22:46.693884+0000","last_active":"2024-03-06T07:22:46.697014+0000","last_peered":"2024-03-06T07:22:46.697014+0000","last_clean":"2024-03-06T07:22:46.697014+0000","last_became_active":"2024-03-06T07:22:46.693528+0000","last_became_peered":"2024-03-06T07:22:46.693528+0000","last_unstale":"2024-03-06T07:22:46.697014+0000","last_undegraded":"2024-03-06T07:22:46.697014+0000","last_fullsized":"2024-03-06T07:22:46.697014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:06.331598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.705678+0000","last_change":"2024-03-06T07:22:45.683697+0000","last_active":"2024-03-06T07:22:46.705678+0000","last_peered":"2024-03-06T07:22:46.705678+0000","last_clean":"2024-03-06T07:22:46.705678+0000","last_became_active":"2024-03-06T07:22:45.683386+0000","last_became_peered":"2024-03-06T07:22:45.683386+0000","last_unstale":"2024-03-06T07:22:46.705678+0000","last_undegraded":"2024-03-06T07:22:46.705678+0000","last_fullsized":"2024-03-06T07:22:46.705678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:56.433777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.701855+0000","last_change":"2024-03-06T07:22:46.697693+0000","last_active":"2024-03-06T07:22:46.701855+0000","last_peered":"2024-03-06T07:22:46.701855+0000","last_clean":"2024-03-06T07:22:46.701855+0000","last_became_active":"2024-03-06T07:22:46.697471+0000","last_became_peered":"2024-03-06T07:22:46.697471+0000","last_unstale":"2024-03-06T07:22:46.701855+0000","last_undegraded":"2024-03-06T07:22:46.701855+0000","last_fullsized":"2024-03-06T07:22:46.701855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:08.126249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698635+0000","last_change":"2024-03-06T07:22:45.681427+0000","last_active":"2024-03-06T07:22:46.698635+0000","last_peered":"2024-03-06T07:22:46.698635+0000","last_clean":"2024-03-06T07:22:46.698635+0000","last_became_active":"2024-03-06T07:22:45.681151+0000","last_became_peered":"2024-03-06T07:22:45.681151+0000","last_unstale":"2024-03-06T07:22:46.698635+0000","last_undegraded":"2024-03-06T07:22:46.698635+0000","last_fullsized":"2024-03-06T07:22:46.698635+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:40.404563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698408+0000","last_change":"2024-03-06T07:22:46.695423+0000","last_active":"2024-03-06T07:22:46.698408+0000","last_peered":"2024-03-06T07:22:46.698408+0000","last_clean":"2024-03-06T07:22:46.698408+0000","last_became_active":"2024-03-06T07:22:46.694312+0000","last_became_peered":"2024-03-06T07:22:46.694312+0000","last_unstale":"2024-03-06T07:22:46.698408+0000","last_undegraded":"2024-03-06T07:22:46.698408+0000","last_fullsized":"2024-03-06T07:22:46.698408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:16.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698248+0000","last_change":"2024-03-06T07:22:45.682417+0000","last_active":"2024-03-06T07:22:46.698248+0000","last_peered":"2024-03-06T07:22:46.698248+0000","last_clean":"2024-03-06T07:22:46.698248+0000","last_became_active":"2024-03-06T07:22:45.682127+0000","last_became_peered":"2024-03-06T07:22:45.682127+0000","last_unstale":"2024-03-06T07:22:46.698248+0000","last_undegraded":"2024-03-06T07:22:46.698248+0000","last_fullsized":"2024-03-06T07:22:46.698248+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:15.110242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699145+0000","last_change":"2024-03-06T07:22:46.696484+0000","last_active":"2024-03-06T07:22:46.699145+0000","last_peered":"2024-03-06T07:22:46.699145+0000","last_clean":"2024-03-06T07:22:46.699145+0000","last_became_active":"2024-03-06T07:22:46.696193+0000","last_became_peered":"2024-03-06T07:22:46.696193+0000","last_unstale":"2024-03-06T07:22:46.699145+0000","last_undegraded":"2024-03-06T07:22:46.699145+0000","last_fullsized":"2024-03-06T07:22:46.699145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.434268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699347+0000","last_change":"2024-03-06T07:22:45.687622+0000","last_active":"2024-03-06T07:22:46.699347+0000","last_peered":"2024-03-06T07:22:46.699347+0000","last_clean":"2024-03-06T07:22:46.699347+0000","last_became_active":"2024-03-06T07:22:45.687347+0000","last_became_peered":"2024-03-06T07:22:45.687347+0000","last_unstale":"2024-03-06T07:22:46.699347+0000","last_undegraded":"2024-03-06T07:22:46.699347+0000","last_fullsized":"2024-03-06T07:22:46.699347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:26.838356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685244+0000","last_change":"2024-03-06T07:22:46.681188+0000","last_active":"2024-03-06T07:22:46.685244+0000","last_peered":"2024-03-06T07:22:46.685244+0000","last_clean":"2024-03-06T07:22:46.685244+0000","last_became_active":"2024-03-06T07:22:46.680900+0000","last_became_peered":"2024-03-06T07:22:46.680900+0000","last_unstale":"2024-03-06T07:22:46.685244+0000","last_undegraded":"2024-03-06T07:22:46.685244+0000","last_fullsized":"2024-03-06T07:22:46.685244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.409708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699274+0000","last_change":"2024-03-06T07:22:45.687631+0000","last_active":"2024-03-06T07:22:46.699274+0000","last_peered":"2024-03-06T07:22:46.699274+0000","last_clean":"2024-03-06T07:22:46.699274+0000","last_became_active":"2024-03-06T07:22:45.687377+0000","last_became_peered":"2024-03-06T07:22:45.687377+0000","last_unstale":"2024-03-06T07:22:46.699274+0000","last_undegraded":"2024-03-06T07:22:46.699274+0000","last_fullsized":"2024-03-06T07:22:46.699274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:42.446950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696166+0000","last_change":"2024-03-06T07:22:46.692340+0000","last_active":"2024-03-06T07:22:46.696166+0000","last_peered":"2024-03-06T07:22:46.696166+0000","last_clean":"2024-03-06T07:22:46.696166+0000","last_became_active":"2024-03-06T07:22:46.691337+0000","last_became_peered":"2024-03-06T07:22:46.691337+0000","last_unstale":"2024-03-06T07:22:46.696166+0000","last_undegraded":"2024-03-06T07:22:46.696166+0000","last_fullsized":"2024-03-06T07:22:46.696166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:35.370622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.675009+0000","last_change":"2024-03-06T07:22:45.679514+0000","last_active":"2024-03-06T07:22:46.675009+0000","last_peered":"2024-03-06T07:22:46.675009+0000","last_clean":"2024-03-06T07:22:46.675009+0000","last_became_active":"2024-03-06T07:22:45.679217+0000","last_became_peered":"2024-03-06T07:22:45.679217+0000","last_unstale":"2024-03-06T07:22:46.675009+0000","last_undegraded":"2024-03-06T07:22:46.675009+0000","last_fullsized":"2024-03-06T07:22:46.675009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:28.279942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.688953+0000","last_change":"2024-03-06T07:22:46.682408+0000","last_active":"2024-03-06T07:22:46.688953+0000","last_peered":"2024-03-06T07:22:46.688953+0000","last_clean":"2024-03-06T07:22:46.688953+0000","last_became_active":"2024-03-06T07:22:46.682152+0000","last_became_peered":"2024-03-06T07:22:46.682152+0000","last_unstale":"2024-03-06T07:22:46.688953+0000","last_undegraded":"2024-03-06T07:22:46.688953+0000","last_fullsized":"2024-03-06T07:22:46.688953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:08.126249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687171+0000","last_change":"2024-03-06T07:22:45.681654+0000","last_active":"2024-03-06T07:22:46.687171+0000","last_peered":"2024-03-06T07:22:46.687171+0000","last_clean":"2024-03-06T07:22:46.687171+0000","last_became_active":"2024-03-06T07:22:45.681349+0000","last_became_peered":"2024-03-06T07:22:45.681349+0000","last_unstale":"2024-03-06T07:22:46.687171+0000","last_undegraded":"2024-03-06T07:22:46.687171+0000","last_fullsized":"2024-03-06T07:22:46.687171+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:25.763463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.691450+0000","last_change":"2024-03-06T07:22:46.686873+0000","last_active":"2024-03-06T07:22:46.691450+0000","last_peered":"2024-03-06T07:22:46.691450+0000","last_clean":"2024-03-06T07:22:46.691450+0000","last_became_active":"2024-03-06T07:22:46.686627+0000","last_became_peered":"2024-03-06T07:22:46.686627+0000","last_unstale":"2024-03-06T07:22:46.691450+0000","last_undegraded":"2024-03-06T07:22:46.691450+0000","last_fullsized":"2024-03-06T07:22:46.691450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:13.159164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676514+0000","last_change":"2024-03-06T07:22:45.678280+0000","last_active":"2024-03-06T07:22:46.676514+0000","last_peered":"2024-03-06T07:22:46.676514+0000","last_clean":"2024-03-06T07:22:46.676514+0000","last_became_active":"2024-03-06T07:22:45.677981+0000","last_became_peered":"2024-03-06T07:22:45.677981+0000","last_unstale":"2024-03-06T07:22:46.676514+0000","last_undegraded":"2024-03-06T07:22:46.676514+0000","last_fullsized":"2024-03-06T07:22:46.676514+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:15.261198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687506+0000","last_change":"2024-03-06T07:22:46.682322+0000","last_active":"2024-03-06T07:22:46.687506+0000","last_peered":"2024-03-06T07:22:46.687506+0000","last_clean":"2024-03-06T07:22:46.687506+0000","last_became_active":"2024-03-06T07:22:46.682050+0000","last_became_peered":"2024-03-06T07:22:46.682050+0000","last_unstale":"2024-03-06T07:22:46.687506+0000","last_undegraded":"2024-03-06T07:22:46.687506+0000","last_fullsized":"2024-03-06T07:22:46.687506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.458500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.674183+0000","last_change":"2024-03-06T07:22:45.687492+0000","last_active":"2024-03-06T07:22:46.674183+0000","last_peered":"2024-03-06T07:22:46.674183+0000","last_clean":"2024-03-06T07:22:46.674183+0000","last_became_active":"2024-03-06T07:22:45.687232+0000","last_became_peered":"2024-03-06T07:22:45.687232+0000","last_unstale":"2024-03-06T07:22:46.674183+0000","last_undegraded":"2024-03-06T07:22:46.674183+0000","last_fullsized":"2024-03-06T07:22:46.674183+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:29.251456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696810+0000","last_change":"2024-03-06T07:22:46.690820+0000","last_active":"2024-03-06T07:22:46.696810+0000","last_peered":"2024-03-06T07:22:46.696810+0000","last_clean":"2024-03-06T07:22:46.696810+0000","last_became_active":"2024-03-06T07:22:46.690598+0000","last_became_peered":"2024-03-06T07:22:46.690598+0000","last_unstale":"2024-03-06T07:22:46.696810+0000","last_undegraded":"2024-03-06T07:22:46.696810+0000","last_fullsized":"2024-03-06T07:22:46.696810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:06.906553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698885+0000","last_change":"2024-03-06T07:22:45.677524+0000","last_active":"2024-03-06T07:22:46.698885+0000","last_peered":"2024-03-06T07:22:46.698885+0000","last_clean":"2024-03-06T07:22:46.698885+0000","last_became_active":"2024-03-06T07:22:45.677212+0000","last_became_peered":"2024-03-06T07:22:45.677212+0000","last_unstale":"2024-03-06T07:22:46.698885+0000","last_undegraded":"2024-03-06T07:22:46.698885+0000","last_fullsized":"2024-03-06T07:22:46.698885+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:01.692220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.690587+0000","last_change":"2024-03-06T07:22:46.684429+0000","last_active":"2024-03-06T07:22:46.690587+0000","last_peered":"2024-03-06T07:22:46.690587+0000","last_clean":"2024-03-06T07:22:46.690587+0000","last_became_active":"2024-03-06T07:22:46.683996+0000","last_became_peered":"2024-03-06T07:22:46.683996+0000","last_unstale":"2024-03-06T07:22:46.690587+0000","last_undegraded":"2024-03-06T07:22:46.690587+0000","last_fullsized":"2024-03-06T07:22:46.690587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.790983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.708456+0000","last_change":"2024-03-06T07:22:45.682558+0000","last_active":"2024-03-06T07:22:46.708456+0000","last_peered":"2024-03-06T07:22:46.708456+0000","last_clean":"2024-03-06T07:22:46.708456+0000","last_became_active":"2024-03-06T07:22:45.682286+0000","last_became_peered":"2024-03-06T07:22:45.682286+0000","last_unstale":"2024-03-06T07:22:46.708456+0000","last_undegraded":"2024-03-06T07:22:46.708456+0000","last_fullsized":"2024-03-06T07:22:46.708456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:33.949823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.688955+0000","last_change":"2024-03-06T07:22:46.683052+0000","last_active":"2024-03-06T07:22:46.688955+0000","last_peered":"2024-03-06T07:22:46.688955+0000","last_clean":"2024-03-06T07:22:46.688955+0000","last_became_active":"2024-03-06T07:22:46.682804+0000","last_became_peered":"2024-03-06T07:22:46.682804+0000","last_unstale":"2024-03-06T07:22:46.688955+0000","last_undegraded":"2024-03-06T07:22:46.688955+0000","last_fullsized":"2024-03-06T07:22:46.688955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.285087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699339+0000","last_change":"2024-03-06T07:22:45.684990+0000","last_active":"2024-03-06T07:22:46.699339+0000","last_peered":"2024-03-06T07:22:46.699339+0000","last_clean":"2024-03-06T07:22:46.699339+0000","last_became_active":"2024-03-06T07:22:45.684009+0000","last_became_peered":"2024-03-06T07:22:45.684009+0000","last_unstale":"2024-03-06T07:22:46.699339+0000","last_undegraded":"2024-03-06T07:22:46.699339+0000","last_fullsized":"2024-03-06T07:22:46.699339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:13.909996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687595+0000","last_change":"2024-03-06T07:22:46.681196+0000","last_active":"2024-03-06T07:22:46.687595+0000","last_peered":"2024-03-06T07:22:46.687595+0000","last_clean":"2024-03-06T07:22:46.687595+0000","last_became_active":"2024-03-06T07:22:46.680311+0000","last_became_peered":"2024-03-06T07:22:46.680311+0000","last_unstale":"2024-03-06T07:22:46.687595+0000","last_undegraded":"2024-03-06T07:22:46.687595+0000","last_fullsized":"2024-03-06T07:22:46.687595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.434268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.673030+0000","last_change":"2024-03-06T07:22:45.682610+0000","last_active":"2024-03-06T07:22:46.673030+0000","last_peered":"2024-03-06T07:22:46.673030+0000","last_clean":"2024-03-06T07:22:46.673030+0000","last_became_active":"2024-03-06T07:22:45.682308+0000","last_became_peered":"2024-03-06T07:22:45.682308+0000","last_unstale":"2024-03-06T07:22:46.673030+0000","last_undegraded":"2024-03-06T07:22:46.673030+0000","last_fullsized":"2024-03-06T07:22:46.673030+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:29.251456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698675+0000","last_change":"2024-03-06T07:22:46.694848+0000","last_active":"2024-03-06T07:22:46.698675+0000","last_peered":"2024-03-06T07:22:46.698675+0000","last_clean":"2024-03-06T07:22:46.698675+0000","last_became_active":"2024-03-06T07:22:46.694590+0000","last_became_peered":"2024-03-06T07:22:46.694590+0000","last_unstale":"2024-03-06T07:22:46.698675+0000","last_undegraded":"2024-03-06T07:22:46.698675+0000","last_fullsized":"2024-03-06T07:22:46.698675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:25.360873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676584+0000","last_change":"2024-03-06T07:22:45.677534+0000","last_active":"2024-03-06T07:22:46.676584+0000","last_peered":"2024-03-06T07:22:46.676584+0000","last_clean":"2024-03-06T07:22:46.676584+0000","last_became_active":"2024-03-06T07:22:45.677269+0000","last_became_peered":"2024-03-06T07:22:45.677269+0000","last_unstale":"2024-03-06T07:22:46.676584+0000","last_undegraded":"2024-03-06T07:22:46.676584+0000","last_fullsized":"2024-03-06T07:22:46.676584+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:37.016501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696982+0000","last_change":"2024-03-06T07:22:46.693884+0000","last_active":"2024-03-06T07:22:46.696982+0000","last_peered":"2024-03-06T07:22:46.696982+0000","last_clean":"2024-03-06T07:22:46.696982+0000","last_became_active":"2024-03-06T07:22:46.693586+0000","last_became_peered":"2024-03-06T07:22:46.693586+0000","last_unstale":"2024-03-06T07:22:46.696982+0000","last_undegraded":"2024-03-06T07:22:46.696982+0000","last_fullsized":"2024-03-06T07:22:46.696982+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:30.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699698+0000","last_change":"2024-03-06T07:22:45.683364+0000","last_active":"2024-03-06T07:22:46.699698+0000","last_peered":"2024-03-06T07:22:46.699698+0000","last_clean":"2024-03-06T07:22:46.699698+0000","last_became_active":"2024-03-06T07:22:45.682941+0000","last_became_peered":"2024-03-06T07:22:45.682941+0000","last_unstale":"2024-03-06T07:22:46.699698+0000","last_undegraded":"2024-03-06T07:22:46.699698+0000","last_fullsized":"2024-03-06T07:22:46.699698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:01.692220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699147+0000","last_change":"2024-03-06T07:22:46.694776+0000","last_active":"2024-03-06T07:22:46.699147+0000","last_peered":"2024-03-06T07:22:46.699147+0000","last_clean":"2024-03-06T07:22:46.699147+0000","last_became_active":"2024-03-06T07:22:46.694447+0000","last_became_peered":"2024-03-06T07:22:46.694447+0000","last_unstale":"2024-03-06T07:22:46.699147+0000","last_undegraded":"2024-03-06T07:22:46.699147+0000","last_fullsized":"2024-03-06T07:22:46.699147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:30.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.697056+0000","last_change":"2024-03-06T07:22:45.680986+0000","last_active":"2024-03-06T07:22:46.697056+0000","last_peered":"2024-03-06T07:22:46.697056+0000","last_clean":"2024-03-06T07:22:46.697056+0000","last_became_active":"2024-03-06T07:22:45.680691+0000","last_became_peered":"2024-03-06T07:22:45.680691+0000","last_unstale":"2024-03-06T07:22:46.697056+0000","last_undegraded":"2024-03-06T07:22:46.697056+0000","last_fullsized":"2024-03-06T07:22:46.697056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:35.146579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699260+0000","last_change":"2024-03-06T07:22:46.696429+0000","last_active":"2024-03-06T07:22:46.699260+0000","last_peered":"2024-03-06T07:22:46.699260+0000","last_clean":"2024-03-06T07:22:46.699260+0000","last_became_active":"2024-03-06T07:22:46.696151+0000","last_became_peered":"2024-03-06T07:22:46.696151+0000","last_unstale":"2024-03-06T07:22:46.699260+0000","last_undegraded":"2024-03-06T07:22:46.699260+0000","last_fullsized":"2024-03-06T07:22:46.699260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:32.008067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.697409+0000","last_change":"2024-03-06T07:22:45.681681+0000","last_active":"2024-03-06T07:22:46.697409+0000","last_peered":"2024-03-06T07:22:46.697409+0000","last_clean":"2024-03-06T07:22:46.697409+0000","last_became_active":"2024-03-06T07:22:45.681370+0000","last_became_peered":"2024-03-06T07:22:45.681370+0000","last_unstale":"2024-03-06T07:22:46.697409+0000","last_undegraded":"2024-03-06T07:22:46.697409+0000","last_fullsized":"2024-03-06T07:22:46.697409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:16.743767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698539+0000","last_change":"2024-03-06T07:22:46.694891+0000","last_active":"2024-03-06T07:22:46.698539+0000","last_peered":"2024-03-06T07:22:46.698539+0000","last_clean":"2024-03-06T07:22:46.698539+0000","last_became_active":"2024-03-06T07:22:46.694581+0000","last_became_peered":"2024-03-06T07:22:46.694581+0000","last_unstale":"2024-03-06T07:22:46.698539+0000","last_undegraded":"2024-03-06T07:22:46.698539+0000","last_fullsized":"2024-03-06T07:22:46.698539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:30.921192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698280+0000","last_change":"2024-03-06T07:22:45.680736+0000","last_active":"2024-03-06T07:22:46.698280+0000","last_peered":"2024-03-06T07:22:46.698280+0000","last_clean":"2024-03-06T07:22:46.698280+0000","last_became_active":"2024-03-06T07:22:45.680418+0000","last_became_peered":"2024-03-06T07:22:45.680418+0000","last_unstale":"2024-03-06T07:22:46.698280+0000","last_undegraded":"2024-03-06T07:22:46.698280+0000","last_fullsized":"2024-03-06T07:22:46.698280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:19.097331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687818+0000","last_change":"2024-03-06T07:22:46.683628+0000","last_active":"2024-03-06T07:22:46.687818+0000","last_peered":"2024-03-06T07:22:46.687818+0000","last_clean":"2024-03-06T07:22:46.687818+0000","last_became_active":"2024-03-06T07:22:46.683348+0000","last_became_peered":"2024-03-06T07:22:46.683348+0000","last_unstale":"2024-03-06T07:22:46.687818+0000","last_undegraded":"2024-03-06T07:22:46.687818+0000","last_fullsized":"2024-03-06T07:22:46.687818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:54.086368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.688873+0000","last_change":"2024-03-06T07:22:45.687626+0000","last_active":"2024-03-06T07:22:46.688873+0000","last_peered":"2024-03-06T07:22:46.688873+0000","last_clean":"2024-03-06T07:22:46.688873+0000","last_became_active":"2024-03-06T07:22:45.687366+0000","last_became_peered":"2024-03-06T07:22:45.687366+0000","last_unstale":"2024-03-06T07:22:46.688873+0000","last_undegraded":"2024-03-06T07:22:46.688873+0000","last_fullsized":"2024-03-06T07:22:46.688873+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:54.715746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.695751+0000","last_change":"2024-03-06T07:22:46.692567+0000","last_active":"2024-03-06T07:22:46.695751+0000","last_peered":"2024-03-06T07:22:46.695751+0000","last_clean":"2024-03-06T07:22:46.695751+0000","last_became_active":"2024-03-06T07:22:46.692286+0000","last_became_peered":"2024-03-06T07:22:46.692286+0000","last_unstale":"2024-03-06T07:22:46.695751+0000","last_undegraded":"2024-03-06T07:22:46.695751+0000","last_fullsized":"2024-03-06T07:22:46.695751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:27.205868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.705713+0000","last_change":"2024-03-06T07:22:45.685383+0000","last_active":"2024-03-06T07:22:46.705713+0000","last_peered":"2024-03-06T07:22:46.705713+0000","last_clean":"2024-03-06T07:22:46.705713+0000","last_became_active":"2024-03-06T07:22:45.685106+0000","last_became_peered":"2024-03-06T07:22:45.685106+0000","last_unstale":"2024-03-06T07:22:46.705713+0000","last_undegraded":"2024-03-06T07:22:46.705713+0000","last_fullsized":"2024-03-06T07:22:46.705713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:12.709958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685965+0000","last_change":"2024-03-06T07:22:46.681534+0000","last_active":"2024-03-06T07:22:46.685965+0000","last_peered":"2024-03-06T07:22:46.685965+0000","last_clean":"2024-03-06T07:22:46.685965+0000","last_became_active":"2024-03-06T07:22:46.681287+0000","last_became_peered":"2024-03-06T07:22:46.681287+0000","last_unstale":"2024-03-06T07:22:46.685965+0000","last_undegraded":"2024-03-06T07:22:46.685965+0000","last_fullsized":"2024-03-06T07:22:46.685965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:17.464263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676841+0000","last_change":"2024-03-06T07:22:45.683095+0000","last_active":"2024-03-06T07:22:46.676841+0000","last_peered":"2024-03-06T07:22:46.676841+0000","last_clean":"2024-03-06T07:22:46.676841+0000","last_became_active":"2024-03-06T07:22:45.682763+0000","last_became_peered":"2024-03-06T07:22:45.682763+0000","last_unstale":"2024-03-06T07:22:46.676841+0000","last_undegraded":"2024-03-06T07:22:46.676841+0000","last_fullsized":"2024-03-06T07:22:46.676841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:42.446950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685939+0000","last_change":"2024-03-06T07:22:46.680842+0000","last_active":"2024-03-06T07:22:46.685939+0000","last_peered":"2024-03-06T07:22:46.685939+0000","last_clean":"2024-03-06T07:22:46.685939+0000","last_became_active":"2024-03-06T07:22:46.680573+0000","last_became_peered":"2024-03-06T07:22:46.680573+0000","last_unstale":"2024-03-06T07:22:46.685939+0000","last_undegraded":"2024-03-06T07:22:46.685939+0000","last_fullsized":"2024-03-06T07:22:46.685939+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:27.562539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.683858+0000","last_change":"2024-03-06T07:22:45.683932+0000","last_active":"2024-03-06T07:22:46.683858+0000","last_peered":"2024-03-06T07:22:46.683858+0000","last_clean":"2024-03-06T07:22:46.683858+0000","last_became_active":"2024-03-06T07:22:45.683635+0000","last_became_peered":"2024-03-06T07:22:45.683635+0000","last_unstale":"2024-03-06T07:22:46.683858+0000","last_undegraded":"2024-03-06T07:22:46.683858+0000","last_fullsized":"2024-03-06T07:22:46.683858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:53.081223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.693899+0000","last_change":"2024-03-06T07:22:46.687248+0000","last_active":"2024-03-06T07:22:46.693899+0000","last_peered":"2024-03-06T07:22:46.693899+0000","last_clean":"2024-03-06T07:22:46.693899+0000","last_became_active":"2024-03-06T07:22:46.687007+0000","last_became_peered":"2024-03-06T07:22:46.687007+0000","last_unstale":"2024-03-06T07:22:46.693899+0000","last_undegraded":"2024-03-06T07:22:46.693899+0000","last_fullsized":"2024-03-06T07:22:46.693899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.458500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.674304+0000","last_change":"2024-03-06T07:22:45.676794+0000","last_active":"2024-03-06T07:22:46.674304+0000","last_peered":"2024-03-06T07:22:46.674304+0000","last_clean":"2024-03-06T07:22:46.674304+0000","last_became_active":"2024-03-06T07:22:45.676537+0000","last_became_peered":"2024-03-06T07:22:45.676537+0000","last_unstale":"2024-03-06T07:22:46.674304+0000","last_undegraded":"2024-03-06T07:22:46.674304+0000","last_fullsized":"2024-03-06T07:22:46.674304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:28.279942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685551+0000","last_change":"2024-03-06T07:22:46.679013+0000","last_active":"2024-03-06T07:22:46.685551+0000","last_peered":"2024-03-06T07:22:46.685551+0000","last_clean":"2024-03-06T07:22:46.685551+0000","last_became_active":"2024-03-06T07:22:46.678774+0000","last_became_peered":"2024-03-06T07:22:46.678774+0000","last_unstale":"2024-03-06T07:22:46.685551+0000","last_undegraded":"2024-03-06T07:22:46.685551+0000","last_fullsized":"2024-03-06T07:22:46.685551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:12.097807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699233+0000","last_change":"2024-03-06T07:22:45.681308+0000","last_active":"2024-03-06T07:22:46.699233+0000","last_peered":"2024-03-06T07:22:46.699233+0000","last_clean":"2024-03-06T07:22:46.699233+0000","last_became_active":"2024-03-06T07:22:45.680925+0000","last_became_peered":"2024-03-06T07:22:45.680925+0000","last_unstale":"2024-03-06T07:22:46.699233+0000","last_undegraded":"2024-03-06T07:22:46.699233+0000","last_fullsized":"2024-03-06T07:22:46.699233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:56.433777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689747+0000","last_change":"2024-03-06T07:22:46.683580+0000","last_active":"2024-03-06T07:22:46.689747+0000","last_peered":"2024-03-06T07:22:46.689747+0000","last_clean":"2024-03-06T07:22:46.689747+0000","last_became_active":"2024-03-06T07:22:46.683274+0000","last_became_peered":"2024-03-06T07:22:46.683274+0000","last_unstale":"2024-03-06T07:22:46.689747+0000","last_undegraded":"2024-03-06T07:22:46.689747+0000","last_fullsized":"2024-03-06T07:22:46.689747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:38.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.674330+0000","last_change":"2024-03-06T07:22:45.684753+0000","last_active":"2024-03-06T07:22:46.674330+0000","last_peered":"2024-03-06T07:22:46.674330+0000","last_clean":"2024-03-06T07:22:46.674330+0000","last_became_active":"2024-03-06T07:22:45.684453+0000","last_became_peered":"2024-03-06T07:22:45.684453+0000","last_unstale":"2024-03-06T07:22:46.674330+0000","last_undegraded":"2024-03-06T07:22:46.674330+0000","last_fullsized":"2024-03-06T07:22:46.674330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:05.326453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.686876+0000","last_change":"2024-03-06T07:22:46.682304+0000","last_active":"2024-03-06T07:22:46.686876+0000","last_peered":"2024-03-06T07:22:46.686876+0000","last_clean":"2024-03-06T07:22:46.686876+0000","last_became_active":"2024-03-06T07:22:46.682045+0000","last_became_peered":"2024-03-06T07:22:46.682045+0000","last_unstale":"2024-03-06T07:22:46.686876+0000","last_undegraded":"2024-03-06T07:22:46.686876+0000","last_fullsized":"2024-03-06T07:22:46.686876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.409708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.672163+0000","last_change":"2024-03-06T07:22:45.681201+0000","last_active":"2024-03-06T07:22:46.672163+0000","last_peered":"2024-03-06T07:22:46.672163+0000","last_clean":"2024-03-06T07:22:46.672163+0000","last_became_active":"2024-03-06T07:22:45.680780+0000","last_became_peered":"2024-03-06T07:22:45.680780+0000","last_unstale":"2024-03-06T07:22:46.672163+0000","last_undegraded":"2024-03-06T07:22:46.672163+0000","last_fullsized":"2024-03-06T07:22:46.672163+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:12.709958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687717+0000","last_change":"2024-03-06T07:22:46.681681+0000","last_active":"2024-03-06T07:22:46.687717+0000","last_peered":"2024-03-06T07:22:46.687717+0000","last_clean":"2024-03-06T07:22:46.687717+0000","last_became_active":"2024-03-06T07:22:46.681457+0000","last_became_peered":"2024-03-06T07:22:46.681457+0000","last_unstale":"2024-03-06T07:22:46.687717+0000","last_undegraded":"2024-03-06T07:22:46.687717+0000","last_fullsized":"2024-03-06T07:22:46.687717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:36.151724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.672592+0000","last_change":"2024-03-06T07:22:45.684589+0000","last_active":"2024-03-06T07:22:46.672592+0000","last_peered":"2024-03-06T07:22:46.672592+0000","last_clean":"2024-03-06T07:22:46.672592+0000","last_became_active":"2024-03-06T07:22:45.684342+0000","last_became_peered":"2024-03-06T07:22:45.684342+0000","last_unstale":"2024-03-06T07:22:46.672592+0000","last_undegraded":"2024-03-06T07:22:46.672592+0000","last_fullsized":"2024-03-06T07:22:46.672592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:35.146579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685085+0000","last_change":"2024-03-06T07:22:46.680473+0000","last_active":"2024-03-06T07:22:46.685085+0000","last_peered":"2024-03-06T07:22:46.685085+0000","last_clean":"2024-03-06T07:22:46.685085+0000","last_became_active":"2024-03-06T07:22:46.680214+0000","last_became_peered":"2024-03-06T07:22:46.680214+0000","last_unstale":"2024-03-06T07:22:46.685085+0000","last_undegraded":"2024-03-06T07:22:46.685085+0000","last_fullsized":"2024-03-06T07:22:46.685085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:08.126249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.673177+0000","last_change":"2024-03-06T07:22:45.681955+0000","last_active":"2024-03-06T07:22:46.673177+0000","last_peered":"2024-03-06T07:22:46.673177+0000","last_clean":"2024-03-06T07:22:46.673177+0000","last_became_active":"2024-03-06T07:22:45.681577+0000","last_became_peered":"2024-03-06T07:22:45.681577+0000","last_unstale":"2024-03-06T07:22:46.673177+0000","last_undegraded":"2024-03-06T07:22:46.673177+0000","last_fullsized":"2024-03-06T07:22:46.673177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:05.326453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685916+0000","last_change":"2024-03-06T07:22:46.680739+0000","last_active":"2024-03-06T07:22:46.685916+0000","last_peered":"2024-03-06T07:22:46.685916+0000","last_clean":"2024-03-06T07:22:46.685916+0000","last_became_active":"2024-03-06T07:22:46.680286+0000","last_became_peered":"2024-03-06T07:22:46.680286+0000","last_unstale":"2024-03-06T07:22:46.685916+0000","last_undegraded":"2024-03-06T07:22:46.685916+0000","last_fullsized":"2024-03-06T07:22:46.685916+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:46.721544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.684035+0000","last_change":"2024-03-06T07:22:45.683758+0000","last_active":"2024-03-06T07:22:46.684035+0000","last_peered":"2024-03-06T07:22:46.684035+0000","last_clean":"2024-03-06T07:22:46.684035+0000","last_became_active":"2024-03-06T07:22:45.683497+0000","last_became_peered":"2024-03-06T07:22:45.683497+0000","last_unstale":"2024-03-06T07:22:46.684035+0000","last_undegraded":"2024-03-06T07:22:46.684035+0000","last_fullsized":"2024-03-06T07:22:46.684035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:26.200722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696201+0000","last_change":"2024-03-06T07:22:46.693567+0000","last_active":"2024-03-06T07:22:46.696201+0000","last_peered":"2024-03-06T07:22:46.696201+0000","last_clean":"2024-03-06T07:22:46.696201+0000","last_became_active":"2024-03-06T07:22:46.693315+0000","last_became_peered":"2024-03-06T07:22:46.693315+0000","last_unstale":"2024-03-06T07:22:46.696201+0000","last_undegraded":"2024-03-06T07:22:46.696201+0000","last_fullsized":"2024-03-06T07:22:46.696201+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:36.151724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.675663+0000","last_change":"2024-03-06T07:22:45.678345+0000","last_active":"2024-03-06T07:22:46.675663+0000","last_peered":"2024-03-06T07:22:46.675663+0000","last_clean":"2024-03-06T07:22:46.675663+0000","last_became_active":"2024-03-06T07:22:45.678073+0000","last_became_peered":"2024-03-06T07:22:45.678073+0000","last_unstale":"2024-03-06T07:22:46.675663+0000","last_undegraded":"2024-03-06T07:22:46.675663+0000","last_fullsized":"2024-03-06T07:22:46.675663+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:29.916047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.692585+0000","last_change":"2024-03-06T07:22:46.686443+0000","last_active":"2024-03-06T07:22:46.692585+0000","last_peered":"2024-03-06T07:22:46.692585+0000","last_clean":"2024-03-06T07:22:46.692585+0000","last_became_active":"2024-03-06T07:22:46.686169+0000","last_became_peered":"2024-03-06T07:22:46.686169+0000","last_unstale":"2024-03-06T07:22:46.692585+0000","last_undegraded":"2024-03-06T07:22:46.692585+0000","last_fullsized":"2024-03-06T07:22:46.692585+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:46.721544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676909+0000","last_change":"2024-03-06T07:22:45.684161+0000","last_active":"2024-03-06T07:22:46.676909+0000","last_peered":"2024-03-06T07:22:46.676909+0000","last_clean":"2024-03-06T07:22:46.676909+0000","last_became_active":"2024-03-06T07:22:45.683873+0000","last_became_peered":"2024-03-06T07:22:45.683873+0000","last_unstale":"2024-03-06T07:22:46.676909+0000","last_undegraded":"2024-03-06T07:22:46.676909+0000","last_fullsized":"2024-03-06T07:22:46.676909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:34.365476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.677839+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:22:46.677839+0000","last_peered":"2024-03-06T07:22:46.677839+0000","last_clean":"2024-03-06T07:22:46.677839+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:22:46.677839+0000","last_undegraded":"2024-03-06T07:22:46.677839+0000","last_fullsized":"2024-03-06T07:22:46.677839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":954368,"data_stored":728572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":145651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":145083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":954368,"data_stored":728572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:49.663 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:22:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.798+0000 7f7399ed4640 1 Processor -- start 2024-03-06T07:22:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.798+0000 7f7399ed4640 1 -- start start 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7399ed4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f739415dfe0 0x7f73940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7399ed4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940cf830 0x7f73940d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7399ed4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73940b3430 con 0x7f73940d27f0 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7399ed4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73940d2c50 con 0x7f739415dfe0 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7399ed4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73940b31e0 con 0x7f73940cf830 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7393fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f73940d27f0 0x7f73940d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41114/0 (socket says 172.21.15.72:41114) 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7393fff640 1 -- 172.21.15.72:0/1353282667 learned_addr learned my addr 172.21.15.72:0/1353282667 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:49.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f73937fe640 1 --2- 172.21.15.72:0/1353282667 >> v2:172.21.15.72:3300/0 conn(0x7f739415dfe0 0x7f73940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f7392ffd640 1 --2- 172.21.15.72:0/1353282667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940cf830 0x7f73940d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f73937fe640 1 -- 172.21.15.72:0/1353282667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940cf830 msgr2=0x7f73940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f73937fe640 1 --2- 172.21.15.72:0/1353282667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940cf830 0x7f73940d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f73937fe640 1 -- 172.21.15.72:0/1353282667 >> v1:172.21.15.72:6789/0 conn(0x7f73940d27f0 legacy=0x7f73940d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f73927fc640 1 -- 172.21.15.72:0/1353282667 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682951483 0 0) 0x7f73940b3430 con 0x7f73940d27f0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.799+0000 7f73937fe640 1 -- 172.21.15.72:0/1353282667 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73940b3710 con 0x7f739415dfe0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.800+0000 7f73937fe640 1 --2- 172.21.15.72:0/1353282667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f73940cf2f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f73840097a0 tx=0x7f738402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=222af3785b016f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.800+0000 7f73927fc640 1 -- 172.21.15.72:0/1353282667 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384035020 con 0x7f739415dfe0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.800+0000 7f73927fc640 1 -- 172.21.15.72:0/1353282667 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7384031830 con 0x7f739415dfe0 2024-03-06T07:22:49.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.800+0000 7f73927fc640 1 -- 172.21.15.72:0/1353282667 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7384031c00 con 0x7f739415dfe0 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 -- 172.21.15.72:0/1353282667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 msgr2=0x7f73940cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 --2- 172.21.15.72:0/1353282667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f73940cf2f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f73840097a0 tx=0x7f738402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 -- 172.21.15.72:0/1353282667 shutdown_connections 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 --2- 172.21.15.72:0/1353282667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940cf830 0x7f73940d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 --2- 172.21.15.72:0/1353282667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f73940cf2f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 -- 172.21.15.72:0/1353282667 >> 172.21.15.72:0/1353282667 conn(0x7f73940d3cd0 msgr2=0x7f73940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 -- 172.21.15.72:0/1353282667 shutdown_connections 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.801+0000 7f7399ed4640 1 -- 172.21.15.72:0/1353282667 wait complete. 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.802+0000 7f7399ed4640 1 Processor -- start 2024-03-06T07:22:49.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.802+0000 7f7399ed4640 1 -- start start 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940cf830 0x7f73940cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940d27f0 0x7f739415e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f739415ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73940d1c70 con 0x7f739415dfe0 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73940b3d60 con 0x7f73940cf830 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73940d1df0 con 0x7f73940d27f0 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f739415ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f739415ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33850/0 (socket says 172.21.15.72:33850) 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 -- 172.21.15.72:0/88077657 learned_addr learned my addr 172.21.15.72:0/88077657 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 -- 172.21.15.72:0/88077657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940d27f0 msgr2=0x7f739415e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940d27f0 0x7f739415e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 -- 172.21.15.72:0/88077657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940cf830 msgr2=0x7f73940cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:49.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940cf830 0x7f73940cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 -- 172.21.15.72:0/88077657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73940b6a40 con 0x7f739415dfe0 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7393fff640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f739415ea20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f738c00c7a0 tx=0x7f738c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c018070 con 0x7f739415dfe0 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c00a640 con 0x7f739415dfe0 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f739415f050 con 0x7f739415dfe0 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.803+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c00aab0 con 0x7f739415dfe0 2024-03-06T07:22:49.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.804+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73940b6ce0 con 0x7f739415dfe0 2024-03-06T07:22:49.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.805+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739404f330 con 0x7f739415dfe0 2024-03-06T07:22:49.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.805+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f738c01b070 con 0x7f739415dfe0 2024-03-06T07:22:49.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.805+0000 7f737bfff640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f738006b500 0x7f738006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:49.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.806+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f738c089720 con 0x7f739415dfe0 2024-03-06T07:22:49.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.806+0000 7f73937fe640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f738006b500 0x7f738006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:49.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.806+0000 7f73937fe640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f738006b500 0x7f738006d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7384036000 tx=0x7f7384030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:49.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:49.814+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f738c058630 con 0x7f739415dfe0 2024-03-06T07:22:50.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.010+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73940b0a60 con 0x7f738006b500 2024-03-06T07:22:50.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.015+0000 7f737bfff640 1 -- 172.21.15.72:0/88077657 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251885 (secure 0 0 0) 0x7f73940b0a60 con 0x7f738006b500 2024-03-06T07:22:50.020 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:50.021 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f738006b500 msgr2=0x7f738006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f738006b500 0x7f738006d9c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7384036000 tx=0x7f7384030000 comp rx=0 tx=0).stop 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 msgr2=0x7f739415ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f739415ea20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f738c00c7a0 tx=0x7f738c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 shutdown_connections 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f738006b500 0x7f738006d9c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739415dfe0 0x7f739415ea20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73940d27f0 0x7f739415e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 --2- 172.21.15.72:0/88077657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73940cf830 0x7f73940cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.018+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 >> 172.21.15.72:0/88077657 conn(0x7f73940d3cd0 msgr2=0x7f739406cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.019+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 shutdown_connections 2024-03-06T07:22:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.019+0000 7f7399ed4640 1 -- 172.21.15.72:0/88077657 wait complete. 2024-03-06T07:22:50.031 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-06T07:22:49.958843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220216,"kb_used_data":4016,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754504,"statfs":{"total":773094113280,"available":772868612096,"internally_reserved":0,"allocated":4112384,"data_stored":2300694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.298189"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.690568+0000","last_change":"2024-03-06T07:22:46.687032+0000","last_active":"2024-03-06T07:22:46.690568+0000","last_peered":"2024-03-06T07:22:46.690568+0000","last_clean":"2024-03-06T07:22:46.690568+0000","last_became_active":"2024-03-06T07:22:46.686520+0000","last_became_peered":"2024-03-06T07:22:46.686520+0000","last_unstale":"2024-03-06T07:22:46.690568+0000","last_undegraded":"2024-03-06T07:22:46.690568+0000","last_fullsized":"2024-03-06T07:22:46.690568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:57.438922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687912+0000","last_change":"2024-03-06T07:22:46.682361+0000","last_active":"2024-03-06T07:22:46.687912+0000","last_peered":"2024-03-06T07:22:46.687912+0000","last_clean":"2024-03-06T07:22:46.687912+0000","last_became_active":"2024-03-06T07:22:46.682082+0000","last_became_peered":"2024-03-06T07:22:46.682082+0000","last_unstale":"2024-03-06T07:22:46.687912+0000","last_undegraded":"2024-03-06T07:22:46.687912+0000","last_fullsized":"2024-03-06T07:22:46.687912+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:13.159164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.690607+0000","last_change":"2024-03-06T07:22:46.684641+0000","last_active":"2024-03-06T07:22:46.690607+0000","last_peered":"2024-03-06T07:22:46.690607+0000","last_clean":"2024-03-06T07:22:46.690607+0000","last_became_active":"2024-03-06T07:22:46.684341+0000","last_became_peered":"2024-03-06T07:22:46.684341+0000","last_unstale":"2024-03-06T07:22:46.690607+0000","last_undegraded":"2024-03-06T07:22:46.690607+0000","last_fullsized":"2024-03-06T07:22:46.690607+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.458500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699355+0000","last_change":"2024-03-06T07:22:46.697560+0000","last_active":"2024-03-06T07:22:46.699355+0000","last_peered":"2024-03-06T07:22:46.699355+0000","last_clean":"2024-03-06T07:22:46.699355+0000","last_became_active":"2024-03-06T07:22:46.695190+0000","last_became_peered":"2024-03-06T07:22:46.695190+0000","last_unstale":"2024-03-06T07:22:46.699355+0000","last_undegraded":"2024-03-06T07:22:46.699355+0000","last_fullsized":"2024-03-06T07:22:46.699355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.954968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685036+0000","last_change":"2024-03-06T07:22:46.678323+0000","last_active":"2024-03-06T07:22:46.685036+0000","last_peered":"2024-03-06T07:22:46.685036+0000","last_clean":"2024-03-06T07:22:46.685036+0000","last_became_active":"2024-03-06T07:22:46.678080+0000","last_became_peered":"2024-03-06T07:22:46.678080+0000","last_unstale":"2024-03-06T07:22:46.685036+0000","last_undegraded":"2024-03-06T07:22:46.685036+0000","last_fullsized":"2024-03-06T07:22:46.685036+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:02:54.163719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689872+0000","last_change":"2024-03-06T07:22:46.684393+0000","last_active":"2024-03-06T07:22:46.689872+0000","last_peered":"2024-03-06T07:22:46.689872+0000","last_clean":"2024-03-06T07:22:46.689872+0000","last_became_active":"2024-03-06T07:22:46.684120+0000","last_became_peered":"2024-03-06T07:22:46.684120+0000","last_unstale":"2024-03-06T07:22:46.689872+0000","last_undegraded":"2024-03-06T07:22:46.689872+0000","last_fullsized":"2024-03-06T07:22:46.689872+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:44.963260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.693921+0000","last_change":"2024-03-06T07:22:46.688586+0000","last_active":"2024-03-06T07:22:46.693921+0000","last_peered":"2024-03-06T07:22:46.693921+0000","last_clean":"2024-03-06T07:22:46.693921+0000","last_became_active":"2024-03-06T07:22:46.688280+0000","last_became_peered":"2024-03-06T07:22:46.688280+0000","last_unstale":"2024-03-06T07:22:46.693921+0000","last_undegraded":"2024-03-06T07:22:46.693921+0000","last_fullsized":"2024-03-06T07:22:46.693921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:13.159164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.692672+0000","last_change":"2024-03-06T07:22:46.686720+0000","last_active":"2024-03-06T07:22:46.692672+0000","last_peered":"2024-03-06T07:22:46.692672+0000","last_clean":"2024-03-06T07:22:46.692672+0000","last_became_active":"2024-03-06T07:22:46.686418+0000","last_became_peered":"2024-03-06T07:22:46.686418+0000","last_unstale":"2024-03-06T07:22:46.692672+0000","last_undegraded":"2024-03-06T07:22:46.692672+0000","last_fullsized":"2024-03-06T07:22:46.692672+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.790983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699076+0000","last_change":"2024-03-06T07:22:46.694110+0000","last_active":"2024-03-06T07:22:46.699076+0000","last_peered":"2024-03-06T07:22:46.699076+0000","last_clean":"2024-03-06T07:22:46.699076+0000","last_became_active":"2024-03-06T07:22:46.693050+0000","last_became_peered":"2024-03-06T07:22:46.693050+0000","last_unstale":"2024-03-06T07:22:46.699076+0000","last_undegraded":"2024-03-06T07:22:46.699076+0000","last_fullsized":"2024-03-06T07:22:46.699076+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:57.438922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.692677+0000","last_change":"2024-03-06T07:22:46.688246+0000","last_active":"2024-03-06T07:22:46.692677+0000","last_peered":"2024-03-06T07:22:46.692677+0000","last_clean":"2024-03-06T07:22:46.692677+0000","last_became_active":"2024-03-06T07:22:46.687953+0000","last_became_peered":"2024-03-06T07:22:46.687953+0000","last_unstale":"2024-03-06T07:22:46.692677+0000","last_undegraded":"2024-03-06T07:22:46.692677+0000","last_fullsized":"2024-03-06T07:22:46.692677+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.409708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689500+0000","last_change":"2024-03-06T07:22:46.684614+0000","last_active":"2024-03-06T07:22:46.689500+0000","last_peered":"2024-03-06T07:22:46.689500+0000","last_clean":"2024-03-06T07:22:46.689500+0000","last_became_active":"2024-03-06T07:22:46.683962+0000","last_became_peered":"2024-03-06T07:22:46.683962+0000","last_unstale":"2024-03-06T07:22:46.689500+0000","last_undegraded":"2024-03-06T07:22:46.689500+0000","last_fullsized":"2024-03-06T07:22:46.689500+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:38.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689921+0000","last_change":"2024-03-06T07:22:46.683785+0000","last_active":"2024-03-06T07:22:46.689921+0000","last_peered":"2024-03-06T07:22:46.689921+0000","last_clean":"2024-03-06T07:22:46.689921+0000","last_became_active":"2024-03-06T07:22:46.683495+0000","last_became_peered":"2024-03-06T07:22:46.683495+0000","last_unstale":"2024-03-06T07:22:46.689921+0000","last_undegraded":"2024-03-06T07:22:46.689921+0000","last_fullsized":"2024-03-06T07:22:46.689921+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:46.721544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.679787+0000","last_change":"2024-03-06T07:22:46.674124+0000","last_active":"2024-03-06T07:22:46.679787+0000","last_peered":"2024-03-06T07:22:46.679787+0000","last_clean":"2024-03-06T07:22:46.679787+0000","last_became_active":"2024-03-06T07:22:46.673799+0000","last_became_peered":"2024-03-06T07:22:46.673799+0000","last_unstale":"2024-03-06T07:22:46.679787+0000","last_undegraded":"2024-03-06T07:22:46.679787+0000","last_fullsized":"2024-03-06T07:22:46.679787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:11:42.429504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689854+0000","last_change":"2024-03-06T07:22:46.681198+0000","last_active":"2024-03-06T07:22:46.689854+0000","last_peered":"2024-03-06T07:22:46.689854+0000","last_clean":"2024-03-06T07:22:46.689854+0000","last_became_active":"2024-03-06T07:22:46.680952+0000","last_became_peered":"2024-03-06T07:22:46.680952+0000","last_unstale":"2024-03-06T07:22:46.689854+0000","last_undegraded":"2024-03-06T07:22:46.689854+0000","last_fullsized":"2024-03-06T07:22:46.689854+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.790983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689778+0000","last_change":"2024-03-06T07:22:46.683267+0000","last_active":"2024-03-06T07:22:46.689778+0000","last_peered":"2024-03-06T07:22:46.689778+0000","last_clean":"2024-03-06T07:22:46.689778+0000","last_became_active":"2024-03-06T07:22:46.682975+0000","last_became_peered":"2024-03-06T07:22:46.682975+0000","last_unstale":"2024-03-06T07:22:46.689778+0000","last_undegraded":"2024-03-06T07:22:46.689778+0000","last_fullsized":"2024-03-06T07:22:46.689778+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:16.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.695748+0000","last_change":"2024-03-06T07:22:46.689884+0000","last_active":"2024-03-06T07:22:46.695748+0000","last_peered":"2024-03-06T07:22:46.695748+0000","last_clean":"2024-03-06T07:22:46.695748+0000","last_became_active":"2024-03-06T07:22:46.689602+0000","last_became_peered":"2024-03-06T07:22:46.689602+0000","last_unstale":"2024-03-06T07:22:46.695748+0000","last_undegraded":"2024-03-06T07:22:46.695748+0000","last_fullsized":"2024-03-06T07:22:46.695748+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:32.008067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.691837+0000","last_change":"2024-03-06T07:22:46.685250+0000","last_active":"2024-03-06T07:22:46.691837+0000","last_peered":"2024-03-06T07:22:46.691837+0000","last_clean":"2024-03-06T07:22:46.691837+0000","last_became_active":"2024-03-06T07:22:46.683498+0000","last_became_peered":"2024-03-06T07:22:46.683498+0000","last_unstale":"2024-03-06T07:22:46.691837+0000","last_undegraded":"2024-03-06T07:22:46.691837+0000","last_fullsized":"2024-03-06T07:22:46.691837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.954968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696841+0000","last_change":"2024-03-06T07:22:46.691530+0000","last_active":"2024-03-06T07:22:46.696841+0000","last_peered":"2024-03-06T07:22:46.696841+0000","last_clean":"2024-03-06T07:22:46.696841+0000","last_became_active":"2024-03-06T07:22:46.691226+0000","last_became_peered":"2024-03-06T07:22:46.691226+0000","last_unstale":"2024-03-06T07:22:46.696841+0000","last_undegraded":"2024-03-06T07:22:46.696841+0000","last_fullsized":"2024-03-06T07:22:46.696841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:32:27.146574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687841+0000","last_change":"2024-03-06T07:22:46.682935+0000","last_active":"2024-03-06T07:22:46.687841+0000","last_peered":"2024-03-06T07:22:46.687841+0000","last_clean":"2024-03-06T07:22:46.687841+0000","last_became_active":"2024-03-06T07:22:46.682679+0000","last_became_peered":"2024-03-06T07:22:46.682679+0000","last_unstale":"2024-03-06T07:22:46.687841+0000","last_undegraded":"2024-03-06T07:22:46.687841+0000","last_fullsized":"2024-03-06T07:22:46.687841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:32.008067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.684977+0000","last_change":"2024-03-06T07:22:46.679651+0000","last_active":"2024-03-06T07:22:46.684977+0000","last_peered":"2024-03-06T07:22:46.684977+0000","last_clean":"2024-03-06T07:22:46.684977+0000","last_became_active":"2024-03-06T07:22:46.679417+0000","last_became_peered":"2024-03-06T07:22:46.679417+0000","last_unstale":"2024-03-06T07:22:46.684977+0000","last_undegraded":"2024-03-06T07:22:46.684977+0000","last_fullsized":"2024-03-06T07:22:46.684977+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:42.877251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696465+0000","last_change":"2024-03-06T07:22:46.692363+0000","last_active":"2024-03-06T07:22:46.696465+0000","last_peered":"2024-03-06T07:22:46.696465+0000","last_clean":"2024-03-06T07:22:46.696465+0000","last_became_active":"2024-03-06T07:22:46.692086+0000","last_became_peered":"2024-03-06T07:22:46.692086+0000","last_unstale":"2024-03-06T07:22:46.696465+0000","last_undegraded":"2024-03-06T07:22:46.696465+0000","last_fullsized":"2024-03-06T07:22:46.696465+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:31.431049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.701850+0000","last_change":"2024-03-06T07:22:46.697671+0000","last_active":"2024-03-06T07:22:46.701850+0000","last_peered":"2024-03-06T07:22:46.701850+0000","last_clean":"2024-03-06T07:22:46.701850+0000","last_became_active":"2024-03-06T07:22:46.697461+0000","last_became_peered":"2024-03-06T07:22:46.697461+0000","last_unstale":"2024-03-06T07:22:46.701850+0000","last_undegraded":"2024-03-06T07:22:46.701850+0000","last_fullsized":"2024-03-06T07:22:46.701850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:06.331598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685041+0000","last_change":"2024-03-06T07:22:46.681333+0000","last_active":"2024-03-06T07:22:46.685041+0000","last_peered":"2024-03-06T07:22:46.685041+0000","last_clean":"2024-03-06T07:22:46.685041+0000","last_became_active":"2024-03-06T07:22:46.681022+0000","last_became_peered":"2024-03-06T07:22:46.681022+0000","last_unstale":"2024-03-06T07:22:46.685041+0000","last_undegraded":"2024-03-06T07:22:46.685041+0000","last_fullsized":"2024-03-06T07:22:46.685041+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:02.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689767+0000","last_change":"2024-03-06T07:22:46.683613+0000","last_active":"2024-03-06T07:22:46.689767+0000","last_peered":"2024-03-06T07:22:46.689767+0000","last_clean":"2024-03-06T07:22:46.689767+0000","last_became_active":"2024-03-06T07:22:46.683123+0000","last_became_peered":"2024-03-06T07:22:46.683123+0000","last_unstale":"2024-03-06T07:22:46.689767+0000","last_undegraded":"2024-03-06T07:22:46.689767+0000","last_fullsized":"2024-03-06T07:22:46.689767+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:34.954968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689163+0000","last_change":"2024-03-06T07:22:46.684110+0000","last_active":"2024-03-06T07:22:46.689163+0000","last_peered":"2024-03-06T07:22:46.689163+0000","last_clean":"2024-03-06T07:22:46.689163+0000","last_became_active":"2024-03-06T07:22:46.683826+0000","last_became_peered":"2024-03-06T07:22:46.683826+0000","last_unstale":"2024-03-06T07:22:46.689163+0000","last_undegraded":"2024-03-06T07:22:46.689163+0000","last_fullsized":"2024-03-06T07:22:46.689163+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:02.697365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687936+0000","last_change":"2024-03-06T07:22:46.683483+0000","last_active":"2024-03-06T07:22:46.687936+0000","last_peered":"2024-03-06T07:22:46.687936+0000","last_clean":"2024-03-06T07:22:46.687936+0000","last_became_active":"2024-03-06T07:22:46.682294+0000","last_became_peered":"2024-03-06T07:22:46.682294+0000","last_unstale":"2024-03-06T07:22:46.687936+0000","last_undegraded":"2024-03-06T07:22:46.687936+0000","last_fullsized":"2024-03-06T07:22:46.687936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:25.360873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699110+0000","last_change":"2024-03-06T07:22:46.696994+0000","last_active":"2024-03-06T07:22:46.699110+0000","last_peered":"2024-03-06T07:22:46.699110+0000","last_clean":"2024-03-06T07:22:46.699110+0000","last_became_active":"2024-03-06T07:22:46.696539+0000","last_became_peered":"2024-03-06T07:22:46.696539+0000","last_unstale":"2024-03-06T07:22:46.699110+0000","last_undegraded":"2024-03-06T07:22:46.699110+0000","last_fullsized":"2024-03-06T07:22:46.699110+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:38.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685266+0000","last_change":"2024-03-06T07:22:46.681842+0000","last_active":"2024-03-06T07:22:46.685266+0000","last_peered":"2024-03-06T07:22:46.685266+0000","last_clean":"2024-03-06T07:22:46.685266+0000","last_became_active":"2024-03-06T07:22:46.680146+0000","last_became_peered":"2024-03-06T07:22:46.680146+0000","last_unstale":"2024-03-06T07:22:46.685266+0000","last_undegraded":"2024-03-06T07:22:46.685266+0000","last_fullsized":"2024-03-06T07:22:46.685266+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:30.921192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696487+0000","last_change":"2024-03-06T07:22:46.692899+0000","last_active":"2024-03-06T07:22:46.696487+0000","last_peered":"2024-03-06T07:22:46.696487+0000","last_clean":"2024-03-06T07:22:46.696487+0000","last_became_active":"2024-03-06T07:22:46.692634+0000","last_became_peered":"2024-03-06T07:22:46.692634+0000","last_unstale":"2024-03-06T07:22:46.696487+0000","last_undegraded":"2024-03-06T07:22:46.696487+0000","last_fullsized":"2024-03-06T07:22:46.696487+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:13.715103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689356+0000","last_change":"2024-03-06T07:22:46.683599+0000","last_active":"2024-03-06T07:22:46.689356+0000","last_peered":"2024-03-06T07:22:46.689356+0000","last_clean":"2024-03-06T07:22:46.689356+0000","last_became_active":"2024-03-06T07:22:46.683183+0000","last_became_peered":"2024-03-06T07:22:46.683183+0000","last_unstale":"2024-03-06T07:22:46.689356+0000","last_undegraded":"2024-03-06T07:22:46.689356+0000","last_fullsized":"2024-03-06T07:22:46.689356+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:16.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687511+0000","last_change":"2024-03-06T07:22:46.680810+0000","last_active":"2024-03-06T07:22:46.687511+0000","last_peered":"2024-03-06T07:22:46.687511+0000","last_clean":"2024-03-06T07:22:46.687511+0000","last_became_active":"2024-03-06T07:22:46.680513+0000","last_became_peered":"2024-03-06T07:22:46.680513+0000","last_unstale":"2024-03-06T07:22:46.687511+0000","last_undegraded":"2024-03-06T07:22:46.687511+0000","last_fullsized":"2024-03-06T07:22:46.687511+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:13.715103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698696+0000","last_change":"2024-03-06T07:22:46.695418+0000","last_active":"2024-03-06T07:22:46.698696+0000","last_peered":"2024-03-06T07:22:46.698696+0000","last_clean":"2024-03-06T07:22:46.698696+0000","last_became_active":"2024-03-06T07:22:46.695161+0000","last_became_peered":"2024-03-06T07:22:46.695161+0000","last_unstale":"2024-03-06T07:22:46.698696+0000","last_undegraded":"2024-03-06T07:22:46.698696+0000","last_fullsized":"2024-03-06T07:22:46.698696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:44.963260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685375+0000","last_change":"2024-03-06T07:22:46.681692+0000","last_active":"2024-03-06T07:22:46.685375+0000","last_peered":"2024-03-06T07:22:46.685375+0000","last_clean":"2024-03-06T07:22:46.685375+0000","last_became_active":"2024-03-06T07:22:46.681454+0000","last_became_peered":"2024-03-06T07:22:46.681454+0000","last_unstale":"2024-03-06T07:22:46.685375+0000","last_undegraded":"2024-03-06T07:22:46.685375+0000","last_fullsized":"2024-03-06T07:22:46.685375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.285087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699391+0000","last_change":"2024-03-06T07:22:45.683890+0000","last_active":"2024-03-06T07:22:46.699391+0000","last_peered":"2024-03-06T07:22:46.699391+0000","last_clean":"2024-03-06T07:22:46.699391+0000","last_became_active":"2024-03-06T07:22:45.683576+0000","last_became_peered":"2024-03-06T07:22:45.683576+0000","last_unstale":"2024-03-06T07:22:46.699391+0000","last_undegraded":"2024-03-06T07:22:46.699391+0000","last_fullsized":"2024-03-06T07:22:46.699391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:11:30.425903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.697014+0000","last_change":"2024-03-06T07:22:46.693884+0000","last_active":"2024-03-06T07:22:46.697014+0000","last_peered":"2024-03-06T07:22:46.697014+0000","last_clean":"2024-03-06T07:22:46.697014+0000","last_became_active":"2024-03-06T07:22:46.693528+0000","last_became_peered":"2024-03-06T07:22:46.693528+0000","last_unstale":"2024-03-06T07:22:46.697014+0000","last_undegraded":"2024-03-06T07:22:46.697014+0000","last_fullsized":"2024-03-06T07:22:46.697014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:06.331598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.705678+0000","last_change":"2024-03-06T07:22:45.683697+0000","last_active":"2024-03-06T07:22:46.705678+0000","last_peered":"2024-03-06T07:22:46.705678+0000","last_clean":"2024-03-06T07:22:46.705678+0000","last_became_active":"2024-03-06T07:22:45.683386+0000","last_became_peered":"2024-03-06T07:22:45.683386+0000","last_unstale":"2024-03-06T07:22:46.705678+0000","last_undegraded":"2024-03-06T07:22:46.705678+0000","last_fullsized":"2024-03-06T07:22:46.705678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:56.433777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.701855+0000","last_change":"2024-03-06T07:22:46.697693+0000","last_active":"2024-03-06T07:22:46.701855+0000","last_peered":"2024-03-06T07:22:46.701855+0000","last_clean":"2024-03-06T07:22:46.701855+0000","last_became_active":"2024-03-06T07:22:46.697471+0000","last_became_peered":"2024-03-06T07:22:46.697471+0000","last_unstale":"2024-03-06T07:22:46.701855+0000","last_undegraded":"2024-03-06T07:22:46.701855+0000","last_fullsized":"2024-03-06T07:22:46.701855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:08.126249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698635+0000","last_change":"2024-03-06T07:22:45.681427+0000","last_active":"2024-03-06T07:22:46.698635+0000","last_peered":"2024-03-06T07:22:46.698635+0000","last_clean":"2024-03-06T07:22:46.698635+0000","last_became_active":"2024-03-06T07:22:45.681151+0000","last_became_peered":"2024-03-06T07:22:45.681151+0000","last_unstale":"2024-03-06T07:22:46.698635+0000","last_undegraded":"2024-03-06T07:22:46.698635+0000","last_fullsized":"2024-03-06T07:22:46.698635+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:40.404563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698408+0000","last_change":"2024-03-06T07:22:46.695423+0000","last_active":"2024-03-06T07:22:46.698408+0000","last_peered":"2024-03-06T07:22:46.698408+0000","last_clean":"2024-03-06T07:22:46.698408+0000","last_became_active":"2024-03-06T07:22:46.694312+0000","last_became_peered":"2024-03-06T07:22:46.694312+0000","last_unstale":"2024-03-06T07:22:46.698408+0000","last_undegraded":"2024-03-06T07:22:46.698408+0000","last_fullsized":"2024-03-06T07:22:46.698408+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:16.266344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698248+0000","last_change":"2024-03-06T07:22:45.682417+0000","last_active":"2024-03-06T07:22:46.698248+0000","last_peered":"2024-03-06T07:22:46.698248+0000","last_clean":"2024-03-06T07:22:46.698248+0000","last_became_active":"2024-03-06T07:22:45.682127+0000","last_became_peered":"2024-03-06T07:22:45.682127+0000","last_unstale":"2024-03-06T07:22:46.698248+0000","last_undegraded":"2024-03-06T07:22:46.698248+0000","last_fullsized":"2024-03-06T07:22:46.698248+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:15.110242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699145+0000","last_change":"2024-03-06T07:22:46.696484+0000","last_active":"2024-03-06T07:22:46.699145+0000","last_peered":"2024-03-06T07:22:46.699145+0000","last_clean":"2024-03-06T07:22:46.699145+0000","last_became_active":"2024-03-06T07:22:46.696193+0000","last_became_peered":"2024-03-06T07:22:46.696193+0000","last_unstale":"2024-03-06T07:22:46.699145+0000","last_undegraded":"2024-03-06T07:22:46.699145+0000","last_fullsized":"2024-03-06T07:22:46.699145+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.434268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699347+0000","last_change":"2024-03-06T07:22:45.687622+0000","last_active":"2024-03-06T07:22:46.699347+0000","last_peered":"2024-03-06T07:22:46.699347+0000","last_clean":"2024-03-06T07:22:46.699347+0000","last_became_active":"2024-03-06T07:22:45.687347+0000","last_became_peered":"2024-03-06T07:22:45.687347+0000","last_unstale":"2024-03-06T07:22:46.699347+0000","last_undegraded":"2024-03-06T07:22:46.699347+0000","last_fullsized":"2024-03-06T07:22:46.699347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:26.838356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685244+0000","last_change":"2024-03-06T07:22:46.681188+0000","last_active":"2024-03-06T07:22:46.685244+0000","last_peered":"2024-03-06T07:22:46.685244+0000","last_clean":"2024-03-06T07:22:46.685244+0000","last_became_active":"2024-03-06T07:22:46.680900+0000","last_became_peered":"2024-03-06T07:22:46.680900+0000","last_unstale":"2024-03-06T07:22:46.685244+0000","last_undegraded":"2024-03-06T07:22:46.685244+0000","last_fullsized":"2024-03-06T07:22:46.685244+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.409708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699274+0000","last_change":"2024-03-06T07:22:45.687631+0000","last_active":"2024-03-06T07:22:46.699274+0000","last_peered":"2024-03-06T07:22:46.699274+0000","last_clean":"2024-03-06T07:22:46.699274+0000","last_became_active":"2024-03-06T07:22:45.687377+0000","last_became_peered":"2024-03-06T07:22:45.687377+0000","last_unstale":"2024-03-06T07:22:46.699274+0000","last_undegraded":"2024-03-06T07:22:46.699274+0000","last_fullsized":"2024-03-06T07:22:46.699274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:42.446950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696166+0000","last_change":"2024-03-06T07:22:46.692340+0000","last_active":"2024-03-06T07:22:46.696166+0000","last_peered":"2024-03-06T07:22:46.696166+0000","last_clean":"2024-03-06T07:22:46.696166+0000","last_became_active":"2024-03-06T07:22:46.691337+0000","last_became_peered":"2024-03-06T07:22:46.691337+0000","last_unstale":"2024-03-06T07:22:46.696166+0000","last_undegraded":"2024-03-06T07:22:46.696166+0000","last_fullsized":"2024-03-06T07:22:46.696166+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:35.370622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.675009+0000","last_change":"2024-03-06T07:22:45.679514+0000","last_active":"2024-03-06T07:22:46.675009+0000","last_peered":"2024-03-06T07:22:46.675009+0000","last_clean":"2024-03-06T07:22:46.675009+0000","last_became_active":"2024-03-06T07:22:45.679217+0000","last_became_peered":"2024-03-06T07:22:45.679217+0000","last_unstale":"2024-03-06T07:22:46.675009+0000","last_undegraded":"2024-03-06T07:22:46.675009+0000","last_fullsized":"2024-03-06T07:22:46.675009+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:28.279942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.688953+0000","last_change":"2024-03-06T07:22:46.682408+0000","last_active":"2024-03-06T07:22:46.688953+0000","last_peered":"2024-03-06T07:22:46.688953+0000","last_clean":"2024-03-06T07:22:46.688953+0000","last_became_active":"2024-03-06T07:22:46.682152+0000","last_became_peered":"2024-03-06T07:22:46.682152+0000","last_unstale":"2024-03-06T07:22:46.688953+0000","last_undegraded":"2024-03-06T07:22:46.688953+0000","last_fullsized":"2024-03-06T07:22:46.688953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:08.126249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687171+0000","last_change":"2024-03-06T07:22:45.681654+0000","last_active":"2024-03-06T07:22:46.687171+0000","last_peered":"2024-03-06T07:22:46.687171+0000","last_clean":"2024-03-06T07:22:46.687171+0000","last_became_active":"2024-03-06T07:22:45.681349+0000","last_became_peered":"2024-03-06T07:22:45.681349+0000","last_unstale":"2024-03-06T07:22:46.687171+0000","last_undegraded":"2024-03-06T07:22:46.687171+0000","last_fullsized":"2024-03-06T07:22:46.687171+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:29:25.763463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.691450+0000","last_change":"2024-03-06T07:22:46.686873+0000","last_active":"2024-03-06T07:22:46.691450+0000","last_peered":"2024-03-06T07:22:46.691450+0000","last_clean":"2024-03-06T07:22:46.691450+0000","last_became_active":"2024-03-06T07:22:46.686627+0000","last_became_peered":"2024-03-06T07:22:46.686627+0000","last_unstale":"2024-03-06T07:22:46.691450+0000","last_undegraded":"2024-03-06T07:22:46.691450+0000","last_fullsized":"2024-03-06T07:22:46.691450+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:01:13.159164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676514+0000","last_change":"2024-03-06T07:22:45.678280+0000","last_active":"2024-03-06T07:22:46.676514+0000","last_peered":"2024-03-06T07:22:46.676514+0000","last_clean":"2024-03-06T07:22:46.676514+0000","last_became_active":"2024-03-06T07:22:45.677981+0000","last_became_peered":"2024-03-06T07:22:45.677981+0000","last_unstale":"2024-03-06T07:22:46.676514+0000","last_undegraded":"2024-03-06T07:22:46.676514+0000","last_fullsized":"2024-03-06T07:22:46.676514+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:10:15.261198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687506+0000","last_change":"2024-03-06T07:22:46.682322+0000","last_active":"2024-03-06T07:22:46.687506+0000","last_peered":"2024-03-06T07:22:46.687506+0000","last_clean":"2024-03-06T07:22:46.687506+0000","last_became_active":"2024-03-06T07:22:46.682050+0000","last_became_peered":"2024-03-06T07:22:46.682050+0000","last_unstale":"2024-03-06T07:22:46.687506+0000","last_undegraded":"2024-03-06T07:22:46.687506+0000","last_fullsized":"2024-03-06T07:22:46.687506+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.458500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.674183+0000","last_change":"2024-03-06T07:22:45.687492+0000","last_active":"2024-03-06T07:22:46.674183+0000","last_peered":"2024-03-06T07:22:46.674183+0000","last_clean":"2024-03-06T07:22:46.674183+0000","last_became_active":"2024-03-06T07:22:45.687232+0000","last_became_peered":"2024-03-06T07:22:45.687232+0000","last_unstale":"2024-03-06T07:22:46.674183+0000","last_undegraded":"2024-03-06T07:22:46.674183+0000","last_fullsized":"2024-03-06T07:22:46.674183+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:29.251456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696810+0000","last_change":"2024-03-06T07:22:46.690820+0000","last_active":"2024-03-06T07:22:46.696810+0000","last_peered":"2024-03-06T07:22:46.696810+0000","last_clean":"2024-03-06T07:22:46.696810+0000","last_became_active":"2024-03-06T07:22:46.690598+0000","last_became_peered":"2024-03-06T07:22:46.690598+0000","last_unstale":"2024-03-06T07:22:46.696810+0000","last_undegraded":"2024-03-06T07:22:46.696810+0000","last_fullsized":"2024-03-06T07:22:46.696810+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:06.906553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698885+0000","last_change":"2024-03-06T07:22:45.677524+0000","last_active":"2024-03-06T07:22:46.698885+0000","last_peered":"2024-03-06T07:22:46.698885+0000","last_clean":"2024-03-06T07:22:46.698885+0000","last_became_active":"2024-03-06T07:22:45.677212+0000","last_became_peered":"2024-03-06T07:22:45.677212+0000","last_unstale":"2024-03-06T07:22:46.698885+0000","last_undegraded":"2024-03-06T07:22:46.698885+0000","last_fullsized":"2024-03-06T07:22:46.698885+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:01.692220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.690587+0000","last_change":"2024-03-06T07:22:46.684429+0000","last_active":"2024-03-06T07:22:46.690587+0000","last_peered":"2024-03-06T07:22:46.690587+0000","last_clean":"2024-03-06T07:22:46.690587+0000","last_became_active":"2024-03-06T07:22:46.683996+0000","last_became_peered":"2024-03-06T07:22:46.683996+0000","last_unstale":"2024-03-06T07:22:46.690587+0000","last_undegraded":"2024-03-06T07:22:46.690587+0000","last_fullsized":"2024-03-06T07:22:46.690587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:54:34.790983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.708456+0000","last_change":"2024-03-06T07:22:45.682558+0000","last_active":"2024-03-06T07:22:46.708456+0000","last_peered":"2024-03-06T07:22:46.708456+0000","last_clean":"2024-03-06T07:22:46.708456+0000","last_became_active":"2024-03-06T07:22:45.682286+0000","last_became_peered":"2024-03-06T07:22:45.682286+0000","last_unstale":"2024-03-06T07:22:46.708456+0000","last_undegraded":"2024-03-06T07:22:46.708456+0000","last_fullsized":"2024-03-06T07:22:46.708456+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:53:33.949823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.688955+0000","last_change":"2024-03-06T07:22:46.683052+0000","last_active":"2024-03-06T07:22:46.688955+0000","last_peered":"2024-03-06T07:22:46.688955+0000","last_clean":"2024-03-06T07:22:46.688955+0000","last_became_active":"2024-03-06T07:22:46.682804+0000","last_became_peered":"2024-03-06T07:22:46.682804+0000","last_unstale":"2024-03-06T07:22:46.688955+0000","last_undegraded":"2024-03-06T07:22:46.688955+0000","last_fullsized":"2024-03-06T07:22:46.688955+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:29.285087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699339+0000","last_change":"2024-03-06T07:22:45.684990+0000","last_active":"2024-03-06T07:22:46.699339+0000","last_peered":"2024-03-06T07:22:46.699339+0000","last_clean":"2024-03-06T07:22:46.699339+0000","last_became_active":"2024-03-06T07:22:45.684009+0000","last_became_peered":"2024-03-06T07:22:45.684009+0000","last_unstale":"2024-03-06T07:22:46.699339+0000","last_undegraded":"2024-03-06T07:22:46.699339+0000","last_fullsized":"2024-03-06T07:22:46.699339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:48:13.909996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687595+0000","last_change":"2024-03-06T07:22:46.681196+0000","last_active":"2024-03-06T07:22:46.687595+0000","last_peered":"2024-03-06T07:22:46.687595+0000","last_clean":"2024-03-06T07:22:46.687595+0000","last_became_active":"2024-03-06T07:22:46.680311+0000","last_became_peered":"2024-03-06T07:22:46.680311+0000","last_unstale":"2024-03-06T07:22:46.687595+0000","last_undegraded":"2024-03-06T07:22:46.687595+0000","last_fullsized":"2024-03-06T07:22:46.687595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:17:40.434268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.673030+0000","last_change":"2024-03-06T07:22:45.682610+0000","last_active":"2024-03-06T07:22:46.673030+0000","last_peered":"2024-03-06T07:22:46.673030+0000","last_clean":"2024-03-06T07:22:46.673030+0000","last_became_active":"2024-03-06T07:22:45.682308+0000","last_became_peered":"2024-03-06T07:22:45.682308+0000","last_unstale":"2024-03-06T07:22:46.673030+0000","last_undegraded":"2024-03-06T07:22:46.673030+0000","last_fullsized":"2024-03-06T07:22:46.673030+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:29.251456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698675+0000","last_change":"2024-03-06T07:22:46.694848+0000","last_active":"2024-03-06T07:22:46.698675+0000","last_peered":"2024-03-06T07:22:46.698675+0000","last_clean":"2024-03-06T07:22:46.698675+0000","last_became_active":"2024-03-06T07:22:46.694590+0000","last_became_peered":"2024-03-06T07:22:46.694590+0000","last_unstale":"2024-03-06T07:22:46.698675+0000","last_undegraded":"2024-03-06T07:22:46.698675+0000","last_fullsized":"2024-03-06T07:22:46.698675+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:03:25.360873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676584+0000","last_change":"2024-03-06T07:22:45.677534+0000","last_active":"2024-03-06T07:22:46.676584+0000","last_peered":"2024-03-06T07:22:46.676584+0000","last_clean":"2024-03-06T07:22:46.676584+0000","last_became_active":"2024-03-06T07:22:45.677269+0000","last_became_peered":"2024-03-06T07:22:45.677269+0000","last_unstale":"2024-03-06T07:22:46.676584+0000","last_undegraded":"2024-03-06T07:22:46.676584+0000","last_fullsized":"2024-03-06T07:22:46.676584+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:37.016501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696982+0000","last_change":"2024-03-06T07:22:46.693884+0000","last_active":"2024-03-06T07:22:46.696982+0000","last_peered":"2024-03-06T07:22:46.696982+0000","last_clean":"2024-03-06T07:22:46.696982+0000","last_became_active":"2024-03-06T07:22:46.693586+0000","last_became_peered":"2024-03-06T07:22:46.693586+0000","last_unstale":"2024-03-06T07:22:46.696982+0000","last_undegraded":"2024-03-06T07:22:46.696982+0000","last_fullsized":"2024-03-06T07:22:46.696982+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:30.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699698+0000","last_change":"2024-03-06T07:22:45.683364+0000","last_active":"2024-03-06T07:22:46.699698+0000","last_peered":"2024-03-06T07:22:46.699698+0000","last_clean":"2024-03-06T07:22:46.699698+0000","last_became_active":"2024-03-06T07:22:45.682941+0000","last_became_peered":"2024-03-06T07:22:45.682941+0000","last_unstale":"2024-03-06T07:22:46.699698+0000","last_undegraded":"2024-03-06T07:22:46.699698+0000","last_fullsized":"2024-03-06T07:22:46.699698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:01.692220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699147+0000","last_change":"2024-03-06T07:22:46.694776+0000","last_active":"2024-03-06T07:22:46.699147+0000","last_peered":"2024-03-06T07:22:46.699147+0000","last_clean":"2024-03-06T07:22:46.699147+0000","last_became_active":"2024-03-06T07:22:46.694447+0000","last_became_peered":"2024-03-06T07:22:46.694447+0000","last_unstale":"2024-03-06T07:22:46.699147+0000","last_undegraded":"2024-03-06T07:22:46.699147+0000","last_fullsized":"2024-03-06T07:22:46.699147+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:12:30.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.697056+0000","last_change":"2024-03-06T07:22:45.680986+0000","last_active":"2024-03-06T07:22:46.697056+0000","last_peered":"2024-03-06T07:22:46.697056+0000","last_clean":"2024-03-06T07:22:46.697056+0000","last_became_active":"2024-03-06T07:22:45.680691+0000","last_became_peered":"2024-03-06T07:22:45.680691+0000","last_unstale":"2024-03-06T07:22:46.697056+0000","last_undegraded":"2024-03-06T07:22:46.697056+0000","last_fullsized":"2024-03-06T07:22:46.697056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:35.146579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699260+0000","last_change":"2024-03-06T07:22:46.696429+0000","last_active":"2024-03-06T07:22:46.699260+0000","last_peered":"2024-03-06T07:22:46.699260+0000","last_clean":"2024-03-06T07:22:46.699260+0000","last_became_active":"2024-03-06T07:22:46.696151+0000","last_became_peered":"2024-03-06T07:22:46.696151+0000","last_unstale":"2024-03-06T07:22:46.699260+0000","last_undegraded":"2024-03-06T07:22:46.699260+0000","last_fullsized":"2024-03-06T07:22:46.699260+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:05:32.008067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.697409+0000","last_change":"2024-03-06T07:22:45.681681+0000","last_active":"2024-03-06T07:22:46.697409+0000","last_peered":"2024-03-06T07:22:46.697409+0000","last_clean":"2024-03-06T07:22:46.697409+0000","last_became_active":"2024-03-06T07:22:45.681370+0000","last_became_peered":"2024-03-06T07:22:45.681370+0000","last_unstale":"2024-03-06T07:22:46.697409+0000","last_undegraded":"2024-03-06T07:22:46.697409+0000","last_fullsized":"2024-03-06T07:22:46.697409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:11:16.743767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698539+0000","last_change":"2024-03-06T07:22:46.694891+0000","last_active":"2024-03-06T07:22:46.698539+0000","last_peered":"2024-03-06T07:22:46.698539+0000","last_clean":"2024-03-06T07:22:46.698539+0000","last_became_active":"2024-03-06T07:22:46.694581+0000","last_became_peered":"2024-03-06T07:22:46.694581+0000","last_unstale":"2024-03-06T07:22:46.698539+0000","last_undegraded":"2024-03-06T07:22:46.698539+0000","last_fullsized":"2024-03-06T07:22:46.698539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:30.921192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.698280+0000","last_change":"2024-03-06T07:22:45.680736+0000","last_active":"2024-03-06T07:22:46.698280+0000","last_peered":"2024-03-06T07:22:46.698280+0000","last_clean":"2024-03-06T07:22:46.698280+0000","last_became_active":"2024-03-06T07:22:45.680418+0000","last_became_peered":"2024-03-06T07:22:45.680418+0000","last_unstale":"2024-03-06T07:22:46.698280+0000","last_undegraded":"2024-03-06T07:22:46.698280+0000","last_fullsized":"2024-03-06T07:22:46.698280+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:01:19.097331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687818+0000","last_change":"2024-03-06T07:22:46.683628+0000","last_active":"2024-03-06T07:22:46.687818+0000","last_peered":"2024-03-06T07:22:46.687818+0000","last_clean":"2024-03-06T07:22:46.687818+0000","last_became_active":"2024-03-06T07:22:46.683348+0000","last_became_peered":"2024-03-06T07:22:46.683348+0000","last_unstale":"2024-03-06T07:22:46.687818+0000","last_undegraded":"2024-03-06T07:22:46.687818+0000","last_fullsized":"2024-03-06T07:22:46.687818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:54.086368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.688873+0000","last_change":"2024-03-06T07:22:45.687626+0000","last_active":"2024-03-06T07:22:46.688873+0000","last_peered":"2024-03-06T07:22:46.688873+0000","last_clean":"2024-03-06T07:22:46.688873+0000","last_became_active":"2024-03-06T07:22:45.687366+0000","last_became_peered":"2024-03-06T07:22:45.687366+0000","last_unstale":"2024-03-06T07:22:46.688873+0000","last_undegraded":"2024-03-06T07:22:46.688873+0000","last_fullsized":"2024-03-06T07:22:46.688873+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:54.715746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.695751+0000","last_change":"2024-03-06T07:22:46.692567+0000","last_active":"2024-03-06T07:22:46.695751+0000","last_peered":"2024-03-06T07:22:46.695751+0000","last_clean":"2024-03-06T07:22:46.695751+0000","last_became_active":"2024-03-06T07:22:46.692286+0000","last_became_peered":"2024-03-06T07:22:46.692286+0000","last_unstale":"2024-03-06T07:22:46.695751+0000","last_undegraded":"2024-03-06T07:22:46.695751+0000","last_fullsized":"2024-03-06T07:22:46.695751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:27.205868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.705713+0000","last_change":"2024-03-06T07:22:45.685383+0000","last_active":"2024-03-06T07:22:46.705713+0000","last_peered":"2024-03-06T07:22:46.705713+0000","last_clean":"2024-03-06T07:22:46.705713+0000","last_became_active":"2024-03-06T07:22:45.685106+0000","last_became_peered":"2024-03-06T07:22:45.685106+0000","last_unstale":"2024-03-06T07:22:46.705713+0000","last_undegraded":"2024-03-06T07:22:46.705713+0000","last_fullsized":"2024-03-06T07:22:46.705713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:12.709958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685965+0000","last_change":"2024-03-06T07:22:46.681534+0000","last_active":"2024-03-06T07:22:46.685965+0000","last_peered":"2024-03-06T07:22:46.685965+0000","last_clean":"2024-03-06T07:22:46.685965+0000","last_became_active":"2024-03-06T07:22:46.681287+0000","last_became_peered":"2024-03-06T07:22:46.681287+0000","last_unstale":"2024-03-06T07:22:46.685965+0000","last_undegraded":"2024-03-06T07:22:46.685965+0000","last_fullsized":"2024-03-06T07:22:46.685965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:17.464263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676841+0000","last_change":"2024-03-06T07:22:45.683095+0000","last_active":"2024-03-06T07:22:46.676841+0000","last_peered":"2024-03-06T07:22:46.676841+0000","last_clean":"2024-03-06T07:22:46.676841+0000","last_became_active":"2024-03-06T07:22:45.682763+0000","last_became_peered":"2024-03-06T07:22:45.682763+0000","last_unstale":"2024-03-06T07:22:46.676841+0000","last_undegraded":"2024-03-06T07:22:46.676841+0000","last_fullsized":"2024-03-06T07:22:46.676841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:42.446950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685939+0000","last_change":"2024-03-06T07:22:46.680842+0000","last_active":"2024-03-06T07:22:46.685939+0000","last_peered":"2024-03-06T07:22:46.685939+0000","last_clean":"2024-03-06T07:22:46.685939+0000","last_became_active":"2024-03-06T07:22:46.680573+0000","last_became_peered":"2024-03-06T07:22:46.680573+0000","last_unstale":"2024-03-06T07:22:46.685939+0000","last_undegraded":"2024-03-06T07:22:46.685939+0000","last_fullsized":"2024-03-06T07:22:46.685939+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:27.562539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.683858+0000","last_change":"2024-03-06T07:22:45.683932+0000","last_active":"2024-03-06T07:22:46.683858+0000","last_peered":"2024-03-06T07:22:46.683858+0000","last_clean":"2024-03-06T07:22:46.683858+0000","last_became_active":"2024-03-06T07:22:45.683635+0000","last_became_peered":"2024-03-06T07:22:45.683635+0000","last_unstale":"2024-03-06T07:22:46.683858+0000","last_undegraded":"2024-03-06T07:22:46.683858+0000","last_fullsized":"2024-03-06T07:22:46.683858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:53.081223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.693899+0000","last_change":"2024-03-06T07:22:46.687248+0000","last_active":"2024-03-06T07:22:46.693899+0000","last_peered":"2024-03-06T07:22:46.693899+0000","last_clean":"2024-03-06T07:22:46.693899+0000","last_became_active":"2024-03-06T07:22:46.687007+0000","last_became_peered":"2024-03-06T07:22:46.687007+0000","last_unstale":"2024-03-06T07:22:46.693899+0000","last_undegraded":"2024-03-06T07:22:46.693899+0000","last_fullsized":"2024-03-06T07:22:46.693899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:43.458500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.674304+0000","last_change":"2024-03-06T07:22:45.676794+0000","last_active":"2024-03-06T07:22:46.674304+0000","last_peered":"2024-03-06T07:22:46.674304+0000","last_clean":"2024-03-06T07:22:46.674304+0000","last_became_active":"2024-03-06T07:22:45.676537+0000","last_became_peered":"2024-03-06T07:22:45.676537+0000","last_unstale":"2024-03-06T07:22:46.674304+0000","last_undegraded":"2024-03-06T07:22:46.674304+0000","last_fullsized":"2024-03-06T07:22:46.674304+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:28.279942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685551+0000","last_change":"2024-03-06T07:22:46.679013+0000","last_active":"2024-03-06T07:22:46.685551+0000","last_peered":"2024-03-06T07:22:46.685551+0000","last_clean":"2024-03-06T07:22:46.685551+0000","last_became_active":"2024-03-06T07:22:46.678774+0000","last_became_peered":"2024-03-06T07:22:46.678774+0000","last_unstale":"2024-03-06T07:22:46.685551+0000","last_undegraded":"2024-03-06T07:22:46.685551+0000","last_fullsized":"2024-03-06T07:22:46.685551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:20:12.097807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.699233+0000","last_change":"2024-03-06T07:22:45.681308+0000","last_active":"2024-03-06T07:22:46.699233+0000","last_peered":"2024-03-06T07:22:46.699233+0000","last_clean":"2024-03-06T07:22:46.699233+0000","last_became_active":"2024-03-06T07:22:45.680925+0000","last_became_peered":"2024-03-06T07:22:45.680925+0000","last_unstale":"2024-03-06T07:22:46.699233+0000","last_undegraded":"2024-03-06T07:22:46.699233+0000","last_fullsized":"2024-03-06T07:22:46.699233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:56.433777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.689747+0000","last_change":"2024-03-06T07:22:46.683580+0000","last_active":"2024-03-06T07:22:46.689747+0000","last_peered":"2024-03-06T07:22:46.689747+0000","last_clean":"2024-03-06T07:22:46.689747+0000","last_became_active":"2024-03-06T07:22:46.683274+0000","last_became_peered":"2024-03-06T07:22:46.683274+0000","last_unstale":"2024-03-06T07:22:46.689747+0000","last_undegraded":"2024-03-06T07:22:46.689747+0000","last_fullsized":"2024-03-06T07:22:46.689747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:38.021646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.674330+0000","last_change":"2024-03-06T07:22:45.684753+0000","last_active":"2024-03-06T07:22:46.674330+0000","last_peered":"2024-03-06T07:22:46.674330+0000","last_clean":"2024-03-06T07:22:46.674330+0000","last_became_active":"2024-03-06T07:22:45.684453+0000","last_became_peered":"2024-03-06T07:22:45.684453+0000","last_unstale":"2024-03-06T07:22:46.674330+0000","last_undegraded":"2024-03-06T07:22:46.674330+0000","last_fullsized":"2024-03-06T07:22:46.674330+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:05.326453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.686876+0000","last_change":"2024-03-06T07:22:46.682304+0000","last_active":"2024-03-06T07:22:46.686876+0000","last_peered":"2024-03-06T07:22:46.686876+0000","last_clean":"2024-03-06T07:22:46.686876+0000","last_became_active":"2024-03-06T07:22:46.682045+0000","last_became_peered":"2024-03-06T07:22:46.682045+0000","last_unstale":"2024-03-06T07:22:46.686876+0000","last_undegraded":"2024-03-06T07:22:46.686876+0000","last_fullsized":"2024-03-06T07:22:46.686876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:18:41.409708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.672163+0000","last_change":"2024-03-06T07:22:45.681201+0000","last_active":"2024-03-06T07:22:46.672163+0000","last_peered":"2024-03-06T07:22:46.672163+0000","last_clean":"2024-03-06T07:22:46.672163+0000","last_became_active":"2024-03-06T07:22:45.680780+0000","last_became_peered":"2024-03-06T07:22:45.680780+0000","last_unstale":"2024-03-06T07:22:46.672163+0000","last_undegraded":"2024-03-06T07:22:46.672163+0000","last_fullsized":"2024-03-06T07:22:46.672163+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:47:12.709958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.687717+0000","last_change":"2024-03-06T07:22:46.681681+0000","last_active":"2024-03-06T07:22:46.687717+0000","last_peered":"2024-03-06T07:22:46.687717+0000","last_clean":"2024-03-06T07:22:46.687717+0000","last_became_active":"2024-03-06T07:22:46.681457+0000","last_became_peered":"2024-03-06T07:22:46.681457+0000","last_unstale":"2024-03-06T07:22:46.687717+0000","last_undegraded":"2024-03-06T07:22:46.687717+0000","last_fullsized":"2024-03-06T07:22:46.687717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:36.151724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.672592+0000","last_change":"2024-03-06T07:22:45.684589+0000","last_active":"2024-03-06T07:22:46.672592+0000","last_peered":"2024-03-06T07:22:46.672592+0000","last_clean":"2024-03-06T07:22:46.672592+0000","last_became_active":"2024-03-06T07:22:45.684342+0000","last_became_peered":"2024-03-06T07:22:45.684342+0000","last_unstale":"2024-03-06T07:22:46.672592+0000","last_undegraded":"2024-03-06T07:22:46.672592+0000","last_fullsized":"2024-03-06T07:22:46.672592+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:35.146579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685085+0000","last_change":"2024-03-06T07:22:46.680473+0000","last_active":"2024-03-06T07:22:46.685085+0000","last_peered":"2024-03-06T07:22:46.685085+0000","last_clean":"2024-03-06T07:22:46.685085+0000","last_became_active":"2024-03-06T07:22:46.680214+0000","last_became_peered":"2024-03-06T07:22:46.680214+0000","last_unstale":"2024-03-06T07:22:46.685085+0000","last_undegraded":"2024-03-06T07:22:46.685085+0000","last_fullsized":"2024-03-06T07:22:46.685085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:08.126249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.673177+0000","last_change":"2024-03-06T07:22:45.681955+0000","last_active":"2024-03-06T07:22:46.673177+0000","last_peered":"2024-03-06T07:22:46.673177+0000","last_clean":"2024-03-06T07:22:46.673177+0000","last_became_active":"2024-03-06T07:22:45.681577+0000","last_became_peered":"2024-03-06T07:22:45.681577+0000","last_unstale":"2024-03-06T07:22:46.673177+0000","last_undegraded":"2024-03-06T07:22:46.673177+0000","last_fullsized":"2024-03-06T07:22:46.673177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:59:05.326453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.685916+0000","last_change":"2024-03-06T07:22:46.680739+0000","last_active":"2024-03-06T07:22:46.685916+0000","last_peered":"2024-03-06T07:22:46.685916+0000","last_clean":"2024-03-06T07:22:46.685916+0000","last_became_active":"2024-03-06T07:22:46.680286+0000","last_became_peered":"2024-03-06T07:22:46.680286+0000","last_unstale":"2024-03-06T07:22:46.685916+0000","last_undegraded":"2024-03-06T07:22:46.685916+0000","last_fullsized":"2024-03-06T07:22:46.685916+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:46.721544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.684035+0000","last_change":"2024-03-06T07:22:45.683758+0000","last_active":"2024-03-06T07:22:46.684035+0000","last_peered":"2024-03-06T07:22:46.684035+0000","last_clean":"2024-03-06T07:22:46.684035+0000","last_became_active":"2024-03-06T07:22:45.683497+0000","last_became_peered":"2024-03-06T07:22:45.683497+0000","last_unstale":"2024-03-06T07:22:46.684035+0000","last_undegraded":"2024-03-06T07:22:46.684035+0000","last_fullsized":"2024-03-06T07:22:46.684035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:50:26.200722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.696201+0000","last_change":"2024-03-06T07:22:46.693567+0000","last_active":"2024-03-06T07:22:46.696201+0000","last_peered":"2024-03-06T07:22:46.696201+0000","last_clean":"2024-03-06T07:22:46.696201+0000","last_became_active":"2024-03-06T07:22:46.693315+0000","last_became_peered":"2024-03-06T07:22:46.693315+0000","last_unstale":"2024-03-06T07:22:46.696201+0000","last_undegraded":"2024-03-06T07:22:46.696201+0000","last_fullsized":"2024-03-06T07:22:46.696201+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:36.151724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.675663+0000","last_change":"2024-03-06T07:22:45.678345+0000","last_active":"2024-03-06T07:22:46.675663+0000","last_peered":"2024-03-06T07:22:46.675663+0000","last_clean":"2024-03-06T07:22:46.675663+0000","last_became_active":"2024-03-06T07:22:45.678073+0000","last_became_peered":"2024-03-06T07:22:45.678073+0000","last_unstale":"2024-03-06T07:22:46.675663+0000","last_undegraded":"2024-03-06T07:22:46.675663+0000","last_fullsized":"2024-03-06T07:22:46.675663+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:29.916047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.692585+0000","last_change":"2024-03-06T07:22:46.686443+0000","last_active":"2024-03-06T07:22:46.692585+0000","last_peered":"2024-03-06T07:22:46.692585+0000","last_clean":"2024-03-06T07:22:46.692585+0000","last_became_active":"2024-03-06T07:22:46.686169+0000","last_became_peered":"2024-03-06T07:22:46.686169+0000","last_unstale":"2024-03-06T07:22:46.692585+0000","last_undegraded":"2024-03-06T07:22:46.692585+0000","last_fullsized":"2024-03-06T07:22:46.692585+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:45.660167+0000","last_clean_scrub_stamp":"2024-03-06T07:22:45.660167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:46.721544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.676909+0000","last_change":"2024-03-06T07:22:45.684161+0000","last_active":"2024-03-06T07:22:46.676909+0000","last_peered":"2024-03-06T07:22:46.676909+0000","last_clean":"2024-03-06T07:22:46.676909+0000","last_became_active":"2024-03-06T07:22:45.683873+0000","last_became_peered":"2024-03-06T07:22:45.683873+0000","last_unstale":"2024-03-06T07:22:46.676909+0000","last_undegraded":"2024-03-06T07:22:46.676909+0000","last_fullsized":"2024-03-06T07:22:46.676909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:22:44.655022+0000","last_clean_scrub_stamp":"2024-03-06T07:22:44.655022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:55:34.365476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-06T07:22:46.677839+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:22:46.677839+0000","last_peered":"2024-03-06T07:22:46.677839+0000","last_clean":"2024-03-06T07:22:46.677839+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:22:46.677839+0000","last_undegraded":"2024-03-06T07:22:46.677839+0000","last_fullsized":"2024-03-06T07:22:46.677839+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":954368,"data_stored":728572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":145651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":380928,"data_stored":145083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":138204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":954368,"data_stored":728572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:22:50.035 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:22:50.035 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:22:50.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.178+0000 7f1314fc7640 1 Processor -- start 2024-03-06T07:22:50.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.178+0000 7f1314fc7640 1 -- start start 2024-03-06T07:22:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.179+0000 7f1314fc7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13101614d0 0x7f13100d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.179+0000 7f1314fc7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f13100d24b0 0x7f13100b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.179+0000 7f1314fc7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13101618b0 con 0x7f13100d1910 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.179+0000 7f1314fc7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13100cf400 con 0x7f13100d24b0 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.179+0000 7f1314fc7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13100cf6a0 con 0x7f13101614d0 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f13100d24b0 0x7f13100b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f1307fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f13100d1910 0x7f13100d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41126/0 (socket says 172.21.15.72:41126) 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f13100d24b0 0x7f13100b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33860/0 (socket says 172.21.15.72:33860) 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 -- 172.21.15.72:0/403666047 learned_addr learned my addr 172.21.15.72:0/403666047 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130e59c640 1 --2- 172.21.15.72:0/403666047 >> v2:172.21.15.146:3300/0 conn(0x7f13101614d0 0x7f13100d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 -- 172.21.15.72:0/403666047 >> v2:172.21.15.146:3300/0 conn(0x7f13101614d0 msgr2=0x7f13100d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 --2- 172.21.15.72:0/403666047 >> v2:172.21.15.146:3300/0 conn(0x7f13101614d0 0x7f13100d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 -- 172.21.15.72:0/403666047 >> v1:172.21.15.72:6789/0 conn(0x7f13100d1910 legacy=0x7f13100d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130dd9b640 1 -- 172.21.15.72:0/403666047 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476750414 0 0) 0x7f13101618b0 con 0x7f13100d1910 2024-03-06T07:22:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.180+0000 7f130ed9d640 1 -- 172.21.15.72:0/403666047 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13100cf980 con 0x7f13100d24b0 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.181+0000 7f130ed9d640 1 --2- 172.21.15.72:0/403666047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13100d24b0 0x7f13100b0e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f131008c470 tx=0x7f130802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbacdbc2f495b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.181+0000 7f130dd9b640 1 -- 172.21.15.72:0/403666047 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130802cb20 con 0x7f13100d24b0 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.181+0000 7f130dd9b640 1 -- 172.21.15.72:0/403666047 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130802ccc0 con 0x7f13100d24b0 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.181+0000 7f130dd9b640 1 -- 172.21.15.72:0/403666047 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308034610 con 0x7f13100d24b0 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 -- 172.21.15.72:0/403666047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13100d24b0 msgr2=0x7f13100b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 --2- 172.21.15.72:0/403666047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13100d24b0 0x7f13100b0e30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f131008c470 tx=0x7f130802bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 -- 172.21.15.72:0/403666047 shutdown_connections 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 --2- 172.21.15.72:0/403666047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13100d24b0 0x7f13100b0e30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 --2- 172.21.15.72:0/403666047 >> v2:172.21.15.146:3300/0 conn(0x7f13101614d0 0x7f13100d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 -- 172.21.15.72:0/403666047 >> 172.21.15.72:0/403666047 conn(0x7f13100d3d40 msgr2=0x7f13100d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 -- 172.21.15.72:0/403666047 shutdown_connections 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.182+0000 7f1314fc7640 1 -- 172.21.15.72:0/403666047 wait complete. 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.183+0000 7f1314fc7640 1 Processor -- start 2024-03-06T07:22:50.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.183+0000 7f1314fc7640 1 -- start start 2024-03-06T07:22:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1314fc7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13100d1910 0x7f13101558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1314fc7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13100d24b0 0x7f13100b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1314fc7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 0x7f13100b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1314fc7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13100b1370 con 0x7f13101614d0 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1314fc7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13100b0910 con 0x7f13100d24b0 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1314fc7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13100b14f0 con 0x7f13100d1910 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 0x7f13100b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 0x7f13100b6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33868/0 (socket says 172.21.15.72:33868) 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 -- 172.21.15.72:0/4276541430 learned_addr learned my addr 172.21.15.72:0/4276541430 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f1307fff640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13100d24b0 0x7f13100b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130e59c640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13100d1910 0x7f13101558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 -- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13100d1910 msgr2=0x7f13101558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13100d1910 0x7f13101558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 -- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13100d24b0 msgr2=0x7f13100b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13100d24b0 0x7f13100b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 -- 172.21.15.72:0/4276541430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13100c85d0 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.184+0000 7f130ed9d640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 0x7f13100b6e70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1308036040 tx=0x7f1308004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.185+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308043030 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.185+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f130803e450 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.185+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13100b74a0 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.185+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1308009510 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.185+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13100b36e0 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.186+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13100b99f0 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.186+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1308046070 con 0x7f13101614d0 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.186+0000 7f1306ffd640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f12f006b4d0 0x7f12f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.187+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13080ac4e0 con 0x7f13101614d0 2024-03-06T07:22:50.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.187+0000 7f130e59c640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f12f006b4d0 0x7f12f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.187+0000 7f130e59c640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f12f006b4d0 0x7f12f006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13000079a0 tx=0x7f1300008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.191+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f130804d070 con 0x7f13101614d0 2024-03-06T07:22:50.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.436+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f1310068ff0 con 0x7f13101614d0 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.438+0000 7f1306ffd640 1 -- 172.21.15.72:0/4276541430 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f13100c85d0 con 0x7f13101614d0 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:22:50.440 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.440+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f12f006b4d0 msgr2=0x7f12f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.440+0000 7f1314fc7640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f12f006b4d0 0x7f12f006d990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f13000079a0 tx=0x7f1300008040 comp rx=0 tx=0).stop 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 msgr2=0x7f13100b6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 0x7f13100b6e70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f1308036040 tx=0x7f1308004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 shutdown_connections 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f12f006b4d0 0x7f12f006d990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13101614d0 0x7f13100b6e70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13100d24b0 0x7f13100b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 --2- 172.21.15.72:0/4276541430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13100d1910 0x7f13101558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 >> 172.21.15.72:0/4276541430 conn(0x7f13100d3d40 msgr2=0x7f13100d1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 shutdown_connections 2024-03-06T07:22:50.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.441+0000 7f1314fc7640 1 -- 172.21.15.72:0/4276541430 wait complete. 2024-03-06T07:22:50.454 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.555+0000 7fdf30379640 1 Processor -- start 2024-03-06T07:22:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.555+0000 7fdf30379640 1 -- start start 2024-03-06T07:22:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf30379640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d2710 0x7fdf280b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf30379640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf280b3430 con 0x7fdf2815dfe0 2024-03-06T07:22:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf30379640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf280b3710 con 0x7fdf280d22b0 2024-03-06T07:22:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf30379640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280b31e0 con 0x7fdf280d2710 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2e115640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdf2815dfe0 0x7fdf280cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41138/0 (socket says 172.21.15.72:41138) 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2e115640 1 -- 172.21.15.72:0/32337754 learned_addr learned my addr 172.21.15.72:0/32337754 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2e916640 1 --2- 172.21.15.72:0/32337754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d2710 0x7fdf280b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101259364 0 0) 0x7fdf280b3430 con 0x7fdf2815dfe0 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf14003520 con 0x7fdf2815dfe0 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581523105 0 0) 0x7fdf280b3710 con 0x7fdf280d22b0 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.556+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf280b3430 con 0x7fdf280d22b0 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.557+0000 7fdf2e916640 1 -- 172.21.15.72:0/32337754 >> v1:172.21.15.146:6789/0 conn(0x7fdf280d22b0 legacy=0x7fdf280b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.557+0000 7fdf2e916640 1 -- 172.21.15.72:0/32337754 >> v1:172.21.15.72:6789/0 conn(0x7fdf2815dfe0 legacy=0x7fdf280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.557+0000 7fdf2e916640 1 -- 172.21.15.72:0/32337754 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf280b39f0 con 0x7fdf280d2710 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.557+0000 7fdf2e916640 1 --2- 172.21.15.72:0/32337754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d2710 0x7fdf280b4900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdf280b3360 tx=0x7fdf2002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2920e98dbb0798e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.557+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20039070 con 0x7fdf280d2710 2024-03-06T07:22:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2002ccd0 con 0x7fdf280d2710 2024-03-06T07:22:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf2d113640 1 -- 172.21.15.72:0/32337754 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20034640 con 0x7fdf280d2710 2024-03-06T07:22:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 -- 172.21.15.72:0/32337754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d2710 msgr2=0x7fdf280b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 --2- 172.21.15.72:0/32337754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d2710 0x7fdf280b4900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdf280b3360 tx=0x7fdf2002bf20 comp rx=0 tx=0).stop 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 -- 172.21.15.72:0/32337754 shutdown_connections 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 --2- 172.21.15.72:0/32337754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d2710 0x7fdf280b4900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 -- 172.21.15.72:0/32337754 >> 172.21.15.72:0/32337754 conn(0x7fdf280d3cd0 msgr2=0x7fdf280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 -- 172.21.15.72:0/32337754 shutdown_connections 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.558+0000 7fdf30379640 1 -- 172.21.15.72:0/32337754 wait complete. 2024-03-06T07:22:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.560+0000 7fdf30379640 1 Processor -- start 2024-03-06T07:22:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.560+0000 7fdf30379640 1 -- start start 2024-03-06T07:22:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf30379640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d22b0 0x7fdf2815c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf30379640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf280d2710 0x7fdf280ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf30379640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 0x7fdf280cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf30379640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280cfa00 con 0x7fdf2815dfe0 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf30379640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280b3d60 con 0x7fdf280d2710 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf30379640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280d1c70 con 0x7fdf280d22b0 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf2e916640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 0x7fdf280cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.561+0000 7fdf2e916640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 0x7fdf280cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33880/0 (socket says 172.21.15.72:33880) 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e115640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d22b0 0x7fdf2815c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 -- 172.21.15.72:0/3859371400 learned_addr learned my addr 172.21.15.72:0/3859371400 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2d914640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf280d2710 0x7fdf280ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 -- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d22b0 msgr2=0x7fdf2815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d22b0 0x7fdf2815c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 -- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf280d2710 msgr2=0x7fdf280ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf280d2710 0x7fdf280ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 -- 172.21.15.72:0/3859371400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf281f7100 con 0x7fdf2815dfe0 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf2e916640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 0x7fdf280cd4b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009eb0 tx=0x7fdf20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20039050 con 0x7fdf2815dfe0 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2003a040 con 0x7fdf2815dfe0 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf281f73e0 con 0x7fdf2815dfe0 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20009450 con 0x7fdf2815dfe0 2024-03-06T07:22:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.562+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf281f79a0 con 0x7fdf2815dfe0 2024-03-06T07:22:50.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.563+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2804f330 con 0x7fdf2815dfe0 2024-03-06T07:22:50.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.564+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf20047020 con 0x7fdf2815dfe0 2024-03-06T07:22:50.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.564+0000 7fdf1effd640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf0c06b500 0x7fdf0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:50.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.564+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf200ac0f0 con 0x7fdf2815dfe0 2024-03-06T07:22:50.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.564+0000 7fdf2e115640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf0c06b500 0x7fdf0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:50.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.564+0000 7fdf2e115640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf0c06b500 0x7fdf0c06d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdf18000f60 tx=0x7fdf18007d70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:50.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.568+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf2004e020 con 0x7fdf2815dfe0 2024-03-06T07:22:50.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.828+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf280b4850 con 0x7fdf2815dfe0 2024-03-06T07:22:50.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.828+0000 7fdf1effd640 1 -- 172.21.15.72:0/3859371400 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fdf2007a920 con 0x7fdf2815dfe0 2024-03-06T07:22:50.830 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:50.831 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:22:50.831 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf0c06b500 msgr2=0x7fdf0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf0c06b500 0x7fdf0c06d9c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdf18000f60 tx=0x7fdf18007d70 comp rx=0 tx=0).stop 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 msgr2=0x7fdf280cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 0x7fdf280cd4b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdf20009eb0 tx=0x7fdf20004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 shutdown_connections 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf0c06b500 0x7fdf0c06d9c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2815dfe0 0x7fdf280cd4b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf280d2710 0x7fdf280ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 --2- 172.21.15.72:0/3859371400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf280d22b0 0x7fdf2815c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 >> 172.21.15.72:0/3859371400 conn(0x7fdf280d3cd0 msgr2=0x7fdf280b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 shutdown_connections 2024-03-06T07:22:50.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:50.830+0000 7fdf30379640 1 -- 172.21.15.72:0/3859371400 wait complete. 2024-03-06T07:22:51.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:22:46.019808+0000', 'modified': '2024-03-06T07:22:48.025103+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5196}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5196': {'gid': 5196, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1659735808', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1659735808}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1659735808}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5196, 'qdb_cluster': [5196]} 2024-03-06T07:22:51.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:22:51.249 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:22:51.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.350+0000 7f7be3281640 1 Processor -- start 2024-03-06T07:22:51.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.350+0000 7f7be3281640 1 -- start start 2024-03-06T07:22:51.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be3281640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7bdc06d3c0 0x7f7bdc0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be3281640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bdc0b1850 0x7f7bdc0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be3281640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bdc0af4b0 con 0x7f7bdc1614a0 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be3281640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc0af790 con 0x7f7bdc0b1850 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be3281640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc0afa70 con 0x7f7bdc06d3c0 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bdc0b1850 0x7f7bdc0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bdc0b1850 0x7f7bdc0b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33888/0 (socket says 172.21.15.72:33888) 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be101d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7bdc1614a0 0x7f7bdc0b0270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41152/0 (socket says 172.21.15.72:41152) 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 -- 172.21.15.72:0/442531782 learned_addr learned my addr 172.21.15.72:0/442531782 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 -- 172.21.15.72:0/442531782 >> v2:172.21.15.146:3300/0 conn(0x7f7bdc06d3c0 msgr2=0x7f7bdc0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 --2- 172.21.15.72:0/442531782 >> v2:172.21.15.146:3300/0 conn(0x7f7bdc06d3c0 0x7f7bdc0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 -- 172.21.15.72:0/442531782 >> v1:172.21.15.72:6789/0 conn(0x7f7bdc1614a0 legacy=0x7f7bdc0b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.351+0000 7f7be181e640 1 -- 172.21.15.72:0/442531782 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0afd50 con 0x7f7bdc0b1850 2024-03-06T07:22:51.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.352+0000 7f7be181e640 1 --2- 172.21.15.72:0/442531782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc0b1850 0x7f7bdc0b5510 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc06bd30 tx=0x7f7bd402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2bcaa4815afcb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.352+0000 7f7bd3fff640 1 -- 172.21.15.72:0/442531782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd402cb20 con 0x7f7bdc0b1850 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.352+0000 7f7bd3fff640 1 -- 172.21.15.72:0/442531782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd402ccc0 con 0x7f7bdc0b1850 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.352+0000 7f7bd3fff640 1 -- 172.21.15.72:0/442531782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4034610 con 0x7f7bdc0b1850 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 -- 172.21.15.72:0/442531782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc0b1850 msgr2=0x7f7bdc0b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 --2- 172.21.15.72:0/442531782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc0b1850 0x7f7bdc0b5510 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc06bd30 tx=0x7f7bd402bf80 comp rx=0 tx=0).stop 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 -- 172.21.15.72:0/442531782 shutdown_connections 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 --2- 172.21.15.72:0/442531782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc0b1850 0x7f7bdc0b5510 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 --2- 172.21.15.72:0/442531782 >> v2:172.21.15.146:3300/0 conn(0x7f7bdc06d3c0 0x7f7bdc0b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 -- 172.21.15.72:0/442531782 >> 172.21.15.72:0/442531782 conn(0x7f7bdc09a9f0 msgr2=0x7f7bdc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 -- 172.21.15.72:0/442531782 shutdown_connections 2024-03-06T07:22:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.353+0000 7f7be3281640 1 -- 172.21.15.72:0/442531782 wait complete. 2024-03-06T07:22:51.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.354+0000 7f7be3281640 1 Processor -- start 2024-03-06T07:22:51.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.354+0000 7f7be3281640 1 -- start start 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be3281640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bdc06d3c0 0x7f7bdc0b04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be3281640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bdc0b1850 0x7f7bdc15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be3281640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 0x7f7bdc14d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be3281640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0b1db0 con 0x7f7bdc1614a0 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be3281640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0b1c30 con 0x7f7bdc06d3c0 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be3281640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0b1f30 con 0x7f7bdc0b1850 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 0x7f7bdc14d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 0x7f7bdc14d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33894/0 (socket says 172.21.15.72:33894) 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 -- 172.21.15.72:0/2022125581 learned_addr learned my addr 172.21.15.72:0/2022125581 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be101d640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bdc06d3c0 0x7f7bdc0b04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be081c640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bdc0b1850 0x7f7bdc15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 -- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bdc0b1850 msgr2=0x7f7bdc15bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bdc0b1850 0x7f7bdc15bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 -- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bdc06d3c0 msgr2=0x7f7bdc0b04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bdc06d3c0 0x7f7bdc0b04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 -- 172.21.15.72:0/2022125581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc15c9b0 con 0x7f7bdc1614a0 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.355+0000 7f7be181e640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 0x7f7bdc14d180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036040 tx=0x7f7bd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.356+0000 7f7bd1ffb640 1 -- 172.21.15.72:0/2022125581 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4043020 con 0x7f7bdc1614a0 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.356+0000 7f7bd1ffb640 1 -- 172.21.15.72:0/2022125581 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd403e450 con 0x7f7bdc1614a0 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.356+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc15cc90 con 0x7f7bdc1614a0 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.356+0000 7f7bd1ffb640 1 -- 172.21.15.72:0/2022125581 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd403e820 con 0x7f7bdc1614a0 2024-03-06T07:22:51.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.356+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc15d250 con 0x7f7bdc1614a0 2024-03-06T07:22:51.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.357+0000 7f7bd1ffb640 1 -- 172.21.15.72:0/2022125581 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bd4046020 con 0x7f7bdc1614a0 2024-03-06T07:22:51.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.358+0000 7f7bd1ffb640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7bc006b4d0 0x7f7bc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.358+0000 7f7bd1ffb640 1 -- 172.21.15.72:0/2022125581 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bd40acd10 con 0x7f7bdc1614a0 2024-03-06T07:22:51.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.358+0000 7f7be101d640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7bc006b4d0 0x7f7bc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.358+0000 7f7be101d640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7bc006b4d0 0x7f7bc006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8007fd0 tx=0x7f7bd8007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.360+0000 7f7be5e62740 1 Processor -- start 2024-03-06T07:22:51.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.360+0000 7f7be5e62740 1 -- start start 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 0x556e00872c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008768c0 0x556e00878360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be5e62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e007dc6b0 con 0x556e00872870 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e00785a60 con 0x556e008768c0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e007d8750 con 0x556e008a2600 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 0x556e00872c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 0x556e00872c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33900/0 (socket says 172.21.15.72:33900) 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 -- 172.21.15.72:0/2893985786 learned_addr learned my addr 172.21.15.72:0/2893985786 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be181e640 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be081c640 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008768c0 0x556e00878360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 -- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 msgr2=0x556e008a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 -- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008768c0 msgr2=0x556e00878360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008768c0 0x556e00878360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.361+0000 7f7be101d640 1 -- 172.21.15.72:0/2893985786 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e007a3f30 con 0x556e00872870 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.362+0000 7f7be101d640 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 0x556e00872c50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8004bf0 tx=0x7f7bd8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b36a33c935426ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.362+0000 7f7bc6ffd640 1 -- 172.21.15.72:0/2893985786 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8026070 con 0x556e00872870 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.362+0000 7f7bc6ffd640 1 -- 172.21.15.72:0/2893985786 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8013b80 con 0x556e00872870 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.362+0000 7f7bc6ffd640 1 -- 172.21.15.72:0/2893985786 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8001d40 con 0x556e00872870 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.362+0000 7f7be5e62740 1 -- 172.21.15.72:0/2893985786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 msgr2=0x556e00872c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 0x556e00872c50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8004bf0 tx=0x7f7bd8013000 comp rx=0 tx=0).stop 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 -- 172.21.15.72:0/2893985786 shutdown_connections 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008768c0 0x556e00878360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2893985786 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e00872870 0x556e00872c50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 -- 172.21.15.72:0/2893985786 >> 172.21.15.72:0/2893985786 conn(0x556e008748f0 msgr2=0x556e008761b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 -- 172.21.15.72:0/2893985786 shutdown_connections 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 -- 172.21.15.72:0/2893985786 wait complete. 2024-03-06T07:22:51.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 -- start start 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.363+0000 7f7be5e62740 1 Processor -- start 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e0089d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e008bd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e00876d50 con 0x556e008a2600 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e00798ea0 con 0x556e008bce50 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e00877030 con 0x556e008a3770 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be5e62740 10 client.0 _ll_register_callbacks cb 0x556e00798870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be101d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e0089d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be101d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e0089d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33902/0 (socket says 172.21.15.72:33902) 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.364+0000 7f7be101d640 1 -- 172.21.15.72:0/4276843645 learned_addr learned my addr 172.21.15.72:0/4276843645 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be081c640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be181e640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e008bd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be101d640 1 -- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 msgr2=0x556e008a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be101d640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be101d640 1 -- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 msgr2=0x556e008bd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be101d640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e008bd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be101d640 1 -- 172.21.15.72:0/4276843645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e00799270 con 0x556e008a2600 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be101d640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e0089d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x556e008a6cf0 tx=0x7f7bd8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9bcf8a344162c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7bbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7bbbfff640 1 -- 172.21.15.72:0/4276843645 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8026040 con 0x556e008a2600 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7bbbfff640 1 -- 172.21.15.72:0/4276843645 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bd8013d30 con 0x556e008a2600 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bd8013b80 con 0x556e008a2600 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7bbbfff640 1 -- 172.21.15.72:0/4276843645 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bd8038050 con 0x556e008a2600 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 10 client.5379 fetch_fsmap learned FSMap version 14 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x556e004f12d0 con 0x556e008a2600 2024-03-06T07:22:51.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7bbbfff640 1 -- 172.21.15.72:0/4276843645 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8013b80 con 0x556e008a2600 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7bbbfff640 1 -- 172.21.15.72:0/4276843645 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 14) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7bd803b020 con 0x556e008a2600 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 10 client.5379 fetch_fsmap finished waiting for FSMap version 14 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 10 client.5379 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 20 client.5379 populate_metadata read hostname 'smithi072' 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e008d9f90 0x556e008dc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 4 client.5379 mds_command: new command op to 5196 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.365+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 --> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e007e9bf0 con 0x556e008d9f90 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.366+0000 7f7be081c640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e008d9f90 0x556e008dc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.366+0000 7f7bbbfff640 10 client.5379 ms_handle_connect on v2:172.21.15.72:6835/1659735808 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.366+0000 7f7be081c640 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e008d9f90 0x556e008dc380 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8722178e81eb10d9 server_cookie=5f4d623fa5a68edb in_seq=0 out_seq=0 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.366+0000 7f7bbbfff640 1 -- 172.21.15.72:0/4276843645 <== mds.0 v2:172.21.15.72:6835/1659735808 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556e007e9bf0 con 0x556e008d9f90 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.366+0000 7f7bbbfff640 10 client.5379 handle_command_reply: tid=0 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.366+0000 7f7be5e62740 1 client.5379 shutdown 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 msgr2=0x556e0089d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e0089d990 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x556e008a6cf0 tx=0x7f7bd8000bc0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 shutdown_connections 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e008d9f90 0x556e008dc380 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e008bd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7bbbfff640 0 client.5379 ms_handle_reset on v2:172.21.15.72:6835/1659735808 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 --2- 172.21.15.72:0/4276843645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e0089d990 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 >> 172.21.15.72:0/4276843645 conn(0x556e00872870 msgr2=0x556e008a05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 shutdown_connections 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 1 -- 172.21.15.72:0/4276843645 wait complete. 2024-03-06T07:22:51.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.367+0000 7f7be5e62740 20 client.5379 trim_cache size 0 max 16384 2024-03-06T07:22:51.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.383+0000 7f7be5e62740 1 Processor -- start 2024-03-06T07:22:51.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.383+0000 7f7be5e62740 1 -- start start 2024-03-06T07:22:51.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e008e15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e00924fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be5e62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1ff2f0 con 0x556e008a2600 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc15eef0 con 0x556e008bce50 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1fed20 con 0x556e008a3770 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e008e15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e008e15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33910/0 (socket says 172.21.15.72:33910) 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 -- 172.21.15.72:0/2771846153 learned_addr learned my addr 172.21.15.72:0/2771846153 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be081c640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be181e640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e00924fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 -- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 msgr2=0x556e008a4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 -- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 msgr2=0x556e00924fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e00924fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 -- 172.21.15.72:0/2771846153 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e008bd600 con 0x556e008a2600 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be081c640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be181e640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e00924fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7be101d640 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e008e15c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8022800 tx=0x7f7bd802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1637a2fce6e9c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7bc67fc640 1 -- 172.21.15.72:0/2771846153 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8026070 con 0x556e008a2600 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7bc67fc640 1 -- 172.21.15.72:0/2771846153 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8030070 con 0x556e008a2600 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.384+0000 7f7bc67fc640 1 -- 172.21.15.72:0/2771846153 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd8022280 con 0x556e008a2600 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 -- 172.21.15.72:0/2771846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 msgr2=0x556e008e15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e008e15c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8022800 tx=0x7f7bd802a000 comp rx=0 tx=0).stop 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 -- 172.21.15.72:0/2771846153 shutdown_connections 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008bce50 0x556e00924fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a3770 0x556e008a4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2771846153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008a2600 0x556e008e15c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 -- 172.21.15.72:0/2771846153 >> 172.21.15.72:0/2771846153 conn(0x556e00872870 msgr2=0x556e008a3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 -- 172.21.15.72:0/2771846153 shutdown_connections 2024-03-06T07:22:51.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.385+0000 7f7be5e62740 1 -- 172.21.15.72:0/2771846153 wait complete. 2024-03-06T07:22:51.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.386+0000 7f7be5e62740 1 -- start start 2024-03-06T07:22:51.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.386+0000 7f7be5e62740 1 Processor -- start 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008e03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008a3770 0x556e00924010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 0x556e008a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e008bded0 con 0x556e008bce50 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e008a4a00 con 0x556e008a3770 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e008b93f0 con 0x556e008a2600 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 10 client.0 _ll_register_callbacks cb 0x7f7bd8024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 0x556e008a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 0x556e008a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33920/0 (socket says 172.21.15.72:33920) 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 -- 172.21.15.72:0/2161694850 learned_addr learned my addr 172.21.15.72:0/2161694850 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be081c640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008a3770 0x556e00924010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be101d640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008e03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 -- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 msgr2=0x556e008e03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008e03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 -- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008a3770 msgr2=0x556e00924010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008a3770 0x556e00924010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 -- 172.21.15.72:0/2161694850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e005bb6b0 con 0x556e008bce50 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be181e640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 0x556e008a01a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4083700 tx=0x7f7bd407b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453ad5169e0d699d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7bbbfff640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7bbbfff640 1 -- 172.21.15.72:0/2161694850 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd407f670 con 0x556e008bce50 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7bbbfff640 1 -- 172.21.15.72:0/2161694850 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bd40837a0 con 0x556e008bce50 2024-03-06T07:22:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bd8030070 con 0x556e008bce50 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7bbbfff640 1 -- 172.21.15.72:0/2161694850 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7bd4083c30 con 0x556e008bce50 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 10 client.5385 fetch_fsmap learned FSMap version 14 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.387+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x556e008e13a0 con 0x556e008bce50 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7bbbfff640 1 -- 172.21.15.72:0/2161694850 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd8030070 con 0x556e008bce50 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7bbbfff640 1 -- 172.21.15.72:0/2161694850 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 14) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f7bd407f150 con 0x556e008bce50 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 10 client.5385 fetch_fsmap finished waiting for FSMap version 14 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 10 client.5385 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 20 client.5385 populate_metadata read hostname 'smithi072' 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e00877de0 0x556e008781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 4 client.5385 mds_command: new command op to 5196 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 --> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556e007e3f40 con 0x556e00877de0 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be101d640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e00877de0 0x556e008781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7bbbfff640 10 client.5385 ms_handle_connect on v2:172.21.15.72:6835/1659735808 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be101d640 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e00877de0 0x556e008781c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a99c0082db5dea09 server_cookie=4bcd6cf5882473d1 in_seq=0 out_seq=0 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7bbbfff640 1 -- 172.21.15.72:0/2161694850 <== mds.0 v2:172.21.15.72:6835/1659735808 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x556e007e3f40 con 0x556e00877de0 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7bbbfff640 10 client.5385 handle_command_reply: tid=0 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 client.5385 shutdown 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 msgr2=0x556e008a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 0x556e008a01a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4083700 tx=0x7f7bd407b120 comp rx=0 tx=0).stop 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 shutdown_connections 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:6835/1659735808,v1:172.21.15.72:6837/1659735808] conn(0x556e00877de0 0x556e008781c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x556e008bce50 0x556e008a01a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x556e008a3770 0x556e00924010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7bbbfff640 0 client.5385 ms_handle_reset on v2:172.21.15.72:6835/1659735808 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 --2- 172.21.15.72:0/2161694850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x556e008a2600 0x556e008e03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.388+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 >> 172.21.15.72:0/2161694850 conn(0x556e00872870 msgr2=0x556e008d80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 shutdown_connections 2024-03-06T07:22:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be5e62740 1 -- 172.21.15.72:0/2161694850 wait complete. 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be5e62740 20 client.5385 trim_cache size 0 max 16384 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5196, 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 4.7124205229999996, 2024-03-06T07:22:51.398 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 14, 2024-03-06T07:22:51.399 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 34, 2024-03-06T07:22:51.399 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 34, 2024-03-06T07:22:51.399 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 11.707500145999999 2024-03-06T07:22:51.399 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:22:51.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7bc006b4d0 msgr2=0x7f7bc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7bc006b4d0 0x7f7bc006d990 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8007fd0 tx=0x7f7bd8007e80 comp rx=0 tx=0).stop 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 msgr2=0x7f7bdc14d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 0x7f7bdc14d180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4036040 tx=0x7f7bd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 shutdown_connections 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7bc006b4d0 0x7f7bc006d990 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bdc1614a0 0x7f7bdc14d180 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bdc0b1850 0x7f7bdc15bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 --2- 172.21.15.72:0/2022125581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bdc06d3c0 0x7f7bdc0b04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 >> 172.21.15.72:0/2022125581 conn(0x7f7bdc09a9f0 msgr2=0x7f7bdc06c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 shutdown_connections 2024-03-06T07:22:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.389+0000 7f7be3281640 1 -- 172.21.15.72:0/2022125581 wait complete. 2024-03-06T07:22:51.402 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.501+0000 7fdefb637640 1 Processor -- start 2024-03-06T07:22:51.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.501+0000 7fdefb637640 1 -- start start 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.501+0000 7fdefb637640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdefb637640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40b3430 con 0x7fdef40b41f0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdefb637640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef40b3710 con 0x7fdef415dfe0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdefb637640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40d2c50 con 0x7fdef40d22b0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdef9bd4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdef40b41f0 0x7fdef40b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41158/0 (socket says 172.21.15.72:41158) 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdef9bd4640 1 -- 172.21.15.72:0/1272938176 learned_addr learned my addr 172.21.15.72:0/1272938176 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdef8bd2640 1 --2- 172.21.15.72:0/1272938176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176359683 0 0) 0x7fdef40b3430 con 0x7fdef40b41f0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdee0003520 con 0x7fdef40b41f0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577033710 0 0) 0x7fdef40b3710 con 0x7fdef415dfe0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef40b3430 con 0x7fdef415dfe0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259981176 0 0) 0x7fdee0003520 con 0x7fdef40b41f0 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdef8bd2640 1 -- 172.21.15.72:0/1272938176 >> v1:172.21.15.146:6789/0 conn(0x7fdef415dfe0 legacy=0x7fdef40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdef8bd2640 1 -- 172.21.15.72:0/1272938176 >> v1:172.21.15.72:6789/0 conn(0x7fdef40b41f0 legacy=0x7fdef40b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.502+0000 7fdef8bd2640 1 -- 172.21.15.72:0/1272938176 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40b39f0 con 0x7fdef40d22b0 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.503+0000 7fdef8bd2640 1 --2- 172.21.15.72:0/1272938176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef40d2710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdee40097d0 tx=0x7fdee402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7471bac25b734369 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.503+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee4037070 con 0x7fdef40d22b0 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.503+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee4030510 con 0x7fdef40d22b0 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.503+0000 7fdeebfff640 1 -- 172.21.15.72:0/1272938176 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec001670 con 0x7fdef40d22b0 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 -- 172.21.15.72:0/1272938176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 msgr2=0x7fdef40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 --2- 172.21.15.72:0/1272938176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef40d2710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdee40097d0 tx=0x7fdee402b800 comp rx=0 tx=0).stop 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 -- 172.21.15.72:0/1272938176 shutdown_connections 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 --2- 172.21.15.72:0/1272938176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef40d2710 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 -- 172.21.15.72:0/1272938176 >> 172.21.15.72:0/1272938176 conn(0x7fdef40d3cd0 msgr2=0x7fdef40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 -- 172.21.15.72:0/1272938176 shutdown_connections 2024-03-06T07:22:51.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.504+0000 7fdefb637640 1 -- 172.21.15.72:0/1272938176 wait complete. 2024-03-06T07:22:51.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.506+0000 7fdefb637640 1 Processor -- start 2024-03-06T07:22:51.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.506+0000 7fdefb637640 1 -- start start 2024-03-06T07:22:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdefb637640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdef40b41f0 0x7fdef414f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdefb637640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef415e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdefb637640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 0x7fdef40b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdefb637640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40b3d60 con 0x7fdef415dfe0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdefb637640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40b31e0 con 0x7fdef40b41f0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdefb637640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef40d1c70 con 0x7fdef40d22b0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 0x7fdef40b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 0x7fdef40b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33922/0 (socket says 172.21.15.72:33922) 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 -- 172.21.15.72:0/769810996 learned_addr learned my addr 172.21.15.72:0/769810996 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef93d3640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdef40b41f0 0x7fdef414f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef8bd2640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef415e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 -- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 msgr2=0x7fdef415e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef415e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 -- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdef40b41f0 msgr2=0x7fdef414f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdef40b41f0 0x7fdef414f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.507+0000 7fdef9bd4640 1 -- 172.21.15.72:0/769810996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef415ed30 con 0x7fdef415dfe0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.508+0000 7fdef9bd4640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 0x7fdef40b2180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c780 tx=0x7fdeec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.508+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec017070 con 0x7fdef415dfe0 2024-03-06T07:22:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.508+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec005ad0 con 0x7fdef415dfe0 2024-03-06T07:22:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.508+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec007070 con 0x7fdef415dfe0 2024-03-06T07:22:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.508+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef415f010 con 0x7fdef415dfe0 2024-03-06T07:22:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.508+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef4160240 con 0x7fdef415dfe0 2024-03-06T07:22:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.509+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef404f330 con 0x7fdef415dfe0 2024-03-06T07:22:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.510+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdeec01a070 con 0x7fdef415dfe0 2024-03-06T07:22:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.510+0000 7fdee9ffb640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.510+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdeec0896b0 con 0x7fdef415dfe0 2024-03-06T07:22:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.510+0000 7fdef93d3640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.511+0000 7fdef93d3640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fded806b4d0 0x7fded806d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdef0001560 tx=0x7fdef0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.515+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdeec0585c0 con 0x7fdef415dfe0 2024-03-06T07:22:51.553 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-03-06T07:22:51.551+0000 7f0c830c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:22:51.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.726+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdef414f5f0 con 0x7fdef415dfe0 2024-03-06T07:22:51.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.726+0000 7fdee9ffb640 1 -- 172.21.15.72:0/769810996 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdeec057ec0 con 0x7fdef415dfe0 2024-03-06T07:22:51.728 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:22:51.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fded806b4d0 msgr2=0x7fded806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fded806b4d0 0x7fded806d990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdef0001560 tx=0x7fdef0009040 comp rx=0 tx=0).stop 2024-03-06T07:22:51.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 msgr2=0x7fdef40b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:51.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 0x7fdef40b2180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c780 tx=0x7fdeec00cc40 comp rx=0 tx=0).stop 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 shutdown_connections 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fded806b4d0 0x7fded806d990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdef415dfe0 0x7fdef40b2180 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdef40d22b0 0x7fdef415e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 --2- 172.21.15.72:0/769810996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdef40b41f0 0x7fdef414f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 >> 172.21.15.72:0/769810996 conn(0x7fdef40d3cd0 msgr2=0x7fdef406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 shutdown_connections 2024-03-06T07:22:51.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:51.728+0000 7fdefb637640 1 -- 172.21.15.72:0/769810996 wait complete. 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:22:51.741 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:51.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:51.741 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:51.757 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42624sec preferred_lft 42624sec 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.758 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:22:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:22:51.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:51.760 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:22:51.824 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:22:51.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:51.825 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:22:51.882 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:22:51.882 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:22:51.882 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:22:51.882 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:22:51.883 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:22:51.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:51.883 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:22:51.883 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:22:51.883 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:22:51.883 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:22:51.883 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:22:51.883 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:22:51.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:51.978 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:22:51.992 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:22:51.993 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:22:51.993 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:51.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:51.993 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:22:52.061 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:22:52.061 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:52.061 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:52.116 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:22:52.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:22:52.116 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:52.171 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:22:52.243 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:22:52.298 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:22:52.299 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:22:52.300 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:52.301 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:52.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:52.302 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:22:52.360 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:22:52.360 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:22:52.402 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:22:52.428 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:22:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:22:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:22:52.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:52.488 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:22:52.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:22:52.487+0000 7feda4bfd480 -1 init, newargv = 0x5654105025b0 newargc=15 2024-03-06T07:22:52.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[52107]: starting ceph client 2024-03-06T07:22:52.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[52107]: starting fuse 2024-03-06T07:22:52.504 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:22:52.504 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:22:52.504 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:22:52.504 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:22:52.504 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:22:52.504 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:22:52.504 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:22:52.504 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:22:52.505 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:22:52.506 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:22:52.506 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:22:52.506 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:22:52.506 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:52.506 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:22:52.506 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:22:52.623 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.52107.asok 2024-03-06T07:22:52.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:52.634 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52107.asok 2024-03-06T07:22:52.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:52.634 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52107.asok status 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "52107", 2024-03-06T07:22:52.772 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5400, 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 5400 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1266747269 2024-03-06T07:22:52.773 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1266747269 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.5400 192.168.0.1:0/1266747269", 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/1266747269", 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 14, 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 34, 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:22:52.774 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:22:52.775 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:22:52.775 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:22:52.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:52.780 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:22:52.780 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:52.796 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:22:52.796 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:52.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:52.796 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:22:52.864 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:22:52.864 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:22:52.864 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:52.864 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:22:52.864 DEBUG:teuthology.orchestra.run.smithi072:> subdir = "delete_me" 2024-03-06T07:22:52.864 DEBUG:teuthology.orchestra.run.smithi072:> size = 1024 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> file_count = 1000 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(0, file_count): 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> filename = "{0}_{1}.bin".format(i, size) 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> f.write(size * '"'"'x'"'"') 2024-03-06T07:22:52.865 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:22:54.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:54.756 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:54.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.899+0000 7f4c8fedf640 1 Processor -- start 2024-03-06T07:22:54.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.900+0000 7f4c8fedf640 1 -- start start 2024-03-06T07:22:54.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.900+0000 7f4c8fedf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c8815dfe0 0x7f4c880cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.900+0000 7f4c8fedf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c880cf830 0x7f4c880d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.900+0000 7f4c8fedf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c880b3430 con 0x7f4c880d27f0 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.900+0000 7f4c8fedf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880b31e0 con 0x7f4c880cf830 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.900+0000 7f4c8fedf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c880b3d60 con 0x7f4c8815dfe0 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c880cf830 0x7f4c880d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c880cf830 0x7f4c880d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44414/0 (socket says 172.21.15.72:44414) 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 -- 172.21.15.72:0/3710633438 learned_addr learned my addr 172.21.15.72:0/3710633438 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 -- 172.21.15.72:0/3710633438 >> v2:172.21.15.146:3300/0 conn(0x7f4c8815dfe0 msgr2=0x7f4c880cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8dc7b640 1 --2- 172.21.15.72:0/3710633438 >> v2:172.21.15.146:3300/0 conn(0x7f4c8815dfe0 0x7f4c880cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 --2- 172.21.15.72:0/3710633438 >> v2:172.21.15.146:3300/0 conn(0x7f4c8815dfe0 0x7f4c880cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 -- 172.21.15.72:0/3710633438 >> v1:172.21.15.146:6789/0 conn(0x7f4c880d27f0 legacy=0x7f4c880d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.901+0000 7f4c8d47a640 1 -- 172.21.15.72:0/3710633438 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880b3710 con 0x7f4c880cf830 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8d47a640 1 --2- 172.21.15.72:0/3710633438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c880d22b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009870 tx=0x7f4c8402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6409efc10090bfa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8cc79640 1 -- 172.21.15.72:0/3710633438 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84035020 con 0x7f4c880cf830 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8cc79640 1 -- 172.21.15.72:0/3710633438 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c84031870 con 0x7f4c880cf830 2024-03-06T07:22:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8cc79640 1 -- 172.21.15.72:0/3710633438 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84031c40 con 0x7f4c880cf830 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 -- 172.21.15.72:0/3710633438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 msgr2=0x7f4c880d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/3710633438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c880d22b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009870 tx=0x7f4c8402b910 comp rx=0 tx=0).stop 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 -- 172.21.15.72:0/3710633438 shutdown_connections 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/3710633438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c880d22b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/3710633438 >> v2:172.21.15.146:3300/0 conn(0x7f4c8815dfe0 0x7f4c880cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 -- 172.21.15.72:0/3710633438 >> 172.21.15.72:0/3710633438 conn(0x7f4c880d3cd0 msgr2=0x7f4c880d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 -- 172.21.15.72:0/3710633438 shutdown_connections 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.902+0000 7f4c8fedf640 1 -- 172.21.15.72:0/3710633438 wait complete. 2024-03-06T07:22:54.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.903+0000 7f4c8fedf640 1 Processor -- start 2024-03-06T07:22:54.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.904+0000 7f4c8fedf640 1 -- start start 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c88161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c880d27f0 0x7f4c880cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8815dfe0 0x7f4c880b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c880d1df0 con 0x7f4c880cf830 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c880d1c70 con 0x7f4c880d27f0 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8815e3c0 con 0x7f4c8815dfe0 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c88161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c88161210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44424/0 (socket says 172.21.15.72:44424) 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 -- 172.21.15.72:0/2530630377 learned_addr learned my addr 172.21.15.72:0/2530630377 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 -- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8815dfe0 msgr2=0x7f4c880b9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8815dfe0 0x7f4c880b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8d47a640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c880d27f0 0x7f4c880cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 -- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c880d27f0 msgr2=0x7f4c880cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c880d27f0 0x7f4c880cc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 -- 172.21.15.72:0/2530630377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c880ccf50 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8dc7b640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c88161210 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800c820 tx=0x7f4c7800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78015070 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c78004240 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7800ce10 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.905+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c880cd230 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.906+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c880b9690 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.907+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c880d2230 con 0x7f4c880cf830 2024-03-06T07:22:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.907+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c78007830 con 0x7f4c880cf830 2024-03-06T07:22:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.907+0000 7f4c7e7fc640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.907+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c7808a0c0 con 0x7f4c880cf830 2024-03-06T07:22:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.908+0000 7f4c8d47a640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.908+0000 7f4c8d47a640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c84036000 tx=0x7f4c84008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:54.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:54.912+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c78058f40 con 0x7f4c880cf830 2024-03-06T07:22:55.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.131+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c8804f330 con 0x7f4c880cf830 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.132+0000 7f4c7e7fc640 1 -- 172.21.15.72:0/2530630377 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f4c78058840 con 0x7f4c880cf830 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:55.134 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:22:55.135 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 msgr2=0x7f4c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4c84036000 tx=0x7f4c84008030 comp rx=0 tx=0).stop 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 msgr2=0x7f4c88161210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c88161210 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800c820 tx=0x7f4c7800cce0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 shutdown_connections 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8815dfe0 0x7f4c880b9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c880d27f0 0x7f4c880cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 --2- 172.21.15.72:0/2530630377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c880cf830 0x7f4c88161210 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 >> 172.21.15.72:0/2530630377 conn(0x7f4c880d3cd0 msgr2=0x7f4c880d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 shutdown_connections 2024-03-06T07:22:55.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.135+0000 7f4c8fedf640 1 -- 172.21.15.72:0/2530630377 wait complete. 2024-03-06T07:22:55.148 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:22:55.413 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:22:55.414 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:22:55.414 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:22:55.415 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:22:55.415 DEBUG:teuthology.orchestra.run.smithi072:> import stat 2024-03-06T07:22:55.415 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:22:55.415 DEBUG:teuthology.orchestra.run.smithi072:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-03-06T07:22:55.415 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:22:55.480 INFO:teuthology.orchestra.run.smithi072.stdout:1099511627776 2024-03-06T07:22:55.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:22:55.489 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:22:55.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.568+0000 7fb19201ef40 1 Processor -- start 2024-03-06T07:22:55.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.568+0000 7fb19201ef40 1 -- start start 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb19201ef40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01b32bc0 0x560e01bc0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb19201ef40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01bdb880 0x560e01bbb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb19201ef40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01b31ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb19201ef40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e01bdbc90 con 0x560e01bdd520 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb19201ef40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e01aa9b70 con 0x560e01b32bc0 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb19201ef40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e01b32fd0 con 0x560e01bdb880 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01b31ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01b31ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44438/0 (socket says 172.21.15.72:44438) 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 -- 172.21.15.72:0/2596302321 learned_addr learned my addr 172.21.15.72:0/2596302321 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:55.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb18f320640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01bdb880 0x560e01bbb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb18fb21640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01b32bc0 0x560e01bc0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 -- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01bdb880 msgr2=0x560e01bbb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01bdb880 0x560e01bbb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 -- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01b32bc0 msgr2=0x560e01bc0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01b32bc0 0x560e01bc0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 -- 172.21.15.72:0/2596302321 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e01b06680 con 0x560e01bdd520 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb18f320640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01bdb880 0x560e01bbb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb18fb21640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01b32bc0 0x560e01bc0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb190322640 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01b31ad0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb18801e210 tx=0x7fb18804f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef19dea15a6ac694 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.569+0000 7fb18e31e640 1 -- 172.21.15.72:0/2596302321 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18805a040 con 0x560e01bdd520 2024-03-06T07:22:55.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb18e31e640 1 -- 172.21.15.72:0/2596302321 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb188064070 con 0x560e01bdd520 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb18e31e640 1 -- 172.21.15.72:0/2596302321 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18804dd10 con 0x560e01bdd520 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 -- 172.21.15.72:0/2596302321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 msgr2=0x560e01b31ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01b31ad0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb18801e210 tx=0x7fb18804f000 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 -- 172.21.15.72:0/2596302321 shutdown_connections 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01b31ad0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01bdb880 0x560e01bbb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 --2- 172.21.15.72:0/2596302321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01b32bc0 0x560e01bc0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 -- 172.21.15.72:0/2596302321 >> 172.21.15.72:0/2596302321 conn(0x560e01b018b0 msgr2=0x560e01b01c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 -- 172.21.15.72:0/2596302321 shutdown_connections 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.570+0000 7fb19201ef40 1 -- 172.21.15.72:0/2596302321 wait complete. 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 Processor -- start 2024-03-06T07:22:55.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 -- start start 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01b32bc0 0x560e01bd3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01bdb880 0x560e01bb90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01bb9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e01b06130 con 0x560e01bdd520 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e01bc06c0 con 0x560e01bdb880 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb19201ef40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e01bbc210 con 0x560e01b32bc0 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb190322640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01bb9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.571+0000 7fb190322640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01bb9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44448/0 (socket says 172.21.15.72:44448) 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 -- 172.21.15.72:0/421858287 learned_addr learned my addr 172.21.15.72:0/421858287 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb18f320640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01bdb880 0x560e01bb90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb18fb21640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01b32bc0 0x560e01bd3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 -- 172.21.15.72:0/421858287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01b32bc0 msgr2=0x560e01bd3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560e01b32bc0 0x560e01bd3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 -- 172.21.15.72:0/421858287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01bdb880 msgr2=0x560e01bb90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560e01bdb880 0x560e01bb90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 -- 172.21.15.72:0/421858287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e01b046f0 con 0x560e01bdd520 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb190322640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01bb9600 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb18801e210 tx=0x7fb188003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb18d31c640 1 -- 172.21.15.72:0/421858287 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188073680 con 0x560e01bdd520 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb18d31c640 1 -- 172.21.15.72:0/421858287 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb188073820 con 0x560e01bdd520 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x560e01b03650 con 0x560e01bdd520 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb18d31c640 1 -- 172.21.15.72:0/421858287 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188073c10 con 0x560e01bdd520 2024-03-06T07:22:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.572+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560e01b03c10 con 0x560e01bdd520 2024-03-06T07:22:55.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.574+0000 7fb18d31c640 1 -- 172.21.15.72:0/421858287 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb188053070 con 0x560e01bdd520 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.574+0000 7fb18d31c640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb17c069eb0 0x7fb17c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.574+0000 7fb18fb21640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb17c069eb0 0x7fb17c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.574+0000 7fb18d31c640 1 -- 172.21.15.72:0/421858287 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1880ce7b0 con 0x560e01bdd520 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x560e01b031c0 0x560e01c70ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v9 -- 0x560e01c71430 con 0x560e01b031c0 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb18f320640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x560e01b031c0 0x560e01c70ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb18fb21640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb17c069eb0 0x7fb17c06c370 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb180001ed0 tx=0x7fb1800017d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb18f320640 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x560e01b031c0 0x560e01c70ef0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:55.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb18f320640 1 -- 172.21.15.72:0/421858287 <== osd.3 v2:172.21.15.72:6808/4245662075 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x7fb18400b960 con 0x560e01b031c0 2024-03-06T07:22:55.582 INFO:teuthology.orchestra.run.smithi072.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-03-06T07:22:55.000000+0000, size 0 2024-03-06T07:22:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x560e01b031c0 msgr2=0x560e01c70ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x560e01b031c0 0x560e01c70ef0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb17c069eb0 msgr2=0x7fb17c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb17c069eb0 0x7fb17c06c370 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb180001ed0 tx=0x7fb1800017d0 comp rx=0 tx=0).stop 2024-03-06T07:22:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 msgr2=0x560e01bb9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.575+0000 7fb19201ef40 1 --2- 172.21.15.72:0/421858287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560e01bdd520 0x560e01bb9600 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb18801e210 tx=0x7fb188003330 comp rx=0 tx=0).stop 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.576+0000 7fb190322640 1 -- 172.21.15.72:0/421858287 reap_dead start 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.576+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 shutdown_connections 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.576+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 >> 172.21.15.72:0/421858287 conn(0x560e01b018b0 msgr2=0x560e01b2faf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.576+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 shutdown_connections 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:55.576+0000 7fb19201ef40 1 -- 172.21.15.72:0/421858287 wait complete. 2024-03-06T07:22:55.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:22:55.584 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:22:57.877 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:57.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.977+0000 7f1ff8993640 1 Processor -- start 2024-03-06T07:22:57.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.977+0000 7f1ff8993640 1 -- start start 2024-03-06T07:22:57.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff8993640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1ff40d2710 0x7f1ff40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff8993640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40b3430 con 0x7f1ff40d22b0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff8993640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40b3710 con 0x7f1ff415dfe0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff8993640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff40b31e0 con 0x7f1ff40d2710 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1febfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1ff40d22b0 0x7f1ff40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56596/0 (socket says 172.21.15.72:56596) 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1febfff640 1 -- 172.21.15.72:0/660837659 learned_addr learned my addr 172.21.15.72:0/660837659 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff2d9d640 1 --2- 172.21.15.72:0/660837659 >> v2:172.21.15.146:3300/0 conn(0x7f1ff40d2710 0x7f1ff40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183084606 0 0) 0x7f1ff40b3430 con 0x7f1ff40d22b0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0003520 con 0x7f1ff40d22b0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764410069 0 0) 0x7f1fe0003520 con 0x7f1ff40d22b0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 >> v2:172.21.15.146:3300/0 conn(0x7f1ff40d2710 msgr2=0x7f1ff40b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 --2- 172.21.15.72:0/660837659 >> v2:172.21.15.146:3300/0 conn(0x7f1ff40d2710 0x7f1ff40b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 >> v1:172.21.15.146:6789/0 conn(0x7f1ff415dfe0 legacy=0x7f1ff40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40b39f0 con 0x7f1ff40d22b0 2024-03-06T07:22:57.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1fdc001670 con 0x7f1ff40d22b0 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 >> v1:172.21.15.72:6789/0 conn(0x7f1ff40d22b0 legacy=0x7f1ff40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00039e0 0x7f1fe0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0004800 0x7f1fe0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe00051c0 0x7f1fe000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40b31e0 con 0x7f1fe00051c0 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe000c090 con 0x7f1fe0004800 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.978+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe000c290 con 0x7f1fe00039e0 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff2d9d640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe00051c0 0x7f1fe000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff259c640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00039e0 0x7f1fe0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1febfff640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0004800 0x7f1fe0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff2d9d640 1 -- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00039e0 msgr2=0x7f1fe0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff2d9d640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00039e0 0x7f1fe0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff2d9d640 1 -- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0004800 msgr2=0x7f1fe0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff2d9d640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0004800 0x7f1fe0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.979+0000 7f1ff2d9d640 1 -- 172.21.15.72:0/660837659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0003520 con 0x7f1fe00051c0 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff2d9d640 1 -- 172.21.15.72:0/660837659 reap_dead start 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff2d9d640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe00051c0 0x7f1fe000b9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009fc0 tx=0x7f1fe0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf6b292dcb19953 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec009c10 con 0x7f1fe00051c0 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec02cca0 con 0x7f1fe00051c0 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff1d9b640 1 -- 172.21.15.72:0/660837659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec003bd0 con 0x7f1fe00051c0 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 -- 172.21.15.72:0/660837659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe00051c0 msgr2=0x7f1fe000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe00051c0 0x7f1fe000b9c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009fc0 tx=0x7f1fe0005830 comp rx=0 tx=0).stop 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 -- 172.21.15.72:0/660837659 shutdown_connections 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 --2- 172.21.15.72:0/660837659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1fe00051c0 0x7f1fe000b9c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 -- 172.21.15.72:0/660837659 >> 172.21.15.72:0/660837659 conn(0x7f1ff40d3cd0 msgr2=0x7f1ff40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:57.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 -- 172.21.15.72:0/660837659 shutdown_connections 2024-03-06T07:22:57.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.980+0000 7f1ff8993640 1 -- 172.21.15.72:0/660837659 wait complete. 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 Processor -- start 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 -- start start 2024-03-06T07:22:57.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe00051c0 0x7f1ff415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 0x7f1ff40b78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff409e6a0 0x7f1ff40b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40d1c70 con 0x7f1ff409e220 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40b3d60 con 0x7f1fe00051c0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.982+0000 7f1ff8993640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40d1df0 con 0x7f1ff409e6a0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 0x7f1ff40b78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 0x7f1ff40b78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44464/0 (socket says 172.21.15.72:44464) 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 -- 172.21.15.72:0/3728237462 learned_addr learned my addr 172.21.15.72:0/3728237462 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1ff259c640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe00051c0 0x7f1ff415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1ff2d9d640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff409e6a0 0x7f1ff40b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 -- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff409e6a0 msgr2=0x7f1ff40b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff409e6a0 0x7f1ff40b7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 -- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe00051c0 msgr2=0x7f1ff415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:57.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe00051c0 0x7f1ff415c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 -- 172.21.15.72:0/3728237462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff415c680 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1febfff640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 0x7f1ff40b78a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0072f0 tx=0x7f1fdc0077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc01a070 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc007e10 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff40b6ae0 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc00fbc0 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.983+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff40b70a0 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.984+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff40d4c10 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.985+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fdc01d070 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.986+0000 7f1feaffd640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fd406b500 0x7f1fd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.986+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fdc089a50 con 0x7f1ff409e220 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.986+0000 7f1ff259c640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fd406b500 0x7f1fd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:57.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.986+0000 7f1ff259c640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fd406b500 0x7f1fd406d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4000c00 tx=0x7f1fe4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:57.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:57.990+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ff40d4c10 con 0x7f1ff409e220 2024-03-06T07:22:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.204+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ff404f330 con 0x7f1ff409e220 2024-03-06T07:22:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.205+0000 7f1feaffd640 1 -- 172.21.15.72:0/3728237462 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f1fdc058310 con 0x7f1ff409e220 2024-03-06T07:22:58.207 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:58.207 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:22:58.208 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fd406b500 msgr2=0x7f1fd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fd406b500 0x7f1fd406d9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4000c00 tx=0x7f1fe4009040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 msgr2=0x7f1ff40b78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 0x7f1ff40b78a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc0072f0 tx=0x7f1fdc0077b0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 shutdown_connections 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fd406b500 0x7f1fd406d9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff409e6a0 0x7f1ff40b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ff409e220 0x7f1ff40b78a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 --2- 172.21.15.72:0/3728237462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe00051c0 0x7f1ff415c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 >> 172.21.15.72:0/3728237462 conn(0x7f1ff40d3cd0 msgr2=0x7f1ff40b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 shutdown_connections 2024-03-06T07:22:58.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.207+0000 7f1ff8993640 1 -- 172.21.15.72:0/3728237462 wait complete. 2024-03-06T07:22:58.223 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:22:58.496 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:22:58.499 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:22:58.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:58.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.602+0000 7f2d36593640 1 Processor -- start 2024-03-06T07:22:58.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.602+0000 7f2d36593640 1 -- start start 2024-03-06T07:22:58.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d36593640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2d301424f0 0x7f2d300cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d36593640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d300cc980 con 0x7f2d3006da40 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d36593640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d300ccc60 con 0x7f2d300cd7a0 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d36593640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d300cd220 con 0x7f2d301424f0 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d2ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2d301424f0 0x7f2d300cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d34b30640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2d3006da40 0x7f2d30071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56608/0 (socket says 172.21.15.72:56608) 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d34b30640 1 -- 172.21.15.72:0/3028884423 learned_addr learned my addr 172.21.15.72:0/3028884423 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.603+0000 7f2d2ffff640 1 -- 172.21.15.72:0/3028884423 >> v1:172.21.15.146:6789/0 conn(0x7f2d300cd7a0 legacy=0x7f2d3006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2ffff640 1 -- 172.21.15.72:0/3028884423 >> v1:172.21.15.72:6789/0 conn(0x7f2d3006da40 legacy=0x7f2d30071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2effd640 1 -- 172.21.15.72:0/3028884423 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278220799 0 0) 0x7f2d300cc980 con 0x7f2d3006da40 2024-03-06T07:22:58.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2ffff640 1 -- 172.21.15.72:0/3028884423 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d300ccf40 con 0x7f2d301424f0 2024-03-06T07:22:58.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2ffff640 1 --2- 172.21.15.72:0/3028884423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc2f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2d200097a0 tx=0x7f2d2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6227474af8d7a99b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2effd640 1 -- 172.21.15.72:0/3028884423 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20035020 con 0x7f2d301424f0 2024-03-06T07:22:58.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2effd640 1 -- 172.21.15.72:0/3028884423 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d20031830 con 0x7f2d301424f0 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.604+0000 7f2d2effd640 1 -- 172.21.15.72:0/3028884423 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d20031c00 con 0x7f2d301424f0 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 -- 172.21.15.72:0/3028884423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 msgr2=0x7f2d300cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 --2- 172.21.15.72:0/3028884423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc2f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2d200097a0 tx=0x7f2d2002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 -- 172.21.15.72:0/3028884423 shutdown_connections 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 --2- 172.21.15.72:0/3028884423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc2f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 -- 172.21.15.72:0/3028884423 >> 172.21.15.72:0/3028884423 conn(0x7f2d300cf8a0 msgr2=0x7f2d300cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 -- 172.21.15.72:0/3028884423 shutdown_connections 2024-03-06T07:22:58.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.605+0000 7f2d36593640 1 -- 172.21.15.72:0/3028884423 wait complete. 2024-03-06T07:22:58.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.606+0000 7f2d36593640 1 Processor -- start 2024-03-06T07:22:58.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.606+0000 7f2d36593640 1 -- start start 2024-03-06T07:22:58.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d36593640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d3006da40 0x7f2d3013e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d36593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d300cd7a0 0x7f2d300cbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d36593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d36593640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3015d070 con 0x7f2d301424f0 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d36593640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3015cef0 con 0x7f2d3006da40 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d36593640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3015d1f0 con 0x7f2d300cd7a0 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44474/0 (socket says 172.21.15.72:44474) 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 -- 172.21.15.72:0/332408586 learned_addr learned my addr 172.21.15.72:0/332408586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d2ffff640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d3006da40 0x7f2d3013e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d2f7fe640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d300cd7a0 0x7f2d300cbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 -- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d300cd7a0 msgr2=0x7f2d300cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d300cd7a0 0x7f2d300cbb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 -- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d3006da40 msgr2=0x7f2d3013e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d3006da40 0x7f2d3013e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.607+0000 7f2d34b30640 1 -- 172.21.15.72:0/332408586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d301f17a0 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.608+0000 7f2d34b30640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800c760 tx=0x7f2d2800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.608+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28018070 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.608+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2800a640 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.608+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3013fc20 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.608+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2800aa30 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.608+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d301401e0 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.609+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d301f1970 con 0x7f2d301424f0 2024-03-06T07:22:58.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.610+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d2801b070 con 0x7f2d301424f0 2024-03-06T07:22:58.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.610+0000 7f2d2cff9640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d1006b4d0 0x7f2d1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:58.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.610+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d280896b0 con 0x7f2d301424f0 2024-03-06T07:22:58.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.610+0000 7f2d2ffff640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d1006b4d0 0x7f2d1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:58.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.610+0000 7f2d2ffff640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d1006b4d0 0x7f2d1006d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d20036000 tx=0x7f2d2003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:58.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.614+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d28057ee0 con 0x7f2d301424f0 2024-03-06T07:22:58.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.889+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d3004f330 con 0x7f2d301424f0 2024-03-06T07:22:58.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.889+0000 7f2d2cff9640 1 -- 172.21.15.72:0/332408586 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f2d2805bf80 con 0x7f2d301424f0 2024-03-06T07:22:58.892 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:58.892 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:22:58.892 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:22:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d1006b4d0 msgr2=0x7f2d1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d1006b4d0 0x7f2d1006d990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d20036000 tx=0x7f2d2003d040 comp rx=0 tx=0).stop 2024-03-06T07:22:58.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 msgr2=0x7f2d300cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc080 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800c760 tx=0x7f2d2800cc20 comp rx=0 tx=0).stop 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 shutdown_connections 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d1006b4d0 0x7f2d1006d990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d301424f0 0x7f2d300cc080 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d300cd7a0 0x7f2d300cbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 --2- 172.21.15.72:0/332408586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d3006da40 0x7f2d3013e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 >> 172.21.15.72:0/332408586 conn(0x7f2d300cf8a0 msgr2=0x7f2d300cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 shutdown_connections 2024-03-06T07:22:58.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:58.891+0000 7f2d36593640 1 -- 172.21.15.72:0/332408586 wait complete. 2024-03-06T07:22:58.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:22:59.023 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_created":1001}} 2024-03-06T07:22:59.023 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-03-06T07:22:59.023 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:22:59.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.122+0000 7fcee23f7640 1 Processor -- start 2024-03-06T07:22:59.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.122+0000 7fcee23f7640 1 -- start start 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee23f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc06c6d0 0x7fcedc15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee23f7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcedc066d90 0x7fcedc0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee23f7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcedc0cd450 con 0x7fcedc0b4f70 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee23f7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcedc15e9f0 con 0x7fcedc066d90 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee23f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0cda10 con 0x7fcedc06c6d0 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcedc066d90 0x7fcedc0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcedc066d90 0x7fcedc0ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44484/0 (socket says 172.21.15.72:44484) 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 -- 172.21.15.72:0/3980966166 learned_addr learned my addr 172.21.15.72:0/3980966166 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcedb7fe640 1 --2- 172.21.15.72:0/3980966166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc06c6d0 0x7fcedc15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 -- 172.21.15.72:0/3980966166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc06c6d0 msgr2=0x7fcedc15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 --2- 172.21.15.72:0/3980966166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc06c6d0 0x7fcedc15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 -- 172.21.15.72:0/3980966166 >> v1:172.21.15.146:6789/0 conn(0x7fcedc0b4f70 legacy=0x7fcedc0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:22:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.123+0000 7fcee0994640 1 -- 172.21.15.72:0/3980966166 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc0cd730 con 0x7fcedc066d90 2024-03-06T07:22:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee0994640 1 --2- 172.21.15.72:0/3980966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0ccb50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fced4009e50 tx=0x7fced402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75034bf482d3d272 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcedaffd640 1 -- 172.21.15.72:0/3980966166 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced402ca70 con 0x7fcedc066d90 2024-03-06T07:22:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcedaffd640 1 -- 172.21.15.72:0/3980966166 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced402cc10 con 0x7fcedc066d90 2024-03-06T07:22:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcedaffd640 1 -- 172.21.15.72:0/3980966166 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced40343f0 con 0x7fcedc066d90 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee23f7640 1 -- 172.21.15.72:0/3980966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 msgr2=0x7fcedc0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee23f7640 1 --2- 172.21.15.72:0/3980966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0ccb50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fced4009e50 tx=0x7fced402bed0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee23f7640 1 -- 172.21.15.72:0/3980966166 shutdown_connections 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee23f7640 1 --2- 172.21.15.72:0/3980966166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0ccb50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee23f7640 1 --2- 172.21.15.72:0/3980966166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc06c6d0 0x7fcedc15ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.124+0000 7fcee23f7640 1 -- 172.21.15.72:0/3980966166 >> 172.21.15.72:0/3980966166 conn(0x7fcedc0724d0 msgr2=0x7fcedc0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.125+0000 7fcee23f7640 1 -- 172.21.15.72:0/3980966166 shutdown_connections 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.125+0000 7fcee23f7640 1 -- 172.21.15.72:0/3980966166 wait complete. 2024-03-06T07:22:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.126+0000 7fcee23f7640 1 Processor -- start 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.126+0000 7fcee23f7640 1 -- start start 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.126+0000 7fcee23f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.126+0000 7fcee23f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc0b4f70 0x7fcedc1473b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee23f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcedc0ceb30 0x7fcedc15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee23f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0cd090 con 0x7fcedc066d90 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee23f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc15f3c0 con 0x7fcedc0ceb30 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee23f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc0cd210 con 0x7fcedc0b4f70 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0d00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44498/0 (socket says 172.21.15.72:44498) 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 -- 172.21.15.72:0/1354408179 learned_addr learned my addr 172.21.15.72:0/1354408179 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee0994640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcedc0ceb30 0x7fcedc15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedb7fe640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc0b4f70 0x7fcedc1473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 -- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc0b4f70 msgr2=0x7fcedc1473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc0b4f70 0x7fcedc1473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 -- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcedc0ceb30 msgr2=0x7fcedc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcedc0ceb30 0x7fcedc15cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 -- 172.21.15.72:0/1354408179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc0cf030 con 0x7fcedc066d90 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcedbfff640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0d00d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00c430 tx=0x7fcecc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc011790 con 0x7fcedc066d90 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc017070 con 0x7fcedc066d90 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcedc0cf310 con 0x7fcedc066d90 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc011bb0 con 0x7fcedc066d90 2024-03-06T07:22:59.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.127+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcedc0cf8d0 con 0x7fcedc066d90 2024-03-06T07:22:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.128+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcedc04f330 con 0x7fcedc066d90 2024-03-06T07:22:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.129+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcecc01c460 con 0x7fcedc066d90 2024-03-06T07:22:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.129+0000 7fced8ff9640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcebc06b500 0x7fcebc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:22:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.129+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcecc089f00 con 0x7fcedc066d90 2024-03-06T07:22:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.129+0000 7fcedb7fe640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcebc06b500 0x7fcebc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:22:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.130+0000 7fcedb7fe640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcebc06b500 0x7fcebc06d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcedc1f72e0 tx=0x7fcec4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:22:59.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.134+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcecc058730 con 0x7fcedc066d90 2024-03-06T07:22:59.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.373+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcedc0b1860 con 0x7fcedc066d90 2024-03-06T07:22:59.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.373+0000 7fced8ff9640 1 -- 172.21.15.72:0/1354408179 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fcecc05c7d0 con 0x7fcedc066d90 2024-03-06T07:22:59.376 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:22:59.376 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:22:59.376 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcebc06b500 msgr2=0x7fcebc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcebc06b500 0x7fcebc06d9c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcedc1f72e0 tx=0x7fcec4008040 comp rx=0 tx=0).stop 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 msgr2=0x7fcedc0d00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0d00d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00c430 tx=0x7fcecc00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 shutdown_connections 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcebc06b500 0x7fcebc06d9c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcedc0ceb30 0x7fcedc15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcedc0b4f70 0x7fcedc1473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 --2- 172.21.15.72:0/1354408179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcedc066d90 0x7fcedc0d00d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.375+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 >> 172.21.15.72:0/1354408179 conn(0x7fcedc0724d0 msgr2=0x7fcedc06bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.376+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 shutdown_connections 2024-03-06T07:22:59.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:22:59.376+0000 7fcee23f7640 1 -- 172.21.15.72:0/1354408179 wait complete. 2024-03-06T07:22:59.388 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:22:59.504 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":427}} 2024-03-06T07:22:59.505 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 427 } } 2024-03-06T07:22:59.505 DEBUG:tasks.ceph_test_case:wait_until_equal: 427 != 1001, waiting (timeout=600)... 2024-03-06T07:23:04.506 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:04.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.606+0000 7f63e995e640 1 Processor -- start 2024-03-06T07:23:04.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.606+0000 7f63e995e640 1 -- start start 2024-03-06T07:23:04.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.607+0000 7f63e995e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63e415dfe0 0x7f63e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.607+0000 7f63e995e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63e40cf830 0x7f63e40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.607+0000 7f63e995e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e40b3430 con 0x7f63e40d27f0 2024-03-06T07:23:04.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.607+0000 7f63e995e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e40b31e0 con 0x7f63e415dfe0 2024-03-06T07:23:04.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.607+0000 7f63e995e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e40b3d60 con 0x7f63e40cf830 2024-03-06T07:23:04.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63e415dfe0 0x7f63e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f63e40d27f0 0x7f63e40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45430/0 (socket says 172.21.15.72:45430) 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e37fe640 1 -- 172.21.15.72:0/137241727 learned_addr learned my addr 172.21.15.72:0/137241727 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e27fc640 1 --2- 172.21.15.72:0/137241727 >> v2:172.21.15.146:3300/0 conn(0x7f63e40cf830 0x7f63e40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e2ffd640 1 -- 172.21.15.72:0/137241727 >> v2:172.21.15.146:3300/0 conn(0x7f63e40cf830 msgr2=0x7f63e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e2ffd640 1 --2- 172.21.15.72:0/137241727 >> v2:172.21.15.146:3300/0 conn(0x7f63e40cf830 0x7f63e40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e1ffb640 1 -- 172.21.15.72:0/137241727 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574725306 0 0) 0x7f63e40b3430 con 0x7f63e40d27f0 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e2ffd640 1 -- 172.21.15.72:0/137241727 >> v1:172.21.15.72:6789/0 conn(0x7f63e40d27f0 legacy=0x7f63e40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e2ffd640 1 -- 172.21.15.72:0/137241727 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e40b3710 con 0x7f63e415dfe0 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.608+0000 7f63e27fc640 1 --2- 172.21.15.72:0/137241727 >> v2:172.21.15.146:3300/0 conn(0x7f63e40cf830 0x7f63e40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e2ffd640 1 --2- 172.21.15.72:0/137241727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e415dfe0 0x7f63e40cf2f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0097a0 tx=0x7f63cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7728ffb06cdc881 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e1ffb640 1 -- 172.21.15.72:0/137241727 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc035020 con 0x7f63e415dfe0 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e1ffb640 1 -- 172.21.15.72:0/137241727 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc031830 con 0x7f63e415dfe0 2024-03-06T07:23:04.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e1ffb640 1 -- 172.21.15.72:0/137241727 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc031c00 con 0x7f63e415dfe0 2024-03-06T07:23:04.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 -- 172.21.15.72:0/137241727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e415dfe0 msgr2=0x7f63e40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 --2- 172.21.15.72:0/137241727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e415dfe0 0x7f63e40cf2f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0097a0 tx=0x7f63cc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 -- 172.21.15.72:0/137241727 shutdown_connections 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 --2- 172.21.15.72:0/137241727 >> v2:172.21.15.146:3300/0 conn(0x7f63e40cf830 0x7f63e40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 --2- 172.21.15.72:0/137241727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e415dfe0 0x7f63e40cf2f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 -- 172.21.15.72:0/137241727 >> 172.21.15.72:0/137241727 conn(0x7f63e40d3cd0 msgr2=0x7f63e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 -- 172.21.15.72:0/137241727 shutdown_connections 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.609+0000 7f63e995e640 1 -- 172.21.15.72:0/137241727 wait complete. 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 Processor -- start 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 -- start start 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e40cf830 0x7f63e40b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 0x7f63e40b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e415dfe0 0x7f63e40b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40d1df0 con 0x7f63e40d27f0 2024-03-06T07:23:04.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e40d1c70 con 0x7f63e40cf830 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.611+0000 7f63e995e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e415e3c0 con 0x7f63e415dfe0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 0x7f63e40b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 0x7f63e40b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56030/0 (socket says 172.21.15.72:56030) 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 -- 172.21.15.72:0/188181699 learned_addr learned my addr 172.21.15.72:0/188181699 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e2ffd640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e40cf830 0x7f63e40b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e37fe640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e415dfe0 0x7f63e40b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 -- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e415dfe0 msgr2=0x7f63e40b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e415dfe0 0x7f63e40b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 -- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e40cf830 msgr2=0x7f63e40b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e40cf830 0x7f63e40b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 -- 172.21.15.72:0/188181699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e40b8430 con 0x7f63e40d27f0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e27fc640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 0x7f63e40b4610 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f63d400c7f0 tx=0x7f63d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4015070 con 0x7f63e40d27f0 2024-03-06T07:23:04.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d4007820 con 0x7f63e40d27f0 2024-03-06T07:23:04.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e40b8710 con 0x7f63e40d27f0 2024-03-06T07:23:04.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4007bf0 con 0x7f63e40d27f0 2024-03-06T07:23:04.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.612+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e40b8cd0 con 0x7f63e40d27f0 2024-03-06T07:23:04.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.613+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e404f330 con 0x7f63e40d27f0 2024-03-06T07:23:04.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.614+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63d4018070 con 0x7f63e40d27f0 2024-03-06T07:23:04.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.614+0000 7f63dbfff640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63c406b4d0 0x7f63c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:04.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.614+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63d408a050 con 0x7f63e40d27f0 2024-03-06T07:23:04.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.614+0000 7f63e2ffd640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63c406b4d0 0x7f63c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:04.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.615+0000 7f63e2ffd640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63c406b4d0 0x7f63c406d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f63cc036000 tx=0x7f63cc03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:04.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.618+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63d4058880 con 0x7f63e40d27f0 2024-03-06T07:23:04.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.832+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e406a1b0 con 0x7f63e40d27f0 2024-03-06T07:23:04.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.833+0000 7f63dbfff640 1 -- 172.21.15.72:0/188181699 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f63d405c920 con 0x7f63e40d27f0 2024-03-06T07:23:04.835 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:04.835 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:23:04.836 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:23:04.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63c406b4d0 msgr2=0x7f63c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63c406b4d0 0x7f63c406d990 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f63cc036000 tx=0x7f63cc03d040 comp rx=0 tx=0).stop 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 msgr2=0x7f63e40b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 0x7f63e40b4610 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f63d400c7f0 tx=0x7f63d400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 shutdown_connections 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63c406b4d0 0x7f63c406d990 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e415dfe0 0x7f63e40b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63e40d27f0 0x7f63e40b4610 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 --2- 172.21.15.72:0/188181699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e40cf830 0x7f63e40b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 >> 172.21.15.72:0/188181699 conn(0x7f63e40d3cd0 msgr2=0x7f63e40d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 shutdown_connections 2024-03-06T07:23:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:04.835+0000 7f63e995e640 1 -- 172.21.15.72:0/188181699 wait complete. 2024-03-06T07:23:04.848 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:23:04.964 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-03-06T07:23:04.965 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-03-06T07:23:04.965 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:05.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.064+0000 7efcb5a0e640 1 Processor -- start 2024-03-06T07:23:05.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.064+0000 7efcb5a0e640 1 -- start start 2024-03-06T07:23:05.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcb5a0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb00d0a80 0x7efcb00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcb5a0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00d25d0 con 0x7efcb00b4170 2024-03-06T07:23:05.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcb5a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb00d2870 con 0x7efcb00d4760 2024-03-06T07:23:05.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcb5a0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb00d4bf0 con 0x7efcb00d0a80 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcaf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb00d0a80 0x7efcb00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcaeffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efcb00b4170 0x7efcb00d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45436/0 (socket says 172.21.15.72:45436) 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcaf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb00d0a80 0x7efcb00b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56042/0 (socket says 172.21.15.72:56042) 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.065+0000 7efcaeffd640 1 -- 172.21.15.72:0/3950697111 learned_addr learned my addr 172.21.15.72:0/3950697111 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3950697111 >> v1:172.21.15.146:6789/0 conn(0x7efcb00d4760 legacy=0x7efcb00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3950697111 >> v1:172.21.15.72:6789/0 conn(0x7efcb00b4170 legacy=0x7efcb00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3950697111 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00d2b50 con 0x7efcb00d0a80 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/3950697111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d0a80 0x7efcb00b6a90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efca8009d90 tx=0x7efca802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4fdb3ba2472c24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcae7fc640 1 -- 172.21.15.72:0/3950697111 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8039070 con 0x7efcb00d0a80 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcae7fc640 1 -- 172.21.15.72:0/3950697111 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca802cc60 con 0x7efcb00d0a80 2024-03-06T07:23:05.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.066+0000 7efcae7fc640 1 -- 172.21.15.72:0/3950697111 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8034660 con 0x7efcb00d0a80 2024-03-06T07:23:05.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3950697111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d0a80 msgr2=0x7efcb00b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3950697111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d0a80 0x7efcb00b6a90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efca8009d90 tx=0x7efca802beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3950697111 shutdown_connections 2024-03-06T07:23:05.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3950697111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d0a80 0x7efcb00b6a90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3950697111 >> 172.21.15.72:0/3950697111 conn(0x7efcb00d3c40 msgr2=0x7efcb00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3950697111 shutdown_connections 2024-03-06T07:23:05.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.067+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3950697111 wait complete. 2024-03-06T07:23:05.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.068+0000 7efcb5a0e640 1 Processor -- start 2024-03-06T07:23:05.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.068+0000 7efcb5a0e640 1 -- start start 2024-03-06T07:23:05.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcb5a0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcb00b4170 0x7efcb00b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcb5a0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcb00d0a80 0x7efcb00cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcb5a0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 0x7efcb00cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcb5a0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00b4dd0 con 0x7efcb0155c40 2024-03-06T07:23:05.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcb5a0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00ce9e0 con 0x7efcb00d0a80 2024-03-06T07:23:05.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcb5a0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00b4f50 con 0x7efcb00b4170 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcaf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 0x7efcb00cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcaf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 0x7efcb00cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56050/0 (socket says 172.21.15.72:56050) 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3022152393 learned_addr learned my addr 172.21.15.72:0/3022152393 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efcaeffd640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcb00b4170 0x7efcb00b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.069+0000 7efca67fc640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcb00d0a80 0x7efcb00cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcb00b4170 msgr2=0x7efcb00b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcb00b4170 0x7efcb00b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcb00d0a80 msgr2=0x7efcb00cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcb00d0a80 0x7efcb00cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3022152393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00c9fc0 con 0x7efcb0155c40 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 0x7efcb00cb250 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efca8009d90 tx=0x7efca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8044030 con 0x7efcb0155c40 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca802ca60 con 0x7efcb0155c40 2024-03-06T07:23:05.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8039040 con 0x7efcb0155c40 2024-03-06T07:23:05.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb00c9440 con 0x7efcb0155c40 2024-03-06T07:23:05.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.070+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb00c9a00 con 0x7efcb0155c40 2024-03-06T07:23:05.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.071+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb00d23c0 con 0x7efcb0155c40 2024-03-06T07:23:05.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.072+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efca8047070 con 0x7efcb0155c40 2024-03-06T07:23:05.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.072+0000 7efcacff9640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc9006b4d0 0x7efc9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.072+0000 7efcaeffd640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc9006b4d0 0x7efc9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.072+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efca80aca70 con 0x7efcb0155c40 2024-03-06T07:23:05.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.073+0000 7efcaeffd640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc9006b4d0 0x7efc9006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efca0000f60 tx=0x7efca0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.077+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcb00d23c0 con 0x7efcb0155c40 2024-03-06T07:23:05.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.292+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcb004f330 con 0x7efcb0155c40 2024-03-06T07:23:05.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.293+0000 7efcacff9640 1 -- 172.21.15.72:0/3022152393 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7efca804e070 con 0x7efcb0155c40 2024-03-06T07:23:05.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:05.295 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:23:05.296 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc9006b4d0 msgr2=0x7efc9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc9006b4d0 0x7efc9006d990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efca0000f60 tx=0x7efca0009040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 msgr2=0x7efcb00cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 0x7efcb00cb250 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7efca8009d90 tx=0x7efca8004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 shutdown_connections 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc9006b4d0 0x7efc9006d990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb0155c40 0x7efcb00cb250 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcb00d0a80 0x7efcb00cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 --2- 172.21.15.72:0/3022152393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcb00b4170 0x7efcb00b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 >> 172.21.15.72:0/3022152393 conn(0x7efcb00d3c40 msgr2=0x7efcb006b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 shutdown_connections 2024-03-06T07:23:05.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.296+0000 7efcb5a0e640 1 -- 172.21.15.72:0/3022152393 wait complete. 2024-03-06T07:23:05.309 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:23:05.468 INFO:teuthology.orchestra.run.smithi072.stdout:{"purge_queue":{"pq_executed":1001}} 2024-03-06T07:23:05.469 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-03-06T07:23:05.469 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-03-06T07:23:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.509+0000 7f066ed32f40 1 Processor -- start 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.509+0000 7f066ed32f40 1 -- start start 2024-03-06T07:23:05.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.509+0000 7f066ed32f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c96cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066ed32f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 0x56547ca00b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066ed32f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 0x56547ca01090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066ed32f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547c96ffc0 con 0x56547c897240 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066ed32f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547c96fe40 con 0x56547c96c080 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066ed32f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56547c970140 con 0x56547c8d43e0 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c96cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c96cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56066/0 (socket says 172.21.15.72:56066) 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 -- 172.21.15.72:0/1046701726 learned_addr learned my addr 172.21.15.72:0/1046701726 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 -- 172.21.15.72:0/1046701726 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 msgr2=0x56547ca00b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 0x56547ca00b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 -- 172.21.15.72:0/1046701726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 msgr2=0x56547ca01090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 0x56547ca01090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 -- 172.21.15.72:0/1046701726 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56547c7c1c50 con 0x56547c897240 2024-03-06T07:23:05.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f066c835640 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c96cd60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0660044720 tx=0x7f066004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5517f293da0f38aa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f0666ffd640 1 -- 172.21.15.72:0/1046701726 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660057040 con 0x56547c897240 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f0666ffd640 1 -- 172.21.15.72:0/1046701726 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0660063070 con 0x56547c897240 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.510+0000 7f0666ffd640 1 -- 172.21.15.72:0/1046701726 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660052b90 con 0x56547c897240 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 -- 172.21.15.72:0/1046701726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 msgr2=0x56547c96cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c96cd60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0660044720 tx=0x7f066004c000 comp rx=0 tx=0).stop 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 -- 172.21.15.72:0/1046701726 shutdown_connections 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 0x56547ca01090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 0x56547ca00b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 --2- 172.21.15.72:0/1046701726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c96cd60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 -- 172.21.15.72:0/1046701726 >> 172.21.15.72:0/1046701726 conn(0x56547c8c0bd0 msgr2=0x56547c8d4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 -- 172.21.15.72:0/1046701726 shutdown_connections 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 -- 172.21.15.72:0/1046701726 wait complete. 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 Processor -- start 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.511+0000 7f066ed32f40 1 -- start start 2024-03-06T07:23:05.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066ed32f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c969020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066ed32f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 0x56547c8d7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066ed32f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 0x56547c8d7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066ed32f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547c96f350 con 0x56547c897240 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066ed32f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56547c96f0b0 con 0x56547c96c080 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066ed32f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56547c967850 con 0x56547c8d43e0 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c969020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c969020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56074/0 (socket says 172.21.15.72:56074) 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 -- 172.21.15.72:0/2492631668 learned_addr learned my addr 172.21.15.72:0/2492631668 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f0667fff640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 0x56547c8d7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066d036640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 0x56547c8d7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 -- 172.21.15.72:0/2492631668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 msgr2=0x56547c8d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56547c8d43e0 0x56547c8d7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 -- 172.21.15.72:0/2492631668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 msgr2=0x56547c8d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56547c96c080 0x56547c8d7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.512+0000 7f066c835640 1 -- 172.21.15.72:0/2492631668 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56547c8d4f30 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.513+0000 7f066c835640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c969020 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0660000c00 tx=0x7f0660002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.513+0000 7f0665ffb640 1 -- 172.21.15.72:0/2492631668 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066006d070 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.513+0000 7f0665ffb640 1 -- 172.21.15.72:0/2492631668 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0660052c10 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.513+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56547c8d4c50 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.513+0000 7f0665ffb640 1 -- 172.21.15.72:0/2492631668 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0660057050 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.513+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56547c8d7240 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.514+0000 7f0665ffb640 1 -- 172.21.15.72:0/2492631668 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0660067420 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f0665ffb640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0650069eb0 0x7f065006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f0665ffb640 1 -- 172.21.15.72:0/2492631668 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06600cc0c0 con 0x56547c897240 2024-03-06T07:23:05.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f0667fff640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0650069eb0 0x7f065006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f066ed32f40 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x56547c8d47c0 0x56547c95bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v9 -- 0x56547c8d84c0 con 0x56547c8d47c0 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f066d036640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x56547c8d47c0 0x56547c95bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.515+0000 7f0667fff640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0650069eb0 0x7f065006c370 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x56547c975fd0 tx=0x7f065800c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066d036640 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x56547c8d47c0 0x56547c95bc60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066d036640 1 -- 172.21.15.72:0/2492631668 <== osd.3 v2:172.21.15.72:6808/4245662075 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f066800f6f0 con 0x56547c8d47c0 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x56547c8d47c0 msgr2=0x56547c95bc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x56547c8d47c0 0x56547c95bc60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0650069eb0 msgr2=0x7f065006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0650069eb0 0x7f065006c370 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x56547c975fd0 tx=0x7f065800c040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 msgr2=0x56547c969020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 --2- 172.21.15.72:0/2492631668 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56547c897240 0x56547c969020 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0660000c00 tx=0x7f0660002fb0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066d036640 1 -- 172.21.15.72:0/2492631668 reap_dead start 2024-03-06T07:23:05.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 shutdown_connections 2024-03-06T07:23:05.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 >> 172.21.15.72:0/2492631668 conn(0x56547c8c0bd0 msgr2=0x56547c8c3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 shutdown_connections 2024-03-06T07:23:05.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.516+0000 7f066ed32f40 1 -- 172.21.15.72:0/2492631668 wait complete. 2024-03-06T07:23:05.527 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:23:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.685+0000 7fd3434ed640 1 Processor -- start 2024-03-06T07:23:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.685+0000 7fd3434ed640 1 -- start start 2024-03-06T07:23:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd3434ed640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd33c0b4460 0x7fd33c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd3434ed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd33c161cf0 0x7fd33c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd3434ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0d22c0 con 0x7fd33c162970 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd3434ed640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c161b00 con 0x7fd33c161cf0 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd3434ed640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33c0cecc0 con 0x7fd33c0b4460 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd341a8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd33c161cf0 0x7fd33c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd341a8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd33c161cf0 0x7fd33c162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56082/0 (socket says 172.21.15.72:56082) 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.686+0000 7fd341a8a640 1 -- 172.21.15.72:0/204933204 learned_addr learned my addr 172.21.15.72:0/204933204 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341a8a640 1 -- 172.21.15.72:0/204933204 >> v2:172.21.15.146:3300/0 conn(0x7fd33c0b4460 msgr2=0x7fd33c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341289640 1 --2- 172.21.15.72:0/204933204 >> v2:172.21.15.146:3300/0 conn(0x7fd33c0b4460 0x7fd33c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341a8a640 1 --2- 172.21.15.72:0/204933204 >> v2:172.21.15.146:3300/0 conn(0x7fd33c0b4460 0x7fd33c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341a8a640 1 -- 172.21.15.72:0/204933204 >> v1:172.21.15.146:6789/0 conn(0x7fd33c162970 legacy=0x7fd33c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341a8a640 1 -- 172.21.15.72:0/204933204 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0d25a0 con 0x7fd33c161cf0 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341289640 1 --2- 172.21.15.72:0/204933204 >> v2:172.21.15.146:3300/0 conn(0x7fd33c0b4460 0x7fd33c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.687+0000 7fd341a8a640 1 --2- 172.21.15.72:0/204933204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c161cf0 0x7fd33c162180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd33c08ddd0 tx=0x7fd33402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75642e5932aef308 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd333fff640 1 -- 172.21.15.72:0/204933204 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33402ca80 con 0x7fd33c161cf0 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd333fff640 1 -- 172.21.15.72:0/204933204 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33402cc20 con 0x7fd33c161cf0 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd333fff640 1 -- 172.21.15.72:0/204933204 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3340343f0 con 0x7fd33c161cf0 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 -- 172.21.15.72:0/204933204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c161cf0 msgr2=0x7fd33c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 --2- 172.21.15.72:0/204933204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c161cf0 0x7fd33c162180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd33c08ddd0 tx=0x7fd33402bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 -- 172.21.15.72:0/204933204 shutdown_connections 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 --2- 172.21.15.72:0/204933204 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c161cf0 0x7fd33c162180 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 --2- 172.21.15.72:0/204933204 >> v2:172.21.15.146:3300/0 conn(0x7fd33c0b4460 0x7fd33c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 -- 172.21.15.72:0/204933204 >> 172.21.15.72:0/204933204 conn(0x7fd33c0d3ce0 msgr2=0x7fd33c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 -- 172.21.15.72:0/204933204 shutdown_connections 2024-03-06T07:23:05.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.688+0000 7fd3434ed640 1 -- 172.21.15.72:0/204933204 wait complete. 2024-03-06T07:23:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.690+0000 7fd3434ed640 1 Processor -- start 2024-03-06T07:23:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.690+0000 7fd3434ed640 1 -- start start 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd3434ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 0x7fd33c1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd3434ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd33c161cf0 0x7fd33c1f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd3434ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd33c162970 0x7fd33c0b7950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd3434ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0ce950 con 0x7fd33c0b4460 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd3434ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0d2b60 con 0x7fd33c162970 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd3434ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33c0b4840 con 0x7fd33c161cf0 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd341289640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 0x7fd33c1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd341289640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 0x7fd33c1f2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56094/0 (socket says 172.21.15.72:56094) 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.691+0000 7fd341289640 1 -- 172.21.15.72:0/1892523980 learned_addr learned my addr 172.21.15.72:0/1892523980 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd340a88640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd33c161cf0 0x7fd33c1f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341a8a640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd33c162970 0x7fd33c0b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341289640 1 -- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd33c161cf0 msgr2=0x7fd33c1f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341289640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd33c161cf0 0x7fd33c1f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341289640 1 -- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd33c162970 msgr2=0x7fd33c0b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341289640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd33c162970 0x7fd33c0b7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341289640 1 -- 172.21.15.72:0/1892523980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0cc520 con 0x7fd33c0b4460 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd341289640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 0x7fd33c1f2e10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd33800c820 tx=0x7fd33800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338015070 con 0x7fd33c0b4460 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3380079d0 con 0x7fd33c0b4460 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c0cc7c0 con 0x7fd33c0b4460 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338007d80 con 0x7fd33c0b4460 2024-03-06T07:23:05.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.692+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c0cca60 con 0x7fd33c0b4460 2024-03-06T07:23:05.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.693+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c04f330 con 0x7fd33c0b4460 2024-03-06T07:23:05.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.693+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd338018070 con 0x7fd33c0b4460 2024-03-06T07:23:05.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.694+0000 7fd331ffb640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd32006b4d0 0x7fd32006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:05.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.694+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd33808a070 con 0x7fd33c0b4460 2024-03-06T07:23:05.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.694+0000 7fd340a88640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd32006b4d0 0x7fd32006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:05.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.694+0000 7fd340a88640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd32006b4d0 0x7fd32006d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd33c162db0 tx=0x7fd32c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:05.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.702+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd338058ef0 con 0x7fd33c0b4460 2024-03-06T07:23:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.950+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fd33c06b180 con 0x7fd33c0b4460 2024-03-06T07:23:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.950+0000 7fd331ffb640 1 -- 172.21.15.72:0/1892523980 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1658 (secure 0 0 0) 0x7fd3380587f0 con 0x7fd33c0b4460 2024-03-06T07:23:05.952 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:05.952 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "total_avail_bytes": 772831985664, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_bytes": 262127616, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_bytes": 262127616, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_ratio": 0.00033906300086528063, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "num_osds": 8, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "num_per_pool_osds": 8, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "stats_by_class": { 2024-03-06T07:23:05.953 INFO:teuthology.orchestra.run.smithi072.stdout: "ssd": { 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "total_bytes": 773094113280, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "total_avail_bytes": 772831985664, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_bytes": 262127616, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_bytes": 262127616, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_ratio": 0.00033906300086528063 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "pools": [ 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "name": ".mgr", 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 1, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "stored": 590368, 2024-03-06T07:23:05.954 INFO:teuthology.orchestra.run.smithi072.stdout: "objects": 2, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "kb_used": 1160, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "bytes_used": 1187840, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "percent_used": 1.6179681097128196e-06, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "max_avail": 367077097472 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephfs_metadata", 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "stored": 3333346, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "objects": 23, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "kb_used": 6568, 2024-03-06T07:23:05.955 INFO:teuthology.orchestra.run.smithi072.stdout: "bytes_used": 6725632, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "percent_used": 9.1609772425726987e-06, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "max_avail": 367077097472 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephfs_data", 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "stored": 0, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "objects": 0, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "kb_used": 0, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "bytes_used": 0, 2024-03-06T07:23:05.956 INFO:teuthology.orchestra.run.smithi072.stdout: "percent_used": 0, 2024-03-06T07:23:05.957 INFO:teuthology.orchestra.run.smithi072.stdout: "max_avail": 367077097472 2024-03-06T07:23:05.957 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:05.957 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:05.957 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:23:05.957 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd32006b4d0 msgr2=0x7fd32006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd32006b4d0 0x7fd32006d990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd33c162db0 tx=0x7fd32c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 msgr2=0x7fd33c1f2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 0x7fd33c1f2e10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd33800c820 tx=0x7fd33800cce0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 shutdown_connections 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd32006b4d0 0x7fd32006d990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd33c162970 0x7fd33c0b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd33c161cf0 0x7fd33c1f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 --2- 172.21.15.72:0/1892523980 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd33c0b4460 0x7fd33c1f2e10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:05.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 >> 172.21.15.72:0/1892523980 conn(0x7fd33c0d3ce0 msgr2=0x7fd33c0cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:05.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 shutdown_connections 2024-03-06T07:23:05.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:05.952+0000 7fd3434ed640 1 -- 172.21.15.72:0/1892523980 wait complete. 2024-03-06T07:23:05.965 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:23:05.965 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:05.965 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:06.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:06.071 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:06.072 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:06.072 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:06.088 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:06.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.224+0000 7f8d0c488640 1 Processor -- start 2024-03-06T07:23:06.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.224+0000 7f8d0c488640 1 -- start start 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.225+0000 7f8d0c488640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d040d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.225+0000 7f8d0c488640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d040d0650 0x7f8d040d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.225+0000 7f8d0c488640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d0415e470 con 0x7f8d040d2920 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.225+0000 7f8d0c488640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d040d18f0 con 0x7f8d040d0650 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.225+0000 7f8d0c488640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040b3cf0 con 0x7f8d0415e090 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0aa25640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d040d0650 0x7f8d040d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d09a23640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8d040d2920 0x7f8d040b35e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45452/0 (socket says 172.21.15.72:45452) 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d09a23640 1 -- 172.21.15.72:0/1915353402 learned_addr learned my addr 172.21.15.72:0/1915353402 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0aa25640 1 -- 172.21.15.72:0/1915353402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 msgr2=0x7f8d040d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0a224640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d040d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0aa25640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d040d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d09222640 1 -- 172.21.15.72:0/1915353402 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401492988 0 0) 0x7f8d0415e470 con 0x7f8d040d2920 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0aa25640 1 -- 172.21.15.72:0/1915353402 >> v1:172.21.15.72:6789/0 conn(0x7f8d040d2920 legacy=0x7f8d040b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0aa25640 1 -- 172.21.15.72:0/1915353402 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d040d1b90 con 0x7f8d040d0650 2024-03-06T07:23:06.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.226+0000 7f8d0a224640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d040d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:06.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0aa25640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d0650 0x7f8d040d0ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009d90 tx=0x7f8cfc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4563b7581815ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d09222640 1 -- 172.21.15.72:0/1915353402 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc039070 con 0x7f8d040d0650 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d09222640 1 -- 172.21.15.72:0/1915353402 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cfc02cc50 con 0x7f8d040d0650 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d09222640 1 -- 172.21.15.72:0/1915353402 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cfc034440 con 0x7f8d040d0650 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 -- 172.21.15.72:0/1915353402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d0650 msgr2=0x7f8d040d0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d0650 0x7f8d040d0ae0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc009d90 tx=0x7f8cfc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 -- 172.21.15.72:0/1915353402 shutdown_connections 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d0650 0x7f8d040d0ae0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 --2- 172.21.15.72:0/1915353402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d040d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 -- 172.21.15.72:0/1915353402 >> 172.21.15.72:0/1915353402 conn(0x7f8d040d3c80 msgr2=0x7f8d040d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 -- 172.21.15.72:0/1915353402 shutdown_connections 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.227+0000 7f8d0c488640 1 -- 172.21.15.72:0/1915353402 wait complete. 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 Processor -- start 2024-03-06T07:23:06.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 -- start start 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040d0650 0x7f8d0415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 0x7f8d040b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d04151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040b2df0 con 0x7f8d040d2920 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040cee30 con 0x7f8d040d0650 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d0c488640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d040ce800 con 0x7f8d0415e090 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d09a23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 0x7f8d040b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d09a23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 0x7f8d040b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56110/0 (socket says 172.21.15.72:56110) 2024-03-06T07:23:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.229+0000 7f8d09a23640 1 -- 172.21.15.72:0/2461715801 learned_addr learned my addr 172.21.15.72:0/2461715801 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d0aa25640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d04151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d0a224640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040d0650 0x7f8d0415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d09a23640 1 -- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 msgr2=0x7f8d04151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d09a23640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d04151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d09a23640 1 -- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040d0650 msgr2=0x7f8d0415bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d09a23640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040d0650 0x7f8d0415bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d09a23640 1 -- 172.21.15.72:0/2461715801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d041f7be0 con 0x7f8d040d2920 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d0aa25640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d04151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d0a224640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040d0650 0x7f8d0415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d09a23640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 0x7f8d040b68a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400c460 tx=0x7f8cf400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4018070 con 0x7f8d040d2920 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf4007500 con 0x7f8d040d2920 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d041f7060 con 0x7f8d040d2920 2024-03-06T07:23:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4004a20 con 0x7f8d040d2920 2024-03-06T07:23:06.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.230+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d041f7620 con 0x7f8d040d2920 2024-03-06T07:23:06.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.231+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0404f330 con 0x7f8d040d2920 2024-03-06T07:23:06.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.232+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8cf401b070 con 0x7f8d040d2920 2024-03-06T07:23:06.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.232+0000 7f8cfaffd640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8ce806b4d0 0x7f8ce806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.232+0000 7f8d0a224640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8ce806b4d0 0x7f8ce806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.232+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8cf4089440 con 0x7f8d040d2920 2024-03-06T07:23:06.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.233+0000 7f8d0a224640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8ce806b4d0 0x7f8ce806d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8d040d28f0 tx=0x7f8d00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.237+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8cf4058350 con 0x7f8d040d2920 2024-03-06T07:23:06.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.448+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d040d1230 con 0x7f8d040d2920 2024-03-06T07:23:06.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.450+0000 7f8cfaffd640 1 -- 172.21.15.72:0/2461715801 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f8cf4057c50 con 0x7f8d040d2920 2024-03-06T07:23:06.452 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:06.452 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":34,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:47.667116+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:06.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8ce806b4d0 msgr2=0x7f8ce806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8ce806b4d0 0x7f8ce806d990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8d040d28f0 tx=0x7f8d00008040 comp rx=0 tx=0).stop 2024-03-06T07:23:06.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 msgr2=0x7f8d040b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 0x7f8d040b68a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400c460 tx=0x7f8cf400c920 comp rx=0 tx=0).stop 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 shutdown_connections 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8ce806b4d0 0x7f8ce806d990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d0415e090 0x7f8d04151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d040d2920 0x7f8d040b68a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 --2- 172.21.15.72:0/2461715801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d040d0650 0x7f8d0415bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 >> 172.21.15.72:0/2461715801 conn(0x7f8d040d3c80 msgr2=0x7f8d040b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 shutdown_connections 2024-03-06T07:23:06.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.452+0000 7f8d0c488640 1 -- 172.21.15.72:0/2461715801 wait complete. 2024-03-06T07:23:06.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:44.323650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:44.983501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:06.465 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:06.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 Processor -- start 2024-03-06T07:23:06.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 -- start start 2024-03-06T07:23:06.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1de00d10c0 0x7f1de00d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1de00d14a0 0x7f1de015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de015eb00 0x7f1de00cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de015c050 con 0x7f1de00d14a0 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de00d0060 con 0x7f1de00d10c0 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.567+0000 7f1de5135640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00cfa50 con 0x7f1de015eb00 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1de00d14a0 0x7f1de015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1de00d14a0 0x7f1de015e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56114/0 (socket says 172.21.15.72:56114) 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 -- 172.21.15.72:0/4048104885 learned_addr learned my addr 172.21.15.72:0/4048104885 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1ddf59e640 1 --2- 172.21.15.72:0/4048104885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de015eb00 0x7f1de00cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dded9d640 1 --2- 172.21.15.72:0/4048104885 >> v2:172.21.15.146:3300/0 conn(0x7f1de00d10c0 0x7f1de00d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 -- 172.21.15.72:0/4048104885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de015eb00 msgr2=0x7f1de00cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 --2- 172.21.15.72:0/4048104885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de015eb00 0x7f1de00cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 -- 172.21.15.72:0/4048104885 >> v2:172.21.15.146:3300/0 conn(0x7f1de00d10c0 msgr2=0x7f1de00d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 --2- 172.21.15.72:0/4048104885 >> v2:172.21.15.146:3300/0 conn(0x7f1de00d10c0 0x7f1de00d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.568+0000 7f1dd7fff640 1 -- 172.21.15.72:0/4048104885 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00d0a30 con 0x7f1de00d14a0 2024-03-06T07:23:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.569+0000 7f1dd7fff640 1 --2- 172.21.15.72:0/4048104885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de00d14a0 0x7f1de015e5c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1de00718d0 tx=0x7f1dc802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfa42096625c7d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.569+0000 7f1dde59c640 1 -- 172.21.15.72:0/4048104885 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8035020 con 0x7f1de00d14a0 2024-03-06T07:23:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.569+0000 7f1dde59c640 1 -- 172.21.15.72:0/4048104885 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc8031870 con 0x7f1de00d14a0 2024-03-06T07:23:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.569+0000 7f1dde59c640 1 -- 172.21.15.72:0/4048104885 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8031c40 con 0x7f1de00d14a0 2024-03-06T07:23:06.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 -- 172.21.15.72:0/4048104885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de00d14a0 msgr2=0x7f1de015e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 --2- 172.21.15.72:0/4048104885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de00d14a0 0x7f1de015e5c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1de00718d0 tx=0x7f1dc802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 -- 172.21.15.72:0/4048104885 shutdown_connections 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 --2- 172.21.15.72:0/4048104885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de015eb00 0x7f1de00cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 --2- 172.21.15.72:0/4048104885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de00d14a0 0x7f1de015e5c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 --2- 172.21.15.72:0/4048104885 >> v2:172.21.15.146:3300/0 conn(0x7f1de00d10c0 0x7f1de00d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 -- 172.21.15.72:0/4048104885 >> 172.21.15.72:0/4048104885 conn(0x7f1de00b3800 msgr2=0x7f1de00b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 -- 172.21.15.72:0/4048104885 shutdown_connections 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.570+0000 7f1de5135640 1 -- 172.21.15.72:0/4048104885 wait complete. 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.571+0000 7f1de5135640 1 Processor -- start 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.571+0000 7f1de5135640 1 -- start start 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1de5135640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00d10c0 0x7f1de0152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1de5135640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de00d14a0 0x7f1de00ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1de5135640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 0x7f1de00afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1de5135640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015f440 con 0x7f1de015eb00 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1de5135640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de015f160 con 0x7f1de00d10c0 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1de5135640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de00cfcf0 con 0x7f1de00d14a0 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 0x7f1de00afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 0x7f1de00afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56124/0 (socket says 172.21.15.72:56124) 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 -- 172.21.15.72:0/3383646323 learned_addr learned my addr 172.21.15.72:0/3383646323 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 -- 172.21.15.72:0/3383646323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de00d14a0 msgr2=0x7f1de00ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de00d14a0 0x7f1de00ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 -- 172.21.15.72:0/3383646323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00d10c0 msgr2=0x7f1de0152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00d10c0 0x7f1de0152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 -- 172.21.15.72:0/3383646323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de009beb0 con 0x7f1de015eb00 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1ddf59e640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 0x7f1de00afd70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800cd70 tx=0x7f1dd800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.572+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8003b90 con 0x7f1de015eb00 2024-03-06T07:23:06.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.573+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd8003d30 con 0x7f1de015eb00 2024-03-06T07:23:06.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.573+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de00cd0d0 con 0x7f1de015eb00 2024-03-06T07:23:06.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.573+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8004240 con 0x7f1de015eb00 2024-03-06T07:23:06.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.573+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de00cdb70 con 0x7f1de015eb00 2024-03-06T07:23:06.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.574+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de006eef0 con 0x7f1de015eb00 2024-03-06T07:23:06.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.574+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1dd80043e0 con 0x7f1de015eb00 2024-03-06T07:23:06.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.574+0000 7f1dd77fe640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dc006b4d0 0x7f1dc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.575+0000 7f1dded9d640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dc006b4d0 0x7f1dc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.575+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1dd808acf0 con 0x7f1de015eb00 2024-03-06T07:23:06.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.575+0000 7f1dded9d640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dc006b4d0 0x7f1dc006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1de015bbc0 tx=0x7f1dd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.579+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1de006eef0 con 0x7f1de015eb00 2024-03-06T07:23:06.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.807+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1de009fe60 con 0x7f1de015eb00 2024-03-06T07:23:06.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.807+0000 7f1dd77fe640 1 -- 172.21.15.72:0/3383646323 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1dd808f070 con 0x7f1de015eb00 2024-03-06T07:23:06.809 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dc006b4d0 msgr2=0x7f1dc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dc006b4d0 0x7f1dc006d990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1de015bbc0 tx=0x7f1dd0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 msgr2=0x7f1de00afd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 0x7f1de00afd70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800cd70 tx=0x7f1dd800a660 comp rx=0 tx=0).stop 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 shutdown_connections 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1dc006b4d0 0x7f1dc006d990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1de015eb00 0x7f1de00afd70 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de00d14a0 0x7f1de00ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 --2- 172.21.15.72:0/3383646323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de00d10c0 0x7f1de0152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 >> 172.21.15.72:0/3383646323 conn(0x7f1de00b3800 msgr2=0x7f1de00cae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 shutdown_connections 2024-03-06T07:23:06.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.809+0000 7f1de5135640 1 -- 172.21.15.72:0/3383646323 wait complete. 2024-03-06T07:23:06.825 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:06.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.931+0000 7f815e0f4640 1 Processor -- start 2024-03-06T07:23:06.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.931+0000 7f815e0f4640 1 -- start start 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f815e0f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f81581614d0 0x7f81580d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f815e0f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f81580d24b0 0x7f81580b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f815e0f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81581618b0 con 0x7f81580d1910 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f815e0f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81580cf400 con 0x7f81581614d0 2024-03-06T07:23:06.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f815e0f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81580cf6a0 con 0x7f81580d24b0 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f81577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f81581614d0 0x7f81580d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f814effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f81580d1910 0x7f81580d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45462/0 (socket says 172.21.15.72:45462) 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f814effd640 1 -- 172.21.15.72:0/3894315443 learned_addr learned my addr 172.21.15.72:0/3894315443 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f81577fe640 1 --2- 172.21.15.72:0/3894315443 >> v2:172.21.15.72:3300/0 conn(0x7f81581614d0 0x7f81580d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56128/0 (socket says 172.21.15.72:56128) 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f81577fe640 1 -- 172.21.15.72:0/3894315443 >> v2:172.21.15.146:3300/0 conn(0x7f81580d24b0 msgr2=0x7f81580b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f81577fe640 1 --2- 172.21.15.72:0/3894315443 >> v2:172.21.15.146:3300/0 conn(0x7f81580d24b0 0x7f81580b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f81577fe640 1 -- 172.21.15.72:0/3894315443 >> v1:172.21.15.72:6789/0 conn(0x7f81580d1910 legacy=0x7f81580d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f81577fe640 1 -- 172.21.15.72:0/3894315443 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81580cf980 con 0x7f81581614d0 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.932+0000 7f8156ffd640 1 -- 172.21.15.72:0/3894315443 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438835359 0 0) 0x7f81581618b0 con 0x7f81580d1910 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.933+0000 7f81577fe640 1 --2- 172.21.15.72:0/3894315443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81581614d0 0x7f81580d4860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81580b2a70 tx=0x7f814802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52042c4dd7e21346 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.933+0000 7f8156ffd640 1 -- 172.21.15.72:0/3894315443 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148035020 con 0x7f81581614d0 2024-03-06T07:23:06.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.933+0000 7f8156ffd640 1 -- 172.21.15.72:0/3894315443 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8148031830 con 0x7f81581614d0 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.933+0000 7f8156ffd640 1 -- 172.21.15.72:0/3894315443 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148031c00 con 0x7f81581614d0 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 -- 172.21.15.72:0/3894315443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81581614d0 msgr2=0x7f81580d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 --2- 172.21.15.72:0/3894315443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81581614d0 0x7f81580d4860 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81580b2a70 tx=0x7f814802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 -- 172.21.15.72:0/3894315443 shutdown_connections 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 --2- 172.21.15.72:0/3894315443 >> v2:172.21.15.146:3300/0 conn(0x7f81580d24b0 0x7f81580b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 --2- 172.21.15.72:0/3894315443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81581614d0 0x7f81580d4860 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 -- 172.21.15.72:0/3894315443 >> 172.21.15.72:0/3894315443 conn(0x7f81580d3d40 msgr2=0x7f81580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 -- 172.21.15.72:0/3894315443 shutdown_connections 2024-03-06T07:23:06.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.934+0000 7f815e0f4640 1 -- 172.21.15.72:0/3894315443 wait complete. 2024-03-06T07:23:06.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.935+0000 7f815e0f4640 1 Processor -- start 2024-03-06T07:23:06.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.935+0000 7f815e0f4640 1 -- start start 2024-03-06T07:23:06.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f815e0f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 0x7f81580b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f815e0f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81580d24b0 0x7f81580cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f815e0f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81581614d0 0x7f815815f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f815e0f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81580b1370 con 0x7f81580d1910 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f815e0f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81580b0910 con 0x7f81581614d0 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f815e0f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81580b14f0 con 0x7f81580d24b0 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 0x7f81580b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 0x7f81580b6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56142/0 (socket says 172.21.15.72:56142) 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 -- 172.21.15.72:0/1557892043 learned_addr learned my addr 172.21.15.72:0/1557892043 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 -- 172.21.15.72:0/1557892043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81580d24b0 msgr2=0x7f81580cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81580d24b0 0x7f81580cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 -- 172.21.15.72:0/1557892043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81581614d0 msgr2=0x7f815815f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81581614d0 0x7f815815f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:06.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 -- 172.21.15.72:0/1557892043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81580b6ff0 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f81577fe640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 0x7f81580b6930 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8148034040 tx=0x7f81480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.936+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148004230 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.937+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81480043d0 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.937+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81580b72d0 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.937+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814803b8b0 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.937+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81580b7890 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.938+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8158068ff0 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.938+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f814804c060 con 0x7f81580d1910 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.938+0000 7f8154ff9640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f813806b500 0x7f813806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:06.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.938+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81480abd30 con 0x7f81580d1910 2024-03-06T07:23:06.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.938+0000 7f814effd640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f813806b500 0x7f813806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:06.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.939+0000 7f814effd640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f813806b500 0x7f813806d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8140007fd0 tx=0x7f8140007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:06.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:06.943+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8148040070 con 0x7f81580d1910 2024-03-06T07:23:07.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.115+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81580b5580 con 0x7f81580d1910 2024-03-06T07:23:07.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.116+0000 7f8154ff9640 1 -- 172.21.15.72:0/1557892043 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f814807abb0 con 0x7f81580d1910 2024-03-06T07:23:07.118 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:23:07.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f813806b500 msgr2=0x7f813806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f813806b500 0x7f813806d9c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8140007fd0 tx=0x7f8140007920 comp rx=0 tx=0).stop 2024-03-06T07:23:07.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 msgr2=0x7f81580b6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 0x7f81580b6930 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8148034040 tx=0x7f81480046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 shutdown_connections 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f813806b500 0x7f813806d9c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81581614d0 0x7f815815f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81580d24b0 0x7f81580cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 --2- 172.21.15.72:0/1557892043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f81580d1910 0x7f81580b6930 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 >> 172.21.15.72:0/1557892043 conn(0x7f81580d3d40 msgr2=0x7f81580b5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 shutdown_connections 2024-03-06T07:23:07.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.118+0000 7f815e0f4640 1 -- 172.21.15.72:0/1557892043 wait complete. 2024-03-06T07:23:07.131 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:07.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.233+0000 7f0c38a15640 1 Processor -- start 2024-03-06T07:23:07.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.233+0000 7f0c38a15640 1 -- start start 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c38a15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3415bbb0 0x7f0c3406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c38a15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c340cea90 con 0x7f0c3406b4f0 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c38a15640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c34161bc0 con 0x7f0c3415ee10 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c38a15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c34161970 con 0x7f0c3415bbb0 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c31d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0c3406b4f0 0x7f0c3415e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45474/0 (socket says 172.21.15.72:45474) 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c31d9b640 1 -- 172.21.15.72:0/2828414670 learned_addr learned my addr 172.21.15.72:0/2828414670 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321663607 0 0) 0x7f0c28001110 con 0x7f0c3406b4f0 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c1c003520 con 0x7f0c3406b4f0 2024-03-06T07:23:07.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3259c640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3415bbb0 0x7f0c3406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677778732 0 0) 0x7f0c1c003520 con 0x7f0c3406b4f0 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3415bbb0 msgr2=0x7f0c3406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3415bbb0 0x7f0c3406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 >> v1:172.21.15.146:6789/0 conn(0x7f0c3415ee10 legacy=0x7f0c34071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34161ea0 con 0x7f0c3406b4f0 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f0c28001840 con 0x7f0c3406b4f0 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 >> v1:172.21.15.72:6789/0 conn(0x7f0c3406b4f0 legacy=0x7f0c3415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c0039e0 0x7f0c1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c1c004340 0x7f0c1c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c1c005020 0x7f0c1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c34161970 con 0x7f0c1c004340 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c00bf00 con 0x7f0c1c005020 2024-03-06T07:23:07.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.234+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1c00c0b0 con 0x7f0c1c0039e0 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c1c004340 0x7f0c1c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c3259c640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c0039e0 0x7f0c1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c32d9d640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c1c005020 0x7f0c1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c31d9b640 1 -- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c0039e0 msgr2=0x7f0c1c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c0039e0 0x7f0c1c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c31d9b640 1 -- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c1c005020 msgr2=0x7f0c1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c1c005020 0x7f0c1c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c31d9b640 1 -- 172.21.15.72:0/2828414670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c1c003520 con 0x7f0c1c004340 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.235+0000 7f0c32d9d640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c1c005020 0x7f0c1c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c1c004340 0x7f0c1c004820 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800c4c0 tx=0x7f0c1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f6158ea82ade43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2800c0e0 con 0x7f0c1c004340 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28030070 con 0x7f0c1c004340 2024-03-06T07:23:07.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c3159a640 1 -- 172.21.15.72:0/2828414670 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280058d0 con 0x7f0c1c004340 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c38a15640 1 -- 172.21.15.72:0/2828414670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c1c004340 msgr2=0x7f0c1c004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c38a15640 1 --2- 172.21.15.72:0/2828414670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c1c004340 0x7f0c1c004820 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0c2800c4c0 tx=0x7f0c1c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c32d9d640 1 -- 172.21.15.72:0/2828414670 reap_dead start 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c38a15640 1 -- 172.21.15.72:0/2828414670 shutdown_connections 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c38a15640 1 -- 172.21.15.72:0/2828414670 >> 172.21.15.72:0/2828414670 conn(0x7f0c340b2660 msgr2=0x7f0c340b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c38a15640 1 -- 172.21.15.72:0/2828414670 shutdown_connections 2024-03-06T07:23:07.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.236+0000 7f0c38a15640 1 -- 172.21.15.72:0/2828414670 wait complete. 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 Processor -- start 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 -- start start 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c004340 0x7f0c3414c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 0x7f0c34154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34159950 0x7f0c341553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3406afb0 con 0x7f0c341594d0 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c341624f0 con 0x7f0c34159950 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c38a15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3406b130 con 0x7f0c1c004340 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c31d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 0x7f0c34154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.238+0000 7f0c31d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 0x7f0c34154e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56160/0 (socket says 172.21.15.72:56160) 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 -- 172.21.15.72:0/3205928202 learned_addr learned my addr 172.21.15.72:0/3205928202 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c32d9d640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34159950 0x7f0c341553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c3259c640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c004340 0x7f0c3414c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 -- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c004340 msgr2=0x7f0c3414c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c004340 0x7f0c3414c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 -- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34159950 msgr2=0x7f0c341553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34159950 0x7f0c341553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 -- 172.21.15.72:0/3205928202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3414cf00 con 0x7f0c341594d0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c31d9b640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 0x7f0c34154e80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0c28001b40 tx=0x7f0c28001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280049e0 con 0x7f0c341594d0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28004b80 con 0x7f0c341594d0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28007070 con 0x7f0c341594d0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3415fcc0 con 0x7f0c341594d0 2024-03-06T07:23:07.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.239+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c34160280 con 0x7f0c341594d0 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.240+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c340733b0 con 0x7f0c341594d0 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.241+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c28007210 con 0x7f0c341594d0 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.241+0000 7f0c22ffd640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c1406b4d0 0x7f0c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.241+0000 7f0c3259c640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c1406b4d0 0x7f0c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.241+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c280abe60 con 0x7f0c341594d0 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.242+0000 7f0c3259c640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c1406b4d0 0x7f0c1406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009bc0 tx=0x7f0c24009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.245+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c340cea90 con 0x7f0c341594d0 2024-03-06T07:23:07.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.467+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0c34068ff0 con 0x7f0c341594d0 2024-03-06T07:23:07.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.468+0000 7f0c22ffd640 1 -- 172.21.15.72:0/3205928202 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0c2807ae30 con 0x7f0c341594d0 2024-03-06T07:23:07.470 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c1406b4d0 msgr2=0x7f0c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c1406b4d0 0x7f0c1406d990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009bc0 tx=0x7f0c24009190 comp rx=0 tx=0).stop 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 msgr2=0x7f0c34154e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 0x7f0c34154e80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0c28001b40 tx=0x7f0c28001c10 comp rx=0 tx=0).stop 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 shutdown_connections 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c1406b4d0 0x7f0c1406d990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34159950 0x7f0c341553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c341594d0 0x7f0c34154e80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 --2- 172.21.15.72:0/3205928202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c1c004340 0x7f0c3414c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.469+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 >> 172.21.15.72:0/3205928202 conn(0x7f0c340b2660 msgr2=0x7f0c340b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.470+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 shutdown_connections 2024-03-06T07:23:07.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.470+0000 7f0c38a15640 1 -- 172.21.15.72:0/3205928202 wait complete. 2024-03-06T07:23:07.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:07.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.582+0000 7fe718fbf640 1 Processor -- start 2024-03-06T07:23:07.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.582+0000 7fe718fbf640 1 -- start start 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe718fbf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7141614a0 0x7fe71406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe718fbf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 0x7fe7140b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe718fbf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7140af4b0 con 0x7fe7140b1850 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe718fbf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7140af790 con 0x7fe7141614a0 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe718fbf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140afa70 con 0x7fe7140b0270 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe712d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe7140b1850 0x7fe71406c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45490/0 (socket says 172.21.15.72:45490) 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe712d9d640 1 -- 172.21.15.72:0/3771298989 learned_addr learned my addr 172.21.15.72:0/3771298989 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525663559 0 0) 0x7fe7140af4b0 con 0x7fe7140b1850 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71259c640 1 --2- 172.21.15.72:0/3771298989 >> v2:172.21.15.146:3300/0 conn(0x7fe7141614a0 0x7fe71406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe711d9b640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 0x7fe7140b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6fc003520 con 0x7fe7140b1850 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570234787 0 0) 0x7fe6fc003520 con 0x7fe7140b1850 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 msgr2=0x7fe7140b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 0x7fe7140b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 >> v2:172.21.15.146:3300/0 conn(0x7fe7141614a0 msgr2=0x7fe71406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 --2- 172.21.15.72:0/3771298989 >> v2:172.21.15.146:3300/0 conn(0x7fe7141614a0 0x7fe71406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7140afd50 con 0x7fe7140b1850 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe70c0019f0 con 0x7fe7140b1850 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.583+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 >> v1:172.21.15.72:6789/0 conn(0x7fe7140b1850 legacy=0x7fe71406c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71159a640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6fc0039b0 0x7fe6fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71159a640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 0x7fe6fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71159a640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc004d40 0x7fe6fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140af790 con 0x7fe6fc0039b0 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140afa70 con 0x7fe6fc004d40 2024-03-06T07:23:07.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc00b9f0 con 0x7fe7140b0270 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71259c640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6fc0039b0 0x7fe6fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe712d9d640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc004d40 0x7fe6fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe711d9b640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 0x7fe6fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71259c640 1 -- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 msgr2=0x7fe6fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71259c640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7140b0270 0x7fe6fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71259c640 1 -- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc004d40 msgr2=0x7fe6fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71259c640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc004d40 0x7fe6fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.584+0000 7fe71259c640 1 -- 172.21.15.72:0/3771298989 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc003520 con 0x7fe6fc0039b0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe71259c640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6fc0039b0 0x7fe6fc003dd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe704009ac0 tx=0x7fe6fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac14d52ce71e63b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704039070 con 0x7fe6fc0039b0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70402c710 con 0x7fe6fc0039b0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe71159a640 1 -- 172.21.15.72:0/3771298989 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704009690 con 0x7fe6fc0039b0 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe718fbf640 1 -- 172.21.15.72:0/3771298989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6fc0039b0 msgr2=0x7fe6fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe718fbf640 1 --2- 172.21.15.72:0/3771298989 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6fc0039b0 0x7fe6fc003dd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe704009ac0 tx=0x7fe6fc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:07.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe712d9d640 1 -- 172.21.15.72:0/3771298989 reap_dead start 2024-03-06T07:23:07.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe718fbf640 1 -- 172.21.15.72:0/3771298989 shutdown_connections 2024-03-06T07:23:07.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe718fbf640 1 -- 172.21.15.72:0/3771298989 >> 172.21.15.72:0/3771298989 conn(0x7fe71409a9f0 msgr2=0x7fe71409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe718fbf640 1 -- 172.21.15.72:0/3771298989 shutdown_connections 2024-03-06T07:23:07.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.585+0000 7fe718fbf640 1 -- 172.21.15.72:0/3771298989 wait complete. 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.587+0000 7fe718fbf640 1 Processor -- start 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.587+0000 7fe718fbf640 1 -- start start 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc0039b0 0x7fe71409e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe71409bc40 0x7fe714158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 0x7fe7141f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140b6520 con 0x7fe7141f70b0 2024-03-06T07:23:07.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140b06b0 con 0x7fe6fc0039b0 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7140b1c80 con 0x7fe71409bc40 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 0x7fe7141f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 0x7fe7141f7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56170/0 (socket says 172.21.15.72:56170) 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 -- 172.21.15.72:0/2884217093 learned_addr learned my addr 172.21.15.72:0/2884217093 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe71259c640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc0039b0 0x7fe71409e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe711d9b640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe71409bc40 0x7fe714158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 -- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe71409bc40 msgr2=0x7fe714158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe71409bc40 0x7fe714158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 -- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc0039b0 msgr2=0x7fe71409e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc0039b0 0x7fe71409e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 -- 172.21.15.72:0/2884217093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71409e810 con 0x7fe7141f70b0 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe71259c640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc0039b0 0x7fe71409e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe711d9b640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe71409bc40 0x7fe714158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:07.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe712d9d640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 0x7fe7141f7530 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe7140721d0 tx=0x7fe70c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00edc0 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70c018070 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71409eaf0 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70c013660 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.588+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71409f0b0 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.589+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7140b4990 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.590+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe70c013800 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.591+0000 7fe702ffd640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe6f406b500 0x7fe6f406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.591+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe70c08a5e0 con 0x7fe7141f70b0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.591+0000 7fe71259c640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe6f406b500 0x7fe6f406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:07.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.591+0000 7fe71259c640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe6f406b500 0x7fe6f406d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe704036000 tx=0x7fe704030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:07.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.595+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7140b4990 con 0x7fe7141f70b0 2024-03-06T07:23:07.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.784+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe71404f330 con 0x7fe7141f70b0 2024-03-06T07:23:07.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.785+0000 7fe702ffd640 1 -- 172.21.15.72:0/2884217093 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v34) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe70c059460 con 0x7fe7141f70b0 2024-03-06T07:23:07.787 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:23:07.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe6f406b500 msgr2=0x7fe6f406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe6f406b500 0x7fe6f406d9c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe704036000 tx=0x7fe704030000 comp rx=0 tx=0).stop 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 msgr2=0x7fe7141f7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 0x7fe7141f7530 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe7140721d0 tx=0x7fe70c00e530 comp rx=0 tx=0).stop 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 shutdown_connections 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe6f406b500 0x7fe6f406d9c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7141f70b0 0x7fe7141f7530 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe71409bc40 0x7fe714158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 --2- 172.21.15.72:0/2884217093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6fc0039b0 0x7fe71409e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 >> 172.21.15.72:0/2884217093 conn(0x7fe71409a9f0 msgr2=0x7fe7140b3830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 shutdown_connections 2024-03-06T07:23:07.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:07.787+0000 7fe718fbf640 1 -- 172.21.15.72:0/2884217093 wait complete. 2024-03-06T07:23:07.800 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:23:07.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.800 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:07.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[52107]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:07.827 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:23:07.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.827 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:07.892 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:07.893 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:07.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:07.910 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:23:07.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.910 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:07.967 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:07.967 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:07.967 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:07.982 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:07.982 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:23:07.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:07.982 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:08.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:08.038 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:08.039 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:08.039 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:08.055 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:08.055 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:23:08.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:08.055 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:08.110 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:08.110 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:08.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 Processor -- start 2024-03-06T07:23:08.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 -- start start 2024-03-06T07:23:08.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78bc15dfe0 0x7f78bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f78bc0d27f0 0x7f78bc0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc0d2c50 con 0x7f78bc0d27f0 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc0b31e0 con 0x7f78bc15dfe0 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.209+0000 7f78c10eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0b3d60 con 0x7f78bc0cf830 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f78bc0d27f0 0x7f78bc0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f78bc0d27f0 0x7f78bc0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56184/0 (socket says 172.21.15.72:56184) 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 -- 172.21.15.72:0/2300969586 learned_addr learned my addr 172.21.15.72:0/2300969586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bad9d640 1 --2- 172.21.15.72:0/2300969586 >> v2:172.21.15.146:3300/0 conn(0x7f78bc15dfe0 0x7f78bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78b3fff640 1 --2- 172.21.15.72:0/2300969586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 -- 172.21.15.72:0/2300969586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 msgr2=0x7f78bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 --2- 172.21.15.72:0/2300969586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 -- 172.21.15.72:0/2300969586 >> v2:172.21.15.146:3300/0 conn(0x7f78bc15dfe0 msgr2=0x7f78bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 --2- 172.21.15.72:0/2300969586 >> v2:172.21.15.146:3300/0 conn(0x7f78bc15dfe0 0x7f78bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bb59e640 1 -- 172.21.15.72:0/2300969586 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc0b3430 con 0x7f78bc0d27f0 2024-03-06T07:23:08.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.210+0000 7f78bad9d640 1 --2- 172.21.15.72:0/2300969586 >> v2:172.21.15.146:3300/0 conn(0x7f78bc15dfe0 0x7f78bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.211+0000 7f78bb59e640 1 --2- 172.21.15.72:0/2300969586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc0d0300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f78bc067990 tx=0x7f78b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1d8a336843a316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.211+0000 7f78ba59c640 1 -- 172.21.15.72:0/2300969586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b402ca70 con 0x7f78bc0d27f0 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.211+0000 7f78ba59c640 1 -- 172.21.15.72:0/2300969586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b402cc10 con 0x7f78bc0d27f0 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.211+0000 7f78ba59c640 1 -- 172.21.15.72:0/2300969586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b40343f0 con 0x7f78bc0d27f0 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 -- 172.21.15.72:0/2300969586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 msgr2=0x7f78bc0d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 --2- 172.21.15.72:0/2300969586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc0d0300 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f78bc067990 tx=0x7f78b402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 -- 172.21.15.72:0/2300969586 shutdown_connections 2024-03-06T07:23:08.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 --2- 172.21.15.72:0/2300969586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc0d0300 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 --2- 172.21.15.72:0/2300969586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 --2- 172.21.15.72:0/2300969586 >> v2:172.21.15.146:3300/0 conn(0x7f78bc15dfe0 0x7f78bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 -- 172.21.15.72:0/2300969586 >> 172.21.15.72:0/2300969586 conn(0x7f78bc0d3cd0 msgr2=0x7f78bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 -- 172.21.15.72:0/2300969586 shutdown_connections 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.212+0000 7f78c10eb640 1 -- 172.21.15.72:0/2300969586 wait complete. 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.214+0000 7f78c10eb640 1 Processor -- start 2024-03-06T07:23:08.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.214+0000 7f78c10eb640 1 -- start start 2024-03-06T07:23:08.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78c10eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0b74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78c10eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc14cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78c10eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78bc15dfe0 0x7f78bc160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78c10eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0d1c70 con 0x7f78bc0d27f0 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78c10eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0b39f0 con 0x7f78bc15dfe0 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78c10eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc0d1df0 con 0x7f78bc0cf830 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78b3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc14cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78b3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc14cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56198/0 (socket says 172.21.15.72:56198) 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.215+0000 7f78b3fff640 1 -- 172.21.15.72:0/3891299873 learned_addr learned my addr 172.21.15.72:0/3891299873 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78bb59e640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78bc15dfe0 0x7f78bc160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78bad9d640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0b74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b3fff640 1 -- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 msgr2=0x7f78bc0b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b3fff640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0b74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b3fff640 1 -- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78bc15dfe0 msgr2=0x7f78bc160350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b3fff640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78bc15dfe0 0x7f78bc160350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b3fff640 1 -- 172.21.15.72:0/3891299873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc15fa40 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78bb59e640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78bc15dfe0 0x7f78bc160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b3fff640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc14cfd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f78a400c7c0 tx=0x7f78a400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4015070 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4004220 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc0caf40 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4007620 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.216+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc14d510 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.217+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78bc0d3740 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.218+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f78a40077c0 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.218+0000 7f78b37fe640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f789c06b500 0x7f789c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.218+0000 7f78bad9d640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f789c06b500 0x7f789c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.218+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a4089fb0 con 0x7f78bc0d27f0 2024-03-06T07:23:08.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.219+0000 7f78bad9d640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f789c06b500 0x7f789c06d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0cd390 tx=0x7f78ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.222+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f78bc0d3740 con 0x7f78bc0d27f0 2024-03-06T07:23:08.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.434+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78bc0b5430 con 0x7f78bc0d27f0 2024-03-06T07:23:08.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.435+0000 7f78b37fe640 1 -- 172.21.15.72:0/3891299873 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f78bc0b5430 con 0x7f78bc0d27f0 2024-03-06T07:23:08.437 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:08.437 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:23:08.437 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:23:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f789c06b500 msgr2=0x7f789c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f789c06b500 0x7f789c06d9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f78bc0cd390 tx=0x7f78ac008040 comp rx=0 tx=0).stop 2024-03-06T07:23:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 msgr2=0x7f78bc14cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc14cfd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f78a400c7c0 tx=0x7f78a400cc80 comp rx=0 tx=0).stop 2024-03-06T07:23:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 shutdown_connections 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f789c06b500 0x7f789c06d9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78bc15dfe0 0x7f78bc160350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78bc0d27f0 0x7f78bc14cfd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 --2- 172.21.15.72:0/3891299873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78bc0cf830 0x7f78bc0b74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 >> 172.21.15.72:0/3891299873 conn(0x7f78bc0d3cd0 msgr2=0x7f78bc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 shutdown_connections 2024-03-06T07:23:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.437+0000 7f78c10eb640 1 -- 172.21.15.72:0/3891299873 wait complete. 2024-03-06T07:23:08.450 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:08.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.548+0000 7fb1e493e640 1 Processor -- start 2024-03-06T07:23:08.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.548+0000 7fb1e493e640 1 -- start start 2024-03-06T07:23:08.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.549+0000 7fb1e493e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1e00afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.549+0000 7fb1e493e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e00b4e30 con 0x7fb1e015ab00 2024-03-06T07:23:08.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.549+0000 7fb1e493e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e00b5110 con 0x7fb1e015bb10 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.549+0000 7fb1e493e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00b0060 con 0x7fb1e00af6c0 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.549+0000 7fb1de59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb1e015ab00 0x7fb1e00b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45504/0 (socket says 172.21.15.72:45504) 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.549+0000 7fb1de59c640 1 -- 172.21.15.72:0/1821352539 learned_addr learned my addr 172.21.15.72:0/1821352539 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1d7fff640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1e00afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054798876 0 0) 0x7fb1e00b4e30 con 0x7fb1e015ab00 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1cc003520 con 0x7fb1e015ab00 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1600135969 0 0) 0x7fb1cc003520 con 0x7fb1e015ab00 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 msgr2=0x7fb1e00afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1e00afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 >> v1:172.21.15.146:6789/0 conn(0x7fb1e015bb10 legacy=0x7fb1e015bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e015b310 con 0x7fb1e015ab00 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fb1d0001670 con 0x7fb1e015ab00 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 >> v1:172.21.15.72:6789/0 conn(0x7fb1e015ab00 legacy=0x7fb1e00b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc0039e0 0x7fb1cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc00b9c0 con 0x7fb1cc004d40 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc00bb40 con 0x7fb1cc0039e0 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1cc00bd40 con 0x7fb1e00af6c0 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ded9d640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1de59c640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc0039e0 0x7fb1cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1d7fff640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ded9d640 1 -- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 msgr2=0x7fb1cc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ded9d640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1cc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ded9d640 1 -- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc0039e0 msgr2=0x7fb1cc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ded9d640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1cc0039e0 0x7fb1cc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.550+0000 7fb1ded9d640 1 -- 172.21.15.72:0/1821352539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc003520 con 0x7fb1cc004d40 2024-03-06T07:23:08.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.551+0000 7fb1d7fff640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e00af6c0 0x7fb1cc004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.551+0000 7fb1ded9d640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8001f60 tx=0x7fb1cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a30311d9b119db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.551+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8001b30 con 0x7fb1cc004d40 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.551+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d802d710 con 0x7fb1cc004d40 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.551+0000 7fb1ddd9b640 1 -- 172.21.15.72:0/1821352539 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d802db00 con 0x7fb1cc004d40 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1e493e640 1 -- 172.21.15.72:0/1821352539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1cc004d40 msgr2=0x7fb1cc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1e493e640 1 --2- 172.21.15.72:0/1821352539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1cc004d40 0x7fb1cc0051c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8001f60 tx=0x7fb1cc005830 comp rx=0 tx=0).stop 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1ded9d640 1 -- 172.21.15.72:0/1821352539 reap_dead start 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1e493e640 1 -- 172.21.15.72:0/1821352539 shutdown_connections 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1e493e640 1 -- 172.21.15.72:0/1821352539 >> 172.21.15.72:0/1821352539 conn(0x7fb1e006c0c0 msgr2=0x7fb1e006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1e493e640 1 -- 172.21.15.72:0/1821352539 shutdown_connections 2024-03-06T07:23:08.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.552+0000 7fb1e493e640 1 -- 172.21.15.72:0/1821352539 wait complete. 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.553+0000 7fb1e493e640 1 Processor -- start 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.553+0000 7fb1e493e640 1 -- start start 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1e493e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc004d40 0x7fb1e006e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1e493e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 0x7fb1e014bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1e493e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e014c500 0x7fb1e00a9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1e493e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0071b20 con 0x7fb1e01546c0 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1e493e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e00b4ac0 con 0x7fb1e014c500 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1e493e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e0071ca0 con 0x7fb1cc004d40 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 0x7fb1e014bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 0x7fb1e014bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56218/0 (socket says 172.21.15.72:56218) 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 -- 172.21.15.72:0/2920721164 learned_addr learned my addr 172.21.15.72:0/2920721164 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:08.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 -- 172.21.15.72:0/2920721164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc004d40 msgr2=0x7fb1e006e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc004d40 0x7fb1e006e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 -- 172.21.15.72:0/2920721164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e014c500 msgr2=0x7fb1e00a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e014c500 0x7fb1e00a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.554+0000 7fb1d7fff640 1 -- 172.21.15.72:0/2920721164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e015d240 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.555+0000 7fb1d7fff640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 0x7fb1e014bfc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8011700 tx=0x7fb1c8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.555+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800f940 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.555+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c800fae0 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.555+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8015460 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.555+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e015d520 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.555+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e015dae0 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.556+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e014c930 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.556+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1c8015600 con 0x7fb1e01546c0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.557+0000 7fb1d6ffd640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.557+0000 7fb1de59c640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.557+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1c808abf0 con 0x7fb1e01546c0 2024-03-06T07:23:08.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.557+0000 7fb1de59c640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1c006b4d0 0x7fb1c006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00011e0 tx=0x7fb1d0001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.561+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1c8019030 con 0x7fb1e01546c0 2024-03-06T07:23:08.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.742+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1e004f330 con 0x7fb1e01546c0 2024-03-06T07:23:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.743+0000 7fb1d6ffd640 1 -- 172.21.15.72:0/2920721164 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fb1c80599e0 con 0x7fb1e01546c0 2024-03-06T07:23:08.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:08.745 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":34,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:47.667116+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:08.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1c006b4d0 msgr2=0x7fb1c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1c006b4d0 0x7fb1c006d990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb1d00011e0 tx=0x7fb1d0001150 comp rx=0 tx=0).stop 2024-03-06T07:23:08.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 msgr2=0x7fb1e014bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 0x7fb1e014bfc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8011700 tx=0x7fb1c8011bc0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 shutdown_connections 2024-03-06T07:23:08.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1c006b4d0 0x7fb1c006d990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e014c500 0x7fb1e00a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e01546c0 0x7fb1e014bfc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 --2- 172.21.15.72:0/2920721164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1cc004d40 0x7fb1e006e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 >> 172.21.15.72:0/2920721164 conn(0x7fb1e006c0c0 msgr2=0x7fb1e0157870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 shutdown_connections 2024-03-06T07:23:08.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.745+0000 7fb1e493e640 1 -- 172.21.15.72:0/2920721164 wait complete. 2024-03-06T07:23:08.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:22:44.323650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:22:44.983501+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:08.759 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:08.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.859+0000 7fa05af0c640 1 Processor -- start 2024-03-06T07:23:08.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.859+0000 7fa05af0c640 1 -- start start 2024-03-06T07:23:08.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa05af0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa05406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa05af0c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0540d0af0 0x7fa0540cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa05af0c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0540cd4b0 con 0x7fa05406fb40 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa05af0c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05406ff80 con 0x7fa0540d0af0 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa05af0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540cd790 con 0x7fa05415c050 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa053fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa05406fb40 0x7fa0540cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45516/0 (socket says 172.21.15.72:45516) 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa053fff640 1 -- 172.21.15.72:0/742021828 learned_addr learned my addr 172.21.15.72:0/742021828 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145188225 0 0) 0x7fa0540cd4b0 con 0x7fa05406fb40 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa040003520 con 0x7fa05406fb40 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0594a9640 1 --2- 172.21.15.72:0/742021828 >> v2:172.21.15.146:3300/0 conn(0x7fa0540d0af0 0x7fa0540cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa058ca8640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa05406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2787430154 0 0) 0x7fa040003520 con 0x7fa05406fb40 2024-03-06T07:23:08.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 msgr2=0x7fa05406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa05406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 >> v2:172.21.15.146:3300/0 conn(0x7fa0540d0af0 msgr2=0x7fa0540cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 --2- 172.21.15.72:0/742021828 >> v2:172.21.15.146:3300/0 conn(0x7fa0540d0af0 0x7fa0540cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0540cda70 con 0x7fa05406fb40 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fa03c001670 con 0x7fa05406fb40 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 >> v1:172.21.15.72:6789/0 conn(0x7fa05406fb40 legacy=0x7fa0540cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0400039b0 0x7fa040003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa040004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040004d40 0x7fa0400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05406ff80 con 0x7fa0400039b0 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0540cd790 con 0x7fa040004d40 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.860+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04000b9f0 con 0x7fa05415c050 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0400039b0 0x7fa040003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa053fff640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa040004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa0594a9640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040004d40 0x7fa0400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 -- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 msgr2=0x7fa040004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa040004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 -- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040004d40 msgr2=0x7fa0400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040004d40 0x7fa0400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 -- 172.21.15.72:0/742021828 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040003520 con 0x7fa0400039b0 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa053fff640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa05415c050 0x7fa040004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa0594a9640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040004d40 0x7fa0400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.861+0000 7fa058ca8640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0400039b0 0x7fa040003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa044009ac0 tx=0x7fa040005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e215e8b2106c137f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044039070 con 0x7fa0400039b0 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04402c6b0 con 0x7fa0400039b0 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa0537fe640 1 -- 172.21.15.72:0/742021828 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044009690 con 0x7fa0400039b0 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa05af0c640 1 -- 172.21.15.72:0/742021828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0400039b0 msgr2=0x7fa040003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa05af0c640 1 --2- 172.21.15.72:0/742021828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0400039b0 0x7fa040003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa044009ac0 tx=0x7fa040005830 comp rx=0 tx=0).stop 2024-03-06T07:23:08.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa0594a9640 1 -- 172.21.15.72:0/742021828 reap_dead start 2024-03-06T07:23:08.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa05af0c640 1 -- 172.21.15.72:0/742021828 shutdown_connections 2024-03-06T07:23:08.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa05af0c640 1 -- 172.21.15.72:0/742021828 >> 172.21.15.72:0/742021828 conn(0x7fa0540b3800 msgr2=0x7fa0540b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:08.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa05af0c640 1 -- 172.21.15.72:0/742021828 shutdown_connections 2024-03-06T07:23:08.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.862+0000 7fa05af0c640 1 -- 172.21.15.72:0/742021828 wait complete. 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 Processor -- start 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 -- start start 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0400039b0 0x7fa05415ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa05414ab50 0x7fa0540d1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 0x7fa0541f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05406f780 con 0x7fa05414afd0 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05406f600 con 0x7fa05414ab50 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.864+0000 7fa05af0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05406f900 con 0x7fa0400039b0 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 0x7fa0541f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 0x7fa0541f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56234/0 (socket says 172.21.15.72:56234) 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 -- 172.21.15.72:0/57857941 learned_addr learned my addr 172.21.15.72:0/57857941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa053fff640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa05414ab50 0x7fa0540d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa058ca8640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0400039b0 0x7fa05415ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 -- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0400039b0 msgr2=0x7fa05415ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0400039b0 0x7fa05415ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 -- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa05414ab50 msgr2=0x7fa0540d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa05414ab50 0x7fa0540d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 -- 172.21.15.72:0/57857941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05415b360 con 0x7fa05414afd0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa053fff640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa05414ab50 0x7fa0540d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa058ca8640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0400039b0 0x7fa05415ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0594a9640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 0x7fa0541f5b50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa0540d0f80 tx=0x7fa04c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c01a020 con 0x7fa05414afd0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c007e60 con 0x7fa05414afd0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa054159390 con 0x7fa05414afd0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c016470 con 0x7fa05414afd0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.865+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa054159950 con 0x7fa05414afd0 2024-03-06T07:23:08.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.866+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05409fe60 con 0x7fa05414afd0 2024-03-06T07:23:08.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.867+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa04c005d10 con 0x7fa05414afd0 2024-03-06T07:23:08.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.867+0000 7fa0517fa640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03406b4d0 0x7fa03406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:08.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.867+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa04c08a8d0 con 0x7fa05414afd0 2024-03-06T07:23:08.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.867+0000 7fa058ca8640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03406b4d0 0x7fa03406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:08.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.868+0000 7fa058ca8640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03406b4d0 0x7fa03406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa044036000 tx=0x7fa044030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:08.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:08.871+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa04c024070 con 0x7fa05414afd0 2024-03-06T07:23:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.048+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa054074580 con 0x7fa05414afd0 2024-03-06T07:23:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.048+0000 7fa0517fa640 1 -- 172.21.15.72:0/57857941 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa05415b360 con 0x7fa05414afd0 2024-03-06T07:23:09.050 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03406b4d0 msgr2=0x7fa03406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03406b4d0 0x7fa03406d990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa044036000 tx=0x7fa044030000 comp rx=0 tx=0).stop 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 msgr2=0x7fa0541f5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 0x7fa0541f5b50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa0540d0f80 tx=0x7fa04c0115a0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 shutdown_connections 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03406b4d0 0x7fa03406d990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa05414afd0 0x7fa0541f5b50 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa05414ab50 0x7fa0540d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 --2- 172.21.15.72:0/57857941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0400039b0 0x7fa05415ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 >> 172.21.15.72:0/57857941 conn(0x7fa0540b3800 msgr2=0x7fa0540aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 shutdown_connections 2024-03-06T07:23:09.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.050+0000 7fa05af0c640 1 -- 172.21.15.72:0/57857941 wait complete. 2024-03-06T07:23:09.063 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:09.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.168+0000 7fa052650640 1 Processor -- start 2024-03-06T07:23:09.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.169+0000 7fa052650640 1 -- start start 2024-03-06T07:23:09.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa052650640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0d10c0 0x7fa04c0d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa052650640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa04c15ecd0 0x7fa04c0cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa052650640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa04c0cfa50 con 0x7fa04c0d14a0 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa052650640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04c15c050 con 0x7fa04c15ecd0 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa052650640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0d0060 con 0x7fa04c0d10c0 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa04c15ecd0 0x7fa04c0cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa04c15ecd0 0x7fa04c0cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56242/0 (socket says 172.21.15.72:56242) 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 -- 172.21.15.72:0/1580423591 learned_addr learned my addr 172.21.15.72:0/1580423591 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa04bfff640 1 --2- 172.21.15.72:0/1580423591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0d10c0 0x7fa04c0d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 -- 172.21.15.72:0/1580423591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0d10c0 msgr2=0x7fa04c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 --2- 172.21.15.72:0/1580423591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0d10c0 0x7fa04c0d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 -- 172.21.15.72:0/1580423591 >> v1:172.21.15.146:6789/0 conn(0x7fa04c0d14a0 legacy=0x7fa04c15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.170+0000 7fa050bed640 1 -- 172.21.15.72:0/1580423591 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c0d0a30 con 0x7fa04c15ecd0 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.171+0000 7fa050bed640 1 --2- 172.21.15.72:0/1580423591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c15ecd0 0x7fa04c0cae80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa044009e50 tx=0x7fa04402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9feae3746c4cccc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.171+0000 7fa04affd640 1 -- 172.21.15.72:0/1580423591 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04402cb00 con 0x7fa04c15ecd0 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.171+0000 7fa04affd640 1 -- 172.21.15.72:0/1580423591 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04402cca0 con 0x7fa04c15ecd0 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.171+0000 7fa04affd640 1 -- 172.21.15.72:0/1580423591 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044034610 con 0x7fa04c15ecd0 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 -- 172.21.15.72:0/1580423591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c15ecd0 msgr2=0x7fa04c0cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 --2- 172.21.15.72:0/1580423591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c15ecd0 0x7fa04c0cae80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa044009e50 tx=0x7fa04402bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 -- 172.21.15.72:0/1580423591 shutdown_connections 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 --2- 172.21.15.72:0/1580423591 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c15ecd0 0x7fa04c0cae80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 --2- 172.21.15.72:0/1580423591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0d10c0 0x7fa04c0d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 -- 172.21.15.72:0/1580423591 >> 172.21.15.72:0/1580423591 conn(0x7fa04c0b3800 msgr2=0x7fa04c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 -- 172.21.15.72:0/1580423591 shutdown_connections 2024-03-06T07:23:09.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.172+0000 7fa052650640 1 -- 172.21.15.72:0/1580423591 wait complete. 2024-03-06T07:23:09.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.173+0000 7fa052650640 1 Processor -- start 2024-03-06T07:23:09.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.173+0000 7fa052650640 1 -- start start 2024-03-06T07:23:09.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa052650640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa04c0d10c0 0x7fa04c0cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa052650640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 0x7fa04c0b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa052650640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0b4e30 0x7fa04c0aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa052650640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c15f440 con 0x7fa04c0d14a0 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa052650640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c15f160 con 0x7fa04c0d10c0 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa052650640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04c0cfcf0 con 0x7fa04c0b4e30 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa04b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 0x7fa04c0b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa04b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 0x7fa04c0b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56258/0 (socket says 172.21.15.72:56258) 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa04b7fe640 1 -- 172.21.15.72:0/2514651334 learned_addr learned my addr 172.21.15.72:0/2514651334 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa04bfff640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa04c0d10c0 0x7fa04c0cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa050bed640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0b4e30 0x7fa04c0aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.174+0000 7fa04b7fe640 1 -- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0b4e30 msgr2=0x7fa04c0aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa04b7fe640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0b4e30 0x7fa04c0aefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa04b7fe640 1 -- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa04c0d10c0 msgr2=0x7fa04c0cea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa04b7fe640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa04c0d10c0 0x7fa04c0cea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa04b7fe640 1 -- 172.21.15.72:0/2514651334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04c1f7bd0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa04bfff640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa04c0d10c0 0x7fa04c0cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa04b7fe640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 0x7fa04c0b48f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa04000c3c0 tx=0x7fa04000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0400052d0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa040005490 con 0x7fa04c0d14a0 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04c0afbb0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa040007260 con 0x7fa04c0d14a0 2024-03-06T07:23:09.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.175+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04c0b0170 con 0x7fa04c0d14a0 2024-03-06T07:23:09.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.176+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04c06eef0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.177+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa040007400 con 0x7fa04c0d14a0 2024-03-06T07:23:09.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.177+0000 7fa048ff9640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.178+0000 7fa04bfff640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.178+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0400896e0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.178+0000 7fa04bfff640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa03c006fd0 tx=0x7fa03c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.182+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa04c06eef0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.358+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa04c09fe60 con 0x7fa04c0d14a0 2024-03-06T07:23:09.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.358+0000 7fa048ff9640 1 -- 172.21.15.72:0/2514651334 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa04c1f7bd0 con 0x7fa04c0d14a0 2024-03-06T07:23:09.360 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03806b4d0 msgr2=0x7fa03806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03806b4d0 0x7fa03806d990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa03c006fd0 tx=0x7fa03c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 msgr2=0x7fa04c0b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 0x7fa04c0b48f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa04000c3c0 tx=0x7fa04000c880 comp rx=0 tx=0).stop 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 shutdown_connections 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa03806b4d0 0x7fa03806d990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa04c0b4e30 0x7fa04c0aefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa04c0d14a0 0x7fa04c0b48f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 --2- 172.21.15.72:0/2514651334 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa04c0d10c0 0x7fa04c0cea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 >> 172.21.15.72:0/2514651334 conn(0x7fa04c0b3800 msgr2=0x7fa04c0b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 shutdown_connections 2024-03-06T07:23:09.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.360+0000 7fa052650640 1 -- 172.21.15.72:0/2514651334 wait complete. 2024-03-06T07:23:09.373 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:09.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.475+0000 7f013de08640 1 Processor -- start 2024-03-06T07:23:09.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.475+0000 7f013de08640 1 -- start start 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f013de08640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01380d14a0 0x7f013815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f013de08640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f013815eb00 0x7f01380cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f013de08640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01380cfa50 con 0x7f01380d10c0 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f013de08640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013815c050 con 0x7f01380d14a0 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f013de08640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01380d0060 con 0x7f013815eb00 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f0136ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01380d14a0 0x7f013815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f01377fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f01380d10c0 0x7f01380d04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45518/0 (socket says 172.21.15.72:45518) 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f01377fe640 1 -- 172.21.15.72:0/4166109447 learned_addr learned my addr 172.21.15.72:0/4166109447 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:09.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f0137fff640 1 --2- 172.21.15.72:0/4166109447 >> v2:172.21.15.146:3300/0 conn(0x7f013815eb00 0x7f01380cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f0136ffd640 1 -- 172.21.15.72:0/4166109447 >> v2:172.21.15.146:3300/0 conn(0x7f013815eb00 msgr2=0x7f01380cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f0136ffd640 1 --2- 172.21.15.72:0/4166109447 >> v2:172.21.15.146:3300/0 conn(0x7f013815eb00 0x7f01380cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f0136ffd640 1 -- 172.21.15.72:0/4166109447 >> v1:172.21.15.72:6789/0 conn(0x7f01380d10c0 legacy=0x7f01380d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.476+0000 7f0136ffd640 1 -- 172.21.15.72:0/4166109447 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380b45c0 con 0x7f01380d14a0 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.477+0000 7f0136ffd640 1 --2- 172.21.15.72:0/4166109447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01380d14a0 0x7f013815e5c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0138071930 tx=0x7f012002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11058ed143b0d02c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.477+0000 7f01367fc640 1 -- 172.21.15.72:0/4166109447 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120035020 con 0x7f01380d14a0 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.477+0000 7f01367fc640 1 -- 172.21.15.72:0/4166109447 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0120031830 con 0x7f01380d14a0 2024-03-06T07:23:09.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.477+0000 7f01367fc640 1 -- 172.21.15.72:0/4166109447 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120031c00 con 0x7f01380d14a0 2024-03-06T07:23:09.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 -- 172.21.15.72:0/4166109447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01380d14a0 msgr2=0x7f013815e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 --2- 172.21.15.72:0/4166109447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01380d14a0 0x7f013815e5c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0138071930 tx=0x7f012002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 -- 172.21.15.72:0/4166109447 shutdown_connections 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 --2- 172.21.15.72:0/4166109447 >> v2:172.21.15.146:3300/0 conn(0x7f013815eb00 0x7f01380cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 --2- 172.21.15.72:0/4166109447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01380d14a0 0x7f013815e5c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 -- 172.21.15.72:0/4166109447 >> 172.21.15.72:0/4166109447 conn(0x7f01380b3800 msgr2=0x7f01380b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 -- 172.21.15.72:0/4166109447 shutdown_connections 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.478+0000 7f013de08640 1 -- 172.21.15.72:0/4166109447 wait complete. 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.479+0000 7f013de08640 1 Processor -- start 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.479+0000 7f013de08640 1 -- start start 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01380d10c0 0x7f01380b64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01380d14a0 0x7f01380cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 0x7f01380cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01380cfcf0 con 0x7f013815eb00 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013815f440 con 0x7f01380d14a0 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380cb3c0 con 0x7f01380d10c0 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 0x7f01380cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 0x7f01380cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56274/0 (socket says 172.21.15.72:56274) 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 -- 172.21.15.72:0/1574944619 learned_addr learned my addr 172.21.15.72:0/1574944619 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0136ffd640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01380d14a0 0x7f01380cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f01377fe640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01380d10c0 0x7f01380b64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 -- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01380d10c0 msgr2=0x7f01380b64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01380d10c0 0x7f01380b64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 -- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01380d14a0 msgr2=0x7f01380cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01380d14a0 0x7f01380cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 -- 172.21.15.72:0/1574944619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01381f7b10 con 0x7f013815eb00 2024-03-06T07:23:09.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0136ffd640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01380d14a0 0x7f01380cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f0137fff640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 0x7f01380cc6c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0130007e60 tx=0x7f013000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130014420 con 0x7f013815eb00 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0130019070 con 0x7f013815eb00 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130014820 con 0x7f013815eb00 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01380cccf0 con 0x7f013815eb00 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.480+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01380cd2b0 con 0x7f013815eb00 2024-03-06T07:23:09.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.481+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013809fe60 con 0x7f013815eb00 2024-03-06T07:23:09.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.482+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01300149c0 con 0x7f013815eb00 2024-03-06T07:23:09.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.482+0000 7f012ffff640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f011806b4d0 0x7f011806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.482+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f013008aa60 con 0x7f013815eb00 2024-03-06T07:23:09.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.482+0000 7f01377fe640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f011806b4d0 0x7f011806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.483+0000 7f01377fe640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f011806b4d0 0x7f011806d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0128000f60 tx=0x7f0128009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.487+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01300598e0 con 0x7f013815eb00 2024-03-06T07:23:09.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.723+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0138068ff0 con 0x7f013815eb00 2024-03-06T07:23:09.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.723+0000 7f012ffff640 1 -- 172.21.15.72:0/1574944619 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7f01300591e0 con 0x7f013815eb00 2024-03-06T07:23:09.726 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f011806b4d0 msgr2=0x7f011806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f011806b4d0 0x7f011806d990 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0128000f60 tx=0x7f0128009040 comp rx=0 tx=0).stop 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 msgr2=0x7f01380cc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 0x7f01380cc6c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0130007e60 tx=0x7f013000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 shutdown_connections 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f011806b4d0 0x7f011806d990 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f013815eb00 0x7f01380cc6c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01380d14a0 0x7f01380cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 --2- 172.21.15.72:0/1574944619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01380d10c0 0x7f01380b64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.725+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 >> 172.21.15.72:0/1574944619 conn(0x7f01380b3800 msgr2=0x7f013815d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.726+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 shutdown_connections 2024-03-06T07:23:09.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.726+0000 7f013de08640 1 -- 172.21.15.72:0/1574944619 wait complete. 2024-03-06T07:23:09.740 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.839+0000 7f6eacc4e640 1 Processor -- start 2024-03-06T07:23:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.839+0000 7f6eacc4e640 1 -- start start 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6eacc4e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6ea80d22b0 0x7f6ea80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6eacc4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b41f0 0x7f6ea80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6eacc4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80b3430 con 0x7f6ea815dfe0 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6eacc4e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea80d2c50 con 0x7f6ea80d22b0 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6eacc4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80b31e0 con 0x7f6ea80b41f0 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6e9ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6ea80d22b0 0x7f6ea80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6ea659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6ea815dfe0 0x7f6ea80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45534/0 (socket says 172.21.15.72:45534) 2024-03-06T07:23:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6ea659c640 1 -- 172.21.15.72:0/2839272361 learned_addr learned my addr 172.21.15.72:0/2839272361 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6e9ffff640 1 -- 172.21.15.72:0/2839272361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b41f0 msgr2=0x7f6ea80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6e9ffff640 1 --2- 172.21.15.72:0/2839272361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b41f0 0x7f6ea80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6e9ffff640 1 -- 172.21.15.72:0/2839272361 >> v1:172.21.15.72:6789/0 conn(0x7f6ea815dfe0 legacy=0x7f6ea80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6e9ffff640 1 -- 172.21.15.72:0/2839272361 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80b3710 con 0x7f6ea80d22b0 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.840+0000 7f6ea5d9b640 1 -- 172.21.15.72:0/2839272361 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051666309 0 0) 0x7f6ea80b3430 con 0x7f6ea815dfe0 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.841+0000 7f6e9ffff640 1 --2- 172.21.15.72:0/2839272361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80d22b0 0x7f6ea80d2710 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6e900097a0 tx=0x7f6e9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa1cccd7e95cca7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.841+0000 7f6ea5d9b640 1 -- 172.21.15.72:0/2839272361 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90035020 con 0x7f6ea80d22b0 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.841+0000 7f6ea5d9b640 1 -- 172.21.15.72:0/2839272361 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90031830 con 0x7f6ea80d22b0 2024-03-06T07:23:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.841+0000 7f6ea5d9b640 1 -- 172.21.15.72:0/2839272361 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90031c00 con 0x7f6ea80d22b0 2024-03-06T07:23:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 -- 172.21.15.72:0/2839272361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80d22b0 msgr2=0x7f6ea80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/2839272361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80d22b0 0x7f6ea80d2710 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6e900097a0 tx=0x7f6e9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 -- 172.21.15.72:0/2839272361 shutdown_connections 2024-03-06T07:23:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/2839272361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b41f0 0x7f6ea80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/2839272361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80d22b0 0x7f6ea80d2710 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 -- 172.21.15.72:0/2839272361 >> 172.21.15.72:0/2839272361 conn(0x7f6ea80d3cd0 msgr2=0x7f6ea80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:09.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 -- 172.21.15.72:0/2839272361 shutdown_connections 2024-03-06T07:23:09.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.842+0000 7f6eacc4e640 1 -- 172.21.15.72:0/2839272361 wait complete. 2024-03-06T07:23:09.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.843+0000 7f6eacc4e640 1 Processor -- start 2024-03-06T07:23:09.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.843+0000 7f6eacc4e640 1 -- start start 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6eacc4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 0x7f6ea814af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6eacc4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80d22b0 0x7f6ea80ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6eacc4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea815dfe0 0x7f6ea814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6eacc4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80cfa00 con 0x7f6ea80b41f0 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6eacc4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80b3d60 con 0x7f6ea815dfe0 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6eacc4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80d1c70 con 0x7f6ea80d22b0 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 0x7f6ea814af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 0x7f6ea814af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56296/0 (socket says 172.21.15.72:56296) 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 -- 172.21.15.72:0/4007294105 learned_addr learned my addr 172.21.15.72:0/4007294105 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 -- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80d22b0 msgr2=0x7f6ea80ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6e9ffff640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80d22b0 0x7f6ea80ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea6d9d640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea815dfe0 0x7f6ea814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80d22b0 0x7f6ea80ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 -- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea815dfe0 msgr2=0x7f6ea814c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea815dfe0 0x7f6ea814c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 -- 172.21.15.72:0/4007294105 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80ca670 con 0x7f6ea80b41f0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea6d9d640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea815dfe0 0x7f6ea814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.844+0000 7f6ea659c640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 0x7f6ea814af90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c430 tx=0x7f6e9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.845+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e980074e0 con 0x7f6ea80b41f0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.845+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e980047e0 con 0x7f6ea80b41f0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.845+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98004bd0 con 0x7f6ea80b41f0 2024-03-06T07:23:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.845+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea814b6c0 con 0x7f6ea80b41f0 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.845+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea814bc80 con 0x7f6ea80b41f0 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.846+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea804f330 con 0x7f6ea80b41f0 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.846+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6e98011070 con 0x7f6ea80b41f0 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.847+0000 7f6e9effd640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6e8806b4d0 0x7f6e8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.847+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e98089de0 con 0x7f6ea80b41f0 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.847+0000 7f6e9ffff640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6e8806b4d0 0x7f6e8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.847+0000 7f6e9ffff640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6e8806b4d0 0x7f6e8806d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6e90036000 tx=0x7f6e90008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:09.851+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6e98058cf0 con 0x7f6ea80b41f0 2024-03-06T07:23:10.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.127+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ea806bfe0 con 0x7f6ea80b41f0 2024-03-06T07:23:10.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.127+0000 7f6e9effd640 1 -- 172.21.15.72:0/4007294105 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f6e980585f0 con 0x7f6ea80b41f0 2024-03-06T07:23:10.130 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:10.130 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:23:10.130 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:23:10.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6e8806b4d0 msgr2=0x7f6e8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6e8806b4d0 0x7f6e8806d990 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6e90036000 tx=0x7f6e90008030 comp rx=0 tx=0).stop 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 msgr2=0x7f6ea814af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 0x7f6ea814af90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800c430 tx=0x7f6e9800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 shutdown_connections 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6e8806b4d0 0x7f6e8806d990 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea815dfe0 0x7f6ea814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80d22b0 0x7f6ea80ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 --2- 172.21.15.72:0/4007294105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea80b41f0 0x7f6ea814af90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 >> 172.21.15.72:0/4007294105 conn(0x7f6ea80d3cd0 msgr2=0x7f6ea806c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 shutdown_connections 2024-03-06T07:23:10.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.130+0000 7f6eacc4e640 1 -- 172.21.15.72:0/4007294105 wait complete. 2024-03-06T07:23:10.148 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:10.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.247+0000 7fa50e5d0640 1 Processor -- start 2024-03-06T07:23:10.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.247+0000 7fa50e5d0640 1 -- start start 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50e5d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5080b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50e5d0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5080b3430 con 0x7fa5080d22b0 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50e5d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5080b3710 con 0x7fa50815dfe0 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50e5d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080b31e0 con 0x7fa5080d2710 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa5077fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa5080d22b0 0x7fa5080b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45548/0 (socket says 172.21.15.72:45548) 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa5077fe640 1 -- 172.21.15.72:0/3057101620 learned_addr learned my addr 172.21.15.72:0/3057101620 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50cb6d640 1 --2- 172.21.15.72:0/3057101620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5080b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987496037 0 0) 0x7fa5080b3430 con 0x7fa5080d22b0 2024-03-06T07:23:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f0003520 con 0x7fa5080d22b0 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733999911 0 0) 0x7fa5080b3710 con 0x7fa50815dfe0 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5080b3430 con 0x7fa50815dfe0 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50cb6d640 1 -- 172.21.15.72:0/3057101620 >> v1:172.21.15.146:6789/0 conn(0x7fa50815dfe0 legacy=0x7fa5080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50cb6d640 1 -- 172.21.15.72:0/3057101620 >> v1:172.21.15.72:6789/0 conn(0x7fa5080d22b0 legacy=0x7fa5080b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1312125451 0 0) 0x7fa4f0003520 con 0x7fa5080d22b0 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.248+0000 7fa50cb6d640 1 -- 172.21.15.72:0/3057101620 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080b39f0 con 0x7fa5080d2710 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.249+0000 7fa50cb6d640 1 --2- 172.21.15.72:0/3057101620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5080b4900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa5080b3360 tx=0x7fa50002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eef6fc217674785 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.249+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500039070 con 0x7fa5080d2710 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.249+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50002c910 con 0x7fa5080d2710 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.249+0000 7fa506ffd640 1 -- 172.21.15.72:0/3057101620 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc001670 con 0x7fa5080d2710 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 -- 172.21.15.72:0/3057101620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 msgr2=0x7fa5080b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/3057101620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5080b4900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa5080b3360 tx=0x7fa50002bde0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 -- 172.21.15.72:0/3057101620 shutdown_connections 2024-03-06T07:23:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/3057101620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5080b4900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 -- 172.21.15.72:0/3057101620 >> 172.21.15.72:0/3057101620 conn(0x7fa5080d3cd0 msgr2=0x7fa5080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 -- 172.21.15.72:0/3057101620 shutdown_connections 2024-03-06T07:23:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.250+0000 7fa50e5d0640 1 -- 172.21.15.72:0/3057101620 wait complete. 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.251+0000 7fa50e5d0640 1 Processor -- start 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.251+0000 7fa50e5d0640 1 -- start start 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50e5d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5080d22b0 0x7fa508150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50e5d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5081f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50e5d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 0x7fa5081f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50e5d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080cfa00 con 0x7fa50815dfe0 2024-03-06T07:23:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50e5d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080b3d60 con 0x7fa5080d22b0 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50e5d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080d1c70 con 0x7fa5080d2710 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 0x7fa5081f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 0x7fa5081f32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56302/0 (socket says 172.21.15.72:56302) 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 -- 172.21.15.72:0/539876300 learned_addr learned my addr 172.21.15.72:0/539876300 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa507fff640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5080d22b0 0x7fa508150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa5077fe640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5081f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 -- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 msgr2=0x7fa5081f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5081f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 -- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5080d22b0 msgr2=0x7fa508150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5080d22b0 0x7fa508150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 -- 172.21.15.72:0/539876300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5081f3830 con 0x7fa50815dfe0 2024-03-06T07:23:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa507fff640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5080d22b0 0x7fa508150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.252+0000 7fa50cb6d640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 0x7fa5081f32f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa500009d90 tx=0x7fa500004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.253+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500039050 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.253+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50003a040 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.253+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5080b1af0 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.253+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500009450 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.253+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5080b20b0 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.254+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50804f330 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.254+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa500047020 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.255+0000 7fa504ff9640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4f406b4d0 0x7fa4f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.255+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5000ac120 con 0x7fa50815dfe0 2024-03-06T07:23:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.255+0000 7fa507fff640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4f406b4d0 0x7fa4f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.255+0000 7fa507fff640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4f406b4d0 0x7fa4f406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8001610 tx=0x7fa4f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.263+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa50004e020 con 0x7fa50815dfe0 2024-03-06T07:23:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.443+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa50815c880 con 0x7fa50815dfe0 2024-03-06T07:23:10.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.445+0000 7fa504ff9640 1 -- 172.21.15.72:0/539876300 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7fa50007afa0 con 0x7fa50815dfe0 2024-03-06T07:23:10.447 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:10.448 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":34,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:47.667116+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4f406b4d0 msgr2=0x7fa4f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4f406b4d0 0x7fa4f406d990 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8001610 tx=0x7fa4f8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 msgr2=0x7fa5081f32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 0x7fa5081f32f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fa500009d90 tx=0x7fa500004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 shutdown_connections 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4f406b4d0 0x7fa4f406d990 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa50815dfe0 0x7fa5081f32f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5080d2710 0x7fa5081f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 --2- 172.21.15.72:0/539876300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5080d22b0 0x7fa508150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 >> 172.21.15.72:0/539876300 conn(0x7fa5080d3cd0 msgr2=0x7fa50806cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 shutdown_connections 2024-03-06T07:23:10.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.447+0000 7fa50e5d0640 1 -- 172.21.15.72:0/539876300 wait complete. 2024-03-06T07:23:10.461 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:10.461 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:10.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.563+0000 7f39e09d7640 1 Processor -- start 2024-03-06T07:23:10.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.563+0000 7f39e09d7640 1 -- start start 2024-03-06T07:23:10.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39e09d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39dc15ddf0 0x7f39dc0cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39e09d7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39dc15e1d0 0x7f39dc0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39e09d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc0d06d0 0x7f39dc0d0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39e09d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39dc0d1080 con 0x7f39dc15ddf0 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39e09d7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39dc0d27c0 con 0x7f39dc15e1d0 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39e09d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d18d0 con 0x7f39dc0d06d0 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39dc15ddf0 0x7f39dc0cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39dc15ddf0 0x7f39dc0cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56318/0 (socket says 172.21.15.72:56318) 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 -- 172.21.15.72:0/3091225834 learned_addr learned my addr 172.21.15.72:0/3091225834 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39d9d9b640 1 --2- 172.21.15.72:0/3091225834 >> v2:172.21.15.146:3300/0 conn(0x7f39dc15e1d0 0x7f39dc0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39dad9d640 1 --2- 172.21.15.72:0/3091225834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc0d06d0 0x7f39dc0d0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 -- 172.21.15.72:0/3091225834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc0d06d0 msgr2=0x7f39dc0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 --2- 172.21.15.72:0/3091225834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc0d06d0 0x7f39dc0d0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 -- 172.21.15.72:0/3091225834 >> v2:172.21.15.146:3300/0 conn(0x7f39dc15e1d0 msgr2=0x7f39dc0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 --2- 172.21.15.72:0/3091225834 >> v2:172.21.15.146:3300/0 conn(0x7f39dc15e1d0 0x7f39dc0cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.565+0000 7f39da59c640 1 -- 172.21.15.72:0/3091225834 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc0d1bb0 con 0x7f39dc15ddf0 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.566+0000 7f39da59c640 1 --2- 172.21.15.72:0/3091225834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15ddf0 0x7f39dc0cf440 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0d39f0 tx=0x7f39d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e70126605d7022 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.566+0000 7f39d959a640 1 -- 172.21.15.72:0/3091225834 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0035020 con 0x7f39dc15ddf0 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.566+0000 7f39d959a640 1 -- 172.21.15.72:0/3091225834 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d0031870 con 0x7f39dc15ddf0 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.566+0000 7f39d959a640 1 -- 172.21.15.72:0/3091225834 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0031c40 con 0x7f39dc15ddf0 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 -- 172.21.15.72:0/3091225834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15ddf0 msgr2=0x7f39dc0cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 --2- 172.21.15.72:0/3091225834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15ddf0 0x7f39dc0cf440 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f39dc0d39f0 tx=0x7f39d002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 -- 172.21.15.72:0/3091225834 shutdown_connections 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 --2- 172.21.15.72:0/3091225834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc0d06d0 0x7f39dc0d0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 --2- 172.21.15.72:0/3091225834 >> v2:172.21.15.146:3300/0 conn(0x7f39dc15e1d0 0x7f39dc0cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 --2- 172.21.15.72:0/3091225834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15ddf0 0x7f39dc0cf440 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 -- 172.21.15.72:0/3091225834 >> 172.21.15.72:0/3091225834 conn(0x7f39dc09cff0 msgr2=0x7f39dc0d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 -- 172.21.15.72:0/3091225834 shutdown_connections 2024-03-06T07:23:10.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.567+0000 7f39e09d7640 1 -- 172.21.15.72:0/3091225834 wait complete. 2024-03-06T07:23:10.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.569+0000 7f39e09d7640 1 Processor -- start 2024-03-06T07:23:10.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.569+0000 7f39e09d7640 1 -- start start 2024-03-06T07:23:10.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39e09d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc0d06d0 0x7f39dc157c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39e09d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc15ddf0 0x7f39dc1f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39e09d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 0x7f39dc14cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39e09d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d2450 con 0x7f39dc15e1d0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39e09d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d2170 con 0x7f39dc0d06d0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39e09d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc0d2c50 con 0x7f39dc15ddf0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 0x7f39dc14cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 0x7f39dc14cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56330/0 (socket says 172.21.15.72:56330) 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 -- 172.21.15.72:0/547090813 learned_addr learned my addr 172.21.15.72:0/547090813 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 -- 172.21.15.72:0/547090813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc15ddf0 msgr2=0x7f39dc1f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc15ddf0 0x7f39dc1f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 -- 172.21.15.72:0/547090813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc0d06d0 msgr2=0x7f39dc157c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.570+0000 7f39dad9d640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc0d06d0 0x7f39dc157c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39dad9d640 1 -- 172.21.15.72:0/547090813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc158150 con 0x7f39dc15e1d0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39dad9d640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 0x7f39dc14cb90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f39d4007e90 tx=0x7f39d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4014420 con 0x7f39dc15e1d0 2024-03-06T07:23:10.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d4019070 con 0x7f39dc15e1d0 2024-03-06T07:23:10.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc0cd980 con 0x7f39dc15e1d0 2024-03-06T07:23:10.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d4014800 con 0x7f39dc15e1d0 2024-03-06T07:23:10.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.571+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc0c7d10 con 0x7f39dc15e1d0 2024-03-06T07:23:10.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.572+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc0cdd70 con 0x7f39dc15e1d0 2024-03-06T07:23:10.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.572+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f39d40149a0 con 0x7f39dc15e1d0 2024-03-06T07:23:10.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.573+0000 7f39caffd640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f39bc06b4d0 0x7f39bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.573+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39d408aa30 con 0x7f39dc15e1d0 2024-03-06T07:23:10.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.573+0000 7f39da59c640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f39bc06b4d0 0x7f39bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.573+0000 7f39da59c640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f39bc06b4d0 0x7f39bc06d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39d0036000 tx=0x7f39d0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.581+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39d40598b0 con 0x7f39dc15e1d0 2024-03-06T07:23:10.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.804+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39dc08dde0 con 0x7f39dc15e1d0 2024-03-06T07:23:10.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.804+0000 7f39caffd640 1 -- 172.21.15.72:0/547090813 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f39dc08dde0 con 0x7f39dc15e1d0 2024-03-06T07:23:10.807 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:10.807 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:23:10.807 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:23:10.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f39bc06b4d0 msgr2=0x7f39bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f39bc06b4d0 0x7f39bc06d990 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39d0036000 tx=0x7f39d0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:10.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 msgr2=0x7f39dc14cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 0x7f39dc14cb90 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f39d4007e90 tx=0x7f39d400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 shutdown_connections 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f39bc06b4d0 0x7f39bc06d990 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39dc15e1d0 0x7f39dc14cb90 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39dc15ddf0 0x7f39dc1f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 --2- 172.21.15.72:0/547090813 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39dc0d06d0 0x7f39dc157c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 >> 172.21.15.72:0/547090813 conn(0x7f39dc09cff0 msgr2=0x7f39dc15e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 shutdown_connections 2024-03-06T07:23:10.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.807+0000 7f39e09d7640 1 -- 172.21.15.72:0/547090813 wait complete. 2024-03-06T07:23:10.821 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:10.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.920+0000 7f358f181640 1 Processor -- start 2024-03-06T07:23:10.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.920+0000 7f358f181640 1 -- start start 2024-03-06T07:23:10.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358f181640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3588066d90 0x7f35880ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358f181640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35880cd450 con 0x7f35880b4f70 2024-03-06T07:23:10.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358f181640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35880cd730 con 0x7f358806c6d0 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358f181640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358815e9f0 con 0x7f3588066d90 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358d71e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3588066d90 0x7f35880ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358cf1d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f35880b4f70 0x7f35880b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45550/0 (socket says 172.21.15.72:45550) 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358cf1d640 1 -- 172.21.15.72:0/2978371633 learned_addr learned my addr 172.21.15.72:0/2978371633 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358d71e640 1 -- 172.21.15.72:0/2978371633 >> v1:172.21.15.146:6789/0 conn(0x7f358806c6d0 legacy=0x7f358815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358d71e640 1 -- 172.21.15.72:0/2978371633 >> v1:172.21.15.72:6789/0 conn(0x7f35880b4f70 legacy=0x7f35880b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f35877fe640 1 -- 172.21.15.72:0/2978371633 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708869914 0 0) 0x7f35880cd450 con 0x7f35880b4f70 2024-03-06T07:23:10.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.921+0000 7f358d71e640 1 -- 172.21.15.72:0/2978371633 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358815e680 con 0x7f3588066d90 2024-03-06T07:23:10.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.922+0000 7f358d71e640 1 --2- 172.21.15.72:0/2978371633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35880ccb50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3580009d90 tx=0x7f358002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11201f4b634d018b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.922+0000 7f35877fe640 1 -- 172.21.15.72:0/2978371633 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580039070 con 0x7f3588066d90 2024-03-06T07:23:10.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.922+0000 7f35877fe640 1 -- 172.21.15.72:0/2978371633 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358002cc60 con 0x7f3588066d90 2024-03-06T07:23:10.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.922+0000 7f35877fe640 1 -- 172.21.15.72:0/2978371633 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580034660 con 0x7f3588066d90 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 -- 172.21.15.72:0/2978371633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 msgr2=0x7f35880ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 --2- 172.21.15.72:0/2978371633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35880ccb50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3580009d90 tx=0x7f358002beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 -- 172.21.15.72:0/2978371633 shutdown_connections 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 --2- 172.21.15.72:0/2978371633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35880ccb50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 -- 172.21.15.72:0/2978371633 >> 172.21.15.72:0/2978371633 conn(0x7f35880724d0 msgr2=0x7f35880728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 -- 172.21.15.72:0/2978371633 shutdown_connections 2024-03-06T07:23:10.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.923+0000 7f358f181640 1 -- 172.21.15.72:0/2978371633 wait complete. 2024-03-06T07:23:10.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.924+0000 7f358f181640 1 Processor -- start 2024-03-06T07:23:10.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.924+0000 7f358f181640 1 -- start start 2024-03-06T07:23:10.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358f181640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35881e78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358f181640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f35880b4f70 0x7f35881484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358f181640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3588159560 0x7f35880c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358f181640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3588161ad0 con 0x7f3588066d90 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358f181640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35880cda10 con 0x7f3588159560 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358f181640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3588161c50 con 0x7f35880b4f70 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35881e78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35881e78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56348/0 (socket says 172.21.15.72:56348) 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 -- 172.21.15.72:0/2892855832 learned_addr learned my addr 172.21.15.72:0/2892855832 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358d71e640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3588159560 0x7f35880c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f3587fff640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f35880b4f70 0x7f35881484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 -- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f35880b4f70 msgr2=0x7f35881484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f35880b4f70 0x7f35881484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 -- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3588159560 msgr2=0x7f35880c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3588159560 0x7f35880c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 -- 172.21.15.72:0/2892855832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358815abf0 con 0x7f3588066d90 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.925+0000 7f358cf1d640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35881e78f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f357800c430 tx=0x7f357800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.926+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35780074e0 con 0x7f3588066d90 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.926+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35780047e0 con 0x7f3588066d90 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.926+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35880ca570 con 0x7f3588066d90 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.926+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578004bd0 con 0x7f3588066d90 2024-03-06T07:23:10.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.926+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35880cab30 con 0x7f3588066d90 2024-03-06T07:23:10.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.927+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358804f330 con 0x7f3588066d90 2024-03-06T07:23:10.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.927+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3578011070 con 0x7f3588066d90 2024-03-06T07:23:10.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.927+0000 7f35857fa640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f356806b500 0x7f356806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:10.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.927+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3578089e70 con 0x7f3588066d90 2024-03-06T07:23:10.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.928+0000 7f3587fff640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f356806b500 0x7f356806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:10.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.928+0000 7f3587fff640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f356806b500 0x7f356806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f358815b980 tx=0x7f3570009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:10.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:10.932+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3578058cf0 con 0x7f3588066d90 2024-03-06T07:23:11.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.147+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35880b1860 con 0x7f3588066d90 2024-03-06T07:23:11.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.148+0000 7f35857fa640 1 -- 172.21.15.72:0/2892855832 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f35780585f0 con 0x7f3588066d90 2024-03-06T07:23:11.150 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:11.150 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:22:46.019808+0000","modified":"2024-03-06T07:22:48.025103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5196},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5196":{"gid":5196,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1659735808","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1659735808},{"type":"v1","addr":"172.21.15.72:6837","nonce":1659735808}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5196,"qdb_cluster":[5196]},"id":2}]} 2024-03-06T07:23:11.150 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 14 2024-03-06T07:23:11.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f356806b500 msgr2=0x7f356806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f356806b500 0x7f356806d9c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f358815b980 tx=0x7f3570009040 comp rx=0 tx=0).stop 2024-03-06T07:23:11.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 msgr2=0x7f35881e78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35881e78f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f357800c430 tx=0x7f357800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 shutdown_connections 2024-03-06T07:23:11.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f356806b500 0x7f356806d9c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3588159560 0x7f35880c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f35880b4f70 0x7f35881484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 --2- 172.21.15.72:0/2892855832 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3588066d90 0x7f35881e78f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 >> 172.21.15.72:0/2892855832 conn(0x7f35880724d0 msgr2=0x7f35880b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 shutdown_connections 2024-03-06T07:23:11.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.150+0000 7f358f181640 1 -- 172.21.15.72:0/2892855832 wait complete. 2024-03-06T07:23:11.163 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:11.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.261+0000 7f2b1dcdf640 1 Processor -- start 2024-03-06T07:23:11.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.261+0000 7f2b1dcdf640 1 -- start start 2024-03-06T07:23:11.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.262+0000 7f2b1dcdf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b180b4f70 0x7f2b180b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.262+0000 7f2b1dcdf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b18066d90 0x7f2b180ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.262+0000 7f2b1dcdf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b180b5890 con 0x7f2b1806c6d0 2024-03-06T07:23:11.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.262+0000 7f2b1dcdf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1815e9f0 con 0x7f2b180b4f70 2024-03-06T07:23:11.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.262+0000 7f2b1dcdf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cda10 con 0x7f2b18066d90 2024-03-06T07:23:11.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b177fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b180b4f70 0x7f2b180b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b17fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b18066d90 0x7f2b180ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b16ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f2b1806c6d0 0x7f2b1815ee80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:60746/0 (socket says 172.21.15.72:60746) 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b16ffd640 1 -- 172.21.15.72:0/4049481133 learned_addr learned my addr 172.21.15.72:0/4049481133 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b177fe640 1 -- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b18066d90 msgr2=0x7f2b180ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b177fe640 1 --2- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b18066d90 0x7f2b180ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b177fe640 1 -- 172.21.15.72:0/4049481133 >> v1:172.21.15.146:6789/0 conn(0x7f2b1806c6d0 legacy=0x7f2b1815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b177fe640 1 -- 172.21.15.72:0/4049481133 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180cd450 con 0x7f2b180b4f70 2024-03-06T07:23:11.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.263+0000 7f2b17fff640 1 --2- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b18066d90 0x7f2b180ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:11.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b177fe640 1 --2- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b180b4f70 0x7f2b180b5350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009d90 tx=0x7f2b0802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c5a0e51475d3d63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b167fc640 1 -- 172.21.15.72:0/4049481133 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08039070 con 0x7f2b180b4f70 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b167fc640 1 -- 172.21.15.72:0/4049481133 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0802cc50 con 0x7f2b180b4f70 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b167fc640 1 -- 172.21.15.72:0/4049481133 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b08034440 con 0x7f2b180b4f70 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b180b4f70 msgr2=0x7f2b180b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b180b4f70 0x7f2b180b5350 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2b08009d90 tx=0x7f2b0802bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/4049481133 shutdown_connections 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b18066d90 0x7f2b180ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/4049481133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b180b4f70 0x7f2b180b5350 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.264+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/4049481133 >> 172.21.15.72:0/4049481133 conn(0x7f2b180724d0 msgr2=0x7f2b180728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.265+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/4049481133 shutdown_connections 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.265+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/4049481133 wait complete. 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.266+0000 7f2b1dcdf640 1 Processor -- start 2024-03-06T07:23:11.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.266+0000 7f2b1dcdf640 1 -- start start 2024-03-06T07:23:11.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.266+0000 7f2b1dcdf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18066d90 0x7f2b180d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.266+0000 7f2b1dcdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 0x7f2b18160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.266+0000 7f2b1dcdf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b180af320 0x7f2b18160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.266+0000 7f2b1dcdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cd090 con 0x7f2b180b4f70 2024-03-06T07:23:11.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b1dcdf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1815e680 con 0x7f2b18066d90 2024-03-06T07:23:11.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b1dcdf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b180cd210 con 0x7f2b180af320 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 0x7f2b18160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 0x7f2b18160640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56352/0 (socket says 172.21.15.72:56352) 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 -- 172.21.15.72:0/2530445752 learned_addr learned my addr 172.21.15.72:0/2530445752 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b177fe640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18066d90 0x7f2b180d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b17fff640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b180af320 0x7f2b18160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 -- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b180af320 msgr2=0x7f2b18160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b180af320 0x7f2b18160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 -- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18066d90 msgr2=0x7f2b180d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18066d90 0x7f2b180d09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 -- 172.21.15.72:0/2530445752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b180d0f20 con 0x7f2b180b4f70 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b16ffd640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 0x7f2b18160640 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000c400 tx=0x7f2b0000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00017070 con 0x7f2b180b4f70 2024-03-06T07:23:11.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b00005ce0 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b180ae560 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b000072f0 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.267+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b180aeb20 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.268+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b18074880 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.269+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b0001a070 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.269+0000 7f2b0ffff640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2af806b4d0 0x7f2af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.269+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b000894d0 con 0x7f2b180b4f70 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.269+0000 7f2b177fe640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2af806b4d0 0x7f2af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.270+0000 7f2b177fe640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2af806b4d0 0x7f2af806d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b08036000 tx=0x7f2b08030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.278+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b00057d10 con 0x7f2b180b4f70 2024-03-06T07:23:11.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.451+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b180b1860 con 0x7f2b180b4f70 2024-03-06T07:23:11.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.453+0000 7f2b0ffff640 1 -- 172.21.15.72:0/2530445752 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16681 (secure 0 0 0) 0x7f2b0005bdb0 con 0x7f2b180b4f70 2024-03-06T07:23:11.455 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:11.455 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":34,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:22:47.667116+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:22:44.323650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-06T07:22:44.983501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:11.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2af806b4d0 msgr2=0x7f2af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2af806b4d0 0x7f2af806d990 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2b08036000 tx=0x7f2b08030000 comp rx=0 tx=0).stop 2024-03-06T07:23:11.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 msgr2=0x7f2b18160640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 0x7f2b18160640 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2b0000c400 tx=0x7f2b0000c800 comp rx=0 tx=0).stop 2024-03-06T07:23:11.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 shutdown_connections 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2af806b4d0 0x7f2af806d990 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b180af320 0x7f2b18160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b180b4f70 0x7f2b18160640 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 --2- 172.21.15.72:0/2530445752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b18066d90 0x7f2b180d09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 >> 172.21.15.72:0/2530445752 conn(0x7f2b180724d0 msgr2=0x7f2b1806bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 shutdown_connections 2024-03-06T07:23:11.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.455+0000 7f2b1dcdf640 1 -- 172.21.15.72:0/2530445752 wait complete. 2024-03-06T07:23:11.469 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.569+0000 7fbbd09e9640 1 Processor -- start 2024-03-06T07:23:11.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.569+0000 7fbbd09e9640 1 -- start start 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbd09e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 0x7fbbcc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbd09e9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbcc0d22c0 con 0x7fbbcc162970 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbd09e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbcc0d25a0 con 0x7fbbcc0b4460 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbd09e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc161b00 con 0x7fbbcc162400 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbc3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbbcc162970 0x7fbbcc161cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45566/0 (socket says 172.21.15.72:45566) 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbc3fff640 1 -- 172.21.15.72:0/3835460725 learned_addr learned my addr 172.21.15.72:0/3835460725 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049705338 0 0) 0x7fbbcc0d22c0 con 0x7fbbcc162970 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbcad9d640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 0x7fbbcc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.570+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb8003520 con 0x7fbbcc162970 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3829067592 0 0) 0x7fbbb8003520 con 0x7fbbcc162970 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 msgr2=0x7fbbcc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 0x7fbbcc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 >> v1:172.21.15.146:6789/0 conn(0x7fbbcc0b4460 legacy=0x7fbbcc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc0d2880 con 0x7fbbcc162970 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fbbb4001670 con 0x7fbbcc162970 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 >> v1:172.21.15.72:6789/0 conn(0x7fbbcc162970 legacy=0x7fbbcc161cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbb80039e0 0x7fbbb8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 0x7fbbb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc161b00 con 0x7fbbb8004800 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb800baa0 con 0x7fbbb80039e0 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb800bca0 con 0x7fbbcc162400 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbca59c640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbb80039e0 0x7fbbb8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbcad9d640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 0x7fbbb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc3fff640 1 -- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 msgr2=0x7fbbb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc162400 0x7fbbb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc3fff640 1 -- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbb80039e0 msgr2=0x7fbbb8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbb80039e0 0x7fbbb8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.571+0000 7fbbc3fff640 1 -- 172.21.15.72:0/3835460725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8003520 con 0x7fbbb8004800 2024-03-06T07:23:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.572+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbb8004c80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4001eb0 tx=0x7fbbb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab17e163ad3fcdf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.572+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007d30 con 0x7fbbb8004800 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.572+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb4030070 con 0x7fbbb8004800 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.572+0000 7fbbc9d9b640 1 -- 172.21.15.72:0/3835460725 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4005d30 con 0x7fbbb8004800 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbd09e9640 1 -- 172.21.15.72:0/3835460725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 msgr2=0x7fbbb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/3835460725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbb8004c80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4001eb0 tx=0x7fbbb8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbcad9d640 1 -- 172.21.15.72:0/3835460725 reap_dead start 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbd09e9640 1 -- 172.21.15.72:0/3835460725 shutdown_connections 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbd09e9640 1 -- 172.21.15.72:0/3835460725 >> 172.21.15.72:0/3835460725 conn(0x7fbbcc0d3ce0 msgr2=0x7fbbcc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbd09e9640 1 -- 172.21.15.72:0/3835460725 shutdown_connections 2024-03-06T07:23:11.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.573+0000 7fbbd09e9640 1 -- 172.21.15.72:0/3835460725 wait complete. 2024-03-06T07:23:11.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.574+0000 7fbbd09e9640 1 Processor -- start 2024-03-06T07:23:11.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.574+0000 7fbbd09e9640 1 -- start start 2024-03-06T07:23:11.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbcc155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc0ccdb0 0x7fbbcc0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbcc0b2590 0x7fbbcc0b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc0ce950 con 0x7fbbb8004800 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc0cecc0 con 0x7fbbcc0b2590 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbcc0b4840 con 0x7fbbcc0ccdb0 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbcc155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbcc155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56374/0 (socket says 172.21.15.72:56374) 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 -- 172.21.15.72:0/1745382093 learned_addr learned my addr 172.21.15.72:0/1745382093 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbcad9d640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbcc0b2590 0x7fbbcc0b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc0ccdb0 0x7fbbcc0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 -- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc0ccdb0 msgr2=0x7fbbcc0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc0ccdb0 0x7fbbcc0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 -- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbcc0b2590 msgr2=0x7fbbcc0b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbcc0b2590 0x7fbbcc0b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 -- 172.21.15.72:0/1745382093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbcc0b6900 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbcad9d640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbcc0b2590 0x7fbbcc0b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbca59c640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbcc155b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00f620 tx=0x7fbbbc00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0052e0 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc018070 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbcc0b6be0 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc003810 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.575+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbcc0b71a0 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.576+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbcc04f330 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.577+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbbbc0039b0 con 0x7fbbb8004800 2024-03-06T07:23:11.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.577+0000 7fbbc2ffd640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbbac06b4d0 0x7fbbac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.577+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbbc08a050 con 0x7fbbb8004800 2024-03-06T07:23:11.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.577+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbbac06b4d0 0x7fbbac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.578+0000 7fbbc3fff640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbbac06b4d0 0x7fbbac06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4034000 tx=0x7fbbb4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.582+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbbbc058ed0 con 0x7fbbb8004800 2024-03-06T07:23:11.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.792+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbbcc06b180 con 0x7fbbb8004800 2024-03-06T07:23:11.794 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:23:11.792+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:11.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.793+0000 7fbbc2ffd640 1 -- 172.21.15.72:0/1745382093 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v15) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbbbc0587d0 con 0x7fbbb8004800 2024-03-06T07:23:11.795 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbbac06b4d0 msgr2=0x7fbbac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbbac06b4d0 0x7fbbac06d990 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4034000 tx=0x7fbbb4009030 comp rx=0 tx=0).stop 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 msgr2=0x7fbbcc155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbcc155b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc00f620 tx=0x7fbbbc00fae0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 shutdown_connections 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbbac06b4d0 0x7fbbac06d990 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbcc0b2590 0x7fbbcc0b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbcc0ccdb0 0x7fbbcc0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 --2- 172.21.15.72:0/1745382093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbbb8004800 0x7fbbcc155b20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.796+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 >> 172.21.15.72:0/1745382093 conn(0x7fbbcc0d3ce0 msgr2=0x7fbbcc0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.797+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 shutdown_connections 2024-03-06T07:23:11.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.797+0000 7fbbd09e9640 1 -- 172.21.15.72:0/1745382093 wait complete. 2024-03-06T07:23:11.810 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:11.835 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:23:11.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.911+0000 7fde870a4640 1 Processor -- start 2024-03-06T07:23:11.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.911+0000 7fde870a4640 1 -- start start 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde870a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde801614d0 0x7fde800d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde870a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde801618b0 con 0x7fde800d24b0 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde870a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde800cf400 con 0x7fde800d1910 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde870a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde800cf6a0 con 0x7fde801614d0 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde84e40640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde801614d0 0x7fde800d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde85641640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fde800d24b0 0x7fde800b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45574/0 (socket says 172.21.15.72:45574) 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.912+0000 7fde85641640 1 -- 172.21.15.72:0/2973135233 learned_addr learned my addr 172.21.15.72:0/2973135233 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde84e40640 1 -- 172.21.15.72:0/2973135233 >> v1:172.21.15.146:6789/0 conn(0x7fde800d1910 legacy=0x7fde800d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde84e40640 1 -- 172.21.15.72:0/2973135233 >> v1:172.21.15.72:6789/0 conn(0x7fde800d24b0 legacy=0x7fde800b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde84e40640 1 -- 172.21.15.72:0/2973135233 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800cf980 con 0x7fde801614d0 2024-03-06T07:23:11.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde84e40640 1 --2- 172.21.15.72:0/2973135233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde801614d0 0x7fde800d4860 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fde8008c470 tx=0x7fde6802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d369ee7e530913 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde7f7fe640 1 -- 172.21.15.72:0/2973135233 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68035020 con 0x7fde801614d0 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde7f7fe640 1 -- 172.21.15.72:0/2973135233 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde68031800 con 0x7fde801614d0 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.913+0000 7fde7f7fe640 1 -- 172.21.15.72:0/2973135233 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68031bd0 con 0x7fde801614d0 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 -- 172.21.15.72:0/2973135233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde801614d0 msgr2=0x7fde800d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 --2- 172.21.15.72:0/2973135233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde801614d0 0x7fde800d4860 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fde8008c470 tx=0x7fde6802b8a0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 -- 172.21.15.72:0/2973135233 shutdown_connections 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 --2- 172.21.15.72:0/2973135233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde801614d0 0x7fde800d4860 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 -- 172.21.15.72:0/2973135233 >> 172.21.15.72:0/2973135233 conn(0x7fde800d3d40 msgr2=0x7fde800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 -- 172.21.15.72:0/2973135233 shutdown_connections 2024-03-06T07:23:11.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.914+0000 7fde870a4640 1 -- 172.21.15.72:0/2973135233 wait complete. 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.915+0000 7fde870a4640 1 Processor -- start 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.915+0000 7fde870a4640 1 -- start start 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde870a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 0x7fde80154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde870a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde800d24b0 0x7fde800cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde870a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde801614d0 0x7fde800cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde870a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b1540 con 0x7fde800d1910 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde870a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b0910 con 0x7fde800d24b0 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde870a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde800b16c0 con 0x7fde801614d0 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 0x7fde80154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 0x7fde80154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56406/0 (socket says 172.21.15.72:56406) 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 -- 172.21.15.72:0/2944485548 learned_addr learned my addr 172.21.15.72:0/2944485548 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 -- 172.21.15.72:0/2944485548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde801614d0 msgr2=0x7fde800cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde801614d0 0x7fde800cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 -- 172.21.15.72:0/2944485548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde800d24b0 msgr2=0x7fde800cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde800d24b0 0x7fde800cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.916+0000 7fde84e40640 1 -- 172.21.15.72:0/2944485548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800cb7f0 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.917+0000 7fde84e40640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 0x7fde80154da0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fde68009890 tx=0x7fde680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.917+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6803ad60 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.917+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde68045450 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.917+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde800cbad0 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.917+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68045840 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.917+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde800cc090 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.918+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde80068ff0 con 0x7fde800d1910 2024-03-06T07:23:11.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.918+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde68038020 con 0x7fde800d1910 2024-03-06T07:23:11.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.918+0000 7fde7d7fa640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde6006b4d0 0x7fde6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:11.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.919+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fde680ac5b0 con 0x7fde800d1910 2024-03-06T07:23:11.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.919+0000 7fde7ffff640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde6006b4d0 0x7fde6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:11.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.919+0000 7fde7ffff640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde6006b4d0 0x7fde6006d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fde8006d7b0 tx=0x7fde700079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:11.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:11.923+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde6807b3d0 con 0x7fde800d1910 2024-03-06T07:23:12.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.131+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fde800cff40 con 0x7fde800d1910 2024-03-06T07:23:12.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.798+0000 7fde7d7fa640 1 -- 172.21.15.72:0/2944485548 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v16) v1 ==== 104+0+0 (secure 0 0 0) 0x7fde6807acd0 con 0x7fde800d1910 2024-03-06T07:23:12.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde6006b4d0 msgr2=0x7fde6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde6006b4d0 0x7fde6006d990 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fde8006d7b0 tx=0x7fde700079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 msgr2=0x7fde80154da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 0x7fde80154da0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fde68009890 tx=0x7fde680046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 shutdown_connections 2024-03-06T07:23:12.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde6006b4d0 0x7fde6006d990 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde801614d0 0x7fde800cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde800d24b0 0x7fde800cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 --2- 172.21.15.72:0/2944485548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde800d1910 0x7fde80154da0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 >> 172.21.15.72:0/2944485548 conn(0x7fde800d3d40 msgr2=0x7fde800b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 shutdown_connections 2024-03-06T07:23:12.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.800+0000 7fde870a4640 1 -- 172.21.15.72:0/2944485548 wait complete. 2024-03-06T07:23:12.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:12.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.957+0000 7fd9273b6640 1 Processor -- start 2024-03-06T07:23:12.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.957+0000 7fd9273b6640 1 -- start start 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd9273b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9200d4630 0x7fd9200d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd9273b6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd9200d2820 0x7fd9200cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd9273b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92015e380 con 0x7fd92015dfa0 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd9273b6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200d2c70 con 0x7fd9200d4630 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd9273b6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9200d2100 con 0x7fd9200d2820 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd924951640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9200d4630 0x7fd9200d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd924951640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9200d4630 0x7fd9200d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44682/0 (socket says 172.21.15.72:44682) 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.958+0000 7fd924951640 1 -- 172.21.15.72:0/197396468 learned_addr learned my addr 172.21.15.72:0/197396468 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.959+0000 7fd925953640 1 --2- 172.21.15.72:0/197396468 >> v2:172.21.15.146:3300/0 conn(0x7fd9200d2820 0x7fd9200cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.959+0000 7fd924951640 1 -- 172.21.15.72:0/197396468 >> v2:172.21.15.146:3300/0 conn(0x7fd9200d2820 msgr2=0x7fd9200cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:12.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.959+0000 7fd924951640 1 --2- 172.21.15.72:0/197396468 >> v2:172.21.15.146:3300/0 conn(0x7fd9200d2820 0x7fd9200cb190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.959+0000 7fd924951640 1 -- 172.21.15.72:0/197396468 >> v1:172.21.15.146:6789/0 conn(0x7fd92015dfa0 legacy=0x7fd9200d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.959+0000 7fd924951640 1 -- 172.21.15.72:0/197396468 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200cf470 con 0x7fd9200d4630 2024-03-06T07:23:12.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.959+0000 7fd924951640 1 --2- 172.21.15.72:0/197396468 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200d22e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd9200ce580 tx=0x7fd91002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a975558c06ba411 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd917fff640 1 -- 172.21.15.72:0/197396468 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910035020 con 0x7fd9200d4630 2024-03-06T07:23:12.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd917fff640 1 -- 172.21.15.72:0/197396468 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910031870 con 0x7fd9200d4630 2024-03-06T07:23:12.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd917fff640 1 -- 172.21.15.72:0/197396468 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910031c40 con 0x7fd9200d4630 2024-03-06T07:23:12.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 -- 172.21.15.72:0/197396468 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 msgr2=0x7fd9200d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 --2- 172.21.15.72:0/197396468 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200d22e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd9200ce580 tx=0x7fd91002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 -- 172.21.15.72:0/197396468 shutdown_connections 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 --2- 172.21.15.72:0/197396468 >> v2:172.21.15.146:3300/0 conn(0x7fd9200d2820 0x7fd9200cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 --2- 172.21.15.72:0/197396468 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200d22e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 -- 172.21.15.72:0/197396468 >> 172.21.15.72:0/197396468 conn(0x7fd92009d000 msgr2=0x7fd9200d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 -- 172.21.15.72:0/197396468 shutdown_connections 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.960+0000 7fd9273b6640 1 -- 172.21.15.72:0/197396468 wait complete. 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.961+0000 7fd9273b6640 1 Processor -- start 2024-03-06T07:23:12.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.961+0000 7fd9273b6640 1 -- start start 2024-03-06T07:23:12.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd9273b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9200d2820 0x7fd9200b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd9273b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd9273b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015dfa0 0x7fd9200b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd9273b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cb6d0 con 0x7fd9200d4630 2024-03-06T07:23:12.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd9273b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cfa30 con 0x7fd92015dfa0 2024-03-06T07:23:12.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd9273b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cb850 con 0x7fd9200d2820 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd924951640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd924951640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44686/0 (socket says 172.21.15.72:44686) 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd924951640 1 -- 172.21.15.72:0/2726594789 learned_addr learned my addr 172.21.15.72:0/2726594789 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd924951640 1 -- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9200d2820 msgr2=0x7fd9200b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd924951640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9200d2820 0x7fd9200b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd924951640 1 -- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015dfa0 msgr2=0x7fd9200b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.962+0000 7fd925953640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015dfa0 0x7fd9200b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd924951640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015dfa0 0x7fd9200b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd924951640 1 -- 172.21.15.72:0/2726594789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9201f74c0 con 0x7fd9200d4630 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd924951640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200c9ef0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd91002b950 tx=0x7fd9100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910040070 con 0x7fd9200d4630 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910030d00 con 0x7fd9200d4630 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910035040 con 0x7fd9200d4630 2024-03-06T07:23:12.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9201f7800 con 0x7fd9200d4630 2024-03-06T07:23:12.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.963+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9201f8040 con 0x7fd9200d4630 2024-03-06T07:23:12.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.964+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92004f330 con 0x7fd9200d4630 2024-03-06T07:23:12.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.964+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd91004b060 con 0x7fd9200d4630 2024-03-06T07:23:12.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.965+0000 7fd915ffb640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:12.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.965+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd9100ac010 con 0x7fd9200d4630 2024-03-06T07:23:12.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.965+0000 7fd925152640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:12.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.965+0000 7fd925152640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0012e0 tx=0x7fd91c007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:12.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:12.969+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd9201f7800 con 0x7fd9200d4630 2024-03-06T07:23:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.141+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd92006bab0 con 0x7fd9200d4630 2024-03-06T07:23:13.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.799+0000 7fd915ffb640 1 -- 172.21.15.72:0/2726594789 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 181+0+0 (secure 0 0 0) 0x7fd91007ae30 con 0x7fd9200d4630 2024-03-06T07:23:13.802 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:13.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 msgr2=0x7fd90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd91c0012e0 tx=0x7fd91c007d50 comp rx=0 tx=0).stop 2024-03-06T07:23:13.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 msgr2=0x7fd9200c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200c9ef0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd91002b950 tx=0x7fd9100046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 shutdown_connections 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015dfa0 0x7fd9200b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9200d4630 0x7fd9200c9ef0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 --2- 172.21.15.72:0/2726594789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9200d2820 0x7fd9200b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 >> 172.21.15.72:0/2726594789 conn(0x7fd92009d000 msgr2=0x7fd92006bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 shutdown_connections 2024-03-06T07:23:13.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.802+0000 7fd9273b6640 1 -- 172.21.15.72:0/2726594789 wait complete. 2024-03-06T07:23:13.818 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:13.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.924+0000 7f7e74e37640 1 Processor -- start 2024-03-06T07:23:13.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.924+0000 7f7e74e37640 1 -- start start 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e74e37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e700ce8c0 0x7f7e700cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e74e37640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e7015c980 0x7f7e70071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e74e37640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700ced50 con 0x7f7e7015c5a0 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e74e37640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e70072a20 con 0x7f7e7015c980 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e74e37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700708b0 con 0x7f7e700ce8c0 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e7015c980 0x7f7e70071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e7015c5a0 0x7f7e7006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58490/0 (socket says 172.21.15.72:58490) 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6e59c640 1 -- 172.21.15.72:0/2756359954 learned_addr learned my addr 172.21.15.72:0/2756359954 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e67fff640 1 --2- 172.21.15.72:0/2756359954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e700ce8c0 0x7f7e700cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6ed9d640 1 -- 172.21.15.72:0/2756359954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e700ce8c0 msgr2=0x7f7e700cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6ed9d640 1 --2- 172.21.15.72:0/2756359954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e700ce8c0 0x7f7e700cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6ed9d640 1 -- 172.21.15.72:0/2756359954 >> v1:172.21.15.72:6789/0 conn(0x7f7e7015c5a0 legacy=0x7f7e7006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6ed9d640 1 -- 172.21.15.72:0/2756359954 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700cf030 con 0x7f7e7015c980 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.925+0000 7f7e6dd9b640 1 -- 172.21.15.72:0/2756359954 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144254919 0 0) 0x7f7e700ced50 con 0x7f7e7015c5a0 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.926+0000 7f7e6ed9d640 1 --2- 172.21.15.72:0/2756359954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e70071300 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009d90 tx=0x7f7e6802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dc10e57e4279acc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.926+0000 7f7e6dd9b640 1 -- 172.21.15.72:0/2756359954 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68039070 con 0x7f7e7015c980 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.926+0000 7f7e6dd9b640 1 -- 172.21.15.72:0/2756359954 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6802cbc0 con 0x7f7e7015c980 2024-03-06T07:23:13.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.926+0000 7f7e6dd9b640 1 -- 172.21.15.72:0/2756359954 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68034440 con 0x7f7e7015c980 2024-03-06T07:23:13.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 -- 172.21.15.72:0/2756359954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 msgr2=0x7f7e70071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2756359954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e70071300 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009d90 tx=0x7f7e6802be10 comp rx=0 tx=0).stop 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 -- 172.21.15.72:0/2756359954 shutdown_connections 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2756359954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e70071300 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2756359954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e700ce8c0 0x7f7e700cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 -- 172.21.15.72:0/2756359954 >> 172.21.15.72:0/2756359954 conn(0x7f7e700b1650 msgr2=0x7f7e700b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 -- 172.21.15.72:0/2756359954 shutdown_connections 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.927+0000 7f7e74e37640 1 -- 172.21.15.72:0/2756359954 wait complete. 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.928+0000 7f7e74e37640 1 Processor -- start 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.928+0000 7f7e74e37640 1 -- start start 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e7015c5a0 0x7f7e701f16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e7013eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e7013c9b0 0x7f7e7015e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700cc870 con 0x7f7e7015c980 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e70070e70 con 0x7f7e7015c5a0 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7015ce30 con 0x7f7e7013c9b0 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e7013eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e7013eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44698/0 (socket says 172.21.15.72:44698) 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 -- 172.21.15.72:0/2460316659 learned_addr learned my addr 172.21.15.72:0/2460316659 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e6e59c640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e7015c5a0 0x7f7e701f16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e6ed9d640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e7013c9b0 0x7f7e7015e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 -- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e7013c9b0 msgr2=0x7f7e7015e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e7013c9b0 0x7f7e7015e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 -- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e7015c5a0 msgr2=0x7f7e701f16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e7015c5a0 0x7f7e701f16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:13.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 -- 172.21.15.72:0/2460316659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7013f080 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e67fff640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e7013eb40 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800e9b0 tx=0x7f7e5800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58017070 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5800c9a0 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e701f1d80 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5800cd90 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.929+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e701f2620 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.930+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e700687d0 con 0x7f7e7015c980 2024-03-06T07:23:13.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.931+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e5801a070 con 0x7f7e7015c980 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.931+0000 7f7e66ffd640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e5006b4d0 0x7f7e5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.931+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f7e5808a050 con 0x7f7e7015c980 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.931+0000 7f7e6e59c640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e5006b4d0 0x7f7e5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:13.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.932+0000 7f7e6e59c640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e5006b4d0 0x7f7e5006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7e60000f60 tx=0x7f7e60007b70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:13.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:13.935+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e7013f080 con 0x7f7e7015c980 2024-03-06T07:23:14.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.106+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7e7006dcf0 con 0x7f7e7015c980 2024-03-06T07:23:14.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.803+0000 7f7e66ffd640 1 -- 172.21.15.72:0/2460316659 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7e580590c0 con 0x7f7e7015c980 2024-03-06T07:23:14.806 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e5006b4d0 msgr2=0x7f7e5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e5006b4d0 0x7f7e5006d990 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7e60000f60 tx=0x7f7e60007b70 comp rx=0 tx=0).stop 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 msgr2=0x7f7e7013eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e7013eb40 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800e9b0 tx=0x7f7e5800ee70 comp rx=0 tx=0).stop 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 shutdown_connections 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e5006b4d0 0x7f7e5006d990 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e7013c9b0 0x7f7e7015e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e7015c980 0x7f7e7013eb40 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 --2- 172.21.15.72:0/2460316659 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e7015c5a0 0x7f7e701f16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 >> 172.21.15.72:0/2460316659 conn(0x7f7e700b1650 msgr2=0x7f7e700afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 shutdown_connections 2024-03-06T07:23:14.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.806+0000 7f7e74e37640 1 -- 172.21.15.72:0/2460316659 wait complete. 2024-03-06T07:23:14.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:14.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.929+0000 7f1b45a40640 1 Processor -- start 2024-03-06T07:23:14.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.929+0000 7f1b45a40640 1 -- start start 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.929+0000 7f1b45a40640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b400d4860 0x7f1b400d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.929+0000 7f1b45a40640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400b0e30 0x7f1b400b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.929+0000 7f1b45a40640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b401618b0 con 0x7f1b401614d0 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.929+0000 7f1b45a40640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b400d4cf0 con 0x7f1b400d4860 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b45a40640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400cf400 con 0x7f1b400b0e30 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b400d4860 0x7f1b400d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b400d4860 0x7f1b400d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44714/0 (socket says 172.21.15.72:44714) 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 -- 172.21.15.72:0/3603056513 learned_addr learned my addr 172.21.15.72:0/3603056513 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 -- 172.21.15.72:0/3603056513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400b0e30 msgr2=0x7f1b400b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 --2- 172.21.15.72:0/3603056513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400b0e30 0x7f1b400b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 -- 172.21.15.72:0/3603056513 >> v1:172.21.15.146:6789/0 conn(0x7f1b401614d0 legacy=0x7f1b400d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.930+0000 7f1b37fff640 1 -- 172.21.15.72:0/3603056513 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400cf6a0 con 0x7f1b400d4860 2024-03-06T07:23:14.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b37fff640 1 --2- 172.21.15.72:0/3603056513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400d4860 0x7f1b400d2020 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009870 tx=0x7f1b2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad3a19cece682c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b3e7fc640 1 -- 172.21.15.72:0/3603056513 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28035020 con 0x7f1b400d4860 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b3e7fc640 1 -- 172.21.15.72:0/3603056513 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b28031640 con 0x7f1b400d4860 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b3e7fc640 1 -- 172.21.15.72:0/3603056513 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2802d040 con 0x7f1b400d4860 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 -- 172.21.15.72:0/3603056513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400d4860 msgr2=0x7f1b400d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 --2- 172.21.15.72:0/3603056513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400d4860 0x7f1b400d2020 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009870 tx=0x7f1b2802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 -- 172.21.15.72:0/3603056513 shutdown_connections 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 --2- 172.21.15.72:0/3603056513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400b0e30 0x7f1b400b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 --2- 172.21.15.72:0/3603056513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400d4860 0x7f1b400d2020 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 -- 172.21.15.72:0/3603056513 >> 172.21.15.72:0/3603056513 conn(0x7f1b400d3d40 msgr2=0x7f1b400d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 -- 172.21.15.72:0/3603056513 shutdown_connections 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.931+0000 7f1b45a40640 1 -- 172.21.15.72:0/3603056513 wait complete. 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 Processor -- start 2024-03-06T07:23:14.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 -- start start 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 0x7f1b400c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400d4860 0x7f1b400ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b401614d0 0x7f1b400b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400b0910 con 0x7f1b400b0e30 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400cfc60 con 0x7f1b401614d0 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b45a40640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b400d2560 con 0x7f1b400d4860 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.933+0000 7f1b3effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 0x7f1b400c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 0x7f1b400c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44726/0 (socket says 172.21.15.72:44726) 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 -- 172.21.15.72:0/746111730 learned_addr learned my addr 172.21.15.72:0/746111730 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3f7fe640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b401614d0 0x7f1b400b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b37fff640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400d4860 0x7f1b400ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 -- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400d4860 msgr2=0x7f1b400ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400d4860 0x7f1b400ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 -- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b401614d0 msgr2=0x7f1b400b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b401614d0 0x7f1b400b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 -- 172.21.15.72:0/746111730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400c7e50 con 0x7f1b400b0e30 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3effd640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 0x7f1b400c7870 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000c430 tx=0x7f1b3000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30011790 con 0x7f1b400b0e30 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b30017070 con 0x7f1b400b0e30 2024-03-06T07:23:14.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b400c8130 con 0x7f1b400b0e30 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30011b70 con 0x7f1b400b0e30 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.934+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b400b6930 con 0x7f1b400b0e30 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.935+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b400b6d50 con 0x7f1b400b0e30 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.935+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b30011d10 con 0x7f1b400b0e30 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.936+0000 7f1b3cff9640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b2006b4d0 0x7f1b2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.936+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1b30089c10 con 0x7f1b400b0e30 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.936+0000 7f1b37fff640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b2006b4d0 0x7f1b2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:14.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.936+0000 7f1b37fff640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b2006b4d0 0x7f1b2006d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1b28036000 tx=0x7f1b2803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:14.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:14.940+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b30058740 con 0x7f1b400b0e30 2024-03-06T07:23:15.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.118+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1b40068ff0 con 0x7f1b400b0e30 2024-03-06T07:23:15.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.122+0000 7f1b3cff9640 1 -- 172.21.15.72:0/746111730 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1b3005c7e0 con 0x7f1b400b0e30 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b2006b4d0 msgr2=0x7f1b2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b2006b4d0 0x7f1b2006d990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1b28036000 tx=0x7f1b2803c040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 msgr2=0x7f1b400c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 0x7f1b400c7870 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000c430 tx=0x7f1b3000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 shutdown_connections 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b2006b4d0 0x7f1b2006d990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b401614d0 0x7f1b400b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b400d4860 0x7f1b400ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 --2- 172.21.15.72:0/746111730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b400b0e30 0x7f1b400c7870 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 >> 172.21.15.72:0/746111730 conn(0x7f1b400d3d40 msgr2=0x7f1b4006bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 shutdown_connections 2024-03-06T07:23:15.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.124+0000 7f1b45a40640 1 -- 172.21.15.72:0/746111730 wait complete. 2024-03-06T07:23:15.137 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-03-06T07:23:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.235+0000 7f50a2525640 1 Processor -- start 2024-03-06T07:23:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.235+0000 7f50a2525640 1 -- start start 2024-03-06T07:23:15.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f50a2525640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f509c0d0320 0x7f509c0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f50a2525640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f509c0b54b0 0x7f509c0cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f50a2525640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f509c15c270 con 0x7f509c0d0700 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f50a2525640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0b04e0 con 0x7f509c0d0320 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f50a2525640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509c0d0e90 con 0x7f509c0b54b0 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f509bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f509c0d0320 0x7f509c0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f509bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f509c0d0320 0x7f509c0b4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44736/0 (socket says 172.21.15.72:44736) 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.236+0000 7f509bfff640 1 -- 172.21.15.72:0/3597300442 learned_addr learned my addr 172.21.15.72:0/3597300442 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f50a0ac2640 1 --2- 172.21.15.72:0/3597300442 >> v2:172.21.15.146:3300/0 conn(0x7f509c0b54b0 0x7f509c0cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f509bfff640 1 -- 172.21.15.72:0/3597300442 >> v2:172.21.15.146:3300/0 conn(0x7f509c0b54b0 msgr2=0x7f509c0cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f509bfff640 1 --2- 172.21.15.72:0/3597300442 >> v2:172.21.15.146:3300/0 conn(0x7f509c0b54b0 0x7f509c0cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f509bfff640 1 -- 172.21.15.72:0/3597300442 >> v1:172.21.15.146:6789/0 conn(0x7f509c0d0700 legacy=0x7f509c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f509bfff640 1 -- 172.21.15.72:0/3597300442 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c0b5ae0 con 0x7f509c0d0320 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f509bfff640 1 --2- 172.21.15.72:0/3597300442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0320 0x7f509c0b4f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f509c073570 tx=0x7f508c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d494cd15981c587 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.237+0000 7f509affd640 1 -- 172.21.15.72:0/3597300442 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c035020 con 0x7f509c0d0320 2024-03-06T07:23:15.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f509affd640 1 -- 172.21.15.72:0/3597300442 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c031870 con 0x7f509c0d0320 2024-03-06T07:23:15.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f509affd640 1 -- 172.21.15.72:0/3597300442 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c031c40 con 0x7f509c0d0320 2024-03-06T07:23:15.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 -- 172.21.15.72:0/3597300442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0320 msgr2=0x7f509c0b4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 --2- 172.21.15.72:0/3597300442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0320 0x7f509c0b4f70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f509c073570 tx=0x7f508c02b910 comp rx=0 tx=0).stop 2024-03-06T07:23:15.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 -- 172.21.15.72:0/3597300442 shutdown_connections 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 --2- 172.21.15.72:0/3597300442 >> v2:172.21.15.146:3300/0 conn(0x7f509c0b54b0 0x7f509c0cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 --2- 172.21.15.72:0/3597300442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0320 0x7f509c0b4f70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 -- 172.21.15.72:0/3597300442 >> 172.21.15.72:0/3597300442 conn(0x7f509c0724d0 msgr2=0x7f509c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 -- 172.21.15.72:0/3597300442 shutdown_connections 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.238+0000 7f50a2525640 1 -- 172.21.15.72:0/3597300442 wait complete. 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.239+0000 7f50a2525640 1 Processor -- start 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.239+0000 7f50a2525640 1 -- start start 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a2525640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f509c0b54b0 0x7f509c156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a2525640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f509c0d0320 0x7f509c099810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a2525640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 0x7f509c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a2525640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0b60a0 con 0x7f509c0d0700 2024-03-06T07:23:15.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a2525640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0b0170 con 0x7f509c0d0320 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a2525640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f509c0ca790 con 0x7f509c0b54b0 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 0x7f509c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 0x7f509c159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44744/0 (socket says 172.21.15.72:44744) 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 -- 172.21.15.72:0/3352130620 learned_addr learned my addr 172.21.15.72:0/3352130620 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f509b7fe640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f509c0d0320 0x7f509c099810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f509bfff640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f509c0b54b0 0x7f509c156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 -- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f509c0b54b0 msgr2=0x7f509c156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f509c0b54b0 0x7f509c156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 -- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f509c0d0320 msgr2=0x7f509c099810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.240+0000 7f50a0ac2640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f509c0d0320 0x7f509c099810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f50a0ac2640 1 -- 172.21.15.72:0/3352130620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509c157480 con 0x7f509c0d0700 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f509b7fe640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f509c0d0320 0x7f509c099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f50a0ac2640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 0x7f509c159e40 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5094007e60 tx=0x7f509400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094014420 con 0x7f509c0d0700 2024-03-06T07:23:15.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5094019070 con 0x7f509c0d0700 2024-03-06T07:23:15.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509c157760 con 0x7f509c0d0700 2024-03-06T07:23:15.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5094014820 con 0x7f509c0d0700 2024-03-06T07:23:15.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.241+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509c157d20 con 0x7f509c0d0700 2024-03-06T07:23:15.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.242+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509c04f330 con 0x7f509c0d0700 2024-03-06T07:23:15.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.242+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f50940149c0 con 0x7f509c0d0700 2024-03-06T07:23:15.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.243+0000 7f5098ff9640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f508806b4d0 0x7f508806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.243+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f509408a790 con 0x7f509c0d0700 2024-03-06T07:23:15.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.243+0000 7f509bfff640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f508806b4d0 0x7f508806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.243+0000 7f509bfff640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f508806b4d0 0x7f508806d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f508c036000 tx=0x7f508c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.247+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5094059910 con 0x7f509c0d0700 2024-03-06T07:23:15.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.483+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) v1 -- 0x7f509c099d50 con 0x7f509c0d0700 2024-03-06T07:23:15.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.805+0000 7f5098ff9640 1 -- 172.21.15.72:0/3352130620 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v99) v1 ==== 132+0+0 (secure 0 0 0) 0x7f5094059210 con 0x7f509c0d0700 2024-03-06T07:23:15.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f508806b4d0 msgr2=0x7f508806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f508806b4d0 0x7f508806d990 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f508c036000 tx=0x7f508c03c040 comp rx=0 tx=0).stop 2024-03-06T07:23:15.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 msgr2=0x7f509c159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 0x7f509c159e40 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5094007e60 tx=0x7f509400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 shutdown_connections 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f508806b4d0 0x7f508806d990 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f509c0d0700 0x7f509c159e40 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f509c0d0320 0x7f509c099810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 --2- 172.21.15.72:0/3352130620 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f509c0b54b0 0x7f509c156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 >> 172.21.15.72:0/3352130620 conn(0x7f509c0724d0 msgr2=0x7f509c0cbaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 shutdown_connections 2024-03-06T07:23:15.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.807+0000 7f50a2525640 1 -- 172.21.15.72:0/3352130620 wait complete. 2024-03-06T07:23:15.821 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-03-06T07:23:15.821 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-03-06T07:23:15.821 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:15.821 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:15.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.918+0000 7f1b30df6640 1 Processor -- start 2024-03-06T07:23:15.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.918+0000 7f1b30df6640 1 -- start start 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.919+0000 7f1b30df6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b2c0cdeb0 0x7f1b2c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.919+0000 7f1b30df6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c071f40 con 0x7f1b2c0cce40 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.919+0000 7f1b30df6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c070a70 con 0x7f1b2c1515e0 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.919+0000 7f1b30df6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c0cd2f0 con 0x7f1b2c0cdeb0 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.919+0000 7f1b29d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1b2c0cce40 0x7f1b2c0cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58492/0 (socket says 172.21.15.72:58492) 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.919+0000 7f1b29d9b640 1 -- 172.21.15.72:0/2010193218 learned_addr learned my addr 172.21.15.72:0/2010193218 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:15.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744678046 0 0) 0x7f1b2c071f40 con 0x7f1b2c0cce40 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2ad9d640 1 --2- 172.21.15.72:0/2010193218 >> v2:172.21.15.146:3300/0 conn(0x7f1b2c0cdeb0 0x7f1b2c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b18003520 con 0x7f1b2c0cce40 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3300598839 0 0) 0x7f1b18003520 con 0x7f1b2c0cce40 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 >> v2:172.21.15.146:3300/0 conn(0x7f1b2c0cdeb0 msgr2=0x7f1b2c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 --2- 172.21.15.72:0/2010193218 >> v2:172.21.15.146:3300/0 conn(0x7f1b2c0cdeb0 0x7f1b2c15b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 >> v1:172.21.15.146:6789/0 conn(0x7f1b2c1515e0 legacy=0x7f1b2c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c070d50 con 0x7f1b2c0cce40 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1b1c001690 con 0x7f1b2c0cce40 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 >> v1:172.21.15.72:6789/0 conn(0x7f1b2c0cce40 legacy=0x7f1b2c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b180039b0 0x7f1b18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b180051c0 0x7f1b1800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0cd2f0 con 0x7f1b18004800 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1800c060 con 0x7f1b180039b0 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1800c260 con 0x7f1b180051c0 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2a59c640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b180039b0 0x7f1b18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2ad9d640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b180051c0 0x7f1b1800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b29d9b640 1 -- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b180051c0 msgr2=0x7f1b1800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b180051c0 0x7f1b1800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b29d9b640 1 -- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b180039b0 msgr2=0x7f1b18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b180039b0 0x7f1b18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b2ad9d640 1 -- 172.21.15.72:0/2010193218 reap_dead start 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.920+0000 7f1b29d9b640 1 -- 172.21.15.72:0/2010193218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18003520 con 0x7f1b18004800 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.921+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b18004c80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00c210 tx=0x7f1b18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5d60cea789ae99 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.921+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c007d10 con 0x7f1b18004800 2024-03-06T07:23:15.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.921+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b1c030070 con 0x7f1b18004800 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.921+0000 7f1b2959a640 1 -- 172.21.15.72:0/2010193218 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c005d10 con 0x7f1b18004800 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 -- 172.21.15.72:0/2010193218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 msgr2=0x7f1b18004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b18004c80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00c210 tx=0x7f1b18005830 comp rx=0 tx=0).stop 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 -- 172.21.15.72:0/2010193218 shutdown_connections 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 --2- 172.21.15.72:0/2010193218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b18004c80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 -- 172.21.15.72:0/2010193218 >> 172.21.15.72:0/2010193218 conn(0x7f1b2c0cf8a0 msgr2=0x7f1b2c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 -- 172.21.15.72:0/2010193218 shutdown_connections 2024-03-06T07:23:15.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.922+0000 7f1b30df6640 1 -- 172.21.15.72:0/2010193218 wait complete. 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.923+0000 7f1b30df6640 1 Processor -- start 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.923+0000 7f1b30df6640 1 -- start start 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b30df6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b2c14b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b30df6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2c1530e0 0x7f1b2c153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b30df6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2c0b3f50 0x7f1b2c0b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b30df6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c15b430 con 0x7f1b18004800 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b30df6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c071310 con 0x7f1b2c0b3f50 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b30df6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c15b5b0 con 0x7f1b2c1530e0 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b2c14b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b2c14b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44756/0 (socket says 172.21.15.72:44756) 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 -- 172.21.15.72:0/1824291926 learned_addr learned my addr 172.21.15.72:0/1824291926 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:15.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2ad9d640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2c0b3f50 0x7f1b2c0b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 -- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2c1530e0 msgr2=0x7f1b2c153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2c1530e0 0x7f1b2c153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2c1530e0 0x7f1b2c153560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 -- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2c0b3f50 msgr2=0x7f1b2c0b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2c0b3f50 0x7f1b2c0b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 -- 172.21.15.72:0/1824291926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c15d9f0 con 0x7f1b18004800 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2ad9d640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2c0b3f50 0x7f1b2c0b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.924+0000 7f1b2a59c640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b2c14b460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400e480 tx=0x7f1b1400e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.925+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b140052e0 con 0x7f1b18004800 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.925+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b14018070 con 0x7f1b18004800 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.925+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c15dcd0 con 0x7f1b18004800 2024-03-06T07:23:15.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.925+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b14003830 con 0x7f1b18004800 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.925+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c15e290 con 0x7f1b18004800 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.926+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c068840 con 0x7f1b18004800 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.926+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b140039d0 con 0x7f1b18004800 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.926+0000 7f1b22ffd640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0c06b500 0x7f1b0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.927+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1b14089ad0 con 0x7f1b18004800 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.927+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0c06b500 0x7f1b0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:15.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.927+0000 7f1b29d9b640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0c06b500 0x7f1b0c06d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c034000 tx=0x7f1b1c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:15.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:15.931+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b14058ce0 con 0x7f1b18004800 2024-03-06T07:23:16.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.104+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b2c04f330 con 0x7f1b18004800 2024-03-06T07:23:16.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.105+0000 7f1b22ffd640 1 -- 172.21.15.72:0/1824291926 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13032 (secure 0 0 0) 0x7f1b140585e0 con 0x7f1b18004800 2024-03-06T07:23:16.107 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:16.108 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":37,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:14.800844+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/1659735808":"2024-03-07T07:23:11.793612+0000","172.21.15.72:6835/1659735808":"2024-03-07T07:23:11.793612+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0c06b500 msgr2=0x7f1b0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0c06b500 0x7f1b0c06d9c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c034000 tx=0x7f1b1c007000 comp rx=0 tx=0).stop 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 msgr2=0x7f1b2c14b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b2c14b460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f1b1400e480 tx=0x7f1b1400e940 comp rx=0 tx=0).stop 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 shutdown_connections 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b0c06b500 0x7f1b0c06d9c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b2c0b3f50 0x7f1b2c0b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b2c1530e0 0x7f1b2c153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 --2- 172.21.15.72:0/1824291926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b18004800 0x7f1b2c14b460 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 >> 172.21.15.72:0/1824291926 conn(0x7f1b2c0cf8a0 msgr2=0x7f1b2c0d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 shutdown_connections 2024-03-06T07:23:16.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.107+0000 7f1b30df6640 1 -- 172.21.15.72:0/1824291926 wait complete. 2024-03-06T07:23:16.121 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:16.121 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:16.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 Processor -- start 2024-03-06T07:23:16.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 -- start start 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facf81614d0 0x7facf80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d1910 0x7facf80d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80d2740 con 0x7facf80d22b0 2024-03-06T07:23:16.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf81618b0 con 0x7facf81614d0 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.222+0000 7facfd78c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf80cf400 con 0x7facf80d1910 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facf81614d0 0x7facf80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf77fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7facf80d22b0 0x7facf80d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:37958/0 (socket says 172.21.15.72:37958) 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf77fe640 1 -- 172.21.15.72:0/3232478743 learned_addr learned my addr 172.21.15.72:0/3232478743 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf67fc640 1 --2- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d1910 0x7facf80d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf5ffb640 1 -- 172.21.15.72:0/3232478743 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936378857 0 0) 0x7facf0001110 con 0x7facf80d22b0 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf5ffb640 1 -- 172.21.15.72:0/3232478743 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face0003520 con 0x7facf80d22b0 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf6ffd640 1 -- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d1910 msgr2=0x7facf80d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf6ffd640 1 --2- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d1910 0x7facf80d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf6ffd640 1 -- 172.21.15.72:0/3232478743 >> v1:172.21.15.146:6789/0 conn(0x7facf80d22b0 legacy=0x7facf80d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.223+0000 7facf6ffd640 1 -- 172.21.15.72:0/3232478743 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf80cf6a0 con 0x7facf81614d0 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.224+0000 7facf6ffd640 1 --2- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf81614d0 0x7facf80d4860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7face8009d90 tx=0x7face802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5b1cce781437f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.224+0000 7facf5ffb640 1 -- 172.21.15.72:0/3232478743 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8039070 con 0x7facf81614d0 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.224+0000 7facf5ffb640 1 -- 172.21.15.72:0/3232478743 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face802cc50 con 0x7facf81614d0 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.224+0000 7facf5ffb640 1 -- 172.21.15.72:0/3232478743 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8034440 con 0x7facf81614d0 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 -- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf81614d0 msgr2=0x7facf80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 --2- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf81614d0 0x7facf80d4860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7face8009d90 tx=0x7face802bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 -- 172.21.15.72:0/3232478743 shutdown_connections 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 --2- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d1910 0x7facf80d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 --2- 172.21.15.72:0/3232478743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf81614d0 0x7facf80d4860 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 -- 172.21.15.72:0/3232478743 >> 172.21.15.72:0/3232478743 conn(0x7facf80d3d40 msgr2=0x7facf80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 -- 172.21.15.72:0/3232478743 shutdown_connections 2024-03-06T07:23:16.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.225+0000 7facfd78c640 1 -- 172.21.15.72:0/3232478743 wait complete. 2024-03-06T07:23:16.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.226+0000 7facfd78c640 1 Processor -- start 2024-03-06T07:23:16.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.226+0000 7facfd78c640 1 -- start start 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf80d1910 0x7facf80cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d22b0 0x7facf80b8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 0x7facf80a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80b0910 con 0x7facf81614d0 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf80cfc60 con 0x7facf80d1910 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf80b1e40 con 0x7facf80d22b0 2024-03-06T07:23:16.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 0x7facf80a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 0x7facf80a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44766/0 (socket says 172.21.15.72:44766) 2024-03-06T07:23:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 -- 172.21.15.72:0/2236028110 learned_addr learned my addr 172.21.15.72:0/2236028110 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf6ffd640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf80d1910 0x7facf80cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf67fc640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d22b0 0x7facf80b8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 -- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d22b0 msgr2=0x7facf80b8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d22b0 0x7facf80b8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 -- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf80d1910 msgr2=0x7facf80cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf80d1910 0x7facf80cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 -- 172.21.15.72:0/2236028110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf80cc760 con 0x7facf81614d0 2024-03-06T07:23:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf67fc640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d22b0 0x7facf80b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facf77fe640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 0x7facf80a1290 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7facf000c730 tx=0x7facf000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0017070 con 0x7facf81614d0 2024-03-06T07:23:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf00058b0 con 0x7facf81614d0 2024-03-06T07:23:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf80cb180 con 0x7facf81614d0 2024-03-06T07:23:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0005ce0 con 0x7facf81614d0 2024-03-06T07:23:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.227+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf80cb740 con 0x7facf81614d0 2024-03-06T07:23:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.228+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf80d39b0 con 0x7facf81614d0 2024-03-06T07:23:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.229+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7facf001a070 con 0x7facf81614d0 2024-03-06T07:23:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.229+0000 7facdf7fe640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7face406b4d0 0x7face406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.229+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7facf0089220 con 0x7facf81614d0 2024-03-06T07:23:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.229+0000 7facf6ffd640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7face406b4d0 0x7face406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.230+0000 7facf6ffd640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7face406b4d0 0x7face406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7face8036000 tx=0x7face8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.234+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7facf80d39b0 con 0x7facf81614d0 2024-03-06T07:23:16.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.405+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7facf80b4080 con 0x7facf81614d0 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.405+0000 7facdf7fe640 1 -- 172.21.15.72:0/2236028110 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7facf80d2740 con 0x7facf81614d0 2024-03-06T07:23:16.408 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7face406b4d0 msgr2=0x7face406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7face406b4d0 0x7face406d990 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7face8036000 tx=0x7face8030000 comp rx=0 tx=0).stop 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 msgr2=0x7facf80a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 0x7facf80a1290 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7facf000c730 tx=0x7facf000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 shutdown_connections 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7face406b4d0 0x7face406d990 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7facf81614d0 0x7facf80a1290 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf80d22b0 0x7facf80b8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 --2- 172.21.15.72:0/2236028110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf80d1910 0x7facf80cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.407+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 >> 172.21.15.72:0/2236028110 conn(0x7facf80d3d40 msgr2=0x7facf806d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.408+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 shutdown_connections 2024-03-06T07:23:16.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.408+0000 7facfd78c640 1 -- 172.21.15.72:0/2236028110 wait complete. 2024-03-06T07:23:16.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:16.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.519+0000 7f5eaf1d0640 1 Processor -- start 2024-03-06T07:23:16.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.519+0000 7f5eaf1d0640 1 -- start start 2024-03-06T07:23:16.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5eaf1d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ea815ee80 0x7f5ea80b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5eaf1d0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ea8070930 0x7f5ea80b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5eaf1d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea8070da0 con 0x7f5ea81443d0 2024-03-06T07:23:16.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5eaf1d0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea80b5060 con 0x7f5ea8070930 2024-03-06T07:23:16.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5eaf1d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea80ce610 con 0x7f5ea815ee80 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ead76d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ea8070930 0x7f5ea80b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ead76d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ea8070930 0x7f5ea80b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44768/0 (socket says 172.21.15.72:44768) 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ead76d640 1 -- 172.21.15.72:0/1090325830 learned_addr learned my addr 172.21.15.72:0/1090325830 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ea7fff640 1 --2- 172.21.15.72:0/1090325830 >> v2:172.21.15.146:3300/0 conn(0x7f5ea815ee80 0x7f5ea80b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ead76d640 1 -- 172.21.15.72:0/1090325830 >> v2:172.21.15.146:3300/0 conn(0x7f5ea815ee80 msgr2=0x7f5ea80b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ead76d640 1 --2- 172.21.15.72:0/1090325830 >> v2:172.21.15.146:3300/0 conn(0x7f5ea815ee80 0x7f5ea80b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.520+0000 7f5ead76d640 1 -- 172.21.15.72:0/1090325830 >> v1:172.21.15.146:6789/0 conn(0x7f5ea81443d0 legacy=0x7f5ea815e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.521+0000 7f5ead76d640 1 -- 172.21.15.72:0/1090325830 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80b3bd0 con 0x7f5ea8070930 2024-03-06T07:23:16.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.521+0000 7f5ead76d640 1 --2- 172.21.15.72:0/1090325830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea80b4b20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5ea806cfc0 tx=0x7f5ea002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6459f7dd8bd9adb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.521+0000 7f5ea77fe640 1 -- 172.21.15.72:0/1090325830 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea002ea90 con 0x7f5ea8070930 2024-03-06T07:23:16.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.521+0000 7f5ea77fe640 1 -- 172.21.15.72:0/1090325830 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea002ec30 con 0x7f5ea8070930 2024-03-06T07:23:16.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.521+0000 7f5ea77fe640 1 -- 172.21.15.72:0/1090325830 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0009600 con 0x7f5ea8070930 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1090325830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 msgr2=0x7f5ea80b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1090325830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea80b4b20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f5ea806cfc0 tx=0x7f5ea002e0f0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1090325830 shutdown_connections 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1090325830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea80b4b20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1090325830 >> v2:172.21.15.146:3300/0 conn(0x7f5ea815ee80 0x7f5ea80b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1090325830 >> 172.21.15.72:0/1090325830 conn(0x7f5ea806b3d0 msgr2=0x7f5ea806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1090325830 shutdown_connections 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.522+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1090325830 wait complete. 2024-03-06T07:23:16.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.523+0000 7f5eaf1d0640 1 Processor -- start 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.523+0000 7f5eaf1d0640 1 -- start start 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eaf1d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eaf1d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ea81443d0 0x7f5ea80af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eaf1d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ea815ee80 0x7f5ea80a9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eaf1d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81447b0 con 0x7f5ea8070930 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eaf1d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80ce2a0 con 0x7f5ea81443d0 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eaf1d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80cdee0 con 0x7f5ea815ee80 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea815a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44782/0 (socket says 172.21.15.72:44782) 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 -- 172.21.15.72:0/1210120744 learned_addr learned my addr 172.21.15.72:0/1210120744 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5ea7fff640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ea81443d0 0x7f5ea80af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5ead76d640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ea815ee80 0x7f5ea80a9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 -- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ea815ee80 msgr2=0x7f5ea80a9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ea815ee80 0x7f5ea80a9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 -- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ea81443d0 msgr2=0x7f5ea80af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ea81443d0 0x7f5ea80af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.524+0000 7f5eacf6c640 1 -- 172.21.15.72:0/1210120744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8141310 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.525+0000 7f5eacf6c640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea815a590 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c430 tx=0x7f5e9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.525+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98011790 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.525+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98017070 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.525+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea81415f0 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.525+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98011b70 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.525+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea8141bb0 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.526+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea804f330 con 0x7f5ea8070930 2024-03-06T07:23:16.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.526+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e98011d10 con 0x7f5ea8070930 2024-03-06T07:23:16.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.527+0000 7f5ea57fa640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.527+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5e98089dc0 con 0x7f5ea8070930 2024-03-06T07:23:16.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.527+0000 7f5ea7fff640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.527+0000 7f5ea7fff640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e9406b4d0 0x7f5e9406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c006fd0 tx=0x7f5e9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.531+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e98058f40 con 0x7f5ea8070930 2024-03-06T07:23:16.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.777+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f5ea8072650 con 0x7f5ea8070930 2024-03-06T07:23:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.812+0000 7f5ea57fa640 1 -- 172.21.15.72:0/1210120744 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v100) v1 ==== 136+0+0 (secure 0 0 0) 0x7f5e98058840 con 0x7f5ea8070930 2024-03-06T07:23:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e9406b4d0 msgr2=0x7f5e9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e9406b4d0 0x7f5e9406d990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c006fd0 tx=0x7f5e9c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 msgr2=0x7f5ea815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea815a590 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c430 tx=0x7f5e9800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 shutdown_connections 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e9406b4d0 0x7f5e9406d990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ea815ee80 0x7f5ea80a9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ea81443d0 0x7f5ea80af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 --2- 172.21.15.72:0/1210120744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ea8070930 0x7f5ea815a590 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 >> 172.21.15.72:0/1210120744 conn(0x7f5ea806b3d0 msgr2=0x7f5ea8071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 shutdown_connections 2024-03-06T07:23:16.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.814+0000 7f5eaf1d0640 1 -- 172.21.15.72:0/1210120744 wait complete. 2024-03-06T07:23:16.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:16.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.928+0000 7f5cb485c640 1 Processor -- start 2024-03-06T07:23:16.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.928+0000 7f5cb485c640 1 -- start start 2024-03-06T07:23:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.928+0000 7f5cb485c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5cb00cff40 0x7f5cb00b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.928+0000 7f5cb485c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb0158ac0 con 0x7f5cb01586e0 2024-03-06T07:23:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.928+0000 7f5cb485c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb015b2a0 con 0x7f5cb00cfa90 2024-03-06T07:23:16.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.928+0000 7f5cb485c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cb00d0390 con 0x7f5cb00cff40 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5caed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5cb00cff40 0x7f5cb00b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5cae59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5cb01586e0 0x7f5cb00aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58506/0 (socket says 172.21.15.72:58506) 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5cae59c640 1 -- 172.21.15.72:0/3292240178 learned_addr learned my addr 172.21.15.72:0/3292240178 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5caed9d640 1 -- 172.21.15.72:0/3292240178 >> v1:172.21.15.146:6789/0 conn(0x7f5cb00cfa90 legacy=0x7f5cb00b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5caed9d640 1 -- 172.21.15.72:0/3292240178 >> v1:172.21.15.72:6789/0 conn(0x7f5cb01586e0 legacy=0x7f5cb00aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5caed9d640 1 -- 172.21.15.72:0/3292240178 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb015b540 con 0x7f5cb00cff40 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.929+0000 7f5cadd9b640 1 -- 172.21.15.72:0/3292240178 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648558047 0 0) 0x7f5cb0158ac0 con 0x7f5cb01586e0 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5caed9d640 1 --2- 172.21.15.72:0/3292240178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb00cff40 0x7f5cb00b3ab0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009d90 tx=0x7f5ca802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7506107bee32bd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cadd9b640 1 -- 172.21.15.72:0/3292240178 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8039070 con 0x7f5cb00cff40 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cadd9b640 1 -- 172.21.15.72:0/3292240178 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca802cc60 con 0x7f5cb00cff40 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cadd9b640 1 -- 172.21.15.72:0/3292240178 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8034660 con 0x7f5cb00cff40 2024-03-06T07:23:16.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 -- 172.21.15.72:0/3292240178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb00cff40 msgr2=0x7f5cb00b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 --2- 172.21.15.72:0/3292240178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb00cff40 0x7f5cb00b3ab0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009d90 tx=0x7f5ca802beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 -- 172.21.15.72:0/3292240178 shutdown_connections 2024-03-06T07:23:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 --2- 172.21.15.72:0/3292240178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb00cff40 0x7f5cb00b3ab0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 -- 172.21.15.72:0/3292240178 >> 172.21.15.72:0/3292240178 conn(0x7f5cb006b190 msgr2=0x7f5cb006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 -- 172.21.15.72:0/3292240178 shutdown_connections 2024-03-06T07:23:16.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.930+0000 7f5cb485c640 1 -- 172.21.15.72:0/3292240178 wait complete. 2024-03-06T07:23:16.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 Processor -- start 2024-03-06T07:23:16.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 -- start start 2024-03-06T07:23:16.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5cb00cfa90 0x7f5cb0147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5cb00cff40 0x7f5cb01f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 0x7f5cb01f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb015bde0 con 0x7f5cb01586e0 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb015bb00 con 0x7f5cb00cfa90 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cb485c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb00af150 con 0x7f5cb00cff40 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5caed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 0x7f5cb01f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5caed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 0x7f5cb01f15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44806/0 (socket says 172.21.15.72:44806) 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5caed9d640 1 -- 172.21.15.72:0/374322682 learned_addr learned my addr 172.21.15.72:0/374322682 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5cae59c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5cb00cfa90 0x7f5cb0147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.932+0000 7f5ca7fff640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5cb00cff40 0x7f5cb01f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5caed9d640 1 -- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5cb00cff40 msgr2=0x7f5cb01f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5caed9d640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5cb00cff40 0x7f5cb01f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5caed9d640 1 -- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5cb00cfa90 msgr2=0x7f5cb0147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5caed9d640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5cb00cfa90 0x7f5cb0147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5caed9d640 1 -- 172.21.15.72:0/374322682 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0148300 con 0x7f5cb01586e0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5cae59c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5cb00cfa90 0x7f5cb0147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5caed9d640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 0x7f5cb01f15a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009d90 tx=0x7f5ca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8044020 con 0x7f5cb01586e0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca803f470 con 0x7f5cb01586e0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb01485e0 con 0x7f5cb01586e0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8039040 con 0x7f5cb01586e0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.933+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb0148ba0 con 0x7f5cb01586e0 2024-03-06T07:23:16.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.934+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb00681e0 con 0x7f5cb01586e0 2024-03-06T07:23:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.934+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ca8047020 con 0x7f5cb01586e0 2024-03-06T07:23:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.935+0000 7f5ca6ffd640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c9006b4d0 0x7f5c9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.935+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5ca80acb50 con 0x7f5cb01586e0 2024-03-06T07:23:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.935+0000 7f5cae59c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c9006b4d0 0x7f5c9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:16.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.935+0000 7f5cae59c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c9006b4d0 0x7f5c9006d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0000f60 tx=0x7f5ca0007cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:16.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:16.939+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5cb00681e0 con 0x7f5cb01586e0 2024-03-06T07:23:17.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.111+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5cb00732b0 con 0x7f5cb01586e0 2024-03-06T07:23:17.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.116+0000 7f5ca6ffd640 1 -- 172.21.15.72:0/374322682 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5ca804e030 con 0x7f5cb01586e0 2024-03-06T07:23:17.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c9006b4d0 msgr2=0x7f5c9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c9006b4d0 0x7f5c9006d990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0000f60 tx=0x7f5ca0007cc0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 msgr2=0x7f5cb01f15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 0x7f5cb01f15a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f5ca8009d90 tx=0x7f5ca8004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:17.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 shutdown_connections 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c9006b4d0 0x7f5c9006d990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5cb01586e0 0x7f5cb01f15a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5cb00cff40 0x7f5cb01f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 --2- 172.21.15.72:0/374322682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5cb00cfa90 0x7f5cb0147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 >> 172.21.15.72:0/374322682 conn(0x7f5cb006b190 msgr2=0x7f5cb0156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 shutdown_connections 2024-03-06T07:23:17.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.118+0000 7f5cb485c640 1 -- 172.21.15.72:0/374322682 wait complete. 2024-03-06T07:23:17.131 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:17.131 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:17.178 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:17.225 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:17.225 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:17.225 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:17.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:17.241 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:17.241 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:17.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:17.258 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:17.258 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:17.296 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:17.296 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:17.296 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:17.314 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:17.314 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:17.314 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:17.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.434+0000 7f8d40e66640 1 Processor -- start 2024-03-06T07:23:17.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.434+0000 7f8d40e66640 1 -- start start 2024-03-06T07:23:17.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d40e66640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8d3c15bb20 0x7f8d3c0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d40e66640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c0cfff0 0x7f8d3c0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d40e66640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d3c0ce4b0 0x7f8d3c0ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d40e66640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c161980 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d40e66640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c161c60 con 0x7f8d3c15bb20 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d40e66640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c161f40 con 0x7f8d3c0cfff0 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d3ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d3c0ce4b0 0x7f8d3c0ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d3ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d3c0ce4b0 0x7f8d3c0ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44818/0 (socket says 172.21.15.72:44818) 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.435+0000 7f8d3ad9d640 1 -- 172.21.15.72:0/3769500852 learned_addr learned my addr 172.21.15.72:0/3769500852 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3a59c640 1 --2- 172.21.15.72:0/3769500852 >> v2:172.21.15.146:3300/0 conn(0x7f8d3c15bb20 0x7f8d3c0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d39d9b640 1 --2- 172.21.15.72:0/3769500852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c0cfff0 0x7f8d3c0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3ad9d640 1 -- 172.21.15.72:0/3769500852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c0cfff0 msgr2=0x7f8d3c0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3ad9d640 1 --2- 172.21.15.72:0/3769500852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c0cfff0 0x7f8d3c0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3ad9d640 1 -- 172.21.15.72:0/3769500852 >> v2:172.21.15.146:3300/0 conn(0x7f8d3c15bb20 msgr2=0x7f8d3c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3ad9d640 1 --2- 172.21.15.72:0/3769500852 >> v2:172.21.15.146:3300/0 conn(0x7f8d3c15bb20 0x7f8d3c0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3ad9d640 1 -- 172.21.15.72:0/3769500852 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c162220 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3ad9d640 1 --2- 172.21.15.72:0/3769500852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c0ce940 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0d2bc0 tx=0x7f8d3402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2ccd350ff62402 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.436+0000 7f8d3959a640 1 -- 172.21.15.72:0/3769500852 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3402ca70 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d3959a640 1 -- 172.21.15.72:0/3769500852 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3402cc10 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d3959a640 1 -- 172.21.15.72:0/3769500852 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d340343f0 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 -- 172.21.15.72:0/3769500852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 msgr2=0x7f8d3c0ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 --2- 172.21.15.72:0/3769500852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c0ce940 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0d2bc0 tx=0x7f8d3402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 -- 172.21.15.72:0/3769500852 shutdown_connections 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 --2- 172.21.15.72:0/3769500852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c0ce940 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 --2- 172.21.15.72:0/3769500852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c0cfff0 0x7f8d3c0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 --2- 172.21.15.72:0/3769500852 >> v2:172.21.15.146:3300/0 conn(0x7f8d3c15bb20 0x7f8d3c0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 -- 172.21.15.72:0/3769500852 >> 172.21.15.72:0/3769500852 conn(0x7f8d3c09a9f0 msgr2=0x7f8d3c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 -- 172.21.15.72:0/3769500852 shutdown_connections 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.437+0000 7f8d40e66640 1 -- 172.21.15.72:0/3769500852 wait complete. 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.438+0000 7f8d40e66640 1 Processor -- start 2024-03-06T07:23:17.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.438+0000 7f8d40e66640 1 -- start start 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d40e66640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d40e66640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0cfff0 0x7f8d3c06ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d40e66640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c15bb20 0x7f8d3c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d40e66640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0d24b0 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d40e66640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0d2330 con 0x7f8d3c0cfff0 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d40e66640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c0d1980 con 0x7f8d3c15bb20 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d3a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d3a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c158ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44832/0 (socket says 172.21.15.72:44832) 2024-03-06T07:23:17.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d3a59c640 1 -- 172.21.15.72:0/2320829073 learned_addr learned my addr 172.21.15.72:0/2320829073 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d39d9b640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0cfff0 0x7f8d3c06ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.439+0000 7f8d3ad9d640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c15bb20 0x7f8d3c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d3a59c640 1 -- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c15bb20 msgr2=0x7f8d3c06d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d3a59c640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c15bb20 0x7f8d3c06d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d3a59c640 1 -- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0cfff0 msgr2=0x7f8d3c06ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d3a59c640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0cfff0 0x7f8d3c06ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d3a59c640 1 -- 172.21.15.72:0/2320829073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c15c150 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d3a59c640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c158ed0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00d620 tx=0x7f8d2c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00b620 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c019070 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c15c430 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c00ba00 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.440+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c15c9f0 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.441+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c06b500 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.441+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d2c00bba0 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.442+0000 7f8d2affd640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d1c06b4d0 0x7f8d1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.442+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8d2c08a380 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.442+0000 7f8d39d9b640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d1c06b4d0 0x7f8d1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.442+0000 7f8d39d9b640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d1c06b4d0 0x7f8d1c06d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c15b8e0 tx=0x7f8d30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.446+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d3c06b500 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.650+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d3c04f330 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.651+0000 7f8d2affd640 1 -- 172.21.15.72:0/2320829073 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f8d2c0594f0 con 0x7f8d3c0ce4b0 2024-03-06T07:23:17.653 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:17.654 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4727,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3953657001","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3953657001},{"type":"v1","addr":"172.21.15.146:6834","nonce":3953657001}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2285423177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2285423177},{"type":"v1","addr":"172.21.15.146:6835","nonce":2285423177}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5187,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2011439933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2011439933},{"type":"v1","addr":"172.21.15.72:6836","nonce":2011439933}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5586,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/4124223780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":4124223780},{"type":"v1","addr":"172.21.15.72:6837","nonce":4124223780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[]} 2024-03-06T07:23:17.654 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 16 2024-03-06T07:23:17.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d1c06b4d0 msgr2=0x7f8d1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d1c06b4d0 0x7f8d1c06d990 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c15b8e0 tx=0x7f8d30008040 comp rx=0 tx=0).stop 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 msgr2=0x7f8d3c158ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c158ed0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00d620 tx=0x7f8d2c00dae0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 shutdown_connections 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d1c06b4d0 0x7f8d1c06d990 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d3c15bb20 0x7f8d3c06d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0cfff0 0x7f8d3c06ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 --2- 172.21.15.72:0/2320829073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d3c0ce4b0 0x7f8d3c158ed0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 >> 172.21.15.72:0/2320829073 conn(0x7f8d3c09a9f0 msgr2=0x7f8d3c0cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 shutdown_connections 2024-03-06T07:23:17.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.653+0000 7f8d40e66640 1 -- 172.21.15.72:0/2320829073 wait complete. 2024-03-06T07:23:17.667 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:17.667 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:17.667 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:17.667 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:17.667 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:17.668 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:17.668 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:17.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:17.668 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:17.668 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:17.668 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:17.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:17.669 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:17.669 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:17.669 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:17.669 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:17.677 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:17.677 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:17.678 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:17.678 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:17.681 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:17.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:17.682 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:17.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:17.719 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:17.720 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:17.720 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:17.721 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:17.723 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:17.723 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:17.724 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:17.724 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:17.724 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:17.795 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:23:17.796 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:23:17.808 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:23:17.809 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.831+0000 7ff04782a640 1 Processor -- start 2024-03-06T07:23:17.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.831+0000 7ff04782a640 1 -- start start 2024-03-06T07:23:17.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff04782a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0401614a0 0x7ff04006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff04782a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 0x7ff0400b5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff04782a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400af4b0 con 0x7ff0400b0270 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff04782a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400af790 con 0x7ff0401614a0 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff04782a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400afa70 con 0x7ff0400b1a20 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff045dc7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 0x7ff0400b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff0455c6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0401614a0 0x7ff04006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff044dc5640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff0400b0270 0x7ff0400b1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:38002/0 (socket says 172.21.15.72:38002) 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff045dc7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 0x7ff0400b5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:44680/0 (socket says 172.21.15.72:44680) 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff044dc5640 1 -- 172.21.15.72:0/2267542459 learned_addr learned my addr 172.21.15.72:0/2267542459 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff045dc7640 1 -- 172.21.15.72:0/2267542459 >> v2:172.21.15.146:3300/0 conn(0x7ff0401614a0 msgr2=0x7ff04006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff045dc7640 1 --2- 172.21.15.72:0/2267542459 >> v2:172.21.15.146:3300/0 conn(0x7ff0401614a0 0x7ff04006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff045dc7640 1 -- 172.21.15.72:0/2267542459 >> v1:172.21.15.146:6789/0 conn(0x7ff0400b0270 legacy=0x7ff0400b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.832+0000 7ff045dc7640 1 -- 172.21.15.72:0/2267542459 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0400afd50 con 0x7ff0400b1a20 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.833+0000 7ff045dc7640 1 --2- 172.21.15.72:0/2267542459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 0x7ff0400b5510 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff038009d90 tx=0x7ff03802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2897ccf0e9c13d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.835+0000 7ff037fff640 1 -- 172.21.15.72:0/2267542459 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038039070 con 0x7ff0400b1a20 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.835+0000 7ff037fff640 1 -- 172.21.15.72:0/2267542459 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03802cbc0 con 0x7ff0400b1a20 2024-03-06T07:23:17.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.835+0000 7ff037fff640 1 -- 172.21.15.72:0/2267542459 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff038034440 con 0x7ff0400b1a20 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.835+0000 7ff04782a640 1 -- 172.21.15.72:0/2267542459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 msgr2=0x7ff0400b5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.835+0000 7ff04782a640 1 --2- 172.21.15.72:0/2267542459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 0x7ff0400b5510 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff038009d90 tx=0x7ff03802be10 comp rx=0 tx=0).stop 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.836+0000 7ff04782a640 1 -- 172.21.15.72:0/2267542459 shutdown_connections 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.836+0000 7ff04782a640 1 --2- 172.21.15.72:0/2267542459 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0400b1a20 0x7ff0400b5510 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.836+0000 7ff04782a640 1 --2- 172.21.15.72:0/2267542459 >> v2:172.21.15.146:3300/0 conn(0x7ff0401614a0 0x7ff04006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.836+0000 7ff04782a640 1 -- 172.21.15.72:0/2267542459 >> 172.21.15.72:0/2267542459 conn(0x7ff04009a9f0 msgr2=0x7ff04009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:17.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.836+0000 7ff04782a640 1 -- 172.21.15.72:0/2267542459 shutdown_connections 2024-03-06T07:23:17.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.836+0000 7ff04782a640 1 -- 172.21.15.72:0/2267542459 wait complete. 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.837+0000 7ff04782a640 1 Processor -- start 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.837+0000 7ff04782a640 1 -- start start 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.837+0000 7ff04782a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0400b0270 0x7ff040156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.837+0000 7ff04782a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 0x7ff04009f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff04782a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0401614a0 0x7ff04015a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff04782a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400b1e30 con 0x7ff0400b1a20 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff04782a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400b06e0 con 0x7ff0400b0270 2024-03-06T07:23:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff04782a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0400b5a50 con 0x7ff0401614a0 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 0x7ff04009f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff045dc7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0401614a0 0x7ff04015a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 0x7ff04009f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44896/0 (socket says 172.21.15.72:44896) 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 -- 172.21.15.72:0/121970533 learned_addr learned my addr 172.21.15.72:0/121970533 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff0455c6640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0400b0270 0x7ff040156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 -- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0401614a0 msgr2=0x7ff04015a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0401614a0 0x7ff04015a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 -- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0400b0270 msgr2=0x7ff040156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0400b0270 0x7ff040156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 -- 172.21.15.72:0/121970533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0401572b0 con 0x7ff0400b1a20 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff044dc5640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 0x7ff04009f490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff03000c450 tx=0x7ff03000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030017070 con 0x7ff0400b1a20 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030005ce0 con 0x7ff0400b1a20 2024-03-06T07:23:17.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0300072b0 con 0x7ff0400b1a20 2024-03-06T07:23:17.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.838+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff040157590 con 0x7ff0400b1a20 2024-03-06T07:23:17.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff040157b50 con 0x7ff0400b1a20 2024-03-06T07:23:17.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.840+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0400b4da0 con 0x7ff0400b1a20 2024-03-06T07:23:17.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.840+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff03001a070 con 0x7ff0400b1a20 2024-03-06T07:23:17.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.840+0000 7ff035ffb640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff02406b4d0 0x7ff02406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:17.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.840+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff030089210 con 0x7ff0400b1a20 2024-03-06T07:23:17.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.840+0000 7ff0455c6640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff02406b4d0 0x7ff02406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:17.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.841+0000 7ff0455c6640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff02406b4d0 0x7ff02406d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009aa0 tx=0x7ff03c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:17.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:17.844+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0400b4da0 con 0x7ff0400b1a20 2024-03-06T07:23:18.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.030+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff040069f60 con 0x7ff0400b1a20 2024-03-06T07:23:18.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.836+0000 7ff035ffb640 1 -- 172.21.15.72:0/121970533 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v42) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff040069f60 con 0x7ff0400b1a20 2024-03-06T07:23:18.838 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:23:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff02406b4d0 msgr2=0x7ff02406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff02406b4d0 0x7ff02406d990 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009aa0 tx=0x7ff03c009190 comp rx=0 tx=0).stop 2024-03-06T07:23:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 msgr2=0x7ff04009f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 0x7ff04009f490 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff03000c450 tx=0x7ff03000c910 comp rx=0 tx=0).stop 2024-03-06T07:23:18.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 shutdown_connections 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff02406b4d0 0x7ff02406d990 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0401614a0 0x7ff04015a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0400b1a20 0x7ff04009f490 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 --2- 172.21.15.72:0/121970533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0400b0270 0x7ff040156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 >> 172.21.15.72:0/121970533 conn(0x7ff04009a9f0 msgr2=0x7ff0400b0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 shutdown_connections 2024-03-06T07:23:18.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.839+0000 7ff04782a640 1 -- 172.21.15.72:0/121970533 wait complete. 2024-03-06T07:23:18.854 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:18.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.954+0000 7fd1dcf20640 1 Processor -- start 2024-03-06T07:23:18.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.954+0000 7fd1dcf20640 1 -- start start 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1dcf20640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd1d80cc9e0 0x7fd1d806db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1dcf20640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1d806e0a0 0x7fd1d80ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1dcf20640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d815e440 con 0x7fd1d80cf100 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1dcf20640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d815e700 con 0x7fd1d806e0a0 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1dcf20640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d815e9e0 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1d806e0a0 0x7fd1d80ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd1d80cf100 0x7fd1d80cc2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58518/0 (socket says 172.21.15.72:58518) 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d659c640 1 -- 172.21.15.72:0/880474703 learned_addr learned my addr 172.21.15.72:0/880474703 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:18.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d6d9d640 1 -- 172.21.15.72:0/880474703 >> v2:172.21.15.146:3300/0 conn(0x7fd1d80cc9e0 msgr2=0x7fd1d806db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1cffff640 1 --2- 172.21.15.72:0/880474703 >> v2:172.21.15.146:3300/0 conn(0x7fd1d80cc9e0 0x7fd1d806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d5d9b640 1 -- 172.21.15.72:0/880474703 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182035317 0 0) 0x7fd1d815e440 con 0x7fd1d80cf100 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d6d9d640 1 --2- 172.21.15.72:0/880474703 >> v2:172.21.15.146:3300/0 conn(0x7fd1d80cc9e0 0x7fd1d806db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d6d9d640 1 -- 172.21.15.72:0/880474703 >> v1:172.21.15.72:6789/0 conn(0x7fd1d80cf100 legacy=0x7fd1d80cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1d6d9d640 1 -- 172.21.15.72:0/880474703 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d815ecc0 con 0x7fd1d806e0a0 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.955+0000 7fd1cffff640 1 --2- 172.21.15.72:0/880474703 >> v2:172.21.15.146:3300/0 conn(0x7fd1d80cc9e0 0x7fd1d806db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.956+0000 7fd1d6d9d640 1 --2- 172.21.15.72:0/880474703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d806e0a0 0x7fd1d80ad860 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80731a0 tx=0x7fd1d002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3694affbdd479d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.956+0000 7fd1d5d9b640 1 -- 172.21.15.72:0/880474703 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0039070 con 0x7fd1d806e0a0 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.956+0000 7fd1d5d9b640 1 -- 172.21.15.72:0/880474703 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d002cbf0 con 0x7fd1d806e0a0 2024-03-06T07:23:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.956+0000 7fd1d5d9b640 1 -- 172.21.15.72:0/880474703 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d0034440 con 0x7fd1d806e0a0 2024-03-06T07:23:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.956+0000 7fd1dcf20640 1 -- 172.21.15.72:0/880474703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d806e0a0 msgr2=0x7fd1d80ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.956+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/880474703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d806e0a0 0x7fd1d80ad860 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80731a0 tx=0x7fd1d002be40 comp rx=0 tx=0).stop 2024-03-06T07:23:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.957+0000 7fd1dcf20640 1 -- 172.21.15.72:0/880474703 shutdown_connections 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.957+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/880474703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d806e0a0 0x7fd1d80ad860 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.957+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/880474703 >> v2:172.21.15.146:3300/0 conn(0x7fd1d80cc9e0 0x7fd1d806db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.957+0000 7fd1dcf20640 1 -- 172.21.15.72:0/880474703 >> 172.21.15.72:0/880474703 conn(0x7fd1d80cf8a0 msgr2=0x7fd1d80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.957+0000 7fd1dcf20640 1 -- 172.21.15.72:0/880474703 shutdown_connections 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.957+0000 7fd1dcf20640 1 -- 172.21.15.72:0/880474703 wait complete. 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.958+0000 7fd1dcf20640 1 Processor -- start 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.958+0000 7fd1dcf20640 1 -- start start 2024-03-06T07:23:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d806e0a0 0x7fd1d8159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 0x7fd1d80c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d80cf100 0x7fd1d8159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d806e4f0 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d80cf4e0 con 0x7fd1d80cf100 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d806e670 con 0x7fd1d806e0a0 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 0x7fd1d80c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 0x7fd1d80c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44914/0 (socket says 172.21.15.72:44914) 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 -- 172.21.15.72:0/667462305 learned_addr learned my addr 172.21.15.72:0/667462305 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1d659c640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d806e0a0 0x7fd1d8159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1d6d9d640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d80cf100 0x7fd1d8159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 -- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d806e0a0 msgr2=0x7fd1d8159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d806e0a0 0x7fd1d8159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 -- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d80cf100 msgr2=0x7fd1d8159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d80cf100 0x7fd1d8159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 -- 172.21.15.72:0/667462305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8072500 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1d659c640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d806e0a0 0x7fd1d8159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1d6d9d640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d80cf100 0x7fd1d8159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1cffff640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 0x7fd1d80c48c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000c820 tx=0x7fd1c000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0015070 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c0007a30 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d80727e0 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0007e00 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.959+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d80c52f0 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.960+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d804f330 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.961+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1c0018070 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.961+0000 7fd1ceffd640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1b806b4d0 0x7fd1b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.961+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd1c0089e00 con 0x7fd1d80cc9e0 2024-03-06T07:23:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.961+0000 7fd1d659c640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1b806b4d0 0x7fd1b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.962+0000 7fd1d659c640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1b806b4d0 0x7fd1b806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8000f60 tx=0x7fd1c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:18.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:18.966+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1c0059060 con 0x7fd1d80cc9e0 2024-03-06T07:23:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.204+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd1d80d0090 con 0x7fd1d80cc9e0 2024-03-06T07:23:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.204+0000 7fd1ceffd640 1 -- 172.21.15.72:0/667462305 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd1c0058960 con 0x7fd1d80cc9e0 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:19.207 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.211 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:23:19.211 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:23:19.211 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.211 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.211 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:19.211 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.214 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.215 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.216 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.217 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:23:19.218 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:23:19.219 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:23:19.220 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:19.221 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:23:19.222 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:23:19.223 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1b806b4d0 msgr2=0x7fd1b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1b806b4d0 0x7fd1b806d990 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8000f60 tx=0x7fd1c8009040 comp rx=0 tx=0).stop 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 msgr2=0x7fd1d80c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 0x7fd1d80c48c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000c820 tx=0x7fd1c000cc20 comp rx=0 tx=0).stop 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 shutdown_connections 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1b806b4d0 0x7fd1b806d990 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d80cf100 0x7fd1d8159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1d80cc9e0 0x7fd1d80c48c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 --2- 172.21.15.72:0/667462305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d806e0a0 0x7fd1d8159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 >> 172.21.15.72:0/667462305 conn(0x7fd1d80cf8a0 msgr2=0x7fd1d80cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.206+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 shutdown_connections 2024-03-06T07:23:19.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.207+0000 7fd1dcf20640 1 -- 172.21.15.72:0/667462305 wait complete. 2024-03-06T07:23:19.226 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:19.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 Processor -- start 2024-03-06T07:23:19.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 -- start start 2024-03-06T07:23:19.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f270415dfe0 0x7f27040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27040cf830 0x7f27040d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27040b3430 con 0x7f27040d27f0 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27040b31e0 con 0x7f27040cf830 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f270b1d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27040b3d60 con 0x7f270415dfe0 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.325+0000 7f2703fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27040cf830 0x7f27040d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2703fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27040cf830 0x7f27040d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44922/0 (socket says 172.21.15.72:44922) 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2703fff640 1 -- 172.21.15.72:0/388545123 learned_addr learned my addr 172.21.15.72:0/388545123 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2708f6c640 1 --2- 172.21.15.72:0/388545123 >> v2:172.21.15.146:3300/0 conn(0x7f270415dfe0 0x7f27040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2703fff640 1 -- 172.21.15.72:0/388545123 >> v2:172.21.15.146:3300/0 conn(0x7f270415dfe0 msgr2=0x7f27040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2703fff640 1 --2- 172.21.15.72:0/388545123 >> v2:172.21.15.146:3300/0 conn(0x7f270415dfe0 0x7f27040cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2703fff640 1 -- 172.21.15.72:0/388545123 >> v1:172.21.15.146:6789/0 conn(0x7f27040d27f0 legacy=0x7f27040d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.326+0000 7f2703fff640 1 -- 172.21.15.72:0/388545123 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27040b3710 con 0x7f27040cf830 2024-03-06T07:23:19.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f2703fff640 1 --2- 172.21.15.72:0/388545123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040cf830 0x7f27040d22b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009870 tx=0x7f26f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cabb5e387efbe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f27037fe640 1 -- 172.21.15.72:0/388545123 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8035020 con 0x7f27040cf830 2024-03-06T07:23:19.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f27037fe640 1 -- 172.21.15.72:0/388545123 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f8031870 con 0x7f27040cf830 2024-03-06T07:23:19.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f27037fe640 1 -- 172.21.15.72:0/388545123 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8031c40 con 0x7f27040cf830 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 -- 172.21.15.72:0/388545123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040cf830 msgr2=0x7f27040d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 --2- 172.21.15.72:0/388545123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040cf830 0x7f27040d22b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009870 tx=0x7f26f802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 -- 172.21.15.72:0/388545123 shutdown_connections 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 --2- 172.21.15.72:0/388545123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040cf830 0x7f27040d22b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 --2- 172.21.15.72:0/388545123 >> v2:172.21.15.146:3300/0 conn(0x7f270415dfe0 0x7f27040cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 -- 172.21.15.72:0/388545123 >> 172.21.15.72:0/388545123 conn(0x7f27040d3cd0 msgr2=0x7f27040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 -- 172.21.15.72:0/388545123 shutdown_connections 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.327+0000 7f270b1d0640 1 -- 172.21.15.72:0/388545123 wait complete. 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.328+0000 7f270b1d0640 1 Processor -- start 2024-03-06T07:23:19.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.328+0000 7f270b1d0640 1 -- start start 2024-03-06T07:23:19.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f270b1d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27040cf830 0x7f2704147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f270b1d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 0x7f27040cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f270b1d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f270415dfe0 0x7f27041604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f270b1d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040d1df0 con 0x7f27040d27f0 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f270b1d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040d1c70 con 0x7f27040cf830 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f270b1d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270415e3c0 con 0x7f270415dfe0 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f2703fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 0x7f27040cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f2703fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 0x7f27040cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44932/0 (socket says 172.21.15.72:44932) 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.329+0000 7f2703fff640 1 -- 172.21.15.72:0/683097124 learned_addr learned my addr 172.21.15.72:0/683097124 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2708f6c640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27040cf830 0x7f2704147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f270976d640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f270415dfe0 0x7f27041604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2703fff640 1 -- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f270415dfe0 msgr2=0x7f27041604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2703fff640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f270415dfe0 0x7f27041604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2703fff640 1 -- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27040cf830 msgr2=0x7f2704147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2703fff640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27040cf830 0x7f2704147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2703fff640 1 -- 172.21.15.72:0/683097124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27040cc490 con 0x7f27040d27f0 2024-03-06T07:23:19.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2708f6c640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27040cf830 0x7f2704147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f2703fff640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 0x7f27040cbdf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009870 tx=0x7f26f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8040070 con 0x7f27040d27f0 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f8030d00 con 0x7f27040d27f0 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27040cc770 con 0x7f27040d27f0 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8035040 con 0x7f27040d27f0 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.330+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27040ccd30 con 0x7f27040d27f0 2024-03-06T07:23:19.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.331+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27040d2230 con 0x7f27040d27f0 2024-03-06T07:23:19.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.332+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f26f804b060 con 0x7f27040d27f0 2024-03-06T07:23:19.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.332+0000 7f27017fa640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26f006b4d0 0x7f26f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.332+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f26f80abd40 con 0x7f27040d27f0 2024-03-06T07:23:19.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.332+0000 7f2708f6c640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26f006b4d0 0x7f26f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.333+0000 7f2708f6c640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26f006b4d0 0x7f26f006d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f26f4006fd0 tx=0x7f26f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.336+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26f802f320 con 0x7f27040d27f0 2024-03-06T07:23:19.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.507+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27040b71c0 con 0x7f27040d27f0 2024-03-06T07:23:19.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.508+0000 7f27017fa640 1 -- 172.21.15.72:0/683097124 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+12903 (secure 0 0 0) 0x7f26f807adf0 con 0x7f27040d27f0 2024-03-06T07:23:19.510 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:19.510 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":42,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:18.834354+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:19.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26f006b4d0 msgr2=0x7f26f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26f006b4d0 0x7f26f006d990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f26f4006fd0 tx=0x7f26f4008040 comp rx=0 tx=0).stop 2024-03-06T07:23:19.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 msgr2=0x7f27040cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 0x7f27040cbdf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009870 tx=0x7f26f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 shutdown_connections 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f26f006b4d0 0x7f26f006d990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f270415dfe0 0x7f27041604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27040d27f0 0x7f27040cbdf0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 --2- 172.21.15.72:0/683097124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27040cf830 0x7f2704147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 >> 172.21.15.72:0/683097124 conn(0x7f27040d3cd0 msgr2=0x7f27040d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 shutdown_connections 2024-03-06T07:23:19.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.510+0000 7f270b1d0640 1 -- 172.21.15.72:0/683097124 wait complete. 2024-03-06T07:23:19.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:19.524 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:19.605 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:19.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:19.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.626+0000 7f3fc186b640 1 Processor -- start 2024-03-06T07:23:19.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.626+0000 7f3fc186b640 1 -- start start 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fc186b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3fbc0701a0 0x7f3fbc06da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fc186b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3fbc0b42f0 0x7f3fbc070580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fc186b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0cf430 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fc186b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc15f6e0 con 0x7f3fbc0b42f0 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fc186b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc06dfa0 con 0x7f3fbc0701a0 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3fbc0b42f0 0x7f3fbc070580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3fbc0b42f0 0x7f3fbc070580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44938/0 (socket says 172.21.15.72:44938) 2024-03-06T07:23:19.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 -- 172.21.15.72:0/2730569818 learned_addr learned my addr 172.21.15.72:0/2730569818 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:19.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fbaffd640 1 --2- 172.21.15.72:0/2730569818 >> v2:172.21.15.146:3300/0 conn(0x7f3fbc0701a0 0x7f3fbc06da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 -- 172.21.15.72:0/2730569818 >> v2:172.21.15.146:3300/0 conn(0x7f3fbc0701a0 msgr2=0x7f3fbc06da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 --2- 172.21.15.72:0/2730569818 >> v2:172.21.15.146:3300/0 conn(0x7f3fbc0701a0 0x7f3fbc06da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 -- 172.21.15.72:0/2730569818 >> v1:172.21.15.146:6789/0 conn(0x7f3fbc0cfa50 legacy=0x7f3fbc0cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.628+0000 7f3fba7fc640 1 -- 172.21.15.72:0/2730569818 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc06e240 con 0x7f3fbc0b42f0 2024-03-06T07:23:19.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.629+0000 7f3fba7fc640 1 --2- 172.21.15.72:0/2730569818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc070580 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0d2c10 tx=0x7f3fac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aab72f3716231e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.629+0000 7f3fb9ffb640 1 -- 172.21.15.72:0/2730569818 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac034070 con 0x7f3fbc0b42f0 2024-03-06T07:23:19.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.629+0000 7f3fb9ffb640 1 -- 172.21.15.72:0/2730569818 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac030860 con 0x7f3fbc0b42f0 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.629+0000 7f3fb9ffb640 1 -- 172.21.15.72:0/2730569818 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac030c30 con 0x7f3fbc0b42f0 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 -- 172.21.15.72:0/2730569818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0b42f0 msgr2=0x7f3fbc070580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 --2- 172.21.15.72:0/2730569818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc070580 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0d2c10 tx=0x7f3fac02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 -- 172.21.15.72:0/2730569818 shutdown_connections 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 --2- 172.21.15.72:0/2730569818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc070580 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 --2- 172.21.15.72:0/2730569818 >> v2:172.21.15.146:3300/0 conn(0x7f3fbc0701a0 0x7f3fbc06da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 -- 172.21.15.72:0/2730569818 >> 172.21.15.72:0/2730569818 conn(0x7f3fbc0b3800 msgr2=0x7f3fbc0b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 -- 172.21.15.72:0/2730569818 shutdown_connections 2024-03-06T07:23:19.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.630+0000 7f3fc186b640 1 -- 172.21.15.72:0/2730569818 wait complete. 2024-03-06T07:23:19.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.631+0000 7f3fc186b640 1 Processor -- start 2024-03-06T07:23:19.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.631+0000 7f3fc186b640 1 -- start start 2024-03-06T07:23:19.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fc186b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fbc0701a0 0x7f3fbc09d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fc186b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc1f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fc186b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 0x7f3fbc09f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fc186b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0ce670 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fc186b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0cae80 con 0x7f3fbc0b42f0 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fc186b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0ce950 con 0x7f3fbc0701a0 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 0x7f3fbc09f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 0x7f3fbc09f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44944/0 (socket says 172.21.15.72:44944) 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 -- 172.21.15.72:0/1085340695 learned_addr learned my addr 172.21.15.72:0/1085340695 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fba7fc640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc1f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbaffd640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fbc0701a0 0x7f3fbc09d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 -- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fbc0701a0 msgr2=0x7f3fbc09d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fbc0701a0 0x7f3fbc09d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 -- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fbc0b42f0 msgr2=0x7f3fbc1f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc1f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbb7fe640 1 -- 172.21.15.72:0/1085340695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc1f49d0 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.632+0000 7f3fbaffd640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fbc0701a0 0x7f3fbc09d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.633+0000 7f3fbb7fe640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 0x7f3fbc09f430 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c7d0 tx=0x7f3fb400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.633+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4003b70 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.633+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb4003d30 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.633+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc1f4cb0 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.633+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb400aa90 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.633+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc09d640 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.634+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc09fe60 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.634+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3fb400ac30 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.634+0000 7f3fa3fff640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3fa806b4d0 0x7f3fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.635+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3fb4089d20 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.635+0000 7f3fbaffd640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3fa806b4d0 0x7f3fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.635+0000 7f3fbaffd640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3fa806b4d0 0x7f3fa806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.639+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fb4058ef0 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.812+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3fbc0cc5c0 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.812+0000 7f3fa3fff640 1 -- 172.21.15.72:0/1085340695 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3fb40587f0 con 0x7f3fbc0cfa50 2024-03-06T07:23:19.814 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3fa806b4d0 msgr2=0x7f3fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3fa806b4d0 0x7f3fa806d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 msgr2=0x7f3fbc09f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 0x7f3fbc09f430 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400c7d0 tx=0x7f3fb400cc90 comp rx=0 tx=0).stop 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 shutdown_connections 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3fa806b4d0 0x7f3fa806d990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fbc0cfa50 0x7f3fbc09f430 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fbc0b42f0 0x7f3fbc1f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 --2- 172.21.15.72:0/1085340695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fbc0701a0 0x7f3fbc09d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 >> 172.21.15.72:0/1085340695 conn(0x7f3fbc0b3800 msgr2=0x7f3fbc0cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 shutdown_connections 2024-03-06T07:23:19.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.814+0000 7f3fc186b640 1 -- 172.21.15.72:0/1085340695 wait complete. 2024-03-06T07:23:19.828 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:19.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.925+0000 7fec23ca4640 1 Processor -- start 2024-03-06T07:23:19.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.926+0000 7fec23ca4640 1 -- start start 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.926+0000 7fec23ca4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c13dc10 0x7fec1c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.926+0000 7fec23ca4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec1c15ef60 0x7fec1c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.926+0000 7fec23ca4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1c15f580 con 0x7fec1c15bf30 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.926+0000 7fec23ca4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1c0cf3f0 con 0x7fec1c15ef60 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.926+0000 7fec23ca4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c15b210 con 0x7fec1c13dc10 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec22241640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec1c15ef60 0x7fec1c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec2123f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fec1c15bf30 0x7fec1c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58522/0 (socket says 172.21.15.72:58522) 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec2123f640 1 -- 172.21.15.72:0/1387200127 learned_addr learned my addr 172.21.15.72:0/1387200127 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec22241640 1 -- 172.21.15.72:0/1387200127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c13dc10 msgr2=0x7fec1c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec22241640 1 --2- 172.21.15.72:0/1387200127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c13dc10 0x7fec1c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec22241640 1 -- 172.21.15.72:0/1387200127 >> v1:172.21.15.72:6789/0 conn(0x7fec1c15bf30 legacy=0x7fec1c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec20a3e640 1 -- 172.21.15.72:0/1387200127 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874267315 0 0) 0x7fec1c15f580 con 0x7fec1c15bf30 2024-03-06T07:23:19.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.927+0000 7fec22241640 1 -- 172.21.15.72:0/1387200127 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c15b450 con 0x7fec1c15ef60 2024-03-06T07:23:19.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec22241640 1 --2- 172.21.15.72:0/1387200127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c15ef60 0x7fec1c0b2e90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fec14009d90 tx=0x7fec1402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=414075dad8f22a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec20a3e640 1 -- 172.21.15.72:0/1387200127 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14039070 con 0x7fec1c15ef60 2024-03-06T07:23:19.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec20a3e640 1 -- 172.21.15.72:0/1387200127 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1402cbc0 con 0x7fec1c15ef60 2024-03-06T07:23:19.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec20a3e640 1 -- 172.21.15.72:0/1387200127 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14034440 con 0x7fec1c15ef60 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 -- 172.21.15.72:0/1387200127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c15ef60 msgr2=0x7fec1c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 --2- 172.21.15.72:0/1387200127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c15ef60 0x7fec1c0b2e90 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fec14009d90 tx=0x7fec1402be10 comp rx=0 tx=0).stop 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 -- 172.21.15.72:0/1387200127 shutdown_connections 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 --2- 172.21.15.72:0/1387200127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c15ef60 0x7fec1c0b2e90 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 --2- 172.21.15.72:0/1387200127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c13dc10 0x7fec1c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 -- 172.21.15.72:0/1387200127 >> 172.21.15.72:0/1387200127 conn(0x7fec1c0cf8a0 msgr2=0x7fec1c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 -- 172.21.15.72:0/1387200127 shutdown_connections 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.928+0000 7fec23ca4640 1 -- 172.21.15.72:0/1387200127 wait complete. 2024-03-06T07:23:19.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.929+0000 7fec23ca4640 1 Processor -- start 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.929+0000 7fec23ca4640 1 -- start start 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec23ca4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 0x7fec1c1e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec23ca4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c15bf30 0x7fec1c1e6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec23ca4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c15ef60 0x7fec1c158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec23ca4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0708d0 con 0x7fec1c13dc10 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec23ca4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c13dff0 con 0x7fec1c15bf30 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec23ca4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c070a50 con 0x7fec1c15ef60 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 0x7fec1c1e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 0x7fec1c1e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44956/0 (socket says 172.21.15.72:44956) 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 -- 172.21.15.72:0/3023853978 learned_addr learned my addr 172.21.15.72:0/3023853978 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 -- 172.21.15.72:0/3023853978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c15ef60 msgr2=0x7fec1c158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c15ef60 0x7fec1c158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 -- 172.21.15.72:0/3023853978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c15bf30 msgr2=0x7fec1c1e6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c15bf30 0x7fec1c1e6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.930+0000 7fec21a40640 1 -- 172.21.15.72:0/3023853978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c1f08d0 con 0x7fec1c13dc10 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.931+0000 7fec21a40640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 0x7fec1c1e5ac0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00d990 tx=0x7fec0c00de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.931+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c01b070 con 0x7fec1c13dc10 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.931+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00edf0 con 0x7fec1c13dc10 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.931+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c157460 con 0x7fec1c13dc10 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.931+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c016680 con 0x7fec1c13dc10 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.931+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c157a20 con 0x7fec1c13dc10 2024-03-06T07:23:19.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.932+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c04f330 con 0x7fec1c13dc10 2024-03-06T07:23:19.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.932+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec0c01e070 con 0x7fec1c13dc10 2024-03-06T07:23:19.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.933+0000 7fec127fc640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec0006b4d0 0x7fec0006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:19.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.933+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fec0c08bd40 con 0x7fec1c13dc10 2024-03-06T07:23:19.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.933+0000 7fec2123f640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec0006b4d0 0x7fec0006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:19.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.933+0000 7fec2123f640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec0006b4d0 0x7fec0006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fec18007fd0 tx=0x7fec18007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:19.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:19.937+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec0c05af10 con 0x7fec1c13dc10 2024-03-06T07:23:20.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.113+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fec1c0d0710 con 0x7fec1c13dc10 2024-03-06T07:23:20.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.113+0000 7fec127fc640 1 -- 172.21.15.72:0/3023853978 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v42) v1 ==== 84+0+65 (secure 0 0 0) 0x7fec0c05a810 con 0x7fec1c13dc10 2024-03-06T07:23:20.115 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:20.115 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:23:20.115 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:20.116 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:23:20.116 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:23:20.116 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:20.116 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:23:20.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec0006b4d0 msgr2=0x7fec0006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec0006b4d0 0x7fec0006d990 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fec18007fd0 tx=0x7fec18007920 comp rx=0 tx=0).stop 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 msgr2=0x7fec1c1e5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 0x7fec1c1e5ac0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00d990 tx=0x7fec0c00de50 comp rx=0 tx=0).stop 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 shutdown_connections 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec0006b4d0 0x7fec0006d990 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c15ef60 0x7fec1c158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c15bf30 0x7fec1c1e6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 --2- 172.21.15.72:0/3023853978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec1c13dc10 0x7fec1c1e5ac0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 >> 172.21.15.72:0/3023853978 conn(0x7fec1c0cf8a0 msgr2=0x7fec1c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 shutdown_connections 2024-03-06T07:23:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.115+0000 7fec23ca4640 1 -- 172.21.15.72:0/3023853978 wait complete. 2024-03-06T07:23:20.128 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:20.128 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:20.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.229+0000 7ff1d30af640 1 Processor -- start 2024-03-06T07:23:20.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.229+0000 7ff1d30af640 1 -- start start 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.230+0000 7ff1d30af640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1cc15dfa0 0x7ff1cc0d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.230+0000 7ff1d30af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc15e380 con 0x7ff1cc0d4630 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.230+0000 7ff1d30af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1cc0d1e30 con 0x7ff1cc0d29f0 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.230+0000 7ff1d30af640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1cc0d2100 con 0x7ff1cc15dfa0 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.230+0000 7ff1cbfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff1cc0d4630 0x7ff1cc0d22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58538/0 (socket says 172.21.15.72:58538) 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.230+0000 7ff1cbfff640 1 -- 172.21.15.72:0/569966886 learned_addr learned my addr 172.21.15.72:0/569966886 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457545981 0 0) 0x7ff1cc15e380 con 0x7ff1cc0d4630 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1b4003520 con 0x7ff1cc0d4630 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1d0e4b640 1 --2- 172.21.15.72:0/569966886 >> v2:172.21.15.146:3300/0 conn(0x7ff1cc15dfa0 0x7ff1cc0d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1089028001 0 0) 0x7ff1b4003520 con 0x7ff1cc0d4630 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 >> v2:172.21.15.146:3300/0 conn(0x7ff1cc15dfa0 msgr2=0x7ff1cc0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 --2- 172.21.15.72:0/569966886 >> v2:172.21.15.146:3300/0 conn(0x7ff1cc15dfa0 0x7ff1cc0d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 >> v1:172.21.15.146:6789/0 conn(0x7ff1cc0d29f0 legacy=0x7ff1cc0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc0cf470 con 0x7ff1cc0d4630 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff1c0001670 con 0x7ff1cc0d4630 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 >> v1:172.21.15.72:6789/0 conn(0x7ff1cc0d4630 legacy=0x7ff1cc0d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b40039b0 0x7ff1b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b40052e0 0x7ff1b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0d2100 con 0x7ff1b4004800 2024-03-06T07:23:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b400c1b0 con 0x7ff1b40052e0 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b400c3b0 con 0x7ff1b40039b0 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1d0e4b640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b40039b0 0x7ff1b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1d164c640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b40052e0 0x7ff1b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cbfff640 1 -- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b40039b0 msgr2=0x7ff1b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b40039b0 0x7ff1b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cbfff640 1 -- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b40052e0 msgr2=0x7ff1b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b40052e0 0x7ff1b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1d164c640 1 -- 172.21.15.72:0/569966886 reap_dead start 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.231+0000 7ff1cbfff640 1 -- 172.21.15.72:0/569966886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b4003520 con 0x7ff1b4004800 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.232+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1b4004c80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0d4aa0 tx=0x7ff1b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b688a5c8b060410 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.232+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0007cf0 con 0x7ff1b4004800 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.232+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c0030070 con 0x7ff1b4004800 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.232+0000 7ff1cb7fe640 1 -- 172.21.15.72:0/569966886 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0005d70 con 0x7ff1b4004800 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 -- 172.21.15.72:0/569966886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 msgr2=0x7ff1b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1b4004c80 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0d4aa0 tx=0x7ff1b4005830 comp rx=0 tx=0).stop 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 -- 172.21.15.72:0/569966886 shutdown_connections 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 --2- 172.21.15.72:0/569966886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1b4004c80 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 -- 172.21.15.72:0/569966886 >> 172.21.15.72:0/569966886 conn(0x7ff1cc09d000 msgr2=0x7ff1cc0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 -- 172.21.15.72:0/569966886 shutdown_connections 2024-03-06T07:23:20.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.233+0000 7ff1d30af640 1 -- 172.21.15.72:0/569966886 wait complete. 2024-03-06T07:23:20.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.234+0000 7ff1d30af640 1 Processor -- start 2024-03-06T07:23:20.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.234+0000 7ff1d30af640 1 -- start start 2024-03-06T07:23:20.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1cc0c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1cc0c88a0 0x7ff1cc0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1cc1532b0 0x7ff1cc156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0d18f0 con 0x7ff1b4004800 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0cfa30 con 0x7ff1cc0c88a0 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc0d1a70 con 0x7ff1cc1532b0 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1cc0c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1cc0c9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44966/0 (socket says 172.21.15.72:44966) 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 -- 172.21.15.72:0/3591312245 learned_addr learned my addr 172.21.15.72:0/3591312245 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 -- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1cc1532b0 msgr2=0x7ff1cc156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1cc1532b0 0x7ff1cc156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 -- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1cc0c88a0 msgr2=0x7ff1cc0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1cc0c88a0 0x7ff1cc0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1cc0c88a0 0x7ff1cc0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 -- 172.21.15.72:0/3591312245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1cc0c8d30 con 0x7ff1b4004800 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1cc0c88a0 0x7ff1cc0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d0e4b640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1cc0c9dd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00b770 tx=0x7ff1bc00bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc018070 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1bc010830 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc0c9010 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc010c00 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.235+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc0c95d0 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.236+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1cc04f330 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.237+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1bc01b070 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.237+0000 7ff1c97fa640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1b806b500 0x7ff1b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.237+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff1bc08a330 con 0x7ff1b4004800 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.237+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1b806b500 0x7ff1b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.238+0000 7ff1cbfff640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1b806b500 0x7ff1b806d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0034000 tx=0x7ff1c0039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.242+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1bc059500 con 0x7ff1b4004800 2024-03-06T07:23:20.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.414+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff1cc0ceec0 con 0x7ff1b4004800 2024-03-06T07:23:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.847+0000 7ff1c97fa640 1 -- 172.21.15.72:0/3591312245 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v43) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff1bc058e00 con 0x7ff1b4004800 2024-03-06T07:23:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:20.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1b806b500 msgr2=0x7ff1b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1b806b500 0x7ff1b806d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff1c0034000 tx=0x7ff1c0039040 comp rx=0 tx=0).stop 2024-03-06T07:23:20.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 msgr2=0x7ff1cc0c9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1cc0c9dd0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00b770 tx=0x7ff1bc00bc30 comp rx=0 tx=0).stop 2024-03-06T07:23:20.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 shutdown_connections 2024-03-06T07:23:20.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1b806b500 0x7ff1b806d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1cc1532b0 0x7ff1cc156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1cc0c88a0 0x7ff1cc0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 --2- 172.21.15.72:0/3591312245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b4004800 0x7ff1cc0c9dd0 secure :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00b770 tx=0x7ff1bc00bc30 comp rx=0 tx=0).stop 2024-03-06T07:23:20.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.850+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 >> 172.21.15.72:0/3591312245 conn(0x7ff1cc09d000 msgr2=0x7ff1cc0cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.851+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 shutdown_connections 2024-03-06T07:23:20.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.851+0000 7ff1d30af640 1 -- 172.21.15.72:0/3591312245 wait complete. 2024-03-06T07:23:20.868 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:20.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.973+0000 7f14faa0e640 1 Processor -- start 2024-03-06T07:23:20.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.973+0000 7f14faa0e640 1 -- start start 2024-03-06T07:23:20.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14faa0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f415a220 0x7f14f40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14faa0e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f14f40712d0 0x7f14f4071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14faa0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14f415de60 0x7f14f415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14faa0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f40cf610 con 0x7f14f415de60 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14faa0e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f4071c70 con 0x7f14f40712d0 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14faa0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40b3320 con 0x7f14f415a220 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14f8fab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14f415de60 0x7f14f415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14f8fab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14f415de60 0x7f14f415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44976/0 (socket says 172.21.15.72:44976) 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.974+0000 7f14f8fab640 1 -- 172.21.15.72:0/3699627487 learned_addr learned my addr 172.21.15.72:0/3699627487 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f8fab640 1 -- 172.21.15.72:0/3699627487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f415a220 msgr2=0x7f14f40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f8fab640 1 --2- 172.21.15.72:0/3699627487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f415a220 0x7f14f40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f8fab640 1 -- 172.21.15.72:0/3699627487 >> v2:172.21.15.146:3300/0 conn(0x7f14f40712d0 msgr2=0x7f14f4071730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f8fab640 1 --2- 172.21.15.72:0/3699627487 >> v2:172.21.15.146:3300/0 conn(0x7f14f40712d0 0x7f14f4071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f8fab640 1 -- 172.21.15.72:0/3699627487 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40728d0 con 0x7f14f415de60 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f8fab640 1 --2- 172.21.15.72:0/3699627487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f415e2f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f14f406eac0 tx=0x7f14ec02e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6124d0f96b957edd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f2ffd640 1 -- 172.21.15.72:0/3699627487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec02ea00 con 0x7f14f415de60 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f2ffd640 1 -- 172.21.15.72:0/3699627487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec02eba0 con 0x7f14f415de60 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.975+0000 7f14f2ffd640 1 -- 172.21.15.72:0/3699627487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec009600 con 0x7f14f415de60 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 -- 172.21.15.72:0/3699627487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 msgr2=0x7f14f415e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 --2- 172.21.15.72:0/3699627487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f415e2f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f14f406eac0 tx=0x7f14ec02e060 comp rx=0 tx=0).stop 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 -- 172.21.15.72:0/3699627487 shutdown_connections 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 --2- 172.21.15.72:0/3699627487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f415e2f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 --2- 172.21.15.72:0/3699627487 >> v2:172.21.15.146:3300/0 conn(0x7f14f40712d0 0x7f14f4071730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 --2- 172.21.15.72:0/3699627487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f415a220 0x7f14f40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 -- 172.21.15.72:0/3699627487 >> 172.21.15.72:0/3699627487 conn(0x7f14f406b190 msgr2=0x7f14f406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 -- 172.21.15.72:0/3699627487 shutdown_connections 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.976+0000 7f14faa0e640 1 -- 172.21.15.72:0/3699627487 wait complete. 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.977+0000 7f14faa0e640 1 Processor -- start 2024-03-06T07:23:20.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.977+0000 7f14faa0e640 1 -- start start 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14faa0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f40712d0 0x7f14f40ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14faa0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f415a220 0x7f14f414c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14faa0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f40ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14faa0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4072030 con 0x7f14f415de60 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14faa0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4072bb0 con 0x7f14f415a220 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14faa0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f4072530 con 0x7f14f40712d0 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f40ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f40ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44988/0 (socket says 172.21.15.72:44988) 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 -- 172.21.15.72:0/106128939 learned_addr learned my addr 172.21.15.72:0/106128939 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 -- 172.21.15.72:0/106128939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f40712d0 msgr2=0x7f14f40ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f40712d0 0x7f14f40ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 -- 172.21.15.72:0/106128939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f415a220 msgr2=0x7f14f414c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:20.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f415a220 0x7f14f414c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.978+0000 7f14f8fab640 1 -- 172.21.15.72:0/106128939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40aa500 con 0x7f14f415de60 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.979+0000 7f14f8fab640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f40ac2d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f14ec038040 tx=0x7f14ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.979+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec045020 con 0x7f14f415de60 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.979+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec040660 con 0x7f14f415de60 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.979+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f414cc60 con 0x7f14f415de60 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.979+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec02ea00 con 0x7f14f415de60 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.979+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f40aaa80 con 0x7f14f415de60 2024-03-06T07:23:20.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.980+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f40732b0 con 0x7f14f415de60 2024-03-06T07:23:20.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.980+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14ec048020 con 0x7f14f415de60 2024-03-06T07:23:20.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.981+0000 7f14f0ff9640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14e006b4d0 0x7f14e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:20.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.981+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f14ec0aeb60 con 0x7f14f415de60 2024-03-06T07:23:20.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.981+0000 7f14f3fff640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14e006b4d0 0x7f14e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:20.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.981+0000 7f14f3fff640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14e006b4d0 0x7f14e006d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009710 tx=0x7f14e8006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:20.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:20.985+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14ec04f030 con 0x7f14f415de60 2024-03-06T07:23:21.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.158+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f14f4068ff0 con 0x7f14f415de60 2024-03-06T07:23:21.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.849+0000 7f14f0ff9640 1 -- 172.21.15.72:0/106128939 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v44) v1 ==== 170+0+0 (secure 0 0 0) 0x7f14ec07dbd0 con 0x7f14f415de60 2024-03-06T07:23:21.852 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.852+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14e006b4d0 msgr2=0x7f14e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.852+0000 7f14faa0e640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14e006b4d0 0x7f14e006d990 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009710 tx=0x7f14e8006c90 comp rx=0 tx=0).stop 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.852+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 msgr2=0x7f14f40ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.852+0000 7f14faa0e640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f40ac2d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f14ec038040 tx=0x7f14ec004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 shutdown_connections 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14e006b4d0 0x7f14e006d990 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14f415de60 0x7f14f40ac2d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14f415a220 0x7f14f414c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 --2- 172.21.15.72:0/106128939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14f40712d0 0x7f14f40ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 >> 172.21.15.72:0/106128939 conn(0x7f14f406b190 msgr2=0x7f14f4158800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 shutdown_connections 2024-03-06T07:23:21.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.853+0000 7f14faa0e640 1 -- 172.21.15.72:0/106128939 wait complete. 2024-03-06T07:23:21.881 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:21.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.987+0000 7fa53f9ee640 1 Processor -- start 2024-03-06T07:23:21.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.987+0000 7fa53f9ee640 1 -- start start 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53f9ee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a490 0x7fa538072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53f9ee640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5380712d0 0x7fa538071710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53f9ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53806caa0 con 0x7fa53815a870 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53f9ee640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa538071c50 con 0x7fa5380712d0 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53f9ee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380b3340 con 0x7fa53815a490 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53cf89640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5380712d0 0x7fa538071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53cf89640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5380712d0 0x7fa538071710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45002/0 (socket says 172.21.15.72:45002) 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53cf89640 1 -- 172.21.15.72:0/3679304751 learned_addr learned my addr 172.21.15.72:0/3679304751 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.988+0000 7fa53d78a640 1 --2- 172.21.15.72:0/3679304751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a490 0x7fa538072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa53cf89640 1 -- 172.21.15.72:0/3679304751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a490 msgr2=0x7fa538072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa53cf89640 1 --2- 172.21.15.72:0/3679304751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a490 0x7fa538072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa53cf89640 1 -- 172.21.15.72:0/3679304751 >> v1:172.21.15.146:6789/0 conn(0x7fa53815a870 legacy=0x7fa53815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa53cf89640 1 -- 172.21.15.72:0/3679304751 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5380b3620 con 0x7fa5380712d0 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa53cf89640 1 --2- 172.21.15.72:0/3679304751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5380712d0 0x7fa538071710 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa52800bb00 tx=0x7fa52802dfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9530415b51671b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa52ffff640 1 -- 172.21.15.72:0/3679304751 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528034070 con 0x7fa5380712d0 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa52ffff640 1 -- 172.21.15.72:0/3679304751 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52802ed60 con 0x7fa5380712d0 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.989+0000 7fa52ffff640 1 -- 172.21.15.72:0/3679304751 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528038620 con 0x7fa5380712d0 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 -- 172.21.15.72:0/3679304751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5380712d0 msgr2=0x7fa538071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/3679304751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5380712d0 0x7fa538071710 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa52800bb00 tx=0x7fa52802dfb0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 -- 172.21.15.72:0/3679304751 shutdown_connections 2024-03-06T07:23:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/3679304751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5380712d0 0x7fa538071710 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/3679304751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a490 0x7fa538072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 -- 172.21.15.72:0/3679304751 >> 172.21.15.72:0/3679304751 conn(0x7fa53806afb0 msgr2=0x7fa53806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 -- 172.21.15.72:0/3679304751 shutdown_connections 2024-03-06T07:23:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.990+0000 7fa53f9ee640 1 -- 172.21.15.72:0/3679304751 wait complete. 2024-03-06T07:23:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.991+0000 7fa53f9ee640 1 Processor -- start 2024-03-06T07:23:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.991+0000 7fa53f9ee640 1 -- start start 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53f9ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5380712d0 0x7fa538141180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53f9ee640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 0x7fa5380ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53f9ee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a870 0x7fa5380adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53f9ee640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380b3a80 con 0x7fa53815a490 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53f9ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5380b3900 con 0x7fa5380712d0 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53f9ee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa538071fb0 con 0x7fa53815a870 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 0x7fa5380ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 0x7fa5380ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45012/0 (socket says 172.21.15.72:45012) 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 -- 172.21.15.72:0/2694226533 learned_addr learned my addr 172.21.15.72:0/2694226533 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53d78a640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5380712d0 0x7fa538141180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53df8b640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a870 0x7fa5380adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 -- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a870 msgr2=0x7fa5380adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a870 0x7fa5380adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 -- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5380712d0 msgr2=0x7fa538141180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5380712d0 0x7fa538141180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 -- 172.21.15.72:0/2694226533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5381f80c0 con 0x7fa53815a490 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.992+0000 7fa53cf89640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 0x7fa5380ad860 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa52803a040 tx=0x7fa5280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.993+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52802eb60 con 0x7fa53815a490 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.993+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5280415a0 con 0x7fa53815a490 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.993+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa538141be0 con 0x7fa53815a490 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.993+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528041970 con 0x7fa53815a490 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.993+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5381421a0 con 0x7fa53815a490 2024-03-06T07:23:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.994+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53804f330 con 0x7fa53815a490 2024-03-06T07:23:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.994+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa528049020 con 0x7fa53815a490 2024-03-06T07:23:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.995+0000 7fa52dffb640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa51c06b4d0 0x7fa51c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.995+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa5280ad090 con 0x7fa53815a490 2024-03-06T07:23:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.995+0000 7fa53d78a640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa51c06b4d0 0x7fa51c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:21.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.995+0000 7fa53d78a640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa51c06b4d0 0x7fa51c06d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa534009a60 tx=0x7fa534003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:22.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:21.999+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa52807c0e0 con 0x7fa53815a490 2024-03-06T07:23:22.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.208+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa5380728c0 con 0x7fa53815a490 2024-03-06T07:23:22.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.860+0000 7fa52dffb640 1 -- 172.21.15.72:0/2694226533 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v18) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa52807b9e0 con 0x7fa53815a490 2024-03-06T07:23:22.862 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 6 and data pool 7 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.872+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa51c06b4d0 msgr2=0x7fa51c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.872+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa51c06b4d0 0x7fa51c06d990 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa534009a60 tx=0x7fa534003990 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.872+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 msgr2=0x7fa5380ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.872+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 0x7fa5380ad860 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa52803a040 tx=0x7fa5280046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.873+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 shutdown_connections 2024-03-06T07:23:22.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.873+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa51c06b4d0 0x7fa51c06d990 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa534009a60 tx=0x7fa534003990 comp rx=0 tx=0).stop 2024-03-06T07:23:22.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.873+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa53815a870 0x7fa5380adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.873+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa53815a490 0x7fa5380ad860 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.873+0000 7fa53f9ee640 1 --2- 172.21.15.72:0/2694226533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5380712d0 0x7fa538141180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:22.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.873+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 >> 172.21.15.72:0/2694226533 conn(0x7fa53806afb0 msgr2=0x7fa5380b2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:22.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.877+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 shutdown_connections 2024-03-06T07:23:22.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:22.881+0000 7fa53f9ee640 1 -- 172.21.15.72:0/2694226533 wait complete. 2024-03-06T07:23:22.903 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:23.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.002+0000 7f7df3437640 1 Processor -- start 2024-03-06T07:23:23.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df3437640 1 -- start start 2024-03-06T07:23:23.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df3437640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7dec0b4170 0x7f7dec0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df3437640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dec0d4760 0x7f7dec0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df3437640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dec0d25d0 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df3437640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0d4bf0 con 0x7f7dec0d4760 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df3437640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dec0ce9e0 con 0x7f7dec0b4170 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df09d2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dec0d4760 0x7f7dec0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df09d2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dec0d4760 0x7f7dec0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51178/0 (socket says 172.21.15.72:51178) 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.003+0000 7f7df09d2640 1 -- 172.21.15.72:0/3275400282 learned_addr learned my addr 172.21.15.72:0/3275400282 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.004+0000 7f7df09d2640 1 -- 172.21.15.72:0/3275400282 >> v2:172.21.15.146:3300/0 conn(0x7f7dec0b4170 msgr2=0x7f7dec0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.004+0000 7f7df09d2640 1 --2- 172.21.15.72:0/3275400282 >> v2:172.21.15.146:3300/0 conn(0x7f7dec0b4170 0x7f7dec0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.004+0000 7f7df09d2640 1 -- 172.21.15.72:0/3275400282 >> v1:172.21.15.146:6789/0 conn(0x7f7dec0b4dd0 legacy=0x7f7dec0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.004+0000 7f7df09d2640 1 -- 172.21.15.72:0/3275400282 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec0d2870 con 0x7f7dec0d4760 2024-03-06T07:23:23.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.004+0000 7f7df09d2640 1 --2- 172.21.15.72:0/3275400282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0d4760 0x7f7dec0d0370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7dec06e030 tx=0x7f7de002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d251471a65992a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7debfff640 1 -- 172.21.15.72:0/3275400282 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0035020 con 0x7f7dec0d4760 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7debfff640 1 -- 172.21.15.72:0/3275400282 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de0031870 con 0x7f7dec0d4760 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7debfff640 1 -- 172.21.15.72:0/3275400282 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de0031c40 con 0x7f7dec0d4760 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 -- 172.21.15.72:0/3275400282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0d4760 msgr2=0x7f7dec0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 --2- 172.21.15.72:0/3275400282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0d4760 0x7f7dec0d0370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7dec06e030 tx=0x7f7de002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 -- 172.21.15.72:0/3275400282 shutdown_connections 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 --2- 172.21.15.72:0/3275400282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0d4760 0x7f7dec0d0370 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 --2- 172.21.15.72:0/3275400282 >> v2:172.21.15.146:3300/0 conn(0x7f7dec0b4170 0x7f7dec0b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 -- 172.21.15.72:0/3275400282 >> 172.21.15.72:0/3275400282 conn(0x7f7dec0d3c40 msgr2=0x7f7dec0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 -- 172.21.15.72:0/3275400282 shutdown_connections 2024-03-06T07:23:23.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.005+0000 7f7df3437640 1 -- 172.21.15.72:0/3275400282 wait complete. 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.006+0000 7f7df3437640 1 Processor -- start 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.006+0000 7f7df3437640 1 -- start start 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df3437640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dec0b4170 0x7f7dec0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df3437640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 0x7f7dec1e6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df3437640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dec0b2c00 0x7f7dec14dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df3437640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b47d0 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df3437640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b4550 con 0x7f7dec0b4170 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df3437640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dec0b4950 con 0x7f7dec0b2c00 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 0x7f7dec1e6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 0x7f7dec1e6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51194/0 (socket says 172.21.15.72:51194) 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 -- 172.21.15.72:0/1760589101 learned_addr learned my addr 172.21.15.72:0/1760589101 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df11d3640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dec0b4170 0x7f7dec0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df19d4640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dec0b2c00 0x7f7dec14dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 -- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dec0b2c00 msgr2=0x7f7dec14dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dec0b2c00 0x7f7dec14dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 -- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dec0b4170 msgr2=0x7f7dec0b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dec0b4170 0x7f7dec0b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 -- 172.21.15.72:0/1760589101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec1e6d40 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df11d3640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dec0b4170 0x7f7dec0b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df19d4640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dec0b2c00 0x7f7dec14dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.007+0000 7f7df09d2640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 0x7f7dec1e6800 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7de0000c00 tx=0x7f7de00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.008+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de003ad60 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.008+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de00443f0 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.008+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7de00447e0 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.008+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dec0b1b60 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.008+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dec0b2120 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.009+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dec04f330 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.009+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7de0038020 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.010+0000 7f7de9ffb640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dd806b500 0x7f7dd806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.010+0000 7f7df11d3640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dd806b500 0x7f7dd806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.010+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7de00ac490 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.010+0000 7f7df11d3640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dd806b500 0x7f7dd806d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7de4007eb0 tx=0x7f7de40073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.014+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7de003f070 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.185+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dec0733b0 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.186+0000 7f7de9ffb640 1 -- 172.21.15.72:0/1760589101 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f7de007b280 con 0x7f7dec0b4dd0 2024-03-06T07:23:23.188 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:23.188 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":45,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:22.209974+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:23.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dd806b500 msgr2=0x7f7dd806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dd806b500 0x7f7dd806d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7de4007eb0 tx=0x7f7de40073c0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 msgr2=0x7f7dec1e6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 0x7f7dec1e6800 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7de0000c00 tx=0x7f7de00046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 shutdown_connections 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dd806b500 0x7f7dd806d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dec0b2c00 0x7f7dec14dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dec0b4dd0 0x7f7dec1e6800 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 --2- 172.21.15.72:0/1760589101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dec0b4170 0x7f7dec0b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 >> 172.21.15.72:0/1760589101 conn(0x7f7dec0d3c40 msgr2=0x7f7dec06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 shutdown_connections 2024-03-06T07:23:23.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.188+0000 7f7df3437640 1 -- 172.21.15.72:0/1760589101 wait complete. 2024-03-06T07:23:23.201 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:23.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.298+0000 7f220b54b640 1 Processor -- start 2024-03-06T07:23:23.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.299+0000 7f220b54b640 1 -- start start 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.299+0000 7f220b54b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f220406c6d0 0x7f220415ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.299+0000 7f220b54b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 0x7f22040ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.299+0000 7f220b54b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22040cd450 con 0x7f22040b4f70 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.299+0000 7f220b54b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220415e9f0 con 0x7f220406c6d0 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.299+0000 7f220b54b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22040cda10 con 0x7f2204066d90 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f22092e7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f22040b4f70 0x7f22040b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32802/0 (socket says 172.21.15.72:32802) 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f22092e7640 1 -- 172.21.15.72:0/182432945 learned_addr learned my addr 172.21.15.72:0/182432945 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f2209ae8640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 0x7f22040ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f2208ae6640 1 --2- 172.21.15.72:0/182432945 >> v2:172.21.15.146:3300/0 conn(0x7f220406c6d0 0x7f220415ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438856194 0 0) 0x7f22040cd450 con 0x7f22040b4f70 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f0003520 con 0x7f22040b4f70 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2374151904 0 0) 0x7f21f0003520 con 0x7f22040b4f70 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 msgr2=0x7f22040ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 0x7f22040ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 >> v2:172.21.15.146:3300/0 conn(0x7f220406c6d0 msgr2=0x7f220415ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 --2- 172.21.15.72:0/182432945 >> v2:172.21.15.146:3300/0 conn(0x7f220406c6d0 0x7f220415ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22040cd730 con 0x7f22040b4f70 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2200001670 con 0x7f22040b4f70 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 >> v1:172.21.15.72:6789/0 conn(0x7f22040b4f70 legacy=0x7f22040b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 0x7f21f00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0003f30 0x7f21f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21f0004d40 0x7f21f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220415e9f0 con 0x7f21f0004d40 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22040cda10 con 0x7f21f0003f30 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f000b9c0 con 0x7f2204066d90 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f2209ae8640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21f0004d40 0x7f21f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f22092e7640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 0x7f21f00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.300+0000 7f2208ae6640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0003f30 0x7f21f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f2209ae8640 1 -- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 msgr2=0x7f21f00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f2209ae8640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2204066d90 0x7f21f00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f2209ae8640 1 -- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0003f30 msgr2=0x7f21f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f2209ae8640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0003f30 0x7f21f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f2209ae8640 1 -- 172.21.15.72:0/182432945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f0003520 con 0x7f21f0004d40 2024-03-06T07:23:23.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f2209ae8640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21f0004d40 0x7f21f00051c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009f90 tx=0x7f21f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a242e6a0235d3de1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc039070 con 0x7f21f0004d40 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc02cbc0 con 0x7f21f0004d40 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.301+0000 7f21f7fff640 1 -- 172.21.15.72:0/182432945 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc009c10 con 0x7f21f0004d40 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f220b54b640 1 -- 172.21.15.72:0/182432945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21f0004d40 msgr2=0x7f21f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f220b54b640 1 --2- 172.21.15.72:0/182432945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21f0004d40 0x7f21f00051c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009f90 tx=0x7f21f0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f2209ae8640 1 -- 172.21.15.72:0/182432945 reap_dead start 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f220b54b640 1 -- 172.21.15.72:0/182432945 shutdown_connections 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f220b54b640 1 -- 172.21.15.72:0/182432945 >> 172.21.15.72:0/182432945 conn(0x7f22040724d0 msgr2=0x7f22040728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f220b54b640 1 -- 172.21.15.72:0/182432945 shutdown_connections 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.302+0000 7f220b54b640 1 -- 172.21.15.72:0/182432945 wait complete. 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.303+0000 7f220b54b640 1 Processor -- start 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.303+0000 7f220b54b640 1 -- start start 2024-03-06T07:23:23.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f220b54b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0004d40 0x7f220414a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f220b54b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f220414ad90 0x7f220415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f220b54b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 0x7f22041f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f220b54b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22040ccb50 con 0x7f220415a670 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f220b54b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220415f3c0 con 0x7f21f0004d40 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f220b54b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22040cccd0 con 0x7f220414ad90 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 0x7f22041f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 0x7f22041f39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51208/0 (socket says 172.21.15.72:51208) 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 -- 172.21.15.72:0/3568849156 learned_addr learned my addr 172.21.15.72:0/3568849156 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f22092e7640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0004d40 0x7f220414a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2208ae6640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f220414ad90 0x7f220415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 -- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f220414ad90 msgr2=0x7f220415a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f220414ad90 0x7f220415a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 -- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0004d40 msgr2=0x7f220414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0004d40 0x7f220414a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 -- 172.21.15.72:0/3568849156 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22041f4700 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.304+0000 7f2209ae8640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 0x7f22041f39d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009f90 tx=0x7f21fc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.305+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc039030 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.305+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc03a040 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.305+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220414b3b0 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.305+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc009c10 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.305+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220414b970 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.306+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220414bb70 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.306+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f21fc045030 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.306+0000 7f21f5ffb640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21e806b500 0x7f21e806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.307+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21fc0abf90 con 0x7f220415a670 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.307+0000 7f22092e7640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21e806b500 0x7f21e806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.307+0000 7f22092e7640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21e806b500 0x7f21e806d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2200001eb0 tx=0x7f2200007970 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.311+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f220414bb70 con 0x7f220415a670 2024-03-06T07:23:23.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.482+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f220404f330 con 0x7f220415a670 2024-03-06T07:23:23.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.484+0000 7f21f5ffb640 1 -- 172.21.15.72:0/3568849156 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16638 (secure 0 0 0) 0x7f21fc04c020 con 0x7f220415a670 2024-03-06T07:23:23.486 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:23.486 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":45,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:22.209974+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.485+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21e806b500 msgr2=0x7f21e806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21e806b500 0x7f21e806d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2200001eb0 tx=0x7f2200007970 comp rx=0 tx=0).stop 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 msgr2=0x7f22041f39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 0x7f22041f39d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009f90 tx=0x7f21fc009750 comp rx=0 tx=0).stop 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 shutdown_connections 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f21e806b500 0x7f21e806d9c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f220415a670 0x7f22041f39d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f220414ad90 0x7f220415a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 --2- 172.21.15.72:0/3568849156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21f0004d40 0x7f220414a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 >> 172.21.15.72:0/3568849156 conn(0x7f22040724d0 msgr2=0x7f22040728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 shutdown_connections 2024-03-06T07:23:23.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.486+0000 7f220b54b640 1 -- 172.21.15.72:0/3568849156 wait complete. 2024-03-06T07:23:23.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:23:23.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.600+0000 7f7d3b015640 1 Processor -- start 2024-03-06T07:23:23.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.600+0000 7f7d3b015640 1 -- start start 2024-03-06T07:23:23.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d3b015640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3415bb40 0x7f7d340b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d3b015640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d3415f7d0 0x7f7d3406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d3b015640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d3406d5c0 0x7f7d340cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d3b015640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3406c910 con 0x7f7d3406d5c0 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d3b015640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3415ff90 con 0x7f7d3415f7d0 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d3b015640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d34160270 con 0x7f7d3415bb40 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d3406d5c0 0x7f7d340cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d3406d5c0 0x7f7d340cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51214/0 (socket says 172.21.15.72:51214) 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 -- 172.21.15.72:0/3782708780 learned_addr learned my addr 172.21.15.72:0/3782708780 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d38db1640 1 --2- 172.21.15.72:0/3782708780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3415bb40 0x7f7d340b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d33fff640 1 --2- 172.21.15.72:0/3782708780 >> v2:172.21.15.146:3300/0 conn(0x7f7d3415f7d0 0x7f7d3406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 -- 172.21.15.72:0/3782708780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3415bb40 msgr2=0x7f7d340b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 --2- 172.21.15.72:0/3782708780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3415bb40 0x7f7d340b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 -- 172.21.15.72:0/3782708780 >> v2:172.21.15.146:3300/0 conn(0x7f7d3415f7d0 msgr2=0x7f7d3406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 --2- 172.21.15.72:0/3782708780 >> v2:172.21.15.146:3300/0 conn(0x7f7d3415f7d0 0x7f7d3406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.601+0000 7f7d395b2640 1 -- 172.21.15.72:0/3782708780 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34160550 con 0x7f7d3406d5c0 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d395b2640 1 --2- 172.21.15.72:0/3782708780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3406d5c0 0x7f7d340cfa90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7d34072cd0 tx=0x7f7d2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3d123d35fe946 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d337fe640 1 -- 172.21.15.72:0/3782708780 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c02ca70 con 0x7f7d3406d5c0 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d337fe640 1 -- 172.21.15.72:0/3782708780 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c02cc10 con 0x7f7d3406d5c0 2024-03-06T07:23:23.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d337fe640 1 -- 172.21.15.72:0/3782708780 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0343f0 con 0x7f7d3406d5c0 2024-03-06T07:23:23.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 -- 172.21.15.72:0/3782708780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3406d5c0 msgr2=0x7f7d340cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3782708780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3406d5c0 0x7f7d340cfa90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7d34072cd0 tx=0x7f7d2c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 -- 172.21.15.72:0/3782708780 shutdown_connections 2024-03-06T07:23:23.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3782708780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3406d5c0 0x7f7d340cfa90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3782708780 >> v2:172.21.15.146:3300/0 conn(0x7f7d3415f7d0 0x7f7d3406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3782708780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3415bb40 0x7f7d340b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.602+0000 7f7d3b015640 1 -- 172.21.15.72:0/3782708780 >> 172.21.15.72:0/3782708780 conn(0x7f7d3406b190 msgr2=0x7f7d3406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.603+0000 7f7d3b015640 1 -- 172.21.15.72:0/3782708780 shutdown_connections 2024-03-06T07:23:23.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.603+0000 7f7d3b015640 1 -- 172.21.15.72:0/3782708780 wait complete. 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.604+0000 7f7d3b015640 1 Processor -- start 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.604+0000 7f7d3b015640 1 -- start start 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.604+0000 7f7d3b015640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3406d5c0 0x7f7d34159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.604+0000 7f7d3b015640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d3415bb40 0x7f7d341f1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d3b015640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 0x7f7d340b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d3b015640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3415bf20 con 0x7f7d3415f7d0 2024-03-06T07:23:23.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d3b015640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3415fc30 con 0x7f7d3415bb40 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d3b015640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3415c0a0 con 0x7f7d3406d5c0 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 0x7f7d340b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 0x7f7d340b0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51226/0 (socket says 172.21.15.72:51226) 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 -- 172.21.15.72:0/3854555725 learned_addr learned my addr 172.21.15.72:0/3854555725 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d33fff640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d3415bb40 0x7f7d341f1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d38db1640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3406d5c0 0x7f7d34159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 -- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3406d5c0 msgr2=0x7f7d34159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3406d5c0 0x7f7d34159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 -- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d3415bb40 msgr2=0x7f7d341f1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d3415bb40 0x7f7d341f1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 -- 172.21.15.72:0/3854555725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3415ac60 con 0x7f7d3415f7d0 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d395b2640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 0x7f7d340b0c10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c036040 tx=0x7f7d2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c043020 con 0x7f7d3415f7d0 2024-03-06T07:23:23.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c03e450 con 0x7f7d3415f7d0 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3415c350 con 0x7f7d3415f7d0 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c009510 con 0x7f7d3415f7d0 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.605+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3415a6a0 con 0x7f7d3415f7d0 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.606+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d340732b0 con 0x7f7d3415f7d0 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.607+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d2c046020 con 0x7f7d3415f7d0 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.607+0000 7f7d317fa640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d1806b4d0 0x7f7d1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.607+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d2c0ac9e0 con 0x7f7d3415f7d0 2024-03-06T07:23:23.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.607+0000 7f7d38db1640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d1806b4d0 0x7f7d1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.607+0000 7f7d38db1640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d1806b4d0 0x7f7d1806d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7d28006fd0 tx=0x7f7d28008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.611+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d2c04d030 con 0x7f7d3415f7d0 2024-03-06T07:23:23.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.818+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7d34068ff0 con 0x7f7d3415f7d0 2024-03-06T07:23:23.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.868+0000 7f7d317fa640 1 -- 172.21.15.72:0/3854555725 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v19) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7d2c07ba10 con 0x7f7d3415f7d0 2024-03-06T07:23:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d1806b4d0 msgr2=0x7f7d1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d1806b4d0 0x7f7d1806d990 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7d28006fd0 tx=0x7f7d28008040 comp rx=0 tx=0).stop 2024-03-06T07:23:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 msgr2=0x7f7d340b0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 0x7f7d340b0c10 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c036040 tx=0x7f7d2c004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 shutdown_connections 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d1806b4d0 0x7f7d1806d990 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d3415f7d0 0x7f7d340b0c10 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d3415bb40 0x7f7d341f1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 --2- 172.21.15.72:0/3854555725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d3406d5c0 0x7f7d34159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 >> 172.21.15.72:0/3854555725 conn(0x7f7d3406b190 msgr2=0x7f7d34156cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 shutdown_connections 2024-03-06T07:23:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.870+0000 7f7d3b015640 1 -- 172.21.15.72:0/3854555725 wait complete. 2024-03-06T07:23:23.885 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:23:23.885 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:23:23.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.988+0000 7febbaaaf640 1 Processor -- start 2024-03-06T07:23:23.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.988+0000 7febbaaaf640 1 -- start start 2024-03-06T07:23:23.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.989+0000 7febbaaaf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 0x7febb40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.989+0000 7febbaaaf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40cd4b0 con 0x7febb415c050 2024-03-06T07:23:23.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.989+0000 7febbaaaf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb40cd790 con 0x7febb40d0af0 2024-03-06T07:23:23.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.989+0000 7febbaaaf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb40cda70 con 0x7febb406fd10 2024-03-06T07:23:23.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.989+0000 7febb884b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7febb415c050 0x7febb406f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32814/0 (socket says 172.21.15.72:32814) 2024-03-06T07:23:23.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.989+0000 7febb884b640 1 -- 172.21.15.72:0/3540278217 learned_addr learned my addr 172.21.15.72:0/3540278217 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb3fff640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 0x7febb40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207645215 0 0) 0x7febb40cd4b0 con 0x7febb415c050 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba0003520 con 0x7febb415c050 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3178642825 0 0) 0x7feba0003520 con 0x7febb415c050 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 msgr2=0x7febb40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 0x7febb40cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 >> v1:172.21.15.146:6789/0 conn(0x7febb40d0af0 legacy=0x7febb40ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb40cdd50 con 0x7febb415c050 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7febac0019f0 con 0x7febb415c050 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 >> v1:172.21.15.72:6789/0 conn(0x7febb415c050 legacy=0x7febb406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 0x7feba00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feba0003f60 0x7feba0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0004d40 0x7feba00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40cda70 con 0x7feba0004d40 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba000bb50 con 0x7feba0003f60 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba000bd50 con 0x7febb406fd10 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb904c640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0004d40 0x7feba00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb884b640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 0x7feba00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb3fff640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feba0003f60 0x7feba0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb904c640 1 -- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 msgr2=0x7feba00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb904c640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febb406fd10 0x7feba00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb904c640 1 -- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feba0003f60 msgr2=0x7feba0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb904c640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feba0003f60 0x7feba0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.990+0000 7febb904c640 1 -- 172.21.15.72:0/3540278217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba0003520 con 0x7feba0004d40 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.991+0000 7febb904c640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0004d40 0x7feba00051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7feba4001720 tx=0x7feba0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8eb6395e8273d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:23.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.991+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4001370 con 0x7feba0004d40 2024-03-06T07:23:23.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.991+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba40052d0 con 0x7feba0004d40 2024-03-06T07:23:23.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.991+0000 7febb37fe640 1 -- 172.21.15.72:0/3540278217 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4003830 con 0x7feba0004d40 2024-03-06T07:23:23.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febbaaaf640 1 -- 172.21.15.72:0/3540278217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0004d40 msgr2=0x7feba00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:23.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febbaaaf640 1 --2- 172.21.15.72:0/3540278217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feba0004d40 0x7feba00051c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7feba4001720 tx=0x7feba0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:23.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febb904c640 1 -- 172.21.15.72:0/3540278217 reap_dead start 2024-03-06T07:23:23.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febbaaaf640 1 -- 172.21.15.72:0/3540278217 shutdown_connections 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febbaaaf640 1 -- 172.21.15.72:0/3540278217 >> 172.21.15.72:0/3540278217 conn(0x7febb40b3800 msgr2=0x7febb40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febbaaaf640 1 -- 172.21.15.72:0/3540278217 shutdown_connections 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.992+0000 7febbaaaf640 1 -- 172.21.15.72:0/3540278217 wait complete. 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.994+0000 7febbaaaf640 1 Processor -- start 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.994+0000 7febbaaaf640 1 -- start start 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.994+0000 7febbaaaf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feba0004d40 0x7febb409f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febb40b55c0 0x7febb40b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 0x7febb40af620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40ccaf0 con 0x7febb409ca30 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb40cc970 con 0x7febb40b55c0 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb4161a90 con 0x7feba0004d40 2024-03-06T07:23:24.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 0x7febb40af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 0x7febb40af620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51240/0 (socket says 172.21.15.72:51240) 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 -- 172.21.15.72:0/4184078323 learned_addr learned my addr 172.21.15.72:0/4184078323 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb884b640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feba0004d40 0x7febb409f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb3fff640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febb40b55c0 0x7febb40b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 -- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feba0004d40 msgr2=0x7febb409f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feba0004d40 0x7febb409f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 -- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febb40b55c0 msgr2=0x7febb40b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febb40b55c0 0x7febb40b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 -- 172.21.15.72:0/4184078323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb409e800 con 0x7febb409ca30 2024-03-06T07:23:24.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb3fff640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febb40b55c0 0x7febb40b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb884b640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feba0004d40 0x7febb409f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb904c640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 0x7febb40af620 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7feba4001840 tx=0x7feba4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4007a30 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba4044070 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb409eae0 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4007680 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.995+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb409f0a0 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.996+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb409fe60 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.997+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feba403b040 con 0x7febb409ca30 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.997+0000 7febb17fa640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb9406b4d0 0x7feb9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.997+0000 7febb884b640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb9406b4d0 0x7feb9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.997+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feba40ab7a0 con 0x7febb409ca30 2024-03-06T07:23:24.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:23.998+0000 7febb884b640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb9406b4d0 0x7feb9406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7febac001b60 tx=0x7febac002140 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.002+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feba407a6b0 con 0x7febb409ca30 2024-03-06T07:23:24.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.218+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7febb4068ff0 con 0x7febb409ca30 2024-03-06T07:23:24.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.876+0000 7febb17fa640 1 -- 172.21.15.72:0/4184078323 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7feba4079fb0 con 0x7febb409ca30 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb9406b4d0 msgr2=0x7feb9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb9406b4d0 0x7feb9406d990 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7febac001b60 tx=0x7febac002140 comp rx=0 tx=0).stop 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 msgr2=0x7febb40af620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 0x7febb40af620 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7feba4001840 tx=0x7feba4001120 comp rx=0 tx=0).stop 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 shutdown_connections 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb9406b4d0 0x7feb9406d990 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7febb409ca30 0x7febb40af620 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febb40b55c0 0x7febb40b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 --2- 172.21.15.72:0/4184078323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feba0004d40 0x7febb409f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 >> 172.21.15.72:0/4184078323 conn(0x7febb40b3800 msgr2=0x7febb4070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 shutdown_connections 2024-03-06T07:23:24.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.879+0000 7febbaaaf640 1 -- 172.21.15.72:0/4184078323 wait complete. 2024-03-06T07:23:24.892 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:23:24.892 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:24.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.990+0000 7fc3b4df7640 1 Processor -- start 2024-03-06T07:23:24.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.990+0000 7fc3b4df7640 1 -- start start 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3b4df7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3b015bb20 0x7fc3b00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3b4df7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc3b00cfff0 0x7fc3b00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3b4df7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b00ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3b4df7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0161980 con 0x7fc3b015bb20 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3b4df7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0161c60 con 0x7fc3b00cfff0 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3b4df7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b0161f40 con 0x7fc3b00ce4b0 2024-03-06T07:23:24.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3b015bb20 0x7fc3b00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3b015bb20 0x7fc3b00d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51252/0 (socket says 172.21.15.72:51252) 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 -- 172.21.15.72:0/575828122 learned_addr learned my addr 172.21.15.72:0/575828122 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 -- 172.21.15.72:0/575828122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 msgr2=0x7fc3b00ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 --2- 172.21.15.72:0/575828122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b00ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 -- 172.21.15.72:0/575828122 >> v2:172.21.15.146:3300/0 conn(0x7fc3b00cfff0 msgr2=0x7fc3b00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 --2- 172.21.15.72:0/575828122 >> v2:172.21.15.146:3300/0 conn(0x7fc3b00cfff0 0x7fc3b00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.991+0000 7fc3ae59c640 1 -- 172.21.15.72:0/575828122 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0162220 con 0x7fc3b015bb20 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.992+0000 7fc3ae59c640 1 --2- 172.21.15.72:0/575828122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b015bb20 0x7fc3b00d1df0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0071290 tx=0x7fc3a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf637f363cde7ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.992+0000 7fc3add9b640 1 -- 172.21.15.72:0/575828122 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0035020 con 0x7fc3b015bb20 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.992+0000 7fc3add9b640 1 -- 172.21.15.72:0/575828122 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a0031870 con 0x7fc3b015bb20 2024-03-06T07:23:24.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.992+0000 7fc3add9b640 1 -- 172.21.15.72:0/575828122 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031c40 con 0x7fc3b015bb20 2024-03-06T07:23:24.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 -- 172.21.15.72:0/575828122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b015bb20 msgr2=0x7fc3b00d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:24.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/575828122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b015bb20 0x7fc3b00d1df0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0071290 tx=0x7fc3a002b910 comp rx=0 tx=0).stop 2024-03-06T07:23:24.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 -- 172.21.15.72:0/575828122 shutdown_connections 2024-03-06T07:23:24.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/575828122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/575828122 >> v2:172.21.15.146:3300/0 conn(0x7fc3b00cfff0 0x7fc3b00d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/575828122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b015bb20 0x7fc3b00d1df0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 -- 172.21.15.72:0/575828122 >> 172.21.15.72:0/575828122 conn(0x7fc3b009a9f0 msgr2=0x7fc3b009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 -- 172.21.15.72:0/575828122 shutdown_connections 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.993+0000 7fc3b4df7640 1 -- 172.21.15.72:0/575828122 wait complete. 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.994+0000 7fc3b4df7640 1 Processor -- start 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.994+0000 7fc3b4df7640 1 -- start start 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3b4df7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b0157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3b4df7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 0x7fc3b015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3b4df7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3b015bb20 0x7fc3b0160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3b4df7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00d24b0 con 0x7fc3b00cfff0 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3b4df7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00d2330 con 0x7fc3b015bb20 2024-03-06T07:23:24.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3b4df7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00d1980 con 0x7fc3b00ce4b0 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 0x7fc3b015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 0x7fc3b015ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51268/0 (socket says 172.21.15.72:51268) 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 -- 172.21.15.72:0/2451481993 learned_addr learned my addr 172.21.15.72:0/2451481993 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3ae59c640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b0157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3aed9d640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3b015bb20 0x7fc3b0160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 -- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 msgr2=0x7fc3b0157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b0157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 -- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3b015bb20 msgr2=0x7fc3b0160810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3b015bb20 0x7fc3b0160810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 -- 172.21.15.72:0/2451481993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b006c3a0 con 0x7fc3b00cfff0 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.995+0000 7fc3a7fff640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 0x7fc3b015ec40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc39800b4a0 tx=0x7fc39800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.996+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398014070 con 0x7fc3b00cfff0 2024-03-06T07:23:25.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.996+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc398010610 con 0x7fc3b00cfff0 2024-03-06T07:23:25.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.996+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc398010a00 con 0x7fc3b00cfff0 2024-03-06T07:23:25.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.996+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b006c680 con 0x7fc3b00cfff0 2024-03-06T07:23:25.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.996+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b006cc40 con 0x7fc3b00cfff0 2024-03-06T07:23:25.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.997+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b004f330 con 0x7fc3b00cfff0 2024-03-06T07:23:25.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.997+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc398017070 con 0x7fc3b00cfff0 2024-03-06T07:23:25.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.997+0000 7fc3a6ffd640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc39006b4d0 0x7fc39006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.998+0000 7fc3ae59c640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc39006b4d0 0x7fc39006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.998+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc398089c40 con 0x7fc3b00cfff0 2024-03-06T07:23:25.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:24.998+0000 7fc3ae59c640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc39006b4d0 0x7fc39006d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036000 tx=0x7fc3a003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.002+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc398058ac0 con 0x7fc3b00cfff0 2024-03-06T07:23:25.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.212+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3b00b1820 con 0x7fc3b00cfff0 2024-03-06T07:23:25.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.213+0000 7fc3a6ffd640 1 -- 172.21.15.72:0/2451481993 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fc3980583c0 con 0x7fc3b00cfff0 2024-03-06T07:23:25.216 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:25.216 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:22.210297+0000","modified":"2024-03-06T07:23:24.220196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5670":{"gid":5670,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/874695441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":874695441},{"type":"v1","addr":"172.21.15.72:6837","nonce":874695441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5670,"qdb_cluster":[5670]},"id":3}]} 2024-03-06T07:23:25.216 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 20 2024-03-06T07:23:25.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc39006b4d0 msgr2=0x7fc39006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc39006b4d0 0x7fc39006d990 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0036000 tx=0x7fc3a003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 msgr2=0x7fc3b015ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 0x7fc3b015ec40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc39800b4a0 tx=0x7fc39800b960 comp rx=0 tx=0).stop 2024-03-06T07:23:25.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 shutdown_connections 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc39006b4d0 0x7fc39006d990 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3b015bb20 0x7fc3b0160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b00cfff0 0x7fc3b015ec40 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 --2- 172.21.15.72:0/2451481993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3b00ce4b0 0x7fc3b0157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 >> 172.21.15.72:0/2451481993 conn(0x7fc3b009a9f0 msgr2=0x7fc3b015d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 shutdown_connections 2024-03-06T07:23:25.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.215+0000 7fc3b4df7640 1 -- 172.21.15.72:0/2451481993 wait complete. 2024-03-06T07:23:25.230 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:25.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.356+0000 7f58784a4640 1 Processor -- start 2024-03-06T07:23:25.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.356+0000 7f58784a4640 1 -- start start 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.357+0000 7f58784a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f58700d22b0 0x7f58700d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.357+0000 7f58784a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58700b3430 con 0x7f587015dfe0 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.357+0000 7f58784a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58700b3710 con 0x7f58700b41f0 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.357+0000 7f58784a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58700d2c50 con 0x7f58700d22b0 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f5875a3f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f58700d22b0 0x7f58700d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f5876240640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f587015dfe0 0x7f58700cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32824/0 (socket says 172.21.15.72:32824) 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f5876240640 1 -- 172.21.15.72:0/1257680648 learned_addr learned my addr 172.21.15.72:0/1257680648 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f5875a3f640 1 -- 172.21.15.72:0/1257680648 >> v1:172.21.15.146:6789/0 conn(0x7f58700b41f0 legacy=0x7f58700b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f5875a3f640 1 -- 172.21.15.72:0/1257680648 >> v1:172.21.15.72:6789/0 conn(0x7f587015dfe0 legacy=0x7f58700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f587523e640 1 -- 172.21.15.72:0/1257680648 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583347752 0 0) 0x7f58700b3430 con 0x7f587015dfe0 2024-03-06T07:23:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.358+0000 7f5875a3f640 1 -- 172.21.15.72:0/1257680648 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58700b39f0 con 0x7f58700d22b0 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.359+0000 7f5875a3f640 1 --2- 172.21.15.72:0/1257680648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700d22b0 0x7f58700d2710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f58600097d0 tx=0x7f586002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eff75340ed3db41 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.359+0000 7f587523e640 1 -- 172.21.15.72:0/1257680648 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860035020 con 0x7f58700d22b0 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.359+0000 7f587523e640 1 -- 172.21.15.72:0/1257680648 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5860031830 con 0x7f58700d22b0 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.359+0000 7f587523e640 1 -- 172.21.15.72:0/1257680648 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5860031c00 con 0x7f58700d22b0 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 -- 172.21.15.72:0/1257680648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700d22b0 msgr2=0x7f58700d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 --2- 172.21.15.72:0/1257680648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700d22b0 0x7f58700d2710 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f58600097d0 tx=0x7f586002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 -- 172.21.15.72:0/1257680648 shutdown_connections 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 --2- 172.21.15.72:0/1257680648 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700d22b0 0x7f58700d2710 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 -- 172.21.15.72:0/1257680648 >> 172.21.15.72:0/1257680648 conn(0x7f58700d3cd0 msgr2=0x7f58700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 -- 172.21.15.72:0/1257680648 shutdown_connections 2024-03-06T07:23:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.360+0000 7f58784a4640 1 -- 172.21.15.72:0/1257680648 wait complete. 2024-03-06T07:23:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.361+0000 7f58784a4640 1 Processor -- start 2024-03-06T07:23:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.361+0000 7f58784a4640 1 -- start start 2024-03-06T07:23:25.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 0x7f58700b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58700d22b0 0x7f58700b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f587015dfe0 0x7f587014f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700b3d60 con 0x7f58700b41f0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58700b31e0 con 0x7f58700d22b0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58700d1c70 con 0x7f587015dfe0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 0x7f58700b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 0x7f58700b75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51292/0 (socket says 172.21.15.72:51292) 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 -- 172.21.15.72:0/668848794 learned_addr learned my addr 172.21.15.72:0/668848794 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5875a3f640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58700d22b0 0x7f58700b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876a41640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f587015dfe0 0x7f587014f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 -- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f587015dfe0 msgr2=0x7f587014f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f587015dfe0 0x7f587014f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 -- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58700d22b0 msgr2=0x7f58700b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58700d22b0 0x7f58700b6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 -- 172.21.15.72:0/668848794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58700b7cc0 con 0x7f58700b41f0 2024-03-06T07:23:25.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5876240640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 0x7f58700b75c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f586800c430 tx=0x7f586800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58680074e0 con 0x7f58700b41f0 2024-03-06T07:23:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58680047e0 con 0x7f58700b41f0 2024-03-06T07:23:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58700b7fa0 con 0x7f58700b41f0 2024-03-06T07:23:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.362+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5868004bd0 con 0x7f58700b41f0 2024-03-06T07:23:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.363+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587015fed0 con 0x7f58700b41f0 2024-03-06T07:23:25.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.364+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587004f330 con 0x7f58700b41f0 2024-03-06T07:23:25.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.364+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5868011070 con 0x7f58700b41f0 2024-03-06T07:23:25.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.364+0000 7f5866ffd640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f585406b4d0 0x7f585406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.365+0000 7f5875a3f640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f585406b4d0 0x7f585406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.365+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5868089ec0 con 0x7f58700b41f0 2024-03-06T07:23:25.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.365+0000 7f5875a3f640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f585406b4d0 0x7f585406d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5860036000 tx=0x7f586003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.369+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5868058d40 con 0x7f58700b41f0 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.547+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f587014f5d0 con 0x7f58700b41f0 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.548+0000 7f5866ffd640 1 -- 172.21.15.72:0/668848794 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16625 (secure 0 0 0) 0x7f5868058640 con 0x7f58700b41f0 2024-03-06T07:23:25.552 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:25.553 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":46,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:23.868650+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:25.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f585406b4d0 msgr2=0x7f585406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f585406b4d0 0x7f585406d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5860036000 tx=0x7f586003c040 comp rx=0 tx=0).stop 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 msgr2=0x7f58700b75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 0x7f58700b75c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f586800c430 tx=0x7f586800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 shutdown_connections 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f585406b4d0 0x7f585406d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f587015dfe0 0x7f587014f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58700d22b0 0x7f58700b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 --2- 172.21.15.72:0/668848794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58700b41f0 0x7f58700b75c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 >> 172.21.15.72:0/668848794 conn(0x7f58700d3cd0 msgr2=0x7f587015d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 shutdown_connections 2024-03-06T07:23:25.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.550+0000 7f58784a4640 1 -- 172.21.15.72:0/668848794 wait complete. 2024-03-06T07:23:25.564 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:23:25.564 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.663+0000 7fcd72b1e640 1 Processor -- start 2024-03-06T07:23:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.663+0000 7fcd72b1e640 1 -- start start 2024-03-06T07:23:25.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd72b1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcd6c1424f0 0x7fcd6c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd72b1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd6c06da40 0x7fcd6c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd72b1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c0cc980 con 0x7fcd6c0cd7a0 2024-03-06T07:23:25.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd72b1e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c0cd220 con 0x7fcd6c1424f0 2024-03-06T07:23:25.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd72b1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c0ccc60 con 0x7fcd6c06da40 2024-03-06T07:23:25.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd708ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcd6c1424f0 0x7fcd6c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd6bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcd6c0cd7a0 0x7fcd6c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32830/0 (socket says 172.21.15.72:32830) 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd6bfff640 1 -- 172.21.15.72:0/1063470033 learned_addr learned my addr 172.21.15.72:0/1063470033 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd710bb640 1 --2- 172.21.15.72:0/1063470033 >> v2:172.21.15.146:3300/0 conn(0x7fcd6c06da40 0x7fcd6c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd708ba640 1 -- 172.21.15.72:0/1063470033 >> v2:172.21.15.146:3300/0 conn(0x7fcd6c06da40 msgr2=0x7fcd6c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd708ba640 1 --2- 172.21.15.72:0/1063470033 >> v2:172.21.15.146:3300/0 conn(0x7fcd6c06da40 0x7fcd6c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd708ba640 1 -- 172.21.15.72:0/1063470033 >> v1:172.21.15.72:6789/0 conn(0x7fcd6c0cd7a0 legacy=0x7fcd6c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd6b7fe640 1 -- 172.21.15.72:0/1063470033 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246771330 0 0) 0x7fcd6c0cc980 con 0x7fcd6c0cd7a0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.664+0000 7fcd708ba640 1 -- 172.21.15.72:0/1063470033 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c0ccf40 con 0x7fcd6c1424f0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.665+0000 7fcd708ba640 1 --2- 172.21.15.72:0/1063470033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c1424f0 0x7fcd6c0cc2f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c068700 tx=0x7fcd6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb3aa34dc0873dda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.665+0000 7fcd6b7fe640 1 -- 172.21.15.72:0/1063470033 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60035020 con 0x7fcd6c1424f0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.665+0000 7fcd6b7fe640 1 -- 172.21.15.72:0/1063470033 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd60031830 con 0x7fcd6c1424f0 2024-03-06T07:23:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.665+0000 7fcd6b7fe640 1 -- 172.21.15.72:0/1063470033 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60031c00 con 0x7fcd6c1424f0 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 -- 172.21.15.72:0/1063470033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c1424f0 msgr2=0x7fcd6c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/1063470033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c1424f0 0x7fcd6c0cc2f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c068700 tx=0x7fcd6002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 -- 172.21.15.72:0/1063470033 shutdown_connections 2024-03-06T07:23:25.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/1063470033 >> v2:172.21.15.146:3300/0 conn(0x7fcd6c06da40 0x7fcd6c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/1063470033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c1424f0 0x7fcd6c0cc2f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 -- 172.21.15.72:0/1063470033 >> 172.21.15.72:0/1063470033 conn(0x7fcd6c0cf8a0 msgr2=0x7fcd6c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 -- 172.21.15.72:0/1063470033 shutdown_connections 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.666+0000 7fcd72b1e640 1 -- 172.21.15.72:0/1063470033 wait complete. 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.667+0000 7fcd72b1e640 1 Processor -- start 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.667+0000 7fcd72b1e640 1 -- start start 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd72b1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 0x7fcd6c13e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd72b1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c0cd7a0 0x7fcd6c1402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd72b1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd6c1424f0 0x7fcd6c1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd72b1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c071ce0 con 0x7fcd6c06da40 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd72b1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c071b60 con 0x7fcd6c0cd7a0 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd72b1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c071e60 con 0x7fcd6c1424f0 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 0x7fcd6c13e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 0x7fcd6c13e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51320/0 (socket says 172.21.15.72:51320) 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 -- 172.21.15.72:0/3426627541 learned_addr learned my addr 172.21.15.72:0/3426627541 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd6bfff640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c0cd7a0 0x7fcd6c1402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd710bb640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd6c1424f0 0x7fcd6c1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 -- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd6c1424f0 msgr2=0x7fcd6c1571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd6c1424f0 0x7fcd6c1571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 -- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c0cd7a0 msgr2=0x7fcd6c1402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c0cd7a0 0x7fcd6c1402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 -- 172.21.15.72:0/3426627541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c13d280 con 0x7fcd6c06da40 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd710bb640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd6c1424f0 0x7fcd6c1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.668+0000 7fcd708ba640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 0x7fcd6c13e720 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcd60000c00 tx=0x7fcd600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.669+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd60035040 con 0x7fcd6c06da40 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.669+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd60004230 con 0x7fcd6c06da40 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.669+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c140a60 con 0x7fcd6c06da40 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.669+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6003a700 con 0x7fcd6c06da40 2024-03-06T07:23:25.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.669+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c141020 con 0x7fcd6c06da40 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.670+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c04f330 con 0x7fcd6c06da40 2024-03-06T07:23:25.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.670+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcd60038020 con 0x7fcd6c06da40 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.670+0000 7fcd697fa640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcd5006b4d0 0x7fcd5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.671+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd600abef0 con 0x7fcd6c06da40 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.671+0000 7fcd6bfff640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcd5006b4d0 0x7fcd5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:25.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.671+0000 7fcd6bfff640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcd5006b4d0 0x7fcd5006d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c13bda0 tx=0x7fcd58007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:25.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.675+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcd6007ae00 con 0x7fcd6c06da40 2024-03-06T07:23:25.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.844+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd6c0d0620 con 0x7fcd5006b4d0 2024-03-06T07:23:25.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.850+0000 7fcd697fa640 1 -- 172.21.15.72:0/3426627541 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246635 (secure 0 0 0) 0x7fcd6c0d0620 con 0x7fcd5006b4d0 2024-03-06T07:23:25.855 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:25.856 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcd5006b4d0 msgr2=0x7fcd5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcd5006b4d0 0x7fcd5006d990 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c13bda0 tx=0x7fcd58007b50 comp rx=0 tx=0).stop 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 msgr2=0x7fcd6c13e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 0x7fcd6c13e720 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcd60000c00 tx=0x7fcd600046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 shutdown_connections 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcd5006b4d0 0x7fcd5006d990 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd6c1424f0 0x7fcd6c1571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c0cd7a0 0x7fcd6c1402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 --2- 172.21.15.72:0/3426627541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd6c06da40 0x7fcd6c13e720 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:25.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 >> 172.21.15.72:0/3426627541 conn(0x7fcd6c0cf8a0 msgr2=0x7fcd6c15cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 shutdown_connections 2024-03-06T07:23:25.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:25.853+0000 7fcd72b1e640 1 -- 172.21.15.72:0/3426627541 wait complete. 2024-03-06T07:23:25.866 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-03-06T07:23:23.964697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254536,"kb_used_data":5096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720184,"statfs":{"total":773094113280,"available":772833468416,"internally_reserved":0,"allocated":5218304,"data_stored":2784288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000672"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856478+0000","last_change":"2024-03-06T07:23:21.856478+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856478+0000","last_undegraded":"2024-03-06T07:23:21.856478+0000","last_fullsized":"2024-03-06T07:23:21.856478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855000+0000","last_change":"2024-03-06T07:23:21.855000+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855000+0000","last_undegraded":"2024-03-06T07:23:21.855000+0000","last_fullsized":"2024-03-06T07:23:21.855000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860791+0000","last_change":"2024-03-06T07:23:21.860791+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860791+0000","last_undegraded":"2024-03-06T07:23:21.860791+0000","last_fullsized":"2024-03-06T07:23:21.860791+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860752+0000","last_change":"2024-03-06T07:23:21.860752+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860752+0000","last_undegraded":"2024-03-06T07:23:21.860752+0000","last_fullsized":"2024-03-06T07:23:21.860752+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853814+0000","last_change":"2024-03-06T07:23:21.853814+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853814+0000","last_undegraded":"2024-03-06T07:23:21.853814+0000","last_fullsized":"2024-03-06T07:23:21.853814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861844+0000","last_change":"2024-03-06T07:23:21.861844+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861844+0000","last_undegraded":"2024-03-06T07:23:21.861844+0000","last_fullsized":"2024-03-06T07:23:21.861844+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855942+0000","last_change":"2024-03-06T07:23:21.855942+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855942+0000","last_undegraded":"2024-03-06T07:23:21.855942+0000","last_fullsized":"2024-03-06T07:23:21.855942+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857730+0000","last_change":"2024-03-06T07:23:21.857730+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857730+0000","last_undegraded":"2024-03-06T07:23:21.857730+0000","last_fullsized":"2024-03-06T07:23:21.857730+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860536+0000","last_change":"2024-03-06T07:23:21.860536+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860536+0000","last_undegraded":"2024-03-06T07:23:21.860536+0000","last_fullsized":"2024-03-06T07:23:21.860536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860872+0000","last_change":"2024-03-06T07:23:21.860872+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860872+0000","last_undegraded":"2024-03-06T07:23:21.860872+0000","last_fullsized":"2024-03-06T07:23:21.860872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860140+0000","last_change":"2024-03-06T07:23:21.860140+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860140+0000","last_undegraded":"2024-03-06T07:23:21.860140+0000","last_fullsized":"2024-03-06T07:23:21.860140+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855745+0000","last_change":"2024-03-06T07:23:21.855745+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855745+0000","last_undegraded":"2024-03-06T07:23:21.855745+0000","last_fullsized":"2024-03-06T07:23:21.855745+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856960+0000","last_change":"2024-03-06T07:23:21.856960+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856960+0000","last_undegraded":"2024-03-06T07:23:21.856960+0000","last_fullsized":"2024-03-06T07:23:21.856960+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860354+0000","last_change":"2024-03-06T07:23:21.860354+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860354+0000","last_undegraded":"2024-03-06T07:23:21.860354+0000","last_fullsized":"2024-03-06T07:23:21.860354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862480+0000","last_change":"2024-03-06T07:23:21.862480+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862480+0000","last_undegraded":"2024-03-06T07:23:21.862480+0000","last_fullsized":"2024-03-06T07:23:21.862480+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859111+0000","last_change":"2024-03-06T07:23:21.859111+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859111+0000","last_undegraded":"2024-03-06T07:23:21.859111+0000","last_fullsized":"2024-03-06T07:23:21.859111+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861819+0000","last_change":"2024-03-06T07:23:21.861819+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861819+0000","last_undegraded":"2024-03-06T07:23:21.861819+0000","last_fullsized":"2024-03-06T07:23:21.861819+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861199+0000","last_change":"2024-03-06T07:23:21.861199+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861199+0000","last_undegraded":"2024-03-06T07:23:21.861199+0000","last_fullsized":"2024-03-06T07:23:21.861199+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860297+0000","last_change":"2024-03-06T07:23:21.860297+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860297+0000","last_undegraded":"2024-03-06T07:23:21.860297+0000","last_fullsized":"2024-03-06T07:23:21.860297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857004+0000","last_change":"2024-03-06T07:23:21.857004+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857004+0000","last_undegraded":"2024-03-06T07:23:21.857004+0000","last_fullsized":"2024-03-06T07:23:21.857004+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863958+0000","last_change":"2024-03-06T07:23:21.863958+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863958+0000","last_undegraded":"2024-03-06T07:23:21.863958+0000","last_fullsized":"2024-03-06T07:23:21.863958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856664+0000","last_change":"2024-03-06T07:23:21.856664+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856664+0000","last_undegraded":"2024-03-06T07:23:21.856664+0000","last_fullsized":"2024-03-06T07:23:21.856664+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858592+0000","last_change":"2024-03-06T07:23:21.858592+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858592+0000","last_undegraded":"2024-03-06T07:23:21.858592+0000","last_fullsized":"2024-03-06T07:23:21.858592+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858616+0000","last_change":"2024-03-06T07:23:21.858616+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858616+0000","last_undegraded":"2024-03-06T07:23:21.858616+0000","last_fullsized":"2024-03-06T07:23:21.858616+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863987+0000","last_change":"2024-03-06T07:23:21.863987+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863987+0000","last_undegraded":"2024-03-06T07:23:21.863987+0000","last_fullsized":"2024-03-06T07:23:21.863987+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860157+0000","last_change":"2024-03-06T07:23:21.860157+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860157+0000","last_undegraded":"2024-03-06T07:23:21.860157+0000","last_fullsized":"2024-03-06T07:23:21.860157+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859204+0000","last_change":"2024-03-06T07:23:21.859204+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859204+0000","last_undegraded":"2024-03-06T07:23:21.859204+0000","last_fullsized":"2024-03-06T07:23:21.859204+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854927+0000","last_change":"2024-03-06T07:23:21.854927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854927+0000","last_undegraded":"2024-03-06T07:23:21.854927+0000","last_fullsized":"2024-03-06T07:23:21.854927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862784+0000","last_change":"2024-03-06T07:23:21.862784+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862784+0000","last_undegraded":"2024-03-06T07:23:21.862784+0000","last_fullsized":"2024-03-06T07:23:21.862784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859904+0000","last_change":"2024-03-06T07:23:21.859904+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859904+0000","last_undegraded":"2024-03-06T07:23:21.859904+0000","last_fullsized":"2024-03-06T07:23:21.859904+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858223+0000","last_change":"2024-03-06T07:23:21.858223+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858223+0000","last_undegraded":"2024-03-06T07:23:21.858223+0000","last_fullsized":"2024-03-06T07:23:21.858223+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860570+0000","last_change":"2024-03-06T07:23:21.860570+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860570+0000","last_undegraded":"2024-03-06T07:23:21.860570+0000","last_fullsized":"2024-03-06T07:23:21.860570+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858941+0000","last_change":"2024-03-06T07:23:21.858941+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858941+0000","last_undegraded":"2024-03-06T07:23:21.858941+0000","last_fullsized":"2024-03-06T07:23:21.858941+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874695+0000","last_change":"2024-03-06T07:23:21.870976+0000","last_active":"2024-03-06T07:23:21.874695+0000","last_peered":"2024-03-06T07:23:21.874695+0000","last_clean":"2024-03-06T07:23:21.874695+0000","last_became_active":"2024-03-06T07:23:21.870706+0000","last_became_peered":"2024-03-06T07:23:21.870706+0000","last_unstale":"2024-03-06T07:23:21.874695+0000","last_undegraded":"2024-03-06T07:23:21.874695+0000","last_fullsized":"2024-03-06T07:23:21.874695+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860916+0000","last_change":"2024-03-06T07:23:21.860916+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860916+0000","last_undegraded":"2024-03-06T07:23:21.860916+0000","last_fullsized":"2024-03-06T07:23:21.860916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.865503+0000","last_change":"2024-03-06T07:23:21.864644+0000","last_active":"2024-03-06T07:23:21.865503+0000","last_peered":"2024-03-06T07:23:21.865503+0000","last_clean":"2024-03-06T07:23:21.865503+0000","last_became_active":"2024-03-06T07:23:21.864375+0000","last_became_peered":"2024-03-06T07:23:21.864375+0000","last_unstale":"2024-03-06T07:23:21.865503+0000","last_undegraded":"2024-03-06T07:23:21.865503+0000","last_fullsized":"2024-03-06T07:23:21.865503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854364+0000","last_change":"2024-03-06T07:23:21.854364+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854364+0000","last_undegraded":"2024-03-06T07:23:21.854364+0000","last_fullsized":"2024-03-06T07:23:21.854364+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869438+0000","last_change":"2024-03-06T07:23:21.868746+0000","last_active":"2024-03-06T07:23:21.869438+0000","last_peered":"2024-03-06T07:23:21.869438+0000","last_clean":"2024-03-06T07:23:21.869438+0000","last_became_active":"2024-03-06T07:23:21.868455+0000","last_became_peered":"2024-03-06T07:23:21.868455+0000","last_unstale":"2024-03-06T07:23:21.869438+0000","last_undegraded":"2024-03-06T07:23:21.869438+0000","last_fullsized":"2024-03-06T07:23:21.869438+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:44.858008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860303+0000","last_change":"2024-03-06T07:23:21.860303+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860303+0000","last_undegraded":"2024-03-06T07:23:21.860303+0000","last_fullsized":"2024-03-06T07:23:21.860303+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870280+0000","last_change":"2024-03-06T07:23:21.865942+0000","last_active":"2024-03-06T07:23:21.870280+0000","last_peered":"2024-03-06T07:23:21.870280+0000","last_clean":"2024-03-06T07:23:21.870280+0000","last_became_active":"2024-03-06T07:23:21.865642+0000","last_became_peered":"2024-03-06T07:23:21.865642+0000","last_unstale":"2024-03-06T07:23:21.870280+0000","last_undegraded":"2024-03-06T07:23:21.870280+0000","last_fullsized":"2024-03-06T07:23:21.870280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859164+0000","last_change":"2024-03-06T07:23:21.859164+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859164+0000","last_undegraded":"2024-03-06T07:23:21.859164+0000","last_fullsized":"2024-03-06T07:23:21.859164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874913+0000","last_change":"2024-03-06T07:23:21.872744+0000","last_active":"2024-03-06T07:23:21.874913+0000","last_peered":"2024-03-06T07:23:21.874913+0000","last_clean":"2024-03-06T07:23:21.874913+0000","last_became_active":"2024-03-06T07:23:21.872444+0000","last_became_peered":"2024-03-06T07:23:21.872444+0000","last_unstale":"2024-03-06T07:23:21.874913+0000","last_undegraded":"2024-03-06T07:23:21.874913+0000","last_fullsized":"2024-03-06T07:23:21.874913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:43.054124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857762+0000","last_change":"2024-03-06T07:23:21.857762+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857762+0000","last_undegraded":"2024-03-06T07:23:21.857762+0000","last_fullsized":"2024-03-06T07:23:21.857762+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870098+0000","last_change":"2024-03-06T07:23:21.868824+0000","last_active":"2024-03-06T07:23:21.870098+0000","last_peered":"2024-03-06T07:23:21.870098+0000","last_clean":"2024-03-06T07:23:21.870098+0000","last_became_active":"2024-03-06T07:23:21.867672+0000","last_became_peered":"2024-03-06T07:23:21.867672+0000","last_unstale":"2024-03-06T07:23:21.870098+0000","last_undegraded":"2024-03-06T07:23:21.870098+0000","last_fullsized":"2024-03-06T07:23:21.870098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857267+0000","last_change":"2024-03-06T07:23:21.857267+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857267+0000","last_undegraded":"2024-03-06T07:23:21.857267+0000","last_fullsized":"2024-03-06T07:23:21.857267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866889+0000","last_change":"2024-03-06T07:23:21.863472+0000","last_active":"2024-03-06T07:23:21.866889+0000","last_peered":"2024-03-06T07:23:21.866889+0000","last_clean":"2024-03-06T07:23:21.866889+0000","last_became_active":"2024-03-06T07:23:21.863161+0000","last_became_peered":"2024-03-06T07:23:21.863161+0000","last_unstale":"2024-03-06T07:23:21.866889+0000","last_undegraded":"2024-03-06T07:23:21.866889+0000","last_fullsized":"2024-03-06T07:23:21.866889+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:42.354236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.865694+0000","last_change":"2024-03-06T07:23:21.865694+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.865694+0000","last_undegraded":"2024-03-06T07:23:21.865694+0000","last_fullsized":"2024-03-06T07:23:21.865694+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867714+0000","last_change":"2024-03-06T07:23:21.864598+0000","last_active":"2024-03-06T07:23:21.867714+0000","last_peered":"2024-03-06T07:23:21.867714+0000","last_clean":"2024-03-06T07:23:21.867714+0000","last_became_active":"2024-03-06T07:23:21.862773+0000","last_became_peered":"2024-03-06T07:23:21.862773+0000","last_unstale":"2024-03-06T07:23:21.867714+0000","last_undegraded":"2024-03-06T07:23:21.867714+0000","last_fullsized":"2024-03-06T07:23:21.867714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860119+0000","last_change":"2024-03-06T07:23:21.860119+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860119+0000","last_undegraded":"2024-03-06T07:23:21.860119+0000","last_fullsized":"2024-03-06T07:23:21.860119+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870825+0000","last_change":"2024-03-06T07:23:21.867046+0000","last_active":"2024-03-06T07:23:21.870825+0000","last_peered":"2024-03-06T07:23:21.870825+0000","last_clean":"2024-03-06T07:23:21.870825+0000","last_became_active":"2024-03-06T07:23:21.866751+0000","last_became_peered":"2024-03-06T07:23:21.866751+0000","last_unstale":"2024-03-06T07:23:21.870825+0000","last_undegraded":"2024-03-06T07:23:21.870825+0000","last_fullsized":"2024-03-06T07:23:21.870825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854878+0000","last_change":"2024-03-06T07:23:21.854878+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854878+0000","last_undegraded":"2024-03-06T07:23:21.854878+0000","last_fullsized":"2024-03-06T07:23:21.854878+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874650+0000","last_change":"2024-03-06T07:23:21.869633+0000","last_active":"2024-03-06T07:23:21.874650+0000","last_peered":"2024-03-06T07:23:21.874650+0000","last_clean":"2024-03-06T07:23:21.874650+0000","last_became_active":"2024-03-06T07:23:21.869349+0000","last_became_peered":"2024-03-06T07:23:21.869349+0000","last_unstale":"2024-03-06T07:23:21.874650+0000","last_undegraded":"2024-03-06T07:23:21.874650+0000","last_fullsized":"2024-03-06T07:23:21.874650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857576+0000","last_change":"2024-03-06T07:23:21.857576+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857576+0000","last_undegraded":"2024-03-06T07:23:21.857576+0000","last_fullsized":"2024-03-06T07:23:21.857576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871236+0000","last_change":"2024-03-06T07:23:21.868933+0000","last_active":"2024-03-06T07:23:21.871236+0000","last_peered":"2024-03-06T07:23:21.871236+0000","last_clean":"2024-03-06T07:23:21.871236+0000","last_became_active":"2024-03-06T07:23:21.868305+0000","last_became_peered":"2024-03-06T07:23:21.868305+0000","last_unstale":"2024-03-06T07:23:21.871236+0000","last_undegraded":"2024-03-06T07:23:21.871236+0000","last_fullsized":"2024-03-06T07:23:21.871236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856248+0000","last_change":"2024-03-06T07:23:21.856248+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856248+0000","last_undegraded":"2024-03-06T07:23:21.856248+0000","last_fullsized":"2024-03-06T07:23:21.856248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866259+0000","last_change":"2024-03-06T07:23:21.862722+0000","last_active":"2024-03-06T07:23:21.866259+0000","last_peered":"2024-03-06T07:23:21.866259+0000","last_clean":"2024-03-06T07:23:21.866259+0000","last_became_active":"2024-03-06T07:23:21.862470+0000","last_became_peered":"2024-03-06T07:23:21.862470+0000","last_unstale":"2024-03-06T07:23:21.866259+0000","last_undegraded":"2024-03-06T07:23:21.866259+0000","last_fullsized":"2024-03-06T07:23:21.866259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:13.201036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860147+0000","last_change":"2024-03-06T07:23:21.860147+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860147+0000","last_undegraded":"2024-03-06T07:23:21.860147+0000","last_fullsized":"2024-03-06T07:23:21.860147+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.877196+0000","last_change":"2024-03-06T07:23:21.875315+0000","last_active":"2024-03-06T07:23:21.877196+0000","last_peered":"2024-03-06T07:23:21.877196+0000","last_clean":"2024-03-06T07:23:21.877196+0000","last_became_active":"2024-03-06T07:23:21.874938+0000","last_became_peered":"2024-03-06T07:23:21.874938+0000","last_unstale":"2024-03-06T07:23:21.877196+0000","last_undegraded":"2024-03-06T07:23:21.877196+0000","last_fullsized":"2024-03-06T07:23:21.877196+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.194391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857385+0000","last_change":"2024-03-06T07:23:21.857385+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857385+0000","last_undegraded":"2024-03-06T07:23:21.857385+0000","last_fullsized":"2024-03-06T07:23:21.857385+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871183+0000","last_change":"2024-03-06T07:23:21.869035+0000","last_active":"2024-03-06T07:23:21.871183+0000","last_peered":"2024-03-06T07:23:21.871183+0000","last_clean":"2024-03-06T07:23:21.871183+0000","last_became_active":"2024-03-06T07:23:21.868739+0000","last_became_peered":"2024-03-06T07:23:21.868739+0000","last_unstale":"2024-03-06T07:23:21.871183+0000","last_undegraded":"2024-03-06T07:23:21.871183+0000","last_fullsized":"2024-03-06T07:23:21.871183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857927+0000","last_change":"2024-03-06T07:23:21.857927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857927+0000","last_undegraded":"2024-03-06T07:23:21.857927+0000","last_fullsized":"2024-03-06T07:23:21.857927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874442+0000","last_change":"2024-03-06T07:23:21.868680+0000","last_active":"2024-03-06T07:23:21.874442+0000","last_peered":"2024-03-06T07:23:21.874442+0000","last_clean":"2024-03-06T07:23:21.874442+0000","last_became_active":"2024-03-06T07:23:21.868416+0000","last_became_peered":"2024-03-06T07:23:21.868416+0000","last_unstale":"2024-03-06T07:23:21.874442+0000","last_undegraded":"2024-03-06T07:23:21.874442+0000","last_fullsized":"2024-03-06T07:23:21.874442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855516+0000","last_change":"2024-03-06T07:23:21.855516+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855516+0000","last_undegraded":"2024-03-06T07:23:21.855516+0000","last_fullsized":"2024-03-06T07:23:21.855516+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874641+0000","last_change":"2024-03-06T07:23:21.870601+0000","last_active":"2024-03-06T07:23:21.874641+0000","last_peered":"2024-03-06T07:23:21.874641+0000","last_clean":"2024-03-06T07:23:21.874641+0000","last_became_active":"2024-03-06T07:23:21.870306+0000","last_became_peered":"2024-03-06T07:23:21.870306+0000","last_unstale":"2024-03-06T07:23:21.874641+0000","last_undegraded":"2024-03-06T07:23:21.874641+0000","last_fullsized":"2024-03-06T07:23:21.874641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874729+0000","last_change":"2024-03-06T07:23:21.872671+0000","last_active":"2024-03-06T07:23:21.874729+0000","last_peered":"2024-03-06T07:23:21.874729+0000","last_clean":"2024-03-06T07:23:21.874729+0000","last_became_active":"2024-03-06T07:23:21.872330+0000","last_became_peered":"2024-03-06T07:23:21.872330+0000","last_unstale":"2024-03-06T07:23:21.874729+0000","last_undegraded":"2024-03-06T07:23:21.874729+0000","last_fullsized":"2024-03-06T07:23:21.874729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:10.134358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856748+0000","last_change":"2024-03-06T07:23:21.856748+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856748+0000","last_undegraded":"2024-03-06T07:23:21.856748+0000","last_fullsized":"2024-03-06T07:23:21.856748+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.872248+0000","last_change":"2024-03-06T07:23:21.870771+0000","last_active":"2024-03-06T07:23:21.872248+0000","last_peered":"2024-03-06T07:23:21.872248+0000","last_clean":"2024-03-06T07:23:21.872248+0000","last_became_active":"2024-03-06T07:23:21.870450+0000","last_became_peered":"2024-03-06T07:23:21.870450+0000","last_unstale":"2024-03-06T07:23:21.872248+0000","last_undegraded":"2024-03-06T07:23:21.872248+0000","last_fullsized":"2024-03-06T07:23:21.872248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859255+0000","last_change":"2024-03-06T07:23:21.859255+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859255+0000","last_undegraded":"2024-03-06T07:23:21.859255+0000","last_fullsized":"2024-03-06T07:23:21.859255+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869938+0000","last_change":"2024-03-06T07:23:21.865980+0000","last_active":"2024-03-06T07:23:21.869938+0000","last_peered":"2024-03-06T07:23:21.869938+0000","last_clean":"2024-03-06T07:23:21.869938+0000","last_became_active":"2024-03-06T07:23:21.865586+0000","last_became_peered":"2024-03-06T07:23:21.865586+0000","last_unstale":"2024-03-06T07:23:21.869938+0000","last_undegraded":"2024-03-06T07:23:21.869938+0000","last_fullsized":"2024-03-06T07:23:21.869938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861702+0000","last_change":"2024-03-06T07:23:21.861702+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861702+0000","last_undegraded":"2024-03-06T07:23:21.861702+0000","last_fullsized":"2024-03-06T07:23:21.861702+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871876+0000","last_change":"2024-03-06T07:23:21.866832+0000","last_active":"2024-03-06T07:23:21.871876+0000","last_peered":"2024-03-06T07:23:21.871876+0000","last_clean":"2024-03-06T07:23:21.871876+0000","last_became_active":"2024-03-06T07:23:21.866546+0000","last_became_peered":"2024-03-06T07:23:21.866546+0000","last_unstale":"2024-03-06T07:23:21.871876+0000","last_undegraded":"2024-03-06T07:23:21.871876+0000","last_fullsized":"2024-03-06T07:23:21.871876+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859270+0000","last_change":"2024-03-06T07:23:21.859270+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859270+0000","last_undegraded":"2024-03-06T07:23:21.859270+0000","last_fullsized":"2024-03-06T07:23:21.859270+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871931+0000","last_change":"2024-03-06T07:23:21.867500+0000","last_active":"2024-03-06T07:23:21.871931+0000","last_peered":"2024-03-06T07:23:21.871931+0000","last_clean":"2024-03-06T07:23:21.871931+0000","last_became_active":"2024-03-06T07:23:21.867223+0000","last_became_peered":"2024-03-06T07:23:21.867223+0000","last_unstale":"2024-03-06T07:23:21.871931+0000","last_undegraded":"2024-03-06T07:23:21.871931+0000","last_fullsized":"2024-03-06T07:23:21.871931+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859561+0000","last_change":"2024-03-06T07:23:21.859561+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859561+0000","last_undegraded":"2024-03-06T07:23:21.859561+0000","last_fullsized":"2024-03-06T07:23:21.859561+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869963+0000","last_change":"2024-03-06T07:23:21.866528+0000","last_active":"2024-03-06T07:23:21.869963+0000","last_peered":"2024-03-06T07:23:21.869963+0000","last_clean":"2024-03-06T07:23:21.869963+0000","last_became_active":"2024-03-06T07:23:21.866281+0000","last_became_peered":"2024-03-06T07:23:21.866281+0000","last_unstale":"2024-03-06T07:23:21.869963+0000","last_undegraded":"2024-03-06T07:23:21.869963+0000","last_fullsized":"2024-03-06T07:23:21.869963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860173+0000","last_change":"2024-03-06T07:23:21.860173+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860173+0000","last_undegraded":"2024-03-06T07:23:21.860173+0000","last_fullsized":"2024-03-06T07:23:21.860173+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874610+0000","last_change":"2024-03-06T07:23:21.870289+0000","last_active":"2024-03-06T07:23:21.874610+0000","last_peered":"2024-03-06T07:23:21.874610+0000","last_clean":"2024-03-06T07:23:21.874610+0000","last_became_active":"2024-03-06T07:23:21.870037+0000","last_became_peered":"2024-03-06T07:23:21.870037+0000","last_unstale":"2024-03-06T07:23:21.874610+0000","last_undegraded":"2024-03-06T07:23:21.874610+0000","last_fullsized":"2024-03-06T07:23:21.874610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857088+0000","last_change":"2024-03-06T07:23:21.857088+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857088+0000","last_undegraded":"2024-03-06T07:23:21.857088+0000","last_fullsized":"2024-03-06T07:23:21.857088+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871188+0000","last_change":"2024-03-06T07:23:21.869803+0000","last_active":"2024-03-06T07:23:21.871188+0000","last_peered":"2024-03-06T07:23:21.871188+0000","last_clean":"2024-03-06T07:23:21.871188+0000","last_became_active":"2024-03-06T07:23:21.869511+0000","last_became_peered":"2024-03-06T07:23:21.869511+0000","last_unstale":"2024-03-06T07:23:21.871188+0000","last_undegraded":"2024-03-06T07:23:21.871188+0000","last_fullsized":"2024-03-06T07:23:21.871188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860466+0000","last_change":"2024-03-06T07:23:21.860466+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860466+0000","last_undegraded":"2024-03-06T07:23:21.860466+0000","last_fullsized":"2024-03-06T07:23:21.860466+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874543+0000","last_change":"2024-03-06T07:23:21.871570+0000","last_active":"2024-03-06T07:23:21.874543+0000","last_peered":"2024-03-06T07:23:21.874543+0000","last_clean":"2024-03-06T07:23:21.874543+0000","last_became_active":"2024-03-06T07:23:21.871278+0000","last_became_peered":"2024-03-06T07:23:21.871278+0000","last_unstale":"2024-03-06T07:23:21.874543+0000","last_undegraded":"2024-03-06T07:23:21.874543+0000","last_fullsized":"2024-03-06T07:23:21.874543+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:00.837068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856546+0000","last_change":"2024-03-06T07:23:21.856546+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856546+0000","last_undegraded":"2024-03-06T07:23:21.856546+0000","last_fullsized":"2024-03-06T07:23:21.856546+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871218+0000","last_change":"2024-03-06T07:23:21.868700+0000","last_active":"2024-03-06T07:23:21.871218+0000","last_peered":"2024-03-06T07:23:21.871218+0000","last_clean":"2024-03-06T07:23:21.871218+0000","last_became_active":"2024-03-06T07:23:21.868258+0000","last_became_peered":"2024-03-06T07:23:21.868258+0000","last_unstale":"2024-03-06T07:23:21.871218+0000","last_undegraded":"2024-03-06T07:23:21.871218+0000","last_fullsized":"2024-03-06T07:23:21.871218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855824+0000","last_change":"2024-03-06T07:23:21.855824+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855824+0000","last_undegraded":"2024-03-06T07:23:21.855824+0000","last_fullsized":"2024-03-06T07:23:21.855824+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869070+0000","last_change":"2024-03-06T07:23:21.868016+0000","last_active":"2024-03-06T07:23:21.869070+0000","last_peered":"2024-03-06T07:23:21.869070+0000","last_clean":"2024-03-06T07:23:21.869070+0000","last_became_active":"2024-03-06T07:23:21.867724+0000","last_became_peered":"2024-03-06T07:23:21.867724+0000","last_unstale":"2024-03-06T07:23:21.869070+0000","last_undegraded":"2024-03-06T07:23:21.869070+0000","last_fullsized":"2024-03-06T07:23:21.869070+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:24.138593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857738+0000","last_change":"2024-03-06T07:23:21.857738+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857738+0000","last_undegraded":"2024-03-06T07:23:21.857738+0000","last_fullsized":"2024-03-06T07:23:21.857738+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870300+0000","last_change":"2024-03-06T07:23:21.867760+0000","last_active":"2024-03-06T07:23:21.870300+0000","last_peered":"2024-03-06T07:23:21.870300+0000","last_clean":"2024-03-06T07:23:21.870300+0000","last_became_active":"2024-03-06T07:23:21.867499+0000","last_became_peered":"2024-03-06T07:23:21.867499+0000","last_unstale":"2024-03-06T07:23:21.870300+0000","last_undegraded":"2024-03-06T07:23:21.870300+0000","last_fullsized":"2024-03-06T07:23:21.870300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863555+0000","last_change":"2024-03-06T07:23:21.863555+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863555+0000","last_undegraded":"2024-03-06T07:23:21.863555+0000","last_fullsized":"2024-03-06T07:23:21.863555+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870077+0000","last_change":"2024-03-06T07:23:21.867044+0000","last_active":"2024-03-06T07:23:21.870077+0000","last_peered":"2024-03-06T07:23:21.870077+0000","last_clean":"2024-03-06T07:23:21.870077+0000","last_became_active":"2024-03-06T07:23:21.866525+0000","last_became_peered":"2024-03-06T07:23:21.866525+0000","last_unstale":"2024-03-06T07:23:21.870077+0000","last_undegraded":"2024-03-06T07:23:21.870077+0000","last_fullsized":"2024-03-06T07:23:21.870077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853974+0000","last_change":"2024-03-06T07:23:21.853974+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853974+0000","last_undegraded":"2024-03-06T07:23:21.853974+0000","last_fullsized":"2024-03-06T07:23:21.853974+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874320+0000","last_change":"2024-03-06T07:23:21.871204+0000","last_active":"2024-03-06T07:23:21.874320+0000","last_peered":"2024-03-06T07:23:21.874320+0000","last_clean":"2024-03-06T07:23:21.874320+0000","last_became_active":"2024-03-06T07:23:21.870923+0000","last_became_peered":"2024-03-06T07:23:21.870923+0000","last_unstale":"2024-03-06T07:23:21.874320+0000","last_undegraded":"2024-03-06T07:23:21.874320+0000","last_fullsized":"2024-03-06T07:23:21.874320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.850435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858703+0000","last_change":"2024-03-06T07:23:21.858703+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858703+0000","last_undegraded":"2024-03-06T07:23:21.858703+0000","last_fullsized":"2024-03-06T07:23:21.858703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874520+0000","last_change":"2024-03-06T07:23:21.867519+0000","last_active":"2024-03-06T07:23:21.874520+0000","last_peered":"2024-03-06T07:23:21.874520+0000","last_clean":"2024-03-06T07:23:21.874520+0000","last_became_active":"2024-03-06T07:23:21.867274+0000","last_became_peered":"2024-03-06T07:23:21.867274+0000","last_unstale":"2024-03-06T07:23:21.874520+0000","last_undegraded":"2024-03-06T07:23:21.874520+0000","last_fullsized":"2024-03-06T07:23:21.874520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867006+0000","last_change":"2024-03-06T07:23:21.863945+0000","last_active":"2024-03-06T07:23:21.867006+0000","last_peered":"2024-03-06T07:23:21.867006+0000","last_clean":"2024-03-06T07:23:21.867006+0000","last_became_active":"2024-03-06T07:23:21.863699+0000","last_became_peered":"2024-03-06T07:23:21.863699+0000","last_unstale":"2024-03-06T07:23:21.867006+0000","last_undegraded":"2024-03-06T07:23:21.867006+0000","last_fullsized":"2024-03-06T07:23:21.867006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.854216+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:21.854216+0000","last_peered":"2024-03-06T07:23:21.854216+0000","last_clean":"2024-03-06T07:23:21.854216+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:21.854216+0000","last_undegraded":"2024-03-06T07:23:21.854216+0000","last_fullsized":"2024-03-06T07:23:21.854216+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:25.871 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:26.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.007+0000 7fa62a4e1640 1 Processor -- start 2024-03-06T07:23:26.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.007+0000 7fa62a4e1640 1 -- start start 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa62a4e1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa62415ac70 0x7fa6240cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa62a4e1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa62406d0d0 0x7fa6240ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa62a4e1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62406efb0 con 0x7fa62415e650 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa62a4e1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62406b170 con 0x7fa62406d0d0 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa62a4e1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62406ec60 con 0x7fa62415ac70 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa6237fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa62406d0d0 0x7fa6240ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa628a7e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa62415e650 0x7fa62415eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32836/0 (socket says 172.21.15.72:32836) 2024-03-06T07:23:26.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.008+0000 7fa628a7e640 1 -- 172.21.15.72:0/2228393719 learned_addr learned my addr 172.21.15.72:0/2228393719 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa623fff640 1 --2- 172.21.15.72:0/2228393719 >> v2:172.21.15.146:3300/0 conn(0x7fa62415ac70 0x7fa6240cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa6237fe640 1 -- 172.21.15.72:0/2228393719 >> v2:172.21.15.146:3300/0 conn(0x7fa62415ac70 msgr2=0x7fa6240cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa6237fe640 1 --2- 172.21.15.72:0/2228393719 >> v2:172.21.15.146:3300/0 conn(0x7fa62415ac70 0x7fa6240cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa6237fe640 1 -- 172.21.15.72:0/2228393719 >> v1:172.21.15.72:6789/0 conn(0x7fa62415e650 legacy=0x7fa62415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa6237fe640 1 -- 172.21.15.72:0/2228393719 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62406e6e0 con 0x7fa62406d0d0 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa622ffd640 1 -- 172.21.15.72:0/2228393719 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967582648 0 0) 0x7fa62406efb0 con 0x7fa62415e650 2024-03-06T07:23:26.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.009+0000 7fa623fff640 1 --2- 172.21.15.72:0/2228393719 >> v2:172.21.15.146:3300/0 conn(0x7fa62415ac70 0x7fa6240cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.010+0000 7fa6237fe640 1 --2- 172.21.15.72:0/2228393719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62406d0d0 0x7fa6240ce1f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa624067a90 tx=0x7fa61802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a5b94b71baf02b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.010+0000 7fa622ffd640 1 -- 172.21.15.72:0/2228393719 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618038070 con 0x7fa62406d0d0 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.010+0000 7fa622ffd640 1 -- 172.21.15.72:0/2228393719 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618031710 con 0x7fa62406d0d0 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.010+0000 7fa622ffd640 1 -- 172.21.15.72:0/2228393719 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618031ae0 con 0x7fa62406d0d0 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.010+0000 7fa62a4e1640 1 -- 172.21.15.72:0/2228393719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62406d0d0 msgr2=0x7fa6240ce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.010+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/2228393719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62406d0d0 0x7fa6240ce1f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fa624067a90 tx=0x7fa61802b7b0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.011+0000 7fa62a4e1640 1 -- 172.21.15.72:0/2228393719 shutdown_connections 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.011+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/2228393719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62406d0d0 0x7fa6240ce1f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.011+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/2228393719 >> v2:172.21.15.146:3300/0 conn(0x7fa62415ac70 0x7fa6240cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.011+0000 7fa62a4e1640 1 -- 172.21.15.72:0/2228393719 >> 172.21.15.72:0/2228393719 conn(0x7fa6240d00e0 msgr2=0x7fa6240d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.011+0000 7fa62a4e1640 1 -- 172.21.15.72:0/2228393719 shutdown_connections 2024-03-06T07:23:26.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.011+0000 7fa62a4e1640 1 -- 172.21.15.72:0/2228393719 wait complete. 2024-03-06T07:23:26.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.012+0000 7fa62a4e1640 1 Processor -- start 2024-03-06T07:23:26.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.012+0000 7fa62a4e1640 1 -- start start 2024-03-06T07:23:26.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa62406d0d0 0x7fa624156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa62415ac70 0x7fa6240c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 0x7fa624158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62406e910 con 0x7fa62415e650 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62406e400 con 0x7fa62406d0d0 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62409dec0 con 0x7fa62415ac70 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 0x7fa624158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 0x7fa624158510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51340/0 (socket says 172.21.15.72:51340) 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 -- 172.21.15.72:0/1622568586 learned_addr learned my addr 172.21.15.72:0/1622568586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa6237fe640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa62415ac70 0x7fa6240c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa623fff640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa62406d0d0 0x7fa624156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 -- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa62415ac70 msgr2=0x7fa6240c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa62415ac70 0x7fa6240c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 -- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa62406d0d0 msgr2=0x7fa624156e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa62406d0d0 0x7fa624156e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 -- 172.21.15.72:0/1622568586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624157470 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa623fff640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa62406d0d0 0x7fa624156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa628a7e640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 0x7fa624158510 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00c7a0 tx=0x7fa61c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c018070 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c00a640 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00aab0 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa624157750 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.013+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa624157d10 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.015+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6240b3380 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.015+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa61c01b070 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.015+0000 7fa620ff9640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa61006b4d0 0x7fa61006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.015+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa61c089700 con 0x7fa62415e650 2024-03-06T07:23:26.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.015+0000 7fa623fff640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa61006b4d0 0x7fa61006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.016+0000 7fa623fff640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa61006b4d0 0x7fa61006d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa62415e620 tx=0x7fa614008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.023+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa6240b3380 con 0x7fa62415e650 2024-03-06T07:23:26.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.298+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa62404f330 con 0x7fa61006b4d0 2024-03-06T07:23:26.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.304+0000 7fa620ff9640 1 -- 172.21.15.72:0/1622568586 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246635 (secure 0 0 0) 0x7fa62404f330 con 0x7fa61006b4d0 2024-03-06T07:23:26.309 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:26.310 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa61006b4d0 msgr2=0x7fa61006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa61006b4d0 0x7fa61006d990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa62415e620 tx=0x7fa614008040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 msgr2=0x7fa624158510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 0x7fa624158510 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00c7a0 tx=0x7fa61c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 shutdown_connections 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa61006b4d0 0x7fa61006d990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62415e650 0x7fa624158510 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa62415ac70 0x7fa6240c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 --2- 172.21.15.72:0/1622568586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa62406d0d0 0x7fa624156e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 >> 172.21.15.72:0/1622568586 conn(0x7fa6240d00e0 msgr2=0x7fa6240b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 shutdown_connections 2024-03-06T07:23:26.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.308+0000 7fa62a4e1640 1 -- 172.21.15.72:0/1622568586 wait complete. 2024-03-06T07:23:26.321 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-06T07:23:25.964943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254536,"kb_used_data":5096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720184,"statfs":{"total":773094113280,"available":772833468416,"internally_reserved":0,"allocated":5218304,"data_stored":2784288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.115574"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856478+0000","last_change":"2024-03-06T07:23:21.856478+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856478+0000","last_undegraded":"2024-03-06T07:23:21.856478+0000","last_fullsized":"2024-03-06T07:23:21.856478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855000+0000","last_change":"2024-03-06T07:23:21.855000+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855000+0000","last_undegraded":"2024-03-06T07:23:21.855000+0000","last_fullsized":"2024-03-06T07:23:21.855000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860791+0000","last_change":"2024-03-06T07:23:21.860791+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860791+0000","last_undegraded":"2024-03-06T07:23:21.860791+0000","last_fullsized":"2024-03-06T07:23:21.860791+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860752+0000","last_change":"2024-03-06T07:23:21.860752+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860752+0000","last_undegraded":"2024-03-06T07:23:21.860752+0000","last_fullsized":"2024-03-06T07:23:21.860752+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853814+0000","last_change":"2024-03-06T07:23:21.853814+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853814+0000","last_undegraded":"2024-03-06T07:23:21.853814+0000","last_fullsized":"2024-03-06T07:23:21.853814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861844+0000","last_change":"2024-03-06T07:23:21.861844+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861844+0000","last_undegraded":"2024-03-06T07:23:21.861844+0000","last_fullsized":"2024-03-06T07:23:21.861844+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855942+0000","last_change":"2024-03-06T07:23:21.855942+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855942+0000","last_undegraded":"2024-03-06T07:23:21.855942+0000","last_fullsized":"2024-03-06T07:23:21.855942+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857730+0000","last_change":"2024-03-06T07:23:21.857730+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857730+0000","last_undegraded":"2024-03-06T07:23:21.857730+0000","last_fullsized":"2024-03-06T07:23:21.857730+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860536+0000","last_change":"2024-03-06T07:23:21.860536+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860536+0000","last_undegraded":"2024-03-06T07:23:21.860536+0000","last_fullsized":"2024-03-06T07:23:21.860536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860872+0000","last_change":"2024-03-06T07:23:21.860872+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860872+0000","last_undegraded":"2024-03-06T07:23:21.860872+0000","last_fullsized":"2024-03-06T07:23:21.860872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860140+0000","last_change":"2024-03-06T07:23:21.860140+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860140+0000","last_undegraded":"2024-03-06T07:23:21.860140+0000","last_fullsized":"2024-03-06T07:23:21.860140+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855745+0000","last_change":"2024-03-06T07:23:21.855745+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855745+0000","last_undegraded":"2024-03-06T07:23:21.855745+0000","last_fullsized":"2024-03-06T07:23:21.855745+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856960+0000","last_change":"2024-03-06T07:23:21.856960+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856960+0000","last_undegraded":"2024-03-06T07:23:21.856960+0000","last_fullsized":"2024-03-06T07:23:21.856960+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860354+0000","last_change":"2024-03-06T07:23:21.860354+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860354+0000","last_undegraded":"2024-03-06T07:23:21.860354+0000","last_fullsized":"2024-03-06T07:23:21.860354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862480+0000","last_change":"2024-03-06T07:23:21.862480+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862480+0000","last_undegraded":"2024-03-06T07:23:21.862480+0000","last_fullsized":"2024-03-06T07:23:21.862480+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859111+0000","last_change":"2024-03-06T07:23:21.859111+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859111+0000","last_undegraded":"2024-03-06T07:23:21.859111+0000","last_fullsized":"2024-03-06T07:23:21.859111+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861819+0000","last_change":"2024-03-06T07:23:21.861819+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861819+0000","last_undegraded":"2024-03-06T07:23:21.861819+0000","last_fullsized":"2024-03-06T07:23:21.861819+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861199+0000","last_change":"2024-03-06T07:23:21.861199+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861199+0000","last_undegraded":"2024-03-06T07:23:21.861199+0000","last_fullsized":"2024-03-06T07:23:21.861199+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860297+0000","last_change":"2024-03-06T07:23:21.860297+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860297+0000","last_undegraded":"2024-03-06T07:23:21.860297+0000","last_fullsized":"2024-03-06T07:23:21.860297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857004+0000","last_change":"2024-03-06T07:23:21.857004+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857004+0000","last_undegraded":"2024-03-06T07:23:21.857004+0000","last_fullsized":"2024-03-06T07:23:21.857004+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863958+0000","last_change":"2024-03-06T07:23:21.863958+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863958+0000","last_undegraded":"2024-03-06T07:23:21.863958+0000","last_fullsized":"2024-03-06T07:23:21.863958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856664+0000","last_change":"2024-03-06T07:23:21.856664+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856664+0000","last_undegraded":"2024-03-06T07:23:21.856664+0000","last_fullsized":"2024-03-06T07:23:21.856664+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858592+0000","last_change":"2024-03-06T07:23:21.858592+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858592+0000","last_undegraded":"2024-03-06T07:23:21.858592+0000","last_fullsized":"2024-03-06T07:23:21.858592+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858616+0000","last_change":"2024-03-06T07:23:21.858616+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858616+0000","last_undegraded":"2024-03-06T07:23:21.858616+0000","last_fullsized":"2024-03-06T07:23:21.858616+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863987+0000","last_change":"2024-03-06T07:23:21.863987+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863987+0000","last_undegraded":"2024-03-06T07:23:21.863987+0000","last_fullsized":"2024-03-06T07:23:21.863987+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860157+0000","last_change":"2024-03-06T07:23:21.860157+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860157+0000","last_undegraded":"2024-03-06T07:23:21.860157+0000","last_fullsized":"2024-03-06T07:23:21.860157+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859204+0000","last_change":"2024-03-06T07:23:21.859204+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859204+0000","last_undegraded":"2024-03-06T07:23:21.859204+0000","last_fullsized":"2024-03-06T07:23:21.859204+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854927+0000","last_change":"2024-03-06T07:23:21.854927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854927+0000","last_undegraded":"2024-03-06T07:23:21.854927+0000","last_fullsized":"2024-03-06T07:23:21.854927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862784+0000","last_change":"2024-03-06T07:23:21.862784+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862784+0000","last_undegraded":"2024-03-06T07:23:21.862784+0000","last_fullsized":"2024-03-06T07:23:21.862784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859904+0000","last_change":"2024-03-06T07:23:21.859904+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859904+0000","last_undegraded":"2024-03-06T07:23:21.859904+0000","last_fullsized":"2024-03-06T07:23:21.859904+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858223+0000","last_change":"2024-03-06T07:23:21.858223+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858223+0000","last_undegraded":"2024-03-06T07:23:21.858223+0000","last_fullsized":"2024-03-06T07:23:21.858223+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860570+0000","last_change":"2024-03-06T07:23:21.860570+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860570+0000","last_undegraded":"2024-03-06T07:23:21.860570+0000","last_fullsized":"2024-03-06T07:23:21.860570+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858941+0000","last_change":"2024-03-06T07:23:21.858941+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858941+0000","last_undegraded":"2024-03-06T07:23:21.858941+0000","last_fullsized":"2024-03-06T07:23:21.858941+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874695+0000","last_change":"2024-03-06T07:23:21.870976+0000","last_active":"2024-03-06T07:23:21.874695+0000","last_peered":"2024-03-06T07:23:21.874695+0000","last_clean":"2024-03-06T07:23:21.874695+0000","last_became_active":"2024-03-06T07:23:21.870706+0000","last_became_peered":"2024-03-06T07:23:21.870706+0000","last_unstale":"2024-03-06T07:23:21.874695+0000","last_undegraded":"2024-03-06T07:23:21.874695+0000","last_fullsized":"2024-03-06T07:23:21.874695+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860916+0000","last_change":"2024-03-06T07:23:21.860916+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860916+0000","last_undegraded":"2024-03-06T07:23:21.860916+0000","last_fullsized":"2024-03-06T07:23:21.860916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.865503+0000","last_change":"2024-03-06T07:23:21.864644+0000","last_active":"2024-03-06T07:23:21.865503+0000","last_peered":"2024-03-06T07:23:21.865503+0000","last_clean":"2024-03-06T07:23:21.865503+0000","last_became_active":"2024-03-06T07:23:21.864375+0000","last_became_peered":"2024-03-06T07:23:21.864375+0000","last_unstale":"2024-03-06T07:23:21.865503+0000","last_undegraded":"2024-03-06T07:23:21.865503+0000","last_fullsized":"2024-03-06T07:23:21.865503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854364+0000","last_change":"2024-03-06T07:23:21.854364+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854364+0000","last_undegraded":"2024-03-06T07:23:21.854364+0000","last_fullsized":"2024-03-06T07:23:21.854364+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869438+0000","last_change":"2024-03-06T07:23:21.868746+0000","last_active":"2024-03-06T07:23:21.869438+0000","last_peered":"2024-03-06T07:23:21.869438+0000","last_clean":"2024-03-06T07:23:21.869438+0000","last_became_active":"2024-03-06T07:23:21.868455+0000","last_became_peered":"2024-03-06T07:23:21.868455+0000","last_unstale":"2024-03-06T07:23:21.869438+0000","last_undegraded":"2024-03-06T07:23:21.869438+0000","last_fullsized":"2024-03-06T07:23:21.869438+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:44.858008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860303+0000","last_change":"2024-03-06T07:23:21.860303+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860303+0000","last_undegraded":"2024-03-06T07:23:21.860303+0000","last_fullsized":"2024-03-06T07:23:21.860303+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870280+0000","last_change":"2024-03-06T07:23:21.865942+0000","last_active":"2024-03-06T07:23:21.870280+0000","last_peered":"2024-03-06T07:23:21.870280+0000","last_clean":"2024-03-06T07:23:21.870280+0000","last_became_active":"2024-03-06T07:23:21.865642+0000","last_became_peered":"2024-03-06T07:23:21.865642+0000","last_unstale":"2024-03-06T07:23:21.870280+0000","last_undegraded":"2024-03-06T07:23:21.870280+0000","last_fullsized":"2024-03-06T07:23:21.870280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859164+0000","last_change":"2024-03-06T07:23:21.859164+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859164+0000","last_undegraded":"2024-03-06T07:23:21.859164+0000","last_fullsized":"2024-03-06T07:23:21.859164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874913+0000","last_change":"2024-03-06T07:23:21.872744+0000","last_active":"2024-03-06T07:23:21.874913+0000","last_peered":"2024-03-06T07:23:21.874913+0000","last_clean":"2024-03-06T07:23:21.874913+0000","last_became_active":"2024-03-06T07:23:21.872444+0000","last_became_peered":"2024-03-06T07:23:21.872444+0000","last_unstale":"2024-03-06T07:23:21.874913+0000","last_undegraded":"2024-03-06T07:23:21.874913+0000","last_fullsized":"2024-03-06T07:23:21.874913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:43.054124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857762+0000","last_change":"2024-03-06T07:23:21.857762+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857762+0000","last_undegraded":"2024-03-06T07:23:21.857762+0000","last_fullsized":"2024-03-06T07:23:21.857762+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870098+0000","last_change":"2024-03-06T07:23:21.868824+0000","last_active":"2024-03-06T07:23:21.870098+0000","last_peered":"2024-03-06T07:23:21.870098+0000","last_clean":"2024-03-06T07:23:21.870098+0000","last_became_active":"2024-03-06T07:23:21.867672+0000","last_became_peered":"2024-03-06T07:23:21.867672+0000","last_unstale":"2024-03-06T07:23:21.870098+0000","last_undegraded":"2024-03-06T07:23:21.870098+0000","last_fullsized":"2024-03-06T07:23:21.870098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857267+0000","last_change":"2024-03-06T07:23:21.857267+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857267+0000","last_undegraded":"2024-03-06T07:23:21.857267+0000","last_fullsized":"2024-03-06T07:23:21.857267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866889+0000","last_change":"2024-03-06T07:23:21.863472+0000","last_active":"2024-03-06T07:23:21.866889+0000","last_peered":"2024-03-06T07:23:21.866889+0000","last_clean":"2024-03-06T07:23:21.866889+0000","last_became_active":"2024-03-06T07:23:21.863161+0000","last_became_peered":"2024-03-06T07:23:21.863161+0000","last_unstale":"2024-03-06T07:23:21.866889+0000","last_undegraded":"2024-03-06T07:23:21.866889+0000","last_fullsized":"2024-03-06T07:23:21.866889+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:42.354236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.865694+0000","last_change":"2024-03-06T07:23:21.865694+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.865694+0000","last_undegraded":"2024-03-06T07:23:21.865694+0000","last_fullsized":"2024-03-06T07:23:21.865694+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867714+0000","last_change":"2024-03-06T07:23:21.864598+0000","last_active":"2024-03-06T07:23:21.867714+0000","last_peered":"2024-03-06T07:23:21.867714+0000","last_clean":"2024-03-06T07:23:21.867714+0000","last_became_active":"2024-03-06T07:23:21.862773+0000","last_became_peered":"2024-03-06T07:23:21.862773+0000","last_unstale":"2024-03-06T07:23:21.867714+0000","last_undegraded":"2024-03-06T07:23:21.867714+0000","last_fullsized":"2024-03-06T07:23:21.867714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860119+0000","last_change":"2024-03-06T07:23:21.860119+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860119+0000","last_undegraded":"2024-03-06T07:23:21.860119+0000","last_fullsized":"2024-03-06T07:23:21.860119+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870825+0000","last_change":"2024-03-06T07:23:21.867046+0000","last_active":"2024-03-06T07:23:21.870825+0000","last_peered":"2024-03-06T07:23:21.870825+0000","last_clean":"2024-03-06T07:23:21.870825+0000","last_became_active":"2024-03-06T07:23:21.866751+0000","last_became_peered":"2024-03-06T07:23:21.866751+0000","last_unstale":"2024-03-06T07:23:21.870825+0000","last_undegraded":"2024-03-06T07:23:21.870825+0000","last_fullsized":"2024-03-06T07:23:21.870825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854878+0000","last_change":"2024-03-06T07:23:21.854878+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854878+0000","last_undegraded":"2024-03-06T07:23:21.854878+0000","last_fullsized":"2024-03-06T07:23:21.854878+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874650+0000","last_change":"2024-03-06T07:23:21.869633+0000","last_active":"2024-03-06T07:23:21.874650+0000","last_peered":"2024-03-06T07:23:21.874650+0000","last_clean":"2024-03-06T07:23:21.874650+0000","last_became_active":"2024-03-06T07:23:21.869349+0000","last_became_peered":"2024-03-06T07:23:21.869349+0000","last_unstale":"2024-03-06T07:23:21.874650+0000","last_undegraded":"2024-03-06T07:23:21.874650+0000","last_fullsized":"2024-03-06T07:23:21.874650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857576+0000","last_change":"2024-03-06T07:23:21.857576+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857576+0000","last_undegraded":"2024-03-06T07:23:21.857576+0000","last_fullsized":"2024-03-06T07:23:21.857576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871236+0000","last_change":"2024-03-06T07:23:21.868933+0000","last_active":"2024-03-06T07:23:21.871236+0000","last_peered":"2024-03-06T07:23:21.871236+0000","last_clean":"2024-03-06T07:23:21.871236+0000","last_became_active":"2024-03-06T07:23:21.868305+0000","last_became_peered":"2024-03-06T07:23:21.868305+0000","last_unstale":"2024-03-06T07:23:21.871236+0000","last_undegraded":"2024-03-06T07:23:21.871236+0000","last_fullsized":"2024-03-06T07:23:21.871236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856248+0000","last_change":"2024-03-06T07:23:21.856248+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856248+0000","last_undegraded":"2024-03-06T07:23:21.856248+0000","last_fullsized":"2024-03-06T07:23:21.856248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866259+0000","last_change":"2024-03-06T07:23:21.862722+0000","last_active":"2024-03-06T07:23:21.866259+0000","last_peered":"2024-03-06T07:23:21.866259+0000","last_clean":"2024-03-06T07:23:21.866259+0000","last_became_active":"2024-03-06T07:23:21.862470+0000","last_became_peered":"2024-03-06T07:23:21.862470+0000","last_unstale":"2024-03-06T07:23:21.866259+0000","last_undegraded":"2024-03-06T07:23:21.866259+0000","last_fullsized":"2024-03-06T07:23:21.866259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:13.201036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860147+0000","last_change":"2024-03-06T07:23:21.860147+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860147+0000","last_undegraded":"2024-03-06T07:23:21.860147+0000","last_fullsized":"2024-03-06T07:23:21.860147+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.877196+0000","last_change":"2024-03-06T07:23:21.875315+0000","last_active":"2024-03-06T07:23:21.877196+0000","last_peered":"2024-03-06T07:23:21.877196+0000","last_clean":"2024-03-06T07:23:21.877196+0000","last_became_active":"2024-03-06T07:23:21.874938+0000","last_became_peered":"2024-03-06T07:23:21.874938+0000","last_unstale":"2024-03-06T07:23:21.877196+0000","last_undegraded":"2024-03-06T07:23:21.877196+0000","last_fullsized":"2024-03-06T07:23:21.877196+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.194391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857385+0000","last_change":"2024-03-06T07:23:21.857385+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857385+0000","last_undegraded":"2024-03-06T07:23:21.857385+0000","last_fullsized":"2024-03-06T07:23:21.857385+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871183+0000","last_change":"2024-03-06T07:23:21.869035+0000","last_active":"2024-03-06T07:23:21.871183+0000","last_peered":"2024-03-06T07:23:21.871183+0000","last_clean":"2024-03-06T07:23:21.871183+0000","last_became_active":"2024-03-06T07:23:21.868739+0000","last_became_peered":"2024-03-06T07:23:21.868739+0000","last_unstale":"2024-03-06T07:23:21.871183+0000","last_undegraded":"2024-03-06T07:23:21.871183+0000","last_fullsized":"2024-03-06T07:23:21.871183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857927+0000","last_change":"2024-03-06T07:23:21.857927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857927+0000","last_undegraded":"2024-03-06T07:23:21.857927+0000","last_fullsized":"2024-03-06T07:23:21.857927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874442+0000","last_change":"2024-03-06T07:23:21.868680+0000","last_active":"2024-03-06T07:23:21.874442+0000","last_peered":"2024-03-06T07:23:21.874442+0000","last_clean":"2024-03-06T07:23:21.874442+0000","last_became_active":"2024-03-06T07:23:21.868416+0000","last_became_peered":"2024-03-06T07:23:21.868416+0000","last_unstale":"2024-03-06T07:23:21.874442+0000","last_undegraded":"2024-03-06T07:23:21.874442+0000","last_fullsized":"2024-03-06T07:23:21.874442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855516+0000","last_change":"2024-03-06T07:23:21.855516+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855516+0000","last_undegraded":"2024-03-06T07:23:21.855516+0000","last_fullsized":"2024-03-06T07:23:21.855516+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874641+0000","last_change":"2024-03-06T07:23:21.870601+0000","last_active":"2024-03-06T07:23:21.874641+0000","last_peered":"2024-03-06T07:23:21.874641+0000","last_clean":"2024-03-06T07:23:21.874641+0000","last_became_active":"2024-03-06T07:23:21.870306+0000","last_became_peered":"2024-03-06T07:23:21.870306+0000","last_unstale":"2024-03-06T07:23:21.874641+0000","last_undegraded":"2024-03-06T07:23:21.874641+0000","last_fullsized":"2024-03-06T07:23:21.874641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874729+0000","last_change":"2024-03-06T07:23:21.872671+0000","last_active":"2024-03-06T07:23:21.874729+0000","last_peered":"2024-03-06T07:23:21.874729+0000","last_clean":"2024-03-06T07:23:21.874729+0000","last_became_active":"2024-03-06T07:23:21.872330+0000","last_became_peered":"2024-03-06T07:23:21.872330+0000","last_unstale":"2024-03-06T07:23:21.874729+0000","last_undegraded":"2024-03-06T07:23:21.874729+0000","last_fullsized":"2024-03-06T07:23:21.874729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:10.134358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856748+0000","last_change":"2024-03-06T07:23:21.856748+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856748+0000","last_undegraded":"2024-03-06T07:23:21.856748+0000","last_fullsized":"2024-03-06T07:23:21.856748+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.872248+0000","last_change":"2024-03-06T07:23:21.870771+0000","last_active":"2024-03-06T07:23:21.872248+0000","last_peered":"2024-03-06T07:23:21.872248+0000","last_clean":"2024-03-06T07:23:21.872248+0000","last_became_active":"2024-03-06T07:23:21.870450+0000","last_became_peered":"2024-03-06T07:23:21.870450+0000","last_unstale":"2024-03-06T07:23:21.872248+0000","last_undegraded":"2024-03-06T07:23:21.872248+0000","last_fullsized":"2024-03-06T07:23:21.872248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859255+0000","last_change":"2024-03-06T07:23:21.859255+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859255+0000","last_undegraded":"2024-03-06T07:23:21.859255+0000","last_fullsized":"2024-03-06T07:23:21.859255+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869938+0000","last_change":"2024-03-06T07:23:21.865980+0000","last_active":"2024-03-06T07:23:21.869938+0000","last_peered":"2024-03-06T07:23:21.869938+0000","last_clean":"2024-03-06T07:23:21.869938+0000","last_became_active":"2024-03-06T07:23:21.865586+0000","last_became_peered":"2024-03-06T07:23:21.865586+0000","last_unstale":"2024-03-06T07:23:21.869938+0000","last_undegraded":"2024-03-06T07:23:21.869938+0000","last_fullsized":"2024-03-06T07:23:21.869938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861702+0000","last_change":"2024-03-06T07:23:21.861702+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861702+0000","last_undegraded":"2024-03-06T07:23:21.861702+0000","last_fullsized":"2024-03-06T07:23:21.861702+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871876+0000","last_change":"2024-03-06T07:23:21.866832+0000","last_active":"2024-03-06T07:23:21.871876+0000","last_peered":"2024-03-06T07:23:21.871876+0000","last_clean":"2024-03-06T07:23:21.871876+0000","last_became_active":"2024-03-06T07:23:21.866546+0000","last_became_peered":"2024-03-06T07:23:21.866546+0000","last_unstale":"2024-03-06T07:23:21.871876+0000","last_undegraded":"2024-03-06T07:23:21.871876+0000","last_fullsized":"2024-03-06T07:23:21.871876+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859270+0000","last_change":"2024-03-06T07:23:21.859270+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859270+0000","last_undegraded":"2024-03-06T07:23:21.859270+0000","last_fullsized":"2024-03-06T07:23:21.859270+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871931+0000","last_change":"2024-03-06T07:23:21.867500+0000","last_active":"2024-03-06T07:23:21.871931+0000","last_peered":"2024-03-06T07:23:21.871931+0000","last_clean":"2024-03-06T07:23:21.871931+0000","last_became_active":"2024-03-06T07:23:21.867223+0000","last_became_peered":"2024-03-06T07:23:21.867223+0000","last_unstale":"2024-03-06T07:23:21.871931+0000","last_undegraded":"2024-03-06T07:23:21.871931+0000","last_fullsized":"2024-03-06T07:23:21.871931+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859561+0000","last_change":"2024-03-06T07:23:21.859561+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859561+0000","last_undegraded":"2024-03-06T07:23:21.859561+0000","last_fullsized":"2024-03-06T07:23:21.859561+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869963+0000","last_change":"2024-03-06T07:23:21.866528+0000","last_active":"2024-03-06T07:23:21.869963+0000","last_peered":"2024-03-06T07:23:21.869963+0000","last_clean":"2024-03-06T07:23:21.869963+0000","last_became_active":"2024-03-06T07:23:21.866281+0000","last_became_peered":"2024-03-06T07:23:21.866281+0000","last_unstale":"2024-03-06T07:23:21.869963+0000","last_undegraded":"2024-03-06T07:23:21.869963+0000","last_fullsized":"2024-03-06T07:23:21.869963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860173+0000","last_change":"2024-03-06T07:23:21.860173+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860173+0000","last_undegraded":"2024-03-06T07:23:21.860173+0000","last_fullsized":"2024-03-06T07:23:21.860173+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874610+0000","last_change":"2024-03-06T07:23:21.870289+0000","last_active":"2024-03-06T07:23:21.874610+0000","last_peered":"2024-03-06T07:23:21.874610+0000","last_clean":"2024-03-06T07:23:21.874610+0000","last_became_active":"2024-03-06T07:23:21.870037+0000","last_became_peered":"2024-03-06T07:23:21.870037+0000","last_unstale":"2024-03-06T07:23:21.874610+0000","last_undegraded":"2024-03-06T07:23:21.874610+0000","last_fullsized":"2024-03-06T07:23:21.874610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857088+0000","last_change":"2024-03-06T07:23:21.857088+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857088+0000","last_undegraded":"2024-03-06T07:23:21.857088+0000","last_fullsized":"2024-03-06T07:23:21.857088+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871188+0000","last_change":"2024-03-06T07:23:21.869803+0000","last_active":"2024-03-06T07:23:21.871188+0000","last_peered":"2024-03-06T07:23:21.871188+0000","last_clean":"2024-03-06T07:23:21.871188+0000","last_became_active":"2024-03-06T07:23:21.869511+0000","last_became_peered":"2024-03-06T07:23:21.869511+0000","last_unstale":"2024-03-06T07:23:21.871188+0000","last_undegraded":"2024-03-06T07:23:21.871188+0000","last_fullsized":"2024-03-06T07:23:21.871188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860466+0000","last_change":"2024-03-06T07:23:21.860466+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860466+0000","last_undegraded":"2024-03-06T07:23:21.860466+0000","last_fullsized":"2024-03-06T07:23:21.860466+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874543+0000","last_change":"2024-03-06T07:23:21.871570+0000","last_active":"2024-03-06T07:23:21.874543+0000","last_peered":"2024-03-06T07:23:21.874543+0000","last_clean":"2024-03-06T07:23:21.874543+0000","last_became_active":"2024-03-06T07:23:21.871278+0000","last_became_peered":"2024-03-06T07:23:21.871278+0000","last_unstale":"2024-03-06T07:23:21.874543+0000","last_undegraded":"2024-03-06T07:23:21.874543+0000","last_fullsized":"2024-03-06T07:23:21.874543+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:00.837068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856546+0000","last_change":"2024-03-06T07:23:21.856546+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856546+0000","last_undegraded":"2024-03-06T07:23:21.856546+0000","last_fullsized":"2024-03-06T07:23:21.856546+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871218+0000","last_change":"2024-03-06T07:23:21.868700+0000","last_active":"2024-03-06T07:23:21.871218+0000","last_peered":"2024-03-06T07:23:21.871218+0000","last_clean":"2024-03-06T07:23:21.871218+0000","last_became_active":"2024-03-06T07:23:21.868258+0000","last_became_peered":"2024-03-06T07:23:21.868258+0000","last_unstale":"2024-03-06T07:23:21.871218+0000","last_undegraded":"2024-03-06T07:23:21.871218+0000","last_fullsized":"2024-03-06T07:23:21.871218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855824+0000","last_change":"2024-03-06T07:23:21.855824+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855824+0000","last_undegraded":"2024-03-06T07:23:21.855824+0000","last_fullsized":"2024-03-06T07:23:21.855824+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869070+0000","last_change":"2024-03-06T07:23:21.868016+0000","last_active":"2024-03-06T07:23:21.869070+0000","last_peered":"2024-03-06T07:23:21.869070+0000","last_clean":"2024-03-06T07:23:21.869070+0000","last_became_active":"2024-03-06T07:23:21.867724+0000","last_became_peered":"2024-03-06T07:23:21.867724+0000","last_unstale":"2024-03-06T07:23:21.869070+0000","last_undegraded":"2024-03-06T07:23:21.869070+0000","last_fullsized":"2024-03-06T07:23:21.869070+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:24.138593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857738+0000","last_change":"2024-03-06T07:23:21.857738+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857738+0000","last_undegraded":"2024-03-06T07:23:21.857738+0000","last_fullsized":"2024-03-06T07:23:21.857738+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870300+0000","last_change":"2024-03-06T07:23:21.867760+0000","last_active":"2024-03-06T07:23:21.870300+0000","last_peered":"2024-03-06T07:23:21.870300+0000","last_clean":"2024-03-06T07:23:21.870300+0000","last_became_active":"2024-03-06T07:23:21.867499+0000","last_became_peered":"2024-03-06T07:23:21.867499+0000","last_unstale":"2024-03-06T07:23:21.870300+0000","last_undegraded":"2024-03-06T07:23:21.870300+0000","last_fullsized":"2024-03-06T07:23:21.870300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863555+0000","last_change":"2024-03-06T07:23:21.863555+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863555+0000","last_undegraded":"2024-03-06T07:23:21.863555+0000","last_fullsized":"2024-03-06T07:23:21.863555+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870077+0000","last_change":"2024-03-06T07:23:21.867044+0000","last_active":"2024-03-06T07:23:21.870077+0000","last_peered":"2024-03-06T07:23:21.870077+0000","last_clean":"2024-03-06T07:23:21.870077+0000","last_became_active":"2024-03-06T07:23:21.866525+0000","last_became_peered":"2024-03-06T07:23:21.866525+0000","last_unstale":"2024-03-06T07:23:21.870077+0000","last_undegraded":"2024-03-06T07:23:21.870077+0000","last_fullsized":"2024-03-06T07:23:21.870077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853974+0000","last_change":"2024-03-06T07:23:21.853974+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853974+0000","last_undegraded":"2024-03-06T07:23:21.853974+0000","last_fullsized":"2024-03-06T07:23:21.853974+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874320+0000","last_change":"2024-03-06T07:23:21.871204+0000","last_active":"2024-03-06T07:23:21.874320+0000","last_peered":"2024-03-06T07:23:21.874320+0000","last_clean":"2024-03-06T07:23:21.874320+0000","last_became_active":"2024-03-06T07:23:21.870923+0000","last_became_peered":"2024-03-06T07:23:21.870923+0000","last_unstale":"2024-03-06T07:23:21.874320+0000","last_undegraded":"2024-03-06T07:23:21.874320+0000","last_fullsized":"2024-03-06T07:23:21.874320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.850435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858703+0000","last_change":"2024-03-06T07:23:21.858703+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858703+0000","last_undegraded":"2024-03-06T07:23:21.858703+0000","last_fullsized":"2024-03-06T07:23:21.858703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874520+0000","last_change":"2024-03-06T07:23:21.867519+0000","last_active":"2024-03-06T07:23:21.874520+0000","last_peered":"2024-03-06T07:23:21.874520+0000","last_clean":"2024-03-06T07:23:21.874520+0000","last_became_active":"2024-03-06T07:23:21.867274+0000","last_became_peered":"2024-03-06T07:23:21.867274+0000","last_unstale":"2024-03-06T07:23:21.874520+0000","last_undegraded":"2024-03-06T07:23:21.874520+0000","last_fullsized":"2024-03-06T07:23:21.874520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867006+0000","last_change":"2024-03-06T07:23:21.863945+0000","last_active":"2024-03-06T07:23:21.867006+0000","last_peered":"2024-03-06T07:23:21.867006+0000","last_clean":"2024-03-06T07:23:21.867006+0000","last_became_active":"2024-03-06T07:23:21.863699+0000","last_became_peered":"2024-03-06T07:23:21.863699+0000","last_unstale":"2024-03-06T07:23:21.867006+0000","last_undegraded":"2024-03-06T07:23:21.867006+0000","last_fullsized":"2024-03-06T07:23:21.867006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.854216+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:21.854216+0000","last_peered":"2024-03-06T07:23:21.854216+0000","last_clean":"2024-03-06T07:23:21.854216+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:21.854216+0000","last_undegraded":"2024-03-06T07:23:21.854216+0000","last_fullsized":"2024-03-06T07:23:21.854216+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:26.325 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.460+0000 7f196e75e640 1 Processor -- start 2024-03-06T07:23:26.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.460+0000 7f196e75e640 1 -- start start 2024-03-06T07:23:26.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196e75e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19680d2710 0x7f19680b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196e75e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680b3430 con 0x7f196815dfe0 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196e75e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680b3710 con 0x7f19680d22b0 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196e75e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19680b31e0 con 0x7f19680d2710 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196ccfb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19680d2710 0x7f19680b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f1967fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f196815dfe0 0x7f19680cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32842/0 (socket says 172.21.15.72:32842) 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f1967fff640 1 -- 172.21.15.72:0/1793713640 learned_addr learned my addr 172.21.15.72:0/1793713640 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196ccfb640 1 -- 172.21.15.72:0/1793713640 >> v1:172.21.15.146:6789/0 conn(0x7f19680d22b0 legacy=0x7f19680b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196ccfb640 1 -- 172.21.15.72:0/1793713640 >> v1:172.21.15.72:6789/0 conn(0x7f196815dfe0 legacy=0x7f19680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f196ccfb640 1 -- 172.21.15.72:0/1793713640 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680b39f0 con 0x7f19680d2710 2024-03-06T07:23:26.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.461+0000 7f1966ffd640 1 -- 172.21.15.72:0/1793713640 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902048079 0 0) 0x7f19680b3430 con 0x7f196815dfe0 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.462+0000 7f196ccfb640 1 --2- 172.21.15.72:0/1793713640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d2710 0x7f19680b4900 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1960009dc0 tx=0x7f196002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1e2a2a2e4cb3d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.462+0000 7f1966ffd640 1 -- 172.21.15.72:0/1793713640 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960039070 con 0x7f19680d2710 2024-03-06T07:23:26.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.462+0000 7f1966ffd640 1 -- 172.21.15.72:0/1793713640 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196002cc40 con 0x7f19680d2710 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.462+0000 7f1966ffd640 1 -- 172.21.15.72:0/1793713640 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960034440 con 0x7f19680d2710 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 -- 172.21.15.72:0/1793713640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d2710 msgr2=0x7f19680b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 --2- 172.21.15.72:0/1793713640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d2710 0x7f19680b4900 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f1960009dc0 tx=0x7f196002be90 comp rx=0 tx=0).stop 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 -- 172.21.15.72:0/1793713640 shutdown_connections 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 --2- 172.21.15.72:0/1793713640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d2710 0x7f19680b4900 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 -- 172.21.15.72:0/1793713640 >> 172.21.15.72:0/1793713640 conn(0x7f19680d3cd0 msgr2=0x7f19680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 -- 172.21.15.72:0/1793713640 shutdown_connections 2024-03-06T07:23:26.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.463+0000 7f196e75e640 1 -- 172.21.15.72:0/1793713640 wait complete. 2024-03-06T07:23:26.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.464+0000 7f196e75e640 1 Processor -- start 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.464+0000 7f196e75e640 1 -- start start 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196e75e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 0x7f196809e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196e75e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19680d2710 0x7f196815b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196e75e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f196815dfe0 0x7f19681601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196e75e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680cfa00 con 0x7f19680d22b0 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196e75e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680b3d60 con 0x7f19680d2710 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196e75e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19680d1c70 con 0x7f196815dfe0 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 0x7f196809e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 0x7f196809e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51368/0 (socket says 172.21.15.72:51368) 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 -- 172.21.15.72:0/3315169307 learned_addr learned my addr 172.21.15.72:0/3315169307 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 -- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f196815dfe0 msgr2=0x7f19681601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196ccfb640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f196815dfe0 0x7f19681601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f19677fe640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19680d2710 0x7f196815b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f196815dfe0 0x7f19681601d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 -- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19680d2710 msgr2=0x7f196815b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19680d2710 0x7f196815b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 -- 172.21.15.72:0/3315169307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968160d40 con 0x7f19680d22b0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f196ccfb640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f196815dfe0 0x7f19681601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f19677fe640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19680d2710 0x7f196815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.465+0000 7f1967fff640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 0x7f196809e670 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f195800c430 tx=0x7f195800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.466+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19580074e0 con 0x7f19680d22b0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.466+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19580047e0 con 0x7f19680d22b0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.466+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968160a60 con 0x7f19680d22b0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.466+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958004bd0 con 0x7f19680d22b0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.466+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196815e820 con 0x7f19680d22b0 2024-03-06T07:23:26.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.467+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196804f330 con 0x7f19680d22b0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.468+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1958011070 con 0x7f19680d22b0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.468+0000 7f1964ff9640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f195406b4d0 0x7f195406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.468+0000 7f19677fe640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f195406b4d0 0x7f195406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.468+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1958089e70 con 0x7f19680d22b0 2024-03-06T07:23:26.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.469+0000 7f19677fe640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f195406b4d0 0x7f195406d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f196815c250 tx=0x7f195c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.472+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1958058cf0 con 0x7f19680d22b0 2024-03-06T07:23:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.647+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f19680b4850 con 0x7f195406b4d0 2024-03-06T07:23:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.652+0000 7f1964ff9640 1 -- 172.21.15.72:0/3315169307 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246635 (secure 0 0 0) 0x7f195c054380 con 0x7f195406b4d0 2024-03-06T07:23:26.657 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:26.658 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f195406b4d0 msgr2=0x7f195406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f195406b4d0 0x7f195406d990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f196815c250 tx=0x7f195c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 msgr2=0x7f196809e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 0x7f196809e670 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f195800c430 tx=0x7f195800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 shutdown_connections 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f195406b4d0 0x7f195406d990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f196815dfe0 0x7f19681601d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19680d2710 0x7f196815b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 --2- 172.21.15.72:0/3315169307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19680d22b0 0x7f196809e670 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 >> 172.21.15.72:0/3315169307 conn(0x7f19680d3cd0 msgr2=0x7f196806ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 shutdown_connections 2024-03-06T07:23:26.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.656+0000 7f196e75e640 1 -- 172.21.15.72:0/3315169307 wait complete. 2024-03-06T07:23:26.669 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-06T07:23:25.964943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254536,"kb_used_data":5096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720184,"statfs":{"total":773094113280,"available":772833468416,"internally_reserved":0,"allocated":5218304,"data_stored":2784288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.115574"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856478+0000","last_change":"2024-03-06T07:23:21.856478+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856478+0000","last_undegraded":"2024-03-06T07:23:21.856478+0000","last_fullsized":"2024-03-06T07:23:21.856478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855000+0000","last_change":"2024-03-06T07:23:21.855000+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855000+0000","last_undegraded":"2024-03-06T07:23:21.855000+0000","last_fullsized":"2024-03-06T07:23:21.855000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860791+0000","last_change":"2024-03-06T07:23:21.860791+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860791+0000","last_undegraded":"2024-03-06T07:23:21.860791+0000","last_fullsized":"2024-03-06T07:23:21.860791+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860752+0000","last_change":"2024-03-06T07:23:21.860752+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860752+0000","last_undegraded":"2024-03-06T07:23:21.860752+0000","last_fullsized":"2024-03-06T07:23:21.860752+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853814+0000","last_change":"2024-03-06T07:23:21.853814+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853814+0000","last_undegraded":"2024-03-06T07:23:21.853814+0000","last_fullsized":"2024-03-06T07:23:21.853814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861844+0000","last_change":"2024-03-06T07:23:21.861844+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861844+0000","last_undegraded":"2024-03-06T07:23:21.861844+0000","last_fullsized":"2024-03-06T07:23:21.861844+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855942+0000","last_change":"2024-03-06T07:23:21.855942+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855942+0000","last_undegraded":"2024-03-06T07:23:21.855942+0000","last_fullsized":"2024-03-06T07:23:21.855942+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857730+0000","last_change":"2024-03-06T07:23:21.857730+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857730+0000","last_undegraded":"2024-03-06T07:23:21.857730+0000","last_fullsized":"2024-03-06T07:23:21.857730+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860536+0000","last_change":"2024-03-06T07:23:21.860536+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860536+0000","last_undegraded":"2024-03-06T07:23:21.860536+0000","last_fullsized":"2024-03-06T07:23:21.860536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860872+0000","last_change":"2024-03-06T07:23:21.860872+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860872+0000","last_undegraded":"2024-03-06T07:23:21.860872+0000","last_fullsized":"2024-03-06T07:23:21.860872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860140+0000","last_change":"2024-03-06T07:23:21.860140+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860140+0000","last_undegraded":"2024-03-06T07:23:21.860140+0000","last_fullsized":"2024-03-06T07:23:21.860140+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855745+0000","last_change":"2024-03-06T07:23:21.855745+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855745+0000","last_undegraded":"2024-03-06T07:23:21.855745+0000","last_fullsized":"2024-03-06T07:23:21.855745+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856960+0000","last_change":"2024-03-06T07:23:21.856960+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856960+0000","last_undegraded":"2024-03-06T07:23:21.856960+0000","last_fullsized":"2024-03-06T07:23:21.856960+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860354+0000","last_change":"2024-03-06T07:23:21.860354+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860354+0000","last_undegraded":"2024-03-06T07:23:21.860354+0000","last_fullsized":"2024-03-06T07:23:21.860354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862480+0000","last_change":"2024-03-06T07:23:21.862480+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862480+0000","last_undegraded":"2024-03-06T07:23:21.862480+0000","last_fullsized":"2024-03-06T07:23:21.862480+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859111+0000","last_change":"2024-03-06T07:23:21.859111+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859111+0000","last_undegraded":"2024-03-06T07:23:21.859111+0000","last_fullsized":"2024-03-06T07:23:21.859111+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861819+0000","last_change":"2024-03-06T07:23:21.861819+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861819+0000","last_undegraded":"2024-03-06T07:23:21.861819+0000","last_fullsized":"2024-03-06T07:23:21.861819+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861199+0000","last_change":"2024-03-06T07:23:21.861199+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861199+0000","last_undegraded":"2024-03-06T07:23:21.861199+0000","last_fullsized":"2024-03-06T07:23:21.861199+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860297+0000","last_change":"2024-03-06T07:23:21.860297+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860297+0000","last_undegraded":"2024-03-06T07:23:21.860297+0000","last_fullsized":"2024-03-06T07:23:21.860297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857004+0000","last_change":"2024-03-06T07:23:21.857004+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857004+0000","last_undegraded":"2024-03-06T07:23:21.857004+0000","last_fullsized":"2024-03-06T07:23:21.857004+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863958+0000","last_change":"2024-03-06T07:23:21.863958+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863958+0000","last_undegraded":"2024-03-06T07:23:21.863958+0000","last_fullsized":"2024-03-06T07:23:21.863958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856664+0000","last_change":"2024-03-06T07:23:21.856664+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856664+0000","last_undegraded":"2024-03-06T07:23:21.856664+0000","last_fullsized":"2024-03-06T07:23:21.856664+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858592+0000","last_change":"2024-03-06T07:23:21.858592+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858592+0000","last_undegraded":"2024-03-06T07:23:21.858592+0000","last_fullsized":"2024-03-06T07:23:21.858592+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858616+0000","last_change":"2024-03-06T07:23:21.858616+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858616+0000","last_undegraded":"2024-03-06T07:23:21.858616+0000","last_fullsized":"2024-03-06T07:23:21.858616+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863987+0000","last_change":"2024-03-06T07:23:21.863987+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863987+0000","last_undegraded":"2024-03-06T07:23:21.863987+0000","last_fullsized":"2024-03-06T07:23:21.863987+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860157+0000","last_change":"2024-03-06T07:23:21.860157+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860157+0000","last_undegraded":"2024-03-06T07:23:21.860157+0000","last_fullsized":"2024-03-06T07:23:21.860157+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859204+0000","last_change":"2024-03-06T07:23:21.859204+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859204+0000","last_undegraded":"2024-03-06T07:23:21.859204+0000","last_fullsized":"2024-03-06T07:23:21.859204+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854927+0000","last_change":"2024-03-06T07:23:21.854927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854927+0000","last_undegraded":"2024-03-06T07:23:21.854927+0000","last_fullsized":"2024-03-06T07:23:21.854927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862784+0000","last_change":"2024-03-06T07:23:21.862784+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862784+0000","last_undegraded":"2024-03-06T07:23:21.862784+0000","last_fullsized":"2024-03-06T07:23:21.862784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859904+0000","last_change":"2024-03-06T07:23:21.859904+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859904+0000","last_undegraded":"2024-03-06T07:23:21.859904+0000","last_fullsized":"2024-03-06T07:23:21.859904+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858223+0000","last_change":"2024-03-06T07:23:21.858223+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858223+0000","last_undegraded":"2024-03-06T07:23:21.858223+0000","last_fullsized":"2024-03-06T07:23:21.858223+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860570+0000","last_change":"2024-03-06T07:23:21.860570+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860570+0000","last_undegraded":"2024-03-06T07:23:21.860570+0000","last_fullsized":"2024-03-06T07:23:21.860570+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858941+0000","last_change":"2024-03-06T07:23:21.858941+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858941+0000","last_undegraded":"2024-03-06T07:23:21.858941+0000","last_fullsized":"2024-03-06T07:23:21.858941+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874695+0000","last_change":"2024-03-06T07:23:21.870976+0000","last_active":"2024-03-06T07:23:21.874695+0000","last_peered":"2024-03-06T07:23:21.874695+0000","last_clean":"2024-03-06T07:23:21.874695+0000","last_became_active":"2024-03-06T07:23:21.870706+0000","last_became_peered":"2024-03-06T07:23:21.870706+0000","last_unstale":"2024-03-06T07:23:21.874695+0000","last_undegraded":"2024-03-06T07:23:21.874695+0000","last_fullsized":"2024-03-06T07:23:21.874695+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860916+0000","last_change":"2024-03-06T07:23:21.860916+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860916+0000","last_undegraded":"2024-03-06T07:23:21.860916+0000","last_fullsized":"2024-03-06T07:23:21.860916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.865503+0000","last_change":"2024-03-06T07:23:21.864644+0000","last_active":"2024-03-06T07:23:21.865503+0000","last_peered":"2024-03-06T07:23:21.865503+0000","last_clean":"2024-03-06T07:23:21.865503+0000","last_became_active":"2024-03-06T07:23:21.864375+0000","last_became_peered":"2024-03-06T07:23:21.864375+0000","last_unstale":"2024-03-06T07:23:21.865503+0000","last_undegraded":"2024-03-06T07:23:21.865503+0000","last_fullsized":"2024-03-06T07:23:21.865503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854364+0000","last_change":"2024-03-06T07:23:21.854364+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854364+0000","last_undegraded":"2024-03-06T07:23:21.854364+0000","last_fullsized":"2024-03-06T07:23:21.854364+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869438+0000","last_change":"2024-03-06T07:23:21.868746+0000","last_active":"2024-03-06T07:23:21.869438+0000","last_peered":"2024-03-06T07:23:21.869438+0000","last_clean":"2024-03-06T07:23:21.869438+0000","last_became_active":"2024-03-06T07:23:21.868455+0000","last_became_peered":"2024-03-06T07:23:21.868455+0000","last_unstale":"2024-03-06T07:23:21.869438+0000","last_undegraded":"2024-03-06T07:23:21.869438+0000","last_fullsized":"2024-03-06T07:23:21.869438+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:44.858008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860303+0000","last_change":"2024-03-06T07:23:21.860303+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860303+0000","last_undegraded":"2024-03-06T07:23:21.860303+0000","last_fullsized":"2024-03-06T07:23:21.860303+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870280+0000","last_change":"2024-03-06T07:23:21.865942+0000","last_active":"2024-03-06T07:23:21.870280+0000","last_peered":"2024-03-06T07:23:21.870280+0000","last_clean":"2024-03-06T07:23:21.870280+0000","last_became_active":"2024-03-06T07:23:21.865642+0000","last_became_peered":"2024-03-06T07:23:21.865642+0000","last_unstale":"2024-03-06T07:23:21.870280+0000","last_undegraded":"2024-03-06T07:23:21.870280+0000","last_fullsized":"2024-03-06T07:23:21.870280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859164+0000","last_change":"2024-03-06T07:23:21.859164+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859164+0000","last_undegraded":"2024-03-06T07:23:21.859164+0000","last_fullsized":"2024-03-06T07:23:21.859164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874913+0000","last_change":"2024-03-06T07:23:21.872744+0000","last_active":"2024-03-06T07:23:21.874913+0000","last_peered":"2024-03-06T07:23:21.874913+0000","last_clean":"2024-03-06T07:23:21.874913+0000","last_became_active":"2024-03-06T07:23:21.872444+0000","last_became_peered":"2024-03-06T07:23:21.872444+0000","last_unstale":"2024-03-06T07:23:21.874913+0000","last_undegraded":"2024-03-06T07:23:21.874913+0000","last_fullsized":"2024-03-06T07:23:21.874913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:43.054124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857762+0000","last_change":"2024-03-06T07:23:21.857762+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857762+0000","last_undegraded":"2024-03-06T07:23:21.857762+0000","last_fullsized":"2024-03-06T07:23:21.857762+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870098+0000","last_change":"2024-03-06T07:23:21.868824+0000","last_active":"2024-03-06T07:23:21.870098+0000","last_peered":"2024-03-06T07:23:21.870098+0000","last_clean":"2024-03-06T07:23:21.870098+0000","last_became_active":"2024-03-06T07:23:21.867672+0000","last_became_peered":"2024-03-06T07:23:21.867672+0000","last_unstale":"2024-03-06T07:23:21.870098+0000","last_undegraded":"2024-03-06T07:23:21.870098+0000","last_fullsized":"2024-03-06T07:23:21.870098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857267+0000","last_change":"2024-03-06T07:23:21.857267+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857267+0000","last_undegraded":"2024-03-06T07:23:21.857267+0000","last_fullsized":"2024-03-06T07:23:21.857267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866889+0000","last_change":"2024-03-06T07:23:21.863472+0000","last_active":"2024-03-06T07:23:21.866889+0000","last_peered":"2024-03-06T07:23:21.866889+0000","last_clean":"2024-03-06T07:23:21.866889+0000","last_became_active":"2024-03-06T07:23:21.863161+0000","last_became_peered":"2024-03-06T07:23:21.863161+0000","last_unstale":"2024-03-06T07:23:21.866889+0000","last_undegraded":"2024-03-06T07:23:21.866889+0000","last_fullsized":"2024-03-06T07:23:21.866889+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:42.354236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.865694+0000","last_change":"2024-03-06T07:23:21.865694+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.865694+0000","last_undegraded":"2024-03-06T07:23:21.865694+0000","last_fullsized":"2024-03-06T07:23:21.865694+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867714+0000","last_change":"2024-03-06T07:23:21.864598+0000","last_active":"2024-03-06T07:23:21.867714+0000","last_peered":"2024-03-06T07:23:21.867714+0000","last_clean":"2024-03-06T07:23:21.867714+0000","last_became_active":"2024-03-06T07:23:21.862773+0000","last_became_peered":"2024-03-06T07:23:21.862773+0000","last_unstale":"2024-03-06T07:23:21.867714+0000","last_undegraded":"2024-03-06T07:23:21.867714+0000","last_fullsized":"2024-03-06T07:23:21.867714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860119+0000","last_change":"2024-03-06T07:23:21.860119+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860119+0000","last_undegraded":"2024-03-06T07:23:21.860119+0000","last_fullsized":"2024-03-06T07:23:21.860119+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870825+0000","last_change":"2024-03-06T07:23:21.867046+0000","last_active":"2024-03-06T07:23:21.870825+0000","last_peered":"2024-03-06T07:23:21.870825+0000","last_clean":"2024-03-06T07:23:21.870825+0000","last_became_active":"2024-03-06T07:23:21.866751+0000","last_became_peered":"2024-03-06T07:23:21.866751+0000","last_unstale":"2024-03-06T07:23:21.870825+0000","last_undegraded":"2024-03-06T07:23:21.870825+0000","last_fullsized":"2024-03-06T07:23:21.870825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854878+0000","last_change":"2024-03-06T07:23:21.854878+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854878+0000","last_undegraded":"2024-03-06T07:23:21.854878+0000","last_fullsized":"2024-03-06T07:23:21.854878+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874650+0000","last_change":"2024-03-06T07:23:21.869633+0000","last_active":"2024-03-06T07:23:21.874650+0000","last_peered":"2024-03-06T07:23:21.874650+0000","last_clean":"2024-03-06T07:23:21.874650+0000","last_became_active":"2024-03-06T07:23:21.869349+0000","last_became_peered":"2024-03-06T07:23:21.869349+0000","last_unstale":"2024-03-06T07:23:21.874650+0000","last_undegraded":"2024-03-06T07:23:21.874650+0000","last_fullsized":"2024-03-06T07:23:21.874650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857576+0000","last_change":"2024-03-06T07:23:21.857576+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857576+0000","last_undegraded":"2024-03-06T07:23:21.857576+0000","last_fullsized":"2024-03-06T07:23:21.857576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871236+0000","last_change":"2024-03-06T07:23:21.868933+0000","last_active":"2024-03-06T07:23:21.871236+0000","last_peered":"2024-03-06T07:23:21.871236+0000","last_clean":"2024-03-06T07:23:21.871236+0000","last_became_active":"2024-03-06T07:23:21.868305+0000","last_became_peered":"2024-03-06T07:23:21.868305+0000","last_unstale":"2024-03-06T07:23:21.871236+0000","last_undegraded":"2024-03-06T07:23:21.871236+0000","last_fullsized":"2024-03-06T07:23:21.871236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856248+0000","last_change":"2024-03-06T07:23:21.856248+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856248+0000","last_undegraded":"2024-03-06T07:23:21.856248+0000","last_fullsized":"2024-03-06T07:23:21.856248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866259+0000","last_change":"2024-03-06T07:23:21.862722+0000","last_active":"2024-03-06T07:23:21.866259+0000","last_peered":"2024-03-06T07:23:21.866259+0000","last_clean":"2024-03-06T07:23:21.866259+0000","last_became_active":"2024-03-06T07:23:21.862470+0000","last_became_peered":"2024-03-06T07:23:21.862470+0000","last_unstale":"2024-03-06T07:23:21.866259+0000","last_undegraded":"2024-03-06T07:23:21.866259+0000","last_fullsized":"2024-03-06T07:23:21.866259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:13.201036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860147+0000","last_change":"2024-03-06T07:23:21.860147+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860147+0000","last_undegraded":"2024-03-06T07:23:21.860147+0000","last_fullsized":"2024-03-06T07:23:21.860147+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.877196+0000","last_change":"2024-03-06T07:23:21.875315+0000","last_active":"2024-03-06T07:23:21.877196+0000","last_peered":"2024-03-06T07:23:21.877196+0000","last_clean":"2024-03-06T07:23:21.877196+0000","last_became_active":"2024-03-06T07:23:21.874938+0000","last_became_peered":"2024-03-06T07:23:21.874938+0000","last_unstale":"2024-03-06T07:23:21.877196+0000","last_undegraded":"2024-03-06T07:23:21.877196+0000","last_fullsized":"2024-03-06T07:23:21.877196+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.194391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857385+0000","last_change":"2024-03-06T07:23:21.857385+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857385+0000","last_undegraded":"2024-03-06T07:23:21.857385+0000","last_fullsized":"2024-03-06T07:23:21.857385+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871183+0000","last_change":"2024-03-06T07:23:21.869035+0000","last_active":"2024-03-06T07:23:21.871183+0000","last_peered":"2024-03-06T07:23:21.871183+0000","last_clean":"2024-03-06T07:23:21.871183+0000","last_became_active":"2024-03-06T07:23:21.868739+0000","last_became_peered":"2024-03-06T07:23:21.868739+0000","last_unstale":"2024-03-06T07:23:21.871183+0000","last_undegraded":"2024-03-06T07:23:21.871183+0000","last_fullsized":"2024-03-06T07:23:21.871183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857927+0000","last_change":"2024-03-06T07:23:21.857927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857927+0000","last_undegraded":"2024-03-06T07:23:21.857927+0000","last_fullsized":"2024-03-06T07:23:21.857927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874442+0000","last_change":"2024-03-06T07:23:21.868680+0000","last_active":"2024-03-06T07:23:21.874442+0000","last_peered":"2024-03-06T07:23:21.874442+0000","last_clean":"2024-03-06T07:23:21.874442+0000","last_became_active":"2024-03-06T07:23:21.868416+0000","last_became_peered":"2024-03-06T07:23:21.868416+0000","last_unstale":"2024-03-06T07:23:21.874442+0000","last_undegraded":"2024-03-06T07:23:21.874442+0000","last_fullsized":"2024-03-06T07:23:21.874442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855516+0000","last_change":"2024-03-06T07:23:21.855516+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855516+0000","last_undegraded":"2024-03-06T07:23:21.855516+0000","last_fullsized":"2024-03-06T07:23:21.855516+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874641+0000","last_change":"2024-03-06T07:23:21.870601+0000","last_active":"2024-03-06T07:23:21.874641+0000","last_peered":"2024-03-06T07:23:21.874641+0000","last_clean":"2024-03-06T07:23:21.874641+0000","last_became_active":"2024-03-06T07:23:21.870306+0000","last_became_peered":"2024-03-06T07:23:21.870306+0000","last_unstale":"2024-03-06T07:23:21.874641+0000","last_undegraded":"2024-03-06T07:23:21.874641+0000","last_fullsized":"2024-03-06T07:23:21.874641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874729+0000","last_change":"2024-03-06T07:23:21.872671+0000","last_active":"2024-03-06T07:23:21.874729+0000","last_peered":"2024-03-06T07:23:21.874729+0000","last_clean":"2024-03-06T07:23:21.874729+0000","last_became_active":"2024-03-06T07:23:21.872330+0000","last_became_peered":"2024-03-06T07:23:21.872330+0000","last_unstale":"2024-03-06T07:23:21.874729+0000","last_undegraded":"2024-03-06T07:23:21.874729+0000","last_fullsized":"2024-03-06T07:23:21.874729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:10.134358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856748+0000","last_change":"2024-03-06T07:23:21.856748+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856748+0000","last_undegraded":"2024-03-06T07:23:21.856748+0000","last_fullsized":"2024-03-06T07:23:21.856748+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.872248+0000","last_change":"2024-03-06T07:23:21.870771+0000","last_active":"2024-03-06T07:23:21.872248+0000","last_peered":"2024-03-06T07:23:21.872248+0000","last_clean":"2024-03-06T07:23:21.872248+0000","last_became_active":"2024-03-06T07:23:21.870450+0000","last_became_peered":"2024-03-06T07:23:21.870450+0000","last_unstale":"2024-03-06T07:23:21.872248+0000","last_undegraded":"2024-03-06T07:23:21.872248+0000","last_fullsized":"2024-03-06T07:23:21.872248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859255+0000","last_change":"2024-03-06T07:23:21.859255+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859255+0000","last_undegraded":"2024-03-06T07:23:21.859255+0000","last_fullsized":"2024-03-06T07:23:21.859255+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869938+0000","last_change":"2024-03-06T07:23:21.865980+0000","last_active":"2024-03-06T07:23:21.869938+0000","last_peered":"2024-03-06T07:23:21.869938+0000","last_clean":"2024-03-06T07:23:21.869938+0000","last_became_active":"2024-03-06T07:23:21.865586+0000","last_became_peered":"2024-03-06T07:23:21.865586+0000","last_unstale":"2024-03-06T07:23:21.869938+0000","last_undegraded":"2024-03-06T07:23:21.869938+0000","last_fullsized":"2024-03-06T07:23:21.869938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861702+0000","last_change":"2024-03-06T07:23:21.861702+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861702+0000","last_undegraded":"2024-03-06T07:23:21.861702+0000","last_fullsized":"2024-03-06T07:23:21.861702+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871876+0000","last_change":"2024-03-06T07:23:21.866832+0000","last_active":"2024-03-06T07:23:21.871876+0000","last_peered":"2024-03-06T07:23:21.871876+0000","last_clean":"2024-03-06T07:23:21.871876+0000","last_became_active":"2024-03-06T07:23:21.866546+0000","last_became_peered":"2024-03-06T07:23:21.866546+0000","last_unstale":"2024-03-06T07:23:21.871876+0000","last_undegraded":"2024-03-06T07:23:21.871876+0000","last_fullsized":"2024-03-06T07:23:21.871876+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859270+0000","last_change":"2024-03-06T07:23:21.859270+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859270+0000","last_undegraded":"2024-03-06T07:23:21.859270+0000","last_fullsized":"2024-03-06T07:23:21.859270+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871931+0000","last_change":"2024-03-06T07:23:21.867500+0000","last_active":"2024-03-06T07:23:21.871931+0000","last_peered":"2024-03-06T07:23:21.871931+0000","last_clean":"2024-03-06T07:23:21.871931+0000","last_became_active":"2024-03-06T07:23:21.867223+0000","last_became_peered":"2024-03-06T07:23:21.867223+0000","last_unstale":"2024-03-06T07:23:21.871931+0000","last_undegraded":"2024-03-06T07:23:21.871931+0000","last_fullsized":"2024-03-06T07:23:21.871931+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859561+0000","last_change":"2024-03-06T07:23:21.859561+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859561+0000","last_undegraded":"2024-03-06T07:23:21.859561+0000","last_fullsized":"2024-03-06T07:23:21.859561+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869963+0000","last_change":"2024-03-06T07:23:21.866528+0000","last_active":"2024-03-06T07:23:21.869963+0000","last_peered":"2024-03-06T07:23:21.869963+0000","last_clean":"2024-03-06T07:23:21.869963+0000","last_became_active":"2024-03-06T07:23:21.866281+0000","last_became_peered":"2024-03-06T07:23:21.866281+0000","last_unstale":"2024-03-06T07:23:21.869963+0000","last_undegraded":"2024-03-06T07:23:21.869963+0000","last_fullsized":"2024-03-06T07:23:21.869963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860173+0000","last_change":"2024-03-06T07:23:21.860173+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860173+0000","last_undegraded":"2024-03-06T07:23:21.860173+0000","last_fullsized":"2024-03-06T07:23:21.860173+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874610+0000","last_change":"2024-03-06T07:23:21.870289+0000","last_active":"2024-03-06T07:23:21.874610+0000","last_peered":"2024-03-06T07:23:21.874610+0000","last_clean":"2024-03-06T07:23:21.874610+0000","last_became_active":"2024-03-06T07:23:21.870037+0000","last_became_peered":"2024-03-06T07:23:21.870037+0000","last_unstale":"2024-03-06T07:23:21.874610+0000","last_undegraded":"2024-03-06T07:23:21.874610+0000","last_fullsized":"2024-03-06T07:23:21.874610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857088+0000","last_change":"2024-03-06T07:23:21.857088+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857088+0000","last_undegraded":"2024-03-06T07:23:21.857088+0000","last_fullsized":"2024-03-06T07:23:21.857088+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871188+0000","last_change":"2024-03-06T07:23:21.869803+0000","last_active":"2024-03-06T07:23:21.871188+0000","last_peered":"2024-03-06T07:23:21.871188+0000","last_clean":"2024-03-06T07:23:21.871188+0000","last_became_active":"2024-03-06T07:23:21.869511+0000","last_became_peered":"2024-03-06T07:23:21.869511+0000","last_unstale":"2024-03-06T07:23:21.871188+0000","last_undegraded":"2024-03-06T07:23:21.871188+0000","last_fullsized":"2024-03-06T07:23:21.871188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860466+0000","last_change":"2024-03-06T07:23:21.860466+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860466+0000","last_undegraded":"2024-03-06T07:23:21.860466+0000","last_fullsized":"2024-03-06T07:23:21.860466+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874543+0000","last_change":"2024-03-06T07:23:21.871570+0000","last_active":"2024-03-06T07:23:21.874543+0000","last_peered":"2024-03-06T07:23:21.874543+0000","last_clean":"2024-03-06T07:23:21.874543+0000","last_became_active":"2024-03-06T07:23:21.871278+0000","last_became_peered":"2024-03-06T07:23:21.871278+0000","last_unstale":"2024-03-06T07:23:21.874543+0000","last_undegraded":"2024-03-06T07:23:21.874543+0000","last_fullsized":"2024-03-06T07:23:21.874543+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:00.837068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856546+0000","last_change":"2024-03-06T07:23:21.856546+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856546+0000","last_undegraded":"2024-03-06T07:23:21.856546+0000","last_fullsized":"2024-03-06T07:23:21.856546+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871218+0000","last_change":"2024-03-06T07:23:21.868700+0000","last_active":"2024-03-06T07:23:21.871218+0000","last_peered":"2024-03-06T07:23:21.871218+0000","last_clean":"2024-03-06T07:23:21.871218+0000","last_became_active":"2024-03-06T07:23:21.868258+0000","last_became_peered":"2024-03-06T07:23:21.868258+0000","last_unstale":"2024-03-06T07:23:21.871218+0000","last_undegraded":"2024-03-06T07:23:21.871218+0000","last_fullsized":"2024-03-06T07:23:21.871218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855824+0000","last_change":"2024-03-06T07:23:21.855824+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855824+0000","last_undegraded":"2024-03-06T07:23:21.855824+0000","last_fullsized":"2024-03-06T07:23:21.855824+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869070+0000","last_change":"2024-03-06T07:23:21.868016+0000","last_active":"2024-03-06T07:23:21.869070+0000","last_peered":"2024-03-06T07:23:21.869070+0000","last_clean":"2024-03-06T07:23:21.869070+0000","last_became_active":"2024-03-06T07:23:21.867724+0000","last_became_peered":"2024-03-06T07:23:21.867724+0000","last_unstale":"2024-03-06T07:23:21.869070+0000","last_undegraded":"2024-03-06T07:23:21.869070+0000","last_fullsized":"2024-03-06T07:23:21.869070+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:24.138593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857738+0000","last_change":"2024-03-06T07:23:21.857738+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857738+0000","last_undegraded":"2024-03-06T07:23:21.857738+0000","last_fullsized":"2024-03-06T07:23:21.857738+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870300+0000","last_change":"2024-03-06T07:23:21.867760+0000","last_active":"2024-03-06T07:23:21.870300+0000","last_peered":"2024-03-06T07:23:21.870300+0000","last_clean":"2024-03-06T07:23:21.870300+0000","last_became_active":"2024-03-06T07:23:21.867499+0000","last_became_peered":"2024-03-06T07:23:21.867499+0000","last_unstale":"2024-03-06T07:23:21.870300+0000","last_undegraded":"2024-03-06T07:23:21.870300+0000","last_fullsized":"2024-03-06T07:23:21.870300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863555+0000","last_change":"2024-03-06T07:23:21.863555+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863555+0000","last_undegraded":"2024-03-06T07:23:21.863555+0000","last_fullsized":"2024-03-06T07:23:21.863555+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870077+0000","last_change":"2024-03-06T07:23:21.867044+0000","last_active":"2024-03-06T07:23:21.870077+0000","last_peered":"2024-03-06T07:23:21.870077+0000","last_clean":"2024-03-06T07:23:21.870077+0000","last_became_active":"2024-03-06T07:23:21.866525+0000","last_became_peered":"2024-03-06T07:23:21.866525+0000","last_unstale":"2024-03-06T07:23:21.870077+0000","last_undegraded":"2024-03-06T07:23:21.870077+0000","last_fullsized":"2024-03-06T07:23:21.870077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853974+0000","last_change":"2024-03-06T07:23:21.853974+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853974+0000","last_undegraded":"2024-03-06T07:23:21.853974+0000","last_fullsized":"2024-03-06T07:23:21.853974+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874320+0000","last_change":"2024-03-06T07:23:21.871204+0000","last_active":"2024-03-06T07:23:21.874320+0000","last_peered":"2024-03-06T07:23:21.874320+0000","last_clean":"2024-03-06T07:23:21.874320+0000","last_became_active":"2024-03-06T07:23:21.870923+0000","last_became_peered":"2024-03-06T07:23:21.870923+0000","last_unstale":"2024-03-06T07:23:21.874320+0000","last_undegraded":"2024-03-06T07:23:21.874320+0000","last_fullsized":"2024-03-06T07:23:21.874320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.850435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858703+0000","last_change":"2024-03-06T07:23:21.858703+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858703+0000","last_undegraded":"2024-03-06T07:23:21.858703+0000","last_fullsized":"2024-03-06T07:23:21.858703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874520+0000","last_change":"2024-03-06T07:23:21.867519+0000","last_active":"2024-03-06T07:23:21.874520+0000","last_peered":"2024-03-06T07:23:21.874520+0000","last_clean":"2024-03-06T07:23:21.874520+0000","last_became_active":"2024-03-06T07:23:21.867274+0000","last_became_peered":"2024-03-06T07:23:21.867274+0000","last_unstale":"2024-03-06T07:23:21.874520+0000","last_undegraded":"2024-03-06T07:23:21.874520+0000","last_fullsized":"2024-03-06T07:23:21.874520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867006+0000","last_change":"2024-03-06T07:23:21.863945+0000","last_active":"2024-03-06T07:23:21.867006+0000","last_peered":"2024-03-06T07:23:21.867006+0000","last_clean":"2024-03-06T07:23:21.867006+0000","last_became_active":"2024-03-06T07:23:21.863699+0000","last_became_peered":"2024-03-06T07:23:21.863699+0000","last_unstale":"2024-03-06T07:23:21.867006+0000","last_undegraded":"2024-03-06T07:23:21.867006+0000","last_fullsized":"2024-03-06T07:23:21.867006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.854216+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:21.854216+0000","last_peered":"2024-03-06T07:23:21.854216+0000","last_clean":"2024-03-06T07:23:21.854216+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:21.854216+0000","last_undegraded":"2024-03-06T07:23:21.854216+0000","last_fullsized":"2024-03-06T07:23:21.854216+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3f is not active+clean 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856478+0000', 'last_change': '2024-03-06T07:23:21.856478+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856478+0000', 'last_undegraded': '2024-03-06T07:23:21.856478+0000', 'last_fullsized': '2024-03-06T07:23:21.856478+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3e is not active+clean 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.855000+0000', 'last_change': '2024-03-06T07:23:21.855000+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.855000+0000', 'last_undegraded': '2024-03-06T07:23:21.855000+0000', 'last_fullsized': '2024-03-06T07:23:21.855000+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3d is not active+clean 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860791+0000', 'last_change': '2024-03-06T07:23:21.860791+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860791+0000', 'last_undegraded': '2024-03-06T07:23:21.860791+0000', 'last_fullsized': '2024-03-06T07:23:21.860791+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3c is not active+clean 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860752+0000', 'last_change': '2024-03-06T07:23:21.860752+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860752+0000', 'last_undegraded': '2024-03-06T07:23:21.860752+0000', 'last_fullsized': '2024-03-06T07:23:21.860752+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3b is not active+clean 2024-03-06T07:23:26.673 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.853814+0000', 'last_change': '2024-03-06T07:23:21.853814+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.853814+0000', 'last_undegraded': '2024-03-06T07:23:21.853814+0000', 'last_fullsized': '2024-03-06T07:23:21.853814+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3a is not active+clean 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.861844+0000', 'last_change': '2024-03-06T07:23:21.861844+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.861844+0000', 'last_undegraded': '2024-03-06T07:23:21.861844+0000', 'last_fullsized': '2024-03-06T07:23:21.861844+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.39 is not active+clean 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.855942+0000', 'last_change': '2024-03-06T07:23:21.855942+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.855942+0000', 'last_undegraded': '2024-03-06T07:23:21.855942+0000', 'last_fullsized': '2024-03-06T07:23:21.855942+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.38 is not active+clean 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857730+0000', 'last_change': '2024-03-06T07:23:21.857730+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857730+0000', 'last_undegraded': '2024-03-06T07:23:21.857730+0000', 'last_fullsized': '2024-03-06T07:23:21.857730+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.37 is not active+clean 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860536+0000', 'last_change': '2024-03-06T07:23:21.860536+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860536+0000', 'last_undegraded': '2024-03-06T07:23:21.860536+0000', 'last_fullsized': '2024-03-06T07:23:21.860536+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.36 is not active+clean 2024-03-06T07:23:26.674 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860872+0000', 'last_change': '2024-03-06T07:23:21.860872+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860872+0000', 'last_undegraded': '2024-03-06T07:23:21.860872+0000', 'last_fullsized': '2024-03-06T07:23:21.860872+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.35 is not active+clean 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860140+0000', 'last_change': '2024-03-06T07:23:21.860140+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860140+0000', 'last_undegraded': '2024-03-06T07:23:21.860140+0000', 'last_fullsized': '2024-03-06T07:23:21.860140+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.34 is not active+clean 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.855745+0000', 'last_change': '2024-03-06T07:23:21.855745+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.855745+0000', 'last_undegraded': '2024-03-06T07:23:21.855745+0000', 'last_fullsized': '2024-03-06T07:23:21.855745+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.33 is not active+clean 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856960+0000', 'last_change': '2024-03-06T07:23:21.856960+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856960+0000', 'last_undegraded': '2024-03-06T07:23:21.856960+0000', 'last_fullsized': '2024-03-06T07:23:21.856960+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.32 is not active+clean 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860354+0000', 'last_change': '2024-03-06T07:23:21.860354+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860354+0000', 'last_undegraded': '2024-03-06T07:23:21.860354+0000', 'last_fullsized': '2024-03-06T07:23:21.860354+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.31 is not active+clean 2024-03-06T07:23:26.675 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.862480+0000', 'last_change': '2024-03-06T07:23:21.862480+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.862480+0000', 'last_undegraded': '2024-03-06T07:23:21.862480+0000', 'last_fullsized': '2024-03-06T07:23:21.862480+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.30 is not active+clean 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859111+0000', 'last_change': '2024-03-06T07:23:21.859111+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859111+0000', 'last_undegraded': '2024-03-06T07:23:21.859111+0000', 'last_fullsized': '2024-03-06T07:23:21.859111+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2f is not active+clean 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.861819+0000', 'last_change': '2024-03-06T07:23:21.861819+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.861819+0000', 'last_undegraded': '2024-03-06T07:23:21.861819+0000', 'last_fullsized': '2024-03-06T07:23:21.861819+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2e is not active+clean 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.861199+0000', 'last_change': '2024-03-06T07:23:21.861199+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.861199+0000', 'last_undegraded': '2024-03-06T07:23:21.861199+0000', 'last_fullsized': '2024-03-06T07:23:21.861199+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2d is not active+clean 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860297+0000', 'last_change': '2024-03-06T07:23:21.860297+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860297+0000', 'last_undegraded': '2024-03-06T07:23:21.860297+0000', 'last_fullsized': '2024-03-06T07:23:21.860297+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2c is not active+clean 2024-03-06T07:23:26.676 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857004+0000', 'last_change': '2024-03-06T07:23:21.857004+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857004+0000', 'last_undegraded': '2024-03-06T07:23:21.857004+0000', 'last_fullsized': '2024-03-06T07:23:21.857004+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2b is not active+clean 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.863958+0000', 'last_change': '2024-03-06T07:23:21.863958+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.863958+0000', 'last_undegraded': '2024-03-06T07:23:21.863958+0000', 'last_fullsized': '2024-03-06T07:23:21.863958+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2a is not active+clean 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856664+0000', 'last_change': '2024-03-06T07:23:21.856664+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856664+0000', 'last_undegraded': '2024-03-06T07:23:21.856664+0000', 'last_fullsized': '2024-03-06T07:23:21.856664+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.29 is not active+clean 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.858592+0000', 'last_change': '2024-03-06T07:23:21.858592+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.858592+0000', 'last_undegraded': '2024-03-06T07:23:21.858592+0000', 'last_fullsized': '2024-03-06T07:23:21.858592+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.28 is not active+clean 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.858616+0000', 'last_change': '2024-03-06T07:23:21.858616+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.858616+0000', 'last_undegraded': '2024-03-06T07:23:21.858616+0000', 'last_fullsized': '2024-03-06T07:23:21.858616+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.27 is not active+clean 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.863987+0000', 'last_change': '2024-03-06T07:23:21.863987+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.863987+0000', 'last_undegraded': '2024-03-06T07:23:21.863987+0000', 'last_fullsized': '2024-03-06T07:23:21.863987+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.677 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.26 is not active+clean 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860157+0000', 'last_change': '2024-03-06T07:23:21.860157+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860157+0000', 'last_undegraded': '2024-03-06T07:23:21.860157+0000', 'last_fullsized': '2024-03-06T07:23:21.860157+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.25 is not active+clean 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859204+0000', 'last_change': '2024-03-06T07:23:21.859204+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859204+0000', 'last_undegraded': '2024-03-06T07:23:21.859204+0000', 'last_fullsized': '2024-03-06T07:23:21.859204+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.24 is not active+clean 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.854927+0000', 'last_change': '2024-03-06T07:23:21.854927+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.854927+0000', 'last_undegraded': '2024-03-06T07:23:21.854927+0000', 'last_fullsized': '2024-03-06T07:23:21.854927+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.23 is not active+clean 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.862784+0000', 'last_change': '2024-03-06T07:23:21.862784+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.862784+0000', 'last_undegraded': '2024-03-06T07:23:21.862784+0000', 'last_fullsized': '2024-03-06T07:23:21.862784+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.22 is not active+clean 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859904+0000', 'last_change': '2024-03-06T07:23:21.859904+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859904+0000', 'last_undegraded': '2024-03-06T07:23:21.859904+0000', 'last_fullsized': '2024-03-06T07:23:21.859904+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.21 is not active+clean 2024-03-06T07:23:26.678 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.858223+0000', 'last_change': '2024-03-06T07:23:21.858223+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.858223+0000', 'last_undegraded': '2024-03-06T07:23:21.858223+0000', 'last_fullsized': '2024-03-06T07:23:21.858223+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.20 is not active+clean 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860570+0000', 'last_change': '2024-03-06T07:23:21.860570+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860570+0000', 'last_undegraded': '2024-03-06T07:23:21.860570+0000', 'last_fullsized': '2024-03-06T07:23:21.860570+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1e is not active+clean 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.858941+0000', 'last_change': '2024-03-06T07:23:21.858941+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.858941+0000', 'last_undegraded': '2024-03-06T07:23:21.858941+0000', 'last_fullsized': '2024-03-06T07:23:21.858941+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1f is not active+clean 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860916+0000', 'last_change': '2024-03-06T07:23:21.860916+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860916+0000', 'last_undegraded': '2024-03-06T07:23:21.860916+0000', 'last_fullsized': '2024-03-06T07:23:21.860916+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1c is not active+clean 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.854364+0000', 'last_change': '2024-03-06T07:23:21.854364+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.854364+0000', 'last_undegraded': '2024-03-06T07:23:21.854364+0000', 'last_fullsized': '2024-03-06T07:23:21.854364+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1d is not active+clean 2024-03-06T07:23:26.679 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860303+0000', 'last_change': '2024-03-06T07:23:21.860303+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860303+0000', 'last_undegraded': '2024-03-06T07:23:21.860303+0000', 'last_fullsized': '2024-03-06T07:23:21.860303+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1a is not active+clean 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859164+0000', 'last_change': '2024-03-06T07:23:21.859164+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859164+0000', 'last_undegraded': '2024-03-06T07:23:21.859164+0000', 'last_fullsized': '2024-03-06T07:23:21.859164+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1b is not active+clean 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857762+0000', 'last_change': '2024-03-06T07:23:21.857762+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857762+0000', 'last_undegraded': '2024-03-06T07:23:21.857762+0000', 'last_fullsized': '2024-03-06T07:23:21.857762+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.18 is not active+clean 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857267+0000', 'last_change': '2024-03-06T07:23:21.857267+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857267+0000', 'last_undegraded': '2024-03-06T07:23:21.857267+0000', 'last_fullsized': '2024-03-06T07:23:21.857267+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.19 is not active+clean 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.865694+0000', 'last_change': '2024-03-06T07:23:21.865694+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.865694+0000', 'last_undegraded': '2024-03-06T07:23:21.865694+0000', 'last_fullsized': '2024-03-06T07:23:21.865694+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.16 is not active+clean 2024-03-06T07:23:26.680 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860119+0000', 'last_change': '2024-03-06T07:23:21.860119+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860119+0000', 'last_undegraded': '2024-03-06T07:23:21.860119+0000', 'last_fullsized': '2024-03-06T07:23:21.860119+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.17 is not active+clean 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.854878+0000', 'last_change': '2024-03-06T07:23:21.854878+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.854878+0000', 'last_undegraded': '2024-03-06T07:23:21.854878+0000', 'last_fullsized': '2024-03-06T07:23:21.854878+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.14 is not active+clean 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857576+0000', 'last_change': '2024-03-06T07:23:21.857576+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857576+0000', 'last_undegraded': '2024-03-06T07:23:21.857576+0000', 'last_fullsized': '2024-03-06T07:23:21.857576+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.15 is not active+clean 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856248+0000', 'last_change': '2024-03-06T07:23:21.856248+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856248+0000', 'last_undegraded': '2024-03-06T07:23:21.856248+0000', 'last_fullsized': '2024-03-06T07:23:21.856248+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.12 is not active+clean 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860147+0000', 'last_change': '2024-03-06T07:23:21.860147+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860147+0000', 'last_undegraded': '2024-03-06T07:23:21.860147+0000', 'last_fullsized': '2024-03-06T07:23:21.860147+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.13 is not active+clean 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857385+0000', 'last_change': '2024-03-06T07:23:21.857385+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857385+0000', 'last_undegraded': '2024-03-06T07:23:21.857385+0000', 'last_fullsized': '2024-03-06T07:23:21.857385+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.10 is not active+clean 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857927+0000', 'last_change': '2024-03-06T07:23:21.857927+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857927+0000', 'last_undegraded': '2024-03-06T07:23:21.857927+0000', 'last_fullsized': '2024-03-06T07:23:21.857927+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.11 is not active+clean 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.855516+0000', 'last_change': '2024-03-06T07:23:21.855516+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.855516+0000', 'last_undegraded': '2024-03-06T07:23:21.855516+0000', 'last_fullsized': '2024-03-06T07:23:21.855516+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.e is not active+clean 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856519+0000', 'last_change': '2024-03-06T07:23:21.856519+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856519+0000', 'last_undegraded': '2024-03-06T07:23:21.856519+0000', 'last_fullsized': '2024-03-06T07:23:21.856519+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.f is not active+clean 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856748+0000', 'last_change': '2024-03-06T07:23:21.856748+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856748+0000', 'last_undegraded': '2024-03-06T07:23:21.856748+0000', 'last_fullsized': '2024-03-06T07:23:21.856748+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.c is not active+clean 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859255+0000', 'last_change': '2024-03-06T07:23:21.859255+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859255+0000', 'last_undegraded': '2024-03-06T07:23:21.859255+0000', 'last_fullsized': '2024-03-06T07:23:21.859255+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.d is not active+clean 2024-03-06T07:23:26.682 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.861702+0000', 'last_change': '2024-03-06T07:23:21.861702+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.861702+0000', 'last_undegraded': '2024-03-06T07:23:21.861702+0000', 'last_fullsized': '2024-03-06T07:23:21.861702+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.a is not active+clean 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859270+0000', 'last_change': '2024-03-06T07:23:21.859270+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859270+0000', 'last_undegraded': '2024-03-06T07:23:21.859270+0000', 'last_fullsized': '2024-03-06T07:23:21.859270+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.b is not active+clean 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.859561+0000', 'last_change': '2024-03-06T07:23:21.859561+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.859561+0000', 'last_undegraded': '2024-03-06T07:23:21.859561+0000', 'last_fullsized': '2024-03-06T07:23:21.859561+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.8 is not active+clean 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860173+0000', 'last_change': '2024-03-06T07:23:21.860173+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860173+0000', 'last_undegraded': '2024-03-06T07:23:21.860173+0000', 'last_fullsized': '2024-03-06T07:23:21.860173+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.9 is not active+clean 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857088+0000', 'last_change': '2024-03-06T07:23:21.857088+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857088+0000', 'last_undegraded': '2024-03-06T07:23:21.857088+0000', 'last_fullsized': '2024-03-06T07:23:21.857088+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.7 is not active+clean 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.860466+0000', 'last_change': '2024-03-06T07:23:21.860466+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.860466+0000', 'last_undegraded': '2024-03-06T07:23:21.860466+0000', 'last_fullsized': '2024-03-06T07:23:21.860466+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:23:26.683 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.4 is not active+clean 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856546+0000', 'last_change': '2024-03-06T07:23:21.856546+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856546+0000', 'last_undegraded': '2024-03-06T07:23:21.856546+0000', 'last_fullsized': '2024-03-06T07:23:21.856546+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.5 is not active+clean 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.855824+0000', 'last_change': '2024-03-06T07:23:21.855824+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.855824+0000', 'last_undegraded': '2024-03-06T07:23:21.855824+0000', 'last_fullsized': '2024-03-06T07:23:21.855824+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.2 is not active+clean 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.857738+0000', 'last_change': '2024-03-06T07:23:21.857738+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.857738+0000', 'last_undegraded': '2024-03-06T07:23:21.857738+0000', 'last_fullsized': '2024-03-06T07:23:21.857738+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.3 is not active+clean 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.863555+0000', 'last_change': '2024-03-06T07:23:21.863555+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.863555+0000', 'last_undegraded': '2024-03-06T07:23:21.863555+0000', 'last_fullsized': '2024-03-06T07:23:21.863555+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.0 is not active+clean 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.853974+0000', 'last_change': '2024-03-06T07:23:21.853974+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.853974+0000', 'last_undegraded': '2024-03-06T07:23:21.853974+0000', 'last_fullsized': '2024-03-06T07:23:21.853974+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:23:26.684 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.1 is not active+clean 2024-03-06T07:23:26.685 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.858703+0000', 'last_change': '2024-03-06T07:23:21.858703+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.858703+0000', 'last_undegraded': '2024-03-06T07:23:21.858703+0000', 'last_fullsized': '2024-03-06T07:23:21.858703+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:23:26.685 INFO:tasks.cephfs.filesystem.ceph_manager:PG 7.6 is not active+clean 2024-03-06T07:23:26.685 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '7.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 44, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:23:21.856519+0000', 'last_change': '2024-03-06T07:23:21.856519+0000', 'last_active': '2024-03-06T07:23:21.848550+0000', 'last_peered': '2024-03-06T07:23:21.848550+0000', 'last_clean': '2024-03-06T07:23:21.848550+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:23:21.856519+0000', 'last_undegraded': '2024-03-06T07:23:21.856519+0000', 'last_fullsized': '2024-03-06T07:23:21.856519+0000', 'mapping_epoch': 44, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 44, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'last_clean_scrub_stamp': '2024-03-06T07:23:21.848550+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:23:26.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:23:26.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.810+0000 7fd72ede2640 1 Processor -- start 2024-03-06T07:23:26.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.810+0000 7fd72ede2640 1 -- start start 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72ede2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 0x7fd7280d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72ede2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd728162970 0x7fd7280b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72ede2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7280d22c0 con 0x7fd728161cf0 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72ede2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd728161b00 con 0x7fd728162970 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72ede2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280cecc0 con 0x7fd7280b4460 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72cb7e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 0x7fd7280d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd727fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd728162970 0x7fd7280b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72cb7e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 0x7fd7280d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:57514/0 (socket says 172.21.15.72:57514) 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd727fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd728162970 0x7fd7280b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:40946/0 (socket says 172.21.15.72:40946) 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72cb7e640 1 -- 172.21.15.72:0/3244231429 learned_addr learned my addr 172.21.15.72:0/3244231429 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72d37f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd728161cf0 0x7fd7280cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:50824/0 (socket says 172.21.15.72:50824) 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72cb7e640 1 -- 172.21.15.72:0/3244231429 >> v2:172.21.15.146:3300/0 conn(0x7fd728162970 msgr2=0x7fd7280b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:26.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.811+0000 7fd72cb7e640 1 --2- 172.21.15.72:0/3244231429 >> v2:172.21.15.146:3300/0 conn(0x7fd728162970 0x7fd7280b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.812+0000 7fd72cb7e640 1 -- 172.21.15.72:0/3244231429 >> v1:172.21.15.146:6789/0 conn(0x7fd728161cf0 legacy=0x7fd7280cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.812+0000 7fd72cb7e640 1 -- 172.21.15.72:0/3244231429 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280d25a0 con 0x7fd7280b4460 2024-03-06T07:23:26.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.812+0000 7fd72cb7e640 1 --2- 172.21.15.72:0/3244231429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 0x7fd7280d4800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009870 tx=0x7fd71c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f52430abe161e34f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd7277fe640 1 -- 172.21.15.72:0/3244231429 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c034070 con 0x7fd7280b4460 2024-03-06T07:23:26.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd7277fe640 1 -- 172.21.15.72:0/3244231429 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71c030840 con 0x7fd7280b4460 2024-03-06T07:23:26.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd7277fe640 1 -- 172.21.15.72:0/3244231429 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c030c10 con 0x7fd7280b4460 2024-03-06T07:23:26.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 -- 172.21.15.72:0/3244231429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 msgr2=0x7fd7280d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 --2- 172.21.15.72:0/3244231429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 0x7fd7280d4800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd71c009870 tx=0x7fd71c02b980 comp rx=0 tx=0).stop 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 -- 172.21.15.72:0/3244231429 shutdown_connections 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 --2- 172.21.15.72:0/3244231429 >> v2:172.21.15.146:3300/0 conn(0x7fd728162970 0x7fd7280b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 --2- 172.21.15.72:0/3244231429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7280b4460 0x7fd7280d4800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 -- 172.21.15.72:0/3244231429 >> 172.21.15.72:0/3244231429 conn(0x7fd7280d3ce0 msgr2=0x7fd7280d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 -- 172.21.15.72:0/3244231429 shutdown_connections 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.813+0000 7fd72ede2640 1 -- 172.21.15.72:0/3244231429 wait complete. 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.814+0000 7fd72ede2640 1 Processor -- start 2024-03-06T07:23:26.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.814+0000 7fd72ede2640 1 -- start start 2024-03-06T07:23:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72ede2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7280b4460 0x7fd7281530e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72ede2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 0x7fd7280c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72ede2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd728162970 0x7fd7280c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72ede2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280ce950 con 0x7fd728161cf0 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72ede2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280d2b60 con 0x7fd7280b4460 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72ede2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7280b3ac0 con 0x7fd728162970 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd727fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 0x7fd7280c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd727fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 0x7fd7280c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51382/0 (socket says 172.21.15.72:51382) 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd727fff640 1 -- 172.21.15.72:0/1875052853 learned_addr learned my addr 172.21.15.72:0/1875052853 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72cb7e640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7280b4460 0x7fd7281530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.815+0000 7fd72d37f640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd728162970 0x7fd7280c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd727fff640 1 -- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd728162970 msgr2=0x7fd7280c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd727fff640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd728162970 0x7fd7280c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd727fff640 1 -- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7280b4460 msgr2=0x7fd7281530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:26.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd727fff640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7280b4460 0x7fd7281530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd727fff640 1 -- 172.21.15.72:0/1875052853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7280c8290 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd72cb7e640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7280b4460 0x7fd7281530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd727fff640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 0x7fd7280c7810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd71400e900 tx=0x7fd71400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714017070 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71400c9a0 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7280c8570 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71400cd90 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.816+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7280c8b30 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.817+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72804f330 con 0x7fd728161cf0 2024-03-06T07:23:26.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.818+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd71401a070 con 0x7fd728161cf0 2024-03-06T07:23:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.818+0000 7fd7257fa640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd70c06b4d0 0x7fd70c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.818+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd71408a9f0 con 0x7fd728161cf0 2024-03-06T07:23:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.818+0000 7fd72cb7e640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd70c06b4d0 0x7fd70c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:26.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.819+0000 7fd72cb7e640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd70c06b4d0 0x7fd70c06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd71c036000 tx=0x7fd71c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:26.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:26.822+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd7140598c0 con 0x7fd728161cf0 2024-03-06T07:23:27.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.075+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd72806b180 con 0x7fd728161cf0 2024-03-06T07:23:27.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.076+0000 7fd7257fa640 1 -- 172.21.15.72:0/1875052853 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7fd7140591c0 con 0x7fd728161cf0 2024-03-06T07:23:27.078 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:27.078 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":121,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709696,"num_in_osds":8,"osd_in_since":1709709687,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":260644864,"bytes_avail":772833468416,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":20,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:23:11.962885+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd70c06b4d0 msgr2=0x7fd70c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd70c06b4d0 0x7fd70c06d990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd71c036000 tx=0x7fd71c03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 msgr2=0x7fd7280c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 0x7fd7280c7810 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd71400e900 tx=0x7fd71400edc0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 shutdown_connections 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd70c06b4d0 0x7fd70c06d990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd728162970 0x7fd7280c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd728161cf0 0x7fd7280c7810 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 --2- 172.21.15.72:0/1875052853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7280b4460 0x7fd7281530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 >> 172.21.15.72:0/1875052853 conn(0x7fd7280d3ce0 msgr2=0x7fd72806cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 shutdown_connections 2024-03-06T07:23:27.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.078+0000 7fd72ede2640 1 -- 172.21.15.72:0/1875052853 wait complete. 2024-03-06T07:23:27.091 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:23:27.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:27.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.203+0000 7f867990c640 1 Processor -- start 2024-03-06T07:23:27.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.203+0000 7f867990c640 1 -- start start 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f867990c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 0x7f86740cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f867990c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8674071f40 con 0x7f86740cd7a0 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f867990c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867415b8b0 con 0x7f867415b430 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f867990c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8674070a70 con 0x7f86741515e0 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86737fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f86740cd7a0 0x7f86740cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32850/0 (socket says 172.21.15.72:32850) 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86737fe640 1 -- 172.21.15.72:0/1876345540 learned_addr learned my addr 172.21.15.72:0/1876345540 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712362480 0 0) 0x7f8674071f40 con 0x7f86740cd7a0 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8660003520 con 0x7f86740cd7a0 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f8672ffd640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 0x7f86740cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774839217 0 0) 0x7f8660003520 con 0x7f86740cd7a0 2024-03-06T07:23:27.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 msgr2=0x7f86740cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 0x7f86740cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 >> v1:172.21.15.146:6789/0 conn(0x7f867415b430 legacy=0x7f8674071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.204+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674070d50 con 0x7f86740cd7a0 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f866c0019f0 con 0x7f86740cd7a0 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 >> v1:172.21.15.72:6789/0 conn(0x7f86740cd7a0 legacy=0x7f86740cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 0x7f86600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8660004010 0x7f8660004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004d40 0x7f86600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8674070a70 con 0x7f8660004010 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866000bb80 con 0x7f8660004d40 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866000bd80 con 0x7f86741515e0 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f8672ffd640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 0x7f86600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f866bfff640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8660004010 0x7f8660004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86737fe640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004d40 0x7f86600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f866bfff640 1 -- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 msgr2=0x7f86600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f866bfff640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86741515e0 0x7f86600039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f866bfff640 1 -- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004d40 msgr2=0x7f86600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f866bfff640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004d40 0x7f86600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f866bfff640 1 -- 172.21.15.72:0/1876345540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660003520 con 0x7f8660004010 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.205+0000 7f86737fe640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004d40 0x7f86600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f866bfff640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8660004010 0x7f8660004800 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f865c001750 tx=0x7f8660005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9348842bae9ad9b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c001370 con 0x7f8660004010 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c0052d0 con 0x7f8660004010 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f86727fc640 1 -- 172.21.15.72:0/1876345540 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c003830 con 0x7f8660004010 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f867990c640 1 -- 172.21.15.72:0/1876345540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8660004010 msgr2=0x7f8660004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f867990c640 1 --2- 172.21.15.72:0/1876345540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8660004010 0x7f8660004800 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f865c001750 tx=0x7f8660005830 comp rx=0 tx=0).stop 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f86737fe640 1 -- 172.21.15.72:0/1876345540 reap_dead start 2024-03-06T07:23:27.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f867990c640 1 -- 172.21.15.72:0/1876345540 shutdown_connections 2024-03-06T07:23:27.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.206+0000 7f867990c640 1 -- 172.21.15.72:0/1876345540 >> 172.21.15.72:0/1876345540 conn(0x7f86740cf8a0 msgr2=0x7f86740cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.207+0000 7f867990c640 1 -- 172.21.15.72:0/1876345540 shutdown_connections 2024-03-06T07:23:27.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.207+0000 7f867990c640 1 -- 172.21.15.72:0/1876345540 wait complete. 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 Processor -- start 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 -- start start 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004010 0x7f8674143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 0x7f867406d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f867406dec0 0x7f867415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86740cce40 con 0x7f86741445c0 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8674071310 con 0x7f8660004010 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f867990c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86740ccfc0 con 0x7f867406dec0 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f866bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 0x7f867406d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f866bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 0x7f867406d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51412/0 (socket says 172.21.15.72:51412) 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.208+0000 7f866bfff640 1 -- 172.21.15.72:0/1695874202 learned_addr learned my addr 172.21.15.72:0/1695874202 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f8672ffd640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004010 0x7f8674143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f86737fe640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f867406dec0 0x7f867415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f866bfff640 1 -- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f867406dec0 msgr2=0x7f867415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f866bfff640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f867406dec0 0x7f867415cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f866bfff640 1 -- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004010 msgr2=0x7f8674143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f866bfff640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004010 0x7f8674143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f866bfff640 1 -- 172.21.15.72:0/1695874202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86741473a0 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f866bfff640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 0x7f867406d980 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f865c001870 tx=0x7f865c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c041070 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c005480 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8674145340 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c001370 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.209+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8674145900 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.210+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8674068840 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.211+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f865c044070 con 0x7f86741445c0 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.211+0000 7f8670ff9640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f865406b4d0 0x7f865406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:27.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.211+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f865c0abc30 con 0x7f86741445c0 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.211+0000 7f8672ffd640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f865406b4d0 0x7f865406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.212+0000 7f8672ffd640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f865406b4d0 0x7f865406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8664006fd0 tx=0x7f8664008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:27.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.216+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f865c07ab40 con 0x7f86741445c0 2024-03-06T07:23:27.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.389+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f867406e2a0 con 0x7f865406b4d0 2024-03-06T07:23:27.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.395+0000 7f8670ff9640 1 -- 172.21.15.72:0/1695874202 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246635 (secure 0 0 0) 0x7f867406e2a0 con 0x7f865406b4d0 2024-03-06T07:23:27.400 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:27.401 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f865406b4d0 msgr2=0x7f865406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f865406b4d0 0x7f865406d990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8664006fd0 tx=0x7f8664008040 comp rx=0 tx=0).stop 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 msgr2=0x7f867406d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 0x7f867406d980 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f865c001870 tx=0x7f865c001120 comp rx=0 tx=0).stop 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 shutdown_connections 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f865406b4d0 0x7f865406d990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f867406dec0 0x7f867415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86741445c0 0x7f867406d980 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 --2- 172.21.15.72:0/1695874202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8660004010 0x7f8674143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 >> 172.21.15.72:0/1695874202 conn(0x7f86740cf8a0 msgr2=0x7f8674150b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 shutdown_connections 2024-03-06T07:23:27.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:27.398+0000 7f867990c640 1 -- 172.21.15.72:0/1695874202 wait complete. 2024-03-06T07:23:27.411 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-03-06T07:23:25.964943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254536,"kb_used_data":5096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720184,"statfs":{"total":773094113280,"available":772833468416,"internally_reserved":0,"allocated":5218304,"data_stored":2784288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.115574"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856478+0000","last_change":"2024-03-06T07:23:21.856478+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856478+0000","last_undegraded":"2024-03-06T07:23:21.856478+0000","last_fullsized":"2024-03-06T07:23:21.856478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855000+0000","last_change":"2024-03-06T07:23:21.855000+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855000+0000","last_undegraded":"2024-03-06T07:23:21.855000+0000","last_fullsized":"2024-03-06T07:23:21.855000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860791+0000","last_change":"2024-03-06T07:23:21.860791+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860791+0000","last_undegraded":"2024-03-06T07:23:21.860791+0000","last_fullsized":"2024-03-06T07:23:21.860791+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860752+0000","last_change":"2024-03-06T07:23:21.860752+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860752+0000","last_undegraded":"2024-03-06T07:23:21.860752+0000","last_fullsized":"2024-03-06T07:23:21.860752+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853814+0000","last_change":"2024-03-06T07:23:21.853814+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853814+0000","last_undegraded":"2024-03-06T07:23:21.853814+0000","last_fullsized":"2024-03-06T07:23:21.853814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861844+0000","last_change":"2024-03-06T07:23:21.861844+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861844+0000","last_undegraded":"2024-03-06T07:23:21.861844+0000","last_fullsized":"2024-03-06T07:23:21.861844+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855942+0000","last_change":"2024-03-06T07:23:21.855942+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855942+0000","last_undegraded":"2024-03-06T07:23:21.855942+0000","last_fullsized":"2024-03-06T07:23:21.855942+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857730+0000","last_change":"2024-03-06T07:23:21.857730+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857730+0000","last_undegraded":"2024-03-06T07:23:21.857730+0000","last_fullsized":"2024-03-06T07:23:21.857730+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860536+0000","last_change":"2024-03-06T07:23:21.860536+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860536+0000","last_undegraded":"2024-03-06T07:23:21.860536+0000","last_fullsized":"2024-03-06T07:23:21.860536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860872+0000","last_change":"2024-03-06T07:23:21.860872+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860872+0000","last_undegraded":"2024-03-06T07:23:21.860872+0000","last_fullsized":"2024-03-06T07:23:21.860872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860140+0000","last_change":"2024-03-06T07:23:21.860140+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860140+0000","last_undegraded":"2024-03-06T07:23:21.860140+0000","last_fullsized":"2024-03-06T07:23:21.860140+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855745+0000","last_change":"2024-03-06T07:23:21.855745+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855745+0000","last_undegraded":"2024-03-06T07:23:21.855745+0000","last_fullsized":"2024-03-06T07:23:21.855745+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856960+0000","last_change":"2024-03-06T07:23:21.856960+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856960+0000","last_undegraded":"2024-03-06T07:23:21.856960+0000","last_fullsized":"2024-03-06T07:23:21.856960+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860354+0000","last_change":"2024-03-06T07:23:21.860354+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860354+0000","last_undegraded":"2024-03-06T07:23:21.860354+0000","last_fullsized":"2024-03-06T07:23:21.860354+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862480+0000","last_change":"2024-03-06T07:23:21.862480+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862480+0000","last_undegraded":"2024-03-06T07:23:21.862480+0000","last_fullsized":"2024-03-06T07:23:21.862480+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859111+0000","last_change":"2024-03-06T07:23:21.859111+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859111+0000","last_undegraded":"2024-03-06T07:23:21.859111+0000","last_fullsized":"2024-03-06T07:23:21.859111+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861819+0000","last_change":"2024-03-06T07:23:21.861819+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861819+0000","last_undegraded":"2024-03-06T07:23:21.861819+0000","last_fullsized":"2024-03-06T07:23:21.861819+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861199+0000","last_change":"2024-03-06T07:23:21.861199+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861199+0000","last_undegraded":"2024-03-06T07:23:21.861199+0000","last_fullsized":"2024-03-06T07:23:21.861199+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860297+0000","last_change":"2024-03-06T07:23:21.860297+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860297+0000","last_undegraded":"2024-03-06T07:23:21.860297+0000","last_fullsized":"2024-03-06T07:23:21.860297+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857004+0000","last_change":"2024-03-06T07:23:21.857004+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857004+0000","last_undegraded":"2024-03-06T07:23:21.857004+0000","last_fullsized":"2024-03-06T07:23:21.857004+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863958+0000","last_change":"2024-03-06T07:23:21.863958+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863958+0000","last_undegraded":"2024-03-06T07:23:21.863958+0000","last_fullsized":"2024-03-06T07:23:21.863958+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856664+0000","last_change":"2024-03-06T07:23:21.856664+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856664+0000","last_undegraded":"2024-03-06T07:23:21.856664+0000","last_fullsized":"2024-03-06T07:23:21.856664+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858592+0000","last_change":"2024-03-06T07:23:21.858592+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858592+0000","last_undegraded":"2024-03-06T07:23:21.858592+0000","last_fullsized":"2024-03-06T07:23:21.858592+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858616+0000","last_change":"2024-03-06T07:23:21.858616+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858616+0000","last_undegraded":"2024-03-06T07:23:21.858616+0000","last_fullsized":"2024-03-06T07:23:21.858616+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863987+0000","last_change":"2024-03-06T07:23:21.863987+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863987+0000","last_undegraded":"2024-03-06T07:23:21.863987+0000","last_fullsized":"2024-03-06T07:23:21.863987+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860157+0000","last_change":"2024-03-06T07:23:21.860157+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860157+0000","last_undegraded":"2024-03-06T07:23:21.860157+0000","last_fullsized":"2024-03-06T07:23:21.860157+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859204+0000","last_change":"2024-03-06T07:23:21.859204+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859204+0000","last_undegraded":"2024-03-06T07:23:21.859204+0000","last_fullsized":"2024-03-06T07:23:21.859204+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854927+0000","last_change":"2024-03-06T07:23:21.854927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854927+0000","last_undegraded":"2024-03-06T07:23:21.854927+0000","last_fullsized":"2024-03-06T07:23:21.854927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.862784+0000","last_change":"2024-03-06T07:23:21.862784+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.862784+0000","last_undegraded":"2024-03-06T07:23:21.862784+0000","last_fullsized":"2024-03-06T07:23:21.862784+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859904+0000","last_change":"2024-03-06T07:23:21.859904+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859904+0000","last_undegraded":"2024-03-06T07:23:21.859904+0000","last_fullsized":"2024-03-06T07:23:21.859904+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858223+0000","last_change":"2024-03-06T07:23:21.858223+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858223+0000","last_undegraded":"2024-03-06T07:23:21.858223+0000","last_fullsized":"2024-03-06T07:23:21.858223+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860570+0000","last_change":"2024-03-06T07:23:21.860570+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860570+0000","last_undegraded":"2024-03-06T07:23:21.860570+0000","last_fullsized":"2024-03-06T07:23:21.860570+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858941+0000","last_change":"2024-03-06T07:23:21.858941+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858941+0000","last_undegraded":"2024-03-06T07:23:21.858941+0000","last_fullsized":"2024-03-06T07:23:21.858941+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874695+0000","last_change":"2024-03-06T07:23:21.870976+0000","last_active":"2024-03-06T07:23:21.874695+0000","last_peered":"2024-03-06T07:23:21.874695+0000","last_clean":"2024-03-06T07:23:21.874695+0000","last_became_active":"2024-03-06T07:23:21.870706+0000","last_became_peered":"2024-03-06T07:23:21.870706+0000","last_unstale":"2024-03-06T07:23:21.874695+0000","last_undegraded":"2024-03-06T07:23:21.874695+0000","last_fullsized":"2024-03-06T07:23:21.874695+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860916+0000","last_change":"2024-03-06T07:23:21.860916+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860916+0000","last_undegraded":"2024-03-06T07:23:21.860916+0000","last_fullsized":"2024-03-06T07:23:21.860916+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.865503+0000","last_change":"2024-03-06T07:23:21.864644+0000","last_active":"2024-03-06T07:23:21.865503+0000","last_peered":"2024-03-06T07:23:21.865503+0000","last_clean":"2024-03-06T07:23:21.865503+0000","last_became_active":"2024-03-06T07:23:21.864375+0000","last_became_peered":"2024-03-06T07:23:21.864375+0000","last_unstale":"2024-03-06T07:23:21.865503+0000","last_undegraded":"2024-03-06T07:23:21.865503+0000","last_fullsized":"2024-03-06T07:23:21.865503+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854364+0000","last_change":"2024-03-06T07:23:21.854364+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854364+0000","last_undegraded":"2024-03-06T07:23:21.854364+0000","last_fullsized":"2024-03-06T07:23:21.854364+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869438+0000","last_change":"2024-03-06T07:23:21.868746+0000","last_active":"2024-03-06T07:23:21.869438+0000","last_peered":"2024-03-06T07:23:21.869438+0000","last_clean":"2024-03-06T07:23:21.869438+0000","last_became_active":"2024-03-06T07:23:21.868455+0000","last_became_peered":"2024-03-06T07:23:21.868455+0000","last_unstale":"2024-03-06T07:23:21.869438+0000","last_undegraded":"2024-03-06T07:23:21.869438+0000","last_fullsized":"2024-03-06T07:23:21.869438+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:44.858008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860303+0000","last_change":"2024-03-06T07:23:21.860303+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860303+0000","last_undegraded":"2024-03-06T07:23:21.860303+0000","last_fullsized":"2024-03-06T07:23:21.860303+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870280+0000","last_change":"2024-03-06T07:23:21.865942+0000","last_active":"2024-03-06T07:23:21.870280+0000","last_peered":"2024-03-06T07:23:21.870280+0000","last_clean":"2024-03-06T07:23:21.870280+0000","last_became_active":"2024-03-06T07:23:21.865642+0000","last_became_peered":"2024-03-06T07:23:21.865642+0000","last_unstale":"2024-03-06T07:23:21.870280+0000","last_undegraded":"2024-03-06T07:23:21.870280+0000","last_fullsized":"2024-03-06T07:23:21.870280+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859164+0000","last_change":"2024-03-06T07:23:21.859164+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859164+0000","last_undegraded":"2024-03-06T07:23:21.859164+0000","last_fullsized":"2024-03-06T07:23:21.859164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874913+0000","last_change":"2024-03-06T07:23:21.872744+0000","last_active":"2024-03-06T07:23:21.874913+0000","last_peered":"2024-03-06T07:23:21.874913+0000","last_clean":"2024-03-06T07:23:21.874913+0000","last_became_active":"2024-03-06T07:23:21.872444+0000","last_became_peered":"2024-03-06T07:23:21.872444+0000","last_unstale":"2024-03-06T07:23:21.874913+0000","last_undegraded":"2024-03-06T07:23:21.874913+0000","last_fullsized":"2024-03-06T07:23:21.874913+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:43.054124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857762+0000","last_change":"2024-03-06T07:23:21.857762+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857762+0000","last_undegraded":"2024-03-06T07:23:21.857762+0000","last_fullsized":"2024-03-06T07:23:21.857762+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870098+0000","last_change":"2024-03-06T07:23:21.868824+0000","last_active":"2024-03-06T07:23:21.870098+0000","last_peered":"2024-03-06T07:23:21.870098+0000","last_clean":"2024-03-06T07:23:21.870098+0000","last_became_active":"2024-03-06T07:23:21.867672+0000","last_became_peered":"2024-03-06T07:23:21.867672+0000","last_unstale":"2024-03-06T07:23:21.870098+0000","last_undegraded":"2024-03-06T07:23:21.870098+0000","last_fullsized":"2024-03-06T07:23:21.870098+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857267+0000","last_change":"2024-03-06T07:23:21.857267+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857267+0000","last_undegraded":"2024-03-06T07:23:21.857267+0000","last_fullsized":"2024-03-06T07:23:21.857267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866889+0000","last_change":"2024-03-06T07:23:21.863472+0000","last_active":"2024-03-06T07:23:21.866889+0000","last_peered":"2024-03-06T07:23:21.866889+0000","last_clean":"2024-03-06T07:23:21.866889+0000","last_became_active":"2024-03-06T07:23:21.863161+0000","last_became_peered":"2024-03-06T07:23:21.863161+0000","last_unstale":"2024-03-06T07:23:21.866889+0000","last_undegraded":"2024-03-06T07:23:21.866889+0000","last_fullsized":"2024-03-06T07:23:21.866889+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:42.354236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.865694+0000","last_change":"2024-03-06T07:23:21.865694+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.865694+0000","last_undegraded":"2024-03-06T07:23:21.865694+0000","last_fullsized":"2024-03-06T07:23:21.865694+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867714+0000","last_change":"2024-03-06T07:23:21.864598+0000","last_active":"2024-03-06T07:23:21.867714+0000","last_peered":"2024-03-06T07:23:21.867714+0000","last_clean":"2024-03-06T07:23:21.867714+0000","last_became_active":"2024-03-06T07:23:21.862773+0000","last_became_peered":"2024-03-06T07:23:21.862773+0000","last_unstale":"2024-03-06T07:23:21.867714+0000","last_undegraded":"2024-03-06T07:23:21.867714+0000","last_fullsized":"2024-03-06T07:23:21.867714+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:25.977053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860119+0000","last_change":"2024-03-06T07:23:21.860119+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860119+0000","last_undegraded":"2024-03-06T07:23:21.860119+0000","last_fullsized":"2024-03-06T07:23:21.860119+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870825+0000","last_change":"2024-03-06T07:23:21.867046+0000","last_active":"2024-03-06T07:23:21.870825+0000","last_peered":"2024-03-06T07:23:21.870825+0000","last_clean":"2024-03-06T07:23:21.870825+0000","last_became_active":"2024-03-06T07:23:21.866751+0000","last_became_peered":"2024-03-06T07:23:21.866751+0000","last_unstale":"2024-03-06T07:23:21.870825+0000","last_undegraded":"2024-03-06T07:23:21.870825+0000","last_fullsized":"2024-03-06T07:23:21.870825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.854878+0000","last_change":"2024-03-06T07:23:21.854878+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.854878+0000","last_undegraded":"2024-03-06T07:23:21.854878+0000","last_fullsized":"2024-03-06T07:23:21.854878+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874650+0000","last_change":"2024-03-06T07:23:21.869633+0000","last_active":"2024-03-06T07:23:21.874650+0000","last_peered":"2024-03-06T07:23:21.874650+0000","last_clean":"2024-03-06T07:23:21.874650+0000","last_became_active":"2024-03-06T07:23:21.869349+0000","last_became_peered":"2024-03-06T07:23:21.869349+0000","last_unstale":"2024-03-06T07:23:21.874650+0000","last_undegraded":"2024-03-06T07:23:21.874650+0000","last_fullsized":"2024-03-06T07:23:21.874650+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:41:32.206617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857576+0000","last_change":"2024-03-06T07:23:21.857576+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857576+0000","last_undegraded":"2024-03-06T07:23:21.857576+0000","last_fullsized":"2024-03-06T07:23:21.857576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871236+0000","last_change":"2024-03-06T07:23:21.868933+0000","last_active":"2024-03-06T07:23:21.871236+0000","last_peered":"2024-03-06T07:23:21.871236+0000","last_clean":"2024-03-06T07:23:21.871236+0000","last_became_active":"2024-03-06T07:23:21.868305+0000","last_became_peered":"2024-03-06T07:23:21.868305+0000","last_unstale":"2024-03-06T07:23:21.871236+0000","last_undegraded":"2024-03-06T07:23:21.871236+0000","last_fullsized":"2024-03-06T07:23:21.871236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856248+0000","last_change":"2024-03-06T07:23:21.856248+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856248+0000","last_undegraded":"2024-03-06T07:23:21.856248+0000","last_fullsized":"2024-03-06T07:23:21.856248+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.866259+0000","last_change":"2024-03-06T07:23:21.862722+0000","last_active":"2024-03-06T07:23:21.866259+0000","last_peered":"2024-03-06T07:23:21.866259+0000","last_clean":"2024-03-06T07:23:21.866259+0000","last_became_active":"2024-03-06T07:23:21.862470+0000","last_became_peered":"2024-03-06T07:23:21.862470+0000","last_unstale":"2024-03-06T07:23:21.866259+0000","last_undegraded":"2024-03-06T07:23:21.866259+0000","last_fullsized":"2024-03-06T07:23:21.866259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:06:13.201036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860147+0000","last_change":"2024-03-06T07:23:21.860147+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860147+0000","last_undegraded":"2024-03-06T07:23:21.860147+0000","last_fullsized":"2024-03-06T07:23:21.860147+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.877196+0000","last_change":"2024-03-06T07:23:21.875315+0000","last_active":"2024-03-06T07:23:21.877196+0000","last_peered":"2024-03-06T07:23:21.877196+0000","last_clean":"2024-03-06T07:23:21.877196+0000","last_became_active":"2024-03-06T07:23:21.874938+0000","last_became_peered":"2024-03-06T07:23:21.874938+0000","last_unstale":"2024-03-06T07:23:21.877196+0000","last_undegraded":"2024-03-06T07:23:21.877196+0000","last_fullsized":"2024-03-06T07:23:21.877196+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.194391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857385+0000","last_change":"2024-03-06T07:23:21.857385+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857385+0000","last_undegraded":"2024-03-06T07:23:21.857385+0000","last_fullsized":"2024-03-06T07:23:21.857385+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871183+0000","last_change":"2024-03-06T07:23:21.869035+0000","last_active":"2024-03-06T07:23:21.871183+0000","last_peered":"2024-03-06T07:23:21.871183+0000","last_clean":"2024-03-06T07:23:21.871183+0000","last_became_active":"2024-03-06T07:23:21.868739+0000","last_became_peered":"2024-03-06T07:23:21.868739+0000","last_unstale":"2024-03-06T07:23:21.871183+0000","last_undegraded":"2024-03-06T07:23:21.871183+0000","last_fullsized":"2024-03-06T07:23:21.871183+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857927+0000","last_change":"2024-03-06T07:23:21.857927+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857927+0000","last_undegraded":"2024-03-06T07:23:21.857927+0000","last_fullsized":"2024-03-06T07:23:21.857927+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874442+0000","last_change":"2024-03-06T07:23:21.868680+0000","last_active":"2024-03-06T07:23:21.874442+0000","last_peered":"2024-03-06T07:23:21.874442+0000","last_clean":"2024-03-06T07:23:21.874442+0000","last_became_active":"2024-03-06T07:23:21.868416+0000","last_became_peered":"2024-03-06T07:23:21.868416+0000","last_unstale":"2024-03-06T07:23:21.874442+0000","last_undegraded":"2024-03-06T07:23:21.874442+0000","last_fullsized":"2024-03-06T07:23:21.874442+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855516+0000","last_change":"2024-03-06T07:23:21.855516+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855516+0000","last_undegraded":"2024-03-06T07:23:21.855516+0000","last_fullsized":"2024-03-06T07:23:21.855516+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874641+0000","last_change":"2024-03-06T07:23:21.870601+0000","last_active":"2024-03-06T07:23:21.874641+0000","last_peered":"2024-03-06T07:23:21.874641+0000","last_clean":"2024-03-06T07:23:21.874641+0000","last_became_active":"2024-03-06T07:23:21.870306+0000","last_became_peered":"2024-03-06T07:23:21.870306+0000","last_unstale":"2024-03-06T07:23:21.874641+0000","last_undegraded":"2024-03-06T07:23:21.874641+0000","last_fullsized":"2024-03-06T07:23:21.874641+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874729+0000","last_change":"2024-03-06T07:23:21.872671+0000","last_active":"2024-03-06T07:23:21.874729+0000","last_peered":"2024-03-06T07:23:21.874729+0000","last_clean":"2024-03-06T07:23:21.874729+0000","last_became_active":"2024-03-06T07:23:21.872330+0000","last_became_peered":"2024-03-06T07:23:21.872330+0000","last_unstale":"2024-03-06T07:23:21.874729+0000","last_undegraded":"2024-03-06T07:23:21.874729+0000","last_fullsized":"2024-03-06T07:23:21.874729+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:54:10.134358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856748+0000","last_change":"2024-03-06T07:23:21.856748+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856748+0000","last_undegraded":"2024-03-06T07:23:21.856748+0000","last_fullsized":"2024-03-06T07:23:21.856748+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.872248+0000","last_change":"2024-03-06T07:23:21.870771+0000","last_active":"2024-03-06T07:23:21.872248+0000","last_peered":"2024-03-06T07:23:21.872248+0000","last_clean":"2024-03-06T07:23:21.872248+0000","last_became_active":"2024-03-06T07:23:21.870450+0000","last_became_peered":"2024-03-06T07:23:21.870450+0000","last_unstale":"2024-03-06T07:23:21.872248+0000","last_undegraded":"2024-03-06T07:23:21.872248+0000","last_fullsized":"2024-03-06T07:23:21.872248+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:01:12.359858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859255+0000","last_change":"2024-03-06T07:23:21.859255+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859255+0000","last_undegraded":"2024-03-06T07:23:21.859255+0000","last_fullsized":"2024-03-06T07:23:21.859255+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869938+0000","last_change":"2024-03-06T07:23:21.865980+0000","last_active":"2024-03-06T07:23:21.869938+0000","last_peered":"2024-03-06T07:23:21.869938+0000","last_clean":"2024-03-06T07:23:21.869938+0000","last_became_active":"2024-03-06T07:23:21.865586+0000","last_became_peered":"2024-03-06T07:23:21.865586+0000","last_unstale":"2024-03-06T07:23:21.869938+0000","last_undegraded":"2024-03-06T07:23:21.869938+0000","last_fullsized":"2024-03-06T07:23:21.869938+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.861702+0000","last_change":"2024-03-06T07:23:21.861702+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.861702+0000","last_undegraded":"2024-03-06T07:23:21.861702+0000","last_fullsized":"2024-03-06T07:23:21.861702+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871876+0000","last_change":"2024-03-06T07:23:21.866832+0000","last_active":"2024-03-06T07:23:21.871876+0000","last_peered":"2024-03-06T07:23:21.871876+0000","last_clean":"2024-03-06T07:23:21.871876+0000","last_became_active":"2024-03-06T07:23:21.866546+0000","last_became_peered":"2024-03-06T07:23:21.866546+0000","last_unstale":"2024-03-06T07:23:21.871876+0000","last_undegraded":"2024-03-06T07:23:21.871876+0000","last_fullsized":"2024-03-06T07:23:21.871876+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859270+0000","last_change":"2024-03-06T07:23:21.859270+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859270+0000","last_undegraded":"2024-03-06T07:23:21.859270+0000","last_fullsized":"2024-03-06T07:23:21.859270+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871931+0000","last_change":"2024-03-06T07:23:21.867500+0000","last_active":"2024-03-06T07:23:21.871931+0000","last_peered":"2024-03-06T07:23:21.871931+0000","last_clean":"2024-03-06T07:23:21.871931+0000","last_became_active":"2024-03-06T07:23:21.867223+0000","last_became_peered":"2024-03-06T07:23:21.867223+0000","last_unstale":"2024-03-06T07:23:21.871931+0000","last_undegraded":"2024-03-06T07:23:21.871931+0000","last_fullsized":"2024-03-06T07:23:21.871931+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.859561+0000","last_change":"2024-03-06T07:23:21.859561+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.859561+0000","last_undegraded":"2024-03-06T07:23:21.859561+0000","last_fullsized":"2024-03-06T07:23:21.859561+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869963+0000","last_change":"2024-03-06T07:23:21.866528+0000","last_active":"2024-03-06T07:23:21.869963+0000","last_peered":"2024-03-06T07:23:21.869963+0000","last_clean":"2024-03-06T07:23:21.869963+0000","last_became_active":"2024-03-06T07:23:21.866281+0000","last_became_peered":"2024-03-06T07:23:21.866281+0000","last_unstale":"2024-03-06T07:23:21.869963+0000","last_undegraded":"2024-03-06T07:23:21.869963+0000","last_fullsized":"2024-03-06T07:23:21.869963+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860173+0000","last_change":"2024-03-06T07:23:21.860173+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860173+0000","last_undegraded":"2024-03-06T07:23:21.860173+0000","last_fullsized":"2024-03-06T07:23:21.860173+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874610+0000","last_change":"2024-03-06T07:23:21.870289+0000","last_active":"2024-03-06T07:23:21.874610+0000","last_peered":"2024-03-06T07:23:21.874610+0000","last_clean":"2024-03-06T07:23:21.874610+0000","last_became_active":"2024-03-06T07:23:21.870037+0000","last_became_peered":"2024-03-06T07:23:21.870037+0000","last_unstale":"2024-03-06T07:23:21.874610+0000","last_undegraded":"2024-03-06T07:23:21.874610+0000","last_fullsized":"2024-03-06T07:23:21.874610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:23:15.687487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857088+0000","last_change":"2024-03-06T07:23:21.857088+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857088+0000","last_undegraded":"2024-03-06T07:23:21.857088+0000","last_fullsized":"2024-03-06T07:23:21.857088+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871188+0000","last_change":"2024-03-06T07:23:21.869803+0000","last_active":"2024-03-06T07:23:21.871188+0000","last_peered":"2024-03-06T07:23:21.871188+0000","last_clean":"2024-03-06T07:23:21.871188+0000","last_became_active":"2024-03-06T07:23:21.869511+0000","last_became_peered":"2024-03-06T07:23:21.869511+0000","last_unstale":"2024-03-06T07:23:21.871188+0000","last_undegraded":"2024-03-06T07:23:21.871188+0000","last_fullsized":"2024-03-06T07:23:21.871188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.860466+0000","last_change":"2024-03-06T07:23:21.860466+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.860466+0000","last_undegraded":"2024-03-06T07:23:21.860466+0000","last_fullsized":"2024-03-06T07:23:21.860466+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874543+0000","last_change":"2024-03-06T07:23:21.871570+0000","last_active":"2024-03-06T07:23:21.874543+0000","last_peered":"2024-03-06T07:23:21.874543+0000","last_clean":"2024-03-06T07:23:21.874543+0000","last_became_active":"2024-03-06T07:23:21.871278+0000","last_became_peered":"2024-03-06T07:23:21.871278+0000","last_unstale":"2024-03-06T07:23:21.874543+0000","last_undegraded":"2024-03-06T07:23:21.874543+0000","last_fullsized":"2024-03-06T07:23:21.874543+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:00.837068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856546+0000","last_change":"2024-03-06T07:23:21.856546+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856546+0000","last_undegraded":"2024-03-06T07:23:21.856546+0000","last_fullsized":"2024-03-06T07:23:21.856546+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.871218+0000","last_change":"2024-03-06T07:23:21.868700+0000","last_active":"2024-03-06T07:23:21.871218+0000","last_peered":"2024-03-06T07:23:21.871218+0000","last_clean":"2024-03-06T07:23:21.871218+0000","last_became_active":"2024-03-06T07:23:21.868258+0000","last_became_peered":"2024-03-06T07:23:21.868258+0000","last_unstale":"2024-03-06T07:23:21.871218+0000","last_undegraded":"2024-03-06T07:23:21.871218+0000","last_fullsized":"2024-03-06T07:23:21.871218+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:34.715695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.855824+0000","last_change":"2024-03-06T07:23:21.855824+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.855824+0000","last_undegraded":"2024-03-06T07:23:21.855824+0000","last_fullsized":"2024-03-06T07:23:21.855824+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.869070+0000","last_change":"2024-03-06T07:23:21.868016+0000","last_active":"2024-03-06T07:23:21.869070+0000","last_peered":"2024-03-06T07:23:21.869070+0000","last_clean":"2024-03-06T07:23:21.869070+0000","last_became_active":"2024-03-06T07:23:21.867724+0000","last_became_peered":"2024-03-06T07:23:21.867724+0000","last_unstale":"2024-03-06T07:23:21.869070+0000","last_undegraded":"2024-03-06T07:23:21.869070+0000","last_fullsized":"2024-03-06T07:23:21.869070+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:24.138593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.857738+0000","last_change":"2024-03-06T07:23:21.857738+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.857738+0000","last_undegraded":"2024-03-06T07:23:21.857738+0000","last_fullsized":"2024-03-06T07:23:21.857738+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870300+0000","last_change":"2024-03-06T07:23:21.867760+0000","last_active":"2024-03-06T07:23:21.870300+0000","last_peered":"2024-03-06T07:23:21.870300+0000","last_clean":"2024-03-06T07:23:21.870300+0000","last_became_active":"2024-03-06T07:23:21.867499+0000","last_became_peered":"2024-03-06T07:23:21.867499+0000","last_unstale":"2024-03-06T07:23:21.870300+0000","last_undegraded":"2024-03-06T07:23:21.870300+0000","last_fullsized":"2024-03-06T07:23:21.870300+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:14.493790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.863555+0000","last_change":"2024-03-06T07:23:21.863555+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.863555+0000","last_undegraded":"2024-03-06T07:23:21.863555+0000","last_fullsized":"2024-03-06T07:23:21.863555+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.870077+0000","last_change":"2024-03-06T07:23:21.867044+0000","last_active":"2024-03-06T07:23:21.870077+0000","last_peered":"2024-03-06T07:23:21.870077+0000","last_clean":"2024-03-06T07:23:21.870077+0000","last_became_active":"2024-03-06T07:23:21.866525+0000","last_became_peered":"2024-03-06T07:23:21.866525+0000","last_unstale":"2024-03-06T07:23:21.870077+0000","last_undegraded":"2024-03-06T07:23:21.870077+0000","last_fullsized":"2024-03-06T07:23:21.870077+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.853974+0000","last_change":"2024-03-06T07:23:21.853974+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.853974+0000","last_undegraded":"2024-03-06T07:23:21.853974+0000","last_fullsized":"2024-03-06T07:23:21.853974+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874320+0000","last_change":"2024-03-06T07:23:21.871204+0000","last_active":"2024-03-06T07:23:21.874320+0000","last_peered":"2024-03-06T07:23:21.874320+0000","last_clean":"2024-03-06T07:23:21.874320+0000","last_became_active":"2024-03-06T07:23:21.870923+0000","last_became_peered":"2024-03-06T07:23:21.870923+0000","last_unstale":"2024-03-06T07:23:21.874320+0000","last_undegraded":"2024-03-06T07:23:21.874320+0000","last_fullsized":"2024-03-06T07:23:21.874320+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.850435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.858703+0000","last_change":"2024-03-06T07:23:21.858703+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.858703+0000","last_undegraded":"2024-03-06T07:23:21.858703+0000","last_fullsized":"2024-03-06T07:23:21.858703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.874520+0000","last_change":"2024-03-06T07:23:21.867519+0000","last_active":"2024-03-06T07:23:21.874520+0000","last_peered":"2024-03-06T07:23:21.874520+0000","last_clean":"2024-03-06T07:23:21.874520+0000","last_became_active":"2024-03-06T07:23:21.867274+0000","last_became_peered":"2024-03-06T07:23:21.867274+0000","last_unstale":"2024-03-06T07:23:21.874520+0000","last_undegraded":"2024-03-06T07:23:21.874520+0000","last_fullsized":"2024-03-06T07:23:21.874520+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:26.588956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-03-06T07:23:21.856519+0000","last_change":"2024-03-06T07:23:21.856519+0000","last_active":"2024-03-06T07:23:21.848550+0000","last_peered":"2024-03-06T07:23:21.848550+0000","last_clean":"2024-03-06T07:23:21.848550+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:21.856519+0000","last_undegraded":"2024-03-06T07:23:21.856519+0000","last_fullsized":"2024-03-06T07:23:21.856519+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.867006+0000","last_change":"2024-03-06T07:23:21.863945+0000","last_active":"2024-03-06T07:23:21.867006+0000","last_peered":"2024-03-06T07:23:21.867006+0000","last_clean":"2024-03-06T07:23:21.867006+0000","last_became_active":"2024-03-06T07:23:21.863699+0000","last_became_peered":"2024-03-06T07:23:21.863699+0000","last_unstale":"2024-03-06T07:23:21.867006+0000","last_undegraded":"2024-03-06T07:23:21.867006+0000","last_fullsized":"2024-03-06T07:23:21.867006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:13.743953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":212,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-03-06T07:23:21.854216+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:21.854216+0000","last_peered":"2024-03-06T07:23:21.854216+0000","last_clean":"2024-03-06T07:23:21.854216+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:21.854216+0000","last_undegraded":"2024-03-06T07:23:21.854216+0000","last_fullsized":"2024-03-06T07:23:21.854216+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":200444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1097728,"data_stored":790812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:28.123 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-03-06T07:23:28.121+0000 7ff3a7a09640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:23:30.416 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:23:30.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.517+0000 7f1f16bc9640 1 Processor -- start 2024-03-06T07:23:30.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.517+0000 7f1f16bc9640 1 -- start start 2024-03-06T07:23:30.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f16bc9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f1015bf30 0x7f1f1015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f16bc9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1015f580 con 0x7f1f100b3060 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f16bc9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1015b450 con 0x7f1f1013dc10 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f16bc9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f100cf3f0 con 0x7f1f1015bf30 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f0ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1f100b3060 0x7f1f1015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:32858/0 (socket says 172.21.15.72:32858) 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f0ffff640 1 -- 172.21.15.72:0/4076678725 learned_addr learned my addr 172.21.15.72:0/4076678725 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f15166640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f1015bf30 0x7f1f1015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f0f7fe640 1 -- 172.21.15.72:0/4076678725 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551541999 0 0) 0x7f1f1015f580 con 0x7f1f100b3060 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f0f7fe640 1 -- 172.21.15.72:0/4076678725 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1efc003520 con 0x7f1f100b3060 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f15166640 1 -- 172.21.15.72:0/4076678725 >> v1:172.21.15.146:6789/0 conn(0x7f1f1013dc10 legacy=0x7f1f100b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f15166640 1 -- 172.21.15.72:0/4076678725 >> v1:172.21.15.72:6789/0 conn(0x7f1f100b3060 legacy=0x7f1f1015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.518+0000 7f1f15166640 1 -- 172.21.15.72:0/4076678725 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1015b730 con 0x7f1f1015bf30 2024-03-06T07:23:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.519+0000 7f1f15166640 1 --2- 172.21.15.72:0/4076678725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1015bf30 0x7f1f1015ef60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009d90 tx=0x7f1f0802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fdcef0144502ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.519+0000 7f1f0f7fe640 1 -- 172.21.15.72:0/4076678725 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f08039070 con 0x7f1f1015bf30 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.519+0000 7f1f0f7fe640 1 -- 172.21.15.72:0/4076678725 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0802cc60 con 0x7f1f1015bf30 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.519+0000 7f1f0f7fe640 1 -- 172.21.15.72:0/4076678725 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f08034640 con 0x7f1f1015bf30 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 -- 172.21.15.72:0/4076678725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1015bf30 msgr2=0x7f1f1015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/4076678725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1015bf30 0x7f1f1015ef60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f1f08009d90 tx=0x7f1f0802beb0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 -- 172.21.15.72:0/4076678725 shutdown_connections 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/4076678725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1015bf30 0x7f1f1015ef60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 -- 172.21.15.72:0/4076678725 >> 172.21.15.72:0/4076678725 conn(0x7f1f100cf8a0 msgr2=0x7f1f100cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 -- 172.21.15.72:0/4076678725 shutdown_connections 2024-03-06T07:23:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.520+0000 7f1f16bc9640 1 -- 172.21.15.72:0/4076678725 wait complete. 2024-03-06T07:23:30.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.521+0000 7f1f16bc9640 1 Processor -- start 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.521+0000 7f1f16bc9640 1 -- start start 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f16bc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f100b3060 0x7f1f100ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f16bc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 0x7f1f101f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f16bc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1015bf30 0x7f1f10140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f16bc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f100718e0 con 0x7f1f1013dc10 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f16bc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1015b210 con 0x7f1f1015bf30 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f16bc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f10071a60 con 0x7f1f100b3060 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f0ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 0x7f1f101f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f0ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 0x7f1f101f1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51432/0 (socket says 172.21.15.72:51432) 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f0ffff640 1 -- 172.21.15.72:0/429550432 learned_addr learned my addr 172.21.15.72:0/429550432 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f15166640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1015bf30 0x7f1f10140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f14965640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f100b3060 0x7f1f100ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f0ffff640 1 -- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f100b3060 msgr2=0x7f1f100ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f0ffff640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f100b3060 0x7f1f100ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.522+0000 7f1f0ffff640 1 -- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1015bf30 msgr2=0x7f1f10140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f0ffff640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1015bf30 0x7f1f10140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f0ffff640 1 -- 172.21.15.72:0/429550432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f101f4f70 con 0x7f1f1013dc10 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f0ffff640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 0x7f1f101f1470 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c530 tx=0x7f1ef800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8017070 con 0x7f1f1013dc10 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef8005ce0 con 0x7f1f1013dc10 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f101f5250 con 0x7f1f1013dc10 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef80072b0 con 0x7f1f1013dc10 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.523+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f101f5810 con 0x7f1f1013dc10 2024-03-06T07:23:30.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.524+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1004f330 con 0x7f1f1013dc10 2024-03-06T07:23:30.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.524+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1ef801a070 con 0x7f1f1013dc10 2024-03-06T07:23:30.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.525+0000 7f1f0d7fa640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ef006b4d0 0x7f1ef006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.525+0000 7f1f14965640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ef006b4d0 0x7f1ef006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.525+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ef8089620 con 0x7f1f1013dc10 2024-03-06T07:23:30.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.525+0000 7f1f14965640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ef006b4d0 0x7f1ef006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f00001640 tx=0x7f1f00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.529+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1ef8058560 con 0x7f1f1013dc10 2024-03-06T07:23:30.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.700+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1f1006eb00 con 0x7f1ef006b4d0 2024-03-06T07:23:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.706+0000 7f1f0d7fa640 1 -- 172.21.15.72:0/429550432 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251959 (secure 0 0 0) 0x7f1f1006eb00 con 0x7f1ef006b4d0 2024-03-06T07:23:30.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:30.711 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ef006b4d0 msgr2=0x7f1ef006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ef006b4d0 0x7f1ef006d990 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1f00001640 tx=0x7f1f00009040 comp rx=0 tx=0).stop 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 msgr2=0x7f1f101f1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 0x7f1f101f1470 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c530 tx=0x7f1ef800c9f0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 shutdown_connections 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ef006b4d0 0x7f1ef006d990 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1015bf30 0x7f1f10140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1013dc10 0x7f1f101f1470 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 --2- 172.21.15.72:0/429550432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f100b3060 0x7f1f100ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 >> 172.21.15.72:0/429550432 conn(0x7f1f100cf8a0 msgr2=0x7f1f100b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 shutdown_connections 2024-03-06T07:23:30.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.709+0000 7f1f16bc9640 1 -- 172.21.15.72:0/429550432 wait complete. 2024-03-06T07:23:30.721 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-03-06T07:23:29.965791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254888,"kb_used_data":5448,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719832,"statfs":{"total":773094113280,"available":772833107968,"internally_reserved":0,"allocated":5578752,"data_stored":2943844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001466"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878586+0000","last_change":"2024-03-06T07:23:22.882347+0000","last_active":"2024-03-06T07:23:23.878586+0000","last_peered":"2024-03-06T07:23:23.878586+0000","last_clean":"2024-03-06T07:23:23.878586+0000","last_became_active":"2024-03-06T07:23:22.882051+0000","last_became_peered":"2024-03-06T07:23:22.882051+0000","last_unstale":"2024-03-06T07:23:23.878586+0000","last_undegraded":"2024-03-06T07:23:23.878586+0000","last_fullsized":"2024-03-06T07:23:23.878586+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:29.216809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878837+0000","last_change":"2024-03-06T07:23:22.877234+0000","last_active":"2024-03-06T07:23:23.878837+0000","last_peered":"2024-03-06T07:23:23.878837+0000","last_clean":"2024-03-06T07:23:23.878837+0000","last_became_active":"2024-03-06T07:23:22.876944+0000","last_became_peered":"2024-03-06T07:23:22.876944+0000","last_unstale":"2024-03-06T07:23:23.878837+0000","last_undegraded":"2024-03-06T07:23:23.878837+0000","last_fullsized":"2024-03-06T07:23:23.878837+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:58.481828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571576+0000","last_change":"2024-03-06T07:23:22.877799+0000","last_active":"2024-03-06T07:23:24.571576+0000","last_peered":"2024-03-06T07:23:24.571576+0000","last_clean":"2024-03-06T07:23:24.571576+0000","last_became_active":"2024-03-06T07:23:22.877542+0000","last_became_peered":"2024-03-06T07:23:22.877542+0000","last_unstale":"2024-03-06T07:23:24.571576+0000","last_undegraded":"2024-03-06T07:23:24.571576+0000","last_fullsized":"2024-03-06T07:23:24.571576+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:24.855485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004900+0000","last_change":"2024-03-06T07:23:22.878477+0000","last_active":"2024-03-06T07:23:24.004900+0000","last_peered":"2024-03-06T07:23:24.004900+0000","last_clean":"2024-03-06T07:23:24.004900+0000","last_became_active":"2024-03-06T07:23:22.877685+0000","last_became_peered":"2024-03-06T07:23:22.877685+0000","last_unstale":"2024-03-06T07:23:24.004900+0000","last_undegraded":"2024-03-06T07:23:24.004900+0000","last_fullsized":"2024-03-06T07:23:24.004900+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:35.144036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878728+0000","last_change":"2024-03-06T07:23:22.868769+0000","last_active":"2024-03-06T07:23:23.878728+0000","last_peered":"2024-03-06T07:23:23.878728+0000","last_clean":"2024-03-06T07:23:23.878728+0000","last_became_active":"2024-03-06T07:23:22.868501+0000","last_became_peered":"2024-03-06T07:23:22.868501+0000","last_unstale":"2024-03-06T07:23:23.878728+0000","last_undegraded":"2024-03-06T07:23:23.878728+0000","last_fullsized":"2024-03-06T07:23:23.878728+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:30.006540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867373+0000","last_change":"2024-03-06T07:23:22.874730+0000","last_active":"2024-03-06T07:23:24.867373+0000","last_peered":"2024-03-06T07:23:24.867373+0000","last_clean":"2024-03-06T07:23:24.867373+0000","last_became_active":"2024-03-06T07:23:22.874457+0000","last_became_peered":"2024-03-06T07:23:22.874457+0000","last_unstale":"2024-03-06T07:23:24.867373+0000","last_undegraded":"2024-03-06T07:23:24.867373+0000","last_fullsized":"2024-03-06T07:23:24.867373+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:57:24.815663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878126+0000","last_change":"2024-03-06T07:23:22.877518+0000","last_active":"2024-03-06T07:23:23.878126+0000","last_peered":"2024-03-06T07:23:23.878126+0000","last_clean":"2024-03-06T07:23:23.878126+0000","last_became_active":"2024-03-06T07:23:22.876800+0000","last_became_peered":"2024-03-06T07:23:22.876800+0000","last_unstale":"2024-03-06T07:23:23.878126+0000","last_undegraded":"2024-03-06T07:23:23.878126+0000","last_fullsized":"2024-03-06T07:23:23.878126+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:49.794812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004545+0000","last_change":"2024-03-06T07:23:22.879636+0000","last_active":"2024-03-06T07:23:24.004545+0000","last_peered":"2024-03-06T07:23:24.004545+0000","last_clean":"2024-03-06T07:23:24.004545+0000","last_became_active":"2024-03-06T07:23:22.877803+0000","last_became_peered":"2024-03-06T07:23:22.877803+0000","last_unstale":"2024-03-06T07:23:24.004545+0000","last_undegraded":"2024-03-06T07:23:24.004545+0000","last_fullsized":"2024-03-06T07:23:24.004545+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:43.366190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005227+0000","last_change":"2024-03-06T07:23:22.876494+0000","last_active":"2024-03-06T07:23:24.005227+0000","last_peered":"2024-03-06T07:23:24.005227+0000","last_clean":"2024-03-06T07:23:24.005227+0000","last_became_active":"2024-03-06T07:23:22.876199+0000","last_became_peered":"2024-03-06T07:23:22.876199+0000","last_unstale":"2024-03-06T07:23:24.005227+0000","last_undegraded":"2024-03-06T07:23:24.005227+0000","last_fullsized":"2024-03-06T07:23:24.005227+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:55.751529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005165+0000","last_change":"2024-03-06T07:23:22.882019+0000","last_active":"2024-03-06T07:23:24.005165+0000","last_peered":"2024-03-06T07:23:24.005165+0000","last_clean":"2024-03-06T07:23:24.005165+0000","last_became_active":"2024-03-06T07:23:22.881760+0000","last_became_peered":"2024-03-06T07:23:22.881760+0000","last_unstale":"2024-03-06T07:23:24.005165+0000","last_undegraded":"2024-03-06T07:23:24.005165+0000","last_fullsized":"2024-03-06T07:23:24.005165+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:29.991967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005005+0000","last_change":"2024-03-06T07:23:22.881529+0000","last_active":"2024-03-06T07:23:24.005005+0000","last_peered":"2024-03-06T07:23:24.005005+0000","last_clean":"2024-03-06T07:23:24.005005+0000","last_became_active":"2024-03-06T07:23:22.881285+0000","last_became_peered":"2024-03-06T07:23:22.881285+0000","last_unstale":"2024-03-06T07:23:24.005005+0000","last_undegraded":"2024-03-06T07:23:24.005005+0000","last_fullsized":"2024-03-06T07:23:24.005005+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:04.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878959+0000","last_change":"2024-03-06T07:23:22.876461+0000","last_active":"2024-03-06T07:23:23.878959+0000","last_peered":"2024-03-06T07:23:23.878959+0000","last_clean":"2024-03-06T07:23:23.878959+0000","last_became_active":"2024-03-06T07:23:22.876206+0000","last_became_peered":"2024-03-06T07:23:22.876206+0000","last_unstale":"2024-03-06T07:23:23.878959+0000","last_undegraded":"2024-03-06T07:23:23.878959+0000","last_fullsized":"2024-03-06T07:23:23.878959+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:38.427751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874771+0000","last_change":"2024-03-06T07:23:22.887541+0000","last_active":"2024-03-06T07:23:23.874771+0000","last_peered":"2024-03-06T07:23:23.874771+0000","last_clean":"2024-03-06T07:23:23.874771+0000","last_became_active":"2024-03-06T07:23:22.886851+0000","last_became_peered":"2024-03-06T07:23:22.886851+0000","last_unstale":"2024-03-06T07:23:23.874771+0000","last_undegraded":"2024-03-06T07:23:23.874771+0000","last_fullsized":"2024-03-06T07:23:23.874771+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:43:45.877625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004701+0000","last_change":"2024-03-06T07:23:22.876901+0000","last_active":"2024-03-06T07:23:24.004701+0000","last_peered":"2024-03-06T07:23:24.004701+0000","last_clean":"2024-03-06T07:23:24.004701+0000","last_became_active":"2024-03-06T07:23:22.876653+0000","last_became_peered":"2024-03-06T07:23:22.876653+0000","last_unstale":"2024-03-06T07:23:24.004701+0000","last_undegraded":"2024-03-06T07:23:24.004701+0000","last_fullsized":"2024-03-06T07:23:24.004701+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:24.852598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867317+0000","last_change":"2024-03-06T07:23:22.885568+0000","last_active":"2024-03-06T07:23:24.867317+0000","last_peered":"2024-03-06T07:23:24.867317+0000","last_clean":"2024-03-06T07:23:24.867317+0000","last_became_active":"2024-03-06T07:23:22.885330+0000","last_became_peered":"2024-03-06T07:23:22.885330+0000","last_unstale":"2024-03-06T07:23:24.867317+0000","last_undegraded":"2024-03-06T07:23:24.867317+0000","last_fullsized":"2024-03-06T07:23:24.867317+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:21.342294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950876+0000","last_change":"2024-03-06T07:23:22.886157+0000","last_active":"2024-03-06T07:23:23.950876+0000","last_peered":"2024-03-06T07:23:23.950876+0000","last_clean":"2024-03-06T07:23:23.950876+0000","last_became_active":"2024-03-06T07:23:22.885902+0000","last_became_peered":"2024-03-06T07:23:22.885902+0000","last_unstale":"2024-03-06T07:23:23.950876+0000","last_undegraded":"2024-03-06T07:23:23.950876+0000","last_fullsized":"2024-03-06T07:23:23.950876+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:44.896627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571739+0000","last_change":"2024-03-06T07:23:22.877022+0000","last_active":"2024-03-06T07:23:24.571739+0000","last_peered":"2024-03-06T07:23:24.571739+0000","last_clean":"2024-03-06T07:23:24.571739+0000","last_became_active":"2024-03-06T07:23:22.876747+0000","last_became_peered":"2024-03-06T07:23:22.876747+0000","last_unstale":"2024-03-06T07:23:24.571739+0000","last_undegraded":"2024-03-06T07:23:24.571739+0000","last_fullsized":"2024-03-06T07:23:24.571739+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:02.793153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571733+0000","last_change":"2024-03-06T07:23:22.886786+0000","last_active":"2024-03-06T07:23:24.571733+0000","last_peered":"2024-03-06T07:23:24.571733+0000","last_clean":"2024-03-06T07:23:24.571733+0000","last_became_active":"2024-03-06T07:23:22.886537+0000","last_became_peered":"2024-03-06T07:23:22.886537+0000","last_unstale":"2024-03-06T07:23:24.571733+0000","last_undegraded":"2024-03-06T07:23:24.571733+0000","last_fullsized":"2024-03-06T07:23:24.571733+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:05.977485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950917+0000","last_change":"2024-03-06T07:23:22.887978+0000","last_active":"2024-03-06T07:23:23.950917+0000","last_peered":"2024-03-06T07:23:23.950917+0000","last_clean":"2024-03-06T07:23:23.950917+0000","last_became_active":"2024-03-06T07:23:22.887710+0000","last_became_peered":"2024-03-06T07:23:22.887710+0000","last_unstale":"2024-03-06T07:23:23.950917+0000","last_undegraded":"2024-03-06T07:23:23.950917+0000","last_fullsized":"2024-03-06T07:23:23.950917+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:14.752947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874884+0000","last_change":"2024-03-06T07:23:22.889564+0000","last_active":"2024-03-06T07:23:23.874884+0000","last_peered":"2024-03-06T07:23:23.874884+0000","last_clean":"2024-03-06T07:23:23.874884+0000","last_became_active":"2024-03-06T07:23:22.889308+0000","last_became_peered":"2024-03-06T07:23:22.889308+0000","last_unstale":"2024-03-06T07:23:23.874884+0000","last_undegraded":"2024-03-06T07:23:23.874884+0000","last_fullsized":"2024-03-06T07:23:23.874884+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:35.423703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867756+0000","last_change":"2024-03-06T07:23:22.874948+0000","last_active":"2024-03-06T07:23:24.867756+0000","last_peered":"2024-03-06T07:23:24.867756+0000","last_clean":"2024-03-06T07:23:24.867756+0000","last_became_active":"2024-03-06T07:23:22.874583+0000","last_became_peered":"2024-03-06T07:23:22.874583+0000","last_unstale":"2024-03-06T07:23:24.867756+0000","last_undegraded":"2024-03-06T07:23:24.867756+0000","last_fullsized":"2024-03-06T07:23:24.867756+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:00.206462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004776+0000","last_change":"2024-03-06T07:23:22.878916+0000","last_active":"2024-03-06T07:23:24.004776+0000","last_peered":"2024-03-06T07:23:24.004776+0000","last_clean":"2024-03-06T07:23:24.004776+0000","last_became_active":"2024-03-06T07:23:22.878658+0000","last_became_peered":"2024-03-06T07:23:22.878658+0000","last_unstale":"2024-03-06T07:23:24.004776+0000","last_undegraded":"2024-03-06T07:23:24.004776+0000","last_fullsized":"2024-03-06T07:23:24.004776+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:58.481828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950683+0000","last_change":"2024-03-06T07:23:22.879875+0000","last_active":"2024-03-06T07:23:23.950683+0000","last_peered":"2024-03-06T07:23:23.950683+0000","last_clean":"2024-03-06T07:23:23.950683+0000","last_became_active":"2024-03-06T07:23:22.879560+0000","last_became_peered":"2024-03-06T07:23:22.879560+0000","last_unstale":"2024-03-06T07:23:23.950683+0000","last_undegraded":"2024-03-06T07:23:23.950683+0000","last_fullsized":"2024-03-06T07:23:23.950683+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:45.867001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950527+0000","last_change":"2024-03-06T07:23:22.880417+0000","last_active":"2024-03-06T07:23:23.950527+0000","last_peered":"2024-03-06T07:23:23.950527+0000","last_clean":"2024-03-06T07:23:23.950527+0000","last_became_active":"2024-03-06T07:23:22.880005+0000","last_became_peered":"2024-03-06T07:23:22.880005+0000","last_unstale":"2024-03-06T07:23:23.950527+0000","last_undegraded":"2024-03-06T07:23:23.950527+0000","last_fullsized":"2024-03-06T07:23:23.950527+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:25.147586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950935+0000","last_change":"2024-03-06T07:23:22.881840+0000","last_active":"2024-03-06T07:23:23.950935+0000","last_peered":"2024-03-06T07:23:23.950935+0000","last_clean":"2024-03-06T07:23:23.950935+0000","last_became_active":"2024-03-06T07:23:22.881321+0000","last_became_peered":"2024-03-06T07:23:22.881321+0000","last_unstale":"2024-03-06T07:23:23.950935+0000","last_undegraded":"2024-03-06T07:23:23.950935+0000","last_fullsized":"2024-03-06T07:23:23.950935+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:15.502783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874894+0000","last_change":"2024-03-06T07:23:22.886951+0000","last_active":"2024-03-06T07:23:23.874894+0000","last_peered":"2024-03-06T07:23:23.874894+0000","last_clean":"2024-03-06T07:23:23.874894+0000","last_became_active":"2024-03-06T07:23:22.885858+0000","last_became_peered":"2024-03-06T07:23:22.885858+0000","last_unstale":"2024-03-06T07:23:23.874894+0000","last_undegraded":"2024-03-06T07:23:23.874894+0000","last_fullsized":"2024-03-06T07:23:23.874894+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:17.537245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571663+0000","last_change":"2024-03-06T07:23:22.876914+0000","last_active":"2024-03-06T07:23:24.571663+0000","last_peered":"2024-03-06T07:23:24.571663+0000","last_clean":"2024-03-06T07:23:24.571663+0000","last_became_active":"2024-03-06T07:23:22.876613+0000","last_became_peered":"2024-03-06T07:23:22.876613+0000","last_unstale":"2024-03-06T07:23:24.571663+0000","last_undegraded":"2024-03-06T07:23:24.571663+0000","last_fullsized":"2024-03-06T07:23:24.571663+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:41.546767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.879114+0000","last_change":"2024-03-06T07:23:22.875780+0000","last_active":"2024-03-06T07:23:23.879114+0000","last_peered":"2024-03-06T07:23:23.879114+0000","last_clean":"2024-03-06T07:23:23.879114+0000","last_became_active":"2024-03-06T07:23:22.875360+0000","last_became_peered":"2024-03-06T07:23:22.875360+0000","last_unstale":"2024-03-06T07:23:23.879114+0000","last_undegraded":"2024-03-06T07:23:23.879114+0000","last_fullsized":"2024-03-06T07:23:23.879114+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:35.144036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950707+0000","last_change":"2024-03-06T07:23:22.878252+0000","last_active":"2024-03-06T07:23:23.950707+0000","last_peered":"2024-03-06T07:23:23.950707+0000","last_clean":"2024-03-06T07:23:23.950707+0000","last_became_active":"2024-03-06T07:23:22.877942+0000","last_became_peered":"2024-03-06T07:23:22.877942+0000","last_unstale":"2024-03-06T07:23:23.950707+0000","last_undegraded":"2024-03-06T07:23:23.950707+0000","last_fullsized":"2024-03-06T07:23:23.950707+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:47.613668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571544+0000","last_change":"2024-03-06T07:23:22.883992+0000","last_active":"2024-03-06T07:23:24.571544+0000","last_peered":"2024-03-06T07:23:24.571544+0000","last_clean":"2024-03-06T07:23:24.571544+0000","last_became_active":"2024-03-06T07:23:22.883683+0000","last_became_peered":"2024-03-06T07:23:22.883683+0000","last_unstale":"2024-03-06T07:23:24.571544+0000","last_undegraded":"2024-03-06T07:23:24.571544+0000","last_fullsized":"2024-03-06T07:23:24.571544+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:01.846061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867379+0000","last_change":"2024-03-06T07:23:22.880395+0000","last_active":"2024-03-06T07:23:24.867379+0000","last_peered":"2024-03-06T07:23:24.867379+0000","last_clean":"2024-03-06T07:23:24.867379+0000","last_became_active":"2024-03-06T07:23:22.880118+0000","last_became_peered":"2024-03-06T07:23:22.880118+0000","last_unstale":"2024-03-06T07:23:24.867379+0000","last_undegraded":"2024-03-06T07:23:24.867379+0000","last_fullsized":"2024-03-06T07:23:24.867379+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:00.878652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874703+0000","last_change":"2024-03-06T07:23:22.877493+0000","last_active":"2024-03-06T07:23:23.874703+0000","last_peered":"2024-03-06T07:23:23.874703+0000","last_clean":"2024-03-06T07:23:23.874703+0000","last_became_active":"2024-03-06T07:23:22.877208+0000","last_became_peered":"2024-03-06T07:23:22.877208+0000","last_unstale":"2024-03-06T07:23:23.874703+0000","last_undegraded":"2024-03-06T07:23:23.874703+0000","last_fullsized":"2024-03-06T07:23:23.874703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:33:03.334957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.196709+0000","last_change":"2024-03-06T07:23:22.889894+0000","last_active":"2024-03-06T07:23:24.196709+0000","last_peered":"2024-03-06T07:23:24.196709+0000","last_clean":"2024-03-06T07:23:24.196709+0000","last_became_active":"2024-03-06T07:23:22.889625+0000","last_became_peered":"2024-03-06T07:23:22.889625+0000","last_unstale":"2024-03-06T07:23:24.196709+0000","last_undegraded":"2024-03-06T07:23:24.196709+0000","last_fullsized":"2024-03-06T07:23:24.196709+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:16.564140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"45'4","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867715+0000","last_change":"2024-03-06T07:23:21.870976+0000","last_active":"2024-03-06T07:23:24.867715+0000","last_peered":"2024-03-06T07:23:24.867715+0000","last_clean":"2024-03-06T07:23:24.867715+0000","last_became_active":"2024-03-06T07:23:21.870706+0000","last_became_peered":"2024-03-06T07:23:21.870706+0000","last_unstale":"2024-03-06T07:23:24.867715+0000","last_undegraded":"2024-03-06T07:23:24.867715+0000","last_fullsized":"2024-03-06T07:23:24.867715+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:23.843604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950970+0000","last_change":"2024-03-06T07:23:22.885967+0000","last_active":"2024-03-06T07:23:23.950970+0000","last_peered":"2024-03-06T07:23:23.950970+0000","last_clean":"2024-03-06T07:23:23.950970+0000","last_became_active":"2024-03-06T07:23:22.885433+0000","last_became_peered":"2024-03-06T07:23:22.885433+0000","last_unstale":"2024-03-06T07:23:23.950970+0000","last_undegraded":"2024-03-06T07:23:23.950970+0000","last_fullsized":"2024-03-06T07:23:23.950970+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:55:35.724689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571826+0000","last_change":"2024-03-06T07:23:21.864644+0000","last_active":"2024-03-06T07:23:24.571826+0000","last_peered":"2024-03-06T07:23:24.571826+0000","last_clean":"2024-03-06T07:23:24.571826+0000","last_became_active":"2024-03-06T07:23:21.864375+0000","last_became_peered":"2024-03-06T07:23:21.864375+0000","last_unstale":"2024-03-06T07:23:24.571826+0000","last_undegraded":"2024-03-06T07:23:24.571826+0000","last_fullsized":"2024-03-06T07:23:24.571826+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:06.179025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.879161+0000","last_change":"2024-03-06T07:23:22.877865+0000","last_active":"2024-03-06T07:23:23.879161+0000","last_peered":"2024-03-06T07:23:23.879161+0000","last_clean":"2024-03-06T07:23:23.879161+0000","last_became_active":"2024-03-06T07:23:22.877542+0000","last_became_peered":"2024-03-06T07:23:22.877542+0000","last_unstale":"2024-03-06T07:23:23.879161+0000","last_undegraded":"2024-03-06T07:23:23.879161+0000","last_fullsized":"2024-03-06T07:23:23.879161+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:04.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.196656+0000","last_change":"2024-03-06T07:23:21.868746+0000","last_active":"2024-03-06T07:23:24.196656+0000","last_peered":"2024-03-06T07:23:24.196656+0000","last_clean":"2024-03-06T07:23:24.196656+0000","last_became_active":"2024-03-06T07:23:21.868455+0000","last_became_peered":"2024-03-06T07:23:21.868455+0000","last_unstale":"2024-03-06T07:23:24.196656+0000","last_undegraded":"2024-03-06T07:23:24.196656+0000","last_fullsized":"2024-03-06T07:23:24.196656+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:43.887633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.196710+0000","last_change":"2024-03-06T07:23:22.886029+0000","last_active":"2024-03-06T07:23:24.196710+0000","last_peered":"2024-03-06T07:23:24.196710+0000","last_clean":"2024-03-06T07:23:24.196710+0000","last_became_active":"2024-03-06T07:23:22.885774+0000","last_became_peered":"2024-03-06T07:23:22.885774+0000","last_unstale":"2024-03-06T07:23:24.196710+0000","last_undegraded":"2024-03-06T07:23:24.196710+0000","last_fullsized":"2024-03-06T07:23:24.196710+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:24.855485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.877998+0000","last_change":"2024-03-06T07:23:21.865942+0000","last_active":"2024-03-06T07:23:23.877998+0000","last_peered":"2024-03-06T07:23:23.877998+0000","last_clean":"2024-03-06T07:23:23.877998+0000","last_became_active":"2024-03-06T07:23:21.865642+0000","last_became_peered":"2024-03-06T07:23:21.865642+0000","last_unstale":"2024-03-06T07:23:23.877998+0000","last_undegraded":"2024-03-06T07:23:23.877998+0000","last_fullsized":"2024-03-06T07:23:23.877998+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:10.194391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571529+0000","last_change":"2024-03-06T07:23:22.887518+0000","last_active":"2024-03-06T07:23:24.571529+0000","last_peered":"2024-03-06T07:23:24.571529+0000","last_clean":"2024-03-06T07:23:24.571529+0000","last_became_active":"2024-03-06T07:23:22.887236+0000","last_became_peered":"2024-03-06T07:23:22.887236+0000","last_unstale":"2024-03-06T07:23:24.571529+0000","last_undegraded":"2024-03-06T07:23:24.571529+0000","last_fullsized":"2024-03-06T07:23:24.571529+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:44.063117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867587+0000","last_change":"2024-03-06T07:23:21.872744+0000","last_active":"2024-03-06T07:23:24.867587+0000","last_peered":"2024-03-06T07:23:24.867587+0000","last_clean":"2024-03-06T07:23:24.867587+0000","last_became_active":"2024-03-06T07:23:21.872444+0000","last_became_peered":"2024-03-06T07:23:21.872444+0000","last_unstale":"2024-03-06T07:23:24.867587+0000","last_undegraded":"2024-03-06T07:23:24.867587+0000","last_fullsized":"2024-03-06T07:23:24.867587+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:04.968492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004759+0000","last_change":"2024-03-06T07:23:22.879986+0000","last_active":"2024-03-06T07:23:24.004759+0000","last_peered":"2024-03-06T07:23:24.004759+0000","last_clean":"2024-03-06T07:23:24.004759+0000","last_became_active":"2024-03-06T07:23:22.879742+0000","last_became_peered":"2024-03-06T07:23:22.879742+0000","last_unstale":"2024-03-06T07:23:24.004759+0000","last_undegraded":"2024-03-06T07:23:24.004759+0000","last_fullsized":"2024-03-06T07:23:24.004759+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:30.006540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571742+0000","last_change":"2024-03-06T07:23:21.868824+0000","last_active":"2024-03-06T07:23:24.571742+0000","last_peered":"2024-03-06T07:23:24.571742+0000","last_clean":"2024-03-06T07:23:24.571742+0000","last_became_active":"2024-03-06T07:23:21.867672+0000","last_became_peered":"2024-03-06T07:23:21.867672+0000","last_unstale":"2024-03-06T07:23:24.571742+0000","last_undegraded":"2024-03-06T07:23:24.571742+0000","last_fullsized":"2024-03-06T07:23:24.571742+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:44.858008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004621+0000","last_change":"2024-03-06T07:23:22.878056+0000","last_active":"2024-03-06T07:23:24.004621+0000","last_peered":"2024-03-06T07:23:24.004621+0000","last_clean":"2024-03-06T07:23:24.004621+0000","last_became_active":"2024-03-06T07:23:22.877480+0000","last_became_peered":"2024-03-06T07:23:22.877480+0000","last_unstale":"2024-03-06T07:23:24.004621+0000","last_undegraded":"2024-03-06T07:23:24.004621+0000","last_fullsized":"2024-03-06T07:23:24.004621+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:02.793153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878542+0000","last_change":"2024-03-06T07:23:21.863472+0000","last_active":"2024-03-06T07:23:23.878542+0000","last_peered":"2024-03-06T07:23:23.878542+0000","last_clean":"2024-03-06T07:23:23.878542+0000","last_became_active":"2024-03-06T07:23:21.863161+0000","last_became_peered":"2024-03-06T07:23:21.863161+0000","last_unstale":"2024-03-06T07:23:23.878542+0000","last_undegraded":"2024-03-06T07:23:23.878542+0000","last_fullsized":"2024-03-06T07:23:23.878542+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:03.038755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950470+0000","last_change":"2024-03-06T07:23:22.888803+0000","last_active":"2024-03-06T07:23:23.950470+0000","last_peered":"2024-03-06T07:23:23.950470+0000","last_clean":"2024-03-06T07:23:23.950470+0000","last_became_active":"2024-03-06T07:23:22.888539+0000","last_became_peered":"2024-03-06T07:23:22.888539+0000","last_unstale":"2024-03-06T07:23:23.950470+0000","last_undegraded":"2024-03-06T07:23:23.950470+0000","last_fullsized":"2024-03-06T07:23:23.950470+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:07.188018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004640+0000","last_change":"2024-03-06T07:23:21.864598+0000","last_active":"2024-03-06T07:23:24.004640+0000","last_peered":"2024-03-06T07:23:24.004640+0000","last_clean":"2024-03-06T07:23:24.004640+0000","last_became_active":"2024-03-06T07:23:21.862773+0000","last_became_peered":"2024-03-06T07:23:21.862773+0000","last_unstale":"2024-03-06T07:23:24.004640+0000","last_undegraded":"2024-03-06T07:23:24.004640+0000","last_fullsized":"2024-03-06T07:23:24.004640+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:00.261574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571707+0000","last_change":"2024-03-06T07:23:22.886135+0000","last_active":"2024-03-06T07:23:24.571707+0000","last_peered":"2024-03-06T07:23:24.571707+0000","last_clean":"2024-03-06T07:23:24.571707+0000","last_became_active":"2024-03-06T07:23:22.885822+0000","last_became_peered":"2024-03-06T07:23:22.885822+0000","last_unstale":"2024-03-06T07:23:24.571707+0000","last_undegraded":"2024-03-06T07:23:24.571707+0000","last_fullsized":"2024-03-06T07:23:24.571707+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:43.366190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571859+0000","last_change":"2024-03-06T07:23:21.867046+0000","last_active":"2024-03-06T07:23:24.571859+0000","last_peered":"2024-03-06T07:23:24.571859+0000","last_clean":"2024-03-06T07:23:24.571859+0000","last_became_active":"2024-03-06T07:23:21.866751+0000","last_became_peered":"2024-03-06T07:23:21.866751+0000","last_unstale":"2024-03-06T07:23:24.571859+0000","last_undegraded":"2024-03-06T07:23:24.571859+0000","last_fullsized":"2024-03-06T07:23:24.571859+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:06:00.261574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878750+0000","last_change":"2024-03-06T07:23:22.885248+0000","last_active":"2024-03-06T07:23:23.878750+0000","last_peered":"2024-03-06T07:23:23.878750+0000","last_clean":"2024-03-06T07:23:23.878750+0000","last_became_active":"2024-03-06T07:23:22.884932+0000","last_became_peered":"2024-03-06T07:23:22.884932+0000","last_unstale":"2024-03-06T07:23:23.878750+0000","last_undegraded":"2024-03-06T07:23:23.878750+0000","last_fullsized":"2024-03-06T07:23:23.878750+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:15.331415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005378+0000","last_change":"2024-03-06T07:23:21.869633+0000","last_active":"2024-03-06T07:23:24.005378+0000","last_peered":"2024-03-06T07:23:24.005378+0000","last_clean":"2024-03-06T07:23:24.005378+0000","last_became_active":"2024-03-06T07:23:21.869349+0000","last_became_peered":"2024-03-06T07:23:21.869349+0000","last_unstale":"2024-03-06T07:23:24.005378+0000","last_undegraded":"2024-03-06T07:23:24.005378+0000","last_fullsized":"2024-03-06T07:23:24.005378+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.850435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867600+0000","last_change":"2024-03-06T07:23:22.884456+0000","last_active":"2024-03-06T07:23:24.867600+0000","last_peered":"2024-03-06T07:23:24.867600+0000","last_clean":"2024-03-06T07:23:24.867600+0000","last_became_active":"2024-03-06T07:23:22.884200+0000","last_became_peered":"2024-03-06T07:23:22.884200+0000","last_unstale":"2024-03-06T07:23:24.867600+0000","last_undegraded":"2024-03-06T07:23:24.867600+0000","last_fullsized":"2024-03-06T07:23:24.867600+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:38.044584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"45'3","reported_seq":17,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878364+0000","last_change":"2024-03-06T07:23:21.868933+0000","last_active":"2024-03-06T07:23:23.878364+0000","last_peered":"2024-03-06T07:23:23.878364+0000","last_clean":"2024-03-06T07:23:23.878364+0000","last_became_active":"2024-03-06T07:23:21.868305+0000","last_became_peered":"2024-03-06T07:23:21.868305+0000","last_unstale":"2024-03-06T07:23:23.878364+0000","last_undegraded":"2024-03-06T07:23:23.878364+0000","last_fullsized":"2024-03-06T07:23:23.878364+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:40.537774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878078+0000","last_change":"2024-03-06T07:23:22.882876+0000","last_active":"2024-03-06T07:23:23.878078+0000","last_peered":"2024-03-06T07:23:23.878078+0000","last_clean":"2024-03-06T07:23:23.878078+0000","last_became_active":"2024-03-06T07:23:22.882606+0000","last_became_peered":"2024-03-06T07:23:22.882606+0000","last_unstale":"2024-03-06T07:23:23.878078+0000","last_undegraded":"2024-03-06T07:23:23.878078+0000","last_fullsized":"2024-03-06T07:23:23.878078+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:04.231248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"45'6","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.875023+0000","last_change":"2024-03-06T07:23:21.862722+0000","last_active":"2024-03-06T07:23:23.875023+0000","last_peered":"2024-03-06T07:23:23.875023+0000","last_clean":"2024-03-06T07:23:23.875023+0000","last_became_active":"2024-03-06T07:23:21.862470+0000","last_became_peered":"2024-03-06T07:23:21.862470+0000","last_unstale":"2024-03-06T07:23:23.875023+0000","last_undegraded":"2024-03-06T07:23:23.875023+0000","last_fullsized":"2024-03-06T07:23:23.875023+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:21.900934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950813+0000","last_change":"2024-03-06T07:23:22.885004+0000","last_active":"2024-03-06T07:23:23.950813+0000","last_peered":"2024-03-06T07:23:23.950813+0000","last_clean":"2024-03-06T07:23:23.950813+0000","last_became_active":"2024-03-06T07:23:22.883994+0000","last_became_peered":"2024-03-06T07:23:22.883994+0000","last_unstale":"2024-03-06T07:23:23.950813+0000","last_undegraded":"2024-03-06T07:23:23.950813+0000","last_fullsized":"2024-03-06T07:23:23.950813+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:29.491922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867681+0000","last_change":"2024-03-06T07:23:21.875315+0000","last_active":"2024-03-06T07:23:24.867681+0000","last_peered":"2024-03-06T07:23:24.867681+0000","last_clean":"2024-03-06T07:23:24.867681+0000","last_became_active":"2024-03-06T07:23:21.874938+0000","last_became_peered":"2024-03-06T07:23:21.874938+0000","last_unstale":"2024-03-06T07:23:24.867681+0000","last_undegraded":"2024-03-06T07:23:24.867681+0000","last_fullsized":"2024-03-06T07:23:24.867681+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:25:28.997547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004887+0000","last_change":"2024-03-06T07:23:22.879385+0000","last_active":"2024-03-06T07:23:24.004887+0000","last_peered":"2024-03-06T07:23:24.004887+0000","last_clean":"2024-03-06T07:23:24.004887+0000","last_became_active":"2024-03-06T07:23:22.879131+0000","last_became_peered":"2024-03-06T07:23:22.879131+0000","last_unstale":"2024-03-06T07:23:24.004887+0000","last_undegraded":"2024-03-06T07:23:24.004887+0000","last_fullsized":"2024-03-06T07:23:24.004887+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:38.427751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878297+0000","last_change":"2024-03-06T07:23:21.869035+0000","last_active":"2024-03-06T07:23:23.878297+0000","last_peered":"2024-03-06T07:23:23.878297+0000","last_clean":"2024-03-06T07:23:23.878297+0000","last_became_active":"2024-03-06T07:23:21.868739+0000","last_became_peered":"2024-03-06T07:23:21.868739+0000","last_unstale":"2024-03-06T07:23:23.878297+0000","last_undegraded":"2024-03-06T07:23:23.878297+0000","last_fullsized":"2024-03-06T07:23:23.878297+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:00.837068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867304+0000","last_change":"2024-03-06T07:23:22.881600+0000","last_active":"2024-03-06T07:23:24.867304+0000","last_peered":"2024-03-06T07:23:24.867304+0000","last_clean":"2024-03-06T07:23:24.867304+0000","last_became_active":"2024-03-06T07:23:22.881328+0000","last_became_peered":"2024-03-06T07:23:22.881328+0000","last_unstale":"2024-03-06T07:23:24.867304+0000","last_undegraded":"2024-03-06T07:23:24.867304+0000","last_fullsized":"2024-03-06T07:23:24.867304+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:17:53.235280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867419+0000","last_change":"2024-03-06T07:23:21.868680+0000","last_active":"2024-03-06T07:23:24.867419+0000","last_peered":"2024-03-06T07:23:24.867419+0000","last_clean":"2024-03-06T07:23:24.867419+0000","last_became_active":"2024-03-06T07:23:21.868416+0000","last_became_peered":"2024-03-06T07:23:21.868416+0000","last_unstale":"2024-03-06T07:23:24.867419+0000","last_undegraded":"2024-03-06T07:23:24.867419+0000","last_fullsized":"2024-03-06T07:23:24.867419+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:30:40.537774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878166+0000","last_change":"2024-03-06T07:23:22.889118+0000","last_active":"2024-03-06T07:23:23.878166+0000","last_peered":"2024-03-06T07:23:23.878166+0000","last_clean":"2024-03-06T07:23:23.878166+0000","last_became_active":"2024-03-06T07:23:22.888618+0000","last_became_peered":"2024-03-06T07:23:22.888618+0000","last_unstale":"2024-03-06T07:23:23.878166+0000","last_undegraded":"2024-03-06T07:23:23.878166+0000","last_fullsized":"2024-03-06T07:23:23.878166+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:29.991967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"45'8","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004717+0000","last_change":"2024-03-06T07:23:21.870601+0000","last_active":"2024-03-06T07:23:24.004717+0000","last_peered":"2024-03-06T07:23:24.004717+0000","last_clean":"2024-03-06T07:23:24.004717+0000","last_became_active":"2024-03-06T07:23:21.870306+0000","last_became_peered":"2024-03-06T07:23:21.870306+0000","last_unstale":"2024-03-06T07:23:24.004717+0000","last_undegraded":"2024-03-06T07:23:24.004717+0000","last_fullsized":"2024-03-06T07:23:24.004717+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:24.138593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874938+0000","last_change":"2024-03-06T07:23:22.880325+0000","last_active":"2024-03-06T07:23:23.874938+0000","last_peered":"2024-03-06T07:23:23.874938+0000","last_clean":"2024-03-06T07:23:23.874938+0000","last_became_active":"2024-03-06T07:23:22.879777+0000","last_became_peered":"2024-03-06T07:23:22.879777+0000","last_unstale":"2024-03-06T07:23:23.874938+0000","last_undegraded":"2024-03-06T07:23:23.874938+0000","last_fullsized":"2024-03-06T07:23:23.874938+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:50:26.986046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874927+0000","last_change":"2024-03-06T07:23:21.872671+0000","last_active":"2024-03-06T07:23:23.874927+0000","last_peered":"2024-03-06T07:23:23.874927+0000","last_clean":"2024-03-06T07:23:23.874927+0000","last_became_active":"2024-03-06T07:23:21.872330+0000","last_became_peered":"2024-03-06T07:23:21.872330+0000","last_unstale":"2024-03-06T07:23:23.874927+0000","last_undegraded":"2024-03-06T07:23:23.874927+0000","last_fullsized":"2024-03-06T07:23:23.874927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:55:09.970373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005358+0000","last_change":"2024-03-06T07:23:22.879435+0000","last_active":"2024-03-06T07:23:24.005358+0000","last_peered":"2024-03-06T07:23:24.005358+0000","last_clean":"2024-03-06T07:23:24.005358+0000","last_became_active":"2024-03-06T07:23:22.879173+0000","last_became_peered":"2024-03-06T07:23:22.879173+0000","last_unstale":"2024-03-06T07:23:24.005358+0000","last_undegraded":"2024-03-06T07:23:24.005358+0000","last_fullsized":"2024-03-06T07:23:24.005358+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:15.331415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571828+0000","last_change":"2024-03-06T07:23:21.870771+0000","last_active":"2024-03-06T07:23:24.571828+0000","last_peered":"2024-03-06T07:23:24.571828+0000","last_clean":"2024-03-06T07:23:24.571828+0000","last_became_active":"2024-03-06T07:23:21.870450+0000","last_became_peered":"2024-03-06T07:23:21.870450+0000","last_unstale":"2024-03-06T07:23:24.571828+0000","last_undegraded":"2024-03-06T07:23:24.571828+0000","last_fullsized":"2024-03-06T07:23:24.571828+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:24.138593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950737+0000","last_change":"2024-03-06T07:23:22.877885+0000","last_active":"2024-03-06T07:23:23.950737+0000","last_peered":"2024-03-06T07:23:23.950737+0000","last_clean":"2024-03-06T07:23:23.950737+0000","last_became_active":"2024-03-06T07:23:22.877602+0000","last_became_peered":"2024-03-06T07:23:22.877602+0000","last_unstale":"2024-03-06T07:23:23.950737+0000","last_undegraded":"2024-03-06T07:23:23.950737+0000","last_fullsized":"2024-03-06T07:23:23.950737+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:27.859429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"45'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.879130+0000","last_change":"2024-03-06T07:23:21.865980+0000","last_active":"2024-03-06T07:23:23.879130+0000","last_peered":"2024-03-06T07:23:23.879130+0000","last_clean":"2024-03-06T07:23:23.879130+0000","last_became_active":"2024-03-06T07:23:21.865586+0000","last_became_peered":"2024-03-06T07:23:21.865586+0000","last_unstale":"2024-03-06T07:23:23.879130+0000","last_undegraded":"2024-03-06T07:23:23.879130+0000","last_fullsized":"2024-03-06T07:23:23.879130+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:00.837068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.950889+0000","last_change":"2024-03-06T07:23:22.882124+0000","last_active":"2024-03-06T07:23:23.950889+0000","last_peered":"2024-03-06T07:23:23.950889+0000","last_clean":"2024-03-06T07:23:23.950889+0000","last_became_active":"2024-03-06T07:23:22.880910+0000","last_became_peered":"2024-03-06T07:23:22.880910+0000","last_unstale":"2024-03-06T07:23:23.950889+0000","last_undegraded":"2024-03-06T07:23:23.950889+0000","last_fullsized":"2024-03-06T07:23:23.950889+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:43.363230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867539+0000","last_change":"2024-03-06T07:23:21.866832+0000","last_active":"2024-03-06T07:23:24.867539+0000","last_peered":"2024-03-06T07:23:24.867539+0000","last_clean":"2024-03-06T07:23:24.867539+0000","last_became_active":"2024-03-06T07:23:21.866546+0000","last_became_peered":"2024-03-06T07:23:21.866546+0000","last_unstale":"2024-03-06T07:23:24.867539+0000","last_undegraded":"2024-03-06T07:23:24.867539+0000","last_fullsized":"2024-03-06T07:23:24.867539+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:25:57.472835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867404+0000","last_change":"2024-03-06T07:23:22.881440+0000","last_active":"2024-03-06T07:23:24.867404+0000","last_peered":"2024-03-06T07:23:24.867404+0000","last_clean":"2024-03-06T07:23:24.867404+0000","last_became_active":"2024-03-06T07:23:22.880981+0000","last_became_peered":"2024-03-06T07:23:22.880981+0000","last_unstale":"2024-03-06T07:23:24.867404+0000","last_undegraded":"2024-03-06T07:23:24.867404+0000","last_fullsized":"2024-03-06T07:23:24.867404+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:14:44.390351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867839+0000","last_change":"2024-03-06T07:23:21.867500+0000","last_active":"2024-03-06T07:23:24.867839+0000","last_peered":"2024-03-06T07:23:24.867839+0000","last_clean":"2024-03-06T07:23:24.867839+0000","last_became_active":"2024-03-06T07:23:21.867223+0000","last_became_peered":"2024-03-06T07:23:21.867223+0000","last_unstale":"2024-03-06T07:23:24.867839+0000","last_undegraded":"2024-03-06T07:23:24.867839+0000","last_fullsized":"2024-03-06T07:23:24.867839+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:39:01.784160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.196536+0000","last_change":"2024-03-06T07:23:22.881430+0000","last_active":"2024-03-06T07:23:24.196536+0000","last_peered":"2024-03-06T07:23:24.196536+0000","last_clean":"2024-03-06T07:23:24.196536+0000","last_became_active":"2024-03-06T07:23:22.881185+0000","last_became_peered":"2024-03-06T07:23:22.881185+0000","last_unstale":"2024-03-06T07:23:24.196536+0000","last_undegraded":"2024-03-06T07:23:24.196536+0000","last_fullsized":"2024-03-06T07:23:24.196536+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:04.047748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878657+0000","last_change":"2024-03-06T07:23:21.866528+0000","last_active":"2024-03-06T07:23:23.878657+0000","last_peered":"2024-03-06T07:23:23.878657+0000","last_clean":"2024-03-06T07:23:23.878657+0000","last_became_active":"2024-03-06T07:23:21.866281+0000","last_became_peered":"2024-03-06T07:23:21.866281+0000","last_unstale":"2024-03-06T07:23:23.878657+0000","last_undegraded":"2024-03-06T07:23:23.878657+0000","last_fullsized":"2024-03-06T07:23:23.878657+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:32:15.555147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.874703+0000","last_change":"2024-03-06T07:23:22.883684+0000","last_active":"2024-03-06T07:23:23.874703+0000","last_peered":"2024-03-06T07:23:23.874703+0000","last_clean":"2024-03-06T07:23:23.874703+0000","last_became_active":"2024-03-06T07:23:22.883354+0000","last_became_peered":"2024-03-06T07:23:22.883354+0000","last_unstale":"2024-03-06T07:23:23.874703+0000","last_undegraded":"2024-03-06T07:23:23.874703+0000","last_fullsized":"2024-03-06T07:23:23.874703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:31:43.094936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004680+0000","last_change":"2024-03-06T07:23:21.870289+0000","last_active":"2024-03-06T07:23:24.004680+0000","last_peered":"2024-03-06T07:23:24.004680+0000","last_clean":"2024-03-06T07:23:24.004680+0000","last_became_active":"2024-03-06T07:23:21.870037+0000","last_became_peered":"2024-03-06T07:23:21.870037+0000","last_unstale":"2024-03-06T07:23:24.004680+0000","last_undegraded":"2024-03-06T07:23:24.004680+0000","last_fullsized":"2024-03-06T07:23:24.004680+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:56:06.179025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004574+0000","last_change":"2024-03-06T07:23:22.875532+0000","last_active":"2024-03-06T07:23:24.004574+0000","last_peered":"2024-03-06T07:23:24.004574+0000","last_clean":"2024-03-06T07:23:24.004574+0000","last_became_active":"2024-03-06T07:23:22.875279+0000","last_became_peered":"2024-03-06T07:23:22.875279+0000","last_unstale":"2024-03-06T07:23:24.004574+0000","last_undegraded":"2024-03-06T07:23:24.004574+0000","last_fullsized":"2024-03-06T07:23:24.004574+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:05.977485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.877882+0000","last_change":"2024-03-06T07:23:21.869803+0000","last_active":"2024-03-06T07:23:23.877882+0000","last_peered":"2024-03-06T07:23:23.877882+0000","last_clean":"2024-03-06T07:23:23.877882+0000","last_became_active":"2024-03-06T07:23:21.869511+0000","last_became_peered":"2024-03-06T07:23:21.869511+0000","last_unstale":"2024-03-06T07:23:23.877882+0000","last_undegraded":"2024-03-06T07:23:23.877882+0000","last_fullsized":"2024-03-06T07:23:23.877882+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:04.968492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.951070+0000","last_change":"2024-03-06T07:23:22.889653+0000","last_active":"2024-03-06T07:23:23.951070+0000","last_peered":"2024-03-06T07:23:23.951070+0000","last_clean":"2024-03-06T07:23:23.951070+0000","last_became_active":"2024-03-06T07:23:22.888821+0000","last_became_peered":"2024-03-06T07:23:22.888821+0000","last_unstale":"2024-03-06T07:23:23.951070+0000","last_undegraded":"2024-03-06T07:23:23.951070+0000","last_fullsized":"2024-03-06T07:23:23.951070+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:39:04.047748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"46'1","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867666+0000","last_change":"2024-03-06T07:23:21.871570+0000","last_active":"2024-03-06T07:23:24.867666+0000","last_peered":"2024-03-06T07:23:24.867666+0000","last_clean":"2024-03-06T07:23:24.867666+0000","last_became_active":"2024-03-06T07:23:21.871278+0000","last_became_peered":"2024-03-06T07:23:21.871278+0000","last_unstale":"2024-03-06T07:23:24.867666+0000","last_undegraded":"2024-03-06T07:23:24.867666+0000","last_fullsized":"2024-03-06T07:23:24.867666+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:42.357197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005061+0000","last_change":"2024-03-06T07:23:22.877950+0000","last_active":"2024-03-06T07:23:24.005061+0000","last_peered":"2024-03-06T07:23:24.005061+0000","last_clean":"2024-03-06T07:23:24.005061+0000","last_became_active":"2024-03-06T07:23:22.877324+0000","last_became_peered":"2024-03-06T07:23:22.877324+0000","last_unstale":"2024-03-06T07:23:24.005061+0000","last_undegraded":"2024-03-06T07:23:24.005061+0000","last_fullsized":"2024-03-06T07:23:24.005061+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:02:49.794812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878732+0000","last_change":"2024-03-06T07:23:21.868700+0000","last_active":"2024-03-06T07:23:23.878732+0000","last_peered":"2024-03-06T07:23:23.878732+0000","last_clean":"2024-03-06T07:23:23.878732+0000","last_became_active":"2024-03-06T07:23:21.868258+0000","last_became_peered":"2024-03-06T07:23:21.868258+0000","last_unstale":"2024-03-06T07:23:23.878732+0000","last_undegraded":"2024-03-06T07:23:23.878732+0000","last_fullsized":"2024-03-06T07:23:23.878732+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:42.357197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878279+0000","last_change":"2024-03-06T07:23:22.886900+0000","last_active":"2024-03-06T07:23:23.878279+0000","last_peered":"2024-03-06T07:23:23.878279+0000","last_clean":"2024-03-06T07:23:23.878279+0000","last_became_active":"2024-03-06T07:23:22.886563+0000","last_became_peered":"2024-03-06T07:23:22.886563+0000","last_unstale":"2024-03-06T07:23:23.878279+0000","last_undegraded":"2024-03-06T07:23:23.878279+0000","last_fullsized":"2024-03-06T07:23:23.878279+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:55.751529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.196530+0000","last_change":"2024-03-06T07:23:21.868016+0000","last_active":"2024-03-06T07:23:24.196530+0000","last_peered":"2024-03-06T07:23:24.196530+0000","last_clean":"2024-03-06T07:23:24.196530+0000","last_became_active":"2024-03-06T07:23:21.867724+0000","last_became_peered":"2024-03-06T07:23:21.867724+0000","last_unstale":"2024-03-06T07:23:24.196530+0000","last_undegraded":"2024-03-06T07:23:24.196530+0000","last_fullsized":"2024-03-06T07:23:24.196530+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:28.482928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867502+0000","last_change":"2024-03-06T07:23:22.875936+0000","last_active":"2024-03-06T07:23:24.867502+0000","last_peered":"2024-03-06T07:23:24.867502+0000","last_clean":"2024-03-06T07:23:24.867502+0000","last_became_active":"2024-03-06T07:23:22.875596+0000","last_became_peered":"2024-03-06T07:23:22.875596+0000","last_unstale":"2024-03-06T07:23:24.867502+0000","last_undegraded":"2024-03-06T07:23:24.867502+0000","last_fullsized":"2024-03-06T07:23:24.867502+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:46.330343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878306+0000","last_change":"2024-03-06T07:23:21.867760+0000","last_active":"2024-03-06T07:23:23.878306+0000","last_peered":"2024-03-06T07:23:23.878306+0000","last_clean":"2024-03-06T07:23:23.878306+0000","last_became_active":"2024-03-06T07:23:21.867499+0000","last_became_peered":"2024-03-06T07:23:21.867499+0000","last_unstale":"2024-03-06T07:23:23.878306+0000","last_undegraded":"2024-03-06T07:23:23.878306+0000","last_fullsized":"2024-03-06T07:23:23.878306+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:10:43.054124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.867493+0000","last_change":"2024-03-06T07:23:22.885430+0000","last_active":"2024-03-06T07:23:24.867493+0000","last_peered":"2024-03-06T07:23:24.867493+0000","last_clean":"2024-03-06T07:23:24.867493+0000","last_became_active":"2024-03-06T07:23:22.884843+0000","last_became_peered":"2024-03-06T07:23:22.884843+0000","last_unstale":"2024-03-06T07:23:24.867493+0000","last_undegraded":"2024-03-06T07:23:24.867493+0000","last_fullsized":"2024-03-06T07:23:24.867493+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:03:53.783028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571788+0000","last_change":"2024-03-06T07:23:21.867044+0000","last_active":"2024-03-06T07:23:24.571788+0000","last_peered":"2024-03-06T07:23:24.571788+0000","last_clean":"2024-03-06T07:23:24.571788+0000","last_became_active":"2024-03-06T07:23:21.866525+0000","last_became_peered":"2024-03-06T07:23:21.866525+0000","last_unstale":"2024-03-06T07:23:24.571788+0000","last_undegraded":"2024-03-06T07:23:24.571788+0000","last_fullsized":"2024-03-06T07:23:24.571788+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:32:26.850435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878493+0000","last_change":"2024-03-06T07:23:22.878586+0000","last_active":"2024-03-06T07:23:23.878493+0000","last_peered":"2024-03-06T07:23:23.878493+0000","last_clean":"2024-03-06T07:23:23.878493+0000","last_became_active":"2024-03-06T07:23:22.878306+0000","last_became_peered":"2024-03-06T07:23:22.878306+0000","last_unstale":"2024-03-06T07:23:23.878493+0000","last_undegraded":"2024-03-06T07:23:23.878493+0000","last_fullsized":"2024-03-06T07:23:23.878493+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:24.852598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.196408+0000","last_change":"2024-03-06T07:23:21.871204+0000","last_active":"2024-03-06T07:23:24.196408+0000","last_peered":"2024-03-06T07:23:24.196408+0000","last_clean":"2024-03-06T07:23:24.196408+0000","last_became_active":"2024-03-06T07:23:21.870923+0000","last_became_peered":"2024-03-06T07:23:21.870923+0000","last_unstale":"2024-03-06T07:23:24.196408+0000","last_undegraded":"2024-03-06T07:23:24.196408+0000","last_fullsized":"2024-03-06T07:23:24.196408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:46.604675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.571582+0000","last_change":"2024-03-06T07:23:22.884861+0000","last_active":"2024-03-06T07:23:24.571582+0000","last_peered":"2024-03-06T07:23:24.571582+0000","last_clean":"2024-03-06T07:23:24.571582+0000","last_became_active":"2024-03-06T07:23:22.884568+0000","last_became_peered":"2024-03-06T07:23:22.884568+0000","last_unstale":"2024-03-06T07:23:24.571582+0000","last_undegraded":"2024-03-06T07:23:24.571582+0000","last_fullsized":"2024-03-06T07:23:24.571582+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:11.203385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.004756+0000","last_change":"2024-03-06T07:23:21.867519+0000","last_active":"2024-03-06T07:23:24.004756+0000","last_peered":"2024-03-06T07:23:24.004756+0000","last_clean":"2024-03-06T07:23:24.004756+0000","last_became_active":"2024-03-06T07:23:21.867274+0000","last_became_peered":"2024-03-06T07:23:21.867274+0000","last_unstale":"2024-03-06T07:23:24.004756+0000","last_undegraded":"2024-03-06T07:23:24.004756+0000","last_fullsized":"2024-03-06T07:23:24.004756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:44.858008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:24.005299+0000","last_change":"2024-03-06T07:23:22.878701+0000","last_active":"2024-03-06T07:23:24.005299+0000","last_peered":"2024-03-06T07:23:24.005299+0000","last_clean":"2024-03-06T07:23:24.005299+0000","last_became_active":"2024-03-06T07:23:22.878386+0000","last_became_peered":"2024-03-06T07:23:22.878386+0000","last_unstale":"2024-03-06T07:23:24.005299+0000","last_undegraded":"2024-03-06T07:23:24.005299+0000","last_fullsized":"2024-03-06T07:23:24.005299+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:21.848550+0000","last_clean_scrub_stamp":"2024-03-06T07:23:21.848550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:45:29.216809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"45'2","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878992+0000","last_change":"2024-03-06T07:23:21.863945+0000","last_active":"2024-03-06T07:23:23.878992+0000","last_peered":"2024-03-06T07:23:23.878992+0000","last_clean":"2024-03-06T07:23:23.878992+0000","last_became_active":"2024-03-06T07:23:21.863699+0000","last_became_peered":"2024-03-06T07:23:21.863699+0000","last_unstale":"2024-03-06T07:23:23.878992+0000","last_undegraded":"2024-03-06T07:23:23.878992+0000","last_fullsized":"2024-03-06T07:23:23.878992+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:20.839557+0000","last_clean_scrub_stamp":"2024-03-06T07:23:20.839557+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:23.846491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":216,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-03-06T07:23:23.878023+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:23.878023+0000","last_peered":"2024-03-06T07:23:23.878023+0000","last_clean":"2024-03-06T07:23:23.878023+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:23.878023+0000","last_undegraded":"2024-03-06T07:23:23.878023+0000","last_fullsized":"2024-03-06T07:23:23.878023+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":810244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574873,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":548864,"data_stored":220458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":219856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":552960,"data_stored":221398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":552960,"data_stored":220424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":557056,"data_stored":221420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":219856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":810188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:23:30.725 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:23:30.725 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:23:30.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.869+0000 7efd0048f640 1 Processor -- start 2024-03-06T07:23:30.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.869+0000 7efd0048f640 1 -- start start 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.869+0000 7efd0048f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf80cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.869+0000 7efd0048f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcf80d0af0 0x7efcf80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efd0048f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80cd1e0 con 0x7efcf815c050 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efd0048f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf80cd4b0 con 0x7efcf80d0af0 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efd0048f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf80cd790 con 0x7efcf806fd10 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfea2c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcf80d0af0 0x7efcf80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfda2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf80cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfea2c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcf80d0af0 0x7efcf80ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:40994/0 (socket says 172.21.15.72:40994) 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfe22b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efcf815c050 0x7efcf806f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:50862/0 (socket says 172.21.15.72:50862) 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfda2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf80cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:57544/0 (socket says 172.21.15.72:57544) 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfea2c640 1 -- 172.21.15.72:0/3245110242 learned_addr learned my addr 172.21.15.72:0/3245110242 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfd229640 1 -- 172.21.15.72:0/3245110242 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240535607 0 0) 0x7efcf80cd1e0 con 0x7efcf815c050 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfda2a640 1 -- 172.21.15.72:0/3245110242 >> v2:172.21.15.146:3300/0 conn(0x7efcf80d0af0 msgr2=0x7efcf80ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfda2a640 1 --2- 172.21.15.72:0/3245110242 >> v2:172.21.15.146:3300/0 conn(0x7efcf80d0af0 0x7efcf80ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfda2a640 1 -- 172.21.15.72:0/3245110242 >> v1:172.21.15.146:6789/0 conn(0x7efcf815c050 legacy=0x7efcf806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.870+0000 7efcfda2a640 1 -- 172.21.15.72:0/3245110242 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80cda70 con 0x7efcf806fd10 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.871+0000 7efcfda2a640 1 --2- 172.21.15.72:0/3245110242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf80cc760 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efce80097a0 tx=0x7efce802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbcddf36e2bf1cda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.871+0000 7efcfd229640 1 -- 172.21.15.72:0/3245110242 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce80315a0 con 0x7efcf806fd10 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.871+0000 7efcfd229640 1 -- 172.21.15.72:0/3245110242 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce8037070 con 0x7efcf806fd10 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.871+0000 7efcfd229640 1 -- 172.21.15.72:0/3245110242 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce8031960 con 0x7efcf806fd10 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 -- 172.21.15.72:0/3245110242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 msgr2=0x7efcf80cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 --2- 172.21.15.72:0/3245110242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf80cc760 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efce80097a0 tx=0x7efce802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 -- 172.21.15.72:0/3245110242 shutdown_connections 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 --2- 172.21.15.72:0/3245110242 >> v2:172.21.15.146:3300/0 conn(0x7efcf80d0af0 0x7efcf80ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 --2- 172.21.15.72:0/3245110242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf80cc760 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 -- 172.21.15.72:0/3245110242 >> 172.21.15.72:0/3245110242 conn(0x7efcf80b3800 msgr2=0x7efcf80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 -- 172.21.15.72:0/3245110242 shutdown_connections 2024-03-06T07:23:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.872+0000 7efd0048f640 1 -- 172.21.15.72:0/3245110242 wait complete. 2024-03-06T07:23:30.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.873+0000 7efd0048f640 1 Processor -- start 2024-03-06T07:23:30.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.873+0000 7efd0048f640 1 -- start start 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efd0048f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf806dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efd0048f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcf80d0af0 0x7efcf8147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efd0048f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 0x7efcf80cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efd0048f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8165c30 con 0x7efcf815c050 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efd0048f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8165ab0 con 0x7efcf80d0af0 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efd0048f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf8165db0 con 0x7efcf806fd10 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 0x7efcf80cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 0x7efcf80cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51446/0 (socket says 172.21.15.72:51446) 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 -- 172.21.15.72:0/1879321126 learned_addr learned my addr 172.21.15.72:0/1879321126 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:30.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfe22b640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf806dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfda2a640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcf80d0af0 0x7efcf8147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 -- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 msgr2=0x7efcf806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf806dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 -- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcf80d0af0 msgr2=0x7efcf8147490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcf80d0af0 0x7efcf8147490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 -- 172.21.15.72:0/1879321126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf81479d0 con 0x7efcf815c050 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efcfea2c640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 0x7efcf80cae80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efcf000efc0 tx=0x7efcf000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.874+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0009e60 con 0x7efcf815c050 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.875+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcf000cd20 con 0x7efcf815c050 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.875+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf815bb90 con 0x7efcf815c050 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.875+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcf0003bd0 con 0x7efcf815c050 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.875+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf80b70e0 con 0x7efcf815c050 2024-03-06T07:23:30.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.876+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf809fe60 con 0x7efcf815c050 2024-03-06T07:23:30.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.876+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcf0005d10 con 0x7efcf815c050 2024-03-06T07:23:30.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.877+0000 7efceeffd640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:30.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.877+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efcf008a610 con 0x7efcf815c050 2024-03-06T07:23:30.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.877+0000 7efcfe22b640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:30.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.877+0000 7efcfe22b640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcdc06b4d0 0x7efcdc06d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efcf4000f60 tx=0x7efcf4007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:30.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:30.883+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcf0022020 con 0x7efcf815c050 2024-03-06T07:23:31.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.152+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7efcf8068ff0 con 0x7efcf815c050 2024-03-06T07:23:31.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.154+0000 7efceeffd640 1 -- 172.21.15.72:0/1879321126 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7efcf0010040 con 0x7efcf815c050 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:31.155 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcdc06b4d0 msgr2=0x7efcdc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcdc06b4d0 0x7efcdc06d990 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efcf4000f60 tx=0x7efcf4007c00 comp rx=0 tx=0).stop 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 msgr2=0x7efcf80cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 0x7efcf80cae80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efcf000efc0 tx=0x7efcf000c490 comp rx=0 tx=0).stop 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 shutdown_connections 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcdc06b4d0 0x7efcdc06d990 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf815c050 0x7efcf80cae80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcf80d0af0 0x7efcf8147490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 --2- 172.21.15.72:0/1879321126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcf806fd10 0x7efcf806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 >> 172.21.15.72:0/1879321126 conn(0x7efcf80b3800 msgr2=0x7efcf80aeae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 shutdown_connections 2024-03-06T07:23:31.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.156+0000 7efd0048f640 1 -- 172.21.15.72:0/1879321126 wait complete. 2024-03-06T07:23:31.168 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:31.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.268+0000 7f1a8d5b6640 1 Processor -- start 2024-03-06T07:23:31.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.268+0000 7f1a8d5b6640 1 -- start start 2024-03-06T07:23:31.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a8d5b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a880b4f70 0x7f1a880b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a8d5b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8806c6d0 0x7f1a8815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a8d5b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a880b5890 con 0x7f1a88066d90 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a8d5b6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8815e9f0 con 0x7f1a880b4f70 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a8d5b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a880cda10 con 0x7f1a8806c6d0 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a880b4f70 0x7f1a880b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a880b4f70 0x7f1a880b5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51452/0 (socket says 172.21.15.72:51452) 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 -- 172.21.15.72:0/404124872 learned_addr learned my addr 172.21.15.72:0/404124872 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a867fc640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8806c6d0 0x7f1a8815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 -- 172.21.15.72:0/404124872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8806c6d0 msgr2=0x7f1a8815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8806c6d0 0x7f1a8815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 -- 172.21.15.72:0/404124872 >> v1:172.21.15.146:6789/0 conn(0x7f1a88066d90 legacy=0x7f1a880ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a86ffd640 1 -- 172.21.15.72:0/404124872 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880cd450 con 0x7f1a880b4f70 2024-03-06T07:23:31.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.269+0000 7f1a867fc640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8806c6d0 0x7f1a8815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.270+0000 7f1a86ffd640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880b4f70 0x7f1a880b5350 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009870 tx=0x7f1a7802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b76c93d9967c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.270+0000 7f1a85ffb640 1 -- 172.21.15.72:0/404124872 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78035020 con 0x7f1a880b4f70 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.270+0000 7f1a85ffb640 1 -- 172.21.15.72:0/404124872 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a78031640 con 0x7f1a880b4f70 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.270+0000 7f1a85ffb640 1 -- 172.21.15.72:0/404124872 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7802d040 con 0x7f1a880b4f70 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/404124872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880b4f70 msgr2=0x7f1a880b5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880b4f70 0x7f1a880b5350 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1a78009870 tx=0x7f1a7802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/404124872 shutdown_connections 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8806c6d0 0x7f1a8815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/404124872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880b4f70 0x7f1a880b5350 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/404124872 >> 172.21.15.72:0/404124872 conn(0x7f1a880724d0 msgr2=0x7f1a880728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/404124872 shutdown_connections 2024-03-06T07:23:31.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.271+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/404124872 wait complete. 2024-03-06T07:23:31.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.272+0000 7f1a8d5b6640 1 Processor -- start 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.272+0000 7f1a8d5b6640 1 -- start start 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a8d5b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a88066d90 0x7f1a88154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a8d5b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a880b4f70 0x7f1a8815a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a8d5b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 0x7f1a880cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a8d5b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8815f3c0 con 0x7f1a880ca0c0 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a8d5b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8815e680 con 0x7f1a88066d90 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a8d5b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88161ad0 con 0x7f1a880b4f70 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 0x7f1a880cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 0x7f1a880cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51460/0 (socket says 172.21.15.72:51460) 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 -- 172.21.15.72:0/1247590579 learned_addr learned my addr 172.21.15.72:0/1247590579 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a867fc640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a880b4f70 0x7f1a8815a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a86ffd640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a88066d90 0x7f1a88154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 -- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a880b4f70 msgr2=0x7f1a8815a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a880b4f70 0x7f1a8815a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 -- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a88066d90 msgr2=0x7f1a88154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a88066d90 0x7f1a88154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 -- 172.21.15.72:0/1247590579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a881554d0 con 0x7f1a880ca0c0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.273+0000 7f1a877fe640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 0x7f1a880cb8a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000c730 tx=0x7f1a8000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.274+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80016020 con 0x7f1a880ca0c0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.274+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8000aa50 con 0x7f1a880ca0c0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.274+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000ce40 con 0x7f1a880ca0c0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.274+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a880ad490 con 0x7f1a880ca0c0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.274+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a880ada50 con 0x7f1a880ca0c0 2024-03-06T07:23:31.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.275+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8804f330 con 0x7f1a880ca0c0 2024-03-06T07:23:31.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.275+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a80019020 con 0x7f1a880ca0c0 2024-03-06T07:23:31.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.276+0000 7f1a6f7fe640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a7406b4d0 0x7f1a7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:31.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.276+0000 7f1a86ffd640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a7406b4d0 0x7f1a7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:31.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.276+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1a8008a220 con 0x7f1a880ca0c0 2024-03-06T07:23:31.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.276+0000 7f1a86ffd640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a7406b4d0 0x7f1a7406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1a78036000 tx=0x7f1a78008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:31.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.280+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a80020030 con 0x7f1a880ca0c0 2024-03-06T07:23:31.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.493+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a880b5f60 con 0x7f1a880ca0c0 2024-03-06T07:23:31.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.494+0000 7f1a6f7fe640 1 -- 172.21.15.72:0/1247590579 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1a800212b0 con 0x7f1a880ca0c0 2024-03-06T07:23:31.495 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:31.495 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:22.210297+0000","modified":"2024-03-06T07:23:24.220196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5670":{"gid":5670,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/874695441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":874695441},{"type":"v1","addr":"172.21.15.72:6837","nonce":874695441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5670,"qdb_cluster":[5670]},"id":3}]} 2024-03-06T07:23:31.495 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 20 2024-03-06T07:23:31.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a7406b4d0 msgr2=0x7f1a7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a7406b4d0 0x7f1a7406d990 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1a78036000 tx=0x7f1a78008030 comp rx=0 tx=0).stop 2024-03-06T07:23:31.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 msgr2=0x7f1a880cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 0x7f1a880cb8a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000c730 tx=0x7f1a8000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 shutdown_connections 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a7406b4d0 0x7f1a7406d990 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a880ca0c0 0x7f1a880cb8a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a880b4f70 0x7f1a8815a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 --2- 172.21.15.72:0/1247590579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a88066d90 0x7f1a88154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 >> 172.21.15.72:0/1247590579 conn(0x7f1a880724d0 msgr2=0x7f1a880afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 shutdown_connections 2024-03-06T07:23:31.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:31.496+0000 7f1a8d5b6640 1 -- 172.21.15.72:0/1247590579 wait complete. 2024-03-06T07:23:31.913 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:22.210297+0000', 'modified': '2024-03-06T07:23:24.220196+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5670': {'gid': 5670, 'name': 'c', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/874695441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 874695441}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 874695441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5670, 'qdb_cluster': [5670]} 2024-03-06T07:23:31.913 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:23:31.913 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:23:32.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.014+0000 7f04120f4640 1 Processor -- start 2024-03-06T07:23:32.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.014+0000 7f04120f4640 1 -- start start 2024-03-06T07:23:32.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f04120f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f040c162970 0x7f040c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f04120f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f040c161cf0 0x7f040c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f04120f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040c0d22c0 con 0x7f040c0b4460 2024-03-06T07:23:32.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f04120f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040c161b00 con 0x7f040c161cf0 2024-03-06T07:23:32.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f04120f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040c0cecc0 con 0x7f040c162970 2024-03-06T07:23:32.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f040bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f040c161cf0 0x7f040c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f040bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f040c161cf0 0x7f040c162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51476/0 (socket says 172.21.15.72:51476) 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.015+0000 7f040bfff640 1 -- 172.21.15.72:0/1666652194 learned_addr learned my addr 172.21.15.72:0/1666652194 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040bfff640 1 -- 172.21.15.72:0/1666652194 >> v2:172.21.15.146:3300/0 conn(0x7f040c162970 msgr2=0x7f040c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040affd640 1 --2- 172.21.15.72:0/1666652194 >> v2:172.21.15.146:3300/0 conn(0x7f040c162970 0x7f040c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040bfff640 1 --2- 172.21.15.72:0/1666652194 >> v2:172.21.15.146:3300/0 conn(0x7f040c162970 0x7f040c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040bfff640 1 -- 172.21.15.72:0/1666652194 >> v1:172.21.15.146:6789/0 conn(0x7f040c0b4460 legacy=0x7f040c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040bfff640 1 -- 172.21.15.72:0/1666652194 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c0d25a0 con 0x7f040c161cf0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040affd640 1 --2- 172.21.15.72:0/1666652194 >> v2:172.21.15.146:3300/0 conn(0x7f040c162970 0x7f040c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.016+0000 7f040bfff640 1 --2- 172.21.15.72:0/1666652194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c162180 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f040c0d17e0 tx=0x7f040402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194376ecd35c0313 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f040a7fc640 1 -- 172.21.15.72:0/1666652194 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040402ca80 con 0x7f040c161cf0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f040a7fc640 1 -- 172.21.15.72:0/1666652194 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040402cc20 con 0x7f040c161cf0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f040a7fc640 1 -- 172.21.15.72:0/1666652194 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040343f0 con 0x7f040c161cf0 2024-03-06T07:23:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 -- 172.21.15.72:0/1666652194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 msgr2=0x7f040c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 --2- 172.21.15.72:0/1666652194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c162180 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f040c0d17e0 tx=0x7f040402bee0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 -- 172.21.15.72:0/1666652194 shutdown_connections 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 --2- 172.21.15.72:0/1666652194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c162180 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 --2- 172.21.15.72:0/1666652194 >> v2:172.21.15.146:3300/0 conn(0x7f040c162970 0x7f040c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 -- 172.21.15.72:0/1666652194 >> 172.21.15.72:0/1666652194 conn(0x7f040c0d3ce0 msgr2=0x7f040c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 -- 172.21.15.72:0/1666652194 shutdown_connections 2024-03-06T07:23:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.017+0000 7f04120f4640 1 -- 172.21.15.72:0/1666652194 wait complete. 2024-03-06T07:23:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.018+0000 7f04120f4640 1 Processor -- start 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.018+0000 7f04120f4640 1 -- start start 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f04120f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f040c0b4460 0x7f040c0b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f04120f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f04120f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f040c162970 0x7f040c1f7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f04120f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0ce950 con 0x7f040c161cf0 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f04120f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0d2b60 con 0x7f040c0b4460 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f04120f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0b4840 con 0x7f040c162970 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f040affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f040affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c0cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51488/0 (socket says 172.21.15.72:51488) 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f040affd640 1 -- 172.21.15.72:0/3271086568 learned_addr learned my addr 172.21.15.72:0/3271086568 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f040b7fe640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f040c0b4460 0x7f040c0b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.019+0000 7f040bfff640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f040c162970 0x7f040c1f7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f040affd640 1 -- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f040c162970 msgr2=0x7f040c1f7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f040affd640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f040c162970 0x7f040c1f7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f040affd640 1 -- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f040c0b4460 msgr2=0x7f040c0b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f040affd640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f040c0b4460 0x7f040c0b0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f040affd640 1 -- 172.21.15.72:0/3271086568 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c0ccbe0 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f040affd640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c0cbe00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f040000c7f0 tx=0x7f040000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f03f3fff640 1 -- 172.21.15.72:0/3271086568 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400015070 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f03f3fff640 1 -- 172.21.15.72:0/3271086568 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0400007a30 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c0ccec0 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f03f3fff640 1 -- 172.21.15.72:0/3271086568 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400007e00 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.020+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c15f690 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.021+0000 7f03f3fff640 1 -- 172.21.15.72:0/3271086568 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0400018070 con 0x7f040c161cf0 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.022+0000 7f03f3fff640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03f806b4d0 0x7f03f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.022+0000 7f03f3fff640 1 -- 172.21.15.72:0/3271086568 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f040008a200 con 0x7f040c161cf0 2024-03-06T07:23:32.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.022+0000 7f040b7fe640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03f806b4d0 0x7f03f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.022+0000 7f040b7fe640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03f806b4d0 0x7f03f806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f03fc006fd0 tx=0x7f03fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.023+0000 7f0414db9740 1 Processor -- start 2024-03-06T07:23:32.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.023+0000 7f0414db9740 1 -- start start 2024-03-06T07:23:32.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f0414db9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 0x55f11b123c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b1278c0 0x55f11b129360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f0414db9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11b08d6b0 con 0x55f11b123870 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11b036a60 con 0x55f11b153600 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f11b089750 con 0x55f11b1278c0 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 0x55f11b123c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 0x55f11b123c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51494/0 (socket says 172.21.15.72:51494) 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 -- 172.21.15.72:0/752242685 learned_addr learned my addr 172.21.15.72:0/752242685 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040bfff640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040affd640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b1278c0 0x55f11b129360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 -- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b1278c0 msgr2=0x55f11b129360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b1278c0 0x55f11b129360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 -- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 msgr2=0x55f11b153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 -- 172.21.15.72:0/752242685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f11b054f30 con 0x55f11b123870 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040bfff640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040affd640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b1278c0 0x55f11b129360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.024+0000 7f040b7fe640 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 0x55f11b123c50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00d860 tx=0x7f03fc010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f0a2c486da4277 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f03f17fa640 1 -- 172.21.15.72:0/752242685 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc004020 con 0x55f11b123870 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f03f17fa640 1 -- 172.21.15.72:0/752242685 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc0041c0 con 0x55f11b123870 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f03f17fa640 1 -- 172.21.15.72:0/752242685 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc0149b0 con 0x55f11b123870 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 -- 172.21.15.72:0/752242685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 msgr2=0x55f11b123c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 0x55f11b123c50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00d860 tx=0x7f03fc010000 comp rx=0 tx=0).stop 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 -- 172.21.15.72:0/752242685 shutdown_connections 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b1278c0 0x55f11b129360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 --2- 172.21.15.72:0/752242685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b123870 0x55f11b123c50 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 -- 172.21.15.72:0/752242685 >> 172.21.15.72:0/752242685 conn(0x55f11b1258f0 msgr2=0x55f11b1271b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 -- 172.21.15.72:0/752242685 shutdown_connections 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.025+0000 7f0414db9740 1 -- 172.21.15.72:0/752242685 wait complete. 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.026+0000 7f0414db9740 1 -- start start 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.026+0000 7f0414db9740 1 Processor -- start 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 0x55f11b16bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b150030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11b127d50 con 0x55f11b153600 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11b049ea0 con 0x55f11b14bbe0 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f11b128030 con 0x55f11b14c130 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f0414db9740 10 client.0 _ll_register_callbacks cb 0x55f11b049870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 0x55f11b16bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 0x55f11b16bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51500/0 (socket says 172.21.15.72:51500) 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 -- 172.21.15.72:0/1696098434 learned_addr learned my addr 172.21.15.72:0/1696098434 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040bfff640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040affd640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 -- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 msgr2=0x55f11b14faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 -- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 msgr2=0x55f11b150030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b150030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 -- 172.21.15.72:0/1696098434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f11b04a270 con 0x55f11b153600 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f040b7fe640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 0x55f11b16bb30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55f11b157cf0 tx=0x7f03fc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8bee1de74eeacd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.027+0000 7f03ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1696098434 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc02e060 con 0x55f11b153600 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1696098434 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f03fc005910 con 0x55f11b153600 2024-03-06T07:23:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03fc0041c0 con 0x55f11b153600 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1696098434 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03fc005b10 con 0x55f11b153600 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f0414db9740 10 client.5835 fetch_fsmap learned FSMap version 20 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55f11ada22d0 con 0x55f11b153600 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1696098434 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc0041c0 con 0x55f11b153600 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.028+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1696098434 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 20) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f03fc013070 con 0x55f11b153600 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f0414db9740 10 client.5835 fetch_fsmap finished waiting for FSMap version 20 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f0414db9740 10 client.5835 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f0414db9740 20 client.5835 populate_metadata read hostname 'smithi072' 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f0414db9740 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b154d70 0x55f11b155150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f0414db9740 4 client.5835 mds_command: new command op to 5670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 --> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f11b09abf0 con 0x55f11b154d70 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f040affd640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b154d70 0x55f11b155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f040affd640 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b154d70 0x55f11b155150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ea21f2e10d2927d server_cookie=61b20d9e5a717620 in_seq=0 out_seq=0 2024-03-06T07:23:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.029+0000 7f03ea7fc640 10 client.5835 ms_handle_connect on v2:172.21.15.72:6835/874695441 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1696098434 <== mds.0 v2:172.21.15.72:6835/874695441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f11b09abf0 con 0x55f11b154d70 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f03ea7fc640 10 client.5835 handle_command_reply: tid=0 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 client.5835 shutdown 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 msgr2=0x55f11b16bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 0x55f11b16bb30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55f11b157cf0 tx=0x7f03fc014000 comp rx=0 tx=0).stop 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 shutdown_connections 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b154d70 0x55f11b155150 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f03ea7fc640 0 client.5835 ms_handle_reset on v2:172.21.15.72:6835/874695441 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b150030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 --2- 172.21.15.72:0/1696098434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b153600 0x55f11b16bb30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.030+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 >> 172.21.15.72:0/1696098434 conn(0x55f11b123870 msgr2=0x55f11b1515c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.031+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 shutdown_connections 2024-03-06T07:23:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.031+0000 7f0414db9740 1 -- 172.21.15.72:0/1696098434 wait complete. 2024-03-06T07:23:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.032+0000 7f0414db9740 20 client.5835 trim_cache size 0 max 16384 2024-03-06T07:23:32.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.048+0000 7f0414db9740 1 Processor -- start 2024-03-06T07:23:32.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.048+0000 7f0414db9740 1 -- start start 2024-03-06T07:23:32.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b1724a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f0414db9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 0x55f11b153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b153600 0x55f11b168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f0414db9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c1ff3e0 con 0x55f11b14c130 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c1fe840 con 0x55f11b14bbe0 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c1fee10 con 0x55f11b153600 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f040affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 0x55f11b153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f040affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 0x55f11b153bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51508/0 (socket says 172.21.15.72:51508) 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.049+0000 7f040affd640 1 -- 172.21.15.72:0/2498445541 learned_addr learned my addr 172.21.15.72:0/2498445541 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040b7fe640 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b1724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040bfff640 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b153600 0x55f11b168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040affd640 1 -- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b153600 msgr2=0x55f11b168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040affd640 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b153600 0x55f11b168fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040affd640 1 -- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 msgr2=0x55f11b1724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040affd640 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b1724a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040affd640 1 -- 172.21.15.72:0/2498445541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f11ae6c6b0 con 0x55f11b14c130 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f040affd640 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 0x55f11b153bd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0400058640 tx=0x7f04000885c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80aa18e23c26f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f03f0ff9640 1 -- 172.21.15.72:0/2498445541 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040001f040 con 0x55f11b14c130 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f03f0ff9640 1 -- 172.21.15.72:0/2498445541 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04000569e0 con 0x55f11b14c130 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.050+0000 7f03f0ff9640 1 -- 172.21.15.72:0/2498445541 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400054fb0 con 0x55f11b14c130 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 -- 172.21.15.72:0/2498445541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 msgr2=0x55f11b153bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 0x55f11b153bd0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0400058640 tx=0x7f04000885c0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 -- 172.21.15.72:0/2498445541 shutdown_connections 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b153600 0x55f11b168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14c130 0x55f11b153bd0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 --2- 172.21.15.72:0/2498445541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b14bbe0 0x55f11b1724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 -- 172.21.15.72:0/2498445541 >> 172.21.15.72:0/2498445541 conn(0x55f11b123870 msgr2=0x55f11b155300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 -- 172.21.15.72:0/2498445541 shutdown_connections 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 -- 172.21.15.72:0/2498445541 wait complete. 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.051+0000 7f0414db9740 1 -- start start 2024-03-06T07:23:32.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.052+0000 7f0414db9740 1 Processor -- start 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 0x55f11b169a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b1de370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11b157480 con 0x55f11b14bbe0 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f11b16f2b0 con 0x55f11b153600 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f11b156d90 con 0x55f11b14c130 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 10 client.0 _ll_register_callbacks cb 0x7f03fc0040f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 0x55f11b169a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 0x55f11b169a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51520/0 (socket says 172.21.15.72:51520) 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 -- 172.21.15.72:0/1561721265 learned_addr learned my addr 172.21.15.72:0/1561721265 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 -- 172.21.15.72:0/1561721265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 msgr2=0x55f11b14e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 -- 172.21.15.72:0/1561721265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 msgr2=0x55f11b1de370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b1de370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 -- 172.21.15.72:0/1561721265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f11b14ed60 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f040b7fe640 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 0x55f11b169a00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00d530 tx=0x7f03fc004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d779982f8212b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f03ea7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1561721265 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc00bd80 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1561721265 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f03fc014e00 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.053+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04000569e0 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1561721265 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03fc048050 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 10 client.5841 fetch_fsmap learned FSMap version 20 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55f11b1925b0 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1561721265 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04000569e0 con 0x55f11b14bbe0 2024-03-06T07:23:32.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1561721265 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 20) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f03fc04b020 con 0x55f11b14bbe0 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 10 client.5841 fetch_fsmap finished waiting for FSMap version 20 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 10 client.5841 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 20 client.5841 populate_metadata read hostname 'smithi072' 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b14f860 0x55f11b14fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 4 client.5841 mds_command: new command op to 5670 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 --> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f11b094f40 con 0x55f11b14f860 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f040affd640 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b14f860 0x55f11b14fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f03ea7fc640 10 client.5841 ms_handle_connect on v2:172.21.15.72:6835/874695441 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f040affd640 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b14f860 0x55f11b14fc40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1ca4dc54a48fd17 server_cookie=28a29dc075bbc820 in_seq=0 out_seq=0 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f03ea7fc640 1 -- 172.21.15.72:0/1561721265 <== mds.0 v2:172.21.15.72:6835/874695441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55f11b094f40 con 0x55f11b14f860 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.054+0000 7f03ea7fc640 10 client.5841 handle_command_reply: tid=0 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 client.5841 shutdown 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 msgr2=0x55f11b169a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 0x55f11b169a00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f03fc00d530 tx=0x7f03fc004680 comp rx=0 tx=0).stop 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 shutdown_connections 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:6835/874695441,v1:172.21.15.72:6837/874695441] conn(0x55f11b14f860 0x55f11b14fc40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f11b153600 0x55f11b1de370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f11b14c130 0x55f11b14e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f03ea7fc640 0 client.5841 ms_handle_reset on v2:172.21.15.72:6835/874695441 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 --2- 172.21.15.72:0/1561721265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f11b14bbe0 0x55f11b169a00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 >> 172.21.15.72:0/1561721265 conn(0x55f11b123870 msgr2=0x55f11b153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 shutdown_connections 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 1 -- 172.21.15.72:0/1561721265 wait complete. 2024-03-06T07:23:32.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f0414db9740 20 client.5841 trim_cache size 0 max 16384 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5670, 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 9.1931472519999993, 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 20, 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 48, 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 46, 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 14.258461818000001 2024-03-06T07:23:32.063 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:23:32.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03f806b4d0 msgr2=0x7f03f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03f806b4d0 0x7f03f806d990 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f03fc006fd0 tx=0x7f03fc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 msgr2=0x7f040c0cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c0cbe00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f040000c7f0 tx=0x7f040000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 shutdown_connections 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03f806b4d0 0x7f03f806d990 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f040c162970 0x7f040c1f7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f040c161cf0 0x7f040c0cbe00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 --2- 172.21.15.72:0/3271086568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f040c0b4460 0x7f040c0b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.055+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 >> 172.21.15.72:0/3271086568 conn(0x7f040c0d3ce0 msgr2=0x7f040c0d42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.056+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 shutdown_connections 2024-03-06T07:23:32.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.056+0000 7f04120f4640 1 -- 172.21.15.72:0/3271086568 wait complete. 2024-03-06T07:23:32.069 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:23:32.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.169+0000 7fefa17f7640 1 Processor -- start 2024-03-06T07:23:32.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.170+0000 7fefa17f7640 1 -- start start 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.170+0000 7fefa17f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 0x7fef9c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.170+0000 7fefa17f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.170+0000 7fefa17f7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c15e380 con 0x7fef9c0d00c0 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.170+0000 7fefa17f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c06ca90 con 0x7fef9c0cf850 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.170+0000 7fefa17f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0cdc10 con 0x7fef9c06cf20 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9a7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 0x7fef9c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9b7fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9affd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fef9c0d00c0 0x7fef9c0d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:50890/0 (socket says 172.21.15.72:50890) 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9a7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 0x7fef9c0cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:41050/0 (socket says 172.21.15.72:41050) 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9affd640 1 -- 172.21.15.72:0/671508376 learned_addr learned my addr 172.21.15.72:0/671508376 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9b7fe640 1 -- 172.21.15.72:0/671508376 >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 msgr2=0x7fef9c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9b7fe640 1 --2- 172.21.15.72:0/671508376 >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 0x7fef9c0cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9b7fe640 1 -- 172.21.15.72:0/671508376 >> v1:172.21.15.146:6789/0 conn(0x7fef9c0d00c0 legacy=0x7fef9c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9b7fe640 1 -- 172.21.15.72:0/671508376 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c06c160 con 0x7fef9c06cf20 2024-03-06T07:23:32.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.171+0000 7fef9a7fc640 1 --2- 172.21.15.72:0/671508376 >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 0x7fef9c0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:32.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.172+0000 7fef9b7fe640 1 --2- 172.21.15.72:0/671508376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c06d3b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef94009d90 tx=0x7fef9402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ef181c3a738470e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.172+0000 7fef99ffb640 1 -- 172.21.15.72:0/671508376 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94039070 con 0x7fef9c06cf20 2024-03-06T07:23:32.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.172+0000 7fef99ffb640 1 -- 172.21.15.72:0/671508376 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9402cc50 con 0x7fef9c06cf20 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.172+0000 7fef99ffb640 1 -- 172.21.15.72:0/671508376 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef94034440 con 0x7fef9c06cf20 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 -- 172.21.15.72:0/671508376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 msgr2=0x7fef9c06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 --2- 172.21.15.72:0/671508376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c06d3b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fef94009d90 tx=0x7fef9402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 -- 172.21.15.72:0/671508376 shutdown_connections 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 --2- 172.21.15.72:0/671508376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c06d3b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 --2- 172.21.15.72:0/671508376 >> v2:172.21.15.146:3300/0 conn(0x7fef9c0cf850 0x7fef9c0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 -- 172.21.15.72:0/671508376 >> 172.21.15.72:0/671508376 conn(0x7fef9c0b35a0 msgr2=0x7fef9c0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 -- 172.21.15.72:0/671508376 shutdown_connections 2024-03-06T07:23:32.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.173+0000 7fefa17f7640 1 -- 172.21.15.72:0/671508376 wait complete. 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.174+0000 7fefa17f7640 1 Processor -- start 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.174+0000 7fefa17f7640 1 -- start start 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fefa17f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c0c5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fefa17f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 0x7fef9c0ca6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fefa17f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef9c0d00c0 0x7fef9c1f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fefa17f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0cd900 con 0x7fef9c0cf850 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fefa17f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c06c720 con 0x7fef9c0d00c0 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fefa17f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c0b1030 con 0x7fef9c06cf20 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 0x7fef9c0ca6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 0x7fef9c0ca6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51522/0 (socket says 172.21.15.72:51522) 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 -- 172.21.15.72:0/2824193811 learned_addr learned my addr 172.21.15.72:0/2824193811 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9b7fe640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef9c0d00c0 0x7fef9c1f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9affd640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c0c5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 -- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 msgr2=0x7fef9c0c5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c0c5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 -- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef9c0d00c0 msgr2=0x7fef9c1f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef9c0d00c0 0x7fef9c1f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 -- 172.21.15.72:0/2824193811 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c1f4420 con 0x7fef9c0cf850 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.175+0000 7fef9a7fc640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 0x7fef9c0ca6e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fef8c00ec10 tx=0x7fef8c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.176+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c017070 con 0x7fef9c0cf850 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.176+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c00cc30 con 0x7fef9c0cf850 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.176+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c1f4700 con 0x7fef9c0cf850 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.176+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c013470 con 0x7fef9c0cf850 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.176+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c0c6410 con 0x7fef9c0cf850 2024-03-06T07:23:32.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.177+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c04f330 con 0x7fef9c0cf850 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.177+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef8c01a070 con 0x7fef9c0cf850 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.177+0000 7fef937fe640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef7c06b4d0 0x7fef7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.178+0000 7fef9affd640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef7c06b4d0 0x7fef7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.178+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fef8c08a4e0 con 0x7fef9c0cf850 2024-03-06T07:23:32.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.178+0000 7fef9affd640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef7c06b4d0 0x7fef7c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0cb240 tx=0x7fef84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:32.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.182+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef8c0593b0 con 0x7fef9c0cf850 2024-03-06T07:23:32.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.398+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fef9c0747a0 con 0x7fef9c0cf850 2024-03-06T07:23:32.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.398+0000 7fef937fe640 1 -- 172.21.15.72:0/2824193811 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7fef8c058cb0 con 0x7fef9c0cf850 2024-03-06T07:23:32.399 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:23:32.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef7c06b4d0 msgr2=0x7fef7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef7c06b4d0 0x7fef7c06d990 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0cb240 tx=0x7fef84009040 comp rx=0 tx=0).stop 2024-03-06T07:23:32.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 msgr2=0x7fef9c0ca6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 0x7fef9c0ca6e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fef8c00ec10 tx=0x7fef8c00c520 comp rx=0 tx=0).stop 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 shutdown_connections 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef7c06b4d0 0x7fef7c06d990 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef9c0d00c0 0x7fef9c1f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef9c0cf850 0x7fef9c0ca6e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 --2- 172.21.15.72:0/2824193811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef9c06cf20 0x7fef9c0c5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 >> 172.21.15.72:0/2824193811 conn(0x7fef9c0b35a0 msgr2=0x7fef9c06d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 shutdown_connections 2024-03-06T07:23:32.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:32.400+0000 7fefa17f7640 1 -- 172.21.15.72:0/2824193811 wait complete. 2024-03-06T07:23:32.412 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:23:32.412 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:23:32.412 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:23:32.412 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:23:32.413 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:32.413 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:23:32.413 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:23:32.413 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:23:32.413 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:32.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.413 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.428 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42583sec preferred_lft 42583sec 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.429 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:32.430 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:23:32.431 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:23:32.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.431 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:23:32.494 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:23:32.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.495 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:23:32.551 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:23:32.551 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:23:32.551 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:23:32.551 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:23:32.551 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:23:32.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.552 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:23:32.552 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:23:32.552 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:23:32.552 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:23:32.552 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:23:32.552 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:23:32.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.646 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:23:32.660 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:23:32.661 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:23:32.661 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:32.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.661 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:23:32.728 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:23:32.728 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:32.728 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:32.784 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:23:32.784 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:23:32.784 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:32.838 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:23:32.905 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:23:32.961 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:23:32.962 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:32.963 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:32.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:32.964 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:23:33.015 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:23:33.015 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:23:33.057 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:23:33.084 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:23:33.143 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:23:33.144 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:23:33.145 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:33.146 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:23:33.146 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:33.146 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:23:33.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:33.146 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:23:33.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:23:33.144+0000 7fd3ef95c480 -1 init, newargv = 0x5630837d25e0 newargc=15 2024-03-06T07:23:33.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[55436]: starting ceph client 2024-03-06T07:23:33.149 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[55436]: starting fuse 2024-03-06T07:23:33.163 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:23:33.163 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:23:33.164 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:23:33.165 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:23:33.284 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.55436.asok 2024-03-06T07:23:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:23:33.295 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55436.asok 2024-03-06T07:23:33.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:33.295 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55436.asok status 2024-03-06T07:23:33.431 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "55436", 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:23:33.432 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5853, 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 5853 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1090814742 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:23:33.433 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1090814742 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.5853 192.168.0.1:0/1090814742", 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/1090814742", 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 20, 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 48, 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:23:33.434 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:23:33.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:33.440 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:23:33.440 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:33.456 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:23:33.456 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:33.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:33.457 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:33.528 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.670+0000 7f9f1fe64640 1 Processor -- start 2024-03-06T07:23:33.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.670+0000 7f9f1fe64640 1 -- start start 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1fe64640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f181614d0 0x7f9f180d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1fe64640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d1910 0x7f9f180d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1fe64640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f180d2740 con 0x7f9f180d22b0 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1fe64640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f181618b0 con 0x7f9f181614d0 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1fe64640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cf400 con 0x7f9f180d1910 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1e401640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9f180d22b0 0x7f9f180d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39850/0 (socket says 172.21.15.72:39850) 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1e401640 1 -- 172.21.15.72:0/2066424786 learned_addr learned my addr 172.21.15.72:0/2066424786 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1dc00640 1 --2- 172.21.15.72:0/2066424786 >> v2:172.21.15.146:3300/0 conn(0x7f9f181614d0 0x7f9f180d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1d3ff640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d1910 0x7f9f180d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.671+0000 7f9f1cbfe640 1 -- 172.21.15.72:0/2066424786 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327349698 0 0) 0x7f9f10001110 con 0x7f9f180d22b0 2024-03-06T07:23:33.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1cbfe640 1 -- 172.21.15.72:0/2066424786 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f04003520 con 0x7f9f180d22b0 2024-03-06T07:23:33.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1dc00640 1 -- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d1910 msgr2=0x7f9f180d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:33.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1dc00640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d1910 0x7f9f180d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1dc00640 1 -- 172.21.15.72:0/2066424786 >> v1:172.21.15.72:6789/0 conn(0x7f9f180d22b0 legacy=0x7f9f180d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1dc00640 1 -- 172.21.15.72:0/2066424786 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f180cf6a0 con 0x7f9f181614d0 2024-03-06T07:23:33.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1d3ff640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d1910 0x7f9f180d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:33.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.672+0000 7f9f1dc00640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180d4860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009d90 tx=0x7f9f1402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caddffefbe26ea4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1cbfe640 1 -- 172.21.15.72:0/2066424786 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14039070 con 0x7f9f181614d0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1cbfe640 1 -- 172.21.15.72:0/2066424786 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1402cc50 con 0x7f9f181614d0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1cbfe640 1 -- 172.21.15.72:0/2066424786 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14034440 con 0x7f9f181614d0 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 -- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 msgr2=0x7f9f180d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180d4860 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9f14009d90 tx=0x7f9f1402bea0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 -- 172.21.15.72:0/2066424786 shutdown_connections 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d1910 0x7f9f180d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/2066424786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180d4860 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 -- 172.21.15.72:0/2066424786 >> 172.21.15.72:0/2066424786 conn(0x7f9f180d3d40 msgr2=0x7f9f180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.673+0000 7f9f1fe64640 1 -- 172.21.15.72:0/2066424786 shutdown_connections 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.674+0000 7f9f1fe64640 1 -- 172.21.15.72:0/2066424786 wait complete. 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 Processor -- start 2024-03-06T07:23:33.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 -- start start 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 0x7f9f1815f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d22b0 0x7f9f180cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b0910 con 0x7f9f180d1910 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180cfc60 con 0x7f9f181614d0 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1fe64640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f180b1e40 con 0x7f9f180d22b0 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1dc00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 0x7f9f1815f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1dc00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 0x7f9f1815f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41014/0 (socket says 172.21.15.72:41014) 2024-03-06T07:23:33.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.675+0000 7f9f1dc00640 1 -- 172.21.15.72:0/1931408990 learned_addr learned my addr 172.21.15.72:0/1931408990 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1e401640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1d3ff640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d22b0 0x7f9f180cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1dc00640 1 -- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d22b0 msgr2=0x7f9f180cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1dc00640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d22b0 0x7f9f180cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1dc00640 1 -- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 msgr2=0x7f9f180cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1dc00640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1dc00640 1 -- 172.21.15.72:0/1931408990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1815f860 con 0x7f9f180d1910 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1d3ff640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d22b0 0x7f9f180cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1dc00640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 0x7f9f1815f320 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9f14000f80 tx=0x7f9f14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14042020 con 0x7f9f180d1910 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1403a040 con 0x7f9f180d1910 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f14039070 con 0x7f9f180d1910 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f180cafb0 con 0x7f9f180d1910 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.676+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f180cb570 con 0x7f9f180d1910 2024-03-06T07:23:33.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.677+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f180cec90 con 0x7f9f180d1910 2024-03-06T07:23:33.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.678+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f14045020 con 0x7f9f180d1910 2024-03-06T07:23:33.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.678+0000 7f9f0e7fc640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9efc06b4d0 0x7f9efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:33.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.678+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9f140ac1d0 con 0x7f9f180d1910 2024-03-06T07:23:33.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.678+0000 7f9f1d3ff640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9efc06b4d0 0x7f9efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:33.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.679+0000 7f9f1d3ff640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9efc06b4d0 0x7f9efc06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9f180a0ea0 tx=0x7f9f08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:33.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.682+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f1404c030 con 0x7f9f180d1910 2024-03-06T07:23:33.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.860+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f9f18068ff0 con 0x7f9f180d1910 2024-03-06T07:23:33.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.863+0000 7f9f0e7fc640 1 -- 172.21.15.72:0/1931408990 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v13)=0 v13) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9f1407b0a0 con 0x7f9f180d1910 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9efc06b4d0 msgr2=0x7f9efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9efc06b4d0 0x7f9efc06d990 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9f180a0ea0 tx=0x7f9f08008040 comp rx=0 tx=0).stop 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 msgr2=0x7f9f1815f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 0x7f9f1815f320 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9f14000f80 tx=0x7f9f14004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 shutdown_connections 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9efc06b4d0 0x7f9efc06d990 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f181614d0 0x7f9f180cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f180d22b0 0x7f9f180cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.865+0000 7f9f1fe64640 1 --2- 172.21.15.72:0/1931408990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f180d1910 0x7f9f1815f320 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.866+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 >> 172.21.15.72:0/1931408990 conn(0x7f9f180d3d40 msgr2=0x7f9f180d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.866+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 shutdown_connections 2024-03-06T07:23:33.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:33.866+0000 7f9f1fe64640 1 -- 172.21.15.72:0/1931408990 wait complete. 2024-03-06T07:23:43.879 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:43.879 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:43.895 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:23:43.895 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:23:43.895 DEBUG:teuthology.orchestra.run.smithi072:> import uuid 2024-03-06T07:23:43.895 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:43.895 DEBUG:teuthology.orchestra.run.smithi072:> n = 51 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = False 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> if 0 > 0: 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = True 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> dpath = os.path.dirname(path) 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> fnameprefix = os.path.basename(path) 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> try: 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(n): 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:23:43.896 DEBUG:teuthology.orchestra.run.smithi072:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> f.write(f"{i}") 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> f.flush() 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(f.fileno()) 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> os.unlink(fpath) 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> if create_hard_links: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> for j in range(0): 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> finally: 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> os.close(dirfd) 2024-03-06T07:23:43.897 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:23:44.097 INFO:teuthology.orchestra.run.smithi072.stderr:Traceback (most recent call last): 2024-03-06T07:23:44.098 INFO:teuthology.orchestra.run.smithi072.stderr: File "", line 20, in 2024-03-06T07:23:44.098 INFO:teuthology.orchestra.run.smithi072.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-03-06T07:23:44.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:23:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:44.108 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:44.109 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:44.204 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:44.245 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:44.245 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:44.245 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:44.260 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:44.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.402+0000 7fb73b181640 1 Processor -- start 2024-03-06T07:23:44.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.402+0000 7fb73b181640 1 -- start start 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb73b181640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 0x7fb73415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb73b181640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73415f580 con 0x7fb73413dc10 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb73b181640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73415b450 con 0x7fb7340b3060 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb73b181640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340cf3f0 con 0x7fb73415bf30 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb738f1d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb73413dc10 0x7fb7340b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44402/0 (socket says 172.21.15.72:44402) 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb738f1d640 1 -- 172.21.15.72:0/1664040865 learned_addr learned my addr 172.21.15.72:0/1664040865 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb73971e640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 0x7fb73415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660784756 0 0) 0x7fb73415f580 con 0x7fb73413dc10 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb720003520 con 0x7fb73413dc10 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.403+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722903042 0 0) 0x7fb73415b450 con 0x7fb7340b3060 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73415f580 con 0x7fb7340b3060 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417694409 0 0) 0x7fb720003520 con 0x7fb73413dc10 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 msgr2=0x7fb73415ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 0x7fb73415ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 >> v1:172.21.15.146:6789/0 conn(0x7fb7340b3060 legacy=0x7fb73415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73415b730 con 0x7fb73413dc10 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fb724001670 con 0x7fb73413dc10 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 >> v1:172.21.15.72:6789/0 conn(0x7fb73413dc10 legacy=0x7fb7340b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 0x7fb720003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7200040d0 0x7fb720004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb720004d40 0x7fb7200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340cf3f0 con 0x7fb720004d40 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72000bb50 con 0x7fb7200040d0 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72000bd20 con 0x7fb73415bf30 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb73971e640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb720004d40 0x7fb7200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb738f1d640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 0x7fb720003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb733fff640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7200040d0 0x7fb720004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb73971e640 1 -- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 msgr2=0x7fb720003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb73971e640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb73415bf30 0x7fb720003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb73971e640 1 -- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7200040d0 msgr2=0x7fb720004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb73971e640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7200040d0 0x7fb720004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.404+0000 7fb73971e640 1 -- 172.21.15.72:0/1664040865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb720003520 con 0x7fb720004d40 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb73971e640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb720004d40 0x7fb7200051c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009fc0 tx=0x7fb720005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd490dffb94b2d6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c039070 con 0x7fb720004d40 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c02cc20 con 0x7fb720004d40 2024-03-06T07:23:44.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb7337fe640 1 -- 172.21.15.72:0/1664040865 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c009c10 con 0x7fb720004d40 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb73b181640 1 -- 172.21.15.72:0/1664040865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb720004d40 msgr2=0x7fb7200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb73b181640 1 --2- 172.21.15.72:0/1664040865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb720004d40 0x7fb7200051c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009fc0 tx=0x7fb720005830 comp rx=0 tx=0).stop 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb73971e640 1 -- 172.21.15.72:0/1664040865 reap_dead start 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb73b181640 1 -- 172.21.15.72:0/1664040865 shutdown_connections 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.405+0000 7fb73b181640 1 -- 172.21.15.72:0/1664040865 >> 172.21.15.72:0/1664040865 conn(0x7fb7340cf8a0 msgr2=0x7fb7340cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.406+0000 7fb73b181640 1 -- 172.21.15.72:0/1664040865 shutdown_connections 2024-03-06T07:23:44.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.406+0000 7fb73b181640 1 -- 172.21.15.72:0/1664040865 wait complete. 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.407+0000 7fb73b181640 1 Processor -- start 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.407+0000 7fb73b181640 1 -- start start 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.407+0000 7fb73b181640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb720004d40 0x7fb734166bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 0x7fb7341525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb734152b30 0x7fb7340cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73415ef60 con 0x7fb73413d530 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73415b210 con 0x7fb720004d40 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73415f0e0 con 0x7fb734152b30 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 0x7fb7341525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 0x7fb7341525f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38636/0 (socket says 172.21.15.72:38636) 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 -- 172.21.15.72:0/833166908 learned_addr learned my addr 172.21.15.72:0/833166908 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb738f1d640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb720004d40 0x7fb734166bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73971e640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb734152b30 0x7fb7340cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 -- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb734152b30 msgr2=0x7fb7340cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb734152b30 0x7fb7340cfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 -- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb720004d40 msgr2=0x7fb734166bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb720004d40 0x7fb734166bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 -- 172.21.15.72:0/833166908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7341f4390 con 0x7fb73413d530 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb738f1d640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb720004d40 0x7fb734166bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73971e640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb734152b30 0x7fb7340cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb733fff640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 0x7fb7341525f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00f7a0 tx=0x7fb71c00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c0052e0 con 0x7fb73413d530 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71c018070 con 0x7fb73413d530 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7341f4670 con 0x7fb73413d530 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c00d970 con 0x7fb73413d530 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.408+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7341f4c30 con 0x7fb73413d530 2024-03-06T07:23:44.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.409+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73404f330 con 0x7fb73413d530 2024-03-06T07:23:44.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.410+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb71c00db10 con 0x7fb73413d530 2024-03-06T07:23:44.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.411+0000 7fb7317fa640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb71406b4d0 0x7fb71406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.411+0000 7fb738f1d640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb71406b4d0 0x7fb71406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.411+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb71c08a250 con 0x7fb73413d530 2024-03-06T07:23:44.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.411+0000 7fb738f1d640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb71406b4d0 0x7fb71406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb724001eb0 tx=0x7fb724009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.415+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb71c01a310 con 0x7fb73413d530 2024-03-06T07:23:44.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.595+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7341412c0 con 0x7fb73413d530 2024-03-06T07:23:44.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.596+0000 7fb7317fa640 1 -- 172.21.15.72:0/833166908 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16777 (secure 0 0 0) 0x7fb71c059090 con 0x7fb73413d530 2024-03-06T07:23:44.598 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:44.598 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":48,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:32.024853+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.32","mappings":[{"from":0,"to":4}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb71406b4d0 msgr2=0x7fb71406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb71406b4d0 0x7fb71406d990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb724001eb0 tx=0x7fb724009040 comp rx=0 tx=0).stop 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 msgr2=0x7fb7341525f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 0x7fb7341525f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00f7a0 tx=0x7fb71c00fc60 comp rx=0 tx=0).stop 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 shutdown_connections 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb71406b4d0 0x7fb71406d990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb734152b30 0x7fb7340cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb73413d530 0x7fb7341525f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 --2- 172.21.15.72:0/833166908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb720004d40 0x7fb734166bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 >> 172.21.15.72:0/833166908 conn(0x7fb7340cf8a0 msgr2=0x7fb734154310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 shutdown_connections 2024-03-06T07:23:44.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.598+0000 7fb73b181640 1 -- 172.21.15.72:0/833166908 wait complete. 2024-03-06T07:23:44.611 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:20.432633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:21.177789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:44.611 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.717+0000 7fa594a0e640 1 Processor -- start 2024-03-06T07:23:44.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.717+0000 7fa594a0e640 1 -- start start 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa594a0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa590168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa594a0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5900d2730 con 0x7fa5901614d0 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa594a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5901618b0 con 0x7fa5900d4860 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa594a0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900d4cf0 con 0x7fa5900b1e40 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa58e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa5901614d0 0x7fa5900d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44408/0 (socket says 172.21.15.72:44408) 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa58e59c640 1 -- 172.21.15.72:0/2968709731 learned_addr learned my addr 172.21.15.72:0/2968709731 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa58ed9d640 1 --2- 172.21.15.72:0/2968709731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa590168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621495524 0 0) 0x7fa5900d2730 con 0x7fa5901614d0 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.718+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa57c003520 con 0x7fa5901614d0 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.719+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968476396 0 0) 0x7fa5901618b0 con 0x7fa5900d4860 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.719+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5900d2730 con 0x7fa5900d4860 2024-03-06T07:23:44.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.719+0000 7fa58ed9d640 1 -- 172.21.15.72:0/2968709731 >> v1:172.21.15.146:6789/0 conn(0x7fa5900d4860 legacy=0x7fa5900d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.719+0000 7fa58ed9d640 1 -- 172.21.15.72:0/2968709731 >> v1:172.21.15.72:6789/0 conn(0x7fa5901614d0 legacy=0x7fa5900d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.719+0000 7fa58ed9d640 1 -- 172.21.15.72:0/2968709731 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5900cf400 con 0x7fa5900b1e40 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.719+0000 7fa58ed9d640 1 --2- 172.21.15.72:0/2968709731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa590168b60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5900cf5d0 tx=0x7fa58802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49c0ba997133cf1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588039070 con 0x7fa5900b1e40 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58802ccd0 con 0x7fa5900b1e40 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa58dd9b640 1 -- 172.21.15.72:0/2968709731 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588034660 con 0x7fa5900b1e40 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 -- 172.21.15.72:0/2968709731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 msgr2=0x7fa590168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 --2- 172.21.15.72:0/2968709731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa590168b60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5900cf5d0 tx=0x7fa58802bf20 comp rx=0 tx=0).stop 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 -- 172.21.15.72:0/2968709731 shutdown_connections 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 --2- 172.21.15.72:0/2968709731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa590168b60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 -- 172.21.15.72:0/2968709731 >> 172.21.15.72:0/2968709731 conn(0x7fa5900d3d40 msgr2=0x7fa5900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:44.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 -- 172.21.15.72:0/2968709731 shutdown_connections 2024-03-06T07:23:44.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.720+0000 7fa594a0e640 1 -- 172.21.15.72:0/2968709731 wait complete. 2024-03-06T07:23:44.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.722+0000 7fa594a0e640 1 Processor -- start 2024-03-06T07:23:44.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.722+0000 7fa594a0e640 1 -- start start 2024-03-06T07:23:44.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa5900b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 0x7fa590153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5901614d0 0x7fa590154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900cfc60 con 0x7fa5900d4860 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900cf980 con 0x7fa5901614d0 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5900b0910 con 0x7fa5900b1e40 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 0x7fa590153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 0x7fa590153b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38644/0 (socket says 172.21.15.72:38644) 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 -- 172.21.15.72:0/991505207 learned_addr learned my addr 172.21.15.72:0/991505207 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa58e59c640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa5900b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa58ed9d640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5901614d0 0x7fa590154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 -- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 msgr2=0x7fa5900b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa5900b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 -- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5901614d0 msgr2=0x7fa590154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5901614d0 0x7fa590154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 -- 172.21.15.72:0/991505207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5900d0890 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa58ed9d640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5901614d0 0x7fa590154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa587fff640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 0x7fa590153b20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa57800c500 tx=0x7fa57800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5780052d0 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa578005490 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5900d0b00 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578012870 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.724+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5900d10c0 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.726+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5900ce170 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.726+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa578012a10 con 0x7fa5900d4860 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.726+0000 7fa586ffd640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa57006b4d0 0x7fa57006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:44.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.726+0000 7fa58e59c640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa57006b4d0 0x7fa57006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:44.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.726+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa578089e50 con 0x7fa5900d4860 2024-03-06T07:23:44.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.727+0000 7fa58e59c640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa57006b4d0 0x7fa57006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa5800011e0 tx=0x7fa580007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:44.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:44.731+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa578058c80 con 0x7fa5900d4860 2024-03-06T07:23:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.026+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa590068ff0 con 0x7fa5900d4860 2024-03-06T07:23:45.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.029+0000 7fa586ffd640 1 -- 172.21.15.72:0/991505207 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa578058580 con 0x7fa5900d4860 2024-03-06T07:23:45.031 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:45.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa57006b4d0 msgr2=0x7fa57006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa57006b4d0 0x7fa57006d990 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa5800011e0 tx=0x7fa580007d30 comp rx=0 tx=0).stop 2024-03-06T07:23:45.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 msgr2=0x7fa590153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 0x7fa590153b20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fa57800c500 tx=0x7fa57800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 shutdown_connections 2024-03-06T07:23:45.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa57006b4d0 0x7fa57006d990 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa5901614d0 0x7fa590154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5900d4860 0x7fa590153b20 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 --2- 172.21.15.72:0/991505207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa5900b1e40 0x7fa5900b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 >> 172.21.15.72:0/991505207 conn(0x7fa5900d3d40 msgr2=0x7fa59015e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 shutdown_connections 2024-03-06T07:23:45.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.031+0000 7fa594a0e640 1 -- 172.21.15.72:0/991505207 wait complete. 2024-03-06T07:23:45.047 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:45.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.148+0000 7f831711e640 1 Processor -- start 2024-03-06T07:23:45.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.148+0000 7f831711e640 1 -- start start 2024-03-06T07:23:45.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f831711e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f83101455d0 0x7f8310072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f831711e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83101459b0 con 0x7f831006afb0 2024-03-06T07:23:45.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f831711e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83100cf1e0 con 0x7f831006b3f0 2024-03-06T07:23:45.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f831711e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83100cfdf0 con 0x7f83101455d0 2024-03-06T07:23:45.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8314eba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f83101455d0 0x7f8310072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8314eba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f83101455d0 0x7f8310072480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38656/0 (socket says 172.21.15.72:38656) 2024-03-06T07:23:45.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8307fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f831006afb0 0x7f831006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44418/0 (socket says 172.21.15.72:44418) 2024-03-06T07:23:45.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8314eba640 1 -- 172.21.15.72:0/2473479019 learned_addr learned my addr 172.21.15.72:0/2473479019 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:45.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8314eba640 1 -- 172.21.15.72:0/2473479019 >> v1:172.21.15.146:6789/0 conn(0x7f831006b3f0 legacy=0x7f831006e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8314eba640 1 -- 172.21.15.72:0/2473479019 >> v1:172.21.15.72:6789/0 conn(0x7f831006afb0 legacy=0x7f831006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.149+0000 7f8314eba640 1 -- 172.21.15.72:0/2473479019 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83100cf4c0 con 0x7f83101455d0 2024-03-06T07:23:45.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f8314eba640 1 --2- 172.21.15.72:0/2473479019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83101455d0 0x7f8310072480 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f830c009870 tx=0x7f830c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b12927a5487316 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f83077fe640 1 -- 172.21.15.72:0/2473479019 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c034070 con 0x7f83101455d0 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f83077fe640 1 -- 172.21.15.72:0/2473479019 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c030860 con 0x7f83101455d0 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f83077fe640 1 -- 172.21.15.72:0/2473479019 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c030c30 con 0x7f83101455d0 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 -- 172.21.15.72:0/2473479019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83101455d0 msgr2=0x7f8310072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 --2- 172.21.15.72:0/2473479019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83101455d0 0x7f8310072480 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f830c009870 tx=0x7f830c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 -- 172.21.15.72:0/2473479019 shutdown_connections 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 --2- 172.21.15.72:0/2473479019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83101455d0 0x7f8310072480 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 -- 172.21.15.72:0/2473479019 >> 172.21.15.72:0/2473479019 conn(0x7f83100b2660 msgr2=0x7f83100b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 -- 172.21.15.72:0/2473479019 shutdown_connections 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.150+0000 7f831711e640 1 -- 172.21.15.72:0/2473479019 wait complete. 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.151+0000 7f831711e640 1 Processor -- start 2024-03-06T07:23:45.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 -- start start 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831006afb0 0x7f8310138a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 0x7f83100b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83101455d0 0x7f83100aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83100729c0 con 0x7f831006b3f0 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83100cfa80 con 0x7f831006afb0 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f831711e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83100b3400 con 0x7f83101455d0 2024-03-06T07:23:45.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.152+0000 7f8307fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 0x7f83100b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 0x7f83100b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38658/0 (socket says 172.21.15.72:38658) 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 -- 172.21.15.72:0/2472377987 learned_addr learned my addr 172.21.15.72:0/2472377987 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8314eba640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831006afb0 0x7f8310138a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f83156bb640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83101455d0 0x7f83100aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 -- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83101455d0 msgr2=0x7f83100aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83101455d0 0x7f83100aebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 -- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831006afb0 msgr2=0x7f8310138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831006afb0 0x7f8310138a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 -- 172.21.15.72:0/2472377987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831015baa0 con 0x7f831006b3f0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f83156bb640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83101455d0 0x7f83100aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f8307fff640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 0x7f83100b47c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f830000c430 tx=0x7f830000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83000074e0 con 0x7f831006b3f0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8300007680 con 0x7f831006b3f0 2024-03-06T07:23:45.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300004a00 con 0x7f831006b3f0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83100ad860 con 0x7f831006b3f0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.153+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83100ade20 con 0x7f831006b3f0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.154+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831006b820 con 0x7f831006b3f0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.154+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f830001a020 con 0x7f831006b3f0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.155+0000 7f83057fa640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82fc06b4d0 0x7f82fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.155+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8300089950 con 0x7f831006b3f0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.155+0000 7f8314eba640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82fc06b4d0 0x7f82fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.155+0000 7f8314eba640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82fc06b4d0 0x7f82fc06d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f830c036000 tx=0x7f830c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.159+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f831006b820 con 0x7f831006b3f0 2024-03-06T07:23:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.335+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8310068ff0 con 0x7f831006b3f0 2024-03-06T07:23:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.335+0000 7f83057fa640 1 -- 172.21.15.72:0/2472377987 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8300058770 con 0x7f831006b3f0 2024-03-06T07:23:45.337 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82fc06b4d0 msgr2=0x7f82fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82fc06b4d0 0x7f82fc06d990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f830c036000 tx=0x7f830c03b040 comp rx=0 tx=0).stop 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 msgr2=0x7f83100b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 0x7f83100b47c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f830000c430 tx=0x7f830000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 shutdown_connections 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82fc06b4d0 0x7f82fc06d990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83101455d0 0x7f83100aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f831006b3f0 0x7f83100b47c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 --2- 172.21.15.72:0/2472377987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f831006afb0 0x7f8310138a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 >> 172.21.15.72:0/2472377987 conn(0x7f83100b2660 msgr2=0x7f831006f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 shutdown_connections 2024-03-06T07:23:45.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.337+0000 7f831711e640 1 -- 172.21.15.72:0/2472377987 wait complete. 2024-03-06T07:23:45.349 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.452+0000 7fd0a5393640 1 Processor -- start 2024-03-06T07:23:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.452+0000 7fd0a5393640 1 -- start start 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd0a5393640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0a00d4860 0x7fd0a00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd0a5393640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a01618b0 con 0x7fd0a01614d0 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd0a5393640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a00cf400 con 0x7fd0a00b0e30 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd0a5393640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a00d4cf0 con 0x7fd0a00d4860 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd0a01614d0 0x7fd0a00d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44432/0 (socket says 172.21.15.72:44432) 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09effd640 1 -- 172.21.15.72:0/1678402489 learned_addr learned my addr 172.21.15.72:0/1678402489 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09e7fc640 1 --2- 172.21.15.72:0/1678402489 >> v2:172.21.15.146:3300/0 conn(0x7fd0a00d4860 0x7fd0a00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426645338 0 0) 0x7fd0a01618b0 con 0x7fd0a01614d0 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd08c003520 con 0x7fd0a01614d0 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3517312071 0 0) 0x7fd08c003520 con 0x7fd0a01614d0 2024-03-06T07:23:45.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 >> v2:172.21.15.146:3300/0 conn(0x7fd0a00d4860 msgr2=0x7fd0a00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 --2- 172.21.15.72:0/1678402489 >> v2:172.21.15.146:3300/0 conn(0x7fd0a00d4860 0x7fd0a00d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 >> v1:172.21.15.146:6789/0 conn(0x7fd0a00b0e30 legacy=0x7fd0a00b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00cf6a0 con 0x7fd0a01614d0 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd098001a10 con 0x7fd0a01614d0 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 >> v1:172.21.15.72:6789/0 conn(0x7fd0a01614d0 legacy=0x7fd0a00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08c0039b0 0x7fd08c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08c004800 0x7fd08c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0051c0 0x7fd08c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00d4cf0 con 0x7fd08c0039b0 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c00c060 con 0x7fd08c0051c0 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.453+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08c00c260 con 0x7fd08c004800 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09effd640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08c0039b0 0x7fd08c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09e7fc640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08c004800 0x7fd08c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09f7fe640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0051c0 0x7fd08c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09effd640 1 -- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08c004800 msgr2=0x7fd08c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09effd640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08c004800 0x7fd08c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09effd640 1 -- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0051c0 msgr2=0x7fd08c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09effd640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0051c0 0x7fd08c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09effd640 1 -- 172.21.15.72:0/1678402489 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c003520 con 0x7fd08c0039b0 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.454+0000 7fd09f7fe640 1 -- 172.21.15.72:0/1678402489 reap_dead start 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd09effd640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08c0039b0 0x7fd08c003dd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd09800a8f0 tx=0x7fd08c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809c602632698c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09800a4c0 con 0x7fd08c0039b0 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd098034650 con 0x7fd08c0039b0 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd09dffb640 1 -- 172.21.15.72:0/1678402489 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd098034a40 con 0x7fd08c0039b0 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 -- 172.21.15.72:0/1678402489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08c0039b0 msgr2=0x7fd08c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08c0039b0 0x7fd08c003dd0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd09800a8f0 tx=0x7fd08c005830 comp rx=0 tx=0).stop 2024-03-06T07:23:45.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 -- 172.21.15.72:0/1678402489 shutdown_connections 2024-03-06T07:23:45.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 --2- 172.21.15.72:0/1678402489 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08c0039b0 0x7fd08c003dd0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 -- 172.21.15.72:0/1678402489 >> 172.21.15.72:0/1678402489 conn(0x7fd0a00d3d40 msgr2=0x7fd0a00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 -- 172.21.15.72:0/1678402489 shutdown_connections 2024-03-06T07:23:45.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.455+0000 7fd0a5393640 1 -- 172.21.15.72:0/1678402489 wait complete. 2024-03-06T07:23:45.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 Processor -- start 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 -- start start 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0039b0 0x7fd0a00c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a00b7310 0x7fd0a015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 0x7fd0a00b9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00b0910 con 0x7fd0a00b90c0 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00cfc60 con 0x7fd08c0039b0 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.457+0000 7fd0a5393640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00d1910 con 0x7fd0a00b7310 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 0x7fd0a00b9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 0x7fd0a00b9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38680/0 (socket says 172.21.15.72:38680) 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 -- 172.21.15.72:0/2018975610 learned_addr learned my addr 172.21.15.72:0/2018975610 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09e7fc640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a00b7310 0x7fd0a015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09effd640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0039b0 0x7fd0a00c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 -- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a00b7310 msgr2=0x7fd0a015cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a00b7310 0x7fd0a015cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 -- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0039b0 msgr2=0x7fd0a00c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0039b0 0x7fd0a00c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 -- 172.21.15.72:0/2018975610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00b7840 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd09f7fe640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 0x7fd0a00b9540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00cadc0 tx=0x7fd08800f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08801a070 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd088005280 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a00b7b20 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd088007030 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.458+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a00b80e0 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.459+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a00b28c0 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.460+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd08801d070 con 0x7fd0a00b90c0 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.460+0000 7fd0977fe640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd08006b4d0 0x7fd08006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.460+0000 7fd09effd640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd08006b4d0 0x7fd08006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.460+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd0880891e0 con 0x7fd0a00b90c0 2024-03-06T07:23:45.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.461+0000 7fd09effd640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd08006b4d0 0x7fd08006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd098034000 tx=0x7fd09802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.465+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0880579d0 con 0x7fd0a00b90c0 2024-03-06T07:23:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.641+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0a0068ff0 con 0x7fd0a00b90c0 2024-03-06T07:23:45.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.642+0000 7fd0977fe640 1 -- 172.21.15.72:0/2018975610 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd08805ba70 con 0x7fd0a00b90c0 2024-03-06T07:23:45.644 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.643+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd08006b4d0 msgr2=0x7fd08006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.643+0000 7fd0a5393640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd08006b4d0 0x7fd08006d990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd098034000 tx=0x7fd09802f000 comp rx=0 tx=0).stop 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.643+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 msgr2=0x7fd0a00b9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.643+0000 7fd0a5393640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 0x7fd0a00b9540 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00cadc0 tx=0x7fd08800f7d0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 shutdown_connections 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd08006b4d0 0x7fd08006d990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0a00b90c0 0x7fd0a00b9540 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a00b7310 0x7fd0a015cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 --2- 172.21.15.72:0/2018975610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08c0039b0 0x7fd0a00c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 >> 172.21.15.72:0/2018975610 conn(0x7fd0a00d3d40 msgr2=0x7fd0a006c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 shutdown_connections 2024-03-06T07:23:45.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.644+0000 7fd0a5393640 1 -- 172.21.15.72:0/2018975610 wait complete. 2024-03-06T07:23:45.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:23:45.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.757+0000 7fd4f27f7640 1 Processor -- start 2024-03-06T07:23:45.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.757+0000 7fd4f27f7640 1 -- start start 2024-03-06T07:23:45.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f27f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec06e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f27f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4ec06b600 0x7fd4ec1622a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f27f7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec0b04c0 con 0x7fd4ec0700a0 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f27f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec1627e0 con 0x7fd4ec06b600 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f27f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec06d8d0 con 0x7fd4ec06b220 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4ec06b600 0x7fd4ec1622a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4ebfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4ec06b600 0x7fd4ec1622a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:35408/0 (socket says 172.21.15.72:35408) 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 -- 172.21.15.72:0/3991259631 learned_addr learned my addr 172.21.15.72:0/3991259631 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4eb7fe640 1 --1- 172.21.15.72:0/3991259631 >> v1:172.21.15.146:6789/0 conn(0x7fd4ec0700a0 0x7fd4ec06eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:42588/0 (socket says 172.21.15.72:42588) 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 -- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 msgr2=0x7fd4ec06e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 --2- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec06e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 -- 172.21.15.72:0/3991259631 >> v1:172.21.15.146:6789/0 conn(0x7fd4ec0700a0 legacy=0x7fd4ec06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.759+0000 7fd4f0d94640 1 -- 172.21.15.72:0/3991259631 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec06db70 con 0x7fd4ec06b600 2024-03-06T07:23:45.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.760+0000 7fd4f0d94640 1 --2- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec06b600 0x7fd4ec1622a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009e50 tx=0x7fd4e402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad9dc02568ef7a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.760+0000 7fd4eaffd640 1 -- 172.21.15.72:0/3991259631 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e402ca70 con 0x7fd4ec06b600 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.760+0000 7fd4eaffd640 1 -- 172.21.15.72:0/3991259631 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e402cc10 con 0x7fd4ec06b600 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.760+0000 7fd4eaffd640 1 -- 172.21.15.72:0/3991259631 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e40343f0 con 0x7fd4ec06b600 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 -- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec06b600 msgr2=0x7fd4ec1622a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec06b600 0x7fd4ec1622a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009e50 tx=0x7fd4e402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 -- 172.21.15.72:0/3991259631 shutdown_connections 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec06b600 0x7fd4ec1622a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/3991259631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec06e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 -- 172.21.15.72:0/3991259631 >> 172.21.15.72:0/3991259631 conn(0x7fd4ec0d0a10 msgr2=0x7fd4ec0d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 -- 172.21.15.72:0/3991259631 shutdown_connections 2024-03-06T07:23:45.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.761+0000 7fd4f27f7640 1 -- 172.21.15.72:0/3991259631 wait complete. 2024-03-06T07:23:45.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.763+0000 7fd4f27f7640 1 Processor -- start 2024-03-06T07:23:45.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.763+0000 7fd4f27f7640 1 -- start start 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f27f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec146020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f27f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 0x7fd4ec158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f27f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec0700a0 0x7fd4ec0c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f27f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec06fb10 con 0x7fd4ec06b600 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f27f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec06e130 con 0x7fd4ec0700a0 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f27f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec06fc90 con 0x7fd4ec06b220 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4eb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 0x7fd4ec158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4eb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 0x7fd4ec158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38694/0 (socket says 172.21.15.72:38694) 2024-03-06T07:23:45.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4eb7fe640 1 -- 172.21.15.72:0/909165424 learned_addr learned my addr 172.21.15.72:0/909165424 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4f0d94640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec0700a0 0x7fd4ec0c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.764+0000 7fd4ebfff640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec146020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f0d94640 1 -- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 msgr2=0x7fd4ec146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f0d94640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec146020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f0d94640 1 -- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 msgr2=0x7fd4ec158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f0d94640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 0x7fd4ec158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f0d94640 1 -- 172.21.15.72:0/909165424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec159f10 con 0x7fd4ec0700a0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4eb7fe640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 0x7fd4ec158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f0d94640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec0700a0 0x7fd4ec0c59b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4032000 tx=0x7fd4e4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4043020 con 0x7fd4ec0700a0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e403e450 con 0x7fd4ec0700a0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec158e70 con 0x7fd4ec0700a0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4009510 con 0x7fd4ec0700a0 2024-03-06T07:23:45.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.765+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec159430 con 0x7fd4ec0700a0 2024-03-06T07:23:45.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.766+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec04f330 con 0x7fd4ec0700a0 2024-03-06T07:23:45.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.767+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd4e4046020 con 0x7fd4ec0700a0 2024-03-06T07:23:45.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.767+0000 7fd4e8ff9640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd4d806b4d0 0x7fd4d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:45.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.767+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd4e40ad220 con 0x7fd4ec0700a0 2024-03-06T07:23:45.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.767+0000 7fd4ebfff640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd4d806b4d0 0x7fd4d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:45.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.768+0000 7fd4ebfff640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd4d806b4d0 0x7fd4d806d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009aa0 tx=0x7fd4dc009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:45.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.772+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd4e404d030 con 0x7fd4ec0700a0 2024-03-06T07:23:45.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.962+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd4ec074120 con 0x7fd4ec0700a0 2024-03-06T07:23:45.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.962+0000 7fd4e8ff9640 1 -- 172.21.15.72:0/909165424 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd4e407b9f0 con 0x7fd4ec0700a0 2024-03-06T07:23:45.964 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd4d806b4d0 msgr2=0x7fd4d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd4d806b4d0 0x7fd4d806d990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009aa0 tx=0x7fd4dc009190 comp rx=0 tx=0).stop 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec0700a0 msgr2=0x7fd4ec0c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec0700a0 0x7fd4ec0c59b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4032000 tx=0x7fd4e4004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 shutdown_connections 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd4d806b4d0 0x7fd4d806d990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec0700a0 0x7fd4ec0c59b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4ec06b600 0x7fd4ec158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 --2- 172.21.15.72:0/909165424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec06b220 0x7fd4ec146020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 >> 172.21.15.72:0/909165424 conn(0x7fd4ec0d0a10 msgr2=0x7fd4ec0b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 shutdown_connections 2024-03-06T07:23:45.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:45.964+0000 7fd4f27f7640 1 -- 172.21.15.72:0/909165424 wait complete. 2024-03-06T07:23:45.977 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:23:45.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:45.977 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:23:46.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[55436]: fuse finished with error 0 and tester_r 0 2024-03-06T07:23:46.004 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:23:46.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:46.004 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:23:46.149 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:46.149 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:46.167 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:46.167 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:23:46.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:46.167 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:23:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:46.224 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:23:46.225 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:46.225 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:46.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:46.240 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:23:46.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:46.240 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:23:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:46.294 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:23:46.294 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:46.294 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:46.311 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:46.311 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:23:46.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:23:46.311 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:23:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:23:46.368 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:23:46.368 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:46.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.467+0000 7f1f21b5c640 1 Processor -- start 2024-03-06T07:23:46.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.467+0000 7f1f21b5c640 1 -- start start 2024-03-06T07:23:46.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f21b5c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f1c0b4170 0x7f1f1c0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f21b5c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c0d25d0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f21b5c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1c0d2870 con 0x7f1f1c0d4760 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f21b5c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c0d4bf0 con 0x7f1f1c0b4170 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f1b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f1c0b4170 0x7f1f1c0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f1bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1f1c0d0a80 0x7f1f1c0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44442/0 (socket says 172.21.15.72:44442) 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.468+0000 7f1f1bfff640 1 -- 172.21.15.72:0/2033230307 learned_addr learned my addr 172.21.15.72:0/2033230307 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.469+0000 7f1f1b7fe640 1 -- 172.21.15.72:0/2033230307 >> v1:172.21.15.146:6789/0 conn(0x7f1f1c0d4760 legacy=0x7f1f1c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.469+0000 7f1f1b7fe640 1 -- 172.21.15.72:0/2033230307 >> v1:172.21.15.72:6789/0 conn(0x7f1f1c0d0a80 legacy=0x7f1f1c0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.469+0000 7f1f1a7fc640 1 -- 172.21.15.72:0/2033230307 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454850458 0 0) 0x7f1f1c0d25d0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.469+0000 7f1f1b7fe640 1 -- 172.21.15.72:0/2033230307 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0d2b50 con 0x7f1f1c0b4170 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.469+0000 7f1f1b7fe640 1 --2- 172.21.15.72:0/2033230307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0b4170 0x7f1f1c0b6a90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0097a0 tx=0x7f1f0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6eff6290c6ef64 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f1a7fc640 1 -- 172.21.15.72:0/2033230307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c035020 con 0x7f1f1c0b4170 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f1a7fc640 1 -- 172.21.15.72:0/2033230307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c031830 con 0x7f1f1c0b4170 2024-03-06T07:23:46.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f1a7fc640 1 -- 172.21.15.72:0/2033230307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c031c00 con 0x7f1f1c0b4170 2024-03-06T07:23:46.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 -- 172.21.15.72:0/2033230307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0b4170 msgr2=0x7f1f1c0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/2033230307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0b4170 0x7f1f1c0b6a90 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0097a0 tx=0x7f1f0c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 -- 172.21.15.72:0/2033230307 shutdown_connections 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/2033230307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0b4170 0x7f1f1c0b6a90 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 -- 172.21.15.72:0/2033230307 >> 172.21.15.72:0/2033230307 conn(0x7f1f1c0d3c40 msgr2=0x7f1f1c0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 -- 172.21.15.72:0/2033230307 shutdown_connections 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.470+0000 7f1f21b5c640 1 -- 172.21.15.72:0/2033230307 wait complete. 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.471+0000 7f1f21b5c640 1 Processor -- start 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.471+0000 7f1f21b5c640 1 -- start start 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f21b5c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f1c0b4170 0x7f1f1c0b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f21b5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 0x7f1f1c0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f21b5c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1c157470 0x7f1f1c0cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f21b5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0b4dd0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f21b5c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0ce9e0 con 0x7f1f1c157470 2024-03-06T07:23:46.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f21b5c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0b4f50 con 0x7f1f1c0b4170 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 0x7f1f1c0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 0x7f1f1c0b1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38704/0 (socket says 172.21.15.72:38704) 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 -- 172.21.15.72:0/3664102487 learned_addr learned my addr 172.21.15.72:0/3664102487 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1b7fe640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f1c0b4170 0x7f1f1c0b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1bfff640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1c157470 0x7f1f1c0cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 -- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f1c0b4170 msgr2=0x7f1f1c0b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f1c0b4170 0x7f1f1c0b2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 -- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1c157470 msgr2=0x7f1f1c0cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1c157470 0x7f1f1c0cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.472+0000 7f1f1affd640 1 -- 172.21.15.72:0/3664102487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0cdcc0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f1bfff640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1c157470 0x7f1f1c0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f1b7fe640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f1c0b4170 0x7f1f1c0b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f1affd640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 0x7f1f1c0b1040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000b420 tx=0x7f1f1000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10011830 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10016070 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c0b1690 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10011c10 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.473+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c0b1c50 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.474+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c04f330 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.474+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1f1001b460 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.475+0000 7f1f03fff640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f0806b4d0 0x7f1f0806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.475+0000 7f1f1b7fe640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f0806b4d0 0x7f1f0806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.475+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1f10089cb0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.475+0000 7f1f1b7fe640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f0806b4d0 0x7f1f0806d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036000 tx=0x7f1f0c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.479+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1f10020020 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.694+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f1c0733b0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.695+0000 7f1f03fff640 1 -- 172.21.15.72:0/3664102487 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f1f10058af0 con 0x7f1f1c0d0a80 2024-03-06T07:23:46.697 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:46.697 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:22.210297+0000","modified":"2024-03-06T07:23:24.220196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5670":{"gid":5670,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/874695441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":874695441},{"type":"v1","addr":"172.21.15.72:6837","nonce":874695441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5670,"qdb_cluster":[5670]},"id":3}]} 2024-03-06T07:23:46.698 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 20 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f0806b4d0 msgr2=0x7f1f0806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f0806b4d0 0x7f1f0806d990 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c036000 tx=0x7f1f0c008030 comp rx=0 tx=0).stop 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 msgr2=0x7f1f1c0b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 0x7f1f1c0b1040 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000b420 tx=0x7f1f1000b8e0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 shutdown_connections 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1f0806b4d0 0x7f1f0806d990 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f1c157470 0x7f1f1c0cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f1c0d0a80 0x7f1f1c0b1040 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 --2- 172.21.15.72:0/3664102487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f1c0b4170 0x7f1f1c0b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 >> 172.21.15.72:0/3664102487 conn(0x7f1f1c0d3c40 msgr2=0x7f1f1c06b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 shutdown_connections 2024-03-06T07:23:46.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.697+0000 7f1f21b5c640 1 -- 172.21.15.72:0/3664102487 wait complete. 2024-03-06T07:23:46.714 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.813+0000 7f2de8ab5640 1 Processor -- start 2024-03-06T07:23:46.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.813+0000 7f2de8ab5640 1 -- start start 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de8ab5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 0x7f2de40ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de8ab5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de40cebe0 con 0x7f2de40d16e0 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de8ab5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de40ceec0 con 0x7f2de415bdd0 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de8ab5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40cf1a0 con 0x7f2de415fa10 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de2d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2de40d16e0 0x7f2de40d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44448/0 (socket says 172.21.15.72:44448) 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de2d9d640 1 -- 172.21.15.72:0/73987108 learned_addr learned my addr 172.21.15.72:0/73987108 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2dd9d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 0x7f2de40ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767586318 0 0) 0x7f2de40cebe0 con 0x7f2de40d16e0 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd0003520 con 0x7f2de40d16e0 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131820482 0 0) 0x7f2dd0003520 con 0x7f2de40d16e0 2024-03-06T07:23:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 msgr2=0x7f2de40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 0x7f2de40ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 >> v1:172.21.15.146:6789/0 conn(0x7f2de415bdd0 legacy=0x7f2de40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de40cc8e0 con 0x7f2de40d16e0 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2ddc0019f0 con 0x7f2de40d16e0 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 >> v1:172.21.15.72:6789/0 conn(0x7f2de40d16e0 legacy=0x7f2de40d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00039e0 0x7f2dd0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dd0004390 0x7f2dd0004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 0x7f2dd0005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40cf1a0 con 0x7f2dd0004390 2024-03-06T07:23:46.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd000baa0 con 0x7f2dd00039e0 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd000bca0 con 0x7f2de415fa10 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2dd9d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dd0004390 0x7f2dd0004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2de2d9d640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 0x7f2dd0005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.814+0000 7f2de259c640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00039e0 0x7f2dd0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2dd9d9b640 1 -- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 msgr2=0x7f2dd0005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2dd9d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de415fa10 0x7f2dd0005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2dd9d9b640 1 -- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00039e0 msgr2=0x7f2dd0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2dd9d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2dd00039e0 0x7f2dd0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2dd9d9b640 1 -- 172.21.15.72:0/73987108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0003520 con 0x7f2dd0004390 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.815+0000 7f2dd9d9b640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dd0004390 0x7f2dd0004860 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009ac0 tx=0x7f2dd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6b210546dfbabf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc039070 con 0x7f2dd0004390 2024-03-06T07:23:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dcc02c6b0 con 0x7f2dd0004390 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de1d9b640 1 -- 172.21.15.72:0/73987108 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc009690 con 0x7f2dd0004390 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de8ab5640 1 -- 172.21.15.72:0/73987108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dd0004390 msgr2=0x7f2dd0004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/73987108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2dd0004390 0x7f2dd0004860 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc009ac0 tx=0x7f2dd0005830 comp rx=0 tx=0).stop 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de2d9d640 1 -- 172.21.15.72:0/73987108 reap_dead start 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de8ab5640 1 -- 172.21.15.72:0/73987108 shutdown_connections 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de8ab5640 1 -- 172.21.15.72:0/73987108 >> 172.21.15.72:0/73987108 conn(0x7f2de409a9f0 msgr2=0x7f2de409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de8ab5640 1 -- 172.21.15.72:0/73987108 shutdown_connections 2024-03-06T07:23:46.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.816+0000 7f2de8ab5640 1 -- 172.21.15.72:0/73987108 wait complete. 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 Processor -- start 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 -- start start 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2dd0004390 0x7f2de41f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de406ee10 0x7f2de40b6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 0x7f2de40b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40ce270 con 0x7f2de4070140 2024-03-06T07:23:46.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415c1b0 con 0x7f2de406ee10 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de8ab5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de40ce3f0 con 0x7f2dd0004390 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 0x7f2de40b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 0x7f2de40b68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38724/0 (socket says 172.21.15.72:38724) 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.818+0000 7f2de2d9d640 1 -- 172.21.15.72:0/1719258477 learned_addr learned my addr 172.21.15.72:0/1719258477 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de259c640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2dd0004390 0x7f2de41f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2dd9d9b640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de406ee10 0x7f2de40b6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de2d9d640 1 -- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2dd0004390 msgr2=0x7f2de41f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de2d9d640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2dd0004390 0x7f2de41f0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de2d9d640 1 -- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de406ee10 msgr2=0x7f2de40b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de2d9d640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de406ee10 0x7f2de40b6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de2d9d640 1 -- 172.21.15.72:0/1719258477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de41f39e0 con 0x7f2de4070140 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de2d9d640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 0x7f2de40b68b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2de406c090 tx=0x7f2ddc007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc00ab50 con 0x7f2de4070140 2024-03-06T07:23:46.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc018070 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de41f3ca0 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0133f0 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.819+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de41f4260 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.820+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de404f330 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.820+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ddc013590 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.821+0000 7f2ddb7fe640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2dc406b500 0x7f2dc406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.821+0000 7f2de259c640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2dc406b500 0x7f2dc406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.821+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ddc08a7d0 con 0x7f2de4070140 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.821+0000 7f2de259c640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2dc406b500 0x7f2dc406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2de4070060 tx=0x7f2dd4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:46.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:46.825+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ddc059610 con 0x7f2de4070140 2024-03-06T07:23:47.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.014+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2de40b7a40 con 0x7f2de4070140 2024-03-06T07:23:47.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.015+0000 7f2ddb7fe640 1 -- 172.21.15.72:0/1719258477 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f2ddc058f10 con 0x7f2de4070140 2024-03-06T07:23:47.017 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:47.017 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":48,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:32.024853+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.32","mappings":[{"from":0,"to":4}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:47.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2dc406b500 msgr2=0x7f2dc406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2dc406b500 0x7f2dc406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2de4070060 tx=0x7f2dd4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 msgr2=0x7f2de40b68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 0x7f2de40b68b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2de406c090 tx=0x7f2ddc007bd0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 shutdown_connections 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2dc406b500 0x7f2dc406d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2de4070140 0x7f2de40b68b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de406ee10 0x7f2de40b6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 --2- 172.21.15.72:0/1719258477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2dd0004390 0x7f2de41f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 >> 172.21.15.72:0/1719258477 conn(0x7f2de409a9f0 msgr2=0x7f2de40ccd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 shutdown_connections 2024-03-06T07:23:47.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.017+0000 7f2de8ab5640 1 -- 172.21.15.72:0/1719258477 wait complete. 2024-03-06T07:23:47.031 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:20.432633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:21.177789+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:47.031 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:47.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.136+0000 7fc61d60f640 1 Processor -- start 2024-03-06T07:23:47.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.136+0000 7fc61d60f640 1 -- start start 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc61d60f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc6180b42e0 0x7fc618165680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc61d60f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61815b260 con 0x7fc618071a70 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc61d60f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc61815b540 con 0x7fc61815ad90 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc61d60f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc61815c020 con 0x7fc6180b42e0 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc616ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc618071a70 0x7fc618071e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44450/0 (socket says 172.21.15.72:44450) 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc616ffd640 1 -- 172.21.15.72:0/2463761373 learned_addr learned my addr 172.21.15.72:0/2463761373 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.137+0000 7fc6177fe640 1 --2- 172.21.15.72:0/2463761373 >> v2:172.21.15.72:3300/0 conn(0x7fc6180b42e0 0x7fc618165680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.138+0000 7fc6177fe640 1 -- 172.21.15.72:0/2463761373 >> v1:172.21.15.146:6789/0 conn(0x7fc61815ad90 legacy=0x7fc6180b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.138+0000 7fc6177fe640 1 -- 172.21.15.72:0/2463761373 >> v1:172.21.15.72:6789/0 conn(0x7fc618071a70 legacy=0x7fc618071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.138+0000 7fc6177fe640 1 -- 172.21.15.72:0/2463761373 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61815b820 con 0x7fc6180b42e0 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.138+0000 7fc6177fe640 1 --2- 172.21.15.72:0/2463761373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6180b42e0 0x7fc618165680 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc610009d90 tx=0x7fc61002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5ead1765f2d71c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc6167fc640 1 -- 172.21.15.72:0/2463761373 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610039070 con 0x7fc6180b42e0 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc6167fc640 1 -- 172.21.15.72:0/2463761373 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61002cbd0 con 0x7fc6180b42e0 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc6167fc640 1 -- 172.21.15.72:0/2463761373 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc610034440 con 0x7fc6180b42e0 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 -- 172.21.15.72:0/2463761373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6180b42e0 msgr2=0x7fc618165680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2463761373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6180b42e0 0x7fc618165680 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc610009d90 tx=0x7fc61002be20 comp rx=0 tx=0).stop 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 -- 172.21.15.72:0/2463761373 shutdown_connections 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2463761373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6180b42e0 0x7fc618165680 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 -- 172.21.15.72:0/2463761373 >> 172.21.15.72:0/2463761373 conn(0x7fc6180b2660 msgr2=0x7fc6180b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 -- 172.21.15.72:0/2463761373 shutdown_connections 2024-03-06T07:23:47.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.139+0000 7fc61d60f640 1 -- 172.21.15.72:0/2463761373 wait complete. 2024-03-06T07:23:47.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.141+0000 7fc61d60f640 1 Processor -- start 2024-03-06T07:23:47.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.141+0000 7fc61d60f640 1 -- start start 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 0x7fc6181412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6180b42e0 0x7fc618156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc61815ad90 0x7fc6181565b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61815d220 con 0x7fc618071a70 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6180b4750 con 0x7fc61815ad90 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61815d3a0 con 0x7fc6180b42e0 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 0x7fc6181412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 0x7fc6181412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38748/0 (socket says 172.21.15.72:38748) 2024-03-06T07:23:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 -- 172.21.15.72:0/2165416494 learned_addr learned my addr 172.21.15.72:0/2165416494 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc6177fe640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc61815ad90 0x7fc6181565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc60e7fc640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6180b42e0 0x7fc618156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 -- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6180b42e0 msgr2=0x7fc618156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6180b42e0 0x7fc618156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 -- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc61815ad90 msgr2=0x7fc6181565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc61815ad90 0x7fc6181565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 -- 172.21.15.72:0/2165416494 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618156c30 con 0x7fc618071a70 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc616ffd640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 0x7fc6181412f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc60800c400 tx=0x7fc60800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608017070 con 0x7fc618071a70 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc608005ce0 con 0x7fc618071a70 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc618156f10 con 0x7fc618071a70 2024-03-06T07:23:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6080072f0 con 0x7fc618071a70 2024-03-06T07:23:47.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.143+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6181574d0 con 0x7fc618071a70 2024-03-06T07:23:47.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.144+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc618073390 con 0x7fc618071a70 2024-03-06T07:23:47.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.145+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc60801a070 con 0x7fc618071a70 2024-03-06T07:23:47.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.145+0000 7fc60ffff640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.145+0000 7fc60e7fc640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.145+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc608089680 con 0x7fc618071a70 2024-03-06T07:23:47.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.146+0000 7fc60e7fc640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc5f806b4d0 0x7fc5f806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.153+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6080584c0 con 0x7fc618071a70 2024-03-06T07:23:47.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.324+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6180d07d0 con 0x7fc618071a70 2024-03-06T07:23:47.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.324+0000 7fc60ffff640 1 -- 172.21.15.72:0/2165416494 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc608057dc0 con 0x7fc618071a70 2024-03-06T07:23:47.326 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc5f806b4d0 msgr2=0x7fc5f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc5f806b4d0 0x7fc5f806d990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc600006fd0 tx=0x7fc600008040 comp rx=0 tx=0).stop 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 msgr2=0x7fc6181412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 0x7fc6181412f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc60800c400 tx=0x7fc60800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 shutdown_connections 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc5f806b4d0 0x7fc5f806d990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc61815ad90 0x7fc6181565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6180b42e0 0x7fc618156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 --2- 172.21.15.72:0/2165416494 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc618071a70 0x7fc6181412f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 >> 172.21.15.72:0/2165416494 conn(0x7fc6180b2660 msgr2=0x7fc618158ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 shutdown_connections 2024-03-06T07:23:47.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.326+0000 7fc61d60f640 1 -- 172.21.15.72:0/2165416494 wait complete. 2024-03-06T07:23:47.341 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:23:47.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.443+0000 7f4480429640 1 Processor -- start 2024-03-06T07:23:47.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.443+0000 7f4480429640 1 -- start start 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f4480429640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d14a0 0x7f447815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f4480429640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f447815eb00 0x7f44780cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f4480429640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44780cfa50 con 0x7f44780d10c0 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f4480429640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447815c050 con 0x7f447815eb00 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f4480429640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44780d0060 con 0x7f44780d14a0 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f447815eb00 0x7f44780cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f447815eb00 0x7f44780cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38760/0 (socket says 172.21.15.72:38760) 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 -- 172.21.15.72:0/932281746 learned_addr learned my addr 172.21.15.72:0/932281746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447d9c4640 1 --2- 172.21.15.72:0/932281746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d14a0 0x7f447815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 -- 172.21.15.72:0/932281746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d14a0 msgr2=0x7f447815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 --2- 172.21.15.72:0/932281746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d14a0 0x7f447815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 -- 172.21.15.72:0/932281746 >> v1:172.21.15.146:6789/0 conn(0x7f44780d10c0 legacy=0x7f44780d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.444+0000 7f447e9c6640 1 -- 172.21.15.72:0/932281746 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44780b45c0 con 0x7f447815eb00 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.445+0000 7f447e9c6640 1 --2- 172.21.15.72:0/932281746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447815eb00 0x7f44780cae80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4470009e50 tx=0x7f447002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f48d2cc6eed4f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.445+0000 7f447d1c3640 1 -- 172.21.15.72:0/932281746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447002cb00 con 0x7f447815eb00 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.445+0000 7f447d1c3640 1 -- 172.21.15.72:0/932281746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447002cca0 con 0x7f447815eb00 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.445+0000 7f447d1c3640 1 -- 172.21.15.72:0/932281746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470034610 con 0x7f447815eb00 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 -- 172.21.15.72:0/932281746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447815eb00 msgr2=0x7f44780cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 --2- 172.21.15.72:0/932281746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447815eb00 0x7f44780cae80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f4470009e50 tx=0x7f447002bf60 comp rx=0 tx=0).stop 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 -- 172.21.15.72:0/932281746 shutdown_connections 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 --2- 172.21.15.72:0/932281746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447815eb00 0x7f44780cae80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 --2- 172.21.15.72:0/932281746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d14a0 0x7f447815e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 -- 172.21.15.72:0/932281746 >> 172.21.15.72:0/932281746 conn(0x7f44780b3800 msgr2=0x7f44780b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 -- 172.21.15.72:0/932281746 shutdown_connections 2024-03-06T07:23:47.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.446+0000 7f4480429640 1 -- 172.21.15.72:0/932281746 wait complete. 2024-03-06T07:23:47.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.448+0000 7f4480429640 1 Processor -- start 2024-03-06T07:23:47.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.448+0000 7f4480429640 1 -- start start 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f4480429640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d10c0 0x7f4478070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f4480429640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 0x7f447814a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f4480429640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f447815eb00 0x7f447815f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f4480429640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44780cfcf0 con 0x7f44780d14a0 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f4480429640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447815f440 con 0x7f447815eb00 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f4480429640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44780cb3c0 con 0x7f44780d10c0 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f447d9c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 0x7f447814a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f447d9c4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 0x7f447814a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38770/0 (socket says 172.21.15.72:38770) 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f447d9c4640 1 -- 172.21.15.72:0/1565082437 learned_addr learned my addr 172.21.15.72:0/1565082437 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f447e1c5640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d10c0 0x7f4478070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.449+0000 7f447e9c6640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f447815eb00 0x7f447815f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447d9c4640 1 -- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d10c0 msgr2=0x7f4478070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447d9c4640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d10c0 0x7f4478070130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447d9c4640 1 -- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f447815eb00 msgr2=0x7f447815f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447d9c4640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f447815eb00 0x7f447815f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447d9c4640 1 -- 172.21.15.72:0/1565082437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44780b59a0 con 0x7f44780d14a0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447e9c6640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f447815eb00 0x7f447815f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f447d9c4640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 0x7f447814a660 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f446800d800 tx=0x7f446800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446800b6c0 con 0x7f44780d14a0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4468019070 con 0x7f44780d14a0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446800ba80 con 0x7f44780d14a0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44780b4e20 con 0x7f44780d14a0 2024-03-06T07:23:47.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.450+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44780b53e0 con 0x7f44780d14a0 2024-03-06T07:23:47.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.451+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4478068ff0 con 0x7f44780d14a0 2024-03-06T07:23:47.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.451+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f446800bc20 con 0x7f44780d14a0 2024-03-06T07:23:47.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.452+0000 7f446effd640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f445c06b500 0x7f445c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.452+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f446808a5a0 con 0x7f44780d14a0 2024-03-06T07:23:47.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.452+0000 7f447e1c5640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f445c06b500 0x7f445c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.452+0000 7f447e1c5640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f445c06b500 0x7f445c06d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4474001780 tx=0x7f4474009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.460+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4468059470 con 0x7f44780d14a0 2024-03-06T07:23:47.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.772+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44780d0c30 con 0x7f44780d14a0 2024-03-06T07:23:47.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.772+0000 7f446effd640 1 -- 172.21.15.72:0/1565082437 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4468058d70 con 0x7f44780d14a0 2024-03-06T07:23:47.774 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.774+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f445c06b500 msgr2=0x7f445c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.774+0000 7f4480429640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f445c06b500 0x7f445c06d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4474001780 tx=0x7f4474009040 comp rx=0 tx=0).stop 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 msgr2=0x7f447814a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 0x7f447814a660 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f446800d800 tx=0x7f446800dcc0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 shutdown_connections 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f445c06b500 0x7f445c06d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f447815eb00 0x7f447815f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44780d14a0 0x7f447814a660 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 --2- 172.21.15.72:0/1565082437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44780d10c0 0x7f4478070130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 >> 172.21.15.72:0/1565082437 conn(0x7f44780b3800 msgr2=0x7f44780d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 shutdown_connections 2024-03-06T07:23:47.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.775+0000 7f4480429640 1 -- 172.21.15.72:0/1565082437 wait complete. 2024-03-06T07:23:47.790 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:23:47.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.891+0000 7f69b0156640 1 Processor -- start 2024-03-06T07:23:47.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.891+0000 7f69b0156640 1 -- start start 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69b0156640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f69a80d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69b0156640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80cfa50 con 0x7f69a80d14a0 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69b0156640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a80d0a30 con 0x7f69a815ecd0 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69b0156640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815c050 con 0x7f69a80d10c0 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69ad6f1640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f69a80d14a0 0x7f69a815e5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44458/0 (socket says 172.21.15.72:44458) 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69ad6f1640 1 -- 172.21.15.72:0/1075232089 learned_addr learned my addr 172.21.15.72:0/1075232089 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217634841 0 0) 0x7f69a80cfa50 con 0x7f69a80d14a0 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6994003520 con 0x7f69a80d14a0 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.892+0000 7f69adef2640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f69a80d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410225282 0 0) 0x7f6994003520 con 0x7f69a80d14a0 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 msgr2=0x7f69a80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f69a80d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 >> v1:172.21.15.146:6789/0 conn(0x7f69a815ecd0 legacy=0x7f69a80cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a80b45c0 con 0x7f69a80d14a0 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f69a4001670 con 0x7f69a80d14a0 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f69a80d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 >> v1:172.21.15.72:6789/0 conn(0x7f69a80d14a0 legacy=0x7f69a815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69940039e0 0x7f6994003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f6994004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6994004d40 0x7f69940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699400baa0 con 0x7f69940039e0 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699400bc50 con 0x7f6994004d40 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699400be50 con 0x7f69a80d10c0 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69940039e0 0x7f6994003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69ad6f1640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f6994004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69ae6f3640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6994004d40 0x7f69940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 -- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 msgr2=0x7f6994004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69a80d10c0 0x7f6994004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 -- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6994004d40 msgr2=0x7f69940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6994004d40 0x7f69940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.893+0000 7f69adef2640 1 -- 172.21.15.72:0/1075232089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994003520 con 0x7f69940039e0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69adef2640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69940039e0 0x7f6994003e00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6998006fd0 tx=0x7f6994005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f400f93b87d54a1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699800f040 con 0x7f69940039e0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699802e070 con 0x7f69940039e0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69acef0640 1 -- 172.21.15.72:0/1075232089 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998004450 con 0x7f69940039e0 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69b0156640 1 -- 172.21.15.72:0/1075232089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69940039e0 msgr2=0x7f6994003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69b0156640 1 --2- 172.21.15.72:0/1075232089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69940039e0 0x7f6994003e00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f6998006fd0 tx=0x7f6994005830 comp rx=0 tx=0).stop 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69ae6f3640 1 -- 172.21.15.72:0/1075232089 reap_dead start 2024-03-06T07:23:47.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69b0156640 1 -- 172.21.15.72:0/1075232089 shutdown_connections 2024-03-06T07:23:47.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.894+0000 7f69b0156640 1 -- 172.21.15.72:0/1075232089 >> 172.21.15.72:0/1075232089 conn(0x7f69a80b3800 msgr2=0x7f69a80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:47.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.895+0000 7f69b0156640 1 -- 172.21.15.72:0/1075232089 shutdown_connections 2024-03-06T07:23:47.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.895+0000 7f69b0156640 1 -- 172.21.15.72:0/1075232089 wait complete. 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 Processor -- start 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 -- start start 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69940039e0 0x7f69a8159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a814f740 0x7f69a80b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 0x7f69a81f76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a815f440 con 0x7f69a80b7470 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80d0060 con 0x7f69a814f740 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69b0156640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a80cfcf0 con 0x7f69940039e0 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69ae6f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 0x7f69a81f76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69ae6f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 0x7f69a81f76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38780/0 (socket says 172.21.15.72:38780) 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.896+0000 7f69ae6f3640 1 -- 172.21.15.72:0/665463442 learned_addr learned my addr 172.21.15.72:0/665463442 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69adef2640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69940039e0 0x7f69a8159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ad6f1640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a814f740 0x7f69a80b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ae6f3640 1 -- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69940039e0 msgr2=0x7f69a8159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ae6f3640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69940039e0 0x7f69a8159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ae6f3640 1 -- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a814f740 msgr2=0x7f69a80b6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ae6f3640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a814f740 0x7f69a80b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ae6f3640 1 -- 172.21.15.72:0/665463442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8159100 con 0x7f69a80b7470 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69ae6f3640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 0x7f69a81f76a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f69a0000f80 tx=0x7f69a000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0003970 con 0x7f69a80b7470 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a0018070 con 0x7f69a80b7470 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a81593e0 con 0x7f69a80b7470 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a0003d50 con 0x7f69a80b7470 2024-03-06T07:23:47.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.897+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a81599a0 con 0x7f69a80b7470 2024-03-06T07:23:47.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.898+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a809fe60 con 0x7f69a80b7470 2024-03-06T07:23:47.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.899+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69a00050d0 con 0x7f69a80b7470 2024-03-06T07:23:47.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.899+0000 7f699e7fc640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f698c06b500 0x7f698c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:47.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.899+0000 7f69adef2640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f698c06b500 0x7f698c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:47.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.899+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f69a0089f30 con 0x7f69a80b7470 2024-03-06T07:23:47.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.899+0000 7f69adef2640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f698c06b500 0x7f698c06d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6998035000 tx=0x7f6998031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:47.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:47.907+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69a0058d70 con 0x7f69a80b7470 2024-03-06T07:23:48.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.085+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69a8159b70 con 0x7f69a80b7470 2024-03-06T07:23:48.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.085+0000 7f699e7fc640 1 -- 172.21.15.72:0/665463442 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7f69a0058670 con 0x7f69a80b7470 2024-03-06T07:23:48.087 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f698c06b500 msgr2=0x7f698c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f698c06b500 0x7f698c06d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6998035000 tx=0x7f6998031000 comp rx=0 tx=0).stop 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 msgr2=0x7f69a81f76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 0x7f69a81f76a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f69a0000f80 tx=0x7f69a000ec10 comp rx=0 tx=0).stop 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 shutdown_connections 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f698c06b500 0x7f698c06d9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69a80b7470 0x7f69a81f76a0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69a814f740 0x7f69a80b6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 --2- 172.21.15.72:0/665463442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69940039e0 0x7f69a8159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 >> 172.21.15.72:0/665463442 conn(0x7f69a80b3800 msgr2=0x7f69a80b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 shutdown_connections 2024-03-06T07:23:48.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.087+0000 7f69b0156640 1 -- 172.21.15.72:0/665463442 wait complete. 2024-03-06T07:23:48.101 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:48.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.200+0000 7f830fd47640 1 Processor -- start 2024-03-06T07:23:48.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830fd47640 1 -- start start 2024-03-06T07:23:48.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830fd47640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f830815dfe0 0x7f83080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830fd47640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f83080d29c0 0x7f83080b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830fd47640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080b3430 con 0x7f83080cf830 2024-03-06T07:23:48.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830fd47640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83080b31e0 con 0x7f83080d29c0 2024-03-06T07:23:48.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830fd47640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83080b3d60 con 0x7f830815dfe0 2024-03-06T07:23:48.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830e2e4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f83080d29c0 0x7f83080b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.201+0000 7f830d2e2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f83080cf830 0x7f83080d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44470/0 (socket says 172.21.15.72:44470) 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830d2e2640 1 -- 172.21.15.72:0/536926314 learned_addr learned my addr 172.21.15.72:0/536926314 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830dae3640 1 --2- 172.21.15.72:0/536926314 >> v2:172.21.15.146:3300/0 conn(0x7f830815dfe0 0x7f83080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830e2e4640 1 -- 172.21.15.72:0/536926314 >> v2:172.21.15.146:3300/0 conn(0x7f830815dfe0 msgr2=0x7f83080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830e2e4640 1 --2- 172.21.15.72:0/536926314 >> v2:172.21.15.146:3300/0 conn(0x7f830815dfe0 0x7f83080cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830e2e4640 1 -- 172.21.15.72:0/536926314 >> v1:172.21.15.72:6789/0 conn(0x7f83080cf830 legacy=0x7f83080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830e2e4640 1 -- 172.21.15.72:0/536926314 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83080b3710 con 0x7f83080d29c0 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830dae3640 1 --2- 172.21.15.72:0/536926314 >> v2:172.21.15.146:3300/0 conn(0x7f830815dfe0 0x7f83080cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:48.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.202+0000 7f830e2e4640 1 --2- 172.21.15.72:0/536926314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080d29c0 0x7f83080b41f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8308067990 tx=0x7f830002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b169533f2e0e1061 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830cae1640 1 -- 172.21.15.72:0/536926314 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300039070 con 0x7f83080d29c0 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830cae1640 1 -- 172.21.15.72:0/536926314 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830002cc80 con 0x7f83080d29c0 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830cae1640 1 -- 172.21.15.72:0/536926314 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300034660 con 0x7f83080d29c0 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 -- 172.21.15.72:0/536926314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080d29c0 msgr2=0x7f83080b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 --2- 172.21.15.72:0/536926314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080d29c0 0x7f83080b41f0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f8308067990 tx=0x7f830002bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 -- 172.21.15.72:0/536926314 shutdown_connections 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 --2- 172.21.15.72:0/536926314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83080d29c0 0x7f83080b41f0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 --2- 172.21.15.72:0/536926314 >> v2:172.21.15.146:3300/0 conn(0x7f830815dfe0 0x7f83080cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 -- 172.21.15.72:0/536926314 >> 172.21.15.72:0/536926314 conn(0x7f83080d3cd0 msgr2=0x7f83080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 -- 172.21.15.72:0/536926314 shutdown_connections 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.203+0000 7f830fd47640 1 -- 172.21.15.72:0/536926314 wait complete. 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.204+0000 7f830fd47640 1 Processor -- start 2024-03-06T07:23:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.204+0000 7f830fd47640 1 -- start start 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830fd47640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83080cf830 0x7f83080cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830fd47640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83080d29c0 0x7f8308151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830fd47640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 0x7f83080cc080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830fd47640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080d1df0 con 0x7f830815dfe0 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830fd47640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080d1c70 con 0x7f83080d29c0 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830fd47640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830815e3c0 con 0x7f83080cf830 2024-03-06T07:23:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830e2e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 0x7f83080cc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830e2e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 0x7f83080cc080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38802/0 (socket says 172.21.15.72:38802) 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830e2e4640 1 -- 172.21.15.72:0/1451713008 learned_addr learned my addr 172.21.15.72:0/1451713008 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830d2e2640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83080d29c0 0x7f8308151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.205+0000 7f830dae3640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83080cf830 0x7f83080cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830e2e4640 1 -- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83080cf830 msgr2=0x7f83080cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830e2e4640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83080cf830 0x7f83080cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830e2e4640 1 -- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83080d29c0 msgr2=0x7f8308151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830e2e4640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83080d29c0 0x7f8308151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830e2e4640 1 -- 172.21.15.72:0/1451713008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83080cc9d0 con 0x7f830815dfe0 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830d2e2640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83080d29c0 0x7f8308151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830dae3640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83080cf830 0x7f83080cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830e2e4640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 0x7f83080cc080 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f830002c4e0 tx=0x7f8300004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8300039050 con 0x7f830815dfe0 2024-03-06T07:23:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8300045070 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83080cccb0 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830003e830 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.206+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83080cd270 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.207+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830804f330 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.208+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f830003e9d0 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.208+0000 7f82fe7fc640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82ec06b4d0 0x7f82ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.208+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f83000ad220 con 0x7f830815dfe0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.208+0000 7f830dae3640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82ec06b4d0 0x7f82ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.208+0000 7f830dae3640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82ec06b4d0 0x7f82ec06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f830400a990 tx=0x7f8304008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.212+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f830007c210 con 0x7f830815dfe0 2024-03-06T07:23:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.423+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83080b2cf0 con 0x7f830815dfe0 2024-03-06T07:23:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.424+0000 7f82fe7fc640 1 -- 172.21.15.72:0/1451713008 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7f830007bb10 con 0x7f830815dfe0 2024-03-06T07:23:48.426 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:48.426 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:22.210297+0000","modified":"2024-03-06T07:23:24.220196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5670":{"gid":5670,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/874695441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":874695441},{"type":"v1","addr":"172.21.15.72:6837","nonce":874695441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5670,"qdb_cluster":[5670]},"id":3}]} 2024-03-06T07:23:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 20 2024-03-06T07:23:48.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.425+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82ec06b4d0 msgr2=0x7f82ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82ec06b4d0 0x7f82ec06d990 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f830400a990 tx=0x7f8304008040 comp rx=0 tx=0).stop 2024-03-06T07:23:48.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 msgr2=0x7f83080cc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 0x7f83080cc080 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f830002c4e0 tx=0x7f8300004a40 comp rx=0 tx=0).stop 2024-03-06T07:23:48.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 shutdown_connections 2024-03-06T07:23:48.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82ec06b4d0 0x7f82ec06d990 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f830815dfe0 0x7f83080cc080 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83080d29c0 0x7f8308151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 --2- 172.21.15.72:0/1451713008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83080cf830 0x7f83080cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 >> 172.21.15.72:0/1451713008 conn(0x7f83080d3cd0 msgr2=0x7f83080b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 shutdown_connections 2024-03-06T07:23:48.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.426+0000 7f830fd47640 1 -- 172.21.15.72:0/1451713008 wait complete. 2024-03-06T07:23:48.439 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.537+0000 7f62f4da0640 1 Processor -- start 2024-03-06T07:23:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.537+0000 7f62f4da0640 1 -- start start 2024-03-06T07:23:48.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62f4da0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62f00cf850 0x7f62f00cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62f4da0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f015e380 con 0x7f62f00d00c0 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62f4da0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f00cdc10 con 0x7f62f006cf20 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62f4da0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62f006ca90 con 0x7f62f00cf850 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62ee59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f62f00d00c0 0x7f62f00d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44486/0 (socket says 172.21.15.72:44486) 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62ee59c640 1 -- 172.21.15.72:0/2628810784 learned_addr learned my addr 172.21.15.72:0/2628810784 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62edd9b640 1 --2- 172.21.15.72:0/2628810784 >> v2:172.21.15.146:3300/0 conn(0x7f62f00cf850 0x7f62f00cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520442839 0 0) 0x7f62f015e380 con 0x7f62f00d00c0 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.538+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62d8003520 con 0x7f62f00d00c0 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420891589 0 0) 0x7f62d8003520 con 0x7f62f00d00c0 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 >> v2:172.21.15.146:3300/0 conn(0x7f62f00cf850 msgr2=0x7f62f00cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 --2- 172.21.15.72:0/2628810784 >> v2:172.21.15.146:3300/0 conn(0x7f62f00cf850 0x7f62f00cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 >> v1:172.21.15.146:6789/0 conn(0x7f62f006cf20 legacy=0x7f62f006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f006c160 con 0x7f62f00d00c0 2024-03-06T07:23:48.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f62e4001670 con 0x7f62f00d00c0 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 >> v1:172.21.15.72:6789/0 conn(0x7f62f00d00c0 legacy=0x7f62f00d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d80039b0 0x7f62d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d80051c0 0x7f62d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f006ca90 con 0x7f62d8004800 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d800c060 con 0x7f62d80039b0 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d800c260 con 0x7f62d80051c0 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62edd9b640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62ee59c640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d80039b0 0x7f62d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62eed9d640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d80051c0 0x7f62d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62edd9b640 1 -- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d80051c0 msgr2=0x7f62d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62edd9b640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d80051c0 0x7f62d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62edd9b640 1 -- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d80039b0 msgr2=0x7f62d8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62edd9b640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d80039b0 0x7f62d8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62edd9b640 1 -- 172.21.15.72:0/2628810784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d8003520 con 0x7f62d8004800 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.539+0000 7f62eed9d640 1 -- 172.21.15.72:0/2628810784 reap_dead start 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.540+0000 7f62edd9b640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62d8004c80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009ac0 tx=0x7f62d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b1664e0ca2db0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.540+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0039070 con 0x7f62d8004800 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.540+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e002c6b0 con 0x7f62d8004800 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.540+0000 7f62ed59a640 1 -- 172.21.15.72:0/2628810784 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e0009690 con 0x7f62d8004800 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 -- 172.21.15.72:0/2628810784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 msgr2=0x7f62d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62d8004c80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009ac0 tx=0x7f62d8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 -- 172.21.15.72:0/2628810784 shutdown_connections 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 --2- 172.21.15.72:0/2628810784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62d8004c80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 -- 172.21.15.72:0/2628810784 >> 172.21.15.72:0/2628810784 conn(0x7f62f00b35a0 msgr2=0x7f62f00b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 -- 172.21.15.72:0/2628810784 shutdown_connections 2024-03-06T07:23:48.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.541+0000 7f62f4da0640 1 -- 172.21.15.72:0/2628810784 wait complete. 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.542+0000 7f62f4da0640 1 Processor -- start 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.542+0000 7f62f4da0640 1 -- start start 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62f00c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62f00b4650 0x7f62f00b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62f01f7050 0x7f62f01f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00b1030 con 0x7f62d8004800 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f006c720 con 0x7f62f00b4650 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f00c87e0 con 0x7f62f01f7050 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62f00c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62f00c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38824/0 (socket says 172.21.15.72:38824) 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 -- 172.21.15.72:0/733640635 learned_addr learned my addr 172.21.15.72:0/733640635 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62edd9b640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62f00b4650 0x7f62f00b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62eed9d640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62f01f7050 0x7f62f01f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 -- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62f01f7050 msgr2=0x7f62f01f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62f01f7050 0x7f62f01f74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 -- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62f00b4650 msgr2=0x7f62f00b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62f00b4650 0x7f62f00b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 -- 172.21.15.72:0/733640635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f00b4b50 con 0x7f62d8004800 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62eed9d640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62f01f7050 0x7f62f01f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62ee59c640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62f00c82a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007310 tx=0x7f62e40077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400f9c0 con 0x7f62d8004800 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e4018070 con 0x7f62d8004800 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f00b4e30 con 0x7f62d8004800 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.543+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400fde0 con 0x7f62d8004800 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.544+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f00b53f0 con 0x7f62d8004800 2024-03-06T07:23:48.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.545+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f004f330 con 0x7f62d8004800 2024-03-06T07:23:48.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.545+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f62e401d460 con 0x7f62d8004800 2024-03-06T07:23:48.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.545+0000 7f62deffd640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f62d006b4d0 0x7f62d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.545+0000 7f62edd9b640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f62d006b4d0 0x7f62d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.546+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f62e408a380 con 0x7f62d8004800 2024-03-06T07:23:48.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.546+0000 7f62edd9b640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f62d006b4d0 0x7f62d006d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f62e0036000 tx=0x7f62e0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.549+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f62e40591c0 con 0x7f62d8004800 2024-03-06T07:23:48.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.725+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62f00ce9b0 con 0x7f62d8004800 2024-03-06T07:23:48.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.726+0000 7f62deffd640 1 -- 172.21.15.72:0/733640635 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f62e4058ac0 con 0x7f62d8004800 2024-03-06T07:23:48.728 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:48.728 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":48,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:32.024853+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.32","mappings":[{"from":0,"to":4}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.727+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f62d006b4d0 msgr2=0x7f62d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f62d006b4d0 0x7f62d006d990 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f62e0036000 tx=0x7f62e0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 msgr2=0x7f62f00c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62f00c82a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f62e4007310 tx=0x7f62e40077d0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 shutdown_connections 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f62d006b4d0 0x7f62d006d990 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62f01f7050 0x7f62f01f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62f00b4650 0x7f62f00b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 --2- 172.21.15.72:0/733640635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62d8004800 0x7f62f00c82a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 >> 172.21.15.72:0/733640635 conn(0x7f62f00b35a0 msgr2=0x7f62f00b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 shutdown_connections 2024-03-06T07:23:48.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.728+0000 7f62f4da0640 1 -- 172.21.15.72:0/733640635 wait complete. 2024-03-06T07:23:48.742 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:23:48.742 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:48.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.841+0000 7fd033278640 1 Processor -- start 2024-03-06T07:23:48.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.841+0000 7fd033278640 1 -- start start 2024-03-06T07:23:48.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd033278640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd02c0d2710 0x7fd02c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd033278640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0b3430 con 0x7fd02c15dfe0 2024-03-06T07:23:48.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd033278640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd02c0b3710 con 0x7fd02c0d22b0 2024-03-06T07:23:48.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd033278640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0b31e0 con 0x7fd02c0d2710 2024-03-06T07:23:48.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd031014640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd02c15dfe0 0x7fd02c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44488/0 (socket says 172.21.15.72:44488) 2024-03-06T07:23:48.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd031014640 1 -- 172.21.15.72:0/2882003646 learned_addr learned my addr 172.21.15.72:0/2882003646 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:48.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008006091 0 0) 0x7fd02c0b3430 con 0x7fd02c15dfe0 2024-03-06T07:23:48.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd018003520 con 0x7fd02c15dfe0 2024-03-06T07:23:48.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.842+0000 7fd031815640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd02c0d2710 0x7fd02c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777511581 0 0) 0x7fd018003520 con 0x7fd02c15dfe0 2024-03-06T07:23:48.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd02c0d2710 msgr2=0x7fd02c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd02c0d2710 0x7fd02c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 >> v1:172.21.15.146:6789/0 conn(0x7fd02c0d22b0 legacy=0x7fd02c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c0b39f0 con 0x7fd02c15dfe0 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd0280019f0 con 0x7fd02c15dfe0 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 >> v1:172.21.15.72:6789/0 conn(0x7fd02c15dfe0 legacy=0x7fd02c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0180039e0 0x7fd018003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd018004800 0x7fd018004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180051c0 0x7fd01800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0b31e0 con 0x7fd0180039e0 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd01800c090 con 0x7fd018004800 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01800c290 con 0x7fd0180051c0 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031815640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd02c0d2710 0x7fd02c0b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:48.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031014640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0180039e0 0x7fd018003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd030813640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd018004800 0x7fd018004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031014640 1 -- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180051c0 msgr2=0x7fd01800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031815640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180051c0 0x7fd01800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031014640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180051c0 0x7fd01800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031014640 1 -- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd018004800 msgr2=0x7fd018004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031014640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd018004800 0x7fd018004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031014640 1 -- 172.21.15.72:0/2882003646 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd018003520 con 0x7fd0180039e0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd031815640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180051c0 0x7fd01800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.843+0000 7fd030813640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd018004800 0x7fd018004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.844+0000 7fd031014640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0180039e0 0x7fd018003e20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd02800a860 tx=0x7fd018005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca8b9c7bfda8f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.844+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02800a480 con 0x7fd0180039e0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.844+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd028034630 con 0x7fd0180039e0 2024-03-06T07:23:48.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.844+0000 7fd023fff640 1 -- 172.21.15.72:0/2882003646 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028034a20 con 0x7fd0180039e0 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd033278640 1 -- 172.21.15.72:0/2882003646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0180039e0 msgr2=0x7fd018003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd033278640 1 --2- 172.21.15.72:0/2882003646 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0180039e0 0x7fd018003e20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd02800a860 tx=0x7fd018005830 comp rx=0 tx=0).stop 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd031815640 1 -- 172.21.15.72:0/2882003646 reap_dead start 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd033278640 1 -- 172.21.15.72:0/2882003646 shutdown_connections 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd033278640 1 -- 172.21.15.72:0/2882003646 >> 172.21.15.72:0/2882003646 conn(0x7fd02c0d3cd0 msgr2=0x7fd02c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd033278640 1 -- 172.21.15.72:0/2882003646 shutdown_connections 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.845+0000 7fd033278640 1 -- 172.21.15.72:0/2882003646 wait complete. 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 Processor -- start 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 -- start start 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180039e0 0x7fd02c15fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 0x7fd02c0ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd02c0cd160 0x7fd02c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0cfa00 con 0x7fd02c14f0b0 2024-03-06T07:23:48.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0b3d60 con 0x7fd02c0cd160 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.846+0000 7fd033278640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02c0d1c70 con 0x7fd0180039e0 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 0x7fd02c0ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 0x7fd02c0ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38842/0 (socket says 172.21.15.72:38842) 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 -- 172.21.15.72:0/2510431258 learned_addr learned my addr 172.21.15.72:0/2510431258 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd031014640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180039e0 0x7fd02c15fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd031815640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd02c0cd160 0x7fd02c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 -- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180039e0 msgr2=0x7fd02c15fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180039e0 0x7fd02c15fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 -- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd02c0cd160 msgr2=0x7fd02c0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd02c0cd160 0x7fd02c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 -- 172.21.15.72:0/2510431258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c160300 con 0x7fd02c14f0b0 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd031014640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180039e0 0x7fd02c15fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd030813640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 0x7fd02c0ccc20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00e630 tx=0x7fd01c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c0052e0 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c018070 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02c1f3940 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.847+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c003810 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.848+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02c1f3f00 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.848+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02c04f330 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.849+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd01c0039b0 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.849+0000 7fd021ffb640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01006b500 0x7fd01006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.849+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd01c089f70 con 0x7fd02c14f0b0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.849+0000 7fd031014640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01006b500 0x7fd01006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:48.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.850+0000 7fd031014640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01006b500 0x7fd01006d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd02800a860 tx=0x7fd02802f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:48.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:48.858+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd01c058db0 con 0x7fd02c14f0b0 2024-03-06T07:23:49.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.078+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd02c1f6bc0 con 0x7fd02c14f0b0 2024-03-06T07:23:49.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.079+0000 7fd021ffb640 1 -- 172.21.15.72:0/2510431258 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7fd01c0586b0 con 0x7fd02c14f0b0 2024-03-06T07:23:49.081 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:49.081 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:22.210297+0000","modified":"2024-03-06T07:23:24.220196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5670":{"gid":5670,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/874695441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":874695441},{"type":"v1","addr":"172.21.15.72:6837","nonce":874695441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5670,"qdb_cluster":[5670]},"id":3}]} 2024-03-06T07:23:49.081 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 20 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.080+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01006b500 msgr2=0x7fd01006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01006b500 0x7fd01006d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd02800a860 tx=0x7fd02802f000 comp rx=0 tx=0).stop 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 msgr2=0x7fd02c0ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 0x7fd02c0ccc20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00e630 tx=0x7fd01c00eaf0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 shutdown_connections 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01006b500 0x7fd01006d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd02c0cd160 0x7fd02c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd02c14f0b0 0x7fd02c0ccc20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 --2- 172.21.15.72:0/2510431258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0180039e0 0x7fd02c15fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 >> 172.21.15.72:0/2510431258 conn(0x7fd02c0d3cd0 msgr2=0x7fd02c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 shutdown_connections 2024-03-06T07:23:49.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.081+0000 7fd033278640 1 -- 172.21.15.72:0/2510431258 wait complete. 2024-03-06T07:23:49.098 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:49.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.200+0000 7efcd5439640 1 Processor -- start 2024-03-06T07:23:49.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.200+0000 7efcd5439640 1 -- start start 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcd5439640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcd01621c0 0x7efcd00cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcd5439640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcd01625a0 0x7efcd015dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcd5439640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcd00ceac0 con 0x7efcd00d4920 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcd5439640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd00cf690 con 0x7efcd01621c0 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcd5439640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcd00d2dc0 con 0x7efcd01625a0 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcceffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcd01621c0 0x7efcd00cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcce7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efcd00d4920 0x7efcd00d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44504/0 (socket says 172.21.15.72:44504) 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcce7fc640 1 -- 172.21.15.72:0/189525239 learned_addr learned my addr 172.21.15.72:0/189525239 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcceffd640 1 -- 172.21.15.72:0/189525239 >> v2:172.21.15.146:3300/0 conn(0x7efcd01625a0 msgr2=0x7efcd015dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcceffd640 1 --2- 172.21.15.72:0/189525239 >> v2:172.21.15.146:3300/0 conn(0x7efcd01625a0 0x7efcd015dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcceffd640 1 -- 172.21.15.72:0/189525239 >> v1:172.21.15.72:6789/0 conn(0x7efcd00d4920 legacy=0x7efcd00d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efccdffb640 1 -- 172.21.15.72:0/189525239 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333961921 0 0) 0x7efcd00ceac0 con 0x7efcd00d4920 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.201+0000 7efcceffd640 1 -- 172.21.15.72:0/189525239 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd00ced60 con 0x7efcd01621c0 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.202+0000 7efcceffd640 1 --2- 172.21.15.72:0/189525239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01621c0 0x7efcd00cfb20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efcd00d3be0 tx=0x7efcc002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed35166e3bd68b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.202+0000 7efccdffb640 1 -- 172.21.15.72:0/189525239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0039070 con 0x7efcd01621c0 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.202+0000 7efccdffb640 1 -- 172.21.15.72:0/189525239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc002cbf0 con 0x7efcd01621c0 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.202+0000 7efccdffb640 1 -- 172.21.15.72:0/189525239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc0034440 con 0x7efcd01621c0 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 -- 172.21.15.72:0/189525239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01621c0 msgr2=0x7efcd00cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 --2- 172.21.15.72:0/189525239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01621c0 0x7efcd00cfb20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7efcd00d3be0 tx=0x7efcc002be40 comp rx=0 tx=0).stop 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 -- 172.21.15.72:0/189525239 shutdown_connections 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 --2- 172.21.15.72:0/189525239 >> v2:172.21.15.146:3300/0 conn(0x7efcd01625a0 0x7efcd015dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 --2- 172.21.15.72:0/189525239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01621c0 0x7efcd00cfb20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 -- 172.21.15.72:0/189525239 >> 172.21.15.72:0/189525239 conn(0x7efcd00d3d10 msgr2=0x7efcd00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 -- 172.21.15.72:0/189525239 shutdown_connections 2024-03-06T07:23:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.203+0000 7efcd5439640 1 -- 172.21.15.72:0/189525239 wait complete. 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.204+0000 7efcd5439640 1 Processor -- start 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.204+0000 7efcd5439640 1 -- start start 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcd5439640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcd00d4920 0x7efcd015add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcd5439640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcd01621c0 0x7efcd01f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcd5439640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 0x7efcd01f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcd5439640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd015e280 con 0x7efcd01625a0 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcd5439640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcd00cf320 con 0x7efcd00d4920 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcd5439640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd015e400 con 0x7efcd01621c0 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 0x7efcd01f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 0x7efcd01f4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38846/0 (socket says 172.21.15.72:38846) 2024-03-06T07:23:49.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 -- 172.21.15.72:0/1123306510 learned_addr learned my addr 172.21.15.72:0/1123306510 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efcceffd640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcd00d4920 0x7efcd015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 -- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcd01621c0 msgr2=0x7efcd01f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcd01621c0 0x7efcd01f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 -- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcd00d4920 msgr2=0x7efcd015add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcd00d4920 0x7efcd015add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 -- 172.21.15.72:0/1123306510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd00cc6a0 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.205+0000 7efccf7fe640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 0x7efcd01f4460 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7efcc800b440 tx=0x7efcc800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.206+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8015020 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.206+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc8011850 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.206+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcd00cc980 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.206+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcc8011c40 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.206+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcd00ccf40 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.207+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcd00cd140 con 0x7efcd01625a0 2024-03-06T07:23:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.207+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcc8018020 con 0x7efcd01625a0 2024-03-06T07:23:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.207+0000 7efcc77fe640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb006b4d0 0x7efcb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.208+0000 7efcceffd640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb006b4d0 0x7efcb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.208+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efcc808a0b0 con 0x7efcd01625a0 2024-03-06T07:23:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.208+0000 7efcceffd640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb006b4d0 0x7efcb006d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efcc0036000 tx=0x7efcc0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.212+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcd00cd140 con 0x7efcd01625a0 2024-03-06T07:23:49.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.422+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcd004f330 con 0x7efcd01625a0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.423+0000 7efcc77fe640 1 -- 172.21.15.72:0/1123306510 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4980 (secure 0 0 0) 0x7efcc801f020 con 0x7efcd01625a0 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:22.210297+0000","modified":"2024-03-06T07:23:24.220196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5670":{"gid":5670,"name":"c","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/874695441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":874695441},{"type":"v1","addr":"172.21.15.72:6837","nonce":874695441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5670,"qdb_cluster":[5670]},"id":3}]} 2024-03-06T07:23:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 20 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.424+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb006b4d0 msgr2=0x7efcb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.424+0000 7efcd5439640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb006b4d0 0x7efcb006d990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7efcc0036000 tx=0x7efcc0030000 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.424+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 msgr2=0x7efcd01f4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.424+0000 7efcd5439640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 0x7efcd01f4460 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7efcc800b440 tx=0x7efcc800b900 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 shutdown_connections 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb006b4d0 0x7efcb006d990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcd01625a0 0x7efcd01f4460 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcd01621c0 0x7efcd01f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 --2- 172.21.15.72:0/1123306510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcd00d4920 0x7efcd015add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 >> 172.21.15.72:0/1123306510 conn(0x7efcd00d3d10 msgr2=0x7efcd00d2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 shutdown_connections 2024-03-06T07:23:49.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.425+0000 7efcd5439640 1 -- 172.21.15.72:0/1123306510 wait complete. 2024-03-06T07:23:49.438 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:49.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.536+0000 7fd207466640 1 Processor -- start 2024-03-06T07:23:49.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.536+0000 7fd207466640 1 -- start start 2024-03-06T07:23:49.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.537+0000 7fd207466640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd20015e190 0x7fd20006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.537+0000 7fd207466640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd200066fe0 con 0x7fd20015ddb0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.537+0000 7fd207466640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2000b5200 con 0x7fd20006b4f0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.537+0000 7fd207466640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd20006b950 con 0x7fd20015e190 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd205202640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd20015ddb0 0x7fd200072ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44520/0 (socket says 172.21.15.72:44520) 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd205202640 1 -- 172.21.15.72:0/738952013 learned_addr learned my addr 172.21.15.72:0/738952013 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd204a01640 1 --2- 172.21.15.72:0/738952013 >> v2:172.21.15.146:3300/0 conn(0x7fd20015e190 0x7fd20006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239894187 0 0) 0x7fd200066fe0 con 0x7fd20015ddb0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1ec003520 con 0x7fd20015ddb0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182588089 0 0) 0x7fd2000b5200 con 0x7fd20006b4f0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd200066fe0 con 0x7fd20006b4f0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761524527 0 0) 0x7fd1ec003520 con 0x7fd20015ddb0 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 >> v2:172.21.15.146:3300/0 conn(0x7fd20015e190 msgr2=0x7fd20006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 --2- 172.21.15.72:0/738952013 >> v2:172.21.15.146:3300/0 conn(0x7fd20015e190 0x7fd20006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 >> v1:172.21.15.146:6789/0 conn(0x7fd20006b4f0 legacy=0x7fd20006d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2000b54e0 con 0x7fd20015ddb0 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd1fc001670 con 0x7fd20015ddb0 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 >> v1:172.21.15.72:6789/0 conn(0x7fd20015ddb0 legacy=0x7fd200072ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ec003b40 0x7fd1ec003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ec004800 0x7fd1ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec0051c0 0x7fd1ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec00bfb0 con 0x7fd1ec004800 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec00c130 con 0x7fd1ec003b40 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec00c330 con 0x7fd1ec0051c0 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.538+0000 7fd204a01640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ec004800 0x7fd1ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd205202640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ec003b40 0x7fd1ec003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd205a03640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec0051c0 0x7fd1ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd204a01640 1 -- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec0051c0 msgr2=0x7fd1ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd204a01640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec0051c0 0x7fd1ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd204a01640 1 -- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ec003b40 msgr2=0x7fd1ec003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd204a01640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ec003b40 0x7fd1ec003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd204a01640 1 -- 172.21.15.72:0/738952013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ec003520 con 0x7fd1ec004800 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd205a03640 1 -- 172.21.15.72:0/738952013 reap_dead start 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.539+0000 7fd204a01640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ec004800 0x7fd1ec004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009a90 tx=0x7fd1ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1235332e7e82e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0039070 con 0x7fd1ec004800 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f002c680 con 0x7fd1ec004800 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd1f7fff640 1 -- 172.21.15.72:0/738952013 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0009690 con 0x7fd1ec004800 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 -- 172.21.15.72:0/738952013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ec004800 msgr2=0x7fd1ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ec004800 0x7fd1ec004c80 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0009a90 tx=0x7fd1ec005830 comp rx=0 tx=0).stop 2024-03-06T07:23:49.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 -- 172.21.15.72:0/738952013 shutdown_connections 2024-03-06T07:23:49.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 --2- 172.21.15.72:0/738952013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ec004800 0x7fd1ec004c80 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 -- 172.21.15.72:0/738952013 >> 172.21.15.72:0/738952013 conn(0x7fd2000b1d00 msgr2=0x7fd2000b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 -- 172.21.15.72:0/738952013 shutdown_connections 2024-03-06T07:23:49.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.540+0000 7fd207466640 1 -- 172.21.15.72:0/738952013 wait complete. 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.542+0000 7fd207466640 1 Processor -- start 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.542+0000 7fd207466640 1 -- start start 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.543+0000 7fd207466640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec004800 0x7fd20009a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.543+0000 7fd207466640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 0x7fd20009d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd207466640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd20009dbb0 0x7fd2001e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd207466640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000b5d80 con 0x7fd2001599b0 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd207466640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000b5aa0 con 0x7fd20009dbb0 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd207466640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2000b4e60 con 0x7fd1ec004800 2024-03-06T07:23:49.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 0x7fd20009d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 0x7fd20009d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38864/0 (socket says 172.21.15.72:38864) 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 -- 172.21.15.72:0/148359949 learned_addr learned my addr 172.21.15.72:0/148359949 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd205a03640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd20009dbb0 0x7fd2001e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd205202640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec004800 0x7fd20009a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 -- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec004800 msgr2=0x7fd20009a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec004800 0x7fd20009a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 -- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd20009dbb0 msgr2=0x7fd2001e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd20009dbb0 0x7fd2001e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 -- 172.21.15.72:0/148359949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd20015e620 con 0x7fd2001599b0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd205a03640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd20009dbb0 0x7fd2001e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd205202640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec004800 0x7fd20009a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd204a01640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 0x7fd20009d670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd1f002c2f0 tx=0x7fd1f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0042020 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1f003a040 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20015e890 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.544+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0039070 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.545+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd200159e10 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.546+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd20004f330 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.546+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1f0045020 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.547+0000 7fd1f5ffb640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1e406b4d0 0x7fd1e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.547+0000 7fd205202640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1e406b4d0 0x7fd1e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.547+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd1f00ab610 con 0x7fd2001599b0 2024-03-06T07:23:49.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.547+0000 7fd205202640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1e406b4d0 0x7fd1e406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc007d20 tx=0x7fd1fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.551+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1f004c070 con 0x7fd2001599b0 2024-03-06T07:23:49.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.729+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd20009df90 con 0x7fd2001599b0 2024-03-06T07:23:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.730+0000 7fd1f5ffb640 1 -- 172.21.15.72:0/148359949 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16777 (secure 0 0 0) 0x7fd1f007a4e0 con 0x7fd2001599b0 2024-03-06T07:23:49.732 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:49.732 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":48,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:32.024853+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:20.432633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:21.177789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.32","mappings":[{"from":0,"to":4}]},{"pgid":"7.3e","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1e406b4d0 msgr2=0x7fd1e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1e406b4d0 0x7fd1e406d990 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc007d20 tx=0x7fd1fc009040 comp rx=0 tx=0).stop 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 msgr2=0x7fd20009d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 0x7fd20009d670 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd1f002c2f0 tx=0x7fd1f0004730 comp rx=0 tx=0).stop 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 shutdown_connections 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1e406b4d0 0x7fd1e406d990 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd20009dbb0 0x7fd2001e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2001599b0 0x7fd20009d670 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 --2- 172.21.15.72:0/148359949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ec004800 0x7fd20009a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 >> 172.21.15.72:0/148359949 conn(0x7fd2000b1d00 msgr2=0x7fd2000afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 shutdown_connections 2024-03-06T07:23:49.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.733+0000 7fd207466640 1 -- 172.21.15.72:0/148359949 wait complete. 2024-03-06T07:23:49.746 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.846+0000 7fa8789a0640 1 Processor -- start 2024-03-06T07:23:49.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.846+0000 7fa8789a0640 1 -- start start 2024-03-06T07:23:49.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8789a0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa874146ac0 0x7fa8740b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8789a0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa87406f500 0x7fa87406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8789a0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8740b5830 con 0x7fa87406afb0 2024-03-06T07:23:49.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8789a0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8740b5b10 con 0x7fa874146ac0 2024-03-06T07:23:49.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8789a0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8740724b0 con 0x7fa87406f500 2024-03-06T07:23:49.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8737fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa874146ac0 0x7fa8740b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa872ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa87406afb0 0x7fa8740d0580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44524/0 (socket says 172.21.15.72:44524) 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa872ffd640 1 -- 172.21.15.72:0/1051696043 learned_addr learned my addr 172.21.15.72:0/1051696043 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa873fff640 1 --2- 172.21.15.72:0/1051696043 >> v2:172.21.15.146:3300/0 conn(0x7fa87406f500 0x7fa87406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8737fe640 1 -- 172.21.15.72:0/1051696043 >> v2:172.21.15.146:3300/0 conn(0x7fa87406f500 msgr2=0x7fa87406b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8737fe640 1 --2- 172.21.15.72:0/1051696043 >> v2:172.21.15.146:3300/0 conn(0x7fa87406f500 0x7fa87406b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8737fe640 1 -- 172.21.15.72:0/1051696043 >> v1:172.21.15.72:6789/0 conn(0x7fa87406afb0 legacy=0x7fa8740d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.847+0000 7fa8737fe640 1 -- 172.21.15.72:0/1051696043 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874072790 con 0x7fa874146ac0 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.848+0000 7fa8737fe640 1 --2- 172.21.15.72:0/1051696043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8740b38f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fa8741479e0 tx=0x7fa86402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f426dba2f5a3b2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.848+0000 7fa8727fc640 1 -- 172.21.15.72:0/1051696043 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864035020 con 0x7fa874146ac0 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.848+0000 7fa8727fc640 1 -- 172.21.15.72:0/1051696043 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864031830 con 0x7fa874146ac0 2024-03-06T07:23:49.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.848+0000 7fa8727fc640 1 -- 172.21.15.72:0/1051696043 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864031c00 con 0x7fa874146ac0 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 -- 172.21.15.72:0/1051696043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 msgr2=0x7fa8740b38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 --2- 172.21.15.72:0/1051696043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8740b38f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fa8741479e0 tx=0x7fa86402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 -- 172.21.15.72:0/1051696043 shutdown_connections 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 --2- 172.21.15.72:0/1051696043 >> v2:172.21.15.146:3300/0 conn(0x7fa87406f500 0x7fa87406b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 --2- 172.21.15.72:0/1051696043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8740b38f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 -- 172.21.15.72:0/1051696043 >> 172.21.15.72:0/1051696043 conn(0x7fa87406c0c0 msgr2=0x7fa87406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 -- 172.21.15.72:0/1051696043 shutdown_connections 2024-03-06T07:23:49.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.849+0000 7fa8789a0640 1 -- 172.21.15.72:0/1051696043 wait complete. 2024-03-06T07:23:49.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:49.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.851+0000 7fa8789a0640 1 Processor -- start 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.851+0000 7fa8789a0640 1 -- start start 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa8789a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa87406afb0 0x7fa87413b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa8789a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa87406f500 0x7fa87413ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa8789a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8741f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa8789a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740720a0 con 0x7fa874146ac0 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa8789a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa874071f20 con 0x7fa87406f500 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa8789a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8740b3e30 con 0x7fa87406afb0 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa873fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8741f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.852+0000 7fa873fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8741f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38878/0 (socket says 172.21.15.72:38878) 2024-03-06T07:23:49.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 -- 172.21.15.72:0/4055790430 learned_addr learned my addr 172.21.15.72:0/4055790430 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:49.856 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa8737fe640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa87406afb0 0x7fa87413b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa872ffd640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa87406f500 0x7fa87413ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 -- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa87406afb0 msgr2=0x7fa87413b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa87406afb0 0x7fa87413b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 -- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa87406f500 msgr2=0x7fa87413ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa87406f500 0x7fa87413ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 -- 172.21.15.72:0/4055790430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87413e330 con 0x7fa874146ac0 2024-03-06T07:23:49.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa873fff640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8741f3c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa86c007d80 tx=0x7fa86c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c014420 con 0x7fa874146ac0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c019070 con 0x7fa874146ac0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87413e610 con 0x7fa874146ac0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c014800 con 0x7fa874146ac0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.853+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87413ebd0 con 0x7fa874146ac0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.854+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8740b53c0 con 0x7fa874146ac0 2024-03-06T07:23:49.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.855+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa86c0149a0 con 0x7fa874146ac0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.856+0000 7fa85bfff640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa86006b4d0 0x7fa86006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.856+0000 7fa8737fe640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa86006b4d0 0x7fa86006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.856+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa86c08a940 con 0x7fa874146ac0 2024-03-06T07:23:49.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.856+0000 7fa8737fe640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa86006b4d0 0x7fa86006d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa864036000 tx=0x7fa86403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:49.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:49.860+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa8740b53c0 con 0x7fa874146ac0 2024-03-06T07:23:50.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.077+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa87406c660 con 0x7fa874146ac0 2024-03-06T07:23:50.080 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:23:50.078+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.081+0000 7fa85bfff640 1 -- 172.21.15.72:0/4055790430 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa86c059140 con 0x7fa874146ac0 2024-03-06T07:23:50.083 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.084+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa86006b4d0 msgr2=0x7fa86006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.084+0000 7fa8789a0640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa86006b4d0 0x7fa86006d990 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa864036000 tx=0x7fa86403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.084+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 msgr2=0x7fa8741f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.084+0000 7fa8789a0640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8741f3c90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fa86c007d80 tx=0x7fa86c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 shutdown_connections 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa86006b4d0 0x7fa86006d990 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa874146ac0 0x7fa8741f3c90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa87406f500 0x7fa87413ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 --2- 172.21.15.72:0/4055790430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa87406afb0 0x7fa87413b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 >> 172.21.15.72:0/4055790430 conn(0x7fa87406c0c0 msgr2=0x7fa8741602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 shutdown_connections 2024-03-06T07:23:50.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.085+0000 7fa8789a0640 1 -- 172.21.15.72:0/4055790430 wait complete. 2024-03-06T07:23:50.099 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:23:50.125 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:23:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.201+0000 7fbc33ff7640 1 Processor -- start 2024-03-06T07:23:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.202+0000 7fbc33ff7640 1 -- start start 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.202+0000 7fbc33ff7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbc2c0cfa90 0x7fbc2c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.202+0000 7fbc33ff7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 0x7fbc2c0b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.202+0000 7fbc33ff7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc2c158ac0 con 0x7fbc2c1586e0 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.202+0000 7fbc33ff7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc2c15b2a0 con 0x7fbc2c0cfa90 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.202+0000 7fbc33ff7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15b540 con 0x7fbc2c0b33a0 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc31d93640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbc2c1586e0 0x7fbc2c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44538/0 (socket says 172.21.15.72:44538) 2024-03-06T07:23:50.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc31d93640 1 -- 172.21.15.72:0/3280733151 learned_addr learned my addr 172.21.15.72:0/3280733151 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006697988 0 0) 0x7fbc2c158ac0 con 0x7fbc2c1586e0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc31592640 1 --2- 172.21.15.72:0/3280733151 >> v2:172.21.15.146:3300/0 conn(0x7fbc2c0cfa90 0x7fbc2c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc18005730 con 0x7fbc2c1586e0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc32594640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 0x7fbc2c0b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1682517070 0 0) 0x7fbc18005730 con 0x7fbc2c1586e0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 msgr2=0x7fbc2c0b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 0x7fbc2c0b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 >> v2:172.21.15.146:3300/0 conn(0x7fbc2c0cfa90 msgr2=0x7fbc2c0cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 --2- 172.21.15.72:0/3280733151 >> v2:172.21.15.146:3300/0 conn(0x7fbc2c0cfa90 0x7fbc2c0cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c15b820 con 0x7fbc2c1586e0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fbc28001670 con 0x7fbc2c1586e0 2024-03-06T07:23:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc31592640 1 --2- 172.21.15.72:0/3280733151 >> v2:172.21.15.146:3300/0 conn(0x7fbc2c0cfa90 0x7fbc2c0cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 >> v1:172.21.15.72:6789/0 conn(0x7fbc2c1586e0 legacy=0x7fbc2c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc18005bc0 0x7fbc18005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc18006a10 0x7fbc18006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 0x7fbc180073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15b2a0 con 0x7fbc18006a10 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15b540 con 0x7fbc18005bc0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1800dc80 con 0x7fbc2c0b33a0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc31592640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc18006a10 0x7fbc18006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc31d93640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc18005bc0 0x7fbc18005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.203+0000 7fbc32594640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 0x7fbc180073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc31592640 1 -- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 msgr2=0x7fbc180073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc31592640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc2c0b33a0 0x7fbc180073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc31592640 1 -- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc18005bc0 msgr2=0x7fbc18005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc31592640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc18005bc0 0x7fbc18005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc31592640 1 -- 172.21.15.72:0/3280733151 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc18005730 con 0x7fbc18006a10 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc31592640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc18006a10 0x7fbc18006e90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c006fd0 tx=0x7fbc18007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db45570edd9b48be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c00f040 con 0x7fbc18006a10 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c004230 con 0x7fbc18006a10 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.204+0000 7fbc30d91640 1 -- 172.21.15.72:0/3280733151 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c003610 con 0x7fbc18006a10 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc33ff7640 1 -- 172.21.15.72:0/3280733151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc18006a10 msgr2=0x7fbc18006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/3280733151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc18006a10 0x7fbc18006e90 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c006fd0 tx=0x7fbc18007a40 comp rx=0 tx=0).stop 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc32594640 1 -- 172.21.15.72:0/3280733151 reap_dead start 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc33ff7640 1 -- 172.21.15.72:0/3280733151 shutdown_connections 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc33ff7640 1 -- 172.21.15.72:0/3280733151 >> 172.21.15.72:0/3280733151 conn(0x7fbc2c06b190 msgr2=0x7fbc2c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc33ff7640 1 -- 172.21.15.72:0/3280733151 shutdown_connections 2024-03-06T07:23:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.205+0000 7fbc33ff7640 1 -- 172.21.15.72:0/3280733151 wait complete. 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.206+0000 7fbc33ff7640 1 Processor -- start 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.206+0000 7fbc33ff7640 1 -- start start 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc33ff7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc18006a10 0x7fbc2c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc33ff7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc2c1572a0 0x7fbc2c144f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc33ff7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 0x7fbc2c1454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc33ff7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c0af150 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc33ff7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c15bde0 con 0x7fbc2c1572a0 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc33ff7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2c071350 con 0x7fbc18006a10 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 0x7fbc2c1454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 0x7fbc2c1454a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38920/0 (socket says 172.21.15.72:38920) 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 -- 172.21.15.72:0/2809003838 learned_addr learned my addr 172.21.15.72:0/2809003838 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc31592640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc2c1572a0 0x7fbc2c144f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc31d93640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc18006a10 0x7fbc2c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 -- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc18006a10 msgr2=0x7fbc2c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc18006a10 0x7fbc2c0adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 -- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc2c1572a0 msgr2=0x7fbc2c144f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:50.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc2c1572a0 0x7fbc2c144f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 -- 172.21.15.72:0/2809003838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c0cd0b0 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc31d93640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc18006a10 0x7fbc2c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.207+0000 7fbc32594640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 0x7fbc2c1454a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0d0910 tx=0x7fbc24007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.208+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc240046a0 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.208+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2401a070 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.208+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2c0cd390 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.208+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24015640 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.208+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2c0cd950 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.209+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2c0732b0 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.209+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc240157e0 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.209+0000 7fbc227fc640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc1006b4d0 0x7fbc1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.209+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbc2408a910 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.209+0000 7fbc31d93640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc1006b4d0 0x7fbc1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:50.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.210+0000 7fbc31d93640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc1006b4d0 0x7fbc1006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbc280015d0 tx=0x7fbc28009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:50.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.214+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc24024020 con 0x7fbc2c0ccaf0 2024-03-06T07:23:50.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:50.422+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbc2c068ff0 con 0x7fbc2c0ccaf0 2024-03-06T07:23:51.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.085+0000 7fbc227fc640 1 -- 172.21.15.72:0/2809003838 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbc24059700 con 0x7fbc2c0ccaf0 2024-03-06T07:23:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc1006b4d0 msgr2=0x7fbc1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc1006b4d0 0x7fbc1006d990 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbc280015d0 tx=0x7fbc28009040 comp rx=0 tx=0).stop 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 msgr2=0x7fbc2c1454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 0x7fbc2c1454a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0d0910 tx=0x7fbc24007700 comp rx=0 tx=0).stop 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 shutdown_connections 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc1006b4d0 0x7fbc1006d990 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc2c0ccaf0 0x7fbc2c1454a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc2c1572a0 0x7fbc2c144f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 --2- 172.21.15.72:0/2809003838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc18006a10 0x7fbc2c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 >> 172.21.15.72:0/2809003838 conn(0x7fbc2c06b190 msgr2=0x7fbc2c06c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 shutdown_connections 2024-03-06T07:23:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.087+0000 7fbc33ff7640 1 -- 172.21.15.72:0/2809003838 wait complete. 2024-03-06T07:23:51.102 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:23:51.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.247+0000 7f18759ac640 1 Processor -- start 2024-03-06T07:23:51.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.247+0000 7f18759ac640 1 -- start start 2024-03-06T07:23:51.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f18759ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f18701602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f18759ac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18700d0130 0x7f187015d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f18759ac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187006f420 con 0x7f18700701e0 2024-03-06T07:23:51.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f18759ac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f187006f700 con 0x7f18700d0130 2024-03-06T07:23:51.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f18759ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f187006f9e0 con 0x7f187015bdd0 2024-03-06T07:23:51.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f186e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18700d0130 0x7f187015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f186f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f18700701e0 0x7f1870070610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44548/0 (socket says 172.21.15.72:44548) 2024-03-06T07:23:51.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.248+0000 7f186f7fe640 1 -- 172.21.15.72:0/3875384866 learned_addr learned my addr 172.21.15.72:0/3875384866 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:51.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186e7fc640 1 -- 172.21.15.72:0/3875384866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 msgr2=0x7f18701602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:51.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186e7fc640 1 --2- 172.21.15.72:0/3875384866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f18701602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186e7fc640 1 -- 172.21.15.72:0/3875384866 >> v1:172.21.15.72:6789/0 conn(0x7f18700701e0 legacy=0x7f1870070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186dffb640 1 -- 172.21.15.72:0/3875384866 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603810237 0 0) 0x7f187006f420 con 0x7f18700701e0 2024-03-06T07:23:51.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186e7fc640 1 -- 172.21.15.72:0/3875384866 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187006fcc0 con 0x7f18700d0130 2024-03-06T07:23:51.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186e7fc640 1 --2- 172.21.15.72:0/3875384866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187015d330 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f18640097a0 tx=0x7f186402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0bdd23d99a246f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.249+0000 7f186dffb640 1 -- 172.21.15.72:0/3875384866 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18640315a0 con 0x7f18700d0130 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f186dffb640 1 -- 172.21.15.72:0/3875384866 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1864037070 con 0x7f18700d0130 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f186dffb640 1 -- 172.21.15.72:0/3875384866 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864031960 con 0x7f18700d0130 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 -- 172.21.15.72:0/3875384866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 msgr2=0x7f187015d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 --2- 172.21.15.72:0/3875384866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187015d330 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f18640097a0 tx=0x7f186402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:51.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 -- 172.21.15.72:0/3875384866 shutdown_connections 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 --2- 172.21.15.72:0/3875384866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187015d330 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 --2- 172.21.15.72:0/3875384866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f18701602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 -- 172.21.15.72:0/3875384866 >> 172.21.15.72:0/3875384866 conn(0x7f187009a9f0 msgr2=0x7f187009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 -- 172.21.15.72:0/3875384866 shutdown_connections 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.250+0000 7f18759ac640 1 -- 172.21.15.72:0/3875384866 wait complete. 2024-03-06T07:23:51.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.251+0000 7f18759ac640 1 Processor -- start 2024-03-06T07:23:51.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.251+0000 7f18759ac640 1 -- start start 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f18759ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18700701e0 0x7f18700c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f18759ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f18759ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f187006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f18759ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187015d9f0 con 0x7f18700d0130 2024-03-06T07:23:51.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f18759ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187015d870 con 0x7f18700701e0 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f18759ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18700d08d0 con 0x7f187015bdd0 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f186e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f186e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187006d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38934/0 (socket says 172.21.15.72:38934) 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f186e7fc640 1 -- 172.21.15.72:0/507711961 learned_addr learned my addr 172.21.15.72:0/507711961 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f186f7fe640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f187006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.252+0000 7f186effd640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18700701e0 0x7f18700c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186e7fc640 1 -- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 msgr2=0x7f187006dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186e7fc640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f187006dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186e7fc640 1 -- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18700701e0 msgr2=0x7f18700c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186e7fc640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18700701e0 0x7f18700c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186e7fc640 1 -- 172.21.15.72:0/507711961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18700b5b20 con 0x7f18700d0130 2024-03-06T07:23:51.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186f7fe640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f187006dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186effd640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18700701e0 0x7f18700c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f186e7fc640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187006d810 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1864034040 tx=0x7f18640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864004230 con 0x7f18700d0130 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18640043f0 con 0x7f18700d0130 2024-03-06T07:23:51.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18700b5e00 con 0x7f18700d0130 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864032880 con 0x7f18700d0130 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.253+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18700b63c0 con 0x7f18700d0130 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.254+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18700c6d40 con 0x7f18700d0130 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.254+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1864032a20 con 0x7f18700d0130 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.255+0000 7f1857fff640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f185c06b4d0 0x7f185c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:51.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.255+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f186402f030 con 0x7f18700d0130 2024-03-06T07:23:51.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.255+0000 7f186effd640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f185c06b4d0 0x7f185c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:51.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.255+0000 7f186effd640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f185c06b4d0 0x7f185c06d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1860007eb0 tx=0x7f18600073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:51.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.259+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18700c6d40 con 0x7f18700d0130 2024-03-06T07:23:51.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:51.428+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f18700b7ad0 con 0x7f18700d0130 2024-03-06T07:23:52.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.089+0000 7f1857fff640 1 -- 172.21.15.72:0/507711961 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1864041070 con 0x7f18700d0130 2024-03-06T07:23:52.091 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:23:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.092+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f185c06b4d0 msgr2=0x7f185c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.092+0000 7f18759ac640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f185c06b4d0 0x7f185c06d990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1860007eb0 tx=0x7f18600073c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.092+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 msgr2=0x7f187006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.092+0000 7f18759ac640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187006d810 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f1864034040 tx=0x7f18640046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 shutdown_connections 2024-03-06T07:23:52.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f185c06b4d0 0x7f185c06d990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f187015bdd0 0x7f187006dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18700d0130 0x7f187006d810 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 --2- 172.21.15.72:0/507711961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18700701e0 0x7f18700c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 >> 172.21.15.72:0/507711961 conn(0x7f187009a9f0 msgr2=0x7f18700ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 shutdown_connections 2024-03-06T07:23:52.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.093+0000 7f18759ac640 1 -- 172.21.15.72:0/507711961 wait complete. 2024-03-06T07:23:52.110 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:23:52.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.213+0000 7f7a8dc17640 1 Processor -- start 2024-03-06T07:23:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.213+0000 7f7a8dc17640 1 -- start start 2024-03-06T07:23:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.214+0000 7f7a8dc17640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a8806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.214+0000 7f7a8dc17640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a88070d10 0x7f7a8815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.214+0000 7f7a8dc17640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8815afe0 con 0x7f7a880708b0 2024-03-06T07:23:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.214+0000 7f7a8dc17640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a880cdc10 con 0x7f7a88070d10 2024-03-06T07:23:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.214+0000 7f7a8dc17640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880cd000 con 0x7f7a8815ac00 2024-03-06T07:23:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a877fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a8806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a87fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a88070d10 0x7f7a8815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a877fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a8806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:58302/0 (socket says 172.21.15.72:58302) 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a877fe640 1 -- 172.21.15.72:0/1031836915 learned_addr learned my addr 172.21.15.72:0/1031836915 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a87fff640 1 -- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 msgr2=0x7f7a8806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a87fff640 1 --2- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a8806c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a87fff640 1 -- 172.21.15.72:0/1031836915 >> v1:172.21.15.146:6789/0 conn(0x7f7a880708b0 legacy=0x7f7a8815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a87fff640 1 -- 172.21.15.72:0/1031836915 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a880cd2e0 con 0x7f7a88070d10 2024-03-06T07:23:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.215+0000 7f7a867fc640 1 -- 172.21.15.72:0/1031836915 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395633214 0 0) 0x7f7a8815afe0 con 0x7f7a880708b0 2024-03-06T07:23:52.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.216+0000 7f7a87fff640 1 --2- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a8815eeb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000bfa0 tx=0x7f7a8002e010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a4834e32c08bf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.216+0000 7f7a867fc640 1 -- 172.21.15.72:0/1031836915 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002e9b0 con 0x7f7a88070d10 2024-03-06T07:23:52.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.216+0000 7f7a867fc640 1 -- 172.21.15.72:0/1031836915 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8002eb50 con 0x7f7a88070d10 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.216+0000 7f7a867fc640 1 -- 172.21.15.72:0/1031836915 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80009540 con 0x7f7a88070d10 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 -- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 msgr2=0x7f7a8815eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a8815eeb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7a8000bfa0 tx=0x7f7a8002e010 comp rx=0 tx=0).stop 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 -- 172.21.15.72:0/1031836915 shutdown_connections 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a8815eeb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/1031836915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a8806c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 -- 172.21.15.72:0/1031836915 >> 172.21.15.72:0/1031836915 conn(0x7f7a880b1650 msgr2=0x7f7a880b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 -- 172.21.15.72:0/1031836915 shutdown_connections 2024-03-06T07:23:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.217+0000 7f7a8dc17640 1 -- 172.21.15.72:0/1031836915 wait complete. 2024-03-06T07:23:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.218+0000 7f7a8dc17640 1 Processor -- start 2024-03-06T07:23:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.218+0000 7f7a8dc17640 1 -- start start 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a8dc17640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 0x7f7a8814a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a8dc17640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a88156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a8dc17640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a881568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a8dc17640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a88071190 con 0x7f7a880708b0 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a8dc17640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a880cd8a0 con 0x7f7a88070d10 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a8dc17640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8806ce50 con 0x7f7a8815ac00 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 0x7f7a8814a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 0x7f7a8814a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38950/0 (socket says 172.21.15.72:38950) 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 -- 172.21.15.72:0/723214293 learned_addr learned my addr 172.21.15.72:0/723214293 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a87fff640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a881568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a86ffd640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a88156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 -- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 msgr2=0x7f7a881568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a881568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 -- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 msgr2=0x7f7a88156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a88156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 -- 172.21.15.72:0/723214293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a88153440 con 0x7f7a880708b0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.219+0000 7f7a877fe640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 0x7f7a8814a200 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e870 tx=0x7f7a7800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.220+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a78017070 con 0x7f7a880708b0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.220+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7800c8e0 con 0x7f7a880708b0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.220+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a88153720 con 0x7f7a880708b0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.220+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7800ccd0 con 0x7f7a880708b0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.220+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a88153ce0 con 0x7f7a880708b0 2024-03-06T07:23:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.221+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a880687d0 con 0x7f7a880708b0 2024-03-06T07:23:52.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.221+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7a7801a070 con 0x7f7a880708b0 2024-03-06T07:23:52.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.221+0000 7f7a6ffff640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7a7406b4d0 0x7f7a7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:52.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.222+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f7a7808a2e0 con 0x7f7a880708b0 2024-03-06T07:23:52.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.222+0000 7f7a86ffd640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7a7406b4d0 0x7f7a7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:52.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.222+0000 7f7a86ffd640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7a7406b4d0 0x7f7a7406d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7a8815d8b0 tx=0x7f7a7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:52.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.226+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7a78058cf0 con 0x7f7a880708b0 2024-03-06T07:23:52.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:52.400+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7a8804f330 con 0x7f7a880708b0 2024-03-06T07:23:53.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.090+0000 7f7a6ffff640 1 -- 172.21.15.72:0/723214293 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7a7805cd90 con 0x7f7a880708b0 2024-03-06T07:23:53.093 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7a7406b4d0 msgr2=0x7f7a7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7a7406b4d0 0x7f7a7406d990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7a8815d8b0 tx=0x7f7a7c009040 comp rx=0 tx=0).stop 2024-03-06T07:23:53.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 msgr2=0x7f7a8814a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 0x7f7a8814a200 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7a7800e870 tx=0x7f7a7800ed30 comp rx=0 tx=0).stop 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 shutdown_connections 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7a7406b4d0 0x7f7a7406d990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a8815ac00 0x7f7a881568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a88070d10 0x7f7a88156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 --2- 172.21.15.72:0/723214293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a880708b0 0x7f7a8814a200 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 >> 172.21.15.72:0/723214293 conn(0x7f7a880b1650 msgr2=0x7f7a880afdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 shutdown_connections 2024-03-06T07:23:53.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.096+0000 7f7a8dc17640 1 -- 172.21.15.72:0/723214293 wait complete. 2024-03-06T07:23:53.114 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:23:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 Processor -- start 2024-03-06T07:23:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 -- start start 2024-03-06T07:23:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0cf9a0 0x7f594c0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f594c066d90 0x7f594c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594c06b710 con 0x7f594c0b5410 2024-03-06T07:23:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594c06b150 con 0x7f594c066d90 2024-03-06T07:23:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.215+0000 7f5952565640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c06b430 con 0x7f594c0cf9a0 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f594c066d90 0x7f594c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594bfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0cf9a0 0x7f594c0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f5950b02640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f594c0b5410 0x7f594c072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57614/0 (socket says 172.21.15.72:57614) 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594b7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f594c066d90 0x7f594c0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:60220/0 (socket says 172.21.15.72:60220) 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f5950b02640 1 -- 172.21.15.72:0/3731337929 learned_addr learned my addr 172.21.15.72:0/3731337929 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594b7fe640 1 -- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0cf9a0 msgr2=0x7f594c0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594b7fe640 1 --2- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0cf9a0 0x7f594c0b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594b7fe640 1 -- 172.21.15.72:0/3731337929 >> v1:172.21.15.146:6789/0 conn(0x7f594c0b5410 legacy=0x7f594c072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594b7fe640 1 -- 172.21.15.72:0/3731337929 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0b0970 con 0x7f594c066d90 2024-03-06T07:23:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.216+0000 7f594bfff640 1 --2- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0cf9a0 0x7f594c0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.217+0000 7f594b7fe640 1 --2- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c066d90 0x7f594c0b4ed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f59340097a0 tx=0x7f593402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8a98536a02f781a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.217+0000 7f594affd640 1 -- 172.21.15.72:0/3731337929 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934035020 con 0x7f594c066d90 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.217+0000 7f594affd640 1 -- 172.21.15.72:0/3731337929 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934031810 con 0x7f594c066d90 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.217+0000 7f594affd640 1 -- 172.21.15.72:0/3731337929 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934031be0 con 0x7f594c066d90 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 -- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c066d90 msgr2=0x7f594c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 --2- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c066d90 0x7f594c0b4ed0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f59340097a0 tx=0x7f593402b8b0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 -- 172.21.15.72:0/3731337929 shutdown_connections 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 --2- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c066d90 0x7f594c0b4ed0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 --2- 172.21.15.72:0/3731337929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0cf9a0 0x7f594c0b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 -- 172.21.15.72:0/3731337929 >> 172.21.15.72:0/3731337929 conn(0x7f594c0b1d70 msgr2=0x7f594c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 -- 172.21.15.72:0/3731337929 shutdown_connections 2024-03-06T07:23:53.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.218+0000 7f5952565640 1 -- 172.21.15.72:0/3731337929 wait complete. 2024-03-06T07:23:53.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.219+0000 7f5952565640 1 Processor -- start 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.219+0000 7f5952565640 1 -- start start 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5952565640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 0x7f594c1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5952565640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0b5410 0x7f594c0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5952565640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c0cf9a0 0x7f594c0ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5952565640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0b6170 con 0x7f594c066d90 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5952565640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0b5ff0 con 0x7f594c0cf9a0 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5952565640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594c0cf050 con 0x7f594c0b5410 2024-03-06T07:23:53.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 0x7f594c1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 0x7f594c1f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39344/0 (socket says 172.21.15.72:39344) 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 -- 172.21.15.72:0/629355001 learned_addr learned my addr 172.21.15.72:0/629355001 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5950b02640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c0cf9a0 0x7f594c0ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594b7fe640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0b5410 0x7f594c0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 -- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0b5410 msgr2=0x7f594c0ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0b5410 0x7f594c0ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 -- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c0cf9a0 msgr2=0x7f594c0ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c0cf9a0 0x7f594c0ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f594bfff640 1 -- 172.21.15.72:0/629355001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594c0cbd60 con 0x7f594c066d90 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.220+0000 7f5950b02640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c0cf9a0 0x7f594c0ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.221+0000 7f594bfff640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 0x7f594c1f3750 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f593c00e960 tx=0x7f593c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.221+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c017070 con 0x7f594c066d90 2024-03-06T07:23:53.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.221+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c00ca00 con 0x7f594c066d90 2024-03-06T07:23:53.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.221+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594c0cc040 con 0x7f594c066d90 2024-03-06T07:23:53.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.221+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c00cdd0 con 0x7f594c066d90 2024-03-06T07:23:53.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.221+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594c0cc600 con 0x7f594c066d90 2024-03-06T07:23:53.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.222+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594c09e3d0 con 0x7f594c066d90 2024-03-06T07:23:53.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.222+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f593c01a070 con 0x7f594c066d90 2024-03-06T07:23:53.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.223+0000 7f5948ff9640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592c06b4d0 0x7f592c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.223+0000 7f594b7fe640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592c06b4d0 0x7f592c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.223+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f593c08a170 con 0x7f594c066d90 2024-03-06T07:23:53.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.223+0000 7f594b7fe640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592c06b4d0 0x7f592c06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5934036000 tx=0x7f593403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.227+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f593c0592f0 con 0x7f594c066d90 2024-03-06T07:23:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.397+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f594c068ff0 con 0x7f594c066d90 2024-03-06T07:23:53.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.400+0000 7f5948ff9640 1 -- 172.21.15.72:0/629355001 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f593c058bf0 con 0x7f594c066d90 2024-03-06T07:23:53.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592c06b4d0 msgr2=0x7f592c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592c06b4d0 0x7f592c06d990 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5934036000 tx=0x7f593403c040 comp rx=0 tx=0).stop 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 msgr2=0x7f594c1f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 0x7f594c1f3750 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f593c00e960 tx=0x7f593c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 shutdown_connections 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592c06b4d0 0x7f592c06d990 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594c0cf9a0 0x7f594c0ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594c0b5410 0x7f594c0ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 --2- 172.21.15.72:0/629355001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594c066d90 0x7f594c1f3750 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 >> 172.21.15.72:0/629355001 conn(0x7f594c0b1d70 msgr2=0x7f594c06de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 shutdown_connections 2024-03-06T07:23:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.402+0000 7f5952565640 1 -- 172.21.15.72:0/629355001 wait complete. 2024-03-06T07:23:53.415 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:23:53.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.515+0000 7f73cd24e640 1 Processor -- start 2024-03-06T07:23:53.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.515+0000 7f73cd24e640 1 -- start start 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73cd24e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73c81586e0 0x7f73c80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73cd24e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f73c80aef80 0x7f73c80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73cd24e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c8158ac0 con 0x7f73c80cffd0 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73cd24e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c815b2a0 con 0x7f73c81586e0 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73cd24e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c815b540 con 0x7f73c80aef80 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73c81586e0 0x7f73c80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73c81586e0 0x7f73c80aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39354/0 (socket says 172.21.15.72:39354) 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 -- 172.21.15.72:0/2994075294 learned_addr learned my addr 172.21.15.72:0/2994075294 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 -- 172.21.15.72:0/2994075294 >> v2:172.21.15.146:3300/0 conn(0x7f73c80aef80 msgr2=0x7f73c80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 --2- 172.21.15.72:0/2994075294 >> v2:172.21.15.146:3300/0 conn(0x7f73c80aef80 0x7f73c80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 -- 172.21.15.72:0/2994075294 >> v1:172.21.15.146:6789/0 conn(0x7f73c80cffd0 legacy=0x7f73c806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.516+0000 7f73c6d9d640 1 -- 172.21.15.72:0/2994075294 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c815b820 con 0x7f73c81586e0 2024-03-06T07:23:53.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73c6d9d640 1 --2- 172.21.15.72:0/2994075294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c81586e0 0x7f73c80aea40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f73c8072e30 tx=0x7f73b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353f4f0ac9c03419 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73c659c640 1 -- 172.21.15.72:0/2994075294 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8035020 con 0x7f73c81586e0 2024-03-06T07:23:53.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73c659c640 1 -- 172.21.15.72:0/2994075294 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8031870 con 0x7f73c81586e0 2024-03-06T07:23:53.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73c659c640 1 -- 172.21.15.72:0/2994075294 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b8031c40 con 0x7f73c81586e0 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73cd24e640 1 -- 172.21.15.72:0/2994075294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c81586e0 msgr2=0x7f73c80aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73cd24e640 1 --2- 172.21.15.72:0/2994075294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c81586e0 0x7f73c80aea40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f73c8072e30 tx=0x7f73b802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73cd24e640 1 -- 172.21.15.72:0/2994075294 shutdown_connections 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73cd24e640 1 --2- 172.21.15.72:0/2994075294 >> v2:172.21.15.146:3300/0 conn(0x7f73c80aef80 0x7f73c80cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73cd24e640 1 --2- 172.21.15.72:0/2994075294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c81586e0 0x7f73c80aea40 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.517+0000 7f73cd24e640 1 -- 172.21.15.72:0/2994075294 >> 172.21.15.72:0/2994075294 conn(0x7f73c806b190 msgr2=0x7f73c806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.518+0000 7f73cd24e640 1 -- 172.21.15.72:0/2994075294 shutdown_connections 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.518+0000 7f73cd24e640 1 -- 172.21.15.72:0/2994075294 wait complete. 2024-03-06T07:23:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.519+0000 7f73cd24e640 1 Processor -- start 2024-03-06T07:23:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.519+0000 7f73cd24e640 1 -- start start 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 0x7f73c80c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73c80cffd0 0x7f73c80cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73c81586e0 0x7f73c80cb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8071350 con 0x7f73c80aef80 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c815bde0 con 0x7f73c80cffd0 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c80714d0 con 0x7f73c81586e0 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 0x7f73c80c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 0x7f73c80c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39358/0 (socket says 172.21.15.72:39358) 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 -- 172.21.15.72:0/4101310623 learned_addr learned my addr 172.21.15.72:0/4101310623 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73be59c640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73c80cffd0 0x7f73c80cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c759e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73c81586e0 0x7f73c80cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 -- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73c81586e0 msgr2=0x7f73c80cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73c81586e0 0x7f73c80cb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 -- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73c80cffd0 msgr2=0x7f73c80cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73c80cffd0 0x7f73c80cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 -- 172.21.15.72:0/4101310623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c80aac40 con 0x7f73c80aef80 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73be59c640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73c80cffd0 0x7f73c80cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c759e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73c81586e0 0x7f73c80cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73c6d9d640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 0x7f73c80c4a50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f73b8000c00 tx=0x7f73b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b803ad60 con 0x7f73c80aef80 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b80443f0 con 0x7f73c80aef80 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73b80447e0 con 0x7f73c80aef80 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.520+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c80aaf20 con 0x7f73c80aef80 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.521+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c80abec0 con 0x7f73c80aef80 2024-03-06T07:23:53.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.521+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c80732b0 con 0x7f73c80aef80 2024-03-06T07:23:53.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.522+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73b804b060 con 0x7f73c80aef80 2024-03-06T07:23:53.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.522+0000 7f73bffff640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.522+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f73b80ac0b0 con 0x7f73c80aef80 2024-03-06T07:23:53.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.522+0000 7f73be59c640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.523+0000 7f73be59c640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f73a806b4d0 0x7f73a806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f73b0006fd0 tx=0x7f73b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.527+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73b803f070 con 0x7f73c80aef80 2024-03-06T07:23:53.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.696+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f73c815a6c0 con 0x7f73c80aef80 2024-03-06T07:23:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.701+0000 7f73bffff640 1 -- 172.21.15.72:0/4101310623 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v15) v1 ==== 108+0+0 (secure 0 0 0) 0x7f73b802f320 con 0x7f73c80aef80 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f73a806b4d0 msgr2=0x7f73a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f73a806b4d0 0x7f73a806d990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f73b0006fd0 tx=0x7f73b0008040 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 msgr2=0x7f73c80c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 0x7f73c80c4a50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f73b8000c00 tx=0x7f73b80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 shutdown_connections 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f73a806b4d0 0x7f73a806d990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73c81586e0 0x7f73c80cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73c80cffd0 0x7f73c80cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 --2- 172.21.15.72:0/4101310623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73c80aef80 0x7f73c80c4a50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 >> 172.21.15.72:0/4101310623 conn(0x7f73c806b190 msgr2=0x7f73c80af570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 shutdown_connections 2024-03-06T07:23:53.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.704+0000 7f73cd24e640 1 -- 172.21.15.72:0/4101310623 wait complete. 2024-03-06T07:23:53.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-03-06T07:23:53.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.822+0000 7f2f152f4640 1 Processor -- start 2024-03-06T07:23:53.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.822+0000 7f2f152f4640 1 -- start start 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f152f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1015ab00 0x7f2f100b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f152f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f100b48f0 0x7f2f100af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f152f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f100afc00 0x7f2f1006e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f152f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f100b4e30 con 0x7f2f100b48f0 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f152f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f100b5110 con 0x7f2f100afc00 2024-03-06T07:23:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f152f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1015b310 con 0x7f2f1015ab00 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f0e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f100b48f0 0x7f2f100af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f0e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f100b48f0 0x7f2f100af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39370/0 (socket says 172.21.15.72:39370) 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f0e7fc640 1 -- 172.21.15.72:0/2808989323 learned_addr learned my addr 172.21.15.72:0/2808989323 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f0f7fe640 1 --2- 172.21.15.72:0/2808989323 >> v2:172.21.15.146:3300/0 conn(0x7f2f100afc00 0x7f2f1006e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.823+0000 7f2f0effd640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1015ab00 0x7f2f100b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0e7fc640 1 -- 172.21.15.72:0/2808989323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1015ab00 msgr2=0x7f2f100b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0e7fc640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1015ab00 0x7f2f100b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0e7fc640 1 -- 172.21.15.72:0/2808989323 >> v2:172.21.15.146:3300/0 conn(0x7f2f100afc00 msgr2=0x7f2f1006e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0e7fc640 1 --2- 172.21.15.72:0/2808989323 >> v2:172.21.15.146:3300/0 conn(0x7f2f100afc00 0x7f2f1006e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0e7fc640 1 -- 172.21.15.72:0/2808989323 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1015b5f0 con 0x7f2f100b48f0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0f7fe640 1 --2- 172.21.15.72:0/2808989323 >> v2:172.21.15.146:3300/0 conn(0x7f2f100afc00 0x7f2f1006e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0effd640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1015ab00 0x7f2f100b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0e7fc640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f100b48f0 0x7f2f100af6c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2f1006e6a0 tx=0x7f2f0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f762fabd28f901d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0dffb640 1 -- 172.21.15.72:0/2808989323 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04035020 con 0x7f2f100b48f0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0dffb640 1 -- 172.21.15.72:0/2808989323 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f04031870 con 0x7f2f100b48f0 2024-03-06T07:23:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.824+0000 7f2f0dffb640 1 -- 172.21.15.72:0/2808989323 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04031c40 con 0x7f2f100b48f0 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 -- 172.21.15.72:0/2808989323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f100b48f0 msgr2=0x7f2f100af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f100b48f0 0x7f2f100af6c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f2f1006e6a0 tx=0x7f2f0402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 -- 172.21.15.72:0/2808989323 shutdown_connections 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2808989323 >> v2:172.21.15.146:3300/0 conn(0x7f2f100afc00 0x7f2f1006e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f100b48f0 0x7f2f100af6c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2808989323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f1015ab00 0x7f2f100b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 -- 172.21.15.72:0/2808989323 >> 172.21.15.72:0/2808989323 conn(0x7f2f1006c0c0 msgr2=0x7f2f1006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 -- 172.21.15.72:0/2808989323 shutdown_connections 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.825+0000 7f2f152f4640 1 -- 172.21.15.72:0/2808989323 wait complete. 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.826+0000 7f2f152f4640 1 Processor -- start 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.826+0000 7f2f152f4640 1 -- start start 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f152f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f100afc00 0x7f2f100cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f152f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100b48f0 0x7f2f100ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f152f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 0x7f2f1006b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f152f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1015b060 con 0x7f2f1015ab00 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f152f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1015aee0 con 0x7f2f100afc00 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f152f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1006ecc0 con 0x7f2f100b48f0 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 0x7f2f1006b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 0x7f2f1006b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39372/0 (socket says 172.21.15.72:39372) 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 -- 172.21.15.72:0/2536193849 learned_addr learned my addr 172.21.15.72:0/2536193849 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0effd640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f100afc00 0x7f2f100cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0e7fc640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100b48f0 0x7f2f100ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 -- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100b48f0 msgr2=0x7f2f100ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100b48f0 0x7f2f100ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 -- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f100afc00 msgr2=0x7f2f100cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f100afc00 0x7f2f100cdc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.827+0000 7f2f0f7fe640 1 -- 172.21.15.72:0/2536193849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f100ce2b0 con 0x7f2f1015ab00 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2f0effd640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f100afc00 0x7f2f100cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:53.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2f0f7fe640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 0x7f2f1006b220 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800e7e0 tx=0x7f2f0800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800f040 con 0x7f2f1015ab00 2024-03-06T07:23:53.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f08004030 con 0x7f2f1015ab00 2024-03-06T07:23:53.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f100ce590 con 0x7f2f1015ab00 2024-03-06T07:23:53.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f080043e0 con 0x7f2f1015ab00 2024-03-06T07:23:53.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.828+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f100ceb50 con 0x7f2f1015ab00 2024-03-06T07:23:53.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.829+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f100b3810 con 0x7f2f1015ab00 2024-03-06T07:23:53.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.829+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f0800c6d0 con 0x7f2f1015ab00 2024-03-06T07:23:53.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.830+0000 7f2ef77fe640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2efc06b4d0 0x7f2efc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:53.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.830+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2f08089a90 con 0x7f2f1015ab00 2024-03-06T07:23:53.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.830+0000 7f2f0effd640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2efc06b4d0 0x7f2efc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:53.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.830+0000 7f2f0effd640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2efc06b4d0 0x7f2efc06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2f000095c0 tx=0x7f2f00009530 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:53.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:53.834+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f100b3810 con 0x7f2f1015ab00 2024-03-06T07:23:54.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.072+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) v1 -- 0x7f2f1004f330 con 0x7f2f1015ab00 2024-03-06T07:23:54.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.098+0000 7f2ef77fe640 1 -- 172.21.15.72:0/2536193849 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v130) v1 ==== 133+0+0 (secure 0 0 0) 0x7f2f08017020 con 0x7f2f1015ab00 2024-03-06T07:23:54.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2efc06b4d0 msgr2=0x7f2efc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2efc06b4d0 0x7f2efc06d990 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2f000095c0 tx=0x7f2f00009530 comp rx=0 tx=0).stop 2024-03-06T07:23:54.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 msgr2=0x7f2f1006b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 0x7f2f1006b220 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800e7e0 tx=0x7f2f0800eca0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 shutdown_connections 2024-03-06T07:23:54.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2efc06b4d0 0x7f2efc06d990 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f1015ab00 0x7f2f1006b220 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f100b48f0 0x7f2f100ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 --2- 172.21.15.72:0/2536193849 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f100afc00 0x7f2f100cdc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 >> 172.21.15.72:0/2536193849 conn(0x7f2f1006c0c0 msgr2=0x7f2f100b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 shutdown_connections 2024-03-06T07:23:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.100+0000 7f2f152f4640 1 -- 172.21.15.72:0/2536193849 wait complete. 2024-03-06T07:23:54.114 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-03-06T07:23:54.114 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-03-06T07:23:54.114 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:23:54.114 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:54.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.212+0000 7f336929d640 1 Processor -- start 2024-03-06T07:23:54.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.212+0000 7f336929d640 1 -- start start 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f336929d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f336413dc10 0x7f33640b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f336929d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 0x7f33640b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f336929d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336415f580 con 0x7f336415e850 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f336929d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33640cf3f0 con 0x7f336413dc10 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f336929d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336415b210 con 0x7f336415bf30 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f33627fc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 0x7f33640b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f3362ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f336413dc10 0x7f33640b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f33627fc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 0x7f33640b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:54888/0 (socket says 172.21.15.72:54888) 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f33637fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f336415e850 0x7f336415eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57628/0 (socket says 172.21.15.72:57628) 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f33627fc640 1 -- 172.21.15.72:0/2698894158 learned_addr learned my addr 172.21.15.72:0/2698894158 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.213+0000 7f3362ffd640 1 --2- 172.21.15.72:0/2698894158 >> v2:172.21.15.146:3300/0 conn(0x7f336413dc10 0x7f33640b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:60266/0 (socket says 172.21.15.72:60266) 2024-03-06T07:23:54.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.214+0000 7f3362ffd640 1 -- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 msgr2=0x7f33640b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.214+0000 7f3362ffd640 1 --2- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 0x7f33640b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.214+0000 7f3362ffd640 1 -- 172.21.15.72:0/2698894158 >> v1:172.21.15.146:6789/0 conn(0x7f336415e850 legacy=0x7f336415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.214+0000 7f3362ffd640 1 -- 172.21.15.72:0/2698894158 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336415b450 con 0x7f336413dc10 2024-03-06T07:23:54.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.214+0000 7f33627fc640 1 --2- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 0x7f33640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f3362ffd640 1 --2- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336413dc10 0x7f33640b2950 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f33540097a0 tx=0x7f335402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbbbd001dfbf3ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f3361ffb640 1 -- 172.21.15.72:0/2698894158 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540315a0 con 0x7f336413dc10 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f3361ffb640 1 -- 172.21.15.72:0/2698894158 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3354037070 con 0x7f336413dc10 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f3361ffb640 1 -- 172.21.15.72:0/2698894158 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354031960 con 0x7f336413dc10 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 -- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336413dc10 msgr2=0x7f33640b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 --2- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336413dc10 0x7f33640b2950 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f33540097a0 tx=0x7f335402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 -- 172.21.15.72:0/2698894158 shutdown_connections 2024-03-06T07:23:54.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 --2- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415bf30 0x7f33640b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 --2- 172.21.15.72:0/2698894158 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336413dc10 0x7f33640b2950 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 -- 172.21.15.72:0/2698894158 >> 172.21.15.72:0/2698894158 conn(0x7f33640cf8a0 msgr2=0x7f33640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 -- 172.21.15.72:0/2698894158 shutdown_connections 2024-03-06T07:23:54.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.215+0000 7f336929d640 1 -- 172.21.15.72:0/2698894158 wait complete. 2024-03-06T07:23:54.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 Processor -- start 2024-03-06T07:23:54.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 -- start start 2024-03-06T07:23:54.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 0x7f336413e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415bf30 0x7f3364133370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e850 0x7f33641f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33640708d0 con 0x7f336413dc10 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336413dff0 con 0x7f336415bf30 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f336929d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3364070a50 con 0x7f336415e850 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 0x7f336413e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f3362ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 0x7f336413e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39388/0 (socket says 172.21.15.72:39388) 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.217+0000 7f3362ffd640 1 -- 172.21.15.72:0/675753562 learned_addr learned my addr 172.21.15.72:0/675753562 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f33627fc640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415bf30 0x7f3364133370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f33637fe640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e850 0x7f33641f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f3362ffd640 1 -- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e850 msgr2=0x7f33641f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f3362ffd640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e850 0x7f33641f6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f3362ffd640 1 -- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415bf30 msgr2=0x7f3364133370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f3362ffd640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415bf30 0x7f3364133370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f3362ffd640 1 -- 172.21.15.72:0/675753562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33641338b0 con 0x7f336413dc10 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f3362ffd640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 0x7f336413e170 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f33540098c0 tx=0x7f33540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354032630 con 0x7f336413dc10 2024-03-06T07:23:54.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33540327f0 con 0x7f336413dc10 2024-03-06T07:23:54.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3354032be0 con 0x7f336413dc10 2024-03-06T07:23:54.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336413b110 con 0x7f336413dc10 2024-03-06T07:23:54.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.218+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336413b6d0 con 0x7f336413dc10 2024-03-06T07:23:54.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.219+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336404f330 con 0x7f336413dc10 2024-03-06T07:23:54.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.220+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f335404e430 con 0x7f336413dc10 2024-03-06T07:23:54.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.220+0000 7f334b7fe640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f335006b4d0 0x7f335006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.220+0000 7f33627fc640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f335006b4d0 0x7f335006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.220+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f335402f030 con 0x7f336413dc10 2024-03-06T07:23:54.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.221+0000 7f33627fc640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f335006b4d0 0x7f335006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f33641f8140 tx=0x7f3358009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.224+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f33540ade20 con 0x7f336413dc10 2024-03-06T07:23:54.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.395+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f336406eb00 con 0x7f336413dc10 2024-03-06T07:23:54.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.396+0000 7f334b7fe640 1 -- 172.21.15.72:0/675753562 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+13030 (secure 0 0 0) 0x7f3354040070 con 0x7f336413dc10 2024-03-06T07:23:54.398 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:54.398 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":51,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:53.090036+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/874695441":"2024-03-07T07:23:50.079129+0000","172.21.15.72:6835/874695441":"2024-03-07T07:23:50.079129+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:54.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f335006b4d0 msgr2=0x7f335006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f335006b4d0 0x7f335006d990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f33641f8140 tx=0x7f3358009240 comp rx=0 tx=0).stop 2024-03-06T07:23:54.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 msgr2=0x7f336413e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 0x7f336413e170 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f33540098c0 tx=0x7f33540046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 shutdown_connections 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f335006b4d0 0x7f335006d990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f336415e850 0x7f33641f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f336415bf30 0x7f3364133370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 --2- 172.21.15.72:0/675753562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336413dc10 0x7f336413e170 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 >> 172.21.15.72:0/675753562 conn(0x7f33640cf8a0 msgr2=0x7f336415a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 shutdown_connections 2024-03-06T07:23:54.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.398+0000 7f336929d640 1 -- 172.21.15.72:0/675753562 wait complete. 2024-03-06T07:23:54.412 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:54.412 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:54.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.513+0000 7f0937b6b640 1 Processor -- start 2024-03-06T07:23:54.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.513+0000 7f0937b6b640 1 -- start start 2024-03-06T07:23:54.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0937b6b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f0930066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0937b6b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09300b0db0 0x7f09300ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0937b6b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09300ce580 0x7f0930072550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0937b6b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093006b050 con 0x7f09300ce580 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0937b6b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093006b2c0 con 0x7f09300b0db0 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0937b6b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09300b0890 con 0x7f093015f4f0 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09300ce580 0x7f0930072550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09300ce580 0x7f0930072550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39394/0 (socket says 172.21.15.72:39394) 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 -- 172.21.15.72:0/1303224303 learned_addr learned my addr 172.21.15.72:0/1303224303 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0935106640 1 --2- 172.21.15.72:0/1303224303 >> v2:172.21.15.146:3300/0 conn(0x7f09300b0db0 0x7f09300ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0935907640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f0930066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 -- 172.21.15.72:0/1303224303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 msgr2=0x7f0930066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f0930066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 -- 172.21.15.72:0/1303224303 >> v2:172.21.15.146:3300/0 conn(0x7f09300b0db0 msgr2=0x7f09300ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 --2- 172.21.15.72:0/1303224303 >> v2:172.21.15.146:3300/0 conn(0x7f09300b0db0 0x7f09300ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0936108640 1 -- 172.21.15.72:0/1303224303 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300672e0 con 0x7f09300ce580 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0935106640 1 --2- 172.21.15.72:0/1303224303 >> v2:172.21.15.146:3300/0 conn(0x7f09300b0db0 0x7f09300ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.514+0000 7f0935907640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f0930066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0936108640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930072550 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0930070b70 tx=0x7f092802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb37dd411461fb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0934905640 1 -- 172.21.15.72:0/1303224303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092802ca70 con 0x7f09300ce580 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0934905640 1 -- 172.21.15.72:0/1303224303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092802cc10 con 0x7f09300ce580 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0934905640 1 -- 172.21.15.72:0/1303224303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09280343f0 con 0x7f09300ce580 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 -- 172.21.15.72:0/1303224303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 msgr2=0x7f0930072550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930072550 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0930070b70 tx=0x7f092802bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 -- 172.21.15.72:0/1303224303 shutdown_connections 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930072550 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 --2- 172.21.15.72:0/1303224303 >> v2:172.21.15.146:3300/0 conn(0x7f09300b0db0 0x7f09300ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 --2- 172.21.15.72:0/1303224303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f0930066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.515+0000 7f0937b6b640 1 -- 172.21.15.72:0/1303224303 >> 172.21.15.72:0/1303224303 conn(0x7f09300b1d70 msgr2=0x7f09300b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.516+0000 7f0937b6b640 1 -- 172.21.15.72:0/1303224303 shutdown_connections 2024-03-06T07:23:54.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.516+0000 7f0937b6b640 1 -- 172.21.15.72:0/1303224303 wait complete. 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 Processor -- start 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 -- start start 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09300b0db0 0x7f09300c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f093015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093015f8d0 con 0x7f09300ce580 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093006b5a0 con 0x7f09300b0db0 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.517+0000 7f0937b6b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09300cc4c0 con 0x7f093015f4f0 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39402/0 (socket says 172.21.15.72:39402) 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 -- 172.21.15.72:0/57193457 learned_addr learned my addr 172.21.15.72:0/57193457 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935907640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09300b0db0 0x7f09300c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0936108640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f093015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 -- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 msgr2=0x7f093015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f093015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 -- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09300b0db0 msgr2=0x7f09300c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09300b0db0 0x7f09300c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 -- 172.21.15.72:0/57193457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300cc1e0 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935907640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09300b0db0 0x7f09300c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0935106640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930152600 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f092000c7f0 tx=0x7f092000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920015070 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0920007a30 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09300c57f0 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920007e00 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.518+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09300c5db0 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.519+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093009e3d0 con 0x7f09300ce580 2024-03-06T07:23:54.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.520+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0920018070 con 0x7f09300ce580 2024-03-06T07:23:54.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.520+0000 7f09267fc640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f091406b4d0 0x7f091406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.520+0000 7f0935907640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f091406b4d0 0x7f091406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.520+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0920089d90 con 0x7f09300ce580 2024-03-06T07:23:54.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.520+0000 7f0935907640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f091406b4d0 0x7f091406d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f092c00a990 tx=0x7f092c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.524+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0920058fa0 con 0x7f09300ce580 2024-03-06T07:23:54.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.694+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f093006cd40 con 0x7f09300ce580 2024-03-06T07:23:54.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.694+0000 7f09267fc640 1 -- 172.21.15.72:0/57193457 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09200588a0 con 0x7f09300ce580 2024-03-06T07:23:54.696 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f091406b4d0 msgr2=0x7f091406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f091406b4d0 0x7f091406d990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f092c00a990 tx=0x7f092c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 msgr2=0x7f0930152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930152600 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f092000c7f0 tx=0x7f092000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 shutdown_connections 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f091406b4d0 0x7f091406d990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f093015f4f0 0x7f093015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09300ce580 0x7f0930152600 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 --2- 172.21.15.72:0/57193457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09300b0db0 0x7f09300c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 >> 172.21.15.72:0/57193457 conn(0x7f09300b1d70 msgr2=0x7f09300cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 shutdown_connections 2024-03-06T07:23:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.696+0000 7f0937b6b640 1 -- 172.21.15.72:0/57193457 wait complete. 2024-03-06T07:23:54.709 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:23:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.808+0000 7f07d2247640 1 Processor -- start 2024-03-06T07:23:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.808+0000 7f07d2247640 1 -- start start 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07d2247640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07cc0b3060 0x7f07cc15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07d2247640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07cc15bf30 0x7f07cc15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07d2247640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc15f2d0 con 0x7f07cc13dc10 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07d2247640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc0cf3f0 con 0x7f07cc15bf30 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07d2247640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc15b210 con 0x7f07cc0b3060 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cbfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07cc15bf30 0x7f07cc15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f07cc13dc10 0x7f07cc0b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57090/0 (socket says 172.21.15.72:57090) 2024-03-06T07:23:54.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cb7fe640 1 -- 172.21.15.72:0/3321747772 learned_addr learned my addr 172.21.15.72:0/3321747772 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07caffd640 1 --2- 172.21.15.72:0/3321747772 >> v2:172.21.15.146:3300/0 conn(0x7f07cc0b3060 0x7f07cc15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cbfff640 1 -- 172.21.15.72:0/3321747772 >> v2:172.21.15.146:3300/0 conn(0x7f07cc0b3060 msgr2=0x7f07cc15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cbfff640 1 --2- 172.21.15.72:0/3321747772 >> v2:172.21.15.146:3300/0 conn(0x7f07cc0b3060 0x7f07cc15e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cbfff640 1 -- 172.21.15.72:0/3321747772 >> v1:172.21.15.72:6789/0 conn(0x7f07cc13dc10 legacy=0x7f07cc0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07cbfff640 1 -- 172.21.15.72:0/3321747772 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc15f580 con 0x7f07cc15bf30 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.809+0000 7f07ca7fc640 1 -- 172.21.15.72:0/3321747772 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576638636 0 0) 0x7f07cc15f2d0 con 0x7f07cc13dc10 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.810+0000 7f07cbfff640 1 --2- 172.21.15.72:0/3321747772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc15bf30 0x7f07cc15ed90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f07cc068700 tx=0x7f07c402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255068346e91a640 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.810+0000 7f07ca7fc640 1 -- 172.21.15.72:0/3321747772 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c402e9d0 con 0x7f07cc15bf30 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.810+0000 7f07ca7fc640 1 -- 172.21.15.72:0/3321747772 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c402eb70 con 0x7f07cc15bf30 2024-03-06T07:23:54.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.810+0000 7f07ca7fc640 1 -- 172.21.15.72:0/3321747772 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4009540 con 0x7f07cc15bf30 2024-03-06T07:23:54.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 -- 172.21.15.72:0/3321747772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc15bf30 msgr2=0x7f07cc15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 --2- 172.21.15.72:0/3321747772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc15bf30 0x7f07cc15ed90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f07cc068700 tx=0x7f07c402e030 comp rx=0 tx=0).stop 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 -- 172.21.15.72:0/3321747772 shutdown_connections 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 --2- 172.21.15.72:0/3321747772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc15bf30 0x7f07cc15ed90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 --2- 172.21.15.72:0/3321747772 >> v2:172.21.15.146:3300/0 conn(0x7f07cc0b3060 0x7f07cc15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 -- 172.21.15.72:0/3321747772 >> 172.21.15.72:0/3321747772 conn(0x7f07cc0cf8a0 msgr2=0x7f07cc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 -- 172.21.15.72:0/3321747772 shutdown_connections 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.811+0000 7f07d2247640 1 -- 172.21.15.72:0/3321747772 wait complete. 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.812+0000 7f07d2247640 1 Processor -- start 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.812+0000 7f07d2247640 1 -- start start 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 0x7f07cc142aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07cc13dc10 0x7f07cc0af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07cc15bf30 0x7f07cc13ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc13dff0 con 0x7f07cc0b3060 2024-03-06T07:23:54.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc15b730 con 0x7f07cc13dc10 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc0cd7a0 con 0x7f07cc15bf30 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 0x7f07cc142aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 0x7f07cc142aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39418/0 (socket says 172.21.15.72:39418) 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 -- 172.21.15.72:0/3152186442 learned_addr learned my addr 172.21.15.72:0/3152186442 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 -- 172.21.15.72:0/3152186442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07cc15bf30 msgr2=0x7f07cc13ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07cc15bf30 0x7f07cc13ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 -- 172.21.15.72:0/3152186442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07cc13dc10 msgr2=0x7f07cc0af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07cc13dc10 0x7f07cc0af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 -- 172.21.15.72:0/3152186442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc143080 con 0x7f07cc0b3060 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07cb7fe640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 0x7f07cc142aa0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f07c000c500 tx=0x7f07c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c0018070 con 0x7f07cc0b3060 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c0007580 con 0x7f07cc0b3060 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc143360 con 0x7f07cc0b3060 2024-03-06T07:23:54.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.813+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c0004a20 con 0x7f07cc0b3060 2024-03-06T07:23:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.814+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc143920 con 0x7f07cc0b3060 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.815+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc143e20 con 0x7f07cc0b3060 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.815+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07c001b070 con 0x7f07cc0b3060 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.815+0000 7f07b3fff640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07b806b4d0 0x7f07b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.815+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f07c00891a0 con 0x7f07cc0b3060 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.815+0000 7f07caffd640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07b806b4d0 0x7f07b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.816+0000 7f07caffd640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07b806b4d0 0x7f07b806d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f07bc006fd0 tx=0x7f07bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:54.819+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07cc143e20 con 0x7f07cc0b3060 2024-03-06T07:23:55.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.057+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f07cc06eb00 con 0x7f07cc0b3060 2024-03-06T07:23:55.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.101+0000 7f07b3fff640 1 -- 172.21.15.72:0/3152186442 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v131) v1 ==== 147+0+0 (secure 0 0 0) 0x7f07c00583b0 con 0x7f07cc0b3060 2024-03-06T07:23:55.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07b806b4d0 msgr2=0x7f07b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07b806b4d0 0x7f07b806d990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f07bc006fd0 tx=0x7f07bc008040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 msgr2=0x7f07cc142aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 0x7f07cc142aa0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f07c000c500 tx=0x7f07c000c9c0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 shutdown_connections 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07b806b4d0 0x7f07b806d990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07cc15bf30 0x7f07cc13ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07cc13dc10 0x7f07cc0af860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 --2- 172.21.15.72:0/3152186442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc0b3060 0x7f07cc142aa0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 >> 172.21.15.72:0/3152186442 conn(0x7f07cc0cf8a0 msgr2=0x7f07cc13c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 shutdown_connections 2024-03-06T07:23:55.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.103+0000 7f07d2247640 1 -- 172.21.15.72:0/3152186442 wait complete. 2024-03-06T07:23:55.116 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:23:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.216+0000 7f47a0015640 1 Processor -- start 2024-03-06T07:23:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.216+0000 7f47a0015640 1 -- start start 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f47a0015640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4798144710 0x7f47980cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f47a0015640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f47980cdab0 0x7f47980cb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f47a0015640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479815b310 con 0x7f47980cf770 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f47a0015640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479815c3b0 con 0x7f47980cdab0 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f47a0015640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479815b5f0 con 0x7f4798144710 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f479e5b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f47980cdab0 0x7f47980cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f479d5b0640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f47980cf770 0x7f47980cd3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57098/0 (socket says 172.21.15.72:57098) 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.217+0000 7f479d5b0640 1 -- 172.21.15.72:0/3528493564 learned_addr learned my addr 172.21.15.72:0/3528493564 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479ddb1640 1 --2- 172.21.15.72:0/3528493564 >> v2:172.21.15.146:3300/0 conn(0x7f4798144710 0x7f47980cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479e5b2640 1 -- 172.21.15.72:0/3528493564 >> v2:172.21.15.146:3300/0 conn(0x7f4798144710 msgr2=0x7f47980cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479e5b2640 1 --2- 172.21.15.72:0/3528493564 >> v2:172.21.15.146:3300/0 conn(0x7f4798144710 0x7f47980cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479e5b2640 1 -- 172.21.15.72:0/3528493564 >> v1:172.21.15.72:6789/0 conn(0x7f47980cf770 legacy=0x7f47980cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479e5b2640 1 -- 172.21.15.72:0/3528493564 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479815b8d0 con 0x7f47980cdab0 2024-03-06T07:23:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479cdaf640 1 -- 172.21.15.72:0/3528493564 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632448000 0 0) 0x7f479815b310 con 0x7f47980cf770 2024-03-06T07:23:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.218+0000 7f479e5b2640 1 --2- 172.21.15.72:0/3528493564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980cb510 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f47980ce730 tx=0x7f479002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39afdd4aab141727 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f479cdaf640 1 -- 172.21.15.72:0/3528493564 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790035020 con 0x7f47980cdab0 2024-03-06T07:23:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f479cdaf640 1 -- 172.21.15.72:0/3528493564 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4790031830 con 0x7f47980cdab0 2024-03-06T07:23:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f479cdaf640 1 -- 172.21.15.72:0/3528493564 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790031c00 con 0x7f47980cdab0 2024-03-06T07:23:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 -- 172.21.15.72:0/3528493564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 msgr2=0x7f47980cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 --2- 172.21.15.72:0/3528493564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980cb510 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f47980ce730 tx=0x7f479002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 -- 172.21.15.72:0/3528493564 shutdown_connections 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 --2- 172.21.15.72:0/3528493564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980cb510 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 --2- 172.21.15.72:0/3528493564 >> v2:172.21.15.146:3300/0 conn(0x7f4798144710 0x7f47980cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 -- 172.21.15.72:0/3528493564 >> 172.21.15.72:0/3528493564 conn(0x7f47980b2660 msgr2=0x7f47980b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 -- 172.21.15.72:0/3528493564 shutdown_connections 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.219+0000 7f47a0015640 1 -- 172.21.15.72:0/3528493564 wait complete. 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.220+0000 7f47a0015640 1 Processor -- start 2024-03-06T07:23:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.220+0000 7f47a0015640 1 -- start start 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f47a0015640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f47a0015640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47980cf770 0x7f479813e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f47a0015640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798144710 0x7f479806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f47a0015640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479815f4c0 con 0x7f47980cdab0 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f47a0015640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479815be90 con 0x7f4798144710 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f47a0015640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f479815f640 con 0x7f47980cf770 2024-03-06T07:23:55.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980b63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39440/0 (socket says 172.21.15.72:39440) 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 -- 172.21.15.72:0/525013624 learned_addr learned my addr 172.21.15.72:0/525013624 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479e5b2640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798144710 0x7f479806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479d5b0640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47980cf770 0x7f479813e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 -- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47980cf770 msgr2=0x7f479813e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47980cf770 0x7f479813e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 -- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798144710 msgr2=0x7f479806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798144710 0x7f479806afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479ddb1640 1 -- 172.21.15.72:0/525013624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980b6930 con 0x7f47980cdab0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479e5b2640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798144710 0x7f479806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.221+0000 7f479d5b0640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47980cf770 0x7f479813e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.222+0000 7f479ddb1640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980b63f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4794007e70 tx=0x7f479400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.222+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794014620 con 0x7f47980cdab0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.222+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4794019070 con 0x7f47980cdab0 2024-03-06T07:23:55.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.222+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479814fc70 con 0x7f47980cdab0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.222+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4794014a00 con 0x7f47980cdab0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.222+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479814cff0 con 0x7f47980cdab0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.223+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47980733b0 con 0x7f47980cdab0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.223+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4794014ba0 con 0x7f47980cdab0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.223+0000 7f478e7fc640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477c06b4d0 0x7f477c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.224+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f479408a6e0 con 0x7f47980cdab0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.224+0000 7f479d5b0640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477c06b4d0 0x7f477c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.224+0000 7f479d5b0640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477c06b4d0 0x7f477c06d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f479815ae50 tx=0x7f4788009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.228+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f47980b6930 con 0x7f47980cdab0 2024-03-06T07:23:55.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.398+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f47980d04f0 con 0x7f47980cdab0 2024-03-06T07:23:55.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.403+0000 7f478e7fc640 1 -- 172.21.15.72:0/525013624 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f47940598f0 con 0x7f47980cdab0 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477c06b4d0 msgr2=0x7f477c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477c06b4d0 0x7f477c06d990 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f479815ae50 tx=0x7f4788009040 comp rx=0 tx=0).stop 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 msgr2=0x7f47980b63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980b63f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4794007e70 tx=0x7f479400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 shutdown_connections 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477c06b4d0 0x7f477c06d990 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798144710 0x7f479806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47980cf770 0x7f479813e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 --2- 172.21.15.72:0/525013624 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cdab0 0x7f47980b63f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.405+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 >> 172.21.15.72:0/525013624 conn(0x7f47980b2660 msgr2=0x7f47980cb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.406+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 shutdown_connections 2024-03-06T07:23:55.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.406+0000 7f47a0015640 1 -- 172.21.15.72:0/525013624 wait complete. 2024-03-06T07:23:55.418 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:23:55.419 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:55.465 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:23:55.507 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:23:55.507 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:55.507 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:55.523 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:55.523 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:55.523 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:55.565 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:55.565 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:23:55.565 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:55.579 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:23:55.580 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:23:55.580 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:23:55.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:23:55.623 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:23:55.623 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.721+0000 7f535cb5e640 1 Processor -- start 2024-03-06T07:23:55.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.721+0000 7f535cb5e640 1 -- start start 2024-03-06T07:23:55.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535cb5e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f535813dc10 0x7f53580b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535cb5e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815ef60 0x7f53580b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535cb5e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535815f580 con 0x7f535815bf30 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535cb5e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53580cf3f0 con 0x7f535813dc10 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535cb5e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535815b210 con 0x7f535815ef60 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535659c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f535813dc10 0x7f53580b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f534ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f535815bf30 0x7f535815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57108/0 (socket says 172.21.15.72:57108) 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f534ffff640 1 -- 172.21.15.72:0/2663107755 learned_addr learned my addr 172.21.15.72:0/2663107755 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535659c640 1 -- 172.21.15.72:0/2663107755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815ef60 msgr2=0x7f53580b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f5356d9d640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815ef60 0x7f53580b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535659c640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815ef60 0x7f53580b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535659c640 1 -- 172.21.15.72:0/2663107755 >> v1:172.21.15.72:6789/0 conn(0x7f535815bf30 legacy=0x7f535815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f535659c640 1 -- 172.21.15.72:0/2663107755 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535815b450 con 0x7f535813dc10 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f5355d9b640 1 -- 172.21.15.72:0/2663107755 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314840094 0 0) 0x7f535815f580 con 0x7f535815bf30 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.722+0000 7f5356d9d640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815ef60 0x7f53580b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.723+0000 7f535659c640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53580b2950 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f53480097a0 tx=0x7f534802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b9ae710df1ed2d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.723+0000 7f5355d9b640 1 -- 172.21.15.72:0/2663107755 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53480315a0 con 0x7f535813dc10 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.723+0000 7f5355d9b640 1 -- 172.21.15.72:0/2663107755 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5348037070 con 0x7f535813dc10 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.723+0000 7f5355d9b640 1 -- 172.21.15.72:0/2663107755 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348031960 con 0x7f535813dc10 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 -- 172.21.15.72:0/2663107755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 msgr2=0x7f53580b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53580b2950 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f53480097a0 tx=0x7f534802b840 comp rx=0 tx=0).stop 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 -- 172.21.15.72:0/2663107755 shutdown_connections 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815ef60 0x7f53580b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 --2- 172.21.15.72:0/2663107755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53580b2950 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 -- 172.21.15.72:0/2663107755 >> 172.21.15.72:0/2663107755 conn(0x7f53580cf8a0 msgr2=0x7f53580cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 -- 172.21.15.72:0/2663107755 shutdown_connections 2024-03-06T07:23:55.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.724+0000 7f535cb5e640 1 -- 172.21.15.72:0/2663107755 wait complete. 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.725+0000 7f535cb5e640 1 Processor -- start 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.725+0000 7f535cb5e640 1 -- start start 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53581506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815bf30 0x7f5358144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f535815ef60 0x7f53581333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580cd7a0 con 0x7f535813dc10 2024-03-06T07:23:55.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535813dff0 con 0x7f535815ef60 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53580cd920 con 0x7f535815bf30 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53581506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53581506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39452/0 (socket says 172.21.15.72:39452) 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 -- 172.21.15.72:0/4225451869 learned_addr learned my addr 172.21.15.72:0/4225451869 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 -- 172.21.15.72:0/4225451869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815bf30 msgr2=0x7f5358144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815bf30 0x7f5358144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 -- 172.21.15.72:0/4225451869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f535815ef60 msgr2=0x7f53581333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f535815ef60 0x7f53581333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 -- 172.21.15.72:0/4225451869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53581360b0 con 0x7f535813dc10 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535659c640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53581506a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f53480097a0 tx=0x7f53480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348004230 con 0x7f535813dc10 2024-03-06T07:23:55.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53480043f0 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535813e2a0 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5348032880 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.726+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535813e860 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.727+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53581552d0 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.728+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5348032a20 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.728+0000 7f534effd640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f533806b4d0 0x7f533806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.728+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f534802f030 con 0x7f535813dc10 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.728+0000 7f534ffff640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f533806b4d0 0x7f533806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:55.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.728+0000 7f534ffff640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f533806b4d0 0x7f533806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5340007fd0 tx=0x7f5340007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:55.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.732+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53581552d0 con 0x7f535813dc10 2024-03-06T07:23:55.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.945+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f535804f330 con 0x7f535813dc10 2024-03-06T07:23:55.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.946+0000 7f534effd640 1 -- 172.21.15.72:0/4225451869 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f5348041070 con 0x7f535813dc10 2024-03-06T07:23:55.948 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:55.949 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5245,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/3774437686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":3774437686},{"type":"v1","addr":"172.21.15.146:6834","nonce":3774437686}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1520118559","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1520118559},{"type":"v1","addr":"172.21.15.146:6835","nonce":1520118559}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5667,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3934203642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3934203642},{"type":"v1","addr":"172.21.15.72:6836","nonce":3934203642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5973,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3581854586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3581854586},{"type":"v1","addr":"172.21.15.72:6837","nonce":3581854586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22}],"filesystems":[]} 2024-03-06T07:23:55.949 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 22 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f533806b4d0 msgr2=0x7f533806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f533806b4d0 0x7f533806d990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5340007fd0 tx=0x7f5340007920 comp rx=0 tx=0).stop 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 msgr2=0x7f53581506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53581506a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f53480097a0 tx=0x7f53480046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 shutdown_connections 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f533806b4d0 0x7f533806d990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f535815ef60 0x7f53581333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f535815bf30 0x7f5358144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 --2- 172.21.15.72:0/4225451869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f535813dc10 0x7f53581506a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 >> 172.21.15.72:0/4225451869 conn(0x7f53580cf8a0 msgr2=0x7f53580b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 shutdown_connections 2024-03-06T07:23:55.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:55.948+0000 7f535cb5e640 1 -- 172.21.15.72:0/4225451869 wait complete. 2024-03-06T07:23:55.961 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:23:55.961 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:23:55.961 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:23:55.961 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:55.962 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:23:55.962 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:23:55.962 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:23:55.962 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:55.962 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:23:55.962 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:23:55.963 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:23:55.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:55.963 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:23:55.963 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:23:55.963 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:23:55.963 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:23:55.971 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:23:55.972 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:23:55.972 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:23:55.973 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:23:55.978 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:23:55.978 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:23:55.979 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:23:55.979 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:23:56.016 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:23:56.016 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:56.017 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:23:56.017 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:56.021 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:23:56.021 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:56.022 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:23:56.022 DEBUG:teuthology.parallel:result is None 2024-03-06T07:23:56.022 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:23:56.092 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:23:56.092 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:23:56.107 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:23:56.113 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:23:56.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.127+0000 7fdcde13a640 1 Processor -- start 2024-03-06T07:23:56.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.127+0000 7fdcde13a640 1 -- start start 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcde13a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd806fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcde13a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcd80d1500 0x7fdcd806afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcde13a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80cd4c0 con 0x7fdcd806b4f0 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcde13a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd80cfeb0 con 0x7fdcd80d1500 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcde13a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80cda40 con 0x7fdcd8145760 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdcd80d1500 0x7fdcd806afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdcd806b4f0 0x7fdcd815f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57146/0 (socket says 172.21.15.72:57146) 2024-03-06T07:23:56.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd7fff640 1 -- 172.21.15.72:0/3701371236 learned_addr learned my addr 172.21.15.72:0/3701371236 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd77fe640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd806fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/3701371236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 msgr2=0x7fdcd806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd6ffd640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd806fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/3701371236 >> v1:172.21.15.72:6789/0 conn(0x7fdcd806b4f0 legacy=0x7fdcd815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd67fc640 1 -- 172.21.15.72:0/3701371236 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568030823 0 0) 0x7fdcd80cd4c0 con 0x7fdcd806b4f0 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/3701371236 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80d0340 con 0x7fdcd80d1500 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.128+0000 7fdcd77fe640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd806fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.129+0000 7fdcd6ffd640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd806afb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0097a0 tx=0x7fdccc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee834872203a8bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.129+0000 7fdcd67fc640 1 -- 172.21.15.72:0/3701371236 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc034070 con 0x7fdcd80d1500 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.129+0000 7fdcd67fc640 1 -- 172.21.15.72:0/3701371236 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc030610 con 0x7fdcd80d1500 2024-03-06T07:23:56.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.129+0000 7fdcd67fc640 1 -- 172.21.15.72:0/3701371236 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc0309e0 con 0x7fdcd80d1500 2024-03-06T07:23:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 -- 172.21.15.72:0/3701371236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 msgr2=0x7fdcd806afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd806afb0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0097a0 tx=0x7fdccc02b960 comp rx=0 tx=0).stop 2024-03-06T07:23:56.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 -- 172.21.15.72:0/3701371236 shutdown_connections 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd806afb0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 --2- 172.21.15.72:0/3701371236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd806fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 -- 172.21.15.72:0/3701371236 >> 172.21.15.72:0/3701371236 conn(0x7fdcd80d0a10 msgr2=0x7fdcd80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 -- 172.21.15.72:0/3701371236 shutdown_connections 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.130+0000 7fdcde13a640 1 -- 172.21.15.72:0/3701371236 wait complete. 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.131+0000 7fdcde13a640 1 Processor -- start 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.131+0000 7fdcde13a640 1 -- start start 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcde13a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd806b4f0 0x7fdcd813ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcde13a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd8153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcde13a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd8153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcde13a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806be80 con 0x7fdcd80d1500 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcde13a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80cfb40 con 0x7fdcd806b4f0 2024-03-06T07:23:56.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcde13a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd806e7b0 con 0x7fdcd8145760 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd8153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd8153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39494/0 (socket says 172.21.15.72:39494) 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/2162792073 learned_addr learned my addr 172.21.15.72:0/2162792073 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd77fe640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd806b4f0 0x7fdcd813ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 msgr2=0x7fdcd8153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd8153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd806b4f0 msgr2=0x7fdcd813ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd806b4f0 0x7fdcd813ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 -- 172.21.15.72:0/2162792073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80cca50 con 0x7fdcd80d1500 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd77fe640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd806b4f0 0x7fdcd813ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcd6ffd640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd8153870 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdccc000c00 tx=0x7fdccc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc047070 con 0x7fdcd80d1500 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.132+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc039a10 con 0x7fdcd80d1500 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.133+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc034040 con 0x7fdcd80d1500 2024-03-06T07:23:56.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.133+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd813f2c0 con 0x7fdcd80d1500 2024-03-06T07:23:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.133+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81547d0 con 0x7fdcd80d1500 2024-03-06T07:23:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.134+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd804f330 con 0x7fdcd80d1500 2024-03-06T07:23:56.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.134+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdccc04a070 con 0x7fdcd80d1500 2024-03-06T07:23:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.134+0000 7fdcbffff640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcc406b4d0 0x7fdcc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.134+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdccc0ab790 con 0x7fdcd80d1500 2024-03-06T07:23:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.135+0000 7fdcd77fe640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcc406b4d0 0x7fdcc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.135+0000 7fdcd77fe640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcc406b4d0 0x7fdcc406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8006fd0 tx=0x7fdcc8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.139+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdccc07a910 con 0x7fdcd80d1500 2024-03-06T07:23:56.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.325+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fdcd80723e0 con 0x7fdcd80d1500 2024-03-06T07:23:56.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.410+0000 7fdcbffff640 1 -- 172.21.15.72:0/2162792073 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdccc07a210 con 0x7fdcd80d1500 2024-03-06T07:23:56.413 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:23:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcc406b4d0 msgr2=0x7fdcc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcc406b4d0 0x7fdcc406d990 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8006fd0 tx=0x7fdcc8008040 comp rx=0 tx=0).stop 2024-03-06T07:23:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 msgr2=0x7fdcd8153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd8153870 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fdccc000c00 tx=0x7fdccc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 shutdown_connections 2024-03-06T07:23:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcc406b4d0 0x7fdcc406d990 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdcd8145760 0x7fdcd8153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdcd80d1500 0x7fdcd8153870 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 --2- 172.21.15.72:0/2162792073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdcd806b4f0 0x7fdcd813ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.413+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 >> 172.21.15.72:0/2162792073 conn(0x7fdcd80d0a10 msgr2=0x7fdcd80b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.414+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 shutdown_connections 2024-03-06T07:23:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.414+0000 7fdcde13a640 1 -- 172.21.15.72:0/2162792073 wait complete. 2024-03-06T07:23:56.428 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:23:56.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.525+0000 7fcde9088640 1 Processor -- start 2024-03-06T07:23:56.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.525+0000 7fcde9088640 1 -- start start 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde9088640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcde41515e0 0x7fcde40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde9088640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcde415b430 0x7fcde4071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde9088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde40cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde9088640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde415b8e0 con 0x7fcde41515e0 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde9088640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde4070a70 con 0x7fcde415b430 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde9088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4070d50 con 0x7fcde40cd7a0 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcde41515e0 0x7fcde40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcde41515e0 0x7fcde40cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39508/0 (socket says 172.21.15.72:39508) 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 -- 172.21.15.72:0/2048791125 learned_addr learned my addr 172.21.15.72:0/2048791125 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde259c640 1 --2- 172.21.15.72:0/2048791125 >> v2:172.21.15.146:3300/0 conn(0x7fcde415b430 0x7fcde4071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde359e640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde40cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 -- 172.21.15.72:0/2048791125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 msgr2=0x7fcde40cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde40cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 -- 172.21.15.72:0/2048791125 >> v2:172.21.15.146:3300/0 conn(0x7fcde415b430 msgr2=0x7fcde4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 --2- 172.21.15.72:0/2048791125 >> v2:172.21.15.146:3300/0 conn(0x7fcde415b430 0x7fcde4071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde2d9d640 1 -- 172.21.15.72:0/2048791125 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4071030 con 0x7fcde41515e0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde259c640 1 --2- 172.21.15.72:0/2048791125 >> v2:172.21.15.146:3300/0 conn(0x7fcde415b430 0x7fcde4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.526+0000 7fcde359e640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde40cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.527+0000 7fcde2d9d640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde40cce40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcde40cc110 tx=0x7fcdd402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3be3d72ce736866 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.527+0000 7fcde1d9b640 1 -- 172.21.15.72:0/2048791125 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd402ca70 con 0x7fcde41515e0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.527+0000 7fcde1d9b640 1 -- 172.21.15.72:0/2048791125 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd402cc10 con 0x7fcde41515e0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.527+0000 7fcde1d9b640 1 -- 172.21.15.72:0/2048791125 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd40343f0 con 0x7fcde41515e0 2024-03-06T07:23:56.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 -- 172.21.15.72:0/2048791125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 msgr2=0x7fcde40cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde40cce40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcde40cc110 tx=0x7fcdd402bed0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 -- 172.21.15.72:0/2048791125 shutdown_connections 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde40cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 --2- 172.21.15.72:0/2048791125 >> v2:172.21.15.146:3300/0 conn(0x7fcde415b430 0x7fcde4071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 --2- 172.21.15.72:0/2048791125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde40cce40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 -- 172.21.15.72:0/2048791125 >> 172.21.15.72:0/2048791125 conn(0x7fcde40cf8a0 msgr2=0x7fcde40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 -- 172.21.15.72:0/2048791125 shutdown_connections 2024-03-06T07:23:56.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.528+0000 7fcde9088640 1 -- 172.21.15.72:0/2048791125 wait complete. 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.529+0000 7fcde9088640 1 Processor -- start 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.529+0000 7fcde9088640 1 -- start start 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde9088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde414b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde9088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde41592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde9088640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde415b430 0x7fcde41f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde9088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4071d70 con 0x7fcde41515e0 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde9088640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde40ce0f0 con 0x7fcde415b430 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde9088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4071ef0 con 0x7fcde40cd7a0 2024-03-06T07:23:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde41592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde41592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39518/0 (socket says 172.21.15.72:39518) 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 -- 172.21.15.72:0/1679052542 learned_addr learned my addr 172.21.15.72:0/1679052542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde359e640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde415b430 0x7fcde41f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde2d9d640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde414b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 -- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 msgr2=0x7fcde414b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde414b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 -- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde415b430 msgr2=0x7fcde41f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde415b430 0x7fcde41f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.530+0000 7fcde259c640 1 -- 172.21.15.72:0/1679052542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde406be10 con 0x7fcde41515e0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.531+0000 7fcde259c640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde41592f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800c7f0 tx=0x7fcdd800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.531+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8015070 con 0x7fcde41515e0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.531+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd8007820 con 0x7fcde41515e0 2024-03-06T07:23:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.531+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde414bbc0 con 0x7fcde41515e0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.531+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd8007bd0 con 0x7fcde41515e0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.531+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde406b290 con 0x7fcde41515e0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.532+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde4068840 con 0x7fcde41515e0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.532+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdd8018070 con 0x7fcde41515e0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.532+0000 7fcdcb7fe640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcdd006b500 0x7fcdd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.533+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcdd8089dc0 con 0x7fcde41515e0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.533+0000 7fcde2d9d640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcdd006b500 0x7fcdd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.533+0000 7fcde2d9d640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcdd006b500 0x7fcdd006d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4035000 tx=0x7fcdd4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.537+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdd8058f90 con 0x7fcde41515e0 2024-03-06T07:23:56.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.774+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcde4159830 con 0x7fcde41515e0 2024-03-06T07:23:56.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.774+0000 7fcdcb7fe640 1 -- 172.21.15.72:0/1679052542 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcdd8058890 con 0x7fcde41515e0 2024-03-06T07:23:56.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:56.777 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:23:56.777 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.778 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:56.779 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.780 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:23:56.781 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.782 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.783 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.784 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.785 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.786 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.787 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.788 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.788 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.788 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.788 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:23:56.788 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:23:56.788 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.789 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.790 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.791 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:23:56.792 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:23:56.793 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:23:56.794 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:23:56.795 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:23:56.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcdd006b500 msgr2=0x7fcdd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcdd006b500 0x7fcdd006d9c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4035000 tx=0x7fcdd4030000 comp rx=0 tx=0).stop 2024-03-06T07:23:56.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 msgr2=0x7fcde41592f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde41592f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fcdd800c7f0 tx=0x7fcdd800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 shutdown_connections 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcdd006b500 0x7fcdd006d9c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcde415b430 0x7fcde41f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcde41515e0 0x7fcde41592f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 --2- 172.21.15.72:0/1679052542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcde40cd7a0 0x7fcde414b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 >> 172.21.15.72:0/1679052542 conn(0x7fcde40cf8a0 msgr2=0x7fcde415dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 shutdown_connections 2024-03-06T07:23:56.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.776+0000 7fcde9088640 1 -- 172.21.15.72:0/1679052542 wait complete. 2024-03-06T07:23:56.797 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:23:56.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.896+0000 7ff2fec81640 1 Processor -- start 2024-03-06T07:23:56.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.896+0000 7ff2fec81640 1 -- start start 2024-03-06T07:23:56.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fec81640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2f80d00c0 0x7ff2f80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fec81640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff2f806cf20 0x7ff2f806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fec81640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f80d09e0 con 0x7ff2f80cf850 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fec81640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f806ca90 con 0x7ff2f80d00c0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fec81640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f815e380 con 0x7ff2f806cf20 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fca1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2f80d00c0 0x7ff2f80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fca1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2f80d00c0 0x7ff2f80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39526/0 (socket says 172.21.15.72:39526) 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fca1d640 1 -- 172.21.15.72:0/708408252 learned_addr learned my addr 172.21.15.72:0/708408252 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.897+0000 7ff2fd21e640 1 --2- 172.21.15.72:0/708408252 >> v2:172.21.15.146:3300/0 conn(0x7ff2f806cf20 0x7ff2f806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2fca1d640 1 -- 172.21.15.72:0/708408252 >> v2:172.21.15.146:3300/0 conn(0x7ff2f806cf20 msgr2=0x7ff2f806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2fca1d640 1 --2- 172.21.15.72:0/708408252 >> v2:172.21.15.146:3300/0 conn(0x7ff2f806cf20 0x7ff2f806d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2fca1d640 1 -- 172.21.15.72:0/708408252 >> v1:172.21.15.146:6789/0 conn(0x7ff2f80cf850 legacy=0x7ff2f80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2fca1d640 1 -- 172.21.15.72:0/708408252 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80cdc10 con 0x7ff2f80d00c0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2fca1d640 1 --2- 172.21.15.72:0/708408252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80d00c0 0x7ff2f80d04a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80cfda0 tx=0x7ff2e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=279d3966e7d2f8b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2f77fe640 1 -- 172.21.15.72:0/708408252 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8035020 con 0x7ff2f80d00c0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2f77fe640 1 -- 172.21.15.72:0/708408252 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8031870 con 0x7ff2f80d00c0 2024-03-06T07:23:56.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.898+0000 7ff2f77fe640 1 -- 172.21.15.72:0/708408252 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8031c40 con 0x7ff2f80d00c0 2024-03-06T07:23:56.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 -- 172.21.15.72:0/708408252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80d00c0 msgr2=0x7ff2f80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 --2- 172.21.15.72:0/708408252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80d00c0 0x7ff2f80d04a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80cfda0 tx=0x7ff2e802b910 comp rx=0 tx=0).stop 2024-03-06T07:23:56.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 -- 172.21.15.72:0/708408252 shutdown_connections 2024-03-06T07:23:56.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 --2- 172.21.15.72:0/708408252 >> v2:172.21.15.146:3300/0 conn(0x7ff2f806cf20 0x7ff2f806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 --2- 172.21.15.72:0/708408252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f80d00c0 0x7ff2f80d04a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 -- 172.21.15.72:0/708408252 >> 172.21.15.72:0/708408252 conn(0x7ff2f80b35a0 msgr2=0x7ff2f80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 -- 172.21.15.72:0/708408252 shutdown_connections 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.899+0000 7ff2fec81640 1 -- 172.21.15.72:0/708408252 wait complete. 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.900+0000 7ff2fec81640 1 Processor -- start 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.900+0000 7ff2fec81640 1 -- start start 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fec81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 0x7ff2f80c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fec81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2f80cf850 0x7ff2f80b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fec81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2f80d00c0 0x7ff2f815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fec81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f806c720 con 0x7ff2f806cf20 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fec81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f806c440 con 0x7ff2f80d00c0 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fec81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f80b1030 con 0x7ff2f80cf850 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 0x7ff2f80c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 0x7ff2f80c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39528/0 (socket says 172.21.15.72:39528) 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 -- 172.21.15.72:0/2595970663 learned_addr learned my addr 172.21.15.72:0/2595970663 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:56.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 -- 172.21.15.72:0/2595970663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2f80cf850 msgr2=0x7ff2f80b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2f80cf850 0x7ff2f80b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 -- 172.21.15.72:0/2595970663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2f80d00c0 msgr2=0x7ff2f815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2f80d00c0 0x7ff2f815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.901+0000 7ff2fca1d640 1 -- 172.21.15.72:0/2595970663 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80c9490 con 0x7ff2f806cf20 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.902+0000 7ff2fca1d640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 0x7ff2f80c8b40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8000c00 tx=0x7ff2e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.902+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e80443f0 con 0x7ff2f806cf20 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.902+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e8044590 con 0x7ff2f806cf20 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.902+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f80c9770 con 0x7ff2f806cf20 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.902+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8044980 con 0x7ff2f806cf20 2024-03-06T07:23:56.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.902+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f80c9d30 con 0x7ff2f806cf20 2024-03-06T07:23:56.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.903+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f804f330 con 0x7ff2f806cf20 2024-03-06T07:23:56.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.903+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2e804b060 con 0x7ff2f806cf20 2024-03-06T07:23:56.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.903+0000 7ff2f57fa640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:56.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.904+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff2e80ac1a0 con 0x7ff2f806cf20 2024-03-06T07:23:56.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.904+0000 7ff2f7fff640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:56.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.904+0000 7ff2f7fff640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80b1ef0 tx=0x7ff2ec0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:56.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:56.908+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2e803f070 con 0x7ff2f806cf20 2024-03-06T07:23:57.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.081+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2f8070880 con 0x7ff2f806cf20 2024-03-06T07:23:57.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.081+0000 7ff2f57fa640 1 -- 172.21.15.72:0/2595970663 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+12903 (secure 0 0 0) 0x7ff2e807b370 con 0x7ff2f806cf20 2024-03-06T07:23:57.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:57.084 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":55,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:56.415132+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:23:57.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 msgr2=0x7ff2e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff2f80b1ef0 tx=0x7ff2ec0079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 msgr2=0x7ff2f80c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 0x7ff2f80c8b40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8000c00 tx=0x7ff2e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 shutdown_connections 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2f80d00c0 0x7ff2f815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2f80cf850 0x7ff2f80b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 --2- 172.21.15.72:0/2595970663 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2f806cf20 0x7ff2f80c8b40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 >> 172.21.15.72:0/2595970663 conn(0x7ff2f80b35a0 msgr2=0x7ff2f80b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 shutdown_connections 2024-03-06T07:23:57.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.083+0000 7ff2fec81640 1 -- 172.21.15.72:0/2595970663 wait complete. 2024-03-06T07:23:57.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:23:57.097 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:23:57.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.198+0000 7f30706ed640 1 Processor -- start 2024-03-06T07:23:57.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.198+0000 7f30706ed640 1 -- start start 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.199+0000 7f30706ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30680cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.199+0000 7f30706ed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30680712d0 0x7f3068071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.199+0000 7f30706ed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680b3320 con 0x7f30680cfa90 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.199+0000 7f30706ed640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680cf610 con 0x7f30680712d0 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.199+0000 7f30706ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3068071c70 con 0x7f306815a220 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306dc88640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30680712d0 0x7f3068071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306ec8a640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f30680cfa90 0x7f306815de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57148/0 (socket says 172.21.15.72:57148) 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306ec8a640 1 -- 172.21.15.72:0/3242968873 learned_addr learned my addr 172.21.15.72:0/3242968873 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:57.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306e489640 1 --2- 172.21.15.72:0/3242968873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30680cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306dc88640 1 -- 172.21.15.72:0/3242968873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 msgr2=0x7f30680cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306dc88640 1 --2- 172.21.15.72:0/3242968873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30680cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306dc88640 1 -- 172.21.15.72:0/3242968873 >> v1:172.21.15.72:6789/0 conn(0x7f30680cfa90 legacy=0x7f306815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306dc88640 1 -- 172.21.15.72:0/3242968873 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680728d0 con 0x7f30680712d0 2024-03-06T07:23:57.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.200+0000 7f306d487640 1 -- 172.21.15.72:0/3242968873 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021424194 0 0) 0x7f30680b3320 con 0x7f30680cfa90 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f306dc88640 1 --2- 172.21.15.72:0/3242968873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680712d0 0x7f3068071730 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f30640097a0 tx=0x7f306402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d529d9f20b071f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f306d487640 1 -- 172.21.15.72:0/3242968873 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30640315a0 con 0x7f30680712d0 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f306d487640 1 -- 172.21.15.72:0/3242968873 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064037070 con 0x7f30680712d0 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f306d487640 1 -- 172.21.15.72:0/3242968873 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064031960 con 0x7f30680712d0 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 -- 172.21.15.72:0/3242968873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680712d0 msgr2=0x7f3068071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 --2- 172.21.15.72:0/3242968873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680712d0 0x7f3068071730 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f30640097a0 tx=0x7f306402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 -- 172.21.15.72:0/3242968873 shutdown_connections 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 --2- 172.21.15.72:0/3242968873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680712d0 0x7f3068071730 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 --2- 172.21.15.72:0/3242968873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30680cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 -- 172.21.15.72:0/3242968873 >> 172.21.15.72:0/3242968873 conn(0x7f306806b190 msgr2=0x7f306806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 -- 172.21.15.72:0/3242968873 shutdown_connections 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.201+0000 7f30706ed640 1 -- 172.21.15.72:0/3242968873 wait complete. 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.202+0000 7f30706ed640 1 Processor -- start 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 -- start start 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680712d0 0x7f3068155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 0x7f306813f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30681f1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3068072030 con 0x7f30680cfa90 2024-03-06T07:23:57.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3068072bb0 con 0x7f30680712d0 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f30706ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3068072530 con 0x7f306815a220 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f306dc88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 0x7f306813f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f306dc88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 0x7f306813f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39554/0 (socket says 172.21.15.72:39554) 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.203+0000 7f306dc88640 1 -- 172.21.15.72:0/1900121073 learned_addr learned my addr 172.21.15.72:0/1900121073 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306ec8a640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30681f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306e489640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680712d0 0x7f3068155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306dc88640 1 -- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 msgr2=0x7f30681f1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306dc88640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30681f1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306dc88640 1 -- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680712d0 msgr2=0x7f3068155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306dc88640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680712d0 0x7f3068155350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306dc88640 1 -- 172.21.15.72:0/1900121073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068155890 con 0x7f30680cfa90 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f306dc88640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 0x7f306813f840 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3064034040 tx=0x7f30640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064032840 con 0x7f30680cfa90 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064032a00 con 0x7f30680cfa90 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3068152810 con 0x7f30680cfa90 2024-03-06T07:23:57.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064032df0 con 0x7f30680cfa90 2024-03-06T07:23:57.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.204+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3068154590 con 0x7f30680cfa90 2024-03-06T07:23:57.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.205+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30680732b0 con 0x7f30680cfa90 2024-03-06T07:23:57.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.205+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f306404e430 con 0x7f30680cfa90 2024-03-06T07:23:57.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.206+0000 7f305affd640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.206+0000 7f306e489640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.206+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f306402f030 con 0x7f30680cfa90 2024-03-06T07:23:57.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.206+0000 7f306e489640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f305c006fd0 tx=0x7f305c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.210+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3068155890 con 0x7f30680cfa90 2024-03-06T07:23:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.381+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3068068ff0 con 0x7f30680cfa90 2024-03-06T07:23:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.381+0000 7f305affd640 1 -- 172.21.15.72:0/1900121073 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3064040070 con 0x7f30680cfa90 2024-03-06T07:23:57.383 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:23:57.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 msgr2=0x7f304c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f305c006fd0 tx=0x7f305c008040 comp rx=0 tx=0).stop 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 msgr2=0x7f306813f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 0x7f306813f840 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3064034040 tx=0x7f30640046c0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 shutdown_connections 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815a220 0x7f30681f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680cfa90 0x7f306813f840 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 --2- 172.21.15.72:0/1900121073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680712d0 0x7f3068155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 >> 172.21.15.72:0/1900121073 conn(0x7f306806b190 msgr2=0x7f30680cbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 shutdown_connections 2024-03-06T07:23:57.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.383+0000 7f30706ed640 1 -- 172.21.15.72:0/1900121073 wait complete. 2024-03-06T07:23:57.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:23:57.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.495+0000 7f7ca6155640 1 Processor -- start 2024-03-06T07:23:57.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.495+0000 7f7ca6155640 1 -- start start 2024-03-06T07:23:57.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7ca6155640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7ca015bf30 0x7f7ca015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7ca6155640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca015f580 con 0x7f7ca00b3060 2024-03-06T07:23:57.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7ca6155640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca015b450 con 0x7f7ca013dc10 2024-03-06T07:23:57.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7ca6155640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca00cf3f0 con 0x7f7ca015bf30 2024-03-06T07:23:57.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7ca015bf30 0x7f7ca015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7ca00b3060 0x7f7ca015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57162/0 (socket says 172.21.15.72:57162) 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9effd640 1 -- 172.21.15.72:0/1516010525 learned_addr learned my addr 172.21.15.72:0/1516010525 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9e7fc640 1 -- 172.21.15.72:0/1516010525 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692272824 0 0) 0x7f7ca015f580 con 0x7f7ca00b3060 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9e7fc640 1 -- 172.21.15.72:0/1516010525 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c88003520 con 0x7f7ca00b3060 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9ffff640 1 -- 172.21.15.72:0/1516010525 >> v1:172.21.15.146:6789/0 conn(0x7f7ca013dc10 legacy=0x7f7ca00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9ffff640 1 -- 172.21.15.72:0/1516010525 >> v1:172.21.15.72:6789/0 conn(0x7f7ca00b3060 legacy=0x7f7ca015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.496+0000 7f7c9ffff640 1 -- 172.21.15.72:0/1516010525 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca015b730 con 0x7f7ca015bf30 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.497+0000 7f7c9ffff640 1 --2- 172.21.15.72:0/1516010525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca015bf30 0x7f7ca015ef60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009d90 tx=0x7f7c9802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc718b4f33598be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.497+0000 7f7c9e7fc640 1 -- 172.21.15.72:0/1516010525 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98039070 con 0x7f7ca015bf30 2024-03-06T07:23:57.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.497+0000 7f7c9e7fc640 1 -- 172.21.15.72:0/1516010525 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9802cbd0 con 0x7f7ca015bf30 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.497+0000 7f7c9e7fc640 1 -- 172.21.15.72:0/1516010525 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98034440 con 0x7f7ca015bf30 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.497+0000 7f7ca6155640 1 -- 172.21.15.72:0/1516010525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca015bf30 msgr2=0x7f7ca015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.497+0000 7f7ca6155640 1 --2- 172.21.15.72:0/1516010525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca015bf30 0x7f7ca015ef60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009d90 tx=0x7f7c9802be20 comp rx=0 tx=0).stop 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.498+0000 7f7ca6155640 1 -- 172.21.15.72:0/1516010525 shutdown_connections 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.498+0000 7f7ca6155640 1 --2- 172.21.15.72:0/1516010525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca015bf30 0x7f7ca015ef60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.498+0000 7f7ca6155640 1 -- 172.21.15.72:0/1516010525 >> 172.21.15.72:0/1516010525 conn(0x7f7ca00cf8a0 msgr2=0x7f7ca00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.498+0000 7f7ca6155640 1 -- 172.21.15.72:0/1516010525 shutdown_connections 2024-03-06T07:23:57.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.498+0000 7f7ca6155640 1 -- 172.21.15.72:0/1516010525 wait complete. 2024-03-06T07:23:57.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.499+0000 7f7ca6155640 1 Processor -- start 2024-03-06T07:23:57.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.499+0000 7f7ca6155640 1 -- start start 2024-03-06T07:23:57.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ca00b3060 0x7f7ca013c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 0x7f7ca015a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ca015bf30 0x7f7ca0153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca00718e0 con 0x7f7ca013dc10 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca015b210 con 0x7f7ca015bf30 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca0071a60 con 0x7f7ca00b3060 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 0x7f7ca015a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 0x7f7ca015a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39586/0 (socket says 172.21.15.72:39586) 2024-03-06T07:23:57.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 -- 172.21.15.72:0/645348348 learned_addr learned my addr 172.21.15.72:0/645348348 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9ffff640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ca015bf30 0x7f7ca0153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 -- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ca00b3060 msgr2=0x7f7ca013c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ca00b3060 0x7f7ca013c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 -- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ca015bf30 msgr2=0x7f7ca0153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ca015bf30 0x7f7ca0153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 -- 172.21.15.72:0/645348348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca0154140 con 0x7f7ca013dc10 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9ffff640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ca015bf30 0x7f7ca0153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:57.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c9effd640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 0x7f7ca015a730 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400c460 tx=0x7f7c9400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94017070 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94005ce0 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca0152b60 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94007290 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.500+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca0153120 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.501+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca004f330 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.502+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7c9401a070 con 0x7f7ca013dc10 2024-03-06T07:23:57.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.502+0000 7f7c87fff640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7c8c06b4d0 0x7f7c8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.502+0000 7f7c9f7fe640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7c8c06b4d0 0x7f7c8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.502+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c94089230 con 0x7f7ca013dc10 2024-03-06T07:23:57.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.503+0000 7f7c9f7fe640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7c8c06b4d0 0x7f7c8c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7c90003cc0 tx=0x7f7c900079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.507+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7c94058410 con 0x7f7ca013dc10 2024-03-06T07:23:57.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.679+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7ca015ad30 con 0x7f7ca013dc10 2024-03-06T07:23:57.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.679+0000 7f7c87fff640 1 -- 172.21.15.72:0/645348348 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v55) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7c94057d10 con 0x7f7ca013dc10 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:23:57.682 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:23:57.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7c8c06b4d0 msgr2=0x7f7c8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7c8c06b4d0 0x7f7c8c06d990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7c90003cc0 tx=0x7f7c900079b0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 msgr2=0x7f7ca015a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 0x7f7ca015a730 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7c9400c460 tx=0x7f7c9400c920 comp rx=0 tx=0).stop 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 shutdown_connections 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7c8c06b4d0 0x7f7c8c06d990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ca015bf30 0x7f7ca0153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ca013dc10 0x7f7ca015a730 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 --2- 172.21.15.72:0/645348348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ca00b3060 0x7f7ca013c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 >> 172.21.15.72:0/645348348 conn(0x7f7ca00cf8a0 msgr2=0x7f7ca013d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.681+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 shutdown_connections 2024-03-06T07:23:57.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.682+0000 7f7ca6155640 1 -- 172.21.15.72:0/645348348 wait complete. 2024-03-06T07:23:57.696 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:23:57.696 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:23:57.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.796+0000 7fc9071a0640 1 Processor -- start 2024-03-06T07:23:57.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc9071a0640 1 -- start start 2024-03-06T07:23:57.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc9071a0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9000708b0 0x7fc900070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc9071a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc9000ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc9071a0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90015afe0 con 0x7fc90015ac00 2024-03-06T07:23:57.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc9071a0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc900071250 con 0x7fc9000708b0 2024-03-06T07:23:57.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc9071a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000cdc10 con 0x7fc9000ce0a0 2024-03-06T07:23:57.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc8fffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9000708b0 0x7fc900070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc904f3c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc90015ac00 0x7fc90006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57176/0 (socket says 172.21.15.72:57176) 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.797+0000 7fc904f3c640 1 -- 172.21.15.72:0/1028379570 learned_addr learned my addr 172.21.15.72:0/1028379570 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc90573d640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc9000ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc8fffff640 1 -- 172.21.15.72:0/1028379570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 msgr2=0x7fc9000ce520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc8fffff640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc9000ce520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc8fffff640 1 -- 172.21.15.72:0/1028379570 >> v1:172.21.15.72:6789/0 conn(0x7fc90015ac00 legacy=0x7fc90006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc8ff7fe640 1 -- 172.21.15.72:0/1028379570 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306740060 0 0) 0x7fc90015afe0 con 0x7fc90015ac00 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc8fffff640 1 -- 172.21.15.72:0/1028379570 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000cd000 con 0x7fc9000708b0 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.798+0000 7fc90573d640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc9000ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc8fffff640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9000708b0 0x7fc900070d10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40097a0 tx=0x7fc8f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe65890b7e83fb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc8ff7fe640 1 -- 172.21.15.72:0/1028379570 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40315a0 con 0x7fc9000708b0 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc8ff7fe640 1 -- 172.21.15.72:0/1028379570 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4037070 con 0x7fc9000708b0 2024-03-06T07:23:57.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc8ff7fe640 1 -- 172.21.15.72:0/1028379570 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4031960 con 0x7fc9000708b0 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 -- 172.21.15.72:0/1028379570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9000708b0 msgr2=0x7fc900070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9000708b0 0x7fc900070d10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc8f40097a0 tx=0x7fc8f402b840 comp rx=0 tx=0).stop 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 -- 172.21.15.72:0/1028379570 shutdown_connections 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc9000ce520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 --2- 172.21.15.72:0/1028379570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9000708b0 0x7fc900070d10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 -- 172.21.15.72:0/1028379570 >> 172.21.15.72:0/1028379570 conn(0x7fc9000b1650 msgr2=0x7fc9000b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:57.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 -- 172.21.15.72:0/1028379570 shutdown_connections 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.799+0000 7fc9071a0640 1 -- 172.21.15.72:0/1028379570 wait complete. 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.800+0000 7fc9071a0640 1 Processor -- start 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.800+0000 7fc9071a0640 1 -- start start 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc9071a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000708b0 0x7fc90015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc9071a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc900151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc9071a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 0x7fc90014f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc9071a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000cd8a0 con 0x7fc90015ac00 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc9071a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000cd5c0 con 0x7fc9000708b0 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc9071a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000cf5d0 con 0x7fc9000ce0a0 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc90573d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 0x7fc90014f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc90573d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 0x7fc90014f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39608/0 (socket says 172.21.15.72:39608) 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc90573d640 1 -- 172.21.15.72:0/84616168 learned_addr learned my addr 172.21.15.72:0/84616168 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.801+0000 7fc904f3c640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000708b0 0x7fc90015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc8fffff640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc900151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc90573d640 1 -- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 msgr2=0x7fc900151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc90573d640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc900151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc90573d640 1 -- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000708b0 msgr2=0x7fc90015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc90573d640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000708b0 0x7fc90015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc90573d640 1 -- 172.21.15.72:0/84616168 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90015d6f0 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc904f3c640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000708b0 0x7fc90015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc90573d640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 0x7fc90014f5b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800cef0 tx=0x7fc8f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f801a070 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f800d860 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000ad890 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800dc50 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.802+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9000ade50 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.803+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9000687d0 con 0x7fc90015ac00 2024-03-06T07:23:57.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.804+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc8f801d070 con 0x7fc90015ac00 2024-03-06T07:23:57.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.804+0000 7fc8fd7fa640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:57.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.804+0000 7fc904f3c640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:57.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.804+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc8f808a6b0 con 0x7fc90015ac00 2024-03-06T07:23:57.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.805+0000 7fc904f3c640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0000f60 tx=0x7fc8f0007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:57.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.808+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc8f80599d0 con 0x7fc90015ac00 2024-03-06T07:23:57.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:57.979+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc90004f330 con 0x7fc90015ac00 2024-03-06T07:23:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.418+0000 7fc8fd7fa640 1 -- 172.21.15.72:0/84616168 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v56) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc8f80592d0 con 0x7fc90015ac00 2024-03-06T07:23:58.420 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc8ec06b4d0 msgr2=0x7fc8ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0000f60 tx=0x7fc8f0007ca0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 msgr2=0x7fc90014f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 0x7fc90014f5b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800cef0 tx=0x7fc8f800a710 comp rx=0 tx=0).stop 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 shutdown_connections 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc8ec06b4d0 0x7fc8ec06d990 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc90015ac00 0x7fc90014f5b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9000ce0a0 0x7fc900151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 --2- 172.21.15.72:0/84616168 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9000708b0 0x7fc90015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.420+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 >> 172.21.15.72:0/84616168 conn(0x7fc9000b1650 msgr2=0x7fc90014e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.421+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 shutdown_connections 2024-03-06T07:23:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.421+0000 7fc9071a0640 1 -- 172.21.15.72:0/84616168 wait complete. 2024-03-06T07:23:58.436 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:23:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.541+0000 7f0c11a2a640 1 Processor -- start 2024-03-06T07:23:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.541+0000 7f0c11a2a640 1 -- start start 2024-03-06T07:23:58.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.541+0000 7f0c11a2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c15bbb0 0x7f0c0c06afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.541+0000 7f0c11a2a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c0cea90 con 0x7f0c0c15ee10 2024-03-06T07:23:58.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.541+0000 7f0c11a2a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c161bc0 con 0x7f0c0c06b4f0 2024-03-06T07:23:58.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.541+0000 7f0c11a2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c161970 con 0x7f0c0c15bbb0 2024-03-06T07:23:58.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0c0c15ee10 0x7f0c0c071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57186/0 (socket says 172.21.15.72:57186) 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0b7fe640 1 -- 172.21.15.72:0/3753832520 learned_addr learned my addr 172.21.15.72:0/3753832520 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007878439 0 0) 0x7f0c04001490 con 0x7f0c0c15ee10 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf8003520 con 0x7f0c0c15ee10 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c15bbb0 0x7f0c0c06afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159434974 0 0) 0x7f0bf8003520 con 0x7f0c0c15ee10 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c15bbb0 msgr2=0x7f0c0c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c15bbb0 0x7f0c0c06afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 >> v1:172.21.15.146:6789/0 conn(0x7f0c0c06b4f0 legacy=0x7f0c0c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c161ea0 con 0x7f0c0c15ee10 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f0c04001bc0 con 0x7f0c0c15ee10 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 >> v1:172.21.15.72:6789/0 conn(0x7f0c0c15ee10 legacy=0x7f0c0c071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0bf8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf8004800 0x7f0bf8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf8005180 0x7f0bf800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c161970 con 0x7f0bf80039e0 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf800bf00 con 0x7f0bf8005180 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf800c0d0 con 0x7f0bf8004800 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0bf8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c03fff640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf8004800 0x7f0bf8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.542+0000 7f0c0b7fe640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf8005180 0x7f0bf800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0affd640 1 -- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf8004800 msgr2=0x7f0bf8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf8004800 0x7f0bf8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0affd640 1 -- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf8005180 msgr2=0x7f0bf800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf8005180 0x7f0bf800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0affd640 1 -- 172.21.15.72:0/3753832520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf8003520 con 0x7f0bf80039e0 2024-03-06T07:23:58.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0bf8003e00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00ba30 tx=0x7f0bf8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78d49a51b8a2649 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc036020 con 0x7f0bf80039e0 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc032640 con 0x7f0bf80039e0 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.543+0000 7f0c0a7fc640 1 -- 172.21.15.72:0/3753832520 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc00b600 con 0x7f0bf80039e0 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3753832520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 msgr2=0x7f0bf8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3753832520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0bf8003e00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00ba30 tx=0x7f0bf8005830 comp rx=0 tx=0).stop 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c0b7fe640 1 -- 172.21.15.72:0/3753832520 reap_dead start 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3753832520 shutdown_connections 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3753832520 >> 172.21.15.72:0/3753832520 conn(0x7f0c0c0b2660 msgr2=0x7f0c0c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3753832520 shutdown_connections 2024-03-06T07:23:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.544+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3753832520 wait complete. 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.545+0000 7f0c11a2a640 1 Processor -- start 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.545+0000 7f0c11a2a640 1 -- start start 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c11a2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0c0c14c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c11a2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0cb1f0 0x7f0c0c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c11a2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c15f9a0 0x7f0c0c1f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c11a2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c06afb0 con 0x7f0bf80039e0 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c11a2a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c1624f0 con 0x7f0c0c15f9a0 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c11a2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c06b130 con 0x7f0c0c0cb1f0 2024-03-06T07:23:58.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0c0c14c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0c0c14c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39624/0 (socket says 172.21.15.72:39624) 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 -- 172.21.15.72:0/3165342688 learned_addr learned my addr 172.21.15.72:0/3165342688 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 -- 172.21.15.72:0/3165342688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0cb1f0 msgr2=0x7f0c0c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0cb1f0 0x7f0c0c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 -- 172.21.15.72:0/3165342688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c15f9a0 msgr2=0x7f0c0c1f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c15f9a0 0x7f0c0c1f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 -- 172.21.15.72:0/3165342688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c15bd30 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c0affd640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0c0c14c570 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00bb20 tx=0x7f0bfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.546+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc041070 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.547+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc030a00 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.547+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c15bff0 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.547+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc036040 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.547+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c0cc5f0 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.548+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c0733b0 con 0x7f0bf80039e0 2024-03-06T07:23:58.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.548+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0bfc039020 con 0x7f0bf80039e0 2024-03-06T07:23:58.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.549+0000 7f0c08ff9640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bec06b4d0 0x7f0bec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:58.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.549+0000 7f0c03fff640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bec06b4d0 0x7f0bec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:58.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.549+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f0bfc0ac1a0 con 0x7f0bf80039e0 2024-03-06T07:23:58.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.549+0000 7f0c03fff640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bec06b4d0 0x7f0bec06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c156600 tx=0x7f0bf4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:58.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.553+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0bfc07b170 con 0x7f0bf80039e0 2024-03-06T07:23:58.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:58.725+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0c0c068ff0 con 0x7f0bf80039e0 2024-03-06T07:23:59.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.424+0000 7f0c08ff9640 1 -- 172.21.15.72:0/3165342688 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v57) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0bfc07aa70 con 0x7f0bf80039e0 2024-03-06T07:23:59.426 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:23:59.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.430+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bec06b4d0 msgr2=0x7f0bec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.430+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bec06b4d0 0x7f0bec06d990 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c156600 tx=0x7f0bf4009040 comp rx=0 tx=0).stop 2024-03-06T07:23:59.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.430+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 msgr2=0x7f0c0c14c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.430+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0c0c14c570 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00bb20 tx=0x7f0bfc004730 comp rx=0 tx=0).stop 2024-03-06T07:23:59.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.432+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 shutdown_connections 2024-03-06T07:23:59.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.432+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bec06b4d0 0x7f0bec06d990 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.432+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c15f9a0 0x7f0c0c1f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.432+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0cb1f0 0x7f0c0c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.432+0000 7f0c11a2a640 1 --2- 172.21.15.72:0/3165342688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf80039e0 0x7f0c0c14c570 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.432+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 >> 172.21.15.72:0/3165342688 conn(0x7f0c0c0b2660 msgr2=0x7f0c0c157090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.433+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 shutdown_connections 2024-03-06T07:23:59.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.435+0000 7f0c11a2a640 1 -- 172.21.15.72:0/3165342688 wait complete. 2024-03-06T07:23:59.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:23:59.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 Processor -- start 2024-03-06T07:23:59.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 -- start start 2024-03-06T07:23:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f735415e0b0 0x7f73540b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f73540b3df0 0x7f73540d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540d28c0 0x7f73540d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73540ce990 con 0x7f735415e0b0 2024-03-06T07:23:59.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73540b4290 con 0x7f73540b3df0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f735a4d4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540b4570 con 0x7f73540d28c0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.557+0000 7f7353fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f735415e0b0 0x7f73540b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f735415e0b0 0x7f73540b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39628/0 (socket says 172.21.15.72:39628) 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 -- 172.21.15.72:0/3126729337 learned_addr learned my addr 172.21.15.72:0/3126729337 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f73537fe640 1 --2- 172.21.15.72:0/3126729337 >> v2:172.21.15.146:3300/0 conn(0x7f73540b3df0 0x7f73540d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7358a71640 1 --2- 172.21.15.72:0/3126729337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540d28c0 0x7f73540d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 -- 172.21.15.72:0/3126729337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540d28c0 msgr2=0x7f73540d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 --2- 172.21.15.72:0/3126729337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540d28c0 0x7f73540d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 -- 172.21.15.72:0/3126729337 >> v2:172.21.15.146:3300/0 conn(0x7f73540b3df0 msgr2=0x7f73540d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 --2- 172.21.15.72:0/3126729337 >> v2:172.21.15.146:3300/0 conn(0x7f73540b3df0 0x7f73540d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 -- 172.21.15.72:0/3126729337 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540b4850 con 0x7f735415e0b0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.558+0000 7f7353fff640 1 --2- 172.21.15.72:0/3126729337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f73540b38b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f735408dbc0 tx=0x7f734402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c58173da55d8bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f7352ffd640 1 -- 172.21.15.72:0/3126729337 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344035020 con 0x7f735415e0b0 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f7352ffd640 1 -- 172.21.15.72:0/3126729337 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7344031870 con 0x7f735415e0b0 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f7352ffd640 1 -- 172.21.15.72:0/3126729337 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344031c40 con 0x7f735415e0b0 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 -- 172.21.15.72:0/3126729337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 msgr2=0x7f73540b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 --2- 172.21.15.72:0/3126729337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f73540b38b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f735408dbc0 tx=0x7f734402b910 comp rx=0 tx=0).stop 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 -- 172.21.15.72:0/3126729337 shutdown_connections 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 --2- 172.21.15.72:0/3126729337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540d28c0 0x7f73540d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 --2- 172.21.15.72:0/3126729337 >> v2:172.21.15.146:3300/0 conn(0x7f73540b3df0 0x7f73540d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 --2- 172.21.15.72:0/3126729337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f73540b38b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 -- 172.21.15.72:0/3126729337 >> 172.21.15.72:0/3126729337 conn(0x7f735408d6f0 msgr2=0x7f735409d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 -- 172.21.15.72:0/3126729337 shutdown_connections 2024-03-06T07:23:59.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.559+0000 7f735a4d4640 1 -- 172.21.15.72:0/3126729337 wait complete. 2024-03-06T07:23:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.560+0000 7f735a4d4640 1 Processor -- start 2024-03-06T07:23:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 -- start start 2024-03-06T07:23:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540b3df0 0x7f73540cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73540d28c0 0x7f735415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f735415f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73540d1cd0 con 0x7f735415e0b0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735415e490 con 0x7f73540d28c0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f735a4d4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540d1e50 con 0x7f73540b3df0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f7358a71640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f735415f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f7358a71640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f735415f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39632/0 (socket says 172.21.15.72:39632) 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f7358a71640 1 -- 172.21.15.72:0/1121143495 learned_addr learned my addr 172.21.15.72:0/1121143495 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f73537fe640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73540d28c0 0x7f735415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.561+0000 7f7353fff640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540b3df0 0x7f73540cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7358a71640 1 -- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540b3df0 msgr2=0x7f73540cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7358a71640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540b3df0 0x7f73540cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7358a71640 1 -- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73540d28c0 msgr2=0x7f735415a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7358a71640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73540d28c0 0x7f735415a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7358a71640 1 -- 172.21.15.72:0/1121143495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735415f990 con 0x7f735415e0b0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7358a71640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f735415f450 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f734c007e40 tx=0x7f734c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c014620 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c019070 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73540cc740 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c014a20 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.562+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73540ccd00 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.563+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735404f330 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.564+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f734c014bc0 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.564+0000 7f7350ff9640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f734006b500 0x7f734006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.564+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f734c08a8c0 con 0x7f735415e0b0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.564+0000 7f7353fff640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f734006b500 0x7f734006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:23:59.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.564+0000 7f7353fff640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f734006b500 0x7f734006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7344036000 tx=0x7f734403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:23:59.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.568+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f734c059150 con 0x7f735415e0b0 2024-03-06T07:23:59.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:23:59.776+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f73540d2160 con 0x7f735415e0b0 2024-03-06T07:24:00.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.440+0000 7f7350ff9640 1 -- 172.21.15.72:0/1121143495 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v25) v1 ==== 172+0+0 (secure 0 0 0) 0x7f734c05d1f0 con 0x7f735415e0b0 2024-03-06T07:24:00.443 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 8 and data pool 9 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f734006b500 msgr2=0x7f734006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f734006b500 0x7f734006d9c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7344036000 tx=0x7f734403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 msgr2=0x7f735415f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f735415f450 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f734c007e40 tx=0x7f734c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 shutdown_connections 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f734006b500 0x7f734006d9c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f735415e0b0 0x7f735415f450 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73540d28c0 0x7f735415a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 --2- 172.21.15.72:0/1121143495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73540b3df0 0x7f73540cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.452+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 >> 172.21.15.72:0/1121143495 conn(0x7f735408d6f0 msgr2=0x7f735406ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.453+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 shutdown_connections 2024-03-06T07:24:00.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.453+0000 7f735a4d4640 1 -- 172.21.15.72:0/1121143495 wait complete. 2024-03-06T07:24:00.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:00.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.568+0000 7f7af5aed640 1 Processor -- start 2024-03-06T07:24:00.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.568+0000 7f7af5aed640 1 -- start start 2024-03-06T07:24:00.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7af5aed640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7af006c6d0 0x7f7af015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7af5aed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7af0066d90 0x7f7af00ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7af5aed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af00cd450 con 0x7f7af00b4f70 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7af5aed640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af015e9f0 con 0x7f7af0066d90 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7af5aed640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af00cda10 con 0x7f7af006c6d0 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aeffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7af0066d90 0x7f7af00ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aef7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7af00b4f70 0x7f7af00b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57202/0 (socket says 172.21.15.72:57202) 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aef7fe640 1 -- 172.21.15.72:0/2078500838 learned_addr learned my addr 172.21.15.72:0/2078500838 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aeffff640 1 -- 172.21.15.72:0/2078500838 >> v2:172.21.15.146:3300/0 conn(0x7f7af006c6d0 msgr2=0x7f7af015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aeffff640 1 --2- 172.21.15.72:0/2078500838 >> v2:172.21.15.146:3300/0 conn(0x7f7af006c6d0 0x7f7af015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aeffff640 1 -- 172.21.15.72:0/2078500838 >> v1:172.21.15.72:6789/0 conn(0x7f7af00b4f70 legacy=0x7f7af00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aeffff640 1 -- 172.21.15.72:0/2078500838 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00cd730 con 0x7f7af0066d90 2024-03-06T07:24:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.569+0000 7f7aee7fc640 1 -- 172.21.15.72:0/2078500838 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594254665 0 0) 0x7f7af00cd450 con 0x7f7af00b4f70 2024-03-06T07:24:00.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.570+0000 7f7aeffff640 1 --2- 172.21.15.72:0/2078500838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af0066d90 0x7f7af00ccb50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ccb00 tx=0x7f7ae802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf71ff132c7d87b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.570+0000 7f7aee7fc640 1 -- 172.21.15.72:0/2078500838 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8039070 con 0x7f7af0066d90 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.570+0000 7f7aee7fc640 1 -- 172.21.15.72:0/2078500838 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae802cc80 con 0x7f7af0066d90 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.570+0000 7f7aee7fc640 1 -- 172.21.15.72:0/2078500838 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8034660 con 0x7f7af0066d90 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 -- 172.21.15.72:0/2078500838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af0066d90 msgr2=0x7f7af00ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 --2- 172.21.15.72:0/2078500838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af0066d90 0x7f7af00ccb50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ccb00 tx=0x7f7ae802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 -- 172.21.15.72:0/2078500838 shutdown_connections 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 --2- 172.21.15.72:0/2078500838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af0066d90 0x7f7af00ccb50 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 --2- 172.21.15.72:0/2078500838 >> v2:172.21.15.146:3300/0 conn(0x7f7af006c6d0 0x7f7af015ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 -- 172.21.15.72:0/2078500838 >> 172.21.15.72:0/2078500838 conn(0x7f7af00724d0 msgr2=0x7f7af00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 -- 172.21.15.72:0/2078500838 shutdown_connections 2024-03-06T07:24:00.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.571+0000 7f7af5aed640 1 -- 172.21.15.72:0/2078500838 wait complete. 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.572+0000 7f7af5aed640 1 Processor -- start 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.572+0000 7f7af5aed640 1 -- start start 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7af5aed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7af0066d90 0x7f7af015b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7af5aed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 0x7f7af015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7af5aed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7af00c9490 0x7f7af014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7af5aed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00cd090 con 0x7f7af00b4f70 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7af5aed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af015f3c0 con 0x7f7af0066d90 2024-03-06T07:24:00.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7af5aed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af00cd210 con 0x7f7af00c9490 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 0x7f7af015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 0x7f7af015d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39650/0 (socket says 172.21.15.72:39650) 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 -- 172.21.15.72:0/1816781108 learned_addr learned my addr 172.21.15.72:0/1816781108 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeffff640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7af00c9490 0x7f7af014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aef7fe640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7af0066d90 0x7f7af015b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 -- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7af00c9490 msgr2=0x7f7af014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7af00c9490 0x7f7af014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 -- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7af0066d90 msgr2=0x7f7af015b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7af0066d90 0x7f7af015b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 -- 172.21.15.72:0/1816781108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00c9a00 con 0x7f7af00b4f70 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeffff640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7af00c9490 0x7f7af014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:00.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aef7fe640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7af0066d90 0x7f7af015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.573+0000 7f7aeeffd640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 0x7f7af015d200 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400b4a0 tx=0x7f7ae400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.574+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4014070 con 0x7f7af00b4f70 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.574+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae4010610 con 0x7f7af00b4f70 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.574+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af00c9ce0 con 0x7f7af00b4f70 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.574+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4010a00 con 0x7f7af00b4f70 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.574+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af00ca2a0 con 0x7f7af00b4f70 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.574+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af004f330 con 0x7f7af00b4f70 2024-03-06T07:24:00.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.575+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ae4017070 con 0x7f7af00b4f70 2024-03-06T07:24:00.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.575+0000 7f7ad7fff640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7adc06b4d0 0x7f7adc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.575+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ae4089c40 con 0x7f7af00b4f70 2024-03-06T07:24:00.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.575+0000 7f7aef7fe640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7adc06b4d0 0x7f7adc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.576+0000 7f7aef7fe640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7adc06b4d0 0x7f7adc06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000f60 tx=0x7f7ae0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.580+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ae4058ac0 con 0x7f7af00b4f70 2024-03-06T07:24:00.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.751+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7af0074ab0 con 0x7f7af00b4f70 2024-03-06T07:24:00.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.752+0000 7f7ad7fff640 1 -- 172.21.15.72:0/1816781108 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f7ae40583c0 con 0x7f7af00b4f70 2024-03-06T07:24:00.754 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:00.755 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":58,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:59.778099+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:00.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7adc06b4d0 msgr2=0x7f7adc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7adc06b4d0 0x7f7adc06d990 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0000f60 tx=0x7f7ae0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 msgr2=0x7f7af015d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 0x7f7af015d200 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400b4a0 tx=0x7f7ae400b960 comp rx=0 tx=0).stop 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 shutdown_connections 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7adc06b4d0 0x7f7adc06d990 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7af00c9490 0x7f7af014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7af00b4f70 0x7f7af015d200 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 --2- 172.21.15.72:0/1816781108 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7af0066d90 0x7f7af015b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 >> 172.21.15.72:0/1816781108 conn(0x7f7af00724d0 msgr2=0x7f7af00b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 shutdown_connections 2024-03-06T07:24:00.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.754+0000 7f7af5aed640 1 -- 172.21.15.72:0/1816781108 wait complete. 2024-03-06T07:24:00.768 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:00.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.866+0000 7f836c281640 1 Processor -- start 2024-03-06T07:24:00.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.866+0000 7f836c281640 1 -- start start 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836c281640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f836406da40 0x7f83640cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836c281640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f836415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836c281640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83640cc980 con 0x7f83641424f0 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836c281640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83640cd220 con 0x7f836406da40 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836c281640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83640ccc60 con 0x7f83640cd7a0 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836a01d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f83641424f0 0x7f836406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57216/0 (socket says 172.21.15.72:57216) 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836a01d640 1 -- 172.21.15.72:0/1899430798 learned_addr learned my addr 172.21.15.72:0/1899430798 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836a81e640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f836415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836981c640 1 --2- 172.21.15.72:0/1899430798 >> v2:172.21.15.146:3300/0 conn(0x7f836406da40 0x7f83640cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003844087 0 0) 0x7f83640cc980 con 0x7f83641424f0 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8350003520 con 0x7f83641424f0 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007724328 0 0) 0x7f8350003520 con 0x7f83641424f0 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 msgr2=0x7f836415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f836415cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 >> v2:172.21.15.146:3300/0 conn(0x7f836406da40 msgr2=0x7f83640cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 --2- 172.21.15.72:0/1899430798 >> v2:172.21.15.146:3300/0 conn(0x7f836406da40 0x7f83640cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83640ccf40 con 0x7f83641424f0 2024-03-06T07:24:00.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f8360001670 con 0x7f83641424f0 2024-03-06T07:24:00.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 >> v1:172.21.15.72:6789/0 conn(0x7f83641424f0 legacy=0x7f836406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8350003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f8350004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8350004d40 0x7f83500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83640cd220 con 0x7f83500039b0 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83640ccc60 con 0x7f8350004d40 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835000b9c0 con 0x7f83640cd7a0 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.867+0000 7f836a01d640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8350003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836981c640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f8350004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a81e640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8350004d40 0x7f83500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a01d640 1 -- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 msgr2=0x7f8350004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a01d640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f8350004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a01d640 1 -- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8350004d40 msgr2=0x7f83500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a01d640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8350004d40 0x7f83500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a01d640 1 -- 172.21.15.72:0/1899430798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8350003520 con 0x7f83500039b0 2024-03-06T07:24:00.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a81e640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8350004d40 0x7f83500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836981c640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640cd7a0 0x7f8350004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.868+0000 7f836a01d640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8350003dd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8360001050 tx=0x7f8350005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d654645b23c2ffb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360007d70 con 0x7f83500039b0 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8360030070 con 0x7f83500039b0 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836901b640 1 -- 172.21.15.72:0/1899430798 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836002cd80 con 0x7f83500039b0 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836c281640 1 -- 172.21.15.72:0/1899430798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 msgr2=0x7f8350003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836c281640 1 --2- 172.21.15.72:0/1899430798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8350003dd0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8360001050 tx=0x7f8350005830 comp rx=0 tx=0).stop 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836a81e640 1 -- 172.21.15.72:0/1899430798 reap_dead start 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836c281640 1 -- 172.21.15.72:0/1899430798 shutdown_connections 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836c281640 1 -- 172.21.15.72:0/1899430798 >> 172.21.15.72:0/1899430798 conn(0x7f83640cf8a0 msgr2=0x7f83640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:00.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836c281640 1 -- 172.21.15.72:0/1899430798 shutdown_connections 2024-03-06T07:24:00.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.869+0000 7f836c281640 1 -- 172.21.15.72:0/1899430798 wait complete. 2024-03-06T07:24:00.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.870+0000 7f836c281640 1 Processor -- start 2024-03-06T07:24:00.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.870+0000 7f836c281640 1 -- start start 2024-03-06T07:24:00.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836c281640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8364157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836c281640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f836415e1e0 0x7f83640b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836c281640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640b0950 0x7f83640b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836c281640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836415d070 con 0x7f83500039b0 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836c281640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836415cef0 con 0x7f836415e1e0 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836c281640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836415d1f0 con 0x7f83640b0950 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8364157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8364157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39672/0 (socket says 172.21.15.72:39672) 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 -- 172.21.15.72:0/1237653121 learned_addr learned my addr 172.21.15.72:0/1237653121 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a81e640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640b0950 0x7f83640b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836981c640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f836415e1e0 0x7f83640b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 -- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640b0950 msgr2=0x7f83640b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640b0950 0x7f83640b0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 -- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f836415e1e0 msgr2=0x7f83640b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.871+0000 7f836a01d640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f836415e1e0 0x7f83640b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:00.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f836a01d640 1 -- 172.21.15.72:0/1237653121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836415f4d0 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f836a01d640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8364157460 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f836000c300 tx=0x7f8360000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360004bf0 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8360004d90 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83640b4120 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360007270 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.872+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836415ec30 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.873+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836404f330 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.873+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8360007410 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.874+0000 7f835affd640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f834806b4d0 0x7f834806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.874+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83600ab380 con 0x7f83500039b0 2024-03-06T07:24:00.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.874+0000 7f836981c640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f834806b4d0 0x7f834806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:00.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.874+0000 7f836981c640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f834806b4d0 0x7f834806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f83640b4c30 tx=0x7f8354008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:00.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:00.878+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f836003e300 con 0x7f83500039b0 2024-03-06T07:24:01.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.049+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f836415bed0 con 0x7f83500039b0 2024-03-06T07:24:01.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.050+0000 7f835affd640 1 -- 172.21.15.72:0/1237653121 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16690 (secure 0 0 0) 0x7f836007a200 con 0x7f83500039b0 2024-03-06T07:24:01.053 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:01.053 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":58,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:23:59.778099+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f834806b4d0 msgr2=0x7f834806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f834806b4d0 0x7f834806d990 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f83640b4c30 tx=0x7f8354008040 comp rx=0 tx=0).stop 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 msgr2=0x7f8364157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8364157460 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f836000c300 tx=0x7f8360000fe0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 shutdown_connections 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f834806b4d0 0x7f834806d990 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83640b0950 0x7f83640b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f836415e1e0 0x7f83640b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 --2- 172.21.15.72:0/1237653121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500039b0 0x7f8364157460 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 >> 172.21.15.72:0/1237653121 conn(0x7f83640cf8a0 msgr2=0x7f8364150a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 shutdown_connections 2024-03-06T07:24:01.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.052+0000 7f836c281640 1 -- 172.21.15.72:0/1237653121 wait complete. 2024-03-06T07:24:01.066 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:01.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.166+0000 7f082f956640 1 Processor -- start 2024-03-06T07:24:01.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.166+0000 7f082f956640 1 -- start start 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082f956640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f082815c050 0x7f082806f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082f956640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f082806fb40 0x7f08280cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082f956640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08280cd4b0 con 0x7f08280ccca0 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082f956640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082806ffb0 con 0x7f082806fb40 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082f956640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08280cd790 con 0x7f082815c050 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082cef1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f082806fb40 0x7f08280cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082def3640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f08280ccca0 0x7f08280cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57232/0 (socket says 172.21.15.72:57232) 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.167+0000 7f082def3640 1 -- 172.21.15.72:0/2906182998 learned_addr learned my addr 172.21.15.72:0/2906182998 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f082d6f2640 1 --2- 172.21.15.72:0/2906182998 >> v2:172.21.15.146:3300/0 conn(0x7f082815c050 0x7f082806f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f082cef1640 1 -- 172.21.15.72:0/2906182998 >> v2:172.21.15.146:3300/0 conn(0x7f082815c050 msgr2=0x7f082806f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f082cef1640 1 --2- 172.21.15.72:0/2906182998 >> v2:172.21.15.146:3300/0 conn(0x7f082815c050 0x7f082806f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f082cef1640 1 -- 172.21.15.72:0/2906182998 >> v1:172.21.15.72:6789/0 conn(0x7f08280ccca0 legacy=0x7f08280cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f082cef1640 1 -- 172.21.15.72:0/2906182998 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280cda70 con 0x7f082806fb40 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f082cef1640 1 --2- 172.21.15.72:0/2906182998 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08280cc760 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0828070ef0 tx=0x7f081802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52db0eab8bd55ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.168+0000 7f081ffff640 1 -- 172.21.15.72:0/2906182998 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08180315a0 con 0x7f082806fb40 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f081ffff640 1 -- 172.21.15.72:0/2906182998 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0818037070 con 0x7f082806fb40 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f081ffff640 1 -- 172.21.15.72:0/2906182998 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818031960 con 0x7f082806fb40 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 -- 172.21.15.72:0/2906182998 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 msgr2=0x7f08280cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 --2- 172.21.15.72:0/2906182998 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08280cc760 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f0828070ef0 tx=0x7f081802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 -- 172.21.15.72:0/2906182998 shutdown_connections 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 --2- 172.21.15.72:0/2906182998 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08280cc760 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 --2- 172.21.15.72:0/2906182998 >> v2:172.21.15.146:3300/0 conn(0x7f082815c050 0x7f082806f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 -- 172.21.15.72:0/2906182998 >> 172.21.15.72:0/2906182998 conn(0x7f08280b3800 msgr2=0x7f08280b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 -- 172.21.15.72:0/2906182998 shutdown_connections 2024-03-06T07:24:01.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.169+0000 7f082f956640 1 -- 172.21.15.72:0/2906182998 wait complete. 2024-03-06T07:24:01.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.170+0000 7f082f956640 1 Processor -- start 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.170+0000 7f082f956640 1 -- start start 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082f956640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08281f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082f956640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08280ccca0 0x7f0828156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082f956640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082815c050 0x7f082814aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082f956640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08280d01f0 con 0x7f082806fb40 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082f956640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08280cd130 con 0x7f08280ccca0 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082f956640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08280d0370 con 0x7f082815c050 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08281f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08281f3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39694/0 (socket says 172.21.15.72:39694) 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 -- 172.21.15.72:0/2684051901 learned_addr learned my addr 172.21.15.72:0/2684051901 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 -- 172.21.15.72:0/2684051901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082815c050 msgr2=0x7f082814aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082815c050 0x7f082814aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 -- 172.21.15.72:0/2684051901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08280ccca0 msgr2=0x7f0828156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08280ccca0 0x7f0828156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.171+0000 7f082d6f2640 1 -- 172.21.15.72:0/2684051901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0828156fb0 con 0x7f082806fb40 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.172+0000 7f082d6f2640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08281f3700 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0824007ef0 tx=0x7f082400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.172+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0824014620 con 0x7f082806fb40 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.172+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0824019070 con 0x7f082806fb40 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.172+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0828157290 con 0x7f082806fb40 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.172+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08240149e0 con 0x7f082806fb40 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.172+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082814a520 con 0x7f082806fb40 2024-03-06T07:24:01.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.173+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08281577b0 con 0x7f082806fb40 2024-03-06T07:24:01.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.173+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0824014b80 con 0x7f082806fb40 2024-03-06T07:24:01.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.174+0000 7f081dffb640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b4d0 0x7f080c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.174+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f082408aba0 con 0x7f082806fb40 2024-03-06T07:24:01.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.174+0000 7f082cef1640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b4d0 0x7f080c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.174+0000 7f082cef1640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b4d0 0x7f080c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0818036000 tx=0x7f0818030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.178+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08281577b0 con 0x7f082806fb40 2024-03-06T07:24:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.385+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f082809fe60 con 0x7f082806fb40 2024-03-06T07:24:01.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.438+0000 7f081dffb640 1 -- 172.21.15.72:0/2684051901 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v26) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0824059a20 con 0x7f082806fb40 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b4d0 msgr2=0x7f080c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b4d0 0x7f080c06d990 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0818036000 tx=0x7f0818030000 comp rx=0 tx=0).stop 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 msgr2=0x7f08281f3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08281f3700 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0824007ef0 tx=0x7f082400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 shutdown_connections 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b4d0 0x7f080c06d990 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082815c050 0x7f082814aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08280ccca0 0x7f0828156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.440+0000 7f082f956640 1 --2- 172.21.15.72:0/2684051901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f082806fb40 0x7f08281f3700 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.441+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 >> 172.21.15.72:0/2684051901 conn(0x7f08280b3800 msgr2=0x7f08280b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.441+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 shutdown_connections 2024-03-06T07:24:01.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.441+0000 7f082f956640 1 -- 172.21.15.72:0/2684051901 wait complete. 2024-03-06T07:24:01.455 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:01.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:01.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.557+0000 7ff404f1e640 1 Processor -- start 2024-03-06T07:24:01.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.557+0000 7ff404f1e640 1 -- start start 2024-03-06T07:24:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.557+0000 7ff404f1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 0x7ff40006f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.557+0000 7ff404f1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff40006fb40 0x7ff4000cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff404f1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4000cd4b0 con 0x7ff4000ccca0 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff404f1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40006ff80 con 0x7ff40006fb40 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff404f1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000cd790 con 0x7ff40015c050 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff4000ccca0 0x7ff4000cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57240/0 (socket says 172.21.15.72:57240) 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fed9d640 1 -- 172.21.15.72:0/1444532305 learned_addr learned my addr 172.21.15.72:0/1444532305 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2096710837 0 0) 0x7ff4000cd4b0 con 0x7ff4000ccca0 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fe59c640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 0x7ff40006f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3f7fff640 1 --2- 172.21.15.72:0/1444532305 >> v2:172.21.15.146:3300/0 conn(0x7ff40006fb40 0x7ff4000cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec003520 con 0x7ff4000ccca0 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1190639543 0 0) 0x7ff3ec003520 con 0x7ff4000ccca0 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 msgr2=0x7ff40006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 0x7ff40006f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 >> v2:172.21.15.146:3300/0 conn(0x7ff40006fb40 msgr2=0x7ff4000cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 --2- 172.21.15.72:0/1444532305 >> v2:172.21.15.146:3300/0 conn(0x7ff40006fb40 0x7ff4000cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000cda70 con 0x7ff4000ccca0 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff3f80019f0 con 0x7ff4000ccca0 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 >> v1:172.21.15.72:6789/0 conn(0x7ff4000ccca0 legacy=0x7ff4000cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ec0039b0 0x7ff3ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 0x7ff3ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ec004d40 0x7ff3ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40006ff80 con 0x7ff3ec004d40 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000cd790 con 0x7ff3ec0039b0 2024-03-06T07:24:01.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec00b9c0 con 0x7ff40015c050 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ec004d40 0x7ff3ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3fe59c640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ec0039b0 0x7ff3ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.558+0000 7ff3f7fff640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 0x7ff3ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.559+0000 7ff3fed9d640 1 -- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 msgr2=0x7ff3ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.559+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff40015c050 0x7ff3ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.559+0000 7ff3fed9d640 1 -- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ec0039b0 msgr2=0x7ff3ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.559+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3ec0039b0 0x7ff3ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.559+0000 7ff3fed9d640 1 -- 172.21.15.72:0/1444532305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ec003520 con 0x7ff3ec004d40 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.559+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ec004d40 0x7ff3ec0051c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80013d0 tx=0x7ff3ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a53c5ca7aa2fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800a4e0 con 0x7ff3ec004d40 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f8034650 con 0x7ff3ec004d40 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff3fdd9b640 1 -- 172.21.15.72:0/1444532305 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8034a40 con 0x7ff3ec004d40 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff404f1e640 1 -- 172.21.15.72:0/1444532305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ec004d40 msgr2=0x7ff3ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff404f1e640 1 --2- 172.21.15.72:0/1444532305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3ec004d40 0x7ff3ec0051c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff3f80013d0 tx=0x7ff3ec005830 comp rx=0 tx=0).stop 2024-03-06T07:24:01.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff3fed9d640 1 -- 172.21.15.72:0/1444532305 reap_dead start 2024-03-06T07:24:01.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff404f1e640 1 -- 172.21.15.72:0/1444532305 shutdown_connections 2024-03-06T07:24:01.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff404f1e640 1 -- 172.21.15.72:0/1444532305 >> 172.21.15.72:0/1444532305 conn(0x7ff4000b3800 msgr2=0x7ff4000b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:01.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff404f1e640 1 -- 172.21.15.72:0/1444532305 shutdown_connections 2024-03-06T07:24:01.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.560+0000 7ff404f1e640 1 -- 172.21.15.72:0/1444532305 wait complete. 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.561+0000 7ff404f1e640 1 Processor -- start 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 -- start start 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4001534c0 0x7ff40009f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4001538a0 0x7ff400157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 0x7ff400155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40006f600 con 0x7ff400158140 2024-03-06T07:24:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4000cd130 con 0x7ff4001538a0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff404f1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40006f780 con 0x7ff4001534c0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff3fed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 0x7ff400155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff3fed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 0x7ff400155bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39726/0 (socket says 172.21.15.72:39726) 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.562+0000 7ff3fed9d640 1 -- 172.21.15.72:0/396437911 learned_addr learned my addr 172.21.15.72:0/396437911 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3f7fff640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4001538a0 0x7ff400157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fe59c640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4001534c0 0x7ff40009f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fed9d640 1 -- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4001534c0 msgr2=0x7ff40009f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4001534c0 0x7ff40009f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fed9d640 1 -- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4001538a0 msgr2=0x7ff400157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4001538a0 0x7ff400157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fed9d640 1 -- 172.21.15.72:0/396437911 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000b5300 con 0x7ff400158140 2024-03-06T07:24:01.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3fed9d640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 0x7ff400155bb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800a0b0 tx=0x7ff3f8001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8039040 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f8045070 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4000b55e0 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8002040 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.563+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4000b5ba0 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.564+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff40009fe60 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.565+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff3f8034a00 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.565+0000 7ff3f6ffd640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff3e006b4d0 0x7ff3e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.565+0000 7ff3fe59c640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff3e006b4d0 0x7ff3e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.565+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3f80ac660 con 0x7ff400158140 2024-03-06T07:24:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.565+0000 7ff3fe59c640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff3e006b4d0 0x7ff3e006d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009980 tx=0x7ff3f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.569+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff3f807b570 con 0x7ff400158140 2024-03-06T07:24:01.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:01.797+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff400068ff0 con 0x7ff400158140 2024-03-06T07:24:02.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.451+0000 7ff3f6ffd640 1 -- 172.21.15.72:0/396437911 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff3f80019b0 con 0x7ff400158140 2024-03-06T07:24:02.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff3e006b4d0 msgr2=0x7ff3e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff3e006b4d0 0x7ff3e006d990 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009980 tx=0x7ff3f0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 msgr2=0x7ff400155bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 0x7ff400155bb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800a0b0 tx=0x7ff3f8001360 comp rx=0 tx=0).stop 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 shutdown_connections 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff3e006b4d0 0x7ff3e006d990 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff400158140 0x7ff400155bb0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4001538a0 0x7ff400157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 --2- 172.21.15.72:0/396437911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4001534c0 0x7ff40009f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 >> 172.21.15.72:0/396437911 conn(0x7ff4000b3800 msgr2=0x7ff4000b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 shutdown_connections 2024-03-06T07:24:02.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.453+0000 7ff404f1e640 1 -- 172.21.15.72:0/396437911 wait complete. 2024-03-06T07:24:02.466 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:02.467 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:02.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.566+0000 7f3469925640 1 Processor -- start 2024-03-06T07:24:02.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.566+0000 7f3469925640 1 -- start start 2024-03-06T07:24:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3469925640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f346415dfe0 0x7f34640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3469925640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f34640cf830 0x7f34640d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3469925640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34640d27f0 0x7f34640d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3469925640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640b31e0 con 0x7f34640cf830 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3469925640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640b3d60 con 0x7f346415dfe0 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3469925640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640b3430 con 0x7f34640d27f0 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f34640cf830 0x7f34640d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f34640cf830 0x7f34640d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33834/0 (socket says 172.21.15.72:33834) 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 -- 172.21.15.72:0/1789539325 learned_addr learned my addr 172.21.15.72:0/1789539325 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3462ffd640 1 --2- 172.21.15.72:0/1789539325 >> v2:172.21.15.146:3300/0 conn(0x7f346415dfe0 0x7f34640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34637fe640 1 --2- 172.21.15.72:0/1789539325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34640d27f0 0x7f34640d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 -- 172.21.15.72:0/1789539325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34640d27f0 msgr2=0x7f34640d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 --2- 172.21.15.72:0/1789539325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34640d27f0 0x7f34640d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 -- 172.21.15.72:0/1789539325 >> v2:172.21.15.146:3300/0 conn(0x7f346415dfe0 msgr2=0x7f34640cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 --2- 172.21.15.72:0/1789539325 >> v2:172.21.15.146:3300/0 conn(0x7f346415dfe0 0x7f34640cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f34627fc640 1 -- 172.21.15.72:0/1789539325 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640b3710 con 0x7f34640cf830 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.567+0000 7f3462ffd640 1 --2- 172.21.15.72:0/1789539325 >> v2:172.21.15.146:3300/0 conn(0x7f346415dfe0 0x7f34640cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.568+0000 7f34627fc640 1 --2- 172.21.15.72:0/1789539325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f34640d22b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3464067990 tx=0x7f345402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a9b2de9799509b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.568+0000 7f3461ffb640 1 -- 172.21.15.72:0/1789539325 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454035020 con 0x7f34640cf830 2024-03-06T07:24:02.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.568+0000 7f3461ffb640 1 -- 172.21.15.72:0/1789539325 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454031870 con 0x7f34640cf830 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.568+0000 7f3461ffb640 1 -- 172.21.15.72:0/1789539325 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454031c40 con 0x7f34640cf830 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 -- 172.21.15.72:0/1789539325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 msgr2=0x7f34640d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 --2- 172.21.15.72:0/1789539325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f34640d22b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3464067990 tx=0x7f345402b910 comp rx=0 tx=0).stop 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 -- 172.21.15.72:0/1789539325 shutdown_connections 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 --2- 172.21.15.72:0/1789539325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34640d27f0 0x7f34640d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 --2- 172.21.15.72:0/1789539325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f34640d22b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 --2- 172.21.15.72:0/1789539325 >> v2:172.21.15.146:3300/0 conn(0x7f346415dfe0 0x7f34640cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 -- 172.21.15.72:0/1789539325 >> 172.21.15.72:0/1789539325 conn(0x7f34640d3cd0 msgr2=0x7f34640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 -- 172.21.15.72:0/1789539325 shutdown_connections 2024-03-06T07:24:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.569+0000 7f3469925640 1 -- 172.21.15.72:0/1789539325 wait complete. 2024-03-06T07:24:02.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.570+0000 7f3469925640 1 Processor -- start 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.570+0000 7f3469925640 1 -- start start 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3469925640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f346415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3469925640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34640d27f0 0x7f34640b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3469925640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f346415dfe0 0x7f34640b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3469925640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640d1df0 con 0x7f34640cf830 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3469925640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34640d1c70 con 0x7f34640d27f0 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3469925640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346415e3c0 con 0x7f346415dfe0 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f346415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f346415c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33846/0 (socket says 172.21.15.72:33846) 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 -- 172.21.15.72:0/4156774022 learned_addr learned my addr 172.21.15.72:0/4156774022 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f34627fc640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34640d27f0 0x7f34640b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f34637fe640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f346415dfe0 0x7f34640b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 -- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f346415dfe0 msgr2=0x7f34640b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f346415dfe0 0x7f34640b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 -- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34640d27f0 msgr2=0x7f34640b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34640d27f0 0x7f34640b68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f3462ffd640 1 -- 172.21.15.72:0/4156774022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640cd0f0 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.571+0000 7f34637fe640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f346415dfe0 0x7f34640b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.572+0000 7f3462ffd640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f346415c310 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f345800d7d0 tx=0x7f345800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.572+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458004030 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.572+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34580041d0 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.572+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34640b7440 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.572+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580096a0 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.572+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34640caf40 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.573+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346404f330 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.573+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3458009840 con 0x7f34640cf830 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.574+0000 7f344bfff640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f345006b4d0 0x7f345006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.574+0000 7f34627fc640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f345006b4d0 0x7f345006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.574+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f345808a4b0 con 0x7f34640cf830 2024-03-06T07:24:02.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.574+0000 7f34627fc640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f345006b4d0 0x7f345006d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3454036000 tx=0x7f345403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.578+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3458059330 con 0x7f34640cf830 2024-03-06T07:24:02.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.791+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34640ce9b0 con 0x7f34640cf830 2024-03-06T07:24:02.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.791+0000 7f344bfff640 1 -- 172.21.15.72:0/4156774022 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f3458058c30 con 0x7f34640cf830 2024-03-06T07:24:02.794 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:02.794 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:02.794 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f345006b4d0 msgr2=0x7f345006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f345006b4d0 0x7f345006d990 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3454036000 tx=0x7f345403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 msgr2=0x7f346415c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f346415c310 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f345800d7d0 tx=0x7f345800dc90 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 shutdown_connections 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f345006b4d0 0x7f345006d990 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f346415dfe0 0x7f34640b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34640d27f0 0x7f34640b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 --2- 172.21.15.72:0/4156774022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34640cf830 0x7f346415c310 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.793+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 >> 172.21.15.72:0/4156774022 conn(0x7f34640d3cd0 msgr2=0x7f346406bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.794+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 shutdown_connections 2024-03-06T07:24:02.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.794+0000 7f3469925640 1 -- 172.21.15.72:0/4156774022 wait complete. 2024-03-06T07:24:02.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:02.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.905+0000 7fefa4b72640 1 Processor -- start 2024-03-06T07:24:02.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.905+0000 7fefa4b72640 1 -- start start 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.906+0000 7fefa4b72640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fefa006c6d0 0x7fefa015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.906+0000 7fefa4b72640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fefa0066d90 0x7fefa00ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.906+0000 7fefa4b72640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefa00cd450 con 0x7fefa00b4f70 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.906+0000 7fefa4b72640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa015e9f0 con 0x7fefa006c6d0 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.906+0000 7fefa4b72640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefa00cda10 con 0x7fefa0066d90 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9dd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fefa006c6d0 0x7fefa015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fefa00b4f70 0x7fefa00b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46372/0 (socket says 172.21.15.72:46372) 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9e59c640 1 -- 172.21.15.72:0/2217191360 learned_addr learned my addr 172.21.15.72:0/2217191360 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9ed9d640 1 --2- 172.21.15.72:0/2217191360 >> v2:172.21.15.146:3300/0 conn(0x7fefa0066d90 0x7fefa00ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9dd9b640 1 -- 172.21.15.72:0/2217191360 >> v2:172.21.15.146:3300/0 conn(0x7fefa0066d90 msgr2=0x7fefa00ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9dd9b640 1 --2- 172.21.15.72:0/2217191360 >> v2:172.21.15.146:3300/0 conn(0x7fefa0066d90 0x7fefa00ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9dd9b640 1 -- 172.21.15.72:0/2217191360 >> v1:172.21.15.72:6789/0 conn(0x7fefa00b4f70 legacy=0x7fefa00b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9d59a640 1 -- 172.21.15.72:0/2217191360 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794153703 0 0) 0x7fefa00cd450 con 0x7fefa00b4f70 2024-03-06T07:24:02.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9dd9b640 1 -- 172.21.15.72:0/2217191360 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa00cd730 con 0x7fefa006c6d0 2024-03-06T07:24:02.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.907+0000 7fef9ed9d640 1 --2- 172.21.15.72:0/2217191360 >> v2:172.21.15.146:3300/0 conn(0x7fefa0066d90 0x7fefa00ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:02.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fef9dd9b640 1 --2- 172.21.15.72:0/2217191360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa006c6d0 0x7fefa015ee80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fefa015e960 tx=0x7fef9002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac463b0dbc87b84d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fef9d59a640 1 -- 172.21.15.72:0/2217191360 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90035020 con 0x7fefa006c6d0 2024-03-06T07:24:02.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fef9d59a640 1 -- 172.21.15.72:0/2217191360 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef90031830 con 0x7fefa006c6d0 2024-03-06T07:24:02.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fef9d59a640 1 -- 172.21.15.72:0/2217191360 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90031c00 con 0x7fefa006c6d0 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 -- 172.21.15.72:0/2217191360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa006c6d0 msgr2=0x7fefa015ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 --2- 172.21.15.72:0/2217191360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa006c6d0 0x7fefa015ee80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fefa015e960 tx=0x7fef9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 -- 172.21.15.72:0/2217191360 shutdown_connections 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 --2- 172.21.15.72:0/2217191360 >> v2:172.21.15.146:3300/0 conn(0x7fefa0066d90 0x7fefa00ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 --2- 172.21.15.72:0/2217191360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa006c6d0 0x7fefa015ee80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 -- 172.21.15.72:0/2217191360 >> 172.21.15.72:0/2217191360 conn(0x7fefa00724d0 msgr2=0x7fefa00728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:02.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 -- 172.21.15.72:0/2217191360 shutdown_connections 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.908+0000 7fefa4b72640 1 -- 172.21.15.72:0/2217191360 wait complete. 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 Processor -- start 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 -- start start 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefa0066d90 0x7fefa00b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 0x7fefa014afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefa014a450 0x7fefa0160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa00cd090 con 0x7fefa00b4f70 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa015f3c0 con 0x7fefa0066d90 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fefa4b72640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefa00cd210 con 0x7fefa014a450 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.910+0000 7fef9dd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 0x7fefa014afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 0x7fefa014afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33872/0 (socket says 172.21.15.72:33872) 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 -- 172.21.15.72:0/3594947640 learned_addr learned my addr 172.21.15.72:0/3594947640 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:02.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9ed9d640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefa014a450 0x7fefa0160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9e59c640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefa0066d90 0x7fefa00b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 -- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefa014a450 msgr2=0x7fefa0160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefa014a450 0x7fefa0160620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 -- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefa0066d90 msgr2=0x7fefa00b54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefa0066d90 0x7fefa00b54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 -- 172.21.15.72:0/3594947640 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa01e7cd0 con 0x7fefa00b4f70 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9ed9d640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefa014a450 0x7fefa0160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9e59c640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefa0066d90 0x7fefa00b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef9dd9b640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 0x7fefa014afd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fef90000c00 tx=0x7fef900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef90035050 con 0x7fefa00b4f70 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef90004230 con 0x7fefa00b4f70 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa00ae230 con 0x7fefa00b4f70 2024-03-06T07:24:02.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9003b800 con 0x7fefa00b4f70 2024-03-06T07:24:02.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.911+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa014b510 con 0x7fefa00b4f70 2024-03-06T07:24:02.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.912+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa004f330 con 0x7fefa00b4f70 2024-03-06T07:24:02.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.913+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef90038020 con 0x7fefa00b4f70 2024-03-06T07:24:02.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.913+0000 7fef96ffd640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef8006b500 0x7fef8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:02.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.913+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef900abf30 con 0x7fefa00b4f70 2024-03-06T07:24:02.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.913+0000 7fef9e59c640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef8006b500 0x7fef8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:02.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.914+0000 7fef9e59c640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef8006b500 0x7fef8006d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef88000f60 tx=0x7fef88007cc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:02.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:02.917+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef90040070 con 0x7fefa00b4f70 2024-03-06T07:24:03.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.097+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefa00b1860 con 0x7fefa00b4f70 2024-03-06T07:24:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.098+0000 7fef96ffd640 1 -- 172.21.15.72:0/3594947640 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fef9007ae40 con 0x7fefa00b4f70 2024-03-06T07:24:03.100 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:03.100 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef8006b500 msgr2=0x7fef8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef8006b500 0x7fef8006d9c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef88000f60 tx=0x7fef88007cc0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 msgr2=0x7fefa014afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 0x7fefa014afd0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fef90000c00 tx=0x7fef900046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 shutdown_connections 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef8006b500 0x7fef8006d9c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fefa014a450 0x7fefa0160620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fefa00b4f70 0x7fefa014afd0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 --2- 172.21.15.72:0/3594947640 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fefa0066d90 0x7fefa00b54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 >> 172.21.15.72:0/3594947640 conn(0x7fefa00724d0 msgr2=0x7fefa00c9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 shutdown_connections 2024-03-06T07:24:03.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.100+0000 7fefa4b72640 1 -- 172.21.15.72:0/3594947640 wait complete. 2024-03-06T07:24:03.113 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:03.113 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:03.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.212+0000 7f18f6bd7640 1 Processor -- start 2024-03-06T07:24:03.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.212+0000 7f18f6bd7640 1 -- start start 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f6bd7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18f015bb20 0x7f18f00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f6bd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f00d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f6bd7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f0161980 con 0x7f18f00ce4b0 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f6bd7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f0161c60 con 0x7f18f015bb20 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f6bd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f0161f40 con 0x7f18f00cfff0 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18f015bb20 0x7f18f00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18f015bb20 0x7f18f00d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33886/0 (socket says 172.21.15.72:33886) 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 -- 172.21.15.72:0/2820185782 learned_addr learned my addr 172.21.15.72:0/2820185782 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f5174640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f18f00ce4b0 0x7f18f00ce940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46378/0 (socket says 172.21.15.72:46378) 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18effff640 1 --2- 172.21.15.72:0/2820185782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f00d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 -- 172.21.15.72:0/2820185782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 msgr2=0x7f18f00d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 --2- 172.21.15.72:0/2820185782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f00d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 -- 172.21.15.72:0/2820185782 >> v1:172.21.15.72:6789/0 conn(0x7f18f00ce4b0 legacy=0x7f18f00ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18ef7fe640 1 -- 172.21.15.72:0/2820185782 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143312136 0 0) 0x7f18f0161980 con 0x7f18f00ce4b0 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.213+0000 7f18f4973640 1 -- 172.21.15.72:0/2820185782 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f0162220 con 0x7f18f015bb20 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.214+0000 7f18f4973640 1 --2- 172.21.15.72:0/2820185782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f00d1df0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009870 tx=0x7f18e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1d83f4910ef1b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.214+0000 7f18ef7fe640 1 -- 172.21.15.72:0/2820185782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4034070 con 0x7f18f015bb20 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.214+0000 7f18ef7fe640 1 -- 172.21.15.72:0/2820185782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4030840 con 0x7f18f015bb20 2024-03-06T07:24:03.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.214+0000 7f18ef7fe640 1 -- 172.21.15.72:0/2820185782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4030c10 con 0x7f18f015bb20 2024-03-06T07:24:03.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.214+0000 7f18f6bd7640 1 -- 172.21.15.72:0/2820185782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 msgr2=0x7f18f00d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.214+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/2820185782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f00d1df0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f18e4009870 tx=0x7f18e402b980 comp rx=0 tx=0).stop 2024-03-06T07:24:03.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.215+0000 7f18f6bd7640 1 -- 172.21.15.72:0/2820185782 shutdown_connections 2024-03-06T07:24:03.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.215+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/2820185782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f00d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.215+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/2820185782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f00d1df0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.215+0000 7f18f6bd7640 1 -- 172.21.15.72:0/2820185782 >> 172.21.15.72:0/2820185782 conn(0x7f18f009a9f0 msgr2=0x7f18f009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.215+0000 7f18f6bd7640 1 -- 172.21.15.72:0/2820185782 shutdown_connections 2024-03-06T07:24:03.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.215+0000 7f18f6bd7640 1 -- 172.21.15.72:0/2820185782 wait complete. 2024-03-06T07:24:03.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.216+0000 7f18f6bd7640 1 Processor -- start 2024-03-06T07:24:03.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.216+0000 7f18f6bd7640 1 -- start start 2024-03-06T07:24:03.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.216+0000 7f18f6bd7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f00ce4b0 0x7f18f014cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f01587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f0158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00d24b0 con 0x7f18f015bb20 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00d2330 con 0x7f18f00ce4b0 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f00d1980 con 0x7f18f00cfff0 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f0158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f0158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33900/0 (socket says 172.21.15.72:33900) 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 -- 172.21.15.72:0/1984983155 learned_addr learned my addr 172.21.15.72:0/1984983155 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f4973640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f00ce4b0 0x7f18f014cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18effff640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f01587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 -- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 msgr2=0x7f18f01587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f01587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 -- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f00ce4b0 msgr2=0x7f18f014cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f00ce4b0 0x7f18f014cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 -- 172.21.15.72:0/1984983155 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f014d430 con 0x7f18f015bb20 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18effff640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f01587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f5174640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f0158cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f18e800c700 tx=0x7f18e800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e8018070 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e800ce10 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e800a860 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f006c000 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.217+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f006c5c0 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.218+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f004f330 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.219+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18e800aa00 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.219+0000 7f18ed7fa640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18d406b4d0 0x7f18d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.219+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18e8089770 con 0x7f18f015bb20 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.219+0000 7f18f4973640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18d406b4d0 0x7f18d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.220+0000 7f18f4973640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18d406b4d0 0x7f18d406d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f18e4036000 tx=0x7f18e4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.224+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18e8058680 con 0x7f18f015bb20 2024-03-06T07:24:03.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.392+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f18f00cbef0 con 0x7f18d406b4d0 2024-03-06T07:24:03.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.398+0000 7f18ed7fa640 1 -- 172.21.15.72:0/1984983155 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237202 (secure 0 0 0) 0x7f18f00cbef0 con 0x7f18d406b4d0 2024-03-06T07:24:03.403 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:03.404 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:03.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.400+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18d406b4d0 msgr2=0x7f18d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.400+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18d406b4d0 0x7f18d406d990 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f18e4036000 tx=0x7f18e4008030 comp rx=0 tx=0).stop 2024-03-06T07:24:03.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.400+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 msgr2=0x7f18f0158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.400+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f0158cf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f18e800c700 tx=0x7f18e800cbc0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.400+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 shutdown_connections 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18d406b4d0 0x7f18d406d990 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f015bb20 0x7f18f0158cf0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f00cfff0 0x7f18f01587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 --2- 172.21.15.72:0/1984983155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f00ce4b0 0x7f18f014cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 >> 172.21.15.72:0/1984983155 conn(0x7f18f009a9f0 msgr2=0x7f18f006ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 shutdown_connections 2024-03-06T07:24:03.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.401+0000 7f18f6bd7640 1 -- 172.21.15.72:0/1984983155 wait complete. 2024-03-06T07:24:03.413 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-03-06T07:24:01.971313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255496,"kb_used_data":6036,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719224,"statfs":{"total":773094113280,"available":772832485376,"internally_reserved":0,"allocated":6180864,"data_stored":3223649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000786"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":234,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.421076+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:56.421076+0000","last_peered":"2024-03-06T07:23:56.421076+0000","last_clean":"2024-03-06T07:23:56.421076+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:56.421076+0000","last_undegraded":"2024-03-06T07:23:56.421076+0000","last_fullsized":"2024-03-06T07:23:56.421076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":845104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":270063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31780,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":254736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1241088,"data_stored":855374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:03.417 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:03.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.552+0000 7f06e08c9640 1 Processor -- start 2024-03-06T07:24:03.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.552+0000 7f06e08c9640 1 -- start start 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.553+0000 7f06e08c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f06dc15fa10 0x7f06dc0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.553+0000 7f06e08c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06dc0cebe0 con 0x7f06dc0d16e0 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.553+0000 7f06e08c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06dc0ceec0 con 0x7f06dc15bdd0 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.553+0000 7f06e08c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06dc0cf1a0 con 0x7f06dc15fa10 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06dad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f06dc0d16e0 0x7f06dc0d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46394/0 (socket says 172.21.15.72:46394) 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06dad9d640 1 -- 172.21.15.72:0/2016195461 learned_addr learned my addr 172.21.15.72:0/2016195461 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/2016195461 >> v2:172.21.15.146:3300/0 conn(0x7f06dc15fa10 0x7f06dc0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847526621 0 0) 0x7f06dc0cebe0 con 0x7f06dc0d16e0 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06c4003520 con 0x7f06dc0d16e0 2024-03-06T07:24:03.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3586395878 0 0) 0x7f06c4003520 con 0x7f06dc0d16e0 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 >> v2:172.21.15.146:3300/0 conn(0x7f06dc15fa10 msgr2=0x7f06dc0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 --2- 172.21.15.72:0/2016195461 >> v2:172.21.15.146:3300/0 conn(0x7f06dc15fa10 0x7f06dc0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 >> v1:172.21.15.146:6789/0 conn(0x7f06dc15bdd0 legacy=0x7f06dc0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc0cc8e0 con 0x7f06dc0d16e0 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f06d4001690 con 0x7f06dc0d16e0 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/2016195461 >> v2:172.21.15.146:3300/0 conn(0x7f06dc15fa10 0x7f06dc0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 >> v1:172.21.15.72:6789/0 conn(0x7f06dc0d16e0 legacy=0x7f06dc0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06c4004800 0x7f06c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06c4005210 0x7f06c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0cf1a0 con 0x7f06c4003a00 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c400c160 con 0x7f06c4004800 2024-03-06T07:24:03.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c400c360 con 0x7f06c4005210 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06da59c640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06c4004800 0x7f06c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06dad9d640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06c4005210 0x7f06c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06da59c640 1 -- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06c4005210 msgr2=0x7f06c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06da59c640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06c4005210 0x7f06c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06da59c640 1 -- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06c4004800 msgr2=0x7f06c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.554+0000 7f06da59c640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06c4004800 0x7f06c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.555+0000 7f06dad9d640 1 -- 172.21.15.72:0/2016195461 reap_dead start 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.555+0000 7f06da59c640 1 -- 172.21.15.72:0/2016195461 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c4003520 con 0x7f06c4003a00 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.555+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06c4004800 0x7f06c4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06da59c640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06c4003e20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f06cc001e20 tx=0x7f06c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289a0bd190944c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc001a70 con 0x7f06c4003a00 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc003b70 con 0x7f06c4003a00 2024-03-06T07:24:03.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06d959a640 1 -- 172.21.15.72:0/2016195461 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc0050d0 con 0x7f06c4003a00 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 -- 172.21.15.72:0/2016195461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 msgr2=0x7f06c4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06c4003e20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f06cc001e20 tx=0x7f06c4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 -- 172.21.15.72:0/2016195461 shutdown_connections 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 --2- 172.21.15.72:0/2016195461 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06c4003e20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 -- 172.21.15.72:0/2016195461 >> 172.21.15.72:0/2016195461 conn(0x7f06dc09a9f0 msgr2=0x7f06dc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 -- 172.21.15.72:0/2016195461 shutdown_connections 2024-03-06T07:24:03.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.556+0000 7f06e08c9640 1 -- 172.21.15.72:0/2016195461 wait complete. 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 Processor -- start 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 -- start start 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06dc14cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06dc158280 0x7f06dc0cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06dc14d730 0x7f06dc0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0c5610 con 0x7f06c4003a00 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc15c1b0 con 0x7f06dc158280 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06e08c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06dc0c5790 con 0x7f06dc14d730 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06da59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06dc14cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06da59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06dc14cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33926/0 (socket says 172.21.15.72:33926) 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.558+0000 7f06da59c640 1 -- 172.21.15.72:0/1251958745 learned_addr learned my addr 172.21.15.72:0/1251958745 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:03.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06dc158280 0x7f06dc0cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06da59c640 1 -- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06dc14d730 msgr2=0x7f06dc0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06da59c640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06dc14d730 0x7f06dc0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06da59c640 1 -- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06dc158280 msgr2=0x7f06dc0cb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06da59c640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06dc158280 0x7f06dc0cb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06da59c640 1 -- 172.21.15.72:0/1251958745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc0c6bc0 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06dc158280 0x7f06dc0cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06da59c640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06dc14cea0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f06cc000f80 tx=0x7f06cc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc003970 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc006b50 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06dc0c6ea0 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc007020 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.559+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06dc0c7460 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.560+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06dc04f330 con 0x7f06c4003a00 2024-03-06T07:24:03.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.561+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06cc049060 con 0x7f06c4003a00 2024-03-06T07:24:03.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.561+0000 7f06caffd640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06bc06b500 0x7f06bc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.561+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06cc0ab410 con 0x7f06c4003a00 2024-03-06T07:24:03.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.561+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06bc06b500 0x7f06bc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.562+0000 7f06d9d9b640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06bc06b500 0x7f06bc06d9c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0c7ab0 tx=0x7f06d0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.566+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06cc033020 con 0x7f06c4003a00 2024-03-06T07:24:03.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.739+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06dc0b7a40 con 0x7f06bc06b500 2024-03-06T07:24:03.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.745+0000 7f06caffd640 1 -- 172.21.15.72:0/1251958745 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+237202 (secure 0 0 0) 0x7f06dc0b7a40 con 0x7f06bc06b500 2024-03-06T07:24:03.749 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:03.750 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06bc06b500 msgr2=0x7f06bc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06bc06b500 0x7f06bc06d9c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0c7ab0 tx=0x7f06d0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 msgr2=0x7f06dc14cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06dc14cea0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f06cc000f80 tx=0x7f06cc0014a0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 shutdown_connections 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06bc06b500 0x7f06bc06d9c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06dc14d730 0x7f06dc0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06dc158280 0x7f06dc0cb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 --2- 172.21.15.72:0/1251958745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06c4003a00 0x7f06dc14cea0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 >> 172.21.15.72:0/1251958745 conn(0x7f06dc09a9f0 msgr2=0x7f06dc06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 shutdown_connections 2024-03-06T07:24:03.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.747+0000 7f06e08c9640 1 -- 172.21.15.72:0/1251958745 wait complete. 2024-03-06T07:24:03.761 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-03-06T07:24:01.971313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255496,"kb_used_data":6036,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719224,"statfs":{"total":773094113280,"available":772832485376,"internally_reserved":0,"allocated":6180864,"data_stored":3223649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000786"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:59.419924+0000","last_change":"2024-03-06T07:23:59.419924+0000","last_active":"2024-03-06T07:23:59.419924+0000","last_peered":"2024-03-06T07:23:59.419924+0000","last_clean":"2024-03-06T07:23:59.419924+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:59.419924+0000","last_undegraded":"2024-03-06T07:23:59.419924+0000","last_fullsized":"2024-03-06T07:23:59.419924+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:23:58.409921+0000","last_change":"2024-03-06T07:23:58.409921+0000","last_active":"2024-03-06T07:23:58.409921+0000","last_peered":"2024-03-06T07:23:58.409921+0000","last_clean":"2024-03-06T07:23:58.409921+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:23:58.409921+0000","last_undegraded":"2024-03-06T07:23:58.409921+0000","last_fullsized":"2024-03-06T07:23:58.409921+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":234,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-03-06T07:23:56.421076+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:23:56.421076+0000","last_peered":"2024-03-06T07:23:56.421076+0000","last_clean":"2024-03-06T07:23:56.421076+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:23:56.421076+0000","last_undegraded":"2024-03-06T07:23:56.421076+0000","last_fullsized":"2024-03-06T07:23:56.421076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":845104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":659456,"data_stored":270063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31780,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340060,"statfs":{"total":96636764160,"available":96604221440,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31776,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340064,"statfs":{"total":96636764160,"available":96604225536,"internally_reserved":0,"allocated":610304,"data_stored":249593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31788,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340052,"statfs":{"total":96636764160,"available":96604213248,"internally_reserved":0,"allocated":622592,"data_stored":254736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1241088,"data_stored":855374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:03.764 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:03.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.899+0000 7f08254f4640 1 Processor -- start 2024-03-06T07:24:03.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.899+0000 7f08254f4640 1 -- start start 2024-03-06T07:24:03.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f08254f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f08201597c0 0x7f082015cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f08254f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f08200d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f08254f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08200cd440 con 0x7f082006f290 2024-03-06T07:24:03.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f08254f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0820159ba0 con 0x7f08201597c0 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f08254f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082015c7d0 con 0x7f082006b6c0 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f081effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f08201597c0 0x7f082015cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f081effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f08201597c0 0x7f082015cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33930/0 (socket says 172.21.15.72:33930) 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f081effd640 1 -- 172.21.15.72:0/4250532798 learned_addr learned my addr 172.21.15.72:0/4250532798 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.900+0000 7f081f7fe640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f08200d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.901+0000 7f081effd640 1 -- 172.21.15.72:0/4250532798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 msgr2=0x7f08200d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.901+0000 7f081effd640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f08200d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.901+0000 7f081effd640 1 -- 172.21.15.72:0/4250532798 >> v1:172.21.15.146:6789/0 conn(0x7f082006f290 legacy=0x7f082006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.901+0000 7f081effd640 1 -- 172.21.15.72:0/4250532798 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08200cd720 con 0x7f08201597c0 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.901+0000 7f081f7fe640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f08200d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.901+0000 7f081effd640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f082015cc60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0810009870 tx=0x7f081002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253fdc38f2277cfc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f081dffb640 1 -- 172.21.15.72:0/4250532798 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810034070 con 0x7f08201597c0 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f081dffb640 1 -- 172.21.15.72:0/4250532798 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0810030840 con 0x7f08201597c0 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f081dffb640 1 -- 172.21.15.72:0/4250532798 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0810030c10 con 0x7f08201597c0 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 -- 172.21.15.72:0/4250532798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 msgr2=0x7f082015cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f082015cc60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0810009870 tx=0x7f081002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 -- 172.21.15.72:0/4250532798 shutdown_connections 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f08200d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 --2- 172.21.15.72:0/4250532798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f082015cc60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 -- 172.21.15.72:0/4250532798 >> 172.21.15.72:0/4250532798 conn(0x7f082006c0c0 msgr2=0x7f082006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 -- 172.21.15.72:0/4250532798 shutdown_connections 2024-03-06T07:24:03.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.902+0000 7f08254f4640 1 -- 172.21.15.72:0/4250532798 wait complete. 2024-03-06T07:24:03.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.904+0000 7f08254f4640 1 Processor -- start 2024-03-06T07:24:03.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.904+0000 7f08254f4640 1 -- start start 2024-03-06T07:24:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f08254f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f082013d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f08254f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f082006f290 0x7f082014fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f08254f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f0820150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f08254f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08200cd0c0 con 0x7f08201597c0 2024-03-06T07:24:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f08254f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082015c150 con 0x7f082006f290 2024-03-06T07:24:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f08254f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08200728d0 con 0x7f082006b6c0 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f081f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f0820150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f081f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f0820150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33940/0 (socket says 172.21.15.72:33940) 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.905+0000 7f081f7fe640 1 -- 172.21.15.72:0/571407214 learned_addr learned my addr 172.21.15.72:0/571407214 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081e7fc640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f082006f290 0x7f082014fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081effd640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f082013d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081f7fe640 1 -- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 msgr2=0x7f082013d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081f7fe640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f082013d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081f7fe640 1 -- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f082006f290 msgr2=0x7f082014fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081f7fe640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f082006f290 0x7f082014fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081f7fe640 1 -- 172.21.15.72:0/571407214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082014f510 con 0x7f08201597c0 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081e7fc640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f082006f290 0x7f082014fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f081f7fe640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f0820150420 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0818007de0 tx=0x7f081800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081801a070 con 0x7f08201597c0 2024-03-06T07:24:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0818003990 con 0x7f08201597c0 2024-03-06T07:24:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0818003d80 con 0x7f08201597c0 2024-03-06T07:24:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082014f7f0 con 0x7f08201597c0 2024-03-06T07:24:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.906+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082014fad0 con 0x7f08201597c0 2024-03-06T07:24:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.907+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08200ce6b0 con 0x7f08201597c0 2024-03-06T07:24:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.908+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f081801d070 con 0x7f08201597c0 2024-03-06T07:24:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.908+0000 7f08077fe640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b500 0x7f080c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.908+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f081808a4e0 con 0x7f08201597c0 2024-03-06T07:24:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.908+0000 7f081effd640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b500 0x7f080c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.909+0000 7f081effd640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b500 0x7f080c06d9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0810036000 tx=0x7f081003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:03.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:03.912+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f08200ce6b0 con 0x7f08201597c0 2024-03-06T07:24:04.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.167+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f082006f0a0 con 0x7f080c06b500 2024-03-06T07:24:04.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.179+0000 7f08077fe640 1 -- 172.21.15.72:0/571407214 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251920 (secure 0 0 0) 0x7f082006f0a0 con 0x7f080c06b500 2024-03-06T07:24:04.183 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:04.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b500 msgr2=0x7f080c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b500 0x7f080c06d9c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0810036000 tx=0x7f081003b040 comp rx=0 tx=0).stop 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 msgr2=0x7f0820150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f0820150420 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f0818007de0 tx=0x7f081800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 shutdown_connections 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f080c06b500 0x7f080c06d9c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f08201597c0 0x7f0820150420 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f082006f290 0x7f082014fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 --2- 172.21.15.72:0/571407214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f082006b6c0 0x7f082013d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 >> 172.21.15.72:0/571407214 conn(0x7f082006c0c0 msgr2=0x7f08200cfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 shutdown_connections 2024-03-06T07:24:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.182+0000 7f08254f4640 1 -- 172.21.15.72:0/571407214 wait complete. 2024-03-06T07:24:04.196 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-03-06T07:24:03.971637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256156,"kb_used_data":6688,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718564,"statfs":{"total":773094113280,"available":772831809536,"internally_reserved":0,"allocated":6848512,"data_stored":3509532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.551913"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449532+0000","last_change":"2024-03-06T07:24:00.439946+0000","last_active":"2024-03-06T07:24:01.449532+0000","last_peered":"2024-03-06T07:24:01.449532+0000","last_clean":"2024-03-06T07:24:01.449532+0000","last_became_active":"2024-03-06T07:24:00.439682+0000","last_became_peered":"2024-03-06T07:24:00.439682+0000","last_unstale":"2024-03-06T07:24:01.449532+0000","last_undegraded":"2024-03-06T07:24:01.449532+0000","last_fullsized":"2024-03-06T07:24:01.449532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:48.774758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452451+0000","last_change":"2024-03-06T07:24:00.446637+0000","last_active":"2024-03-06T07:24:00.452451+0000","last_peered":"2024-03-06T07:24:00.452451+0000","last_clean":"2024-03-06T07:24:00.452451+0000","last_became_active":"2024-03-06T07:24:00.446373+0000","last_became_peered":"2024-03-06T07:24:00.446373+0000","last_unstale":"2024-03-06T07:24:00.452451+0000","last_undegraded":"2024-03-06T07:24:00.452451+0000","last_fullsized":"2024-03-06T07:24:00.452451+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:11.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587704+0000","last_change":"2024-03-06T07:24:00.439609+0000","last_active":"2024-03-06T07:24:01.587704+0000","last_peered":"2024-03-06T07:24:01.587704+0000","last_clean":"2024-03-06T07:24:01.587704+0000","last_became_active":"2024-03-06T07:24:00.439333+0000","last_became_peered":"2024-03-06T07:24:00.439333+0000","last_unstale":"2024-03-06T07:24:01.587704+0000","last_undegraded":"2024-03-06T07:24:01.587704+0000","last_fullsized":"2024-03-06T07:24:01.587704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448757+0000","last_change":"2024-03-06T07:24:00.445347+0000","last_active":"2024-03-06T07:24:01.448757+0000","last_peered":"2024-03-06T07:24:01.448757+0000","last_clean":"2024-03-06T07:24:01.448757+0000","last_became_active":"2024-03-06T07:24:00.445031+0000","last_became_peered":"2024-03-06T07:24:00.445031+0000","last_unstale":"2024-03-06T07:24:01.448757+0000","last_undegraded":"2024-03-06T07:24:01.448757+0000","last_fullsized":"2024-03-06T07:24:01.448757+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449309+0000","last_change":"2024-03-06T07:24:00.441074+0000","last_active":"2024-03-06T07:24:01.449309+0000","last_peered":"2024-03-06T07:24:01.449309+0000","last_clean":"2024-03-06T07:24:01.449309+0000","last_became_active":"2024-03-06T07:24:00.440804+0000","last_became_peered":"2024-03-06T07:24:00.440804+0000","last_unstale":"2024-03-06T07:24:01.449309+0000","last_undegraded":"2024-03-06T07:24:01.449309+0000","last_fullsized":"2024-03-06T07:24:01.449309+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:39.417435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.456804+0000","last_change":"2024-03-06T07:24:00.453304+0000","last_active":"2024-03-06T07:24:00.456804+0000","last_peered":"2024-03-06T07:24:00.456804+0000","last_clean":"2024-03-06T07:24:00.456804+0000","last_became_active":"2024-03-06T07:24:00.452795+0000","last_became_peered":"2024-03-06T07:24:00.452795+0000","last_unstale":"2024-03-06T07:24:00.456804+0000","last_undegraded":"2024-03-06T07:24:00.456804+0000","last_fullsized":"2024-03-06T07:24:00.456804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448469+0000","last_change":"2024-03-06T07:24:00.444902+0000","last_active":"2024-03-06T07:24:01.448469+0000","last_peered":"2024-03-06T07:24:01.448469+0000","last_clean":"2024-03-06T07:24:01.448469+0000","last_became_active":"2024-03-06T07:24:00.444629+0000","last_became_peered":"2024-03-06T07:24:00.444629+0000","last_unstale":"2024-03-06T07:24:01.448469+0000","last_undegraded":"2024-03-06T07:24:01.448469+0000","last_fullsized":"2024-03-06T07:24:01.448469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:54.193912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448269+0000","last_change":"2024-03-06T07:24:00.439953+0000","last_active":"2024-03-06T07:24:00.448269+0000","last_peered":"2024-03-06T07:24:00.448269+0000","last_clean":"2024-03-06T07:24:00.448269+0000","last_became_active":"2024-03-06T07:24:00.439683+0000","last_became_peered":"2024-03-06T07:24:00.439683+0000","last_unstale":"2024-03-06T07:24:00.448269+0000","last_undegraded":"2024-03-06T07:24:00.448269+0000","last_fullsized":"2024-03-06T07:24:00.448269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448813+0000","last_change":"2024-03-06T07:24:00.444198+0000","last_active":"2024-03-06T07:24:01.448813+0000","last_peered":"2024-03-06T07:24:01.448813+0000","last_clean":"2024-03-06T07:24:01.448813+0000","last_became_active":"2024-03-06T07:24:00.443885+0000","last_became_peered":"2024-03-06T07:24:00.443885+0000","last_unstale":"2024-03-06T07:24:01.448813+0000","last_undegraded":"2024-03-06T07:24:01.448813+0000","last_fullsized":"2024-03-06T07:24:01.448813+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:22.426550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448763+0000","last_change":"2024-03-06T07:24:00.444858+0000","last_active":"2024-03-06T07:24:00.448763+0000","last_peered":"2024-03-06T07:24:00.448763+0000","last_clean":"2024-03-06T07:24:00.448763+0000","last_became_active":"2024-03-06T07:24:00.444401+0000","last_became_peered":"2024-03-06T07:24:00.444401+0000","last_unstale":"2024-03-06T07:24:00.448763+0000","last_undegraded":"2024-03-06T07:24:00.448763+0000","last_fullsized":"2024-03-06T07:24:00.448763+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:58.913668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.453709+0000","last_change":"2024-03-06T07:24:00.447374+0000","last_active":"2024-03-06T07:24:00.453709+0000","last_peered":"2024-03-06T07:24:00.453709+0000","last_clean":"2024-03-06T07:24:00.453709+0000","last_became_active":"2024-03-06T07:24:00.445602+0000","last_became_peered":"2024-03-06T07:24:00.445602+0000","last_unstale":"2024-03-06T07:24:00.453709+0000","last_undegraded":"2024-03-06T07:24:00.453709+0000","last_fullsized":"2024-03-06T07:24:00.453709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:21.961725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448447+0000","last_change":"2024-03-06T07:24:00.447595+0000","last_active":"2024-03-06T07:24:01.448447+0000","last_peered":"2024-03-06T07:24:01.448447+0000","last_clean":"2024-03-06T07:24:01.448447+0000","last_became_active":"2024-03-06T07:24:00.446729+0000","last_became_peered":"2024-03-06T07:24:00.446729+0000","last_unstale":"2024-03-06T07:24:01.448447+0000","last_undegraded":"2024-03-06T07:24:01.448447+0000","last_fullsized":"2024-03-06T07:24:01.448447+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.447121+0000","last_change":"2024-03-06T07:24:00.444770+0000","last_active":"2024-03-06T07:24:00.447121+0000","last_peered":"2024-03-06T07:24:00.447121+0000","last_clean":"2024-03-06T07:24:00.447121+0000","last_became_active":"2024-03-06T07:24:00.444233+0000","last_became_peered":"2024-03-06T07:24:00.444233+0000","last_unstale":"2024-03-06T07:24:00.447121+0000","last_undegraded":"2024-03-06T07:24:00.447121+0000","last_fullsized":"2024-03-06T07:24:00.447121+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:36.621918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448903+0000","last_change":"2024-03-06T07:24:00.452714+0000","last_active":"2024-03-06T07:24:01.448903+0000","last_peered":"2024-03-06T07:24:01.448903+0000","last_clean":"2024-03-06T07:24:01.448903+0000","last_became_active":"2024-03-06T07:24:00.452377+0000","last_became_peered":"2024-03-06T07:24:00.452377+0000","last_unstale":"2024-03-06T07:24:01.448903+0000","last_undegraded":"2024-03-06T07:24:01.448903+0000","last_fullsized":"2024-03-06T07:24:01.448903+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:30.827190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448497+0000","last_change":"2024-03-06T07:24:00.448276+0000","last_active":"2024-03-06T07:24:01.448497+0000","last_peered":"2024-03-06T07:24:01.448497+0000","last_clean":"2024-03-06T07:24:01.448497+0000","last_became_active":"2024-03-06T07:24:00.447961+0000","last_became_peered":"2024-03-06T07:24:00.447961+0000","last_unstale":"2024-03-06T07:24:01.448497+0000","last_undegraded":"2024-03-06T07:24:01.448497+0000","last_fullsized":"2024-03-06T07:24:01.448497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:57.067784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451491+0000","last_change":"2024-03-06T07:24:00.444103+0000","last_active":"2024-03-06T07:24:00.451491+0000","last_peered":"2024-03-06T07:24:00.451491+0000","last_clean":"2024-03-06T07:24:00.451491+0000","last_became_active":"2024-03-06T07:24:00.443851+0000","last_became_peered":"2024-03-06T07:24:00.443851+0000","last_unstale":"2024-03-06T07:24:00.451491+0000","last_undegraded":"2024-03-06T07:24:00.451491+0000","last_fullsized":"2024-03-06T07:24:00.451491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:35.722017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452542+0000","last_change":"2024-03-06T07:24:00.443021+0000","last_active":"2024-03-06T07:24:00.452542+0000","last_peered":"2024-03-06T07:24:00.452542+0000","last_clean":"2024-03-06T07:24:00.452542+0000","last_became_active":"2024-03-06T07:24:00.442730+0000","last_became_peered":"2024-03-06T07:24:00.442730+0000","last_unstale":"2024-03-06T07:24:00.452542+0000","last_undegraded":"2024-03-06T07:24:00.452542+0000","last_fullsized":"2024-03-06T07:24:00.452542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587794+0000","last_change":"2024-03-06T07:24:00.451841+0000","last_active":"2024-03-06T07:24:01.587794+0000","last_peered":"2024-03-06T07:24:01.587794+0000","last_clean":"2024-03-06T07:24:01.587794+0000","last_became_active":"2024-03-06T07:24:00.451473+0000","last_became_peered":"2024-03-06T07:24:00.451473+0000","last_unstale":"2024-03-06T07:24:01.587794+0000","last_undegraded":"2024-03-06T07:24:01.587794+0000","last_fullsized":"2024-03-06T07:24:01.587794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:15.579899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449397+0000","last_change":"2024-03-06T07:24:00.439220+0000","last_active":"2024-03-06T07:24:01.449397+0000","last_peered":"2024-03-06T07:24:01.449397+0000","last_clean":"2024-03-06T07:24:01.449397+0000","last_became_active":"2024-03-06T07:24:00.438971+0000","last_became_peered":"2024-03-06T07:24:00.438971+0000","last_unstale":"2024-03-06T07:24:01.449397+0000","last_undegraded":"2024-03-06T07:24:01.449397+0000","last_fullsized":"2024-03-06T07:24:01.449397+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.634491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452285+0000","last_change":"2024-03-06T07:24:00.447150+0000","last_active":"2024-03-06T07:24:00.452285+0000","last_peered":"2024-03-06T07:24:00.452285+0000","last_clean":"2024-03-06T07:24:00.452285+0000","last_became_active":"2024-03-06T07:24:00.446837+0000","last_became_peered":"2024-03-06T07:24:00.446837+0000","last_unstale":"2024-03-06T07:24:00.452285+0000","last_undegraded":"2024-03-06T07:24:00.452285+0000","last_fullsized":"2024-03-06T07:24:00.452285+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:39.577599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587434+0000","last_change":"2024-03-06T07:24:00.442324+0000","last_active":"2024-03-06T07:24:01.587434+0000","last_peered":"2024-03-06T07:24:01.587434+0000","last_clean":"2024-03-06T07:24:01.587434+0000","last_became_active":"2024-03-06T07:24:00.442041+0000","last_became_peered":"2024-03-06T07:24:00.442041+0000","last_unstale":"2024-03-06T07:24:01.587434+0000","last_undegraded":"2024-03-06T07:24:01.587434+0000","last_fullsized":"2024-03-06T07:24:01.587434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451994+0000","last_change":"2024-03-06T07:24:00.442786+0000","last_active":"2024-03-06T07:24:00.451994+0000","last_peered":"2024-03-06T07:24:00.451994+0000","last_clean":"2024-03-06T07:24:00.451994+0000","last_became_active":"2024-03-06T07:24:00.442534+0000","last_became_peered":"2024-03-06T07:24:00.442534+0000","last_unstale":"2024-03-06T07:24:00.451994+0000","last_undegraded":"2024-03-06T07:24:00.451994+0000","last_fullsized":"2024-03-06T07:24:00.451994+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587327+0000","last_change":"2024-03-06T07:24:00.445639+0000","last_active":"2024-03-06T07:24:01.587327+0000","last_peered":"2024-03-06T07:24:01.587327+0000","last_clean":"2024-03-06T07:24:01.587327+0000","last_became_active":"2024-03-06T07:24:00.445190+0000","last_became_peered":"2024-03-06T07:24:00.445190+0000","last_unstale":"2024-03-06T07:24:01.587327+0000","last_undegraded":"2024-03-06T07:24:01.587327+0000","last_fullsized":"2024-03-06T07:24:01.587327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.443994+0000","last_change":"2024-03-06T07:24:00.441727+0000","last_active":"2024-03-06T07:24:00.443994+0000","last_peered":"2024-03-06T07:24:00.443994+0000","last_clean":"2024-03-06T07:24:00.443994+0000","last_became_active":"2024-03-06T07:24:00.440305+0000","last_became_peered":"2024-03-06T07:24:00.440305+0000","last_unstale":"2024-03-06T07:24:00.443994+0000","last_undegraded":"2024-03-06T07:24:00.443994+0000","last_fullsized":"2024-03-06T07:24:00.443994+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:52.902789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449608+0000","last_change":"2024-03-06T07:24:00.446374+0000","last_active":"2024-03-06T07:24:01.449608+0000","last_peered":"2024-03-06T07:24:01.449608+0000","last_clean":"2024-03-06T07:24:01.449608+0000","last_became_active":"2024-03-06T07:24:00.446119+0000","last_became_peered":"2024-03-06T07:24:00.446119+0000","last_unstale":"2024-03-06T07:24:01.449608+0000","last_undegraded":"2024-03-06T07:24:01.449608+0000","last_fullsized":"2024-03-06T07:24:01.449608+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:20.937564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448327+0000","last_change":"2024-03-06T07:24:00.448205+0000","last_active":"2024-03-06T07:24:01.448327+0000","last_peered":"2024-03-06T07:24:01.448327+0000","last_clean":"2024-03-06T07:24:01.448327+0000","last_became_active":"2024-03-06T07:24:00.445713+0000","last_became_peered":"2024-03-06T07:24:00.445713+0000","last_unstale":"2024-03-06T07:24:01.448327+0000","last_undegraded":"2024-03-06T07:24:01.448327+0000","last_fullsized":"2024-03-06T07:24:01.448327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451941+0000","last_change":"2024-03-06T07:24:00.446705+0000","last_active":"2024-03-06T07:24:00.451941+0000","last_peered":"2024-03-06T07:24:00.451941+0000","last_clean":"2024-03-06T07:24:00.451941+0000","last_became_active":"2024-03-06T07:24:00.446464+0000","last_became_peered":"2024-03-06T07:24:00.446464+0000","last_unstale":"2024-03-06T07:24:00.451941+0000","last_undegraded":"2024-03-06T07:24:00.451941+0000","last_fullsized":"2024-03-06T07:24:00.451941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:41.593226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449580+0000","last_change":"2024-03-06T07:24:00.447941+0000","last_active":"2024-03-06T07:24:01.449580+0000","last_peered":"2024-03-06T07:24:01.449580+0000","last_clean":"2024-03-06T07:24:01.449580+0000","last_became_active":"2024-03-06T07:24:00.447687+0000","last_became_peered":"2024-03-06T07:24:00.447687+0000","last_unstale":"2024-03-06T07:24:01.449580+0000","last_undegraded":"2024-03-06T07:24:01.449580+0000","last_fullsized":"2024-03-06T07:24:01.449580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:19.118141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452493+0000","last_change":"2024-03-06T07:24:00.442005+0000","last_active":"2024-03-06T07:24:00.452493+0000","last_peered":"2024-03-06T07:24:00.452493+0000","last_clean":"2024-03-06T07:24:00.452493+0000","last_became_active":"2024-03-06T07:24:00.441729+0000","last_became_peered":"2024-03-06T07:24:00.441729+0000","last_unstale":"2024-03-06T07:24:00.452493+0000","last_undegraded":"2024-03-06T07:24:00.452493+0000","last_fullsized":"2024-03-06T07:24:00.452493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587466+0000","last_change":"2024-03-06T07:24:00.440566+0000","last_active":"2024-03-06T07:24:01.587466+0000","last_peered":"2024-03-06T07:24:01.587466+0000","last_clean":"2024-03-06T07:24:01.587466+0000","last_became_active":"2024-03-06T07:24:00.440019+0000","last_became_peered":"2024-03-06T07:24:00.440019+0000","last_unstale":"2024-03-06T07:24:01.587466+0000","last_undegraded":"2024-03-06T07:24:01.587466+0000","last_fullsized":"2024-03-06T07:24:01.587466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449286+0000","last_change":"2024-03-06T07:24:00.445742+0000","last_active":"2024-03-06T07:24:01.449286+0000","last_peered":"2024-03-06T07:24:01.449286+0000","last_clean":"2024-03-06T07:24:01.449286+0000","last_became_active":"2024-03-06T07:24:00.445483+0000","last_became_peered":"2024-03-06T07:24:00.445483+0000","last_unstale":"2024-03-06T07:24:01.449286+0000","last_undegraded":"2024-03-06T07:24:01.449286+0000","last_fullsized":"2024-03-06T07:24:01.449286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:02.426858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451511+0000","last_change":"2024-03-06T07:24:00.444808+0000","last_active":"2024-03-06T07:24:00.451511+0000","last_peered":"2024-03-06T07:24:00.451511+0000","last_clean":"2024-03-06T07:24:00.451511+0000","last_became_active":"2024-03-06T07:24:00.444516+0000","last_became_peered":"2024-03-06T07:24:00.444516+0000","last_unstale":"2024-03-06T07:24:00.451511+0000","last_undegraded":"2024-03-06T07:24:00.451511+0000","last_fullsized":"2024-03-06T07:24:00.451511+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:13.209868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.456968+0000","last_change":"2024-03-06T07:24:00.447645+0000","last_active":"2024-03-06T07:24:00.456968+0000","last_peered":"2024-03-06T07:24:00.456968+0000","last_clean":"2024-03-06T07:24:00.456968+0000","last_became_active":"2024-03-06T07:24:00.445733+0000","last_became_peered":"2024-03-06T07:24:00.445733+0000","last_unstale":"2024-03-06T07:24:00.456968+0000","last_undegraded":"2024-03-06T07:24:00.456968+0000","last_fullsized":"2024-03-06T07:24:00.456968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:47.112827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.461097+0000","last_change":"2024-03-06T07:23:59.439938+0000","last_active":"2024-03-06T07:24:00.461097+0000","last_peered":"2024-03-06T07:24:00.461097+0000","last_clean":"2024-03-06T07:24:00.461097+0000","last_became_active":"2024-03-06T07:23:59.439672+0000","last_became_peered":"2024-03-06T07:23:59.439672+0000","last_unstale":"2024-03-06T07:24:00.461097+0000","last_undegraded":"2024-03-06T07:24:00.461097+0000","last_fullsized":"2024-03-06T07:24:00.461097+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.454911+0000","last_change":"2024-03-06T07:24:00.448305+0000","last_active":"2024-03-06T07:24:00.454911+0000","last_peered":"2024-03-06T07:24:00.454911+0000","last_clean":"2024-03-06T07:24:00.454911+0000","last_became_active":"2024-03-06T07:24:00.447621+0000","last_became_peered":"2024-03-06T07:24:00.447621+0000","last_unstale":"2024-03-06T07:24:00.454911+0000","last_undegraded":"2024-03-06T07:24:00.454911+0000","last_fullsized":"2024-03-06T07:24:00.454911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:23.901716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460260+0000","last_change":"2024-03-06T07:23:59.440673+0000","last_active":"2024-03-06T07:24:00.460260+0000","last_peered":"2024-03-06T07:24:00.460260+0000","last_clean":"2024-03-06T07:24:00.460260+0000","last_became_active":"2024-03-06T07:23:59.440422+0000","last_became_peered":"2024-03-06T07:23:59.440422+0000","last_unstale":"2024-03-06T07:24:00.460260+0000","last_undegraded":"2024-03-06T07:24:00.460260+0000","last_fullsized":"2024-03-06T07:24:00.460260+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:23.304839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455265+0000","last_change":"2024-03-06T07:24:00.449741+0000","last_active":"2024-03-06T07:24:00.455265+0000","last_peered":"2024-03-06T07:24:00.455265+0000","last_clean":"2024-03-06T07:24:00.455265+0000","last_became_active":"2024-03-06T07:24:00.449211+0000","last_became_peered":"2024-03-06T07:24:00.449211+0000","last_unstale":"2024-03-06T07:24:00.455265+0000","last_undegraded":"2024-03-06T07:24:00.455265+0000","last_fullsized":"2024-03-06T07:24:00.455265+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460671+0000","last_change":"2024-03-06T07:23:59.435437+0000","last_active":"2024-03-06T07:24:00.460671+0000","last_peered":"2024-03-06T07:24:00.460671+0000","last_clean":"2024-03-06T07:24:00.460671+0000","last_became_active":"2024-03-06T07:23:59.435184+0000","last_became_peered":"2024-03-06T07:23:59.435184+0000","last_unstale":"2024-03-06T07:24:00.460671+0000","last_undegraded":"2024-03-06T07:24:00.460671+0000","last_fullsized":"2024-03-06T07:24:00.460671+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455259+0000","last_change":"2024-03-06T07:24:00.449515+0000","last_active":"2024-03-06T07:24:00.455259+0000","last_peered":"2024-03-06T07:24:00.455259+0000","last_clean":"2024-03-06T07:24:00.455259+0000","last_became_active":"2024-03-06T07:24:00.449232+0000","last_became_peered":"2024-03-06T07:24:00.449232+0000","last_unstale":"2024-03-06T07:24:00.455259+0000","last_undegraded":"2024-03-06T07:24:00.455259+0000","last_fullsized":"2024-03-06T07:24:00.455259+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:37.348299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448545+0000","last_change":"2024-03-06T07:23:59.439760+0000","last_active":"2024-03-06T07:24:01.448545+0000","last_peered":"2024-03-06T07:24:01.448545+0000","last_clean":"2024-03-06T07:24:01.448545+0000","last_became_active":"2024-03-06T07:23:59.439480+0000","last_became_peered":"2024-03-06T07:23:59.439480+0000","last_unstale":"2024-03-06T07:24:01.448545+0000","last_undegraded":"2024-03-06T07:24:01.448545+0000","last_fullsized":"2024-03-06T07:24:01.448545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:21.416548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452045+0000","last_change":"2024-03-06T07:24:00.443357+0000","last_active":"2024-03-06T07:24:00.452045+0000","last_peered":"2024-03-06T07:24:00.452045+0000","last_clean":"2024-03-06T07:24:00.452045+0000","last_became_active":"2024-03-06T07:24:00.443130+0000","last_became_peered":"2024-03-06T07:24:00.443130+0000","last_unstale":"2024-03-06T07:24:00.452045+0000","last_undegraded":"2024-03-06T07:24:00.452045+0000","last_fullsized":"2024-03-06T07:24:00.452045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.121640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449204+0000","last_change":"2024-03-06T07:23:59.441469+0000","last_active":"2024-03-06T07:24:01.449204+0000","last_peered":"2024-03-06T07:24:01.449204+0000","last_clean":"2024-03-06T07:24:01.449204+0000","last_became_active":"2024-03-06T07:23:59.441163+0000","last_became_peered":"2024-03-06T07:23:59.441163+0000","last_unstale":"2024-03-06T07:24:01.449204+0000","last_undegraded":"2024-03-06T07:24:01.449204+0000","last_fullsized":"2024-03-06T07:24:01.449204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:36.767833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452510+0000","last_change":"2024-03-06T07:24:00.443896+0000","last_active":"2024-03-06T07:24:00.452510+0000","last_peered":"2024-03-06T07:24:00.452510+0000","last_clean":"2024-03-06T07:24:00.452510+0000","last_became_active":"2024-03-06T07:24:00.443638+0000","last_became_peered":"2024-03-06T07:24:00.443638+0000","last_unstale":"2024-03-06T07:24:00.452510+0000","last_undegraded":"2024-03-06T07:24:00.452510+0000","last_fullsized":"2024-03-06T07:24:00.452510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:45.446431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.459348+0000","last_change":"2024-03-06T07:23:59.438876+0000","last_active":"2024-03-06T07:24:00.459348+0000","last_peered":"2024-03-06T07:24:00.459348+0000","last_clean":"2024-03-06T07:24:00.459348+0000","last_became_active":"2024-03-06T07:23:59.438581+0000","last_became_peered":"2024-03-06T07:23:59.438581+0000","last_unstale":"2024-03-06T07:24:00.459348+0000","last_undegraded":"2024-03-06T07:24:00.459348+0000","last_fullsized":"2024-03-06T07:24:00.459348+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.057781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.436978+0000","last_change":"2024-03-06T07:24:00.435008+0000","last_active":"2024-03-06T07:24:00.436978+0000","last_peered":"2024-03-06T07:24:00.436978+0000","last_clean":"2024-03-06T07:24:00.436978+0000","last_became_active":"2024-03-06T07:24:00.433660+0000","last_became_peered":"2024-03-06T07:24:00.433660+0000","last_unstale":"2024-03-06T07:24:00.436978+0000","last_undegraded":"2024-03-06T07:24:00.436978+0000","last_fullsized":"2024-03-06T07:24:00.436978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:06.788182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449045+0000","last_change":"2024-03-06T07:23:59.438832+0000","last_active":"2024-03-06T07:24:01.449045+0000","last_peered":"2024-03-06T07:24:01.449045+0000","last_clean":"2024-03-06T07:24:01.449045+0000","last_became_active":"2024-03-06T07:23:59.438564+0000","last_became_peered":"2024-03-06T07:23:59.438564+0000","last_unstale":"2024-03-06T07:24:01.449045+0000","last_undegraded":"2024-03-06T07:24:01.449045+0000","last_fullsized":"2024-03-06T07:24:01.449045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.053733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452404+0000","last_change":"2024-03-06T07:24:00.443103+0000","last_active":"2024-03-06T07:24:00.452404+0000","last_peered":"2024-03-06T07:24:00.452404+0000","last_clean":"2024-03-06T07:24:00.452404+0000","last_became_active":"2024-03-06T07:24:00.442853+0000","last_became_peered":"2024-03-06T07:24:00.442853+0000","last_unstale":"2024-03-06T07:24:00.452404+0000","last_undegraded":"2024-03-06T07:24:00.452404+0000","last_fullsized":"2024-03-06T07:24:00.452404+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:15.579899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448304+0000","last_change":"2024-03-06T07:23:59.440421+0000","last_active":"2024-03-06T07:24:01.448304+0000","last_peered":"2024-03-06T07:24:01.448304+0000","last_clean":"2024-03-06T07:24:01.448304+0000","last_became_active":"2024-03-06T07:23:59.440160+0000","last_became_peered":"2024-03-06T07:23:59.440160+0000","last_unstale":"2024-03-06T07:24:01.448304+0000","last_undegraded":"2024-03-06T07:24:01.448304+0000","last_fullsized":"2024-03-06T07:24:01.448304+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:12.543578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449819+0000","last_change":"2024-03-06T07:24:00.447813+0000","last_active":"2024-03-06T07:24:01.449819+0000","last_peered":"2024-03-06T07:24:01.449819+0000","last_clean":"2024-03-06T07:24:01.449819+0000","last_became_active":"2024-03-06T07:24:00.447532+0000","last_became_peered":"2024-03-06T07:24:00.447532+0000","last_unstale":"2024-03-06T07:24:01.449819+0000","last_undegraded":"2024-03-06T07:24:01.449819+0000","last_fullsized":"2024-03-06T07:24:01.449819+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:13.646985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587813+0000","last_change":"2024-03-06T07:23:59.443581+0000","last_active":"2024-03-06T07:24:01.587813+0000","last_peered":"2024-03-06T07:24:01.587813+0000","last_clean":"2024-03-06T07:24:01.587813+0000","last_became_active":"2024-03-06T07:23:59.443241+0000","last_became_peered":"2024-03-06T07:23:59.443241+0000","last_unstale":"2024-03-06T07:24:01.587813+0000","last_undegraded":"2024-03-06T07:24:01.587813+0000","last_fullsized":"2024-03-06T07:24:01.587813+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:29.129622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587863+0000","last_change":"2024-03-06T07:24:00.449617+0000","last_active":"2024-03-06T07:24:01.587863+0000","last_peered":"2024-03-06T07:24:01.587863+0000","last_clean":"2024-03-06T07:24:01.587863+0000","last_became_active":"2024-03-06T07:24:00.447044+0000","last_became_peered":"2024-03-06T07:24:00.447044+0000","last_unstale":"2024-03-06T07:24:01.587863+0000","last_undegraded":"2024-03-06T07:24:01.587863+0000","last_fullsized":"2024-03-06T07:24:01.587863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:45.446431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.436569+0000","last_change":"2024-03-06T07:23:59.442149+0000","last_active":"2024-03-06T07:24:00.436569+0000","last_peered":"2024-03-06T07:24:00.436569+0000","last_clean":"2024-03-06T07:24:00.436569+0000","last_became_active":"2024-03-06T07:23:59.441779+0000","last_became_peered":"2024-03-06T07:23:59.441779+0000","last_unstale":"2024-03-06T07:24:00.436569+0000","last_undegraded":"2024-03-06T07:24:00.436569+0000","last_fullsized":"2024-03-06T07:24:00.436569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:19.338232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448460+0000","last_change":"2024-03-06T07:24:00.445041+0000","last_active":"2024-03-06T07:24:00.448460+0000","last_peered":"2024-03-06T07:24:00.448460+0000","last_clean":"2024-03-06T07:24:00.448460+0000","last_became_active":"2024-03-06T07:24:00.444726+0000","last_became_peered":"2024-03-06T07:24:00.444726+0000","last_unstale":"2024-03-06T07:24:00.448460+0000","last_undegraded":"2024-03-06T07:24:00.448460+0000","last_fullsized":"2024-03-06T07:24:00.448460+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:38.450025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587905+0000","last_change":"2024-03-06T07:23:59.441620+0000","last_active":"2024-03-06T07:24:01.587905+0000","last_peered":"2024-03-06T07:24:01.587905+0000","last_clean":"2024-03-06T07:24:01.587905+0000","last_became_active":"2024-03-06T07:23:59.441352+0000","last_became_peered":"2024-03-06T07:23:59.441352+0000","last_unstale":"2024-03-06T07:24:01.587905+0000","last_undegraded":"2024-03-06T07:24:01.587905+0000","last_fullsized":"2024-03-06T07:24:01.587905+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.053733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587694+0000","last_change":"2024-03-06T07:24:00.449597+0000","last_active":"2024-03-06T07:24:01.587694+0000","last_peered":"2024-03-06T07:24:01.587694+0000","last_clean":"2024-03-06T07:24:01.587694+0000","last_became_active":"2024-03-06T07:24:00.449118+0000","last_became_peered":"2024-03-06T07:24:00.449118+0000","last_unstale":"2024-03-06T07:24:01.587694+0000","last_undegraded":"2024-03-06T07:24:01.587694+0000","last_fullsized":"2024-03-06T07:24:01.587694+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449762+0000","last_change":"2024-03-06T07:23:59.444415+0000","last_active":"2024-03-06T07:24:01.449762+0000","last_peered":"2024-03-06T07:24:01.449762+0000","last_clean":"2024-03-06T07:24:01.449762+0000","last_became_active":"2024-03-06T07:23:59.444120+0000","last_became_peered":"2024-03-06T07:23:59.444120+0000","last_unstale":"2024-03-06T07:24:01.449762+0000","last_undegraded":"2024-03-06T07:24:01.449762+0000","last_fullsized":"2024-03-06T07:24:01.449762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:14.605955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448246+0000","last_change":"2024-03-06T07:24:00.449289+0000","last_active":"2024-03-06T07:24:01.448246+0000","last_peered":"2024-03-06T07:24:01.448246+0000","last_clean":"2024-03-06T07:24:01.448246+0000","last_became_active":"2024-03-06T07:24:00.447448+0000","last_became_peered":"2024-03-06T07:24:00.447448+0000","last_unstale":"2024-03-06T07:24:01.448246+0000","last_undegraded":"2024-03-06T07:24:01.448246+0000","last_fullsized":"2024-03-06T07:24:01.448246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460960+0000","last_change":"2024-03-06T07:23:59.443439+0000","last_active":"2024-03-06T07:24:00.460960+0000","last_peered":"2024-03-06T07:24:00.460960+0000","last_clean":"2024-03-06T07:24:00.460960+0000","last_became_active":"2024-03-06T07:23:59.443188+0000","last_became_peered":"2024-03-06T07:23:59.443188+0000","last_unstale":"2024-03-06T07:24:00.460960+0000","last_undegraded":"2024-03-06T07:24:00.460960+0000","last_fullsized":"2024-03-06T07:24:00.460960+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:47.951502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587625+0000","last_change":"2024-03-06T07:24:00.434521+0000","last_active":"2024-03-06T07:24:01.587625+0000","last_peered":"2024-03-06T07:24:01.587625+0000","last_clean":"2024-03-06T07:24:01.587625+0000","last_became_active":"2024-03-06T07:24:00.434247+0000","last_became_peered":"2024-03-06T07:24:00.434247+0000","last_unstale":"2024-03-06T07:24:01.587625+0000","last_undegraded":"2024-03-06T07:24:01.587625+0000","last_fullsized":"2024-03-06T07:24:01.587625+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.431173+0000","last_change":"2024-03-06T07:23:59.441682+0000","last_active":"2024-03-06T07:24:00.431173+0000","last_peered":"2024-03-06T07:24:00.431173+0000","last_clean":"2024-03-06T07:24:00.431173+0000","last_became_active":"2024-03-06T07:23:59.441406+0000","last_became_peered":"2024-03-06T07:23:59.441406+0000","last_unstale":"2024-03-06T07:24:00.431173+0000","last_undegraded":"2024-03-06T07:24:00.431173+0000","last_fullsized":"2024-03-06T07:24:00.431173+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:26.356183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448151+0000","last_change":"2024-03-06T07:24:00.451006+0000","last_active":"2024-03-06T07:24:01.448151+0000","last_peered":"2024-03-06T07:24:01.448151+0000","last_clean":"2024-03-06T07:24:01.448151+0000","last_became_active":"2024-03-06T07:24:00.450626+0000","last_became_peered":"2024-03-06T07:24:00.450626+0000","last_unstale":"2024-03-06T07:24:01.448151+0000","last_undegraded":"2024-03-06T07:24:01.448151+0000","last_fullsized":"2024-03-06T07:24:01.448151+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:14.268540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449005+0000","last_change":"2024-03-06T07:23:59.437649+0000","last_active":"2024-03-06T07:24:01.449005+0000","last_peered":"2024-03-06T07:24:01.449005+0000","last_clean":"2024-03-06T07:24:01.449005+0000","last_became_active":"2024-03-06T07:23:59.437377+0000","last_became_peered":"2024-03-06T07:23:59.437377+0000","last_unstale":"2024-03-06T07:24:01.449005+0000","last_undegraded":"2024-03-06T07:24:01.449005+0000","last_fullsized":"2024-03-06T07:24:01.449005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587470+0000","last_change":"2024-03-06T07:24:00.446910+0000","last_active":"2024-03-06T07:24:01.587470+0000","last_peered":"2024-03-06T07:24:01.587470+0000","last_clean":"2024-03-06T07:24:01.587470+0000","last_became_active":"2024-03-06T07:24:00.445795+0000","last_became_peered":"2024-03-06T07:24:00.445795+0000","last_unstale":"2024-03-06T07:24:01.587470+0000","last_undegraded":"2024-03-06T07:24:01.587470+0000","last_fullsized":"2024-03-06T07:24:01.587470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":22,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587273+0000","last_change":"2024-03-06T07:23:59.442282+0000","last_active":"2024-03-06T07:24:01.587273+0000","last_peered":"2024-03-06T07:24:01.587273+0000","last_clean":"2024-03-06T07:24:01.587273+0000","last_became_active":"2024-03-06T07:23:59.441992+0000","last_became_peered":"2024-03-06T07:23:59.441992+0000","last_unstale":"2024-03-06T07:24:01.587273+0000","last_undegraded":"2024-03-06T07:24:01.587273+0000","last_fullsized":"2024-03-06T07:24:01.587273+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:12.239306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587803+0000","last_change":"2024-03-06T07:24:00.448694+0000","last_active":"2024-03-06T07:24:01.587803+0000","last_peered":"2024-03-06T07:24:01.587803+0000","last_clean":"2024-03-06T07:24:01.587803+0000","last_became_active":"2024-03-06T07:24:00.448369+0000","last_became_peered":"2024-03-06T07:24:00.448369+0000","last_unstale":"2024-03-06T07:24:01.587803+0000","last_undegraded":"2024-03-06T07:24:01.587803+0000","last_fullsized":"2024-03-06T07:24:01.587803+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.121640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460842+0000","last_change":"2024-03-06T07:23:59.442248+0000","last_active":"2024-03-06T07:24:00.460842+0000","last_peered":"2024-03-06T07:24:00.460842+0000","last_clean":"2024-03-06T07:24:00.460842+0000","last_became_active":"2024-03-06T07:23:59.441980+0000","last_became_peered":"2024-03-06T07:23:59.441980+0000","last_unstale":"2024-03-06T07:24:00.460842+0000","last_undegraded":"2024-03-06T07:24:00.460842+0000","last_fullsized":"2024-03-06T07:24:00.460842+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:42.855432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448632+0000","last_change":"2024-03-06T07:24:00.446282+0000","last_active":"2024-03-06T07:24:01.448632+0000","last_peered":"2024-03-06T07:24:01.448632+0000","last_clean":"2024-03-06T07:24:01.448632+0000","last_became_active":"2024-03-06T07:24:00.445994+0000","last_became_peered":"2024-03-06T07:24:00.445994+0000","last_unstale":"2024-03-06T07:24:01.448632+0000","last_undegraded":"2024-03-06T07:24:01.448632+0000","last_fullsized":"2024-03-06T07:24:01.448632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460629+0000","last_change":"2024-03-06T07:23:59.440130+0000","last_active":"2024-03-06T07:24:00.460629+0000","last_peered":"2024-03-06T07:24:00.460629+0000","last_clean":"2024-03-06T07:24:00.460629+0000","last_became_active":"2024-03-06T07:23:59.439642+0000","last_became_peered":"2024-03-06T07:23:59.439642+0000","last_unstale":"2024-03-06T07:24:00.460629+0000","last_undegraded":"2024-03-06T07:24:00.460629+0000","last_fullsized":"2024-03-06T07:24:00.460629+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.817188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452262+0000","last_change":"2024-03-06T07:24:00.445971+0000","last_active":"2024-03-06T07:24:00.452262+0000","last_peered":"2024-03-06T07:24:00.452262+0000","last_clean":"2024-03-06T07:24:00.452262+0000","last_became_active":"2024-03-06T07:24:00.445707+0000","last_became_peered":"2024-03-06T07:24:00.445707+0000","last_unstale":"2024-03-06T07:24:00.452262+0000","last_undegraded":"2024-03-06T07:24:00.452262+0000","last_fullsized":"2024-03-06T07:24:00.452262+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:26.491484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448963+0000","last_change":"2024-03-06T07:23:59.438479+0000","last_active":"2024-03-06T07:24:01.448963+0000","last_peered":"2024-03-06T07:24:01.448963+0000","last_clean":"2024-03-06T07:24:01.448963+0000","last_became_active":"2024-03-06T07:23:59.438153+0000","last_became_peered":"2024-03-06T07:23:59.438153+0000","last_unstale":"2024-03-06T07:24:01.448963+0000","last_undegraded":"2024-03-06T07:24:01.448963+0000","last_fullsized":"2024-03-06T07:24:01.448963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587879+0000","last_change":"2024-03-06T07:24:00.448405+0000","last_active":"2024-03-06T07:24:01.587879+0000","last_peered":"2024-03-06T07:24:01.587879+0000","last_clean":"2024-03-06T07:24:01.587879+0000","last_became_active":"2024-03-06T07:24:00.448018+0000","last_became_peered":"2024-03-06T07:24:00.448018+0000","last_unstale":"2024-03-06T07:24:01.587879+0000","last_undegraded":"2024-03-06T07:24:01.587879+0000","last_fullsized":"2024-03-06T07:24:01.587879+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:26.491484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.438159+0000","last_change":"2024-03-06T07:23:59.440654+0000","last_active":"2024-03-06T07:24:00.438159+0000","last_peered":"2024-03-06T07:24:00.438159+0000","last_clean":"2024-03-06T07:24:00.438159+0000","last_became_active":"2024-03-06T07:23:59.440414+0000","last_became_peered":"2024-03-06T07:23:59.440414+0000","last_unstale":"2024-03-06T07:24:00.438159+0000","last_undegraded":"2024-03-06T07:24:00.438159+0000","last_fullsized":"2024-03-06T07:24:00.438159+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:21.416548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448400+0000","last_change":"2024-03-06T07:24:00.447077+0000","last_active":"2024-03-06T07:24:01.448400+0000","last_peered":"2024-03-06T07:24:01.448400+0000","last_clean":"2024-03-06T07:24:01.448400+0000","last_became_active":"2024-03-06T07:24:00.446780+0000","last_became_peered":"2024-03-06T07:24:00.446780+0000","last_unstale":"2024-03-06T07:24:01.448400+0000","last_undegraded":"2024-03-06T07:24:01.448400+0000","last_fullsized":"2024-03-06T07:24:01.448400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460124+0000","last_change":"2024-03-06T07:23:59.440760+0000","last_active":"2024-03-06T07:24:00.460124+0000","last_peered":"2024-03-06T07:24:00.460124+0000","last_clean":"2024-03-06T07:24:00.460124+0000","last_became_active":"2024-03-06T07:23:59.440490+0000","last_became_peered":"2024-03-06T07:23:59.440490+0000","last_unstale":"2024-03-06T07:24:00.460124+0000","last_undegraded":"2024-03-06T07:24:00.460124+0000","last_fullsized":"2024-03-06T07:24:00.460124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:36.338297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.453192+0000","last_change":"2024-03-06T07:24:00.447433+0000","last_active":"2024-03-06T07:24:00.453192+0000","last_peered":"2024-03-06T07:24:00.453192+0000","last_clean":"2024-03-06T07:24:00.453192+0000","last_became_active":"2024-03-06T07:24:00.447195+0000","last_became_peered":"2024-03-06T07:24:00.447195+0000","last_unstale":"2024-03-06T07:24:00.453192+0000","last_undegraded":"2024-03-06T07:24:00.453192+0000","last_fullsized":"2024-03-06T07:24:00.453192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:43.865434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.434712+0000","last_change":"2024-03-06T07:23:59.439766+0000","last_active":"2024-03-06T07:24:00.434712+0000","last_peered":"2024-03-06T07:24:00.434712+0000","last_clean":"2024-03-06T07:24:00.434712+0000","last_became_active":"2024-03-06T07:23:59.439309+0000","last_became_peered":"2024-03-06T07:23:59.439309+0000","last_unstale":"2024-03-06T07:24:00.434712+0000","last_undegraded":"2024-03-06T07:24:00.434712+0000","last_fullsized":"2024-03-06T07:24:00.434712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:32.312900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455249+0000","last_change":"2024-03-06T07:24:00.450031+0000","last_active":"2024-03-06T07:24:00.455249+0000","last_peered":"2024-03-06T07:24:00.455249+0000","last_clean":"2024-03-06T07:24:00.455249+0000","last_became_active":"2024-03-06T07:24:00.449743+0000","last_became_peered":"2024-03-06T07:24:00.449743+0000","last_unstale":"2024-03-06T07:24:00.455249+0000","last_undegraded":"2024-03-06T07:24:00.455249+0000","last_fullsized":"2024-03-06T07:24:00.455249+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448089+0000","last_change":"2024-03-06T07:23:59.438050+0000","last_active":"2024-03-06T07:24:01.448089+0000","last_peered":"2024-03-06T07:24:01.448089+0000","last_clean":"2024-03-06T07:24:01.448089+0000","last_became_active":"2024-03-06T07:23:59.437721+0000","last_became_peered":"2024-03-06T07:23:59.437721+0000","last_unstale":"2024-03-06T07:24:01.448089+0000","last_undegraded":"2024-03-06T07:24:01.448089+0000","last_fullsized":"2024-03-06T07:24:01.448089+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.057781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448906+0000","last_change":"2024-03-06T07:24:00.445997+0000","last_active":"2024-03-06T07:24:01.448906+0000","last_peered":"2024-03-06T07:24:01.448906+0000","last_clean":"2024-03-06T07:24:01.448906+0000","last_became_active":"2024-03-06T07:24:00.445250+0000","last_became_peered":"2024-03-06T07:24:00.445250+0000","last_unstale":"2024-03-06T07:24:01.448906+0000","last_undegraded":"2024-03-06T07:24:01.448906+0000","last_fullsized":"2024-03-06T07:24:01.448906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455796+0000","last_change":"2024-03-06T07:23:59.435108+0000","last_active":"2024-03-06T07:24:00.455796+0000","last_peered":"2024-03-06T07:24:00.455796+0000","last_clean":"2024-03-06T07:24:00.455796+0000","last_became_active":"2024-03-06T07:23:59.434845+0000","last_became_peered":"2024-03-06T07:23:59.434845+0000","last_unstale":"2024-03-06T07:24:00.455796+0000","last_undegraded":"2024-03-06T07:24:00.455796+0000","last_fullsized":"2024-03-06T07:24:00.455796+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:35.611916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448816+0000","last_change":"2024-03-06T07:24:00.449232+0000","last_active":"2024-03-06T07:24:01.448816+0000","last_peered":"2024-03-06T07:24:01.448816+0000","last_clean":"2024-03-06T07:24:01.448816+0000","last_became_active":"2024-03-06T07:24:00.448976+0000","last_became_peered":"2024-03-06T07:24:00.448976+0000","last_unstale":"2024-03-06T07:24:01.448816+0000","last_undegraded":"2024-03-06T07:24:01.448816+0000","last_fullsized":"2024-03-06T07:24:01.448816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.446278+0000","last_change":"2024-03-06T07:23:59.439820+0000","last_active":"2024-03-06T07:24:01.446278+0000","last_peered":"2024-03-06T07:24:01.446278+0000","last_clean":"2024-03-06T07:24:01.446278+0000","last_became_active":"2024-03-06T07:23:59.439328+0000","last_became_peered":"2024-03-06T07:23:59.439328+0000","last_unstale":"2024-03-06T07:24:01.446278+0000","last_undegraded":"2024-03-06T07:24:01.446278+0000","last_fullsized":"2024-03-06T07:24:01.446278+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:12.636982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448440+0000","last_change":"2024-03-06T07:24:00.444894+0000","last_active":"2024-03-06T07:24:00.448440+0000","last_peered":"2024-03-06T07:24:00.448440+0000","last_clean":"2024-03-06T07:24:00.448440+0000","last_became_active":"2024-03-06T07:24:00.444614+0000","last_became_peered":"2024-03-06T07:24:00.444614+0000","last_unstale":"2024-03-06T07:24:00.448440+0000","last_undegraded":"2024-03-06T07:24:00.448440+0000","last_fullsized":"2024-03-06T07:24:00.448440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:02.387037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.431703+0000","last_change":"2024-03-06T07:23:59.440546+0000","last_active":"2024-03-06T07:24:00.431703+0000","last_peered":"2024-03-06T07:24:00.431703+0000","last_clean":"2024-03-06T07:24:00.431703+0000","last_became_active":"2024-03-06T07:23:59.440268+0000","last_became_peered":"2024-03-06T07:23:59.440268+0000","last_unstale":"2024-03-06T07:24:00.431703+0000","last_undegraded":"2024-03-06T07:24:00.431703+0000","last_fullsized":"2024-03-06T07:24:00.431703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:32.312900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455109+0000","last_change":"2024-03-06T07:24:00.448723+0000","last_active":"2024-03-06T07:24:00.455109+0000","last_peered":"2024-03-06T07:24:00.455109+0000","last_clean":"2024-03-06T07:24:00.455109+0000","last_became_active":"2024-03-06T07:24:00.448168+0000","last_became_peered":"2024-03-06T07:24:00.448168+0000","last_unstale":"2024-03-06T07:24:00.455109+0000","last_undegraded":"2024-03-06T07:24:00.455109+0000","last_fullsized":"2024-03-06T07:24:00.455109+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:48.961505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448645+0000","last_change":"2024-03-06T07:23:59.440547+0000","last_active":"2024-03-06T07:24:01.448645+0000","last_peered":"2024-03-06T07:24:01.448645+0000","last_clean":"2024-03-06T07:24:01.448645+0000","last_became_active":"2024-03-06T07:23:59.440261+0000","last_became_peered":"2024-03-06T07:23:59.440261+0000","last_unstale":"2024-03-06T07:24:01.448645+0000","last_undegraded":"2024-03-06T07:24:01.448645+0000","last_fullsized":"2024-03-06T07:24:01.448645+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.817188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452240+0000","last_change":"2024-03-06T07:24:00.450414+0000","last_active":"2024-03-06T07:24:00.452240+0000","last_peered":"2024-03-06T07:24:00.452240+0000","last_clean":"2024-03-06T07:24:00.452240+0000","last_became_active":"2024-03-06T07:24:00.450152+0000","last_became_peered":"2024-03-06T07:24:00.450152+0000","last_unstale":"2024-03-06T07:24:00.452240+0000","last_undegraded":"2024-03-06T07:24:00.452240+0000","last_fullsized":"2024-03-06T07:24:00.452240+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:59.617656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.435133+0000","last_change":"2024-03-06T07:23:59.436261+0000","last_active":"2024-03-06T07:24:00.435133+0000","last_peered":"2024-03-06T07:24:00.435133+0000","last_clean":"2024-03-06T07:24:00.435133+0000","last_became_active":"2024-03-06T07:23:59.436007+0000","last_became_peered":"2024-03-06T07:23:59.436007+0000","last_unstale":"2024-03-06T07:24:00.435133+0000","last_undegraded":"2024-03-06T07:24:00.435133+0000","last_fullsized":"2024-03-06T07:24:00.435133+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:05.778180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587653+0000","last_change":"2024-03-06T07:24:00.441206+0000","last_active":"2024-03-06T07:24:01.587653+0000","last_peered":"2024-03-06T07:24:01.587653+0000","last_clean":"2024-03-06T07:24:01.587653+0000","last_became_active":"2024-03-06T07:24:00.440923+0000","last_became_peered":"2024-03-06T07:24:00.440923+0000","last_unstale":"2024-03-06T07:24:01.587653+0000","last_undegraded":"2024-03-06T07:24:01.587653+0000","last_fullsized":"2024-03-06T07:24:01.587653+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449011+0000","last_change":"2024-03-06T07:23:59.443108+0000","last_active":"2024-03-06T07:24:01.449011+0000","last_peered":"2024-03-06T07:24:01.449011+0000","last_clean":"2024-03-06T07:24:01.449011+0000","last_became_active":"2024-03-06T07:23:59.442847+0000","last_became_peered":"2024-03-06T07:23:59.442847+0000","last_unstale":"2024-03-06T07:24:01.449011+0000","last_undegraded":"2024-03-06T07:24:01.449011+0000","last_fullsized":"2024-03-06T07:24:01.449011+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:09.321526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.447348+0000","last_change":"2024-03-06T07:24:00.441365+0000","last_active":"2024-03-06T07:24:00.447348+0000","last_peered":"2024-03-06T07:24:00.447348+0000","last_clean":"2024-03-06T07:24:00.447348+0000","last_became_active":"2024-03-06T07:24:00.440667+0000","last_became_peered":"2024-03-06T07:24:00.440667+0000","last_unstale":"2024-03-06T07:24:00.447348+0000","last_undegraded":"2024-03-06T07:24:00.447348+0000","last_fullsized":"2024-03-06T07:24:00.447348+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:30.806654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.435506+0000","last_change":"2024-03-06T07:23:59.438513+0000","last_active":"2024-03-06T07:24:00.435506+0000","last_peered":"2024-03-06T07:24:00.435506+0000","last_clean":"2024-03-06T07:24:00.435506+0000","last_became_active":"2024-03-06T07:23:59.438207+0000","last_became_peered":"2024-03-06T07:23:59.438207+0000","last_unstale":"2024-03-06T07:24:00.435506+0000","last_undegraded":"2024-03-06T07:24:00.435506+0000","last_fullsized":"2024-03-06T07:24:00.435506+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.892786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452382+0000","last_change":"2024-03-06T07:24:00.440384+0000","last_active":"2024-03-06T07:24:00.452382+0000","last_peered":"2024-03-06T07:24:00.452382+0000","last_clean":"2024-03-06T07:24:00.452382+0000","last_became_active":"2024-03-06T07:24:00.440128+0000","last_became_peered":"2024-03-06T07:24:00.440128+0000","last_unstale":"2024-03-06T07:24:00.452382+0000","last_undegraded":"2024-03-06T07:24:00.452382+0000","last_fullsized":"2024-03-06T07:24:00.452382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.435351+0000","last_change":"2024-03-06T07:23:59.437764+0000","last_active":"2024-03-06T07:24:00.435351+0000","last_peered":"2024-03-06T07:24:00.435351+0000","last_clean":"2024-03-06T07:24:00.435351+0000","last_became_active":"2024-03-06T07:23:59.437126+0000","last_became_peered":"2024-03-06T07:23:59.437126+0000","last_unstale":"2024-03-06T07:24:00.435351+0000","last_undegraded":"2024-03-06T07:24:00.435351+0000","last_fullsized":"2024-03-06T07:24:00.435351+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:46.102824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452476+0000","last_change":"2024-03-06T07:24:00.440544+0000","last_active":"2024-03-06T07:24:00.452476+0000","last_peered":"2024-03-06T07:24:00.452476+0000","last_clean":"2024-03-06T07:24:00.452476+0000","last_became_active":"2024-03-06T07:24:00.440284+0000","last_became_peered":"2024-03-06T07:24:00.440284+0000","last_unstale":"2024-03-06T07:24:00.452476+0000","last_undegraded":"2024-03-06T07:24:00.452476+0000","last_fullsized":"2024-03-06T07:24:00.452476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.431265+0000","last_change":"2024-03-06T07:23:59.444024+0000","last_active":"2024-03-06T07:24:00.431265+0000","last_peered":"2024-03-06T07:24:00.431265+0000","last_clean":"2024-03-06T07:24:00.431265+0000","last_became_active":"2024-03-06T07:23:59.443744+0000","last_became_peered":"2024-03-06T07:23:59.443744+0000","last_unstale":"2024-03-06T07:24:00.431265+0000","last_undegraded":"2024-03-06T07:24:00.431265+0000","last_fullsized":"2024-03-06T07:24:00.431265+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:06.553338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:01.587213+0000","last_peered":"2024-03-06T07:24:01.587213+0000","last_clean":"2024-03-06T07:24:01.587213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:01.587213+0000","last_undegraded":"2024-03-06T07:24:01.587213+0000","last_fullsized":"2024-03-06T07:24:01.587213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1298432,"data_stored":879450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":696320,"data_stored":288922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":295874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":720896,"data_stored":296834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":288978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":288978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":289012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":881484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:04.202 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 Processor -- start 2024-03-06T07:24:04.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 -- start start 2024-03-06T07:24:04.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6a74162970 0x7f6a740d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74161cf0 0x7f6a74162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a740d22c0 con 0x7f6a740b4460 2024-03-06T07:24:04.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a74161b00 con 0x7f6a74162970 2024-03-06T07:24:04.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.335+0000 7f6a78d20640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740cecc0 con 0x7f6a74161cf0 2024-03-06T07:24:04.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a7259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6a740b4460 0x7f6a740b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46410/0 (socket says 172.21.15.72:46410) 2024-03-06T07:24:04.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a7259c640 1 -- 172.21.15.72:0/244205914 learned_addr learned my addr 172.21.15.72:0/244205914 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:04.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a6bfff640 1 --2- 172.21.15.72:0/244205914 >> v2:172.21.15.146:3300/0 conn(0x7f6a74162970 0x7f6a740d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a72d9d640 1 --2- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74161cf0 0x7f6a74162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a71d9b640 1 -- 172.21.15.72:0/244205914 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517140366 0 0) 0x7f6a740d22c0 con 0x7f6a740b4460 2024-03-06T07:24:04.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a71d9b640 1 -- 172.21.15.72:0/244205914 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a60003520 con 0x7f6a740b4460 2024-03-06T07:24:04.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a6bfff640 1 -- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74161cf0 msgr2=0x7f6a74162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a6bfff640 1 --2- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74161cf0 0x7f6a74162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a71d9b640 1 -- 172.21.15.72:0/244205914 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237113168 0 0) 0x7f6a60003520 con 0x7f6a740b4460 2024-03-06T07:24:04.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a6bfff640 1 -- 172.21.15.72:0/244205914 >> v1:172.21.15.72:6789/0 conn(0x7f6a740b4460 legacy=0x7f6a740b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.336+0000 7f6a6bfff640 1 -- 172.21.15.72:0/244205914 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a740d25a0 con 0x7f6a74162970 2024-03-06T07:24:04.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.337+0000 7f6a6bfff640 1 --2- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74162970 0x7f6a740d4800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097a0 tx=0x7f6a5c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e68ae9c3b9c9d86 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a71d9b640 1 -- 172.21.15.72:0/244205914 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c037070 con 0x7f6a74162970 2024-03-06T07:24:04.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a71d9b640 1 -- 172.21.15.72:0/244205914 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c030510 con 0x7f6a74162970 2024-03-06T07:24:04.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a71d9b640 1 -- 172.21.15.72:0/244205914 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64001670 con 0x7f6a74162970 2024-03-06T07:24:04.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 -- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74162970 msgr2=0x7f6a740d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 --2- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74162970 0x7f6a740d4800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097a0 tx=0x7f6a5c02b790 comp rx=0 tx=0).stop 2024-03-06T07:24:04.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 -- 172.21.15.72:0/244205914 shutdown_connections 2024-03-06T07:24:04.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 --2- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74161cf0 0x7f6a74162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 --2- 172.21.15.72:0/244205914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74162970 0x7f6a740d4800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 -- 172.21.15.72:0/244205914 >> 172.21.15.72:0/244205914 conn(0x7f6a740d3ce0 msgr2=0x7f6a740d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 -- 172.21.15.72:0/244205914 shutdown_connections 2024-03-06T07:24:04.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.338+0000 7f6a78d20640 1 -- 172.21.15.72:0/244205914 wait complete. 2024-03-06T07:24:04.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.339+0000 7f6a78d20640 1 Processor -- start 2024-03-06T07:24:04.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 -- start start 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 0x7f6a74155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74161cf0 0x7f6a740b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74162970 0x7f6a740b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740ce950 con 0x7f6a740b4460 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740d2b60 con 0x7f6a74161cf0 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a78d20640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a740b4840 con 0x7f6a74162970 2024-03-06T07:24:04.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a7259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 0x7f6a74155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a7259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 0x7f6a74155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33956/0 (socket says 172.21.15.72:33956) 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.340+0000 7f6a7259c640 1 -- 172.21.15.72:0/1136995268 learned_addr learned my addr 172.21.15.72:0/1136995268 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a6bfff640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74161cf0 0x7f6a740b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a7259c640 1 -- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74162970 msgr2=0x7f6a740b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a7259c640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74162970 0x7f6a740b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a7259c640 1 -- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74161cf0 msgr2=0x7f6a740b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a7259c640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74161cf0 0x7f6a740b1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a7259c640 1 -- 172.21.15.72:0/1136995268 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74156320 con 0x7f6a740b4460 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a7259c640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 0x7f6a74155de0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c750 tx=0x7f6a6400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64005ad0 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64005c90 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a740b6c20 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640072b0 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.341+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a740b9060 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.342+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7404f330 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.342+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6a64007450 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.343+0000 7f6a6affd640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a5406b4d0 0x7f6a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.343+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a64089fa0 con 0x7f6a740b4460 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.343+0000 7f6a6bfff640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a5406b4d0 0x7f6a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.343+0000 7f6a6bfff640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a5406b4d0 0x7f6a5406d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c034000 tx=0x7f6a5c02f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.347+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6a74156320 con 0x7f6a740b4460 2024-03-06T07:24:04.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.718+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6a740c77b0 con 0x7f6a740b4460 2024-03-06T07:24:04.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.719+0000 7f6a6affd640 1 -- 172.21.15.72:0/1136995268 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7f6a64058da0 con 0x7f6a740b4460 2024-03-06T07:24:04.721 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:04.721 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":159,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709696,"num_in_osds":8,"osd_in_since":1709709687,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":262303744,"bytes_avail":772831809536,"bytes_total":773094113280,"write_bytes_sec":2582,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":27,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":6057}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:23:11.962885+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.720+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a5406b4d0 msgr2=0x7f6a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.720+0000 7f6a78d20640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a5406b4d0 0x7f6a5406d990 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c034000 tx=0x7f6a5c02f000 comp rx=0 tx=0).stop 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.720+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 msgr2=0x7f6a74155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.720+0000 7f6a78d20640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 0x7f6a74155de0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c750 tx=0x7f6a6400cc10 comp rx=0 tx=0).stop 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 shutdown_connections 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6a5406b4d0 0x7f6a5406d990 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a74162970 0x7f6a740b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a74161cf0 0x7f6a740b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 --2- 172.21.15.72:0/1136995268 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a740b4460 0x7f6a74155de0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 >> 172.21.15.72:0/1136995268 conn(0x7f6a740d3ce0 msgr2=0x7f6a740d2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 shutdown_connections 2024-03-06T07:24:04.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.721+0000 7f6a78d20640 1 -- 172.21.15.72:0/1136995268 wait complete. 2024-03-06T07:24:04.733 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:04.734 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:04.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.832+0000 7fd0dc8c9640 1 Processor -- start 2024-03-06T07:24:04.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.832+0000 7fd0dc8c9640 1 -- start start 2024-03-06T07:24:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.833+0000 7fd0dc8c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0d80d22b0 0x7fd0d80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.833+0000 7fd0dc8c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d80b3430 con 0x7fd0d815dfe0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.833+0000 7fd0dc8c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d80b3710 con 0x7fd0d80b41f0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.833+0000 7fd0dc8c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d80d2c50 con 0x7fd0d80d22b0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd0d815dfe0 0x7fd0d80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46412/0 (socket says 172.21.15.72:46412) 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d659c640 1 -- 172.21.15.72:0/3950940003 learned_addr learned my addr 172.21.15.72:0/3950940003 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207043165 0 0) 0x7fd0d80b3430 con 0x7fd0d815dfe0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0c4003520 con 0x7fd0d815dfe0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0cffff640 1 --2- 172.21.15.72:0/3950940003 >> v2:172.21.15.146:3300/0 conn(0x7fd0d80d22b0 0x7fd0d80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104816166 0 0) 0x7fd0c4003520 con 0x7fd0d815dfe0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 >> v2:172.21.15.146:3300/0 conn(0x7fd0d80d22b0 msgr2=0x7fd0d80d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 --2- 172.21.15.72:0/3950940003 >> v2:172.21.15.146:3300/0 conn(0x7fd0d80d22b0 0x7fd0d80d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 >> v1:172.21.15.146:6789/0 conn(0x7fd0d80b41f0 legacy=0x7fd0d80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80b39f0 con 0x7fd0d815dfe0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd0c8001670 con 0x7fd0d815dfe0 2024-03-06T07:24:04.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 >> v1:172.21.15.72:6789/0 conn(0x7fd0d815dfe0 legacy=0x7fd0d80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0c40039b0 0x7fd0c4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0c40052e0 0x7fd0c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c400c1b0 con 0x7fd0c4004800 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c400c360 con 0x7fd0c40052e0 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0c400c560 con 0x7fd0c40039b0 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0cffff640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d6d9d640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0c40052e0 0x7fd0c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.834+0000 7fd0d659c640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0c40039b0 0x7fd0c4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0cffff640 1 -- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0c40039b0 msgr2=0x7fd0c4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0cffff640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0c40039b0 0x7fd0c4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0cffff640 1 -- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0c40052e0 msgr2=0x7fd0c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0cffff640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0c40052e0 0x7fd0c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0cffff640 1 -- 172.21.15.72:0/3950940003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c4003520 con 0x7fd0c4004800 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0d6d9d640 1 -- 172.21.15.72:0/3950940003 reap_dead start 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0cffff640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0c4004c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000a970 tx=0x7fd0c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdcf51bb94c3f64c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.835+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0035020 con 0x7fd0c4004800 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c0031680 con 0x7fd0c4004800 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0d5d9b640 1 -- 172.21.15.72:0/3950940003 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c000a570 con 0x7fd0c4004800 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/3950940003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 msgr2=0x7fd0c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0c4004c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000a970 tx=0x7fd0c4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/3950940003 shutdown_connections 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/3950940003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0c4004c80 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/3950940003 >> 172.21.15.72:0/3950940003 conn(0x7fd0d80d3cd0 msgr2=0x7fd0d80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/3950940003 shutdown_connections 2024-03-06T07:24:04.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.836+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/3950940003 wait complete. 2024-03-06T07:24:04.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 Processor -- start 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 -- start start 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0d8152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d80ccca0 0x7fd0d80cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d8160a30 0x7fd0d8160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80b3d60 con 0x7fd0c4004800 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80b31e0 con 0x7fd0d80ccca0 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.838+0000 7fd0dc8c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d815e3c0 con 0x7fd0d8160a30 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0d8152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0d8152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33976/0 (socket says 172.21.15.72:33976) 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 -- 172.21.15.72:0/4118098959 learned_addr learned my addr 172.21.15.72:0/4118098959 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d6d9d640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d8160a30 0x7fd0d8160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0cffff640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d80ccca0 0x7fd0d80cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 -- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d8160a30 msgr2=0x7fd0d8160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d8160a30 0x7fd0d8160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 -- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d80ccca0 msgr2=0x7fd0d80cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d80ccca0 0x7fd0d80cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 -- 172.21.15.72:0/4118098959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d81532e0 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d6d9d640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d8160a30 0x7fd0d8160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0cffff640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d80ccca0 0x7fd0d80cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0d659c640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0d8152da0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80072c0 tx=0x7fd0c8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c801a070 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c800f960 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800fd30 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d80cdb60 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.839+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d80cb220 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.840+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d804f330 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.841+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0c8014450 con 0x7fd0c4004800 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.841+0000 7fd0ceffd640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0b806b500 0x7fd0b806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:04.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.841+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0c8089b20 con 0x7fd0c4004800 2024-03-06T07:24:04.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.841+0000 7fd0cffff640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0b806b500 0x7fd0b806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:04.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.842+0000 7fd0cffff640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0b806b500 0x7fd0b806d9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0036000 tx=0x7fd0c003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:04.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:04.846+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0d81532e0 con 0x7fd0c4004800 2024-03-06T07:24:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.020+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0d81e6b80 con 0x7fd0b806b500 2024-03-06T07:24:05.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.026+0000 7fd0ceffd640 1 -- 172.21.15.72:0/4118098959 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251920 (secure 0 0 0) 0x7fd0d81e6b80 con 0x7fd0b806b500 2024-03-06T07:24:05.031 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:05.032 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:05.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0b806b500 msgr2=0x7fd0b806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0b806b500 0x7fd0b806d9c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0036000 tx=0x7fd0c003c040 comp rx=0 tx=0).stop 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 msgr2=0x7fd0d8152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0d8152da0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80072c0 tx=0x7fd0c8007780 comp rx=0 tx=0).stop 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 shutdown_connections 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0b806b500 0x7fd0b806d9c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d8160a30 0x7fd0d8160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d80ccca0 0x7fd0d80cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 --2- 172.21.15.72:0/4118098959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0c4004800 0x7fd0d8152da0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 >> 172.21.15.72:0/4118098959 conn(0x7fd0d80d3cd0 msgr2=0x7fd0d806c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 shutdown_connections 2024-03-06T07:24:05.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:05.029+0000 7fd0dc8c9640 1 -- 172.21.15.72:0/4118098959 wait complete. 2024-03-06T07:24:05.045 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-03-06T07:24:03.971637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256156,"kb_used_data":6688,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718564,"statfs":{"total":773094113280,"available":772831809536,"internally_reserved":0,"allocated":6848512,"data_stored":3509532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.551913"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449532+0000","last_change":"2024-03-06T07:24:00.439946+0000","last_active":"2024-03-06T07:24:01.449532+0000","last_peered":"2024-03-06T07:24:01.449532+0000","last_clean":"2024-03-06T07:24:01.449532+0000","last_became_active":"2024-03-06T07:24:00.439682+0000","last_became_peered":"2024-03-06T07:24:00.439682+0000","last_unstale":"2024-03-06T07:24:01.449532+0000","last_undegraded":"2024-03-06T07:24:01.449532+0000","last_fullsized":"2024-03-06T07:24:01.449532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:48.774758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452451+0000","last_change":"2024-03-06T07:24:00.446637+0000","last_active":"2024-03-06T07:24:00.452451+0000","last_peered":"2024-03-06T07:24:00.452451+0000","last_clean":"2024-03-06T07:24:00.452451+0000","last_became_active":"2024-03-06T07:24:00.446373+0000","last_became_peered":"2024-03-06T07:24:00.446373+0000","last_unstale":"2024-03-06T07:24:00.452451+0000","last_undegraded":"2024-03-06T07:24:00.452451+0000","last_fullsized":"2024-03-06T07:24:00.452451+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:11.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587704+0000","last_change":"2024-03-06T07:24:00.439609+0000","last_active":"2024-03-06T07:24:01.587704+0000","last_peered":"2024-03-06T07:24:01.587704+0000","last_clean":"2024-03-06T07:24:01.587704+0000","last_became_active":"2024-03-06T07:24:00.439333+0000","last_became_peered":"2024-03-06T07:24:00.439333+0000","last_unstale":"2024-03-06T07:24:01.587704+0000","last_undegraded":"2024-03-06T07:24:01.587704+0000","last_fullsized":"2024-03-06T07:24:01.587704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448757+0000","last_change":"2024-03-06T07:24:00.445347+0000","last_active":"2024-03-06T07:24:01.448757+0000","last_peered":"2024-03-06T07:24:01.448757+0000","last_clean":"2024-03-06T07:24:01.448757+0000","last_became_active":"2024-03-06T07:24:00.445031+0000","last_became_peered":"2024-03-06T07:24:00.445031+0000","last_unstale":"2024-03-06T07:24:01.448757+0000","last_undegraded":"2024-03-06T07:24:01.448757+0000","last_fullsized":"2024-03-06T07:24:01.448757+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449309+0000","last_change":"2024-03-06T07:24:00.441074+0000","last_active":"2024-03-06T07:24:01.449309+0000","last_peered":"2024-03-06T07:24:01.449309+0000","last_clean":"2024-03-06T07:24:01.449309+0000","last_became_active":"2024-03-06T07:24:00.440804+0000","last_became_peered":"2024-03-06T07:24:00.440804+0000","last_unstale":"2024-03-06T07:24:01.449309+0000","last_undegraded":"2024-03-06T07:24:01.449309+0000","last_fullsized":"2024-03-06T07:24:01.449309+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:39.417435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.456804+0000","last_change":"2024-03-06T07:24:00.453304+0000","last_active":"2024-03-06T07:24:00.456804+0000","last_peered":"2024-03-06T07:24:00.456804+0000","last_clean":"2024-03-06T07:24:00.456804+0000","last_became_active":"2024-03-06T07:24:00.452795+0000","last_became_peered":"2024-03-06T07:24:00.452795+0000","last_unstale":"2024-03-06T07:24:00.456804+0000","last_undegraded":"2024-03-06T07:24:00.456804+0000","last_fullsized":"2024-03-06T07:24:00.456804+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448469+0000","last_change":"2024-03-06T07:24:00.444902+0000","last_active":"2024-03-06T07:24:01.448469+0000","last_peered":"2024-03-06T07:24:01.448469+0000","last_clean":"2024-03-06T07:24:01.448469+0000","last_became_active":"2024-03-06T07:24:00.444629+0000","last_became_peered":"2024-03-06T07:24:00.444629+0000","last_unstale":"2024-03-06T07:24:01.448469+0000","last_undegraded":"2024-03-06T07:24:01.448469+0000","last_fullsized":"2024-03-06T07:24:01.448469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:54.193912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448269+0000","last_change":"2024-03-06T07:24:00.439953+0000","last_active":"2024-03-06T07:24:00.448269+0000","last_peered":"2024-03-06T07:24:00.448269+0000","last_clean":"2024-03-06T07:24:00.448269+0000","last_became_active":"2024-03-06T07:24:00.439683+0000","last_became_peered":"2024-03-06T07:24:00.439683+0000","last_unstale":"2024-03-06T07:24:00.448269+0000","last_undegraded":"2024-03-06T07:24:00.448269+0000","last_fullsized":"2024-03-06T07:24:00.448269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448813+0000","last_change":"2024-03-06T07:24:00.444198+0000","last_active":"2024-03-06T07:24:01.448813+0000","last_peered":"2024-03-06T07:24:01.448813+0000","last_clean":"2024-03-06T07:24:01.448813+0000","last_became_active":"2024-03-06T07:24:00.443885+0000","last_became_peered":"2024-03-06T07:24:00.443885+0000","last_unstale":"2024-03-06T07:24:01.448813+0000","last_undegraded":"2024-03-06T07:24:01.448813+0000","last_fullsized":"2024-03-06T07:24:01.448813+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:22.426550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448763+0000","last_change":"2024-03-06T07:24:00.444858+0000","last_active":"2024-03-06T07:24:00.448763+0000","last_peered":"2024-03-06T07:24:00.448763+0000","last_clean":"2024-03-06T07:24:00.448763+0000","last_became_active":"2024-03-06T07:24:00.444401+0000","last_became_peered":"2024-03-06T07:24:00.444401+0000","last_unstale":"2024-03-06T07:24:00.448763+0000","last_undegraded":"2024-03-06T07:24:00.448763+0000","last_fullsized":"2024-03-06T07:24:00.448763+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:34:58.913668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.453709+0000","last_change":"2024-03-06T07:24:00.447374+0000","last_active":"2024-03-06T07:24:00.453709+0000","last_peered":"2024-03-06T07:24:00.453709+0000","last_clean":"2024-03-06T07:24:00.453709+0000","last_became_active":"2024-03-06T07:24:00.445602+0000","last_became_peered":"2024-03-06T07:24:00.445602+0000","last_unstale":"2024-03-06T07:24:00.453709+0000","last_undegraded":"2024-03-06T07:24:00.453709+0000","last_fullsized":"2024-03-06T07:24:00.453709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:21.961725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448447+0000","last_change":"2024-03-06T07:24:00.447595+0000","last_active":"2024-03-06T07:24:01.448447+0000","last_peered":"2024-03-06T07:24:01.448447+0000","last_clean":"2024-03-06T07:24:01.448447+0000","last_became_active":"2024-03-06T07:24:00.446729+0000","last_became_peered":"2024-03-06T07:24:00.446729+0000","last_unstale":"2024-03-06T07:24:01.448447+0000","last_undegraded":"2024-03-06T07:24:01.448447+0000","last_fullsized":"2024-03-06T07:24:01.448447+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.447121+0000","last_change":"2024-03-06T07:24:00.444770+0000","last_active":"2024-03-06T07:24:00.447121+0000","last_peered":"2024-03-06T07:24:00.447121+0000","last_clean":"2024-03-06T07:24:00.447121+0000","last_became_active":"2024-03-06T07:24:00.444233+0000","last_became_peered":"2024-03-06T07:24:00.444233+0000","last_unstale":"2024-03-06T07:24:00.447121+0000","last_undegraded":"2024-03-06T07:24:00.447121+0000","last_fullsized":"2024-03-06T07:24:00.447121+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:36.621918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448903+0000","last_change":"2024-03-06T07:24:00.452714+0000","last_active":"2024-03-06T07:24:01.448903+0000","last_peered":"2024-03-06T07:24:01.448903+0000","last_clean":"2024-03-06T07:24:01.448903+0000","last_became_active":"2024-03-06T07:24:00.452377+0000","last_became_peered":"2024-03-06T07:24:00.452377+0000","last_unstale":"2024-03-06T07:24:01.448903+0000","last_undegraded":"2024-03-06T07:24:01.448903+0000","last_fullsized":"2024-03-06T07:24:01.448903+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:30.827190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448497+0000","last_change":"2024-03-06T07:24:00.448276+0000","last_active":"2024-03-06T07:24:01.448497+0000","last_peered":"2024-03-06T07:24:01.448497+0000","last_clean":"2024-03-06T07:24:01.448497+0000","last_became_active":"2024-03-06T07:24:00.447961+0000","last_became_peered":"2024-03-06T07:24:00.447961+0000","last_unstale":"2024-03-06T07:24:01.448497+0000","last_undegraded":"2024-03-06T07:24:01.448497+0000","last_fullsized":"2024-03-06T07:24:01.448497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:57.067784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451491+0000","last_change":"2024-03-06T07:24:00.444103+0000","last_active":"2024-03-06T07:24:00.451491+0000","last_peered":"2024-03-06T07:24:00.451491+0000","last_clean":"2024-03-06T07:24:00.451491+0000","last_became_active":"2024-03-06T07:24:00.443851+0000","last_became_peered":"2024-03-06T07:24:00.443851+0000","last_unstale":"2024-03-06T07:24:00.451491+0000","last_undegraded":"2024-03-06T07:24:00.451491+0000","last_fullsized":"2024-03-06T07:24:00.451491+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:23:35.722017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452542+0000","last_change":"2024-03-06T07:24:00.443021+0000","last_active":"2024-03-06T07:24:00.452542+0000","last_peered":"2024-03-06T07:24:00.452542+0000","last_clean":"2024-03-06T07:24:00.452542+0000","last_became_active":"2024-03-06T07:24:00.442730+0000","last_became_peered":"2024-03-06T07:24:00.442730+0000","last_unstale":"2024-03-06T07:24:00.452542+0000","last_undegraded":"2024-03-06T07:24:00.452542+0000","last_fullsized":"2024-03-06T07:24:00.452542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587794+0000","last_change":"2024-03-06T07:24:00.451841+0000","last_active":"2024-03-06T07:24:01.587794+0000","last_peered":"2024-03-06T07:24:01.587794+0000","last_clean":"2024-03-06T07:24:01.587794+0000","last_became_active":"2024-03-06T07:24:00.451473+0000","last_became_peered":"2024-03-06T07:24:00.451473+0000","last_unstale":"2024-03-06T07:24:01.587794+0000","last_undegraded":"2024-03-06T07:24:01.587794+0000","last_fullsized":"2024-03-06T07:24:01.587794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:15.579899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449397+0000","last_change":"2024-03-06T07:24:00.439220+0000","last_active":"2024-03-06T07:24:01.449397+0000","last_peered":"2024-03-06T07:24:01.449397+0000","last_clean":"2024-03-06T07:24:01.449397+0000","last_became_active":"2024-03-06T07:24:00.438971+0000","last_became_peered":"2024-03-06T07:24:00.438971+0000","last_unstale":"2024-03-06T07:24:01.449397+0000","last_undegraded":"2024-03-06T07:24:01.449397+0000","last_fullsized":"2024-03-06T07:24:01.449397+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.634491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452285+0000","last_change":"2024-03-06T07:24:00.447150+0000","last_active":"2024-03-06T07:24:00.452285+0000","last_peered":"2024-03-06T07:24:00.452285+0000","last_clean":"2024-03-06T07:24:00.452285+0000","last_became_active":"2024-03-06T07:24:00.446837+0000","last_became_peered":"2024-03-06T07:24:00.446837+0000","last_unstale":"2024-03-06T07:24:00.452285+0000","last_undegraded":"2024-03-06T07:24:00.452285+0000","last_fullsized":"2024-03-06T07:24:00.452285+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:39.577599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587434+0000","last_change":"2024-03-06T07:24:00.442324+0000","last_active":"2024-03-06T07:24:01.587434+0000","last_peered":"2024-03-06T07:24:01.587434+0000","last_clean":"2024-03-06T07:24:01.587434+0000","last_became_active":"2024-03-06T07:24:00.442041+0000","last_became_peered":"2024-03-06T07:24:00.442041+0000","last_unstale":"2024-03-06T07:24:01.587434+0000","last_undegraded":"2024-03-06T07:24:01.587434+0000","last_fullsized":"2024-03-06T07:24:01.587434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451994+0000","last_change":"2024-03-06T07:24:00.442786+0000","last_active":"2024-03-06T07:24:00.451994+0000","last_peered":"2024-03-06T07:24:00.451994+0000","last_clean":"2024-03-06T07:24:00.451994+0000","last_became_active":"2024-03-06T07:24:00.442534+0000","last_became_peered":"2024-03-06T07:24:00.442534+0000","last_unstale":"2024-03-06T07:24:00.451994+0000","last_undegraded":"2024-03-06T07:24:00.451994+0000","last_fullsized":"2024-03-06T07:24:00.451994+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587327+0000","last_change":"2024-03-06T07:24:00.445639+0000","last_active":"2024-03-06T07:24:01.587327+0000","last_peered":"2024-03-06T07:24:01.587327+0000","last_clean":"2024-03-06T07:24:01.587327+0000","last_became_active":"2024-03-06T07:24:00.445190+0000","last_became_peered":"2024-03-06T07:24:00.445190+0000","last_unstale":"2024-03-06T07:24:01.587327+0000","last_undegraded":"2024-03-06T07:24:01.587327+0000","last_fullsized":"2024-03-06T07:24:01.587327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.443994+0000","last_change":"2024-03-06T07:24:00.441727+0000","last_active":"2024-03-06T07:24:00.443994+0000","last_peered":"2024-03-06T07:24:00.443994+0000","last_clean":"2024-03-06T07:24:00.443994+0000","last_became_active":"2024-03-06T07:24:00.440305+0000","last_became_peered":"2024-03-06T07:24:00.440305+0000","last_unstale":"2024-03-06T07:24:00.443994+0000","last_undegraded":"2024-03-06T07:24:00.443994+0000","last_fullsized":"2024-03-06T07:24:00.443994+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:52.902789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449608+0000","last_change":"2024-03-06T07:24:00.446374+0000","last_active":"2024-03-06T07:24:01.449608+0000","last_peered":"2024-03-06T07:24:01.449608+0000","last_clean":"2024-03-06T07:24:01.449608+0000","last_became_active":"2024-03-06T07:24:00.446119+0000","last_became_peered":"2024-03-06T07:24:00.446119+0000","last_unstale":"2024-03-06T07:24:01.449608+0000","last_undegraded":"2024-03-06T07:24:01.449608+0000","last_fullsized":"2024-03-06T07:24:01.449608+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:20.937564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448327+0000","last_change":"2024-03-06T07:24:00.448205+0000","last_active":"2024-03-06T07:24:01.448327+0000","last_peered":"2024-03-06T07:24:01.448327+0000","last_clean":"2024-03-06T07:24:01.448327+0000","last_became_active":"2024-03-06T07:24:00.445713+0000","last_became_peered":"2024-03-06T07:24:00.445713+0000","last_unstale":"2024-03-06T07:24:01.448327+0000","last_undegraded":"2024-03-06T07:24:01.448327+0000","last_fullsized":"2024-03-06T07:24:01.448327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451941+0000","last_change":"2024-03-06T07:24:00.446705+0000","last_active":"2024-03-06T07:24:00.451941+0000","last_peered":"2024-03-06T07:24:00.451941+0000","last_clean":"2024-03-06T07:24:00.451941+0000","last_became_active":"2024-03-06T07:24:00.446464+0000","last_became_peered":"2024-03-06T07:24:00.446464+0000","last_unstale":"2024-03-06T07:24:00.451941+0000","last_undegraded":"2024-03-06T07:24:00.451941+0000","last_fullsized":"2024-03-06T07:24:00.451941+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:41.593226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449580+0000","last_change":"2024-03-06T07:24:00.447941+0000","last_active":"2024-03-06T07:24:01.449580+0000","last_peered":"2024-03-06T07:24:01.449580+0000","last_clean":"2024-03-06T07:24:01.449580+0000","last_became_active":"2024-03-06T07:24:00.447687+0000","last_became_peered":"2024-03-06T07:24:00.447687+0000","last_unstale":"2024-03-06T07:24:01.449580+0000","last_undegraded":"2024-03-06T07:24:01.449580+0000","last_fullsized":"2024-03-06T07:24:01.449580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:19.118141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452493+0000","last_change":"2024-03-06T07:24:00.442005+0000","last_active":"2024-03-06T07:24:00.452493+0000","last_peered":"2024-03-06T07:24:00.452493+0000","last_clean":"2024-03-06T07:24:00.452493+0000","last_became_active":"2024-03-06T07:24:00.441729+0000","last_became_peered":"2024-03-06T07:24:00.441729+0000","last_unstale":"2024-03-06T07:24:00.452493+0000","last_undegraded":"2024-03-06T07:24:00.452493+0000","last_fullsized":"2024-03-06T07:24:00.452493+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587466+0000","last_change":"2024-03-06T07:24:00.440566+0000","last_active":"2024-03-06T07:24:01.587466+0000","last_peered":"2024-03-06T07:24:01.587466+0000","last_clean":"2024-03-06T07:24:01.587466+0000","last_became_active":"2024-03-06T07:24:00.440019+0000","last_became_peered":"2024-03-06T07:24:00.440019+0000","last_unstale":"2024-03-06T07:24:01.587466+0000","last_undegraded":"2024-03-06T07:24:01.587466+0000","last_fullsized":"2024-03-06T07:24:01.587466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449286+0000","last_change":"2024-03-06T07:24:00.445742+0000","last_active":"2024-03-06T07:24:01.449286+0000","last_peered":"2024-03-06T07:24:01.449286+0000","last_clean":"2024-03-06T07:24:01.449286+0000","last_became_active":"2024-03-06T07:24:00.445483+0000","last_became_peered":"2024-03-06T07:24:00.445483+0000","last_unstale":"2024-03-06T07:24:01.449286+0000","last_undegraded":"2024-03-06T07:24:01.449286+0000","last_fullsized":"2024-03-06T07:24:01.449286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:02.426858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.451511+0000","last_change":"2024-03-06T07:24:00.444808+0000","last_active":"2024-03-06T07:24:00.451511+0000","last_peered":"2024-03-06T07:24:00.451511+0000","last_clean":"2024-03-06T07:24:00.451511+0000","last_became_active":"2024-03-06T07:24:00.444516+0000","last_became_peered":"2024-03-06T07:24:00.444516+0000","last_unstale":"2024-03-06T07:24:00.451511+0000","last_undegraded":"2024-03-06T07:24:00.451511+0000","last_fullsized":"2024-03-06T07:24:00.451511+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:13.209868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.456968+0000","last_change":"2024-03-06T07:24:00.447645+0000","last_active":"2024-03-06T07:24:00.456968+0000","last_peered":"2024-03-06T07:24:00.456968+0000","last_clean":"2024-03-06T07:24:00.456968+0000","last_became_active":"2024-03-06T07:24:00.445733+0000","last_became_peered":"2024-03-06T07:24:00.445733+0000","last_unstale":"2024-03-06T07:24:00.456968+0000","last_undegraded":"2024-03-06T07:24:00.456968+0000","last_fullsized":"2024-03-06T07:24:00.456968+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:47.112827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.461097+0000","last_change":"2024-03-06T07:23:59.439938+0000","last_active":"2024-03-06T07:24:00.461097+0000","last_peered":"2024-03-06T07:24:00.461097+0000","last_clean":"2024-03-06T07:24:00.461097+0000","last_became_active":"2024-03-06T07:23:59.439672+0000","last_became_peered":"2024-03-06T07:23:59.439672+0000","last_unstale":"2024-03-06T07:24:00.461097+0000","last_undegraded":"2024-03-06T07:24:00.461097+0000","last_fullsized":"2024-03-06T07:24:00.461097+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.454911+0000","last_change":"2024-03-06T07:24:00.448305+0000","last_active":"2024-03-06T07:24:00.454911+0000","last_peered":"2024-03-06T07:24:00.454911+0000","last_clean":"2024-03-06T07:24:00.454911+0000","last_became_active":"2024-03-06T07:24:00.447621+0000","last_became_peered":"2024-03-06T07:24:00.447621+0000","last_unstale":"2024-03-06T07:24:00.454911+0000","last_undegraded":"2024-03-06T07:24:00.454911+0000","last_fullsized":"2024-03-06T07:24:00.454911+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:47:23.901716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460260+0000","last_change":"2024-03-06T07:23:59.440673+0000","last_active":"2024-03-06T07:24:00.460260+0000","last_peered":"2024-03-06T07:24:00.460260+0000","last_clean":"2024-03-06T07:24:00.460260+0000","last_became_active":"2024-03-06T07:23:59.440422+0000","last_became_peered":"2024-03-06T07:23:59.440422+0000","last_unstale":"2024-03-06T07:24:00.460260+0000","last_undegraded":"2024-03-06T07:24:00.460260+0000","last_fullsized":"2024-03-06T07:24:00.460260+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:23.304839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455265+0000","last_change":"2024-03-06T07:24:00.449741+0000","last_active":"2024-03-06T07:24:00.455265+0000","last_peered":"2024-03-06T07:24:00.455265+0000","last_clean":"2024-03-06T07:24:00.455265+0000","last_became_active":"2024-03-06T07:24:00.449211+0000","last_became_peered":"2024-03-06T07:24:00.449211+0000","last_unstale":"2024-03-06T07:24:00.455265+0000","last_undegraded":"2024-03-06T07:24:00.455265+0000","last_fullsized":"2024-03-06T07:24:00.455265+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460671+0000","last_change":"2024-03-06T07:23:59.435437+0000","last_active":"2024-03-06T07:24:00.460671+0000","last_peered":"2024-03-06T07:24:00.460671+0000","last_clean":"2024-03-06T07:24:00.460671+0000","last_became_active":"2024-03-06T07:23:59.435184+0000","last_became_peered":"2024-03-06T07:23:59.435184+0000","last_unstale":"2024-03-06T07:24:00.460671+0000","last_undegraded":"2024-03-06T07:24:00.460671+0000","last_fullsized":"2024-03-06T07:24:00.460671+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455259+0000","last_change":"2024-03-06T07:24:00.449515+0000","last_active":"2024-03-06T07:24:00.455259+0000","last_peered":"2024-03-06T07:24:00.455259+0000","last_clean":"2024-03-06T07:24:00.455259+0000","last_became_active":"2024-03-06T07:24:00.449232+0000","last_became_peered":"2024-03-06T07:24:00.449232+0000","last_unstale":"2024-03-06T07:24:00.455259+0000","last_undegraded":"2024-03-06T07:24:00.455259+0000","last_fullsized":"2024-03-06T07:24:00.455259+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:37.348299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448545+0000","last_change":"2024-03-06T07:23:59.439760+0000","last_active":"2024-03-06T07:24:01.448545+0000","last_peered":"2024-03-06T07:24:01.448545+0000","last_clean":"2024-03-06T07:24:01.448545+0000","last_became_active":"2024-03-06T07:23:59.439480+0000","last_became_peered":"2024-03-06T07:23:59.439480+0000","last_unstale":"2024-03-06T07:24:01.448545+0000","last_undegraded":"2024-03-06T07:24:01.448545+0000","last_fullsized":"2024-03-06T07:24:01.448545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:21.416548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452045+0000","last_change":"2024-03-06T07:24:00.443357+0000","last_active":"2024-03-06T07:24:00.452045+0000","last_peered":"2024-03-06T07:24:00.452045+0000","last_clean":"2024-03-06T07:24:00.452045+0000","last_became_active":"2024-03-06T07:24:00.443130+0000","last_became_peered":"2024-03-06T07:24:00.443130+0000","last_unstale":"2024-03-06T07:24:00.452045+0000","last_undegraded":"2024-03-06T07:24:00.452045+0000","last_fullsized":"2024-03-06T07:24:00.452045+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.121640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449204+0000","last_change":"2024-03-06T07:23:59.441469+0000","last_active":"2024-03-06T07:24:01.449204+0000","last_peered":"2024-03-06T07:24:01.449204+0000","last_clean":"2024-03-06T07:24:01.449204+0000","last_became_active":"2024-03-06T07:23:59.441163+0000","last_became_peered":"2024-03-06T07:23:59.441163+0000","last_unstale":"2024-03-06T07:24:01.449204+0000","last_undegraded":"2024-03-06T07:24:01.449204+0000","last_fullsized":"2024-03-06T07:24:01.449204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:36.767833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452510+0000","last_change":"2024-03-06T07:24:00.443896+0000","last_active":"2024-03-06T07:24:00.452510+0000","last_peered":"2024-03-06T07:24:00.452510+0000","last_clean":"2024-03-06T07:24:00.452510+0000","last_became_active":"2024-03-06T07:24:00.443638+0000","last_became_peered":"2024-03-06T07:24:00.443638+0000","last_unstale":"2024-03-06T07:24:00.452510+0000","last_undegraded":"2024-03-06T07:24:00.452510+0000","last_fullsized":"2024-03-06T07:24:00.452510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:45.446431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.459348+0000","last_change":"2024-03-06T07:23:59.438876+0000","last_active":"2024-03-06T07:24:00.459348+0000","last_peered":"2024-03-06T07:24:00.459348+0000","last_clean":"2024-03-06T07:24:00.459348+0000","last_became_active":"2024-03-06T07:23:59.438581+0000","last_became_peered":"2024-03-06T07:23:59.438581+0000","last_unstale":"2024-03-06T07:24:00.459348+0000","last_undegraded":"2024-03-06T07:24:00.459348+0000","last_fullsized":"2024-03-06T07:24:00.459348+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.057781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.436978+0000","last_change":"2024-03-06T07:24:00.435008+0000","last_active":"2024-03-06T07:24:00.436978+0000","last_peered":"2024-03-06T07:24:00.436978+0000","last_clean":"2024-03-06T07:24:00.436978+0000","last_became_active":"2024-03-06T07:24:00.433660+0000","last_became_peered":"2024-03-06T07:24:00.433660+0000","last_unstale":"2024-03-06T07:24:00.436978+0000","last_undegraded":"2024-03-06T07:24:00.436978+0000","last_fullsized":"2024-03-06T07:24:00.436978+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:06.788182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449045+0000","last_change":"2024-03-06T07:23:59.438832+0000","last_active":"2024-03-06T07:24:01.449045+0000","last_peered":"2024-03-06T07:24:01.449045+0000","last_clean":"2024-03-06T07:24:01.449045+0000","last_became_active":"2024-03-06T07:23:59.438564+0000","last_became_peered":"2024-03-06T07:23:59.438564+0000","last_unstale":"2024-03-06T07:24:01.449045+0000","last_undegraded":"2024-03-06T07:24:01.449045+0000","last_fullsized":"2024-03-06T07:24:01.449045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.053733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452404+0000","last_change":"2024-03-06T07:24:00.443103+0000","last_active":"2024-03-06T07:24:00.452404+0000","last_peered":"2024-03-06T07:24:00.452404+0000","last_clean":"2024-03-06T07:24:00.452404+0000","last_became_active":"2024-03-06T07:24:00.442853+0000","last_became_peered":"2024-03-06T07:24:00.442853+0000","last_unstale":"2024-03-06T07:24:00.452404+0000","last_undegraded":"2024-03-06T07:24:00.452404+0000","last_fullsized":"2024-03-06T07:24:00.452404+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:15.579899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448304+0000","last_change":"2024-03-06T07:23:59.440421+0000","last_active":"2024-03-06T07:24:01.448304+0000","last_peered":"2024-03-06T07:24:01.448304+0000","last_clean":"2024-03-06T07:24:01.448304+0000","last_became_active":"2024-03-06T07:23:59.440160+0000","last_became_peered":"2024-03-06T07:23:59.440160+0000","last_unstale":"2024-03-06T07:24:01.448304+0000","last_undegraded":"2024-03-06T07:24:01.448304+0000","last_fullsized":"2024-03-06T07:24:01.448304+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:12.543578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449819+0000","last_change":"2024-03-06T07:24:00.447813+0000","last_active":"2024-03-06T07:24:01.449819+0000","last_peered":"2024-03-06T07:24:01.449819+0000","last_clean":"2024-03-06T07:24:01.449819+0000","last_became_active":"2024-03-06T07:24:00.447532+0000","last_became_peered":"2024-03-06T07:24:00.447532+0000","last_unstale":"2024-03-06T07:24:01.449819+0000","last_undegraded":"2024-03-06T07:24:01.449819+0000","last_fullsized":"2024-03-06T07:24:01.449819+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:13.646985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587813+0000","last_change":"2024-03-06T07:23:59.443581+0000","last_active":"2024-03-06T07:24:01.587813+0000","last_peered":"2024-03-06T07:24:01.587813+0000","last_clean":"2024-03-06T07:24:01.587813+0000","last_became_active":"2024-03-06T07:23:59.443241+0000","last_became_peered":"2024-03-06T07:23:59.443241+0000","last_unstale":"2024-03-06T07:24:01.587813+0000","last_undegraded":"2024-03-06T07:24:01.587813+0000","last_fullsized":"2024-03-06T07:24:01.587813+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:29.129622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587863+0000","last_change":"2024-03-06T07:24:00.449617+0000","last_active":"2024-03-06T07:24:01.587863+0000","last_peered":"2024-03-06T07:24:01.587863+0000","last_clean":"2024-03-06T07:24:01.587863+0000","last_became_active":"2024-03-06T07:24:00.447044+0000","last_became_peered":"2024-03-06T07:24:00.447044+0000","last_unstale":"2024-03-06T07:24:01.587863+0000","last_undegraded":"2024-03-06T07:24:01.587863+0000","last_fullsized":"2024-03-06T07:24:01.587863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:45.446431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.436569+0000","last_change":"2024-03-06T07:23:59.442149+0000","last_active":"2024-03-06T07:24:00.436569+0000","last_peered":"2024-03-06T07:24:00.436569+0000","last_clean":"2024-03-06T07:24:00.436569+0000","last_became_active":"2024-03-06T07:23:59.441779+0000","last_became_peered":"2024-03-06T07:23:59.441779+0000","last_unstale":"2024-03-06T07:24:00.436569+0000","last_undegraded":"2024-03-06T07:24:00.436569+0000","last_fullsized":"2024-03-06T07:24:00.436569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:19.338232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448460+0000","last_change":"2024-03-06T07:24:00.445041+0000","last_active":"2024-03-06T07:24:00.448460+0000","last_peered":"2024-03-06T07:24:00.448460+0000","last_clean":"2024-03-06T07:24:00.448460+0000","last_became_active":"2024-03-06T07:24:00.444726+0000","last_became_peered":"2024-03-06T07:24:00.444726+0000","last_unstale":"2024-03-06T07:24:00.448460+0000","last_undegraded":"2024-03-06T07:24:00.448460+0000","last_fullsized":"2024-03-06T07:24:00.448460+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:38.450025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587905+0000","last_change":"2024-03-06T07:23:59.441620+0000","last_active":"2024-03-06T07:24:01.587905+0000","last_peered":"2024-03-06T07:24:01.587905+0000","last_clean":"2024-03-06T07:24:01.587905+0000","last_became_active":"2024-03-06T07:23:59.441352+0000","last_became_peered":"2024-03-06T07:23:59.441352+0000","last_unstale":"2024-03-06T07:24:01.587905+0000","last_undegraded":"2024-03-06T07:24:01.587905+0000","last_fullsized":"2024-03-06T07:24:01.587905+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.053733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587694+0000","last_change":"2024-03-06T07:24:00.449597+0000","last_active":"2024-03-06T07:24:01.587694+0000","last_peered":"2024-03-06T07:24:01.587694+0000","last_clean":"2024-03-06T07:24:01.587694+0000","last_became_active":"2024-03-06T07:24:00.449118+0000","last_became_peered":"2024-03-06T07:24:00.449118+0000","last_unstale":"2024-03-06T07:24:01.587694+0000","last_undegraded":"2024-03-06T07:24:01.587694+0000","last_fullsized":"2024-03-06T07:24:01.587694+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449762+0000","last_change":"2024-03-06T07:23:59.444415+0000","last_active":"2024-03-06T07:24:01.449762+0000","last_peered":"2024-03-06T07:24:01.449762+0000","last_clean":"2024-03-06T07:24:01.449762+0000","last_became_active":"2024-03-06T07:23:59.444120+0000","last_became_peered":"2024-03-06T07:23:59.444120+0000","last_unstale":"2024-03-06T07:24:01.449762+0000","last_undegraded":"2024-03-06T07:24:01.449762+0000","last_fullsized":"2024-03-06T07:24:01.449762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:14.605955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448246+0000","last_change":"2024-03-06T07:24:00.449289+0000","last_active":"2024-03-06T07:24:01.448246+0000","last_peered":"2024-03-06T07:24:01.448246+0000","last_clean":"2024-03-06T07:24:01.448246+0000","last_became_active":"2024-03-06T07:24:00.447448+0000","last_became_peered":"2024-03-06T07:24:00.447448+0000","last_unstale":"2024-03-06T07:24:01.448246+0000","last_undegraded":"2024-03-06T07:24:01.448246+0000","last_fullsized":"2024-03-06T07:24:01.448246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460960+0000","last_change":"2024-03-06T07:23:59.443439+0000","last_active":"2024-03-06T07:24:00.460960+0000","last_peered":"2024-03-06T07:24:00.460960+0000","last_clean":"2024-03-06T07:24:00.460960+0000","last_became_active":"2024-03-06T07:23:59.443188+0000","last_became_peered":"2024-03-06T07:23:59.443188+0000","last_unstale":"2024-03-06T07:24:00.460960+0000","last_undegraded":"2024-03-06T07:24:00.460960+0000","last_fullsized":"2024-03-06T07:24:00.460960+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:47.951502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587625+0000","last_change":"2024-03-06T07:24:00.434521+0000","last_active":"2024-03-06T07:24:01.587625+0000","last_peered":"2024-03-06T07:24:01.587625+0000","last_clean":"2024-03-06T07:24:01.587625+0000","last_became_active":"2024-03-06T07:24:00.434247+0000","last_became_peered":"2024-03-06T07:24:00.434247+0000","last_unstale":"2024-03-06T07:24:01.587625+0000","last_undegraded":"2024-03-06T07:24:01.587625+0000","last_fullsized":"2024-03-06T07:24:01.587625+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.431173+0000","last_change":"2024-03-06T07:23:59.441682+0000","last_active":"2024-03-06T07:24:00.431173+0000","last_peered":"2024-03-06T07:24:00.431173+0000","last_clean":"2024-03-06T07:24:00.431173+0000","last_became_active":"2024-03-06T07:23:59.441406+0000","last_became_peered":"2024-03-06T07:23:59.441406+0000","last_unstale":"2024-03-06T07:24:00.431173+0000","last_undegraded":"2024-03-06T07:24:00.431173+0000","last_fullsized":"2024-03-06T07:24:00.431173+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:26.356183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448151+0000","last_change":"2024-03-06T07:24:00.451006+0000","last_active":"2024-03-06T07:24:01.448151+0000","last_peered":"2024-03-06T07:24:01.448151+0000","last_clean":"2024-03-06T07:24:01.448151+0000","last_became_active":"2024-03-06T07:24:00.450626+0000","last_became_peered":"2024-03-06T07:24:00.450626+0000","last_unstale":"2024-03-06T07:24:01.448151+0000","last_undegraded":"2024-03-06T07:24:01.448151+0000","last_fullsized":"2024-03-06T07:24:01.448151+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:14.268540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449005+0000","last_change":"2024-03-06T07:23:59.437649+0000","last_active":"2024-03-06T07:24:01.449005+0000","last_peered":"2024-03-06T07:24:01.449005+0000","last_clean":"2024-03-06T07:24:01.449005+0000","last_became_active":"2024-03-06T07:23:59.437377+0000","last_became_peered":"2024-03-06T07:23:59.437377+0000","last_unstale":"2024-03-06T07:24:01.449005+0000","last_undegraded":"2024-03-06T07:24:01.449005+0000","last_fullsized":"2024-03-06T07:24:01.449005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587470+0000","last_change":"2024-03-06T07:24:00.446910+0000","last_active":"2024-03-06T07:24:01.587470+0000","last_peered":"2024-03-06T07:24:01.587470+0000","last_clean":"2024-03-06T07:24:01.587470+0000","last_became_active":"2024-03-06T07:24:00.445795+0000","last_became_peered":"2024-03-06T07:24:00.445795+0000","last_unstale":"2024-03-06T07:24:01.587470+0000","last_undegraded":"2024-03-06T07:24:01.587470+0000","last_fullsized":"2024-03-06T07:24:01.587470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":22,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587273+0000","last_change":"2024-03-06T07:23:59.442282+0000","last_active":"2024-03-06T07:24:01.587273+0000","last_peered":"2024-03-06T07:24:01.587273+0000","last_clean":"2024-03-06T07:24:01.587273+0000","last_became_active":"2024-03-06T07:23:59.441992+0000","last_became_peered":"2024-03-06T07:23:59.441992+0000","last_unstale":"2024-03-06T07:24:01.587273+0000","last_undegraded":"2024-03-06T07:24:01.587273+0000","last_fullsized":"2024-03-06T07:24:01.587273+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:12.239306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587803+0000","last_change":"2024-03-06T07:24:00.448694+0000","last_active":"2024-03-06T07:24:01.587803+0000","last_peered":"2024-03-06T07:24:01.587803+0000","last_clean":"2024-03-06T07:24:01.587803+0000","last_became_active":"2024-03-06T07:24:00.448369+0000","last_became_peered":"2024-03-06T07:24:00.448369+0000","last_unstale":"2024-03-06T07:24:01.587803+0000","last_undegraded":"2024-03-06T07:24:01.587803+0000","last_fullsized":"2024-03-06T07:24:01.587803+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.121640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460842+0000","last_change":"2024-03-06T07:23:59.442248+0000","last_active":"2024-03-06T07:24:00.460842+0000","last_peered":"2024-03-06T07:24:00.460842+0000","last_clean":"2024-03-06T07:24:00.460842+0000","last_became_active":"2024-03-06T07:23:59.441980+0000","last_became_peered":"2024-03-06T07:23:59.441980+0000","last_unstale":"2024-03-06T07:24:00.460842+0000","last_undegraded":"2024-03-06T07:24:00.460842+0000","last_fullsized":"2024-03-06T07:24:00.460842+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:42.855432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448632+0000","last_change":"2024-03-06T07:24:00.446282+0000","last_active":"2024-03-06T07:24:01.448632+0000","last_peered":"2024-03-06T07:24:01.448632+0000","last_clean":"2024-03-06T07:24:01.448632+0000","last_became_active":"2024-03-06T07:24:00.445994+0000","last_became_peered":"2024-03-06T07:24:00.445994+0000","last_unstale":"2024-03-06T07:24:01.448632+0000","last_undegraded":"2024-03-06T07:24:01.448632+0000","last_fullsized":"2024-03-06T07:24:01.448632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460629+0000","last_change":"2024-03-06T07:23:59.440130+0000","last_active":"2024-03-06T07:24:00.460629+0000","last_peered":"2024-03-06T07:24:00.460629+0000","last_clean":"2024-03-06T07:24:00.460629+0000","last_became_active":"2024-03-06T07:23:59.439642+0000","last_became_peered":"2024-03-06T07:23:59.439642+0000","last_unstale":"2024-03-06T07:24:00.460629+0000","last_undegraded":"2024-03-06T07:24:00.460629+0000","last_fullsized":"2024-03-06T07:24:00.460629+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.817188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452262+0000","last_change":"2024-03-06T07:24:00.445971+0000","last_active":"2024-03-06T07:24:00.452262+0000","last_peered":"2024-03-06T07:24:00.452262+0000","last_clean":"2024-03-06T07:24:00.452262+0000","last_became_active":"2024-03-06T07:24:00.445707+0000","last_became_peered":"2024-03-06T07:24:00.445707+0000","last_unstale":"2024-03-06T07:24:00.452262+0000","last_undegraded":"2024-03-06T07:24:00.452262+0000","last_fullsized":"2024-03-06T07:24:00.452262+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:26.491484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448963+0000","last_change":"2024-03-06T07:23:59.438479+0000","last_active":"2024-03-06T07:24:01.448963+0000","last_peered":"2024-03-06T07:24:01.448963+0000","last_clean":"2024-03-06T07:24:01.448963+0000","last_became_active":"2024-03-06T07:23:59.438153+0000","last_became_peered":"2024-03-06T07:23:59.438153+0000","last_unstale":"2024-03-06T07:24:01.448963+0000","last_undegraded":"2024-03-06T07:24:01.448963+0000","last_fullsized":"2024-03-06T07:24:01.448963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587879+0000","last_change":"2024-03-06T07:24:00.448405+0000","last_active":"2024-03-06T07:24:01.587879+0000","last_peered":"2024-03-06T07:24:01.587879+0000","last_clean":"2024-03-06T07:24:01.587879+0000","last_became_active":"2024-03-06T07:24:00.448018+0000","last_became_peered":"2024-03-06T07:24:00.448018+0000","last_unstale":"2024-03-06T07:24:01.587879+0000","last_undegraded":"2024-03-06T07:24:01.587879+0000","last_fullsized":"2024-03-06T07:24:01.587879+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:26.491484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.438159+0000","last_change":"2024-03-06T07:23:59.440654+0000","last_active":"2024-03-06T07:24:00.438159+0000","last_peered":"2024-03-06T07:24:00.438159+0000","last_clean":"2024-03-06T07:24:00.438159+0000","last_became_active":"2024-03-06T07:23:59.440414+0000","last_became_peered":"2024-03-06T07:23:59.440414+0000","last_unstale":"2024-03-06T07:24:00.438159+0000","last_undegraded":"2024-03-06T07:24:00.438159+0000","last_fullsized":"2024-03-06T07:24:00.438159+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:21.416548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448400+0000","last_change":"2024-03-06T07:24:00.447077+0000","last_active":"2024-03-06T07:24:01.448400+0000","last_peered":"2024-03-06T07:24:01.448400+0000","last_clean":"2024-03-06T07:24:01.448400+0000","last_became_active":"2024-03-06T07:24:00.446780+0000","last_became_peered":"2024-03-06T07:24:00.446780+0000","last_unstale":"2024-03-06T07:24:01.448400+0000","last_undegraded":"2024-03-06T07:24:01.448400+0000","last_fullsized":"2024-03-06T07:24:01.448400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.460124+0000","last_change":"2024-03-06T07:23:59.440760+0000","last_active":"2024-03-06T07:24:00.460124+0000","last_peered":"2024-03-06T07:24:00.460124+0000","last_clean":"2024-03-06T07:24:00.460124+0000","last_became_active":"2024-03-06T07:23:59.440490+0000","last_became_peered":"2024-03-06T07:23:59.440490+0000","last_unstale":"2024-03-06T07:24:00.460124+0000","last_undegraded":"2024-03-06T07:24:00.460124+0000","last_fullsized":"2024-03-06T07:24:00.460124+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:36.338297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.453192+0000","last_change":"2024-03-06T07:24:00.447433+0000","last_active":"2024-03-06T07:24:00.453192+0000","last_peered":"2024-03-06T07:24:00.453192+0000","last_clean":"2024-03-06T07:24:00.453192+0000","last_became_active":"2024-03-06T07:24:00.447195+0000","last_became_peered":"2024-03-06T07:24:00.447195+0000","last_unstale":"2024-03-06T07:24:00.453192+0000","last_undegraded":"2024-03-06T07:24:00.453192+0000","last_fullsized":"2024-03-06T07:24:00.453192+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:43.865434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.434712+0000","last_change":"2024-03-06T07:23:59.439766+0000","last_active":"2024-03-06T07:24:00.434712+0000","last_peered":"2024-03-06T07:24:00.434712+0000","last_clean":"2024-03-06T07:24:00.434712+0000","last_became_active":"2024-03-06T07:23:59.439309+0000","last_became_peered":"2024-03-06T07:23:59.439309+0000","last_unstale":"2024-03-06T07:24:00.434712+0000","last_undegraded":"2024-03-06T07:24:00.434712+0000","last_fullsized":"2024-03-06T07:24:00.434712+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:32.312900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455249+0000","last_change":"2024-03-06T07:24:00.450031+0000","last_active":"2024-03-06T07:24:00.455249+0000","last_peered":"2024-03-06T07:24:00.455249+0000","last_clean":"2024-03-06T07:24:00.455249+0000","last_became_active":"2024-03-06T07:24:00.449743+0000","last_became_peered":"2024-03-06T07:24:00.449743+0000","last_unstale":"2024-03-06T07:24:00.455249+0000","last_undegraded":"2024-03-06T07:24:00.455249+0000","last_fullsized":"2024-03-06T07:24:00.455249+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448089+0000","last_change":"2024-03-06T07:23:59.438050+0000","last_active":"2024-03-06T07:24:01.448089+0000","last_peered":"2024-03-06T07:24:01.448089+0000","last_clean":"2024-03-06T07:24:01.448089+0000","last_became_active":"2024-03-06T07:23:59.437721+0000","last_became_peered":"2024-03-06T07:23:59.437721+0000","last_unstale":"2024-03-06T07:24:01.448089+0000","last_undegraded":"2024-03-06T07:24:01.448089+0000","last_fullsized":"2024-03-06T07:24:01.448089+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.057781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448906+0000","last_change":"2024-03-06T07:24:00.445997+0000","last_active":"2024-03-06T07:24:01.448906+0000","last_peered":"2024-03-06T07:24:01.448906+0000","last_clean":"2024-03-06T07:24:01.448906+0000","last_became_active":"2024-03-06T07:24:00.445250+0000","last_became_peered":"2024-03-06T07:24:00.445250+0000","last_unstale":"2024-03-06T07:24:01.448906+0000","last_undegraded":"2024-03-06T07:24:01.448906+0000","last_fullsized":"2024-03-06T07:24:01.448906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455796+0000","last_change":"2024-03-06T07:23:59.435108+0000","last_active":"2024-03-06T07:24:00.455796+0000","last_peered":"2024-03-06T07:24:00.455796+0000","last_clean":"2024-03-06T07:24:00.455796+0000","last_became_active":"2024-03-06T07:23:59.434845+0000","last_became_peered":"2024-03-06T07:23:59.434845+0000","last_unstale":"2024-03-06T07:24:00.455796+0000","last_undegraded":"2024-03-06T07:24:00.455796+0000","last_fullsized":"2024-03-06T07:24:00.455796+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:35.611916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448816+0000","last_change":"2024-03-06T07:24:00.449232+0000","last_active":"2024-03-06T07:24:01.448816+0000","last_peered":"2024-03-06T07:24:01.448816+0000","last_clean":"2024-03-06T07:24:01.448816+0000","last_became_active":"2024-03-06T07:24:00.448976+0000","last_became_peered":"2024-03-06T07:24:00.448976+0000","last_unstale":"2024-03-06T07:24:01.448816+0000","last_undegraded":"2024-03-06T07:24:01.448816+0000","last_fullsized":"2024-03-06T07:24:01.448816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.446278+0000","last_change":"2024-03-06T07:23:59.439820+0000","last_active":"2024-03-06T07:24:01.446278+0000","last_peered":"2024-03-06T07:24:01.446278+0000","last_clean":"2024-03-06T07:24:01.446278+0000","last_became_active":"2024-03-06T07:23:59.439328+0000","last_became_peered":"2024-03-06T07:23:59.439328+0000","last_unstale":"2024-03-06T07:24:01.446278+0000","last_undegraded":"2024-03-06T07:24:01.446278+0000","last_fullsized":"2024-03-06T07:24:01.446278+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:12.636982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.448440+0000","last_change":"2024-03-06T07:24:00.444894+0000","last_active":"2024-03-06T07:24:00.448440+0000","last_peered":"2024-03-06T07:24:00.448440+0000","last_clean":"2024-03-06T07:24:00.448440+0000","last_became_active":"2024-03-06T07:24:00.444614+0000","last_became_peered":"2024-03-06T07:24:00.444614+0000","last_unstale":"2024-03-06T07:24:00.448440+0000","last_undegraded":"2024-03-06T07:24:00.448440+0000","last_fullsized":"2024-03-06T07:24:00.448440+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:02.387037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.431703+0000","last_change":"2024-03-06T07:23:59.440546+0000","last_active":"2024-03-06T07:24:00.431703+0000","last_peered":"2024-03-06T07:24:00.431703+0000","last_clean":"2024-03-06T07:24:00.431703+0000","last_became_active":"2024-03-06T07:23:59.440268+0000","last_became_peered":"2024-03-06T07:23:59.440268+0000","last_unstale":"2024-03-06T07:24:00.431703+0000","last_undegraded":"2024-03-06T07:24:00.431703+0000","last_fullsized":"2024-03-06T07:24:00.431703+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:32.312900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.455109+0000","last_change":"2024-03-06T07:24:00.448723+0000","last_active":"2024-03-06T07:24:00.455109+0000","last_peered":"2024-03-06T07:24:00.455109+0000","last_clean":"2024-03-06T07:24:00.455109+0000","last_became_active":"2024-03-06T07:24:00.448168+0000","last_became_peered":"2024-03-06T07:24:00.448168+0000","last_unstale":"2024-03-06T07:24:00.455109+0000","last_undegraded":"2024-03-06T07:24:00.455109+0000","last_fullsized":"2024-03-06T07:24:00.455109+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:48.961505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448645+0000","last_change":"2024-03-06T07:23:59.440547+0000","last_active":"2024-03-06T07:24:01.448645+0000","last_peered":"2024-03-06T07:24:01.448645+0000","last_clean":"2024-03-06T07:24:01.448645+0000","last_became_active":"2024-03-06T07:23:59.440261+0000","last_became_peered":"2024-03-06T07:23:59.440261+0000","last_unstale":"2024-03-06T07:24:01.448645+0000","last_undegraded":"2024-03-06T07:24:01.448645+0000","last_fullsized":"2024-03-06T07:24:01.448645+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.817188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452240+0000","last_change":"2024-03-06T07:24:00.450414+0000","last_active":"2024-03-06T07:24:00.452240+0000","last_peered":"2024-03-06T07:24:00.452240+0000","last_clean":"2024-03-06T07:24:00.452240+0000","last_became_active":"2024-03-06T07:24:00.450152+0000","last_became_peered":"2024-03-06T07:24:00.450152+0000","last_unstale":"2024-03-06T07:24:00.452240+0000","last_undegraded":"2024-03-06T07:24:00.452240+0000","last_fullsized":"2024-03-06T07:24:00.452240+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:59.617656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.435133+0000","last_change":"2024-03-06T07:23:59.436261+0000","last_active":"2024-03-06T07:24:00.435133+0000","last_peered":"2024-03-06T07:24:00.435133+0000","last_clean":"2024-03-06T07:24:00.435133+0000","last_became_active":"2024-03-06T07:23:59.436007+0000","last_became_peered":"2024-03-06T07:23:59.436007+0000","last_unstale":"2024-03-06T07:24:00.435133+0000","last_undegraded":"2024-03-06T07:24:00.435133+0000","last_fullsized":"2024-03-06T07:24:00.435133+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:05.778180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587653+0000","last_change":"2024-03-06T07:24:00.441206+0000","last_active":"2024-03-06T07:24:01.587653+0000","last_peered":"2024-03-06T07:24:01.587653+0000","last_clean":"2024-03-06T07:24:01.587653+0000","last_became_active":"2024-03-06T07:24:00.440923+0000","last_became_peered":"2024-03-06T07:24:00.440923+0000","last_unstale":"2024-03-06T07:24:01.587653+0000","last_undegraded":"2024-03-06T07:24:01.587653+0000","last_fullsized":"2024-03-06T07:24:01.587653+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449011+0000","last_change":"2024-03-06T07:23:59.443108+0000","last_active":"2024-03-06T07:24:01.449011+0000","last_peered":"2024-03-06T07:24:01.449011+0000","last_clean":"2024-03-06T07:24:01.449011+0000","last_became_active":"2024-03-06T07:23:59.442847+0000","last_became_peered":"2024-03-06T07:23:59.442847+0000","last_unstale":"2024-03-06T07:24:01.449011+0000","last_undegraded":"2024-03-06T07:24:01.449011+0000","last_fullsized":"2024-03-06T07:24:01.449011+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:09.321526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.447348+0000","last_change":"2024-03-06T07:24:00.441365+0000","last_active":"2024-03-06T07:24:00.447348+0000","last_peered":"2024-03-06T07:24:00.447348+0000","last_clean":"2024-03-06T07:24:00.447348+0000","last_became_active":"2024-03-06T07:24:00.440667+0000","last_became_peered":"2024-03-06T07:24:00.440667+0000","last_unstale":"2024-03-06T07:24:00.447348+0000","last_undegraded":"2024-03-06T07:24:00.447348+0000","last_fullsized":"2024-03-06T07:24:00.447348+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:18:30.806654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.435506+0000","last_change":"2024-03-06T07:23:59.438513+0000","last_active":"2024-03-06T07:24:00.435506+0000","last_peered":"2024-03-06T07:24:00.435506+0000","last_clean":"2024-03-06T07:24:00.435506+0000","last_became_active":"2024-03-06T07:23:59.438207+0000","last_became_peered":"2024-03-06T07:23:59.438207+0000","last_unstale":"2024-03-06T07:24:00.435506+0000","last_undegraded":"2024-03-06T07:24:00.435506+0000","last_fullsized":"2024-03-06T07:24:00.435506+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.892786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452382+0000","last_change":"2024-03-06T07:24:00.440384+0000","last_active":"2024-03-06T07:24:00.452382+0000","last_peered":"2024-03-06T07:24:00.452382+0000","last_clean":"2024-03-06T07:24:00.452382+0000","last_became_active":"2024-03-06T07:24:00.440128+0000","last_became_peered":"2024-03-06T07:24:00.440128+0000","last_unstale":"2024-03-06T07:24:00.452382+0000","last_undegraded":"2024-03-06T07:24:00.452382+0000","last_fullsized":"2024-03-06T07:24:00.452382+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.435351+0000","last_change":"2024-03-06T07:23:59.437764+0000","last_active":"2024-03-06T07:24:00.435351+0000","last_peered":"2024-03-06T07:24:00.435351+0000","last_clean":"2024-03-06T07:24:00.435351+0000","last_became_active":"2024-03-06T07:23:59.437126+0000","last_became_peered":"2024-03-06T07:23:59.437126+0000","last_unstale":"2024-03-06T07:24:00.435351+0000","last_undegraded":"2024-03-06T07:24:00.435351+0000","last_fullsized":"2024-03-06T07:24:00.435351+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:46.102824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.452476+0000","last_change":"2024-03-06T07:24:00.440544+0000","last_active":"2024-03-06T07:24:00.452476+0000","last_peered":"2024-03-06T07:24:00.452476+0000","last_clean":"2024-03-06T07:24:00.452476+0000","last_became_active":"2024-03-06T07:24:00.440284+0000","last_became_peered":"2024-03-06T07:24:00.440284+0000","last_unstale":"2024-03-06T07:24:00.452476+0000","last_undegraded":"2024-03-06T07:24:00.452476+0000","last_fullsized":"2024-03-06T07:24:00.452476+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-06T07:24:00.431265+0000","last_change":"2024-03-06T07:23:59.444024+0000","last_active":"2024-03-06T07:24:00.431265+0000","last_peered":"2024-03-06T07:24:00.431265+0000","last_clean":"2024-03-06T07:24:00.431265+0000","last_became_active":"2024-03-06T07:23:59.443744+0000","last_became_peered":"2024-03-06T07:23:59.443744+0000","last_unstale":"2024-03-06T07:24:00.431265+0000","last_undegraded":"2024-03-06T07:24:00.431265+0000","last_fullsized":"2024-03-06T07:24:00.431265+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:06.553338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:01.587213+0000","last_peered":"2024-03-06T07:24:01.587213+0000","last_clean":"2024-03-06T07:24:01.587213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:01.587213+0000","last_undegraded":"2024-03-06T07:24:01.587213+0000","last_fullsized":"2024-03-06T07:24:01.587213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1298432,"data_stored":879450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":696320,"data_stored":288922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":295874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":720896,"data_stored":296834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":288978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":288978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":289012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1306624,"data_stored":881484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:06.415 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-03-06T07:24:06.413+0000 7f02f24d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:24:08.050 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:08.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.150+0000 7f477d4fb640 1 Processor -- start 2024-03-06T07:24:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.150+0000 7f477d4fb640 1 -- start start 2024-03-06T07:24:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f477d4fb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f477815bdd0 0x7f47780c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f477d4fb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47780cebe0 con 0x7f477815fa10 2024-03-06T07:24:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f477d4fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47780ceec0 con 0x7f47780d16e0 2024-03-06T07:24:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f477d4fb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47780ce980 con 0x7f477815bdd0 2024-03-06T07:24:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f4776ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f477815bdd0 0x7f47780c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f47767fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f477815fa10 0x7f47780ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46424/0 (socket says 172.21.15.72:46424) 2024-03-06T07:24:08.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f4776ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f477815bdd0 0x7f47780c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33992/0 (socket says 172.21.15.72:33992) 2024-03-06T07:24:08.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f47767fc640 1 -- 172.21.15.72:0/775426023 learned_addr learned my addr 172.21.15.72:0/775426023 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:08.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f4776ffd640 1 -- 172.21.15.72:0/775426023 >> v1:172.21.15.146:6789/0 conn(0x7f47780d16e0 legacy=0x7f47780d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f4776ffd640 1 -- 172.21.15.72:0/775426023 >> v1:172.21.15.72:6789/0 conn(0x7f477815fa10 legacy=0x7f47780ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.152+0000 7f4776ffd640 1 -- 172.21.15.72:0/775426023 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47780cf1a0 con 0x7f477815bdd0 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.153+0000 7f4776ffd640 1 --2- 172.21.15.72:0/775426023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f477815bdd0 0x7f47780c5610 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f47680097a0 tx=0x7f476802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e563df1a6a18132 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.153+0000 7f4775ffb640 1 -- 172.21.15.72:0/775426023 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47680315a0 con 0x7f477815bdd0 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.153+0000 7f4775ffb640 1 -- 172.21.15.72:0/775426023 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4768037070 con 0x7f477815bdd0 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.153+0000 7f4775ffb640 1 -- 172.21.15.72:0/775426023 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768031960 con 0x7f477815bdd0 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 -- 172.21.15.72:0/775426023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f477815bdd0 msgr2=0x7f47780c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 --2- 172.21.15.72:0/775426023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f477815bdd0 0x7f47780c5610 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f47680097a0 tx=0x7f476802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:08.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 -- 172.21.15.72:0/775426023 shutdown_connections 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 --2- 172.21.15.72:0/775426023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f477815bdd0 0x7f47780c5610 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 -- 172.21.15.72:0/775426023 >> 172.21.15.72:0/775426023 conn(0x7f477809a9f0 msgr2=0x7f477809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 -- 172.21.15.72:0/775426023 shutdown_connections 2024-03-06T07:24:08.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.154+0000 7f477d4fb640 1 -- 172.21.15.72:0/775426023 wait complete. 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.155+0000 7f477d4fb640 1 Processor -- start 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.155+0000 7f477d4fb640 1 -- start start 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f477d4fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 0x7f47781551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f477d4fb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477815bdd0 0x7f47780b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f477d4fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477815fa10 0x7f477815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f477d4fb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47780cc5d0 con 0x7f47780d16e0 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f477d4fb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47780c5b50 con 0x7f477815bdd0 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f477d4fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477815c1b0 con 0x7f477815fa10 2024-03-06T07:24:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 0x7f47781551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 0x7f47781551b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34004/0 (socket says 172.21.15.72:34004) 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 -- 172.21.15.72:0/792107378 learned_addr learned my addr 172.21.15.72:0/792107378 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f47767fc640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477815bdd0 0x7f47780b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f47777fe640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477815fa10 0x7f477815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 -- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477815fa10 msgr2=0x7f477815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477815fa10 0x7f477815d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 -- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477815bdd0 msgr2=0x7f47780b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477815bdd0 0x7f47780b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 -- 172.21.15.72:0/792107378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47780b5ac0 con 0x7f47780d16e0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f4776ffd640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 0x7f47781551b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4768034040 tx=0x7f47680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768032840 con 0x7f47780d16e0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.156+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4768032a00 con 0x7f47780d16e0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.157+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47780b5da0 con 0x7f47780d16e0 2024-03-06T07:24:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.157+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4768032df0 con 0x7f47780d16e0 2024-03-06T07:24:08.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.157+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47780b6360 con 0x7f47780d16e0 2024-03-06T07:24:08.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.158+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f477804f330 con 0x7f47780d16e0 2024-03-06T07:24:08.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.158+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f476803b3f0 con 0x7f47780d16e0 2024-03-06T07:24:08.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.158+0000 7f475f7fe640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f476406b4d0 0x7f476406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.159+0000 7f47767fc640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f476406b4d0 0x7f476406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.159+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f476802f030 con 0x7f47780d16e0 2024-03-06T07:24:08.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.159+0000 7f47767fc640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f476406b4d0 0x7f476406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f477815e500 tx=0x7f476c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.163+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4768040070 con 0x7f47780d16e0 2024-03-06T07:24:08.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.339+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f477815ff10 con 0x7f476406b4d0 2024-03-06T07:24:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.345+0000 7f475f7fe640 1 -- 172.21.15.72:0/792107378 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251867 (secure 0 0 0) 0x7f477815ff10 con 0x7f476406b4d0 2024-03-06T07:24:08.349 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:08.351 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:08.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f476406b4d0 msgr2=0x7f476406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f476406b4d0 0x7f476406d990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f477815e500 tx=0x7f476c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 msgr2=0x7f47781551b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 0x7f47781551b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4768034040 tx=0x7f47680046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 shutdown_connections 2024-03-06T07:24:08.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f476406b4d0 0x7f476406d990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f477815fa10 0x7f477815d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f477815bdd0 0x7f47780b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 --2- 172.21.15.72:0/792107378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47780d16e0 0x7f47781551b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 >> 172.21.15.72:0/792107378 conn(0x7f477809a9f0 msgr2=0x7f4778070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 shutdown_connections 2024-03-06T07:24:08.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.348+0000 7f477d4fb640 1 -- 172.21.15.72:0/792107378 wait complete. 2024-03-06T07:24:08.361 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-03-06T07:24:07.972316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256228,"kb_used_data":6760,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718492,"statfs":{"total":773094113280,"available":772831735808,"internally_reserved":0,"allocated":6922240,"data_stored":3547332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001410"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449532+0000","last_change":"2024-03-06T07:24:00.439946+0000","last_active":"2024-03-06T07:24:01.449532+0000","last_peered":"2024-03-06T07:24:01.449532+0000","last_clean":"2024-03-06T07:24:01.449532+0000","last_became_active":"2024-03-06T07:24:00.439682+0000","last_became_peered":"2024-03-06T07:24:00.439682+0000","last_unstale":"2024-03-06T07:24:01.449532+0000","last_undegraded":"2024-03-06T07:24:01.449532+0000","last_fullsized":"2024-03-06T07:24:01.449532+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:10:48.774758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465902+0000","last_change":"2024-03-06T07:24:00.446637+0000","last_active":"2024-03-06T07:24:02.465902+0000","last_peered":"2024-03-06T07:24:02.465902+0000","last_clean":"2024-03-06T07:24:02.465902+0000","last_became_active":"2024-03-06T07:24:00.446373+0000","last_became_peered":"2024-03-06T07:24:00.446373+0000","last_unstale":"2024-03-06T07:24:02.465902+0000","last_undegraded":"2024-03-06T07:24:02.465902+0000","last_fullsized":"2024-03-06T07:24:02.465902+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:19.588199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587704+0000","last_change":"2024-03-06T07:24:00.439609+0000","last_active":"2024-03-06T07:24:01.587704+0000","last_peered":"2024-03-06T07:24:01.587704+0000","last_clean":"2024-03-06T07:24:01.587704+0000","last_became_active":"2024-03-06T07:24:00.439333+0000","last_became_peered":"2024-03-06T07:24:00.439333+0000","last_unstale":"2024-03-06T07:24:01.587704+0000","last_undegraded":"2024-03-06T07:24:01.587704+0000","last_fullsized":"2024-03-06T07:24:01.587704+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448757+0000","last_change":"2024-03-06T07:24:00.445347+0000","last_active":"2024-03-06T07:24:01.448757+0000","last_peered":"2024-03-06T07:24:01.448757+0000","last_clean":"2024-03-06T07:24:01.448757+0000","last_became_active":"2024-03-06T07:24:00.445031+0000","last_became_peered":"2024-03-06T07:24:00.445031+0000","last_unstale":"2024-03-06T07:24:01.448757+0000","last_undegraded":"2024-03-06T07:24:01.448757+0000","last_fullsized":"2024-03-06T07:24:01.448757+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449309+0000","last_change":"2024-03-06T07:24:00.441074+0000","last_active":"2024-03-06T07:24:01.449309+0000","last_peered":"2024-03-06T07:24:01.449309+0000","last_clean":"2024-03-06T07:24:01.449309+0000","last_became_active":"2024-03-06T07:24:00.440804+0000","last_became_peered":"2024-03-06T07:24:00.440804+0000","last_unstale":"2024-03-06T07:24:01.449309+0000","last_undegraded":"2024-03-06T07:24:01.449309+0000","last_fullsized":"2024-03-06T07:24:01.449309+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:50:39.417435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469421+0000","last_change":"2024-03-06T07:24:00.453304+0000","last_active":"2024-03-06T07:24:02.469421+0000","last_peered":"2024-03-06T07:24:02.469421+0000","last_clean":"2024-03-06T07:24:02.469421+0000","last_became_active":"2024-03-06T07:24:00.452795+0000","last_became_peered":"2024-03-06T07:24:00.452795+0000","last_unstale":"2024-03-06T07:24:02.469421+0000","last_undegraded":"2024-03-06T07:24:02.469421+0000","last_fullsized":"2024-03-06T07:24:02.469421+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:15.579899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448469+0000","last_change":"2024-03-06T07:24:00.444902+0000","last_active":"2024-03-06T07:24:01.448469+0000","last_peered":"2024-03-06T07:24:01.448469+0000","last_clean":"2024-03-06T07:24:01.448469+0000","last_became_active":"2024-03-06T07:24:00.444629+0000","last_became_peered":"2024-03-06T07:24:00.444629+0000","last_unstale":"2024-03-06T07:24:01.448469+0000","last_undegraded":"2024-03-06T07:24:01.448469+0000","last_fullsized":"2024-03-06T07:24:01.448469+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:54.193912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465613+0000","last_change":"2024-03-06T07:24:00.439953+0000","last_active":"2024-03-06T07:24:02.465613+0000","last_peered":"2024-03-06T07:24:02.465613+0000","last_clean":"2024-03-06T07:24:02.465613+0000","last_became_active":"2024-03-06T07:24:00.439683+0000","last_became_peered":"2024-03-06T07:24:00.439683+0000","last_unstale":"2024-03-06T07:24:02.465613+0000","last_undegraded":"2024-03-06T07:24:02.465613+0000","last_fullsized":"2024-03-06T07:24:02.465613+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:58:53.406964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448813+0000","last_change":"2024-03-06T07:24:00.444198+0000","last_active":"2024-03-06T07:24:01.448813+0000","last_peered":"2024-03-06T07:24:01.448813+0000","last_clean":"2024-03-06T07:24:01.448813+0000","last_became_active":"2024-03-06T07:24:00.443885+0000","last_became_peered":"2024-03-06T07:24:00.443885+0000","last_unstale":"2024-03-06T07:24:01.448813+0000","last_undegraded":"2024-03-06T07:24:01.448813+0000","last_fullsized":"2024-03-06T07:24:01.448813+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:22.426550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.278218+0000","last_change":"2024-03-06T07:24:00.444858+0000","last_active":"2024-03-06T07:24:03.278218+0000","last_peered":"2024-03-06T07:24:03.278218+0000","last_clean":"2024-03-06T07:24:03.278218+0000","last_became_active":"2024-03-06T07:24:00.444401+0000","last_became_peered":"2024-03-06T07:24:00.444401+0000","last_unstale":"2024-03-06T07:24:03.278218+0000","last_undegraded":"2024-03-06T07:24:03.278218+0000","last_fullsized":"2024-03-06T07:24:03.278218+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:13.553580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.278176+0000","last_change":"2024-03-06T07:24:00.447374+0000","last_active":"2024-03-06T07:24:03.278176+0000","last_peered":"2024-03-06T07:24:03.278176+0000","last_clean":"2024-03-06T07:24:03.278176+0000","last_became_active":"2024-03-06T07:24:00.445602+0000","last_became_peered":"2024-03-06T07:24:00.445602+0000","last_unstale":"2024-03-06T07:24:03.278176+0000","last_undegraded":"2024-03-06T07:24:03.278176+0000","last_fullsized":"2024-03-06T07:24:03.278176+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:10.331528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448447+0000","last_change":"2024-03-06T07:24:00.447595+0000","last_active":"2024-03-06T07:24:01.448447+0000","last_peered":"2024-03-06T07:24:01.448447+0000","last_clean":"2024-03-06T07:24:01.448447+0000","last_became_active":"2024-03-06T07:24:00.446729+0000","last_became_peered":"2024-03-06T07:24:00.446729+0000","last_unstale":"2024-03-06T07:24:01.448447+0000","last_undegraded":"2024-03-06T07:24:01.448447+0000","last_fullsized":"2024-03-06T07:24:01.448447+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.156829+0000","last_change":"2024-03-06T07:24:00.444770+0000","last_active":"2024-03-06T07:24:02.156829+0000","last_peered":"2024-03-06T07:24:02.156829+0000","last_clean":"2024-03-06T07:24:02.156829+0000","last_became_active":"2024-03-06T07:24:00.444233+0000","last_became_peered":"2024-03-06T07:24:00.444233+0000","last_unstale":"2024-03-06T07:24:02.156829+0000","last_undegraded":"2024-03-06T07:24:02.156829+0000","last_fullsized":"2024-03-06T07:24:02.156829+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:15:21.961725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448903+0000","last_change":"2024-03-06T07:24:00.452714+0000","last_active":"2024-03-06T07:24:01.448903+0000","last_peered":"2024-03-06T07:24:01.448903+0000","last_clean":"2024-03-06T07:24:01.448903+0000","last_became_active":"2024-03-06T07:24:00.452377+0000","last_became_peered":"2024-03-06T07:24:00.452377+0000","last_unstale":"2024-03-06T07:24:01.448903+0000","last_undegraded":"2024-03-06T07:24:01.448903+0000","last_fullsized":"2024-03-06T07:24:01.448903+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:30.827190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448497+0000","last_change":"2024-03-06T07:24:00.448276+0000","last_active":"2024-03-06T07:24:01.448497+0000","last_peered":"2024-03-06T07:24:01.448497+0000","last_clean":"2024-03-06T07:24:01.448497+0000","last_became_active":"2024-03-06T07:24:00.447961+0000","last_became_peered":"2024-03-06T07:24:00.447961+0000","last_unstale":"2024-03-06T07:24:01.448497+0000","last_undegraded":"2024-03-06T07:24:01.448497+0000","last_fullsized":"2024-03-06T07:24:01.448497+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:57.067784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469327+0000","last_change":"2024-03-06T07:24:00.444103+0000","last_active":"2024-03-06T07:24:02.469327+0000","last_peered":"2024-03-06T07:24:02.469327+0000","last_clean":"2024-03-06T07:24:02.469327+0000","last_became_active":"2024-03-06T07:24:00.443851+0000","last_became_peered":"2024-03-06T07:24:00.443851+0000","last_unstale":"2024-03-06T07:24:02.469327+0000","last_undegraded":"2024-03-06T07:24:02.469327+0000","last_fullsized":"2024-03-06T07:24:02.469327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466046+0000","last_change":"2024-03-06T07:24:00.443021+0000","last_active":"2024-03-06T07:24:02.466046+0000","last_peered":"2024-03-06T07:24:02.466046+0000","last_clean":"2024-03-06T07:24:02.466046+0000","last_became_active":"2024-03-06T07:24:00.442730+0000","last_became_peered":"2024-03-06T07:24:00.442730+0000","last_unstale":"2024-03-06T07:24:02.466046+0000","last_undegraded":"2024-03-06T07:24:02.466046+0000","last_fullsized":"2024-03-06T07:24:02.466046+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:26:48.309672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587794+0000","last_change":"2024-03-06T07:24:00.451841+0000","last_active":"2024-03-06T07:24:01.587794+0000","last_peered":"2024-03-06T07:24:01.587794+0000","last_clean":"2024-03-06T07:24:01.587794+0000","last_became_active":"2024-03-06T07:24:00.451473+0000","last_became_peered":"2024-03-06T07:24:00.451473+0000","last_unstale":"2024-03-06T07:24:01.587794+0000","last_undegraded":"2024-03-06T07:24:01.587794+0000","last_fullsized":"2024-03-06T07:24:01.587794+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:16:15.579899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449397+0000","last_change":"2024-03-06T07:24:00.439220+0000","last_active":"2024-03-06T07:24:01.449397+0000","last_peered":"2024-03-06T07:24:01.449397+0000","last_clean":"2024-03-06T07:24:01.449397+0000","last_became_active":"2024-03-06T07:24:00.438971+0000","last_became_peered":"2024-03-06T07:24:00.438971+0000","last_unstale":"2024-03-06T07:24:01.449397+0000","last_undegraded":"2024-03-06T07:24:01.449397+0000","last_fullsized":"2024-03-06T07:24:01.449397+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:11:21.634491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465841+0000","last_change":"2024-03-06T07:24:00.447150+0000","last_active":"2024-03-06T07:24:02.465841+0000","last_peered":"2024-03-06T07:24:02.465841+0000","last_clean":"2024-03-06T07:24:02.465841+0000","last_became_active":"2024-03-06T07:24:00.446837+0000","last_became_peered":"2024-03-06T07:24:00.446837+0000","last_unstale":"2024-03-06T07:24:02.465841+0000","last_undegraded":"2024-03-06T07:24:02.465841+0000","last_fullsized":"2024-03-06T07:24:02.465841+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:52:20.061391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587434+0000","last_change":"2024-03-06T07:24:00.442324+0000","last_active":"2024-03-06T07:24:01.587434+0000","last_peered":"2024-03-06T07:24:01.587434+0000","last_clean":"2024-03-06T07:24:01.587434+0000","last_became_active":"2024-03-06T07:24:00.442041+0000","last_became_peered":"2024-03-06T07:24:00.442041+0000","last_unstale":"2024-03-06T07:24:01.587434+0000","last_undegraded":"2024-03-06T07:24:01.587434+0000","last_fullsized":"2024-03-06T07:24:01.587434+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465644+0000","last_change":"2024-03-06T07:24:00.442786+0000","last_active":"2024-03-06T07:24:02.465644+0000","last_peered":"2024-03-06T07:24:02.465644+0000","last_clean":"2024-03-06T07:24:02.465644+0000","last_became_active":"2024-03-06T07:24:00.442534+0000","last_became_peered":"2024-03-06T07:24:00.442534+0000","last_unstale":"2024-03-06T07:24:02.465644+0000","last_undegraded":"2024-03-06T07:24:02.465644+0000","last_fullsized":"2024-03-06T07:24:02.465644+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:21:57.054784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587327+0000","last_change":"2024-03-06T07:24:00.445639+0000","last_active":"2024-03-06T07:24:01.587327+0000","last_peered":"2024-03-06T07:24:01.587327+0000","last_clean":"2024-03-06T07:24:01.587327+0000","last_became_active":"2024-03-06T07:24:00.445190+0000","last_became_peered":"2024-03-06T07:24:00.445190+0000","last_unstale":"2024-03-06T07:24:01.587327+0000","last_undegraded":"2024-03-06T07:24:01.587327+0000","last_fullsized":"2024-03-06T07:24:01.587327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.156761+0000","last_change":"2024-03-06T07:24:00.441727+0000","last_active":"2024-03-06T07:24:02.156761+0000","last_peered":"2024-03-06T07:24:02.156761+0000","last_clean":"2024-03-06T07:24:02.156761+0000","last_became_active":"2024-03-06T07:24:00.440305+0000","last_became_peered":"2024-03-06T07:24:00.440305+0000","last_unstale":"2024-03-06T07:24:02.156761+0000","last_undegraded":"2024-03-06T07:24:02.156761+0000","last_fullsized":"2024-03-06T07:24:02.156761+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:19:38.450025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449608+0000","last_change":"2024-03-06T07:24:00.446374+0000","last_active":"2024-03-06T07:24:01.449608+0000","last_peered":"2024-03-06T07:24:01.449608+0000","last_clean":"2024-03-06T07:24:01.449608+0000","last_became_active":"2024-03-06T07:24:00.446119+0000","last_became_peered":"2024-03-06T07:24:00.446119+0000","last_unstale":"2024-03-06T07:24:01.449608+0000","last_undegraded":"2024-03-06T07:24:01.449608+0000","last_fullsized":"2024-03-06T07:24:01.449608+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:37:20.937564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448327+0000","last_change":"2024-03-06T07:24:00.448205+0000","last_active":"2024-03-06T07:24:01.448327+0000","last_peered":"2024-03-06T07:24:01.448327+0000","last_clean":"2024-03-06T07:24:01.448327+0000","last_became_active":"2024-03-06T07:24:00.445713+0000","last_became_peered":"2024-03-06T07:24:00.445713+0000","last_unstale":"2024-03-06T07:24:01.448327+0000","last_undegraded":"2024-03-06T07:24:01.448327+0000","last_fullsized":"2024-03-06T07:24:01.448327+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:20.348235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469723+0000","last_change":"2024-03-06T07:24:00.446705+0000","last_active":"2024-03-06T07:24:02.469723+0000","last_peered":"2024-03-06T07:24:02.469723+0000","last_clean":"2024-03-06T07:24:02.469723+0000","last_became_active":"2024-03-06T07:24:00.446464+0000","last_became_peered":"2024-03-06T07:24:00.446464+0000","last_unstale":"2024-03-06T07:24:02.469723+0000","last_undegraded":"2024-03-06T07:24:02.469723+0000","last_fullsized":"2024-03-06T07:24:02.469723+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:07:11.486466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449580+0000","last_change":"2024-03-06T07:24:00.447941+0000","last_active":"2024-03-06T07:24:01.449580+0000","last_peered":"2024-03-06T07:24:01.449580+0000","last_clean":"2024-03-06T07:24:01.449580+0000","last_became_active":"2024-03-06T07:24:00.447687+0000","last_became_peered":"2024-03-06T07:24:00.447687+0000","last_unstale":"2024-03-06T07:24:01.449580+0000","last_undegraded":"2024-03-06T07:24:01.449580+0000","last_fullsized":"2024-03-06T07:24:01.449580+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:31:19.118141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466082+0000","last_change":"2024-03-06T07:24:00.442005+0000","last_active":"2024-03-06T07:24:02.466082+0000","last_peered":"2024-03-06T07:24:02.466082+0000","last_clean":"2024-03-06T07:24:02.466082+0000","last_became_active":"2024-03-06T07:24:00.441729+0000","last_became_peered":"2024-03-06T07:24:00.441729+0000","last_unstale":"2024-03-06T07:24:02.466082+0000","last_undegraded":"2024-03-06T07:24:02.466082+0000","last_fullsized":"2024-03-06T07:24:02.466082+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:40.808994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587466+0000","last_change":"2024-03-06T07:24:00.440566+0000","last_active":"2024-03-06T07:24:01.587466+0000","last_peered":"2024-03-06T07:24:01.587466+0000","last_clean":"2024-03-06T07:24:01.587466+0000","last_became_active":"2024-03-06T07:24:00.440019+0000","last_became_peered":"2024-03-06T07:24:00.440019+0000","last_unstale":"2024-03-06T07:24:01.587466+0000","last_undegraded":"2024-03-06T07:24:01.587466+0000","last_fullsized":"2024-03-06T07:24:01.587466+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449286+0000","last_change":"2024-03-06T07:24:00.445742+0000","last_active":"2024-03-06T07:24:01.449286+0000","last_peered":"2024-03-06T07:24:01.449286+0000","last_clean":"2024-03-06T07:24:01.449286+0000","last_became_active":"2024-03-06T07:24:00.445483+0000","last_became_peered":"2024-03-06T07:24:00.445483+0000","last_unstale":"2024-03-06T07:24:01.449286+0000","last_undegraded":"2024-03-06T07:24:01.449286+0000","last_fullsized":"2024-03-06T07:24:01.449286+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:28:02.426858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469391+0000","last_change":"2024-03-06T07:24:00.444808+0000","last_active":"2024-03-06T07:24:02.469391+0000","last_peered":"2024-03-06T07:24:02.469391+0000","last_clean":"2024-03-06T07:24:02.469391+0000","last_became_active":"2024-03-06T07:24:00.444516+0000","last_became_peered":"2024-03-06T07:24:00.444516+0000","last_unstale":"2024-03-06T07:24:02.469391+0000","last_undegraded":"2024-03-06T07:24:02.469391+0000","last_fullsized":"2024-03-06T07:24:02.469391+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469790+0000","last_change":"2024-03-06T07:24:00.447645+0000","last_active":"2024-03-06T07:24:02.469790+0000","last_peered":"2024-03-06T07:24:02.469790+0000","last_clean":"2024-03-06T07:24:02.469790+0000","last_became_active":"2024-03-06T07:24:00.445733+0000","last_became_peered":"2024-03-06T07:24:00.445733+0000","last_unstale":"2024-03-06T07:24:02.469790+0000","last_undegraded":"2024-03-06T07:24:02.469790+0000","last_fullsized":"2024-03-06T07:24:02.469790+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:39.577599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469830+0000","last_change":"2024-03-06T07:23:59.439938+0000","last_active":"2024-03-06T07:24:02.469830+0000","last_peered":"2024-03-06T07:24:02.469830+0000","last_clean":"2024-03-06T07:24:02.469830+0000","last_became_active":"2024-03-06T07:23:59.439672+0000","last_became_peered":"2024-03-06T07:23:59.439672+0000","last_unstale":"2024-03-06T07:24:02.469830+0000","last_undegraded":"2024-03-06T07:24:02.469830+0000","last_fullsized":"2024-03-06T07:24:02.469830+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.278339+0000","last_change":"2024-03-06T07:24:00.448305+0000","last_active":"2024-03-06T07:24:03.278339+0000","last_peered":"2024-03-06T07:24:03.278339+0000","last_clean":"2024-03-06T07:24:03.278339+0000","last_became_active":"2024-03-06T07:24:00.447621+0000","last_became_peered":"2024-03-06T07:24:00.447621+0000","last_unstale":"2024-03-06T07:24:03.278339+0000","last_undegraded":"2024-03-06T07:24:03.278339+0000","last_fullsized":"2024-03-06T07:24:03.278339+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:40.784770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465921+0000","last_change":"2024-03-06T07:23:59.440673+0000","last_active":"2024-03-06T07:24:02.465921+0000","last_peered":"2024-03-06T07:24:02.465921+0000","last_clean":"2024-03-06T07:24:02.465921+0000","last_became_active":"2024-03-06T07:23:59.440422+0000","last_became_peered":"2024-03-06T07:23:59.440422+0000","last_unstale":"2024-03-06T07:24:02.465921+0000","last_undegraded":"2024-03-06T07:24:02.465921+0000","last_fullsized":"2024-03-06T07:24:02.465921+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:23.304839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469689+0000","last_change":"2024-03-06T07:24:00.449741+0000","last_active":"2024-03-06T07:24:02.469689+0000","last_peered":"2024-03-06T07:24:02.469689+0000","last_clean":"2024-03-06T07:24:02.469689+0000","last_became_active":"2024-03-06T07:24:00.449211+0000","last_became_peered":"2024-03-06T07:24:00.449211+0000","last_unstale":"2024-03-06T07:24:02.469689+0000","last_undegraded":"2024-03-06T07:24:02.469689+0000","last_fullsized":"2024-03-06T07:24:02.469689+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:45.446431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469833+0000","last_change":"2024-03-06T07:23:59.435437+0000","last_active":"2024-03-06T07:24:02.469833+0000","last_peered":"2024-03-06T07:24:02.469833+0000","last_clean":"2024-03-06T07:24:02.469833+0000","last_became_active":"2024-03-06T07:23:59.435184+0000","last_became_peered":"2024-03-06T07:23:59.435184+0000","last_unstale":"2024-03-06T07:24:02.469833+0000","last_undegraded":"2024-03-06T07:24:02.469833+0000","last_fullsized":"2024-03-06T07:24:02.469833+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469717+0000","last_change":"2024-03-06T07:24:00.449515+0000","last_active":"2024-03-06T07:24:02.469717+0000","last_peered":"2024-03-06T07:24:02.469717+0000","last_clean":"2024-03-06T07:24:02.469717+0000","last_became_active":"2024-03-06T07:24:00.449232+0000","last_became_peered":"2024-03-06T07:24:00.449232+0000","last_unstale":"2024-03-06T07:24:02.469717+0000","last_undegraded":"2024-03-06T07:24:02.469717+0000","last_fullsized":"2024-03-06T07:24:02.469717+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:26.491484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448545+0000","last_change":"2024-03-06T07:23:59.439760+0000","last_active":"2024-03-06T07:24:01.448545+0000","last_peered":"2024-03-06T07:24:01.448545+0000","last_clean":"2024-03-06T07:24:01.448545+0000","last_became_active":"2024-03-06T07:23:59.439480+0000","last_became_peered":"2024-03-06T07:23:59.439480+0000","last_unstale":"2024-03-06T07:24:01.448545+0000","last_undegraded":"2024-03-06T07:24:01.448545+0000","last_fullsized":"2024-03-06T07:24:01.448545+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:21.416548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465767+0000","last_change":"2024-03-06T07:24:00.443357+0000","last_active":"2024-03-06T07:24:02.465767+0000","last_peered":"2024-03-06T07:24:02.465767+0000","last_clean":"2024-03-06T07:24:02.465767+0000","last_became_active":"2024-03-06T07:24:00.443130+0000","last_became_peered":"2024-03-06T07:24:00.443130+0000","last_unstale":"2024-03-06T07:24:02.465767+0000","last_undegraded":"2024-03-06T07:24:02.465767+0000","last_fullsized":"2024-03-06T07:24:02.465767+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:51.909308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449204+0000","last_change":"2024-03-06T07:23:59.441469+0000","last_active":"2024-03-06T07:24:01.449204+0000","last_peered":"2024-03-06T07:24:01.449204+0000","last_clean":"2024-03-06T07:24:01.449204+0000","last_became_active":"2024-03-06T07:23:59.441163+0000","last_became_peered":"2024-03-06T07:23:59.441163+0000","last_unstale":"2024-03-06T07:24:01.449204+0000","last_undegraded":"2024-03-06T07:24:01.449204+0000","last_fullsized":"2024-03-06T07:24:01.449204+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:36.767833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465913+0000","last_change":"2024-03-06T07:24:00.443896+0000","last_active":"2024-03-06T07:24:02.465913+0000","last_peered":"2024-03-06T07:24:02.465913+0000","last_clean":"2024-03-06T07:24:02.465913+0000","last_became_active":"2024-03-06T07:24:00.443638+0000","last_became_peered":"2024-03-06T07:24:00.443638+0000","last_unstale":"2024-03-06T07:24:02.465913+0000","last_undegraded":"2024-03-06T07:24:02.465913+0000","last_fullsized":"2024-03-06T07:24:02.465913+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:06.136349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469402+0000","last_change":"2024-03-06T07:23:59.438876+0000","last_active":"2024-03-06T07:24:02.469402+0000","last_peered":"2024-03-06T07:24:02.469402+0000","last_clean":"2024-03-06T07:24:02.469402+0000","last_became_active":"2024-03-06T07:23:59.438581+0000","last_became_peered":"2024-03-06T07:23:59.438581+0000","last_unstale":"2024-03-06T07:24:02.469402+0000","last_undegraded":"2024-03-06T07:24:02.469402+0000","last_fullsized":"2024-03-06T07:24:02.469402+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.057781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.156686+0000","last_change":"2024-03-06T07:24:00.435008+0000","last_active":"2024-03-06T07:24:02.156686+0000","last_peered":"2024-03-06T07:24:02.156686+0000","last_clean":"2024-03-06T07:24:02.156686+0000","last_became_active":"2024-03-06T07:24:00.433660+0000","last_became_peered":"2024-03-06T07:24:00.433660+0000","last_unstale":"2024-03-06T07:24:02.156686+0000","last_undegraded":"2024-03-06T07:24:02.156686+0000","last_fullsized":"2024-03-06T07:24:02.156686+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:02.387037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449045+0000","last_change":"2024-03-06T07:23:59.438832+0000","last_active":"2024-03-06T07:24:01.449045+0000","last_peered":"2024-03-06T07:24:01.449045+0000","last_clean":"2024-03-06T07:24:01.449045+0000","last_became_active":"2024-03-06T07:23:59.438564+0000","last_became_peered":"2024-03-06T07:23:59.438564+0000","last_unstale":"2024-03-06T07:24:01.449045+0000","last_undegraded":"2024-03-06T07:24:01.449045+0000","last_fullsized":"2024-03-06T07:24:01.449045+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.053733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465711+0000","last_change":"2024-03-06T07:24:00.443103+0000","last_active":"2024-03-06T07:24:02.465711+0000","last_peered":"2024-03-06T07:24:02.465711+0000","last_clean":"2024-03-06T07:24:02.465711+0000","last_became_active":"2024-03-06T07:24:00.442853+0000","last_became_peered":"2024-03-06T07:24:00.442853+0000","last_unstale":"2024-03-06T07:24:02.465711+0000","last_undegraded":"2024-03-06T07:24:02.465711+0000","last_fullsized":"2024-03-06T07:24:02.465711+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:48.180932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448304+0000","last_change":"2024-03-06T07:23:59.440421+0000","last_active":"2024-03-06T07:24:01.448304+0000","last_peered":"2024-03-06T07:24:01.448304+0000","last_clean":"2024-03-06T07:24:01.448304+0000","last_became_active":"2024-03-06T07:23:59.440160+0000","last_became_peered":"2024-03-06T07:23:59.440160+0000","last_unstale":"2024-03-06T07:24:01.448304+0000","last_undegraded":"2024-03-06T07:24:01.448304+0000","last_fullsized":"2024-03-06T07:24:01.448304+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:12.543578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449819+0000","last_change":"2024-03-06T07:24:00.447813+0000","last_active":"2024-03-06T07:24:01.449819+0000","last_peered":"2024-03-06T07:24:01.449819+0000","last_clean":"2024-03-06T07:24:01.449819+0000","last_became_active":"2024-03-06T07:24:00.447532+0000","last_became_peered":"2024-03-06T07:24:00.447532+0000","last_unstale":"2024-03-06T07:24:01.449819+0000","last_undegraded":"2024-03-06T07:24:01.449819+0000","last_fullsized":"2024-03-06T07:24:01.449819+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:13.646985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587813+0000","last_change":"2024-03-06T07:23:59.443581+0000","last_active":"2024-03-06T07:24:01.587813+0000","last_peered":"2024-03-06T07:24:01.587813+0000","last_clean":"2024-03-06T07:24:01.587813+0000","last_became_active":"2024-03-06T07:23:59.443241+0000","last_became_peered":"2024-03-06T07:23:59.443241+0000","last_unstale":"2024-03-06T07:24:01.587813+0000","last_undegraded":"2024-03-06T07:24:01.587813+0000","last_fullsized":"2024-03-06T07:24:01.587813+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:10:29.129622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587863+0000","last_change":"2024-03-06T07:24:00.449617+0000","last_active":"2024-03-06T07:24:01.587863+0000","last_peered":"2024-03-06T07:24:01.587863+0000","last_clean":"2024-03-06T07:24:01.587863+0000","last_became_active":"2024-03-06T07:24:00.447044+0000","last_became_peered":"2024-03-06T07:24:00.447044+0000","last_unstale":"2024-03-06T07:24:01.587863+0000","last_undegraded":"2024-03-06T07:24:01.587863+0000","last_fullsized":"2024-03-06T07:24:01.587863+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:05:45.446431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466022+0000","last_change":"2024-03-06T07:23:59.442149+0000","last_active":"2024-03-06T07:24:02.466022+0000","last_peered":"2024-03-06T07:24:02.466022+0000","last_clean":"2024-03-06T07:24:02.466022+0000","last_became_active":"2024-03-06T07:23:59.441779+0000","last_became_peered":"2024-03-06T07:23:59.441779+0000","last_unstale":"2024-03-06T07:24:02.466022+0000","last_undegraded":"2024-03-06T07:24:02.466022+0000","last_fullsized":"2024-03-06T07:24:02.466022+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:19.338232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.278086+0000","last_change":"2024-03-06T07:24:00.445041+0000","last_active":"2024-03-06T07:24:03.278086+0000","last_peered":"2024-03-06T07:24:03.278086+0000","last_clean":"2024-03-06T07:24:03.278086+0000","last_became_active":"2024-03-06T07:24:00.444726+0000","last_became_peered":"2024-03-06T07:24:00.444726+0000","last_unstale":"2024-03-06T07:24:03.278086+0000","last_undegraded":"2024-03-06T07:24:03.278086+0000","last_fullsized":"2024-03-06T07:24:03.278086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:15.615958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":17,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587905+0000","last_change":"2024-03-06T07:23:59.441620+0000","last_active":"2024-03-06T07:24:01.587905+0000","last_peered":"2024-03-06T07:24:01.587905+0000","last_clean":"2024-03-06T07:24:01.587905+0000","last_became_active":"2024-03-06T07:23:59.441352+0000","last_became_peered":"2024-03-06T07:23:59.441352+0000","last_unstale":"2024-03-06T07:24:01.587905+0000","last_undegraded":"2024-03-06T07:24:01.587905+0000","last_fullsized":"2024-03-06T07:24:01.587905+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:53.053733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587694+0000","last_change":"2024-03-06T07:24:00.449597+0000","last_active":"2024-03-06T07:24:01.587694+0000","last_peered":"2024-03-06T07:24:01.587694+0000","last_clean":"2024-03-06T07:24:01.587694+0000","last_became_active":"2024-03-06T07:24:00.449118+0000","last_became_peered":"2024-03-06T07:24:00.449118+0000","last_unstale":"2024-03-06T07:24:01.587694+0000","last_undegraded":"2024-03-06T07:24:01.587694+0000","last_fullsized":"2024-03-06T07:24:01.587694+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449762+0000","last_change":"2024-03-06T07:23:59.444415+0000","last_active":"2024-03-06T07:24:01.449762+0000","last_peered":"2024-03-06T07:24:01.449762+0000","last_clean":"2024-03-06T07:24:01.449762+0000","last_became_active":"2024-03-06T07:23:59.444120+0000","last_became_peered":"2024-03-06T07:23:59.444120+0000","last_unstale":"2024-03-06T07:24:01.449762+0000","last_undegraded":"2024-03-06T07:24:01.449762+0000","last_fullsized":"2024-03-06T07:24:01.449762+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:14.605955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448246+0000","last_change":"2024-03-06T07:24:00.449289+0000","last_active":"2024-03-06T07:24:01.448246+0000","last_peered":"2024-03-06T07:24:01.448246+0000","last_clean":"2024-03-06T07:24:01.448246+0000","last_became_active":"2024-03-06T07:24:00.447448+0000","last_became_peered":"2024-03-06T07:24:00.447448+0000","last_unstale":"2024-03-06T07:24:01.448246+0000","last_undegraded":"2024-03-06T07:24:01.448246+0000","last_fullsized":"2024-03-06T07:24:01.448246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:24.314842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466008+0000","last_change":"2024-03-06T07:23:59.443439+0000","last_active":"2024-03-06T07:24:02.466008+0000","last_peered":"2024-03-06T07:24:02.466008+0000","last_clean":"2024-03-06T07:24:02.466008+0000","last_became_active":"2024-03-06T07:23:59.443188+0000","last_became_peered":"2024-03-06T07:23:59.443188+0000","last_unstale":"2024-03-06T07:24:02.466008+0000","last_undegraded":"2024-03-06T07:24:02.466008+0000","last_fullsized":"2024-03-06T07:24:02.466008+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:38:47.951502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587625+0000","last_change":"2024-03-06T07:24:00.434521+0000","last_active":"2024-03-06T07:24:01.587625+0000","last_peered":"2024-03-06T07:24:01.587625+0000","last_clean":"2024-03-06T07:24:01.587625+0000","last_became_active":"2024-03-06T07:24:00.434247+0000","last_became_peered":"2024-03-06T07:24:00.434247+0000","last_unstale":"2024-03-06T07:24:01.587625+0000","last_undegraded":"2024-03-06T07:24:01.587625+0000","last_fullsized":"2024-03-06T07:24:01.587625+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.156658+0000","last_change":"2024-03-06T07:23:59.441682+0000","last_active":"2024-03-06T07:24:02.156658+0000","last_peered":"2024-03-06T07:24:02.156658+0000","last_clean":"2024-03-06T07:24:02.156658+0000","last_became_active":"2024-03-06T07:23:59.441406+0000","last_became_peered":"2024-03-06T07:23:59.441406+0000","last_unstale":"2024-03-06T07:24:02.156658+0000","last_undegraded":"2024-03-06T07:24:02.156658+0000","last_fullsized":"2024-03-06T07:24:02.156658+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:03:26.356183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448151+0000","last_change":"2024-03-06T07:24:00.451006+0000","last_active":"2024-03-06T07:24:01.448151+0000","last_peered":"2024-03-06T07:24:01.448151+0000","last_clean":"2024-03-06T07:24:01.448151+0000","last_became_active":"2024-03-06T07:24:00.450626+0000","last_became_peered":"2024-03-06T07:24:00.450626+0000","last_unstale":"2024-03-06T07:24:01.448151+0000","last_undegraded":"2024-03-06T07:24:01.448151+0000","last_fullsized":"2024-03-06T07:24:01.448151+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:14.268540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449005+0000","last_change":"2024-03-06T07:23:59.437649+0000","last_active":"2024-03-06T07:24:01.449005+0000","last_peered":"2024-03-06T07:24:01.449005+0000","last_clean":"2024-03-06T07:24:01.449005+0000","last_became_active":"2024-03-06T07:23:59.437377+0000","last_became_peered":"2024-03-06T07:23:59.437377+0000","last_unstale":"2024-03-06T07:24:01.449005+0000","last_undegraded":"2024-03-06T07:24:01.449005+0000","last_fullsized":"2024-03-06T07:24:01.449005+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:09:53.183909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587470+0000","last_change":"2024-03-06T07:24:00.446910+0000","last_active":"2024-03-06T07:24:01.587470+0000","last_peered":"2024-03-06T07:24:01.587470+0000","last_clean":"2024-03-06T07:24:01.587470+0000","last_became_active":"2024-03-06T07:24:00.445795+0000","last_became_peered":"2024-03-06T07:24:00.445795+0000","last_unstale":"2024-03-06T07:24:01.587470+0000","last_undegraded":"2024-03-06T07:24:01.587470+0000","last_fullsized":"2024-03-06T07:24:01.587470+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:27.281955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":22,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587273+0000","last_change":"2024-03-06T07:23:59.442282+0000","last_active":"2024-03-06T07:24:01.587273+0000","last_peered":"2024-03-06T07:24:01.587273+0000","last_clean":"2024-03-06T07:24:01.587273+0000","last_became_active":"2024-03-06T07:23:59.441992+0000","last_became_peered":"2024-03-06T07:23:59.441992+0000","last_unstale":"2024-03-06T07:24:01.587273+0000","last_undegraded":"2024-03-06T07:24:01.587273+0000","last_fullsized":"2024-03-06T07:24:01.587273+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:54:12.239306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587803+0000","last_change":"2024-03-06T07:24:00.448694+0000","last_active":"2024-03-06T07:24:01.587803+0000","last_peered":"2024-03-06T07:24:01.587803+0000","last_clean":"2024-03-06T07:24:01.587803+0000","last_became_active":"2024-03-06T07:24:00.448369+0000","last_became_peered":"2024-03-06T07:24:00.448369+0000","last_unstale":"2024-03-06T07:24:01.587803+0000","last_undegraded":"2024-03-06T07:24:01.587803+0000","last_fullsized":"2024-03-06T07:24:01.587803+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.121640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466039+0000","last_change":"2024-03-06T07:23:59.442248+0000","last_active":"2024-03-06T07:24:02.466039+0000","last_peered":"2024-03-06T07:24:02.466039+0000","last_clean":"2024-03-06T07:24:02.466039+0000","last_became_active":"2024-03-06T07:23:59.441980+0000","last_became_peered":"2024-03-06T07:23:59.441980+0000","last_unstale":"2024-03-06T07:24:02.466039+0000","last_undegraded":"2024-03-06T07:24:02.466039+0000","last_fullsized":"2024-03-06T07:24:02.466039+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:42.855432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448632+0000","last_change":"2024-03-06T07:24:00.446282+0000","last_active":"2024-03-06T07:24:01.448632+0000","last_peered":"2024-03-06T07:24:01.448632+0000","last_clean":"2024-03-06T07:24:01.448632+0000","last_became_active":"2024-03-06T07:24:00.445994+0000","last_became_peered":"2024-03-06T07:24:00.445994+0000","last_unstale":"2024-03-06T07:24:01.448632+0000","last_undegraded":"2024-03-06T07:24:01.448632+0000","last_fullsized":"2024-03-06T07:24:01.448632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:28:59.378316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469747+0000","last_change":"2024-03-06T07:23:59.440130+0000","last_active":"2024-03-06T07:24:02.469747+0000","last_peered":"2024-03-06T07:24:02.469747+0000","last_clean":"2024-03-06T07:24:02.469747+0000","last_became_active":"2024-03-06T07:23:59.439642+0000","last_became_peered":"2024-03-06T07:23:59.439642+0000","last_unstale":"2024-03-06T07:24:02.469747+0000","last_undegraded":"2024-03-06T07:24:02.469747+0000","last_fullsized":"2024-03-06T07:24:02.469747+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.817188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465953+0000","last_change":"2024-03-06T07:24:00.445971+0000","last_active":"2024-03-06T07:24:02.465953+0000","last_peered":"2024-03-06T07:24:02.465953+0000","last_clean":"2024-03-06T07:24:02.465953+0000","last_became_active":"2024-03-06T07:24:00.445707+0000","last_became_peered":"2024-03-06T07:24:00.445707+0000","last_unstale":"2024-03-06T07:24:02.465953+0000","last_undegraded":"2024-03-06T07:24:02.465953+0000","last_fullsized":"2024-03-06T07:24:02.465953+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:13:33.378144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448963+0000","last_change":"2024-03-06T07:23:59.438479+0000","last_active":"2024-03-06T07:24:01.448963+0000","last_peered":"2024-03-06T07:24:01.448963+0000","last_clean":"2024-03-06T07:24:01.448963+0000","last_became_active":"2024-03-06T07:23:59.438153+0000","last_became_peered":"2024-03-06T07:23:59.438153+0000","last_unstale":"2024-03-06T07:24:01.448963+0000","last_undegraded":"2024-03-06T07:24:01.448963+0000","last_fullsized":"2024-03-06T07:24:01.448963+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:52:13.258538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587879+0000","last_change":"2024-03-06T07:24:00.448405+0000","last_active":"2024-03-06T07:24:01.587879+0000","last_peered":"2024-03-06T07:24:01.587879+0000","last_clean":"2024-03-06T07:24:01.587879+0000","last_became_active":"2024-03-06T07:24:00.448018+0000","last_became_peered":"2024-03-06T07:24:00.448018+0000","last_unstale":"2024-03-06T07:24:01.587879+0000","last_undegraded":"2024-03-06T07:24:01.587879+0000","last_fullsized":"2024-03-06T07:24:01.587879+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:44:26.491484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469639+0000","last_change":"2024-03-06T07:23:59.440654+0000","last_active":"2024-03-06T07:24:02.469639+0000","last_peered":"2024-03-06T07:24:02.469639+0000","last_clean":"2024-03-06T07:24:02.469639+0000","last_became_active":"2024-03-06T07:23:59.440414+0000","last_became_peered":"2024-03-06T07:23:59.440414+0000","last_unstale":"2024-03-06T07:24:02.469639+0000","last_undegraded":"2024-03-06T07:24:02.469639+0000","last_fullsized":"2024-03-06T07:24:02.469639+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:54:21.416548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448400+0000","last_change":"2024-03-06T07:24:00.447077+0000","last_active":"2024-03-06T07:24:01.448400+0000","last_peered":"2024-03-06T07:24:01.448400+0000","last_clean":"2024-03-06T07:24:01.448400+0000","last_became_active":"2024-03-06T07:24:00.446780+0000","last_became_peered":"2024-03-06T07:24:00.446780+0000","last_unstale":"2024-03-06T07:24:01.448400+0000","last_undegraded":"2024-03-06T07:24:01.448400+0000","last_fullsized":"2024-03-06T07:24:01.448400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:36.842064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466118+0000","last_change":"2024-03-06T07:23:59.440760+0000","last_active":"2024-03-06T07:24:02.466118+0000","last_peered":"2024-03-06T07:24:02.466118+0000","last_clean":"2024-03-06T07:24:02.466118+0000","last_became_active":"2024-03-06T07:23:59.440490+0000","last_became_peered":"2024-03-06T07:23:59.440490+0000","last_unstale":"2024-03-06T07:24:02.466118+0000","last_undegraded":"2024-03-06T07:24:02.466118+0000","last_fullsized":"2024-03-06T07:24:02.466118+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:54:36.338297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469607+0000","last_change":"2024-03-06T07:24:00.447433+0000","last_active":"2024-03-06T07:24:02.469607+0000","last_peered":"2024-03-06T07:24:02.469607+0000","last_clean":"2024-03-06T07:24:02.469607+0000","last_became_active":"2024-03-06T07:24:00.447195+0000","last_became_peered":"2024-03-06T07:24:00.447195+0000","last_unstale":"2024-03-06T07:24:02.469607+0000","last_undegraded":"2024-03-06T07:24:02.469607+0000","last_fullsized":"2024-03-06T07:24:02.469607+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:30.121640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.277964+0000","last_change":"2024-03-06T07:23:59.439766+0000","last_active":"2024-03-06T07:24:03.277964+0000","last_peered":"2024-03-06T07:24:03.277964+0000","last_clean":"2024-03-06T07:24:03.277964+0000","last_became_active":"2024-03-06T07:23:59.439309+0000","last_became_peered":"2024-03-06T07:23:59.439309+0000","last_unstale":"2024-03-06T07:24:03.277964+0000","last_undegraded":"2024-03-06T07:24:03.277964+0000","last_fullsized":"2024-03-06T07:24:03.277964+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:32.312900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469363+0000","last_change":"2024-03-06T07:24:00.450031+0000","last_active":"2024-03-06T07:24:02.469363+0000","last_peered":"2024-03-06T07:24:02.469363+0000","last_clean":"2024-03-06T07:24:02.469363+0000","last_became_active":"2024-03-06T07:24:00.449743+0000","last_became_peered":"2024-03-06T07:24:00.449743+0000","last_unstale":"2024-03-06T07:24:02.469363+0000","last_undegraded":"2024-03-06T07:24:02.469363+0000","last_fullsized":"2024-03-06T07:24:02.469363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448089+0000","last_change":"2024-03-06T07:23:59.438050+0000","last_active":"2024-03-06T07:24:01.448089+0000","last_peered":"2024-03-06T07:24:01.448089+0000","last_clean":"2024-03-06T07:24:01.448089+0000","last_became_active":"2024-03-06T07:23:59.437721+0000","last_became_peered":"2024-03-06T07:23:59.437721+0000","last_unstale":"2024-03-06T07:24:01.448089+0000","last_undegraded":"2024-03-06T07:24:01.448089+0000","last_fullsized":"2024-03-06T07:24:01.448089+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:56.057781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448906+0000","last_change":"2024-03-06T07:24:00.445997+0000","last_active":"2024-03-06T07:24:01.448906+0000","last_peered":"2024-03-06T07:24:01.448906+0000","last_clean":"2024-03-06T07:24:01.448906+0000","last_became_active":"2024-03-06T07:24:00.445250+0000","last_became_peered":"2024-03-06T07:24:00.445250+0000","last_unstale":"2024-03-06T07:24:01.448906+0000","last_undegraded":"2024-03-06T07:24:01.448906+0000","last_fullsized":"2024-03-06T07:24:01.448906+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:09:31.312775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":16,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.278296+0000","last_change":"2024-03-06T07:23:59.435108+0000","last_active":"2024-03-06T07:24:03.278296+0000","last_peered":"2024-03-06T07:24:03.278296+0000","last_clean":"2024-03-06T07:24:03.278296+0000","last_became_active":"2024-03-06T07:23:59.434845+0000","last_became_peered":"2024-03-06T07:23:59.434845+0000","last_unstale":"2024-03-06T07:24:03.278296+0000","last_undegraded":"2024-03-06T07:24:03.278296+0000","last_fullsized":"2024-03-06T07:24:03.278296+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:04:35.611916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448816+0000","last_change":"2024-03-06T07:24:00.449232+0000","last_active":"2024-03-06T07:24:01.448816+0000","last_peered":"2024-03-06T07:24:01.448816+0000","last_clean":"2024-03-06T07:24:01.448816+0000","last_became_active":"2024-03-06T07:24:00.448976+0000","last_became_peered":"2024-03-06T07:24:00.448976+0000","last_unstale":"2024-03-06T07:24:01.448816+0000","last_undegraded":"2024-03-06T07:24:01.448816+0000","last_fullsized":"2024-03-06T07:24:01.448816+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:11:51.763728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"58'1","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469859+0000","last_change":"2024-03-06T07:23:59.439820+0000","last_active":"2024-03-06T07:24:02.469859+0000","last_peered":"2024-03-06T07:24:02.469859+0000","last_clean":"2024-03-06T07:24:02.469859+0000","last_became_active":"2024-03-06T07:23:59.439328+0000","last_became_peered":"2024-03-06T07:23:59.439328+0000","last_unstale":"2024-03-06T07:24:02.469859+0000","last_undegraded":"2024-03-06T07:24:02.469859+0000","last_fullsized":"2024-03-06T07:24:02.469859+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:12.636982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.278120+0000","last_change":"2024-03-06T07:24:00.444894+0000","last_active":"2024-03-06T07:24:03.278120+0000","last_peered":"2024-03-06T07:24:03.278120+0000","last_clean":"2024-03-06T07:24:03.278120+0000","last_became_active":"2024-03-06T07:24:00.444614+0000","last_became_peered":"2024-03-06T07:24:00.444614+0000","last_unstale":"2024-03-06T07:24:03.278120+0000","last_undegraded":"2024-03-06T07:24:03.278120+0000","last_fullsized":"2024-03-06T07:24:03.278120+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:19:37.777835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.156787+0000","last_change":"2024-03-06T07:23:59.440546+0000","last_active":"2024-03-06T07:24:02.156787+0000","last_peered":"2024-03-06T07:24:02.156787+0000","last_clean":"2024-03-06T07:24:02.156787+0000","last_became_active":"2024-03-06T07:23:59.440268+0000","last_became_peered":"2024-03-06T07:23:59.440268+0000","last_unstale":"2024-03-06T07:24:02.156787+0000","last_undegraded":"2024-03-06T07:24:02.156787+0000","last_fullsized":"2024-03-06T07:24:02.156787+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:53:32.312900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.469860+0000","last_change":"2024-03-06T07:24:00.448723+0000","last_active":"2024-03-06T07:24:02.469860+0000","last_peered":"2024-03-06T07:24:02.469860+0000","last_clean":"2024-03-06T07:24:02.469860+0000","last_became_active":"2024-03-06T07:24:00.448168+0000","last_became_peered":"2024-03-06T07:24:00.448168+0000","last_unstale":"2024-03-06T07:24:02.469860+0000","last_undegraded":"2024-03-06T07:24:02.469860+0000","last_fullsized":"2024-03-06T07:24:02.469860+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:59.617656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.448645+0000","last_change":"2024-03-06T07:23:59.440547+0000","last_active":"2024-03-06T07:24:01.448645+0000","last_peered":"2024-03-06T07:24:01.448645+0000","last_clean":"2024-03-06T07:24:01.448645+0000","last_became_active":"2024-03-06T07:23:59.440261+0000","last_became_peered":"2024-03-06T07:23:59.440261+0000","last_unstale":"2024-03-06T07:24:01.448645+0000","last_undegraded":"2024-03-06T07:24:01.448645+0000","last_fullsized":"2024-03-06T07:24:01.448645+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:12:29.817188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465928+0000","last_change":"2024-03-06T07:24:00.450414+0000","last_active":"2024-03-06T07:24:02.465928+0000","last_peered":"2024-03-06T07:24:02.465928+0000","last_clean":"2024-03-06T07:24:02.465928+0000","last_became_active":"2024-03-06T07:24:00.450152+0000","last_became_peered":"2024-03-06T07:24:00.450152+0000","last_unstale":"2024-03-06T07:24:02.465928+0000","last_undegraded":"2024-03-06T07:24:02.465928+0000","last_fullsized":"2024-03-06T07:24:02.465928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:51:56.363484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.277936+0000","last_change":"2024-03-06T07:23:59.436261+0000","last_active":"2024-03-06T07:24:03.277936+0000","last_peered":"2024-03-06T07:24:03.277936+0000","last_clean":"2024-03-06T07:24:03.277936+0000","last_became_active":"2024-03-06T07:23:59.436007+0000","last_became_peered":"2024-03-06T07:23:59.436007+0000","last_unstale":"2024-03-06T07:24:03.277936+0000","last_undegraded":"2024-03-06T07:24:03.277936+0000","last_fullsized":"2024-03-06T07:24:03.277936+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:46:05.778180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587653+0000","last_change":"2024-03-06T07:24:00.441206+0000","last_active":"2024-03-06T07:24:01.587653+0000","last_peered":"2024-03-06T07:24:01.587653+0000","last_clean":"2024-03-06T07:24:01.587653+0000","last_became_active":"2024-03-06T07:24:00.440923+0000","last_became_peered":"2024-03-06T07:24:00.440923+0000","last_unstale":"2024-03-06T07:24:01.587653+0000","last_undegraded":"2024-03-06T07:24:01.587653+0000","last_fullsized":"2024-03-06T07:24:01.587653+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:56:59.383865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.449011+0000","last_change":"2024-03-06T07:23:59.443108+0000","last_active":"2024-03-06T07:24:01.449011+0000","last_peered":"2024-03-06T07:24:01.449011+0000","last_clean":"2024-03-06T07:24:01.449011+0000","last_became_active":"2024-03-06T07:23:59.442847+0000","last_became_peered":"2024-03-06T07:23:59.442847+0000","last_unstale":"2024-03-06T07:24:01.449011+0000","last_undegraded":"2024-03-06T07:24:01.449011+0000","last_fullsized":"2024-03-06T07:24:01.449011+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:52:09.321526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.277899+0000","last_change":"2024-03-06T07:24:00.441365+0000","last_active":"2024-03-06T07:24:03.277899+0000","last_peered":"2024-03-06T07:24:03.277899+0000","last_clean":"2024-03-06T07:24:03.277899+0000","last_became_active":"2024-03-06T07:24:00.440667+0000","last_became_peered":"2024-03-06T07:24:00.440667+0000","last_unstale":"2024-03-06T07:24:03.277899+0000","last_undegraded":"2024-03-06T07:24:03.277899+0000","last_fullsized":"2024-03-06T07:24:03.277899+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:04:31.354402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:03.277919+0000","last_change":"2024-03-06T07:23:59.438513+0000","last_active":"2024-03-06T07:24:03.277919+0000","last_peered":"2024-03-06T07:24:03.277919+0000","last_clean":"2024-03-06T07:24:03.277919+0000","last_became_active":"2024-03-06T07:23:59.438207+0000","last_became_peered":"2024-03-06T07:23:59.438207+0000","last_unstale":"2024-03-06T07:24:03.277919+0000","last_undegraded":"2024-03-06T07:24:03.277919+0000","last_fullsized":"2024-03-06T07:24:03.277919+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:25:51.892786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.466059+0000","last_change":"2024-03-06T07:24:00.440384+0000","last_active":"2024-03-06T07:24:02.466059+0000","last_peered":"2024-03-06T07:24:02.466059+0000","last_clean":"2024-03-06T07:24:02.466059+0000","last_became_active":"2024-03-06T07:24:00.440128+0000","last_became_peered":"2024-03-06T07:24:00.440128+0000","last_unstale":"2024-03-06T07:24:02.466059+0000","last_undegraded":"2024-03-06T07:24:02.466059+0000","last_fullsized":"2024-03-06T07:24:02.466059+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:21.071067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465713+0000","last_change":"2024-03-06T07:23:59.437764+0000","last_active":"2024-03-06T07:24:02.465713+0000","last_peered":"2024-03-06T07:24:02.465713+0000","last_clean":"2024-03-06T07:24:02.465713+0000","last_became_active":"2024-03-06T07:23:59.437126+0000","last_became_peered":"2024-03-06T07:23:59.437126+0000","last_unstale":"2024-03-06T07:24:02.465713+0000","last_undegraded":"2024-03-06T07:24:02.465713+0000","last_fullsized":"2024-03-06T07:24:02.465713+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:33:46.102824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.465643+0000","last_change":"2024-03-06T07:24:00.440544+0000","last_active":"2024-03-06T07:24:02.465643+0000","last_peered":"2024-03-06T07:24:02.465643+0000","last_clean":"2024-03-06T07:24:02.465643+0000","last_became_active":"2024-03-06T07:24:00.440284+0000","last_became_peered":"2024-03-06T07:24:00.440284+0000","last_unstale":"2024-03-06T07:24:02.465643+0000","last_undegraded":"2024-03-06T07:24:02.465643+0000","last_fullsized":"2024-03-06T07:24:02.465643+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:59.419924+0000","last_clean_scrub_stamp":"2024-03-06T07:23:59.419924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:28:54.261448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:02.156736+0000","last_change":"2024-03-06T07:23:59.444024+0000","last_active":"2024-03-06T07:24:02.156736+0000","last_peered":"2024-03-06T07:24:02.156736+0000","last_clean":"2024-03-06T07:24:02.156736+0000","last_became_active":"2024-03-06T07:23:59.443744+0000","last_became_peered":"2024-03-06T07:23:59.443744+0000","last_unstale":"2024-03-06T07:24:02.156736+0000","last_undegraded":"2024-03-06T07:24:02.156736+0000","last_fullsized":"2024-03-06T07:24:02.156736+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:23:58.409921+0000","last_clean_scrub_stamp":"2024-03-06T07:23:58.409921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:53:06.553338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":242,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-06T07:24:01.587213+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:01.587213+0000","last_peered":"2024-03-06T07:24:01.587213+0000","last_clean":"2024-03-06T07:24:01.587213+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:01.587213+0000","last_undegraded":"2024-03-06T07:24:01.587213+0000","last_fullsized":"2024-03-06T07:24:01.587213+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32464,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339376,"statfs":{"total":96636764160,"available":96603521024,"internally_reserved":0,"allocated":1310720,"data_stored":885750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":708608,"data_stored":295222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":295874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":720896,"data_stored":296834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":716800,"data_stored":295278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":295278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":712704,"data_stored":295312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":887784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:08.365 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:24:08.365 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:24:08.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.511+0000 7feb961bc640 1 Processor -- start 2024-03-06T07:24:08.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.511+0000 7feb961bc640 1 -- start start 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb961bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feb901614d0 0x7feb900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb961bc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb901618b0 con 0x7feb900d1910 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb961bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb900cf400 con 0x7feb900d24b0 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb961bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb900cf6a0 con 0x7feb901614d0 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb86ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7feb900d1910 0x7feb900d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46426/0 (socket says 172.21.15.72:46426) 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb86ffd640 1 -- 172.21.15.72:0/3077874803 learned_addr learned my addr 172.21.15.72:0/3077874803 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239109377 0 0) 0x7feb901618b0 con 0x7feb900d1910 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.512+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb7c003520 con 0x7feb900d1910 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/3077874803 >> v2:172.21.15.146:3300/0 conn(0x7feb901614d0 0x7feb900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570825092 0 0) 0x7feb7c003520 con 0x7feb900d1910 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 >> v2:172.21.15.146:3300/0 conn(0x7feb901614d0 msgr2=0x7feb900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 --2- 172.21.15.72:0/3077874803 >> v2:172.21.15.146:3300/0 conn(0x7feb901614d0 0x7feb900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 >> v1:172.21.15.146:6789/0 conn(0x7feb900d24b0 legacy=0x7feb900b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900cf980 con 0x7feb900d1910 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7feb78001670 con 0x7feb900d1910 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/3077874803 >> v2:172.21.15.146:3300/0 conn(0x7feb901614d0 0x7feb900d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 >> v1:172.21.15.72:6789/0 conn(0x7feb900d1910 legacy=0x7feb900d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb7c0039b0 0x7feb7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7c004800 0x7feb7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0051c0 0x7feb7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb900cf6a0 con 0x7feb7c0039b0 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c00c110 con 0x7feb7c0051c0 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7c00c310 con 0x7feb7c004800 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb7c0039b0 0x7feb7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8ffff640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0051c0 0x7feb7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb86ffd640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7c004800 0x7feb7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 -- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7c004800 msgr2=0x7feb7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb7c004800 0x7feb7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 -- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0051c0 msgr2=0x7feb7c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0051c0 0x7feb7c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8f7fe640 1 -- 172.21.15.72:0/3077874803 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c003520 con 0x7feb7c0039b0 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.513+0000 7feb8ffff640 1 -- 172.21.15.72:0/3077874803 reap_dead start 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.514+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb7c0039b0 0x7feb7c003dd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7feb800098a0 tx=0x7feb7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61482d9c0e35af20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.514+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000f040 con 0x7feb7c0039b0 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.514+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80004230 con 0x7feb7c0039b0 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.514+0000 7feb8effd640 1 -- 172.21.15.72:0/3077874803 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80003610 con 0x7feb7c0039b0 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.514+0000 7feb961bc640 1 -- 172.21.15.72:0/3077874803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb7c0039b0 msgr2=0x7feb7c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.515+0000 7feb961bc640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb7c0039b0 0x7feb7c003dd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7feb800098a0 tx=0x7feb7c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.515+0000 7feb961bc640 1 -- 172.21.15.72:0/3077874803 shutdown_connections 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.515+0000 7feb961bc640 1 --2- 172.21.15.72:0/3077874803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb7c0039b0 0x7feb7c003dd0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.515+0000 7feb961bc640 1 -- 172.21.15.72:0/3077874803 >> 172.21.15.72:0/3077874803 conn(0x7feb900d3d40 msgr2=0x7feb900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.515+0000 7feb961bc640 1 -- 172.21.15.72:0/3077874803 shutdown_connections 2024-03-06T07:24:08.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.515+0000 7feb961bc640 1 -- 172.21.15.72:0/3077874803 wait complete. 2024-03-06T07:24:08.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.516+0000 7feb961bc640 1 Processor -- start 2024-03-06T07:24:08.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.516+0000 7feb961bc640 1 -- start start 2024-03-06T07:24:08.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0039b0 0x7feb90160310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb900cb820 0x7feb900c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 0x7feb900ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb900d4860 con 0x7feb900cd8e0 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb900b0910 con 0x7feb7c0039b0 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb900d49e0 con 0x7feb900cb820 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 0x7feb900ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 0x7feb900ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34022/0 (socket says 172.21.15.72:34022) 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 -- 172.21.15.72:0/4096437087 learned_addr learned my addr 172.21.15.72:0/4096437087 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb86ffd640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb900cb820 0x7feb900c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0039b0 0x7feb90160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 -- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb900cb820 msgr2=0x7feb900c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb900cb820 0x7feb900c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 -- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0039b0 msgr2=0x7feb90160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0039b0 0x7feb90160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 -- 172.21.15.72:0/4096437087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb900cbd90 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb86ffd640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb900cb820 0x7feb900c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0039b0 0x7feb90160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8ffff640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 0x7feb900ca130 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7feb900b2ae0 tx=0x7feb8800e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88019070 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb88006d60 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb900cc070 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.517+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb88003bb0 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.518+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb900cc630 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.519+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb90068ff0 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.519+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feb8801c070 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.519+0000 7feb8cff9640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb7006b4d0 0x7feb7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.520+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb7006b4d0 0x7feb7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.520+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb88089540 con 0x7feb900cd8e0 2024-03-06T07:24:08.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.520+0000 7feb8f7fe640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb7006b4d0 0x7feb7006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feb80035000 tx=0x7feb8003a040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.524+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feb88058510 con 0x7feb900cd8e0 2024-03-06T07:24:08.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.781+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feb900a1c10 con 0x7feb900cd8e0 2024-03-06T07:24:08.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.783+0000 7feb8cff9640 1 -- 172.21.15.72:0/4096437087 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7feb88057e10 con 0x7feb900cd8e0 2024-03-06T07:24:08.785 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:24:08.785 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:24:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:24:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:24:08.786 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:08.786 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.785+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb7006b4d0 msgr2=0x7feb7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.785+0000 7feb961bc640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb7006b4d0 0x7feb7006d990 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feb80035000 tx=0x7feb8003a040 comp rx=0 tx=0).stop 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.785+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 msgr2=0x7feb900ca130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.785+0000 7feb961bc640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 0x7feb900ca130 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7feb900b2ae0 tx=0x7feb8800e980 comp rx=0 tx=0).stop 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 shutdown_connections 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feb7006b4d0 0x7feb7006d990 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb900cd8e0 0x7feb900ca130 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb900cb820 0x7feb900c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 --2- 172.21.15.72:0/4096437087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb7c0039b0 0x7feb90160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 >> 172.21.15.72:0/4096437087 conn(0x7feb900d3d40 msgr2=0x7feb900b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 shutdown_connections 2024-03-06T07:24:08.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.786+0000 7feb961bc640 1 -- 172.21.15.72:0/4096437087 wait complete. 2024-03-06T07:24:08.799 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:08.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.898+0000 7f684aa40640 1 Processor -- start 2024-03-06T07:24:08.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f684aa40640 1 -- start start 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f684aa40640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f684415dfe0 0x7f68440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f684aa40640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68440cf830 0x7f68440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f684aa40640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440b3430 con 0x7f68440d27f0 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f684aa40640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440b31e0 con 0x7f684415dfe0 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f684aa40640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68440b3d60 con 0x7f68440cf830 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f6843fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f684415dfe0 0x7f68440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f6848fdd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f68440d27f0 0x7f68440d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46436/0 (socket says 172.21.15.72:46436) 2024-03-06T07:24:08.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.899+0000 7f6848fdd640 1 -- 172.21.15.72:0/4270209626 learned_addr learned my addr 172.21.15.72:0/4270209626 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:08.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f68437fe640 1 --2- 172.21.15.72:0/4270209626 >> v2:172.21.15.146:3300/0 conn(0x7f68440cf830 0x7f68440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f6843fff640 1 -- 172.21.15.72:0/4270209626 >> v2:172.21.15.146:3300/0 conn(0x7f68440cf830 msgr2=0x7f68440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:08.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f6843fff640 1 --2- 172.21.15.72:0/4270209626 >> v2:172.21.15.146:3300/0 conn(0x7f68440cf830 0x7f68440d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f6843fff640 1 -- 172.21.15.72:0/4270209626 >> v1:172.21.15.72:6789/0 conn(0x7f68440d27f0 legacy=0x7f68440d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f6843fff640 1 -- 172.21.15.72:0/4270209626 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68440b3710 con 0x7f684415dfe0 2024-03-06T07:24:08.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f6842ffd640 1 -- 172.21.15.72:0/4270209626 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146754154 0 0) 0x7f68440b3430 con 0x7f68440d27f0 2024-03-06T07:24:08.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.900+0000 7f6843fff640 1 --2- 172.21.15.72:0/4270209626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684415dfe0 0x7f68440cf2f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f68340097a0 tx=0x7f683402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69531be0f94be652 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f6842ffd640 1 -- 172.21.15.72:0/4270209626 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834035020 con 0x7f684415dfe0 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f6842ffd640 1 -- 172.21.15.72:0/4270209626 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6834031830 con 0x7f684415dfe0 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f6842ffd640 1 -- 172.21.15.72:0/4270209626 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834031c00 con 0x7f684415dfe0 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 -- 172.21.15.72:0/4270209626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684415dfe0 msgr2=0x7f68440cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 --2- 172.21.15.72:0/4270209626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684415dfe0 0x7f68440cf2f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f68340097a0 tx=0x7f683402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 -- 172.21.15.72:0/4270209626 shutdown_connections 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 --2- 172.21.15.72:0/4270209626 >> v2:172.21.15.146:3300/0 conn(0x7f68440cf830 0x7f68440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 --2- 172.21.15.72:0/4270209626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f684415dfe0 0x7f68440cf2f0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 -- 172.21.15.72:0/4270209626 >> 172.21.15.72:0/4270209626 conn(0x7f68440d3cd0 msgr2=0x7f68440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 -- 172.21.15.72:0/4270209626 shutdown_connections 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.901+0000 7f684aa40640 1 -- 172.21.15.72:0/4270209626 wait complete. 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.902+0000 7f684aa40640 1 Processor -- start 2024-03-06T07:24:08.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 -- start start 2024-03-06T07:24:08.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68440cf830 0x7f68440b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 0x7f68441f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684415dfe0 0x7f684415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68440d1df0 con 0x7f68440d27f0 2024-03-06T07:24:08.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68440d1c70 con 0x7f684415dfe0 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f684aa40640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684415e3c0 con 0x7f68440cf830 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f68437fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 0x7f68441f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f68437fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 0x7f68441f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34040/0 (socket says 172.21.15.72:34040) 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.903+0000 7f68437fe640 1 -- 172.21.15.72:0/487046015 learned_addr learned my addr 172.21.15.72:0/487046015 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f6848fdd640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684415dfe0 0x7f684415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f6843fff640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68440cf830 0x7f68440b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f68437fe640 1 -- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68440cf830 msgr2=0x7f68440b8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f68437fe640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68440cf830 0x7f68440b8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f68437fe640 1 -- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684415dfe0 msgr2=0x7f684415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f68437fe640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684415dfe0 0x7f684415b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f68437fe640 1 -- 172.21.15.72:0/487046015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68440b95a0 con 0x7f68440d27f0 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f6843fff640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68440cf830 0x7f68440b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f68437fe640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 0x7f68441f3d50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f683800d5c0 tx=0x7f683800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68380125f0 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6838019070 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68380129d0 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68441f6d60 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.904+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68441f7320 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.905+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684404f330 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.905+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6838012b70 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.906+0000 7f6840ff9640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f683006b4d0 0x7f683006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.906+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f683808a4e0 con 0x7f68440d27f0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.906+0000 7f6843fff640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f683006b4d0 0x7f683006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:08.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.906+0000 7f6843fff640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f683006b4d0 0x7f683006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6834036000 tx=0x7f6834008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:08.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:08.910+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68380593f0 con 0x7f68440d27f0 2024-03-06T07:24:09.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.118+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68440747a0 con 0x7f68440d27f0 2024-03-06T07:24:09.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.118+0000 7f6840ff9640 1 -- 172.21.15.72:0/487046015 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f6838058cf0 con 0x7f68440d27f0 2024-03-06T07:24:09.121 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:09.121 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:09.121 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f683006b4d0 msgr2=0x7f683006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f683006b4d0 0x7f683006d990 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6834036000 tx=0x7f6834008030 comp rx=0 tx=0).stop 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 msgr2=0x7f68441f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 0x7f68441f3d50 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f683800d5c0 tx=0x7f683800da80 comp rx=0 tx=0).stop 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 shutdown_connections 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f683006b4d0 0x7f683006d990 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684415dfe0 0x7f684415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68440d27f0 0x7f68441f3d50 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 --2- 172.21.15.72:0/487046015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68440cf830 0x7f68440b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 >> 172.21.15.72:0/487046015 conn(0x7f68440d3cd0 msgr2=0x7f68440d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 shutdown_connections 2024-03-06T07:24:09.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.121+0000 7f684aa40640 1 -- 172.21.15.72:0/487046015 wait complete. 2024-03-06T07:24:09.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:23:59.778464+0000', 'modified': '2024-03-06T07:24:01.798443+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6057}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6057': {'gid': 6057, 'name': 'c', 'rank': 0, 'incarnation': 25, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/3136290264', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 3136290264}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 3136290264}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6057, 'qdb_cluster': [6057]} 2024-03-06T07:24:09.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:24:09.541 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:24:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.646+0000 7fd1b2e2a640 1 Processor -- start 2024-03-06T07:24:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.646+0000 7fd1b2e2a640 1 -- start start 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b2e2a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd1ac0b4460 0x7fd1ac0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b2e2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b2e2a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1ac161cf0 0x7fd1ac162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b2e2a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ac161b00 con 0x7fd1ac161cf0 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b2e2a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1ac0cecc0 con 0x7fd1ac0b4460 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b2e2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0d22c0 con 0x7fd1ac162970 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.647+0000 7fd1b13c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1ac161cf0 0x7fd1ac162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1ac161cf0 0x7fd1ac162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34050/0 (socket says 172.21.15.72:34050) 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3988902050 learned_addr learned my addr 172.21.15.72:0/3988902050 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3988902050 >> v2:172.21.15.146:3300/0 conn(0x7fd1ac0b4460 0x7fd1ac0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3988902050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3988902050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 msgr2=0x7fd1ac0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3988902050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3988902050 >> v2:172.21.15.146:3300/0 conn(0x7fd1ac0b4460 msgr2=0x7fd1ac0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3988902050 >> v2:172.21.15.146:3300/0 conn(0x7fd1ac0b4460 0x7fd1ac0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3988902050 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac0d25a0 con 0x7fd1ac161cf0 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.648+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3988902050 >> v2:172.21.15.146:3300/0 conn(0x7fd1ac0b4460 0x7fd1ac0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3988902050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac162180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac08ddd0 tx=0x7fd1a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47f761e664d8d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1ab7fe640 1 -- 172.21.15.72:0/3988902050 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a402ca70 con 0x7fd1ac161cf0 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1ab7fe640 1 -- 172.21.15.72:0/3988902050 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a402cc10 con 0x7fd1ac161cf0 2024-03-06T07:24:09.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1ab7fe640 1 -- 172.21.15.72:0/3988902050 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40343f0 con 0x7fd1ac161cf0 2024-03-06T07:24:09.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3988902050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 msgr2=0x7fd1ac162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3988902050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac162180 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac08ddd0 tx=0x7fd1a402bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3988902050 shutdown_connections 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3988902050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac162180 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3988902050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3988902050 >> v2:172.21.15.146:3300/0 conn(0x7fd1ac0b4460 0x7fd1ac0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3988902050 >> 172.21.15.72:0/3988902050 conn(0x7fd1ac0d3ce0 msgr2=0x7fd1ac0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3988902050 shutdown_connections 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.649+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3988902050 wait complete. 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.651+0000 7fd1b2e2a640 1 Processor -- start 2024-03-06T07:24:09.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.651+0000 7fd1b2e2a640 1 -- start start 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b2e2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ac0b4460 0x7fd1ac1f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b2e2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b2e2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac1561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b2e2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0ce950 con 0x7fd1ac161cf0 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b2e2a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0d2b60 con 0x7fd1ac0b4460 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b2e2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0b3ac0 con 0x7fd1ac162970 2024-03-06T07:24:09.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac0c9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34056/0 (socket says 172.21.15.72:34056) 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 -- 172.21.15.72:0/3591145115 learned_addr learned my addr 172.21.15.72:0/3591145115 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 -- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 msgr2=0x7fd1ac1561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac1561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 -- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ac0b4460 msgr2=0x7fd1ac1f7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ac0b4460 0x7fd1ac1f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ac0b4460 0x7fd1ac1f7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1abfff640 1 -- 172.21.15.72:0/3591145115 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac0c95c0 con 0x7fd1ac161cf0 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.652+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ac0b4460 0x7fd1ac1f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.653+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac0c9080 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd19800e9b0 tx=0x7fd19800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.653+0000 7fd1a97fa640 1 -- 172.21.15.72:0/3591145115 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198017070 con 0x7fd1ac161cf0 2024-03-06T07:24:09.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.653+0000 7fd1a97fa640 1 -- 172.21.15.72:0/3591145115 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19800c9a0 con 0x7fd1ac161cf0 2024-03-06T07:24:09.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.653+0000 7fd1a97fa640 1 -- 172.21.15.72:0/3591145115 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19800cd70 con 0x7fd1ac161cf0 2024-03-06T07:24:09.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.653+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ac0b6a30 con 0x7fd1ac161cf0 2024-03-06T07:24:09.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.653+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ac0b6ff0 con 0x7fd1ac161cf0 2024-03-06T07:24:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.655+0000 7fd1a97fa640 1 -- 172.21.15.72:0/3591145115 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd19801a070 con 0x7fd1ac161cf0 2024-03-06T07:24:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.655+0000 7fd1a97fa640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19006b500 0x7fd19006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.655+0000 7fd1a97fa640 1 -- 172.21.15.72:0/3591145115 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd19808a080 con 0x7fd1ac161cf0 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.655+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19006b500 0x7fd19006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.655+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19006b500 0x7fd19006d9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0006fd0 tx=0x7fd1a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.656+0000 7fd1b5a13740 1 Processor -- start 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.656+0000 7fd1b5a13740 1 -- start start 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1321a870 0x560a1321ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1321e8c0 0x560a13220360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a1324aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a131846b0 con 0x560a1324a600 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1312da60 con 0x560a1321e8c0 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a13180750 con 0x560a1321a870 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b13c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a1324aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b13c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a1324aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34062/0 (socket says 172.21.15.72:34062) 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3702566117 learned_addr learned my addr 172.21.15.72:0/3702566117 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1321e8c0 0x560a13220360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1321a870 msgr2=0x560a1321ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.657+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1321a870 0x560a1321ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1321a870 0x560a1321ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1321e8c0 msgr2=0x560a13220360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1321e8c0 0x560a13220360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b13c7640 1 -- 172.21.15.72:0/3702566117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a1314bf30 con 0x560a1324a600 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a1324aa80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009f40 tx=0x7fd1a402e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcd9e311566e549 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1967fc640 1 -- 172.21.15.72:0/3702566117 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4009510 con 0x560a1324a600 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1967fc640 1 -- 172.21.15.72:0/3702566117 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4044070 con 0x560a1324a600 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1967fc640 1 -- 172.21.15.72:0/3702566117 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a402cab0 con 0x560a1324a600 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3702566117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 msgr2=0x560a1324aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a1324aa80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009f40 tx=0x7fd1a402e070 comp rx=0 tx=0).stop 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3702566117 shutdown_connections 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a1324aa80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1321e8c0 0x560a13220360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3702566117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1321a870 0x560a1321ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.658+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3702566117 >> 172.21.15.72:0/3702566117 conn(0x560a1321c8f0 msgr2=0x560a1321e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.659+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3702566117 shutdown_connections 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.659+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3702566117 wait complete. 2024-03-06T07:24:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.659+0000 7fd1b5a13740 1 -- start start 2024-03-06T07:24:09.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.660+0000 7fd1b5a13740 1 Processor -- start 2024-03-06T07:24:09.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13242ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13243010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a132407c0 0x560a13240c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1321ed50 con 0x560a1324a600 2024-03-06T07:24:09.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a13140ea0 con 0x560a132407c0 2024-03-06T07:24:09.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a1321f030 con 0x560a1323ec30 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b5a13740 10 client.0 _ll_register_callbacks cb 0x560a13140870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13242ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13242ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34078/0 (socket says 172.21.15.72:34078) 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 -- 172.21.15.72:0/1111069669 learned_addr learned my addr 172.21.15.72:0/1111069669 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a132407c0 0x560a13240c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1abfff640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13243010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 -- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 msgr2=0x560a13243010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13243010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 -- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a132407c0 msgr2=0x560a13240c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a132407c0 0x560a13240c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 -- 172.21.15.72:0/1111069669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a13141270 con 0x560a1324a600 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13242ad0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000efc0 tx=0x7fd1a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2158cd2814e36cd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.661+0000 7fd18b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd18b7fe640 1 -- 172.21.15.72:0/1111069669 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a000f040 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd18b7fe640 1 -- 172.21.15.72:0/1111069669 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1a0015dd0 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1a4044070 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd18b7fe640 1 -- 172.21.15.72:0/1111069669 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1a0005990 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 10 client.6246 fetch_fsmap learned FSMap version 27 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x560a12e992d0 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd18b7fe640 1 -- 172.21.15.72:0/1111069669 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4044070 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd18b7fe640 1 -- 172.21.15.72:0/1111069669 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 27) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd1a0030080 con 0x560a1324a600 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 10 client.6246 fetch_fsmap finished waiting for FSMap version 27 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 10 client.6246 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:09.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 20 client.6246 populate_metadata read hostname 'smithi072' 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a1322d480 0x560a13246a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 4 client.6246 mds_command: new command op to 6057 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 --> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a13191bf0 con 0x560a1322d480 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.662+0000 7fd1abfff640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a1322d480 0x560a13246a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd18b7fe640 10 client.6246 ms_handle_connect on v2:172.21.15.72:6835/3136290264 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1abfff640 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a1322d480 0x560a13246a40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d7d8ced856f2f2 server_cookie=55852c4ede2c9a28 in_seq=0 out_seq=0 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd18b7fe640 1 -- 172.21.15.72:0/1111069669 <== mds.0 v2:172.21.15.72:6835/3136290264 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560a13191bf0 con 0x560a1322d480 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd18b7fe640 10 client.6246 handle_command_reply: tid=0 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 client.6246 shutdown 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 msgr2=0x560a13242ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13242ad0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000efc0 tx=0x7fd1a0004730 comp rx=0 tx=0).stop 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 shutdown_connections 2024-03-06T07:24:09.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a1322d480 0x560a13246a40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a132407c0 0x560a13240c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13243010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1111069669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13242ad0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd18b7fe640 0 client.6246 ms_handle_reset on v2:172.21.15.72:6835/3136290264 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.663+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 >> 172.21.15.72:0/1111069669 conn(0x560a1321a870 msgr2=0x560a132485c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.664+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 shutdown_connections 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.664+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1111069669 wait complete. 2024-03-06T07:24:09.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.664+0000 7fd1b5a13740 20 client.6246 trim_cache size 0 max 16384 2024-03-06T07:24:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.695+0000 7fd1b5a13740 1 Processor -- start 2024-03-06T07:24:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.695+0000 7fd1b5a13740 1 -- start start 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1323ec30 0x560a1321e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a132407c0 0x560a13242c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13287e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0c83d0 con 0x560a1324a600 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0c7800 con 0x560a1323ec30 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac0c7dd0 con 0x560a132407c0 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b13c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13287e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b13c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13287e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34090/0 (socket says 172.21.15.72:34090) 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b13c7640 1 -- 172.21.15.72:0/1390279649 learned_addr learned my addr 172.21.15.72:0/1390279649 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1323ec30 0x560a1321e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b13c7640 1 -- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a132407c0 msgr2=0x560a13242c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.696+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a132407c0 0x560a13242c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b13c7640 1 -- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1323ec30 msgr2=0x560a1321e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1323ec30 0x560a1321e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b13c7640 1 -- 172.21.15.72:0/1390279649 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a13240f90 con 0x560a1324a600 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13287e40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x560a1324ecf0 tx=0x7fd1a4030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123b0305f79b4616 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd195ffb640 1 -- 172.21.15.72:0/1390279649 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4003b90 con 0x560a1324a600 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd195ffb640 1 -- 172.21.15.72:0/1390279649 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4044040 con 0x560a1324a600 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd195ffb640 1 -- 172.21.15.72:0/1390279649 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a402c870 con 0x560a1324a600 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1390279649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 msgr2=0x560a13287e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13287e40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x560a1324ecf0 tx=0x7fd1a4030000 comp rx=0 tx=0).stop 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1390279649 shutdown_connections 2024-03-06T07:24:09.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a1324a600 0x560a13287e40 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a132407c0 0x560a13242c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/1390279649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1323ec30 0x560a1321e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1390279649 >> 172.21.15.72:0/1390279649 conn(0x560a1321a870 msgr2=0x560a132476d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1390279649 shutdown_connections 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.697+0000 7fd1b5a13740 1 -- 172.21.15.72:0/1390279649 wait complete. 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.698+0000 7fd1b5a13740 1 -- start start 2024-03-06T07:24:09.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.698+0000 7fd1b5a13740 1 Processor -- start 2024-03-06T07:24:09.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13249570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 0x560a13249ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1324a600 0x560a13249ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a1322d300 con 0x560a132407c0 2024-03-06T07:24:09.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a132472d0 con 0x560a1324a600 2024-03-06T07:24:09.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560a1322d8a0 con 0x560a1323ec30 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b5a13740 10 client.0 _ll_register_callbacks cb 0x7fd1a000ec40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 0x560a13249ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 0x560a13249ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34096/0 (socket says 172.21.15.72:34096) 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 -- 172.21.15.72:0/3463100982 learned_addr learned my addr 172.21.15.72:0/3463100982 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b13c7640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1324a600 0x560a13249ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13249570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 -- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 msgr2=0x560a13249570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13249570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 -- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1324a600 msgr2=0x560a13249ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1324a600 0x560a13249ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 -- 172.21.15.72:0/3463100982 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a12f636b0 con 0x560a132407c0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd1abfff640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 0x560a13249ab0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd198054cf0 tx=0x7fd198059100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b579364f7bf10f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.699+0000 7fd18b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd18b7fe640 1 -- 172.21.15.72:0/3463100982 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198085070 con 0x560a132407c0 2024-03-06T07:24:09.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd18b7fe640 1 -- 172.21.15.72:0/3463100982 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd198054f00 con 0x560a132407c0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1a4044040 con 0x560a132407c0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd18b7fe640 1 -- 172.21.15.72:0/3463100982 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd198021040 con 0x560a132407c0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 10 client.6252 fetch_fsmap learned FSMap version 27 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x560a13247c90 con 0x560a132407c0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd18b7fe640 1 -- 172.21.15.72:0/3463100982 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4044040 con 0x560a132407c0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd18b7fe640 1 -- 172.21.15.72:0/3463100982 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 27) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd198088020 con 0x560a132407c0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 10 client.6252 fetch_fsmap finished waiting for FSMap version 27 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 10 client.6252 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 20 client.6252 populate_metadata read hostname 'smithi072' 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a132d4ad0 0x560a132d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 4 client.6252 mds_command: new command op to 6057 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 --> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560a1318bf40 con 0x560a132d4ad0 2024-03-06T07:24:09.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a132d4ad0 0x560a132d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd18b7fe640 10 client.6252 ms_handle_connect on v2:172.21.15.72:6835/3136290264 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.700+0000 7fd1b0bc6640 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a132d4ad0 0x560a132d4eb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcd7a6523075be5e server_cookie=462e95c5201c6888 in_seq=0 out_seq=0 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd18b7fe640 1 -- 172.21.15.72:0/3463100982 <== mds.0 v2:172.21.15.72:6835/3136290264 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x560a1318bf40 con 0x560a132d4ad0 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd18b7fe640 10 client.6252 handle_command_reply: tid=0 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 client.6252 shutdown 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 msgr2=0x560a13249ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 0x560a13249ab0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd198054cf0 tx=0x7fd198059100 comp rx=0 tx=0).stop 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 shutdown_connections 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:6835/3136290264,v1:172.21.15.72:6837/3136290264] conn(0x560a132d4ad0 0x560a132d4eb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560a1324a600 0x560a13249ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560a132407c0 0x560a13249ab0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 --2- 172.21.15.72:0/3463100982 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560a1323ec30 0x560a13249570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 >> 172.21.15.72:0/3463100982 conn(0x560a1321a870 msgr2=0x560a13220090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd18b7fe640 0 client.6252 ms_handle_reset on v2:172.21.15.72:6835/3136290264 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 shutdown_connections 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 1 -- 172.21.15.72:0/3463100982 wait complete. 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b5a13740 20 client.6252 trim_cache size 0 max 16384 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6057, 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:24:09.710 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:24:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 9.2613483179999996, 2024-03-06T07:24:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 27, 2024-03-06T07:24:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 59, 2024-03-06T07:24:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 59, 2024-03-06T07:24:09.711 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 13.608920263 2024-03-06T07:24:09.711 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19006b500 msgr2=0x7fd19006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19006b500 0x7fd19006d9c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0006fd0 tx=0x7fd1a0008040 comp rx=0 tx=0).stop 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 msgr2=0x7fd1ac0c9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.701+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac0c9080 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fd19800e9b0 tx=0x7fd19800ee70 comp rx=0 tx=0).stop 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 shutdown_connections 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19006b500 0x7fd19006d9c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1ac162970 0x7fd1ac1561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1ac161cf0 0x7fd1ac0c9080 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 --2- 172.21.15.72:0/3591145115 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1ac0b4460 0x7fd1ac1f7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 >> 172.21.15.72:0/3591145115 conn(0x7fd1ac0d3ce0 msgr2=0x7fd1ac06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 shutdown_connections 2024-03-06T07:24:09.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.702+0000 7fd1b2e2a640 1 -- 172.21.15.72:0/3591145115 wait complete. 2024-03-06T07:24:09.714 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:09.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.812+0000 7f82a437a640 1 Processor -- start 2024-03-06T07:24:09.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.812+0000 7f82a437a640 1 -- start start 2024-03-06T07:24:09.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.813+0000 7f82a437a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c15bdd0 0x7f829c0c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.813+0000 7f82a437a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0cebe0 con 0x7f829c0d16e0 2024-03-06T07:24:09.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.813+0000 7f82a437a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0ceec0 con 0x7f829c15fa10 2024-03-06T07:24:09.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.813+0000 7f82a437a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0ce980 con 0x7f829c15bdd0 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.813+0000 7f82a2917640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f829c0d16e0 0x7f829c0d1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46440/0 (socket says 172.21.15.72:46440) 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.813+0000 7f82a2917640 1 -- 172.21.15.72:0/2296372557 learned_addr learned my addr 172.21.15.72:0/2296372557 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a2116640 1 --2- 172.21.15.72:0/2296372557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c15bdd0 0x7f829c0c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123796152 0 0) 0x7f829c0cebe0 con 0x7f829c0d16e0 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8288003520 con 0x7f829c0d16e0 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557298711 0 0) 0x7f829c0ceec0 con 0x7f829c15fa10 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0cebe0 con 0x7f829c15fa10 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947530122 0 0) 0x7f8288003520 con 0x7f829c0d16e0 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a2116640 1 -- 172.21.15.72:0/2296372557 >> v1:172.21.15.146:6789/0 conn(0x7f829c15fa10 legacy=0x7f829c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a2116640 1 -- 172.21.15.72:0/2296372557 >> v1:172.21.15.72:6789/0 conn(0x7f829c0d16e0 legacy=0x7f829c0d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.814+0000 7f82a2116640 1 -- 172.21.15.72:0/2296372557 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0cf1a0 con 0x7f829c15bdd0 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.815+0000 7f82a2116640 1 --2- 172.21.15.72:0/2296372557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c15bdd0 0x7f829c0c5610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f828c0097d0 tx=0x7f828c02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=260613d519e864df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.815+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828c037070 con 0x7f829c15bdd0 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.815+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828c030510 con 0x7f829c15bdd0 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.815+0000 7f82a1114640 1 -- 172.21.15.72:0/2296372557 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82940019f0 con 0x7f829c15bdd0 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 -- 172.21.15.72:0/2296372557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c15bdd0 msgr2=0x7f829c0c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 --2- 172.21.15.72:0/2296372557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c15bdd0 0x7f829c0c5610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f828c0097d0 tx=0x7f828c02b770 comp rx=0 tx=0).stop 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 -- 172.21.15.72:0/2296372557 shutdown_connections 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 --2- 172.21.15.72:0/2296372557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c15bdd0 0x7f829c0c5610 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 -- 172.21.15.72:0/2296372557 >> 172.21.15.72:0/2296372557 conn(0x7f829c09a9f0 msgr2=0x7f829c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 -- 172.21.15.72:0/2296372557 shutdown_connections 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.816+0000 7f82a437a640 1 -- 172.21.15.72:0/2296372557 wait complete. 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.817+0000 7f82a437a640 1 Processor -- start 2024-03-06T07:24:09.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.817+0000 7f82a437a640 1 -- start start 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c0d16e0 0x7f829c15c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15bdd0 0x7f829c06f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 0x7f829c0b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0cc5d0 con 0x7f829c15fa10 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c0c5b50 con 0x7f829c15bdd0 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c15c1b0 con 0x7f829c0d16e0 2024-03-06T07:24:09.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 0x7f829c0b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 0x7f829c0b6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34106/0 (socket says 172.21.15.72:34106) 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 -- 172.21.15.72:0/1025933409 learned_addr learned my addr 172.21.15.72:0/1025933409 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2116640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c0d16e0 0x7f829c15c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a1915640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15bdd0 0x7f829c06f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 -- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c0d16e0 msgr2=0x7f829c15c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c0d16e0 0x7f829c15c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 -- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15bdd0 msgr2=0x7f829c06f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15bdd0 0x7f829c06f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 -- 172.21.15.72:0/1025933409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c1e5a30 con 0x7f829c15fa10 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a1915640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15bdd0 0x7f829c06f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2116640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c0d16e0 0x7f829c15c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a2917640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 0x7f829c0b6b90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f829400cc50 tx=0x7f829400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8294017070 con 0x7f829c15fa10 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8294012400 con 0x7f829c15fa10 2024-03-06T07:24:09.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f829400ce60 con 0x7f829c15fa10 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c1e5d10 con 0x7f829c15fa10 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.818+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c1e62d0 con 0x7f829c15fa10 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.819+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c04f330 con 0x7f829c15fa10 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.820+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f829401a070 con 0x7f829c15fa10 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.820+0000 7f8292ffd640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f828006b4d0 0x7f828006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.820+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f829408a110 con 0x7f829c15fa10 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.820+0000 7f82a2116640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f828006b4d0 0x7f828006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:09.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.821+0000 7f82a2116640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f828006b4d0 0x7f828006d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f828c034000 tx=0x7f828c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:09.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:09.825+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8294059020 con 0x7f829c15fa10 2024-03-06T07:24:10.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.040+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f829c0b7a40 con 0x7f829c15fa10 2024-03-06T07:24:10.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.040+0000 7f8292ffd640 1 -- 172.21.15.72:0/1025933409 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8294058920 con 0x7f829c15fa10 2024-03-06T07:24:10.043 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f828006b4d0 msgr2=0x7f828006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f828006b4d0 0x7f828006d990 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f828c034000 tx=0x7f828c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 msgr2=0x7f829c0b6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 0x7f829c0b6b90 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f829400cc50 tx=0x7f829400a430 comp rx=0 tx=0).stop 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 shutdown_connections 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f828006b4d0 0x7f828006d990 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f829c15fa10 0x7f829c0b6b90 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f829c15bdd0 0x7f829c06f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 --2- 172.21.15.72:0/1025933409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f829c0d16e0 0x7f829c15c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:10.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 >> 172.21.15.72:0/1025933409 conn(0x7f829c09a9f0 msgr2=0x7f829c0c7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:10.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 shutdown_connections 2024-03-06T07:24:10.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10.042+0000 7f82a437a640 1 -- 172.21.15.72:0/1025933409 wait complete. 2024-03-06T07:24:10.058 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:10.058 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:10.059 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:10.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.059 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:24:10.075 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:10.075 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:10.075 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:10.076 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42545sec preferred_lft 42545sec 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.077 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:10.078 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:10.078 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:24:10.078 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:10.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.078 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:10.141 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:24:10.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.143 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:24:10.198 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:24:10.198 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:10.198 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:10.198 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:10.198 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:10.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.199 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:24:10.199 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:24:10.199 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:10.199 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:10.199 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:10.199 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:24:10.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.292 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:24:10.307 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:10.307 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:10.307 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:10.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:24:10.374 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:10.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:10.374 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:10.429 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:10.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:10.430 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:10.486 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:24:10.557 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:10.611 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:10.612 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:10.613 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:24:10.614 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:10.614 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:10.614 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:10.614 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:10.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.614 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:24:10.665 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:10.666 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:10.708 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:24:10.735 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:10.790 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:10.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.793 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:24:10.797 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:24:10.795+0000 7f527d443480 -1 init, newargv = 0x564bf27ed550 newargc=15 2024-03-06T07:24:10.797 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[58343]: starting ceph client 2024-03-06T07:24:10.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[58343]: starting fuse 2024-03-06T07:24:10.811 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:24:10.812 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:10.812 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:10.813 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:10.814 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:10.814 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:24:10.932 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.58343.asok 2024-03-06T07:24:10.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:10.943 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58343.asok 2024-03-06T07:24:10.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:10.943 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58343.asok status 2024-03-06T07:24:11.079 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "58343", 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:11.080 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6264, 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 6264 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3697323750 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3697323750 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.6264 192.168.0.1:0/3697323750", 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/3697323750", 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 27, 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 59, 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:24:11.082 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:24:11.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:11.088 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:24:11.088 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:11.103 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:24:11.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:11.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:11.103 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:11.171 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-03-06T07:24:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.312+0000 7f799f1f7640 1 Processor -- start 2024-03-06T07:24:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 -- start start 2024-03-06T07:24:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79981614d0 0x7f79980d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79980d1910 0x7f79980d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79980d22b0 0x7f79980d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79980d2740 con 0x7f79980d1910 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79981618b0 con 0x7f79980d22b0 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f799f1f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79980cf400 con 0x7f79981614d0 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f7997fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79980d1910 0x7f79980d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f7997fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f79980d1910 0x7f79980d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34110/0 (socket says 172.21.15.72:34110) 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.313+0000 7f7997fff640 1 -- 172.21.15.72:0/2375626615 learned_addr learned my addr 172.21.15.72:0/2375626615 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f799cf93640 1 --2- 172.21.15.72:0/2375626615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79981614d0 0x7f79980d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f799d794640 1 --2- 172.21.15.72:0/2375626615 >> v2:172.21.15.146:3300/0 conn(0x7f79980d22b0 0x7f79980d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f7997fff640 1 -- 172.21.15.72:0/2375626615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79981614d0 msgr2=0x7f79980d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f7997fff640 1 --2- 172.21.15.72:0/2375626615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79981614d0 0x7f79980d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f7997fff640 1 -- 172.21.15.72:0/2375626615 >> v2:172.21.15.146:3300/0 conn(0x7f79980d22b0 msgr2=0x7f79980d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f7997fff640 1 --2- 172.21.15.72:0/2375626615 >> v2:172.21.15.146:3300/0 conn(0x7f79980d22b0 0x7f79980d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f7997fff640 1 -- 172.21.15.72:0/2375626615 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980cf6a0 con 0x7f79980d1910 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.314+0000 7f7997fff640 1 --2- 172.21.15.72:0/2375626615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79980d1910 0x7f79980d1d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f79980b57d0 tx=0x7f798c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3868a2efd9a0a689 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f79977fe640 1 -- 172.21.15.72:0/2375626615 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c035020 con 0x7f79980d1910 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f79977fe640 1 -- 172.21.15.72:0/2375626615 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f798c031870 con 0x7f79980d1910 2024-03-06T07:24:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f79977fe640 1 -- 172.21.15.72:0/2375626615 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798c031c40 con 0x7f79980d1910 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 -- 172.21.15.72:0/2375626615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79980d1910 msgr2=0x7f79980d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 --2- 172.21.15.72:0/2375626615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79980d1910 0x7f79980d1d70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f79980b57d0 tx=0x7f798c02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 -- 172.21.15.72:0/2375626615 shutdown_connections 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 --2- 172.21.15.72:0/2375626615 >> v2:172.21.15.146:3300/0 conn(0x7f79980d22b0 0x7f79980d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 --2- 172.21.15.72:0/2375626615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79980d1910 0x7f79980d1d70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 --2- 172.21.15.72:0/2375626615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79981614d0 0x7f79980d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 -- 172.21.15.72:0/2375626615 >> 172.21.15.72:0/2375626615 conn(0x7f79980d3d40 msgr2=0x7f79980d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 -- 172.21.15.72:0/2375626615 shutdown_connections 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.315+0000 7f799f1f7640 1 -- 172.21.15.72:0/2375626615 wait complete. 2024-03-06T07:24:11.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.316+0000 7f799f1f7640 1 Processor -- start 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.316+0000 7f799f1f7640 1 -- start start 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799f1f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980d1910 0x7f79980b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799f1f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980d22b0 0x7f79980b9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799f1f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 0x7f79981f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799f1f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79980b0910 con 0x7f79981614d0 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799f1f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79980cfc60 con 0x7f79980d1910 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799f1f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79980d0be0 con 0x7f79980d22b0 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 0x7f79981f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 0x7f79981f6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34114/0 (socket says 172.21.15.72:34114) 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 -- 172.21.15.72:0/3286750814 learned_addr learned my addr 172.21.15.72:0/3286750814 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f7997fff640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980d22b0 0x7f79980b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799cf93640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980d1910 0x7f79980b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 -- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980d22b0 msgr2=0x7f79980b9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980d22b0 0x7f79980b9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 -- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980d1910 msgr2=0x7f79980b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980d1910 0x7f79980b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.317+0000 7f799d794640 1 -- 172.21.15.72:0/3286750814 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799815d120 con 0x7f79981614d0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f799cf93640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980d1910 0x7f79980b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f799d794640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 0x7f79981f6e90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7990007e40 tx=0x7f799000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990014620 con 0x7f79981614d0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7990019070 con 0x7f79981614d0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799815d400 con 0x7f79981614d0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7990014a20 con 0x7f79981614d0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.318+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799815d9c0 con 0x7f79981614d0 2024-03-06T07:24:11.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.319+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7998068ff0 con 0x7f79981614d0 2024-03-06T07:24:11.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.319+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7990014bc0 con 0x7f79981614d0 2024-03-06T07:24:11.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.320+0000 7f79957fa640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f798406b4d0 0x7f798406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.320+0000 7f799cf93640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f798406b4d0 0x7f798406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.320+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f799008aa40 con 0x7f79981614d0 2024-03-06T07:24:11.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.320+0000 7f799cf93640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f798406b4d0 0x7f798406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7988006fd0 tx=0x7f7988008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.324+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7990059950 con 0x7f79981614d0 2024-03-06T07:24:11.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.504+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f79980a1c10 con 0x7f79981614d0 2024-03-06T07:24:11.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.507+0000 7f79957fa640 1 -- 172.21.15.72:0/3286750814 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v17)=0 v17) v1 ==== 124+0+0 (secure 0 0 0) 0x7f799008ae20 con 0x7f79981614d0 2024-03-06T07:24:11.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f798406b4d0 msgr2=0x7f798406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f798406b4d0 0x7f798406d990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7988006fd0 tx=0x7f7988008040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 msgr2=0x7f79981f6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 0x7f79981f6e90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f7990007e40 tx=0x7f799000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 shutdown_connections 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f798406b4d0 0x7f798406d990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f79981614d0 0x7f79981f6e90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980d22b0 0x7f79980b9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 --2- 172.21.15.72:0/3286750814 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980d1910 0x7f79980b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 >> 172.21.15.72:0/3286750814 conn(0x7f79980d3d40 msgr2=0x7f79980b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 shutdown_connections 2024-03-06T07:24:11.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.509+0000 7f799f1f7640 1 -- 172.21.15.72:0/3286750814 wait complete. 2024-03-06T07:24:11.522 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-03-06T07:24:11.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.633+0000 7f3b928b5640 1 Processor -- start 2024-03-06T07:24:11.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.633+0000 7f3b928b5640 1 -- start start 2024-03-06T07:24:11.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b928b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b8c0b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b928b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c158ac0 con 0x7f3b8c1586e0 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b928b5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8c15b2a0 con 0x7f3b8c0cfa90 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b928b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c15b540 con 0x7f3b8c0cff40 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b8bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3b8c1586e0 0x7f3b8c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46450/0 (socket says 172.21.15.72:46450) 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b8bfff640 1 -- 172.21.15.72:0/4218718717 learned_addr learned my addr 172.21.15.72:0/4218718717 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b90e52640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b8c0b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317829738 0 0) 0x7f3b8c158ac0 con 0x7f3b8c1586e0 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.634+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b74005730 con 0x7f3b8c1586e0 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546401215 0 0) 0x7f3b8c15b2a0 con 0x7f3b8c0cfa90 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b8c158ac0 con 0x7f3b8c0cfa90 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2832806545 0 0) 0x7f3b74005730 con 0x7f3b8c1586e0 2024-03-06T07:24:11.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 msgr2=0x7f3b8c0b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b8c0b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 >> v1:172.21.15.146:6789/0 conn(0x7f3b8c0cfa90 legacy=0x7f3b8c0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c15b820 con 0x7f3b8c1586e0 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3b80001670 con 0x7f3b8c1586e0 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b8c0b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 >> v1:172.21.15.72:6789/0 conn(0x7f3b8c1586e0 legacy=0x7f3b8c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b74005d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b74006310 0x7f3b74006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b74006f50 0x7f3b740073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c15b540 con 0x7f3b74006f50 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7400dce0 con 0x7f3b74006310 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7400dee0 con 0x7f3b8c0cff40 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b74006f50 0x7f3b740073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8bfff640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b74005d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8b7fe640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b74006310 0x7f3b74006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 -- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 msgr2=0x7f3b74005d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b74005d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 -- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b74006310 msgr2=0x7f3b74006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b74006310 0x7f3b74006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b90e52640 1 -- 172.21.15.72:0/4218718717 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74005730 con 0x7f3b74006f50 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8b7fe640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b74006310 0x7f3b74006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.635+0000 7f3b8bfff640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c0cff40 0x7f3b74005d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b90e52640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b74006f50 0x7f3b740073d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400e320 tx=0x7f3b74007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e342685cadfd12 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84009dc0 con 0x7f3b74006f50 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b8402cd60 con 0x7f3b74006f50 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b8affd640 1 -- 172.21.15.72:0/4218718717 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84003b90 con 0x7f3b74006f50 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b928b5640 1 -- 172.21.15.72:0/4218718717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b74006f50 msgr2=0x7f3b740073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b928b5640 1 --2- 172.21.15.72:0/4218718717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b74006f50 0x7f3b740073d0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400e320 tx=0x7f3b74007a40 comp rx=0 tx=0).stop 2024-03-06T07:24:11.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b90e52640 1 -- 172.21.15.72:0/4218718717 reap_dead start 2024-03-06T07:24:11.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b928b5640 1 -- 172.21.15.72:0/4218718717 shutdown_connections 2024-03-06T07:24:11.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.636+0000 7f3b928b5640 1 -- 172.21.15.72:0/4218718717 >> 172.21.15.72:0/4218718717 conn(0x7f3b8c06b190 msgr2=0x7f3b8c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.637+0000 7f3b928b5640 1 -- 172.21.15.72:0/4218718717 shutdown_connections 2024-03-06T07:24:11.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.637+0000 7f3b928b5640 1 -- 172.21.15.72:0/4218718717 wait complete. 2024-03-06T07:24:11.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.638+0000 7f3b928b5640 1 Processor -- start 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.639+0000 7f3b928b5640 1 -- start start 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b928b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 0x7f3b8c146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b928b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b8c0abce0 0x7f3b8c0ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b928b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c148590 0x7f3b8c148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b928b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0d0360 con 0x7f3b8c145ec0 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b928b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c15bde0 con 0x7f3b8c0abce0 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b928b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8c0af150 con 0x7f3b8c148590 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 0x7f3b8c146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 0x7f3b8c146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34138/0 (socket says 172.21.15.72:34138) 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 -- 172.21.15.72:0/2881361145 learned_addr learned my addr 172.21.15.72:0/2881361145 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8b7fe640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b8c0abce0 0x7f3b8c0ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b90e52640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c148590 0x7f3b8c148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 -- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c148590 msgr2=0x7f3b8c148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c148590 0x7f3b8c148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 -- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b8c0abce0 msgr2=0x7f3b8c0ac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b8c0abce0 0x7f3b8c0ac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 -- 172.21.15.72:0/2881361145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c149440 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8b7fe640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b8c0abce0 0x7f3b8c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.640+0000 7f3b8bfff640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 0x7f3b8c146680 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000e7b0 tx=0x7f3b8000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.641+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8000c870 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.641+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80018070 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.641+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8c1e20e0 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.641+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8000cc10 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.641+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b8c1e26a0 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.642+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8c0732b0 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.642+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3b8001d460 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.643+0000 7f3b88ff9640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b7806b500 0x7f3b7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.643+0000 7f3b8b7fe640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b7806b500 0x7f3b7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.643+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b8008a0b0 con 0x7f3b8c145ec0 2024-03-06T07:24:11.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.643+0000 7f3b8b7fe640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b7806b500 0x7f3b7806d9c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0abc80 tx=0x7f3b7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:11.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.647+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3b80058fc0 con 0x7f3b8c145ec0 2024-03-06T07:24:11.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.884+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f3b8c1e2870 con 0x7f3b8c145ec0 2024-03-06T07:24:11.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.887+0000 7f3b88ff9640 1 -- 172.21.15.72:0/2881361145 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v18)=0 v18) v1 ==== 116+0+0 (secure 0 0 0) 0x7f3b800588c0 con 0x7f3b8c145ec0 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b7806b500 msgr2=0x7f3b7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b7806b500 0x7f3b7806d9c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0abc80 tx=0x7f3b7c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 msgr2=0x7f3b8c146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 0x7f3b8c146680 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000e7b0 tx=0x7f3b8000ec70 comp rx=0 tx=0).stop 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 shutdown_connections 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b7806b500 0x7f3b7806d9c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b8c148590 0x7f3b8c148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b8c0abce0 0x7f3b8c0ac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 --2- 172.21.15.72:0/2881361145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b8c145ec0 0x7f3b8c146680 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:11.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 >> 172.21.15.72:0/2881361145 conn(0x7f3b8c06b190 msgr2=0x7f3b8c06db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:11.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 shutdown_connections 2024-03-06T07:24:11.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:11.890+0000 7f3b928b5640 1 -- 172.21.15.72:0/2881361145 wait complete. 2024-03-06T07:24:19.916 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:19.919 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:21.903 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:21.903 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> import uuid 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> n = 50 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = False 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> if 0 > 0: 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = True 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:21.920 DEBUG:teuthology.orchestra.run.smithi072:> dpath = os.path.dirname(path) 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> try: 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(n): 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> f.write(f"{i}") 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> f.flush() 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(f.fileno()) 2024-03-06T07:24:21.921 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> os.unlink(fpath) 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> if create_hard_links: 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> for j in range(0): 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> finally: 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> os.close(dirfd) 2024-03-06T07:24:21.922 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:24:22.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:22.140 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:22.140 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:22.196 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.340+0000 7f731828f640 1 Processor -- start 2024-03-06T07:24:22.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.340+0000 7f731828f640 1 -- start start 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731828f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 0x7f731015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731828f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f73100cd7a0 0x7f73100cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731828f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7310071f40 con 0x7f73101515e0 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731828f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100cd2c0 con 0x7f73100cd7a0 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731828f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7310070a70 con 0x7f73100cce40 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731602b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f73101515e0 0x7f7310071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44156/0 (socket says 172.21.15.72:44156) 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731602b640 1 -- 172.21.15.72:0/1456636725 learned_addr learned my addr 172.21.15.72:0/1456636725 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731682c640 1 --2- 172.21.15.72:0/1456636725 >> v2:172.21.15.146:3300/0 conn(0x7f73100cd7a0 0x7f73100cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f731582a640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 0x7f731015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412056618 0 0) 0x7f7310071f40 con 0x7f73101515e0 2024-03-06T07:24:22.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72fc003520 con 0x7f73101515e0 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706941185 0 0) 0x7f72fc003520 con 0x7f73101515e0 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 msgr2=0x7f731015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 0x7f731015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 >> v2:172.21.15.146:3300/0 conn(0x7f73100cd7a0 msgr2=0x7f73100cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 --2- 172.21.15.72:0/1456636725 >> v2:172.21.15.146:3300/0 conn(0x7f73100cd7a0 0x7f73100cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7310070d50 con 0x7f73101515e0 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f730c001670 con 0x7f73101515e0 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.341+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 >> v1:172.21.15.72:6789/0 conn(0x7f73101515e0 legacy=0x7f7310071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f7315029640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 0x7f72fc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f7315029640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72fc003f30 0x7f72fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f7315029640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc004d40 0x7f72fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100cd2c0 con 0x7f72fc003f30 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7310070a70 con 0x7f72fc004d40 2024-03-06T07:24:22.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc00b9c0 con 0x7f73100cce40 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731582a640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72fc003f30 0x7f72fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731602b640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 0x7f72fc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731682c640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc004d40 0x7f72fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731582a640 1 -- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 msgr2=0x7f72fc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731582a640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73100cce40 0x7f72fc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731582a640 1 -- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc004d40 msgr2=0x7f72fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731582a640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72fc004d40 0x7f72fc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.342+0000 7f731582a640 1 -- 172.21.15.72:0/1456636725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc003520 con 0x7f72fc003f30 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731582a640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72fc003f30 0x7f72fc004800 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7300009ac0 tx=0x7f72fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c8042007ef4f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300039070 con 0x7f72fc003f30 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730002c6b0 con 0x7f72fc003f30 2024-03-06T07:24:22.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f7315029640 1 -- 172.21.15.72:0/1456636725 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300009690 con 0x7f72fc003f30 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731828f640 1 -- 172.21.15.72:0/1456636725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72fc003f30 msgr2=0x7f72fc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731828f640 1 --2- 172.21.15.72:0/1456636725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72fc003f30 0x7f72fc004800 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f7300009ac0 tx=0x7f72fc005830 comp rx=0 tx=0).stop 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731682c640 1 -- 172.21.15.72:0/1456636725 reap_dead start 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731828f640 1 -- 172.21.15.72:0/1456636725 shutdown_connections 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731828f640 1 -- 172.21.15.72:0/1456636725 >> 172.21.15.72:0/1456636725 conn(0x7f73100cf8a0 msgr2=0x7f73100cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731828f640 1 -- 172.21.15.72:0/1456636725 shutdown_connections 2024-03-06T07:24:22.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.343+0000 7f731828f640 1 -- 172.21.15.72:0/1456636725 wait complete. 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 Processor -- start 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 -- start start 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72fc003f30 0x7f73101f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731014f930 0x7f731014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 0x7f73101450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100ce110 con 0x7f7310144c30 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7310071310 con 0x7f731014f930 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731828f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731015b430 con 0x7f72fc003f30 2024-03-06T07:24:22.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731682c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 0x7f73101450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731682c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 0x7f73101450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56788/0 (socket says 172.21.15.72:56788) 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.345+0000 7f731682c640 1 -- 172.21.15.72:0/1295114034 learned_addr learned my addr 172.21.15.72:0/1295114034 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731582a640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731014f930 0x7f731014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731602b640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72fc003f30 0x7f73101f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731682c640 1 -- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72fc003f30 msgr2=0x7f73101f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731682c640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72fc003f30 0x7f73101f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731682c640 1 -- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731014f930 msgr2=0x7f731014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731682c640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731014f930 0x7f731014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731682c640 1 -- 172.21.15.72:0/1295114034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73101459a0 con 0x7f7310144c30 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731602b640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72fc003f30 0x7f73101f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731582a640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731014f930 0x7f731014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731682c640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 0x7f73101450b0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f73100731f0 tx=0x7f730800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730800fde0 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7308015430 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7310145c80 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308015820 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.346+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7310146240 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.347+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731004f330 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.348+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f73080159c0 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.348+0000 7f7306ffd640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72f406b470 0x7f72f406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.348+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f730808b140 con 0x7f7310144c30 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.348+0000 7f731602b640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72f406b470 0x7f72f406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.349+0000 7f731602b640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72f406b470 0x7f72f406d930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f731015b170 tx=0x7f730c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.353+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7308019030 con 0x7f7310144c30 2024-03-06T07:24:22.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.529+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73100b4c30 con 0x7f7310144c30 2024-03-06T07:24:22.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.530+0000 7f7306ffd640 1 -- 172.21.15.72:0/1295114034 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f7308010040 con 0x7f7310144c30 2024-03-06T07:24:22.533 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:22.533 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:22.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72f406b470 msgr2=0x7f72f406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72f406b470 0x7f72f406d930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f731015b170 tx=0x7f730c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 msgr2=0x7f73101450b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 0x7f73101450b0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f73100731f0 tx=0x7f730800f580 comp rx=0 tx=0).stop 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 shutdown_connections 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72f406b470 0x7f72f406d930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7310144c30 0x7f73101450b0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731014f930 0x7f731014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 --2- 172.21.15.72:0/1295114034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72fc003f30 0x7f73101f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 >> 172.21.15.72:0/1295114034 conn(0x7f73100cf8a0 msgr2=0x7f73100716f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 shutdown_connections 2024-03-06T07:24:22.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.532+0000 7f731828f640 1 -- 172.21.15.72:0/1295114034 wait complete. 2024-03-06T07:24:22.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:57.998737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:58.744228+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:22.546 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:22.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.648+0000 7fde1f948640 1 Processor -- start 2024-03-06T07:24:22.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.649+0000 7fde1f948640 1 -- start start 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.649+0000 7fde1f948640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 0x7fde18168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.649+0000 7fde1f948640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde180d2730 con 0x7fde181614d0 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.649+0000 7fde1f948640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde181618b0 con 0x7fde180d4860 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.649+0000 7fde1f948640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde180d4cf0 con 0x7fde180b1e40 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde1d6e4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fde181614d0 0x7fde180d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53620/0 (socket says 172.21.15.72:53620) 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde1d6e4640 1 -- 172.21.15.72:0/1323828367 learned_addr learned my addr 172.21.15.72:0/1323828367 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:22.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde1dee5640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 0x7fde18168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174772937 0 0) 0x7fde180d2730 con 0x7fde181614d0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde04003520 con 0x7fde181614d0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989036653 0 0) 0x7fde181618b0 con 0x7fde180d4860 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde180d2730 con 0x7fde180d4860 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108180721 0 0) 0x7fde04003520 con 0x7fde181614d0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 msgr2=0x7fde18168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 0x7fde18168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 >> v1:172.21.15.146:6789/0 conn(0x7fde180d4860 legacy=0x7fde180d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde180cf400 con 0x7fde181614d0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fde14001670 con 0x7fde181614d0 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 >> v1:172.21.15.72:6789/0 conn(0x7fde181614d0 legacy=0x7fde180d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04003b70 0x7fde04003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 0x7fde040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180d4cf0 con 0x7fde04004800 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0400ba70 con 0x7fde04003b70 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0400bc70 con 0x7fde180b1e40 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde1cee3640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde1d6e4640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04003b70 0x7fde04003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.650+0000 7fde1dee5640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 0x7fde040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde1cee3640 1 -- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 msgr2=0x7fde040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde1cee3640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b1e40 0x7fde040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde1cee3640 1 -- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04003b70 msgr2=0x7fde04003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde1cee3640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde04003b70 0x7fde04003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde1cee3640 1 -- 172.21.15.72:0/1323828367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde04003520 con 0x7fde04004800 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde1cee3640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde04004c80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fde080018a0 tx=0x7fde04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba5b6746a4611d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08037070 con 0x7fde04004800 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde080052d0 con 0x7fde04004800 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.651+0000 7fde0ffff640 1 -- 172.21.15.72:0/1323828367 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde080014f0 con 0x7fde04004800 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1f948640 1 -- 172.21.15.72:0/1323828367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 msgr2=0x7fde04004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1f948640 1 --2- 172.21.15.72:0/1323828367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde04004c80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fde080018a0 tx=0x7fde04005830 comp rx=0 tx=0).stop 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1dee5640 1 -- 172.21.15.72:0/1323828367 reap_dead start 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1f948640 1 -- 172.21.15.72:0/1323828367 shutdown_connections 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1f948640 1 -- 172.21.15.72:0/1323828367 >> 172.21.15.72:0/1323828367 conn(0x7fde180d3d40 msgr2=0x7fde180d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1f948640 1 -- 172.21.15.72:0/1323828367 shutdown_connections 2024-03-06T07:24:22.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.652+0000 7fde1f948640 1 -- 172.21.15.72:0/1323828367 wait complete. 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.653+0000 7fde1f948640 1 Processor -- start 2024-03-06T07:24:22.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.653+0000 7fde1f948640 1 -- start start 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1f948640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde180b9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1f948640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde1815bcf0 0x7fde180cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1f948640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b3540 0x7fde180b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1f948640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180cfc60 con 0x7fde04004800 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1f948640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180cf980 con 0x7fde1815bcf0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1f948640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde180b0910 con 0x7fde180b3540 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde180b9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde180b9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58618/0 (socket says 172.21.15.72:58618) 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 -- 172.21.15.72:0/2575874824 learned_addr learned my addr 172.21.15.72:0/2575874824 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1dee5640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b3540 0x7fde180b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1cee3640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde1815bcf0 0x7fde180cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 -- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b3540 msgr2=0x7fde180b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b3540 0x7fde180b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 -- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde1815bcf0 msgr2=0x7fde180cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde1815bcf0 0x7fde180cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1d6e4640 1 -- 172.21.15.72:0/2575874824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde180cc4b0 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1dee5640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b3540 0x7fde180b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.654+0000 7fde1cee3640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde1815bcf0 0x7fde180cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.655+0000 7fde1d6e4640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde180b9020 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fde14007340 tx=0x7fde14007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.655+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1400f940 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.655+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde14018070 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.655+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde180cc790 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.655+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1400fd60 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.655+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde180ccd50 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.656+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde18068ff0 con 0x7fde04004800 2024-03-06T07:24:22.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.656+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde1401d460 con 0x7fde04004800 2024-03-06T07:24:22.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.657+0000 7fde0dffb640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddfc06b500 0x7fddfc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.657+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde1408a2f0 con 0x7fde04004800 2024-03-06T07:24:22.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.657+0000 7fde1cee3640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddfc06b500 0x7fddfc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.657+0000 7fde1cee3640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddfc06b500 0x7fddfc06d9c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fde08034000 tx=0x7fde08031000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.661+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde140590e0 con 0x7fde04004800 2024-03-06T07:24:22.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.839+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde180b5590 con 0x7fde04004800 2024-03-06T07:24:22.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.839+0000 7fde0dffb640 1 -- 172.21.15.72:0/2575874824 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde140589e0 con 0x7fde04004800 2024-03-06T07:24:22.841 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddfc06b500 msgr2=0x7fddfc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddfc06b500 0x7fddfc06d9c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fde08034000 tx=0x7fde08031000 comp rx=0 tx=0).stop 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 msgr2=0x7fde180b9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde180b9020 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fde14007340 tx=0x7fde14007800 comp rx=0 tx=0).stop 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 shutdown_connections 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddfc06b500 0x7fddfc06d9c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde180b3540 0x7fde180b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde1815bcf0 0x7fde180cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 --2- 172.21.15.72:0/2575874824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde04004800 0x7fde180b9020 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.841+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 >> 172.21.15.72:0/2575874824 conn(0x7fde180d3d40 msgr2=0x7fde180b5830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.842+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 shutdown_connections 2024-03-06T07:24:22.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.842+0000 7fde1f948640 1 -- 172.21.15.72:0/2575874824 wait complete. 2024-03-06T07:24:22.856 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:22.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.958+0000 7f3d5a90c640 1 Processor -- start 2024-03-06T07:24:22.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.958+0000 7f3d5a90c640 1 -- start start 2024-03-06T07:24:22.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.959+0000 7f3d5a90c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d5415c050 0x7f3d5406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.959+0000 7f3d5a90c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d5406fb40 0x7f3d540cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d5a90c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d540cd4b0 con 0x7f3d540ccca0 2024-03-06T07:24:22.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d5a90c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5406ff80 con 0x7f3d5406fb40 2024-03-06T07:24:22.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d5a90c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540cd790 con 0x7f3d5415c050 2024-03-06T07:24:22.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d537fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d5406fb40 0x7f3d540cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d58ea9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3d540ccca0 0x7f3d540cfae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53624/0 (socket says 172.21.15.72:53624) 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d537fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d5406fb40 0x7f3d540cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58624/0 (socket says 172.21.15.72:58624) 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d58ea9640 1 -- 172.21.15.72:0/3864898589 learned_addr learned my addr 172.21.15.72:0/3864898589 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d53fff640 1 --2- 172.21.15.72:0/3864898589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d5415c050 0x7f3d5406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d537fe640 1 -- 172.21.15.72:0/3864898589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d5415c050 msgr2=0x7f3d5406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d537fe640 1 --2- 172.21.15.72:0/3864898589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d5415c050 0x7f3d5406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d537fe640 1 -- 172.21.15.72:0/3864898589 >> v1:172.21.15.72:6789/0 conn(0x7f3d540ccca0 legacy=0x7f3d540cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d537fe640 1 -- 172.21.15.72:0/3864898589 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d540cda70 con 0x7f3d5406fb40 2024-03-06T07:24:22.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.960+0000 7f3d52ffd640 1 -- 172.21.15.72:0/3864898589 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069456828 0 0) 0x7f3d540cd4b0 con 0x7f3d540ccca0 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.961+0000 7f3d537fe640 1 --2- 172.21.15.72:0/3864898589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5406fb40 0x7f3d540cc760 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5d960880d469ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.961+0000 7f3d52ffd640 1 -- 172.21.15.72:0/3864898589 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480315a0 con 0x7f3d5406fb40 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.961+0000 7f3d52ffd640 1 -- 172.21.15.72:0/3864898589 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d48037070 con 0x7f3d5406fb40 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.961+0000 7f3d52ffd640 1 -- 172.21.15.72:0/3864898589 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48031960 con 0x7f3d5406fb40 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 -- 172.21.15.72:0/3864898589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5406fb40 msgr2=0x7f3d540cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/3864898589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5406fb40 0x7f3d540cc760 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 -- 172.21.15.72:0/3864898589 shutdown_connections 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/3864898589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5406fb40 0x7f3d540cc760 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/3864898589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d5415c050 0x7f3d5406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 -- 172.21.15.72:0/3864898589 >> 172.21.15.72:0/3864898589 conn(0x7f3d540b3800 msgr2=0x7f3d540b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 -- 172.21.15.72:0/3864898589 shutdown_connections 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.962+0000 7f3d5a90c640 1 -- 172.21.15.72:0/3864898589 wait complete. 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.963+0000 7f3d5a90c640 1 Processor -- start 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.963+0000 7f3d5a90c640 1 -- start start 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d5a90c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5406fb40 0x7f3d5414a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d5a90c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d540ccca0 0x7f3d541557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d5a90c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 0x7f3d540cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d5a90c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540d01f0 con 0x7f3d5415c050 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d5a90c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540cd130 con 0x7f3d5406fb40 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d5a90c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d540d0370 con 0x7f3d540ccca0 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 0x7f3d540cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 0x7f3d540cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58640/0 (socket says 172.21.15.72:58640) 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 -- 172.21.15.72:0/1127880837 learned_addr learned my addr 172.21.15.72:0/1127880837 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d53fff640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5406fb40 0x7f3d5414a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 -- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d540ccca0 msgr2=0x7f3d541557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d537fe640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d540ccca0 0x7f3d541557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d540ccca0 0x7f3d541557f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 -- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5406fb40 msgr2=0x7f3d5414a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5406fb40 0x7f3d5414a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 -- 172.21.15.72:0/1127880837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d540b6450 con 0x7f3d5415c050 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d53fff640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5406fb40 0x7f3d5414a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d537fe640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d540ccca0 0x7f3d541557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.964+0000 7f3d58ea9640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 0x7f3d540cae80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c7a0 tx=0x7f3d4c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.965+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c018070 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.965+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c00a640 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.965+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d540b6730 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.965+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c00aab0 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.965+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d540b6cf0 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.966+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5409fe60 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.967+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d4c01b070 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.967+0000 7f3d50ff9640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d4006b4d0 0x7f3d4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.967+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d4c0898d0 con 0x7f3d5415c050 2024-03-06T07:24:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.967+0000 7f3d53fff640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d4006b4d0 0x7f3d4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:22.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.968+0000 7f3d53fff640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d4006b4d0 0x7f3d4006d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d44006fd0 tx=0x7f3d44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:22.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:22.971+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d4c0586c0 con 0x7f3d5415c050 2024-03-06T07:24:23.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.148+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d54068ff0 con 0x7f3d5415c050 2024-03-06T07:24:23.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.149+0000 7f3d50ff9640 1 -- 172.21.15.72:0/1127880837 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3d4c057fc0 con 0x7f3d5415c050 2024-03-06T07:24:23.151 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d4006b4d0 msgr2=0x7f3d4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d4006b4d0 0x7f3d4006d990 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3d44006fd0 tx=0x7f3d44008040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 msgr2=0x7f3d540cae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 0x7f3d540cae80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c7a0 tx=0x7f3d4c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 shutdown_connections 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d4006b4d0 0x7f3d4006d990 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d5415c050 0x7f3d540cae80 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d540ccca0 0x7f3d541557f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 --2- 172.21.15.72:0/1127880837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5406fb40 0x7f3d5414a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 >> 172.21.15.72:0/1127880837 conn(0x7f3d540b3800 msgr2=0x7f3d540b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 shutdown_connections 2024-03-06T07:24:23.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.151+0000 7f3d5a90c640 1 -- 172.21.15.72:0/1127880837 wait complete. 2024-03-06T07:24:23.163 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:23.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.267+0000 7f3bb200e640 1 Processor -- start 2024-03-06T07:24:23.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.267+0000 7f3bb200e640 1 -- start start 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3bb200e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 0x7f3bac0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3bb200e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac1618b0 con 0x7f3bac0d1910 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3bb200e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bac0cf400 con 0x7f3bac0d2480 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3bb200e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0cf6a0 con 0x7f3bac1614d0 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baaffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3bac0d1910 0x7f3bac0d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53634/0 (socket says 172.21.15.72:53634) 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baaffd640 1 -- 172.21.15.72:0/3167439314 learned_addr learned my addr 172.21.15.72:0/3167439314 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 0x7f3bac0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677687504 0 0) 0x7f3bac1618b0 con 0x7f3bac0d1910 2024-03-06T07:24:23.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b94003520 con 0x7f3bac0d1910 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279095473 0 0) 0x7f3b94003520 con 0x7f3bac0d1910 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 msgr2=0x7f3bac0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 0x7f3bac0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 >> v1:172.21.15.146:6789/0 conn(0x7f3bac0d2480 legacy=0x7f3bac0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac0cf980 con 0x7f3bac0d1910 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.268+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3b9c001670 con 0x7f3bac0d1910 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 >> v1:172.21.15.72:6789/0 conn(0x7f3bac0d1910 legacy=0x7f3bac0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b940039e0 0x7f3b94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 0x7f3b94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b94004d40 0x7f3b940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0cf6a0 con 0x7f3b940039e0 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9400ba70 con 0x7f3b94004d40 2024-03-06T07:24:23.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9400bc70 con 0x7f3bac1614d0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b940039e0 0x7f3b94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3babfff640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b94004d40 0x7f3b940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3baaffd640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 0x7f3b94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3bab7fe640 1 -- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 msgr2=0x7f3b94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac1614d0 0x7f3b94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3bab7fe640 1 -- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b94004d40 msgr2=0x7f3b940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b94004d40 0x7f3b940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.269+0000 7f3bab7fe640 1 -- 172.21.15.72:0/3167439314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94003520 con 0x7f3b940039e0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b940039e0 0x7f3b94003e00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009a90 tx=0x7f3b94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24ab606e4c14929 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0039070 con 0x7f3b940039e0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba002c680 con 0x7f3b940039e0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3baa7fc640 1 -- 172.21.15.72:0/3167439314 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0009690 con 0x7f3b940039e0 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bb200e640 1 -- 172.21.15.72:0/3167439314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b940039e0 msgr2=0x7f3b94003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bb200e640 1 --2- 172.21.15.72:0/3167439314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b940039e0 0x7f3b94003e00 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009a90 tx=0x7f3b94005830 comp rx=0 tx=0).stop 2024-03-06T07:24:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3babfff640 1 -- 172.21.15.72:0/3167439314 reap_dead start 2024-03-06T07:24:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bb200e640 1 -- 172.21.15.72:0/3167439314 shutdown_connections 2024-03-06T07:24:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bb200e640 1 -- 172.21.15.72:0/3167439314 >> 172.21.15.72:0/3167439314 conn(0x7f3bac0d3d40 msgr2=0x7f3bac0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bb200e640 1 -- 172.21.15.72:0/3167439314 shutdown_connections 2024-03-06T07:24:23.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.270+0000 7f3bb200e640 1 -- 172.21.15.72:0/3167439314 wait complete. 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.272+0000 7f3bb200e640 1 Processor -- start 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.272+0000 7f3bb200e640 1 -- start start 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b940039e0 0x7f3bac1e6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac15d7c0 0x7f3bac0cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 0x7f3bac1f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0d4860 con 0x7f3bac1f7480 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0b0910 con 0x7f3b940039e0 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac0d49e0 con 0x7f3bac15d7c0 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 0x7f3bac1f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 0x7f3bac1f7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58658/0 (socket says 172.21.15.72:58658) 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 -- 172.21.15.72:0/164339355 learned_addr learned my addr 172.21.15.72:0/164339355 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b940039e0 0x7f3bac1e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3baaffd640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac15d7c0 0x7f3bac0cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 -- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac15d7c0 msgr2=0x7f3bac0cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac15d7c0 0x7f3bac0cc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 -- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b940039e0 msgr2=0x7f3bac1e6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b940039e0 0x7f3bac1e6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 -- 172.21.15.72:0/164339355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac15dc40 con 0x7f3bac1f7480 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b940039e0 0x7f3bac1e6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3babfff640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 0x7f3bac1f7900 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3bac08c470 tx=0x7f3ba400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba401a070 con 0x7f3bac1f7480 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4006d60 con 0x7f3bac1f7480 2024-03-06T07:24:23.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac0cd030 con 0x7f3bac1f7480 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4003b90 con 0x7f3bac1f7480 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.274+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac0cd5f0 con 0x7f3bac1f7480 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.276+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bac15de10 con 0x7f3bac1f7480 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.276+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3ba401d070 con 0x7f3bac1f7480 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.276+0000 7f3b93fff640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b9806b500 0x7f3b9806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.276+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b9806b500 0x7f3b9806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.276+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ba4089830 con 0x7f3bac1f7480 2024-03-06T07:24:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.277+0000 7f3bab7fe640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b9806b500 0x7f3b9806d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0036000 tx=0x7f3ba0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.280+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3ba40586b0 con 0x7f3bac1f7480 2024-03-06T07:24:23.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.524+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3bac0cd7c0 con 0x7f3bac1f7480 2024-03-06T07:24:23.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.524+0000 7f3b93fff640 1 -- 172.21.15.72:0/164339355 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3bac0cd7c0 con 0x7f3bac1f7480 2024-03-06T07:24:23.526 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b9806b500 msgr2=0x7f3b9806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b9806b500 0x7f3b9806d9c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0036000 tx=0x7f3ba0030000 comp rx=0 tx=0).stop 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 msgr2=0x7f3bac1f7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 0x7f3bac1f7900 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3bac08c470 tx=0x7f3ba400fd40 comp rx=0 tx=0).stop 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 shutdown_connections 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3b9806b500 0x7f3b9806d9c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bac1f7480 0x7f3bac1f7900 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bac15d7c0 0x7f3bac0cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 --2- 172.21.15.72:0/164339355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b940039e0 0x7f3bac1e6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 >> 172.21.15.72:0/164339355 conn(0x7f3bac0d3d40 msgr2=0x7f3bac0c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 shutdown_connections 2024-03-06T07:24:23.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.527+0000 7f3bb200e640 1 -- 172.21.15.72:0/164339355 wait complete. 2024-03-06T07:24:23.543 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:23.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.643+0000 7f9ba4ec0640 1 Processor -- start 2024-03-06T07:24:23.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.643+0000 7f9ba4ec0640 1 -- start start 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9ba4ec0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9ba00d0d90 0x7f9ba00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9ba4ec0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba00c90f0 0x7f9ba00ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9ba4ec0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba00d25e0 con 0x7f9ba015e000 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9ba4ec0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba00c7c50 con 0x7f9ba00d0d90 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9ba4ec0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00d2880 con 0x7f9ba00c90f0 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9ba00d0d90 0x7f9ba00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9ba00d0d90 0x7f9ba00ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58672/0 (socket says 172.21.15.72:58672) 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 -- 172.21.15.72:0/1238480019 learned_addr learned my addr 172.21.15.72:0/1238480019 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 -- 172.21.15.72:0/1238480019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba00c90f0 msgr2=0x7f9ba00ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b9ed9d640 1 --2- 172.21.15.72:0/1238480019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba00c90f0 0x7f9ba00ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 --2- 172.21.15.72:0/1238480019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba00c90f0 0x7f9ba00ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 -- 172.21.15.72:0/1238480019 >> v1:172.21.15.146:6789/0 conn(0x7f9ba015e000 legacy=0x7f9ba00d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:23.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.644+0000 7f9b97fff640 1 -- 172.21.15.72:0/1238480019 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba00d2b40 con 0x7f9ba00d0d90 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.645+0000 7f9b97fff640 1 --2- 172.21.15.72:0/1238480019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba00ce7c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009870 tx=0x7f9b8802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b418043d591f7833 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.645+0000 7f9b9dd9b640 1 -- 172.21.15.72:0/1238480019 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88034070 con 0x7f9ba00d0d90 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.645+0000 7f9b9dd9b640 1 -- 172.21.15.72:0/1238480019 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b88030840 con 0x7f9ba00d0d90 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.645+0000 7f9b9dd9b640 1 -- 172.21.15.72:0/1238480019 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88030c10 con 0x7f9ba00d0d90 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/1238480019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 msgr2=0x7f9ba00ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/1238480019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba00ce7c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009870 tx=0x7f9b8802b980 comp rx=0 tx=0).stop 2024-03-06T07:24:23.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/1238480019 shutdown_connections 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/1238480019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba00c90f0 0x7f9ba00ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/1238480019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba00ce7c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/1238480019 >> 172.21.15.72:0/1238480019 conn(0x7f9ba00d3cf0 msgr2=0x7f9ba00d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/1238480019 shutdown_connections 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.646+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/1238480019 wait complete. 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.648+0000 7f9ba4ec0640 1 Processor -- start 2024-03-06T07:24:23.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.648+0000 7f9ba4ec0640 1 -- start start 2024-03-06T07:24:23.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9ba4ec0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba00c90f0 0x7f9ba015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9ba4ec0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba009fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9ba4ec0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba015e000 0x7f9ba00a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9ba4ec0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba015e3e0 con 0x7f9ba00d0d90 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9ba4ec0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00d22d0 con 0x7f9ba00c90f0 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9ba4ec0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba00d4810 con 0x7f9ba015e000 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9b97fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba009fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9b97fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba009fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58688/0 (socket says 172.21.15.72:58688) 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9b97fff640 1 -- 172.21.15.72:0/2646886664 learned_addr learned my addr 172.21.15.72:0/2646886664 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9b9ed9d640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba015e000 0x7f9ba00a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.649+0000 7f9b9e59c640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba00c90f0 0x7f9ba015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b97fff640 1 -- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba015e000 msgr2=0x7f9ba00a0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b97fff640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba015e000 0x7f9ba00a0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b97fff640 1 -- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba00c90f0 msgr2=0x7f9ba015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b97fff640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba00c90f0 0x7f9ba015cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b97fff640 1 -- 172.21.15.72:0/2646886664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba01f3b80 con 0x7f9ba00d0d90 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b9e59c640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba00c90f0 0x7f9ba015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b97fff640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba009fbe0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009870 tx=0x7f9b88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88048070 con 0x7f9ba00d0d90 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b88030640 con 0x7f9ba00d0d90 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba01e8610 con 0x7f9ba00d0d90 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88034040 con 0x7f9ba00d0d90 2024-03-06T07:24:23.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.650+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba01e8bd0 con 0x7f9ba00d0d90 2024-03-06T07:24:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.651+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba004f330 con 0x7f9ba00d0d90 2024-03-06T07:24:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.651+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b8804b070 con 0x7f9ba00d0d90 2024-03-06T07:24:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.652+0000 7f9b96ffd640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b8006b500 0x7f9b8006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.652+0000 7f9b9e59c640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b8006b500 0x7f9b8006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.652+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b880ac360 con 0x7f9ba00d0d90 2024-03-06T07:24:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.652+0000 7f9b9e59c640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b8006b500 0x7f9b8006d9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b90006fd0 tx=0x7f9b90008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:23.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.656+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ba01f3b80 con 0x7f9ba00d0d90 2024-03-06T07:24:23.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.869+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9ba00cddf0 con 0x7f9ba00d0d90 2024-03-06T07:24:23.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.869+0000 7f9b96ffd640 1 -- 172.21.15.72:0/2646886664 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9b8807b260 con 0x7f9ba00d0d90 2024-03-06T07:24:23.871 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b8006b500 msgr2=0x7f9b8006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b8006b500 0x7f9b8006d9c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b90006fd0 tx=0x7f9b90008040 comp rx=0 tx=0).stop 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 msgr2=0x7f9ba009fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba009fbe0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009870 tx=0x7f9b88004730 comp rx=0 tx=0).stop 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 shutdown_connections 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b8006b500 0x7f9b8006d9c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ba015e000 0x7f9ba00a0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ba00d0d90 0x7f9ba009fbe0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 --2- 172.21.15.72:0/2646886664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ba00c90f0 0x7f9ba015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:23.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 >> 172.21.15.72:0/2646886664 conn(0x7f9ba00d3cf0 msgr2=0x7f9ba00cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:23.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 shutdown_connections 2024-03-06T07:24:23.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:23.871+0000 7f9ba4ec0640 1 -- 172.21.15.72:0/2646886664 wait complete. 2024-03-06T07:24:23.884 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:23.884 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:23.899 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:24.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.038+0000 7f01de8c9640 1 Processor -- start 2024-03-06T07:24:24.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.038+0000 7f01de8c9640 1 -- start start 2024-03-06T07:24:24.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01de8c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01d80cfff0 0x7f01d80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01de8c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d8161980 con 0x7f01d80cc8b0 2024-03-06T07:24:24.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01de8c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d8161c60 con 0x7f01d815bb20 2024-03-06T07:24:24.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01de8c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01d80d2500 con 0x7f01d80cfff0 2024-03-06T07:24:24.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01dce66640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f01d80cc8b0 0x7f01d80ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53640/0 (socket says 172.21.15.72:53640) 2024-03-06T07:24:24.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01dce66640 1 -- 172.21.15.72:0/2400215417 learned_addr learned my addr 172.21.15.72:0/2400215417 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.039+0000 7f01d77fe640 1 --2- 172.21.15.72:0/2400215417 >> v2:172.21.15.146:3300/0 conn(0x7f01d80cfff0 0x7f01d80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431731279 0 0) 0x7f01d8161980 con 0x7f01d80cc8b0 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c0003520 con 0x7f01d80cc8b0 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161960287 0 0) 0x7f01d8161c60 con 0x7f01d815bb20 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d8161980 con 0x7f01d815bb20 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3339835037 0 0) 0x7f01c0003520 con 0x7f01d80cc8b0 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 >> v2:172.21.15.146:3300/0 conn(0x7f01d80cfff0 msgr2=0x7f01d80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 --2- 172.21.15.72:0/2400215417 >> v2:172.21.15.146:3300/0 conn(0x7f01d80cfff0 0x7f01d80d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 >> v1:172.21.15.146:6789/0 conn(0x7f01d815bb20 legacy=0x7f01d80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8161f40 con 0x7f01d80cc8b0 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f01d0001970 con 0x7f01d80cc8b0 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 >> v1:172.21.15.72:6789/0 conn(0x7f01d80cc8b0 legacy=0x7f01d80ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c0003b90 0x7f01c0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c0004800 0x7f01c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c00051c0 0x7f01c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80d2500 con 0x7f01c0004800 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c000c090 con 0x7f01c00051c0 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c000c290 con 0x7f01c0003b90 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d77fe640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c0004800 0x7f01c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01dce66640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c00051c0 0x7f01c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d7fff640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c0003b90 0x7f01c0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d77fe640 1 -- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c0003b90 msgr2=0x7f01c0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d77fe640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01c0003b90 0x7f01c0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d77fe640 1 -- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c00051c0 msgr2=0x7f01c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d77fe640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c00051c0 0x7f01c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01d77fe640 1 -- 172.21.15.72:0/2400215417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c0003520 con 0x7f01c0004800 2024-03-06T07:24:24.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.040+0000 7f01dce66640 1 -- 172.21.15.72:0/2400215417 reap_dead start 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.041+0000 7f01d77fe640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c0004800 0x7f01c0004c80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009ac0 tx=0x7f01c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11432d615da4560b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.041+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc039070 con 0x7f01c0004800 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.041+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc02c6b0 con 0x7f01c0004800 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.041+0000 7f01d6ffd640 1 -- 172.21.15.72:0/2400215417 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc009690 con 0x7f01c0004800 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 -- 172.21.15.72:0/2400215417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c0004800 msgr2=0x7f01c0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c0004800 0x7f01c0004c80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009ac0 tx=0x7f01c0005830 comp rx=0 tx=0).stop 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 -- 172.21.15.72:0/2400215417 shutdown_connections 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 --2- 172.21.15.72:0/2400215417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c0004800 0x7f01c0004c80 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 -- 172.21.15.72:0/2400215417 >> 172.21.15.72:0/2400215417 conn(0x7f01d809a9f0 msgr2=0x7f01d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 -- 172.21.15.72:0/2400215417 shutdown_connections 2024-03-06T07:24:24.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.042+0000 7f01de8c9640 1 -- 172.21.15.72:0/2400215417 wait complete. 2024-03-06T07:24:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.043+0000 7f01de8c9640 1 Processor -- start 2024-03-06T07:24:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.043+0000 7f01de8c9640 1 -- start start 2024-03-06T07:24:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c0004800 0x7f01d815aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 0x7f01d806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01d81f30a0 0x7f01d806cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80d1980 con 0x7f01d814a980 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d8162500 con 0x7f01c0004800 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80d1b00 con 0x7f01d81f30a0 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 0x7f01d806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 0x7f01d806f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58702/0 (socket says 172.21.15.72:58702) 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 -- 172.21.15.72:0/1970701058 learned_addr learned my addr 172.21.15.72:0/1970701058 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01dce66640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01d81f30a0 0x7f01d806cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d7fff640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c0004800 0x7f01d815aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 -- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01d81f30a0 msgr2=0x7f01d806cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01d81f30a0 0x7f01d806cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 -- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c0004800 msgr2=0x7f01d815aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c0004800 0x7f01d815aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 -- 172.21.15.72:0/1970701058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d8149ec0 con 0x7f01d814a980 2024-03-06T07:24:24.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01dce66640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01d81f30a0 0x7f01d806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d7fff640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c0004800 0x7f01d815aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d77fe640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 0x7f01d806f0e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f01cc037040 tx=0x7f01cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc039050 con 0x7f01d814a980 2024-03-06T07:24:24.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01cc045070 con 0x7f01d814a980 2024-03-06T07:24:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.044+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d806f850 con 0x7f01d814a980 2024-03-06T07:24:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.045+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0044e0 con 0x7f01d814a980 2024-03-06T07:24:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.045+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d81f3520 con 0x7f01d814a980 2024-03-06T07:24:24.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.045+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d804f330 con 0x7f01d814a980 2024-03-06T07:24:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.046+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01cc003a50 con 0x7f01d814a980 2024-03-06T07:24:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.046+0000 7f01d4ff9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01c406b4d0 0x7f01c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.046+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01cc0abd00 con 0x7f01d814a980 2024-03-06T07:24:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.046+0000 7f01d7fff640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01c406b4d0 0x7f01c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.047+0000 7f01d7fff640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01c406b4d0 0x7f01c406d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f01c80016b0 tx=0x7f01c8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.055+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01cc04e020 con 0x7f01d814a980 2024-03-06T07:24:24.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.232+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d80b1820 con 0x7f01d814a980 2024-03-06T07:24:24.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.233+0000 7f01d4ff9640 1 -- 172.21.15.72:0/1970701058 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f01cc07aca0 con 0x7f01d814a980 2024-03-06T07:24:24.236 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:24.236 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01c406b4d0 msgr2=0x7f01c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01c406b4d0 0x7f01c406d990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f01c80016b0 tx=0x7f01c8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 msgr2=0x7f01d806f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 0x7f01d806f0e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f01cc037040 tx=0x7f01cc004730 comp rx=0 tx=0).stop 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 shutdown_connections 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f01c406b4d0 0x7f01c406d990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01d81f30a0 0x7f01d806cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01d814a980 0x7f01d806f0e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 --2- 172.21.15.72:0/1970701058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01c0004800 0x7f01d815aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 >> 172.21.15.72:0/1970701058 conn(0x7f01d809a9f0 msgr2=0x7f01d809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.235+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 shutdown_connections 2024-03-06T07:24:24.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.236+0000 7f01de8c9640 1 -- 172.21.15.72:0/1970701058 wait complete. 2024-03-06T07:24:24.248 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:57.998737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:58.744228+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:24.249 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:24.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.350+0000 7f738489a640 1 Processor -- start 2024-03-06T07:24:24.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.350+0000 7f738489a640 1 -- start start 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f738489a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f73800d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f738489a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73800d1910 0x7f73800d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f738489a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73800d2740 con 0x7f73800d22b0 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f738489a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73801618b0 con 0x7f73800d1910 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f738489a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73800cf400 con 0x7f73801614d0 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73800d1910 0x7f73800d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73800d1910 0x7f73800d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58716/0 (socket says 172.21.15.72:58716) 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 -- 172.21.15.72:0/1534783005 learned_addr learned my addr 172.21.15.72:0/1534783005 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737e59c640 1 --2- 172.21.15.72:0/1534783005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f73800d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 -- 172.21.15.72:0/1534783005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 msgr2=0x7f73800d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 --2- 172.21.15.72:0/1534783005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f73800d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 -- 172.21.15.72:0/1534783005 >> v1:172.21.15.146:6789/0 conn(0x7f73800d22b0 legacy=0x7f73800d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.351+0000 7f737dd9b640 1 -- 172.21.15.72:0/1534783005 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73800cf6a0 con 0x7f73800d1910 2024-03-06T07:24:24.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.352+0000 7f737dd9b640 1 --2- 172.21.15.72:0/1534783005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800d1d70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7374009870 tx=0x7f737402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4a6a0bbfc94ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.352+0000 7f737d59a640 1 -- 172.21.15.72:0/1534783005 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374034070 con 0x7f73800d1910 2024-03-06T07:24:24.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.352+0000 7f737d59a640 1 -- 172.21.15.72:0/1534783005 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374030840 con 0x7f73800d1910 2024-03-06T07:24:24.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.352+0000 7f737d59a640 1 -- 172.21.15.72:0/1534783005 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374030c10 con 0x7f73800d1910 2024-03-06T07:24:24.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 -- 172.21.15.72:0/1534783005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 msgr2=0x7f73800d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 --2- 172.21.15.72:0/1534783005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800d1d70 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f7374009870 tx=0x7f737402b980 comp rx=0 tx=0).stop 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 -- 172.21.15.72:0/1534783005 shutdown_connections 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 --2- 172.21.15.72:0/1534783005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800d1d70 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 --2- 172.21.15.72:0/1534783005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f73800d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 -- 172.21.15.72:0/1534783005 >> 172.21.15.72:0/1534783005 conn(0x7f73800d3d40 msgr2=0x7f73800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 -- 172.21.15.72:0/1534783005 shutdown_connections 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.353+0000 7f738489a640 1 -- 172.21.15.72:0/1534783005 wait complete. 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.354+0000 7f738489a640 1 Processor -- start 2024-03-06T07:24:24.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.354+0000 7f738489a640 1 -- start start 2024-03-06T07:24:24.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f738489a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f738489a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73800d22b0 0x7f73800ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f738489a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f738014fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f738489a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b0910 con 0x7f73800d1910 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f738489a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73800cfc60 con 0x7f73800d22b0 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f738489a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73800b1e40 con 0x7f73801614d0 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58730/0 (socket says 172.21.15.72:58730) 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 -- 172.21.15.72:0/586961239 learned_addr learned my addr 172.21.15.72:0/586961239 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:24.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737dd9b640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73800d22b0 0x7f73800ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737ed9d640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f738014fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 -- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 msgr2=0x7f738014fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f738014fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 -- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73800d22b0 msgr2=0x7f73800ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73800d22b0 0x7f73800ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 -- 172.21.15.72:0/586961239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73800b8460 con 0x7f73800d1910 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737ed9d640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f738014fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.355+0000 7f737e59c640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800b42a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f737000e8a0 tx=0x7f737000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.356+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370017070 con 0x7f73800d1910 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.356+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737000c940 con 0x7f73800d1910 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.356+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73800b8740 con 0x7f73800d1910 2024-03-06T07:24:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.356+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737000cd30 con 0x7f73800d1910 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.356+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73800b8d00 con 0x7f73800d1910 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.357+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7380068ff0 con 0x7f73800d1910 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.357+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f737001a070 con 0x7f73800d1910 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.358+0000 7f736effd640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f736006b4d0 0x7f736006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.358+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f737008a3a0 con 0x7f73800d1910 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.358+0000 7f737dd9b640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f736006b4d0 0x7f736006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.358+0000 7f737dd9b640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f736006b4d0 0x7f736006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7374036000 tx=0x7f737403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.362+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73700592b0 con 0x7f73800d1910 2024-03-06T07:24:24.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.574+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f73800b9b10 con 0x7f73800d1910 2024-03-06T07:24:24.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.574+0000 7f736effd640 1 -- 172.21.15.72:0/586961239 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7370058bb0 con 0x7f73800d1910 2024-03-06T07:24:24.576 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:24:24.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f736006b4d0 msgr2=0x7f736006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f736006b4d0 0x7f736006d990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7374036000 tx=0x7f737403b040 comp rx=0 tx=0).stop 2024-03-06T07:24:24.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 msgr2=0x7f73800b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800b42a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f737000e8a0 tx=0x7f737000ed60 comp rx=0 tx=0).stop 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 shutdown_connections 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f736006b4d0 0x7f736006d990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73801614d0 0x7f738014fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73800d22b0 0x7f73800ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 --2- 172.21.15.72:0/586961239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73800d1910 0x7f73800b42a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 >> 172.21.15.72:0/586961239 conn(0x7f73800d3d40 msgr2=0x7f738006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 shutdown_connections 2024-03-06T07:24:24.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.576+0000 7f738489a640 1 -- 172.21.15.72:0/586961239 wait complete. 2024-03-06T07:24:24.589 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:24.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.689+0000 7f57cb466640 1 Processor -- start 2024-03-06T07:24:24.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.690+0000 7f57cb466640 1 -- start start 2024-03-06T07:24:24.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.690+0000 7f57cb466640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f57c40719e0 0x7f57c40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.690+0000 7f57cb466640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f57c406c9f0 0x7f57c406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.690+0000 7f57cb466640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c40b3320 con 0x7f57c415a220 2024-03-06T07:24:24.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.690+0000 7f57cb466640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c40cf610 con 0x7f57c406c9f0 2024-03-06T07:24:24.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.690+0000 7f57cb466640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c40728d0 con 0x7f57c40719e0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9a03640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f57c406c9f0 0x7f57c406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9202640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f57c415a220 0x7f57c40712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:49954/0 (socket says 172.21.15.72:49954) 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9202640 1 -- 172.21.15.72:0/2442920955 learned_addr learned my addr 172.21.15.72:0/2442920955 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c8a01640 1 --2- 172.21.15.72:0/2442920955 >> v2:172.21.15.146:3300/0 conn(0x7f57c40719e0 0x7f57c40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c3fff640 1 -- 172.21.15.72:0/2442920955 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938856862 0 0) 0x7f57c40b3320 con 0x7f57c415a220 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9a03640 1 -- 172.21.15.72:0/2442920955 >> v2:172.21.15.146:3300/0 conn(0x7f57c40719e0 msgr2=0x7f57c40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9a03640 1 --2- 172.21.15.72:0/2442920955 >> v2:172.21.15.146:3300/0 conn(0x7f57c40719e0 0x7f57c40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9a03640 1 -- 172.21.15.72:0/2442920955 >> v1:172.21.15.146:6789/0 conn(0x7f57c415a220 legacy=0x7f57c40712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.691+0000 7f57c9a03640 1 -- 172.21.15.72:0/2442920955 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c40b3590 con 0x7f57c406c9f0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.692+0000 7f57c9a03640 1 --2- 172.21.15.72:0/2442920955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c406c9f0 0x7f57c406ce40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f57c40d0980 tx=0x7f57b402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7788fe9ba7e52ff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.692+0000 7f57c3fff640 1 -- 172.21.15.72:0/2442920955 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4039070 con 0x7f57c406c9f0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.692+0000 7f57c3fff640 1 -- 172.21.15.72:0/2442920955 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b402cbf0 con 0x7f57c406c9f0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.692+0000 7f57c3fff640 1 -- 172.21.15.72:0/2442920955 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4034440 con 0x7f57c406c9f0 2024-03-06T07:24:24.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 -- 172.21.15.72:0/2442920955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c406c9f0 msgr2=0x7f57c406ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 --2- 172.21.15.72:0/2442920955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c406c9f0 0x7f57c406ce40 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f57c40d0980 tx=0x7f57b402be40 comp rx=0 tx=0).stop 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 -- 172.21.15.72:0/2442920955 shutdown_connections 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 --2- 172.21.15.72:0/2442920955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c406c9f0 0x7f57c406ce40 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 --2- 172.21.15.72:0/2442920955 >> v2:172.21.15.146:3300/0 conn(0x7f57c40719e0 0x7f57c40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 -- 172.21.15.72:0/2442920955 >> 172.21.15.72:0/2442920955 conn(0x7f57c406b190 msgr2=0x7f57c406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 -- 172.21.15.72:0/2442920955 shutdown_connections 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.693+0000 7f57cb466640 1 -- 172.21.15.72:0/2442920955 wait complete. 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.694+0000 7f57cb466640 1 Processor -- start 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.694+0000 7f57cb466640 1 -- start start 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57cb466640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57c406c9f0 0x7f57c4150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57cb466640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 0x7f57c4153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57cb466640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57c415a220 0x7f57c4152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57cb466640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4072530 con 0x7f57c40719e0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57cb466640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4072030 con 0x7f57c406c9f0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57cb466640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c40726b0 con 0x7f57c415a220 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 0x7f57c4153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 0x7f57c4153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58742/0 (socket says 172.21.15.72:58742) 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 -- 172.21.15.72:0/3792759650 learned_addr learned my addr 172.21.15.72:0/3792759650 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c9202640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57c406c9f0 0x7f57c4150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c9a03640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57c415a220 0x7f57c4152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 -- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57c415a220 msgr2=0x7f57c4152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57c415a220 0x7f57c4152b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 -- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57c406c9f0 msgr2=0x7f57c4150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57c406c9f0 0x7f57c4150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 -- 172.21.15.72:0/3792759650 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c40b1160 con 0x7f57c40719e0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.695+0000 7f57c8a01640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 0x7f57c4153860 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f57b800e910 tx=0x7f57b800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.696+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8016070 con 0x7f57c40719e0 2024-03-06T07:24:24.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.696+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b800cc40 con 0x7f57c40719e0 2024-03-06T07:24:24.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.696+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c41520b0 con 0x7f57c40719e0 2024-03-06T07:24:24.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.696+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b800b040 con 0x7f57c40719e0 2024-03-06T07:24:24.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.696+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c4152670 con 0x7f57c40719e0 2024-03-06T07:24:24.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.697+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c40732b0 con 0x7f57c40719e0 2024-03-06T07:24:24.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.697+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57b8019070 con 0x7f57c40719e0 2024-03-06T07:24:24.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.697+0000 7f57c1ffb640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57b006b4d0 0x7f57b006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:24.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.698+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57b8089e00 con 0x7f57c40719e0 2024-03-06T07:24:24.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.698+0000 7f57c9202640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57b006b4d0 0x7f57b006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:24.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.698+0000 7f57c9202640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57b006b4d0 0x7f57b006d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f57bc000f60 tx=0x7f57bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:24.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.702+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f57b8058c80 con 0x7f57c40719e0 2024-03-06T07:24:24.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.926+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f57c4068ff0 con 0x7f57c40719e0 2024-03-06T07:24:24.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.926+0000 7f57c1ffb640 1 -- 172.21.15.72:0/3792759650 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f57b8058580 con 0x7f57c40719e0 2024-03-06T07:24:24.929 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:24:24.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57b006b4d0 msgr2=0x7f57b006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57b006b4d0 0x7f57b006d990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f57bc000f60 tx=0x7f57bc009040 comp rx=0 tx=0).stop 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 msgr2=0x7f57c4153860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 0x7f57c4153860 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f57b800e910 tx=0x7f57b800edd0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 shutdown_connections 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57b006b4d0 0x7f57b006d990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57c415a220 0x7f57c4152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c40719e0 0x7f57c4153860 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 --2- 172.21.15.72:0/3792759650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57c406c9f0 0x7f57c4150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 >> 172.21.15.72:0/3792759650 conn(0x7f57c406b190 msgr2=0x7f57c40cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 shutdown_connections 2024-03-06T07:24:24.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:24.928+0000 7f57cb466640 1 -- 172.21.15.72:0/3792759650 wait complete. 2024-03-06T07:24:24.941 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:25.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.044+0000 7f329b278640 1 Processor -- start 2024-03-06T07:24:25.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.044+0000 7f329b278640 1 -- start start 2024-03-06T07:24:25.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f329b278640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32940b1e40 0x7f3294168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f329b278640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32940d2730 con 0x7f32941614d0 2024-03-06T07:24:25.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f329b278640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32941618b0 con 0x7f32940d4860 2024-03-06T07:24:25.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f329b278640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32940d4cf0 con 0x7f32940b1e40 2024-03-06T07:24:25.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f3299815640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f32940b1e40 0x7f3294168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f3299014640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f32941614d0 0x7f32940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53656/0 (socket says 172.21.15.72:53656) 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f3299014640 1 -- 172.21.15.72:0/44483872 learned_addr learned my addr 172.21.15.72:0/44483872 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f3299815640 1 -- 172.21.15.72:0/44483872 >> v1:172.21.15.146:6789/0 conn(0x7f32940d4860 legacy=0x7f32940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f3299815640 1 -- 172.21.15.72:0/44483872 >> v1:172.21.15.72:6789/0 conn(0x7f32941614d0 legacy=0x7f32940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f328bfff640 1 -- 172.21.15.72:0/44483872 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735827162 0 0) 0x7f32940d2730 con 0x7f32941614d0 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.045+0000 7f3299815640 1 -- 172.21.15.72:0/44483872 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940cf400 con 0x7f32940b1e40 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.046+0000 7f3299815640 1 --2- 172.21.15.72:0/44483872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32940b1e40 0x7f3294168b60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f32940cf5d0 tx=0x7f328c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523e37a29c75976e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.046+0000 7f328bfff640 1 -- 172.21.15.72:0/44483872 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c039070 con 0x7f32940b1e40 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.046+0000 7f328bfff640 1 -- 172.21.15.72:0/44483872 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c02cbc0 con 0x7f32940b1e40 2024-03-06T07:24:25.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.046+0000 7f328bfff640 1 -- 172.21.15.72:0/44483872 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c034440 con 0x7f32940b1e40 2024-03-06T07:24:25.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 -- 172.21.15.72:0/44483872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32940b1e40 msgr2=0x7f3294168b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 --2- 172.21.15.72:0/44483872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32940b1e40 0x7f3294168b60 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f32940cf5d0 tx=0x7f328c02be10 comp rx=0 tx=0).stop 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 -- 172.21.15.72:0/44483872 shutdown_connections 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 --2- 172.21.15.72:0/44483872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32940b1e40 0x7f3294168b60 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 -- 172.21.15.72:0/44483872 >> 172.21.15.72:0/44483872 conn(0x7f32940d3d40 msgr2=0x7f32940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 -- 172.21.15.72:0/44483872 shutdown_connections 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.047+0000 7f329b278640 1 -- 172.21.15.72:0/44483872 wait complete. 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.048+0000 7f329b278640 1 Processor -- start 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.048+0000 7f329b278640 1 -- start start 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f329b278640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32940b1e40 0x7f32940cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f329b278640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32940d4860 0x7f32940cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f329b278640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 0x7f32940cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f329b278640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32940cfc60 con 0x7f32941614d0 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f329b278640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32940cf980 con 0x7f32940b1e40 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f329b278640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32940b0910 con 0x7f32940d4860 2024-03-06T07:24:25.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 0x7f32940cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 0x7f32940cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58762/0 (socket says 172.21.15.72:58762) 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 -- 172.21.15.72:0/1839466016 learned_addr learned my addr 172.21.15.72:0/1839466016 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3298813640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32940d4860 0x7f32940cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299014640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32940b1e40 0x7f32940cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 -- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32940d4860 msgr2=0x7f32940cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32940d4860 0x7f32940cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 -- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32940b1e40 msgr2=0x7f32940cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32940b1e40 0x7f32940cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 -- 172.21.15.72:0/1839466016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32940cbd50 con 0x7f32941614d0 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3298813640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32940d4860 0x7f32940cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.049+0000 7f3299815640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 0x7f32940cce20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f328c009e80 tx=0x7f328c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.050+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c044020 con 0x7f32941614d0 2024-03-06T07:24:25.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.050+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328c03f470 con 0x7f32941614d0 2024-03-06T07:24:25.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.050+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328c039040 con 0x7f32941614d0 2024-03-06T07:24:25.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.050+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32940cc030 con 0x7f32941614d0 2024-03-06T07:24:25.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.050+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32940c7d20 con 0x7f32941614d0 2024-03-06T07:24:25.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.051+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32940cd480 con 0x7f32941614d0 2024-03-06T07:24:25.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.051+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f328c047020 con 0x7f32941614d0 2024-03-06T07:24:25.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.052+0000 7f3289ffb640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.052+0000 7f3299014640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.052+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f328c0acd80 con 0x7f32941614d0 2024-03-06T07:24:25.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.052+0000 7f3299014640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3284000f60 tx=0x7f3284007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.056+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f328c04e030 con 0x7f32941614d0 2024-03-06T07:24:25.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.232+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3294068ff0 con 0x7f32941614d0 2024-03-06T07:24:25.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.233+0000 7f3289ffb640 1 -- 172.21.15.72:0/1839466016 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f328c07bc90 con 0x7f32941614d0 2024-03-06T07:24:25.235 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 msgr2=0x7f327806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3284000f60 tx=0x7f3284007d50 comp rx=0 tx=0).stop 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 msgr2=0x7f32940cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 0x7f32940cce20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f328c009e80 tx=0x7f328c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 shutdown_connections 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32941614d0 0x7f32940cce20 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32940d4860 0x7f32940cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 --2- 172.21.15.72:0/1839466016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32940b1e40 0x7f32940cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 >> 172.21.15.72:0/1839466016 conn(0x7f32940d3d40 msgr2=0x7f329406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 shutdown_connections 2024-03-06T07:24:25.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.235+0000 7f329b278640 1 -- 172.21.15.72:0/1839466016 wait complete. 2024-03-06T07:24:25.251 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:25.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.351+0000 7f5a019ac640 1 Processor -- start 2024-03-06T07:24:25.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.351+0000 7f5a019ac640 1 -- start start 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f5a019ac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59fc0670a0 0x7f59fc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f5a019ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0b50d0 0x7f59fc0ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f5a019ac640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59fc0b5c30 con 0x7f59fc0d02b0 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f5a019ac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59fc0cd940 con 0x7f59fc0670a0 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f5a019ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0b5f10 con 0x7f59fc0b50d0 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f59f3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59fc0670a0 0x7f59fc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f59f3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59fc0670a0 0x7f59fc06b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58776/0 (socket says 172.21.15.72:58776) 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f59f3fff640 1 -- 172.21.15.72:0/3318562531 learned_addr learned my addr 172.21.15.72:0/3318562531 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.352+0000 7f59fb7fe640 1 --2- 172.21.15.72:0/3318562531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0b50d0 0x7f59fc0ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.353+0000 7f59f3fff640 1 -- 172.21.15.72:0/3318562531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0b50d0 msgr2=0x7f59fc0ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:25.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.353+0000 7f59f3fff640 1 --2- 172.21.15.72:0/3318562531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0b50d0 0x7f59fc0ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.353+0000 7f59f3fff640 1 -- 172.21.15.72:0/3318562531 >> v1:172.21.15.146:6789/0 conn(0x7f59fc0d02b0 legacy=0x7f59fc06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:25.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.353+0000 7f59f3fff640 1 -- 172.21.15.72:0/3318562531 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0cd5d0 con 0x7f59fc0670a0 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f59f3fff640 1 --2- 172.21.15.72:0/3318562531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0670a0 0x7f59fc06b6c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f59fc06f9e0 tx=0x7f59e402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6391b91c59da8478 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f59fa7fc640 1 -- 172.21.15.72:0/3318562531 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e40315a0 con 0x7f59fc0670a0 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f59fa7fc640 1 -- 172.21.15.72:0/3318562531 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e4037070 con 0x7f59fc0670a0 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f59fa7fc640 1 -- 172.21.15.72:0/3318562531 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e4031960 con 0x7f59fc0670a0 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 -- 172.21.15.72:0/3318562531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0670a0 msgr2=0x7f59fc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 --2- 172.21.15.72:0/3318562531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0670a0 0x7f59fc06b6c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f59fc06f9e0 tx=0x7f59e402b840 comp rx=0 tx=0).stop 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 -- 172.21.15.72:0/3318562531 shutdown_connections 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 --2- 172.21.15.72:0/3318562531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0b50d0 0x7f59fc0ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 --2- 172.21.15.72:0/3318562531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0670a0 0x7f59fc06b6c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 -- 172.21.15.72:0/3318562531 >> 172.21.15.72:0/3318562531 conn(0x7f59fc072520 msgr2=0x7f59fc072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 -- 172.21.15.72:0/3318562531 shutdown_connections 2024-03-06T07:24:25.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.354+0000 7f5a019ac640 1 -- 172.21.15.72:0/3318562531 wait complete. 2024-03-06T07:24:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 Processor -- start 2024-03-06T07:24:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 -- start start 2024-03-06T07:24:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0670a0 0x7f59fc1f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59fc0b50d0 0x7f59fc0c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 0x7f59fc1f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc06bd80 con 0x7f59fc0d02b0 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc06bc00 con 0x7f59fc0b50d0 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f5a019ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59fc0d0690 con 0x7f59fc0670a0 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f59fb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 0x7f59fc1f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f59fb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 0x7f59fc1f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58784/0 (socket says 172.21.15.72:58784) 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.356+0000 7f59fb7fe640 1 -- 172.21.15.72:0/844725745 learned_addr learned my addr 172.21.15.72:0/844725745 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59f3fff640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59fc0b50d0 0x7f59fc0c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59faffd640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0670a0 0x7f59fc1f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59fb7fe640 1 -- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0670a0 msgr2=0x7f59fc1f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59fb7fe640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0670a0 0x7f59fc1f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59fb7fe640 1 -- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59fc0b50d0 msgr2=0x7f59fc0c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59fb7fe640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59fc0b50d0 0x7f59fc0c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59fb7fe640 1 -- 172.21.15.72:0/844725745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc0c6390 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59f3fff640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59fc0b50d0 0x7f59fc0c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59fb7fe640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 0x7f59fc1f74b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f59f400cda0 tx=0x7f59f400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4003b90 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f4003d30 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59fc0c6670 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4004240 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.357+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59fc0afe90 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.358+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59fc04f330 con 0x7f59fc0d02b0 2024-03-06T07:24:25.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.358+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59f40043e0 con 0x7f59fc0d02b0 2024-03-06T07:24:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.359+0000 7f59f8ff9640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59dc06b4d0 0x7f59dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.359+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59f408ad80 con 0x7f59fc0d02b0 2024-03-06T07:24:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.359+0000 7f59faffd640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59dc06b4d0 0x7f59dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:25.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.359+0000 7f59faffd640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59dc06b4d0 0x7f59dc06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006fd0 tx=0x7f59ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:25.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.363+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59f401d340 con 0x7f59fc0d02b0 2024-03-06T07:24:25.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.552+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f59fc06e850 con 0x7f59fc0d02b0 2024-03-06T07:24:25.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.553+0000 7f59f8ff9640 1 -- 172.21.15.72:0/844725745 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7f59f4059c00 con 0x7f59fc0d02b0 2024-03-06T07:24:25.555 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:24:25.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59dc06b4d0 msgr2=0x7f59dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59dc06b4d0 0x7f59dc06d990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006fd0 tx=0x7f59ec008040 comp rx=0 tx=0).stop 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 msgr2=0x7f59fc1f74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 0x7f59fc1f74b0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f59f400cda0 tx=0x7f59f400a660 comp rx=0 tx=0).stop 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 shutdown_connections 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59dc06b4d0 0x7f59dc06d990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59fc0d02b0 0x7f59fc1f74b0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59fc0b50d0 0x7f59fc0c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 --2- 172.21.15.72:0/844725745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59fc0670a0 0x7f59fc1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 >> 172.21.15.72:0/844725745 conn(0x7f59fc072520 msgr2=0x7f59fc15bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 shutdown_connections 2024-03-06T07:24:25.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:25.555+0000 7f5a019ac640 1 -- 172.21.15.72:0/844725745 wait complete. 2024-03-06T07:24:25.568 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:24:25.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:25.568 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:25.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[58343]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:25.595 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:31.696 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:24:31.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:31.697 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:31.712 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.849+0000 7fd1ad8eb640 1 Processor -- start 2024-03-06T07:24:31.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.850+0000 7fd1ad8eb640 1 -- start start 2024-03-06T07:24:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1ad8eb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1a81515e0 0x7fd1a80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1ad8eb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd1a815b430 0x7fd1a8071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1ad8eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a815b8e0 con 0x7fd1a80cd7a0 2024-03-06T07:24:31.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1ad8eb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a8070a70 con 0x7fd1a81515e0 2024-03-06T07:24:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1ad8eb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a8070d50 con 0x7fd1a815b430 2024-03-06T07:24:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1a81515e0 0x7fd1a80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1a81515e0 0x7fd1a80cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58790/0 (socket says 172.21.15.72:58790) 2024-03-06T07:24:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 -- 172.21.15.72:0/292142185 learned_addr learned my addr 172.21.15.72:0/292142185 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:31.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a67fc640 1 --2- 172.21.15.72:0/292142185 >> v2:172.21.15.146:3300/0 conn(0x7fd1a815b430 0x7fd1a8071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 -- 172.21.15.72:0/292142185 >> v2:172.21.15.146:3300/0 conn(0x7fd1a815b430 msgr2=0x7fd1a8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 --2- 172.21.15.72:0/292142185 >> v2:172.21.15.146:3300/0 conn(0x7fd1a815b430 0x7fd1a8071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 -- 172.21.15.72:0/292142185 >> v1:172.21.15.146:6789/0 conn(0x7fd1a80cd7a0 legacy=0x7fd1a80cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a6ffd640 1 -- 172.21.15.72:0/292142185 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8071030 con 0x7fd1a81515e0 2024-03-06T07:24:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.851+0000 7fd1a67fc640 1 --2- 172.21.15.72:0/292142185 >> v2:172.21.15.146:3300/0 conn(0x7fd1a815b430 0x7fd1a8071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:31.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.852+0000 7fd1a6ffd640 1 --2- 172.21.15.72:0/292142185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a80cce40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd1a806eb50 tx=0x7fd19802cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e02abf6aac691d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.852+0000 7fd1a5ffb640 1 -- 172.21.15.72:0/292142185 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19803a070 con 0x7fd1a81515e0 2024-03-06T07:24:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.852+0000 7fd1a5ffb640 1 -- 172.21.15.72:0/292142185 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19802db80 con 0x7fd1a81515e0 2024-03-06T07:24:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.852+0000 7fd1a5ffb640 1 -- 172.21.15.72:0/292142185 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19800b040 con 0x7fd1a81515e0 2024-03-06T07:24:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/292142185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 msgr2=0x7fd1a80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/292142185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a80cce40 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fd1a806eb50 tx=0x7fd19802cdd0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/292142185 shutdown_connections 2024-03-06T07:24:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/292142185 >> v2:172.21.15.146:3300/0 conn(0x7fd1a815b430 0x7fd1a8071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/292142185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a80cce40 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/292142185 >> 172.21.15.72:0/292142185 conn(0x7fd1a80cf8a0 msgr2=0x7fd1a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/292142185 shutdown_connections 2024-03-06T07:24:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.853+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/292142185 wait complete. 2024-03-06T07:24:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.854+0000 7fd1ad8eb640 1 Processor -- start 2024-03-06T07:24:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.854+0000 7fd1ad8eb640 1 -- start start 2024-03-06T07:24:31.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1ad8eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1a80cd7a0 0x7fd1a815e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1ad8eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a81489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1ad8eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1a815b430 0x7fd1a81426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1ad8eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8071ef0 con 0x7fd1a81515e0 2024-03-06T07:24:31.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1ad8eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a8071d70 con 0x7fd1a815b430 2024-03-06T07:24:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1ad8eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a80cc3a0 con 0x7fd1a80cd7a0 2024-03-06T07:24:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a81489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a81489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58792/0 (socket says 172.21.15.72:58792) 2024-03-06T07:24:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 -- 172.21.15.72:0/1754110 learned_addr learned my addr 172.21.15.72:0/1754110 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 -- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1a80cd7a0 msgr2=0x7fd1a815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a6ffd640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1a80cd7a0 0x7fd1a815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a77fe640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1a815b430 0x7fd1a81426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1a80cd7a0 0x7fd1a815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 -- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1a815b430 msgr2=0x7fd1a81426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:31.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1a815b430 0x7fd1a81426f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:31.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a67fc640 1 -- 172.21.15.72:0/1754110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8149110 con 0x7fd1a81515e0 2024-03-06T07:24:31.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a6ffd640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1a80cd7a0 0x7fd1a815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:31.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.855+0000 7fd1a77fe640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1a815b430 0x7fd1a81426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:31.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.856+0000 7fd1a67fc640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a81489c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c7f0 tx=0x7fd19c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.856+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c015070 con 0x7fd1a81515e0 2024-03-06T07:24:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.856+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c007820 con 0x7fd1a81515e0 2024-03-06T07:24:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.856+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a81493f0 con 0x7fd1a81515e0 2024-03-06T07:24:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.856+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c007bd0 con 0x7fd1a81515e0 2024-03-06T07:24:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.856+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a815bdd0 con 0x7fd1a81515e0 2024-03-06T07:24:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.857+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a8068840 con 0x7fd1a81515e0 2024-03-06T07:24:31.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.857+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd19c018070 con 0x7fd1a81515e0 2024-03-06T07:24:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.858+0000 7fd18ffff640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19406b4d0 0x7fd19406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.858+0000 7fd1a6ffd640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19406b4d0 0x7fd19406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.858+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd19c08a3b0 con 0x7fd1a81515e0 2024-03-06T07:24:31.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.858+0000 7fd1a6ffd640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19406b4d0 0x7fd19406d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd198037000 tx=0x7fd19800c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:31.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:31.862+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd19c059190 con 0x7fd1a81515e0 2024-03-06T07:24:32.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.080+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1a804f330 con 0x7fd1a81515e0 2024-03-06T07:24:32.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.081+0000 7fd18ffff640 1 -- 172.21.15.72:0/1754110 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fd19c058a90 con 0x7fd1a81515e0 2024-03-06T07:24:32.083 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:32.084 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:32.085 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:32.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.082+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19406b4d0 msgr2=0x7fd19406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.082+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19406b4d0 0x7fd19406d990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd198037000 tx=0x7fd19800c040 comp rx=0 tx=0).stop 2024-03-06T07:24:32.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.082+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 msgr2=0x7fd1a81489c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.082+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a81489c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c7f0 tx=0x7fd19c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 shutdown_connections 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd19406b4d0 0x7fd19406d990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1a815b430 0x7fd1a81426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1a81515e0 0x7fd1a81489c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 --2- 172.21.15.72:0/1754110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1a80cd7a0 0x7fd1a815e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 >> 172.21.15.72:0/1754110 conn(0x7fd1a80cf8a0 msgr2=0x7fd1a8072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 shutdown_connections 2024-03-06T07:24:32.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.083+0000 7fd1ad8eb640 1 -- 172.21.15.72:0/1754110 wait complete. 2024-03-06T07:24:32.097 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dirfrag split /subdir 0/0 1 2024-03-06T07:24:32.211 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-03-06T07:24:32.211 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:32.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.310+0000 7fd0406a4640 1 Processor -- start 2024-03-06T07:24:32.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.310+0000 7fd0406a4640 1 -- start start 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd0406a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd03815bdd0 0x7fd0380c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd0406a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd03815fa10 0x7fd0380ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd0406a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0380cebe0 con 0x7fd0380d16e0 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd0406a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0380ceec0 con 0x7fd03815bdd0 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd0406a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380cf1a0 con 0x7fd03815fa10 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd03815bdd0 0x7fd0380c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd03815bdd0 0x7fd0380c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58804/0 (socket says 172.21.15.72:58804) 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 -- 172.21.15.72:0/3542636497 learned_addr learned my addr 172.21.15.72:0/3542636497 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:32.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03dc3f640 1 --2- 172.21.15.72:0/3542636497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd03815fa10 0x7fd0380ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 -- 172.21.15.72:0/3542636497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd03815fa10 msgr2=0x7fd0380ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 --2- 172.21.15.72:0/3542636497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd03815fa10 0x7fd0380ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 -- 172.21.15.72:0/3542636497 >> v1:172.21.15.146:6789/0 conn(0x7fd0380d16e0 legacy=0x7fd0380d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.311+0000 7fd03e440640 1 -- 172.21.15.72:0/3542636497 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380cc8e0 con 0x7fd03815bdd0 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.312+0000 7fd03e440640 1 --2- 172.21.15.72:0/3542636497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815bdd0 0x7fd0380c5610 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd034009870 tx=0x7fd03402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfae8a2279270b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.312+0000 7fd03d43e640 1 -- 172.21.15.72:0/3542636497 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd034035020 con 0x7fd03815bdd0 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.312+0000 7fd03d43e640 1 -- 172.21.15.72:0/3542636497 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd034031640 con 0x7fd03815bdd0 2024-03-06T07:24:32.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.312+0000 7fd03d43e640 1 -- 172.21.15.72:0/3542636497 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03402d040 con 0x7fd03815bdd0 2024-03-06T07:24:32.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.312+0000 7fd0406a4640 1 -- 172.21.15.72:0/3542636497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815bdd0 msgr2=0x7fd0380c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.312+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3542636497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815bdd0 0x7fd0380c5610 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fd034009870 tx=0x7fd03402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.313+0000 7fd0406a4640 1 -- 172.21.15.72:0/3542636497 shutdown_connections 2024-03-06T07:24:32.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.313+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3542636497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd03815fa10 0x7fd0380ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.313+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3542636497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815bdd0 0x7fd0380c5610 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.313+0000 7fd0406a4640 1 -- 172.21.15.72:0/3542636497 >> 172.21.15.72:0/3542636497 conn(0x7fd03809a9f0 msgr2=0x7fd03809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.313+0000 7fd0406a4640 1 -- 172.21.15.72:0/3542636497 shutdown_connections 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.313+0000 7fd0406a4640 1 -- 172.21.15.72:0/3542636497 wait complete. 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.314+0000 7fd0406a4640 1 Processor -- start 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.314+0000 7fd0406a4640 1 -- start start 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.314+0000 7fd0406a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0380d16e0 0x7fd03806c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.314+0000 7fd0406a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd03815bdd0 0x7fd0381e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd0406a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 0x7fd03806d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd0406a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03815c1b0 con 0x7fd03815fa10 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd0406a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380cc5d0 con 0x7fd03815bdd0 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd0406a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380ce7b0 con 0x7fd0380d16e0 2024-03-06T07:24:32.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 0x7fd03806d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 0x7fd03806d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58816/0 (socket says 172.21.15.72:58816) 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 -- 172.21.15.72:0/3009674780 learned_addr learned my addr 172.21.15.72:0/3009674780 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03dc3f640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd03815bdd0 0x7fd0381e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03e440640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0380d16e0 0x7fd03806c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 -- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0380d16e0 msgr2=0x7fd03806c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0380d16e0 0x7fd03806c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 -- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd03815bdd0 msgr2=0x7fd0381e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd03815bdd0 0x7fd0381e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 -- 172.21.15.72:0/3009674780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0380ca970 con 0x7fd03815fa10 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd03ec41640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 0x7fd03806d8d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd03000c7a0 tx=0x7fd03000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd030003b70 con 0x7fd03815fa10 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd030003d30 con 0x7fd03815fa10 2024-03-06T07:24:32.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03806c830 con 0x7fd03815fa10 2024-03-06T07:24:32.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03000aa90 con 0x7fd03815fa10 2024-03-06T07:24:32.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.315+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03806cdf0 con 0x7fd03815fa10 2024-03-06T07:24:32.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.316+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03804f330 con 0x7fd03815fa10 2024-03-06T07:24:32.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.317+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd03000ac30 con 0x7fd03815fa10 2024-03-06T07:24:32.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.317+0000 7fd02effd640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01c06b500 0x7fd01c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.317+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd030089fa0 con 0x7fd03815fa10 2024-03-06T07:24:32.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.317+0000 7fd03e440640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01c06b500 0x7fd01c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.318+0000 7fd03e440640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01c06b500 0x7fd01c06d9c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd034036000 tx=0x7fd03403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.321+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd0300587d0 con 0x7fd03815fa10 2024-03-06T07:24:32.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.541+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0380b7a40 con 0x7fd03815fa10 2024-03-06T07:24:32.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.542+0000 7fd02effd640 1 -- 172.21.15.72:0/3009674780 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fd03005c870 con 0x7fd03815fa10 2024-03-06T07:24:32.544 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:32.544 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:32.544 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01c06b500 msgr2=0x7fd01c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01c06b500 0x7fd01c06d9c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd034036000 tx=0x7fd03403c040 comp rx=0 tx=0).stop 2024-03-06T07:24:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 msgr2=0x7fd03806d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 0x7fd03806d8d0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd03000c7a0 tx=0x7fd03000cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 shutdown_connections 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd01c06b500 0x7fd01c06d9c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd03815fa10 0x7fd03806d8d0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd03815bdd0 0x7fd0381e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 --2- 172.21.15.72:0/3009674780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0380d16e0 0x7fd03806c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 >> 172.21.15.72:0/3009674780 conn(0x7fd03809a9f0 msgr2=0x7fd0380702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 shutdown_connections 2024-03-06T07:24:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.544+0000 7fd0406a4640 1 -- 172.21.15.72:0/3009674780 wait complete. 2024-03-06T07:24:32.557 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:24:32.686 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:24:32.686 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:24:32.686 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:24:32.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.814+0000 7f8633eec640 1 Processor -- start 2024-03-06T07:24:32.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.814+0000 7f8633eec640 1 -- start start 2024-03-06T07:24:32.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8633eec640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f862c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8633eec640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f862c15bf30 0x7f862c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8633eec640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862c15f580 con 0x7f862c15e850 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8633eec640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0cf3f0 con 0x7f862c15bf30 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8633eec640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c15b210 con 0x7f862c13dc10 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8632489640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f862c15e850 0x7f862c15eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57696/0 (socket says 172.21.15.72:57696) 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8632489640 1 -- 172.21.15.72:0/3957269269 learned_addr learned my addr 172.21.15.72:0/3957269269 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224445754 0 0) 0x7f862c15f580 con 0x7f862c15e850 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8631487640 1 --2- 172.21.15.72:0/3957269269 >> v2:172.21.15.146:3300/0 conn(0x7f862c15bf30 0x7f862c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8618003520 con 0x7f862c15e850 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8631c88640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f862c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1636519616 0 0) 0x7f8618003520 con 0x7f862c15e850 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 msgr2=0x7f862c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f862c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 >> v2:172.21.15.146:3300/0 conn(0x7f862c15bf30 msgr2=0x7f862c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 --2- 172.21.15.72:0/3957269269 >> v2:172.21.15.146:3300/0 conn(0x7f862c15bf30 0x7f862c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c15b450 con 0x7f862c15e850 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f86240019f0 con 0x7f862c15e850 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 >> v1:172.21.15.72:6789/0 conn(0x7f862c15e850 legacy=0x7f862c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f86180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8618003f30 0x7f8618004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8618004d40 0x7f86180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0cf3f0 con 0x7f8618004d40 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c15b210 con 0x7f8618003f30 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.815+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861800ba10 con 0x7f862c13dc10 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8632489640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8618004d40 0x7f86180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8631487640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8618003f30 0x7f8618004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8631c88640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f86180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8632489640 1 -- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 msgr2=0x7f86180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8632489640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f86180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8632489640 1 -- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8618003f30 msgr2=0x7f8618004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8632489640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8618003f30 0x7f8618004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8632489640 1 -- 172.21.15.72:0/3957269269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8618003520 con 0x7f8618004d40 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.816+0000 7f8631c88640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f862c13dc10 0x7f86180039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8632489640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8618004d40 0x7f86180051c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f862400a8a0 tx=0x7f8618005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8be6afcd7cd2ed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862400a4c0 con 0x7f8618004d40 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624034630 con 0x7f8618004d40 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8630c86640 1 -- 172.21.15.72:0/3957269269 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624034a20 con 0x7f8618004d40 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8633eec640 1 -- 172.21.15.72:0/3957269269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8618004d40 msgr2=0x7f86180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8633eec640 1 --2- 172.21.15.72:0/3957269269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8618004d40 0x7f86180051c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f862400a8a0 tx=0x7f8618005830 comp rx=0 tx=0).stop 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8632489640 1 -- 172.21.15.72:0/3957269269 reap_dead start 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8633eec640 1 -- 172.21.15.72:0/3957269269 shutdown_connections 2024-03-06T07:24:32.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8633eec640 1 -- 172.21.15.72:0/3957269269 >> 172.21.15.72:0/3957269269 conn(0x7f862c0cf8a0 msgr2=0x7f862c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:32.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8633eec640 1 -- 172.21.15.72:0/3957269269 shutdown_connections 2024-03-06T07:24:32.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.817+0000 7f8633eec640 1 -- 172.21.15.72:0/3957269269 wait complete. 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.818+0000 7f8633eec640 1 Processor -- start 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.818+0000 7f8633eec640 1 -- start start 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8633eec640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8618004d40 0x7f862c13fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8633eec640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f862c06d380 0x7f862c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8633eec640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 0x7f862c153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8633eec640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0b2950 con 0x7f862c153820 2024-03-06T07:24:32.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8633eec640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c13dff0 con 0x7f862c06d380 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8633eec640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0b2ad0 con 0x7f8618004d40 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8632489640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 0x7f862c153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8632489640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 0x7f862c153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36434/0 (socket says 172.21.15.72:36434) 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8632489640 1 -- 172.21.15.72:0/1606312398 learned_addr learned my addr 172.21.15.72:0/1606312398 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8631c88640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8618004d40 0x7f862c13fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.819+0000 7f8631487640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f862c06d380 0x7f862c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8632489640 1 -- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8618004d40 msgr2=0x7f862c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8632489640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8618004d40 0x7f862c13fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8632489640 1 -- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f862c06d380 msgr2=0x7f862c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8632489640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f862c06d380 0x7f862c06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8632489640 1 -- 172.21.15.72:0/1606312398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c13c820 con 0x7f862c153820 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8632489640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 0x7f862c153ca0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f862400a0b0 tx=0x7f862c15b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624039040 con 0x7f862c153820 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8624043070 con 0x7f862c153820 2024-03-06T07:24:32.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c13cb00 con 0x7f862c153820 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8624002030 con 0x7f862c153820 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.820+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c13d0c0 con 0x7f862c153820 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.821+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862c04f330 con 0x7f862c153820 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.822+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8624034630 con 0x7f862c153820 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.822+0000 7f86227fc640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f861006b500 0x7f861006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.822+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86240abe30 con 0x7f862c153820 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.822+0000 7f8631c88640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f861006b500 0x7f861006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:32.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.823+0000 7f8631c88640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f861006b500 0x7f861006d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8628009a90 tx=0x7f8628008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:32.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:32.831+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f862407ad40 con 0x7f862c153820 2024-03-06T07:24:33.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.063+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f862c06eb00 con 0x7f862c153820 2024-03-06T07:24:33.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.063+0000 7f86227fc640 1 -- 172.21.15.72:0/1606312398 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7f862407a640 con 0x7f862c153820 2024-03-06T07:24:33.066 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f861006b500 msgr2=0x7f861006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f861006b500 0x7f861006d9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8628009a90 tx=0x7f8628008040 comp rx=0 tx=0).stop 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 msgr2=0x7f862c153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 0x7f862c153ca0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f862400a0b0 tx=0x7f862c15b620 comp rx=0 tx=0).stop 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 shutdown_connections 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f861006b500 0x7f861006d9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f862c153820 0x7f862c153ca0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f862c06d380 0x7f862c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 --2- 172.21.15.72:0/1606312398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8618004d40 0x7f862c13fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:33.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 >> 172.21.15.72:0/1606312398 conn(0x7f862c0cf8a0 msgr2=0x7f862c15a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:33.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 shutdown_connections 2024-03-06T07:24:33.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33.066+0000 7f8633eec640 1 -- 172.21.15.72:0/1606312398 wait complete. 2024-03-06T07:24:33.078 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:24:33.078 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:24:33.078 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:24:33.078 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:24:33.079 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:33.079 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:24:33.079 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:24:33.079 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:24:33.079 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:33.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.079 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:24:33.094 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:24:33.095 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42522sec preferred_lft 42522sec 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.096 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:24:33.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.097 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:24:33.159 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:24:33.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.161 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:24:33.216 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:24:33.216 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:24:33.216 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:24:33.216 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:24:33.216 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:24:33.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.216 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:24:33.217 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:24:33.217 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:24:33.217 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:24:33.217 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:24:33.217 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:24:33.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.310 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:24:33.324 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:24:33.324 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:24:33.325 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:33.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.325 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:24:33.391 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:24:33.391 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:33.391 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:33.446 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:24:33.447 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:24:33.447 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:33.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:24:33.568 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:33.624 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:24:33.624 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:24:33.624 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:24:33.624 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:24:33.624 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:33.625 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:33.626 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:33.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.628 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:24:33.682 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:24:33.682 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:24:33.725 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:24:33.752 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:24:33.808 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:24:33.808 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:24:33.809 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:24:33.810 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:24:33.811 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:33.811 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:24:33.811 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:33.811 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:24:33.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.811 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:24:33.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:24:33.809+0000 7fbb81c3e480 -1 init, newargv = 0x55b584ac47e0 newargc=15 2024-03-06T07:24:33.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[59507]: starting ceph client 2024-03-06T07:24:33.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[59507]: starting fuse 2024-03-06T07:24:33.827 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:24:33.827 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:24:33.828 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:24:33.829 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:24:33.956 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.59507.asok 2024-03-06T07:24:33.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:33.967 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.59507.asok 2024-03-06T07:24:33.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:33.967 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.59507.asok status 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "59507", 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:34.105 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6366, 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 6366 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:34.106 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3217709788 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3217709788 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.6366 192.168.0.1:0/3217709788", 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/3217709788", 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 27, 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 59, 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:24:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:24:34.108 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:24:34.108 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:24:34.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:34.113 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:24:34.113 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:24:34.128 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:34.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:34.128 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:34.194 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:34.194 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> import uuid 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> n = 75 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = False 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> if 0 > 0: 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = True 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> dpath = os.path.dirname(path) 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> fnameprefix = os.path.basename(path) 2024-03-06T07:24:34.249 DEBUG:teuthology.orchestra.run.smithi072:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> try: 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(n): 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> f.write(f"{i}") 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> f.flush() 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(f.fileno()) 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> os.unlink(fpath) 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> if create_hard_links: 2024-03-06T07:24:34.250 DEBUG:teuthology.orchestra.run.smithi072:> for j in range(0): 2024-03-06T07:24:34.251 DEBUG:teuthology.orchestra.run.smithi072:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:24:34.251 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:24:34.251 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:24:34.251 DEBUG:teuthology.orchestra.run.smithi072:> finally: 2024-03-06T07:24:34.251 DEBUG:teuthology.orchestra.run.smithi072:> os.close(dirfd) 2024-03-06T07:24:34.251 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:24:34.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:24:34.763 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:34.764 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:34.864 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:34.905 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:34.905 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:34.905 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:34.921 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.061+0000 7fe1cfb1e640 1 Processor -- start 2024-03-06T07:24:35.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.061+0000 7fe1cfb1e640 1 -- start start 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cfb1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe1c815dfe0 0x7fe1c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cfb1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c80cf830 0x7fe1c80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cfb1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe1c80d27f0 0x7fe1c80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cfb1e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c80d2c50 con 0x7fe1c815dfe0 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cfb1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c80b31e0 con 0x7fe1c80d27f0 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cfb1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80b3d60 con 0x7fe1c80cf830 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cd8ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe1c815dfe0 0x7fe1c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cd8ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe1c815dfe0 0x7fe1c80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36440/0 (socket says 172.21.15.72:36440) 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cd8ba640 1 -- 172.21.15.72:0/302175413 learned_addr learned my addr 172.21.15.72:0/302175413 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1ce0bb640 1 --2- 172.21.15.72:0/302175413 >> v2:172.21.15.146:3300/0 conn(0x7fe1c80d27f0 0x7fe1c80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.062+0000 7fe1cd0b9640 1 --2- 172.21.15.72:0/302175413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c80cf830 0x7fe1c80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cd8ba640 1 -- 172.21.15.72:0/302175413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c80cf830 msgr2=0x7fe1c80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/302175413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c80cf830 0x7fe1c80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cd8ba640 1 -- 172.21.15.72:0/302175413 >> v2:172.21.15.146:3300/0 conn(0x7fe1c80d27f0 msgr2=0x7fe1c80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/302175413 >> v2:172.21.15.146:3300/0 conn(0x7fe1c80d27f0 0x7fe1c80d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cd8ba640 1 -- 172.21.15.72:0/302175413 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80b3430 con 0x7fe1c815dfe0 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/302175413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c815dfe0 0x7fe1c80cf2f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8067ad0 tx=0x7fe1b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cadc080d3bbc5b54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.063+0000 7fe1cc8b8640 1 -- 172.21.15.72:0/302175413 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8035020 con 0x7fe1c815dfe0 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cc8b8640 1 -- 172.21.15.72:0/302175413 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b8031870 con 0x7fe1c815dfe0 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cc8b8640 1 -- 172.21.15.72:0/302175413 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b8031c40 con 0x7fe1c815dfe0 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/302175413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c815dfe0 msgr2=0x7fe1c80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/302175413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c815dfe0 0x7fe1c80cf2f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8067ad0 tx=0x7fe1b802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/302175413 shutdown_connections 2024-03-06T07:24:35.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/302175413 >> v2:172.21.15.146:3300/0 conn(0x7fe1c80d27f0 0x7fe1c80d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/302175413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c80cf830 0x7fe1c80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/302175413 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c815dfe0 0x7fe1c80cf2f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/302175413 >> 172.21.15.72:0/302175413 conn(0x7fe1c80d3cd0 msgr2=0x7fe1c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/302175413 shutdown_connections 2024-03-06T07:24:35.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.064+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/302175413 wait complete. 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.066+0000 7fe1cfb1e640 1 Processor -- start 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.066+0000 7fe1cfb1e640 1 -- start start 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cfb1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1c80cf830 0x7fe1c81f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cfb1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 0x7fe1c8160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cfb1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c815dfe0 0x7fe1c815fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cfb1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80d1c70 con 0x7fe1c80d27f0 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cfb1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80b39f0 con 0x7fe1c80cf830 2024-03-06T07:24:35.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cfb1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c80d1df0 con 0x7fe1c815dfe0 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 0x7fe1c8160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 0x7fe1c8160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36446/0 (socket says 172.21.15.72:36446) 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 -- 172.21.15.72:0/3266871573 learned_addr learned my addr 172.21.15.72:0/3266871573 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1ce0bb640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c815dfe0 0x7fe1c815fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1c80cf830 0x7fe1c81f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 -- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c815dfe0 msgr2=0x7fe1c815fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c815dfe0 0x7fe1c815fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 -- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1c80cf830 msgr2=0x7fe1c81f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1c80cf830 0x7fe1c81f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 -- 172.21.15.72:0/3266871573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c80b9740 con 0x7fe1c80d27f0 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1ce0bb640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c815dfe0 0x7fe1c815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1c80cf830 0x7fe1c81f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.067+0000 7fe1cd0b9640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 0x7fe1c8160ad0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400d7d0 tx=0x7fe1c400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.068+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c40096a0 con 0x7fe1c80d27f0 2024-03-06T07:24:35.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.068+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4019070 con 0x7fe1c80d27f0 2024-03-06T07:24:35.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.068+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4009a60 con 0x7fe1c80d27f0 2024-03-06T07:24:35.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.068+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c80b9a20 con 0x7fe1c80d27f0 2024-03-06T07:24:35.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.068+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c80cceb0 con 0x7fe1c80d27f0 2024-03-06T07:24:35.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.069+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c80b5200 con 0x7fe1c80d27f0 2024-03-06T07:24:35.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.069+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1c4009c00 con 0x7fe1c80d27f0 2024-03-06T07:24:35.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.070+0000 7fe1be7fc640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.070+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1c408a620 con 0x7fe1c80d27f0 2024-03-06T07:24:35.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.070+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.070+0000 7fe1cd8ba640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8036000 tx=0x7fe1b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.074+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1c80b5200 con 0x7fe1c80d27f0 2024-03-06T07:24:35.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.339+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1c804f330 con 0x7fe1c80d27f0 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.340+0000 7fe1be7fc640 1 -- 172.21.15.72:0/3266871573 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fe1c40594a0 con 0x7fe1c80d27f0 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:35.342 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1ac06b4d0 msgr2=0x7fe1ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8036000 tx=0x7fe1b8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:35.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 msgr2=0x7fe1c8160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 0x7fe1c8160ad0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe1c400d7d0 tx=0x7fe1c400dc90 comp rx=0 tx=0).stop 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 shutdown_connections 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1ac06b4d0 0x7fe1ac06d990 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1c815dfe0 0x7fe1c815fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1c80d27f0 0x7fe1c8160ad0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 --2- 172.21.15.72:0/3266871573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1c80cf830 0x7fe1c81f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 >> 172.21.15.72:0/3266871573 conn(0x7fe1c80d3cd0 msgr2=0x7fe1c806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 shutdown_connections 2024-03-06T07:24:35.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.342+0000 7fe1cfb1e640 1 -- 172.21.15.72:0/3266871573 wait complete. 2024-03-06T07:24:35.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:57.998737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:58.744228+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:35.359 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:35.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.462+0000 7f6582ff6640 1 Processor -- start 2024-03-06T07:24:35.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.462+0000 7f6582ff6640 1 -- start start 2024-03-06T07:24:35.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6582ff6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f657c06fd10 0x7f657c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6582ff6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c0d0af0 0x7f657c0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6582ff6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c0cd1e0 con 0x7f657c15c050 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6582ff6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c0cd4b0 con 0x7f657c06fd10 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6582ff6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0cd790 con 0x7f657c0d0af0 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f657bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f657c06fd10 0x7f657c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6581593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c0d0af0 0x7f657c0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6580d92640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f657c15c050 0x7f657c06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:32998/0 (socket says 172.21.15.72:32998) 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6580d92640 1 -- 172.21.15.72:0/232816118 learned_addr learned my addr 172.21.15.72:0/232816118 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f657bfff640 1 -- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c0d0af0 msgr2=0x7f657c0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f657bfff640 1 --2- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c0d0af0 0x7f657c0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f657bfff640 1 -- 172.21.15.72:0/232816118 >> v1:172.21.15.146:6789/0 conn(0x7f657c15c050 legacy=0x7f657c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f657b7fe640 1 -- 172.21.15.72:0/232816118 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222073255 0 0) 0x7f657c0cd1e0 con 0x7f657c15c050 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f657bfff640 1 -- 172.21.15.72:0/232816118 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0cda70 con 0x7f657c06fd10 2024-03-06T07:24:35.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.463+0000 7f6581593640 1 --2- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c0d0af0 0x7f657c0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.464+0000 7f657bfff640 1 --2- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c06fd10 0x7f657c0cc760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f65680097a0 tx=0x7f656802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be419e639307c541 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.464+0000 7f657b7fe640 1 -- 172.21.15.72:0/232816118 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680315a0 con 0x7f657c06fd10 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.464+0000 7f657b7fe640 1 -- 172.21.15.72:0/232816118 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568037070 con 0x7f657c06fd10 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.464+0000 7f657b7fe640 1 -- 172.21.15.72:0/232816118 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568031960 con 0x7f657c06fd10 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 -- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c06fd10 msgr2=0x7f657c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 --2- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c06fd10 0x7f657c0cc760 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f65680097a0 tx=0x7f656802b840 comp rx=0 tx=0).stop 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 -- 172.21.15.72:0/232816118 shutdown_connections 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 --2- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c0d0af0 0x7f657c0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 --2- 172.21.15.72:0/232816118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c06fd10 0x7f657c0cc760 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 -- 172.21.15.72:0/232816118 >> 172.21.15.72:0/232816118 conn(0x7f657c0b3800 msgr2=0x7f657c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 -- 172.21.15.72:0/232816118 shutdown_connections 2024-03-06T07:24:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.465+0000 7f6582ff6640 1 -- 172.21.15.72:0/232816118 wait complete. 2024-03-06T07:24:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.466+0000 7f6582ff6640 1 Processor -- start 2024-03-06T07:24:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.466+0000 7f6582ff6640 1 -- start start 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6582ff6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c06fd10 0x7f657c09f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6582ff6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c0d0af0 0x7f657c15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6582ff6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 0x7f657c0b7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6582ff6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c165c30 con 0x7f657c15c050 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6582ff6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c165ab0 con 0x7f657c0d0af0 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6582ff6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c165db0 con 0x7f657c06fd10 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6581593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 0x7f657c0b7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6581593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 0x7f657c0b7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36448/0 (socket says 172.21.15.72:36448) 2024-03-06T07:24:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6581593640 1 -- 172.21.15.72:0/638948657 learned_addr learned my addr 172.21.15.72:0/638948657 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f657bfff640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c0d0af0 0x7f657c15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.467+0000 7f6580d92640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c06fd10 0x7f657c09f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6581593640 1 -- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c06fd10 msgr2=0x7f657c09f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6581593640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c06fd10 0x7f657c09f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6581593640 1 -- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c0d0af0 msgr2=0x7f657c15a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6581593640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c0d0af0 0x7f657c15a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6581593640 1 -- 172.21.15.72:0/638948657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c1596e0 con 0x7f657c15c050 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6581593640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 0x7f657c0b7020 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6574009e10 tx=0x7f657400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574018020 con 0x7f657c15c050 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574014480 con 0x7f657c15c050 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657c1599c0 con 0x7f657c15c050 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574014850 con 0x7f657c15c050 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.468+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c159f80 con 0x7f657c15c050 2024-03-06T07:24:35.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.469+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c13ee50 con 0x7f657c15c050 2024-03-06T07:24:35.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.469+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6574005d10 con 0x7f657c15c050 2024-03-06T07:24:35.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.470+0000 7f65797fa640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.470+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f657408ae40 con 0x7f657c15c050 2024-03-06T07:24:35.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.470+0000 7f6580d92640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.470+0000 7f6580d92640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f656006b4d0 0x7f656006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6570007fd0 tx=0x7f6570007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.474+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f657c13ee50 con 0x7f657c15c050 2024-03-06T07:24:35.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.652+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f657c09fe60 con 0x7f657c15c050 2024-03-06T07:24:35.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.652+0000 7f65797fa640 1 -- 172.21.15.72:0/638948657 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6574022030 con 0x7f657c15c050 2024-03-06T07:24:35.654 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:24:35.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f656006b4d0 msgr2=0x7f656006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f656006b4d0 0x7f656006d990 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6570007fd0 tx=0x7f6570007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 msgr2=0x7f657c0b7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 0x7f657c0b7020 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6574009e10 tx=0x7f657400c490 comp rx=0 tx=0).stop 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 shutdown_connections 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f656006b4d0 0x7f656006d990 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f657c15c050 0x7f657c0b7020 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f657c0d0af0 0x7f657c15a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 --2- 172.21.15.72:0/638948657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f657c06fd10 0x7f657c09f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 >> 172.21.15.72:0/638948657 conn(0x7f657c0b3800 msgr2=0x7f657c158db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 shutdown_connections 2024-03-06T07:24:35.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.654+0000 7f6582ff6640 1 -- 172.21.15.72:0/638948657 wait complete. 2024-03-06T07:24:35.667 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.769+0000 7f370b956640 1 Processor -- start 2024-03-06T07:24:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.769+0000 7f370b956640 1 -- start start 2024-03-06T07:24:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f370b956640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f370415def0 0x7f37040b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f370b956640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 0x7f37040ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f370b956640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370406d030 con 0x7f370406cab0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f370b956640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37040cf1c0 con 0x7f370415def0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f370b956640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37040cc760 con 0x7f37040cea50 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f3709ef3640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 0x7f37040ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f3709ef3640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 0x7f37040ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:34564/0 (socket says 172.21.15.72:34564) 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f37096f2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f370415def0 0x7f37040b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.770+0000 7f3709ef3640 1 -- 172.21.15.72:0/1037666252 learned_addr learned my addr 172.21.15.72:0/1037666252 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f37096f2640 1 -- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 msgr2=0x7f37040ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f37096f2640 1 --2- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 0x7f37040ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f37096f2640 1 -- 172.21.15.72:0/1037666252 >> v1:172.21.15.146:6789/0 conn(0x7f370406cab0 legacy=0x7f37040ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f37096f2640 1 -- 172.21.15.72:0/1037666252 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370406d310 con 0x7f370415def0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f3709ef3640 1 --2- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 0x7f37040ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f37096f2640 1 --2- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040b41b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f37000097a0 tx=0x7f370002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c25ccb552ae29d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.771+0000 7f36fbfff640 1 -- 172.21.15.72:0/1037666252 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700034070 con 0x7f370415def0 2024-03-06T07:24:35.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f36fbfff640 1 -- 172.21.15.72:0/1037666252 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3700030610 con 0x7f370415def0 2024-03-06T07:24:35.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f36fbfff640 1 -- 172.21.15.72:0/1037666252 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37000309e0 con 0x7f370415def0 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 -- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 msgr2=0x7f37040b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 --2- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040b41b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f37000097a0 tx=0x7f370002b960 comp rx=0 tx=0).stop 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 -- 172.21.15.72:0/1037666252 shutdown_connections 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 --2- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37040cea50 0x7f37040ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 --2- 172.21.15.72:0/1037666252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040b41b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 -- 172.21.15.72:0/1037666252 >> 172.21.15.72:0/1037666252 conn(0x7f37040b3690 msgr2=0x7f37040b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 -- 172.21.15.72:0/1037666252 shutdown_connections 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.772+0000 7f370b956640 1 -- 172.21.15.72:0/1037666252 wait complete. 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.773+0000 7f370b956640 1 Processor -- start 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.773+0000 7f370b956640 1 -- start start 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f370b956640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370406cab0 0x7f3704156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f370b956640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 0x7f37040cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f370b956640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f370b956640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370415e2d0 con 0x7f37040cea50 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f370b956640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37040cc3f0 con 0x7f370415def0 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f370b956640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370415e450 con 0x7f370406cab0 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f3708ef1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 0x7f37040cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f3708ef1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 0x7f37040cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36462/0 (socket says 172.21.15.72:36462) 2024-03-06T07:24:35.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f3708ef1640 1 -- 172.21.15.72:0/776774896 learned_addr learned my addr 172.21.15.72:0/776774896 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f37096f2640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370406cab0 0x7f3704156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.774+0000 7f3709ef3640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f37096f2640 1 -- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 msgr2=0x7f37040cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f37096f2640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f37096f2640 1 -- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 msgr2=0x7f37040cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f37096f2640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 0x7f37040cb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f37096f2640 1 -- 172.21.15.72:0/776774896 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37040c8c30 con 0x7f370406cab0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f3708ef1640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 0x7f37040cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f37096f2640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370406cab0 0x7f3704156930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3700033000 tx=0x7f37000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3700034040 con 0x7f370406cab0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370003aaa0 con 0x7f370406cab0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37040b55b0 con 0x7f370406cab0 2024-03-06T07:24:35.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37000443f0 con 0x7f370406cab0 2024-03-06T07:24:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.775+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37040c98a0 con 0x7f370406cab0 2024-03-06T07:24:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.776+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370409fcf0 con 0x7f370406cab0 2024-03-06T07:24:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.777+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f370004b060 con 0x7f370406cab0 2024-03-06T07:24:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.777+0000 7f36f9ffb640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f36e806b4d0 0x7f36e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:35.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.777+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37000abdd0 con 0x7f370406cab0 2024-03-06T07:24:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.777+0000 7f3708ef1640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f36e806b4d0 0x7f36e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:35.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.778+0000 7f3708ef1640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f36e806b4d0 0x7f36e806d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37040cc070 tx=0x7f36f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:35.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.782+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f370003f070 con 0x7f370406cab0 2024-03-06T07:24:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.963+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3704068ff0 con 0x7f370406cab0 2024-03-06T07:24:35.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.963+0000 7f36f9ffb640 1 -- 172.21.15.72:0/776774896 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f37000ad0a0 con 0x7f370406cab0 2024-03-06T07:24:35.965 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f36e806b4d0 msgr2=0x7f36e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f36e806b4d0 0x7f36e806d990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37040cc070 tx=0x7f36f4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370406cab0 msgr2=0x7f3704156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370406cab0 0x7f3704156930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3700033000 tx=0x7f37000046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 shutdown_connections 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f36e806b4d0 0x7f36e806d990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f370415def0 0x7f37040cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37040cea50 0x7f37040cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 --2- 172.21.15.72:0/776774896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f370406cab0 0x7f3704156930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:35.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.965+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 >> 172.21.15.72:0/776774896 conn(0x7f37040b3690 msgr2=0x7f37040b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:35.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.966+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 shutdown_connections 2024-03-06T07:24:35.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:35.966+0000 7f370b956640 1 -- 172.21.15.72:0/776774896 wait complete. 2024-03-06T07:24:35.981 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.120+0000 7f916aa23640 1 Processor -- start 2024-03-06T07:24:36.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.120+0000 7f916aa23640 1 -- start start 2024-03-06T07:24:36.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f916aa23640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f91640d4860 0x7f91640d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f916aa23640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91640b0e30 0x7f91640b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f916aa23640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91641618b0 con 0x7f91641614d0 2024-03-06T07:24:36.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f916aa23640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91640d4cf0 con 0x7f91640b0e30 2024-03-06T07:24:36.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f916aa23640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91640cf400 con 0x7f91640d4860 2024-03-06T07:24:36.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9168fc0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91640b0e30 0x7f91640b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9163fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f91641614d0 0x7f91640d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57700/0 (socket says 172.21.15.72:57700) 2024-03-06T07:24:36.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9163fff640 1 -- 172.21.15.72:0/70766327 learned_addr learned my addr 172.21.15.72:0/70766327 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:36.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9168fc0640 1 -- 172.21.15.72:0/70766327 >> v2:172.21.15.146:3300/0 conn(0x7f91640d4860 msgr2=0x7f91640d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f91637fe640 1 --2- 172.21.15.72:0/70766327 >> v2:172.21.15.146:3300/0 conn(0x7f91640d4860 0x7f91640d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9168fc0640 1 --2- 172.21.15.72:0/70766327 >> v2:172.21.15.146:3300/0 conn(0x7f91640d4860 0x7f91640d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9162ffd640 1 -- 172.21.15.72:0/70766327 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418230396 0 0) 0x7f91641618b0 con 0x7f91641614d0 2024-03-06T07:24:36.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9168fc0640 1 -- 172.21.15.72:0/70766327 >> v1:172.21.15.72:6789/0 conn(0x7f91641614d0 legacy=0x7f91640d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f9168fc0640 1 -- 172.21.15.72:0/70766327 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640cf6a0 con 0x7f91640b0e30 2024-03-06T07:24:36.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.121+0000 7f91637fe640 1 --2- 172.21.15.72:0/70766327 >> v2:172.21.15.146:3300/0 conn(0x7f91640d4860 0x7f91640d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:36.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.122+0000 7f9168fc0640 1 --2- 172.21.15.72:0/70766327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91640b0e30 0x7f91640b1260 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f91640ce550 tx=0x7f915c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7654d3719f1c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.122+0000 7f9162ffd640 1 -- 172.21.15.72:0/70766327 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c039070 con 0x7f91640b0e30 2024-03-06T07:24:36.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.122+0000 7f9162ffd640 1 -- 172.21.15.72:0/70766327 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c02cc80 con 0x7f91640b0e30 2024-03-06T07:24:36.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.122+0000 7f9162ffd640 1 -- 172.21.15.72:0/70766327 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c034660 con 0x7f91640b0e30 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 -- 172.21.15.72:0/70766327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91640b0e30 msgr2=0x7f91640b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 --2- 172.21.15.72:0/70766327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91640b0e30 0x7f91640b1260 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f91640ce550 tx=0x7f915c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 -- 172.21.15.72:0/70766327 shutdown_connections 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 --2- 172.21.15.72:0/70766327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91640b0e30 0x7f91640b1260 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 --2- 172.21.15.72:0/70766327 >> v2:172.21.15.146:3300/0 conn(0x7f91640d4860 0x7f91640d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 -- 172.21.15.72:0/70766327 >> 172.21.15.72:0/70766327 conn(0x7f91640d3d40 msgr2=0x7f91640d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 -- 172.21.15.72:0/70766327 shutdown_connections 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.123+0000 7f916aa23640 1 -- 172.21.15.72:0/70766327 wait complete. 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.124+0000 7f916aa23640 1 Processor -- start 2024-03-06T07:24:36.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.124+0000 7f916aa23640 1 -- start start 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f916aa23640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91640b0e30 0x7f91640ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f916aa23640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91640d4860 0x7f916414cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f916aa23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 0x7f916415da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f916aa23640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640b0910 con 0x7f91641614d0 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f916aa23640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91640cfc60 con 0x7f91640d4860 2024-03-06T07:24:36.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f916aa23640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91640d2560 con 0x7f91640b0e30 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f9168fc0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 0x7f916415da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f9168fc0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 0x7f916415da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36486/0 (socket says 172.21.15.72:36486) 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f9168fc0640 1 -- 172.21.15.72:0/3345325896 learned_addr learned my addr 172.21.15.72:0/3345325896 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f91637fe640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91640d4860 0x7f916414cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.125+0000 7f9163fff640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91640b0e30 0x7f91640ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9168fc0640 1 -- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91640b0e30 msgr2=0x7f91640ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9168fc0640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91640b0e30 0x7f91640ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9168fc0640 1 -- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91640d4860 msgr2=0x7f916414cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9168fc0640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91640d4860 0x7f916414cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9168fc0640 1 -- 172.21.15.72:0/3345325896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640cd6e0 con 0x7f91641614d0 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f91637fe640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91640d4860 0x7f916414cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9163fff640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91640b0e30 0x7f91640ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9168fc0640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 0x7f916415da50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f915c037040 tx=0x7f915c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c044020 con 0x7f91641614d0 2024-03-06T07:24:36.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c03a040 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c039050 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91640cd9c0 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.126+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916414d1d0 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.127+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91640d0f70 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.127+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f915c003b90 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.128+0000 7f9160ff9640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f914406b500 0x7f914406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.128+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f915c0acaa0 con 0x7f91641614d0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.128+0000 7f9163fff640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f914406b500 0x7f914406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.128+0000 7f9163fff640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f914406b500 0x7f914406d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f914c000f60 tx=0x7f914c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.136+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f915c04e020 con 0x7f91641614d0 2024-03-06T07:24:36.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.312+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9164068ff0 con 0x7f91641614d0 2024-03-06T07:24:36.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.313+0000 7f9160ff9640 1 -- 172.21.15.72:0/3345325896 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f915c07b9b0 con 0x7f91641614d0 2024-03-06T07:24:36.315 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:24:36.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f914406b500 msgr2=0x7f914406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f914406b500 0x7f914406d9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f914c000f60 tx=0x7f914c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 msgr2=0x7f916415da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 0x7f916415da50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f915c037040 tx=0x7f915c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 shutdown_connections 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f914406b500 0x7f914406d9c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91641614d0 0x7f916415da50 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91640d4860 0x7f916414cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 --2- 172.21.15.72:0/3345325896 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91640b0e30 0x7f91640ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 >> 172.21.15.72:0/3345325896 conn(0x7f91640d3d40 msgr2=0x7f91640d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 shutdown_connections 2024-03-06T07:24:36.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.315+0000 7f916aa23640 1 -- 172.21.15.72:0/3345325896 wait complete. 2024-03-06T07:24:36.328 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:24:36.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.427+0000 7fcba2015640 1 Processor -- start 2024-03-06T07:24:36.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.427+0000 7fcba2015640 1 -- start start 2024-03-06T07:24:36.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcba2015640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb9c0b4c00 0x7fcb9c0b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcba2015640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb9c15ba50 0x7fcb9c15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcba2015640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c15d510 con 0x7fcb9c0b28a0 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcba2015640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c0b3420 con 0x7fcb9c0b4c00 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcba2015640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c15e0b0 con 0x7fcb9c15ba50 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcb9b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb9c0b4c00 0x7fcb9c0b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcb9b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb9c0b4c00 0x7fcb9c0b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36490/0 (socket says 172.21.15.72:36490) 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.428+0000 7fcb9b7fe640 1 -- 172.21.15.72:0/964492671 learned_addr learned my addr 172.21.15.72:0/964492671 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.429+0000 7fcb9b7fe640 1 -- 172.21.15.72:0/964492671 >> v2:172.21.15.146:3300/0 conn(0x7fcb9c15ba50 msgr2=0x7fcb9c15bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.429+0000 7fcb9b7fe640 1 --2- 172.21.15.72:0/964492671 >> v2:172.21.15.146:3300/0 conn(0x7fcb9c15ba50 0x7fcb9c15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.429+0000 7fcb9b7fe640 1 -- 172.21.15.72:0/964492671 >> v1:172.21.15.146:6789/0 conn(0x7fcb9c0b28a0 legacy=0x7fcb9c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.429+0000 7fcb9b7fe640 1 -- 172.21.15.72:0/964492671 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c15d780 con 0x7fcb9c0b4c00 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.429+0000 7fcb9b7fe640 1 --2- 172.21.15.72:0/964492671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c0b4c00 0x7fcb9c0b4fe0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c06b930 tx=0x7fcb8c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e0ac89a93fecd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.429+0000 7fcb9a7fc640 1 -- 172.21.15.72:0/964492671 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c034070 con 0x7fcb9c0b4c00 2024-03-06T07:24:36.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcb9a7fc640 1 -- 172.21.15.72:0/964492671 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c030860 con 0x7fcb9c0b4c00 2024-03-06T07:24:36.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcb9a7fc640 1 -- 172.21.15.72:0/964492671 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c030c30 con 0x7fcb9c0b4c00 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 -- 172.21.15.72:0/964492671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c0b4c00 msgr2=0x7fcb9c0b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 --2- 172.21.15.72:0/964492671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c0b4c00 0x7fcb9c0b4fe0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c06b930 tx=0x7fcb8c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 -- 172.21.15.72:0/964492671 shutdown_connections 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 --2- 172.21.15.72:0/964492671 >> v2:172.21.15.146:3300/0 conn(0x7fcb9c15ba50 0x7fcb9c15bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 --2- 172.21.15.72:0/964492671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c0b4c00 0x7fcb9c0b4fe0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 -- 172.21.15.72:0/964492671 >> 172.21.15.72:0/964492671 conn(0x7fcb9c09a9f0 msgr2=0x7fcb9c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 -- 172.21.15.72:0/964492671 shutdown_connections 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.430+0000 7fcba2015640 1 -- 172.21.15.72:0/964492671 wait complete. 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.431+0000 7fcba2015640 1 Processor -- start 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.431+0000 7fcba2015640 1 -- start start 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcba2015640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb9c0b28a0 0x7fcb9c143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcba2015640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb9c0b4c00 0x7fcb9c1521d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcba2015640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 0x7fcb9c152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcba2015640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c15f550 con 0x7fcb9c15ba50 2024-03-06T07:24:36.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcba2015640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c15dd40 con 0x7fcb9c0b28a0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcba2015640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c15f6d0 con 0x7fcb9c0b4c00 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 0x7fcb9c152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 0x7fcb9c152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36502/0 (socket says 172.21.15.72:36502) 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 -- 172.21.15.72:0/3925954125 learned_addr learned my addr 172.21.15.72:0/3925954125 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9b7fe640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb9c0b28a0 0x7fcb9c143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9affd640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb9c0b4c00 0x7fcb9c1521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 -- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb9c0b4c00 msgr2=0x7fcb9c1521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb9c0b4c00 0x7fcb9c1521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 -- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb9c0b28a0 msgr2=0x7fcb9c143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb9c0b28a0 0x7fcb9c143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.432+0000 7fcb9bfff640 1 -- 172.21.15.72:0/3925954125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c13fa20 con 0x7fcb9c15ba50 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcb9affd640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb9c0b4c00 0x7fcb9c1521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcb9bfff640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 0x7fcb9c152710 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400b970 tx=0x7fcb9400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94019070 con 0x7fcb9c15ba50 2024-03-06T07:24:36.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9400cb60 con 0x7fcb9c15ba50 2024-03-06T07:24:36.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c153f40 con 0x7fcb9c15ba50 2024-03-06T07:24:36.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94014440 con 0x7fcb9c15ba50 2024-03-06T07:24:36.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.433+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c154500 con 0x7fcb9c15ba50 2024-03-06T07:24:36.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.434+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c09b520 con 0x7fcb9c15ba50 2024-03-06T07:24:36.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.434+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb9401c070 con 0x7fcb9c15ba50 2024-03-06T07:24:36.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.435+0000 7fcb93fff640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:36.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.435+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb94089fb0 con 0x7fcb9c15ba50 2024-03-06T07:24:36.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.435+0000 7fcb9b7fe640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:36.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.435+0000 7fcb9b7fe640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c036000 tx=0x7fcb8c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:36.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.441+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb9c09b520 con 0x7fcb9c15ba50 2024-03-06T07:24:36.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.680+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcb9c04f330 con 0x7fcb9c15ba50 2024-03-06T07:24:36.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.680+0000 7fcb93fff640 1 -- 172.21.15.72:0/3925954125 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcb94058db0 con 0x7fcb9c15ba50 2024-03-06T07:24:36.682 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:24:36.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb7c06b4d0 msgr2=0x7fcb7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c036000 tx=0x7fcb8c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 msgr2=0x7fcb9c152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 0x7fcb9c152710 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400b970 tx=0x7fcb9400be30 comp rx=0 tx=0).stop 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 shutdown_connections 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb7c06b4d0 0x7fcb7c06d990 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb9c15ba50 0x7fcb9c152710 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb9c0b4c00 0x7fcb9c1521d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 --2- 172.21.15.72:0/3925954125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb9c0b28a0 0x7fcb9c143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 >> 172.21.15.72:0/3925954125 conn(0x7fcb9c09a9f0 msgr2=0x7fcb9c15bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 shutdown_connections 2024-03-06T07:24:36.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:36.682+0000 7fcba2015640 1 -- 172.21.15.72:0/3925954125 wait complete. 2024-03-06T07:24:36.695 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:24:36.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:36.696 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:24:36.720 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[59507]: fuse finished with error 0 and tester_r 0 2024-03-06T07:24:36.722 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:24:36.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:36.722 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:24:36.807 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:24:36.807 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:36.826 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:36.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:24:36.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:36.826 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:24:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:36.883 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:24:36.883 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:36.883 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:36.898 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:36.899 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:24:36.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:36.899 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:24:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:36.953 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:24:36.954 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:24:36.954 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:36.970 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:36.970 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:24:36.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:24:36.971 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:24:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:24:37.027 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:24:37.027 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:37.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.152+0000 7fde72efb640 1 Processor -- start 2024-03-06T07:24:37.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.152+0000 7fde72efb640 1 -- start start 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde72efb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde6c0d22b0 0x7fde6c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde72efb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0b3430 con 0x7fde6c15dfe0 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde72efb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0b3710 con 0x7fde6c0b41f0 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde72efb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6c0d2c50 con 0x7fde6c0d22b0 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde6bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde6c0d22b0 0x7fde6c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde70c97640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fde6c15dfe0 0x7fde6c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57708/0 (socket says 172.21.15.72:57708) 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.153+0000 7fde70c97640 1 -- 172.21.15.72:0/2732579133 learned_addr learned my addr 172.21.15.72:0/2732579133 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:37.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.154+0000 7fde6bfff640 1 -- 172.21.15.72:0/2732579133 >> v1:172.21.15.146:6789/0 conn(0x7fde6c0b41f0 legacy=0x7fde6c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.154+0000 7fde6bfff640 1 -- 172.21.15.72:0/2732579133 >> v1:172.21.15.72:6789/0 conn(0x7fde6c15dfe0 legacy=0x7fde6c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.154+0000 7fde6bfff640 1 -- 172.21.15.72:0/2732579133 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0b39f0 con 0x7fde6c0d22b0 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.154+0000 7fde6b7fe640 1 -- 172.21.15.72:0/2732579133 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899031175 0 0) 0x7fde6c0b3430 con 0x7fde6c15dfe0 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.154+0000 7fde6bfff640 1 --2- 172.21.15.72:0/2732579133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c0d22b0 0x7fde6c0d2710 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fde580097d0 tx=0x7fde5802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d094b0a361622034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde6b7fe640 1 -- 172.21.15.72:0/2732579133 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde58034070 con 0x7fde6c0d22b0 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde6b7fe640 1 -- 172.21.15.72:0/2732579133 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58030610 con 0x7fde6c0d22b0 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde6b7fe640 1 -- 172.21.15.72:0/2732579133 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde580309e0 con 0x7fde6c0d22b0 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 -- 172.21.15.72:0/2732579133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c0d22b0 msgr2=0x7fde6c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 --2- 172.21.15.72:0/2732579133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c0d22b0 0x7fde6c0d2710 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fde580097d0 tx=0x7fde5802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 -- 172.21.15.72:0/2732579133 shutdown_connections 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 --2- 172.21.15.72:0/2732579133 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c0d22b0 0x7fde6c0d2710 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 -- 172.21.15.72:0/2732579133 >> 172.21.15.72:0/2732579133 conn(0x7fde6c0d3cd0 msgr2=0x7fde6c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 -- 172.21.15.72:0/2732579133 shutdown_connections 2024-03-06T07:24:37.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.155+0000 7fde72efb640 1 -- 172.21.15.72:0/2732579133 wait complete. 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.156+0000 7fde72efb640 1 Processor -- start 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.156+0000 7fde72efb640 1 -- start start 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde6c0b41f0 0x7fde6c160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde6c0d22b0 0x7fde6c0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 0x7fde6c15fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b3d60 con 0x7fde6c15dfe0 2024-03-06T07:24:37.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0b31e0 con 0x7fde6c0b41f0 2024-03-06T07:24:37.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6c0d1c70 con 0x7fde6c0d22b0 2024-03-06T07:24:37.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 0x7fde6c15fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 0x7fde6c15fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36514/0 (socket says 172.21.15.72:36514) 2024-03-06T07:24:37.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 -- 172.21.15.72:0/4181308690 learned_addr learned my addr 172.21.15.72:0/4181308690 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:37.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde6bfff640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde6c0d22b0 0x7fde6c0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde70c97640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde6c0b41f0 0x7fde6c160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 -- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde6c0d22b0 msgr2=0x7fde6c0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde6c0d22b0 0x7fde6c0b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 -- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde6c0b41f0 msgr2=0x7fde6c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde6c0b41f0 0x7fde6c160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 -- 172.21.15.72:0/4181308690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c15e910 con 0x7fde6c15dfe0 2024-03-06T07:24:37.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde6bfff640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde6c0d22b0 0x7fde6c0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde70c97640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde6c0b41f0 0x7fde6c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde71498640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 0x7fde6c15fdb0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fde6400c7a0 tx=0x7fde6400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde64003b70 con 0x7fde6c15dfe0 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde64003d30 con 0x7fde6c15dfe0 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c15ebf0 con 0x7fde6c15dfe0 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.158+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6400aa90 con 0x7fde6c15dfe0 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.159+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c15f1b0 con 0x7fde6c15dfe0 2024-03-06T07:24:37.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.160+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c04f330 con 0x7fde6c15dfe0 2024-03-06T07:24:37.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.160+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde6400ac30 con 0x7fde6c15dfe0 2024-03-06T07:24:37.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.161+0000 7fde697fa640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde5006b4d0 0x7fde5006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.161+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde6408a150 con 0x7fde6c15dfe0 2024-03-06T07:24:37.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.161+0000 7fde70c97640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde5006b4d0 0x7fde5006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.161+0000 7fde70c97640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde5006b4d0 0x7fde5006d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fde60007fd0 tx=0x7fde60007dc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.165+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde64058fd0 con 0x7fde6c15dfe0 2024-03-06T07:24:37.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.475+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde6c0cea90 con 0x7fde6c15dfe0 2024-03-06T07:24:37.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.476+0000 7fde697fa640 1 -- 172.21.15.72:0/4181308690 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fde640588d0 con 0x7fde6c15dfe0 2024-03-06T07:24:37.478 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:37.478 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:37.478 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:37.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde5006b4d0 msgr2=0x7fde5006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde5006b4d0 0x7fde5006d990 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fde60007fd0 tx=0x7fde60007dc0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 msgr2=0x7fde6c15fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 0x7fde6c15fdb0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fde6400c7a0 tx=0x7fde6400cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:37.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 shutdown_connections 2024-03-06T07:24:37.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde5006b4d0 0x7fde5006d990 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde6c15dfe0 0x7fde6c15fdb0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde6c0d22b0 0x7fde6c0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 --2- 172.21.15.72:0/4181308690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde6c0b41f0 0x7fde6c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 >> 172.21.15.72:0/4181308690 conn(0x7fde6c0d3cd0 msgr2=0x7fde6c15d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 shutdown_connections 2024-03-06T07:24:37.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.478+0000 7fde72efb640 1 -- 172.21.15.72:0/4181308690 wait complete. 2024-03-06T07:24:37.491 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:37.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.593+0000 7f93867eb640 1 Processor -- start 2024-03-06T07:24:37.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.593+0000 7f93867eb640 1 -- start start 2024-03-06T07:24:37.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f93867eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 0x7f93800cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f93867eb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93800b3430 con 0x7f93800d29c0 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f93867eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93800b3710 con 0x7f93800cf830 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f93867eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93800b31e0 con 0x7f938015dfe0 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f9384d88640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f93800d29c0 0x7f93800b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57712/0 (socket says 172.21.15.72:57712) 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f9384d88640 1 -- 172.21.15.72:0/528862566 learned_addr learned my addr 172.21.15.72:0/528862566 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937ffff640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 0x7f93800cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932670933 0 0) 0x7f93800b3430 con 0x7f93800d29c0 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9368003520 con 0x7f93800d29c0 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074447670 0 0) 0x7f93800b3710 con 0x7f93800cf830 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93800b3430 con 0x7f93800cf830 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696310227 0 0) 0x7f9368003520 con 0x7f93800d29c0 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 msgr2=0x7f93800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 0x7f93800cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 >> v1:172.21.15.146:6789/0 conn(0x7f93800cf830 legacy=0x7f93800d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93800b39f0 con 0x7f93800d29c0 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f93780019f0 con 0x7f93800d29c0 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 >> v1:172.21.15.72:6789/0 conn(0x7f93800d29c0 legacy=0x7f93800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f9368003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 0x7f9368004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368004d40 0x7f93680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93800b31e0 con 0x7f9368003b70 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936800ba70 con 0x7f9368004d40 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936800bc70 con 0x7f938015dfe0 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.594+0000 7f937ffff640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f9368003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937f7fe640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 0x7f9368004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f9384d88640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368004d40 0x7f93680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937ffff640 1 -- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 msgr2=0x7f9368004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937ffff640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f938015dfe0 0x7f9368004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937ffff640 1 -- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368004d40 msgr2=0x7f93680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937ffff640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368004d40 0x7f93680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937ffff640 1 -- 172.21.15.72:0/528862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9368003520 con 0x7f9368003b70 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f9384d88640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9368004d40 0x7f93680051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.595+0000 7f937ffff640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f9368003f70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9370009ac0 tx=0x7f9368005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8fe34276e526e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370039070 con 0x7f9368003b70 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937002c6b0 con 0x7f9368003b70 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f937effd640 1 -- 172.21.15.72:0/528862566 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370009690 con 0x7f9368003b70 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f93867eb640 1 -- 172.21.15.72:0/528862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 msgr2=0x7f9368003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f93867eb640 1 --2- 172.21.15.72:0/528862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f9368003f70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9370009ac0 tx=0x7f9368005830 comp rx=0 tx=0).stop 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f9384d88640 1 -- 172.21.15.72:0/528862566 reap_dead start 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f93867eb640 1 -- 172.21.15.72:0/528862566 shutdown_connections 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f93867eb640 1 -- 172.21.15.72:0/528862566 >> 172.21.15.72:0/528862566 conn(0x7f93800d3cd0 msgr2=0x7f93800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f93867eb640 1 -- 172.21.15.72:0/528862566 shutdown_connections 2024-03-06T07:24:37.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.596+0000 7f93867eb640 1 -- 172.21.15.72:0/528862566 wait complete. 2024-03-06T07:24:37.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.598+0000 7f93867eb640 1 Processor -- start 2024-03-06T07:24:37.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.598+0000 7f93867eb640 1 -- start start 2024-03-06T07:24:37.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.599+0000 7f93867eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f938015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.599+0000 7f93867eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93800b90e0 0x7f93800b9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.599+0000 7f93867eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9380151180 0x7f9380151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.599+0000 7f93867eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f938015e3c0 con 0x7f9368003b70 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.599+0000 7f93867eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93800b3d60 con 0x7f93800b90e0 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.599+0000 7f93867eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938015e540 con 0x7f9380151180 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f938015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f938015c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36546/0 (socket says 172.21.15.72:36546) 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 -- 172.21.15.72:0/622952305 learned_addr learned my addr 172.21.15.72:0/622952305 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937f7fe640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93800b90e0 0x7f93800b9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f9384d88640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9380151180 0x7f9380151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 -- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9380151180 msgr2=0x7f9380151600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9380151180 0x7f9380151600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 -- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93800b90e0 msgr2=0x7f93800b9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93800b90e0 0x7f93800b9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 -- 172.21.15.72:0/622952305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938015e910 con 0x7f9368003b70 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937f7fe640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93800b90e0 0x7f93800b9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:37.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937ffff640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f938015c030 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9370000c00 tx=0x7f9370004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370042020 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937003a040 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f938015ebb0 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9370039070 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.600+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9380152420 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.602+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f938004f330 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.602+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9370045020 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.602+0000 7f937cff9640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f936c06b500 0x7f936c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.602+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93700ab570 con 0x7f9368003b70 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.602+0000 7f937f7fe640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f936c06b500 0x7f936c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.603+0000 7f937f7fe640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f936c06b500 0x7f936c06d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f93800b6af0 tx=0x7f9374009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.606+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f937004c070 con 0x7f9368003b70 2024-03-06T07:24:37.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.812+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93800ceac0 con 0x7f9368003b70 2024-03-06T07:24:37.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.813+0000 7f937cff9640 1 -- 172.21.15.72:0/622952305 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7f937007a480 con 0x7f9368003b70 2024-03-06T07:24:37.816 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:37.816 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:37.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f936c06b500 msgr2=0x7f936c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f936c06b500 0x7f936c06d9c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f93800b6af0 tx=0x7f9374009040 comp rx=0 tx=0).stop 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 msgr2=0x7f938015c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f938015c030 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9370000c00 tx=0x7f9370004730 comp rx=0 tx=0).stop 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 shutdown_connections 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f936c06b500 0x7f936c06d9c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9380151180 0x7f9380151600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93800b90e0 0x7f93800b9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 --2- 172.21.15.72:0/622952305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9368003b70 0x7f938015c030 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 >> 172.21.15.72:0/622952305 conn(0x7f93800d3cd0 msgr2=0x7f93800cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 shutdown_connections 2024-03-06T07:24:37.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.815+0000 7f93867eb640 1 -- 172.21.15.72:0/622952305 wait complete. 2024-03-06T07:24:37.828 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:23:57.998737+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:23:58.744228+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:37.829 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:37.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.930+0000 7f6b641b6640 1 Processor -- start 2024-03-06T07:24:37.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.931+0000 7f6b641b6640 1 -- start start 2024-03-06T07:24:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.931+0000 7f6b641b6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6b5c0d14a0 0x7f6b5c15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.931+0000 7f6b641b6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0cfa50 con 0x7f6b5c15eb00 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.931+0000 7f6b641b6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0b45c0 con 0x7f6b5c0d10c0 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.931+0000 7f6b641b6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c15c050 con 0x7f6b5c0d14a0 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b62753640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6b5c15eb00 0x7f6b5c0cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57726/0 (socket says 172.21.15.72:57726) 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b62753640 1 -- 172.21.15.72:0/1218680402 learned_addr learned my addr 172.21.15.72:0/1218680402 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b61751640 1 --2- 172.21.15.72:0/1218680402 >> v2:172.21.15.146:3300/0 conn(0x7f6b5c0d14a0 0x7f6b5c15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857160493 0 0) 0x7f6b5c0cfa50 con 0x7f6b5c15eb00 2024-03-06T07:24:37.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b48003520 con 0x7f6b5c15eb00 2024-03-06T07:24:37.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2084009137 0 0) 0x7f6b48003520 con 0x7f6b5c15eb00 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 >> v2:172.21.15.146:3300/0 conn(0x7f6b5c0d14a0 msgr2=0x7f6b5c15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 --2- 172.21.15.72:0/1218680402 >> v2:172.21.15.146:3300/0 conn(0x7f6b5c0d14a0 0x7f6b5c15e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 >> v1:172.21.15.146:6789/0 conn(0x7f6b5c0d10c0 legacy=0x7f6b5c0d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c15f160 con 0x7f6b5c15eb00 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6b540019f0 con 0x7f6b5c15eb00 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 >> v1:172.21.15.72:6789/0 conn(0x7f6b5c15eb00 legacy=0x7f6b5c0cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b480039b0 0x7f6b48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b48004800 0x7f6b48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b480051c0 0x7f6b4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4800c030 con 0x7f6b48004800 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4800c1e0 con 0x7f6b480051c0 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4800c3e0 con 0x7f6b480039b0 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b61751640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b48004800 0x7f6b48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b61f52640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b480039b0 0x7f6b48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.932+0000 7f6b62753640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b480051c0 0x7f6b4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b61751640 1 -- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b480039b0 msgr2=0x7f6b48003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b61751640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b480039b0 0x7f6b48003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b61751640 1 -- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b480051c0 msgr2=0x7f6b4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b61751640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b480051c0 0x7f6b4800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b61751640 1 -- 172.21.15.72:0/1218680402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b48003520 con 0x7f6b48004800 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b62753640 1 -- 172.21.15.72:0/1218680402 reap_dead start 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b61751640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b48004800 0x7f6b48004c80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009ac0 tx=0x7f6b48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe73ce98d03bc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c039070 con 0x7f6b48004800 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c02c6b0 con 0x7f6b48004800 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.933+0000 7f6b60f50640 1 -- 172.21.15.72:0/1218680402 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c009690 con 0x7f6b48004800 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 -- 172.21.15.72:0/1218680402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b48004800 msgr2=0x7f6b48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b48004800 0x7f6b48004c80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c009ac0 tx=0x7f6b48005830 comp rx=0 tx=0).stop 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 -- 172.21.15.72:0/1218680402 shutdown_connections 2024-03-06T07:24:37.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 --2- 172.21.15.72:0/1218680402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b48004800 0x7f6b48004c80 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 -- 172.21.15.72:0/1218680402 >> 172.21.15.72:0/1218680402 conn(0x7f6b5c0b3800 msgr2=0x7f6b5c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:37.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 -- 172.21.15.72:0/1218680402 shutdown_connections 2024-03-06T07:24:37.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.934+0000 7f6b641b6640 1 -- 172.21.15.72:0/1218680402 wait complete. 2024-03-06T07:24:37.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.935+0000 7f6b641b6640 1 Processor -- start 2024-03-06T07:24:37.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.935+0000 7f6b641b6640 1 -- start start 2024-03-06T07:24:37.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b641b6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b48004800 0x7f6b5c1e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b641b6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b5c14a250 0x7f6b5c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b641b6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 0x7f6b5c14d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b641b6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c0cfcf0 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b641b6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c0d0060 con 0x7f6b5c14a250 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b641b6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c15e5c0 con 0x7f6b48004800 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 0x7f6b5c14d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 0x7f6b5c14d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36562/0 (socket says 172.21.15.72:36562) 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 -- 172.21.15.72:0/633018003 learned_addr learned my addr 172.21.15.72:0/633018003 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b61f52640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b48004800 0x7f6b5c1e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b61751640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b5c14a250 0x7f6b5c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 -- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b48004800 msgr2=0x7f6b5c1e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b48004800 0x7f6b5c1e6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 -- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b5c14a250 msgr2=0x7f6b5c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b5c14a250 0x7f6b5c14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 -- 172.21.15.72:0/633018003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c14b040 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.936+0000 7f6b62753640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 0x7f6b5c14d620 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6b54000f80 tx=0x7f6b54007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.937+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5400ab00 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.937+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b54019070 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.937+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c1e6ea0 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.937+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b540133f0 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.937+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c14b5c0 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.938+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c0703a0 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.939+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6b54013590 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.939+0000 7f6b527fc640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6b4006b500 0x7f6b4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.939+0000 7f6b61f52640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6b4006b500 0x7f6b4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.939+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6b5408a6f0 con 0x7f6b5c14d1a0 2024-03-06T07:24:37.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.939+0000 7f6b61f52640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6b4006b500 0x7f6b4006d9c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6b58000c00 tx=0x7f6b58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:37.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:37.943+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6b5c0703a0 con 0x7f6b5c14d1a0 2024-03-06T07:24:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.114+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b5c09fe60 con 0x7f6b5c14d1a0 2024-03-06T07:24:38.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.114+0000 7f6b527fc640 1 -- 172.21.15.72:0/633018003 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b54059570 con 0x7f6b5c14d1a0 2024-03-06T07:24:38.116 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:24:38.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6b4006b500 msgr2=0x7f6b4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6b4006b500 0x7f6b4006d9c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6b58000c00 tx=0x7f6b58009040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 msgr2=0x7f6b5c14d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 0x7f6b5c14d620 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6b54000f80 tx=0x7f6b54007bd0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 shutdown_connections 2024-03-06T07:24:38.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6b4006b500 0x7f6b4006d9c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6b5c14d1a0 0x7f6b5c14d620 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b5c14a250 0x7f6b5c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 --2- 172.21.15.72:0/633018003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b48004800 0x7f6b5c1e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 >> 172.21.15.72:0/633018003 conn(0x7f6b5c0b3800 msgr2=0x7f6b5c160080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 shutdown_connections 2024-03-06T07:24:38.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.116+0000 7f6b641b6640 1 -- 172.21.15.72:0/633018003 wait complete. 2024-03-06T07:24:38.129 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:24:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 Processor -- start 2024-03-06T07:24:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 -- start start 2024-03-06T07:24:38.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff4b40d4860 0x7ff4b40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b40b0e30 0x7ff4b40b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b41618b0 con 0x7ff4b41614d0 2024-03-06T07:24:38.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4b40d4cf0 con 0x7ff4b40d4860 2024-03-06T07:24:38.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.232+0000 7ff4b9ec9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40cf400 con 0x7ff4b40b0e30 2024-03-06T07:24:38.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff4b40d4860 0x7ff4b40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff4b41614d0 0x7ff4b40d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57728/0 (socket says 172.21.15.72:57728) 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b37fe640 1 -- 172.21.15.72:0/2964533723 learned_addr learned my addr 172.21.15.72:0/2964533723 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/2964533723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b40b0e30 msgr2=0x7ff4b40b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b3fff640 1 --2- 172.21.15.72:0/2964533723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b40b0e30 0x7ff4b40b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b2ffd640 1 --2- 172.21.15.72:0/2964533723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b40b0e30 0x7ff4b40b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b27fc640 1 -- 172.21.15.72:0/2964533723 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039236382 0 0) 0x7ff4b41618b0 con 0x7ff4b41614d0 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/2964533723 >> v1:172.21.15.72:6789/0 conn(0x7ff4b41614d0 legacy=0x7ff4b40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.233+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/2964533723 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40cf6a0 con 0x7ff4b40d4860 2024-03-06T07:24:38.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.234+0000 7ff4b2ffd640 1 --2- 172.21.15.72:0/2964533723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b40d2020 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80097a0 tx=0x7ff4a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d89b46aa1225ee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.234+0000 7ff4b27fc640 1 -- 172.21.15.72:0/2964533723 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8035020 con 0x7ff4b40d4860 2024-03-06T07:24:38.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.234+0000 7ff4b27fc640 1 -- 172.21.15.72:0/2964533723 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a8031830 con 0x7ff4b40d4860 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.234+0000 7ff4b27fc640 1 -- 172.21.15.72:0/2964533723 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8031c00 con 0x7ff4b40d4860 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/2964533723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 msgr2=0x7ff4b40d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/2964533723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b40d2020 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80097a0 tx=0x7ff4a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/2964533723 shutdown_connections 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/2964533723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b40b0e30 0x7ff4b40b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/2964533723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b40d2020 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/2964533723 >> 172.21.15.72:0/2964533723 conn(0x7ff4b40d3d40 msgr2=0x7ff4b40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/2964533723 shutdown_connections 2024-03-06T07:24:38.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.235+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/2964533723 wait complete. 2024-03-06T07:24:38.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.236+0000 7ff4b9ec9640 1 Processor -- start 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.236+0000 7ff4b9ec9640 1 -- start start 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b9ec9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4b40b0e30 0x7ff4b40ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b9ec9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b4155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b9ec9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b41614d0 0x7ff4b415dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b9ec9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40b0910 con 0x7ff4b40d4860 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b9ec9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40cfc60 con 0x7ff4b40b0e30 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b9ec9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b40d2560 con 0x7ff4b41614d0 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b4155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b4155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36580/0 (socket says 172.21.15.72:36580) 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/3417455126 learned_addr learned my addr 172.21.15.72:0/3417455126 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b37fe640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4b40b0e30 0x7ff4b40ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b3fff640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b41614d0 0x7ff4b415dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b41614d0 msgr2=0x7ff4b415dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b41614d0 0x7ff4b415dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4b40b0e30 msgr2=0x7ff4b40ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4b40b0e30 0x7ff4b40ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 -- 172.21.15.72:0/3417455126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b40b8730 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b37fe640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4b40b0e30 0x7ff4b40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff4b2ffd640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b4155db0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff4a802bee0 tx=0x7ff4a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8041070 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.237+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4a803a950 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.238+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b40b8a10 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.238+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a8035040 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.238+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b40b8fd0 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.239+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4b40670d0 con 0x7ff4b40d4860 2024-03-06T07:24:38.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.239+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4a804c060 con 0x7ff4b40d4860 2024-03-06T07:24:38.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.239+0000 7ff49bfff640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.239+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4a80abc60 con 0x7ff4b40d4860 2024-03-06T07:24:38.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.239+0000 7ff4b37fe640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.240+0000 7ff4b37fe640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff4a006b4d0 0x7ff4a006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4000f60 tx=0x7ff4a4007c10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.244+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff4b40670d0 con 0x7ff4b40d4860 2024-03-06T07:24:38.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.420+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4b4068ff0 con 0x7ff4b40d4860 2024-03-06T07:24:38.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.421+0000 7ff49bfff640 1 -- 172.21.15.72:0/3417455126 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4a807aae0 con 0x7ff4b40d4860 2024-03-06T07:24:38.423 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.422+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff4a006b4d0 msgr2=0x7ff4a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.422+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff4a006b4d0 0x7ff4a006d990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4000f60 tx=0x7ff4a4007c10 comp rx=0 tx=0).stop 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 msgr2=0x7ff4b4155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b4155db0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7ff4a802bee0 tx=0x7ff4a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 shutdown_connections 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff4a006b4d0 0x7ff4a006d990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff4b41614d0 0x7ff4b415dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4b40d4860 0x7ff4b4155db0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 --2- 172.21.15.72:0/3417455126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff4b40b0e30 0x7ff4b40ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 >> 172.21.15.72:0/3417455126 conn(0x7ff4b40d3d40 msgr2=0x7ff4b40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 shutdown_connections 2024-03-06T07:24:38.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.423+0000 7ff4b9ec9640 1 -- 172.21.15.72:0/3417455126 wait complete. 2024-03-06T07:24:38.436 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:24:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.537+0000 7f390beb8640 1 Processor -- start 2024-03-06T07:24:38.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.538+0000 7f390beb8640 1 -- start start 2024-03-06T07:24:38.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f390beb8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f390415c050 0x7f390406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f390beb8640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f39040cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f390beb8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39040cd4b0 con 0x7f390406fb40 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f390beb8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f390406ffb0 con 0x7f390415c050 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f390beb8640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39040cd790 con 0x7f39040d0af0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f3909453640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f390406fb40 0x7f39040cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57732/0 (socket says 172.21.15.72:57732) 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f3909453640 1 -- 172.21.15.72:0/79745021 learned_addr learned my addr 172.21.15.72:0/79745021 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f390a455640 1 --2- 172.21.15.72:0/79745021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f39040cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f3909c54640 1 --2- 172.21.15.72:0/79745021 >> v2:172.21.15.146:3300/0 conn(0x7f390415c050 0x7f390406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f3908c52640 1 -- 172.21.15.72:0/79745021 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942096333 0 0) 0x7f39040cd4b0 con 0x7f390406fb40 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.539+0000 7f3908c52640 1 -- 172.21.15.72:0/79745021 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f0003520 con 0x7f390406fb40 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.540+0000 7f390a455640 1 -- 172.21.15.72:0/79745021 >> v2:172.21.15.146:3300/0 conn(0x7f390415c050 msgr2=0x7f390406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.540+0000 7f390a455640 1 --2- 172.21.15.72:0/79745021 >> v2:172.21.15.146:3300/0 conn(0x7f390415c050 0x7f390406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.540+0000 7f390a455640 1 -- 172.21.15.72:0/79745021 >> v1:172.21.15.72:6789/0 conn(0x7f390406fb40 legacy=0x7f39040cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.540+0000 7f3908c52640 1 -- 172.21.15.72:0/79745021 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 374968293 0 0) 0x7f38f0003520 con 0x7f390406fb40 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.540+0000 7f390a455640 1 -- 172.21.15.72:0/79745021 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040cda70 con 0x7f39040d0af0 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.540+0000 7f390a455640 1 --2- 172.21.15.72:0/79745021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f39040cce70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009d90 tx=0x7f38fc02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b80f5ea71208056 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f3908c52640 1 -- 172.21.15.72:0/79745021 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc039070 con 0x7f39040d0af0 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f3908c52640 1 -- 172.21.15.72:0/79745021 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38fc02cbc0 con 0x7f39040d0af0 2024-03-06T07:24:38.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f3908c52640 1 -- 172.21.15.72:0/79745021 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc034440 con 0x7f39040d0af0 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 -- 172.21.15.72:0/79745021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 msgr2=0x7f39040cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 --2- 172.21.15.72:0/79745021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f39040cce70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009d90 tx=0x7f38fc02be10 comp rx=0 tx=0).stop 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 -- 172.21.15.72:0/79745021 shutdown_connections 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 --2- 172.21.15.72:0/79745021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f39040cce70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 --2- 172.21.15.72:0/79745021 >> v2:172.21.15.146:3300/0 conn(0x7f390415c050 0x7f390406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 -- 172.21.15.72:0/79745021 >> 172.21.15.72:0/79745021 conn(0x7f39040b3800 msgr2=0x7f39040b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 -- 172.21.15.72:0/79745021 shutdown_connections 2024-03-06T07:24:38.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.541+0000 7f390beb8640 1 -- 172.21.15.72:0/79745021 wait complete. 2024-03-06T07:24:38.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.543+0000 7f390beb8640 1 Processor -- start 2024-03-06T07:24:38.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.543+0000 7f390beb8640 1 -- start start 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f390beb8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 0x7f390414da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f390beb8640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f3904156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f390beb8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f390415c050 0x7f3904157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f390beb8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904165c30 con 0x7f390406fb40 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f390beb8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904165ab0 con 0x7f390415c050 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f390beb8640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3904165db0 con 0x7f39040d0af0 2024-03-06T07:24:38.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f3909c54640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 0x7f390414da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f3909c54640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 0x7f390414da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36596/0 (socket says 172.21.15.72:36596) 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.544+0000 7f3909c54640 1 -- 172.21.15.72:0/1091196753 learned_addr learned my addr 172.21.15.72:0/1091196753 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909453640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f3904156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f390a455640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f390415c050 0x7f3904157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909c54640 1 -- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 msgr2=0x7f3904156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909c54640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f3904156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909c54640 1 -- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f390415c050 msgr2=0x7f3904157030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909c54640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f390415c050 0x7f3904157030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909c54640 1 -- 172.21.15.72:0/1091196753 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390415a7a0 con 0x7f390406fb40 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f3909c54640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 0x7f390414da70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f390000e9e0 tx=0x7f390000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900017070 con 0x7f390406fb40 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390000c9d0 con 0x7f390406fb40 2024-03-06T07:24:38.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390415aa40 con 0x7f390406fb40 2024-03-06T07:24:38.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390000cdc0 con 0x7f390406fb40 2024-03-06T07:24:38.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.545+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390415b000 con 0x7f390406fb40 2024-03-06T07:24:38.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.546+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390409fe60 con 0x7f390406fb40 2024-03-06T07:24:38.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.546+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f390001a070 con 0x7f390406fb40 2024-03-06T07:24:38.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.547+0000 7f38fa7fc640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f38e806b4d0 0x7f38e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:38.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.547+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f390008a220 con 0x7f390406fb40 2024-03-06T07:24:38.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.547+0000 7f3909453640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f38e806b4d0 0x7f38e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:38.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.547+0000 7f3909453640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f38e806b4d0 0x7f38e806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f390409fe30 tx=0x7f38f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:38.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.555+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f39000590b0 con 0x7f390406fb40 2024-03-06T07:24:38.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.860+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3904068ff0 con 0x7f390406fb40 2024-03-06T07:24:38.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.860+0000 7f38fa7fc640 1 -- 172.21.15.72:0/1091196753 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39000589b0 con 0x7f390406fb40 2024-03-06T07:24:38.863 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:24:38.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f38e806b4d0 msgr2=0x7f38e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f38e806b4d0 0x7f38e806d990 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f390409fe30 tx=0x7f38f4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 msgr2=0x7f390414da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 0x7f390414da70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f390000e9e0 tx=0x7f390000eea0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 shutdown_connections 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f38e806b4d0 0x7f38e806d990 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f390415c050 0x7f3904157030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39040d0af0 0x7f3904156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 --2- 172.21.15.72:0/1091196753 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f390406fb40 0x7f390414da70 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 >> 172.21.15.72:0/1091196753 conn(0x7f39040b3800 msgr2=0x7f3904158b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 shutdown_connections 2024-03-06T07:24:38.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.862+0000 7f390beb8640 1 -- 172.21.15.72:0/1091196753 wait complete. 2024-03-06T07:24:38.878 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:38.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.995+0000 7fc2224ed640 1 Processor -- start 2024-03-06T07:24:38.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.995+0000 7fc2224ed640 1 -- start start 2024-03-06T07:24:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc2224ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 0x7fc21c15ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc2224ed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21c15f580 con 0x7fc21c0b3060 2024-03-06T07:24:39.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc2224ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21c15b450 con 0x7fc21c13dc10 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc2224ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0cf3f0 con 0x7fc21c15bf30 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc21c0b3060 0x7fc21c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57742/0 (socket says 172.21.15.72:57742) 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21b7fe640 1 -- 172.21.15.72:0/1255985011 learned_addr learned my addr 172.21.15.72:0/1255985011 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc220a8a640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 0x7fc21c15ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557395296 0 0) 0x7fc21c15f580 con 0x7fc21c0b3060 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204005730 con 0x7fc21c0b3060 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83385265 0 0) 0x7fc204005730 con 0x7fc21c0b3060 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 msgr2=0x7fc21c15ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 0x7fc21c15ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 >> v1:172.21.15.146:6789/0 conn(0x7fc21c13dc10 legacy=0x7fc21c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c15b730 con 0x7fc21c0b3060 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fc20c001670 con 0x7fc21c0b3060 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 >> v1:172.21.15.72:6789/0 conn(0x7fc21c0b3060 legacy=0x7fc21c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 0x7fc204005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204006170 0x7fc204006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc204006f50 0x7fc2040073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c0cf3f0 con 0x7fc204006170 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20400dd60 con 0x7fc204006f50 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.996+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20400df60 con 0x7fc21c15bf30 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21b7fe640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204006170 0x7fc204006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc220a8a640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc204006f50 0x7fc2040073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21bfff640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 0x7fc204005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21b7fe640 1 -- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 msgr2=0x7fc204005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21b7fe640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc21c15bf30 0x7fc204005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21b7fe640 1 -- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc204006f50 msgr2=0x7fc2040073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21b7fe640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc204006f50 0x7fc2040073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.997+0000 7fc21b7fe640 1 -- 172.21.15.72:0/1255985011 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc204005730 con 0x7fc204006170 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc21b7fe640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204006170 0x7fc204006a10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c240 tx=0x7fc204007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d71041837c3ca79a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c007cf0 con 0x7fc204006170 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c030070 con 0x7fc204006170 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc21affd640 1 -- 172.21.15.72:0/1255985011 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c005d30 con 0x7fc204006170 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc2224ed640 1 -- 172.21.15.72:0/1255985011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204006170 msgr2=0x7fc204006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc2224ed640 1 --2- 172.21.15.72:0/1255985011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc204006170 0x7fc204006a10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc20c00c240 tx=0x7fc204007a40 comp rx=0 tx=0).stop 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc220a8a640 1 -- 172.21.15.72:0/1255985011 reap_dead start 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc2224ed640 1 -- 172.21.15.72:0/1255985011 shutdown_connections 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.998+0000 7fc2224ed640 1 -- 172.21.15.72:0/1255985011 >> 172.21.15.72:0/1255985011 conn(0x7fc21c0cf8a0 msgr2=0x7fc21c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.999+0000 7fc2224ed640 1 -- 172.21.15.72:0/1255985011 shutdown_connections 2024-03-06T07:24:39.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:38.999+0000 7fc2224ed640 1 -- 172.21.15.72:0/1255985011 wait complete. 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.000+0000 7fc2224ed640 1 Processor -- start 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.000+0000 7fc2224ed640 1 -- start start 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.000+0000 7fc2224ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc204006170 0x7fc21c1417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.000+0000 7fc2224ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc21c13d210 0x7fc21c15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.000+0000 7fc2224ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 0x7fc21c06d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc2224ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c15ef60 con 0x7fc21c15e1d0 2024-03-06T07:24:39.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc2224ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c15b210 con 0x7fc21c13d210 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc2224ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c15f0e0 con 0x7fc204006170 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 0x7fc21c06d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 0x7fc21c06d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36610/0 (socket says 172.21.15.72:36610) 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 -- 172.21.15.72:0/2856825082 learned_addr learned my addr 172.21.15.72:0/2856825082 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc21bfff640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc204006170 0x7fc21c1417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc21b7fe640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc21c13d210 0x7fc21c15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 -- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc204006170 msgr2=0x7fc21c1417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc204006170 0x7fc21c1417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 -- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc21c13d210 msgr2=0x7fc21c15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc21c13d210 0x7fc21c15dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 -- 172.21.15.72:0/2856825082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c0cc1f0 con 0x7fc21c15e1d0 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc220a8a640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 0x7fc21c06d2b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0b3a00 tx=0x7fc21400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21400fe10 con 0x7fc21c15e1d0 2024-03-06T07:24:39.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21401d070 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c13e2a0 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214015670 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.001+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c13e860 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.003+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c04f330 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.003+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc214015810 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.003+0000 7fc218ff9640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc20806b500 0x7fc20806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.003+0000 7fc21bfff640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc20806b500 0x7fc20806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.003+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc21408ab90 con 0x7fc21c15e1d0 2024-03-06T07:24:39.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.004+0000 7fc21bfff640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc20806b500 0x7fc20806d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc2100035f0 tx=0x7fc210009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.008+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc214010040 con 0x7fc21c15e1d0 2024-03-06T07:24:39.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.224+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc21c0b29f0 con 0x7fc21c15e1d0 2024-03-06T07:24:39.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.224+0000 7fc218ff9640 1 -- 172.21.15.72:0/2856825082 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fc214059980 con 0x7fc21c15e1d0 2024-03-06T07:24:39.226 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:39.226 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:39.227 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:39.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc20806b500 msgr2=0x7fc20806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc20806b500 0x7fc20806d9c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc2100035f0 tx=0x7fc210009040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 msgr2=0x7fc21c06d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 0x7fc21c06d2b0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc21c0b3a00 tx=0x7fc21400f580 comp rx=0 tx=0).stop 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 shutdown_connections 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc20806b500 0x7fc20806d9c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc21c15e1d0 0x7fc21c06d2b0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc21c13d210 0x7fc21c15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 --2- 172.21.15.72:0/2856825082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc204006170 0x7fc21c1417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 >> 172.21.15.72:0/2856825082 conn(0x7fc21c0cf8a0 msgr2=0x7fc21c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 shutdown_connections 2024-03-06T07:24:39.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.226+0000 7fc2224ed640 1 -- 172.21.15.72:0/2856825082 wait complete. 2024-03-06T07:24:39.239 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.338+0000 7fe5e808d640 1 Processor -- start 2024-03-06T07:24:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.338+0000 7fe5e808d640 1 -- start start 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e808d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e808d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe5e00d4760 0x7fe5e00d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e808d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5e00d25d0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e808d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5e00d4bf0 con 0x7fe5e00d4760 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e808d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00ce9e0 con 0x7fe5e00b4170 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e5628640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe5e00d4760 0x7fe5e00d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e5628640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe5e00d4760 0x7fe5e00d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36626/0 (socket says 172.21.15.72:36626) 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.339+0000 7fe5e5628640 1 -- 172.21.15.72:0/2046169602 learned_addr learned my addr 172.21.15.72:0/2046169602 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e5628640 1 -- 172.21.15.72:0/2046169602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 msgr2=0x7fe5e00b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e5e29640 1 --2- 172.21.15.72:0/2046169602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e5628640 1 --2- 172.21.15.72:0/2046169602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e5628640 1 -- 172.21.15.72:0/2046169602 >> v1:172.21.15.146:6789/0 conn(0x7fe5e00b4dd0 legacy=0x7fe5e00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e5628640 1 -- 172.21.15.72:0/2046169602 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00d2870 con 0x7fe5e00d4760 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e5628640 1 --2- 172.21.15.72:0/2046169602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00d4760 0x7fe5e00d0370 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009870 tx=0x7fe5d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52794ff3eafc53ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e4e27640 1 -- 172.21.15.72:0/2046169602 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0035020 con 0x7fe5e00d4760 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e4e27640 1 -- 172.21.15.72:0/2046169602 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d0031640 con 0x7fe5e00d4760 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.340+0000 7fe5e4e27640 1 -- 172.21.15.72:0/2046169602 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d002d040 con 0x7fe5e00d4760 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 -- 172.21.15.72:0/2046169602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00d4760 msgr2=0x7fe5e00d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 --2- 172.21.15.72:0/2046169602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00d4760 0x7fe5e00d0370 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0009870 tx=0x7fe5d002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 -- 172.21.15.72:0/2046169602 shutdown_connections 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 --2- 172.21.15.72:0/2046169602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00d4760 0x7fe5e00d0370 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 --2- 172.21.15.72:0/2046169602 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 -- 172.21.15.72:0/2046169602 >> 172.21.15.72:0/2046169602 conn(0x7fe5e00d3c40 msgr2=0x7fe5e00d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 -- 172.21.15.72:0/2046169602 shutdown_connections 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.341+0000 7fe5e808d640 1 -- 172.21.15.72:0/2046169602 wait complete. 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.342+0000 7fe5e808d640 1 Processor -- start 2024-03-06T07:24:39.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.342+0000 7fe5e808d640 1 -- start start 2024-03-06T07:24:39.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e808d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e808d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 0x7fe5e00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e808d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5e014d540 0x7fe5e00cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e808d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00b47d0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e808d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00b4550 con 0x7fe5e014d540 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e808d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5e00b4950 con 0x7fe5e00b4170 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 0x7fe5e00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 0x7fe5e00b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36628/0 (socket says 172.21.15.72:36628) 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 -- 172.21.15.72:0/1731770320 learned_addr learned my addr 172.21.15.72:0/1731770320 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5e29640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 -- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 msgr2=0x7fe5e00ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 -- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5e014d540 msgr2=0x7fe5e00cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5e014d540 0x7fe5e00cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 -- 172.21.15.72:0/1731770320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e00c99c0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.343+0000 7fe5e5628640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 0x7fe5e00b07f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.344+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0040070 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.344+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5d0039bf0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.344+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5e00b31b0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.344+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0035040 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.344+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5e00c9c30 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.345+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5e004f330 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.345+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe5d0038020 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.345+0000 7fe5d67fc640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe5c406b4d0 0x7fe5c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.346+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5d00abcf0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.346+0000 7fe5e5e29640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe5c406b4d0 0x7fe5c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.346+0000 7fe5e5e29640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe5c406b4d0 0x7fe5c406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.350+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5d00b0070 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.530+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5e00733b0 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.532+0000 7fe5d67fc640 1 -- 172.21.15.72:0/1731770320 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fe5d007ac00 con 0x7fe5e00b4dd0 2024-03-06T07:24:39.534 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:39.534 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe5c406b4d0 msgr2=0x7fe5c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe5c406b4d0 0x7fe5c406d990 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc008040 comp rx=0 tx=0).stop 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 msgr2=0x7fe5e00b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 0x7fe5e00b07f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0000c00 tx=0x7fe5d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 shutdown_connections 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe5c406b4d0 0x7fe5c406d990 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe5e014d540 0x7fe5e00cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5e00b4dd0 0x7fe5e00b07f0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 --2- 172.21.15.72:0/1731770320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5e00b4170 0x7fe5e00ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.534+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 >> 172.21.15.72:0/1731770320 conn(0x7fe5e00d3c40 msgr2=0x7fe5e00b6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.535+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 shutdown_connections 2024-03-06T07:24:39.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.535+0000 7fe5e808d640 1 -- 172.21.15.72:0/1731770320 wait complete. 2024-03-06T07:24:39.547 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:24:39.548 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:39.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.650+0000 7f7fad981640 1 Processor -- start 2024-03-06T07:24:39.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.650+0000 7f7fad981640 1 -- start start 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fad981640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7fa806c6d0 0x7f7fa815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fad981640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7fa8066d90 0x7f7fa80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fad981640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa80cd450 con 0x7f7fa80b4f70 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fad981640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa815e9f0 con 0x7f7fa806c6d0 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fad981640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa80cda10 con 0x7f7fa8066d90 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7fa806c6d0 0x7f7fa815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7fa806c6d0 0x7f7fa815ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36630/0 (socket says 172.21.15.72:36630) 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 -- 172.21.15.72:0/628489437 learned_addr learned my addr 172.21.15.72:0/628489437 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 -- 172.21.15.72:0/628489437 >> v2:172.21.15.146:3300/0 conn(0x7f7fa8066d90 msgr2=0x7f7fa80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa77fe640 1 --2- 172.21.15.72:0/628489437 >> v2:172.21.15.146:3300/0 conn(0x7f7fa8066d90 0x7f7fa80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 --2- 172.21.15.72:0/628489437 >> v2:172.21.15.146:3300/0 conn(0x7f7fa8066d90 0x7f7fa80ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 -- 172.21.15.72:0/628489437 >> v1:172.21.15.146:6789/0 conn(0x7f7fa80b4f70 legacy=0x7f7fa80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa67fc640 1 -- 172.21.15.72:0/628489437 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa80cd730 con 0x7f7fa806c6d0 2024-03-06T07:24:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.651+0000 7f7fa77fe640 1 --2- 172.21.15.72:0/628489437 >> v2:172.21.15.146:3300/0 conn(0x7f7fa8066d90 0x7f7fa80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.652+0000 7f7fa67fc640 1 --2- 172.21.15.72:0/628489437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa806c6d0 0x7f7fa815ee80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7fa815e960 tx=0x7f7f9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6228e14dcb1d75d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.652+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/628489437 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98035020 con 0x7f7fa806c6d0 2024-03-06T07:24:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.652+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/628489437 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f98031870 con 0x7f7fa806c6d0 2024-03-06T07:24:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.652+0000 7f7fa5ffb640 1 -- 172.21.15.72:0/628489437 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f98031c40 con 0x7f7fa806c6d0 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.652+0000 7f7fad981640 1 -- 172.21.15.72:0/628489437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa806c6d0 msgr2=0x7f7fa815ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.652+0000 7f7fad981640 1 --2- 172.21.15.72:0/628489437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa806c6d0 0x7f7fa815ee80 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7fa815e960 tx=0x7f7f9802b910 comp rx=0 tx=0).stop 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.653+0000 7f7fad981640 1 -- 172.21.15.72:0/628489437 shutdown_connections 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.653+0000 7f7fad981640 1 --2- 172.21.15.72:0/628489437 >> v2:172.21.15.146:3300/0 conn(0x7f7fa8066d90 0x7f7fa80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.653+0000 7f7fad981640 1 --2- 172.21.15.72:0/628489437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa806c6d0 0x7f7fa815ee80 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.653+0000 7f7fad981640 1 -- 172.21.15.72:0/628489437 >> 172.21.15.72:0/628489437 conn(0x7f7fa80724d0 msgr2=0x7f7fa80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.653+0000 7f7fad981640 1 -- 172.21.15.72:0/628489437 shutdown_connections 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.653+0000 7f7fad981640 1 -- 172.21.15.72:0/628489437 wait complete. 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.654+0000 7f7fad981640 1 Processor -- start 2024-03-06T07:24:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.654+0000 7f7fad981640 1 -- start start 2024-03-06T07:24:39.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.654+0000 7f7fad981640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa8066d90 0x7f7fa81556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.654+0000 7f7fad981640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa80b4f70 0x7f7fa8158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fad981640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 0x7f7fa80c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fad981640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa80cd090 con 0x7f7fa8159110 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fad981640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa815f3c0 con 0x7f7fa80b4f70 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fad981640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa80cd210 con 0x7f7fa8066d90 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 0x7f7fa80c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 0x7f7fa80c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36642/0 (socket says 172.21.15.72:36642) 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 -- 172.21.15.72:0/781359590 learned_addr learned my addr 172.21.15.72:0/781359590 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa67fc640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa80b4f70 0x7f7fa8158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa6ffd640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa8066d90 0x7f7fa81556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 -- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa8066d90 msgr2=0x7f7fa81556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa8066d90 0x7f7fa81556a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 -- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa80b4f70 msgr2=0x7f7fa8158bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa80b4f70 0x7f7fa8158bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 -- 172.21.15.72:0/781359590 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa81574b0 con 0x7f7fa8159110 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fa77fe640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 0x7f7fa80c9490 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000ce70 tx=0x7f7fa000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0003b90 con 0x7f7fa8159110 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa0003d50 con 0x7f7fa8159110 2024-03-06T07:24:39.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa8157790 con 0x7f7fa8159110 2024-03-06T07:24:39.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0004260 con 0x7f7fa8159110 2024-03-06T07:24:39.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.655+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa8157d50 con 0x7f7fa8159110 2024-03-06T07:24:39.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.656+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa804f330 con 0x7f7fa8159110 2024-03-06T07:24:39.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.658+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7fa0004400 con 0x7f7fa8159110 2024-03-06T07:24:39.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.658+0000 7f7f9ffff640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7f8806b4d0 0x7f7f8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.658+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fa008aac0 con 0x7f7fa8159110 2024-03-06T07:24:39.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.658+0000 7f7fa6ffd640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7f8806b4d0 0x7f7f8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.658+0000 7f7fa6ffd640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7f8806b4d0 0x7f7f8806d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8159720 tx=0x7f7f90009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.662+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7fa00592f0 con 0x7f7fa8159110 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.878+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fa8074ab0 con 0x7f7fa8159110 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.878+0000 7f7f9ffff640 1 -- 172.21.15.72:0/781359590 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7f7fa005d390 con 0x7f7fa8159110 2024-03-06T07:24:39.880 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:39.881 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7f8806b4d0 msgr2=0x7f7f8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7f8806b4d0 0x7f7f8806d990 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8159720 tx=0x7f7f90009060 comp rx=0 tx=0).stop 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 msgr2=0x7f7fa80c9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 0x7f7fa80c9490 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7fa000ce70 tx=0x7f7fa000a710 comp rx=0 tx=0).stop 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 shutdown_connections 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7f8806b4d0 0x7f7f8806d990 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7fa8159110 0x7f7fa80c9490 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa80b4f70 0x7f7fa8158bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 --2- 172.21.15.72:0/781359590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fa8066d90 0x7f7fa81556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 >> 172.21.15.72:0/781359590 conn(0x7f7fa80724d0 msgr2=0x7f7fa80b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 shutdown_connections 2024-03-06T07:24:39.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.880+0000 7f7fad981640 1 -- 172.21.15.72:0/781359590 wait complete. 2024-03-06T07:24:39.893 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:39.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.992+0000 7fb9cddc9640 1 Processor -- start 2024-03-06T07:24:39.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.992+0000 7fb9cddc9640 1 -- start start 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9cddc9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9c80d04a0 0x7fb9c80ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9cddc9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c8161980 con 0x7fb9c80cfff0 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9cddc9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c8161c60 con 0x7fb9c815bb20 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9cddc9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c80d2500 con 0x7fb9c80d04a0 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9c7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9c80d04a0 0x7fb9c80ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9c6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb9c80cfff0 0x7fb9c80cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57758/0 (socket says 172.21.15.72:57758) 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.993+0000 7fb9c6ffd640 1 -- 172.21.15.72:0/4044975774 learned_addr learned my addr 172.21.15.72:0/4044975774 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.994+0000 7fb9c7fff640 1 -- 172.21.15.72:0/4044975774 >> v1:172.21.15.146:6789/0 conn(0x7fb9c815bb20 legacy=0x7fb9c80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.994+0000 7fb9c7fff640 1 -- 172.21.15.72:0/4044975774 >> v1:172.21.15.72:6789/0 conn(0x7fb9c80cfff0 legacy=0x7fb9c80cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.994+0000 7fb9c7fff640 1 -- 172.21.15.72:0/4044975774 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c8161f40 con 0x7fb9c80d04a0 2024-03-06T07:24:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.994+0000 7fb9c67fc640 1 -- 172.21.15.72:0/4044975774 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195489337 0 0) 0x7fb9c8161980 con 0x7fb9c80cfff0 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.994+0000 7fb9c7fff640 1 --2- 172.21.15.72:0/4044975774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80ccfc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009d90 tx=0x7fb9c002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70effa27729cb9e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9c67fc640 1 -- 172.21.15.72:0/4044975774 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0039070 con 0x7fb9c80d04a0 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9c67fc640 1 -- 172.21.15.72:0/4044975774 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9c002cd00 con 0x7fb9c80d04a0 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9c67fc640 1 -- 172.21.15.72:0/4044975774 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0034660 con 0x7fb9c80d04a0 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 -- 172.21.15.72:0/4044975774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 msgr2=0x7fb9c80ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/4044975774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80ccfc0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009d90 tx=0x7fb9c002bf50 comp rx=0 tx=0).stop 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 -- 172.21.15.72:0/4044975774 shutdown_connections 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/4044975774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80ccfc0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 -- 172.21.15.72:0/4044975774 >> 172.21.15.72:0/4044975774 conn(0x7fb9c809a9f0 msgr2=0x7fb9c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 -- 172.21.15.72:0/4044975774 shutdown_connections 2024-03-06T07:24:39.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.995+0000 7fb9cddc9640 1 -- 172.21.15.72:0/4044975774 wait complete. 2024-03-06T07:24:39.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.996+0000 7fb9cddc9640 1 Processor -- start 2024-03-06T07:24:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.996+0000 7fb9cddc9640 1 -- start start 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9cddc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c80cfff0 0x7fb9c815a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9cddc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9cddc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c815bb20 0x7fb9c81609d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9cddc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80d08d0 con 0x7fb9c80d04a0 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9cddc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c8162500 con 0x7fb9c815bb20 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9cddc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c80d1980 con 0x7fb9c80cfff0 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9c6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9c6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80c64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36654/0 (socket says 172.21.15.72:36654) 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9c6ffd640 1 -- 172.21.15.72:0/2017628172 learned_addr learned my addr 172.21.15.72:0/2017628172 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9c7fff640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c815bb20 0x7fb9c81609d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.997+0000 7fb9c77fe640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c80cfff0 0x7fb9c815a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9c6ffd640 1 -- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c80cfff0 msgr2=0x7fb9c815a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9c6ffd640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c80cfff0 0x7fb9c815a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9c6ffd640 1 -- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c815bb20 msgr2=0x7fb9c81609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9c6ffd640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c815bb20 0x7fb9c81609d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9c6ffd640 1 -- 172.21.15.72:0/2017628172 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c815e210 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9c6ffd640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80c64e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c420 tx=0x7fb9bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc0074e0 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc0047e0 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c815aa00 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc004c10 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.998+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c815e4b0 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.999+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c8068200 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:39.999+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb9bc011070 con 0x7fb9c80d04a0 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.000+0000 7fb9affff640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb9b406b4d0 0x7fb9b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.000+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9bc089d10 con 0x7fb9c80d04a0 2024-03-06T07:24:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.000+0000 7fb9c77fe640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb9b406b4d0 0x7fb9b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.000+0000 7fb9c77fe640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb9b406b4d0 0x7fb9b406d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000f60 tx=0x7fb9b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.004+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb9c8068200 con 0x7fb9c80d04a0 2024-03-06T07:24:40.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.218+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c80b3030 con 0x7fb9c80d04a0 2024-03-06T07:24:40.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.218+0000 7fb9affff640 1 -- 172.21.15.72:0/2017628172 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4983 (secure 0 0 0) 0x7fb9bc058b90 con 0x7fb9c80d04a0 2024-03-06T07:24:40.221 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:40.221 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":27,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:23:59.778464+0000","modified":"2024-03-06T07:24:01.798443+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6057},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6057":{"gid":6057,"name":"c","rank":0,"incarnation":25,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3136290264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3136290264},{"type":"v1","addr":"172.21.15.72:6837","nonce":3136290264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6057,"qdb_cluster":[6057]},"id":4}]} 2024-03-06T07:24:40.221 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 27 2024-03-06T07:24:40.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb9b406b4d0 msgr2=0x7fb9b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb9b406b4d0 0x7fb9b406d990 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8000f60 tx=0x7fb9b8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 msgr2=0x7fb9c80c64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80c64e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00c420 tx=0x7fb9bc00c8e0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 shutdown_connections 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb9b406b4d0 0x7fb9b406d990 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9c815bb20 0x7fb9c81609d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9c80d04a0 0x7fb9c80c64e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 --2- 172.21.15.72:0/2017628172 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c80cfff0 0x7fb9c815a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 >> 172.21.15.72:0/2017628172 conn(0x7fb9c809a9f0 msgr2=0x7fb9c8070150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 shutdown_connections 2024-03-06T07:24:40.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.220+0000 7fb9cddc9640 1 -- 172.21.15.72:0/2017628172 wait complete. 2024-03-06T07:24:40.233 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:40.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.332+0000 7fcceeb56640 1 Processor -- start 2024-03-06T07:24:40.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.332+0000 7fcceeb56640 1 -- start start 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcceeb56640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcce815dfe0 0x7fcce80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcceeb56640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcce80d29c0 0x7fcce80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcceeb56640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce80b3430 con 0x7fcce80cf830 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcceeb56640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80b31e0 con 0x7fcce80d29c0 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcceeb56640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce80b3d60 con 0x7fcce815dfe0 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcce80d29c0 0x7fcce80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcce80d29c0 0x7fcce80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36656/0 (socket says 172.21.15.72:36656) 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 -- 172.21.15.72:0/3469244289 learned_addr learned my addr 172.21.15.72:0/3469244289 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fccec8f2640 1 --2- 172.21.15.72:0/3469244289 >> v2:172.21.15.146:3300/0 conn(0x7fcce815dfe0 0x7fcce80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 -- 172.21.15.72:0/3469244289 >> v2:172.21.15.146:3300/0 conn(0x7fcce815dfe0 msgr2=0x7fcce80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 --2- 172.21.15.72:0/3469244289 >> v2:172.21.15.146:3300/0 conn(0x7fcce815dfe0 0x7fcce80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 -- 172.21.15.72:0/3469244289 >> v1:172.21.15.146:6789/0 conn(0x7fcce80cf830 legacy=0x7fcce80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.333+0000 7fcced0f3640 1 -- 172.21.15.72:0/3469244289 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce80b3710 con 0x7fcce80d29c0 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.334+0000 7fcced0f3640 1 --2- 172.21.15.72:0/3469244289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80d29c0 0x7fcce80b41f0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcce8067990 tx=0x7fcce002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb38bc44c6c0da3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.334+0000 7fcce77fe640 1 -- 172.21.15.72:0/3469244289 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce002cb20 con 0x7fcce80d29c0 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.334+0000 7fcce77fe640 1 -- 172.21.15.72:0/3469244289 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce002ccc0 con 0x7fcce80d29c0 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.334+0000 7fcce77fe640 1 -- 172.21.15.72:0/3469244289 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0034610 con 0x7fcce80d29c0 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 -- 172.21.15.72:0/3469244289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80d29c0 msgr2=0x7fcce80b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 --2- 172.21.15.72:0/3469244289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80d29c0 0x7fcce80b41f0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcce8067990 tx=0x7fcce002bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 -- 172.21.15.72:0/3469244289 shutdown_connections 2024-03-06T07:24:40.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 --2- 172.21.15.72:0/3469244289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80d29c0 0x7fcce80b41f0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 --2- 172.21.15.72:0/3469244289 >> v2:172.21.15.146:3300/0 conn(0x7fcce815dfe0 0x7fcce80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 -- 172.21.15.72:0/3469244289 >> 172.21.15.72:0/3469244289 conn(0x7fcce80d3cd0 msgr2=0x7fcce80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 -- 172.21.15.72:0/3469244289 shutdown_connections 2024-03-06T07:24:40.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.335+0000 7fcceeb56640 1 -- 172.21.15.72:0/3469244289 wait complete. 2024-03-06T07:24:40.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.337+0000 7fcceeb56640 1 Processor -- start 2024-03-06T07:24:40.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.337+0000 7fcceeb56640 1 -- start start 2024-03-06T07:24:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fcceeb56640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 0x7fcce81f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fcceeb56640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce80d29c0 0x7fcce80ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fcceeb56640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce815dfe0 0x7fcce815fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fcceeb56640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80d1df0 con 0x7fcce80cf830 2024-03-06T07:24:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fcceeb56640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce80d1c70 con 0x7fcce80d29c0 2024-03-06T07:24:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fcceeb56640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce815e3c0 con 0x7fcce815dfe0 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 0x7fcce81f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 0x7fcce81f4490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36668/0 (socket says 172.21.15.72:36668) 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 -- 172.21.15.72:0/540979743 learned_addr learned my addr 172.21.15.72:0/540979743 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 -- 172.21.15.72:0/540979743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce815dfe0 msgr2=0x7fcce815fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce815dfe0 0x7fcce815fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 -- 172.21.15.72:0/540979743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce80d29c0 msgr2=0x7fcce80ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce80d29c0 0x7fcce80ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.338+0000 7fccec8f2640 1 -- 172.21.15.72:0/540979743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce81f49d0 con 0x7fcce80cf830 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.339+0000 7fccec8f2640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 0x7fcce81f4490 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00d620 tx=0x7fccdc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.339+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc00b620 con 0x7fcce80cf830 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.339+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc019070 con 0x7fcce80cf830 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.339+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce80cc170 con 0x7fcce80cf830 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.339+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc00ba00 con 0x7fcce80cf830 2024-03-06T07:24:40.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.339+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce80cb690 con 0x7fcce80cf830 2024-03-06T07:24:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.340+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce804f330 con 0x7fcce80cf830 2024-03-06T07:24:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.340+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fccdc00bba0 con 0x7fcce80cf830 2024-03-06T07:24:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.340+0000 7fcce57fa640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.341+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccdc08a620 con 0x7fcce80cf830 2024-03-06T07:24:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.341+0000 7fcce7fff640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.341+0000 7fcce7fff640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcccc06b4d0 0x7fcccc06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fccd4000f60 tx=0x7fccd40079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.349+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcce81f49d0 con 0x7fcce80cf830 2024-03-06T07:24:40.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.529+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcce80008d0 con 0x7fcce80cf830 2024-03-06T07:24:40.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.530+0000 7fcce57fa640 1 -- 172.21.15.72:0/540979743 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16677 (secure 0 0 0) 0x7fccdc0594a0 con 0x7fcce80cf830 2024-03-06T07:24:40.533 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:40.533 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:01.438914+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:23:57.998737+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-06T07:23:58.744228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.532+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcccc06b4d0 msgr2=0x7fcccc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.532+0000 7fcceeb56640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcccc06b4d0 0x7fcccc06d990 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fccd4000f60 tx=0x7fccd40079d0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 msgr2=0x7fcce81f4490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 0x7fcce81f4490 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00d620 tx=0x7fccdc00dae0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 shutdown_connections 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcccc06b4d0 0x7fcccc06d990 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce815dfe0 0x7fcce815fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce80d29c0 0x7fcce80ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 --2- 172.21.15.72:0/540979743 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcce80cf830 0x7fcce81f4490 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 >> 172.21.15.72:0/540979743 conn(0x7fcce80d3cd0 msgr2=0x7fcce806d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 shutdown_connections 2024-03-06T07:24:40.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.533+0000 7fcceeb56640 1 -- 172.21.15.72:0/540979743 wait complete. 2024-03-06T07:24:40.546 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:24:40.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.645+0000 7f69573f6640 1 Processor -- start 2024-03-06T07:24:40.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.646+0000 7f69573f6640 1 -- start start 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.646+0000 7f69573f6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69500b1a20 0x7f6950168bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.646+0000 7f69573f6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69500af4b0 con 0x7f695006d3c0 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.646+0000 7f69573f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69500af790 con 0x7f69501614a0 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.646+0000 7f69573f6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69500afa70 con 0x7f69500b1a20 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6955993640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69500b1a20 0x7f6950168bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6955993640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69500b1a20 0x7f6950168bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36674/0 (socket says 172.21.15.72:36674) 2024-03-06T07:24:40.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6955993640 1 -- 172.21.15.72:0/2159215454 learned_addr learned my addr 172.21.15.72:0/2159215454 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6954991640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f695006d3c0 0x7f69500b1310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57762/0 (socket says 172.21.15.72:57762) 2024-03-06T07:24:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6955993640 1 -- 172.21.15.72:0/2159215454 >> v1:172.21.15.146:6789/0 conn(0x7f69501614a0 legacy=0x7f69500b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6955993640 1 -- 172.21.15.72:0/2159215454 >> v1:172.21.15.72:6789/0 conn(0x7f695006d3c0 legacy=0x7f69500b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.647+0000 7f6955993640 1 -- 172.21.15.72:0/2159215454 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69500afd50 con 0x7f69500b1a20 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f6955993640 1 --2- 172.21.15.72:0/2159215454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f6950168bd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6948009e50 tx=0x7f694802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c1e6020ac2e205 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f6947fff640 1 -- 172.21.15.72:0/2159215454 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694802cb20 con 0x7f69500b1a20 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f6947fff640 1 -- 172.21.15.72:0/2159215454 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694802ccc0 con 0x7f69500b1a20 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f6947fff640 1 -- 172.21.15.72:0/2159215454 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6948034610 con 0x7f69500b1a20 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 -- 172.21.15.72:0/2159215454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 msgr2=0x7f6950168bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 --2- 172.21.15.72:0/2159215454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f6950168bd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6948009e50 tx=0x7f694802bf80 comp rx=0 tx=0).stop 2024-03-06T07:24:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 -- 172.21.15.72:0/2159215454 shutdown_connections 2024-03-06T07:24:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 --2- 172.21.15.72:0/2159215454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f6950168bd0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 -- 172.21.15.72:0/2159215454 >> 172.21.15.72:0/2159215454 conn(0x7f695009a9f0 msgr2=0x7f695009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 -- 172.21.15.72:0/2159215454 shutdown_connections 2024-03-06T07:24:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.648+0000 7f69573f6640 1 -- 172.21.15.72:0/2159215454 wait complete. 2024-03-06T07:24:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.650+0000 7f69573f6640 1 Processor -- start 2024-03-06T07:24:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.650+0000 7f69573f6640 1 -- start start 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.651+0000 7f69573f6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006d3c0 0x7f695015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.651+0000 7f69573f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f69501475a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.651+0000 7f69573f6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69501614a0 0x7f69500b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.651+0000 7f69573f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69500b1e00 con 0x7f69500b1a20 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.651+0000 7f69573f6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69500b6520 con 0x7f69501614a0 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.651+0000 7f69573f6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69500b1f80 con 0x7f695006d3c0 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f69501475a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f69501475a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36686/0 (socket says 172.21.15.72:36686) 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 -- 172.21.15.72:0/2309725448 learned_addr learned my addr 172.21.15.72:0/2309725448 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6955192640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006d3c0 0x7f695015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6955993640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69501614a0 0x7f69500b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 -- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006d3c0 msgr2=0x7f695015b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006d3c0 0x7f695015b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 -- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69501614a0 msgr2=0x7f69500b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69501614a0 0x7f69500b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 -- 172.21.15.72:0/2309725448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950147ae0 con 0x7f69500b1a20 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6955993640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69501614a0 0x7f69500b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6955192640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006d3c0 0x7f695015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6954991640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f69501475a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f694000c3c0 tx=0x7f694000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69400074e0 con 0x7f69500b1a20 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6940007680 con 0x7f69500b1a20 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695015a720 con 0x7f69500b1a20 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6940004a00 con 0x7f69500b1a20 2024-03-06T07:24:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.652+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695015ace0 con 0x7f69500b1a20 2024-03-06T07:24:40.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.653+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695004f330 con 0x7f69500b1a20 2024-03-06T07:24:40.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.654+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f694001a020 con 0x7f69500b1a20 2024-03-06T07:24:40.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.654+0000 7f6945ffb640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693406b4d0 0x7f693406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.654+0000 7f6955192640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693406b4d0 0x7f693406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.654+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6940089970 con 0x7f69500b1a20 2024-03-06T07:24:40.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.655+0000 7f6955192640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693406b4d0 0x7f693406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f694c003eb0 tx=0x7f694c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:40.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.658+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69400587f0 con 0x7f69500b1a20 2024-03-06T07:24:40.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.873+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f69500aec10 con 0x7f69500b1a20 2024-03-06T07:24:40.875 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:24:40.874+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:24:40.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.875+0000 7f6945ffb640 1 -- 172.21.15.72:0/2309725448 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7f69400580f0 con 0x7f69500b1a20 2024-03-06T07:24:40.877 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:24:40.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693406b4d0 msgr2=0x7f693406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693406b4d0 0x7f693406d990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f694c003eb0 tx=0x7f694c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 msgr2=0x7f69501475a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f69501475a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f694000c3c0 tx=0x7f694000c880 comp rx=0 tx=0).stop 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 shutdown_connections 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f693406b4d0 0x7f693406d990 secure :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f694c003eb0 tx=0x7f694c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69501614a0 0x7f69500b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69500b1a20 0x7f69501475a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 --2- 172.21.15.72:0/2309725448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695006d3c0 0x7f695015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.878+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 >> 172.21.15.72:0/2309725448 conn(0x7f695009a9f0 msgr2=0x7f69501607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:40.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.879+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 shutdown_connections 2024-03-06T07:24:40.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.879+0000 7f69573f6640 1 -- 172.21.15.72:0/2309725448 wait complete. 2024-03-06T07:24:40.892 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:24:40.917 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:24:40.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.993+0000 7f3845d5e640 1 Processor -- start 2024-03-06T07:24:40.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f3845d5e640 1 -- start start 2024-03-06T07:24:40.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f3845d5e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38400d06e0 0x7f384015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f3845d5e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f38400cf200 0x7f38400cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f3845d5e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38400d47f0 con 0x7f38400d0ac0 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f3845d5e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38400ced70 con 0x7f38400cf200 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f3845d5e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38400d1ab0 con 0x7f38400d06e0 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.994+0000 7f383ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f38400cf200 0x7f38400cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f38400cf200 0x7f38400cf650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36706/0 (socket says 172.21.15.72:36706) 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383ffff640 1 -- 172.21.15.72:0/2591026638 learned_addr learned my addr 172.21.15.72:0/2591026638 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383f7fe640 1 --2- 172.21.15.72:0/2591026638 >> v2:172.21.15.146:3300/0 conn(0x7f38400d06e0 0x7f384015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383ffff640 1 -- 172.21.15.72:0/2591026638 >> v2:172.21.15.146:3300/0 conn(0x7f38400d06e0 msgr2=0x7f384015fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383ffff640 1 --2- 172.21.15.72:0/2591026638 >> v2:172.21.15.146:3300/0 conn(0x7f38400d06e0 0x7f384015fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:40.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383ffff640 1 -- 172.21.15.72:0/2591026638 >> v1:172.21.15.146:6789/0 conn(0x7f38400d0ac0 legacy=0x7f38400b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.995+0000 7f383ffff640 1 -- 172.21.15.72:0/2591026638 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38400d1d90 con 0x7f38400cf200 2024-03-06T07:24:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f383ffff640 1 --2- 172.21.15.72:0/2591026638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400cf200 0x7f38400cf650 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f38400b6850 tx=0x7f383802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4745f1b165104af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f383e7fc640 1 -- 172.21.15.72:0/2591026638 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383802ca80 con 0x7f38400cf200 2024-03-06T07:24:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f383e7fc640 1 -- 172.21.15.72:0/2591026638 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383802cc20 con 0x7f38400cf200 2024-03-06T07:24:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f383e7fc640 1 -- 172.21.15.72:0/2591026638 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380343f0 con 0x7f38400cf200 2024-03-06T07:24:41.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 -- 172.21.15.72:0/2591026638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400cf200 msgr2=0x7f38400cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 --2- 172.21.15.72:0/2591026638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400cf200 0x7f38400cf650 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f38400b6850 tx=0x7f383802bee0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 -- 172.21.15.72:0/2591026638 shutdown_connections 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 --2- 172.21.15.72:0/2591026638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400cf200 0x7f38400cf650 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 --2- 172.21.15.72:0/2591026638 >> v2:172.21.15.146:3300/0 conn(0x7f38400d06e0 0x7f384015fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 -- 172.21.15.72:0/2591026638 >> 172.21.15.72:0/2591026638 conn(0x7f38400d3cd0 msgr2=0x7f38400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 -- 172.21.15.72:0/2591026638 shutdown_connections 2024-03-06T07:24:41.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.996+0000 7f3845d5e640 1 -- 172.21.15.72:0/2591026638 wait complete. 2024-03-06T07:24:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.997+0000 7f3845d5e640 1 Processor -- start 2024-03-06T07:24:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 -- start start 2024-03-06T07:24:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38400cf200 0x7f3840151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38400d06e0 0x7f38400c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 0x7f38400c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400b44d0 con 0x7f38400d0ac0 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38400ce990 con 0x7f38400cf200 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f3845d5e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38400d0f30 con 0x7f38400d06e0 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f383ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 0x7f38400c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f383ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 0x7f38400c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36720/0 (socket says 172.21.15.72:36720) 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.998+0000 7f383ffff640 1 -- 172.21.15.72:0/618097106 learned_addr learned my addr 172.21.15.72:0/618097106 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383f7fe640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38400cf200 0x7f3840151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383effd640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38400d06e0 0x7f38400c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383ffff640 1 -- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38400d06e0 msgr2=0x7f38400c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383ffff640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38400d06e0 0x7f38400c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383ffff640 1 -- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38400cf200 msgr2=0x7f3840151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383ffff640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38400cf200 0x7f3840151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383ffff640 1 -- 172.21.15.72:0/618097106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38401522d0 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383f7fe640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38400cf200 0x7f3840151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f383ffff640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 0x7f38400c91e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3838009f40 tx=0x7f3838004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838043020 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383803e450 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838009510 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38401543b0 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:40.999+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38400c84a0 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.000+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38400a1c80 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.000+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3838046020 con 0x7f38400d0ac0 2024-03-06T07:24:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.001+0000 7f3827fff640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f382c06b4d0 0x7f382c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.001+0000 7f383f7fe640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f382c06b4d0 0x7f382c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.001+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f38380aca20 con 0x7f38400d0ac0 2024-03-06T07:24:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.001+0000 7f383f7fe640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f382c06b4d0 0x7f382c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3830006fd0 tx=0x7f3830008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.005+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f383804d030 con 0x7f38400d0ac0 2024-03-06T07:24:41.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.217+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3840068ff0 con 0x7f38400d0ac0 2024-03-06T07:24:41.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.876+0000 7f3827fff640 1 -- 172.21.15.72:0/618097106 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f383807b9f0 con 0x7f38400d0ac0 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.879+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f382c06b4d0 msgr2=0x7f382c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.879+0000 7f3845d5e640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f382c06b4d0 0x7f382c06d990 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3830006fd0 tx=0x7f3830008040 comp rx=0 tx=0).stop 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 msgr2=0x7f38400c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 0x7f38400c91e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3838009f40 tx=0x7f3838004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 shutdown_connections 2024-03-06T07:24:41.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f382c06b4d0 0x7f382c06d990 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38400d0ac0 0x7f38400c91e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38400d06e0 0x7f38400c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 --2- 172.21.15.72:0/618097106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38400cf200 0x7f3840151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 >> 172.21.15.72:0/618097106 conn(0x7f38400d3cd0 msgr2=0x7f3840161dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 shutdown_connections 2024-03-06T07:24:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:41.880+0000 7f3845d5e640 1 -- 172.21.15.72:0/618097106 wait complete. 2024-03-06T07:24:41.895 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:24:42.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.046+0000 7f4e58925640 1 Processor -- start 2024-03-06T07:24:42.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.046+0000 7f4e58925640 1 -- start start 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e58925640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e540ce8c0 0x7f4e540cc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e58925640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5415c980 0x7f4e54071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e58925640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e540ced50 con 0x7f4e5415c5a0 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e58925640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e54072a20 con 0x7f4e540ce8c0 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e58925640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540708b0 con 0x7f4e5415c980 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e540ce8c0 0x7f4e540cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e540ce8c0 0x7f4e540cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36722/0 (socket says 172.21.15.72:36722) 2024-03-06T07:24:42.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 -- 172.21.15.72:0/252323 learned_addr learned my addr 172.21.15.72:0/252323 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e52d9d640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5415c980 0x7f4e54071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 -- 172.21.15.72:0/252323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5415c980 msgr2=0x7f4e54071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5415c980 0x7f4e54071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 -- 172.21.15.72:0/252323 >> v1:172.21.15.146:6789/0 conn(0x7f4e5415c5a0 legacy=0x7f4e5406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e4bfff640 1 -- 172.21.15.72:0/252323 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e540cf030 con 0x7f4e540ce8c0 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.047+0000 7f4e52d9d640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5415c980 0x7f4e54071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.048+0000 7f4e4bfff640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce8c0 0x7f4e540cc330 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009870 tx=0x7f4e3c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54960c453be01fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.048+0000 7f4e51d9b640 1 -- 172.21.15.72:0/252323 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c035020 con 0x7f4e540ce8c0 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.048+0000 7f4e51d9b640 1 -- 172.21.15.72:0/252323 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c031640 con 0x7f4e540ce8c0 2024-03-06T07:24:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.048+0000 7f4e51d9b640 1 -- 172.21.15.72:0/252323 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c02d040 con 0x7f4e540ce8c0 2024-03-06T07:24:42.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 -- 172.21.15.72:0/252323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce8c0 msgr2=0x7f4e540cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce8c0 0x7f4e540cc330 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c009870 tx=0x7f4e3c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 -- 172.21.15.72:0/252323 shutdown_connections 2024-03-06T07:24:42.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5415c980 0x7f4e54071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 --2- 172.21.15.72:0/252323 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce8c0 0x7f4e540cc330 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 -- 172.21.15.72:0/252323 >> 172.21.15.72:0/252323 conn(0x7f4e540b1650 msgr2=0x7f4e540b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 -- 172.21.15.72:0/252323 shutdown_connections 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.049+0000 7f4e58925640 1 -- 172.21.15.72:0/252323 wait complete. 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.050+0000 7f4e58925640 1 Processor -- start 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.050+0000 7f4e58925640 1 -- start start 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e58925640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 0x7f4e54156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e58925640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e5415c980 0x7f4e54156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e58925640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5413b630 0x7f4e54158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e58925640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e540cc870 con 0x7f4e5415c5a0 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e58925640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e54070e70 con 0x7f4e5415c980 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e58925640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5415ce30 con 0x7f4e5413b630 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 0x7f4e54156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 0x7f4e54156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36734/0 (socket says 172.21.15.72:36734) 2024-03-06T07:24:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 -- 172.21.15.72:0/2643642845 learned_addr learned my addr 172.21.15.72:0/2643642845 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e52d9d640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5413b630 0x7f4e54158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 -- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5413b630 msgr2=0x7f4e54158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e4bfff640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e5415c980 0x7f4e54156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5413b630 0x7f4e54158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 -- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e5415c980 msgr2=0x7f4e54156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e5415c980 0x7f4e54156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 -- 172.21.15.72:0/2643642845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e540aef90 con 0x7f4e5415c5a0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e5259c640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 0x7f4e54156600 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400c430 tx=0x7f4e4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.051+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440052d0 con 0x7f4e5415c5a0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.052+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e44005490 con 0x7f4e5415c5a0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.052+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e44007260 con 0x7f4e5415c5a0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.052+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e54159220 con 0x7f4e5415c5a0 2024-03-06T07:24:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.052+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e541597e0 con 0x7f4e5415c5a0 2024-03-06T07:24:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.053+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e540687d0 con 0x7f4e5415c5a0 2024-03-06T07:24:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.053+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e44007400 con 0x7f4e5415c5a0 2024-03-06T07:24:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.053+0000 7f4e4affd640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e3406b4d0 0x7f4e3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.054+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e44089740 con 0x7f4e5415c5a0 2024-03-06T07:24:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.054+0000 7f4e4bfff640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e3406b4d0 0x7f4e3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.054+0000 7f4e4bfff640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e3406b4d0 0x7f4e3406d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c036000 tx=0x7f4e3c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.058+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e540aef90 con 0x7f4e5415c5a0 2024-03-06T07:24:42.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.228+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4e54155ca0 con 0x7f4e5415c5a0 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.881+0000 7f4e4affd640 1 -- 172.21.15.72:0/2643642845 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4e44058560 con 0x7f4e5415c5a0 2024-03-06T07:24:42.883 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.884+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e3406b4d0 msgr2=0x7f4e3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.884+0000 7f4e58925640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e3406b4d0 0x7f4e3406d990 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c036000 tx=0x7f4e3c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.884+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 msgr2=0x7f4e54156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.884+0000 7f4e58925640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 0x7f4e54156600 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f4e4400c430 tx=0x7f4e4400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 shutdown_connections 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e3406b4d0 0x7f4e3406d990 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e5413b630 0x7f4e54158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e5415c980 0x7f4e54156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 --2- 172.21.15.72:0/2643642845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415c5a0 0x7f4e54156600 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:42.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 >> 172.21.15.72:0/2643642845 conn(0x7f4e540b1650 msgr2=0x7f4e540aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:42.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 shutdown_connections 2024-03-06T07:24:42.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:42.885+0000 7f4e58925640 1 -- 172.21.15.72:0/2643642845 wait complete. 2024-03-06T07:24:42.901 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:24:43.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.003+0000 7f4aaa93e640 1 Processor -- start 2024-03-06T07:24:43.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.003+0000 7f4aaa93e640 1 -- start start 2024-03-06T07:24:43.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aaa93e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4aa40708f0 0x7f4aa40d0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aaa93e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa406f420 con 0x7f4aa415bdd0 2024-03-06T07:24:43.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aaa93e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa406f700 con 0x7f4aa41602b0 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aaa93e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa415da40 con 0x7f4aa40708f0 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aa8edb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4aa40708f0 0x7f4aa40d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aa3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4aa415bdd0 0x7f4aa415d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49658/0 (socket says 172.21.15.72:49658) 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.004+0000 7f4aa3fff640 1 -- 172.21.15.72:0/2370710071 learned_addr learned my addr 172.21.15.72:0/2370710071 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa8edb640 1 -- 172.21.15.72:0/2370710071 >> v1:172.21.15.146:6789/0 conn(0x7f4aa41602b0 legacy=0x7f4aa40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa8edb640 1 -- 172.21.15.72:0/2370710071 >> v1:172.21.15.72:6789/0 conn(0x7f4aa415bdd0 legacy=0x7f4aa415d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa8edb640 1 -- 172.21.15.72:0/2370710071 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa406f9e0 con 0x7f4aa40708f0 2024-03-06T07:24:43.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa2ffd640 1 -- 172.21.15.72:0/2370710071 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972400637 0 0) 0x7f4aa406f420 con 0x7f4aa415bdd0 2024-03-06T07:24:43.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa8edb640 1 --2- 172.21.15.72:0/2370710071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa40708f0 0x7f4aa40d0130 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009d90 tx=0x7f4a9c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf0f381ea42ab63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa2ffd640 1 -- 172.21.15.72:0/2370710071 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c039070 con 0x7f4aa40708f0 2024-03-06T07:24:43.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa2ffd640 1 -- 172.21.15.72:0/2370710071 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c02cc60 con 0x7f4aa40708f0 2024-03-06T07:24:43.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.005+0000 7f4aa2ffd640 1 -- 172.21.15.72:0/2370710071 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c034660 con 0x7f4aa40708f0 2024-03-06T07:24:43.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 -- 172.21.15.72:0/2370710071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa40708f0 msgr2=0x7f4aa40d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/2370710071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa40708f0 0x7f4aa40d0130 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009d90 tx=0x7f4a9c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 -- 172.21.15.72:0/2370710071 shutdown_connections 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/2370710071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa40708f0 0x7f4aa40d0130 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 -- 172.21.15.72:0/2370710071 >> 172.21.15.72:0/2370710071 conn(0x7f4aa409a9f0 msgr2=0x7f4aa409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 -- 172.21.15.72:0/2370710071 shutdown_connections 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.006+0000 7f4aaa93e640 1 -- 172.21.15.72:0/2370710071 wait complete. 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.007+0000 7f4aaa93e640 1 Processor -- start 2024-03-06T07:24:43.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.007+0000 7f4aaa93e640 1 -- start start 2024-03-06T07:24:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aaa93e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4aa40708f0 0x7f4aa40c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aaa93e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa415bdd0 0x7f4aa41f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aaa93e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 0x7f4aa41f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aaa93e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40d0670 con 0x7f4aa41602b0 2024-03-06T07:24:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aaa93e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40d08d0 con 0x7f4aa415bdd0 2024-03-06T07:24:43.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aaa93e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40d0a50 con 0x7f4aa40708f0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 0x7f4aa41f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 0x7f4aa41f7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56330/0 (socket says 172.21.15.72:56330) 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 -- 172.21.15.72:0/3784186450 learned_addr learned my addr 172.21.15.72:0/3784186450 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa37fe640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa415bdd0 0x7f4aa41f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa3fff640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4aa40708f0 0x7f4aa40c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 -- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4aa40708f0 msgr2=0x7f4aa40c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4aa40708f0 0x7f4aa40c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 -- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa415bdd0 msgr2=0x7f4aa41f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa415bdd0 0x7f4aa41f75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa8edb640 1 -- 172.21.15.72:0/3784186450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40b0a60 con 0x7f4aa41602b0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.008+0000 7f4aa3fff640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4aa40708f0 0x7f4aa40c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.009+0000 7f4aa8edb640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 0x7f4aa41f7af0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009d90 tx=0x7f4a9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.009+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c044030 con 0x7f4aa41602b0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.009+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c02ca60 con 0x7f4aa41602b0 2024-03-06T07:24:43.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.009+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa406c1d0 con 0x7f4aa41602b0 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.009+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c039040 con 0x7f4aa41602b0 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.009+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa406c790 con 0x7f4aa41602b0 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.010+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa404f330 con 0x7f4aa41602b0 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.010+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a9c047070 con 0x7f4aa41602b0 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.011+0000 7f4aa0ff9640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a9006b4d0 0x7f4a9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.011+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f4a9c0ac930 con 0x7f4aa41602b0 2024-03-06T07:24:43.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.011+0000 7f4aa3fff640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a9006b4d0 0x7f4a9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:43.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.011+0000 7f4aa3fff640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a9006b4d0 0x7f4a9006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4a94007fd0 tx=0x7f4a94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:43.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.015+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a9c04e070 con 0x7f4aa41602b0 2024-03-06T07:24:43.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.188+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4aa40b7ad0 con 0x7f4aa41602b0 2024-03-06T07:24:43.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.883+0000 7f4aa0ff9640 1 -- 172.21.15.72:0/3784186450 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4a9c07b910 con 0x7f4aa41602b0 2024-03-06T07:24:43.885 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:24:43.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.885+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a9006b4d0 msgr2=0x7f4a9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.885+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a9006b4d0 0x7f4a9006d990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4a94007fd0 tx=0x7f4a94009040 comp rx=0 tx=0).stop 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.885+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 msgr2=0x7f4aa41f7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.885+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 0x7f4aa41f7af0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c009d90 tx=0x7f4a9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.886+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 shutdown_connections 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.886+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a9006b4d0 0x7f4a9006d990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.886+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aa41602b0 0x7f4aa41f7af0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.886+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4aa415bdd0 0x7f4aa41f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.886+0000 7f4aaa93e640 1 --2- 172.21.15.72:0/3784186450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4aa40708f0 0x7f4aa40c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:43.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.886+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 >> 172.21.15.72:0/3784186450 conn(0x7f4aa409a9f0 msgr2=0x7f4aa40b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:43.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.887+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 shutdown_connections 2024-03-06T07:24:43.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:43.887+0000 7f4aaa93e640 1 -- 172.21.15.72:0/3784186450 wait complete. 2024-03-06T07:24:43.903 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:24:44.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.005+0000 7f202b35c640 1 Processor -- start 2024-03-06T07:24:44.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.005+0000 7f202b35c640 1 -- start start 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f202b35c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 0x7f20240b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f202b35c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2024066d90 0x7f20240b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f202b35c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f202406b710 con 0x7f20240b5410 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f202b35c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f202406b150 con 0x7f2024066d90 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f202b35c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202406b430 con 0x7f20240cf9a0 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f20298f9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f20240b5410 0x7f2024072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49662/0 (socket says 172.21.15.72:49662) 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f20298f9640 1 -- 172.21.15.72:0/1747891661 learned_addr learned my addr 172.21.15.72:0/1747891661 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844543142 0 0) 0x7f202406b710 con 0x7f20240b5410 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2010003520 con 0x7f20240b5410 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f20288f7640 1 --2- 172.21.15.72:0/1747891661 >> v2:172.21.15.146:3300/0 conn(0x7f2024066d90 0x7f20240b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f20290f8640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 0x7f20240b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476370797 0 0) 0x7f2010003520 con 0x7f20240b5410 2024-03-06T07:24:44.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 msgr2=0x7f20240b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 0x7f20240b5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 >> v2:172.21.15.146:3300/0 conn(0x7f2024066d90 msgr2=0x7f20240b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 --2- 172.21.15.72:0/1747891661 >> v2:172.21.15.146:3300/0 conn(0x7f2024066d90 0x7f20240b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240b0970 con 0x7f20240b5410 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f201c0019f0 con 0x7f20240b5410 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.006+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 >> v1:172.21.15.72:6789/0 conn(0x7f20240b5410 legacy=0x7f2024072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f201bfff640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100039b0 0x7f2010003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f201bfff640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2010004800 0x7f2010004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f201bfff640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 0x7f20100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202406b150 con 0x7f2010004800 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202406b430 con 0x7f20100039b0 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201000ba70 con 0x7f20240cf9a0 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20288f7640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2010004800 0x7f2010004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20290f8640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100039b0 0x7f2010003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20298f9640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 0x7f20100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20288f7640 1 -- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 msgr2=0x7f20100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20288f7640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20240cf9a0 0x7f20100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20288f7640 1 -- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100039b0 msgr2=0x7f2010003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20288f7640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100039b0 0x7f2010003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.007+0000 7f20288f7640 1 -- 172.21.15.72:0/1747891661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010003520 con 0x7f2010004800 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f20288f7640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2010004800 0x7f2010004c80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f20240671b0 tx=0x7f2010005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918582876265727b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2014036020 con 0x7f2010004800 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20140325f0 con 0x7f2010004800 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f201bfff640 1 -- 172.21.15.72:0/1747891661 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201400b600 con 0x7f2010004800 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f202b35c640 1 -- 172.21.15.72:0/1747891661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2010004800 msgr2=0x7f2010004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f202b35c640 1 --2- 172.21.15.72:0/1747891661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2010004800 0x7f2010004c80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f20240671b0 tx=0x7f2010005830 comp rx=0 tx=0).stop 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f20298f9640 1 -- 172.21.15.72:0/1747891661 reap_dead start 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f202b35c640 1 -- 172.21.15.72:0/1747891661 shutdown_connections 2024-03-06T07:24:44.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f202b35c640 1 -- 172.21.15.72:0/1747891661 >> 172.21.15.72:0/1747891661 conn(0x7f20240b1d70 msgr2=0x7f20240b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f202b35c640 1 -- 172.21.15.72:0/1747891661 shutdown_connections 2024-03-06T07:24:44.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.008+0000 7f202b35c640 1 -- 172.21.15.72:0/1747891661 wait complete. 2024-03-06T07:24:44.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 Processor -- start 2024-03-06T07:24:44.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 -- start start 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010004800 0x7f20240cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20240ccaf0 0x7f20241e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 0x7f20241e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240cf1d0 con 0x7f20241e72e0 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20240cf050 con 0x7f20240ccaf0 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f202b35c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20240cfd80 con 0x7f2010004800 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f20298f9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 0x7f20241e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f20298f9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 0x7f20241e7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56344/0 (socket says 172.21.15.72:56344) 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f20298f9640 1 -- 172.21.15.72:0/345113833 learned_addr learned my addr 172.21.15.72:0/345113833 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f20290f8640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010004800 0x7f20240cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.010+0000 7f20288f7640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20240ccaf0 0x7f20241e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f20298f9640 1 -- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010004800 msgr2=0x7f20240cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f20298f9640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010004800 0x7f20240cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f20298f9640 1 -- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20240ccaf0 msgr2=0x7f20241e6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f20298f9640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20240ccaf0 0x7f20241e6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f20298f9640 1 -- 172.21.15.72:0/345113833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20240ccf70 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f20298f9640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 0x7f20241e7760 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f20240710f0 tx=0x7f201c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c00ec10 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201c018070 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20240cbbd0 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c013420 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.011+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f202415f980 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.012+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f202409e3d0 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.012+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f201c0135c0 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.013+0000 7f2019ffb640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f200806b500 0x7f200806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.013+0000 7f20290f8640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f200806b500 0x7f200806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.013+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f201c08a110 con 0x7f20241e72e0 2024-03-06T07:24:44.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.013+0000 7f20290f8640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f200806b500 0x7f200806d9c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2020006fd0 tx=0x7f2020008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.017+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f201c058cd0 con 0x7f20241e72e0 2024-03-06T07:24:44.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.189+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2024068ff0 con 0x7f20241e72e0 2024-03-06T07:24:44.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.192+0000 7f2019ffb640 1 -- 172.21.15.72:0/345113833 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f201c05cd70 con 0x7f20241e72e0 2024-03-06T07:24:44.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f200806b500 msgr2=0x7f200806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f200806b500 0x7f200806d9c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2020006fd0 tx=0x7f2020008040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 msgr2=0x7f20241e7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 0x7f20241e7760 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f20240710f0 tx=0x7f201c00e530 comp rx=0 tx=0).stop 2024-03-06T07:24:44.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 shutdown_connections 2024-03-06T07:24:44.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f200806b500 0x7f200806d9c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20241e72e0 0x7f20241e7760 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20240ccaf0 0x7f20241e6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 --2- 172.21.15.72:0/345113833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010004800 0x7f20240cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 >> 172.21.15.72:0/345113833 conn(0x7f20240b1d70 msgr2=0x7f2024157540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 shutdown_connections 2024-03-06T07:24:44.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.195+0000 7f202b35c640 1 -- 172.21.15.72:0/345113833 wait complete. 2024-03-06T07:24:44.208 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-03-06T07:24:44.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.308+0000 7f3899455640 1 Processor -- start 2024-03-06T07:24:44.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.308+0000 7f3899455640 1 -- start start 2024-03-06T07:24:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3899455640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38940d4860 0x7f38940d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3899455640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38941618b0 con 0x7f38941614d0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3899455640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38940cf400 con 0x7f38940b0e30 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3899455640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38940d4cf0 con 0x7f38940d4860 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3892ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f38941614d0 0x7f38940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49666/0 (socket says 172.21.15.72:49666) 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3892ffd640 1 -- 172.21.15.72:0/3168162736 learned_addr learned my addr 172.21.15.72:0/3168162736 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f38927fc640 1 --2- 172.21.15.72:0/3168162736 >> v2:172.21.15.146:3300/0 conn(0x7f38940d4860 0x7f38940d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147296220 0 0) 0x7f38941618b0 con 0x7f38941614d0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f387c003520 con 0x7f38941614d0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 953555773 0 0) 0x7f387c003520 con 0x7f38941614d0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 >> v2:172.21.15.146:3300/0 conn(0x7f38940d4860 msgr2=0x7f38940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 --2- 172.21.15.72:0/3168162736 >> v2:172.21.15.146:3300/0 conn(0x7f38940d4860 0x7f38940d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 >> v1:172.21.15.146:6789/0 conn(0x7f38940b0e30 legacy=0x7f38940b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940cf6a0 con 0x7f38941614d0 2024-03-06T07:24:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3884001690 con 0x7f38941614d0 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 >> v1:172.21.15.72:6789/0 conn(0x7f38941614d0 legacy=0x7f38940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c0039b0 0x7f387c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f387c004800 0x7f387c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.309+0000 7f3891ffb640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387c0051c0 0x7f387c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38940d4cf0 con 0x7f387c004800 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c00c060 con 0x7f387c0039b0 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387c00c260 con 0x7f387c0051c0 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38927fc640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f387c004800 0x7f387c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f3892ffd640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c0039b0 0x7f387c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38937fe640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387c0051c0 0x7f387c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38927fc640 1 -- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387c0051c0 msgr2=0x7f387c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38927fc640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f387c0051c0 0x7f387c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38927fc640 1 -- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c0039b0 msgr2=0x7f387c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38927fc640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c0039b0 0x7f387c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38927fc640 1 -- 172.21.15.72:0/3168162736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c003520 con 0x7f387c004800 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.310+0000 7f38937fe640 1 -- 172.21.15.72:0/3168162736 reap_dead start 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f38927fc640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f387c004800 0x7f387c004c80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3888009ac0 tx=0x7f387c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847db98a224b9356 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888039070 con 0x7f387c004800 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388802c6b0 con 0x7f387c004800 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3891ffb640 1 -- 172.21.15.72:0/3168162736 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888009690 con 0x7f387c004800 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 -- 172.21.15.72:0/3168162736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f387c004800 msgr2=0x7f387c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f387c004800 0x7f387c004c80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3888009ac0 tx=0x7f387c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 -- 172.21.15.72:0/3168162736 shutdown_connections 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 --2- 172.21.15.72:0/3168162736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f387c004800 0x7f387c004c80 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 -- 172.21.15.72:0/3168162736 >> 172.21.15.72:0/3168162736 conn(0x7f38940d3d40 msgr2=0x7f38940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 -- 172.21.15.72:0/3168162736 shutdown_connections 2024-03-06T07:24:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.311+0000 7f3899455640 1 -- 172.21.15.72:0/3168162736 wait complete. 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 Processor -- start 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 -- start start 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c004800 0x7f3894154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 0x7f38940c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38940ca0b0 0x7f38940c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38940b0910 con 0x7f38940c9c30 2024-03-06T07:24:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38940cfc60 con 0x7f387c004800 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f3899455640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38940d1910 con 0x7f38940ca0b0 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f38927fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 0x7f38940c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f38927fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 0x7f38940c8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56366/0 (socket says 172.21.15.72:56366) 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.313+0000 7f38927fc640 1 -- 172.21.15.72:0/1363141085 learned_addr learned my addr 172.21.15.72:0/1363141085 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f3892ffd640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c004800 0x7f3894154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38937fe640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38940ca0b0 0x7f38940c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38927fc640 1 -- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38940ca0b0 msgr2=0x7f38940c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38927fc640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38940ca0b0 0x7f38940c8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38927fc640 1 -- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c004800 msgr2=0x7f3894154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38927fc640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c004800 0x7f3894154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38927fc640 1 -- 172.21.15.72:0/1363141085 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38940c8b90 con 0x7f38940c9c30 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f38927fc640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 0x7f38940c8010 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3888009be0 tx=0x7f3888004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888042020 con 0x7f38940c9c30 2024-03-06T07:24:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388803a040 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38940c8e70 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888039070 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.314+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38940c9430 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.315+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3894068ff0 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.315+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3888045020 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.316+0000 7f387b7fe640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f388006b500 0x7f388006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.316+0000 7f3892ffd640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f388006b500 0x7f388006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.316+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f38880aac80 con 0x7f38940c9c30 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.316+0000 7f3892ffd640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f388006b500 0x7f388006d9c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f38940c7340 tx=0x7f3884009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.320+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3888079e90 con 0x7f38940c9c30 2024-03-06T07:24:44.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.492+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f38940d2a20 con 0x7f38940c9c30 2024-03-06T07:24:44.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.496+0000 7f387b7fe640 1 -- 172.21.15.72:0/1363141085 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7f3888079790 con 0x7f38940c9c30 2024-03-06T07:24:44.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f388006b500 msgr2=0x7f388006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f388006b500 0x7f388006d9c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f38940c7340 tx=0x7f3884009040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 msgr2=0x7f38940c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 0x7f38940c8010 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3888009be0 tx=0x7f3888004730 comp rx=0 tx=0).stop 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 shutdown_connections 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f388006b500 0x7f388006d9c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f38940ca0b0 0x7f38940c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f38940c9c30 0x7f38940c8010 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 --2- 172.21.15.72:0/1363141085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f387c004800 0x7f3894154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 >> 172.21.15.72:0/1363141085 conn(0x7f38940d3d40 msgr2=0x7f38940b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 shutdown_connections 2024-03-06T07:24:44.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.498+0000 7f3899455640 1 -- 172.21.15.72:0/1363141085 wait complete. 2024-03-06T07:24:44.513 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:24:44.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.612+0000 7f27e23d0640 1 Processor -- start 2024-03-06T07:24:44.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.612+0000 7f27e23d0640 1 -- start start 2024-03-06T07:24:44.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27e23d0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27dc0cf9a0 0x7f27dc0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27e23d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc066d90 0x7f27dc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27e23d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27dc06b710 con 0x7f27dc0b5410 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27e23d0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27dc06b150 con 0x7f27dc0cf9a0 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27e23d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc06b430 con 0x7f27dc066d90 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27dc0cf9a0 0x7f27dc0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27dc0cf9a0 0x7f27dc0b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56372/0 (socket says 172.21.15.72:56372) 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 -- 172.21.15.72:0/1021516731 learned_addr learned my addr 172.21.15.72:0/1021516731 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27db7fe640 1 --2- 172.21.15.72:0/1021516731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc066d90 0x7f27dc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 -- 172.21.15.72:0/1021516731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc066d90 msgr2=0x7f27dc0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 --2- 172.21.15.72:0/1021516731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc066d90 0x7f27dc0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 -- 172.21.15.72:0/1021516731 >> v1:172.21.15.146:6789/0 conn(0x7f27dc0b5410 legacy=0x7f27dc072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.613+0000 7f27dbfff640 1 -- 172.21.15.72:0/1021516731 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0b0970 con 0x7f27dc0cf9a0 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.614+0000 7f27dbfff640 1 --2- 172.21.15.72:0/1021516731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0cf9a0 0x7f27dc0b5ab0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009870 tx=0x7f27cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9fc83e35aaaa30 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.614+0000 7f27daffd640 1 -- 172.21.15.72:0/1021516731 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc034070 con 0x7f27dc0cf9a0 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.614+0000 7f27daffd640 1 -- 172.21.15.72:0/1021516731 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc030840 con 0x7f27dc0cf9a0 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.614+0000 7f27daffd640 1 -- 172.21.15.72:0/1021516731 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc030c10 con 0x7f27dc0cf9a0 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 -- 172.21.15.72:0/1021516731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0cf9a0 msgr2=0x7f27dc0b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 --2- 172.21.15.72:0/1021516731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0cf9a0 0x7f27dc0b5ab0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009870 tx=0x7f27cc02b980 comp rx=0 tx=0).stop 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 -- 172.21.15.72:0/1021516731 shutdown_connections 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 --2- 172.21.15.72:0/1021516731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc066d90 0x7f27dc0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 --2- 172.21.15.72:0/1021516731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0cf9a0 0x7f27dc0b5ab0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 -- 172.21.15.72:0/1021516731 >> 172.21.15.72:0/1021516731 conn(0x7f27dc0b1d70 msgr2=0x7f27dc0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 -- 172.21.15.72:0/1021516731 shutdown_connections 2024-03-06T07:24:44.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.615+0000 7f27e23d0640 1 -- 172.21.15.72:0/1021516731 wait complete. 2024-03-06T07:24:44.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.616+0000 7f27e23d0640 1 Processor -- start 2024-03-06T07:24:44.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.616+0000 7f27e23d0640 1 -- start start 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e23d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27dc066d90 0x7f27dc14b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e23d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 0x7f27dc15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e23d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc0cf9a0 0x7f27dc0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e23d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc0b6170 con 0x7f27dc0b5410 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e23d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc0b5ff0 con 0x7f27dc066d90 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e23d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc0cf050 con 0x7f27dc0cf9a0 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 0x7f27dc15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 0x7f27dc15eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56384/0 (socket says 172.21.15.72:56384) 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 -- 172.21.15.72:0/2359695550 learned_addr learned my addr 172.21.15.72:0/2359695550 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:44.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27dbfff640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27dc066d90 0x7f27dc14b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27e096d640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc0cf9a0 0x7f27dc0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 -- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc0cf9a0 msgr2=0x7f27dc0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc0cf9a0 0x7f27dc0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 -- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27dc066d90 msgr2=0x7f27dc14b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27dc066d90 0x7f27dc14b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 -- 172.21.15.72:0/2359695550 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc15f8e0 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27db7fe640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 0x7f27dc15eeb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f27d000d800 tx=0x7f27d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.617+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d000b6c0 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.618+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d0019070 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.618+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc15fbc0 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.618+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d000ba80 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.618+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc160180 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.619+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc09e3d0 con 0x7f27dc0b5410 2024-03-06T07:24:44.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.619+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27d000bc20 con 0x7f27dc0b5410 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.619+0000 7f27d8ff9640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27c806b4d0 0x7f27c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.620+0000 7f27dbfff640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27c806b4d0 0x7f27c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.620+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f27d008a3b0 con 0x7f27dc0b5410 2024-03-06T07:24:44.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.620+0000 7f27dbfff640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27c806b4d0 0x7f27c806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f27cc036000 tx=0x7f27cc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.624+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27d0059530 con 0x7f27dc0b5410 2024-03-06T07:24:44.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.796+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f27dc068ff0 con 0x7f27dc0b5410 2024-03-06T07:24:44.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.802+0000 7f27d8ff9640 1 -- 172.21.15.72:0/2359695550 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v21) v1 ==== 108+0+0 (secure 0 0 0) 0x7f27d0058e30 con 0x7f27dc0b5410 2024-03-06T07:24:44.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27c806b4d0 msgr2=0x7f27c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27c806b4d0 0x7f27c806d990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f27cc036000 tx=0x7f27cc03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:44.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 msgr2=0x7f27dc15eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 0x7f27dc15eeb0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f27d000d800 tx=0x7f27d000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 shutdown_connections 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27c806b4d0 0x7f27c806d990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27dc0cf9a0 0x7f27dc0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27dc0b5410 0x7f27dc15eeb0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 --2- 172.21.15.72:0/2359695550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27dc066d90 0x7f27dc14b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 >> 172.21.15.72:0/2359695550 conn(0x7f27dc0b1d70 msgr2=0x7f27dc161670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 shutdown_connections 2024-03-06T07:24:44.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.804+0000 7f27e23d0640 1 -- 172.21.15.72:0/2359695550 wait complete. 2024-03-06T07:24:44.817 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-03-06T07:24:44.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.917+0000 7f4bdf881640 1 Processor -- start 2024-03-06T07:24:44.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.917+0000 7f4bdf881640 1 -- start start 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdf881640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4bd80b3060 0x7f4bd815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdf881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 0x7f4bd815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdf881640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bd815f2d0 con 0x7f4bd813dc10 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdf881640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bd80cf3f0 con 0x7f4bd80b3060 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdf881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd815b210 con 0x7f4bd815bf30 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdd61d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4bd813dc10 0x7f4bd80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49668/0 (socket says 172.21.15.72:49668) 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdd61d640 1 -- 172.21.15.72:0/1405171013 learned_addr learned my addr 172.21.15.72:0/1405171013 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394770550 0 0) 0x7f4bd815f2d0 con 0x7f4bd813dc10 2024-03-06T07:24:44.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdde1e640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 0x7f4bd815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bdce1c640 1 --2- 172.21.15.72:0/1405171013 >> v2:172.21.15.146:3300/0 conn(0x7f4bd80b3060 0x7f4bd815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.918+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc4003520 con 0x7f4bd813dc10 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602077511 0 0) 0x7f4bc4003520 con 0x7f4bd813dc10 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 msgr2=0x7f4bd815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 0x7f4bd815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 >> v2:172.21.15.146:3300/0 conn(0x7f4bd80b3060 msgr2=0x7f4bd815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 --2- 172.21.15.72:0/1405171013 >> v2:172.21.15.146:3300/0 conn(0x7f4bd80b3060 0x7f4bd815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd815f580 con 0x7f4bd813dc10 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f4bd40019f0 con 0x7f4bd813dc10 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 >> v1:172.21.15.72:6789/0 conn(0x7f4bd813dc10 legacy=0x7f4bd80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc40039b0 0x7f4bc4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc4004800 0x7f4bc4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 0x7f4bc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd80cf3f0 con 0x7f4bc40039b0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd815b210 con 0x7f4bc4004800 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc400b9c0 con 0x7f4bd815bf30 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc40039b0 0x7f4bc4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdce1c640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc4004800 0x7f4bc4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdde1e640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 0x7f4bc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdd61d640 1 -- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 msgr2=0x7f4bc40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd815bf30 0x7f4bc40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdd61d640 1 -- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc4004800 msgr2=0x7f4bc4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc4004800 0x7f4bc4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.919+0000 7f4bdd61d640 1 -- 172.21.15.72:0/1405171013 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc4003520 con 0x7f4bc40039b0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.920+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc40039b0 0x7f4bc4003dd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400a950 tx=0x7f4bc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47305caa5369240 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.920+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400a4c0 con 0x7f4bc40039b0 2024-03-06T07:24:44.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.920+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd4034650 con 0x7f4bc40039b0 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.920+0000 7f4bcffff640 1 -- 172.21.15.72:0/1405171013 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4034a40 con 0x7f4bc40039b0 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdf881640 1 -- 172.21.15.72:0/1405171013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc40039b0 msgr2=0x7f4bc4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdf881640 1 --2- 172.21.15.72:0/1405171013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bc40039b0 0x7f4bc4003dd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400a950 tx=0x7f4bc4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdde1e640 1 -- 172.21.15.72:0/1405171013 reap_dead start 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdf881640 1 -- 172.21.15.72:0/1405171013 shutdown_connections 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdf881640 1 -- 172.21.15.72:0/1405171013 >> 172.21.15.72:0/1405171013 conn(0x7f4bd80cf8a0 msgr2=0x7f4bd80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdf881640 1 -- 172.21.15.72:0/1405171013 shutdown_connections 2024-03-06T07:24:44.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.921+0000 7f4bdf881640 1 -- 172.21.15.72:0/1405171013 wait complete. 2024-03-06T07:24:44.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.922+0000 7f4bdf881640 1 Processor -- start 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.922+0000 7f4bdf881640 1 -- start start 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.922+0000 7f4bdf881640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc40039b0 0x7f4bd813a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.922+0000 7f4bdf881640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 0x7f4bd8140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdf881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd8142550 0x7f4bd8140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdf881640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd813dff0 con 0x7f4bd813d2a0 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdf881640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd815b730 con 0x7f4bc40039b0 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdf881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bd815ed90 con 0x7f4bd8142550 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 0x7f4bd8140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 0x7f4bd8140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56394/0 (socket says 172.21.15.72:56394) 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 -- 172.21.15.72:0/941696734 learned_addr learned my addr 172.21.15.72:0/941696734 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc40039b0 0x7f4bd813a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdde1e640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd8142550 0x7f4bd8140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 -- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd8142550 msgr2=0x7f4bd8140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd8142550 0x7f4bd8140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 -- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc40039b0 msgr2=0x7f4bd813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc40039b0 0x7f4bd813a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 -- 172.21.15.72:0/941696734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8142b10 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdce1c640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 0x7f4bd8140600 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8011960 tx=0x7f4bc8011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8009590 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bc800f9d0 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bd8142df0 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.923+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800fdc0 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.924+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bd81433b0 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.925+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bd804f330 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.925+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4bc8012040 con 0x7f4bd813d2a0 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.925+0000 7f4bcdffb640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:44.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.925+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4bc808a210 con 0x7f4bd813d2a0 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.925+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.926+0000 7f4bdd61d640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4034000 tx=0x7f4bd4036040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:44.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:44.930+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4bc8019030 con 0x7f4bd813d2a0 2024-03-06T07:24:45.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.171+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) v1 -- 0x7f4bd806eb00 con 0x7f4bd813d2a0 2024-03-06T07:24:45.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.893+0000 7f4bcdffb640 1 -- 172.21.15.72:0/941696734 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v172) v1 ==== 144+0+0 (secure 0 0 0) 0x7f4bc8059390 con 0x7f4bd813d2a0 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4bbc06b500 msgr2=0x7f4bbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4034000 tx=0x7f4bd4036040 comp rx=0 tx=0).stop 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 msgr2=0x7f4bd8140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 0x7f4bd8140600 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4bc8011960 tx=0x7f4bc8011d60 comp rx=0 tx=0).stop 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 shutdown_connections 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4bbc06b500 0x7f4bbc06d9c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4bd8142550 0x7f4bd8140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4bd813d2a0 0x7f4bd8140600 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 --2- 172.21.15.72:0/941696734 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4bc40039b0 0x7f4bd813a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 >> 172.21.15.72:0/941696734 conn(0x7f4bd80cf8a0 msgr2=0x7f4bd815c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 shutdown_connections 2024-03-06T07:24:45.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:45.897+0000 7f4bdf881640 1 -- 172.21.15.72:0/941696734 wait complete. 2024-03-06T07:24:45.912 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-03-06T07:24:45.912 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-03-06T07:24:45.912 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:24:45.913 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:46.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.017+0000 7f9e4474b640 1 Processor -- start 2024-03-06T07:24:46.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.017+0000 7f9e4474b640 1 -- start start 2024-03-06T07:24:46.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e4474b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9e3c06c6d0 0x7f9e3c15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e4474b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3c0cd450 con 0x7f9e3c0b4f70 2024-03-06T07:24:46.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e4474b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3c0cd730 con 0x7f9e3c066d90 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e4474b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e3c15e9f0 con 0x7f9e3c06c6d0 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e41ce6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9e3c06c6d0 0x7f9e3c15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e424e7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9e3c0b4f70 0x7f9e3c0b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49670/0 (socket says 172.21.15.72:49670) 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.018+0000 7f9e424e7640 1 -- 172.21.15.72:0/764856605 learned_addr learned my addr 172.21.15.72:0/764856605 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e41ce6640 1 -- 172.21.15.72:0/764856605 >> v1:172.21.15.146:6789/0 conn(0x7f9e3c066d90 legacy=0x7f9e3c0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e41ce6640 1 -- 172.21.15.72:0/764856605 >> v1:172.21.15.72:6789/0 conn(0x7f9e3c0b4f70 legacy=0x7f9e3c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e41ce6640 1 -- 172.21.15.72:0/764856605 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c15e680 con 0x7f9e3c06c6d0 2024-03-06T07:24:46.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e41ce6640 1 --2- 172.21.15.72:0/764856605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c06c6d0 0x7f9e3c15ee80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0097a0 tx=0x7f9e2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec75719a73552fe server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e414e5640 1 -- 172.21.15.72:0/764856605 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c035020 con 0x7f9e3c06c6d0 2024-03-06T07:24:46.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e414e5640 1 -- 172.21.15.72:0/764856605 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e2c031830 con 0x7f9e3c06c6d0 2024-03-06T07:24:46.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.019+0000 7f9e414e5640 1 -- 172.21.15.72:0/764856605 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c031c00 con 0x7f9e3c06c6d0 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 -- 172.21.15.72:0/764856605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c06c6d0 msgr2=0x7f9e3c15ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 --2- 172.21.15.72:0/764856605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c06c6d0 0x7f9e3c15ee80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0097a0 tx=0x7f9e2c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 -- 172.21.15.72:0/764856605 shutdown_connections 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 --2- 172.21.15.72:0/764856605 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c06c6d0 0x7f9e3c15ee80 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 -- 172.21.15.72:0/764856605 >> 172.21.15.72:0/764856605 conn(0x7f9e3c0724d0 msgr2=0x7f9e3c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 -- 172.21.15.72:0/764856605 shutdown_connections 2024-03-06T07:24:46.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.020+0000 7f9e4474b640 1 -- 172.21.15.72:0/764856605 wait complete. 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.021+0000 7f9e4474b640 1 Processor -- start 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.021+0000 7f9e4474b640 1 -- start start 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e4474b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 0x7f9e3c1e9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e4474b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e3c0b4f70 0x7f9e3c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e4474b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e3c14e090 0x7f9e3c14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e4474b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c161ad0 con 0x7f9e3c066d90 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e4474b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c0cda10 con 0x7f9e3c14e090 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e4474b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3c161c50 con 0x7f9e3c0b4f70 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 0x7f9e3c1e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 0x7f9e3c1e9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56412/0 (socket says 172.21.15.72:56412) 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 -- 172.21.15.72:0/2549510416 learned_addr learned my addr 172.21.15.72:0/2549510416 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e41ce6640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e3c0b4f70 0x7f9e3c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e42ce8640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e3c14e090 0x7f9e3c14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 -- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e3c0b4f70 msgr2=0x7f9e3c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e3c0b4f70 0x7f9e3c15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 -- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e3c14e090 msgr2=0x7f9e3c14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e3c14e090 0x7f9e3c14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e424e7640 1 -- 172.21.15.72:0/2549510416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3c1e8770 con 0x7f9e3c066d90 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e42ce8640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e3c14e090 0x7f9e3c14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.022+0000 7f9e41ce6640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e3c0b4f70 0x7f9e3c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.023+0000 7f9e424e7640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 0x7f9e3c1e9810 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800c3c0 tx=0x7f9e3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.023+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38017070 con 0x7f9e3c066d90 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.023+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e38005ce0 con 0x7f9e3c066d90 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.023+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e3c1e8a50 con 0x7f9e3c066d90 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.023+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e380072f0 con 0x7f9e3c066d90 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.023+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e3c1e9010 con 0x7f9e3c066d90 2024-03-06T07:24:46.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.024+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3c04f330 con 0x7f9e3c066d90 2024-03-06T07:24:46.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.024+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9e3801a070 con 0x7f9e3c066d90 2024-03-06T07:24:46.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.024+0000 7f9e32ffd640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9e2006b4d0 0x7f9e2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.025+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9e380892e0 con 0x7f9e3c066d90 2024-03-06T07:24:46.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.025+0000 7f9e41ce6640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9e2006b4d0 0x7f9e2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.025+0000 7f9e41ce6640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9e2006b4d0 0x7f9e2006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c036000 tx=0x7f9e2c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.029+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9e38058460 con 0x7f9e3c066d90 2024-03-06T07:24:46.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.202+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e3c0b1860 con 0x7f9e3c066d90 2024-03-06T07:24:46.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.203+0000 7f9e32ffd640 1 -- 172.21.15.72:0/2549510416 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+13032 (secure 0 0 0) 0x7f9e38057d60 con 0x7f9e3c066d90 2024-03-06T07:24:46.205 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:46.205 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":62,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:43.882524+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/3136290264":"2024-03-07T07:24:40.874771+0000","172.21.15.72:6835/3136290264":"2024-03-07T07:24:40.874771+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:46.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9e2006b4d0 msgr2=0x7f9e2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9e2006b4d0 0x7f9e2006d990 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c036000 tx=0x7f9e2c008030 comp rx=0 tx=0).stop 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 msgr2=0x7f9e3c1e9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 0x7f9e3c1e9810 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f9e3800c3c0 tx=0x7f9e3800c880 comp rx=0 tx=0).stop 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 shutdown_connections 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9e2006b4d0 0x7f9e2006d990 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e3c14e090 0x7f9e3c14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e3c0b4f70 0x7f9e3c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 --2- 172.21.15.72:0/2549510416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9e3c066d90 0x7f9e3c1e9810 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.205+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 >> 172.21.15.72:0/2549510416 conn(0x7f9e3c0724d0 msgr2=0x7f9e3c0cca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.206+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 shutdown_connections 2024-03-06T07:24:46.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.206+0000 7f9e4474b640 1 -- 172.21.15.72:0/2549510416 wait complete. 2024-03-06T07:24:46.218 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:46.219 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:46.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.319+0000 7f33a0bf7640 1 Processor -- start 2024-03-06T07:24:46.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.319+0000 7f33a0bf7640 1 -- start start 2024-03-06T07:24:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f33a0bf7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f339c1614d0 0x7f339c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f33a0bf7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f339c0d1910 0x7f339c0d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f33a0bf7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d22e0 0x7f339c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f33a0bf7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c0d2750 con 0x7f339c1614d0 2024-03-06T07:24:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f33a0bf7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339c1618b0 con 0x7f339c0d1910 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f33a0bf7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0cf400 con 0x7f339c0d22e0 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f339a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f339c1614d0 0x7f339c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f339a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f339c1614d0 0x7f339c0d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56414/0 (socket says 172.21.15.72:56414) 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.320+0000 7f339a59c640 1 -- 172.21.15.72:0/2279264931 learned_addr learned my addr 172.21.15.72:0/2279264931 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f3399d9b640 1 --2- 172.21.15.72:0/2279264931 >> v2:172.21.15.146:3300/0 conn(0x7f339c0d1910 0x7f339c0d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339ad9d640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d22e0 0x7f339c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339a59c640 1 -- 172.21.15.72:0/2279264931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d22e0 msgr2=0x7f339c0d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339a59c640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d22e0 0x7f339c0d06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339a59c640 1 -- 172.21.15.72:0/2279264931 >> v2:172.21.15.146:3300/0 conn(0x7f339c0d1910 msgr2=0x7f339c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339a59c640 1 --2- 172.21.15.72:0/2279264931 >> v2:172.21.15.146:3300/0 conn(0x7f339c0d1910 0x7f339c0d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339a59c640 1 -- 172.21.15.72:0/2279264931 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0cf6a0 con 0x7f339c1614d0 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339ad9d640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d22e0 0x7f339c0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339a59c640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c0d4860 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f339c0ce550 tx=0x7f338c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6e081a23f7bf4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339959a640 1 -- 172.21.15.72:0/2279264931 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c02ca70 con 0x7f339c1614d0 2024-03-06T07:24:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.321+0000 7f339959a640 1 -- 172.21.15.72:0/2279264931 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c02cc10 con 0x7f339c1614d0 2024-03-06T07:24:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f339959a640 1 -- 172.21.15.72:0/2279264931 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0343f0 con 0x7f339c1614d0 2024-03-06T07:24:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2279264931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 msgr2=0x7f339c0d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c0d4860 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f339c0ce550 tx=0x7f338c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2279264931 shutdown_connections 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d22e0 0x7f339c0d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2279264931 >> v2:172.21.15.146:3300/0 conn(0x7f339c0d1910 0x7f339c0d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2279264931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c0d4860 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2279264931 >> 172.21.15.72:0/2279264931 conn(0x7f339c0d3d40 msgr2=0x7f339c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2279264931 shutdown_connections 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.322+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2279264931 wait complete. 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.323+0000 7f33a0bf7640 1 Processor -- start 2024-03-06T07:24:46.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 -- start start 2024-03-06T07:24:46.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d1910 0x7f339c0c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f339c0d22e0 0x7f339c0b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c1f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0b0910 con 0x7f339c1614d0 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0cfc60 con 0x7f339c0d22e0 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f33a0bf7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339c0d0be0 con 0x7f339c0d1910 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f339ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c1f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f339ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c1f3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56420/0 (socket says 172.21.15.72:56420) 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f339ad9d640 1 -- 172.21.15.72:0/2147573752 learned_addr learned my addr 172.21.15.72:0/2147573752 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.324+0000 7f3399d9b640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f339c0d22e0 0x7f339c0b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339a59c640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d1910 0x7f339c0c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339ad9d640 1 -- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d1910 msgr2=0x7f339c0c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339ad9d640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d1910 0x7f339c0c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339ad9d640 1 -- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f339c0d22e0 msgr2=0x7f339c0b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339ad9d640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f339c0d22e0 0x7f339c0b7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339ad9d640 1 -- 172.21.15.72:0/2147573752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c0ca9a0 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339a59c640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d1910 0x7f339c0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f339ad9d640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c1f3060 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f339400c820 tx=0x7f339400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394015070 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33940079d0 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c0b80f0 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3394007dc0 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.325+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c0cb060 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.326+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c068ff0 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.326+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3394018070 con 0x7f339c1614d0 2024-03-06T07:24:46.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.327+0000 7f338affd640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f337c06b4d0 0x7f337c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.327+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3394089cd0 con 0x7f339c1614d0 2024-03-06T07:24:46.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.327+0000 7f339a59c640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f337c06b4d0 0x7f337c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.327+0000 7f339a59c640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f337c06b4d0 0x7f337c06d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f338c035000 tx=0x7f338c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.331+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3394058ee0 con 0x7f339c1614d0 2024-03-06T07:24:46.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.501+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f339c0cae10 con 0x7f339c1614d0 2024-03-06T07:24:46.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.501+0000 7f338affd640 1 -- 172.21.15.72:0/2147573752 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33940587e0 con 0x7f339c1614d0 2024-03-06T07:24:46.503 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:24:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f337c06b4d0 msgr2=0x7f337c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f337c06b4d0 0x7f337c06d990 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f338c035000 tx=0x7f338c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 msgr2=0x7f339c1f3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c1f3060 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f339400c820 tx=0x7f339400cce0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 shutdown_connections 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f337c06b4d0 0x7f337c06d990 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f339c1614d0 0x7f339c1f3060 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f339c0d22e0 0x7f339c0b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 --2- 172.21.15.72:0/2147573752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f339c0d1910 0x7f339c0c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 >> 172.21.15.72:0/2147573752 conn(0x7f339c0d3d40 msgr2=0x7f339c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 shutdown_connections 2024-03-06T07:24:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.503+0000 7f33a0bf7640 1 -- 172.21.15.72:0/2147573752 wait complete. 2024-03-06T07:24:46.516 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.614+0000 7fcbedb72640 1 Processor -- start 2024-03-06T07:24:46.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.614+0000 7fcbedb72640 1 -- start start 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbedb72640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcbe806da40 0x7fcbe80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbedb72640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 0x7fcbe815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbedb72640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80cc980 con 0x7fcbe81424f0 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbedb72640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe80cd220 con 0x7fcbe806da40 2024-03-06T07:24:46.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbedb72640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80ccc60 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 0x7fcbe815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbe6ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcbe806da40 0x7fcbe80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbe7fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 0x7fcbe815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:49596/0 (socket says 172.21.15.72:49596) 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.615+0000 7fcbe7fff640 1 -- 172.21.15.72:0/1939159547 learned_addr learned my addr 172.21.15.72:0/1939159547 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.616+0000 7fcbe7fff640 1 -- 172.21.15.72:0/1939159547 >> v2:172.21.15.146:3300/0 conn(0x7fcbe806da40 msgr2=0x7fcbe80cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.616+0000 7fcbe7fff640 1 --2- 172.21.15.72:0/1939159547 >> v2:172.21.15.146:3300/0 conn(0x7fcbe806da40 0x7fcbe80cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.616+0000 7fcbe7fff640 1 -- 172.21.15.72:0/1939159547 >> v1:172.21.15.146:6789/0 conn(0x7fcbe81424f0 legacy=0x7fcbe806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.616+0000 7fcbe7fff640 1 -- 172.21.15.72:0/1939159547 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80ccf40 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.616+0000 7fcbe6ffd640 1 --2- 172.21.15.72:0/1939159547 >> v2:172.21.15.146:3300/0 conn(0x7fcbe806da40 0x7fcbe80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbe7fff640 1 --2- 172.21.15.72:0/1939159547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 0x7fcbe815cef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009e50 tx=0x7fcbe002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2885730c033862 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbe67fc640 1 -- 172.21.15.72:0/1939159547 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe002cb00 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbe67fc640 1 -- 172.21.15.72:0/1939159547 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe002cca0 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbe67fc640 1 -- 172.21.15.72:0/1939159547 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0034610 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 -- 172.21.15.72:0/1939159547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 msgr2=0x7fcbe815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1939159547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 0x7fcbe815cef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009e50 tx=0x7fcbe002bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 -- 172.21.15.72:0/1939159547 shutdown_connections 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1939159547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe80cd7a0 0x7fcbe815cef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1939159547 >> v2:172.21.15.146:3300/0 conn(0x7fcbe806da40 0x7fcbe80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 -- 172.21.15.72:0/1939159547 >> 172.21.15.72:0/1939159547 conn(0x7fcbe80cf8a0 msgr2=0x7fcbe80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 -- 172.21.15.72:0/1939159547 shutdown_connections 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.617+0000 7fcbedb72640 1 -- 172.21.15.72:0/1939159547 wait complete. 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.618+0000 7fcbedb72640 1 Processor -- start 2024-03-06T07:24:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 -- start start 2024-03-06T07:24:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe806da40 0x7fcbe815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 0x7fcbe81e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbe81424f0 0x7fcbe81e98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8071ce0 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8071b60 con 0x7fcbe81424f0 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbedb72640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8071e60 con 0x7fcbe806da40 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbe6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 0x7fcbe81e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbe6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 0x7fcbe81e8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56424/0 (socket says 172.21.15.72:56424) 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbe6ffd640 1 -- 172.21.15.72:0/1594413106 learned_addr learned my addr 172.21.15.72:0/1594413106 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbe7fff640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbe81424f0 0x7fcbe81e98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.619+0000 7fcbe77fe640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe806da40 0x7fcbe815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbe6ffd640 1 -- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe806da40 msgr2=0x7fcbe815aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbe6ffd640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe806da40 0x7fcbe815aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbe6ffd640 1 -- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbe81424f0 msgr2=0x7fcbe81e98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbe6ffd640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbe81424f0 0x7fcbe81e98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbe6ffd640 1 -- 172.21.15.72:0/1594413106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe81e7e90 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbe6ffd640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 0x7fcbe81e8f30 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00ec10 tx=0x7fcbdc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc017070 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc00cc30 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe81e8170 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc013470 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.620+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe81e8730 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.621+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe804f330 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.622+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcbdc01a070 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.622+0000 7fcbcffff640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcbd406b500 0x7fcbd406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.622+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcbdc08a1e0 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.622+0000 7fcbe77fe640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcbd406b500 0x7fcbd406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.623+0000 7fcbe77fe640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcbd406b500 0x7fcbd406d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80016e0 tx=0x7fcbd8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:46.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.626+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcbdc058c80 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.864+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7fcbe80d0620 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.900+0000 7fcbcffff640 1 -- 172.21.15.72:0/1594413106 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v173) v1 ==== 143+0+0 (secure 0 0 0) 0x7fcbdc05cd20 con 0x7fcbe80cd7a0 2024-03-06T07:24:46.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcbd406b500 msgr2=0x7fcbd406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcbd406b500 0x7fcbd406d9c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80016e0 tx=0x7fcbd8009040 comp rx=0 tx=0).stop 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 msgr2=0x7fcbe81e8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 0x7fcbe81e8f30 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00ec10 tx=0x7fcbdc00c520 comp rx=0 tx=0).stop 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 shutdown_connections 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcbd406b500 0x7fcbd406d9c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbe81424f0 0x7fcbe81e98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcbe80cd7a0 0x7fcbe81e8f30 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 --2- 172.21.15.72:0/1594413106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbe806da40 0x7fcbe815aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 >> 172.21.15.72:0/1594413106 conn(0x7fcbe80cf8a0 msgr2=0x7fcbe815cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 shutdown_connections 2024-03-06T07:24:46.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:46.902+0000 7fcbedb72640 1 -- 172.21.15.72:0/1594413106 wait complete. 2024-03-06T07:24:46.915 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:24:47.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.017+0000 7f60e4ced640 1 Processor -- start 2024-03-06T07:24:47.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.017+0000 7f60e4ced640 1 -- start start 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60e4ced640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f60e015def0 0x7f60e00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60e4ced640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e006cab0 0x7f60e00ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60e4ced640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e006d030 con 0x7f60e00ce880 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60e4ced640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e00cecf0 con 0x7f60e015def0 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60e4ced640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00cf1c0 con 0x7f60e006cab0 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f60e015def0 0x7f60e00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f60e015def0 0x7f60e00b41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56430/0 (socket says 172.21.15.72:56430) 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 -- 172.21.15.72:0/2797057153 learned_addr learned my addr 172.21.15.72:0/2797057153 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 -- 172.21.15.72:0/2797057153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e006cab0 msgr2=0x7f60e00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 --2- 172.21.15.72:0/2797057153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e006cab0 0x7f60e00ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 -- 172.21.15.72:0/2797057153 >> v1:172.21.15.146:6789/0 conn(0x7f60e00ce880 legacy=0x7f60e00cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.018+0000 7f60de59c640 1 -- 172.21.15.72:0/2797057153 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e006d310 con 0x7f60e015def0 2024-03-06T07:24:47.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.019+0000 7f60de59c640 1 --2- 172.21.15.72:0/2797057153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e015def0 0x7f60e00b41b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009870 tx=0x7f60d002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5119860584b63f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.019+0000 7f60dd59a640 1 -- 172.21.15.72:0/2797057153 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0034070 con 0x7f60e015def0 2024-03-06T07:24:47.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.019+0000 7f60dd59a640 1 -- 172.21.15.72:0/2797057153 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d0030840 con 0x7f60e015def0 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.019+0000 7f60dd59a640 1 -- 172.21.15.72:0/2797057153 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0030c10 con 0x7f60e015def0 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 -- 172.21.15.72:0/2797057153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e015def0 msgr2=0x7f60e00b41b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2797057153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e015def0 0x7f60e00b41b0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009870 tx=0x7f60d002b980 comp rx=0 tx=0).stop 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 -- 172.21.15.72:0/2797057153 shutdown_connections 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2797057153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e006cab0 0x7f60e00ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2797057153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e015def0 0x7f60e00b41b0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 -- 172.21.15.72:0/2797057153 >> 172.21.15.72:0/2797057153 conn(0x7f60e00b3690 msgr2=0x7f60e00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 -- 172.21.15.72:0/2797057153 shutdown_connections 2024-03-06T07:24:47.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.020+0000 7f60e4ced640 1 -- 172.21.15.72:0/2797057153 wait complete. 2024-03-06T07:24:47.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.021+0000 7f60e4ced640 1 Processor -- start 2024-03-06T07:24:47.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.021+0000 7f60e4ced640 1 -- start start 2024-03-06T07:24:47.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60e4ced640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 0x7f60e01f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60e4ced640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60e00ce880 0x7f60e00b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60e4ced640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e015def0 0x7f60e014e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60e4ced640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00cc3f0 con 0x7f60e006cab0 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60e4ced640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e00cc760 con 0x7f60e00ce880 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60e4ced640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e015e2d0 con 0x7f60e015def0 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 0x7f60e01f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 0x7f60e01f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56444/0 (socket says 172.21.15.72:56444) 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 -- 172.21.15.72:0/2684565 learned_addr learned my addr 172.21.15.72:0/2684565 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 -- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e015def0 msgr2=0x7f60e014e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e015def0 0x7f60e014e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 -- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60e00ce880 msgr2=0x7f60e00b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60ddd9b640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60e00ce880 0x7f60e00b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60e00ce880 0x7f60e00b4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60de59c640 1 -- 172.21.15.72:0/2684565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e015f160 con 0x7f60e006cab0 2024-03-06T07:24:47.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.022+0000 7f60ddd9b640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60e00ce880 0x7f60e00b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:47.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.023+0000 7f60de59c640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 0x7f60e01f3ad0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009870 tx=0x7f60d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.023+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0034040 con 0x7f60e006cab0 2024-03-06T07:24:47.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.023+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d0039d90 con 0x7f60e006cab0 2024-03-06T07:24:47.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.023+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e015f400 con 0x7f60e006cab0 2024-03-06T07:24:47.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.023+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d0043610 con 0x7f60e006cab0 2024-03-06T07:24:47.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.023+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e015f9c0 con 0x7f60e006cab0 2024-03-06T07:24:47.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.024+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e009fcf0 con 0x7f60e006cab0 2024-03-06T07:24:47.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.024+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60d004a060 con 0x7f60e006cab0 2024-03-06T07:24:47.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.024+0000 7f60ceffd640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60c006b4d0 0x7f60c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.024+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f60d00ac140 con 0x7f60e006cab0 2024-03-06T07:24:47.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.025+0000 7f60ddd9b640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60c006b4d0 0x7f60c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.025+0000 7f60ddd9b640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60c006b4d0 0x7f60c006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006fd0 tx=0x7f60d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.029+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60d003e070 con 0x7f60e006cab0 2024-03-06T07:24:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.202+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f60e0068ff0 con 0x7f60e006cab0 2024-03-06T07:24:47.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.205+0000 7f60ceffd640 1 -- 172.21.15.72:0/2684565 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f60d002d330 con 0x7f60e006cab0 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60c006b4d0 msgr2=0x7f60c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60c006b4d0 0x7f60c006d990 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006fd0 tx=0x7f60d4008040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 msgr2=0x7f60e01f3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 0x7f60e01f3ad0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009870 tx=0x7f60d00046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 shutdown_connections 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60c006b4d0 0x7f60c006d990 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60e015def0 0x7f60e014e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60e00ce880 0x7f60e00b4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 --2- 172.21.15.72:0/2684565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60e006cab0 0x7f60e01f3ad0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 >> 172.21.15.72:0/2684565 conn(0x7f60e00b3690 msgr2=0x7f60e006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 shutdown_connections 2024-03-06T07:24:47.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.207+0000 7f60e4ced640 1 -- 172.21.15.72:0/2684565 wait complete. 2024-03-06T07:24:47.220 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:24:47.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:47.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:24:47.308 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:24:47.309 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:47.309 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.324 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:47.324 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:24:47.325 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:47.342 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:24:47.343 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.379 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:24:47.379 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:24:47.379 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:24:47.400 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:24:47.401 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:24:47.401 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:47.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.517+0000 7fa3b7b7a640 1 Processor -- start 2024-03-06T07:24:47.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.517+0000 7fa3b7b7a640 1 -- start start 2024-03-06T07:24:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b7b7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b7b7a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3b015bf30 0x7fa3b015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b7b7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b015f2d0 con 0x7fa3b013dc10 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b7b7a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b00cf3f0 con 0x7fa3b015bf30 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b7b7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b015b210 con 0x7fa3b00b3060 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3b015bf30 0x7fa3b015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3b015bf30 0x7fa3b015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56446/0 (socket says 172.21.15.72:56446) 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 -- 172.21.15.72:0/2322054757 learned_addr learned my addr 172.21.15.72:0/2322054757 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b5115640 1 --2- 172.21.15.72:0/2322054757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 -- 172.21.15.72:0/2322054757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 msgr2=0x7fa3b015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 --2- 172.21.15.72:0/2322054757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 -- 172.21.15.72:0/2322054757 >> v1:172.21.15.146:6789/0 conn(0x7fa3b013dc10 legacy=0x7fa3b00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.518+0000 7fa3b6117640 1 -- 172.21.15.72:0/2322054757 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b015f580 con 0x7fa3b015bf30 2024-03-06T07:24:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b6117640 1 --2- 172.21.15.72:0/2322054757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b015bf30 0x7fa3b015ed90 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009e50 tx=0x7fa3a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24b752a26c72037 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b4914640 1 -- 172.21.15.72:0/2322054757 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a802ca70 con 0x7fa3b015bf30 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b4914640 1 -- 172.21.15.72:0/2322054757 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a802cc10 con 0x7fa3b015bf30 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b4914640 1 -- 172.21.15.72:0/2322054757 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a80343f0 con 0x7fa3b015bf30 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2322054757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b015bf30 msgr2=0x7fa3b015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2322054757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b015bf30 0x7fa3b015ed90 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa3a8009e50 tx=0x7fa3a802bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2322054757 shutdown_connections 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2322054757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b015bf30 0x7fa3b015ed90 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2322054757 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b015e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.519+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2322054757 >> 172.21.15.72:0/2322054757 conn(0x7fa3b00cf8a0 msgr2=0x7fa3b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.520+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2322054757 shutdown_connections 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.520+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2322054757 wait complete. 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 Processor -- start 2024-03-06T07:24:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 -- start start 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b01546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 0x7fa3b013e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b015bf30 0x7fa3b00714f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b013dff0 con 0x7fa3b013dc10 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b015b730 con 0x7fa3b015bf30 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b7b7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b00708d0 con 0x7fa3b00b3060 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b5115640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 0x7fa3b013e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b5115640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 0x7fa3b013e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56458/0 (socket says 172.21.15.72:56458) 2024-03-06T07:24:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.521+0000 7fa3b5115640 1 -- 172.21.15.72:0/2660775367 learned_addr learned my addr 172.21.15.72:0/2660775367 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5916640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b01546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b6117640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b015bf30 0x7fa3b00714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5115640 1 -- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 msgr2=0x7fa3b01546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5115640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b01546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5115640 1 -- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b015bf30 msgr2=0x7fa3b00714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5115640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b015bf30 0x7fa3b00714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5115640 1 -- 172.21.15.72:0/2660775367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b0154be0 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b5115640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 0x7fa3b013e170 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000d800 tx=0x7fa3a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000b6c0 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0019070 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a000bac0 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b013ea70 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.522+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b013f030 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.523+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b004f330 con 0x7fa3b013dc10 2024-03-06T07:24:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.524+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3a000bc60 con 0x7fa3b013dc10 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.524+0000 7fa3a67fc640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa39406b4d0 0x7fa39406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.524+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa3a008a350 con 0x7fa3b013dc10 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.524+0000 7fa3b5916640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa39406b4d0 0x7fa39406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.524+0000 7fa3b5916640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa39406b4d0 0x7fa39406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac007fd0 tx=0x7fa3ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.528+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3a0059560 con 0x7fa3b013dc10 2024-03-06T07:24:47.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.734+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3b00b05b0 con 0x7fa3b013dc10 2024-03-06T07:24:47.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.735+0000 7fa3a67fc640 1 -- 172.21.15.72:0/2660775367 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fa3a0058e60 con 0x7fa3b013dc10 2024-03-06T07:24:47.737 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:47.737 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/2388675791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2388675791},{"type":"v1","addr":"172.21.15.146:6833","nonce":2388675791}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/2473611413","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2473611413},{"type":"v1","addr":"172.21.15.146:6835","nonce":2473611413}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":6054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3906125441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3906125441},{"type":"v1","addr":"172.21.15.72:6836","nonce":3906125441}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":6477,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3588775179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3588775179},{"type":"v1","addr":"172.21.15.72:6837","nonce":3588775179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2024-03-06T07:24:47.737 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 29 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa39406b4d0 msgr2=0x7fa39406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa39406b4d0 0x7fa39406d990 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac007fd0 tx=0x7fa3ac009040 comp rx=0 tx=0).stop 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 msgr2=0x7fa3b013e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 0x7fa3b013e170 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fa3a000d800 tx=0x7fa3a000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 shutdown_connections 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa39406b4d0 0x7fa39406d990 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b015bf30 0x7fa3b00714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3b013dc10 0x7fa3b013e170 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 --2- 172.21.15.72:0/2660775367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b00b3060 0x7fa3b01546a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 >> 172.21.15.72:0/2660775367 conn(0x7fa3b00cf8a0 msgr2=0x7fa3b00afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 shutdown_connections 2024-03-06T07:24:47.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.737+0000 7fa3b7b7a640 1 -- 172.21.15.72:0/2660775367 wait complete. 2024-03-06T07:24:47.750 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:24:47.750 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:24:47.751 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:24:47.751 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:47.751 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:24:47.751 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:24:47.751 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:24:47.751 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:47.751 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:24:47.751 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:24:47.752 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:24:47.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:47.752 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:24:47.752 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:24:47.752 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:24:47.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:24:47.760 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:24:47.761 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:24:47.761 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:24:47.762 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:24:47.764 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:24:47.764 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:24:47.765 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:24:47.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:24:47.803 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:24:47.803 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:47.804 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:24:47.804 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:47.806 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:24:47.806 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:47.807 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:24:47.807 DEBUG:teuthology.parallel:result is None 2024-03-06T07:24:47.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:24:47.879 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:24:47.879 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:24:47.889 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:24:47.889 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:24:47.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.911+0000 7f8b675d0640 1 Processor -- start 2024-03-06T07:24:47.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.911+0000 7f8b675d0640 1 -- start start 2024-03-06T07:24:47.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b675d0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b600b4c00 0x7f8b600b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b675d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b600b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b675d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8b6015e540 0x7f8b6015e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b675d0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6015e0b0 con 0x7f8b600b4c00 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b675d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6015d510 con 0x7f8b6015e540 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b675d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6015d780 con 0x7f8b600b28a0 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b600b4c00 0x7f8b600b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b600b4c00 0x7f8b600b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56502/0 (socket says 172.21.15.72:56502) 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 -- 172.21.15.72:0/736730886 learned_addr learned my addr 172.21.15.72:0/736730886 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b64b6b640 1 --2- 172.21.15.72:0/736730886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b600b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b65b6d640 1 --2- 172.21.15.72:0/736730886 >> v2:172.21.15.146:3300/0 conn(0x7f8b6015e540 0x7f8b6015e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 -- 172.21.15.72:0/736730886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 msgr2=0x7f8b600b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 --2- 172.21.15.72:0/736730886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b600b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 -- 172.21.15.72:0/736730886 >> v2:172.21.15.146:3300/0 conn(0x7f8b6015e540 msgr2=0x7f8b6015e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:47.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 --2- 172.21.15.72:0/736730886 >> v2:172.21.15.146:3300/0 conn(0x7f8b6015e540 0x7f8b6015e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b6536c640 1 -- 172.21.15.72:0/736730886 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6015da60 con 0x7f8b600b4c00 2024-03-06T07:24:47.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.912+0000 7f8b65b6d640 1 --2- 172.21.15.72:0/736730886 >> v2:172.21.15.146:3300/0 conn(0x7f8b6015e540 0x7f8b6015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:47.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.913+0000 7f8b6536c640 1 --2- 172.21.15.72:0/736730886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b600b4fe0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8b6006ba80 tx=0x7f8b5c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20af71a539deed85 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.913+0000 7f8b57fff640 1 -- 172.21.15.72:0/736730886 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c034070 con 0x7f8b600b4c00 2024-03-06T07:24:47.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.913+0000 7f8b57fff640 1 -- 172.21.15.72:0/736730886 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c030850 con 0x7f8b600b4c00 2024-03-06T07:24:47.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.913+0000 7f8b57fff640 1 -- 172.21.15.72:0/736730886 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c030c20 con 0x7f8b600b4c00 2024-03-06T07:24:47.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 -- 172.21.15.72:0/736730886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 msgr2=0x7f8b600b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 --2- 172.21.15.72:0/736730886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b600b4fe0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8b6006ba80 tx=0x7f8b5c02b990 comp rx=0 tx=0).stop 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 -- 172.21.15.72:0/736730886 shutdown_connections 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 --2- 172.21.15.72:0/736730886 >> v2:172.21.15.146:3300/0 conn(0x7f8b6015e540 0x7f8b6015e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 --2- 172.21.15.72:0/736730886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b600b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 --2- 172.21.15.72:0/736730886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b600b4fe0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 -- 172.21.15.72:0/736730886 >> 172.21.15.72:0/736730886 conn(0x7f8b6009a9f0 msgr2=0x7f8b6009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 -- 172.21.15.72:0/736730886 shutdown_connections 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.914+0000 7f8b675d0640 1 -- 172.21.15.72:0/736730886 wait complete. 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.915+0000 7f8b675d0640 1 Processor -- start 2024-03-06T07:24:47.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.915+0000 7f8b675d0640 1 -- start start 2024-03-06T07:24:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b675d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b6014a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b675d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b6014b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b675d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b6015e540 0x7f8b6006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b675d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b600b33a0 con 0x7f8b600b4c00 2024-03-06T07:24:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b675d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b600b3220 con 0x7f8b6015e540 2024-03-06T07:24:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b675d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6015eec0 con 0x7f8b600b28a0 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b6014b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b6014b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56506/0 (socket says 172.21.15.72:56506) 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 -- 172.21.15.72:0/424826043 learned_addr learned my addr 172.21.15.72:0/424826043 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b6536c640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b6014a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b65b6d640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b6015e540 0x7f8b6006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 -- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 msgr2=0x7f8b6014a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b6014a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 -- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b6015e540 msgr2=0x7f8b6006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b6015e540 0x7f8b6006d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 -- 172.21.15.72:0/424826043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b600d12d0 con 0x7f8b600b4c00 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b6536c640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b6014a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.916+0000 7f8b64b6b640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b6014b740 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c7f0 tx=0x7f8b5000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.917+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50015070 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.917+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b50004220 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.917+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b50007620 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.917+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6006cb70 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.917+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b600d1850 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.918+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6004f330 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.918+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8b500079d0 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.918+0000 7f8b55ffb640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b4406b4d0 0x7f8b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.919+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8b50089fe0 con 0x7f8b600b4c00 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.919+0000 7f8b6536c640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b4406b4d0 0x7f8b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:47.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.919+0000 7f8b6536c640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b4406b4d0 0x7f8b4406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c036000 tx=0x7f8b5c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:47.925+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8b500590d0 con 0x7f8b600b4c00 2024-03-06T07:24:48.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.111+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f8b600b5d90 con 0x7f8b600b4c00 2024-03-06T07:24:48.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.933+0000 7f8b55ffb640 1 -- 172.21.15.72:0/424826043 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8b500589d0 con 0x7f8b600b4c00 2024-03-06T07:24:48.935 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b4406b4d0 msgr2=0x7f8b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b4406b4d0 0x7f8b4406d990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c036000 tx=0x7f8b5c03b040 comp rx=0 tx=0).stop 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 msgr2=0x7f8b6014b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b6014b740 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000c7f0 tx=0x7f8b5000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 shutdown_connections 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b4406b4d0 0x7f8b4406d990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b6015e540 0x7f8b6006d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b600b4c00 0x7f8b6014b740 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 --2- 172.21.15.72:0/424826043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b600b28a0 0x7f8b6014a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.936+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 >> 172.21.15.72:0/424826043 conn(0x7f8b6009a9f0 msgr2=0x7f8b6015be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.937+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 shutdown_connections 2024-03-06T07:24:48.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:48.937+0000 7f8b675d0640 1 -- 172.21.15.72:0/424826043 wait complete. 2024-03-06T07:24:48.952 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:24:49.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.054+0000 7ff0b86fb640 1 Processor -- start 2024-03-06T07:24:49.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.054+0000 7ff0b86fb640 1 -- start start 2024-03-06T07:24:49.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.055+0000 7ff0b86fb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff0b013dc10 0x7ff0b00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.055+0000 7ff0b86fb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b015f580 con 0x7ff0b015bf30 2024-03-06T07:24:49.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.055+0000 7ff0b86fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b015b450 con 0x7ff0b015ef60 2024-03-06T07:24:49.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.055+0000 7ff0b86fb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b00cf3f0 con 0x7ff0b013dc10 2024-03-06T07:24:49.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5c96640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff0b015bf30 0x7ff0b015e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49704/0 (socket says 172.21.15.72:49704) 2024-03-06T07:24:49.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5c96640 1 -- 172.21.15.72:0/2429539910 learned_addr learned my addr 172.21.15.72:0/2429539910 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b6497640 1 --2- 172.21.15.72:0/2429539910 >> v2:172.21.15.146:3300/0 conn(0x7ff0b013dc10 0x7ff0b00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128332237 0 0) 0x7ff0b015f580 con 0x7ff0b015bf30 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff09c003520 con 0x7ff0b015bf30 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3159509522 0 0) 0x7ff09c003520 con 0x7ff0b015bf30 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 >> v2:172.21.15.146:3300/0 conn(0x7ff0b013dc10 msgr2=0x7ff0b00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 --2- 172.21.15.72:0/2429539910 >> v2:172.21.15.146:3300/0 conn(0x7ff0b013dc10 0x7ff0b00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 >> v1:172.21.15.146:6789/0 conn(0x7ff0b015ef60 legacy=0x7ff0b00708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b015b730 con 0x7ff0b015bf30 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff0a0001670 con 0x7ff0b015bf30 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 >> v1:172.21.15.72:6789/0 conn(0x7ff0b015bf30 legacy=0x7ff0b015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff09c0039b0 0x7ff09c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff09c004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff09c005120 0x7ff09c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b00cf3f0 con 0x7ff09c0043b0 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c00c060 con 0x7ff09c0039b0 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c00c260 con 0x7ff09c005120 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff09c004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b6497640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff09c0039b0 0x7ff09c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.056+0000 7ff0b6c98640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff09c005120 0x7ff09c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5c96640 1 -- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff09c005120 msgr2=0x7ff09c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff09c005120 0x7ff09c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5c96640 1 -- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff09c0039b0 msgr2=0x7ff09c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff09c0039b0 0x7ff09c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5c96640 1 -- 172.21.15.72:0/2429539910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c003520 con 0x7ff09c0043b0 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b6c98640 1 -- 172.21.15.72:0/2429539910 reap_dead start 2024-03-06T07:24:49.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff09c004880 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000c240 tx=0x7ff09c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3b191cf0f1e8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0007d10 con 0x7ff09c0043b0 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a0030070 con 0x7ff09c0043b0 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.057+0000 7ff0b5495640 1 -- 172.21.15.72:0/2429539910 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0005d70 con 0x7ff09c0043b0 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2429539910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 msgr2=0x7ff09c004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff09c004880 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000c240 tx=0x7ff09c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2429539910 shutdown_connections 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2429539910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff09c004880 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2429539910 >> 172.21.15.72:0/2429539910 conn(0x7ff0b00cf8a0 msgr2=0x7ff0b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2429539910 shutdown_connections 2024-03-06T07:24:49.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.058+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2429539910 wait complete. 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.059+0000 7ff0b86fb640 1 Processor -- start 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.059+0000 7ff0b86fb640 1 -- start start 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b86fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff0b0141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b86fb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b0139f30 0x7ff0b0154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b86fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b0142d90 0x7ff0b0143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b86fb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b015e850 con 0x7ff09c0043b0 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b86fb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b015b210 con 0x7ff0b0139f30 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b86fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b015e9d0 con 0x7ff0b0142d90 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff0b0141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff0b0141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56528/0 (socket says 172.21.15.72:56528) 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 -- 172.21.15.72:0/2607960124 learned_addr learned my addr 172.21.15.72:0/2607960124 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6c98640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b0142d90 0x7ff0b0143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b0139f30 0x7ff0b0154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 -- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b0142d90 msgr2=0x7ff0b0143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b0142d90 0x7ff0b0143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 -- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b0139f30 msgr2=0x7ff0b0154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b0139f30 0x7ff0b0154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 -- 172.21.15.72:0/2607960124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0142500 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6c98640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b0142d90 0x7ff0b0143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.060+0000 7ff0b6497640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff0b0141ee0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac011ad0 tx=0x7ff0ac011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.061+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00fbb0 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.061+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac00fd50 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.061+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b01427e0 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.061+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac014640 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.061+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b013e680 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.062+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b004f330 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.062+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff0ac012040 con 0x7ff09c0043b0 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.062+0000 7ff0a6ffd640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09406b4d0 0x7ff09406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.063+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff0ac08a5c0 con 0x7ff09c0043b0 2024-03-06T07:24:49.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.063+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09406b4d0 0x7ff09406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.063+0000 7ff0b5c96640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09406b4d0 0x7ff09406d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0034000 tx=0x7ff0a0039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.067+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0ac019030 con 0x7ff09c0043b0 2024-03-06T07:24:49.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.306+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff0b006eb00 con 0x7ff09c0043b0 2024-03-06T07:24:49.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.306+0000 7ff0a6ffd640 1 -- 172.21.15.72:0/2607960124 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff0ac059790 con 0x7ff09c0043b0 2024-03-06T07:24:49.308 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:49.308 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:24:49.308 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:24:49.310 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:24:49.311 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.312 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.315 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.316 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.318 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:24:49.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:49.320 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:24:49.321 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09406b4d0 msgr2=0x7ff09406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09406b4d0 0x7ff09406d990 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0034000 tx=0x7ff0a0039040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 msgr2=0x7ff0b0141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff0b0141ee0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac011ad0 tx=0x7ff0ac011f90 comp rx=0 tx=0).stop 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 shutdown_connections 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff09406b4d0 0x7ff09406d990 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0b0142d90 0x7ff0b0143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0b0139f30 0x7ff0b0154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 --2- 172.21.15.72:0/2607960124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff09c0043b0 0x7ff0b0141ee0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 >> 172.21.15.72:0/2607960124 conn(0x7ff0b00cf8a0 msgr2=0x7ff0b00711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 shutdown_connections 2024-03-06T07:24:49.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.308+0000 7ff0b86fb640 1 -- 172.21.15.72:0/2607960124 wait complete. 2024-03-06T07:24:49.323 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:49.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.421+0000 7fad67d65640 1 Processor -- start 2024-03-06T07:24:49.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.421+0000 7fad67d65640 1 -- start start 2024-03-06T07:24:49.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad67d65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad6013dc10 0x7fad600b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad67d65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6015bf30 0x7fad600b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad67d65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6015f580 con 0x7fad6015e850 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad67d65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad600cf3f0 con 0x7fad6013dc10 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad67d65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6015b210 con 0x7fad6015bf30 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad65b01640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad6013dc10 0x7fad600b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad66302640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fad6015e850 0x7fad6015eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49708/0 (socket says 172.21.15.72:49708) 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad66302640 1 -- 172.21.15.72:0/1594876501 learned_addr learned my addr 172.21.15.72:0/1594876501 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad65300640 1 --2- 172.21.15.72:0/1594876501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6015bf30 0x7fad600b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad65b01640 1 -- 172.21.15.72:0/1594876501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6015bf30 msgr2=0x7fad600b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad65b01640 1 --2- 172.21.15.72:0/1594876501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6015bf30 0x7fad600b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad65b01640 1 -- 172.21.15.72:0/1594876501 >> v1:172.21.15.72:6789/0 conn(0x7fad6015e850 legacy=0x7fad6015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.422+0000 7fad65b01640 1 -- 172.21.15.72:0/1594876501 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6015b450 con 0x7fad6013dc10 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.423+0000 7fad65b01640 1 --2- 172.21.15.72:0/1594876501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6013dc10 0x7fad600b2950 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009d90 tx=0x7fad5c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843d57f19dbe537f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.423+0000 7fad64aff640 1 -- 172.21.15.72:0/1594876501 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c039070 con 0x7fad6013dc10 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.423+0000 7fad64aff640 1 -- 172.21.15.72:0/1594876501 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5c02cbc0 con 0x7fad6013dc10 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.423+0000 7fad64aff640 1 -- 172.21.15.72:0/1594876501 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5c034440 con 0x7fad6013dc10 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 -- 172.21.15.72:0/1594876501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6013dc10 msgr2=0x7fad600b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 --2- 172.21.15.72:0/1594876501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6013dc10 0x7fad600b2950 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fad5c009d90 tx=0x7fad5c02be10 comp rx=0 tx=0).stop 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 -- 172.21.15.72:0/1594876501 shutdown_connections 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 --2- 172.21.15.72:0/1594876501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6015bf30 0x7fad600b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 --2- 172.21.15.72:0/1594876501 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6013dc10 0x7fad600b2950 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 -- 172.21.15.72:0/1594876501 >> 172.21.15.72:0/1594876501 conn(0x7fad600cf8a0 msgr2=0x7fad600cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 -- 172.21.15.72:0/1594876501 shutdown_connections 2024-03-06T07:24:49.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.424+0000 7fad67d65640 1 -- 172.21.15.72:0/1594876501 wait complete. 2024-03-06T07:24:49.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.425+0000 7fad67d65640 1 Processor -- start 2024-03-06T07:24:49.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.425+0000 7fad67d65640 1 -- start start 2024-03-06T07:24:49.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6013dc10 0x7fad6013cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad6015bf30 0x7fad601532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 0x7fad60153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad600708d0 con 0x7fad6015e850 2024-03-06T07:24:49.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6013dff0 con 0x7fad6015bf30 2024-03-06T07:24:49.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad60070a50 con 0x7fad6013dc10 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 0x7fad60153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 0x7fad60153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56554/0 (socket says 172.21.15.72:56554) 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 -- 172.21.15.72:0/2432189018 learned_addr learned my addr 172.21.15.72:0/2432189018 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad65b01640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6013dc10 0x7fad6013cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad65300640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad6015bf30 0x7fad601532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 -- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6013dc10 msgr2=0x7fad6013cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6013dc10 0x7fad6013cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 -- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad6015bf30 msgr2=0x7fad601532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad6015bf30 0x7fad601532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 -- 172.21.15.72:0/2432189018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6013e650 con 0x7fad6015e850 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad65b01640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6013dc10 0x7fad6013cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad65300640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad6015bf30 0x7fad601532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad66302640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 0x7fad60153db0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fad5800c3a0 tx=0x7fad5800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad58017070 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad58005ce0 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad580072d0 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6013e930 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.426+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6013eef0 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.427+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6004f330 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.428+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad5801a070 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.428+0000 7fad567fc640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad4406b4d0 0x7fad4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.428+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fad58089090 con 0x7fad6015e850 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.428+0000 7fad65b01640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad4406b4d0 0x7fad4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.429+0000 7fad65b01640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad4406b4d0 0x7fad4406d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fad5c036000 tx=0x7fad5c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.433+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad58057c10 con 0x7fad6015e850 2024-03-06T07:24:49.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.604+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad60071b80 con 0x7fad6015e850 2024-03-06T07:24:49.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.604+0000 7fad567fc640 1 -- 172.21.15.72:0/2432189018 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+12903 (secure 0 0 0) 0x7fad5805bcb0 con 0x7fad6015e850 2024-03-06T07:24:49.606 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:49.606 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":67,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:48.927191+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad4406b4d0 msgr2=0x7fad4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad4406b4d0 0x7fad4406d990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fad5c036000 tx=0x7fad5c030000 comp rx=0 tx=0).stop 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 msgr2=0x7fad60153db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 0x7fad60153db0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fad5800c3a0 tx=0x7fad5800c860 comp rx=0 tx=0).stop 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 shutdown_connections 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad4406b4d0 0x7fad4406d990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad6015e850 0x7fad60153db0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad6015bf30 0x7fad601532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 --2- 172.21.15.72:0/2432189018 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad6013dc10 0x7fad6013cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 >> 172.21.15.72:0/2432189018 conn(0x7fad600cf8a0 msgr2=0x7fad600722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 shutdown_connections 2024-03-06T07:24:49.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.606+0000 7fad67d65640 1 -- 172.21.15.72:0/2432189018 wait complete. 2024-03-06T07:24:49.620 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:24:49.620 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:24:49.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.721+0000 7f64becfb640 1 Processor -- start 2024-03-06T07:24:49.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.721+0000 7f64becfb640 1 -- start start 2024-03-06T07:24:49.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.721+0000 7f64becfb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64b80b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64becfb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b806d030 con 0x7f64b806cab0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64becfb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b806d310 con 0x7f64b80cea50 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64becfb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b80cf1c0 con 0x7f64b815def0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f64b806cab0 0x7f64b80ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49718/0 (socket says 172.21.15.72:49718) 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b7fff640 1 -- 172.21.15.72:0/437761233 learned_addr learned my addr 172.21.15.72:0/437761233 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64bca97640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64b80b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982143614 0 0) 0x7f64b806d030 con 0x7f64b806cab0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4003520 con 0x7f64b806cab0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155812800 0 0) 0x7f64a4003520 con 0x7f64b806cab0 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 msgr2=0x7f64b80b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64b80b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 >> v1:172.21.15.146:6789/0 conn(0x7f64b80cea50 legacy=0x7f64b80ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b80cc110 con 0x7f64b806cab0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f64a0001670 con 0x7f64b806cab0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 >> v1:172.21.15.72:6789/0 conn(0x7f64b806cab0 legacy=0x7f64b80ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64a40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64a4003f20 0x7f64a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4004d40 0x7f64a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a400b9f0 con 0x7f64a4003f20 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a400bba0 con 0x7f64a4004d40 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a400bda0 con 0x7f64b815def0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.722+0000 7f64b7fff640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64a4003f20 0x7f64a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64bca97640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64a40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64bd298640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4004d40 0x7f64a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b7fff640 1 -- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 msgr2=0x7f64a40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b7fff640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64a40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b7fff640 1 -- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4004d40 msgr2=0x7f64a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b7fff640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4004d40 0x7f64a40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b7fff640 1 -- 172.21.15.72:0/437761233 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4003520 con 0x7f64a4003f20 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64bca97640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b815def0 0x7f64a40039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b7fff640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64a4003f20 0x7f64a4004800 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c290 tx=0x7f64a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473e37196e2f498c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.723+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0007d50 con 0x7f64a4003f20 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0030070 con 0x7f64a4003f20 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64b77fe640 1 -- 172.21.15.72:0/437761233 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0005d70 con 0x7f64a4003f20 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64becfb640 1 -- 172.21.15.72:0/437761233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64a4003f20 msgr2=0x7f64a4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64becfb640 1 --2- 172.21.15.72:0/437761233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64a4003f20 0x7f64a4004800 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f64a000c290 tx=0x7f64a4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64bd298640 1 -- 172.21.15.72:0/437761233 reap_dead start 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64becfb640 1 -- 172.21.15.72:0/437761233 shutdown_connections 2024-03-06T07:24:49.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64becfb640 1 -- 172.21.15.72:0/437761233 >> 172.21.15.72:0/437761233 conn(0x7f64b80b3690 msgr2=0x7f64b80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64becfb640 1 -- 172.21.15.72:0/437761233 shutdown_connections 2024-03-06T07:24:49.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.724+0000 7f64becfb640 1 -- 172.21.15.72:0/437761233 wait complete. 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.725+0000 7f64becfb640 1 Processor -- start 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.725+0000 7f64becfb640 1 -- start start 2024-03-06T07:24:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64becfb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4003f20 0x7f64b815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64becfb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 0x7f64b8160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64becfb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b8160840 0x7f64b8160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64becfb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b815e2d0 con 0x7f64b815e5d0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64becfb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b80cc760 con 0x7f64a4003f20 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64becfb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64b815e450 con 0x7f64b8160840 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 0x7f64b8160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 0x7f64b8160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56578/0 (socket says 172.21.15.72:56578) 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 -- 172.21.15.72:0/4234468558 learned_addr learned my addr 172.21.15.72:0/4234468558 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64bd298640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b8160840 0x7f64b8160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64bca97640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4003f20 0x7f64b815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 -- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b8160840 msgr2=0x7f64b8160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b8160840 0x7f64b8160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 -- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4003f20 msgr2=0x7f64b815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4003f20 0x7f64b815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.726+0000 7f64b7fff640 1 -- 172.21.15.72:0/4234468558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b80b7cc0 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64bca97640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4003f20 0x7f64b815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64bd298640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b8160840 0x7f64b8160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64b7fff640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 0x7f64b8160300 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f64a002c4c0 tx=0x7f64a0001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0007d50 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a0004c10 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b815fbf0 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0007070 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.727+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b80c58c0 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.728+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b809fcf0 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.728+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f64a0047020 con 0x7f64b815e5d0 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.729+0000 7f64b57fa640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f649806b4d0 0x7f649806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.729+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f64a00ab110 con 0x7f64b815e5d0 2024-03-06T07:24:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.729+0000 7f64bca97640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f649806b4d0 0x7f649806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.729+0000 7f64bca97640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f649806b4d0 0x7f649806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f64b0007560 tx=0x7f64b000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.733+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f64a007a2f0 con 0x7f64b815e5d0 2024-03-06T07:24:49.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.903+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64b8068ff0 con 0x7f64b815e5d0 2024-03-06T07:24:49.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.904+0000 7f64b57fa640 1 -- 172.21.15.72:0/4234468558 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64a0079bf0 con 0x7f64b815e5d0 2024-03-06T07:24:49.906 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.905+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f649806b4d0 msgr2=0x7f649806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.905+0000 7f64becfb640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f649806b4d0 0x7f649806d990 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f64b0007560 tx=0x7f64b000b040 comp rx=0 tx=0).stop 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.905+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 msgr2=0x7f64b8160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.905+0000 7f64becfb640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 0x7f64b8160300 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f64a002c4c0 tx=0x7f64a0001a40 comp rx=0 tx=0).stop 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 shutdown_connections 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f649806b4d0 0x7f649806d990 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64b8160840 0x7f64b8160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f64b815e5d0 0x7f64b8160300 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 --2- 172.21.15.72:0/4234468558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64a4003f20 0x7f64b815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 >> 172.21.15.72:0/4234468558 conn(0x7f64b80b3690 msgr2=0x7f64b806f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 shutdown_connections 2024-03-06T07:24:49.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:49.906+0000 7f64becfb640 1 -- 172.21.15.72:0/4234468558 wait complete. 2024-03-06T07:24:49.919 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:24:50.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.016+0000 7efd8919a640 1 Processor -- start 2024-03-06T07:24:50.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.016+0000 7efd8919a640 1 -- start start 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd8919a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd840b6550 0x7efd840cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd8919a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd840cef50 con 0x7efd840d02c0 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd8919a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8415c620 con 0x7efd840d06a0 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd8919a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd840b50b0 con 0x7efd840b6550 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd82d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efd840d02c0 0x7efd840ce130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49724/0 (socket says 172.21.15.72:49724) 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd82d9d640 1 -- 172.21.15.72:0/2749293428 learned_addr learned my addr 172.21.15.72:0/2749293428 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314049230 0 0) 0x7efd840cef50 con 0x7efd840d02c0 2024-03-06T07:24:50.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd6c003520 con 0x7efd840d02c0 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd8259c640 1 --2- 172.21.15.72:0/2749293428 >> v2:172.21.15.146:3300/0 conn(0x7efd840b6550 0x7efd840cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2384212628 0 0) 0x7efd6c003520 con 0x7efd840d02c0 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 >> v2:172.21.15.146:3300/0 conn(0x7efd840b6550 msgr2=0x7efd840cf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd81d9b640 1 --2- 172.21.15.72:0/2749293428 >> v2:172.21.15.146:3300/0 conn(0x7efd840b6550 0x7efd840cf850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.017+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 >> v1:172.21.15.146:6789/0 conn(0x7efd840d06a0 legacy=0x7efd840ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd840cd800 con 0x7efd840d02c0 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7efd74001670 con 0x7efd840d02c0 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 --2- 172.21.15.72:0/2749293428 >> v2:172.21.15.146:3300/0 conn(0x7efd840b6550 0x7efd840cf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 >> v1:172.21.15.72:6789/0 conn(0x7efd840d02c0 legacy=0x7efd840ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd6c0039b0 0x7efd6c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd6c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6c0051c0 0x7efd6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840b50b0 con 0x7efd6c004800 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c00c110 con 0x7efd6c0051c0 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c00c310 con 0x7efd6c0039b0 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd6c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd82d9d640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd6c0039b0 0x7efd6c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8359e640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6c0051c0 0x7efd6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 -- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd6c0039b0 msgr2=0x7efd6c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd6c0039b0 0x7efd6c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 -- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6c0051c0 msgr2=0x7efd6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd6c0051c0 0x7efd6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8359e640 1 -- 172.21.15.72:0/2749293428 reap_dead start 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.018+0000 7efd8259c640 1 -- 172.21.15.72:0/2749293428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c003520 con 0x7efd6c004800 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8259c640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd6c004c80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7efd840b69e0 tx=0x7efd6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62133cad319dbdda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78034070 con 0x7efd6c004800 2024-03-06T07:24:50.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd78030400 con 0x7efd6c004800 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd81d9b640 1 -- 172.21.15.72:0/2749293428 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7800f040 con 0x7efd6c004800 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 -- 172.21.15.72:0/2749293428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 msgr2=0x7efd6c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd6c004c80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7efd840b69e0 tx=0x7efd6c005830 comp rx=0 tx=0).stop 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 -- 172.21.15.72:0/2749293428 shutdown_connections 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 --2- 172.21.15.72:0/2749293428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd6c004c80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 -- 172.21.15.72:0/2749293428 >> 172.21.15.72:0/2749293428 conn(0x7efd840b35a0 msgr2=0x7efd840b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 -- 172.21.15.72:0/2749293428 shutdown_connections 2024-03-06T07:24:50.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.019+0000 7efd8919a640 1 -- 172.21.15.72:0/2749293428 wait complete. 2024-03-06T07:24:50.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.021+0000 7efd8919a640 1 Processor -- start 2024-03-06T07:24:50.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.021+0000 7efd8919a640 1 -- start start 2024-03-06T07:24:50.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd84156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd84155210 0x7efd840ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd8414e310 0x7efd840cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840cf850 con 0x7efd6c004800 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd840b4d40 con 0x7efd84155210 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd840cf9d0 con 0x7efd8414e310 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd84156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd84156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56598/0 (socket says 172.21.15.72:56598) 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 -- 172.21.15.72:0/2561367059 learned_addr learned my addr 172.21.15.72:0/2561367059 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 -- 172.21.15.72:0/2561367059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd8414e310 msgr2=0x7efd840cabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd8414e310 0x7efd840cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 -- 172.21.15.72:0/2561367059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd84155210 msgr2=0x7efd840ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd84155210 0x7efd840ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 -- 172.21.15.72:0/2561367059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd840c8480 con 0x7efd6c004800 2024-03-06T07:24:50.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd82d9d640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd84156c80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7efd7400e800 tx=0x7efd7400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400c7f0 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd74018070 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd840c8760 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400cbd0 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.022+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd841571c0 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.023+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8404f330 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.024+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd74013420 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.024+0000 7efd6b7fe640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.024+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efd74089a10 con 0x7efd6c004800 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.024+0000 7efd8259c640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.024+0000 7efd8259c640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efd78036000 tx=0x7efd78030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.028+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd74058c70 con 0x7efd6c004800 2024-03-06T07:24:50.064 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:50.066 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:24:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.206+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efd840cf3d0 con 0x7efd6c004800 2024-03-06T07:24:50.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.206+0000 7efd6b7fe640 1 -- 172.21.15.72:0/2561367059 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+65 (secure 0 0 0) 0x7efd74058570 con 0x7efd6c004800 2024-03-06T07:24:50.208 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:50.209 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:24:50.209 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:24:50.209 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:24:50.209 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:24:50.209 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:24:50.209 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:24:50.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 msgr2=0x7efd7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efd78036000 tx=0x7efd78030000 comp rx=0 tx=0).stop 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 msgr2=0x7efd84156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd84156c80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7efd7400e800 tx=0x7efd7400ecc0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 shutdown_connections 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd7006b500 0x7efd7006d9c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd8414e310 0x7efd840cabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd84155210 0x7efd840ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 --2- 172.21.15.72:0/2561367059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd6c004800 0x7efd84156c80 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 >> 172.21.15.72:0/2561367059 conn(0x7efd840b35a0 msgr2=0x7efd840ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 shutdown_connections 2024-03-06T07:24:50.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.208+0000 7efd8919a640 1 -- 172.21.15.72:0/2561367059 wait complete. 2024-03-06T07:24:50.221 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:24:50.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:24:50.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.323+0000 7fa07bc8d640 1 Processor -- start 2024-03-06T07:24:50.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.324+0000 7fa07bc8d640 1 -- start start 2024-03-06T07:24:50.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.324+0000 7fa07bc8d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa07415ac00 0x7fa07406c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.324+0000 7fa07bc8d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0740708b0 0x7fa074070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.324+0000 7fa07bc8d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07415afe0 con 0x7fa07415e7a0 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.324+0000 7fa07bc8d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa074071250 con 0x7fa07415ac00 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.324+0000 7fa07bc8d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0740cdc10 con 0x7fa0740708b0 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa07415ac00 0x7fa07406c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa07415ac00 0x7fa07406c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56612/0 (socket says 172.21.15.72:56612) 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 -- 172.21.15.72:0/387503010 learned_addr learned my addr 172.21.15.72:0/387503010 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 -- 172.21.15.72:0/387503010 >> v2:172.21.15.146:3300/0 conn(0x7fa0740708b0 msgr2=0x7fa074070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 --2- 172.21.15.72:0/387503010 >> v2:172.21.15.146:3300/0 conn(0x7fa0740708b0 0x7fa074070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 -- 172.21.15.72:0/387503010 >> v1:172.21.15.146:6789/0 conn(0x7fa07415e7a0 legacy=0x7fa07415ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 -- 172.21.15.72:0/387503010 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0740cd000 con 0x7fa07415ac00 2024-03-06T07:24:50.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.325+0000 7fa079a29640 1 --2- 172.21.15.72:0/387503010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415ac00 0x7fa07406c910 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa074074f80 tx=0x7fa07002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900d9f8e1c98285a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa078a27640 1 -- 172.21.15.72:0/387503010 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa07002e070 con 0x7fa07415ac00 2024-03-06T07:24:50.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa078a27640 1 -- 172.21.15.72:0/387503010 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa070037070 con 0x7fa07415ac00 2024-03-06T07:24:50.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa078a27640 1 -- 172.21.15.72:0/387503010 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa070031a00 con 0x7fa07415ac00 2024-03-06T07:24:50.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 -- 172.21.15.72:0/387503010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415ac00 msgr2=0x7fa07406c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/387503010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415ac00 0x7fa07406c910 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fa074074f80 tx=0x7fa07002b880 comp rx=0 tx=0).stop 2024-03-06T07:24:50.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 -- 172.21.15.72:0/387503010 shutdown_connections 2024-03-06T07:24:50.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/387503010 >> v2:172.21.15.146:3300/0 conn(0x7fa0740708b0 0x7fa074070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/387503010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415ac00 0x7fa07406c910 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 -- 172.21.15.72:0/387503010 >> 172.21.15.72:0/387503010 conn(0x7fa0740b1650 msgr2=0x7fa0740b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 -- 172.21.15.72:0/387503010 shutdown_connections 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.326+0000 7fa07bc8d640 1 -- 172.21.15.72:0/387503010 wait complete. 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.327+0000 7fa07bc8d640 1 Processor -- start 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.327+0000 7fa07bc8d640 1 -- start start 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07bc8d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0740708b0 0x7fa07414b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07bc8d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa07415ac00 0x7fa0740aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07bc8d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 0x7fa074158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07bc8d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740cd8a0 con 0x7fa07415e7a0 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07bc8d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0740cd5c0 con 0x7fa0740708b0 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07bc8d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07406ce50 con 0x7fa07415ac00 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07a22a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 0x7fa074158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07a22a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 0x7fa074158780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56622/0 (socket says 172.21.15.72:56622) 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07a22a640 1 -- 172.21.15.72:0/4144636491 learned_addr learned my addr 172.21.15.72:0/4144636491 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa079a29640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0740708b0 0x7fa07414b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa07a22a640 1 -- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa07415ac00 msgr2=0x7fa0740aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.328+0000 7fa079228640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa07415ac00 0x7fa0740aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07a22a640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa07415ac00 0x7fa0740aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07a22a640 1 -- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0740708b0 msgr2=0x7fa07414b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07a22a640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0740708b0 0x7fa07414b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07a22a640 1 -- 172.21.15.72:0/4144636491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07414b900 con 0x7fa07415e7a0 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa079a29640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0740708b0 0x7fa07414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07a22a640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 0x7fa074158780 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b990 tx=0x7fa06c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c019070 con 0x7fa07415e7a0 2024-03-06T07:24:50.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c00cba0 con 0x7fa07415e7a0 2024-03-06T07:24:50.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0740ad9e0 con 0x7fa07415e7a0 2024-03-06T07:24:50.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06c014440 con 0x7fa07415e7a0 2024-03-06T07:24:50.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.329+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0740adfa0 con 0x7fa07415e7a0 2024-03-06T07:24:50.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.330+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07404f330 con 0x7fa07415e7a0 2024-03-06T07:24:50.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.330+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa06c01c070 con 0x7fa07415e7a0 2024-03-06T07:24:50.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.331+0000 7fa06a7fc640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa05806b4d0 0x7fa05806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:50.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.331+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa06c089c30 con 0x7fa07415e7a0 2024-03-06T07:24:50.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.331+0000 7fa079a29640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa05806b4d0 0x7fa05806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:50.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.331+0000 7fa079a29640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa05806b4d0 0x7fa05806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa070036000 tx=0x7fa070008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:50.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.335+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa07414b900 con 0x7fa07415e7a0 2024-03-06T07:24:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.506+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa074074880 con 0x7fa07415e7a0 2024-03-06T07:24:50.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.943+0000 7fa06a7fc640 1 -- 172.21.15.72:0/4144636491 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v68) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa06c058e00 con 0x7fa07415e7a0 2024-03-06T07:24:50.945 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:24:50.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa05806b4d0 msgr2=0x7fa05806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa05806b4d0 0x7fa05806d990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa070036000 tx=0x7fa070008030 comp rx=0 tx=0).stop 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 msgr2=0x7fa074158780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 0x7fa074158780 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa06c00b990 tx=0x7fa06c00be50 comp rx=0 tx=0).stop 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 shutdown_connections 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa05806b4d0 0x7fa05806d990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa07415e7a0 0x7fa074158780 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa07415ac00 0x7fa0740aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 --2- 172.21.15.72:0/4144636491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0740708b0 0x7fa07414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 >> 172.21.15.72:0/4144636491 conn(0x7fa0740b1650 msgr2=0x7fa0740b3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 shutdown_connections 2024-03-06T07:24:50.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:50.945+0000 7fa07bc8d640 1 -- 172.21.15.72:0/4144636491 wait complete. 2024-03-06T07:24:50.961 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:24:51.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.066+0000 7f98ffb8a640 1 Processor -- start 2024-03-06T07:24:51.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98ffb8a640 1 -- start start 2024-03-06T07:24:51.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98ffb8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98f81614d0 0x7f98f80d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98ffb8a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f81618b0 con 0x7f98f80d1910 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98ffb8a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f80cf400 con 0x7f98f80d24b0 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98ffb8a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f80cf6a0 con 0x7f98f81614d0 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98fd926640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98f81614d0 0x7f98f80d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.067+0000 7f98fd125640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f98f80d1910 0x7f98f80d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49740/0 (socket says 172.21.15.72:49740) 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.068+0000 7f98fd125640 1 -- 172.21.15.72:0/494138544 learned_addr learned my addr 172.21.15.72:0/494138544 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.068+0000 7f98fd926640 1 -- 172.21.15.72:0/494138544 >> v1:172.21.15.146:6789/0 conn(0x7f98f80d24b0 legacy=0x7f98f80b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.068+0000 7f98fd926640 1 -- 172.21.15.72:0/494138544 >> v1:172.21.15.72:6789/0 conn(0x7f98f80d1910 legacy=0x7f98f80d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.068+0000 7f98fc924640 1 -- 172.21.15.72:0/494138544 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942501260 0 0) 0x7f98f81618b0 con 0x7f98f80d1910 2024-03-06T07:24:51.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.068+0000 7f98fd926640 1 -- 172.21.15.72:0/494138544 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f80cf980 con 0x7f98f81614d0 2024-03-06T07:24:51.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98fd926640 1 --2- 172.21.15.72:0/494138544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f81614d0 0x7f98f80d4860 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f98e80097a0 tx=0x7f98e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f42fc92b0d5bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98fc924640 1 -- 172.21.15.72:0/494138544 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8034070 con 0x7f98f81614d0 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98fc924640 1 -- 172.21.15.72:0/494138544 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e8030610 con 0x7f98f81614d0 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98fc924640 1 -- 172.21.15.72:0/494138544 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e80309e0 con 0x7f98f81614d0 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 -- 172.21.15.72:0/494138544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f81614d0 msgr2=0x7f98f80d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/494138544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f81614d0 0x7f98f80d4860 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f98e80097a0 tx=0x7f98e802b960 comp rx=0 tx=0).stop 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 -- 172.21.15.72:0/494138544 shutdown_connections 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/494138544 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f81614d0 0x7f98f80d4860 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 -- 172.21.15.72:0/494138544 >> 172.21.15.72:0/494138544 conn(0x7f98f80d3d40 msgr2=0x7f98f80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 -- 172.21.15.72:0/494138544 shutdown_connections 2024-03-06T07:24:51.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.069+0000 7f98ffb8a640 1 -- 172.21.15.72:0/494138544 wait complete. 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.070+0000 7f98ffb8a640 1 Processor -- start 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.070+0000 7f98ffb8a640 1 -- start start 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98ffb8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 0x7f98f815a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98ffb8a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d24b0 0x7f98f80c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98ffb8a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f81614d0 0x7f98f80cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98ffb8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f80b1540 con 0x7f98f80d1910 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98ffb8a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f80b0910 con 0x7f98f80d24b0 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98ffb8a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f80b16c0 con 0x7f98f81614d0 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fd926640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 0x7f98f815a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fd926640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 0x7f98f815a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56630/0 (socket says 172.21.15.72:56630) 2024-03-06T07:24:51.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fd926640 1 -- 172.21.15.72:0/3120260230 learned_addr learned my addr 172.21.15.72:0/3120260230 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fe127640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f81614d0 0x7f98f80cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fd125640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d24b0 0x7f98f80c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fd926640 1 -- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f81614d0 msgr2=0x7f98f80cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.071+0000 7f98fd926640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f81614d0 0x7f98f80cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98fd926640 1 -- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d24b0 msgr2=0x7f98f80c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98fd926640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d24b0 0x7f98f80c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98fd926640 1 -- 172.21.15.72:0/3120260230 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f80b9920 con 0x7f98f80d1910 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98fd926640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 0x7f98f815a700 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f98e802bf70 tx=0x7f98e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8034040 con 0x7f98f80d1910 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e803acc0 con 0x7f98f80d1910 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f80c7db0 con 0x7f98f80d1910 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8044630 con 0x7f98f80d1910 2024-03-06T07:24:51.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.072+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f80c9e80 con 0x7f98f80d1910 2024-03-06T07:24:51.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.073+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f80d14c0 con 0x7f98f80d1910 2024-03-06T07:24:51.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.074+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98e8037070 con 0x7f98f80d1910 2024-03-06T07:24:51.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.074+0000 7f98ee7fc640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98dc06b500 0x7f98dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:51.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.074+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f98e80abed0 con 0x7f98f80d1910 2024-03-06T07:24:51.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.074+0000 7f98fd125640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98dc06b500 0x7f98dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:51.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.074+0000 7f98fd125640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98dc06b500 0x7f98dc06d9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f98f80d0c70 tx=0x7f98f4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:51.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.079+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98e803f070 con 0x7f98f80d1910 2024-03-06T07:24:51.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.250+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f98f8068ff0 con 0x7f98f80d1910 2024-03-06T07:24:51.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.945+0000 7f98ee7fc640 1 -- 172.21.15.72:0/3120260230 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v69) v1 ==== 170+0+0 (secure 0 0 0) 0x7f98e807afc0 con 0x7f98f80d1910 2024-03-06T07:24:51.947 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:24:51.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98dc06b500 msgr2=0x7f98dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98dc06b500 0x7f98dc06d9c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f98f80d0c70 tx=0x7f98f4009040 comp rx=0 tx=0).stop 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 msgr2=0x7f98f815a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 0x7f98f815a700 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f98e802bf70 tx=0x7f98e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 shutdown_connections 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98dc06b500 0x7f98dc06d9c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f81614d0 0x7f98f80cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d24b0 0x7f98f80c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 --2- 172.21.15.72:0/3120260230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d1910 0x7f98f815a700 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:51.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.949+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 >> 172.21.15.72:0/3120260230 conn(0x7f98f80d3d40 msgr2=0x7f98f80ce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:51.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.950+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 shutdown_connections 2024-03-06T07:24:51.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:51.951+0000 7f98ffb8a640 1 -- 172.21.15.72:0/3120260230 wait complete. 2024-03-06T07:24:51.977 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:24:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.083+0000 7f1b0cf93640 1 Processor -- start 2024-03-06T07:24:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.083+0000 7f1b0cf93640 1 -- start start 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0cf93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b080b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0cf93640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b0815ba80 0x7f1b080b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0cf93640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0815e8a0 con 0x7f1b08158800 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0cf93640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0815e650 con 0x7f1b0815ba80 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0cf93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0815eb80 con 0x7f1b08158420 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0659c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b080b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b06d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b0815ba80 0x7f1b080b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b05d9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1b08158800 0x7f1b0815b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:42478/0 (socket says 172.21.15.72:42478) 2024-03-06T07:24:52.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b05d9b640 1 -- 172.21.15.72:0/3498873125 learned_addr learned my addr 172.21.15.72:0/3498873125 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0659c640 1 -- 172.21.15.72:0/3498873125 >> v2:172.21.15.146:3300/0 conn(0x7f1b0815ba80 msgr2=0x7f1b080b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0659c640 1 --2- 172.21.15.72:0/3498873125 >> v2:172.21.15.146:3300/0 conn(0x7f1b0815ba80 0x7f1b080b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0659c640 1 -- 172.21.15.72:0/3498873125 >> v1:172.21.15.146:6789/0 conn(0x7f1b08158800 legacy=0x7f1b0815b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b0659c640 1 -- 172.21.15.72:0/3498873125 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0815ee60 con 0x7f1b08158420 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.084+0000 7f1b06d9d640 1 --2- 172.21.15.72:0/3498873125 >> v2:172.21.15.146:3300/0 conn(0x7f1b0815ba80 0x7f1b080b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.085+0000 7f1b0659c640 1 --2- 172.21.15.72:0/3498873125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b080b35c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1af80097a0 tx=0x7f1af802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0285c24d30f6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.085+0000 7f1b0559a640 1 -- 172.21.15.72:0/3498873125 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8031780 con 0x7f1b08158420 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.085+0000 7f1b0559a640 1 -- 172.21.15.72:0/3498873125 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af8037070 con 0x7f1b08158420 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.085+0000 7f1b0559a640 1 -- 172.21.15.72:0/3498873125 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8031b40 con 0x7f1b08158420 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 -- 172.21.15.72:0/3498873125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 msgr2=0x7f1b080b35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/3498873125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b080b35c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1af80097a0 tx=0x7f1af802b820 comp rx=0 tx=0).stop 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 -- 172.21.15.72:0/3498873125 shutdown_connections 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/3498873125 >> v2:172.21.15.146:3300/0 conn(0x7f1b0815ba80 0x7f1b080b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/3498873125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b080b35c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 -- 172.21.15.72:0/3498873125 >> 172.21.15.72:0/3498873125 conn(0x7f1b0806afb0 msgr2=0x7f1b0806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 -- 172.21.15.72:0/3498873125 shutdown_connections 2024-03-06T07:24:52.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.086+0000 7f1b0cf93640 1 -- 172.21.15.72:0/3498873125 wait complete. 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.087+0000 7f1b0cf93640 1 Processor -- start 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.087+0000 7f1b0cf93640 1 -- start start 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0cf93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b08150be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0cf93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 0x7f1b081502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0cf93640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b081534e0 0x7f1b08153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0cf93640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08071ba0 con 0x7f1b0815ba80 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0cf93640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08071a20 con 0x7f1b081534e0 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0cf93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08071d20 con 0x7f1b08158420 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b05d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 0x7f1b081502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b05d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 0x7f1b081502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56642/0 (socket says 172.21.15.72:56642) 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b05d9b640 1 -- 172.21.15.72:0/1006582677 learned_addr learned my addr 172.21.15.72:0/1006582677 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:52.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b0659c640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b08150be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b06d9d640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b081534e0 0x7f1b08153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.088+0000 7f1b05d9b640 1 -- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 msgr2=0x7f1b08150be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b05d9b640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b08150be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b05d9b640 1 -- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b081534e0 msgr2=0x7f1b08153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b05d9b640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b081534e0 0x7f1b08153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b05d9b640 1 -- 172.21.15.72:0/1006582677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b080cfc40 con 0x7f1b0815ba80 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b05d9b640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 0x7f1b081502a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00c430 tx=0x7f1afc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc017070 con 0x7f1b0815ba80 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1afc005ce0 con 0x7f1b0815ba80 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b080af4a0 con 0x7f1b0815ba80 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1afc0072f0 con 0x7f1b0815ba80 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.089+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b080afa20 con 0x7f1b0815ba80 2024-03-06T07:24:52.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.090+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0804f330 con 0x7f1b0815ba80 2024-03-06T07:24:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.090+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1afc01a070 con 0x7f1b0815ba80 2024-03-06T07:24:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.091+0000 7f1af6ffd640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae806b500 0x7f1ae806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.091+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1afc0893c0 con 0x7f1b0815ba80 2024-03-06T07:24:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.091+0000 7f1b0659c640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae806b500 0x7f1ae806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:52.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.091+0000 7f1b0659c640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae806b500 0x7f1ae806d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1af8036000 tx=0x7f1af803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:52.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.095+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1afc057ce0 con 0x7f1b0815ba80 2024-03-06T07:24:52.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.302+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1b080b2a30 con 0x7f1b0815ba80 2024-03-06T07:24:52.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.967+0000 7f1af6ffd640 1 -- 172.21.15.72:0/1006582677 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v31) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1afc05bd80 con 0x7f1b0815ba80 2024-03-06T07:24:52.972 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 10 and data pool 11 2024-03-06T07:24:52.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.974+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae806b500 msgr2=0x7f1ae806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.974+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae806b500 0x7f1ae806d9c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1af8036000 tx=0x7f1af803d040 comp rx=0 tx=0).stop 2024-03-06T07:24:52.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.974+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 msgr2=0x7f1b081502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:52.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.974+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 0x7f1b081502a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00c430 tx=0x7f1afc00c830 comp rx=0 tx=0).stop 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.976+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 shutdown_connections 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.976+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae806b500 0x7f1ae806d9c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.976+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b081534e0 0x7f1b08153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.976+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b0815ba80 0x7f1b081502a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.976+0000 7f1b0cf93640 1 --2- 172.21.15.72:0/1006582677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b08158420 0x7f1b08150be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:52.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.976+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 >> 172.21.15.72:0/1006582677 conn(0x7f1b0806afb0 msgr2=0x7f1b0815cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:52.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.978+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 shutdown_connections 2024-03-06T07:24:52.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:52.979+0000 7f1b0cf93640 1 -- 172.21.15.72:0/1006582677 wait complete. 2024-03-06T07:24:52.998 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:53.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.098+0000 7f2f46e0f640 1 Processor -- start 2024-03-06T07:24:53.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.098+0000 7f2f46e0f640 1 -- start start 2024-03-06T07:24:53.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f46e0f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f400d2710 0x7f2f400b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f46e0f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400b3430 con 0x7f2f400d22b0 2024-03-06T07:24:53.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f46e0f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400b3710 con 0x7f2f4015dfe0 2024-03-06T07:24:53.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f46e0f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f400b31e0 con 0x7f2f400d2710 2024-03-06T07:24:53.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f453ac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f400d2710 0x7f2f400b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f3ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2f400d22b0 0x7f2f400b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37722/0 (socket says 172.21.15.72:37722) 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f3ffff640 1 -- 172.21.15.72:0/2169690601 learned_addr learned my addr 172.21.15.72:0/2169690601 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f453ac640 1 -- 172.21.15.72:0/2169690601 >> v1:172.21.15.146:6789/0 conn(0x7f2f4015dfe0 legacy=0x7f2f400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f453ac640 1 -- 172.21.15.72:0/2169690601 >> v1:172.21.15.72:6789/0 conn(0x7f2f400d22b0 legacy=0x7f2f400b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f453ac640 1 -- 172.21.15.72:0/2169690601 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f400b39f0 con 0x7f2f400d2710 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.099+0000 7f2f3f7fe640 1 -- 172.21.15.72:0/2169690601 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029483731 0 0) 0x7f2f400b3430 con 0x7f2f400d22b0 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.100+0000 7f2f453ac640 1 --2- 172.21.15.72:0/2169690601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f400d2710 0x7f2f400b4900 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009dc0 tx=0x7f2f3802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed1332e95aa4a43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.100+0000 7f2f3f7fe640 1 -- 172.21.15.72:0/2169690601 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38039070 con 0x7f2f400d2710 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.100+0000 7f2f3f7fe640 1 -- 172.21.15.72:0/2169690601 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3802cc40 con 0x7f2f400d2710 2024-03-06T07:24:53.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.100+0000 7f2f3f7fe640 1 -- 172.21.15.72:0/2169690601 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38034440 con 0x7f2f400d2710 2024-03-06T07:24:53.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 -- 172.21.15.72:0/2169690601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f400d2710 msgr2=0x7f2f400b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/2169690601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f400d2710 0x7f2f400b4900 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009dc0 tx=0x7f2f3802be90 comp rx=0 tx=0).stop 2024-03-06T07:24:53.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 -- 172.21.15.72:0/2169690601 shutdown_connections 2024-03-06T07:24:53.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/2169690601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f400d2710 0x7f2f400b4900 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 -- 172.21.15.72:0/2169690601 >> 172.21.15.72:0/2169690601 conn(0x7f2f400d3cd0 msgr2=0x7f2f400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 -- 172.21.15.72:0/2169690601 shutdown_connections 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.101+0000 7f2f46e0f640 1 -- 172.21.15.72:0/2169690601 wait complete. 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.102+0000 7f2f46e0f640 1 Processor -- start 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.102+0000 7f2f46e0f640 1 -- start start 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f46e0f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f400d22b0 0x7f2f401f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f46e0f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f400d2710 0x7f2f401f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f46e0f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 0x7f2f400b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f46e0f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400cfa00 con 0x7f2f4015dfe0 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f46e0f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b3d60 con 0x7f2f400d2710 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f46e0f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400d1c70 con 0x7f2f400d22b0 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 0x7f2f400b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 0x7f2f400b89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37652/0 (socket says 172.21.15.72:37652) 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 -- 172.21.15.72:0/211545073 learned_addr learned my addr 172.21.15.72:0/211545073 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f44bab640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f400d22b0 0x7f2f401f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f3ffff640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f400d2710 0x7f2f401f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 -- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f400d22b0 msgr2=0x7f2f401f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f400d22b0 0x7f2f401f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 -- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f400d2710 msgr2=0x7f2f401f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f400d2710 0x7f2f401f42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 -- 172.21.15.72:0/211545073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f400b7e50 con 0x7f2f4015dfe0 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f44bab640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f400d22b0 0x7f2f401f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f3ffff640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f400d2710 0x7f2f401f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f453ac640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 0x7f2f400b89d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2f38037040 tx=0x7f2f38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.103+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38044020 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.104+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f38034df0 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.104+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f400b6db0 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.104+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38039040 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.104+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f400b7370 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.104+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4004f330 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.105+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f38047020 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.105+0000 7f2f3d7fa640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f2406b4d0 0x7f2f2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.105+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f380acc00 con 0x7f2f4015dfe0 2024-03-06T07:24:53.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.105+0000 7f2f44bab640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f2406b4d0 0x7f2f2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.106+0000 7f2f44bab640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f2406b4d0 0x7f2f2406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400a810 tx=0x7f2f34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.110+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f380b3080 con 0x7f2f4015dfe0 2024-03-06T07:24:53.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.282+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f400747a0 con 0x7f2f4015dfe0 2024-03-06T07:24:53.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.283+0000 7f2f3d7fa640 1 -- 172.21.15.72:0/211545073 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16693 (secure 0 0 0) 0x7f2f3804e030 con 0x7f2f4015dfe0 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:53.285 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":70,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:52.303565+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f2406b4d0 msgr2=0x7f2f2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f2406b4d0 0x7f2f2406d990 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400a810 tx=0x7f2f34008040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 msgr2=0x7f2f400b89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 0x7f2f400b89d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f2f38037040 tx=0x7f2f38004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 shutdown_connections 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f2406b4d0 0x7f2f2406d990 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f4015dfe0 0x7f2f400b89d0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f400d2710 0x7f2f401f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 --2- 172.21.15.72:0/211545073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f400d22b0 0x7f2f401f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 >> 172.21.15.72:0/211545073 conn(0x7f2f400d3cd0 msgr2=0x7f2f4006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 shutdown_connections 2024-03-06T07:24:53.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.285+0000 7f2f46e0f640 1 -- 172.21.15.72:0/211545073 wait complete. 2024-03-06T07:24:53.298 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:53.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.395+0000 7f3291ca4640 1 Processor -- start 2024-03-06T07:24:53.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.395+0000 7f3291ca4640 1 -- start start 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f3291ca4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f328c0cce40 0x7f328c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f3291ca4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c071f40 con 0x7f328c1515e0 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f3291ca4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c070a70 con 0x7f328c0cd7a0 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f3291ca4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c0cd2f0 con 0x7f328c0cce40 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f328c0cce40 0x7f328c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f328c1515e0 0x7f328c071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37728/0 (socket says 172.21.15.72:37728) 2024-03-06T07:24:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328b7fe640 1 -- 172.21.15.72:0/343280301 learned_addr learned my addr 172.21.15.72:0/343280301 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328affd640 1 -- 172.21.15.72:0/343280301 >> v1:172.21.15.146:6789/0 conn(0x7f328c0cd7a0 legacy=0x7f328c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328affd640 1 -- 172.21.15.72:0/343280301 >> v1:172.21.15.72:6789/0 conn(0x7f328c1515e0 legacy=0x7f328c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328a7fc640 1 -- 172.21.15.72:0/343280301 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419645044 0 0) 0x7f328c071f40 con 0x7f328c1515e0 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.396+0000 7f328affd640 1 -- 172.21.15.72:0/343280301 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c070d50 con 0x7f328c0cce40 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.397+0000 7f328affd640 1 --2- 172.21.15.72:0/343280301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c0cce40 0x7f328c15b430 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f328000a940 tx=0x7f328002ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff73df9aa2eecbf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.397+0000 7f328a7fc640 1 -- 172.21.15.72:0/343280301 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328003a070 con 0x7f328c0cce40 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.397+0000 7f328a7fc640 1 -- 172.21.15.72:0/343280301 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328002dbe0 con 0x7f328c0cce40 2024-03-06T07:24:53.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.397+0000 7f328a7fc640 1 -- 172.21.15.72:0/343280301 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328000b040 con 0x7f328c0cce40 2024-03-06T07:24:53.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 -- 172.21.15.72:0/343280301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c0cce40 msgr2=0x7f328c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 --2- 172.21.15.72:0/343280301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c0cce40 0x7f328c15b430 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f328000a940 tx=0x7f328002ce30 comp rx=0 tx=0).stop 2024-03-06T07:24:53.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 -- 172.21.15.72:0/343280301 shutdown_connections 2024-03-06T07:24:53.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 --2- 172.21.15.72:0/343280301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c0cce40 0x7f328c15b430 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 -- 172.21.15.72:0/343280301 >> 172.21.15.72:0/343280301 conn(0x7f328c0cf8a0 msgr2=0x7f328c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 -- 172.21.15.72:0/343280301 shutdown_connections 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.398+0000 7f3291ca4640 1 -- 172.21.15.72:0/343280301 wait complete. 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.399+0000 7f3291ca4640 1 Processor -- start 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.399+0000 7f3291ca4640 1 -- start start 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f328c0cce40 0x7f328c1e58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f328c0cd7a0 0x7f328c1f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 0x7f328c1f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0cc3a0 con 0x7f328c1515e0 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c071310 con 0x7f328c0cd7a0 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c0cc520 con 0x7f328c0cce40 2024-03-06T07:24:53.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 0x7f328c1f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 0x7f328c1f5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37660/0 (socket says 172.21.15.72:37660) 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 -- 172.21.15.72:0/631608307 learned_addr learned my addr 172.21.15.72:0/631608307 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328b7fe640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f328c0cce40 0x7f328c1e58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328affd640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f328c0cd7a0 0x7f328c1f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 -- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f328c0cce40 msgr2=0x7f328c1e58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f328c0cce40 0x7f328c1e58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 -- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f328c0cd7a0 msgr2=0x7f328c1f53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f328c0cd7a0 0x7f328c1f53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 -- 172.21.15.72:0/631608307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c1e5e30 con 0x7f328c1515e0 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328b7fe640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f328c0cce40 0x7f328c1e58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328affd640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f328c0cd7a0 0x7f328c1f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f328bfff640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 0x7f328c1f5920 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f328400c7a0 tx=0x7f328400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284003b70 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3284003d30 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c1f5f50 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f328400aa30 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.400+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c1e4e10 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.401+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c068840 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.402+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f328400abd0 con 0x7f328c1515e0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.402+0000 7f3273fff640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.403+0000 7f328b7fe640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.403+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f328408a040 con 0x7f328c1515e0 2024-03-06T07:24:53.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.403+0000 7f328b7fe640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f327c000f60 tx=0x7f327c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.407+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3284058f50 con 0x7f328c1515e0 2024-03-06T07:24:53.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.579+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f328c0b4c30 con 0x7f328c1515e0 2024-03-06T07:24:53.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.580+0000 7f3273fff640 1 -- 172.21.15.72:0/631608307 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16693 (secure 0 0 0) 0x7f3284058850 con 0x7f328c1515e0 2024-03-06T07:24:53.582 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:53.582 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":70,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:52.303565+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 msgr2=0x7f327806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f327c000f60 tx=0x7f327c009040 comp rx=0 tx=0).stop 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 msgr2=0x7f328c1f5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 0x7f328c1f5920 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f328400c7a0 tx=0x7f328400cc60 comp rx=0 tx=0).stop 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 shutdown_connections 2024-03-06T07:24:53.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f327806b4d0 0x7f327806d990 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f328c1515e0 0x7f328c1f5920 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f328c0cd7a0 0x7f328c1f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 --2- 172.21.15.72:0/631608307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f328c0cce40 0x7f328c1e58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 >> 172.21.15.72:0/631608307 conn(0x7f328c0cf8a0 msgr2=0x7f328c0ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 shutdown_connections 2024-03-06T07:24:53.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.582+0000 7f3291ca4640 1 -- 172.21.15.72:0/631608307 wait complete. 2024-03-06T07:24:53.596 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:24:53.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.696+0000 7f10d25c7640 1 Processor -- start 2024-03-06T07:24:53.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.696+0000 7f10d25c7640 1 -- start start 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10d25c7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f10cc0d4860 0x7f10cc0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10d25c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 0x7f10cc0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10d25c7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc1618b0 con 0x7f10cc1614d0 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10d25c7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc0d4cf0 con 0x7f10cc0d4860 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10d25c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0cf400 con 0x7f10cc0b0e30 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10cbfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f10cc1614d0 0x7f10cc0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37744/0 (socket says 172.21.15.72:37744) 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.697+0000 7f10cbfff640 1 -- 172.21.15.72:0/4239271880 learned_addr learned my addr 172.21.15.72:0/4239271880 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:53.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746402919 0 0) 0x7f10cc1618b0 con 0x7f10cc1614d0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4003520 con 0x7f10cc1614d0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10d0b64640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 0x7f10cc0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/4239271880 >> v2:172.21.15.146:3300/0 conn(0x7f10cc0d4860 0x7f10cc0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4219655896 0 0) 0x7f10b4003520 con 0x7f10cc1614d0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 msgr2=0x7f10cc0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 0x7f10cc0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 >> v2:172.21.15.146:3300/0 conn(0x7f10cc0d4860 msgr2=0x7f10cc0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 --2- 172.21.15.72:0/4239271880 >> v2:172.21.15.146:3300/0 conn(0x7f10cc0d4860 0x7f10cc0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc0cf6a0 con 0x7f10cc1614d0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f10bc001670 con 0x7f10cc1614d0 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/4239271880 >> v2:172.21.15.146:3300/0 conn(0x7f10cc0d4860 0x7f10cc0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 >> v1:172.21.15.72:6789/0 conn(0x7f10cc1614d0 legacy=0x7f10cc0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10b40039b0 0x7f10b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10b4004800 0x7f10b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 0x7f10b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0d4cf0 con 0x7f10b4004800 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0cf400 con 0x7f10b40039b0 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b400b9c0 con 0x7f10cc0b0e30 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10b4004800 0x7f10b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10d0b64640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 0x7f10b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cbfff640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10b40039b0 0x7f10b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 -- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 msgr2=0x7f10b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10cc0b0e30 0x7f10b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 -- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10b40039b0 msgr2=0x7f10b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10b40039b0 0x7f10b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cb7fe640 1 -- 172.21.15.72:0/4239271880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4003520 con 0x7f10b4004800 2024-03-06T07:24:53.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.698+0000 7f10cbfff640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10b40039b0 0x7f10b4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.699+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10b4004800 0x7f10b4004c80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098a0 tx=0x7f10b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1749205926b17fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.699+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c000f040 con 0x7f10b4004800 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.699+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0004230 con 0x7f10b4004800 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.699+0000 7f10caffd640 1 -- 172.21.15.72:0/4239271880 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0003610 con 0x7f10b4004800 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d25c7640 1 -- 172.21.15.72:0/4239271880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10b4004800 msgr2=0x7f10b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d25c7640 1 --2- 172.21.15.72:0/4239271880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10b4004800 0x7f10b4004c80 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098a0 tx=0x7f10b4005830 comp rx=0 tx=0).stop 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d0b64640 1 -- 172.21.15.72:0/4239271880 reap_dead start 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d25c7640 1 -- 172.21.15.72:0/4239271880 shutdown_connections 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d25c7640 1 -- 172.21.15.72:0/4239271880 >> 172.21.15.72:0/4239271880 conn(0x7f10cc0d3d40 msgr2=0x7f10cc0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d25c7640 1 -- 172.21.15.72:0/4239271880 shutdown_connections 2024-03-06T07:24:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.700+0000 7f10d25c7640 1 -- 172.21.15.72:0/4239271880 wait complete. 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.701+0000 7f10d25c7640 1 Processor -- start 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.701+0000 7f10d25c7640 1 -- start start 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10b4004800 0x7f10cc1f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 0x7f10cc160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10cc0c7870 0x7f10cc0c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0b0910 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0cfc60 con 0x7f10cc0c7870 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0d2560 con 0x7f10b4004800 2024-03-06T07:24:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 0x7f10cc160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 0x7f10cc160050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37674/0 (socket says 172.21.15.72:37674) 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 -- 172.21.15.72:0/914107357 learned_addr learned my addr 172.21.15.72:0/914107357 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cbfff640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10b4004800 0x7f10cc1f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d0b64640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10cc0c7870 0x7f10cc0c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 -- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10b4004800 msgr2=0x7f10cc1f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10b4004800 0x7f10cc1f43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 -- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10cc0c7870 msgr2=0x7f10cc0c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10cc0c7870 0x7f10cc0c7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 -- 172.21.15.72:0/914107357 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc1f48e0 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cbfff640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10b4004800 0x7f10cc1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d0b64640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10cc0c7870 0x7f10cc0c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10cb7fe640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 0x7f10cc160050 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f10c002bff0 tx=0x7f10c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0047070 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0042400 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc0cafb0 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.702+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00427d0 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.703+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc0cb570 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.703+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc068ff0 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.704+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f10c004a070 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.704+0000 7f10c8ff9640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.704+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10c00abb30 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.704+0000 7f10cbfff640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.705+0000 7f10cbfff640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f10b806b4d0 0x7f10b806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f10bc000fb0 tx=0x7f10bc005d60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:53.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.708+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f10cc1f48e0 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.917+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f10cc0d13e0 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.970+0000 7f10c8ff9640 1 -- 172.21.15.72:0/914107357 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7f10c0033070 con 0x7f10cc0cd9c0 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.972+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f10b806b4d0 msgr2=0x7f10b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.972+0000 7f10d25c7640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f10b806b4d0 0x7f10b806d990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f10bc000fb0 tx=0x7f10bc005d60 comp rx=0 tx=0).stop 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.972+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 msgr2=0x7f10cc160050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.972+0000 7f10d25c7640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 0x7f10cc160050 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f10c002bff0 tx=0x7f10c0004730 comp rx=0 tx=0).stop 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 shutdown_connections 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f10b806b4d0 0x7f10b806d990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10cc0c7870 0x7f10cc0c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10cc0cd9c0 0x7f10cc160050 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 --2- 172.21.15.72:0/914107357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10b4004800 0x7f10cc1f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 >> 172.21.15.72:0/914107357 conn(0x7f10cc0d3d40 msgr2=0x7f10cc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:53.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 shutdown_connections 2024-03-06T07:24:53.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:53.973+0000 7f10d25c7640 1 -- 172.21.15.72:0/914107357 wait complete. 2024-03-06T07:24:53.986 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:24:53.987 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.087+0000 7f0f01c8f640 1 Processor -- start 2024-03-06T07:24:54.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.087+0000 7f0f01c8f640 1 -- start start 2024-03-06T07:24:54.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0f01c8f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0efc1586e0 0x7f0efc0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0f01c8f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc158ac0 con 0x7f0efc0aef80 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0f01c8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0efc15b2a0 con 0x7f0efc0d01a0 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0f01c8f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0efc15b540 con 0x7f0efc1586e0 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0efb7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0efc1586e0 0x7f0efc0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0efaffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0efc0aef80 0x7f0efc0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37760/0 (socket says 172.21.15.72:37760) 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0efaffd640 1 -- 172.21.15.72:0/440205828 learned_addr learned my addr 172.21.15.72:0/440205828 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0efb7fe640 1 -- 172.21.15.72:0/440205828 >> v1:172.21.15.146:6789/0 conn(0x7f0efc0d01a0 legacy=0x7f0efc0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.088+0000 7f0efb7fe640 1 -- 172.21.15.72:0/440205828 >> v1:172.21.15.72:6789/0 conn(0x7f0efc0aef80 legacy=0x7f0efc0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.089+0000 7f0efa7fc640 1 -- 172.21.15.72:0/440205828 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973166386 0 0) 0x7f0efc158ac0 con 0x7f0efc0aef80 2024-03-06T07:24:54.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.089+0000 7f0efb7fe640 1 -- 172.21.15.72:0/440205828 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc15b820 con 0x7f0efc1586e0 2024-03-06T07:24:54.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.089+0000 7f0efb7fe640 1 --2- 172.21.15.72:0/440205828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc1586e0 0x7f0efc0aea40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0097a0 tx=0x7f0eec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0d48e3fe25117d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.089+0000 7f0efa7fc640 1 -- 172.21.15.72:0/440205828 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec035020 con 0x7f0efc1586e0 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.089+0000 7f0efa7fc640 1 -- 172.21.15.72:0/440205828 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eec031830 con 0x7f0efc1586e0 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.089+0000 7f0efa7fc640 1 -- 172.21.15.72:0/440205828 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec031c00 con 0x7f0efc1586e0 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 -- 172.21.15.72:0/440205828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc1586e0 msgr2=0x7f0efc0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/440205828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc1586e0 0x7f0efc0aea40 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0097a0 tx=0x7f0eec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 -- 172.21.15.72:0/440205828 shutdown_connections 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/440205828 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc1586e0 0x7f0efc0aea40 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 -- 172.21.15.72:0/440205828 >> 172.21.15.72:0/440205828 conn(0x7f0efc06b190 msgr2=0x7f0efc06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 -- 172.21.15.72:0/440205828 shutdown_connections 2024-03-06T07:24:54.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.090+0000 7f0f01c8f640 1 -- 172.21.15.72:0/440205828 wait complete. 2024-03-06T07:24:54.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.091+0000 7f0f01c8f640 1 Processor -- start 2024-03-06T07:24:54.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.091+0000 7f0f01c8f640 1 -- start start 2024-03-06T07:24:54.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0f01c8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0efc0aef80 0x7f0efc1493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0f01c8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 0x7f0efc1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0f01c8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0efc1586e0 0x7f0efc152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0f01c8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc071350 con 0x7f0efc0d01a0 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0f01c8f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc15bde0 con 0x7f0efc0aef80 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0f01c8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0efc0714d0 con 0x7f0efc1586e0 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 0x7f0efc1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 0x7f0efc1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37684/0 (socket says 172.21.15.72:37684) 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 -- 172.21.15.72:0/3741917378 learned_addr learned my addr 172.21.15.72:0/3741917378 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efb7fe640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0efc0aef80 0x7f0efc1493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efbfff640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0efc1586e0 0x7f0efc152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 -- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0efc1586e0 msgr2=0x7f0efc152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0efc1586e0 0x7f0efc152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 -- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0efc0aef80 msgr2=0x7f0efc1493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0efc0aef80 0x7f0efc1493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 -- 172.21.15.72:0/3741917378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc06e310 con 0x7f0efc0d01a0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.092+0000 7f0efaffd640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 0x7f0efc1527a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c460 tx=0x7f0ef000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.093+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0018070 con 0x7f0efc0d01a0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.093+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef0007520 con 0x7f0efc0d01a0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.093+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0004a20 con 0x7f0efc0d01a0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.093+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0efc14fe80 con 0x7f0efc0d01a0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.093+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0efc150440 con 0x7f0efc0d01a0 2024-03-06T07:24:54.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.094+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0efc0732b0 con 0x7f0efc0d01a0 2024-03-06T07:24:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.094+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ef001b070 con 0x7f0efc0d01a0 2024-03-06T07:24:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.094+0000 7f0ee3fff640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee806b4d0 0x7f0ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.094+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ef00894f0 con 0x7f0efc0d01a0 2024-03-06T07:24:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.094+0000 7f0efb7fe640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee806b4d0 0x7f0ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:54.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.095+0000 7f0efb7fe640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee806b4d0 0x7f0ee806d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0eec036000 tx=0x7f0eec008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:54.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.099+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ef0057d20 con 0x7f0efc0d01a0 2024-03-06T07:24:54.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.307+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f0efc0723b0 con 0x7f0efc0d01a0 2024-03-06T07:24:54.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.982+0000 7f0ee3fff640 1 -- 172.21.15.72:0/3741917378 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0ef005bdc0 con 0x7f0efc0d01a0 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee806b4d0 msgr2=0x7f0ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee806b4d0 0x7f0ee806d990 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0eec036000 tx=0x7f0eec008030 comp rx=0 tx=0).stop 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 msgr2=0x7f0efc1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 0x7f0efc1527a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c460 tx=0x7f0ef000c920 comp rx=0 tx=0).stop 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 shutdown_connections 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee806b4d0 0x7f0ee806d990 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0efc1586e0 0x7f0efc152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0efc0d01a0 0x7f0efc1527a0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 --2- 172.21.15.72:0/3741917378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0efc0aef80 0x7f0efc1493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.984+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 >> 172.21.15.72:0/3741917378 conn(0x7f0efc06b190 msgr2=0x7f0efc0b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.985+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 shutdown_connections 2024-03-06T07:24:54.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:54.985+0000 7f0f01c8f640 1 -- 172.21.15.72:0/3741917378 wait complete. 2024-03-06T07:24:54.998 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:24:54.998 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:24:55.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.098+0000 7fb0a10b5640 1 Processor -- start 2024-03-06T07:24:55.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.098+0000 7fb0a10b5640 1 -- start start 2024-03-06T07:24:55.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb0a10b5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb09c06da40 0x7fb09c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb0a10b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb09c0cd7a0 0x7fb09c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb0a10b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09c0cc980 con 0x7fb09c1424f0 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb0a10b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09c0cd220 con 0x7fb09c0cd7a0 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb0a10b5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb09c0ccc60 con 0x7fb09c06da40 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09b59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb09c0cd7a0 0x7fb09c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb09c1424f0 0x7fb09c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37774/0 (socket says 172.21.15.72:37774) 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09ad9d640 1 -- 172.21.15.72:0/672967236 learned_addr learned my addr 172.21.15.72:0/672967236 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb093fff640 1 --2- 172.21.15.72:0/672967236 >> v2:172.21.15.146:3300/0 conn(0x7fb09c06da40 0x7fb09c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09b59e640 1 -- 172.21.15.72:0/672967236 >> v2:172.21.15.146:3300/0 conn(0x7fb09c06da40 msgr2=0x7fb09c0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09b59e640 1 --2- 172.21.15.72:0/672967236 >> v2:172.21.15.146:3300/0 conn(0x7fb09c06da40 0x7fb09c0cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09b59e640 1 -- 172.21.15.72:0/672967236 >> v1:172.21.15.72:6789/0 conn(0x7fb09c1424f0 legacy=0x7fb09c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09a59c640 1 -- 172.21.15.72:0/672967236 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253229604 0 0) 0x7fb09c0cc980 con 0x7fb09c1424f0 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.099+0000 7fb09b59e640 1 -- 172.21.15.72:0/672967236 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c0ccf40 con 0x7fb09c0cd7a0 2024-03-06T07:24:55.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb09b59e640 1 --2- 172.21.15.72:0/672967236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c15cef0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0b2a00 tx=0x7fb09402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4bbb8dbadef235 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb09a59c640 1 -- 172.21.15.72:0/672967236 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094039070 con 0x7fb09c0cd7a0 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb09a59c640 1 -- 172.21.15.72:0/672967236 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09402cc80 con 0x7fb09c0cd7a0 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb09a59c640 1 -- 172.21.15.72:0/672967236 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094034660 con 0x7fb09c0cd7a0 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb0a10b5640 1 -- 172.21.15.72:0/672967236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c0cd7a0 msgr2=0x7fb09c15cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/672967236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c15cef0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0b2a00 tx=0x7fb09402bed0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb0a10b5640 1 -- 172.21.15.72:0/672967236 shutdown_connections 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/672967236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c15cef0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/672967236 >> v2:172.21.15.146:3300/0 conn(0x7fb09c06da40 0x7fb09c0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.100+0000 7fb0a10b5640 1 -- 172.21.15.72:0/672967236 >> 172.21.15.72:0/672967236 conn(0x7fb09c0cf8a0 msgr2=0x7fb09c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.101+0000 7fb0a10b5640 1 -- 172.21.15.72:0/672967236 shutdown_connections 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.101+0000 7fb0a10b5640 1 -- 172.21.15.72:0/672967236 wait complete. 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 Processor -- start 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 -- start start 2024-03-06T07:24:55.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb09c06da40 0x7fb09c0ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c06b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 0x7fb09c0accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c071ce0 con 0x7fb09c1424f0 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c071b60 con 0x7fb09c0cd7a0 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.102+0000 7fb0a10b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c071e60 con 0x7fb09c06da40 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 0x7fb09c0accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 0x7fb09c0accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37694/0 (socket says 172.21.15.72:37694) 2024-03-06T07:24:55.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 -- 172.21.15.72:0/994827495 learned_addr learned my addr 172.21.15.72:0/994827495 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09ad9d640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb09c06da40 0x7fb09c0ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb093fff640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c06b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 -- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb09c06da40 msgr2=0x7fb09c0ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb09c06da40 0x7fb09c0ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 -- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb09c0cd7a0 msgr2=0x7fb09c06b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c06b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 -- 172.21.15.72:0/994827495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c0adac0 con 0x7fb09c1424f0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb093fff640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c06b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb09b59e640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 0x7fb09c0accb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb094009e80 tx=0x7fb094004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094039050 con 0x7fb09c1424f0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb094045070 con 0x7fb09c1424f0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c0ad7e0 con 0x7fb09c1424f0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094009450 con 0x7fb09c1424f0 2024-03-06T07:24:55.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.103+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c0ab930 con 0x7fb09c1424f0 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.104+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c04f330 con 0x7fb09c1424f0 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.104+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb09403e850 con 0x7fb09c1424f0 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.105+0000 7fb0937fe640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb07c06b4d0 0x7fb07c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.105+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0940ad0f0 con 0x7fb09c1424f0 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.105+0000 7fb09ad9d640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb07c06b4d0 0x7fb07c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.105+0000 7fb09ad9d640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb07c06b4d0 0x7fb07c06d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb08c007fd0 tx=0x7fb08c007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.109+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb09c0adac0 con 0x7fb09c1424f0 2024-03-06T07:24:55.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.315+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb09c0d0620 con 0x7fb09c1424f0 2024-03-06T07:24:55.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.316+0000 7fb0937fe640 1 -- 172.21.15.72:0/994827495 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fb09407c120 con 0x7fb09c1424f0 2024-03-06T07:24:55.318 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:55.318 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:24:55.319 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:24:55.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb07c06b4d0 msgr2=0x7fb07c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb07c06b4d0 0x7fb07c06d990 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb08c007fd0 tx=0x7fb08c007de0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 msgr2=0x7fb09c0accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 0x7fb09c0accb0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fb094009e80 tx=0x7fb094004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 shutdown_connections 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb07c06b4d0 0x7fb07c06d990 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb09c1424f0 0x7fb09c0accb0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb09c0cd7a0 0x7fb09c06b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 --2- 172.21.15.72:0/994827495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb09c06da40 0x7fb09c0ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 >> 172.21.15.72:0/994827495 conn(0x7fb09c0cf8a0 msgr2=0x7fb09c06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 shutdown_connections 2024-03-06T07:24:55.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.318+0000 7fb0a10b5640 1 -- 172.21.15.72:0/994827495 wait complete. 2024-03-06T07:24:55.331 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:24:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.429+0000 7fa09ccc3640 1 Processor -- start 2024-03-06T07:24:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.429+0000 7fa09ccc3640 1 -- start start 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa09ccc3640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa09ccc3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa0980b41f0 0x7fa0980b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa09ccc3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0980b3430 con 0x7fa09815dfe0 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa09ccc3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0980d2c50 con 0x7fa0980b41f0 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa09ccc3640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0980b31e0 con 0x7fa0980d22b0 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa096d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa0980b41f0 0x7fa0980b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa096d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa0980b41f0 0x7fa0980b4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37696/0 (socket says 172.21.15.72:37696) 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.430+0000 7fa096d9d640 1 -- 172.21.15.72:0/3807328132 learned_addr learned my addr 172.21.15.72:0/3807328132 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.431+0000 7fa095d9b640 1 --2- 172.21.15.72:0/3807328132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.431+0000 7fa096d9d640 1 -- 172.21.15.72:0/3807328132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 msgr2=0x7fa0980d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.431+0000 7fa096d9d640 1 --2- 172.21.15.72:0/3807328132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.431+0000 7fa096d9d640 1 -- 172.21.15.72:0/3807328132 >> v1:172.21.15.146:6789/0 conn(0x7fa09815dfe0 legacy=0x7fa0980cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.431+0000 7fa096d9d640 1 -- 172.21.15.72:0/3807328132 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0980b3710 con 0x7fa0980b41f0 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.431+0000 7fa096d9d640 1 --2- 172.21.15.72:0/3807328132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0980b41f0 0x7fa0980b4650 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa090009e50 tx=0x7fa09002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c1023e41f21297 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09559a640 1 -- 172.21.15.72:0/3807328132 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09002cb00 con 0x7fa0980b41f0 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09559a640 1 -- 172.21.15.72:0/3807328132 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09002cca0 con 0x7fa0980b41f0 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09559a640 1 -- 172.21.15.72:0/3807328132 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090034610 con 0x7fa0980b41f0 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3807328132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0980b41f0 msgr2=0x7fa0980b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3807328132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0980b41f0 0x7fa0980b4650 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa090009e50 tx=0x7fa09002bf60 comp rx=0 tx=0).stop 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3807328132 shutdown_connections 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3807328132 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0980b41f0 0x7fa0980b4650 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3807328132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3807328132 >> 172.21.15.72:0/3807328132 conn(0x7fa0980d3cd0 msgr2=0x7fa0980d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3807328132 shutdown_connections 2024-03-06T07:24:55.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.432+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3807328132 wait complete. 2024-03-06T07:24:55.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 Processor -- start 2024-03-06T07:24:55.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 -- start start 2024-03-06T07:24:55.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0980b41f0 0x7fa0980b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 0x7fa0981610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0980cfa00 con 0x7fa09815dfe0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0980b3d60 con 0x7fa0980b41f0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09ccc3640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0980d1c70 con 0x7fa0980d22b0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa096d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 0x7fa0981610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa096d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 0x7fa0981610d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37710/0 (socket says 172.21.15.72:37710) 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa096d9d640 1 -- 172.21.15.72:0/3849125729 learned_addr learned my addr 172.21.15.72:0/3849125729 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa09659c640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0980b41f0 0x7fa0980b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.434+0000 7fa095d9b640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa096d9d640 1 -- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 msgr2=0x7fa0980b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa096d9d640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980b8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa096d9d640 1 -- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0980b41f0 msgr2=0x7fa0980b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa096d9d640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0980b41f0 0x7fa0980b9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa096d9d640 1 -- 172.21.15.72:0/3849125729 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09815a700 con 0x7fa09815dfe0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa096d9d640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 0x7fa0981610d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa090036040 tx=0x7fa090004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090043020 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09002ce50 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0980b86f0 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090009510 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.435+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0980b8cb0 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.436+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa09804f330 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.436+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa090005d10 con 0x7fa09815dfe0 2024-03-06T07:24:55.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.437+0000 7fa08effd640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa07806b4d0 0x7fa07806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.437+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0900ac960 con 0x7fa09815dfe0 2024-03-06T07:24:55.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.437+0000 7fa09659c640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa07806b4d0 0x7fa07806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.437+0000 7fa09659c640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa07806b4d0 0x7fa07806d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa0800016e0 tx=0x7fa080009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.441+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa09007b190 con 0x7fa09815dfe0 2024-03-06T07:24:55.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.617+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0980ce910 con 0x7fa09815dfe0 2024-03-06T07:24:55.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.618+0000 7fa08effd640 1 -- 172.21.15.72:0/3849125729 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16680 (secure 0 0 0) 0x7fa09007f230 con 0x7fa09815dfe0 2024-03-06T07:24:55.620 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:55.621 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":71,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:53.957942+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:24:55.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa07806b4d0 msgr2=0x7fa07806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa07806b4d0 0x7fa07806d990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa0800016e0 tx=0x7fa080009040 comp rx=0 tx=0).stop 2024-03-06T07:24:55.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 msgr2=0x7fa0981610d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 0x7fa0981610d0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa090036040 tx=0x7fa090004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 shutdown_connections 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa07806b4d0 0x7fa07806d990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa09815dfe0 0x7fa0981610d0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0980d22b0 0x7fa0980b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 --2- 172.21.15.72:0/3849125729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0980b41f0 0x7fa0980b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 >> 172.21.15.72:0/3849125729 conn(0x7fa0980d3cd0 msgr2=0x7fa09806d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 shutdown_connections 2024-03-06T07:24:55.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.620+0000 7fa09ccc3640 1 -- 172.21.15.72:0/3849125729 wait complete. 2024-03-06T07:24:55.633 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:24:55.633 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.731+0000 7fd2ecd7a640 1 Processor -- start 2024-03-06T07:24:55.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.732+0000 7fd2ecd7a640 1 -- start start 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.732+0000 7fd2ecd7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e806c6d0 0x7fd2e815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.732+0000 7fd2ecd7a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2e8066d90 0x7fd2e80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.732+0000 7fd2ecd7a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80cd450 con 0x7fd2e80b4f70 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.732+0000 7fd2ecd7a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e815e9f0 con 0x7fd2e8066d90 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.732+0000 7fd2ecd7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80cda10 con 0x7fd2e806c6d0 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd2e80b4f70 0x7fd2e80b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37776/0 (socket says 172.21.15.72:37776) 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e659c640 1 -- 172.21.15.72:0/1914995739 learned_addr learned my addr 172.21.15.72:0/1914995739 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e5d9b640 1 --2- 172.21.15.72:0/1914995739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e806c6d0 0x7fd2e815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e6d9d640 1 --2- 172.21.15.72:0/1914995739 >> v2:172.21.15.146:3300/0 conn(0x7fd2e8066d90 0x7fd2e80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e559a640 1 -- 172.21.15.72:0/1914995739 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531190860 0 0) 0x7fd2e80cd450 con 0x7fd2e80b4f70 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e559a640 1 -- 172.21.15.72:0/1914995739 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d0003520 con 0x7fd2e80b4f70 2024-03-06T07:24:55.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e5d9b640 1 -- 172.21.15.72:0/1914995739 >> v2:172.21.15.146:3300/0 conn(0x7fd2e8066d90 msgr2=0x7fd2e80ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e5d9b640 1 --2- 172.21.15.72:0/1914995739 >> v2:172.21.15.146:3300/0 conn(0x7fd2e8066d90 0x7fd2e80ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e5d9b640 1 -- 172.21.15.72:0/1914995739 >> v1:172.21.15.72:6789/0 conn(0x7fd2e80b4f70 legacy=0x7fd2e80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e5d9b640 1 -- 172.21.15.72:0/1914995739 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80cd730 con 0x7fd2e806c6d0 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.733+0000 7fd2e6d9d640 1 --2- 172.21.15.72:0/1914995739 >> v2:172.21.15.146:3300/0 conn(0x7fd2e8066d90 0x7fd2e80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2e5d9b640 1 --2- 172.21.15.72:0/1914995739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e806c6d0 0x7fd2e815ee80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87ecc3edf2b5c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2e559a640 1 -- 172.21.15.72:0/1914995739 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8035020 con 0x7fd2e806c6d0 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2e559a640 1 -- 172.21.15.72:0/1914995739 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8031830 con 0x7fd2e806c6d0 2024-03-06T07:24:55.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2e559a640 1 -- 172.21.15.72:0/1914995739 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8031be0 con 0x7fd2e806c6d0 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/1914995739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e806c6d0 msgr2=0x7fd2e815ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/1914995739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e806c6d0 0x7fd2e815ee80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/1914995739 shutdown_connections 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/1914995739 >> v2:172.21.15.146:3300/0 conn(0x7fd2e8066d90 0x7fd2e80ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/1914995739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e806c6d0 0x7fd2e815ee80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/1914995739 >> 172.21.15.72:0/1914995739 conn(0x7fd2e80724d0 msgr2=0x7fd2e80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.734+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/1914995739 shutdown_connections 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.735+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/1914995739 wait complete. 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 Processor -- start 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 -- start start 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 0x7fd2e80cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80b4f70 0x7fd2e80d0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e814d520 0x7fd2e806d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80cd090 con 0x7fd2e8066d90 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e815f3c0 con 0x7fd2e80b4f70 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.736+0000 7fd2ecd7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80cd210 con 0x7fd2e814d520 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 0x7fd2e80cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 0x7fd2e80cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37716/0 (socket says 172.21.15.72:37716) 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 -- 172.21.15.72:0/3941875518 learned_addr learned my addr 172.21.15.72:0/3941875518 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e5d9b640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80b4f70 0x7fd2e80d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e6d9d640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e814d520 0x7fd2e806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 -- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e814d520 msgr2=0x7fd2e806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e814d520 0x7fd2e806d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 -- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80b4f70 msgr2=0x7fd2e80d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80b4f70 0x7fd2e80d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 -- 172.21.15.72:0/3941875518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80d0870 con 0x7fd2e8066d90 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2e659c640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 0x7fd2e80cde40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00c390 tx=0x7fd2dc00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc017070 con 0x7fd2e8066d90 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc005ce0 con 0x7fd2e8066d90 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e81f7080 con 0x7fd2e8066d90 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc0072f0 con 0x7fd2e8066d90 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.737+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e81f7640 con 0x7fd2e8066d90 2024-03-06T07:24:55.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.738+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e804f330 con 0x7fd2e8066d90 2024-03-06T07:24:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.739+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2dc01a070 con 0x7fd2e8066d90 2024-03-06T07:24:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.739+0000 7fd2d6ffd640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.739+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2dc0894a0 con 0x7fd2e8066d90 2024-03-06T07:24:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.739+0000 7fd2e5d9b640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:55.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.739+0000 7fd2e5d9b640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:55.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.743+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2dc057ce0 con 0x7fd2e8066d90 2024-03-06T07:24:55.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.912+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2e81f7e40 con 0x7fd2c806b4d0 2024-03-06T07:24:55.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.918+0000 7fd2d6ffd640 1 -- 172.21.15.72:0/3941875518 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7fd2e81f7e40 con 0x7fd2c806b4d0 2024-03-06T07:24:55.923 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:55.925 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:55.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 msgr2=0x7fd2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d8008030 comp rx=0 tx=0).stop 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 msgr2=0x7fd2e80cde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 0x7fd2e80cde40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00c390 tx=0x7fd2dc00c790 comp rx=0 tx=0).stop 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 shutdown_connections 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e814d520 0x7fd2e806d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80b4f70 0x7fd2e80d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 --2- 172.21.15.72:0/3941875518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e8066d90 0x7fd2e80cde40 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 >> 172.21.15.72:0/3941875518 conn(0x7fd2e80724d0 msgr2=0x7fd2e80728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 shutdown_connections 2024-03-06T07:24:55.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:55.921+0000 7fd2ecd7a640 1 -- 172.21.15.72:0/3941875518 wait complete. 2024-03-06T07:24:55.934 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-03-06T07:24:53.980256+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257064,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717656,"statfs":{"total":773094113280,"available":772830879744,"internally_reserved":0,"allocated":7806976,"data_stored":3935832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000600"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955811+0000","last_change":"2024-03-06T07:24:51.955811+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955811+0000","last_undegraded":"2024-03-06T07:24:51.955811+0000","last_fullsized":"2024-03-06T07:24:51.955811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956564+0000","last_change":"2024-03-06T07:24:51.956564+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956564+0000","last_undegraded":"2024-03-06T07:24:51.956564+0000","last_fullsized":"2024-03-06T07:24:51.956564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949967+0000","last_change":"2024-03-06T07:24:51.949967+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949967+0000","last_undegraded":"2024-03-06T07:24:51.949967+0000","last_fullsized":"2024-03-06T07:24:51.949967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954692+0000","last_change":"2024-03-06T07:24:51.954692+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954692+0000","last_undegraded":"2024-03-06T07:24:51.954692+0000","last_fullsized":"2024-03-06T07:24:51.954692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.957393+0000","last_change":"2024-03-06T07:24:51.957393+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.957393+0000","last_undegraded":"2024-03-06T07:24:51.957393+0000","last_fullsized":"2024-03-06T07:24:51.957393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956546+0000","last_change":"2024-03-06T07:24:51.956546+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956546+0000","last_undegraded":"2024-03-06T07:24:51.956546+0000","last_fullsized":"2024-03-06T07:24:51.956546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950665+0000","last_change":"2024-03-06T07:24:51.950665+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950665+0000","last_undegraded":"2024-03-06T07:24:51.950665+0000","last_fullsized":"2024-03-06T07:24:51.950665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954574+0000","last_change":"2024-03-06T07:24:51.954574+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954574+0000","last_undegraded":"2024-03-06T07:24:51.954574+0000","last_fullsized":"2024-03-06T07:24:51.954574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956937+0000","last_change":"2024-03-06T07:24:51.956937+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956937+0000","last_undegraded":"2024-03-06T07:24:51.956937+0000","last_fullsized":"2024-03-06T07:24:51.956937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953539+0000","last_change":"2024-03-06T07:24:51.953539+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953539+0000","last_undegraded":"2024-03-06T07:24:51.953539+0000","last_fullsized":"2024-03-06T07:24:51.953539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954128+0000","last_change":"2024-03-06T07:24:51.954128+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954128+0000","last_undegraded":"2024-03-06T07:24:51.954128+0000","last_fullsized":"2024-03-06T07:24:51.954128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954265+0000","last_change":"2024-03-06T07:24:51.954265+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954265+0000","last_undegraded":"2024-03-06T07:24:51.954265+0000","last_fullsized":"2024-03-06T07:24:51.954265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956604+0000","last_change":"2024-03-06T07:24:51.956604+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956604+0000","last_undegraded":"2024-03-06T07:24:51.956604+0000","last_fullsized":"2024-03-06T07:24:51.956604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955792+0000","last_change":"2024-03-06T07:24:51.955792+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955792+0000","last_undegraded":"2024-03-06T07:24:51.955792+0000","last_fullsized":"2024-03-06T07:24:51.955792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956049+0000","last_change":"2024-03-06T07:24:51.956049+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956049+0000","last_undegraded":"2024-03-06T07:24:51.956049+0000","last_fullsized":"2024-03-06T07:24:51.956049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951835+0000","last_change":"2024-03-06T07:24:51.951835+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951835+0000","last_undegraded":"2024-03-06T07:24:51.951835+0000","last_fullsized":"2024-03-06T07:24:51.951835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956373+0000","last_change":"2024-03-06T07:24:51.956373+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956373+0000","last_undegraded":"2024-03-06T07:24:51.956373+0000","last_fullsized":"2024-03-06T07:24:51.956373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953867+0000","last_change":"2024-03-06T07:24:51.953867+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953867+0000","last_undegraded":"2024-03-06T07:24:51.953867+0000","last_fullsized":"2024-03-06T07:24:51.953867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955657+0000","last_change":"2024-03-06T07:24:51.955657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955657+0000","last_undegraded":"2024-03-06T07:24:51.955657+0000","last_fullsized":"2024-03-06T07:24:51.955657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955630+0000","last_change":"2024-03-06T07:24:51.955630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955630+0000","last_undegraded":"2024-03-06T07:24:51.955630+0000","last_fullsized":"2024-03-06T07:24:51.955630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954554+0000","last_change":"2024-03-06T07:24:51.954554+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954554+0000","last_undegraded":"2024-03-06T07:24:51.954554+0000","last_fullsized":"2024-03-06T07:24:51.954554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952981+0000","last_change":"2024-03-06T07:24:51.952981+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952981+0000","last_undegraded":"2024-03-06T07:24:51.952981+0000","last_fullsized":"2024-03-06T07:24:51.952981+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952956+0000","last_change":"2024-03-06T07:24:51.952956+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952956+0000","last_undegraded":"2024-03-06T07:24:51.952956+0000","last_fullsized":"2024-03-06T07:24:51.952956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951873+0000","last_change":"2024-03-06T07:24:51.951873+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951873+0000","last_undegraded":"2024-03-06T07:24:51.951873+0000","last_fullsized":"2024-03-06T07:24:51.951873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953582+0000","last_change":"2024-03-06T07:24:51.953582+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953582+0000","last_undegraded":"2024-03-06T07:24:51.953582+0000","last_fullsized":"2024-03-06T07:24:51.953582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950728+0000","last_change":"2024-03-06T07:24:51.950728+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950728+0000","last_undegraded":"2024-03-06T07:24:51.950728+0000","last_fullsized":"2024-03-06T07:24:51.950728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949352+0000","last_change":"2024-03-06T07:24:51.949352+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949352+0000","last_undegraded":"2024-03-06T07:24:51.949352+0000","last_fullsized":"2024-03-06T07:24:51.949352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953082+0000","last_change":"2024-03-06T07:24:51.953082+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953082+0000","last_undegraded":"2024-03-06T07:24:51.953082+0000","last_fullsized":"2024-03-06T07:24:51.953082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949990+0000","last_change":"2024-03-06T07:24:51.949990+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949990+0000","last_undegraded":"2024-03-06T07:24:51.949990+0000","last_fullsized":"2024-03-06T07:24:51.949990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950896+0000","last_change":"2024-03-06T07:24:51.950896+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950896+0000","last_undegraded":"2024-03-06T07:24:51.950896+0000","last_fullsized":"2024-03-06T07:24:51.950896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949630+0000","last_change":"2024-03-06T07:24:51.949630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949630+0000","last_undegraded":"2024-03-06T07:24:51.949630+0000","last_fullsized":"2024-03-06T07:24:51.949630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954177+0000","last_change":"2024-03-06T07:24:51.954177+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954177+0000","last_undegraded":"2024-03-06T07:24:51.954177+0000","last_fullsized":"2024-03-06T07:24:51.954177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953434+0000","last_change":"2024-03-06T07:24:51.953434+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953434+0000","last_undegraded":"2024-03-06T07:24:51.953434+0000","last_fullsized":"2024-03-06T07:24:51.953434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966812+0000","last_change":"2024-03-06T07:24:51.962181+0000","last_active":"2024-03-06T07:24:51.966812+0000","last_peered":"2024-03-06T07:24:51.966812+0000","last_clean":"2024-03-06T07:24:51.966812+0000","last_became_active":"2024-03-06T07:24:51.961932+0000","last_became_peered":"2024-03-06T07:24:51.961932+0000","last_unstale":"2024-03-06T07:24:51.966812+0000","last_undegraded":"2024-03-06T07:24:51.966812+0000","last_fullsized":"2024-03-06T07:24:51.966812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955019+0000","last_change":"2024-03-06T07:24:51.955019+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955019+0000","last_undegraded":"2024-03-06T07:24:51.955019+0000","last_fullsized":"2024-03-06T07:24:51.955019+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964255+0000","last_change":"2024-03-06T07:24:51.962165+0000","last_active":"2024-03-06T07:24:51.964255+0000","last_peered":"2024-03-06T07:24:51.964255+0000","last_clean":"2024-03-06T07:24:51.964255+0000","last_became_active":"2024-03-06T07:24:51.961847+0000","last_became_peered":"2024-03-06T07:24:51.961847+0000","last_unstale":"2024-03-06T07:24:51.964255+0000","last_undegraded":"2024-03-06T07:24:51.964255+0000","last_fullsized":"2024-03-06T07:24:51.964255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956036+0000","last_change":"2024-03-06T07:24:51.956036+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956036+0000","last_undegraded":"2024-03-06T07:24:51.956036+0000","last_fullsized":"2024-03-06T07:24:51.956036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967209+0000","last_change":"2024-03-06T07:24:51.965487+0000","last_active":"2024-03-06T07:24:51.967209+0000","last_peered":"2024-03-06T07:24:51.967209+0000","last_clean":"2024-03-06T07:24:51.967209+0000","last_became_active":"2024-03-06T07:24:51.965210+0000","last_became_peered":"2024-03-06T07:24:51.965210+0000","last_unstale":"2024-03-06T07:24:51.967209+0000","last_undegraded":"2024-03-06T07:24:51.967209+0000","last_fullsized":"2024-03-06T07:24:51.967209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:18.007631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953707+0000","last_change":"2024-03-06T07:24:51.953707+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953707+0000","last_undegraded":"2024-03-06T07:24:51.953707+0000","last_fullsized":"2024-03-06T07:24:51.953707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967180+0000","last_change":"2024-03-06T07:24:51.963677+0000","last_active":"2024-03-06T07:24:51.967180+0000","last_peered":"2024-03-06T07:24:51.967180+0000","last_clean":"2024-03-06T07:24:51.967180+0000","last_became_active":"2024-03-06T07:24:51.963383+0000","last_became_peered":"2024-03-06T07:24:51.963383+0000","last_unstale":"2024-03-06T07:24:51.967180+0000","last_undegraded":"2024-03-06T07:24:51.967180+0000","last_fullsized":"2024-03-06T07:24:51.967180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:05.163132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949619+0000","last_change":"2024-03-06T07:24:51.949619+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949619+0000","last_undegraded":"2024-03-06T07:24:51.949619+0000","last_fullsized":"2024-03-06T07:24:51.949619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963718+0000","last_change":"2024-03-06T07:24:51.962180+0000","last_active":"2024-03-06T07:24:51.963718+0000","last_peered":"2024-03-06T07:24:51.963718+0000","last_clean":"2024-03-06T07:24:51.963718+0000","last_became_active":"2024-03-06T07:24:51.961911+0000","last_became_peered":"2024-03-06T07:24:51.961911+0000","last_unstale":"2024-03-06T07:24:51.963718+0000","last_undegraded":"2024-03-06T07:24:51.963718+0000","last_fullsized":"2024-03-06T07:24:51.963718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:40.456147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954185+0000","last_change":"2024-03-06T07:24:51.954185+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954185+0000","last_undegraded":"2024-03-06T07:24:51.954185+0000","last_fullsized":"2024-03-06T07:24:51.954185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967273+0000","last_change":"2024-03-06T07:24:51.964997+0000","last_active":"2024-03-06T07:24:51.967273+0000","last_peered":"2024-03-06T07:24:51.967273+0000","last_clean":"2024-03-06T07:24:51.967273+0000","last_became_active":"2024-03-06T07:24:51.964708+0000","last_became_peered":"2024-03-06T07:24:51.964708+0000","last_unstale":"2024-03-06T07:24:51.967273+0000","last_undegraded":"2024-03-06T07:24:51.967273+0000","last_fullsized":"2024-03-06T07:24:51.967273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:33.109374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954642+0000","last_change":"2024-03-06T07:24:51.954642+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954642+0000","last_undegraded":"2024-03-06T07:24:51.954642+0000","last_fullsized":"2024-03-06T07:24:51.954642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963914+0000","last_change":"2024-03-06T07:24:51.962002+0000","last_active":"2024-03-06T07:24:51.963914+0000","last_peered":"2024-03-06T07:24:51.963914+0000","last_clean":"2024-03-06T07:24:51.963914+0000","last_became_active":"2024-03-06T07:24:51.961717+0000","last_became_peered":"2024-03-06T07:24:51.961717+0000","last_unstale":"2024-03-06T07:24:51.963914+0000","last_undegraded":"2024-03-06T07:24:51.963914+0000","last_fullsized":"2024-03-06T07:24:51.963914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954942+0000","last_change":"2024-03-06T07:24:51.954942+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954942+0000","last_undegraded":"2024-03-06T07:24:51.954942+0000","last_fullsized":"2024-03-06T07:24:51.954942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961350+0000","last_change":"2024-03-06T07:24:51.957748+0000","last_active":"2024-03-06T07:24:51.961350+0000","last_peered":"2024-03-06T07:24:51.961350+0000","last_clean":"2024-03-06T07:24:51.961350+0000","last_became_active":"2024-03-06T07:24:51.957465+0000","last_became_peered":"2024-03-06T07:24:51.957465+0000","last_unstale":"2024-03-06T07:24:51.961350+0000","last_undegraded":"2024-03-06T07:24:51.961350+0000","last_fullsized":"2024-03-06T07:24:51.961350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:53.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949800+0000","last_change":"2024-03-06T07:24:51.949800+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949800+0000","last_undegraded":"2024-03-06T07:24:51.949800+0000","last_fullsized":"2024-03-06T07:24:51.949800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967114+0000","last_change":"2024-03-06T07:24:51.962228+0000","last_active":"2024-03-06T07:24:51.967114+0000","last_peered":"2024-03-06T07:24:51.967114+0000","last_clean":"2024-03-06T07:24:51.967114+0000","last_became_active":"2024-03-06T07:24:51.961977+0000","last_became_peered":"2024-03-06T07:24:51.961977+0000","last_unstale":"2024-03-06T07:24:51.967114+0000","last_undegraded":"2024-03-06T07:24:51.967114+0000","last_fullsized":"2024-03-06T07:24:51.967114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:07.096047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955422+0000","last_change":"2024-03-06T07:24:51.955422+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955422+0000","last_undegraded":"2024-03-06T07:24:51.955422+0000","last_fullsized":"2024-03-06T07:24:51.955422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969507+0000","last_change":"2024-03-06T07:24:51.965861+0000","last_active":"2024-03-06T07:24:51.969507+0000","last_peered":"2024-03-06T07:24:51.969507+0000","last_clean":"2024-03-06T07:24:51.969507+0000","last_became_active":"2024-03-06T07:24:51.965498+0000","last_became_peered":"2024-03-06T07:24:51.965498+0000","last_unstale":"2024-03-06T07:24:51.969507+0000","last_undegraded":"2024-03-06T07:24:51.969507+0000","last_fullsized":"2024-03-06T07:24:51.969507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955625+0000","last_change":"2024-03-06T07:24:51.955625+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955625+0000","last_undegraded":"2024-03-06T07:24:51.955625+0000","last_fullsized":"2024-03-06T07:24:51.955625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.959867+0000","last_change":"2024-03-06T07:24:51.958271+0000","last_active":"2024-03-06T07:24:51.959867+0000","last_peered":"2024-03-06T07:24:51.959867+0000","last_clean":"2024-03-06T07:24:51.959867+0000","last_became_active":"2024-03-06T07:24:51.957988+0000","last_became_peered":"2024-03-06T07:24:51.957988+0000","last_unstale":"2024-03-06T07:24:51.959867+0000","last_undegraded":"2024-03-06T07:24:51.959867+0000","last_fullsized":"2024-03-06T07:24:51.959867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:30.871115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955262+0000","last_change":"2024-03-06T07:24:51.955262+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955262+0000","last_undegraded":"2024-03-06T07:24:51.955262+0000","last_fullsized":"2024-03-06T07:24:51.955262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961753+0000","last_change":"2024-03-06T07:24:51.958913+0000","last_active":"2024-03-06T07:24:51.961753+0000","last_peered":"2024-03-06T07:24:51.961753+0000","last_clean":"2024-03-06T07:24:51.961753+0000","last_became_active":"2024-03-06T07:24:51.958624+0000","last_became_peered":"2024-03-06T07:24:51.958624+0000","last_unstale":"2024-03-06T07:24:51.961753+0000","last_undegraded":"2024-03-06T07:24:51.961753+0000","last_fullsized":"2024-03-06T07:24:51.961753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954553+0000","last_change":"2024-03-06T07:24:51.954553+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954553+0000","last_undegraded":"2024-03-06T07:24:51.954553+0000","last_fullsized":"2024-03-06T07:24:51.954553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.962777+0000","last_change":"2024-03-06T07:24:51.960455+0000","last_active":"2024-03-06T07:24:51.962777+0000","last_peered":"2024-03-06T07:24:51.962777+0000","last_clean":"2024-03-06T07:24:51.962777+0000","last_became_active":"2024-03-06T07:24:51.960189+0000","last_became_peered":"2024-03-06T07:24:51.960189+0000","last_unstale":"2024-03-06T07:24:51.962777+0000","last_undegraded":"2024-03-06T07:24:51.962777+0000","last_fullsized":"2024-03-06T07:24:51.962777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949626+0000","last_change":"2024-03-06T07:24:51.949626+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949626+0000","last_undegraded":"2024-03-06T07:24:51.949626+0000","last_fullsized":"2024-03-06T07:24:51.949626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967289+0000","last_change":"2024-03-06T07:24:51.965187+0000","last_active":"2024-03-06T07:24:51.967289+0000","last_peered":"2024-03-06T07:24:51.967289+0000","last_clean":"2024-03-06T07:24:51.967289+0000","last_became_active":"2024-03-06T07:24:51.964934+0000","last_became_peered":"2024-03-06T07:24:51.964934+0000","last_unstale":"2024-03-06T07:24:51.967289+0000","last_undegraded":"2024-03-06T07:24:51.967289+0000","last_fullsized":"2024-03-06T07:24:51.967289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:35.381581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954508+0000","last_change":"2024-03-06T07:24:51.954508+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954508+0000","last_undegraded":"2024-03-06T07:24:51.954508+0000","last_fullsized":"2024-03-06T07:24:51.954508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966779+0000","last_change":"2024-03-06T07:24:51.961258+0000","last_active":"2024-03-06T07:24:51.966779+0000","last_peered":"2024-03-06T07:24:51.966779+0000","last_clean":"2024-03-06T07:24:51.966779+0000","last_became_active":"2024-03-06T07:24:51.961000+0000","last_became_peered":"2024-03-06T07:24:51.961000+0000","last_unstale":"2024-03-06T07:24:51.966779+0000","last_undegraded":"2024-03-06T07:24:51.966779+0000","last_fullsized":"2024-03-06T07:24:51.966779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954701+0000","last_change":"2024-03-06T07:24:51.954701+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954701+0000","last_undegraded":"2024-03-06T07:24:51.954701+0000","last_fullsized":"2024-03-06T07:24:51.954701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966456+0000","last_change":"2024-03-06T07:24:51.959310+0000","last_active":"2024-03-06T07:24:51.966456+0000","last_peered":"2024-03-06T07:24:51.966456+0000","last_clean":"2024-03-06T07:24:51.966456+0000","last_became_active":"2024-03-06T07:24:51.958975+0000","last_became_peered":"2024-03-06T07:24:51.958975+0000","last_unstale":"2024-03-06T07:24:51.966456+0000","last_undegraded":"2024-03-06T07:24:51.966456+0000","last_fullsized":"2024-03-06T07:24:51.966456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:22.828922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953114+0000","last_change":"2024-03-06T07:24:51.953114+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953114+0000","last_undegraded":"2024-03-06T07:24:51.953114+0000","last_fullsized":"2024-03-06T07:24:51.953114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969566+0000","last_change":"2024-03-06T07:24:51.966524+0000","last_active":"2024-03-06T07:24:51.969566+0000","last_peered":"2024-03-06T07:24:51.969566+0000","last_clean":"2024-03-06T07:24:51.969566+0000","last_became_active":"2024-03-06T07:24:51.966147+0000","last_became_peered":"2024-03-06T07:24:51.966147+0000","last_unstale":"2024-03-06T07:24:51.969566+0000","last_undegraded":"2024-03-06T07:24:51.969566+0000","last_fullsized":"2024-03-06T07:24:51.969566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954975+0000","last_change":"2024-03-06T07:24:51.954975+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954975+0000","last_undegraded":"2024-03-06T07:24:51.954975+0000","last_fullsized":"2024-03-06T07:24:51.954975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967206+0000","last_change":"2024-03-06T07:24:51.965299+0000","last_active":"2024-03-06T07:24:51.967206+0000","last_peered":"2024-03-06T07:24:51.967206+0000","last_clean":"2024-03-06T07:24:51.967206+0000","last_became_active":"2024-03-06T07:24:51.964994+0000","last_became_peered":"2024-03-06T07:24:51.964994+0000","last_unstale":"2024-03-06T07:24:51.967206+0000","last_undegraded":"2024-03-06T07:24:51.967206+0000","last_fullsized":"2024-03-06T07:24:51.967206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:36.962579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953854+0000","last_change":"2024-03-06T07:24:51.953854+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953854+0000","last_undegraded":"2024-03-06T07:24:51.953854+0000","last_fullsized":"2024-03-06T07:24:51.953854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972229+0000","last_change":"2024-03-06T07:24:51.967402+0000","last_active":"2024-03-06T07:24:51.972229+0000","last_peered":"2024-03-06T07:24:51.972229+0000","last_clean":"2024-03-06T07:24:51.972229+0000","last_became_active":"2024-03-06T07:24:51.967118+0000","last_became_peered":"2024-03-06T07:24:51.967118+0000","last_unstale":"2024-03-06T07:24:51.972229+0000","last_undegraded":"2024-03-06T07:24:51.972229+0000","last_fullsized":"2024-03-06T07:24:51.972229+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954657+0000","last_change":"2024-03-06T07:24:51.954657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954657+0000","last_undegraded":"2024-03-06T07:24:51.954657+0000","last_fullsized":"2024-03-06T07:24:51.954657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972236+0000","last_change":"2024-03-06T07:24:51.967728+0000","last_active":"2024-03-06T07:24:51.972236+0000","last_peered":"2024-03-06T07:24:51.972236+0000","last_clean":"2024-03-06T07:24:51.972236+0000","last_became_active":"2024-03-06T07:24:51.967414+0000","last_became_peered":"2024-03-06T07:24:51.967414+0000","last_unstale":"2024-03-06T07:24:51.972236+0000","last_undegraded":"2024-03-06T07:24:51.972236+0000","last_fullsized":"2024-03-06T07:24:51.972236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949563+0000","last_change":"2024-03-06T07:24:51.949563+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949563+0000","last_undegraded":"2024-03-06T07:24:51.949563+0000","last_fullsized":"2024-03-06T07:24:51.949563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967254+0000","last_change":"2024-03-06T07:24:51.964639+0000","last_active":"2024-03-06T07:24:51.967254+0000","last_peered":"2024-03-06T07:24:51.967254+0000","last_clean":"2024-03-06T07:24:51.967254+0000","last_became_active":"2024-03-06T07:24:51.964394+0000","last_became_peered":"2024-03-06T07:24:51.964394+0000","last_unstale":"2024-03-06T07:24:51.967254+0000","last_undegraded":"2024-03-06T07:24:51.967254+0000","last_fullsized":"2024-03-06T07:24:51.967254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.726015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948968+0000","last_change":"2024-03-06T07:24:51.948968+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948968+0000","last_undegraded":"2024-03-06T07:24:51.948968+0000","last_fullsized":"2024-03-06T07:24:51.948968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961413+0000","last_change":"2024-03-06T07:24:51.957428+0000","last_active":"2024-03-06T07:24:51.961413+0000","last_peered":"2024-03-06T07:24:51.961413+0000","last_clean":"2024-03-06T07:24:51.961413+0000","last_became_active":"2024-03-06T07:24:51.957114+0000","last_became_peered":"2024-03-06T07:24:51.957114+0000","last_unstale":"2024-03-06T07:24:51.961413+0000","last_undegraded":"2024-03-06T07:24:51.961413+0000","last_fullsized":"2024-03-06T07:24:51.961413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.825819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949253+0000","last_change":"2024-03-06T07:24:51.949253+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949253+0000","last_undegraded":"2024-03-06T07:24:51.949253+0000","last_fullsized":"2024-03-06T07:24:51.949253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964318+0000","last_change":"2024-03-06T07:24:51.962417+0000","last_active":"2024-03-06T07:24:51.964318+0000","last_peered":"2024-03-06T07:24:51.964318+0000","last_clean":"2024-03-06T07:24:51.964318+0000","last_became_active":"2024-03-06T07:24:51.962166+0000","last_became_peered":"2024-03-06T07:24:51.962166+0000","last_unstale":"2024-03-06T07:24:51.964318+0000","last_undegraded":"2024-03-06T07:24:51.964318+0000","last_fullsized":"2024-03-06T07:24:51.964318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.253518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953963+0000","last_change":"2024-03-06T07:24:51.953963+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953963+0000","last_undegraded":"2024-03-06T07:24:51.953963+0000","last_fullsized":"2024-03-06T07:24:51.953963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967382+0000","last_change":"2024-03-06T07:24:51.965725+0000","last_active":"2024-03-06T07:24:51.967382+0000","last_peered":"2024-03-06T07:24:51.967382+0000","last_clean":"2024-03-06T07:24:51.967382+0000","last_became_active":"2024-03-06T07:24:51.965329+0000","last_became_peered":"2024-03-06T07:24:51.965329+0000","last_unstale":"2024-03-06T07:24:51.967382+0000","last_undegraded":"2024-03-06T07:24:51.967382+0000","last_fullsized":"2024-03-06T07:24:51.967382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.628974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949805+0000","last_change":"2024-03-06T07:24:51.949805+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949805+0000","last_undegraded":"2024-03-06T07:24:51.949805+0000","last_fullsized":"2024-03-06T07:24:51.949805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966811+0000","last_change":"2024-03-06T07:24:51.961973+0000","last_active":"2024-03-06T07:24:51.966811+0000","last_peered":"2024-03-06T07:24:51.966811+0000","last_clean":"2024-03-06T07:24:51.966811+0000","last_became_active":"2024-03-06T07:24:51.961700+0000","last_became_peered":"2024-03-06T07:24:51.961700+0000","last_unstale":"2024-03-06T07:24:51.966811+0000","last_undegraded":"2024-03-06T07:24:51.966811+0000","last_fullsized":"2024-03-06T07:24:51.966811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952214+0000","last_change":"2024-03-06T07:24:51.952214+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952214+0000","last_undegraded":"2024-03-06T07:24:51.952214+0000","last_fullsized":"2024-03-06T07:24:51.952214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963087+0000","last_change":"2024-03-06T07:24:51.959714+0000","last_active":"2024-03-06T07:24:51.963087+0000","last_peered":"2024-03-06T07:24:51.963087+0000","last_clean":"2024-03-06T07:24:51.963087+0000","last_became_active":"2024-03-06T07:24:51.959436+0000","last_became_peered":"2024-03-06T07:24:51.959436+0000","last_unstale":"2024-03-06T07:24:51.963087+0000","last_undegraded":"2024-03-06T07:24:51.963087+0000","last_fullsized":"2024-03-06T07:24:51.963087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:13.477872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952016+0000","last_change":"2024-03-06T07:24:51.952016+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952016+0000","last_undegraded":"2024-03-06T07:24:51.952016+0000","last_fullsized":"2024-03-06T07:24:51.952016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.968632+0000","last_change":"2024-03-06T07:24:51.968045+0000","last_active":"2024-03-06T07:24:51.968632+0000","last_peered":"2024-03-06T07:24:51.968632+0000","last_clean":"2024-03-06T07:24:51.968632+0000","last_became_active":"2024-03-06T07:24:51.967814+0000","last_became_peered":"2024-03-06T07:24:51.967814+0000","last_unstale":"2024-03-06T07:24:51.968632+0000","last_undegraded":"2024-03-06T07:24:51.968632+0000","last_fullsized":"2024-03-06T07:24:51.968632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:37.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954421+0000","last_change":"2024-03-06T07:24:51.954421+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954421+0000","last_undegraded":"2024-03-06T07:24:51.954421+0000","last_fullsized":"2024-03-06T07:24:51.954421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961459+0000","last_change":"2024-03-06T07:24:51.958710+0000","last_active":"2024-03-06T07:24:51.961459+0000","last_peered":"2024-03-06T07:24:51.961459+0000","last_clean":"2024-03-06T07:24:51.961459+0000","last_became_active":"2024-03-06T07:24:51.958435+0000","last_became_peered":"2024-03-06T07:24:51.958435+0000","last_unstale":"2024-03-06T07:24:51.961459+0000","last_undegraded":"2024-03-06T07:24:51.961459+0000","last_fullsized":"2024-03-06T07:24:51.961459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:29.966173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956317+0000","last_change":"2024-03-06T07:24:51.956317+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956317+0000","last_undegraded":"2024-03-06T07:24:51.956317+0000","last_fullsized":"2024-03-06T07:24:51.956317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964468+0000","last_change":"2024-03-06T07:24:51.962531+0000","last_active":"2024-03-06T07:24:51.964468+0000","last_peered":"2024-03-06T07:24:51.964468+0000","last_clean":"2024-03-06T07:24:51.964468+0000","last_became_active":"2024-03-06T07:24:51.962253+0000","last_became_peered":"2024-03-06T07:24:51.962253+0000","last_unstale":"2024-03-06T07:24:51.964468+0000","last_undegraded":"2024-03-06T07:24:51.964468+0000","last_fullsized":"2024-03-06T07:24:51.964468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:11.864382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953329+0000","last_change":"2024-03-06T07:24:51.953329+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953329+0000","last_undegraded":"2024-03-06T07:24:51.953329+0000","last_fullsized":"2024-03-06T07:24:51.953329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967111+0000","last_change":"2024-03-06T07:24:51.963025+0000","last_active":"2024-03-06T07:24:51.967111+0000","last_peered":"2024-03-06T07:24:51.967111+0000","last_clean":"2024-03-06T07:24:51.967111+0000","last_became_active":"2024-03-06T07:24:51.962710+0000","last_became_peered":"2024-03-06T07:24:51.962710+0000","last_unstale":"2024-03-06T07:24:51.967111+0000","last_undegraded":"2024-03-06T07:24:51.967111+0000","last_fullsized":"2024-03-06T07:24:51.967111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:21.637787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954067+0000","last_change":"2024-03-06T07:24:51.954067+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954067+0000","last_undegraded":"2024-03-06T07:24:51.954067+0000","last_fullsized":"2024-03-06T07:24:51.954067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967246+0000","last_change":"2024-03-06T07:24:51.963634+0000","last_active":"2024-03-06T07:24:51.967246+0000","last_peered":"2024-03-06T07:24:51.967246+0000","last_clean":"2024-03-06T07:24:51.967246+0000","last_became_active":"2024-03-06T07:24:51.963368+0000","last_became_peered":"2024-03-06T07:24:51.963368+0000","last_unstale":"2024-03-06T07:24:51.967246+0000","last_undegraded":"2024-03-06T07:24:51.967246+0000","last_fullsized":"2024-03-06T07:24:51.967246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:22.343338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948955+0000","last_change":"2024-03-06T07:24:51.948955+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948955+0000","last_undegraded":"2024-03-06T07:24:51.948955+0000","last_fullsized":"2024-03-06T07:24:51.948955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967349+0000","last_change":"2024-03-06T07:24:51.964342+0000","last_active":"2024-03-06T07:24:51.967349+0000","last_peered":"2024-03-06T07:24:51.967349+0000","last_clean":"2024-03-06T07:24:51.967349+0000","last_became_active":"2024-03-06T07:24:51.964058+0000","last_became_peered":"2024-03-06T07:24:51.964058+0000","last_unstale":"2024-03-06T07:24:51.967349+0000","last_undegraded":"2024-03-06T07:24:51.967349+0000","last_fullsized":"2024-03-06T07:24:51.967349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:27.238164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.956184+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:51.956184+0000","last_peered":"2024-03-06T07:24:51.956184+0000","last_clean":"2024-03-06T07:24:51.956184+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:51.956184+0000","last_undegraded":"2024-03-06T07:24:51.956184+0000","last_fullsized":"2024-03-06T07:24:51.956184+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:55.938 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:56.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.072+0000 7f1ce4888640 1 Processor -- start 2024-03-06T07:24:56.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.072+0000 7f1ce4888640 1 -- start start 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1ce4888640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1ce00d22b0 0x7f1ce00d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1ce4888640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00b41f0 0x7f1ce00b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1ce4888640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce00b3430 con 0x7f1ce015dfe0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1ce4888640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce00d2c50 con 0x7f1ce00d22b0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1ce4888640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00b31e0 con 0x7f1ce00b41f0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cde59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1ce015dfe0 0x7f1ce00cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37786/0 (socket says 172.21.15.72:37786) 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cde59c640 1 -- 172.21.15.72:0/110118649 learned_addr learned my addr 172.21.15.72:0/110118649 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cded9d640 1 --2- 172.21.15.72:0/110118649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00b41f0 0x7f1ce00b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cddd9b640 1 --2- 172.21.15.72:0/110118649 >> v2:172.21.15.146:3300/0 conn(0x7f1ce00d22b0 0x7f1ce00d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cdd59a640 1 -- 172.21.15.72:0/110118649 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474100295 0 0) 0x7f1ce00b3430 con 0x7f1ce015dfe0 2024-03-06T07:24:56.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cdd59a640 1 -- 172.21.15.72:0/110118649 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cc8003520 con 0x7f1ce015dfe0 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cded9d640 1 -- 172.21.15.72:0/110118649 >> v2:172.21.15.146:3300/0 conn(0x7f1ce00d22b0 msgr2=0x7f1ce00d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cded9d640 1 --2- 172.21.15.72:0/110118649 >> v2:172.21.15.146:3300/0 conn(0x7f1ce00d22b0 0x7f1ce00d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cded9d640 1 -- 172.21.15.72:0/110118649 >> v1:172.21.15.72:6789/0 conn(0x7f1ce015dfe0 legacy=0x7f1ce00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.073+0000 7f1cded9d640 1 -- 172.21.15.72:0/110118649 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00b3710 con 0x7f1ce00b41f0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.074+0000 7f1cded9d640 1 --2- 172.21.15.72:0/110118649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00b41f0 0x7f1ce00b4680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009d90 tx=0x7f1cd802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d48821051a9369 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.074+0000 7f1cdd59a640 1 -- 172.21.15.72:0/110118649 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8039070 con 0x7f1ce00b41f0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.074+0000 7f1cdd59a640 1 -- 172.21.15.72:0/110118649 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd802cc50 con 0x7f1ce00b41f0 2024-03-06T07:24:56.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1cdd59a640 1 -- 172.21.15.72:0/110118649 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8034440 con 0x7f1ce00b41f0 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 -- 172.21.15.72:0/110118649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00b41f0 msgr2=0x7f1ce00b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 --2- 172.21.15.72:0/110118649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00b41f0 0x7f1ce00b4680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009d90 tx=0x7f1cd802bea0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 -- 172.21.15.72:0/110118649 shutdown_connections 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 --2- 172.21.15.72:0/110118649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00b41f0 0x7f1ce00b4680 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 --2- 172.21.15.72:0/110118649 >> v2:172.21.15.146:3300/0 conn(0x7f1ce00d22b0 0x7f1ce00d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 -- 172.21.15.72:0/110118649 >> 172.21.15.72:0/110118649 conn(0x7f1ce00d3cd0 msgr2=0x7f1ce00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 -- 172.21.15.72:0/110118649 shutdown_connections 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.075+0000 7f1ce4888640 1 -- 172.21.15.72:0/110118649 wait complete. 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.076+0000 7f1ce4888640 1 Processor -- start 2024-03-06T07:24:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.076+0000 7f1ce4888640 1 -- start start 2024-03-06T07:24:56.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1ce4888640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce00b41f0 0x7f1ce00cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1ce4888640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00d22b0 0x7f1ce01603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1ce4888640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 0x7f1ce0160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1ce4888640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00cfa00 con 0x7f1ce015dfe0 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1ce4888640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00b3d60 con 0x7f1ce00b41f0 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1ce4888640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00d1c70 con 0x7f1ce00d22b0 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cded9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 0x7f1ce0160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cded9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 0x7f1ce0160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37728/0 (socket says 172.21.15.72:37728) 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cded9d640 1 -- 172.21.15.72:0/1077438622 learned_addr learned my addr 172.21.15.72:0/1077438622 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cde59c640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce00b41f0 0x7f1ce00cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cddd9b640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00d22b0 0x7f1ce01603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cded9d640 1 -- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00d22b0 msgr2=0x7f1ce01603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cded9d640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00d22b0 0x7f1ce01603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.077+0000 7f1cded9d640 1 -- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce00b41f0 msgr2=0x7f1ce00cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1cded9d640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce00b41f0 0x7f1ce00cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1cded9d640 1 -- 172.21.15.72:0/1077438622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce01f7010 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1cded9d640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 0x7f1ce0160910 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009d90 tx=0x7f1cd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8039050 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce01f72f0 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd8045070 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8009450 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.078+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce00cbed0 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.079+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce004f330 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.079+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1cd803e610 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.080+0000 7f1cceffd640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1cc006b4d0 0x7f1cc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.080+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cd80aca70 con 0x7f1ce015dfe0 2024-03-06T07:24:56.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.080+0000 7f1cde59c640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1cc006b4d0 0x7f1cc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.080+0000 7f1cde59c640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1cc006b4d0 0x7f1cc006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0005880 tx=0x7f1cd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.084+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1cd807b2b0 con 0x7f1ce015dfe0 2024-03-06T07:24:56.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.255+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1ce01f6dc0 con 0x7f1cc006b4d0 2024-03-06T07:24:56.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.261+0000 7f1cceffd640 1 -- 172.21.15.72:0/1077438622 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f1ce01f6dc0 con 0x7f1cc006b4d0 2024-03-06T07:24:56.266 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:56.267 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:56.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1cc006b4d0 msgr2=0x7f1cc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1cc006b4d0 0x7f1cc006d990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0005880 tx=0x7f1cd0009040 comp rx=0 tx=0).stop 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 msgr2=0x7f1ce0160910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 0x7f1ce0160910 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f1cd8009d90 tx=0x7f1cd8004a40 comp rx=0 tx=0).stop 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 shutdown_connections 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1cc006b4d0 0x7f1cc006d990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1ce015dfe0 0x7f1ce0160910 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce00d22b0 0x7f1ce01603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 --2- 172.21.15.72:0/1077438622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce00b41f0 0x7f1ce00cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 >> 172.21.15.72:0/1077438622 conn(0x7f1ce00d3cd0 msgr2=0x7f1ce006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 shutdown_connections 2024-03-06T07:24:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.264+0000 7f1ce4888640 1 -- 172.21.15.72:0/1077438622 wait complete. 2024-03-06T07:24:56.277 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-06T07:24:55.980586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257064,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717656,"statfs":{"total":773094113280,"available":772830879744,"internally_reserved":0,"allocated":7806976,"data_stored":3935832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.035949"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955811+0000","last_change":"2024-03-06T07:24:51.955811+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955811+0000","last_undegraded":"2024-03-06T07:24:51.955811+0000","last_fullsized":"2024-03-06T07:24:51.955811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956564+0000","last_change":"2024-03-06T07:24:51.956564+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956564+0000","last_undegraded":"2024-03-06T07:24:51.956564+0000","last_fullsized":"2024-03-06T07:24:51.956564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949967+0000","last_change":"2024-03-06T07:24:51.949967+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949967+0000","last_undegraded":"2024-03-06T07:24:51.949967+0000","last_fullsized":"2024-03-06T07:24:51.949967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954692+0000","last_change":"2024-03-06T07:24:51.954692+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954692+0000","last_undegraded":"2024-03-06T07:24:51.954692+0000","last_fullsized":"2024-03-06T07:24:51.954692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.957393+0000","last_change":"2024-03-06T07:24:51.957393+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.957393+0000","last_undegraded":"2024-03-06T07:24:51.957393+0000","last_fullsized":"2024-03-06T07:24:51.957393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956546+0000","last_change":"2024-03-06T07:24:51.956546+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956546+0000","last_undegraded":"2024-03-06T07:24:51.956546+0000","last_fullsized":"2024-03-06T07:24:51.956546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950665+0000","last_change":"2024-03-06T07:24:51.950665+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950665+0000","last_undegraded":"2024-03-06T07:24:51.950665+0000","last_fullsized":"2024-03-06T07:24:51.950665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954574+0000","last_change":"2024-03-06T07:24:51.954574+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954574+0000","last_undegraded":"2024-03-06T07:24:51.954574+0000","last_fullsized":"2024-03-06T07:24:51.954574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956937+0000","last_change":"2024-03-06T07:24:51.956937+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956937+0000","last_undegraded":"2024-03-06T07:24:51.956937+0000","last_fullsized":"2024-03-06T07:24:51.956937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953539+0000","last_change":"2024-03-06T07:24:51.953539+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953539+0000","last_undegraded":"2024-03-06T07:24:51.953539+0000","last_fullsized":"2024-03-06T07:24:51.953539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954128+0000","last_change":"2024-03-06T07:24:51.954128+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954128+0000","last_undegraded":"2024-03-06T07:24:51.954128+0000","last_fullsized":"2024-03-06T07:24:51.954128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954265+0000","last_change":"2024-03-06T07:24:51.954265+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954265+0000","last_undegraded":"2024-03-06T07:24:51.954265+0000","last_fullsized":"2024-03-06T07:24:51.954265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956604+0000","last_change":"2024-03-06T07:24:51.956604+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956604+0000","last_undegraded":"2024-03-06T07:24:51.956604+0000","last_fullsized":"2024-03-06T07:24:51.956604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955792+0000","last_change":"2024-03-06T07:24:51.955792+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955792+0000","last_undegraded":"2024-03-06T07:24:51.955792+0000","last_fullsized":"2024-03-06T07:24:51.955792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956049+0000","last_change":"2024-03-06T07:24:51.956049+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956049+0000","last_undegraded":"2024-03-06T07:24:51.956049+0000","last_fullsized":"2024-03-06T07:24:51.956049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951835+0000","last_change":"2024-03-06T07:24:51.951835+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951835+0000","last_undegraded":"2024-03-06T07:24:51.951835+0000","last_fullsized":"2024-03-06T07:24:51.951835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956373+0000","last_change":"2024-03-06T07:24:51.956373+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956373+0000","last_undegraded":"2024-03-06T07:24:51.956373+0000","last_fullsized":"2024-03-06T07:24:51.956373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953867+0000","last_change":"2024-03-06T07:24:51.953867+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953867+0000","last_undegraded":"2024-03-06T07:24:51.953867+0000","last_fullsized":"2024-03-06T07:24:51.953867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955657+0000","last_change":"2024-03-06T07:24:51.955657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955657+0000","last_undegraded":"2024-03-06T07:24:51.955657+0000","last_fullsized":"2024-03-06T07:24:51.955657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955630+0000","last_change":"2024-03-06T07:24:51.955630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955630+0000","last_undegraded":"2024-03-06T07:24:51.955630+0000","last_fullsized":"2024-03-06T07:24:51.955630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954554+0000","last_change":"2024-03-06T07:24:51.954554+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954554+0000","last_undegraded":"2024-03-06T07:24:51.954554+0000","last_fullsized":"2024-03-06T07:24:51.954554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952981+0000","last_change":"2024-03-06T07:24:51.952981+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952981+0000","last_undegraded":"2024-03-06T07:24:51.952981+0000","last_fullsized":"2024-03-06T07:24:51.952981+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952956+0000","last_change":"2024-03-06T07:24:51.952956+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952956+0000","last_undegraded":"2024-03-06T07:24:51.952956+0000","last_fullsized":"2024-03-06T07:24:51.952956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951873+0000","last_change":"2024-03-06T07:24:51.951873+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951873+0000","last_undegraded":"2024-03-06T07:24:51.951873+0000","last_fullsized":"2024-03-06T07:24:51.951873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953582+0000","last_change":"2024-03-06T07:24:51.953582+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953582+0000","last_undegraded":"2024-03-06T07:24:51.953582+0000","last_fullsized":"2024-03-06T07:24:51.953582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950728+0000","last_change":"2024-03-06T07:24:51.950728+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950728+0000","last_undegraded":"2024-03-06T07:24:51.950728+0000","last_fullsized":"2024-03-06T07:24:51.950728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949352+0000","last_change":"2024-03-06T07:24:51.949352+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949352+0000","last_undegraded":"2024-03-06T07:24:51.949352+0000","last_fullsized":"2024-03-06T07:24:51.949352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953082+0000","last_change":"2024-03-06T07:24:51.953082+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953082+0000","last_undegraded":"2024-03-06T07:24:51.953082+0000","last_fullsized":"2024-03-06T07:24:51.953082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949990+0000","last_change":"2024-03-06T07:24:51.949990+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949990+0000","last_undegraded":"2024-03-06T07:24:51.949990+0000","last_fullsized":"2024-03-06T07:24:51.949990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950896+0000","last_change":"2024-03-06T07:24:51.950896+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950896+0000","last_undegraded":"2024-03-06T07:24:51.950896+0000","last_fullsized":"2024-03-06T07:24:51.950896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949630+0000","last_change":"2024-03-06T07:24:51.949630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949630+0000","last_undegraded":"2024-03-06T07:24:51.949630+0000","last_fullsized":"2024-03-06T07:24:51.949630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954177+0000","last_change":"2024-03-06T07:24:51.954177+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954177+0000","last_undegraded":"2024-03-06T07:24:51.954177+0000","last_fullsized":"2024-03-06T07:24:51.954177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953434+0000","last_change":"2024-03-06T07:24:51.953434+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953434+0000","last_undegraded":"2024-03-06T07:24:51.953434+0000","last_fullsized":"2024-03-06T07:24:51.953434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966812+0000","last_change":"2024-03-06T07:24:51.962181+0000","last_active":"2024-03-06T07:24:51.966812+0000","last_peered":"2024-03-06T07:24:51.966812+0000","last_clean":"2024-03-06T07:24:51.966812+0000","last_became_active":"2024-03-06T07:24:51.961932+0000","last_became_peered":"2024-03-06T07:24:51.961932+0000","last_unstale":"2024-03-06T07:24:51.966812+0000","last_undegraded":"2024-03-06T07:24:51.966812+0000","last_fullsized":"2024-03-06T07:24:51.966812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955019+0000","last_change":"2024-03-06T07:24:51.955019+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955019+0000","last_undegraded":"2024-03-06T07:24:51.955019+0000","last_fullsized":"2024-03-06T07:24:51.955019+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964255+0000","last_change":"2024-03-06T07:24:51.962165+0000","last_active":"2024-03-06T07:24:51.964255+0000","last_peered":"2024-03-06T07:24:51.964255+0000","last_clean":"2024-03-06T07:24:51.964255+0000","last_became_active":"2024-03-06T07:24:51.961847+0000","last_became_peered":"2024-03-06T07:24:51.961847+0000","last_unstale":"2024-03-06T07:24:51.964255+0000","last_undegraded":"2024-03-06T07:24:51.964255+0000","last_fullsized":"2024-03-06T07:24:51.964255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956036+0000","last_change":"2024-03-06T07:24:51.956036+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956036+0000","last_undegraded":"2024-03-06T07:24:51.956036+0000","last_fullsized":"2024-03-06T07:24:51.956036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967209+0000","last_change":"2024-03-06T07:24:51.965487+0000","last_active":"2024-03-06T07:24:51.967209+0000","last_peered":"2024-03-06T07:24:51.967209+0000","last_clean":"2024-03-06T07:24:51.967209+0000","last_became_active":"2024-03-06T07:24:51.965210+0000","last_became_peered":"2024-03-06T07:24:51.965210+0000","last_unstale":"2024-03-06T07:24:51.967209+0000","last_undegraded":"2024-03-06T07:24:51.967209+0000","last_fullsized":"2024-03-06T07:24:51.967209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:18.007631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953707+0000","last_change":"2024-03-06T07:24:51.953707+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953707+0000","last_undegraded":"2024-03-06T07:24:51.953707+0000","last_fullsized":"2024-03-06T07:24:51.953707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967180+0000","last_change":"2024-03-06T07:24:51.963677+0000","last_active":"2024-03-06T07:24:51.967180+0000","last_peered":"2024-03-06T07:24:51.967180+0000","last_clean":"2024-03-06T07:24:51.967180+0000","last_became_active":"2024-03-06T07:24:51.963383+0000","last_became_peered":"2024-03-06T07:24:51.963383+0000","last_unstale":"2024-03-06T07:24:51.967180+0000","last_undegraded":"2024-03-06T07:24:51.967180+0000","last_fullsized":"2024-03-06T07:24:51.967180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:05.163132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949619+0000","last_change":"2024-03-06T07:24:51.949619+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949619+0000","last_undegraded":"2024-03-06T07:24:51.949619+0000","last_fullsized":"2024-03-06T07:24:51.949619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963718+0000","last_change":"2024-03-06T07:24:51.962180+0000","last_active":"2024-03-06T07:24:51.963718+0000","last_peered":"2024-03-06T07:24:51.963718+0000","last_clean":"2024-03-06T07:24:51.963718+0000","last_became_active":"2024-03-06T07:24:51.961911+0000","last_became_peered":"2024-03-06T07:24:51.961911+0000","last_unstale":"2024-03-06T07:24:51.963718+0000","last_undegraded":"2024-03-06T07:24:51.963718+0000","last_fullsized":"2024-03-06T07:24:51.963718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:40.456147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954185+0000","last_change":"2024-03-06T07:24:51.954185+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954185+0000","last_undegraded":"2024-03-06T07:24:51.954185+0000","last_fullsized":"2024-03-06T07:24:51.954185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967273+0000","last_change":"2024-03-06T07:24:51.964997+0000","last_active":"2024-03-06T07:24:51.967273+0000","last_peered":"2024-03-06T07:24:51.967273+0000","last_clean":"2024-03-06T07:24:51.967273+0000","last_became_active":"2024-03-06T07:24:51.964708+0000","last_became_peered":"2024-03-06T07:24:51.964708+0000","last_unstale":"2024-03-06T07:24:51.967273+0000","last_undegraded":"2024-03-06T07:24:51.967273+0000","last_fullsized":"2024-03-06T07:24:51.967273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:33.109374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954642+0000","last_change":"2024-03-06T07:24:51.954642+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954642+0000","last_undegraded":"2024-03-06T07:24:51.954642+0000","last_fullsized":"2024-03-06T07:24:51.954642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963914+0000","last_change":"2024-03-06T07:24:51.962002+0000","last_active":"2024-03-06T07:24:51.963914+0000","last_peered":"2024-03-06T07:24:51.963914+0000","last_clean":"2024-03-06T07:24:51.963914+0000","last_became_active":"2024-03-06T07:24:51.961717+0000","last_became_peered":"2024-03-06T07:24:51.961717+0000","last_unstale":"2024-03-06T07:24:51.963914+0000","last_undegraded":"2024-03-06T07:24:51.963914+0000","last_fullsized":"2024-03-06T07:24:51.963914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954942+0000","last_change":"2024-03-06T07:24:51.954942+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954942+0000","last_undegraded":"2024-03-06T07:24:51.954942+0000","last_fullsized":"2024-03-06T07:24:51.954942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961350+0000","last_change":"2024-03-06T07:24:51.957748+0000","last_active":"2024-03-06T07:24:51.961350+0000","last_peered":"2024-03-06T07:24:51.961350+0000","last_clean":"2024-03-06T07:24:51.961350+0000","last_became_active":"2024-03-06T07:24:51.957465+0000","last_became_peered":"2024-03-06T07:24:51.957465+0000","last_unstale":"2024-03-06T07:24:51.961350+0000","last_undegraded":"2024-03-06T07:24:51.961350+0000","last_fullsized":"2024-03-06T07:24:51.961350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:53.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949800+0000","last_change":"2024-03-06T07:24:51.949800+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949800+0000","last_undegraded":"2024-03-06T07:24:51.949800+0000","last_fullsized":"2024-03-06T07:24:51.949800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967114+0000","last_change":"2024-03-06T07:24:51.962228+0000","last_active":"2024-03-06T07:24:51.967114+0000","last_peered":"2024-03-06T07:24:51.967114+0000","last_clean":"2024-03-06T07:24:51.967114+0000","last_became_active":"2024-03-06T07:24:51.961977+0000","last_became_peered":"2024-03-06T07:24:51.961977+0000","last_unstale":"2024-03-06T07:24:51.967114+0000","last_undegraded":"2024-03-06T07:24:51.967114+0000","last_fullsized":"2024-03-06T07:24:51.967114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:07.096047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955422+0000","last_change":"2024-03-06T07:24:51.955422+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955422+0000","last_undegraded":"2024-03-06T07:24:51.955422+0000","last_fullsized":"2024-03-06T07:24:51.955422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969507+0000","last_change":"2024-03-06T07:24:51.965861+0000","last_active":"2024-03-06T07:24:51.969507+0000","last_peered":"2024-03-06T07:24:51.969507+0000","last_clean":"2024-03-06T07:24:51.969507+0000","last_became_active":"2024-03-06T07:24:51.965498+0000","last_became_peered":"2024-03-06T07:24:51.965498+0000","last_unstale":"2024-03-06T07:24:51.969507+0000","last_undegraded":"2024-03-06T07:24:51.969507+0000","last_fullsized":"2024-03-06T07:24:51.969507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955625+0000","last_change":"2024-03-06T07:24:51.955625+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955625+0000","last_undegraded":"2024-03-06T07:24:51.955625+0000","last_fullsized":"2024-03-06T07:24:51.955625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.959867+0000","last_change":"2024-03-06T07:24:51.958271+0000","last_active":"2024-03-06T07:24:51.959867+0000","last_peered":"2024-03-06T07:24:51.959867+0000","last_clean":"2024-03-06T07:24:51.959867+0000","last_became_active":"2024-03-06T07:24:51.957988+0000","last_became_peered":"2024-03-06T07:24:51.957988+0000","last_unstale":"2024-03-06T07:24:51.959867+0000","last_undegraded":"2024-03-06T07:24:51.959867+0000","last_fullsized":"2024-03-06T07:24:51.959867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:30.871115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955262+0000","last_change":"2024-03-06T07:24:51.955262+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955262+0000","last_undegraded":"2024-03-06T07:24:51.955262+0000","last_fullsized":"2024-03-06T07:24:51.955262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961753+0000","last_change":"2024-03-06T07:24:51.958913+0000","last_active":"2024-03-06T07:24:51.961753+0000","last_peered":"2024-03-06T07:24:51.961753+0000","last_clean":"2024-03-06T07:24:51.961753+0000","last_became_active":"2024-03-06T07:24:51.958624+0000","last_became_peered":"2024-03-06T07:24:51.958624+0000","last_unstale":"2024-03-06T07:24:51.961753+0000","last_undegraded":"2024-03-06T07:24:51.961753+0000","last_fullsized":"2024-03-06T07:24:51.961753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954553+0000","last_change":"2024-03-06T07:24:51.954553+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954553+0000","last_undegraded":"2024-03-06T07:24:51.954553+0000","last_fullsized":"2024-03-06T07:24:51.954553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.962777+0000","last_change":"2024-03-06T07:24:51.960455+0000","last_active":"2024-03-06T07:24:51.962777+0000","last_peered":"2024-03-06T07:24:51.962777+0000","last_clean":"2024-03-06T07:24:51.962777+0000","last_became_active":"2024-03-06T07:24:51.960189+0000","last_became_peered":"2024-03-06T07:24:51.960189+0000","last_unstale":"2024-03-06T07:24:51.962777+0000","last_undegraded":"2024-03-06T07:24:51.962777+0000","last_fullsized":"2024-03-06T07:24:51.962777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949626+0000","last_change":"2024-03-06T07:24:51.949626+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949626+0000","last_undegraded":"2024-03-06T07:24:51.949626+0000","last_fullsized":"2024-03-06T07:24:51.949626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967289+0000","last_change":"2024-03-06T07:24:51.965187+0000","last_active":"2024-03-06T07:24:51.967289+0000","last_peered":"2024-03-06T07:24:51.967289+0000","last_clean":"2024-03-06T07:24:51.967289+0000","last_became_active":"2024-03-06T07:24:51.964934+0000","last_became_peered":"2024-03-06T07:24:51.964934+0000","last_unstale":"2024-03-06T07:24:51.967289+0000","last_undegraded":"2024-03-06T07:24:51.967289+0000","last_fullsized":"2024-03-06T07:24:51.967289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:35.381581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954508+0000","last_change":"2024-03-06T07:24:51.954508+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954508+0000","last_undegraded":"2024-03-06T07:24:51.954508+0000","last_fullsized":"2024-03-06T07:24:51.954508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966779+0000","last_change":"2024-03-06T07:24:51.961258+0000","last_active":"2024-03-06T07:24:51.966779+0000","last_peered":"2024-03-06T07:24:51.966779+0000","last_clean":"2024-03-06T07:24:51.966779+0000","last_became_active":"2024-03-06T07:24:51.961000+0000","last_became_peered":"2024-03-06T07:24:51.961000+0000","last_unstale":"2024-03-06T07:24:51.966779+0000","last_undegraded":"2024-03-06T07:24:51.966779+0000","last_fullsized":"2024-03-06T07:24:51.966779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954701+0000","last_change":"2024-03-06T07:24:51.954701+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954701+0000","last_undegraded":"2024-03-06T07:24:51.954701+0000","last_fullsized":"2024-03-06T07:24:51.954701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966456+0000","last_change":"2024-03-06T07:24:51.959310+0000","last_active":"2024-03-06T07:24:51.966456+0000","last_peered":"2024-03-06T07:24:51.966456+0000","last_clean":"2024-03-06T07:24:51.966456+0000","last_became_active":"2024-03-06T07:24:51.958975+0000","last_became_peered":"2024-03-06T07:24:51.958975+0000","last_unstale":"2024-03-06T07:24:51.966456+0000","last_undegraded":"2024-03-06T07:24:51.966456+0000","last_fullsized":"2024-03-06T07:24:51.966456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:22.828922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953114+0000","last_change":"2024-03-06T07:24:51.953114+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953114+0000","last_undegraded":"2024-03-06T07:24:51.953114+0000","last_fullsized":"2024-03-06T07:24:51.953114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969566+0000","last_change":"2024-03-06T07:24:51.966524+0000","last_active":"2024-03-06T07:24:51.969566+0000","last_peered":"2024-03-06T07:24:51.969566+0000","last_clean":"2024-03-06T07:24:51.969566+0000","last_became_active":"2024-03-06T07:24:51.966147+0000","last_became_peered":"2024-03-06T07:24:51.966147+0000","last_unstale":"2024-03-06T07:24:51.969566+0000","last_undegraded":"2024-03-06T07:24:51.969566+0000","last_fullsized":"2024-03-06T07:24:51.969566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954975+0000","last_change":"2024-03-06T07:24:51.954975+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954975+0000","last_undegraded":"2024-03-06T07:24:51.954975+0000","last_fullsized":"2024-03-06T07:24:51.954975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967206+0000","last_change":"2024-03-06T07:24:51.965299+0000","last_active":"2024-03-06T07:24:51.967206+0000","last_peered":"2024-03-06T07:24:51.967206+0000","last_clean":"2024-03-06T07:24:51.967206+0000","last_became_active":"2024-03-06T07:24:51.964994+0000","last_became_peered":"2024-03-06T07:24:51.964994+0000","last_unstale":"2024-03-06T07:24:51.967206+0000","last_undegraded":"2024-03-06T07:24:51.967206+0000","last_fullsized":"2024-03-06T07:24:51.967206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:36.962579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953854+0000","last_change":"2024-03-06T07:24:51.953854+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953854+0000","last_undegraded":"2024-03-06T07:24:51.953854+0000","last_fullsized":"2024-03-06T07:24:51.953854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972229+0000","last_change":"2024-03-06T07:24:51.967402+0000","last_active":"2024-03-06T07:24:51.972229+0000","last_peered":"2024-03-06T07:24:51.972229+0000","last_clean":"2024-03-06T07:24:51.972229+0000","last_became_active":"2024-03-06T07:24:51.967118+0000","last_became_peered":"2024-03-06T07:24:51.967118+0000","last_unstale":"2024-03-06T07:24:51.972229+0000","last_undegraded":"2024-03-06T07:24:51.972229+0000","last_fullsized":"2024-03-06T07:24:51.972229+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954657+0000","last_change":"2024-03-06T07:24:51.954657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954657+0000","last_undegraded":"2024-03-06T07:24:51.954657+0000","last_fullsized":"2024-03-06T07:24:51.954657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972236+0000","last_change":"2024-03-06T07:24:51.967728+0000","last_active":"2024-03-06T07:24:51.972236+0000","last_peered":"2024-03-06T07:24:51.972236+0000","last_clean":"2024-03-06T07:24:51.972236+0000","last_became_active":"2024-03-06T07:24:51.967414+0000","last_became_peered":"2024-03-06T07:24:51.967414+0000","last_unstale":"2024-03-06T07:24:51.972236+0000","last_undegraded":"2024-03-06T07:24:51.972236+0000","last_fullsized":"2024-03-06T07:24:51.972236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949563+0000","last_change":"2024-03-06T07:24:51.949563+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949563+0000","last_undegraded":"2024-03-06T07:24:51.949563+0000","last_fullsized":"2024-03-06T07:24:51.949563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967254+0000","last_change":"2024-03-06T07:24:51.964639+0000","last_active":"2024-03-06T07:24:51.967254+0000","last_peered":"2024-03-06T07:24:51.967254+0000","last_clean":"2024-03-06T07:24:51.967254+0000","last_became_active":"2024-03-06T07:24:51.964394+0000","last_became_peered":"2024-03-06T07:24:51.964394+0000","last_unstale":"2024-03-06T07:24:51.967254+0000","last_undegraded":"2024-03-06T07:24:51.967254+0000","last_fullsized":"2024-03-06T07:24:51.967254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.726015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948968+0000","last_change":"2024-03-06T07:24:51.948968+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948968+0000","last_undegraded":"2024-03-06T07:24:51.948968+0000","last_fullsized":"2024-03-06T07:24:51.948968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961413+0000","last_change":"2024-03-06T07:24:51.957428+0000","last_active":"2024-03-06T07:24:51.961413+0000","last_peered":"2024-03-06T07:24:51.961413+0000","last_clean":"2024-03-06T07:24:51.961413+0000","last_became_active":"2024-03-06T07:24:51.957114+0000","last_became_peered":"2024-03-06T07:24:51.957114+0000","last_unstale":"2024-03-06T07:24:51.961413+0000","last_undegraded":"2024-03-06T07:24:51.961413+0000","last_fullsized":"2024-03-06T07:24:51.961413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.825819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949253+0000","last_change":"2024-03-06T07:24:51.949253+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949253+0000","last_undegraded":"2024-03-06T07:24:51.949253+0000","last_fullsized":"2024-03-06T07:24:51.949253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964318+0000","last_change":"2024-03-06T07:24:51.962417+0000","last_active":"2024-03-06T07:24:51.964318+0000","last_peered":"2024-03-06T07:24:51.964318+0000","last_clean":"2024-03-06T07:24:51.964318+0000","last_became_active":"2024-03-06T07:24:51.962166+0000","last_became_peered":"2024-03-06T07:24:51.962166+0000","last_unstale":"2024-03-06T07:24:51.964318+0000","last_undegraded":"2024-03-06T07:24:51.964318+0000","last_fullsized":"2024-03-06T07:24:51.964318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.253518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953963+0000","last_change":"2024-03-06T07:24:51.953963+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953963+0000","last_undegraded":"2024-03-06T07:24:51.953963+0000","last_fullsized":"2024-03-06T07:24:51.953963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967382+0000","last_change":"2024-03-06T07:24:51.965725+0000","last_active":"2024-03-06T07:24:51.967382+0000","last_peered":"2024-03-06T07:24:51.967382+0000","last_clean":"2024-03-06T07:24:51.967382+0000","last_became_active":"2024-03-06T07:24:51.965329+0000","last_became_peered":"2024-03-06T07:24:51.965329+0000","last_unstale":"2024-03-06T07:24:51.967382+0000","last_undegraded":"2024-03-06T07:24:51.967382+0000","last_fullsized":"2024-03-06T07:24:51.967382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.628974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949805+0000","last_change":"2024-03-06T07:24:51.949805+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949805+0000","last_undegraded":"2024-03-06T07:24:51.949805+0000","last_fullsized":"2024-03-06T07:24:51.949805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966811+0000","last_change":"2024-03-06T07:24:51.961973+0000","last_active":"2024-03-06T07:24:51.966811+0000","last_peered":"2024-03-06T07:24:51.966811+0000","last_clean":"2024-03-06T07:24:51.966811+0000","last_became_active":"2024-03-06T07:24:51.961700+0000","last_became_peered":"2024-03-06T07:24:51.961700+0000","last_unstale":"2024-03-06T07:24:51.966811+0000","last_undegraded":"2024-03-06T07:24:51.966811+0000","last_fullsized":"2024-03-06T07:24:51.966811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952214+0000","last_change":"2024-03-06T07:24:51.952214+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952214+0000","last_undegraded":"2024-03-06T07:24:51.952214+0000","last_fullsized":"2024-03-06T07:24:51.952214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963087+0000","last_change":"2024-03-06T07:24:51.959714+0000","last_active":"2024-03-06T07:24:51.963087+0000","last_peered":"2024-03-06T07:24:51.963087+0000","last_clean":"2024-03-06T07:24:51.963087+0000","last_became_active":"2024-03-06T07:24:51.959436+0000","last_became_peered":"2024-03-06T07:24:51.959436+0000","last_unstale":"2024-03-06T07:24:51.963087+0000","last_undegraded":"2024-03-06T07:24:51.963087+0000","last_fullsized":"2024-03-06T07:24:51.963087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:13.477872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952016+0000","last_change":"2024-03-06T07:24:51.952016+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952016+0000","last_undegraded":"2024-03-06T07:24:51.952016+0000","last_fullsized":"2024-03-06T07:24:51.952016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.968632+0000","last_change":"2024-03-06T07:24:51.968045+0000","last_active":"2024-03-06T07:24:51.968632+0000","last_peered":"2024-03-06T07:24:51.968632+0000","last_clean":"2024-03-06T07:24:51.968632+0000","last_became_active":"2024-03-06T07:24:51.967814+0000","last_became_peered":"2024-03-06T07:24:51.967814+0000","last_unstale":"2024-03-06T07:24:51.968632+0000","last_undegraded":"2024-03-06T07:24:51.968632+0000","last_fullsized":"2024-03-06T07:24:51.968632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:37.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954421+0000","last_change":"2024-03-06T07:24:51.954421+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954421+0000","last_undegraded":"2024-03-06T07:24:51.954421+0000","last_fullsized":"2024-03-06T07:24:51.954421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961459+0000","last_change":"2024-03-06T07:24:51.958710+0000","last_active":"2024-03-06T07:24:51.961459+0000","last_peered":"2024-03-06T07:24:51.961459+0000","last_clean":"2024-03-06T07:24:51.961459+0000","last_became_active":"2024-03-06T07:24:51.958435+0000","last_became_peered":"2024-03-06T07:24:51.958435+0000","last_unstale":"2024-03-06T07:24:51.961459+0000","last_undegraded":"2024-03-06T07:24:51.961459+0000","last_fullsized":"2024-03-06T07:24:51.961459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:29.966173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956317+0000","last_change":"2024-03-06T07:24:51.956317+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956317+0000","last_undegraded":"2024-03-06T07:24:51.956317+0000","last_fullsized":"2024-03-06T07:24:51.956317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964468+0000","last_change":"2024-03-06T07:24:51.962531+0000","last_active":"2024-03-06T07:24:51.964468+0000","last_peered":"2024-03-06T07:24:51.964468+0000","last_clean":"2024-03-06T07:24:51.964468+0000","last_became_active":"2024-03-06T07:24:51.962253+0000","last_became_peered":"2024-03-06T07:24:51.962253+0000","last_unstale":"2024-03-06T07:24:51.964468+0000","last_undegraded":"2024-03-06T07:24:51.964468+0000","last_fullsized":"2024-03-06T07:24:51.964468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:11.864382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953329+0000","last_change":"2024-03-06T07:24:51.953329+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953329+0000","last_undegraded":"2024-03-06T07:24:51.953329+0000","last_fullsized":"2024-03-06T07:24:51.953329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967111+0000","last_change":"2024-03-06T07:24:51.963025+0000","last_active":"2024-03-06T07:24:51.967111+0000","last_peered":"2024-03-06T07:24:51.967111+0000","last_clean":"2024-03-06T07:24:51.967111+0000","last_became_active":"2024-03-06T07:24:51.962710+0000","last_became_peered":"2024-03-06T07:24:51.962710+0000","last_unstale":"2024-03-06T07:24:51.967111+0000","last_undegraded":"2024-03-06T07:24:51.967111+0000","last_fullsized":"2024-03-06T07:24:51.967111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:21.637787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954067+0000","last_change":"2024-03-06T07:24:51.954067+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954067+0000","last_undegraded":"2024-03-06T07:24:51.954067+0000","last_fullsized":"2024-03-06T07:24:51.954067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967246+0000","last_change":"2024-03-06T07:24:51.963634+0000","last_active":"2024-03-06T07:24:51.967246+0000","last_peered":"2024-03-06T07:24:51.967246+0000","last_clean":"2024-03-06T07:24:51.967246+0000","last_became_active":"2024-03-06T07:24:51.963368+0000","last_became_peered":"2024-03-06T07:24:51.963368+0000","last_unstale":"2024-03-06T07:24:51.967246+0000","last_undegraded":"2024-03-06T07:24:51.967246+0000","last_fullsized":"2024-03-06T07:24:51.967246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:22.343338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948955+0000","last_change":"2024-03-06T07:24:51.948955+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948955+0000","last_undegraded":"2024-03-06T07:24:51.948955+0000","last_fullsized":"2024-03-06T07:24:51.948955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967349+0000","last_change":"2024-03-06T07:24:51.964342+0000","last_active":"2024-03-06T07:24:51.967349+0000","last_peered":"2024-03-06T07:24:51.967349+0000","last_clean":"2024-03-06T07:24:51.967349+0000","last_became_active":"2024-03-06T07:24:51.964058+0000","last_became_peered":"2024-03-06T07:24:51.964058+0000","last_unstale":"2024-03-06T07:24:51.967349+0000","last_undegraded":"2024-03-06T07:24:51.967349+0000","last_fullsized":"2024-03-06T07:24:51.967349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:27.238164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.956184+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:51.956184+0000","last_peered":"2024-03-06T07:24:51.956184+0000","last_clean":"2024-03-06T07:24:51.956184+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:51.956184+0000","last_undegraded":"2024-03-06T07:24:51.956184+0000","last_fullsized":"2024-03-06T07:24:51.956184+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:56.281 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.414+0000 7fe1add56640 1 Processor -- start 2024-03-06T07:24:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.414+0000 7fe1add56640 1 -- start start 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1add56640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a81424f0 0x7fe1a80cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1add56640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe1a80cd7a0 0x7fe1a815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1add56640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80cc980 con 0x7fe1a806d330 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1add56640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80cd220 con 0x7fe1a80cd7a0 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1add56640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80ccc60 con 0x7fe1a81424f0 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1a7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe1a806d330 0x7fe1a8071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37794/0 (socket says 172.21.15.72:37794) 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1a7fff640 1 -- 172.21.15.72:0/786672914 learned_addr learned my addr 172.21.15.72:0/786672914 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1a77fe640 1 --2- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a81424f0 0x7fe1a80cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1a6ffd640 1 --2- 172.21.15.72:0/786672914 >> v2:172.21.15.146:3300/0 conn(0x7fe1a80cd7a0 0x7fe1a815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.415+0000 7fe1a67fc640 1 -- 172.21.15.72:0/786672914 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618849425 0 0) 0x7fe1a80cc980 con 0x7fe1a806d330 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.416+0000 7fe1a67fc640 1 -- 172.21.15.72:0/786672914 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe190003520 con 0x7fe1a806d330 2024-03-06T07:24:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.416+0000 7fe1a6ffd640 1 -- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a81424f0 msgr2=0x7fe1a80cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.416+0000 7fe1a6ffd640 1 --2- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a81424f0 0x7fe1a80cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.416+0000 7fe1a6ffd640 1 -- 172.21.15.72:0/786672914 >> v1:172.21.15.72:6789/0 conn(0x7fe1a806d330 legacy=0x7fe1a8071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.416+0000 7fe1a6ffd640 1 -- 172.21.15.72:0/786672914 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80ccf40 con 0x7fe1a80cd7a0 2024-03-06T07:24:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.416+0000 7fe1a6ffd640 1 --2- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a80cd7a0 0x7fe1a815cef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0097a0 tx=0x7fe19c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76424b6b9bc3911f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1a67fc640 1 -- 172.21.15.72:0/786672914 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c035020 con 0x7fe1a80cd7a0 2024-03-06T07:24:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1a67fc640 1 -- 172.21.15.72:0/786672914 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19c031830 con 0x7fe1a80cd7a0 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1a67fc640 1 -- 172.21.15.72:0/786672914 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19c031c00 con 0x7fe1a80cd7a0 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 -- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a80cd7a0 msgr2=0x7fe1a815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 --2- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a80cd7a0 0x7fe1a815cef0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0097a0 tx=0x7fe19c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 -- 172.21.15.72:0/786672914 shutdown_connections 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 --2- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a80cd7a0 0x7fe1a815cef0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 --2- 172.21.15.72:0/786672914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a81424f0 0x7fe1a80cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 -- 172.21.15.72:0/786672914 >> 172.21.15.72:0/786672914 conn(0x7fe1a80cf8a0 msgr2=0x7fe1a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 -- 172.21.15.72:0/786672914 shutdown_connections 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.417+0000 7fe1add56640 1 -- 172.21.15.72:0/786672914 wait complete. 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.418+0000 7fe1add56640 1 Processor -- start 2024-03-06T07:24:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.418+0000 7fe1add56640 1 -- start start 2024-03-06T07:24:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1add56640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a806d330 0x7fe1a815b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1add56640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a80cd7a0 0x7fe1a80b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1add56640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 0x7fe1a80b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1add56640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a806d940 con 0x7fe1a81424f0 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1add56640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a806d7c0 con 0x7fe1a806d330 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1add56640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a806dac0 con 0x7fe1a80cd7a0 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 0x7fe1a80b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 0x7fe1a80b10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37742/0 (socket says 172.21.15.72:37742) 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1a7fff640 1 -- 172.21.15.72:0/4057606678 learned_addr learned my addr 172.21.15.72:0/4057606678 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1a77fe640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a806d330 0x7fe1a815b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.419+0000 7fe1a6ffd640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a80cd7a0 0x7fe1a80b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1a7fff640 1 -- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a80cd7a0 msgr2=0x7fe1a80b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1a7fff640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a80cd7a0 0x7fe1a80b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1a7fff640 1 -- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a806d330 msgr2=0x7fe1a815b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1a7fff640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a806d330 0x7fe1a815b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1a7fff640 1 -- 172.21.15.72:0/4057606678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a81442c0 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1a7fff640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 0x7fe1a80b10c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000cb10 tx=0x7fe1a000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0017070 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a000ac70 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a8144560 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0012620 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.420+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a8144b20 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.421+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a804f330 con 0x7fe1a81424f0 2024-03-06T07:24:56.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.421+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe1a001a070 con 0x7fe1a81424f0 2024-03-06T07:24:56.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.422+0000 7fe18ffff640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe19406b4d0 0x7fe19406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.422+0000 7fe1a77fe640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe19406b4d0 0x7fe19406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.422+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1a0089f00 con 0x7fe1a81424f0 2024-03-06T07:24:56.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.422+0000 7fe1a77fe640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe19406b4d0 0x7fe19406d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe198009aa0 tx=0x7fe198009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.426+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe1a000ae20 con 0x7fe1a81424f0 2024-03-06T07:24:56.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.594+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe1a80d0620 con 0x7fe19406b4d0 2024-03-06T07:24:56.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.600+0000 7fe18ffff640 1 -- 172.21.15.72:0/4057606678 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7fe1a80d0620 con 0x7fe19406b4d0 2024-03-06T07:24:56.605 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:56.607 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:56.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe19406b4d0 msgr2=0x7fe19406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe19406b4d0 0x7fe19406d990 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe198009aa0 tx=0x7fe198009190 comp rx=0 tx=0).stop 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 msgr2=0x7fe1a80b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 0x7fe1a80b10c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe1a000cb10 tx=0x7fe1a000a060 comp rx=0 tx=0).stop 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 shutdown_connections 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe19406b4d0 0x7fe19406d990 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe1a81424f0 0x7fe1a80b10c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1a80cd7a0 0x7fe1a80b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 --2- 172.21.15.72:0/4057606678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a806d330 0x7fe1a815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 >> 172.21.15.72:0/4057606678 conn(0x7fe1a80cf8a0 msgr2=0x7fe1a815c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 shutdown_connections 2024-03-06T07:24:56.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.603+0000 7fe1add56640 1 -- 172.21.15.72:0/4057606678 wait complete. 2024-03-06T07:24:56.616 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-06T07:24:55.980586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257064,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717656,"statfs":{"total":773094113280,"available":772830879744,"internally_reserved":0,"allocated":7806976,"data_stored":3935832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.035949"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955811+0000","last_change":"2024-03-06T07:24:51.955811+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955811+0000","last_undegraded":"2024-03-06T07:24:51.955811+0000","last_fullsized":"2024-03-06T07:24:51.955811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956564+0000","last_change":"2024-03-06T07:24:51.956564+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956564+0000","last_undegraded":"2024-03-06T07:24:51.956564+0000","last_fullsized":"2024-03-06T07:24:51.956564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949967+0000","last_change":"2024-03-06T07:24:51.949967+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949967+0000","last_undegraded":"2024-03-06T07:24:51.949967+0000","last_fullsized":"2024-03-06T07:24:51.949967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954692+0000","last_change":"2024-03-06T07:24:51.954692+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954692+0000","last_undegraded":"2024-03-06T07:24:51.954692+0000","last_fullsized":"2024-03-06T07:24:51.954692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.957393+0000","last_change":"2024-03-06T07:24:51.957393+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.957393+0000","last_undegraded":"2024-03-06T07:24:51.957393+0000","last_fullsized":"2024-03-06T07:24:51.957393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956546+0000","last_change":"2024-03-06T07:24:51.956546+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956546+0000","last_undegraded":"2024-03-06T07:24:51.956546+0000","last_fullsized":"2024-03-06T07:24:51.956546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950665+0000","last_change":"2024-03-06T07:24:51.950665+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950665+0000","last_undegraded":"2024-03-06T07:24:51.950665+0000","last_fullsized":"2024-03-06T07:24:51.950665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954574+0000","last_change":"2024-03-06T07:24:51.954574+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954574+0000","last_undegraded":"2024-03-06T07:24:51.954574+0000","last_fullsized":"2024-03-06T07:24:51.954574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956937+0000","last_change":"2024-03-06T07:24:51.956937+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956937+0000","last_undegraded":"2024-03-06T07:24:51.956937+0000","last_fullsized":"2024-03-06T07:24:51.956937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953539+0000","last_change":"2024-03-06T07:24:51.953539+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953539+0000","last_undegraded":"2024-03-06T07:24:51.953539+0000","last_fullsized":"2024-03-06T07:24:51.953539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954128+0000","last_change":"2024-03-06T07:24:51.954128+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954128+0000","last_undegraded":"2024-03-06T07:24:51.954128+0000","last_fullsized":"2024-03-06T07:24:51.954128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954265+0000","last_change":"2024-03-06T07:24:51.954265+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954265+0000","last_undegraded":"2024-03-06T07:24:51.954265+0000","last_fullsized":"2024-03-06T07:24:51.954265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956604+0000","last_change":"2024-03-06T07:24:51.956604+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956604+0000","last_undegraded":"2024-03-06T07:24:51.956604+0000","last_fullsized":"2024-03-06T07:24:51.956604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955792+0000","last_change":"2024-03-06T07:24:51.955792+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955792+0000","last_undegraded":"2024-03-06T07:24:51.955792+0000","last_fullsized":"2024-03-06T07:24:51.955792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956049+0000","last_change":"2024-03-06T07:24:51.956049+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956049+0000","last_undegraded":"2024-03-06T07:24:51.956049+0000","last_fullsized":"2024-03-06T07:24:51.956049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951835+0000","last_change":"2024-03-06T07:24:51.951835+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951835+0000","last_undegraded":"2024-03-06T07:24:51.951835+0000","last_fullsized":"2024-03-06T07:24:51.951835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956373+0000","last_change":"2024-03-06T07:24:51.956373+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956373+0000","last_undegraded":"2024-03-06T07:24:51.956373+0000","last_fullsized":"2024-03-06T07:24:51.956373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953867+0000","last_change":"2024-03-06T07:24:51.953867+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953867+0000","last_undegraded":"2024-03-06T07:24:51.953867+0000","last_fullsized":"2024-03-06T07:24:51.953867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955657+0000","last_change":"2024-03-06T07:24:51.955657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955657+0000","last_undegraded":"2024-03-06T07:24:51.955657+0000","last_fullsized":"2024-03-06T07:24:51.955657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955630+0000","last_change":"2024-03-06T07:24:51.955630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955630+0000","last_undegraded":"2024-03-06T07:24:51.955630+0000","last_fullsized":"2024-03-06T07:24:51.955630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954554+0000","last_change":"2024-03-06T07:24:51.954554+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954554+0000","last_undegraded":"2024-03-06T07:24:51.954554+0000","last_fullsized":"2024-03-06T07:24:51.954554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952981+0000","last_change":"2024-03-06T07:24:51.952981+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952981+0000","last_undegraded":"2024-03-06T07:24:51.952981+0000","last_fullsized":"2024-03-06T07:24:51.952981+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952956+0000","last_change":"2024-03-06T07:24:51.952956+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952956+0000","last_undegraded":"2024-03-06T07:24:51.952956+0000","last_fullsized":"2024-03-06T07:24:51.952956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951873+0000","last_change":"2024-03-06T07:24:51.951873+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951873+0000","last_undegraded":"2024-03-06T07:24:51.951873+0000","last_fullsized":"2024-03-06T07:24:51.951873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953582+0000","last_change":"2024-03-06T07:24:51.953582+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953582+0000","last_undegraded":"2024-03-06T07:24:51.953582+0000","last_fullsized":"2024-03-06T07:24:51.953582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950728+0000","last_change":"2024-03-06T07:24:51.950728+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950728+0000","last_undegraded":"2024-03-06T07:24:51.950728+0000","last_fullsized":"2024-03-06T07:24:51.950728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949352+0000","last_change":"2024-03-06T07:24:51.949352+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949352+0000","last_undegraded":"2024-03-06T07:24:51.949352+0000","last_fullsized":"2024-03-06T07:24:51.949352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953082+0000","last_change":"2024-03-06T07:24:51.953082+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953082+0000","last_undegraded":"2024-03-06T07:24:51.953082+0000","last_fullsized":"2024-03-06T07:24:51.953082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949990+0000","last_change":"2024-03-06T07:24:51.949990+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949990+0000","last_undegraded":"2024-03-06T07:24:51.949990+0000","last_fullsized":"2024-03-06T07:24:51.949990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950896+0000","last_change":"2024-03-06T07:24:51.950896+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950896+0000","last_undegraded":"2024-03-06T07:24:51.950896+0000","last_fullsized":"2024-03-06T07:24:51.950896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949630+0000","last_change":"2024-03-06T07:24:51.949630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949630+0000","last_undegraded":"2024-03-06T07:24:51.949630+0000","last_fullsized":"2024-03-06T07:24:51.949630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954177+0000","last_change":"2024-03-06T07:24:51.954177+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954177+0000","last_undegraded":"2024-03-06T07:24:51.954177+0000","last_fullsized":"2024-03-06T07:24:51.954177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953434+0000","last_change":"2024-03-06T07:24:51.953434+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953434+0000","last_undegraded":"2024-03-06T07:24:51.953434+0000","last_fullsized":"2024-03-06T07:24:51.953434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966812+0000","last_change":"2024-03-06T07:24:51.962181+0000","last_active":"2024-03-06T07:24:51.966812+0000","last_peered":"2024-03-06T07:24:51.966812+0000","last_clean":"2024-03-06T07:24:51.966812+0000","last_became_active":"2024-03-06T07:24:51.961932+0000","last_became_peered":"2024-03-06T07:24:51.961932+0000","last_unstale":"2024-03-06T07:24:51.966812+0000","last_undegraded":"2024-03-06T07:24:51.966812+0000","last_fullsized":"2024-03-06T07:24:51.966812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955019+0000","last_change":"2024-03-06T07:24:51.955019+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955019+0000","last_undegraded":"2024-03-06T07:24:51.955019+0000","last_fullsized":"2024-03-06T07:24:51.955019+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964255+0000","last_change":"2024-03-06T07:24:51.962165+0000","last_active":"2024-03-06T07:24:51.964255+0000","last_peered":"2024-03-06T07:24:51.964255+0000","last_clean":"2024-03-06T07:24:51.964255+0000","last_became_active":"2024-03-06T07:24:51.961847+0000","last_became_peered":"2024-03-06T07:24:51.961847+0000","last_unstale":"2024-03-06T07:24:51.964255+0000","last_undegraded":"2024-03-06T07:24:51.964255+0000","last_fullsized":"2024-03-06T07:24:51.964255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956036+0000","last_change":"2024-03-06T07:24:51.956036+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956036+0000","last_undegraded":"2024-03-06T07:24:51.956036+0000","last_fullsized":"2024-03-06T07:24:51.956036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967209+0000","last_change":"2024-03-06T07:24:51.965487+0000","last_active":"2024-03-06T07:24:51.967209+0000","last_peered":"2024-03-06T07:24:51.967209+0000","last_clean":"2024-03-06T07:24:51.967209+0000","last_became_active":"2024-03-06T07:24:51.965210+0000","last_became_peered":"2024-03-06T07:24:51.965210+0000","last_unstale":"2024-03-06T07:24:51.967209+0000","last_undegraded":"2024-03-06T07:24:51.967209+0000","last_fullsized":"2024-03-06T07:24:51.967209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:18.007631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953707+0000","last_change":"2024-03-06T07:24:51.953707+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953707+0000","last_undegraded":"2024-03-06T07:24:51.953707+0000","last_fullsized":"2024-03-06T07:24:51.953707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967180+0000","last_change":"2024-03-06T07:24:51.963677+0000","last_active":"2024-03-06T07:24:51.967180+0000","last_peered":"2024-03-06T07:24:51.967180+0000","last_clean":"2024-03-06T07:24:51.967180+0000","last_became_active":"2024-03-06T07:24:51.963383+0000","last_became_peered":"2024-03-06T07:24:51.963383+0000","last_unstale":"2024-03-06T07:24:51.967180+0000","last_undegraded":"2024-03-06T07:24:51.967180+0000","last_fullsized":"2024-03-06T07:24:51.967180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:05.163132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949619+0000","last_change":"2024-03-06T07:24:51.949619+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949619+0000","last_undegraded":"2024-03-06T07:24:51.949619+0000","last_fullsized":"2024-03-06T07:24:51.949619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963718+0000","last_change":"2024-03-06T07:24:51.962180+0000","last_active":"2024-03-06T07:24:51.963718+0000","last_peered":"2024-03-06T07:24:51.963718+0000","last_clean":"2024-03-06T07:24:51.963718+0000","last_became_active":"2024-03-06T07:24:51.961911+0000","last_became_peered":"2024-03-06T07:24:51.961911+0000","last_unstale":"2024-03-06T07:24:51.963718+0000","last_undegraded":"2024-03-06T07:24:51.963718+0000","last_fullsized":"2024-03-06T07:24:51.963718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:40.456147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954185+0000","last_change":"2024-03-06T07:24:51.954185+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954185+0000","last_undegraded":"2024-03-06T07:24:51.954185+0000","last_fullsized":"2024-03-06T07:24:51.954185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967273+0000","last_change":"2024-03-06T07:24:51.964997+0000","last_active":"2024-03-06T07:24:51.967273+0000","last_peered":"2024-03-06T07:24:51.967273+0000","last_clean":"2024-03-06T07:24:51.967273+0000","last_became_active":"2024-03-06T07:24:51.964708+0000","last_became_peered":"2024-03-06T07:24:51.964708+0000","last_unstale":"2024-03-06T07:24:51.967273+0000","last_undegraded":"2024-03-06T07:24:51.967273+0000","last_fullsized":"2024-03-06T07:24:51.967273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:33.109374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954642+0000","last_change":"2024-03-06T07:24:51.954642+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954642+0000","last_undegraded":"2024-03-06T07:24:51.954642+0000","last_fullsized":"2024-03-06T07:24:51.954642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963914+0000","last_change":"2024-03-06T07:24:51.962002+0000","last_active":"2024-03-06T07:24:51.963914+0000","last_peered":"2024-03-06T07:24:51.963914+0000","last_clean":"2024-03-06T07:24:51.963914+0000","last_became_active":"2024-03-06T07:24:51.961717+0000","last_became_peered":"2024-03-06T07:24:51.961717+0000","last_unstale":"2024-03-06T07:24:51.963914+0000","last_undegraded":"2024-03-06T07:24:51.963914+0000","last_fullsized":"2024-03-06T07:24:51.963914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954942+0000","last_change":"2024-03-06T07:24:51.954942+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954942+0000","last_undegraded":"2024-03-06T07:24:51.954942+0000","last_fullsized":"2024-03-06T07:24:51.954942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961350+0000","last_change":"2024-03-06T07:24:51.957748+0000","last_active":"2024-03-06T07:24:51.961350+0000","last_peered":"2024-03-06T07:24:51.961350+0000","last_clean":"2024-03-06T07:24:51.961350+0000","last_became_active":"2024-03-06T07:24:51.957465+0000","last_became_peered":"2024-03-06T07:24:51.957465+0000","last_unstale":"2024-03-06T07:24:51.961350+0000","last_undegraded":"2024-03-06T07:24:51.961350+0000","last_fullsized":"2024-03-06T07:24:51.961350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:53.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949800+0000","last_change":"2024-03-06T07:24:51.949800+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949800+0000","last_undegraded":"2024-03-06T07:24:51.949800+0000","last_fullsized":"2024-03-06T07:24:51.949800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967114+0000","last_change":"2024-03-06T07:24:51.962228+0000","last_active":"2024-03-06T07:24:51.967114+0000","last_peered":"2024-03-06T07:24:51.967114+0000","last_clean":"2024-03-06T07:24:51.967114+0000","last_became_active":"2024-03-06T07:24:51.961977+0000","last_became_peered":"2024-03-06T07:24:51.961977+0000","last_unstale":"2024-03-06T07:24:51.967114+0000","last_undegraded":"2024-03-06T07:24:51.967114+0000","last_fullsized":"2024-03-06T07:24:51.967114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:07.096047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955422+0000","last_change":"2024-03-06T07:24:51.955422+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955422+0000","last_undegraded":"2024-03-06T07:24:51.955422+0000","last_fullsized":"2024-03-06T07:24:51.955422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969507+0000","last_change":"2024-03-06T07:24:51.965861+0000","last_active":"2024-03-06T07:24:51.969507+0000","last_peered":"2024-03-06T07:24:51.969507+0000","last_clean":"2024-03-06T07:24:51.969507+0000","last_became_active":"2024-03-06T07:24:51.965498+0000","last_became_peered":"2024-03-06T07:24:51.965498+0000","last_unstale":"2024-03-06T07:24:51.969507+0000","last_undegraded":"2024-03-06T07:24:51.969507+0000","last_fullsized":"2024-03-06T07:24:51.969507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955625+0000","last_change":"2024-03-06T07:24:51.955625+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955625+0000","last_undegraded":"2024-03-06T07:24:51.955625+0000","last_fullsized":"2024-03-06T07:24:51.955625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.959867+0000","last_change":"2024-03-06T07:24:51.958271+0000","last_active":"2024-03-06T07:24:51.959867+0000","last_peered":"2024-03-06T07:24:51.959867+0000","last_clean":"2024-03-06T07:24:51.959867+0000","last_became_active":"2024-03-06T07:24:51.957988+0000","last_became_peered":"2024-03-06T07:24:51.957988+0000","last_unstale":"2024-03-06T07:24:51.959867+0000","last_undegraded":"2024-03-06T07:24:51.959867+0000","last_fullsized":"2024-03-06T07:24:51.959867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:30.871115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955262+0000","last_change":"2024-03-06T07:24:51.955262+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955262+0000","last_undegraded":"2024-03-06T07:24:51.955262+0000","last_fullsized":"2024-03-06T07:24:51.955262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961753+0000","last_change":"2024-03-06T07:24:51.958913+0000","last_active":"2024-03-06T07:24:51.961753+0000","last_peered":"2024-03-06T07:24:51.961753+0000","last_clean":"2024-03-06T07:24:51.961753+0000","last_became_active":"2024-03-06T07:24:51.958624+0000","last_became_peered":"2024-03-06T07:24:51.958624+0000","last_unstale":"2024-03-06T07:24:51.961753+0000","last_undegraded":"2024-03-06T07:24:51.961753+0000","last_fullsized":"2024-03-06T07:24:51.961753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954553+0000","last_change":"2024-03-06T07:24:51.954553+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954553+0000","last_undegraded":"2024-03-06T07:24:51.954553+0000","last_fullsized":"2024-03-06T07:24:51.954553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.962777+0000","last_change":"2024-03-06T07:24:51.960455+0000","last_active":"2024-03-06T07:24:51.962777+0000","last_peered":"2024-03-06T07:24:51.962777+0000","last_clean":"2024-03-06T07:24:51.962777+0000","last_became_active":"2024-03-06T07:24:51.960189+0000","last_became_peered":"2024-03-06T07:24:51.960189+0000","last_unstale":"2024-03-06T07:24:51.962777+0000","last_undegraded":"2024-03-06T07:24:51.962777+0000","last_fullsized":"2024-03-06T07:24:51.962777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949626+0000","last_change":"2024-03-06T07:24:51.949626+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949626+0000","last_undegraded":"2024-03-06T07:24:51.949626+0000","last_fullsized":"2024-03-06T07:24:51.949626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967289+0000","last_change":"2024-03-06T07:24:51.965187+0000","last_active":"2024-03-06T07:24:51.967289+0000","last_peered":"2024-03-06T07:24:51.967289+0000","last_clean":"2024-03-06T07:24:51.967289+0000","last_became_active":"2024-03-06T07:24:51.964934+0000","last_became_peered":"2024-03-06T07:24:51.964934+0000","last_unstale":"2024-03-06T07:24:51.967289+0000","last_undegraded":"2024-03-06T07:24:51.967289+0000","last_fullsized":"2024-03-06T07:24:51.967289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:35.381581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954508+0000","last_change":"2024-03-06T07:24:51.954508+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954508+0000","last_undegraded":"2024-03-06T07:24:51.954508+0000","last_fullsized":"2024-03-06T07:24:51.954508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966779+0000","last_change":"2024-03-06T07:24:51.961258+0000","last_active":"2024-03-06T07:24:51.966779+0000","last_peered":"2024-03-06T07:24:51.966779+0000","last_clean":"2024-03-06T07:24:51.966779+0000","last_became_active":"2024-03-06T07:24:51.961000+0000","last_became_peered":"2024-03-06T07:24:51.961000+0000","last_unstale":"2024-03-06T07:24:51.966779+0000","last_undegraded":"2024-03-06T07:24:51.966779+0000","last_fullsized":"2024-03-06T07:24:51.966779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954701+0000","last_change":"2024-03-06T07:24:51.954701+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954701+0000","last_undegraded":"2024-03-06T07:24:51.954701+0000","last_fullsized":"2024-03-06T07:24:51.954701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966456+0000","last_change":"2024-03-06T07:24:51.959310+0000","last_active":"2024-03-06T07:24:51.966456+0000","last_peered":"2024-03-06T07:24:51.966456+0000","last_clean":"2024-03-06T07:24:51.966456+0000","last_became_active":"2024-03-06T07:24:51.958975+0000","last_became_peered":"2024-03-06T07:24:51.958975+0000","last_unstale":"2024-03-06T07:24:51.966456+0000","last_undegraded":"2024-03-06T07:24:51.966456+0000","last_fullsized":"2024-03-06T07:24:51.966456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:22.828922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953114+0000","last_change":"2024-03-06T07:24:51.953114+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953114+0000","last_undegraded":"2024-03-06T07:24:51.953114+0000","last_fullsized":"2024-03-06T07:24:51.953114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969566+0000","last_change":"2024-03-06T07:24:51.966524+0000","last_active":"2024-03-06T07:24:51.969566+0000","last_peered":"2024-03-06T07:24:51.969566+0000","last_clean":"2024-03-06T07:24:51.969566+0000","last_became_active":"2024-03-06T07:24:51.966147+0000","last_became_peered":"2024-03-06T07:24:51.966147+0000","last_unstale":"2024-03-06T07:24:51.969566+0000","last_undegraded":"2024-03-06T07:24:51.969566+0000","last_fullsized":"2024-03-06T07:24:51.969566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954975+0000","last_change":"2024-03-06T07:24:51.954975+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954975+0000","last_undegraded":"2024-03-06T07:24:51.954975+0000","last_fullsized":"2024-03-06T07:24:51.954975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967206+0000","last_change":"2024-03-06T07:24:51.965299+0000","last_active":"2024-03-06T07:24:51.967206+0000","last_peered":"2024-03-06T07:24:51.967206+0000","last_clean":"2024-03-06T07:24:51.967206+0000","last_became_active":"2024-03-06T07:24:51.964994+0000","last_became_peered":"2024-03-06T07:24:51.964994+0000","last_unstale":"2024-03-06T07:24:51.967206+0000","last_undegraded":"2024-03-06T07:24:51.967206+0000","last_fullsized":"2024-03-06T07:24:51.967206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:36.962579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953854+0000","last_change":"2024-03-06T07:24:51.953854+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953854+0000","last_undegraded":"2024-03-06T07:24:51.953854+0000","last_fullsized":"2024-03-06T07:24:51.953854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972229+0000","last_change":"2024-03-06T07:24:51.967402+0000","last_active":"2024-03-06T07:24:51.972229+0000","last_peered":"2024-03-06T07:24:51.972229+0000","last_clean":"2024-03-06T07:24:51.972229+0000","last_became_active":"2024-03-06T07:24:51.967118+0000","last_became_peered":"2024-03-06T07:24:51.967118+0000","last_unstale":"2024-03-06T07:24:51.972229+0000","last_undegraded":"2024-03-06T07:24:51.972229+0000","last_fullsized":"2024-03-06T07:24:51.972229+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954657+0000","last_change":"2024-03-06T07:24:51.954657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954657+0000","last_undegraded":"2024-03-06T07:24:51.954657+0000","last_fullsized":"2024-03-06T07:24:51.954657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972236+0000","last_change":"2024-03-06T07:24:51.967728+0000","last_active":"2024-03-06T07:24:51.972236+0000","last_peered":"2024-03-06T07:24:51.972236+0000","last_clean":"2024-03-06T07:24:51.972236+0000","last_became_active":"2024-03-06T07:24:51.967414+0000","last_became_peered":"2024-03-06T07:24:51.967414+0000","last_unstale":"2024-03-06T07:24:51.972236+0000","last_undegraded":"2024-03-06T07:24:51.972236+0000","last_fullsized":"2024-03-06T07:24:51.972236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949563+0000","last_change":"2024-03-06T07:24:51.949563+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949563+0000","last_undegraded":"2024-03-06T07:24:51.949563+0000","last_fullsized":"2024-03-06T07:24:51.949563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967254+0000","last_change":"2024-03-06T07:24:51.964639+0000","last_active":"2024-03-06T07:24:51.967254+0000","last_peered":"2024-03-06T07:24:51.967254+0000","last_clean":"2024-03-06T07:24:51.967254+0000","last_became_active":"2024-03-06T07:24:51.964394+0000","last_became_peered":"2024-03-06T07:24:51.964394+0000","last_unstale":"2024-03-06T07:24:51.967254+0000","last_undegraded":"2024-03-06T07:24:51.967254+0000","last_fullsized":"2024-03-06T07:24:51.967254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.726015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948968+0000","last_change":"2024-03-06T07:24:51.948968+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948968+0000","last_undegraded":"2024-03-06T07:24:51.948968+0000","last_fullsized":"2024-03-06T07:24:51.948968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961413+0000","last_change":"2024-03-06T07:24:51.957428+0000","last_active":"2024-03-06T07:24:51.961413+0000","last_peered":"2024-03-06T07:24:51.961413+0000","last_clean":"2024-03-06T07:24:51.961413+0000","last_became_active":"2024-03-06T07:24:51.957114+0000","last_became_peered":"2024-03-06T07:24:51.957114+0000","last_unstale":"2024-03-06T07:24:51.961413+0000","last_undegraded":"2024-03-06T07:24:51.961413+0000","last_fullsized":"2024-03-06T07:24:51.961413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.825819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949253+0000","last_change":"2024-03-06T07:24:51.949253+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949253+0000","last_undegraded":"2024-03-06T07:24:51.949253+0000","last_fullsized":"2024-03-06T07:24:51.949253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964318+0000","last_change":"2024-03-06T07:24:51.962417+0000","last_active":"2024-03-06T07:24:51.964318+0000","last_peered":"2024-03-06T07:24:51.964318+0000","last_clean":"2024-03-06T07:24:51.964318+0000","last_became_active":"2024-03-06T07:24:51.962166+0000","last_became_peered":"2024-03-06T07:24:51.962166+0000","last_unstale":"2024-03-06T07:24:51.964318+0000","last_undegraded":"2024-03-06T07:24:51.964318+0000","last_fullsized":"2024-03-06T07:24:51.964318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.253518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953963+0000","last_change":"2024-03-06T07:24:51.953963+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953963+0000","last_undegraded":"2024-03-06T07:24:51.953963+0000","last_fullsized":"2024-03-06T07:24:51.953963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967382+0000","last_change":"2024-03-06T07:24:51.965725+0000","last_active":"2024-03-06T07:24:51.967382+0000","last_peered":"2024-03-06T07:24:51.967382+0000","last_clean":"2024-03-06T07:24:51.967382+0000","last_became_active":"2024-03-06T07:24:51.965329+0000","last_became_peered":"2024-03-06T07:24:51.965329+0000","last_unstale":"2024-03-06T07:24:51.967382+0000","last_undegraded":"2024-03-06T07:24:51.967382+0000","last_fullsized":"2024-03-06T07:24:51.967382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.628974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949805+0000","last_change":"2024-03-06T07:24:51.949805+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949805+0000","last_undegraded":"2024-03-06T07:24:51.949805+0000","last_fullsized":"2024-03-06T07:24:51.949805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966811+0000","last_change":"2024-03-06T07:24:51.961973+0000","last_active":"2024-03-06T07:24:51.966811+0000","last_peered":"2024-03-06T07:24:51.966811+0000","last_clean":"2024-03-06T07:24:51.966811+0000","last_became_active":"2024-03-06T07:24:51.961700+0000","last_became_peered":"2024-03-06T07:24:51.961700+0000","last_unstale":"2024-03-06T07:24:51.966811+0000","last_undegraded":"2024-03-06T07:24:51.966811+0000","last_fullsized":"2024-03-06T07:24:51.966811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952214+0000","last_change":"2024-03-06T07:24:51.952214+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952214+0000","last_undegraded":"2024-03-06T07:24:51.952214+0000","last_fullsized":"2024-03-06T07:24:51.952214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963087+0000","last_change":"2024-03-06T07:24:51.959714+0000","last_active":"2024-03-06T07:24:51.963087+0000","last_peered":"2024-03-06T07:24:51.963087+0000","last_clean":"2024-03-06T07:24:51.963087+0000","last_became_active":"2024-03-06T07:24:51.959436+0000","last_became_peered":"2024-03-06T07:24:51.959436+0000","last_unstale":"2024-03-06T07:24:51.963087+0000","last_undegraded":"2024-03-06T07:24:51.963087+0000","last_fullsized":"2024-03-06T07:24:51.963087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:13.477872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952016+0000","last_change":"2024-03-06T07:24:51.952016+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952016+0000","last_undegraded":"2024-03-06T07:24:51.952016+0000","last_fullsized":"2024-03-06T07:24:51.952016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.968632+0000","last_change":"2024-03-06T07:24:51.968045+0000","last_active":"2024-03-06T07:24:51.968632+0000","last_peered":"2024-03-06T07:24:51.968632+0000","last_clean":"2024-03-06T07:24:51.968632+0000","last_became_active":"2024-03-06T07:24:51.967814+0000","last_became_peered":"2024-03-06T07:24:51.967814+0000","last_unstale":"2024-03-06T07:24:51.968632+0000","last_undegraded":"2024-03-06T07:24:51.968632+0000","last_fullsized":"2024-03-06T07:24:51.968632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:37.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954421+0000","last_change":"2024-03-06T07:24:51.954421+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954421+0000","last_undegraded":"2024-03-06T07:24:51.954421+0000","last_fullsized":"2024-03-06T07:24:51.954421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961459+0000","last_change":"2024-03-06T07:24:51.958710+0000","last_active":"2024-03-06T07:24:51.961459+0000","last_peered":"2024-03-06T07:24:51.961459+0000","last_clean":"2024-03-06T07:24:51.961459+0000","last_became_active":"2024-03-06T07:24:51.958435+0000","last_became_peered":"2024-03-06T07:24:51.958435+0000","last_unstale":"2024-03-06T07:24:51.961459+0000","last_undegraded":"2024-03-06T07:24:51.961459+0000","last_fullsized":"2024-03-06T07:24:51.961459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:29.966173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956317+0000","last_change":"2024-03-06T07:24:51.956317+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956317+0000","last_undegraded":"2024-03-06T07:24:51.956317+0000","last_fullsized":"2024-03-06T07:24:51.956317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964468+0000","last_change":"2024-03-06T07:24:51.962531+0000","last_active":"2024-03-06T07:24:51.964468+0000","last_peered":"2024-03-06T07:24:51.964468+0000","last_clean":"2024-03-06T07:24:51.964468+0000","last_became_active":"2024-03-06T07:24:51.962253+0000","last_became_peered":"2024-03-06T07:24:51.962253+0000","last_unstale":"2024-03-06T07:24:51.964468+0000","last_undegraded":"2024-03-06T07:24:51.964468+0000","last_fullsized":"2024-03-06T07:24:51.964468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:11.864382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953329+0000","last_change":"2024-03-06T07:24:51.953329+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953329+0000","last_undegraded":"2024-03-06T07:24:51.953329+0000","last_fullsized":"2024-03-06T07:24:51.953329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967111+0000","last_change":"2024-03-06T07:24:51.963025+0000","last_active":"2024-03-06T07:24:51.967111+0000","last_peered":"2024-03-06T07:24:51.967111+0000","last_clean":"2024-03-06T07:24:51.967111+0000","last_became_active":"2024-03-06T07:24:51.962710+0000","last_became_peered":"2024-03-06T07:24:51.962710+0000","last_unstale":"2024-03-06T07:24:51.967111+0000","last_undegraded":"2024-03-06T07:24:51.967111+0000","last_fullsized":"2024-03-06T07:24:51.967111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:21.637787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954067+0000","last_change":"2024-03-06T07:24:51.954067+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954067+0000","last_undegraded":"2024-03-06T07:24:51.954067+0000","last_fullsized":"2024-03-06T07:24:51.954067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967246+0000","last_change":"2024-03-06T07:24:51.963634+0000","last_active":"2024-03-06T07:24:51.967246+0000","last_peered":"2024-03-06T07:24:51.967246+0000","last_clean":"2024-03-06T07:24:51.967246+0000","last_became_active":"2024-03-06T07:24:51.963368+0000","last_became_peered":"2024-03-06T07:24:51.963368+0000","last_unstale":"2024-03-06T07:24:51.967246+0000","last_undegraded":"2024-03-06T07:24:51.967246+0000","last_fullsized":"2024-03-06T07:24:51.967246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:22.343338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948955+0000","last_change":"2024-03-06T07:24:51.948955+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948955+0000","last_undegraded":"2024-03-06T07:24:51.948955+0000","last_fullsized":"2024-03-06T07:24:51.948955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967349+0000","last_change":"2024-03-06T07:24:51.964342+0000","last_active":"2024-03-06T07:24:51.967349+0000","last_peered":"2024-03-06T07:24:51.967349+0000","last_clean":"2024-03-06T07:24:51.967349+0000","last_became_active":"2024-03-06T07:24:51.964058+0000","last_became_peered":"2024-03-06T07:24:51.964058+0000","last_unstale":"2024-03-06T07:24:51.967349+0000","last_undegraded":"2024-03-06T07:24:51.967349+0000","last_fullsized":"2024-03-06T07:24:51.967349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:27.238164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.956184+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:51.956184+0000","last_peered":"2024-03-06T07:24:51.956184+0000","last_clean":"2024-03-06T07:24:51.956184+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:51.956184+0000","last_undegraded":"2024-03-06T07:24:51.956184+0000","last_fullsized":"2024-03-06T07:24:51.956184+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:56.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3f is not active+clean 2024-03-06T07:24:56.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955811+0000', 'last_change': '2024-03-06T07:24:51.955811+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955811+0000', 'last_undegraded': '2024-03-06T07:24:51.955811+0000', 'last_fullsized': '2024-03-06T07:24:51.955811+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3e is not active+clean 2024-03-06T07:24:56.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956564+0000', 'last_change': '2024-03-06T07:24:51.956564+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956564+0000', 'last_undegraded': '2024-03-06T07:24:51.956564+0000', 'last_fullsized': '2024-03-06T07:24:51.956564+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3d is not active+clean 2024-03-06T07:24:56.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949967+0000', 'last_change': '2024-03-06T07:24:51.949967+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949967+0000', 'last_undegraded': '2024-03-06T07:24:51.949967+0000', 'last_fullsized': '2024-03-06T07:24:51.949967+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3c is not active+clean 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954692+0000', 'last_change': '2024-03-06T07:24:51.954692+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954692+0000', 'last_undegraded': '2024-03-06T07:24:51.954692+0000', 'last_fullsized': '2024-03-06T07:24:51.954692+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3b is not active+clean 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.957393+0000', 'last_change': '2024-03-06T07:24:51.957393+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.957393+0000', 'last_undegraded': '2024-03-06T07:24:51.957393+0000', 'last_fullsized': '2024-03-06T07:24:51.957393+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3a is not active+clean 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956546+0000', 'last_change': '2024-03-06T07:24:51.956546+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956546+0000', 'last_undegraded': '2024-03-06T07:24:51.956546+0000', 'last_fullsized': '2024-03-06T07:24:51.956546+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.39 is not active+clean 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.950665+0000', 'last_change': '2024-03-06T07:24:51.950665+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.950665+0000', 'last_undegraded': '2024-03-06T07:24:51.950665+0000', 'last_fullsized': '2024-03-06T07:24:51.950665+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.38 is not active+clean 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954574+0000', 'last_change': '2024-03-06T07:24:51.954574+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954574+0000', 'last_undegraded': '2024-03-06T07:24:51.954574+0000', 'last_fullsized': '2024-03-06T07:24:51.954574+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.37 is not active+clean 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956937+0000', 'last_change': '2024-03-06T07:24:51.956937+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956937+0000', 'last_undegraded': '2024-03-06T07:24:51.956937+0000', 'last_fullsized': '2024-03-06T07:24:51.956937+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.36 is not active+clean 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953539+0000', 'last_change': '2024-03-06T07:24:51.953539+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953539+0000', 'last_undegraded': '2024-03-06T07:24:51.953539+0000', 'last_fullsized': '2024-03-06T07:24:51.953539+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.35 is not active+clean 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954128+0000', 'last_change': '2024-03-06T07:24:51.954128+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954128+0000', 'last_undegraded': '2024-03-06T07:24:51.954128+0000', 'last_fullsized': '2024-03-06T07:24:51.954128+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.34 is not active+clean 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954265+0000', 'last_change': '2024-03-06T07:24:51.954265+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954265+0000', 'last_undegraded': '2024-03-06T07:24:51.954265+0000', 'last_fullsized': '2024-03-06T07:24:51.954265+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.33 is not active+clean 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956604+0000', 'last_change': '2024-03-06T07:24:51.956604+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956604+0000', 'last_undegraded': '2024-03-06T07:24:51.956604+0000', 'last_fullsized': '2024-03-06T07:24:51.956604+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.32 is not active+clean 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955792+0000', 'last_change': '2024-03-06T07:24:51.955792+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955792+0000', 'last_undegraded': '2024-03-06T07:24:51.955792+0000', 'last_fullsized': '2024-03-06T07:24:51.955792+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.31 is not active+clean 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956049+0000', 'last_change': '2024-03-06T07:24:51.956049+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956049+0000', 'last_undegraded': '2024-03-06T07:24:51.956049+0000', 'last_fullsized': '2024-03-06T07:24:51.956049+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.30 is not active+clean 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.951835+0000', 'last_change': '2024-03-06T07:24:51.951835+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.951835+0000', 'last_undegraded': '2024-03-06T07:24:51.951835+0000', 'last_fullsized': '2024-03-06T07:24:51.951835+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2f is not active+clean 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956373+0000', 'last_change': '2024-03-06T07:24:51.956373+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956373+0000', 'last_undegraded': '2024-03-06T07:24:51.956373+0000', 'last_fullsized': '2024-03-06T07:24:51.956373+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2e is not active+clean 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953867+0000', 'last_change': '2024-03-06T07:24:51.953867+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953867+0000', 'last_undegraded': '2024-03-06T07:24:51.953867+0000', 'last_fullsized': '2024-03-06T07:24:51.953867+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2d is not active+clean 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955657+0000', 'last_change': '2024-03-06T07:24:51.955657+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955657+0000', 'last_undegraded': '2024-03-06T07:24:51.955657+0000', 'last_fullsized': '2024-03-06T07:24:51.955657+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2c is not active+clean 2024-03-06T07:24:56.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955630+0000', 'last_change': '2024-03-06T07:24:51.955630+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955630+0000', 'last_undegraded': '2024-03-06T07:24:51.955630+0000', 'last_fullsized': '2024-03-06T07:24:51.955630+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2b is not active+clean 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954554+0000', 'last_change': '2024-03-06T07:24:51.954554+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954554+0000', 'last_undegraded': '2024-03-06T07:24:51.954554+0000', 'last_fullsized': '2024-03-06T07:24:51.954554+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2a is not active+clean 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.952981+0000', 'last_change': '2024-03-06T07:24:51.952981+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.952981+0000', 'last_undegraded': '2024-03-06T07:24:51.952981+0000', 'last_fullsized': '2024-03-06T07:24:51.952981+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.29 is not active+clean 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.952956+0000', 'last_change': '2024-03-06T07:24:51.952956+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.952956+0000', 'last_undegraded': '2024-03-06T07:24:51.952956+0000', 'last_fullsized': '2024-03-06T07:24:51.952956+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.28 is not active+clean 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.951873+0000', 'last_change': '2024-03-06T07:24:51.951873+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.951873+0000', 'last_undegraded': '2024-03-06T07:24:51.951873+0000', 'last_fullsized': '2024-03-06T07:24:51.951873+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.27 is not active+clean 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953582+0000', 'last_change': '2024-03-06T07:24:51.953582+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953582+0000', 'last_undegraded': '2024-03-06T07:24:51.953582+0000', 'last_fullsized': '2024-03-06T07:24:51.953582+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.26 is not active+clean 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.950728+0000', 'last_change': '2024-03-06T07:24:51.950728+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.950728+0000', 'last_undegraded': '2024-03-06T07:24:51.950728+0000', 'last_fullsized': '2024-03-06T07:24:51.950728+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.25 is not active+clean 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949352+0000', 'last_change': '2024-03-06T07:24:51.949352+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949352+0000', 'last_undegraded': '2024-03-06T07:24:51.949352+0000', 'last_fullsized': '2024-03-06T07:24:51.949352+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.24 is not active+clean 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953082+0000', 'last_change': '2024-03-06T07:24:51.953082+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953082+0000', 'last_undegraded': '2024-03-06T07:24:51.953082+0000', 'last_fullsized': '2024-03-06T07:24:51.953082+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.23 is not active+clean 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949990+0000', 'last_change': '2024-03-06T07:24:51.949990+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949990+0000', 'last_undegraded': '2024-03-06T07:24:51.949990+0000', 'last_fullsized': '2024-03-06T07:24:51.949990+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.22 is not active+clean 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.950896+0000', 'last_change': '2024-03-06T07:24:51.950896+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.950896+0000', 'last_undegraded': '2024-03-06T07:24:51.950896+0000', 'last_fullsized': '2024-03-06T07:24:51.950896+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.21 is not active+clean 2024-03-06T07:24:56.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949630+0000', 'last_change': '2024-03-06T07:24:51.949630+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949630+0000', 'last_undegraded': '2024-03-06T07:24:51.949630+0000', 'last_fullsized': '2024-03-06T07:24:51.949630+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.20 is not active+clean 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954177+0000', 'last_change': '2024-03-06T07:24:51.954177+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954177+0000', 'last_undegraded': '2024-03-06T07:24:51.954177+0000', 'last_fullsized': '2024-03-06T07:24:51.954177+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953434+0000', 'last_change': '2024-03-06T07:24:51.953434+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953434+0000', 'last_undegraded': '2024-03-06T07:24:51.953434+0000', 'last_fullsized': '2024-03-06T07:24:51.953434+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955019+0000', 'last_change': '2024-03-06T07:24:51.955019+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955019+0000', 'last_undegraded': '2024-03-06T07:24:51.955019+0000', 'last_fullsized': '2024-03-06T07:24:51.955019+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956036+0000', 'last_change': '2024-03-06T07:24:51.956036+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956036+0000', 'last_undegraded': '2024-03-06T07:24:51.956036+0000', 'last_fullsized': '2024-03-06T07:24:51.956036+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953707+0000', 'last_change': '2024-03-06T07:24:51.953707+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953707+0000', 'last_undegraded': '2024-03-06T07:24:51.953707+0000', 'last_fullsized': '2024-03-06T07:24:51.953707+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-03-06T07:24:56.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949619+0000', 'last_change': '2024-03-06T07:24:51.949619+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949619+0000', 'last_undegraded': '2024-03-06T07:24:51.949619+0000', 'last_fullsized': '2024-03-06T07:24:51.949619+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954185+0000', 'last_change': '2024-03-06T07:24:51.954185+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954185+0000', 'last_undegraded': '2024-03-06T07:24:51.954185+0000', 'last_fullsized': '2024-03-06T07:24:51.954185+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954642+0000', 'last_change': '2024-03-06T07:24:51.954642+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954642+0000', 'last_undegraded': '2024-03-06T07:24:51.954642+0000', 'last_fullsized': '2024-03-06T07:24:51.954642+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954942+0000', 'last_change': '2024-03-06T07:24:51.954942+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954942+0000', 'last_undegraded': '2024-03-06T07:24:51.954942+0000', 'last_fullsized': '2024-03-06T07:24:51.954942+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949800+0000', 'last_change': '2024-03-06T07:24:51.949800+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949800+0000', 'last_undegraded': '2024-03-06T07:24:51.949800+0000', 'last_fullsized': '2024-03-06T07:24:51.949800+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-03-06T07:24:56.627 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955422+0000', 'last_change': '2024-03-06T07:24:51.955422+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955422+0000', 'last_undegraded': '2024-03-06T07:24:51.955422+0000', 'last_fullsized': '2024-03-06T07:24:51.955422+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955625+0000', 'last_change': '2024-03-06T07:24:51.955625+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955625+0000', 'last_undegraded': '2024-03-06T07:24:51.955625+0000', 'last_fullsized': '2024-03-06T07:24:51.955625+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.955262+0000', 'last_change': '2024-03-06T07:24:51.955262+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.955262+0000', 'last_undegraded': '2024-03-06T07:24:51.955262+0000', 'last_fullsized': '2024-03-06T07:24:51.955262+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954553+0000', 'last_change': '2024-03-06T07:24:51.954553+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954553+0000', 'last_undegraded': '2024-03-06T07:24:51.954553+0000', 'last_fullsized': '2024-03-06T07:24:51.954553+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949626+0000', 'last_change': '2024-03-06T07:24:51.949626+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949626+0000', 'last_undegraded': '2024-03-06T07:24:51.949626+0000', 'last_fullsized': '2024-03-06T07:24:51.949626+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954508+0000', 'last_change': '2024-03-06T07:24:51.954508+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954508+0000', 'last_undegraded': '2024-03-06T07:24:51.954508+0000', 'last_fullsized': '2024-03-06T07:24:51.954508+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:56.628 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954701+0000', 'last_change': '2024-03-06T07:24:51.954701+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954701+0000', 'last_undegraded': '2024-03-06T07:24:51.954701+0000', 'last_fullsized': '2024-03-06T07:24:51.954701+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953114+0000', 'last_change': '2024-03-06T07:24:51.953114+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953114+0000', 'last_undegraded': '2024-03-06T07:24:51.953114+0000', 'last_fullsized': '2024-03-06T07:24:51.953114+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954975+0000', 'last_change': '2024-03-06T07:24:51.954975+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954975+0000', 'last_undegraded': '2024-03-06T07:24:51.954975+0000', 'last_fullsized': '2024-03-06T07:24:51.954975+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953854+0000', 'last_change': '2024-03-06T07:24:51.953854+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953854+0000', 'last_undegraded': '2024-03-06T07:24:51.953854+0000', 'last_fullsized': '2024-03-06T07:24:51.953854+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954657+0000', 'last_change': '2024-03-06T07:24:51.954657+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954657+0000', 'last_undegraded': '2024-03-06T07:24:51.954657+0000', 'last_fullsized': '2024-03-06T07:24:51.954657+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.629 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949563+0000', 'last_change': '2024-03-06T07:24:51.949563+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949563+0000', 'last_undegraded': '2024-03-06T07:24:51.949563+0000', 'last_fullsized': '2024-03-06T07:24:51.949563+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.948968+0000', 'last_change': '2024-03-06T07:24:51.948968+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.948968+0000', 'last_undegraded': '2024-03-06T07:24:51.948968+0000', 'last_fullsized': '2024-03-06T07:24:51.948968+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949253+0000', 'last_change': '2024-03-06T07:24:51.949253+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949253+0000', 'last_undegraded': '2024-03-06T07:24:51.949253+0000', 'last_fullsized': '2024-03-06T07:24:51.949253+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953963+0000', 'last_change': '2024-03-06T07:24:51.953963+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953963+0000', 'last_undegraded': '2024-03-06T07:24:51.953963+0000', 'last_fullsized': '2024-03-06T07:24:51.953963+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.949805+0000', 'last_change': '2024-03-06T07:24:51.949805+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.949805+0000', 'last_undegraded': '2024-03-06T07:24:51.949805+0000', 'last_fullsized': '2024-03-06T07:24:51.949805+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-03-06T07:24:56.630 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.952214+0000', 'last_change': '2024-03-06T07:24:51.952214+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.952214+0000', 'last_undegraded': '2024-03-06T07:24:51.952214+0000', 'last_fullsized': '2024-03-06T07:24:51.952214+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.952016+0000', 'last_change': '2024-03-06T07:24:51.952016+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.952016+0000', 'last_undegraded': '2024-03-06T07:24:51.952016+0000', 'last_fullsized': '2024-03-06T07:24:51.952016+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954421+0000', 'last_change': '2024-03-06T07:24:51.954421+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954421+0000', 'last_undegraded': '2024-03-06T07:24:51.954421+0000', 'last_fullsized': '2024-03-06T07:24:51.954421+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.956317+0000', 'last_change': '2024-03-06T07:24:51.956317+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.956317+0000', 'last_undegraded': '2024-03-06T07:24:51.956317+0000', 'last_fullsized': '2024-03-06T07:24:51.956317+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.953329+0000', 'last_change': '2024-03-06T07:24:51.953329+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.953329+0000', 'last_undegraded': '2024-03-06T07:24:51.953329+0000', 'last_fullsized': '2024-03-06T07:24:51.953329+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.954067+0000', 'last_change': '2024-03-06T07:24:51.954067+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.954067+0000', 'last_undegraded': '2024-03-06T07:24:51.954067+0000', 'last_fullsized': '2024-03-06T07:24:51.954067+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-03-06T07:24:56.631 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 69, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:24:51.948955+0000', 'last_change': '2024-03-06T07:24:51.948955+0000', 'last_active': '2024-03-06T07:24:51.944226+0000', 'last_peered': '2024-03-06T07:24:51.944226+0000', 'last_clean': '2024-03-06T07:24:51.944226+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:24:51.948955+0000', 'last_undegraded': '2024-03-06T07:24:51.948955+0000', 'last_fullsized': '2024-03-06T07:24:51.948955+0000', 'mapping_epoch': 69, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 69, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'last_clean_scrub_stamp': '2024-03-06T07:24:51.944226+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:24:56.632 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:24:56.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.754+0000 7f2396484640 1 Processor -- start 2024-03-06T07:24:56.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.754+0000 7f2396484640 1 -- start start 2024-03-06T07:24:56.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f2396484640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f23900b3240 0x7f23900b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f2396484640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239015e3f0 con 0x7f239015e010 2024-03-06T07:24:56.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f2396484640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23900b1c10 con 0x7f23900b2910 2024-03-06T07:24:56.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f2396484640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23900b1f40 con 0x7f23900b3240 2024-03-06T07:24:56.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f239015e010 0x7f23900cedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37810/0 (socket says 172.21.15.72:37810) 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238ffff640 1 -- 172.21.15.72:0/1832519338 learned_addr learned my addr 172.21.15.72:0/1832519338 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238f7fe640 1 --2- 172.21.15.72:0/1832519338 >> v2:172.21.15.146:3300/0 conn(0x7f23900b3240 0x7f23900b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238effd640 1 -- 172.21.15.72:0/1832519338 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202031865 0 0) 0x7f239015e3f0 con 0x7f239015e010 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238effd640 1 -- 172.21.15.72:0/1832519338 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378003520 con 0x7f239015e010 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238f7fe640 1 -- 172.21.15.72:0/1832519338 >> v1:172.21.15.146:6789/0 conn(0x7f23900b2910 legacy=0x7f23900d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238f7fe640 1 -- 172.21.15.72:0/1832519338 >> v1:172.21.15.72:6789/0 conn(0x7f239015e010 legacy=0x7f23900cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.755+0000 7f238f7fe640 1 -- 172.21.15.72:0/1832519338 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239006d1b0 con 0x7f23900b3240 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.756+0000 7f238f7fe640 1 --2- 172.21.15.72:0/1832519338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900b3240 0x7f23900b23d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f23840097a0 tx=0x7f238402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28a3343e42e980e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.756+0000 7f238effd640 1 -- 172.21.15.72:0/1832519338 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384035020 con 0x7f23900b3240 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.756+0000 7f238effd640 1 -- 172.21.15.72:0/1832519338 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384031830 con 0x7f23900b3240 2024-03-06T07:24:56.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.756+0000 7f238effd640 1 -- 172.21.15.72:0/1832519338 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384031c00 con 0x7f23900b3240 2024-03-06T07:24:56.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 -- 172.21.15.72:0/1832519338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900b3240 msgr2=0x7f23900b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 --2- 172.21.15.72:0/1832519338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900b3240 0x7f23900b23d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f23840097a0 tx=0x7f238402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 -- 172.21.15.72:0/1832519338 shutdown_connections 2024-03-06T07:24:56.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 --2- 172.21.15.72:0/1832519338 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900b3240 0x7f23900b23d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 -- 172.21.15.72:0/1832519338 >> 172.21.15.72:0/1832519338 conn(0x7f239009a9f0 msgr2=0x7f239009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 -- 172.21.15.72:0/1832519338 shutdown_connections 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.757+0000 7f2396484640 1 -- 172.21.15.72:0/1832519338 wait complete. 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.758+0000 7f2396484640 1 Processor -- start 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.758+0000 7f2396484640 1 -- start start 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2396484640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239015e010 0x7f23900b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2396484640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 0x7f23900c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2396484640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900c5700 0x7f23900c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2396484640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23900d21e0 con 0x7f23900c79d0 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2396484640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239006ced0 con 0x7f23900c5700 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2396484640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23900d2360 con 0x7f239015e010 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 0x7f23900c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 0x7f23900c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37756/0 (socket says 172.21.15.72:37756) 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 -- 172.21.15.72:0/3180995617 learned_addr learned my addr 172.21.15.72:0/3180995617 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f2394a21640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900c5700 0x7f23900c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238ffff640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239015e010 0x7f23900b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 -- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239015e010 msgr2=0x7f23900b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239015e010 0x7f23900b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 -- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900c5700 msgr2=0x7f23900c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900c5700 0x7f23900c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 -- 172.21.15.72:0/3180995617 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23900b6fb0 con 0x7f23900c79d0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238f7fe640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 0x7f23900c7e50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2384009890 tx=0x7f23840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.759+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384035040 con 0x7f23900c79d0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.760+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238403b920 con 0x7f23900c79d0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.760+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23900c6480 con 0x7f23900c79d0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.760+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238403bd10 con 0x7f23900c79d0 2024-03-06T07:24:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.760+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23900c6a40 con 0x7f23900c79d0 2024-03-06T07:24:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.761+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239004f330 con 0x7f23900c79d0 2024-03-06T07:24:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.761+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2384038020 con 0x7f23900c79d0 2024-03-06T07:24:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.761+0000 7f238cff9640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.761+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23840ac190 con 0x7f23900c79d0 2024-03-06T07:24:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.761+0000 7f238ffff640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.762+0000 7f238ffff640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f23800011e0 tx=0x7f2380007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:56.766+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2384031800 con 0x7f23900c79d0 2024-03-06T07:24:57.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.006+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f23900d2980 con 0x7f23900c79d0 2024-03-06T07:24:57.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.006+0000 7f238cff9640 1 -- 172.21.15.72:0/3180995617 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7f238407a9d0 con 0x7f23900c79d0 2024-03-06T07:24:57.008 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:57.008 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":211,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":71,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709696,"num_in_osds":8,"osd_in_since":1709709687,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":263233536,"bytes_avail":772830879744,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":33,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"b","status":"up:active","gid":6579}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:23:11.962885+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:24:57.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 msgr2=0x7f237c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f23800011e0 tx=0x7f2380007d30 comp rx=0 tx=0).stop 2024-03-06T07:24:57.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 msgr2=0x7f23900c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 0x7f23900c7e50 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2384009890 tx=0x7f23840046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 shutdown_connections 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23900c5700 0x7f23900c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23900c79d0 0x7f23900c7e50 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 --2- 172.21.15.72:0/3180995617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239015e010 0x7f23900b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 >> 172.21.15.72:0/3180995617 conn(0x7f239009a9f0 msgr2=0x7f23900702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 shutdown_connections 2024-03-06T07:24:57.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.008+0000 7f2396484640 1 -- 172.21.15.72:0/3180995617 wait complete. 2024-03-06T07:24:57.021 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:24:57.021 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:24:57.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 Processor -- start 2024-03-06T07:24:57.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 -- start start 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77a80cf830 0x7f77a80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77a80d27f0 0x7f77a80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a80d2c50 con 0x7f77a80cf830 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a80b31e0 con 0x7f77a80d27f0 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.119+0000 7f77ad5c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80b3d60 con 0x7f77a815dfe0 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77a80cf830 0x7f77a80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77a80cf830 0x7f77a80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37758/0 (socket says 172.21.15.72:37758) 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 -- 172.21.15.72:0/4133509908 learned_addr learned my addr 172.21.15.72:0/4133509908 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a6ffd640 1 --2- 172.21.15.72:0/4133509908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a77fe640 1 --2- 172.21.15.72:0/4133509908 >> v2:172.21.15.146:3300/0 conn(0x7f77a80d27f0 0x7f77a80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 -- 172.21.15.72:0/4133509908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 msgr2=0x7f77a80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 --2- 172.21.15.72:0/4133509908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 -- 172.21.15.72:0/4133509908 >> v2:172.21.15.146:3300/0 conn(0x7f77a80d27f0 msgr2=0x7f77a80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:24:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 --2- 172.21.15.72:0/4133509908 >> v2:172.21.15.146:3300/0 conn(0x7f77a80d27f0 0x7f77a80d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.120+0000 7f77a67fc640 1 -- 172.21.15.72:0/4133509908 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80b3430 con 0x7f77a80cf830 2024-03-06T07:24:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77a67fc640 1 --2- 172.21.15.72:0/4133509908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80cf830 0x7f77a80d22b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f77a8067ad0 tx=0x7f779c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a76fb3f2923af server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77a5ffb640 1 -- 172.21.15.72:0/4133509908 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c035020 con 0x7f77a80cf830 2024-03-06T07:24:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77a5ffb640 1 -- 172.21.15.72:0/4133509908 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c031870 con 0x7f77a80cf830 2024-03-06T07:24:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77a5ffb640 1 -- 172.21.15.72:0/4133509908 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c031c40 con 0x7f77a80cf830 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 -- 172.21.15.72:0/4133509908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80cf830 msgr2=0x7f77a80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/4133509908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80cf830 0x7f77a80d22b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f77a8067ad0 tx=0x7f779c02b910 comp rx=0 tx=0).stop 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 -- 172.21.15.72:0/4133509908 shutdown_connections 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/4133509908 >> v2:172.21.15.146:3300/0 conn(0x7f77a80d27f0 0x7f77a80d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/4133509908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80cf830 0x7f77a80d22b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/4133509908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 -- 172.21.15.72:0/4133509908 >> 172.21.15.72:0/4133509908 conn(0x7f77a80d3cd0 msgr2=0x7f77a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 -- 172.21.15.72:0/4133509908 shutdown_connections 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.121+0000 7f77ad5c9640 1 -- 172.21.15.72:0/4133509908 wait complete. 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 Processor -- start 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 -- start start 2024-03-06T07:24:57.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a80cf830 0x7f77a814e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 0x7f77a8161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80b8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80d1c70 con 0x7f77a80d27f0 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80b39f0 con 0x7f77a80cf830 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77ad5c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a80d1df0 con 0x7f77a815dfe0 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77a67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 0x7f77a8161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77a67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 0x7f77a8161040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37762/0 (socket says 172.21.15.72:37762) 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.123+0000 7f77a67fc640 1 -- 172.21.15.72:0/3804523043 learned_addr learned my addr 172.21.15.72:0/3804523043 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a6ffd640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a80cf830 0x7f77a814e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a77fe640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a67fc640 1 -- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 msgr2=0x7f77a80b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a67fc640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a67fc640 1 -- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a80cf830 msgr2=0x7f77a814e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a67fc640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a80cf830 0x7f77a814e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a67fc640 1 -- 172.21.15.72:0/3804523043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a80b6b20 con 0x7f77a80d27f0 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77a67fc640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 0x7f77a8161040 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f779c02b950 tx=0x7f779c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c040070 con 0x7f77a80d27f0 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c031e30 con 0x7f77a80d27f0 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a80b6e00 con 0x7f77a80d27f0 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c035040 con 0x7f77a80d27f0 2024-03-06T07:24:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.124+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a80b73c0 con 0x7f77a80d27f0 2024-03-06T07:24:57.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.125+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a804f330 con 0x7f77a80d27f0 2024-03-06T07:24:57.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.125+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f779c038020 con 0x7f77a80d27f0 2024-03-06T07:24:57.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.126+0000 7f778f7fe640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f779406b4d0 0x7f779406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:24:57.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.126+0000 7f77a6ffd640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f779406b4d0 0x7f779406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:24:57.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.126+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f779c0abe80 con 0x7f77a80d27f0 2024-03-06T07:24:57.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.126+0000 7f77a6ffd640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f779406b4d0 0x7f779406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f77a806ccf0 tx=0x7f7798009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:24:57.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.130+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f779c07a6b0 con 0x7f77a80d27f0 2024-03-06T07:24:57.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.299+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77a80b68d0 con 0x7f779406b4d0 2024-03-06T07:24:57.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.304+0000 7f778f7fe640 1 -- 172.21.15.72:0/3804523043 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f77a80b68d0 con 0x7f779406b4d0 2024-03-06T07:24:57.309 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:24:57.311 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:24:57.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f779406b4d0 msgr2=0x7f779406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f779406b4d0 0x7f779406d990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f77a806ccf0 tx=0x7f7798009190 comp rx=0 tx=0).stop 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 msgr2=0x7f77a8161040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 0x7f77a8161040 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f779c02b950 tx=0x7f779c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 shutdown_connections 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f779406b4d0 0x7f779406d990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77a815dfe0 0x7f77a80b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77a80d27f0 0x7f77a8161040 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 --2- 172.21.15.72:0/3804523043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77a80cf830 0x7f77a814e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 >> 172.21.15.72:0/3804523043 conn(0x7f77a80d3cd0 msgr2=0x7f77a80b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 shutdown_connections 2024-03-06T07:24:57.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:24:57.307+0000 7f77ad5c9640 1 -- 172.21.15.72:0/3804523043 wait complete. 2024-03-06T07:24:57.321 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-03-06T07:24:55.980586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257064,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717656,"statfs":{"total":773094113280,"available":772830879744,"internally_reserved":0,"allocated":7806976,"data_stored":3935832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.035949"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955811+0000","last_change":"2024-03-06T07:24:51.955811+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955811+0000","last_undegraded":"2024-03-06T07:24:51.955811+0000","last_fullsized":"2024-03-06T07:24:51.955811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956564+0000","last_change":"2024-03-06T07:24:51.956564+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956564+0000","last_undegraded":"2024-03-06T07:24:51.956564+0000","last_fullsized":"2024-03-06T07:24:51.956564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949967+0000","last_change":"2024-03-06T07:24:51.949967+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949967+0000","last_undegraded":"2024-03-06T07:24:51.949967+0000","last_fullsized":"2024-03-06T07:24:51.949967+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954692+0000","last_change":"2024-03-06T07:24:51.954692+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954692+0000","last_undegraded":"2024-03-06T07:24:51.954692+0000","last_fullsized":"2024-03-06T07:24:51.954692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.957393+0000","last_change":"2024-03-06T07:24:51.957393+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.957393+0000","last_undegraded":"2024-03-06T07:24:51.957393+0000","last_fullsized":"2024-03-06T07:24:51.957393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956546+0000","last_change":"2024-03-06T07:24:51.956546+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956546+0000","last_undegraded":"2024-03-06T07:24:51.956546+0000","last_fullsized":"2024-03-06T07:24:51.956546+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950665+0000","last_change":"2024-03-06T07:24:51.950665+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950665+0000","last_undegraded":"2024-03-06T07:24:51.950665+0000","last_fullsized":"2024-03-06T07:24:51.950665+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954574+0000","last_change":"2024-03-06T07:24:51.954574+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954574+0000","last_undegraded":"2024-03-06T07:24:51.954574+0000","last_fullsized":"2024-03-06T07:24:51.954574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956937+0000","last_change":"2024-03-06T07:24:51.956937+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956937+0000","last_undegraded":"2024-03-06T07:24:51.956937+0000","last_fullsized":"2024-03-06T07:24:51.956937+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953539+0000","last_change":"2024-03-06T07:24:51.953539+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953539+0000","last_undegraded":"2024-03-06T07:24:51.953539+0000","last_fullsized":"2024-03-06T07:24:51.953539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954128+0000","last_change":"2024-03-06T07:24:51.954128+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954128+0000","last_undegraded":"2024-03-06T07:24:51.954128+0000","last_fullsized":"2024-03-06T07:24:51.954128+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954265+0000","last_change":"2024-03-06T07:24:51.954265+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954265+0000","last_undegraded":"2024-03-06T07:24:51.954265+0000","last_fullsized":"2024-03-06T07:24:51.954265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956604+0000","last_change":"2024-03-06T07:24:51.956604+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956604+0000","last_undegraded":"2024-03-06T07:24:51.956604+0000","last_fullsized":"2024-03-06T07:24:51.956604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955792+0000","last_change":"2024-03-06T07:24:51.955792+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955792+0000","last_undegraded":"2024-03-06T07:24:51.955792+0000","last_fullsized":"2024-03-06T07:24:51.955792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956049+0000","last_change":"2024-03-06T07:24:51.956049+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956049+0000","last_undegraded":"2024-03-06T07:24:51.956049+0000","last_fullsized":"2024-03-06T07:24:51.956049+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951835+0000","last_change":"2024-03-06T07:24:51.951835+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951835+0000","last_undegraded":"2024-03-06T07:24:51.951835+0000","last_fullsized":"2024-03-06T07:24:51.951835+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956373+0000","last_change":"2024-03-06T07:24:51.956373+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956373+0000","last_undegraded":"2024-03-06T07:24:51.956373+0000","last_fullsized":"2024-03-06T07:24:51.956373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953867+0000","last_change":"2024-03-06T07:24:51.953867+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953867+0000","last_undegraded":"2024-03-06T07:24:51.953867+0000","last_fullsized":"2024-03-06T07:24:51.953867+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955657+0000","last_change":"2024-03-06T07:24:51.955657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955657+0000","last_undegraded":"2024-03-06T07:24:51.955657+0000","last_fullsized":"2024-03-06T07:24:51.955657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955630+0000","last_change":"2024-03-06T07:24:51.955630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955630+0000","last_undegraded":"2024-03-06T07:24:51.955630+0000","last_fullsized":"2024-03-06T07:24:51.955630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954554+0000","last_change":"2024-03-06T07:24:51.954554+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954554+0000","last_undegraded":"2024-03-06T07:24:51.954554+0000","last_fullsized":"2024-03-06T07:24:51.954554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952981+0000","last_change":"2024-03-06T07:24:51.952981+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952981+0000","last_undegraded":"2024-03-06T07:24:51.952981+0000","last_fullsized":"2024-03-06T07:24:51.952981+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952956+0000","last_change":"2024-03-06T07:24:51.952956+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952956+0000","last_undegraded":"2024-03-06T07:24:51.952956+0000","last_fullsized":"2024-03-06T07:24:51.952956+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.951873+0000","last_change":"2024-03-06T07:24:51.951873+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.951873+0000","last_undegraded":"2024-03-06T07:24:51.951873+0000","last_fullsized":"2024-03-06T07:24:51.951873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953582+0000","last_change":"2024-03-06T07:24:51.953582+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953582+0000","last_undegraded":"2024-03-06T07:24:51.953582+0000","last_fullsized":"2024-03-06T07:24:51.953582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950728+0000","last_change":"2024-03-06T07:24:51.950728+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950728+0000","last_undegraded":"2024-03-06T07:24:51.950728+0000","last_fullsized":"2024-03-06T07:24:51.950728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949352+0000","last_change":"2024-03-06T07:24:51.949352+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949352+0000","last_undegraded":"2024-03-06T07:24:51.949352+0000","last_fullsized":"2024-03-06T07:24:51.949352+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953082+0000","last_change":"2024-03-06T07:24:51.953082+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953082+0000","last_undegraded":"2024-03-06T07:24:51.953082+0000","last_fullsized":"2024-03-06T07:24:51.953082+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949990+0000","last_change":"2024-03-06T07:24:51.949990+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949990+0000","last_undegraded":"2024-03-06T07:24:51.949990+0000","last_fullsized":"2024-03-06T07:24:51.949990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.950896+0000","last_change":"2024-03-06T07:24:51.950896+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.950896+0000","last_undegraded":"2024-03-06T07:24:51.950896+0000","last_fullsized":"2024-03-06T07:24:51.950896+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949630+0000","last_change":"2024-03-06T07:24:51.949630+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949630+0000","last_undegraded":"2024-03-06T07:24:51.949630+0000","last_fullsized":"2024-03-06T07:24:51.949630+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954177+0000","last_change":"2024-03-06T07:24:51.954177+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954177+0000","last_undegraded":"2024-03-06T07:24:51.954177+0000","last_fullsized":"2024-03-06T07:24:51.954177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953434+0000","last_change":"2024-03-06T07:24:51.953434+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953434+0000","last_undegraded":"2024-03-06T07:24:51.953434+0000","last_fullsized":"2024-03-06T07:24:51.953434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966812+0000","last_change":"2024-03-06T07:24:51.962181+0000","last_active":"2024-03-06T07:24:51.966812+0000","last_peered":"2024-03-06T07:24:51.966812+0000","last_clean":"2024-03-06T07:24:51.966812+0000","last_became_active":"2024-03-06T07:24:51.961932+0000","last_became_peered":"2024-03-06T07:24:51.961932+0000","last_unstale":"2024-03-06T07:24:51.966812+0000","last_undegraded":"2024-03-06T07:24:51.966812+0000","last_fullsized":"2024-03-06T07:24:51.966812+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955019+0000","last_change":"2024-03-06T07:24:51.955019+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955019+0000","last_undegraded":"2024-03-06T07:24:51.955019+0000","last_fullsized":"2024-03-06T07:24:51.955019+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964255+0000","last_change":"2024-03-06T07:24:51.962165+0000","last_active":"2024-03-06T07:24:51.964255+0000","last_peered":"2024-03-06T07:24:51.964255+0000","last_clean":"2024-03-06T07:24:51.964255+0000","last_became_active":"2024-03-06T07:24:51.961847+0000","last_became_peered":"2024-03-06T07:24:51.961847+0000","last_unstale":"2024-03-06T07:24:51.964255+0000","last_undegraded":"2024-03-06T07:24:51.964255+0000","last_fullsized":"2024-03-06T07:24:51.964255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956036+0000","last_change":"2024-03-06T07:24:51.956036+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956036+0000","last_undegraded":"2024-03-06T07:24:51.956036+0000","last_fullsized":"2024-03-06T07:24:51.956036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967209+0000","last_change":"2024-03-06T07:24:51.965487+0000","last_active":"2024-03-06T07:24:51.967209+0000","last_peered":"2024-03-06T07:24:51.967209+0000","last_clean":"2024-03-06T07:24:51.967209+0000","last_became_active":"2024-03-06T07:24:51.965210+0000","last_became_peered":"2024-03-06T07:24:51.965210+0000","last_unstale":"2024-03-06T07:24:51.967209+0000","last_undegraded":"2024-03-06T07:24:51.967209+0000","last_fullsized":"2024-03-06T07:24:51.967209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:18.007631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953707+0000","last_change":"2024-03-06T07:24:51.953707+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953707+0000","last_undegraded":"2024-03-06T07:24:51.953707+0000","last_fullsized":"2024-03-06T07:24:51.953707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967180+0000","last_change":"2024-03-06T07:24:51.963677+0000","last_active":"2024-03-06T07:24:51.967180+0000","last_peered":"2024-03-06T07:24:51.967180+0000","last_clean":"2024-03-06T07:24:51.967180+0000","last_became_active":"2024-03-06T07:24:51.963383+0000","last_became_peered":"2024-03-06T07:24:51.963383+0000","last_unstale":"2024-03-06T07:24:51.967180+0000","last_undegraded":"2024-03-06T07:24:51.967180+0000","last_fullsized":"2024-03-06T07:24:51.967180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:05.163132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949619+0000","last_change":"2024-03-06T07:24:51.949619+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949619+0000","last_undegraded":"2024-03-06T07:24:51.949619+0000","last_fullsized":"2024-03-06T07:24:51.949619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963718+0000","last_change":"2024-03-06T07:24:51.962180+0000","last_active":"2024-03-06T07:24:51.963718+0000","last_peered":"2024-03-06T07:24:51.963718+0000","last_clean":"2024-03-06T07:24:51.963718+0000","last_became_active":"2024-03-06T07:24:51.961911+0000","last_became_peered":"2024-03-06T07:24:51.961911+0000","last_unstale":"2024-03-06T07:24:51.963718+0000","last_undegraded":"2024-03-06T07:24:51.963718+0000","last_fullsized":"2024-03-06T07:24:51.963718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:40.456147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954185+0000","last_change":"2024-03-06T07:24:51.954185+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954185+0000","last_undegraded":"2024-03-06T07:24:51.954185+0000","last_fullsized":"2024-03-06T07:24:51.954185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967273+0000","last_change":"2024-03-06T07:24:51.964997+0000","last_active":"2024-03-06T07:24:51.967273+0000","last_peered":"2024-03-06T07:24:51.967273+0000","last_clean":"2024-03-06T07:24:51.967273+0000","last_became_active":"2024-03-06T07:24:51.964708+0000","last_became_peered":"2024-03-06T07:24:51.964708+0000","last_unstale":"2024-03-06T07:24:51.967273+0000","last_undegraded":"2024-03-06T07:24:51.967273+0000","last_fullsized":"2024-03-06T07:24:51.967273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:33.109374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954642+0000","last_change":"2024-03-06T07:24:51.954642+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954642+0000","last_undegraded":"2024-03-06T07:24:51.954642+0000","last_fullsized":"2024-03-06T07:24:51.954642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963914+0000","last_change":"2024-03-06T07:24:51.962002+0000","last_active":"2024-03-06T07:24:51.963914+0000","last_peered":"2024-03-06T07:24:51.963914+0000","last_clean":"2024-03-06T07:24:51.963914+0000","last_became_active":"2024-03-06T07:24:51.961717+0000","last_became_peered":"2024-03-06T07:24:51.961717+0000","last_unstale":"2024-03-06T07:24:51.963914+0000","last_undegraded":"2024-03-06T07:24:51.963914+0000","last_fullsized":"2024-03-06T07:24:51.963914+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954942+0000","last_change":"2024-03-06T07:24:51.954942+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954942+0000","last_undegraded":"2024-03-06T07:24:51.954942+0000","last_fullsized":"2024-03-06T07:24:51.954942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961350+0000","last_change":"2024-03-06T07:24:51.957748+0000","last_active":"2024-03-06T07:24:51.961350+0000","last_peered":"2024-03-06T07:24:51.961350+0000","last_clean":"2024-03-06T07:24:51.961350+0000","last_became_active":"2024-03-06T07:24:51.957465+0000","last_became_peered":"2024-03-06T07:24:51.957465+0000","last_unstale":"2024-03-06T07:24:51.961350+0000","last_undegraded":"2024-03-06T07:24:51.961350+0000","last_fullsized":"2024-03-06T07:24:51.961350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:53.903184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949800+0000","last_change":"2024-03-06T07:24:51.949800+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949800+0000","last_undegraded":"2024-03-06T07:24:51.949800+0000","last_fullsized":"2024-03-06T07:24:51.949800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967114+0000","last_change":"2024-03-06T07:24:51.962228+0000","last_active":"2024-03-06T07:24:51.967114+0000","last_peered":"2024-03-06T07:24:51.967114+0000","last_clean":"2024-03-06T07:24:51.967114+0000","last_became_active":"2024-03-06T07:24:51.961977+0000","last_became_peered":"2024-03-06T07:24:51.961977+0000","last_unstale":"2024-03-06T07:24:51.967114+0000","last_undegraded":"2024-03-06T07:24:51.967114+0000","last_fullsized":"2024-03-06T07:24:51.967114+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:07.096047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955422+0000","last_change":"2024-03-06T07:24:51.955422+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955422+0000","last_undegraded":"2024-03-06T07:24:51.955422+0000","last_fullsized":"2024-03-06T07:24:51.955422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969507+0000","last_change":"2024-03-06T07:24:51.965861+0000","last_active":"2024-03-06T07:24:51.969507+0000","last_peered":"2024-03-06T07:24:51.969507+0000","last_clean":"2024-03-06T07:24:51.969507+0000","last_became_active":"2024-03-06T07:24:51.965498+0000","last_became_peered":"2024-03-06T07:24:51.965498+0000","last_unstale":"2024-03-06T07:24:51.969507+0000","last_undegraded":"2024-03-06T07:24:51.969507+0000","last_fullsized":"2024-03-06T07:24:51.969507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955625+0000","last_change":"2024-03-06T07:24:51.955625+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955625+0000","last_undegraded":"2024-03-06T07:24:51.955625+0000","last_fullsized":"2024-03-06T07:24:51.955625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.959867+0000","last_change":"2024-03-06T07:24:51.958271+0000","last_active":"2024-03-06T07:24:51.959867+0000","last_peered":"2024-03-06T07:24:51.959867+0000","last_clean":"2024-03-06T07:24:51.959867+0000","last_became_active":"2024-03-06T07:24:51.957988+0000","last_became_peered":"2024-03-06T07:24:51.957988+0000","last_unstale":"2024-03-06T07:24:51.959867+0000","last_undegraded":"2024-03-06T07:24:51.959867+0000","last_fullsized":"2024-03-06T07:24:51.959867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:30.871115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.955262+0000","last_change":"2024-03-06T07:24:51.955262+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.955262+0000","last_undegraded":"2024-03-06T07:24:51.955262+0000","last_fullsized":"2024-03-06T07:24:51.955262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961753+0000","last_change":"2024-03-06T07:24:51.958913+0000","last_active":"2024-03-06T07:24:51.961753+0000","last_peered":"2024-03-06T07:24:51.961753+0000","last_clean":"2024-03-06T07:24:51.961753+0000","last_became_active":"2024-03-06T07:24:51.958624+0000","last_became_peered":"2024-03-06T07:24:51.958624+0000","last_unstale":"2024-03-06T07:24:51.961753+0000","last_undegraded":"2024-03-06T07:24:51.961753+0000","last_fullsized":"2024-03-06T07:24:51.961753+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:40.477652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954553+0000","last_change":"2024-03-06T07:24:51.954553+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954553+0000","last_undegraded":"2024-03-06T07:24:51.954553+0000","last_fullsized":"2024-03-06T07:24:51.954553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.962777+0000","last_change":"2024-03-06T07:24:51.960455+0000","last_active":"2024-03-06T07:24:51.962777+0000","last_peered":"2024-03-06T07:24:51.962777+0000","last_clean":"2024-03-06T07:24:51.962777+0000","last_became_active":"2024-03-06T07:24:51.960189+0000","last_became_peered":"2024-03-06T07:24:51.960189+0000","last_unstale":"2024-03-06T07:24:51.962777+0000","last_undegraded":"2024-03-06T07:24:51.962777+0000","last_fullsized":"2024-03-06T07:24:51.962777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949626+0000","last_change":"2024-03-06T07:24:51.949626+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949626+0000","last_undegraded":"2024-03-06T07:24:51.949626+0000","last_fullsized":"2024-03-06T07:24:51.949626+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967289+0000","last_change":"2024-03-06T07:24:51.965187+0000","last_active":"2024-03-06T07:24:51.967289+0000","last_peered":"2024-03-06T07:24:51.967289+0000","last_clean":"2024-03-06T07:24:51.967289+0000","last_became_active":"2024-03-06T07:24:51.964934+0000","last_became_peered":"2024-03-06T07:24:51.964934+0000","last_unstale":"2024-03-06T07:24:51.967289+0000","last_undegraded":"2024-03-06T07:24:51.967289+0000","last_fullsized":"2024-03-06T07:24:51.967289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:35.381581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954508+0000","last_change":"2024-03-06T07:24:51.954508+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954508+0000","last_undegraded":"2024-03-06T07:24:51.954508+0000","last_fullsized":"2024-03-06T07:24:51.954508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966779+0000","last_change":"2024-03-06T07:24:51.961258+0000","last_active":"2024-03-06T07:24:51.966779+0000","last_peered":"2024-03-06T07:24:51.966779+0000","last_clean":"2024-03-06T07:24:51.966779+0000","last_became_active":"2024-03-06T07:24:51.961000+0000","last_became_peered":"2024-03-06T07:24:51.961000+0000","last_unstale":"2024-03-06T07:24:51.966779+0000","last_undegraded":"2024-03-06T07:24:51.966779+0000","last_fullsized":"2024-03-06T07:24:51.966779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:32.325142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954701+0000","last_change":"2024-03-06T07:24:51.954701+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954701+0000","last_undegraded":"2024-03-06T07:24:51.954701+0000","last_fullsized":"2024-03-06T07:24:51.954701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966456+0000","last_change":"2024-03-06T07:24:51.959310+0000","last_active":"2024-03-06T07:24:51.966456+0000","last_peered":"2024-03-06T07:24:51.966456+0000","last_clean":"2024-03-06T07:24:51.966456+0000","last_became_active":"2024-03-06T07:24:51.958975+0000","last_became_peered":"2024-03-06T07:24:51.958975+0000","last_unstale":"2024-03-06T07:24:51.966456+0000","last_undegraded":"2024-03-06T07:24:51.966456+0000","last_fullsized":"2024-03-06T07:24:51.966456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:22.828922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953114+0000","last_change":"2024-03-06T07:24:51.953114+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953114+0000","last_undegraded":"2024-03-06T07:24:51.953114+0000","last_fullsized":"2024-03-06T07:24:51.953114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.969566+0000","last_change":"2024-03-06T07:24:51.966524+0000","last_active":"2024-03-06T07:24:51.969566+0000","last_peered":"2024-03-06T07:24:51.969566+0000","last_clean":"2024-03-06T07:24:51.969566+0000","last_became_active":"2024-03-06T07:24:51.966147+0000","last_became_peered":"2024-03-06T07:24:51.966147+0000","last_unstale":"2024-03-06T07:24:51.969566+0000","last_undegraded":"2024-03-06T07:24:51.969566+0000","last_fullsized":"2024-03-06T07:24:51.969566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:55:28.864447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954975+0000","last_change":"2024-03-06T07:24:51.954975+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954975+0000","last_undegraded":"2024-03-06T07:24:51.954975+0000","last_fullsized":"2024-03-06T07:24:51.954975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967206+0000","last_change":"2024-03-06T07:24:51.965299+0000","last_active":"2024-03-06T07:24:51.967206+0000","last_peered":"2024-03-06T07:24:51.967206+0000","last_clean":"2024-03-06T07:24:51.967206+0000","last_became_active":"2024-03-06T07:24:51.964994+0000","last_became_peered":"2024-03-06T07:24:51.964994+0000","last_unstale":"2024-03-06T07:24:51.967206+0000","last_undegraded":"2024-03-06T07:24:51.967206+0000","last_fullsized":"2024-03-06T07:24:51.967206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:36.962579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953854+0000","last_change":"2024-03-06T07:24:51.953854+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953854+0000","last_undegraded":"2024-03-06T07:24:51.953854+0000","last_fullsized":"2024-03-06T07:24:51.953854+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972229+0000","last_change":"2024-03-06T07:24:51.967402+0000","last_active":"2024-03-06T07:24:51.972229+0000","last_peered":"2024-03-06T07:24:51.972229+0000","last_clean":"2024-03-06T07:24:51.972229+0000","last_became_active":"2024-03-06T07:24:51.967118+0000","last_became_peered":"2024-03-06T07:24:51.967118+0000","last_unstale":"2024-03-06T07:24:51.972229+0000","last_undegraded":"2024-03-06T07:24:51.972229+0000","last_fullsized":"2024-03-06T07:24:51.972229+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:15.830989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954657+0000","last_change":"2024-03-06T07:24:51.954657+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954657+0000","last_undegraded":"2024-03-06T07:24:51.954657+0000","last_fullsized":"2024-03-06T07:24:51.954657+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.972236+0000","last_change":"2024-03-06T07:24:51.967728+0000","last_active":"2024-03-06T07:24:51.972236+0000","last_peered":"2024-03-06T07:24:51.972236+0000","last_clean":"2024-03-06T07:24:51.972236+0000","last_became_active":"2024-03-06T07:24:51.967414+0000","last_became_peered":"2024-03-06T07:24:51.967414+0000","last_unstale":"2024-03-06T07:24:51.972236+0000","last_undegraded":"2024-03-06T07:24:51.972236+0000","last_fullsized":"2024-03-06T07:24:51.972236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949563+0000","last_change":"2024-03-06T07:24:51.949563+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949563+0000","last_undegraded":"2024-03-06T07:24:51.949563+0000","last_fullsized":"2024-03-06T07:24:51.949563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967254+0000","last_change":"2024-03-06T07:24:51.964639+0000","last_active":"2024-03-06T07:24:51.967254+0000","last_peered":"2024-03-06T07:24:51.967254+0000","last_clean":"2024-03-06T07:24:51.967254+0000","last_became_active":"2024-03-06T07:24:51.964394+0000","last_became_peered":"2024-03-06T07:24:51.964394+0000","last_unstale":"2024-03-06T07:24:51.967254+0000","last_undegraded":"2024-03-06T07:24:51.967254+0000","last_fullsized":"2024-03-06T07:24:51.967254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:04.726015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948968+0000","last_change":"2024-03-06T07:24:51.948968+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948968+0000","last_undegraded":"2024-03-06T07:24:51.948968+0000","last_fullsized":"2024-03-06T07:24:51.948968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961413+0000","last_change":"2024-03-06T07:24:51.957428+0000","last_active":"2024-03-06T07:24:51.961413+0000","last_peered":"2024-03-06T07:24:51.961413+0000","last_clean":"2024-03-06T07:24:51.961413+0000","last_became_active":"2024-03-06T07:24:51.957114+0000","last_became_peered":"2024-03-06T07:24:51.957114+0000","last_unstale":"2024-03-06T07:24:51.961413+0000","last_undegraded":"2024-03-06T07:24:51.961413+0000","last_fullsized":"2024-03-06T07:24:51.961413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:39.825819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949253+0000","last_change":"2024-03-06T07:24:51.949253+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949253+0000","last_undegraded":"2024-03-06T07:24:51.949253+0000","last_fullsized":"2024-03-06T07:24:51.949253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964318+0000","last_change":"2024-03-06T07:24:51.962417+0000","last_active":"2024-03-06T07:24:51.964318+0000","last_peered":"2024-03-06T07:24:51.964318+0000","last_clean":"2024-03-06T07:24:51.964318+0000","last_became_active":"2024-03-06T07:24:51.962166+0000","last_became_peered":"2024-03-06T07:24:51.962166+0000","last_unstale":"2024-03-06T07:24:51.964318+0000","last_undegraded":"2024-03-06T07:24:51.964318+0000","last_fullsized":"2024-03-06T07:24:51.964318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.253518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953963+0000","last_change":"2024-03-06T07:24:51.953963+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953963+0000","last_undegraded":"2024-03-06T07:24:51.953963+0000","last_fullsized":"2024-03-06T07:24:51.953963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967382+0000","last_change":"2024-03-06T07:24:51.965725+0000","last_active":"2024-03-06T07:24:51.967382+0000","last_peered":"2024-03-06T07:24:51.967382+0000","last_clean":"2024-03-06T07:24:51.967382+0000","last_became_active":"2024-03-06T07:24:51.965329+0000","last_became_peered":"2024-03-06T07:24:51.965329+0000","last_unstale":"2024-03-06T07:24:51.967382+0000","last_undegraded":"2024-03-06T07:24:51.967382+0000","last_fullsized":"2024-03-06T07:24:51.967382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:38.628974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.949805+0000","last_change":"2024-03-06T07:24:51.949805+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.949805+0000","last_undegraded":"2024-03-06T07:24:51.949805+0000","last_fullsized":"2024-03-06T07:24:51.949805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.966811+0000","last_change":"2024-03-06T07:24:51.961973+0000","last_active":"2024-03-06T07:24:51.966811+0000","last_peered":"2024-03-06T07:24:51.966811+0000","last_clean":"2024-03-06T07:24:51.966811+0000","last_became_active":"2024-03-06T07:24:51.961700+0000","last_became_peered":"2024-03-06T07:24:51.961700+0000","last_unstale":"2024-03-06T07:24:51.966811+0000","last_undegraded":"2024-03-06T07:24:51.966811+0000","last_fullsized":"2024-03-06T07:24:51.966811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:12.587214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952214+0000","last_change":"2024-03-06T07:24:51.952214+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952214+0000","last_undegraded":"2024-03-06T07:24:51.952214+0000","last_fullsized":"2024-03-06T07:24:51.952214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.963087+0000","last_change":"2024-03-06T07:24:51.959714+0000","last_active":"2024-03-06T07:24:51.963087+0000","last_peered":"2024-03-06T07:24:51.963087+0000","last_clean":"2024-03-06T07:24:51.963087+0000","last_became_active":"2024-03-06T07:24:51.959436+0000","last_became_peered":"2024-03-06T07:24:51.959436+0000","last_unstale":"2024-03-06T07:24:51.963087+0000","last_undegraded":"2024-03-06T07:24:51.963087+0000","last_fullsized":"2024-03-06T07:24:51.963087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:16:13.477872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.952016+0000","last_change":"2024-03-06T07:24:51.952016+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.952016+0000","last_undegraded":"2024-03-06T07:24:51.952016+0000","last_fullsized":"2024-03-06T07:24:51.952016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.968632+0000","last_change":"2024-03-06T07:24:51.968045+0000","last_active":"2024-03-06T07:24:51.968632+0000","last_peered":"2024-03-06T07:24:51.968632+0000","last_clean":"2024-03-06T07:24:51.968632+0000","last_became_active":"2024-03-06T07:24:51.967814+0000","last_became_peered":"2024-03-06T07:24:51.967814+0000","last_unstale":"2024-03-06T07:24:51.968632+0000","last_undegraded":"2024-03-06T07:24:51.968632+0000","last_fullsized":"2024-03-06T07:24:51.968632+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:37.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954421+0000","last_change":"2024-03-06T07:24:51.954421+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954421+0000","last_undegraded":"2024-03-06T07:24:51.954421+0000","last_fullsized":"2024-03-06T07:24:51.954421+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.961459+0000","last_change":"2024-03-06T07:24:51.958710+0000","last_active":"2024-03-06T07:24:51.961459+0000","last_peered":"2024-03-06T07:24:51.961459+0000","last_clean":"2024-03-06T07:24:51.961459+0000","last_became_active":"2024-03-06T07:24:51.958435+0000","last_became_peered":"2024-03-06T07:24:51.958435+0000","last_unstale":"2024-03-06T07:24:51.961459+0000","last_undegraded":"2024-03-06T07:24:51.961459+0000","last_fullsized":"2024-03-06T07:24:51.961459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:20:29.966173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.956317+0000","last_change":"2024-03-06T07:24:51.956317+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.956317+0000","last_undegraded":"2024-03-06T07:24:51.956317+0000","last_fullsized":"2024-03-06T07:24:51.956317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.964468+0000","last_change":"2024-03-06T07:24:51.962531+0000","last_active":"2024-03-06T07:24:51.964468+0000","last_peered":"2024-03-06T07:24:51.964468+0000","last_clean":"2024-03-06T07:24:51.964468+0000","last_became_active":"2024-03-06T07:24:51.962253+0000","last_became_peered":"2024-03-06T07:24:51.962253+0000","last_unstale":"2024-03-06T07:24:51.964468+0000","last_undegraded":"2024-03-06T07:24:51.964468+0000","last_fullsized":"2024-03-06T07:24:51.964468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:11.864382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.953329+0000","last_change":"2024-03-06T07:24:51.953329+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.953329+0000","last_undegraded":"2024-03-06T07:24:51.953329+0000","last_fullsized":"2024-03-06T07:24:51.953329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967111+0000","last_change":"2024-03-06T07:24:51.963025+0000","last_active":"2024-03-06T07:24:51.967111+0000","last_peered":"2024-03-06T07:24:51.967111+0000","last_clean":"2024-03-06T07:24:51.967111+0000","last_became_active":"2024-03-06T07:24:51.962710+0000","last_became_peered":"2024-03-06T07:24:51.962710+0000","last_unstale":"2024-03-06T07:24:51.967111+0000","last_undegraded":"2024-03-06T07:24:51.967111+0000","last_fullsized":"2024-03-06T07:24:51.967111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:21.637787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.954067+0000","last_change":"2024-03-06T07:24:51.954067+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.954067+0000","last_undegraded":"2024-03-06T07:24:51.954067+0000","last_fullsized":"2024-03-06T07:24:51.954067+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967246+0000","last_change":"2024-03-06T07:24:51.963634+0000","last_active":"2024-03-06T07:24:51.967246+0000","last_peered":"2024-03-06T07:24:51.967246+0000","last_clean":"2024-03-06T07:24:51.967246+0000","last_became_active":"2024-03-06T07:24:51.963368+0000","last_became_peered":"2024-03-06T07:24:51.963368+0000","last_unstale":"2024-03-06T07:24:51.967246+0000","last_undegraded":"2024-03-06T07:24:51.967246+0000","last_fullsized":"2024-03-06T07:24:51.967246+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:22.343338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":69,"state":"creating+peering","last_fresh":"2024-03-06T07:24:51.948955+0000","last_change":"2024-03-06T07:24:51.948955+0000","last_active":"2024-03-06T07:24:51.944226+0000","last_peered":"2024-03-06T07:24:51.944226+0000","last_clean":"2024-03-06T07:24:51.944226+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:24:51.948955+0000","last_undegraded":"2024-03-06T07:24:51.948955+0000","last_fullsized":"2024-03-06T07:24:51.948955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.967349+0000","last_change":"2024-03-06T07:24:51.964342+0000","last_active":"2024-03-06T07:24:51.967349+0000","last_peered":"2024-03-06T07:24:51.967349+0000","last_clean":"2024-03-06T07:24:51.967349+0000","last_became_active":"2024-03-06T07:24:51.964058+0000","last_became_peered":"2024-03-06T07:24:51.964058+0000","last_unstale":"2024-03-06T07:24:51.967349+0000","last_undegraded":"2024-03-06T07:24:51.967349+0000","last_fullsized":"2024-03-06T07:24:51.967349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:27.238164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":260,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-03-06T07:24:51.956184+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:51.956184+0000","last_peered":"2024-03-06T07:24:51.956184+0000","last_clean":"2024-03-06T07:24:51.956184+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:51.956184+0000","last_undegraded":"2024-03-06T07:24:51.956184+0000","last_fullsized":"2024-03-06T07:24:51.956184+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":344387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":934755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:24:58.213 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-03-06T07:24:58.211+0000 7fd68671d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:00.325 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:00.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.436+0000 7f8dd5dd7640 1 Processor -- start 2024-03-06T07:25:00.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.436+0000 7f8dd5dd7640 1 -- start start 2024-03-06T07:25:00.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dd5dd7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8dd01424f0 0x7f8dd00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dd5dd7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8dd00cd7a0 0x7f8dd015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dd5dd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd006d330 0x7f8dd006d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dd5dd7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00cd220 con 0x7f8dd01424f0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dd5dd7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dd00cc980 con 0x7f8dd00cd7a0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dd5dd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd00ccc60 con 0x7f8dd006d330 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dcf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8dd01424f0 0x7f8dd00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dcf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8dd01424f0 0x7f8dd00cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37768/0 (socket says 172.21.15.72:37768) 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dcf7fe640 1 -- 172.21.15.72:0/394015399 learned_addr learned my addr 172.21.15.72:0/394015399 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dceffd640 1 --2- 172.21.15.72:0/394015399 >> v2:172.21.15.146:3300/0 conn(0x7f8dd00cd7a0 0x7f8dd015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dcffff640 1 --2- 172.21.15.72:0/394015399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd006d330 0x7f8dd006d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dcf7fe640 1 -- 172.21.15.72:0/394015399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd006d330 msgr2=0x7f8dd006d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.437+0000 7f8dcf7fe640 1 --2- 172.21.15.72:0/394015399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd006d330 0x7f8dd006d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.438+0000 7f8dcf7fe640 1 -- 172.21.15.72:0/394015399 >> v2:172.21.15.146:3300/0 conn(0x7f8dd00cd7a0 msgr2=0x7f8dd015cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.438+0000 7f8dcf7fe640 1 --2- 172.21.15.72:0/394015399 >> v2:172.21.15.146:3300/0 conn(0x7f8dd00cd7a0 0x7f8dd015cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.438+0000 7f8dcf7fe640 1 -- 172.21.15.72:0/394015399 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00ccf40 con 0x7f8dd01424f0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.438+0000 7f8dcf7fe640 1 --2- 172.21.15.72:0/394015399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd00cc2f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8dd00b2a00 tx=0x7f8dc002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d5918ae737ef2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.438+0000 7f8dce7fc640 1 -- 172.21.15.72:0/394015399 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0035020 con 0x7f8dd01424f0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.438+0000 7f8dce7fc640 1 -- 172.21.15.72:0/394015399 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc0031870 con 0x7f8dd01424f0 2024-03-06T07:25:00.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dce7fc640 1 -- 172.21.15.72:0/394015399 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0031c40 con 0x7f8dd01424f0 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/394015399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 msgr2=0x7f8dd00cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/394015399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd00cc2f0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8dd00b2a00 tx=0x7f8dc002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/394015399 shutdown_connections 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/394015399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd006d330 0x7f8dd006d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/394015399 >> v2:172.21.15.146:3300/0 conn(0x7f8dd00cd7a0 0x7f8dd015cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/394015399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd00cc2f0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/394015399 >> 172.21.15.72:0/394015399 conn(0x7f8dd00cf8a0 msgr2=0x7f8dd00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/394015399 shutdown_connections 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.439+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/394015399 wait complete. 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.440+0000 7f8dd5dd7640 1 Processor -- start 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.440+0000 7f8dd5dd7640 1 -- start start 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dd5dd7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8dd006d330 0x7f8dd00cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dd5dd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd00cd7a0 0x7f8dd015df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dd5dd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd01e7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dd5dd7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0071ce0 con 0x7f8dd01424f0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dd5dd7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0071b60 con 0x7f8dd006d330 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dd5dd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dd0071e60 con 0x7f8dd00cd7a0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dcffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd01e7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dcffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd01e7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37776/0 (socket says 172.21.15.72:37776) 2024-03-06T07:25:00.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dceffd640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd00cd7a0 0x7f8dd015df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dcffff640 1 -- 172.21.15.72:0/1622626299 learned_addr learned my addr 172.21.15.72:0/1622626299 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.441+0000 7f8dcf7fe640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8dd006d330 0x7f8dd00cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dcffff640 1 -- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd00cd7a0 msgr2=0x7f8dd015df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dcffff640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd00cd7a0 0x7f8dd015df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dcffff640 1 -- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8dd006d330 msgr2=0x7f8dd00cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dcffff640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8dd006d330 0x7f8dd00cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dcffff640 1 -- 172.21.15.72:0/1622626299 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd00aa140 con 0x7f8dd01424f0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dceffd640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd00cd7a0 0x7f8dd015df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dcffff640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd01e7a50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800eed0 tx=0x7f8dc800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc8018020 con 0x7f8dd01424f0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dc8014400 con 0x7f8dd01424f0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dd00aa420 con 0x7f8dd01424f0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc80147f0 con 0x7f8dd01424f0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.442+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dd00aa9e0 con 0x7f8dd01424f0 2024-03-06T07:25:00.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.443+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd004f330 con 0x7f8dd01424f0 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.444+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8dc8005d10 con 0x7f8dd01424f0 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.444+0000 7f8db7fff640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.444+0000 7f8dcf7fe640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.444+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8dc808aab0 con 0x7f8dd01424f0 2024-03-06T07:25:00.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.444+0000 7f8dcf7fe640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0036000 tx=0x7f8dc003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.448+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8dc8022030 con 0x7f8dd01424f0 2024-03-06T07:25:00.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.616+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8dd015f330 con 0x7f8dbc06b4d0 2024-03-06T07:25:00.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.622+0000 7f8db7fff640 1 -- 172.21.15.72:0/1622626299 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f8dc00858c0 con 0x7f8dbc06b4d0 2024-03-06T07:25:00.627 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:00.628 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8dbc06b4d0 msgr2=0x7f8dbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8dc0036000 tx=0x7f8dc003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 msgr2=0x7f8dd01e7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd01e7a50 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f8dc800eed0 tx=0x7f8dc800b480 comp rx=0 tx=0).stop 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 shutdown_connections 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8dbc06b4d0 0x7f8dbc06d990 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8dd01424f0 0x7f8dd01e7a50 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8dd00cd7a0 0x7f8dd015df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 --2- 172.21.15.72:0/1622626299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8dd006d330 0x7f8dd00cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.625+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 >> 172.21.15.72:0/1622626299 conn(0x7f8dd00cf8a0 msgr2=0x7f8dd006c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.626+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 shutdown_connections 2024-03-06T07:25:00.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.626+0000 7f8dd5dd7640 1 -- 172.21.15.72:0/1622626299 wait complete. 2024-03-06T07:25:00.638 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-03-06T07:24:59.981547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257420,"kb_used_data":7976,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717300,"statfs":{"total":773094113280,"available":772830515200,"internally_reserved":0,"allocated":8167424,"data_stored":4095452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001636"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.594085+0000","last_change":"2024-03-06T07:24:52.985614+0000","last_active":"2024-03-06T07:24:54.594085+0000","last_peered":"2024-03-06T07:24:54.594085+0000","last_clean":"2024-03-06T07:24:54.594085+0000","last_became_active":"2024-03-06T07:24:52.985363+0000","last_became_peered":"2024-03-06T07:24:52.985363+0000","last_unstale":"2024-03-06T07:24:54.594085+0000","last_undegraded":"2024-03-06T07:24:54.594085+0000","last_fullsized":"2024-03-06T07:24:54.594085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:32.087262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906340+0000","last_change":"2024-03-06T07:24:52.981108+0000","last_active":"2024-03-06T07:24:54.906340+0000","last_peered":"2024-03-06T07:24:54.906340+0000","last_clean":"2024-03-06T07:24:54.906340+0000","last_became_active":"2024-03-06T07:24:52.980286+0000","last_became_peered":"2024-03-06T07:24:52.980286+0000","last_unstale":"2024-03-06T07:24:54.906340+0000","last_undegraded":"2024-03-06T07:24:54.906340+0000","last_fullsized":"2024-03-06T07:24:54.906340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.833974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961842+0000","last_change":"2024-03-06T07:24:52.976969+0000","last_active":"2024-03-06T07:24:53.961842+0000","last_peered":"2024-03-06T07:24:53.961842+0000","last_clean":"2024-03-06T07:24:53.961842+0000","last_became_active":"2024-03-06T07:24:52.976673+0000","last_became_peered":"2024-03-06T07:24:52.976673+0000","last_unstale":"2024-03-06T07:24:53.961842+0000","last_undegraded":"2024-03-06T07:24:53.961842+0000","last_fullsized":"2024-03-06T07:24:53.961842+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:10:15.014044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960954+0000","last_change":"2024-03-06T07:24:52.980732+0000","last_active":"2024-03-06T07:24:53.960954+0000","last_peered":"2024-03-06T07:24:53.960954+0000","last_clean":"2024-03-06T07:24:53.960954+0000","last_became_active":"2024-03-06T07:24:52.980398+0000","last_became_peered":"2024-03-06T07:24:52.980398+0000","last_unstale":"2024-03-06T07:24:53.960954+0000","last_undegraded":"2024-03-06T07:24:53.960954+0000","last_fullsized":"2024-03-06T07:24:53.960954+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:52.141958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905777+0000","last_change":"2024-03-06T07:24:52.974126+0000","last_active":"2024-03-06T07:24:54.905777+0000","last_peered":"2024-03-06T07:24:54.905777+0000","last_clean":"2024-03-06T07:24:54.905777+0000","last_became_active":"2024-03-06T07:24:52.973836+0000","last_became_peered":"2024-03-06T07:24:52.973836+0000","last_unstale":"2024-03-06T07:24:54.905777+0000","last_undegraded":"2024-03-06T07:24:54.905777+0000","last_fullsized":"2024-03-06T07:24:54.905777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:13:23.351492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905983+0000","last_change":"2024-03-06T07:24:52.982069+0000","last_active":"2024-03-06T07:24:54.905983+0000","last_peered":"2024-03-06T07:24:54.905983+0000","last_clean":"2024-03-06T07:24:54.905983+0000","last_became_active":"2024-03-06T07:24:52.980388+0000","last_became_peered":"2024-03-06T07:24:52.980388+0000","last_unstale":"2024-03-06T07:24:54.905983+0000","last_undegraded":"2024-03-06T07:24:54.905983+0000","last_fullsized":"2024-03-06T07:24:54.905983+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:21:44.433610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962297+0000","last_change":"2024-03-06T07:24:52.966400+0000","last_active":"2024-03-06T07:24:53.962297+0000","last_peered":"2024-03-06T07:24:53.962297+0000","last_clean":"2024-03-06T07:24:53.962297+0000","last_became_active":"2024-03-06T07:24:52.966103+0000","last_became_peered":"2024-03-06T07:24:52.966103+0000","last_unstale":"2024-03-06T07:24:53.962297+0000","last_undegraded":"2024-03-06T07:24:53.962297+0000","last_fullsized":"2024-03-06T07:24:53.962297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:29.397779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906085+0000","last_change":"2024-03-06T07:24:52.982739+0000","last_active":"2024-03-06T07:24:54.906085+0000","last_peered":"2024-03-06T07:24:54.906085+0000","last_clean":"2024-03-06T07:24:54.906085+0000","last_became_active":"2024-03-06T07:24:52.980531+0000","last_became_peered":"2024-03-06T07:24:52.980531+0000","last_unstale":"2024-03-06T07:24:54.906085+0000","last_undegraded":"2024-03-06T07:24:54.906085+0000","last_fullsized":"2024-03-06T07:24:54.906085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:53:12.585693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906081+0000","last_change":"2024-03-06T07:24:52.980054+0000","last_active":"2024-03-06T07:24:54.906081+0000","last_peered":"2024-03-06T07:24:54.906081+0000","last_clean":"2024-03-06T07:24:54.906081+0000","last_became_active":"2024-03-06T07:24:52.979795+0000","last_became_peered":"2024-03-06T07:24:52.979795+0000","last_unstale":"2024-03-06T07:24:54.906081+0000","last_undegraded":"2024-03-06T07:24:54.906081+0000","last_fullsized":"2024-03-06T07:24:54.906081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:36.389736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906931+0000","last_change":"2024-03-06T07:24:52.982560+0000","last_active":"2024-03-06T07:24:54.906931+0000","last_peered":"2024-03-06T07:24:54.906931+0000","last_clean":"2024-03-06T07:24:54.906931+0000","last_became_active":"2024-03-06T07:24:52.981581+0000","last_became_peered":"2024-03-06T07:24:52.981581+0000","last_unstale":"2024-03-06T07:24:54.906931+0000","last_undegraded":"2024-03-06T07:24:54.906931+0000","last_fullsized":"2024-03-06T07:24:54.906931+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:38.490267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593992+0000","last_change":"2024-03-06T07:24:52.979585+0000","last_active":"2024-03-06T07:24:54.593992+0000","last_peered":"2024-03-06T07:24:54.593992+0000","last_clean":"2024-03-06T07:24:54.593992+0000","last_became_active":"2024-03-06T07:24:52.979239+0000","last_became_peered":"2024-03-06T07:24:52.979239+0000","last_unstale":"2024-03-06T07:24:54.593992+0000","last_undegraded":"2024-03-06T07:24:54.593992+0000","last_fullsized":"2024-03-06T07:24:54.593992+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:43.326254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960938+0000","last_change":"2024-03-06T07:24:52.970675+0000","last_active":"2024-03-06T07:24:53.960938+0000","last_peered":"2024-03-06T07:24:53.960938+0000","last_clean":"2024-03-06T07:24:53.960938+0000","last_became_active":"2024-03-06T07:24:52.970216+0000","last_became_peered":"2024-03-06T07:24:52.970216+0000","last_unstale":"2024-03-06T07:24:53.960938+0000","last_undegraded":"2024-03-06T07:24:53.960938+0000","last_fullsized":"2024-03-06T07:24:53.960938+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:32.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978694+0000","last_change":"2024-03-06T07:24:52.983564+0000","last_active":"2024-03-06T07:24:53.978694+0000","last_peered":"2024-03-06T07:24:53.978694+0000","last_clean":"2024-03-06T07:24:53.978694+0000","last_became_active":"2024-03-06T07:24:52.983279+0000","last_became_peered":"2024-03-06T07:24:52.983279+0000","last_unstale":"2024-03-06T07:24:53.978694+0000","last_undegraded":"2024-03-06T07:24:53.978694+0000","last_fullsized":"2024-03-06T07:24:53.978694+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:32.076982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905847+0000","last_change":"2024-03-06T07:24:52.970590+0000","last_active":"2024-03-06T07:24:54.905847+0000","last_peered":"2024-03-06T07:24:54.905847+0000","last_clean":"2024-03-06T07:24:54.905847+0000","last_became_active":"2024-03-06T07:24:52.969167+0000","last_became_peered":"2024-03-06T07:24:52.969167+0000","last_unstale":"2024-03-06T07:24:54.905847+0000","last_undegraded":"2024-03-06T07:24:54.905847+0000","last_fullsized":"2024-03-06T07:24:54.905847+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:24:28.246319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978677+0000","last_change":"2024-03-06T07:24:52.966520+0000","last_active":"2024-03-06T07:24:53.978677+0000","last_peered":"2024-03-06T07:24:53.978677+0000","last_clean":"2024-03-06T07:24:53.978677+0000","last_became_active":"2024-03-06T07:24:52.966211+0000","last_became_peered":"2024-03-06T07:24:52.966211+0000","last_unstale":"2024-03-06T07:24:53.978677+0000","last_undegraded":"2024-03-06T07:24:53.978677+0000","last_fullsized":"2024-03-06T07:24:53.978677+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:49.339193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905687+0000","last_change":"2024-03-06T07:24:52.989079+0000","last_active":"2024-03-06T07:24:54.905687+0000","last_peered":"2024-03-06T07:24:54.905687+0000","last_clean":"2024-03-06T07:24:54.905687+0000","last_became_active":"2024-03-06T07:24:52.988792+0000","last_became_peered":"2024-03-06T07:24:52.988792+0000","last_unstale":"2024-03-06T07:24:54.905687+0000","last_undegraded":"2024-03-06T07:24:54.905687+0000","last_fullsized":"2024-03-06T07:24:54.905687+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:27:40.833974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906163+0000","last_change":"2024-03-06T07:24:52.979909+0000","last_active":"2024-03-06T07:24:54.906163+0000","last_peered":"2024-03-06T07:24:54.906163+0000","last_clean":"2024-03-06T07:24:54.906163+0000","last_became_active":"2024-03-06T07:24:52.979625+0000","last_became_peered":"2024-03-06T07:24:52.979625+0000","last_unstale":"2024-03-06T07:24:54.906163+0000","last_undegraded":"2024-03-06T07:24:54.906163+0000","last_fullsized":"2024-03-06T07:24:54.906163+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:39:41.485807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960952+0000","last_change":"2024-03-06T07:24:52.984813+0000","last_active":"2024-03-06T07:24:53.960952+0000","last_peered":"2024-03-06T07:24:53.960952+0000","last_clean":"2024-03-06T07:24:53.960952+0000","last_became_active":"2024-03-06T07:24:52.984437+0000","last_became_peered":"2024-03-06T07:24:52.984437+0000","last_unstale":"2024-03-06T07:24:53.960952+0000","last_undegraded":"2024-03-06T07:24:53.960952+0000","last_fullsized":"2024-03-06T07:24:53.960952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:34.730449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906275+0000","last_change":"2024-03-06T07:24:52.983725+0000","last_active":"2024-03-06T07:24:54.906275+0000","last_peered":"2024-03-06T07:24:54.906275+0000","last_clean":"2024-03-06T07:24:54.906275+0000","last_became_active":"2024-03-06T07:24:52.983035+0000","last_became_peered":"2024-03-06T07:24:52.983035+0000","last_unstale":"2024-03-06T07:24:54.906275+0000","last_undegraded":"2024-03-06T07:24:54.906275+0000","last_fullsized":"2024-03-06T07:24:54.906275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:52:48.887786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978865+0000","last_change":"2024-03-06T07:24:52.971375+0000","last_active":"2024-03-06T07:24:53.978865+0000","last_peered":"2024-03-06T07:24:53.978865+0000","last_clean":"2024-03-06T07:24:53.978865+0000","last_became_active":"2024-03-06T07:24:52.971043+0000","last_became_peered":"2024-03-06T07:24:52.971043+0000","last_unstale":"2024-03-06T07:24:53.978865+0000","last_undegraded":"2024-03-06T07:24:53.978865+0000","last_fullsized":"2024-03-06T07:24:53.978865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:43.326254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906647+0000","last_change":"2024-03-06T07:24:52.971043+0000","last_active":"2024-03-06T07:24:54.906647+0000","last_peered":"2024-03-06T07:24:54.906647+0000","last_clean":"2024-03-06T07:24:54.906647+0000","last_became_active":"2024-03-06T07:24:52.970761+0000","last_became_peered":"2024-03-06T07:24:52.970761+0000","last_unstale":"2024-03-06T07:24:54.906647+0000","last_undegraded":"2024-03-06T07:24:54.906647+0000","last_fullsized":"2024-03-06T07:24:54.906647+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:30.375555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905839+0000","last_change":"2024-03-06T07:24:52.984292+0000","last_active":"2024-03-06T07:24:54.905839+0000","last_peered":"2024-03-06T07:24:54.905839+0000","last_clean":"2024-03-06T07:24:54.905839+0000","last_became_active":"2024-03-06T07:24:52.982586+0000","last_became_peered":"2024-03-06T07:24:52.982586+0000","last_unstale":"2024-03-06T07:24:54.905839+0000","last_undegraded":"2024-03-06T07:24:54.905839+0000","last_fullsized":"2024-03-06T07:24:54.905839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:33.333297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593823+0000","last_change":"2024-03-06T07:24:52.984932+0000","last_active":"2024-03-06T07:24:54.593823+0000","last_peered":"2024-03-06T07:24:54.593823+0000","last_clean":"2024-03-06T07:24:54.593823+0000","last_became_active":"2024-03-06T07:24:52.984601+0000","last_became_peered":"2024-03-06T07:24:52.984601+0000","last_unstale":"2024-03-06T07:24:54.593823+0000","last_undegraded":"2024-03-06T07:24:54.593823+0000","last_fullsized":"2024-03-06T07:24:54.593823+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:32.076982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593769+0000","last_change":"2024-03-06T07:24:52.985437+0000","last_active":"2024-03-06T07:24:54.593769+0000","last_peered":"2024-03-06T07:24:54.593769+0000","last_clean":"2024-03-06T07:24:54.593769+0000","last_became_active":"2024-03-06T07:24:52.985154+0000","last_became_peered":"2024-03-06T07:24:52.985154+0000","last_unstale":"2024-03-06T07:24:54.593769+0000","last_undegraded":"2024-03-06T07:24:54.593769+0000","last_fullsized":"2024-03-06T07:24:54.593769+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:41.464301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.907177+0000","last_change":"2024-03-06T07:24:52.973222+0000","last_active":"2024-03-06T07:24:54.907177+0000","last_peered":"2024-03-06T07:24:54.907177+0000","last_clean":"2024-03-06T07:24:54.907177+0000","last_became_active":"2024-03-06T07:24:52.972893+0000","last_became_peered":"2024-03-06T07:24:52.972893+0000","last_unstale":"2024-03-06T07:24:54.907177+0000","last_undegraded":"2024-03-06T07:24:54.907177+0000","last_fullsized":"2024-03-06T07:24:54.907177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:48.683813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962563+0000","last_change":"2024-03-06T07:24:52.984723+0000","last_active":"2024-03-06T07:24:53.962563+0000","last_peered":"2024-03-06T07:24:53.962563+0000","last_clean":"2024-03-06T07:24:53.962563+0000","last_became_active":"2024-03-06T07:24:52.984471+0000","last_became_peered":"2024-03-06T07:24:52.984471+0000","last_unstale":"2024-03-06T07:24:53.962563+0000","last_undegraded":"2024-03-06T07:24:53.962563+0000","last_fullsized":"2024-03-06T07:24:53.962563+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:49.339193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961876+0000","last_change":"2024-03-06T07:24:52.978857+0000","last_active":"2024-03-06T07:24:53.961876+0000","last_peered":"2024-03-06T07:24:53.961876+0000","last_clean":"2024-03-06T07:24:53.961876+0000","last_became_active":"2024-03-06T07:24:52.978591+0000","last_became_peered":"2024-03-06T07:24:52.978591+0000","last_unstale":"2024-03-06T07:24:53.961876+0000","last_undegraded":"2024-03-06T07:24:53.961876+0000","last_fullsized":"2024-03-06T07:24:53.961876+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:22:47.864881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906015+0000","last_change":"2024-03-06T07:24:52.977156+0000","last_active":"2024-03-06T07:24:54.906015+0000","last_peered":"2024-03-06T07:24:54.906015+0000","last_clean":"2024-03-06T07:24:54.906015+0000","last_became_active":"2024-03-06T07:24:52.976817+0000","last_became_peered":"2024-03-06T07:24:52.976817+0000","last_unstale":"2024-03-06T07:24:54.906015+0000","last_undegraded":"2024-03-06T07:24:54.906015+0000","last_fullsized":"2024-03-06T07:24:54.906015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:13.595369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961800+0000","last_change":"2024-03-06T07:24:52.982005+0000","last_active":"2024-03-06T07:24:53.961800+0000","last_peered":"2024-03-06T07:24:53.961800+0000","last_clean":"2024-03-06T07:24:53.961800+0000","last_became_active":"2024-03-06T07:24:52.981754+0000","last_became_peered":"2024-03-06T07:24:52.981754+0000","last_unstale":"2024-03-06T07:24:53.961800+0000","last_undegraded":"2024-03-06T07:24:53.961800+0000","last_fullsized":"2024-03-06T07:24:53.961800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:26.213758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962422+0000","last_change":"2024-03-06T07:24:52.980875+0000","last_active":"2024-03-06T07:24:53.962422+0000","last_peered":"2024-03-06T07:24:53.962422+0000","last_clean":"2024-03-06T07:24:53.962422+0000","last_became_active":"2024-03-06T07:24:52.980592+0000","last_became_peered":"2024-03-06T07:24:52.980592+0000","last_unstale":"2024-03-06T07:24:53.962422+0000","last_undegraded":"2024-03-06T07:24:53.962422+0000","last_fullsized":"2024-03-06T07:24:53.962422+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:09.181538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962523+0000","last_change":"2024-03-06T07:24:52.977195+0000","last_active":"2024-03-06T07:24:53.962523+0000","last_peered":"2024-03-06T07:24:53.962523+0000","last_clean":"2024-03-06T07:24:53.962523+0000","last_became_active":"2024-03-06T07:24:52.976901+0000","last_became_peered":"2024-03-06T07:24:52.976901+0000","last_unstale":"2024-03-06T07:24:53.962523+0000","last_undegraded":"2024-03-06T07:24:53.962523+0000","last_fullsized":"2024-03-06T07:24:53.962523+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:31.879270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906177+0000","last_change":"2024-03-06T07:24:52.983374+0000","last_active":"2024-03-06T07:24:54.906177+0000","last_peered":"2024-03-06T07:24:54.906177+0000","last_clean":"2024-03-06T07:24:54.906177+0000","last_became_active":"2024-03-06T07:24:52.982398+0000","last_became_peered":"2024-03-06T07:24:52.982398+0000","last_unstale":"2024-03-06T07:24:54.906177+0000","last_undegraded":"2024-03-06T07:24:54.906177+0000","last_fullsized":"2024-03-06T07:24:54.906177+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:34:58.660651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.594136+0000","last_change":"2024-03-06T07:24:52.988255+0000","last_active":"2024-03-06T07:24:54.594136+0000","last_peered":"2024-03-06T07:24:54.594136+0000","last_clean":"2024-03-06T07:24:54.594136+0000","last_became_active":"2024-03-06T07:24:52.987982+0000","last_became_peered":"2024-03-06T07:24:52.987982+0000","last_unstale":"2024-03-06T07:24:54.594136+0000","last_undegraded":"2024-03-06T07:24:54.594136+0000","last_fullsized":"2024-03-06T07:24:54.594136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:52.255518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"70'4","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.594146+0000","last_change":"2024-03-06T07:24:51.962181+0000","last_active":"2024-03-06T07:24:54.594146+0000","last_peered":"2024-03-06T07:24:54.594146+0000","last_clean":"2024-03-06T07:24:54.594146+0000","last_became_active":"2024-03-06T07:24:51.961932+0000","last_became_peered":"2024-03-06T07:24:51.961932+0000","last_unstale":"2024-03-06T07:24:54.594146+0000","last_undegraded":"2024-03-06T07:24:54.594146+0000","last_fullsized":"2024-03-06T07:24:54.594146+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.253518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906158+0000","last_change":"2024-03-06T07:24:52.976977+0000","last_active":"2024-03-06T07:24:54.906158+0000","last_peered":"2024-03-06T07:24:54.906158+0000","last_clean":"2024-03-06T07:24:54.906158+0000","last_became_active":"2024-03-06T07:24:52.976707+0000","last_became_peered":"2024-03-06T07:24:52.976707+0000","last_unstale":"2024-03-06T07:24:54.906158+0000","last_undegraded":"2024-03-06T07:24:54.906158+0000","last_fullsized":"2024-03-06T07:24:54.906158+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:34:39.637129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978674+0000","last_change":"2024-03-06T07:24:51.962165+0000","last_active":"2024-03-06T07:24:53.978674+0000","last_peered":"2024-03-06T07:24:53.978674+0000","last_clean":"2024-03-06T07:24:53.978674+0000","last_became_active":"2024-03-06T07:24:51.961847+0000","last_became_peered":"2024-03-06T07:24:51.961847+0000","last_unstale":"2024-03-06T07:24:53.978674+0000","last_undegraded":"2024-03-06T07:24:53.978674+0000","last_fullsized":"2024-03-06T07:24:53.978674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:50.254064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906516+0000","last_change":"2024-03-06T07:24:52.977490+0000","last_active":"2024-03-06T07:24:54.906516+0000","last_peered":"2024-03-06T07:24:54.906516+0000","last_clean":"2024-03-06T07:24:54.906516+0000","last_became_active":"2024-03-06T07:24:52.977234+0000","last_became_peered":"2024-03-06T07:24:52.977234+0000","last_unstale":"2024-03-06T07:24:54.906516+0000","last_undegraded":"2024-03-06T07:24:54.906516+0000","last_fullsized":"2024-03-06T07:24:54.906516+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:33.333297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905889+0000","last_change":"2024-03-06T07:24:51.965487+0000","last_active":"2024-03-06T07:24:54.905889+0000","last_peered":"2024-03-06T07:24:54.905889+0000","last_clean":"2024-03-06T07:24:54.905889+0000","last_became_active":"2024-03-06T07:24:51.965210+0000","last_became_peered":"2024-03-06T07:24:51.965210+0000","last_unstale":"2024-03-06T07:24:54.905889+0000","last_undegraded":"2024-03-06T07:24:54.905889+0000","last_fullsized":"2024-03-06T07:24:54.905889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:31.093746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593995+0000","last_change":"2024-03-06T07:24:52.982320+0000","last_active":"2024-03-06T07:24:54.593995+0000","last_peered":"2024-03-06T07:24:54.593995+0000","last_clean":"2024-03-06T07:24:54.593995+0000","last_became_active":"2024-03-06T07:24:52.982017+0000","last_became_peered":"2024-03-06T07:24:52.982017+0000","last_unstale":"2024-03-06T07:24:54.593995+0000","last_undegraded":"2024-03-06T07:24:54.593995+0000","last_fullsized":"2024-03-06T07:24:54.593995+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:00:26.918506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906593+0000","last_change":"2024-03-06T07:24:51.963677+0000","last_active":"2024-03-06T07:24:54.906593+0000","last_peered":"2024-03-06T07:24:54.906593+0000","last_clean":"2024-03-06T07:24:54.906593+0000","last_became_active":"2024-03-06T07:24:51.963383+0000","last_became_peered":"2024-03-06T07:24:51.963383+0000","last_unstale":"2024-03-06T07:24:54.906593+0000","last_undegraded":"2024-03-06T07:24:54.906593+0000","last_fullsized":"2024-03-06T07:24:54.906593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:36.962579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962585+0000","last_change":"2024-03-06T07:24:52.976965+0000","last_active":"2024-03-06T07:24:53.962585+0000","last_peered":"2024-03-06T07:24:53.962585+0000","last_clean":"2024-03-06T07:24:53.962585+0000","last_became_active":"2024-03-06T07:24:52.976684+0000","last_became_peered":"2024-03-06T07:24:52.976684+0000","last_unstale":"2024-03-06T07:24:53.962585+0000","last_undegraded":"2024-03-06T07:24:53.962585+0000","last_fullsized":"2024-03-06T07:24:53.962585+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:32.076982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961893+0000","last_change":"2024-03-06T07:24:51.962180+0000","last_active":"2024-03-06T07:24:53.961893+0000","last_peered":"2024-03-06T07:24:53.961893+0000","last_clean":"2024-03-06T07:24:53.961893+0000","last_became_active":"2024-03-06T07:24:51.961911+0000","last_became_peered":"2024-03-06T07:24:51.961911+0000","last_unstale":"2024-03-06T07:24:53.961893+0000","last_undegraded":"2024-03-06T07:24:53.961893+0000","last_fullsized":"2024-03-06T07:24:53.961893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:55:31.068827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906677+0000","last_change":"2024-03-06T07:24:52.975097+0000","last_active":"2024-03-06T07:24:54.906677+0000","last_peered":"2024-03-06T07:24:54.906677+0000","last_clean":"2024-03-06T07:24:54.906677+0000","last_became_active":"2024-03-06T07:24:52.974825+0000","last_became_peered":"2024-03-06T07:24:52.974825+0000","last_unstale":"2024-03-06T07:24:54.906677+0000","last_undegraded":"2024-03-06T07:24:54.906677+0000","last_fullsized":"2024-03-06T07:24:54.906677+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:11.261672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906071+0000","last_change":"2024-03-06T07:24:51.964997+0000","last_active":"2024-03-06T07:24:54.906071+0000","last_peered":"2024-03-06T07:24:54.906071+0000","last_clean":"2024-03-06T07:24:54.906071+0000","last_became_active":"2024-03-06T07:24:51.964708+0000","last_became_peered":"2024-03-06T07:24:51.964708+0000","last_unstale":"2024-03-06T07:24:54.906071+0000","last_undegraded":"2024-03-06T07:24:54.906071+0000","last_fullsized":"2024-03-06T07:24:54.906071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:12:43.279875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905580+0000","last_change":"2024-03-06T07:24:52.970184+0000","last_active":"2024-03-06T07:24:54.905580+0000","last_peered":"2024-03-06T07:24:54.905580+0000","last_clean":"2024-03-06T07:24:54.905580+0000","last_became_active":"2024-03-06T07:24:52.969878+0000","last_became_peered":"2024-03-06T07:24:52.969878+0000","last_unstale":"2024-03-06T07:24:54.905580+0000","last_undegraded":"2024-03-06T07:24:54.905580+0000","last_fullsized":"2024-03-06T07:24:54.905580+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:16:49.592086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960594+0000","last_change":"2024-03-06T07:24:51.962002+0000","last_active":"2024-03-06T07:24:53.960594+0000","last_peered":"2024-03-06T07:24:53.960594+0000","last_clean":"2024-03-06T07:24:53.960594+0000","last_became_active":"2024-03-06T07:24:51.961717+0000","last_became_peered":"2024-03-06T07:24:51.961717+0000","last_unstale":"2024-03-06T07:24:53.960594+0000","last_undegraded":"2024-03-06T07:24:53.960594+0000","last_fullsized":"2024-03-06T07:24:53.960594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:28.358211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978602+0000","last_change":"2024-03-06T07:24:52.971534+0000","last_active":"2024-03-06T07:24:53.978602+0000","last_peered":"2024-03-06T07:24:53.978602+0000","last_clean":"2024-03-06T07:24:53.978602+0000","last_became_active":"2024-03-06T07:24:52.971254+0000","last_became_peered":"2024-03-06T07:24:52.971254+0000","last_unstale":"2024-03-06T07:24:53.978602+0000","last_undegraded":"2024-03-06T07:24:53.978602+0000","last_fullsized":"2024-03-06T07:24:53.978602+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:41.464301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905583+0000","last_change":"2024-03-06T07:24:51.957748+0000","last_active":"2024-03-06T07:24:54.905583+0000","last_peered":"2024-03-06T07:24:54.905583+0000","last_clean":"2024-03-06T07:24:54.905583+0000","last_became_active":"2024-03-06T07:24:51.957465+0000","last_became_peered":"2024-03-06T07:24:51.957465+0000","last_unstale":"2024-03-06T07:24:54.905583+0000","last_undegraded":"2024-03-06T07:24:54.905583+0000","last_fullsized":"2024-03-06T07:24:54.905583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:44:07.132105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962745+0000","last_change":"2024-03-06T07:24:52.976256+0000","last_active":"2024-03-06T07:24:53.962745+0000","last_peered":"2024-03-06T07:24:53.962745+0000","last_clean":"2024-03-06T07:24:53.962745+0000","last_became_active":"2024-03-06T07:24:52.975970+0000","last_became_peered":"2024-03-06T07:24:52.975970+0000","last_unstale":"2024-03-06T07:24:53.962745+0000","last_undegraded":"2024-03-06T07:24:53.962745+0000","last_fullsized":"2024-03-06T07:24:53.962745+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:03:43.326254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.907262+0000","last_change":"2024-03-06T07:24:51.962228+0000","last_active":"2024-03-06T07:24:54.907262+0000","last_peered":"2024-03-06T07:24:54.907262+0000","last_clean":"2024-03-06T07:24:54.907262+0000","last_became_active":"2024-03-06T07:24:51.961977+0000","last_became_peered":"2024-03-06T07:24:51.961977+0000","last_unstale":"2024-03-06T07:24:54.907262+0000","last_undegraded":"2024-03-06T07:24:54.907262+0000","last_fullsized":"2024-03-06T07:24:54.907262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:44.923111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593998+0000","last_change":"2024-03-06T07:24:52.993193+0000","last_active":"2024-03-06T07:24:54.593998+0000","last_peered":"2024-03-06T07:24:54.593998+0000","last_clean":"2024-03-06T07:24:54.593998+0000","last_became_active":"2024-03-06T07:24:52.992873+0000","last_became_peered":"2024-03-06T07:24:52.992873+0000","last_unstale":"2024-03-06T07:24:54.593998+0000","last_undegraded":"2024-03-06T07:24:54.593998+0000","last_fullsized":"2024-03-06T07:24:54.593998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:21:21.885385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906494+0000","last_change":"2024-03-06T07:24:51.965861+0000","last_active":"2024-03-06T07:24:54.906494+0000","last_peered":"2024-03-06T07:24:54.906494+0000","last_clean":"2024-03-06T07:24:54.906494+0000","last_became_active":"2024-03-06T07:24:51.965498+0000","last_became_peered":"2024-03-06T07:24:51.965498+0000","last_unstale":"2024-03-06T07:24:54.906494+0000","last_undegraded":"2024-03-06T07:24:54.906494+0000","last_fullsized":"2024-03-06T07:24:54.906494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:21.637787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906575+0000","last_change":"2024-03-06T07:24:52.979937+0000","last_active":"2024-03-06T07:24:54.906575+0000","last_peered":"2024-03-06T07:24:54.906575+0000","last_clean":"2024-03-06T07:24:54.906575+0000","last_became_active":"2024-03-06T07:24:52.979664+0000","last_became_peered":"2024-03-06T07:24:52.979664+0000","last_unstale":"2024-03-06T07:24:54.906575+0000","last_undegraded":"2024-03-06T07:24:54.906575+0000","last_fullsized":"2024-03-06T07:24:54.906575+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:13.595369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"70'3","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961928+0000","last_change":"2024-03-06T07:24:51.958271+0000","last_active":"2024-03-06T07:24:53.961928+0000","last_peered":"2024-03-06T07:24:53.961928+0000","last_clean":"2024-03-06T07:24:53.961928+0000","last_became_active":"2024-03-06T07:24:51.957988+0000","last_became_peered":"2024-03-06T07:24:51.957988+0000","last_unstale":"2024-03-06T07:24:53.961928+0000","last_undegraded":"2024-03-06T07:24:53.961928+0000","last_fullsized":"2024-03-06T07:24:53.961928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:48.331038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905819+0000","last_change":"2024-03-06T07:24:52.977102+0000","last_active":"2024-03-06T07:24:54.905819+0000","last_peered":"2024-03-06T07:24:54.905819+0000","last_clean":"2024-03-06T07:24:54.905819+0000","last_became_active":"2024-03-06T07:24:52.976765+0000","last_became_peered":"2024-03-06T07:24:52.976765+0000","last_unstale":"2024-03-06T07:24:54.905819+0000","last_undegraded":"2024-03-06T07:24:54.905819+0000","last_fullsized":"2024-03-06T07:24:54.905819+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:06.171287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"70'6","reported_seq":20,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961032+0000","last_change":"2024-03-06T07:24:51.958913+0000","last_active":"2024-03-06T07:24:53.961032+0000","last_peered":"2024-03-06T07:24:53.961032+0000","last_clean":"2024-03-06T07:24:53.961032+0000","last_became_active":"2024-03-06T07:24:51.958624+0000","last_became_peered":"2024-03-06T07:24:51.958624+0000","last_unstale":"2024-03-06T07:24:53.961032+0000","last_undegraded":"2024-03-06T07:24:53.961032+0000","last_fullsized":"2024-03-06T07:24:53.961032+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:50.900012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593800+0000","last_change":"2024-03-06T07:24:52.978411+0000","last_active":"2024-03-06T07:24:54.593800+0000","last_peered":"2024-03-06T07:24:54.593800+0000","last_clean":"2024-03-06T07:24:54.593800+0000","last_became_active":"2024-03-06T07:24:52.978123+0000","last_became_peered":"2024-03-06T07:24:52.978123+0000","last_unstale":"2024-03-06T07:24:54.593800+0000","last_undegraded":"2024-03-06T07:24:54.593800+0000","last_fullsized":"2024-03-06T07:24:54.593800+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:49.339193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960711+0000","last_change":"2024-03-06T07:24:51.960455+0000","last_active":"2024-03-06T07:24:53.960711+0000","last_peered":"2024-03-06T07:24:53.960711+0000","last_clean":"2024-03-06T07:24:53.960711+0000","last_became_active":"2024-03-06T07:24:51.960189+0000","last_became_peered":"2024-03-06T07:24:51.960189+0000","last_unstale":"2024-03-06T07:24:53.960711+0000","last_undegraded":"2024-03-06T07:24:53.960711+0000","last_fullsized":"2024-03-06T07:24:53.960711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:18.798102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962517+0000","last_change":"2024-03-06T07:24:52.969998+0000","last_active":"2024-03-06T07:24:53.962517+0000","last_peered":"2024-03-06T07:24:53.962517+0000","last_clean":"2024-03-06T07:24:53.962517+0000","last_became_active":"2024-03-06T07:24:52.969275+0000","last_became_peered":"2024-03-06T07:24:52.969275+0000","last_unstale":"2024-03-06T07:24:53.962517+0000","last_undegraded":"2024-03-06T07:24:53.962517+0000","last_fullsized":"2024-03-06T07:24:53.962517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:53:41.464301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906318+0000","last_change":"2024-03-06T07:24:51.965187+0000","last_active":"2024-03-06T07:24:54.906318+0000","last_peered":"2024-03-06T07:24:54.906318+0000","last_clean":"2024-03-06T07:24:54.906318+0000","last_became_active":"2024-03-06T07:24:51.964934+0000","last_became_peered":"2024-03-06T07:24:51.964934+0000","last_unstale":"2024-03-06T07:24:54.906318+0000","last_undegraded":"2024-03-06T07:24:54.906318+0000","last_fullsized":"2024-03-06T07:24:54.906318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:10:22.828922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978572+0000","last_change":"2024-03-06T07:24:52.983621+0000","last_active":"2024-03-06T07:24:53.978572+0000","last_peered":"2024-03-06T07:24:53.978572+0000","last_clean":"2024-03-06T07:24:53.978572+0000","last_became_active":"2024-03-06T07:24:52.983353+0000","last_became_peered":"2024-03-06T07:24:52.983353+0000","last_unstale":"2024-03-06T07:24:53.978572+0000","last_undegraded":"2024-03-06T07:24:53.978572+0000","last_fullsized":"2024-03-06T07:24:53.978572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:19:31.879270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962298+0000","last_change":"2024-03-06T07:24:51.961258+0000","last_active":"2024-03-06T07:24:53.962298+0000","last_peered":"2024-03-06T07:24:53.962298+0000","last_clean":"2024-03-06T07:24:53.962298+0000","last_became_active":"2024-03-06T07:24:51.961000+0000","last_became_peered":"2024-03-06T07:24:51.961000+0000","last_unstale":"2024-03-06T07:24:53.962298+0000","last_undegraded":"2024-03-06T07:24:53.962298+0000","last_fullsized":"2024-03-06T07:24:53.962298+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905662+0000","last_change":"2024-03-06T07:24:52.976206+0000","last_active":"2024-03-06T07:24:54.905662+0000","last_peered":"2024-03-06T07:24:54.905662+0000","last_clean":"2024-03-06T07:24:54.905662+0000","last_became_active":"2024-03-06T07:24:52.975937+0000","last_became_peered":"2024-03-06T07:24:52.975937+0000","last_unstale":"2024-03-06T07:24:54.905662+0000","last_undegraded":"2024-03-06T07:24:54.905662+0000","last_fullsized":"2024-03-06T07:24:54.905662+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:55:14.950853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"70'8","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905676+0000","last_change":"2024-03-06T07:24:51.959310+0000","last_active":"2024-03-06T07:24:54.905676+0000","last_peered":"2024-03-06T07:24:54.905676+0000","last_clean":"2024-03-06T07:24:54.905676+0000","last_became_active":"2024-03-06T07:24:51.958975+0000","last_became_peered":"2024-03-06T07:24:51.958975+0000","last_unstale":"2024-03-06T07:24:54.905676+0000","last_undegraded":"2024-03-06T07:24:54.905676+0000","last_fullsized":"2024-03-06T07:24:54.905676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:00:51.133803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.907022+0000","last_change":"2024-03-06T07:24:52.967591+0000","last_active":"2024-03-06T07:24:54.907022+0000","last_peered":"2024-03-06T07:24:54.907022+0000","last_clean":"2024-03-06T07:24:54.907022+0000","last_became_active":"2024-03-06T07:24:52.967156+0000","last_became_peered":"2024-03-06T07:24:52.967156+0000","last_unstale":"2024-03-06T07:24:54.907022+0000","last_undegraded":"2024-03-06T07:24:54.907022+0000","last_fullsized":"2024-03-06T07:24:54.907022+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:51.303846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906206+0000","last_change":"2024-03-06T07:24:51.966524+0000","last_active":"2024-03-06T07:24:54.906206+0000","last_peered":"2024-03-06T07:24:54.906206+0000","last_clean":"2024-03-06T07:24:54.906206+0000","last_became_active":"2024-03-06T07:24:51.966147+0000","last_became_peered":"2024-03-06T07:24:51.966147+0000","last_unstale":"2024-03-06T07:24:54.906206+0000","last_undegraded":"2024-03-06T07:24:54.906206+0000","last_fullsized":"2024-03-06T07:24:54.906206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:28.358211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.594135+0000","last_change":"2024-03-06T07:24:52.983822+0000","last_active":"2024-03-06T07:24:54.594135+0000","last_peered":"2024-03-06T07:24:54.594135+0000","last_clean":"2024-03-06T07:24:54.594135+0000","last_became_active":"2024-03-06T07:24:52.983550+0000","last_became_peered":"2024-03-06T07:24:52.983550+0000","last_unstale":"2024-03-06T07:24:54.594135+0000","last_undegraded":"2024-03-06T07:24:54.594135+0000","last_fullsized":"2024-03-06T07:24:54.594135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:49:42.526852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.907022+0000","last_change":"2024-03-06T07:24:51.965299+0000","last_active":"2024-03-06T07:24:54.907022+0000","last_peered":"2024-03-06T07:24:54.907022+0000","last_clean":"2024-03-06T07:24:54.907022+0000","last_became_active":"2024-03-06T07:24:51.964994+0000","last_became_peered":"2024-03-06T07:24:51.964994+0000","last_unstale":"2024-03-06T07:24:54.907022+0000","last_undegraded":"2024-03-06T07:24:54.907022+0000","last_fullsized":"2024-03-06T07:24:54.907022+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:29:45.777595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961026+0000","last_change":"2024-03-06T07:24:52.970473+0000","last_active":"2024-03-06T07:24:53.961026+0000","last_peered":"2024-03-06T07:24:53.961026+0000","last_clean":"2024-03-06T07:24:53.961026+0000","last_became_active":"2024-03-06T07:24:52.970033+0000","last_became_peered":"2024-03-06T07:24:52.970033+0000","last_unstale":"2024-03-06T07:24:53.961026+0000","last_undegraded":"2024-03-06T07:24:53.961026+0000","last_fullsized":"2024-03-06T07:24:53.961026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:59:45.931266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"70'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906357+0000","last_change":"2024-03-06T07:24:51.967402+0000","last_active":"2024-03-06T07:24:54.906357+0000","last_peered":"2024-03-06T07:24:54.906357+0000","last_clean":"2024-03-06T07:24:54.906357+0000","last_became_active":"2024-03-06T07:24:51.967118+0000","last_became_peered":"2024-03-06T07:24:51.967118+0000","last_unstale":"2024-03-06T07:24:54.906357+0000","last_undegraded":"2024-03-06T07:24:54.906357+0000","last_fullsized":"2024-03-06T07:24:54.906357+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:11.864382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905925+0000","last_change":"2024-03-06T07:24:52.969679+0000","last_active":"2024-03-06T07:24:54.905925+0000","last_peered":"2024-03-06T07:24:54.905925+0000","last_clean":"2024-03-06T07:24:54.905925+0000","last_became_active":"2024-03-06T07:24:52.969303+0000","last_became_peered":"2024-03-06T07:24:52.969303+0000","last_unstale":"2024-03-06T07:24:54.905925+0000","last_undegraded":"2024-03-06T07:24:54.905925+0000","last_fullsized":"2024-03-06T07:24:54.905925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:17:34.117528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593983+0000","last_change":"2024-03-06T07:24:51.967728+0000","last_active":"2024-03-06T07:24:54.593983+0000","last_peered":"2024-03-06T07:24:54.593983+0000","last_clean":"2024-03-06T07:24:54.593983+0000","last_became_active":"2024-03-06T07:24:51.967414+0000","last_became_peered":"2024-03-06T07:24:51.967414+0000","last_unstale":"2024-03-06T07:24:54.593983+0000","last_undegraded":"2024-03-06T07:24:54.593983+0000","last_fullsized":"2024-03-06T07:24:54.593983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:42:29.367400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961834+0000","last_change":"2024-03-06T07:24:52.978285+0000","last_active":"2024-03-06T07:24:53.961834+0000","last_peered":"2024-03-06T07:24:53.961834+0000","last_clean":"2024-03-06T07:24:53.961834+0000","last_became_active":"2024-03-06T07:24:52.977984+0000","last_became_peered":"2024-03-06T07:24:52.977984+0000","last_unstale":"2024-03-06T07:24:53.961834+0000","last_undegraded":"2024-03-06T07:24:53.961834+0000","last_fullsized":"2024-03-06T07:24:53.961834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:20.859484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906430+0000","last_change":"2024-03-06T07:24:51.964639+0000","last_active":"2024-03-06T07:24:54.906430+0000","last_peered":"2024-03-06T07:24:54.906430+0000","last_clean":"2024-03-06T07:24:54.906430+0000","last_became_active":"2024-03-06T07:24:51.964394+0000","last_became_peered":"2024-03-06T07:24:51.964394+0000","last_unstale":"2024-03-06T07:24:54.906430+0000","last_undegraded":"2024-03-06T07:24:54.906430+0000","last_fullsized":"2024-03-06T07:24:54.906430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:17:07.096047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961658+0000","last_change":"2024-03-06T07:24:52.990238+0000","last_active":"2024-03-06T07:24:53.961658+0000","last_peered":"2024-03-06T07:24:53.961658+0000","last_clean":"2024-03-06T07:24:53.961658+0000","last_became_active":"2024-03-06T07:24:52.989912+0000","last_became_peered":"2024-03-06T07:24:52.989912+0000","last_unstale":"2024-03-06T07:24:53.961658+0000","last_undegraded":"2024-03-06T07:24:53.961658+0000","last_fullsized":"2024-03-06T07:24:53.961658+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:33:52.255518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962633+0000","last_change":"2024-03-06T07:24:51.957428+0000","last_active":"2024-03-06T07:24:53.962633+0000","last_peered":"2024-03-06T07:24:53.962633+0000","last_clean":"2024-03-06T07:24:53.962633+0000","last_became_active":"2024-03-06T07:24:51.957114+0000","last_became_peered":"2024-03-06T07:24:51.957114+0000","last_unstale":"2024-03-06T07:24:53.962633+0000","last_undegraded":"2024-03-06T07:24:53.962633+0000","last_fullsized":"2024-03-06T07:24:53.962633+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:10.253518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961714+0000","last_change":"2024-03-06T07:24:52.990522+0000","last_active":"2024-03-06T07:24:53.961714+0000","last_peered":"2024-03-06T07:24:53.961714+0000","last_clean":"2024-03-06T07:24:53.961714+0000","last_became_active":"2024-03-06T07:24:52.990232+0000","last_became_peered":"2024-03-06T07:24:52.990232+0000","last_unstale":"2024-03-06T07:24:53.961714+0000","last_undegraded":"2024-03-06T07:24:53.961714+0000","last_fullsized":"2024-03-06T07:24:53.961714+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:26.124316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978500+0000","last_change":"2024-03-06T07:24:51.962417+0000","last_active":"2024-03-06T07:24:53.978500+0000","last_peered":"2024-03-06T07:24:53.978500+0000","last_clean":"2024-03-06T07:24:53.978500+0000","last_became_active":"2024-03-06T07:24:51.962166+0000","last_became_peered":"2024-03-06T07:24:51.962166+0000","last_unstale":"2024-03-06T07:24:53.978500+0000","last_undegraded":"2024-03-06T07:24:53.978500+0000","last_fullsized":"2024-03-06T07:24:53.978500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:47.675658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978699+0000","last_change":"2024-03-06T07:24:52.969475+0000","last_active":"2024-03-06T07:24:53.978699+0000","last_peered":"2024-03-06T07:24:53.978699+0000","last_clean":"2024-03-06T07:24:53.978699+0000","last_became_active":"2024-03-06T07:24:52.969201+0000","last_became_peered":"2024-03-06T07:24:52.969201+0000","last_unstale":"2024-03-06T07:24:53.978699+0000","last_undegraded":"2024-03-06T07:24:53.978699+0000","last_fullsized":"2024-03-06T07:24:53.978699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:24.250545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906306+0000","last_change":"2024-03-06T07:24:51.965725+0000","last_active":"2024-03-06T07:24:54.906306+0000","last_peered":"2024-03-06T07:24:54.906306+0000","last_clean":"2024-03-06T07:24:54.906306+0000","last_became_active":"2024-03-06T07:24:51.965329+0000","last_became_peered":"2024-03-06T07:24:51.965329+0000","last_unstale":"2024-03-06T07:24:54.906306+0000","last_undegraded":"2024-03-06T07:24:54.906306+0000","last_fullsized":"2024-03-06T07:24:54.906306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:49:18.798102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962736+0000","last_change":"2024-03-06T07:24:52.966748+0000","last_active":"2024-03-06T07:24:53.962736+0000","last_peered":"2024-03-06T07:24:53.962736+0000","last_clean":"2024-03-06T07:24:53.962736+0000","last_became_active":"2024-03-06T07:24:52.966406+0000","last_became_peered":"2024-03-06T07:24:52.966406+0000","last_unstale":"2024-03-06T07:24:53.962736+0000","last_undegraded":"2024-03-06T07:24:53.962736+0000","last_fullsized":"2024-03-06T07:24:53.962736+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:24.250545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"71'1","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.962777+0000","last_change":"2024-03-06T07:24:51.961973+0000","last_active":"2024-03-06T07:24:53.962777+0000","last_peered":"2024-03-06T07:24:53.962777+0000","last_clean":"2024-03-06T07:24:53.962777+0000","last_became_active":"2024-03-06T07:24:51.961700+0000","last_became_peered":"2024-03-06T07:24:51.961700+0000","last_unstale":"2024-03-06T07:24:53.962777+0000","last_undegraded":"2024-03-06T07:24:53.962777+0000","last_fullsized":"2024-03-06T07:24:53.962777+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:37.482112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593918+0000","last_change":"2024-03-06T07:24:52.988998+0000","last_active":"2024-03-06T07:24:54.593918+0000","last_peered":"2024-03-06T07:24:54.593918+0000","last_clean":"2024-03-06T07:24:54.593918+0000","last_became_active":"2024-03-06T07:24:52.988713+0000","last_became_peered":"2024-03-06T07:24:52.988713+0000","last_unstale":"2024-03-06T07:24:54.593918+0000","last_undegraded":"2024-03-06T07:24:54.593918+0000","last_fullsized":"2024-03-06T07:24:54.593918+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:23:24.250545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905912+0000","last_change":"2024-03-06T07:24:51.959714+0000","last_active":"2024-03-06T07:24:54.905912+0000","last_peered":"2024-03-06T07:24:54.905912+0000","last_clean":"2024-03-06T07:24:54.905912+0000","last_became_active":"2024-03-06T07:24:51.959436+0000","last_became_peered":"2024-03-06T07:24:51.959436+0000","last_unstale":"2024-03-06T07:24:54.905912+0000","last_undegraded":"2024-03-06T07:24:54.905912+0000","last_fullsized":"2024-03-06T07:24:54.905912+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:29.293983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.594127+0000","last_change":"2024-03-06T07:24:52.984940+0000","last_active":"2024-03-06T07:24:54.594127+0000","last_peered":"2024-03-06T07:24:54.594127+0000","last_clean":"2024-03-06T07:24:54.594127+0000","last_became_active":"2024-03-06T07:24:52.984637+0000","last_became_peered":"2024-03-06T07:24:52.984637+0000","last_unstale":"2024-03-06T07:24:54.594127+0000","last_undegraded":"2024-03-06T07:24:54.594127+0000","last_fullsized":"2024-03-06T07:24:54.594127+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:57:19.407891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.978947+0000","last_change":"2024-03-06T07:24:51.968045+0000","last_active":"2024-03-06T07:24:53.978947+0000","last_peered":"2024-03-06T07:24:53.978947+0000","last_clean":"2024-03-06T07:24:53.978947+0000","last_became_active":"2024-03-06T07:24:51.967814+0000","last_became_peered":"2024-03-06T07:24:51.967814+0000","last_unstale":"2024-03-06T07:24:53.978947+0000","last_undegraded":"2024-03-06T07:24:53.978947+0000","last_fullsized":"2024-03-06T07:24:53.978947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:05.344124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905932+0000","last_change":"2024-03-06T07:24:52.979268+0000","last_active":"2024-03-06T07:24:54.905932+0000","last_peered":"2024-03-06T07:24:54.905932+0000","last_clean":"2024-03-06T07:24:54.905932+0000","last_became_active":"2024-03-06T07:24:52.978780+0000","last_became_peered":"2024-03-06T07:24:52.978780+0000","last_unstale":"2024-03-06T07:24:54.905932+0000","last_undegraded":"2024-03-06T07:24:54.905932+0000","last_fullsized":"2024-03-06T07:24:54.905932+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:05.734170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905756+0000","last_change":"2024-03-06T07:24:51.958710+0000","last_active":"2024-03-06T07:24:54.905756+0000","last_peered":"2024-03-06T07:24:54.905756+0000","last_clean":"2024-03-06T07:24:54.905756+0000","last_became_active":"2024-03-06T07:24:51.958435+0000","last_became_peered":"2024-03-06T07:24:51.958435+0000","last_unstale":"2024-03-06T07:24:54.905756+0000","last_undegraded":"2024-03-06T07:24:54.905756+0000","last_fullsized":"2024-03-06T07:24:54.905756+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:05:22.870549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906159+0000","last_change":"2024-03-06T07:24:52.981124+0000","last_active":"2024-03-06T07:24:54.906159+0000","last_peered":"2024-03-06T07:24:54.906159+0000","last_clean":"2024-03-06T07:24:54.906159+0000","last_became_active":"2024-03-06T07:24:52.980834+0000","last_became_peered":"2024-03-06T07:24:52.980834+0000","last_unstale":"2024-03-06T07:24:54.906159+0000","last_undegraded":"2024-03-06T07:24:54.906159+0000","last_fullsized":"2024-03-06T07:24:54.906159+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:12.112501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960611+0000","last_change":"2024-03-06T07:24:51.962531+0000","last_active":"2024-03-06T07:24:53.960611+0000","last_peered":"2024-03-06T07:24:53.960611+0000","last_clean":"2024-03-06T07:24:53.960611+0000","last_became_active":"2024-03-06T07:24:51.962253+0000","last_became_peered":"2024-03-06T07:24:51.962253+0000","last_unstale":"2024-03-06T07:24:53.960611+0000","last_undegraded":"2024-03-06T07:24:53.960611+0000","last_fullsized":"2024-03-06T07:24:53.960611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:50.894464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.960668+0000","last_change":"2024-03-06T07:24:52.975730+0000","last_active":"2024-03-06T07:24:53.960668+0000","last_peered":"2024-03-06T07:24:53.960668+0000","last_clean":"2024-03-06T07:24:53.960668+0000","last_became_active":"2024-03-06T07:24:52.975396+0000","last_became_peered":"2024-03-06T07:24:52.975396+0000","last_unstale":"2024-03-06T07:24:53.960668+0000","last_undegraded":"2024-03-06T07:24:53.960668+0000","last_fullsized":"2024-03-06T07:24:53.960668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:04.010768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.905689+0000","last_change":"2024-03-06T07:24:51.963025+0000","last_active":"2024-03-06T07:24:54.905689+0000","last_peered":"2024-03-06T07:24:54.905689+0000","last_clean":"2024-03-06T07:24:54.905689+0000","last_became_active":"2024-03-06T07:24:51.962710+0000","last_became_peered":"2024-03-06T07:24:51.962710+0000","last_unstale":"2024-03-06T07:24:54.905689+0000","last_undegraded":"2024-03-06T07:24:54.905689+0000","last_fullsized":"2024-03-06T07:24:54.905689+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:54:33.722295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906366+0000","last_change":"2024-03-06T07:24:52.979911+0000","last_active":"2024-03-06T07:24:54.906366+0000","last_peered":"2024-03-06T07:24:54.906366+0000","last_clean":"2024-03-06T07:24:54.906366+0000","last_became_active":"2024-03-06T07:24:52.979635+0000","last_became_peered":"2024-03-06T07:24:52.979635+0000","last_unstale":"2024-03-06T07:24:54.906366+0000","last_undegraded":"2024-03-06T07:24:54.906366+0000","last_fullsized":"2024-03-06T07:24:54.906366+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:25.902446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906178+0000","last_change":"2024-03-06T07:24:51.963634+0000","last_active":"2024-03-06T07:24:54.906178+0000","last_peered":"2024-03-06T07:24:54.906178+0000","last_clean":"2024-03-06T07:24:54.906178+0000","last_became_active":"2024-03-06T07:24:51.963368+0000","last_became_peered":"2024-03-06T07:24:51.963368+0000","last_unstale":"2024-03-06T07:24:54.906178+0000","last_undegraded":"2024-03-06T07:24:54.906178+0000","last_fullsized":"2024-03-06T07:24:54.906178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:57:50.900012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:53.961804+0000","last_change":"2024-03-06T07:24:52.990500+0000","last_active":"2024-03-06T07:24:53.961804+0000","last_peered":"2024-03-06T07:24:53.961804+0000","last_clean":"2024-03-06T07:24:53.961804+0000","last_became_active":"2024-03-06T07:24:52.989913+0000","last_became_peered":"2024-03-06T07:24:52.989913+0000","last_unstale":"2024-03-06T07:24:53.961804+0000","last_undegraded":"2024-03-06T07:24:53.961804+0000","last_fullsized":"2024-03-06T07:24:53.961804+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:51.944226+0000","last_clean_scrub_stamp":"2024-03-06T07:24:51.944226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:17:39.470413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"70'2","reported_seq":16,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.906306+0000","last_change":"2024-03-06T07:24:51.964342+0000","last_active":"2024-03-06T07:24:54.906306+0000","last_peered":"2024-03-06T07:24:54.906306+0000","last_clean":"2024-03-06T07:24:54.906306+0000","last_became_active":"2024-03-06T07:24:51.964058+0000","last_became_peered":"2024-03-06T07:24:51.964058+0000","last_unstale":"2024-03-06T07:24:54.906306+0000","last_undegraded":"2024-03-06T07:24:54.906306+0000","last_fullsized":"2024-03-06T07:24:54.906306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:24:50.936071+0000","last_clean_scrub_stamp":"2024-03-06T07:24:50.936071+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:29:50.894464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":264,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-06T07:24:54.593717+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:24:54.593717+0000","last_peered":"2024-03-06T07:24:54.593717+0000","last_clean":"2024-03-06T07:24:54.593717+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:24:54.593717+0000","last_undegraded":"2024-03-06T07:24:54.593717+0000","last_fullsized":"2024-03-06T07:24:54.593717+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32612,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1466368,"data_stored":954721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":363829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":868352,"data_stored":363827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32028,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339812,"statfs":{"total":96636764160,"available":96603967488,"internally_reserved":0,"allocated":868352,"data_stored":363807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":363849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":876544,"data_stored":365349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32044,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339796,"statfs":{"total":96636764160,"available":96603951104,"internally_reserved":0,"allocated":884736,"data_stored":365965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1458176,"data_stored":954105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:00.642 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:00.643 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:00.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.783+0000 7fc6ae79a640 1 Processor -- start 2024-03-06T07:25:00.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.783+0000 7fc6ae79a640 1 -- start start 2024-03-06T07:25:00.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6ae79a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6ae79a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6a806d5c0 0x7fc6a80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6ae79a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a815ff90 con 0x7fc6a815bb40 2024-03-06T07:25:00.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6ae79a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a80b4dd0 con 0x7fc6a806d5c0 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6ae79a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a806c910 con 0x7fc6a80b4970 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6a7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc6a815bb40 0x7fc6a815f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37822/0 (socket says 172.21.15.72:37822) 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6a7fff640 1 -- 172.21.15.72:0/2344526919 learned_addr learned my addr 172.21.15.72:0/2344526919 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6a77fe640 1 --2- 172.21.15.72:0/2344526919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.784+0000 7fc6acd37640 1 --2- 172.21.15.72:0/2344526919 >> v2:172.21.15.146:3300/0 conn(0x7fc6a806d5c0 0x7fc6a80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a6ffd640 1 -- 172.21.15.72:0/2344526919 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112431792 0 0) 0x7fc6a815ff90 con 0x7fc6a815bb40 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a6ffd640 1 -- 172.21.15.72:0/2344526919 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc690003520 con 0x7fc6a815bb40 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a77fe640 1 -- 172.21.15.72:0/2344526919 >> v2:172.21.15.146:3300/0 conn(0x7fc6a806d5c0 msgr2=0x7fc6a80cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a77fe640 1 --2- 172.21.15.72:0/2344526919 >> v2:172.21.15.146:3300/0 conn(0x7fc6a806d5c0 0x7fc6a80cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a77fe640 1 -- 172.21.15.72:0/2344526919 >> v1:172.21.15.72:6789/0 conn(0x7fc6a815bb40 legacy=0x7fc6a815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a77fe640 1 -- 172.21.15.72:0/2344526919 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8160270 con 0x7fc6a80b4970 2024-03-06T07:25:00.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.785+0000 7fc6a77fe640 1 --2- 172.21.15.72:0/2344526919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a806d080 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0097a0 tx=0x7fc69c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e71f178db8281ec server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6a6ffd640 1 -- 172.21.15.72:0/2344526919 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c035020 con 0x7fc6a80b4970 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6a6ffd640 1 -- 172.21.15.72:0/2344526919 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c031830 con 0x7fc6a80b4970 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6a6ffd640 1 -- 172.21.15.72:0/2344526919 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c031be0 con 0x7fc6a80b4970 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 -- 172.21.15.72:0/2344526919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 msgr2=0x7fc6a806d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/2344526919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a806d080 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0097a0 tx=0x7fc69c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 -- 172.21.15.72:0/2344526919 shutdown_connections 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/2344526919 >> v2:172.21.15.146:3300/0 conn(0x7fc6a806d5c0 0x7fc6a80cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/2344526919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a806d080 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 -- 172.21.15.72:0/2344526919 >> 172.21.15.72:0/2344526919 conn(0x7fc6a806b190 msgr2=0x7fc6a806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 -- 172.21.15.72:0/2344526919 shutdown_connections 2024-03-06T07:25:00.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.786+0000 7fc6ae79a640 1 -- 172.21.15.72:0/2344526919 wait complete. 2024-03-06T07:25:00.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.787+0000 7fc6ae79a640 1 Processor -- start 2024-03-06T07:25:00.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.787+0000 7fc6ae79a640 1 -- start start 2024-03-06T07:25:00.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6ae79a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a806d5c0 0x7fc6a815ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6ae79a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6ae79a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 0x7fc6a80b1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6ae79a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a815bf20 con 0x7fc6a815bb40 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6ae79a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a806cb60 con 0x7fc6a806d5c0 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6ae79a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a815c0a0 con 0x7fc6a80b4970 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6acd37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 0x7fc6a80b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6acd37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 0x7fc6a80b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37782/0 (socket says 172.21.15.72:37782) 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6acd37640 1 -- 172.21.15.72:0/3204671888 learned_addr learned my addr 172.21.15.72:0/3204671888 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6a7fff640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a806d5c0 0x7fc6a815ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.788+0000 7fc6a77fe640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6acd37640 1 -- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 msgr2=0x7fc6a814d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6acd37640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a814d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6acd37640 1 -- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a806d5c0 msgr2=0x7fc6a815ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6acd37640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a806d5c0 0x7fc6a815ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6acd37640 1 -- 172.21.15.72:0/3204671888 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a814dd40 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6acd37640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 0x7fc6a80b1860 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000efc0 tx=0x7fc6a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0018020 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a0014480 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a8158c80 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0014870 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.789+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a815c6d0 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.790+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a80732b0 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.790+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6a0005d10 con 0x7fc6a815bb40 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.791+0000 7fc6a4ff9640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69406b4d0 0x7fc69406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:00.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.791+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a008aba0 con 0x7fc6a815bb40 2024-03-06T07:25:00.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.791+0000 7fc6a7fff640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69406b4d0 0x7fc69406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:00.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.791+0000 7fc6a7fff640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69406b4d0 0x7fc69406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc698005880 tx=0x7fc698009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:00.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:00.795+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6a0022070 con 0x7fc6a815bb40 2024-03-06T07:25:01.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.039+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc6a8153c80 con 0x7fc6a815bb40 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.040+0000 7fc6a4ff9640 1 -- 172.21.15.72:0/3204671888 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc6a0010040 con 0x7fc6a815bb40 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:25:01.042 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:25:01.043 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:25:01.043 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:25:01.043 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69406b4d0 msgr2=0x7fc69406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69406b4d0 0x7fc69406d990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc698005880 tx=0x7fc698009040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 msgr2=0x7fc6a80b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 0x7fc6a80b1860 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000efc0 tx=0x7fc6a000c490 comp rx=0 tx=0).stop 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 shutdown_connections 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69406b4d0 0x7fc69406d990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6a815bb40 0x7fc6a80b1860 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6a80b4970 0x7fc6a814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 --2- 172.21.15.72:0/3204671888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6a806d5c0 0x7fc6a815ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 >> 172.21.15.72:0/3204671888 conn(0x7fc6a806b190 msgr2=0x7fc6a80ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 shutdown_connections 2024-03-06T07:25:01.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.042+0000 7fc6ae79a640 1 -- 172.21.15.72:0/3204671888 wait complete. 2024-03-06T07:25:01.055 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:01.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.154+0000 7f770ebf4640 1 Processor -- start 2024-03-06T07:25:01.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.154+0000 7f770ebf4640 1 -- start start 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.154+0000 7f770ebf4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77080d2710 0x7f77080b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.154+0000 7f770ebf4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b3430 con 0x7f77080d22b0 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.154+0000 7f770ebf4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b3710 con 0x7f770815dfe0 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.154+0000 7f770ebf4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77080b31e0 con 0x7f77080d2710 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f770d191640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f77080d2710 0x7f77080b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f7707fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f77080d22b0 0x7f77080b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37826/0 (socket says 172.21.15.72:37826) 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f7707fff640 1 -- 172.21.15.72:0/584715737 learned_addr learned my addr 172.21.15.72:0/584715737 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:01.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f770d191640 1 -- 172.21.15.72:0/584715737 >> v1:172.21.15.146:6789/0 conn(0x7f770815dfe0 legacy=0x7f77080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f770d191640 1 -- 172.21.15.72:0/584715737 >> v1:172.21.15.72:6789/0 conn(0x7f77080d22b0 legacy=0x7f77080b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f77077fe640 1 -- 172.21.15.72:0/584715737 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372604661 0 0) 0x7f77080b3710 con 0x7f770815dfe0 2024-03-06T07:25:01.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.155+0000 7f770d191640 1 -- 172.21.15.72:0/584715737 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080b39f0 con 0x7f77080d2710 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770d191640 1 --2- 172.21.15.72:0/584715737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d2710 0x7f77080b4900 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f7700009dc0 tx=0x7f770002bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d7177f2b367638 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f77077fe640 1 -- 172.21.15.72:0/584715737 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700039070 con 0x7f77080d2710 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f77077fe640 1 -- 172.21.15.72:0/584715737 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770002caa0 con 0x7f77080d2710 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f77077fe640 1 -- 172.21.15.72:0/584715737 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700034440 con 0x7f77080d2710 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 -- 172.21.15.72:0/584715737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d2710 msgr2=0x7f77080b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 --2- 172.21.15.72:0/584715737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d2710 0x7f77080b4900 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f7700009dc0 tx=0x7f770002bd70 comp rx=0 tx=0).stop 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 -- 172.21.15.72:0/584715737 shutdown_connections 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 --2- 172.21.15.72:0/584715737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f77080d2710 0x7f77080b4900 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 -- 172.21.15.72:0/584715737 >> 172.21.15.72:0/584715737 conn(0x7f77080d3cd0 msgr2=0x7f77080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 -- 172.21.15.72:0/584715737 shutdown_connections 2024-03-06T07:25:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.156+0000 7f770ebf4640 1 -- 172.21.15.72:0/584715737 wait complete. 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.157+0000 7f770ebf4640 1 Processor -- start 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.157+0000 7f770ebf4640 1 -- start start 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770ebf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77080d22b0 0x7f77080cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770ebf4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77080d2710 0x7f770815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770ebf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 0x7f770815f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770ebf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080cfa00 con 0x7f770815dfe0 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770ebf4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080b3d60 con 0x7f77080d2710 2024-03-06T07:25:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770ebf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d1c70 con 0x7f77080d22b0 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770d191640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 0x7f770815f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770d191640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 0x7f770815f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37804/0 (socket says 172.21.15.72:37804) 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.158+0000 7f770d191640 1 -- 172.21.15.72:0/151107052 learned_addr learned my addr 172.21.15.72:0/151107052 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f7707fff640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77080d2710 0x7f770815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770c990640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77080d22b0 0x7f77080cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770d191640 1 -- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77080d22b0 msgr2=0x7f77080cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770d191640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77080d22b0 0x7f77080cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770d191640 1 -- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77080d2710 msgr2=0x7f770815f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770d191640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77080d2710 0x7f770815f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770d191640 1 -- 172.21.15.72:0/151107052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770815a9d0 con 0x7f770815dfe0 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770d191640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 0x7f770815f950 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f7700037040 tx=0x7f7700004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700044020 con 0x7f770815dfe0 2024-03-06T07:25:01.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700034cf0 con 0x7f770815dfe0 2024-03-06T07:25:01.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770815ff80 con 0x7f770815dfe0 2024-03-06T07:25:01.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700039040 con 0x7f770815dfe0 2024-03-06T07:25:01.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.159+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7708160540 con 0x7f770815dfe0 2024-03-06T07:25:01.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.160+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770804f330 con 0x7f770815dfe0 2024-03-06T07:25:01.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.160+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7700047020 con 0x7f770815dfe0 2024-03-06T07:25:01.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.161+0000 7f77057fa640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76ec06b4d0 0x7f76ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.161+0000 7f770c990640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76ec06b4d0 0x7f76ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.161+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77000ac900 con 0x7f770815dfe0 2024-03-06T07:25:01.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.161+0000 7f770c990640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76ec06b4d0 0x7f76ec06d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f76fc000f60 tx=0x7f76fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.165+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f770004e030 con 0x7f770815dfe0 2024-03-06T07:25:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.372+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77080b4720 con 0x7f770815dfe0 2024-03-06T07:25:01.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.372+0000 7f77057fa640 1 -- 172.21.15.72:0/151107052 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f770007b130 con 0x7f770815dfe0 2024-03-06T07:25:01.374 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:01.374 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76ec06b4d0 msgr2=0x7f76ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76ec06b4d0 0x7f76ec06d990 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f76fc000f60 tx=0x7f76fc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 msgr2=0x7f770815f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 0x7f770815f950 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f7700037040 tx=0x7f7700004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 shutdown_connections 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76ec06b4d0 0x7f76ec06d990 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770815dfe0 0x7f770815f950 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77080d2710 0x7f770815f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 --2- 172.21.15.72:0/151107052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77080d22b0 0x7f77080cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 >> 172.21.15.72:0/151107052 conn(0x7f77080d3cd0 msgr2=0x7f77080b4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 shutdown_connections 2024-03-06T07:25:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.374+0000 7f770ebf4640 1 -- 172.21.15.72:0/151107052 wait complete. 2024-03-06T07:25:01.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:24:52.303862+0000', 'modified': '2024-03-06T07:24:54.309372+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6579}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6579': {'gid': 6579, 'name': 'b', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6835/2032377664', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6833', 'nonce': 2032377664}, {'type': 'v1', 'addr': '172.21.15.146:6835', 'nonce': 2032377664}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6579, 'qdb_cluster': [6579]} 2024-03-06T07:25:01.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:01.790 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:25:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.889+0000 7fde0fd3a640 1 Processor -- start 2024-03-06T07:25:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.889+0000 7fde0fd3a640 1 -- start start 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0fd3a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fde080b4460 0x7fde080d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0fd3a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde08162970 0x7fde080b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0fd3a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde08162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0fd3a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde08161b00 con 0x7fde08162970 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0fd3a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde080cecc0 con 0x7fde080b4460 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0fd3a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080d22c0 con 0x7fde08161cf0 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0d2d5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde08162970 0x7fde080b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0d2d5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde08162970 0x7fde080b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37816/0 (socket says 172.21.15.72:37816) 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0d2d5640 1 -- 172.21.15.72:0/1790140590 learned_addr learned my addr 172.21.15.72:0/1790140590 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/1790140590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde08162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.890+0000 7fde0dad6640 1 --2- 172.21.15.72:0/1790140590 >> v2:172.21.15.146:3300/0 conn(0x7fde080b4460 0x7fde080d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0d2d5640 1 -- 172.21.15.72:0/1790140590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 msgr2=0x7fde08162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/1790140590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde08162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0d2d5640 1 -- 172.21.15.72:0/1790140590 >> v2:172.21.15.146:3300/0 conn(0x7fde080b4460 msgr2=0x7fde080d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/1790140590 >> v2:172.21.15.146:3300/0 conn(0x7fde080b4460 0x7fde080d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0d2d5640 1 -- 172.21.15.72:0/1790140590 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080d25a0 con 0x7fde08162970 2024-03-06T07:25:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/1790140590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde08162970 0x7fde080b3580 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fde0808ddd0 tx=0x7fddf802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e9576fa931aace server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.891+0000 7fde0cad4640 1 -- 172.21.15.72:0/1790140590 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8035020 con 0x7fde08162970 2024-03-06T07:25:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0cad4640 1 -- 172.21.15.72:0/1790140590 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf8031870 con 0x7fde08162970 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0cad4640 1 -- 172.21.15.72:0/1790140590 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf8031c40 con 0x7fde08162970 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1790140590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde08162970 msgr2=0x7fde080b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1790140590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde08162970 0x7fde080b3580 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fde0808ddd0 tx=0x7fddf802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1790140590 shutdown_connections 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1790140590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde08162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1790140590 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde08162970 0x7fde080b3580 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1790140590 >> v2:172.21.15.146:3300/0 conn(0x7fde080b4460 0x7fde080d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1790140590 >> 172.21.15.72:0/1790140590 conn(0x7fde080d3ce0 msgr2=0x7fde080d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1790140590 shutdown_connections 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.892+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1790140590 wait complete. 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.893+0000 7fde0fd3a640 1 Processor -- start 2024-03-06T07:25:01.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.893+0000 7fde0fd3a640 1 -- start start 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0fd3a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 0x7fde080cc260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0fd3a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde081562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0fd3a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde08162970 0x7fde080b8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0fd3a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080ce950 con 0x7fde080b4460 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0fd3a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080d2b60 con 0x7fde08162970 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0fd3a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080b3ac0 con 0x7fde08161cf0 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 0x7fde080cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 0x7fde080cc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37830/0 (socket says 172.21.15.72:37830) 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 -- 172.21.15.72:0/1605727102 learned_addr learned my addr 172.21.15.72:0/1605727102 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde081562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde08162970 0x7fde080b8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 -- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 msgr2=0x7fde081562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde081562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 -- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde08162970 msgr2=0x7fde080b8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde08162970 0x7fde080b8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.894+0000 7fde0dad6640 1 -- 172.21.15.72:0/1605727102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080ccab0 con 0x7fde080b4460 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.895+0000 7fde0dad6640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 0x7fde080cc260 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fde0400efc0 tx=0x7fde0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.895+0000 7fddfe7fc640 1 -- 172.21.15.72:0/1605727102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04018020 con 0x7fde080b4460 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.895+0000 7fddfe7fc640 1 -- 172.21.15.72:0/1605727102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04014480 con 0x7fde080b4460 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.895+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde080ccd90 con 0x7fde080b4460 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.895+0000 7fddfe7fc640 1 -- 172.21.15.72:0/1605727102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04014850 con 0x7fde080b4460 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.895+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde080cc7a0 con 0x7fde080b4460 2024-03-06T07:25:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.896+0000 7fddfe7fc640 1 -- 172.21.15.72:0/1605727102 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde04005d10 con 0x7fde080b4460 2024-03-06T07:25:01.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.896+0000 7fddfe7fc640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddec06b4d0 0x7fddec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.897+0000 7fddfe7fc640 1 -- 172.21.15.72:0/1605727102 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde0408ab50 con 0x7fde080b4460 2024-03-06T07:25:01.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.897+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddec06b4d0 0x7fddec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.897+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddec06b4d0 0x7fddec06d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036000 tx=0x7fddf8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 Processor -- start 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 -- start start 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 0x5566cb0e3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb0e78c0 0x5566cb0e9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb113a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cb04d6b0 con 0x5566cb113600 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566caff6a60 con 0x5566cb0e3870 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.898+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566cb049750 con 0x5566cb0e78c0 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 0x5566cb0e3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37844/0 (socket says 172.21.15.72:37844) 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 -- 172.21.15.72:0/2004595034 learned_addr learned my addr 172.21.15.72:0/2004595034 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0dad6640 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 0x5566cb0e3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:36192/0 (socket says 172.21.15.72:36192) 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb0e78c0 0x5566cb0e9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 -- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb0e78c0 msgr2=0x5566cb0e9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb0e78c0 0x5566cb0e9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 -- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 msgr2=0x5566cb0e3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 0x5566cb0e3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 -- 172.21.15.72:0/2004595034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566cb014f30 con 0x5566cb113600 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0dad6640 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 0x5566cb0e3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb113a80 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fde00009bb0 tx=0x7fde00011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37886266d7c7e309 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fddeb7fe640 1 -- 172.21.15.72:0/2004595034 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00014440 con 0x5566cb113600 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fddeb7fe640 1 -- 172.21.15.72:0/2004595034 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde000145e0 con 0x5566cb113600 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.899+0000 7fddeb7fe640 1 -- 172.21.15.72:0/2004595034 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000149b0 con 0x5566cb113600 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 -- 172.21.15.72:0/2004595034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 msgr2=0x5566cb113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb113a80 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fde00009bb0 tx=0x7fde00011690 comp rx=0 tx=0).stop 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 -- 172.21.15.72:0/2004595034 shutdown_connections 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb113a80 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb0e78c0 0x5566cb0e9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 --2- 172.21.15.72:0/2004595034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb0e3870 0x5566cb0e3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 -- 172.21.15.72:0/2004595034 >> 172.21.15.72:0/2004595034 conn(0x5566cb0e58f0 msgr2=0x5566cb0e71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 -- 172.21.15.72:0/2004595034 shutdown_connections 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 -- 172.21.15.72:0/2004595034 wait complete. 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.900+0000 7fde129fc740 1 -- start start 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 Processor -- start 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb10aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb116480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb10fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cb0e7d50 con 0x5566cb113600 2024-03-06T07:25:01.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cb009ea0 con 0x5566cb10c7f0 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566cb0e8030 con 0x5566cb10f820 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde129fc740 10 client.0 _ll_register_callbacks cb 0x5566cb009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb10aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb10aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37846/0 (socket says 172.21.15.72:37846) 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.901+0000 7fde0dad6640 1 -- 172.21.15.72:0/717958597 learned_addr learned my addr 172.21.15.72:0/717958597 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb116480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb10fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0dad6640 1 -- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 msgr2=0x5566cb10fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0dad6640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb10fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0dad6640 1 -- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 msgr2=0x5566cb116480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0dad6640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb116480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0dad6640 1 -- 172.21.15.72:0/717958597 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566cb00a270 con 0x5566cb113600 2024-03-06T07:25:01.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb10fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde0dad6640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb10aa80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fde04059080 tx=0x7fde04059250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d283cb104b058cb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fdde8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fdde8ff9640 1 -- 172.21.15.72:0/717958597 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04087060 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fdde8ff9640 1 -- 172.21.15.72:0/717958597 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde040589d0 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde000145e0 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fdde8ff9640 1 -- 172.21.15.72:0/717958597 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde0405f3b0 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 10 client.6741 fetch_fsmap learned FSMap version 33 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5566cad622d0 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fdde8ff9640 1 -- 172.21.15.72:0/717958597 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde000145e0 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fdde8ff9640 1 -- 172.21.15.72:0/717958597 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fde0405f550 con 0x5566cb113600 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 10 client.6741 fetch_fsmap finished waiting for FSMap version 33 2024-03-06T07:25:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 10 client.6741 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 20 client.6741 populate_metadata read hostname 'smithi072' 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb12d910 0x5566cb14d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 4 client.6741 mds_command: new command op to 6579 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.902+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 --> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566cb05abf0 con 0x5566cb12d910 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.903+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb12d910 0x5566cb14d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.903+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb12d910 0x5566cb14d360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f4587b4de91c52 server_cookie=32e45b3490aef22d in_seq=0 out_seq=0 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.903+0000 7fdde8ff9640 10 client.6741 ms_handle_connect on v2:172.21.15.146:6833/2032377664 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fdde8ff9640 1 -- 172.21.15.72:0/717958597 <== mds.0 v2:172.21.15.146:6833/2032377664 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5566cb05abf0 con 0x5566cb12d910 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fdde8ff9640 10 client.6741 handle_command_reply: tid=0 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 client.6741 shutdown 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 msgr2=0x5566cb10aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb10aa80 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fde04059080 tx=0x7fde04059250 comp rx=0 tx=0).stop 2024-03-06T07:25:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 shutdown_connections 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb12d910 0x5566cb14d360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb10fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb116480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 --2- 172.21.15.72:0/717958597 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb10aa80 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 >> 172.21.15.72:0/717958597 conn(0x5566cb0e3870 msgr2=0x5566cb1115c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fdde8ff9640 0 client.6741 ms_handle_reset on v2:172.21.15.146:6833/2032377664 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 shutdown_connections 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 1 -- 172.21.15.72:0/717958597 wait complete. 2024-03-06T07:25:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.904+0000 7fde129fc740 20 client.6741 trim_cache size 0 max 16384 2024-03-06T07:25:01.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 Processor -- start 2024-03-06T07:25:01.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 -- start start 2024-03-06T07:25:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 0x5566cb1162c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb0e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb113600 0x5566cb113f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde08200280 con 0x5566cb10c7f0 2024-03-06T07:25:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080b8300 con 0x5566cb113600 2024-03-06T07:25:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde080cbb40 con 0x5566cb10f820 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 0x5566cb1162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.921+0000 7fde0dad6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 0x5566cb1162c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37850/0 (socket says 172.21.15.72:37850) 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 -- 172.21.15.72:0/3216218348 learned_addr learned my addr 172.21.15.72:0/3216218348 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb0e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb113600 0x5566cb113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 -- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 msgr2=0x5566cb0e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb0e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 -- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb113600 msgr2=0x5566cb113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb113600 0x5566cb113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 -- 172.21.15.72:0/3216218348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566cb0ea2a0 con 0x5566cb10c7f0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 0x5566cb1162c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c090 tx=0x7fde0405b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8012328657e8a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fddeaffd640 1 -- 172.21.15.72:0/3216218348 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04082070 con 0x5566cb10c7f0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fddeaffd640 1 -- 172.21.15.72:0/3216218348 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04055430 con 0x5566cb10c7f0 2024-03-06T07:25:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fddeaffd640 1 -- 172.21.15.72:0/3216218348 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde04058a10 con 0x5566cb10c7f0 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde129fc740 1 -- 172.21.15.72:0/3216218348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 msgr2=0x5566cb1162c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.922+0000 7fde129fc740 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 0x5566cb1162c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c090 tx=0x7fde0405b000 comp rx=0 tx=0).stop 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 -- 172.21.15.72:0/3216218348 shutdown_connections 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb113600 0x5566cb113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb0e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 --2- 172.21.15.72:0/3216218348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb10c7f0 0x5566cb1162c0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 -- 172.21.15.72:0/3216218348 >> 172.21.15.72:0/3216218348 conn(0x5566cb0e3870 msgr2=0x5566cb117260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 -- 172.21.15.72:0/3216218348 shutdown_connections 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 -- 172.21.15.72:0/3216218348 wait complete. 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 -- start start 2024-03-06T07:25:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.923+0000 7fde129fc740 1 Processor -- start 2024-03-06T07:25:01.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb1a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb0e8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566cb12c810 con 0x5566cb113600 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde080b7d00 con 0x5566cb10c7f0 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566cb12e1b0 con 0x5566cb10f820 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde129fc740 10 client.0 _ll_register_callbacks cb 0x7fde0400cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb0e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb0e8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37866/0 (socket says 172.21.15.72:37866) 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 -- 172.21.15.72:0/3819211540 learned_addr learned my addr 172.21.15.72:0/3819211540 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb1a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0d2d5640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 -- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 msgr2=0x5566cb1525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb1525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 -- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 msgr2=0x5566cb1a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb1a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0e2d7640 1 -- 172.21.15.72:0/3819211540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566cae2c6b0 con 0x5566cb113600 2024-03-06T07:25:01.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.924+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb1a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde0e2d7640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb0e8470 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fde00009b50 tx=0x7fde00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e2af6aeb4a842c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fdde8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fdde8ff9640 1 -- 172.21.15.72:0/3819211540 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00003610 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fdde8ff9640 1 -- 172.21.15.72:0/3819211540 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde000037b0 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566cb10bbd0 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fdde8ff9640 1 -- 172.21.15.72:0/3819211540 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde000039b0 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 10 client.6747 fetch_fsmap learned FSMap version 33 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x5566cb108670 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fdde8ff9640 1 -- 172.21.15.72:0/3819211540 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566cb10bbd0 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fdde8ff9640 1 -- 172.21.15.72:0/3819211540 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 33) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fde00003b50 con 0x5566cb113600 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 10 client.6747 fetch_fsmap finished waiting for FSMap version 33 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 10 client.6747 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 20 client.6747 populate_metadata read hostname 'smithi072' 2024-03-06T07:25:01.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb0e89b0 0x5566cb111300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 4 client.6747 mds_command: new command op to 6579 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 --> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5566cb054f40 con 0x5566cb0e89b0 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.925+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb0e89b0 0x5566cb111300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fdde8ff9640 10 client.6747 ms_handle_connect on v2:172.21.15.146:6833/2032377664 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde0dad6640 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb0e89b0 0x5566cb111300 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a01dbfbc81b9936b server_cookie=3a8c0df397a0a3e3 in_seq=0 out_seq=0 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fdde8ff9640 1 -- 172.21.15.72:0/3819211540 <== mds.0 v2:172.21.15.146:6833/2032377664 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5566cb054f40 con 0x5566cb0e89b0 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fdde8ff9640 10 client.6747 handle_command_reply: tid=0 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 client.6747 shutdown 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 msgr2=0x5566cb0e8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb0e8470 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fde00009b50 tx=0x7fde00004730 comp rx=0 tx=0).stop 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 shutdown_connections 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:6833/2032377664,v1:172.21.15.146:6835/2032377664] conn(0x5566cb0e89b0 0x5566cb111300 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5566cb113600 0x5566cb0e8470 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5566cb10f820 0x5566cb1525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fdde8ff9640 0 client.6747 ms_handle_reset on v2:172.21.15.146:6833/2032377664 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 --2- 172.21.15.72:0/3819211540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5566cb10c7f0 0x5566cb1a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 >> 172.21.15.72:0/3819211540 conn(0x5566cb0e3870 msgr2=0x5566cb113d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 shutdown_connections 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 1 -- 172.21.15.72:0/3819211540 wait complete. 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.926+0000 7fde129fc740 20 client.6747 trim_cache size 0 max 16384 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:25:01.935 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6579, 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 8.9576541830000007, 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 33, 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 71, 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 71, 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 14.037608090999999 2024-03-06T07:25:01.936 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddec06b4d0 msgr2=0x7fddec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddec06b4d0 0x7fddec06d990 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fddf8036000 tx=0x7fddf8008030 comp rx=0 tx=0).stop 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 msgr2=0x7fde080cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 0x7fde080cc260 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fde0400efc0 tx=0x7fde0400c490 comp rx=0 tx=0).stop 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 shutdown_connections 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddec06b4d0 0x7fddec06d990 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde08162970 0x7fde080b8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde08161cf0 0x7fde081562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 --2- 172.21.15.72:0/1605727102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde080b4460 0x7fde080cc260 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 >> 172.21.15.72:0/1605727102 conn(0x7fde080d3ce0 msgr2=0x7fde080cf150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 shutdown_connections 2024-03-06T07:25:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:01.927+0000 7fde0fd3a640 1 -- 172.21.15.72:0/1605727102 wait complete. 2024-03-06T07:25:01.940 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:02.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.038+0000 7fa98493a640 1 Processor -- start 2024-03-06T07:25:02.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.038+0000 7fa98493a640 1 -- start start 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa98493a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa98015fa10 0x7fa9800ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa98493a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9800cebe0 con 0x7fa98015bdd0 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa98493a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9800ceec0 con 0x7fa9800d16e0 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa98493a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9800cf1a0 con 0x7fa98015fa10 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa97e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa98015bdd0 0x7fa9800c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37830/0 (socket says 172.21.15.72:37830) 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa97e59c640 1 -- 172.21.15.72:0/162024319 learned_addr learned my addr 172.21.15.72:0/162024319 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.039+0000 7fa97dd9b640 1 --2- 172.21.15.72:0/162024319 >> v2:172.21.15.146:3300/0 conn(0x7fa98015fa10 0x7fa9800ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145639328 0 0) 0x7fa9800cebe0 con 0x7fa98015bdd0 2024-03-06T07:25:02.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa968003520 con 0x7fa98015bdd0 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842734785 0 0) 0x7fa9800ceec0 con 0x7fa9800d16e0 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9800cebe0 con 0x7fa9800d16e0 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97dd9b640 1 -- 172.21.15.72:0/162024319 >> v1:172.21.15.146:6789/0 conn(0x7fa9800d16e0 legacy=0x7fa9800d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97dd9b640 1 -- 172.21.15.72:0/162024319 >> v1:172.21.15.72:6789/0 conn(0x7fa98015bdd0 legacy=0x7fa9800c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.040+0000 7fa97dd9b640 1 -- 172.21.15.72:0/162024319 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800cc8e0 con 0x7fa98015fa10 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.041+0000 7fa97dd9b640 1 --2- 172.21.15.72:0/162024319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015fa10 0x7fa9800ce270 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa9740097a0 tx=0x7fa97402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9df577f64594b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.041+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974035020 con 0x7fa98015fa10 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.041+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa974031830 con 0x7fa98015fa10 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.041+0000 7fa97d59a640 1 -- 172.21.15.72:0/162024319 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa974031be0 con 0x7fa98015fa10 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.041+0000 7fa98493a640 1 -- 172.21.15.72:0/162024319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015fa10 msgr2=0x7fa9800ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.041+0000 7fa98493a640 1 --2- 172.21.15.72:0/162024319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015fa10 0x7fa9800ce270 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa9740097a0 tx=0x7fa97402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.042+0000 7fa98493a640 1 -- 172.21.15.72:0/162024319 shutdown_connections 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.042+0000 7fa98493a640 1 --2- 172.21.15.72:0/162024319 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015fa10 0x7fa9800ce270 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.042+0000 7fa98493a640 1 -- 172.21.15.72:0/162024319 >> 172.21.15.72:0/162024319 conn(0x7fa98009a9f0 msgr2=0x7fa98009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.042+0000 7fa98493a640 1 -- 172.21.15.72:0/162024319 shutdown_connections 2024-03-06T07:25:02.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.042+0000 7fa98493a640 1 -- 172.21.15.72:0/162024319 wait complete. 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.043+0000 7fa98493a640 1 Processor -- start 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.043+0000 7fa98493a640 1 -- start start 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 0x7fa980154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015bdd0 0x7fa9800c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98015fa10 0x7fa9800c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9800ce7b0 con 0x7fa9800d16e0 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98015c1b0 con 0x7fa98015bdd0 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9800ce930 con 0x7fa98015fa10 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 0x7fa980154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 0x7fa980154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37878/0 (socket says 172.21.15.72:37878) 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 -- 172.21.15.72:0/3161500922 learned_addr learned my addr 172.21.15.72:0/3161500922 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97dd9b640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015bdd0 0x7fa9800c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97ed9d640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98015fa10 0x7fa9800c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 -- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98015fa10 msgr2=0x7fa9800c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98015fa10 0x7fa9800c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 -- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015bdd0 msgr2=0x7fa9800c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015bdd0 0x7fa9800c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 -- 172.21.15.72:0/3161500922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9800cb590 con 0x7fa9800d16e0 2024-03-06T07:25:02.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97dd9b640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015bdd0 0x7fa9800c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa97e59c640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 0x7fa980154f00 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa97000cba0 tx=0x7fa97000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970017070 con 0x7fa9800d16e0 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97000ad40 con 0x7fa9800d16e0 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98015d700 con 0x7fa9800d16e0 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.044+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970012620 con 0x7fa9800d16e0 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.045+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9800cbae0 con 0x7fa9800d16e0 2024-03-06T07:25:02.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.045+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98004f330 con 0x7fa9800d16e0 2024-03-06T07:25:02.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.046+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa97001a070 con 0x7fa9800d16e0 2024-03-06T07:25:02.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.046+0000 7fa96effd640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa96006b4d0 0x7fa96006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:02.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.046+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa970089ff0 con 0x7fa9800d16e0 2024-03-06T07:25:02.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.046+0000 7fa97dd9b640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa96006b4d0 0x7fa96006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:02.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.047+0000 7fa97dd9b640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa96006b4d0 0x7fa96006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa974036000 tx=0x7fa974008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:02.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.050+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa970058830 con 0x7fa9800d16e0 2024-03-06T07:25:02.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.262+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa9800b7a40 con 0x7fa9800d16e0 2024-03-06T07:25:02.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.262+0000 7fa96effd640 1 -- 172.21.15.72:0/3161500922 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa97005c8d0 con 0x7fa9800d16e0 2024-03-06T07:25:02.265 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:02.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa96006b4d0 msgr2=0x7fa96006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa96006b4d0 0x7fa96006d990 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa974036000 tx=0x7fa974008030 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 msgr2=0x7fa980154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 0x7fa980154f00 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fa97000cba0 tx=0x7fa97000a430 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 shutdown_connections 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa96006b4d0 0x7fa96006d990 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98015fa10 0x7fa9800c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa98015bdd0 0x7fa9800c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 --2- 172.21.15.72:0/3161500922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9800d16e0 0x7fa980154f00 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 >> 172.21.15.72:0/3161500922 conn(0x7fa98009a9f0 msgr2=0x7fa9800c5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 shutdown_connections 2024-03-06T07:25:02.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:02.264+0000 7fa98493a640 1 -- 172.21.15.72:0/3161500922 wait complete. 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:02.277 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:02.278 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:02.278 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:02.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.278 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:02.293 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42493sec preferred_lft 42493sec 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.294 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:02.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.295 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:02.358 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:25:02.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.360 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:25:02.415 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:25:02.415 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:02.415 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:25:02.416 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:25:02.416 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:02.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.416 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:25:02.416 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:25:02.416 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:25:02.416 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:25:02.416 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:25:02.416 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:25:02.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.510 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:25:02.525 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:02.525 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:02.525 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:02.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.525 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:25:02.592 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:02.593 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:02.593 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:02.648 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:02.648 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:02.703 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:25:02.769 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:02.825 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:25:02.825 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:25:02.825 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:02.826 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:02.827 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:02.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:02.828 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:25:02.880 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:02.880 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:02.923 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:25:02.949 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:03.004 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:25:03.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:25:03.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:25:03.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:03.007 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:03.007 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:03.007 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:03.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:03.008 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:25:03.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:25:03.006+0000 7f856f241480 -1 init, newargv = 0x55ea1c69c5c0 newargc=15 2024-03-06T07:25:03.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[62400]: starting ceph client 2024-03-06T07:25:03.012 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[62400]: starting fuse 2024-03-06T07:25:03.022 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:25:03.023 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:03.023 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:03.024 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:03.025 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:25:03.143 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.62400.asok 2024-03-06T07:25:03.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:03.153 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.62400.asok 2024-03-06T07:25:03.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:03.153 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.62400.asok status 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "62400", 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:25:03.288 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6759, 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 6759 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1834139235 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:03.289 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1834139235 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.6759 192.168.0.1:0/1834139235", 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/1834139235", 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 33, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 71, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:25:03.290 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:25:03.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:03.296 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:25:03.296 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:03.311 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:25:03.311 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:03.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:03.311 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:03.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-03-06T07:25:03.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.520+0000 7f88b05eb640 1 Processor -- start 2024-03-06T07:25:03.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.520+0000 7f88b05eb640 1 -- start start 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88b05eb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f88a80d4860 0x7f88a80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88b05eb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a81618b0 con 0x7f88a80b0e30 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88b05eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88a80cf400 con 0x7f88a81614d0 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88b05eb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88a80d4cf0 con 0x7f88a80d4860 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88adb86640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f88a80d4860 0x7f88a80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88aeb88640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f88a80b0e30 0x7f88a80b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41102/0 (socket says 172.21.15.72:41102) 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.521+0000 7f88aeb88640 1 -- 172.21.15.72:0/830781781 learned_addr learned my addr 172.21.15.72:0/830781781 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.522+0000 7f88adb86640 1 -- 172.21.15.72:0/830781781 >> v1:172.21.15.146:6789/0 conn(0x7f88a81614d0 legacy=0x7f88a80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.522+0000 7f88adb86640 1 -- 172.21.15.72:0/830781781 >> v1:172.21.15.72:6789/0 conn(0x7f88a80b0e30 legacy=0x7f88a80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.522+0000 7f88adb86640 1 -- 172.21.15.72:0/830781781 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80cf6a0 con 0x7f88a80d4860 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.522+0000 7f88adb86640 1 --2- 172.21.15.72:0/830781781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80d4860 0x7f88a80d2020 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f88980097a0 tx=0x7f889802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a21cc561972072 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88ad385640 1 -- 172.21.15.72:0/830781781 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8898034070 con 0x7f88a80d4860 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88ad385640 1 -- 172.21.15.72:0/830781781 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8898030610 con 0x7f88a80d4860 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88ad385640 1 -- 172.21.15.72:0/830781781 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88980309e0 con 0x7f88a80d4860 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 -- 172.21.15.72:0/830781781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80d4860 msgr2=0x7f88a80d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 --2- 172.21.15.72:0/830781781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80d4860 0x7f88a80d2020 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f88980097a0 tx=0x7f889802b960 comp rx=0 tx=0).stop 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 -- 172.21.15.72:0/830781781 shutdown_connections 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 --2- 172.21.15.72:0/830781781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80d4860 0x7f88a80d2020 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 -- 172.21.15.72:0/830781781 >> 172.21.15.72:0/830781781 conn(0x7f88a80d3d40 msgr2=0x7f88a80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 -- 172.21.15.72:0/830781781 shutdown_connections 2024-03-06T07:25:03.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.523+0000 7f88b05eb640 1 -- 172.21.15.72:0/830781781 wait complete. 2024-03-06T07:25:03.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.524+0000 7f88b05eb640 1 Processor -- start 2024-03-06T07:25:03.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.524+0000 7f88b05eb640 1 -- start start 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88b05eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 0x7f88a80ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88b05eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88a80d4860 0x7f88a80b7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88b05eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88a81614d0 0x7f88a80b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88b05eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80b0910 con 0x7f88a80b0e30 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88b05eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80cfc60 con 0x7f88a80d4860 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88b05eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88a80d2560 con 0x7f88a81614d0 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88ae387640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 0x7f88a80ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88ae387640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 0x7f88a80ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44294/0 (socket says 172.21.15.72:44294) 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88ae387640 1 -- 172.21.15.72:0/4283392339 learned_addr learned my addr 172.21.15.72:0/4283392339 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88adb86640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88a80d4860 0x7f88a80b7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.525+0000 7f88aeb88640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88a81614d0 0x7f88a80b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88ae387640 1 -- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88a81614d0 msgr2=0x7f88a80b7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88ae387640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88a81614d0 0x7f88a80b7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88ae387640 1 -- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88a80d4860 msgr2=0x7f88a80b7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88ae387640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88a80d4860 0x7f88a80b7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88ae387640 1 -- 172.21.15.72:0/4283392339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a80ca690 con 0x7f88a80b0e30 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88ae387640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 0x7f88a80ca150 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f88a400c730 tx=0x7f88a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4016020 con 0x7f88a80b0e30 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4006d50 con 0x7f88a80b0e30 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a8150d60 con 0x7f88a80b0e30 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4003bb0 con 0x7f88a80b0e30 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.526+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a8151320 con 0x7f88a80b0e30 2024-03-06T07:25:03.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.527+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a8068ff0 con 0x7f88a80b0e30 2024-03-06T07:25:03.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.527+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88a400a850 con 0x7f88a80b0e30 2024-03-06T07:25:03.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.528+0000 7f889effd640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f888c06b500 0x7f888c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:03.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.528+0000 7f88adb86640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f888c06b500 0x7f888c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:03.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.528+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88a4089c60 con 0x7f88a80b0e30 2024-03-06T07:25:03.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.528+0000 7f88adb86640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f888c06b500 0x7f888c06d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8898036000 tx=0x7f889803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:03.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.532+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f88a4020020 con 0x7f88a80b0e30 2024-03-06T07:25:03.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.701+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) v1 -- 0x7f88a8069e80 con 0x7f88a80b0e30 2024-03-06T07:25:03.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.704+0000 7f889effd640 1 -- 172.21.15.72:0/4283392339 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v23)=0 v23) v1 ==== 124+0+0 (secure 0 0 0) 0x7f88a4058520 con 0x7f88a80b0e30 2024-03-06T07:25:03.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f888c06b500 msgr2=0x7f888c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f888c06b500 0x7f888c06d9c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8898036000 tx=0x7f889803b040 comp rx=0 tx=0).stop 2024-03-06T07:25:03.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 msgr2=0x7f88a80ca150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:03.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 0x7f88a80ca150 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f88a400c730 tx=0x7f88a400cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 shutdown_connections 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f888c06b500 0x7f888c06d9c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88a81614d0 0x7f88a80b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88a80d4860 0x7f88a80b7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 --2- 172.21.15.72:0/4283392339 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88a80b0e30 0x7f88a80ca150 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 >> 172.21.15.72:0/4283392339 conn(0x7f88a80d3d40 msgr2=0x7f88a80cce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 shutdown_connections 2024-03-06T07:25:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:03.706+0000 7f88b05eb640 1 -- 172.21.15.72:0/4283392339 wait complete. 2024-03-06T07:25:13.718 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:13.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.827+0000 7f31bdcdf640 1 Processor -- start 2024-03-06T07:25:13.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.827+0000 7f31bdcdf640 1 -- start start 2024-03-06T07:25:13.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31bdcdf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f31b81515e0 0x7f31b80cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31bdcdf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b815b430 0x7f31b8071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31bdcdf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b815b8b0 con 0x7f31b80cc340 2024-03-06T07:25:13.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31bdcdf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b8070a70 con 0x7f31b81515e0 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31bdcdf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b8070d50 con 0x7f31b815b430 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f31b81515e0 0x7f31b80cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f31b81515e0 0x7f31b80cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52220/0 (socket says 172.21.15.72:52220) 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 -- 172.21.15.72:0/2721027405 learned_addr learned my addr 172.21.15.72:0/2721027405 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b6ffd640 1 --2- 172.21.15.72:0/2721027405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b815b430 0x7f31b8071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 -- 172.21.15.72:0/2721027405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b815b430 msgr2=0x7f31b8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 --2- 172.21.15.72:0/2721027405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b815b430 0x7f31b8071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 -- 172.21.15.72:0/2721027405 >> v1:172.21.15.146:6789/0 conn(0x7f31b80cc340 legacy=0x7f31b80cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.828+0000 7f31b77fe640 1 -- 172.21.15.72:0/2721027405 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8071030 con 0x7f31b81515e0 2024-03-06T07:25:13.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.829+0000 7f31b77fe640 1 --2- 172.21.15.72:0/2721027405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b81515e0 0x7f31b80cce40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f31a800bb10 tx=0x7f31a802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d3180a945620d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.829+0000 7f31b67fc640 1 -- 172.21.15.72:0/2721027405 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a803b070 con 0x7f31b81515e0 2024-03-06T07:25:13.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.829+0000 7f31b67fc640 1 -- 172.21.15.72:0/2721027405 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a8036440 con 0x7f31b81515e0 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.829+0000 7f31b67fc640 1 -- 172.21.15.72:0/2721027405 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8036810 con 0x7f31b81515e0 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 -- 172.21.15.72:0/2721027405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b81515e0 msgr2=0x7f31b80cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/2721027405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b81515e0 0x7f31b80cce40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f31a800bb10 tx=0x7f31a802e150 comp rx=0 tx=0).stop 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 -- 172.21.15.72:0/2721027405 shutdown_connections 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/2721027405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b815b430 0x7f31b8071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/2721027405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b81515e0 0x7f31b80cce40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 -- 172.21.15.72:0/2721027405 >> 172.21.15.72:0/2721027405 conn(0x7f31b80cf8a0 msgr2=0x7f31b80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 -- 172.21.15.72:0/2721027405 shutdown_connections 2024-03-06T07:25:13.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.830+0000 7f31bdcdf640 1 -- 172.21.15.72:0/2721027405 wait complete. 2024-03-06T07:25:13.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.831+0000 7f31bdcdf640 1 Processor -- start 2024-03-06T07:25:13.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.831+0000 7f31bdcdf640 1 -- start start 2024-03-06T07:25:13.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31bdcdf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 0x7f31b813cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31bdcdf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b81515e0 0x7f31b813f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31bdcdf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31b815b430 0x7f31b813fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31bdcdf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b8071ef0 con 0x7f31b80cc340 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31bdcdf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b8071d70 con 0x7f31b815b430 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31bdcdf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80cc770 con 0x7f31b81515e0 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 0x7f31b813cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 0x7f31b813cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52226/0 (socket says 172.21.15.72:52226) 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 -- 172.21.15.72:0/55646481 learned_addr learned my addr 172.21.15.72:0/55646481 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b6ffd640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b81515e0 0x7f31b813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b7fff640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31b815b430 0x7f31b813fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 -- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b81515e0 msgr2=0x7f31b813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b81515e0 0x7f31b813f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 -- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31b815b430 msgr2=0x7f31b813fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31b815b430 0x7f31b813fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 -- 172.21.15.72:0/55646481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b813e1f0 con 0x7f31b80cc340 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.832+0000 7f31b77fe640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 0x7f31b813cc00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f31a800b2c0 tx=0x7f31a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.833+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8045030 con 0x7f31b80cc340 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.833+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a8034440 con 0x7f31b80cc340 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.833+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b813e4d0 con 0x7f31b80cc340 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.833+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a803b040 con 0x7f31b80cc340 2024-03-06T07:25:13.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.833+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b813ea90 con 0x7f31b80cc340 2024-03-06T07:25:13.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.834+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b8068840 con 0x7f31b80cc340 2024-03-06T07:25:13.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.834+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f31a8048070 con 0x7f31b80cc340 2024-03-06T07:25:13.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.835+0000 7f319ffff640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f31a406b4d0 0x7f31a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:13.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.835+0000 7f31b6ffd640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f31a406b4d0 0x7f31a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:13.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.835+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f31a80aeac0 con 0x7f31b80cc340 2024-03-06T07:25:13.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.835+0000 7f31b6ffd640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f31a406b4d0 0x7f31a406d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f31b80c7300 tx=0x7f31ac009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:13.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:13.839+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f31a807daf0 con 0x7f31b80cc340 2024-03-06T07:25:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.047+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31b804f330 con 0x7f31b80cc340 2024-03-06T07:25:14.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.047+0000 7f319ffff640 1 -- 172.21.15.72:0/55646481 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f31a807d3f0 con 0x7f31b80cc340 2024-03-06T07:25:14.050 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:14.050 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:14.050 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f31a406b4d0 msgr2=0x7f31a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f31a406b4d0 0x7f31a406d990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f31b80c7300 tx=0x7f31ac009190 comp rx=0 tx=0).stop 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 msgr2=0x7f31b813cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 0x7f31b813cc00 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f31a800b2c0 tx=0x7f31a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 shutdown_connections 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f31a406b4d0 0x7f31a406d990 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31b815b430 0x7f31b813fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f31b81515e0 0x7f31b813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 --2- 172.21.15.72:0/55646481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31b80cc340 0x7f31b813cc00 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.049+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 >> 172.21.15.72:0/55646481 conn(0x7f31b80cf8a0 msgr2=0x7f31b8072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.050+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 shutdown_connections 2024-03-06T07:25:14.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.050+0000 7f31bdcdf640 1 -- 172.21.15.72:0/55646481 wait complete. 2024-03-06T07:25:14.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:14.178 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_created":0}} 2024-03-06T07:25:14.179 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-03-06T07:25:14.179 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:14.179 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:14.194 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> import uuid 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> n = 100 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = False 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> if 0 > 0: 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = True 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> dpath = os.path.dirname(path) 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> try: 2024-03-06T07:25:14.195 DEBUG:teuthology.orchestra.run.smithi072:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(n): 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> f.write(f"{i}") 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> f.flush() 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(f.fileno()) 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> os.unlink(fpath) 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> if create_hard_links: 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> for j in range(0): 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:14.196 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:25:14.197 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:25:14.197 DEBUG:teuthology.orchestra.run.smithi072:> finally: 2024-03-06T07:25:14.197 DEBUG:teuthology.orchestra.run.smithi072:> os.close(dirfd) 2024-03-06T07:25:14.197 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:25:14.343 INFO:teuthology.orchestra.run.smithi072.stderr:Traceback (most recent call last): 2024-03-06T07:25:14.344 INFO:teuthology.orchestra.run.smithi072.stderr: File "", line 26, in 2024-03-06T07:25:14.344 INFO:teuthology.orchestra.run.smithi072.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-03-06T07:25:14.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-06T07:25:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:14.354 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:14.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.492+0000 7f5b79977640 1 Processor -- start 2024-03-06T07:25:14.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.492+0000 7f5b79977640 1 -- start start 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b79977640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5b740b4170 0x7f5b740b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b79977640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740d25d0 con 0x7f5b740d0a80 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b79977640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b740d2870 con 0x7f5b740d4760 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b79977640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b740d4bf0 con 0x7f5b740b4170 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b737fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5b740d0a80 0x7f5b740cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56494/0 (socket says 172.21.15.72:56494) 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b737fe640 1 -- 172.21.15.72:0/4021081575 learned_addr learned my addr 172.21.15.72:0/4021081575 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b72ffd640 1 --2- 172.21.15.72:0/4021081575 >> v2:172.21.15.146:3300/0 conn(0x7f5b740b4170 0x7f5b740b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308788477 0 0) 0x7f5b740d25d0 con 0x7f5b740d0a80 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.493+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c003520 con 0x7f5b740d0a80 2024-03-06T07:25:14.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832979230 0 0) 0x7f5b740d2870 con 0x7f5b740d4760 2024-03-06T07:25:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b740d25d0 con 0x7f5b740d4760 2024-03-06T07:25:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1024618066 0 0) 0x7f5b5c003520 con 0x7f5b740d0a80 2024-03-06T07:25:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b72ffd640 1 -- 172.21.15.72:0/4021081575 >> v1:172.21.15.146:6789/0 conn(0x7f5b740d4760 legacy=0x7f5b740d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b72ffd640 1 -- 172.21.15.72:0/4021081575 >> v1:172.21.15.72:6789/0 conn(0x7f5b740d0a80 legacy=0x7f5b740cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b72ffd640 1 -- 172.21.15.72:0/4021081575 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b740d2b50 con 0x7f5b740b4170 2024-03-06T07:25:14.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.494+0000 7f5b72ffd640 1 --2- 172.21.15.72:0/4021081575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740b4170 0x7f5b740b6a90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009d90 tx=0x7f5b6402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eddf7072193b6fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64039070 con 0x7f5b740b4170 2024-03-06T07:25:14.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6402c940 con 0x7f5b740b4170 2024-03-06T07:25:14.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b71ffb640 1 -- 172.21.15.72:0/4021081575 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c001670 con 0x7f5b740b4170 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 -- 172.21.15.72:0/4021081575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740b4170 msgr2=0x7f5b740b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 --2- 172.21.15.72:0/4021081575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740b4170 0x7f5b740b6a90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009d90 tx=0x7f5b6402be10 comp rx=0 tx=0).stop 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 -- 172.21.15.72:0/4021081575 shutdown_connections 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 --2- 172.21.15.72:0/4021081575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740b4170 0x7f5b740b6a90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 -- 172.21.15.72:0/4021081575 >> 172.21.15.72:0/4021081575 conn(0x7f5b740d3c40 msgr2=0x7f5b740d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 -- 172.21.15.72:0/4021081575 shutdown_connections 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.495+0000 7f5b79977640 1 -- 172.21.15.72:0/4021081575 wait complete. 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.496+0000 7f5b79977640 1 Processor -- start 2024-03-06T07:25:14.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 -- start start 2024-03-06T07:25:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b740b4170 0x7f5b740a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740d0a80 0x7f5b74153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 0x7f5b740ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b740b4dd0 con 0x7f5b74154280 2024-03-06T07:25:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b740ce9e0 con 0x7f5b740d0a80 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b79977640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b740b4f50 con 0x7f5b740b4170 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b737fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 0x7f5b740ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b737fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 0x7f5b740ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52238/0 (socket says 172.21.15.72:52238) 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b737fe640 1 -- 172.21.15.72:0/279636248 learned_addr learned my addr 172.21.15.72:0/279636248 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b727fc640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740d0a80 0x7f5b74153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.497+0000 7f5b72ffd640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b740b4170 0x7f5b740a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b737fe640 1 -- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b740b4170 msgr2=0x7f5b740a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b737fe640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b740b4170 0x7f5b740a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b737fe640 1 -- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740d0a80 msgr2=0x7f5b74153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b737fe640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740d0a80 0x7f5b74153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b737fe640 1 -- 172.21.15.72:0/279636248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7409e8f0 con 0x7f5b74154280 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b737fe640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 0x7f5b740ccdb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c780 tx=0x7f5b6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c017070 con 0x7f5b74154280 2024-03-06T07:25:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c005ad0 con 0x7f5b74154280 2024-03-06T07:25:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7409ebd0 con 0x7f5b74154280 2024-03-06T07:25:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c007070 con 0x7f5b74154280 2024-03-06T07:25:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.498+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b740a1720 con 0x7f5b74154280 2024-03-06T07:25:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.499+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7404f330 con 0x7f5b74154280 2024-03-06T07:25:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.499+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b6c01a070 con 0x7f5b74154280 2024-03-06T07:25:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.500+0000 7f5b5bfff640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b6006b4d0 0x7f5b6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.500+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b6c0896a0 con 0x7f5b74154280 2024-03-06T07:25:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.500+0000 7f5b72ffd640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b6006b4d0 0x7f5b6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.500+0000 7f5b72ffd640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b6006b4d0 0x7f5b6006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5b64036000 tx=0x7f5b64030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.504+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b6c057ed0 con 0x7f5b74154280 2024-03-06T07:25:14.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.711+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b740733b0 con 0x7f5b74154280 2024-03-06T07:25:14.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.712+0000 7f5b5bfff640 1 -- 172.21.15.72:0/279636248 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f5b6c05bf70 con 0x7f5b74154280 2024-03-06T07:25:14.714 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:14.714 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:14.714 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b6006b4d0 msgr2=0x7f5b6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b6006b4d0 0x7f5b6006d990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5b64036000 tx=0x7f5b64030000 comp rx=0 tx=0).stop 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 msgr2=0x7f5b740ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 0x7f5b740ccdb0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c00c780 tx=0x7f5b6c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 shutdown_connections 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b6006b4d0 0x7f5b6006d990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b74154280 0x7f5b740ccdb0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b740d0a80 0x7f5b74153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 --2- 172.21.15.72:0/279636248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b740b4170 0x7f5b740a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.713+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 >> 172.21.15.72:0/279636248 conn(0x7f5b740d3c40 msgr2=0x7f5b740b91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.714+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 shutdown_connections 2024-03-06T07:25:14.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.714+0000 7f5b79977640 1 -- 172.21.15.72:0/279636248 wait complete. 2024-03-06T07:25:14.727 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:25:14.843 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_created":10}} 2024-03-06T07:25:14.844 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-03-06T07:25:14.844 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:14.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.942+0000 7f8f4635e640 1 Processor -- start 2024-03-06T07:25:14.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.942+0000 7f8f4635e640 1 -- start start 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.942+0000 7f8f4635e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f4015bf30 0x7f8f4015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.942+0000 7f8f4635e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4015f580 con 0x7f8f4013dc10 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.942+0000 7f8f4635e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4015b450 con 0x7f8f400b3060 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.942+0000 7f8f4635e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400cf3f0 con 0x7f8f4015bf30 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f3ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8f4013dc10 0x7f8f400b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56500/0 (socket says 172.21.15.72:56500) 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f3ffff640 1 -- 172.21.15.72:0/4088165291 learned_addr learned my addr 172.21.15.72:0/4088165291 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f448fb640 1 --2- 172.21.15.72:0/4088165291 >> v2:172.21.15.146:3300/0 conn(0x7f8f4015bf30 0x7f8f4015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916375760 0 0) 0x7f8f4015f580 con 0x7f8f4013dc10 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f28003520 con 0x7f8f4013dc10 2024-03-06T07:25:14.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616950023 0 0) 0x7f8f4015b450 con 0x7f8f400b3060 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4015f580 con 0x7f8f400b3060 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f448fb640 1 -- 172.21.15.72:0/4088165291 >> v1:172.21.15.146:6789/0 conn(0x7f8f400b3060 legacy=0x7f8f4015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f448fb640 1 -- 172.21.15.72:0/4088165291 >> v1:172.21.15.72:6789/0 conn(0x7f8f4013dc10 legacy=0x7f8f400b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.943+0000 7f8f448fb640 1 -- 172.21.15.72:0/4088165291 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4015b730 con 0x7f8f4015bf30 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.944+0000 7f8f448fb640 1 --2- 172.21.15.72:0/4088165291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f4015bf30 0x7f8f4015ef60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009d90 tx=0x7f8f3802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b463f32b4a56950 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.944+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38039070 con 0x7f8f4015bf30 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.944+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3802cc60 con 0x7f8f4015bf30 2024-03-06T07:25:14.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.944+0000 7f8f3effd640 1 -- 172.21.15.72:0/4088165291 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38034660 con 0x7f8f4015bf30 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.944+0000 7f8f4635e640 1 -- 172.21.15.72:0/4088165291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f4015bf30 msgr2=0x7f8f4015ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.944+0000 7f8f4635e640 1 --2- 172.21.15.72:0/4088165291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f4015bf30 0x7f8f4015ef60 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009d90 tx=0x7f8f3802beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.945+0000 7f8f4635e640 1 -- 172.21.15.72:0/4088165291 shutdown_connections 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.945+0000 7f8f4635e640 1 --2- 172.21.15.72:0/4088165291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f4015bf30 0x7f8f4015ef60 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.945+0000 7f8f4635e640 1 -- 172.21.15.72:0/4088165291 >> 172.21.15.72:0/4088165291 conn(0x7f8f400cf8a0 msgr2=0x7f8f400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.945+0000 7f8f4635e640 1 -- 172.21.15.72:0/4088165291 shutdown_connections 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.945+0000 7f8f4635e640 1 -- 172.21.15.72:0/4088165291 wait complete. 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.946+0000 7f8f4635e640 1 Processor -- start 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.946+0000 7f8f4635e640 1 -- start start 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f400b3060 0x7f8f400cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4013dc10 0x7f8f401436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 0x7f8f40143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400718e0 con 0x7f8f4015bf30 2024-03-06T07:25:14.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4015b210 con 0x7f8f400b3060 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f40071a60 con 0x7f8f4013dc10 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 0x7f8f40143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 0x7f8f40143c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52246/0 (socket says 172.21.15.72:52246) 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 -- 172.21.15.72:0/1247576824 learned_addr learned my addr 172.21.15.72:0/1247576824 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f3ffff640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f400b3060 0x7f8f400cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f3f7fe640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4013dc10 0x7f8f401436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 -- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4013dc10 msgr2=0x7f8f401436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4013dc10 0x7f8f401436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 -- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f400b3060 msgr2=0x7f8f400cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f400b3060 0x7f8f400cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 -- 172.21.15.72:0/1247576824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f401f13a0 con 0x7f8f4015bf30 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f448fb640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 0x7f8f40143c10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009d90 tx=0x7f8f38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38039050 con 0x7f8f4015bf30 2024-03-06T07:25:14.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38045070 con 0x7f8f4015bf30 2024-03-06T07:25:14.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f401f1650 con 0x7f8f4015bf30 2024-03-06T07:25:14.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38009450 con 0x7f8f4015bf30 2024-03-06T07:25:14.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.947+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f401f1c10 con 0x7f8f4015bf30 2024-03-06T07:25:14.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.948+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4004f330 con 0x7f8f4015bf30 2024-03-06T07:25:14.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.949+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f3803e850 con 0x7f8f4015bf30 2024-03-06T07:25:14.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.949+0000 7f8f3cff9640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:14.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.949+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f380ace90 con 0x7f8f4015bf30 2024-03-06T07:25:14.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.949+0000 7f8f3ffff640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:14.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.949+0000 7f8f3ffff640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8f30000f60 tx=0x7f8f30007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:14.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:14.953+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f3807b7e0 con 0x7f8f4015bf30 2024-03-06T07:25:15.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.161+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f400748f0 con 0x7f8f4015bf30 2024-03-06T07:25:15.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.161+0000 7f8f3cff9640 1 -- 172.21.15.72:0/1247576824 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f8f3807f880 con 0x7f8f4015bf30 2024-03-06T07:25:15.163 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:15.164 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:15.164 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b4d0 msgr2=0x7f8f2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8f30000f60 tx=0x7f8f30007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 msgr2=0x7f8f40143c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 0x7f8f40143c10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009d90 tx=0x7f8f38004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 shutdown_connections 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b4d0 0x7f8f2c06d990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4015bf30 0x7f8f40143c10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4013dc10 0x7f8f401436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 --2- 172.21.15.72:0/1247576824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f400b3060 0x7f8f400cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 >> 172.21.15.72:0/1247576824 conn(0x7f8f400cf8a0 msgr2=0x7f8f4006d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 shutdown_connections 2024-03-06T07:25:15.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:15.163+0000 7f8f4635e640 1 -- 172.21.15.72:0/1247576824 wait complete. 2024-03-06T07:25:15.176 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:15.294 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:25:15.294 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:25:15.294 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-03-06T07:25:20.225 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:20.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:20.295 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:20.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 Processor -- start 2024-03-06T07:25:20.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 -- start start 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf400cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf4006cf20 0x7fbf4006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4015e380 con 0x7fbf400d00c0 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4006ca90 con 0x7fbf4006cf20 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.402+0000 7fbf4553a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400cdc10 con 0x7fbf400cf850 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3effd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbf400d00c0 0x7fbf400d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:43020/0 (socket says 172.21.15.72:43020) 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3e7fc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf400cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf4006cf20 0x7fbf4006d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3effd640 1 -- 172.21.15.72:0/10308313 learned_addr learned my addr 172.21.15.72:0/10308313 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 msgr2=0x7fbf400cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3f7fe640 1 --2- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf400cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/10308313 >> v1:172.21.15.146:6789/0 conn(0x7fbf400d00c0 legacy=0x7fbf400d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/10308313 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4006c160 con 0x7fbf4006cf20 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.403+0000 7fbf3dffb640 1 -- 172.21.15.72:0/10308313 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165180746 0 0) 0x7fbf4015e380 con 0x7fbf400d00c0 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.404+0000 7fbf3f7fe640 1 --2- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf4006d3b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009d90 tx=0x7fbf3802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79aa7d016941abfa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.404+0000 7fbf3dffb640 1 -- 172.21.15.72:0/10308313 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38039070 con 0x7fbf4006cf20 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.404+0000 7fbf3dffb640 1 -- 172.21.15.72:0/10308313 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3802cc50 con 0x7fbf4006cf20 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.404+0000 7fbf3dffb640 1 -- 172.21.15.72:0/10308313 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38034440 con 0x7fbf4006cf20 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 -- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 msgr2=0x7fbf4006d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 --2- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf4006d3b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009d90 tx=0x7fbf3802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 -- 172.21.15.72:0/10308313 shutdown_connections 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 --2- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf4006d3b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 --2- 172.21.15.72:0/10308313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf400cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 -- 172.21.15.72:0/10308313 >> 172.21.15.72:0/10308313 conn(0x7fbf400b35a0 msgr2=0x7fbf400b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 -- 172.21.15.72:0/10308313 shutdown_connections 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.405+0000 7fbf4553a640 1 -- 172.21.15.72:0/10308313 wait complete. 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.406+0000 7fbf4553a640 1 Processor -- start 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.406+0000 7fbf4553a640 1 -- start start 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf4553a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf401f3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf4553a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf4015d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf4553a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 0x7fbf401f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf4553a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400cd900 con 0x7fbf400d00c0 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf4553a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4006c720 con 0x7fbf4006cf20 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf4553a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf400b1030 con 0x7fbf400cf850 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 0x7fbf401f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf401f3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 0x7fbf401f7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52250/0 (socket says 172.21.15.72:52250) 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3effd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf401f3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:58332/0 (socket says 172.21.15.72:58332) 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3e7fc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf4015d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/784270379 learned_addr learned my addr 172.21.15.72:0/784270379 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 msgr2=0x7fbf4015d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf4015d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 msgr2=0x7fbf401f3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf401f3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3f7fe640 1 -- 172.21.15.72:0/784270379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf401f8110 con 0x7fbf400d00c0 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.407+0000 7fbf3effd640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf401f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.408+0000 7fbf3f7fe640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 0x7fbf401f7bd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009eb0 tx=0x7fbf38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.408+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38039050 con 0x7fbf400d00c0 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.408+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38045070 con 0x7fbf400d00c0 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.408+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf401f4290 con 0x7fbf400d00c0 2024-03-06T07:25:20.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.408+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38009450 con 0x7fbf400d00c0 2024-03-06T07:25:20.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.408+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf401f4850 con 0x7fbf400d00c0 2024-03-06T07:25:20.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.409+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4004f330 con 0x7fbf400d00c0 2024-03-06T07:25:20.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.409+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf3803e610 con 0x7fbf400d00c0 2024-03-06T07:25:20.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.410+0000 7fbf277fe640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.410+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf380acab0 con 0x7fbf400d00c0 2024-03-06T07:25:20.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.410+0000 7fbf3effd640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.410+0000 7fbf3effd640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbf30007fd0 tx=0x7fbf30007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.414+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf401f8110 con 0x7fbf400d00c0 2024-03-06T07:25:20.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.620+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf400b0d90 con 0x7fbf400d00c0 2024-03-06T07:25:20.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.621+0000 7fbf277fe640 1 -- 172.21.15.72:0/784270379 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fbf3807b2e0 con 0x7fbf400d00c0 2024-03-06T07:25:20.623 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:20.623 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:20.623 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf2c06b4d0 msgr2=0x7fbf2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbf30007fd0 tx=0x7fbf30007de0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 msgr2=0x7fbf401f7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 0x7fbf401f7bd0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009eb0 tx=0x7fbf38004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 shutdown_connections 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf2c06b4d0 0x7fbf2c06d990 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf400d00c0 0x7fbf401f7bd0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf400cf850 0x7fbf4015d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 --2- 172.21.15.72:0/784270379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf4006cf20 0x7fbf401f3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 >> 172.21.15.72:0/784270379 conn(0x7fbf400b35a0 msgr2=0x7fbf400b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 shutdown_connections 2024-03-06T07:25:20.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.623+0000 7fbf4553a640 1 -- 172.21.15.72:0/784270379 wait complete. 2024-03-06T07:25:20.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:25:20.756 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-03-06T07:25:20.756 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-03-06T07:25:20.757 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:20.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.854+0000 7f9fcbe9d640 1 Processor -- start 2024-03-06T07:25:20.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.854+0000 7f9fcbe9d640 1 -- start start 2024-03-06T07:25:20.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fcbe9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9fc413dc10 0x7f9fc40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fcbe9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fcbe9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc415f580 con 0x7f9fc415e850 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fcbe9d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc40cf3f0 con 0x7f9fc413dc10 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fcbe9d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc415b210 con 0x7f9fc415bf30 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fca43a640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9fc415e850 0x7f9fc415eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56508/0 (socket says 172.21.15.72:56508) 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fca43a640 1 -- 172.21.15.72:0/3156023811 learned_addr learned my addr 172.21.15.72:0/3156023811 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9438640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9c39640 1 --2- 172.21.15.72:0/3156023811 >> v2:172.21.15.146:3300/0 conn(0x7f9fc413dc10 0x7f9fc40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc8c37640 1 -- 172.21.15.72:0/3156023811 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218969818 0 0) 0x7f9fc415f580 con 0x7f9fc415e850 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc8c37640 1 -- 172.21.15.72:0/3156023811 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb0003520 con 0x7f9fc415e850 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9c39640 1 -- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 msgr2=0x7f9fc40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9c39640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9c39640 1 -- 172.21.15.72:0/3156023811 >> v1:172.21.15.72:6789/0 conn(0x7f9fc415e850 legacy=0x7f9fc415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9c39640 1 -- 172.21.15.72:0/3156023811 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc415b450 con 0x7f9fc413dc10 2024-03-06T07:25:20.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.855+0000 7f9fc9438640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:20.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.856+0000 7f9fc9c39640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc413dc10 0x7f9fc40b2950 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00097a0 tx=0x7f9fc002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a7e9f779435f486 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.856+0000 7f9fc8c37640 1 -- 172.21.15.72:0/3156023811 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc00315a0 con 0x7f9fc413dc10 2024-03-06T07:25:20.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.856+0000 7f9fc8c37640 1 -- 172.21.15.72:0/3156023811 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc0037070 con 0x7f9fc413dc10 2024-03-06T07:25:20.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.856+0000 7f9fc8c37640 1 -- 172.21.15.72:0/3156023811 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0031940 con 0x7f9fc413dc10 2024-03-06T07:25:20.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc413dc10 msgr2=0x7f9fc40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc413dc10 0x7f9fc40b2950 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00097a0 tx=0x7f9fc002b840 comp rx=0 tx=0).stop 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/3156023811 shutdown_connections 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/3156023811 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc413dc10 0x7f9fc40b2950 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/3156023811 >> 172.21.15.72:0/3156023811 conn(0x7f9fc40cf8a0 msgr2=0x7f9fc40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/3156023811 shutdown_connections 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.857+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/3156023811 wait complete. 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.858+0000 7f9fcbe9d640 1 Processor -- start 2024-03-06T07:25:20.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.858+0000 7f9fcbe9d640 1 -- start start 2024-03-06T07:25:20.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 0x7f9fc4140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc4142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc415e850 0x7f9fc4143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc40708d0 con 0x7f9fc413dc10 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc413dff0 con 0x7f9fc415e850 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc4070a50 con 0x7f9fc415bf30 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 0x7f9fc4140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 0x7f9fc4140750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52264/0 (socket says 172.21.15.72:52264) 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 -- 172.21.15.72:0/2996157107 learned_addr learned my addr 172.21.15.72:0/2996157107 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9438640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc4142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fca43a640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc415e850 0x7f9fc4143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 -- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 msgr2=0x7f9fc4142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc4142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 -- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc415e850 msgr2=0x7f9fc4143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc415e850 0x7f9fc4143320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 -- 172.21.15.72:0/2996157107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc4144130 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fca43a640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc415e850 0x7f9fc4143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fc9c39640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 0x7f9fc4140750 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9fc002b880 tx=0x7f9fc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0032630 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc00327f0 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc413fc70 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0032be0 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.859+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc4144410 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.860+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc404f330 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.861+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9fc003b3f0 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.861+0000 7f9fba7fc640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9fa806b4d0 0x7f9fa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.861+0000 7f9fc9438640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9fa806b4d0 0x7f9fa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.861+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fc002f030 con 0x7f9fc413dc10 2024-03-06T07:25:20.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.862+0000 7f9fc9438640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9fa806b4d0 0x7f9fa806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009560 tx=0x7f9fb4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:20.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:20.866+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9fc0040070 con 0x7f9fc413dc10 2024-03-06T07:25:21.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.075+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fc406eb00 con 0x7f9fc413dc10 2024-03-06T07:25:21.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.075+0000 7f9fba7fc640 1 -- 172.21.15.72:0/2996157107 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f9fc007ad10 con 0x7f9fc413dc10 2024-03-06T07:25:21.078 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:21.078 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:21.078 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9fa806b4d0 msgr2=0x7f9fa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9fa806b4d0 0x7f9fa806d990 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4009560 tx=0x7f9fb4009190 comp rx=0 tx=0).stop 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 msgr2=0x7f9fc4140750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 0x7f9fc4140750 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f9fc002b880 tx=0x7f9fc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 shutdown_connections 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9fa806b4d0 0x7f9fa806d990 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fc415e850 0x7f9fc4143320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fc415bf30 0x7f9fc4142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 --2- 172.21.15.72:0/2996157107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fc413dc10 0x7f9fc4140750 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 >> 172.21.15.72:0/2996157107 conn(0x7f9fc40cf8a0 msgr2=0x7f9fc4071fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 shutdown_connections 2024-03-06T07:25:21.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.077+0000 7f9fcbe9d640 1 -- 172.21.15.72:0/2996157107 wait complete. 2024-03-06T07:25:21.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:25:21.206 INFO:teuthology.orchestra.run.smithi146.stdout:{"purge_queue":{"pq_executed":10}} 2024-03-06T07:25:21.206 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-03-06T07:25:21.207 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:21.207 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> import uuid 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> n = 10 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = False 2024-03-06T07:25:21.222 DEBUG:teuthology.orchestra.run.smithi072:> if 0 > 0: 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> create_hard_links = True 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> dpath = os.path.dirname(path) 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> fnameprefix = os.path.basename(path) 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> os.makedirs(dpath, exist_ok=True) 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> try: 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(n): 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> with open(fpath, '"'"'w'"'"') as f: 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> f.write(f"{i}") 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:25:21.223 DEBUG:teuthology.orchestra.run.smithi072:> f.flush() 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(f.fileno()) 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> os.unlink(fpath) 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> if True: 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> if create_hard_links: 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> for j in range(0): 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> if False: 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> os.fsync(dirfd) 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> finally: 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> os.close(dirfd) 2024-03-06T07:25:21.224 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:25:21.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:21.397 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:21.397 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:21.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:21.531 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:21.531 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:21.531 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:21.546 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:21.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 Processor -- start 2024-03-06T07:25:21.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 -- start start 2024-03-06T07:25:21.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0f44158400 0x7f0f440cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f440cf870 0x7f0f4406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f440cfd80 con 0x7f0f441587e0 2024-03-06T07:25:21.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4415e590 con 0x7f0f440cf870 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.684+0000 7f0f4a6f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f440d0060 con 0x7f0f44158400 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f437fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f440cf870 0x7f0f4406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f48c91640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0f441587e0 0x7f0f44161bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56520/0 (socket says 172.21.15.72:56520) 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f48c91640 1 -- 172.21.15.72:0/2871370717 learned_addr learned my addr 172.21.15.72:0/2871370717 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f43fff640 1 --2- 172.21.15.72:0/2871370717 >> v2:172.21.15.146:3300/0 conn(0x7f0f44158400 0x7f0f440cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f437fe640 1 -- 172.21.15.72:0/2871370717 >> v2:172.21.15.146:3300/0 conn(0x7f0f44158400 msgr2=0x7f0f440cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f437fe640 1 --2- 172.21.15.72:0/2871370717 >> v2:172.21.15.146:3300/0 conn(0x7f0f44158400 0x7f0f440cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f437fe640 1 -- 172.21.15.72:0/2871370717 >> v1:172.21.15.72:6789/0 conn(0x7f0f441587e0 legacy=0x7f0f44161bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f437fe640 1 -- 172.21.15.72:0/2871370717 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4415e7d0 con 0x7f0f440cf870 2024-03-06T07:25:21.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.685+0000 7f0f42ffd640 1 -- 172.21.15.72:0/2871370717 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930597943 0 0) 0x7f0f440cfd80 con 0x7f0f441587e0 2024-03-06T07:25:21.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f437fe640 1 --2- 172.21.15.72:0/2871370717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f4406afb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0f4406bbf0 tx=0x7f0f3802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401e52d58d287a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f42ffd640 1 -- 172.21.15.72:0/2871370717 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f380315a0 con 0x7f0f440cf870 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f42ffd640 1 -- 172.21.15.72:0/2871370717 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f38037070 con 0x7f0f440cf870 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f42ffd640 1 -- 172.21.15.72:0/2871370717 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f38031960 con 0x7f0f440cf870 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/2871370717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 msgr2=0x7f0f4406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/2871370717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f4406afb0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0f4406bbf0 tx=0x7f0f3802b840 comp rx=0 tx=0).stop 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/2871370717 shutdown_connections 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/2871370717 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f4406afb0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/2871370717 >> v2:172.21.15.146:3300/0 conn(0x7f0f44158400 0x7f0f440cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/2871370717 >> 172.21.15.72:0/2871370717 conn(0x7f0f4406c0c0 msgr2=0x7f0f4406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/2871370717 shutdown_connections 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.686+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/2871370717 wait complete. 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.687+0000 7f0f4a6f4640 1 Processor -- start 2024-03-06T07:25:21.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 -- start start 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f44146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f44158400 0x7f0f441e6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f441587e0 0x7f0f44071980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440cd260 con 0x7f0f440cf870 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4415ed90 con 0x7f0f44158400 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f4a6f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440cd3e0 con 0x7f0f441587e0 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f44146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f43fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f44146150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52286/0 (socket says 172.21.15.72:52286) 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f43fff640 1 -- 172.21.15.72:0/1536150875 learned_addr learned my addr 172.21.15.72:0/1536150875 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:21.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f437fe640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f44158400 0x7f0f441e6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.688+0000 7f0f48c91640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f441587e0 0x7f0f44071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f43fff640 1 -- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f441587e0 msgr2=0x7f0f44071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f43fff640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f441587e0 0x7f0f44071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f43fff640 1 -- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f44158400 msgr2=0x7f0f441e6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f43fff640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f44158400 0x7f0f441e6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f43fff640 1 -- 172.21.15.72:0/1536150875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f441e91d0 con 0x7f0f440cf870 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f48c91640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f441587e0 0x7f0f44071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f43fff640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f44146150 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400d830 tx=0x7f0f3400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400b6c0 con 0x7f0f440cf870 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f34019070 con 0x7f0f440cf870 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f441e94b0 con 0x7f0f440cf870 2024-03-06T07:25:21.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3400ba80 con 0x7f0f440cf870 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.689+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f441e9a70 con 0x7f0f440cf870 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.690+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4404f330 con 0x7f0f440cf870 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.690+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0f3401e460 con 0x7f0f440cf870 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.691+0000 7f0f40ff9640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f3006b500 0x7f0f3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.691+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0f3408a800 con 0x7f0f440cf870 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.691+0000 7f0f437fe640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f3006b500 0x7f0f3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.691+0000 7f0f437fe640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f3006b500 0x7f0f3006d9c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0f38036000 tx=0x7f0f38008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.695+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0f34059680 con 0x7f0f440cf870 2024-03-06T07:25:21.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.869+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f440ce380 con 0x7f0f440cf870 2024-03-06T07:25:21.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.870+0000 7f0f40ff9640 1 -- 172.21.15.72:0/1536150875 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f0f34058f80 con 0x7f0f440cf870 2024-03-06T07:25:21.872 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:21.872 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":71,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:53.957942+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:21.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f3006b500 msgr2=0x7f0f3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f3006b500 0x7f0f3006d9c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0f38036000 tx=0x7f0f38008030 comp rx=0 tx=0).stop 2024-03-06T07:25:21.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 msgr2=0x7f0f44146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f44146150 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400d830 tx=0x7f0f3400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 shutdown_connections 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0f3006b500 0x7f0f3006d9c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f441587e0 0x7f0f44071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f44158400 0x7f0f441e6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 --2- 172.21.15.72:0/1536150875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f440cf870 0x7f0f44146150 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 >> 172.21.15.72:0/1536150875 conn(0x7f0f4406c0c0 msgr2=0x7f0f440b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 shutdown_connections 2024-03-06T07:25:21.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.872+0000 7f0f4a6f4640 1 -- 172.21.15.72:0/1536150875 wait complete. 2024-03-06T07:25:21.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:50.525301+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:51.269562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:21.885 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:21.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.986+0000 7ffad880e640 1 Processor -- start 2024-03-06T07:25:21.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.986+0000 7ffad880e640 1 -- start start 2024-03-06T07:25:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad880e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffad41614d0 0x7ffad40d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad880e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad41618b0 con 0x7ffad40d24b0 2024-03-06T07:25:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad880e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad40cf400 con 0x7ffad40d1910 2024-03-06T07:25:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad880e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad40cf6a0 con 0x7ffad41614d0 2024-03-06T07:25:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad2d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ffad40d24b0 0x7ffad40b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56524/0 (socket says 172.21.15.72:56524) 2024-03-06T07:25:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad2d9d640 1 -- 172.21.15.72:0/3572176057 learned_addr learned my addr 172.21.15.72:0/3572176057 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad259c640 1 --2- 172.21.15.72:0/3572176057 >> v2:172.21.15.146:3300/0 conn(0x7ffad41614d0 0x7ffad40d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167913266 0 0) 0x7ffad41618b0 con 0x7ffad40d24b0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac0003520 con 0x7ffad40d24b0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957458291 0 0) 0x7ffad40cf400 con 0x7ffad40d1910 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad41618b0 con 0x7ffad40d1910 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad259c640 1 -- 172.21.15.72:0/3572176057 >> v1:172.21.15.146:6789/0 conn(0x7ffad40d1910 legacy=0x7ffad40d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939962554 0 0) 0x7ffac0003520 con 0x7ffad40d24b0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad259c640 1 -- 172.21.15.72:0/3572176057 >> v1:172.21.15.72:6789/0 conn(0x7ffad40d24b0 legacy=0x7ffad40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.987+0000 7ffad259c640 1 -- 172.21.15.72:0/3572176057 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40cf980 con 0x7ffad41614d0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.988+0000 7ffad259c640 1 --2- 172.21.15.72:0/3572176057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad41614d0 0x7ffad40d4860 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffac40097a0 tx=0x7ffac402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2484ae48738a2337 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.988+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac4038070 con 0x7ffad41614d0 2024-03-06T07:25:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.988+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac4031500 con 0x7ffad41614d0 2024-03-06T07:25:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.988+0000 7ffad1d9b640 1 -- 172.21.15.72:0/3572176057 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc0019f0 con 0x7ffad41614d0 2024-03-06T07:25:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 -- 172.21.15.72:0/3572176057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad41614d0 msgr2=0x7ffad40d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 --2- 172.21.15.72:0/3572176057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad41614d0 0x7ffad40d4860 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ffac40097a0 tx=0x7ffac402b820 comp rx=0 tx=0).stop 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 -- 172.21.15.72:0/3572176057 shutdown_connections 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 --2- 172.21.15.72:0/3572176057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad41614d0 0x7ffad40d4860 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 -- 172.21.15.72:0/3572176057 >> 172.21.15.72:0/3572176057 conn(0x7ffad40d3d40 msgr2=0x7ffad40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 -- 172.21.15.72:0/3572176057 shutdown_connections 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.989+0000 7ffad880e640 1 -- 172.21.15.72:0/3572176057 wait complete. 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.990+0000 7ffad880e640 1 Processor -- start 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.990+0000 7ffad880e640 1 -- start start 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad880e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad40d1910 0x7ffad40d41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad880e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad40d24b0 0x7ffad40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad880e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 0x7ffad4153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad880e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b1540 con 0x7ffad41614d0 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad880e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b0910 con 0x7ffad40d1910 2024-03-06T07:25:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad880e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad40b16c0 con 0x7ffad40d24b0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 0x7ffad4153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 0x7ffad4153160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52290/0 (socket says 172.21.15.72:52290) 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 -- 172.21.15.72:0/376260260 learned_addr learned my addr 172.21.15.72:0/376260260 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad259c640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad40d1910 0x7ffad40d41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffacbfff640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad40d24b0 0x7ffad40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 -- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad40d24b0 msgr2=0x7ffad40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad40d24b0 0x7ffad40cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 -- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad40d1910 msgr2=0x7ffad40d41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad40d1910 0x7ffad40d41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.991+0000 7ffad2d9d640 1 -- 172.21.15.72:0/376260260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40b8360 con 0x7ffad41614d0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.992+0000 7ffad2d9d640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 0x7ffad4153160 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ffacc001f60 tx=0x7ffacc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.992+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc017070 con 0x7ffad41614d0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.992+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc012400 con 0x7ffad41614d0 2024-03-06T07:25:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.992+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc0127d0 con 0x7ffad41614d0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.992+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad40cdb20 con 0x7ffad41614d0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.992+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad40c8db0 con 0x7ffad41614d0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.993+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad4068ff0 con 0x7ffad41614d0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.993+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffacc01a070 con 0x7ffad41614d0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.993+0000 7ffacaffd640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab406b4d0 0x7ffab406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.994+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffacc08a120 con 0x7ffad41614d0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.994+0000 7ffad259c640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab406b4d0 0x7ffab406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.994+0000 7ffad259c640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab406b4d0 0x7ffab406d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffac4035000 tx=0x7ffac4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:21.998+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffad40cdb20 con 0x7ffad41614d0 2024-03-06T07:25:22.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.168+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffad40d0e30 con 0x7ffad41614d0 2024-03-06T07:25:22.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.169+0000 7ffacaffd640 1 -- 172.21.15.72:0/376260260 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffad40b8360 con 0x7ffad41614d0 2024-03-06T07:25:22.171 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:25:22.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab406b4d0 msgr2=0x7ffab406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab406b4d0 0x7ffab406d990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ffac4035000 tx=0x7ffac4008030 comp rx=0 tx=0).stop 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 msgr2=0x7ffad4153160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 0x7ffad4153160 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ffacc001f60 tx=0x7ffacc00a430 comp rx=0 tx=0).stop 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 shutdown_connections 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab406b4d0 0x7ffab406d990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad41614d0 0x7ffad4153160 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad40d24b0 0x7ffad40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 --2- 172.21.15.72:0/376260260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad40d1910 0x7ffad40d41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 >> 172.21.15.72:0/376260260 conn(0x7ffad40d3d40 msgr2=0x7ffad40ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 shutdown_connections 2024-03-06T07:25:22.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.171+0000 7ffad880e640 1 -- 172.21.15.72:0/376260260 wait complete. 2024-03-06T07:25:22.184 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:22.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.285+0000 7fc3f0a0e640 1 Processor -- start 2024-03-06T07:25:22.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.285+0000 7fc3f0a0e640 1 -- start start 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3f0a0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3ec0cf9a0 0x7fc3ec0b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3f0a0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3f0a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ec06b710 con 0x7fc3ec0b5410 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3f0a0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ec06b150 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3f0a0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec06b430 con 0x7fc3ec066d90 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3ec0cf9a0 0x7fc3ec0b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3e9d9b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3ec0cf9a0 0x7fc3ec0b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52304/0 (socket says 172.21.15.72:52304) 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 -- 172.21.15.72:0/258385641 learned_addr learned my addr 172.21.15.72:0/258385641 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 -- 172.21.15.72:0/258385641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 msgr2=0x7fc3ec0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 --2- 172.21.15.72:0/258385641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 -- 172.21.15.72:0/258385641 >> v1:172.21.15.146:6789/0 conn(0x7fc3ec0b5410 legacy=0x7fc3ec072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3e959a640 1 -- 172.21.15.72:0/258385641 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650160340 0 0) 0x7fc3ec06b710 con 0x7fc3ec0b5410 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3ea59c640 1 -- 172.21.15.72:0/258385641 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec0b0970 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.286+0000 7fc3e9d9b640 1 --2- 172.21.15.72:0/258385641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.287+0000 7fc3ea59c640 1 --2- 172.21.15.72:0/258385641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec0b5ab0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009870 tx=0x7fc3dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87bcbcdcd167009 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.287+0000 7fc3e959a640 1 -- 172.21.15.72:0/258385641 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc034070 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.287+0000 7fc3e959a640 1 -- 172.21.15.72:0/258385641 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3dc030840 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.287+0000 7fc3e959a640 1 -- 172.21.15.72:0/258385641 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc030c10 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/258385641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 msgr2=0x7fc3ec0b5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/258385641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec0b5ab0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc009870 tx=0x7fc3dc02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/258385641 shutdown_connections 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/258385641 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/258385641 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec0b5ab0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/258385641 >> 172.21.15.72:0/258385641 conn(0x7fc3ec0b1d70 msgr2=0x7fc3ec0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/258385641 shutdown_connections 2024-03-06T07:25:22.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.288+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/258385641 wait complete. 2024-03-06T07:25:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.289+0000 7fc3f0a0e640 1 Processor -- start 2024-03-06T07:25:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.289+0000 7fc3f0a0e640 1 -- start start 2024-03-06T07:25:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3f0a0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3f0a0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3ec0b5410 0x7fc3ec15cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3f0a0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec15d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3f0a0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0b6170 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3f0a0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0b5ff0 con 0x7fc3ec0b5410 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3f0a0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec0cf050 con 0x7fc3ec066d90 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec15d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec15d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52312/0 (socket says 172.21.15.72:52312) 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 -- 172.21.15.72:0/2018421347 learned_addr learned my addr 172.21.15.72:0/2018421347 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3e9d9b640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3ec0b5410 0x7fc3ec15cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ea59c640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 -- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 msgr2=0x7fc3ec1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 -- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3ec0b5410 msgr2=0x7fc3ec15cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3ec0b5410 0x7fc3ec15cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 -- 172.21.15.72:0/2018421347 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec1f7410 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.290+0000 7fc3ead9d640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec15d590 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c790 tx=0x7fc3e400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.291+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4003da0 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.291+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4005070 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.291+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec1f4420 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.291+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4005460 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.291+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec1f49e0 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.292+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec1f75e0 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.292+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3e4005600 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.292+0000 7fc3daffd640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.292+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3e4089f80 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.292+0000 7fc3ea59c640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.293+0000 7fc3ea59c640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc036000 tx=0x7fc3dc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.297+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3e40587b0 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.466+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3ec0d0150 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.466+0000 7fc3daffd640 1 -- 172.21.15.72:0/2018421347 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc3e405c850 con 0x7fc3ec0cf9a0 2024-03-06T07:25:22.469 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3cc06b4d0 msgr2=0x7fc3cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc036000 tx=0x7fc3dc03c040 comp rx=0 tx=0).stop 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 msgr2=0x7fc3ec15d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec15d590 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c790 tx=0x7fc3e400cc50 comp rx=0 tx=0).stop 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 shutdown_connections 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3cc06b4d0 0x7fc3cc06d990 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec0cf9a0 0x7fc3ec15d590 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3ec0b5410 0x7fc3ec15cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 --2- 172.21.15.72:0/2018421347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3ec066d90 0x7fc3ec1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 >> 172.21.15.72:0/2018421347 conn(0x7fc3ec0b1d70 msgr2=0x7fc3ec0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 shutdown_connections 2024-03-06T07:25:22.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.468+0000 7fc3f0a0e640 1 -- 172.21.15.72:0/2018421347 wait complete. 2024-03-06T07:25:22.481 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:22.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.581+0000 7fc31c73e640 1 Processor -- start 2024-03-06T07:25:22.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.581+0000 7fc31c73e640 1 -- start start 2024-03-06T07:25:22.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.582+0000 7fc31c73e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3140d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.582+0000 7fc31c73e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc3140d2480 0x7fc3140b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.582+0000 7fc31c73e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3141618b0 con 0x7fc3140d1910 2024-03-06T07:25:22.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.582+0000 7fc31c73e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3140cf400 con 0x7fc3140d2480 2024-03-06T07:25:22.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.582+0000 7fc31c73e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140cf6a0 con 0x7fc3141614d0 2024-03-06T07:25:22.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc3140d2480 0x7fc3140b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31a4da640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3140d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc3140d2480 0x7fc3140b0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:35112/0 (socket says 172.21.15.72:35112) 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 -- 172.21.15.72:0/2918376765 learned_addr learned my addr 172.21.15.72:0/2918376765 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc319cd9640 1 --1- 172.21.15.72:0/2918376765 >> v1:172.21.15.146:6789/0 conn(0x7fc3140d1910 0x7fc3140d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:42752/0 (socket says 172.21.15.72:42752) 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 -- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 msgr2=0x7fc3140d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 --2- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3140d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 -- 172.21.15.72:0/2918376765 >> v1:172.21.15.146:6789/0 conn(0x7fc3140d1910 legacy=0x7fc3140d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc3194d8640 1 -- 172.21.15.72:0/2918376765 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012208755 0 0) 0x7fc3141618b0 con 0x7fc3140d1910 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31acdb640 1 -- 172.21.15.72:0/2918376765 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140cf980 con 0x7fc3140d2480 2024-03-06T07:25:22.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.583+0000 7fc31a4da640 1 --2- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3140d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc31acdb640 1 --2- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140b0e30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc304009e50 tx=0x7fc30402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6e461465c0947e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc3194d8640 1 -- 172.21.15.72:0/2918376765 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30402cb00 con 0x7fc3140d2480 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc3194d8640 1 -- 172.21.15.72:0/2918376765 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30402cca0 con 0x7fc3140d2480 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc3194d8640 1 -- 172.21.15.72:0/2918376765 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc304034610 con 0x7fc3140d2480 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc31c73e640 1 -- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 msgr2=0x7fc3140b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc31c73e640 1 --2- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140b0e30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc304009e50 tx=0x7fc30402bf60 comp rx=0 tx=0).stop 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc31c73e640 1 -- 172.21.15.72:0/2918376765 shutdown_connections 2024-03-06T07:25:22.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc31c73e640 1 --2- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140b0e30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.584+0000 7fc31c73e640 1 --2- 172.21.15.72:0/2918376765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3140d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.585+0000 7fc31c73e640 1 -- 172.21.15.72:0/2918376765 >> 172.21.15.72:0/2918376765 conn(0x7fc3140d3d40 msgr2=0x7fc3140d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.585+0000 7fc31c73e640 1 -- 172.21.15.72:0/2918376765 shutdown_connections 2024-03-06T07:25:22.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.585+0000 7fc31c73e640 1 -- 172.21.15.72:0/2918376765 wait complete. 2024-03-06T07:25:22.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 Processor -- start 2024-03-06T07:25:22.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 -- start start 2024-03-06T07:25:22.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 0x7fc3141f7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3141f1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140b1370 con 0x7fc3140d1910 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140b0910 con 0x7fc3140d2480 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31c73e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3140b14f0 con 0x7fc3141614d0 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.586+0000 7fc31a4da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 0x7fc3141f7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 0x7fc3141f7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58812/0 (socket says 172.21.15.72:58812) 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 -- 172.21.15.72:0/1772543361 learned_addr learned my addr 172.21.15.72:0/1772543361 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc319cd9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31acdb640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3141f1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 -- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 msgr2=0x7fc3141f1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3141f1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 -- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 msgr2=0x7fc3140cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 -- 172.21.15.72:0/1772543361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3140b8010 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31a4da640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 0x7fc3141f7d10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00e960 tx=0x7fc30c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c017070 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30c00ca00 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3140b82f0 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30c00cdd0 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.587+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3141f7510 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.588+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc314068ff0 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.588+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc30c01a070 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.589+0000 7fc312ffd640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30006b4d0 0x7fc30006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.589+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc30c08a3b0 con 0x7fc3140d1910 2024-03-06T07:25:22.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.589+0000 7fc319cd9640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30006b4d0 0x7fc30006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.589+0000 7fc319cd9640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30006b4d0 0x7fc30006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc308000f60 tx=0x7fc308007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.593+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc30c058be0 con 0x7fc3140d1910 2024-03-06T07:25:22.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.765+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3140d1f80 con 0x7fc3140d1910 2024-03-06T07:25:22.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.765+0000 7fc312ffd640 1 -- 172.21.15.72:0/1772543361 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc30c05cc80 con 0x7fc3140d1910 2024-03-06T07:25:22.768 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:25:22.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30006b4d0 msgr2=0x7fc30006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30006b4d0 0x7fc30006d990 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc308000f60 tx=0x7fc308007d50 comp rx=0 tx=0).stop 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 msgr2=0x7fc3141f7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 0x7fc3141f7d10 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fc30c00e960 tx=0x7fc30c00ee20 comp rx=0 tx=0).stop 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 shutdown_connections 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30006b4d0 0x7fc30006d990 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3141614d0 0x7fc3141f1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3140d2480 0x7fc3140cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 --2- 172.21.15.72:0/1772543361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3140d1910 0x7fc3141f7d10 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 >> 172.21.15.72:0/1772543361 conn(0x7fc3140d3d40 msgr2=0x7fc3140cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 shutdown_connections 2024-03-06T07:25:22.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.767+0000 7fc31c73e640 1 -- 172.21.15.72:0/1772543361 wait complete. 2024-03-06T07:25:22.780 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:22.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.878+0000 7f86613d7640 1 Processor -- start 2024-03-06T07:25:22.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.878+0000 7f86613d7640 1 -- start start 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f86613d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f865c0b4fe0 0x7f865c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f86613d7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f865c15ba50 0x7f865c15bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f86613d7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f865c15d510 con 0x7f865c0b4c00 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f86613d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c15e0b0 con 0x7f865c0b4fe0 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f86613d7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f865c15d780 con 0x7f865c15ba50 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f865a7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f865c0b4fe0 0x7f865c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f865a7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f865c0b4fe0 0x7f865c0b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58828/0 (socket says 172.21.15.72:58828) 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f865a7fc640 1 -- 172.21.15.72:0/1170221057 learned_addr learned my addr 172.21.15.72:0/1170221057 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:22.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.879+0000 7f865b7fe640 1 --2- 172.21.15.72:0/1170221057 >> v2:172.21.15.146:3300/0 conn(0x7f865c15ba50 0x7f865c15bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.880+0000 7f865a7fc640 1 -- 172.21.15.72:0/1170221057 >> v2:172.21.15.146:3300/0 conn(0x7f865c15ba50 msgr2=0x7f865c15bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:22.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.880+0000 7f865a7fc640 1 --2- 172.21.15.72:0/1170221057 >> v2:172.21.15.146:3300/0 conn(0x7f865c15ba50 0x7f865c15bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.880+0000 7f865a7fc640 1 -- 172.21.15.72:0/1170221057 >> v1:172.21.15.146:6789/0 conn(0x7f865c0b4c00 legacy=0x7f865c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.880+0000 7f865a7fc640 1 -- 172.21.15.72:0/1170221057 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c15da60 con 0x7f865c0b4fe0 2024-03-06T07:25:22.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.880+0000 7f865a7fc640 1 --2- 172.21.15.72:0/1170221057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4fe0 0x7f865c0b2fb0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f865c06b930 tx=0x7f865002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3489f2d3c7236679 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f8659ffb640 1 -- 172.21.15.72:0/1170221057 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865002f850 con 0x7f865c0b4fe0 2024-03-06T07:25:22.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f8659ffb640 1 -- 172.21.15.72:0/1170221057 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865002f9f0 con 0x7f865c0b4fe0 2024-03-06T07:25:22.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f8659ffb640 1 -- 172.21.15.72:0/1170221057 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865002fdc0 con 0x7f865c0b4fe0 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 -- 172.21.15.72:0/1170221057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4fe0 msgr2=0x7f865c0b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 --2- 172.21.15.72:0/1170221057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4fe0 0x7f865c0b2fb0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f865c06b930 tx=0x7f865002ba30 comp rx=0 tx=0).stop 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 -- 172.21.15.72:0/1170221057 shutdown_connections 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 --2- 172.21.15.72:0/1170221057 >> v2:172.21.15.146:3300/0 conn(0x7f865c15ba50 0x7f865c15bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 --2- 172.21.15.72:0/1170221057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4fe0 0x7f865c0b2fb0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 -- 172.21.15.72:0/1170221057 >> 172.21.15.72:0/1170221057 conn(0x7f865c09a9f0 msgr2=0x7f865c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 -- 172.21.15.72:0/1170221057 shutdown_connections 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.881+0000 7f86613d7640 1 -- 172.21.15.72:0/1170221057 wait complete. 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.882+0000 7f86613d7640 1 Processor -- start 2024-03-06T07:25:22.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.882+0000 7f86613d7640 1 -- start start 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f86613d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 0x7f865c14ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f86613d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f865c0b4fe0 0x7f865c1f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f86613d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f865c15ba50 0x7f865c154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f86613d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c15f6d0 con 0x7f865c0b4c00 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f86613d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865c15f550 con 0x7f865c15ba50 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f86613d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f865c15c400 con 0x7f865c0b4fe0 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 0x7f865c14ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 0x7f865c14ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58844/0 (socket says 172.21.15.72:58844) 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 -- 172.21.15.72:0/2797498398 learned_addr learned my addr 172.21.15.72:0/2797498398 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865b7fe640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f865c15ba50 0x7f865c154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865a7fc640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f865c0b4fe0 0x7f865c1f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 -- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f865c0b4fe0 msgr2=0x7f865c1f2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f865c0b4fe0 0x7f865c1f2000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 -- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f865c15ba50 msgr2=0x7f865c154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.883+0000 7f865affd640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f865c15ba50 0x7f865c154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f865affd640 1 -- 172.21.15.72:0/2797498398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c153770 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f865affd640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 0x7f865c14ece0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c790 tx=0x7f864c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c016020 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c006d50 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865c153a50 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c003bb0 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.884+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865c154010 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.885+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865c04f330 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.885+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f864c00a850 con 0x7f865c0b4c00 2024-03-06T07:25:22.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.886+0000 7f86437fe640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f864806b4d0 0x7f864806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:22.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.886+0000 7f865a7fc640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f864806b4d0 0x7f864806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:22.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.886+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f864c089d70 con 0x7f865c0b4c00 2024-03-06T07:25:22.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.886+0000 7f865a7fc640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f864806b4d0 0x7f864806d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8650036000 tx=0x7f8650008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:22.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:22.890+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f864c020020 con 0x7f865c0b4c00 2024-03-06T07:25:23.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.074+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f865c09b650 con 0x7f865c0b4c00 2024-03-06T07:25:23.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.074+0000 7f86437fe640 1 -- 172.21.15.72:0/2797498398 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v71) v1 ==== 80+0+0 (secure 0 0 0) 0x7f864c058610 con 0x7f865c0b4c00 2024-03-06T07:25:23.076 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:25:23.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f864806b4d0 msgr2=0x7f864806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f864806b4d0 0x7f864806d990 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8650036000 tx=0x7f8650008030 comp rx=0 tx=0).stop 2024-03-06T07:25:23.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 msgr2=0x7f865c14ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 0x7f865c14ece0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c790 tx=0x7f864c00cc50 comp rx=0 tx=0).stop 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 shutdown_connections 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f864806b4d0 0x7f864806d990 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f865c15ba50 0x7f865c154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f865c0b4fe0 0x7f865c1f2000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 --2- 172.21.15.72:0/2797498398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865c0b4c00 0x7f865c14ece0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 >> 172.21.15.72:0/2797498398 conn(0x7f865c09a9f0 msgr2=0x7f865c158af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 shutdown_connections 2024-03-06T07:25:23.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.076+0000 7f86613d7640 1 -- 172.21.15.72:0/2797498398 wait complete. 2024-03-06T07:25:23.088 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:25:23.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:23.089 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:23.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[62400]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:23.116 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:25:23.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:23.116 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:23.224 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:25:23.225 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:23.240 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:23.240 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:25:23.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:23.241 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:23.295 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:23.295 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:23.295 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:23.311 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:23.311 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:25:23.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:23.311 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:23.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:23.365 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:23.365 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:25:23.365 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:23.380 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:23.380 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:25:23.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:23.381 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:23.435 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:23.435 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:23.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.533+0000 7fb4833ee640 1 Processor -- start 2024-03-06T07:25:23.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.533+0000 7fb4833ee640 1 -- start start 2024-03-06T07:25:23.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb4833ee640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb47c0cce40 0x7fb47c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb4833ee640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c071f40 con 0x7fb47c1515e0 2024-03-06T07:25:23.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb4833ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c070a70 con 0x7fb47c0cd7a0 2024-03-06T07:25:23.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb4833ee640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0cd2f0 con 0x7fb47c0cce40 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb480989640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb47c0cce40 0x7fb47c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb48118a640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb47c1515e0 0x7fb47c071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36720/0 (socket says 172.21.15.72:36720) 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb48118a640 1 -- 172.21.15.72:0/719491442 learned_addr learned my addr 172.21.15.72:0/719491442 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb480989640 1 -- 172.21.15.72:0/719491442 >> v1:172.21.15.146:6789/0 conn(0x7fb47c0cd7a0 legacy=0x7fb47c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb480989640 1 -- 172.21.15.72:0/719491442 >> v1:172.21.15.72:6789/0 conn(0x7fb47c1515e0 legacy=0x7fb47c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb473fff640 1 -- 172.21.15.72:0/719491442 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514324888 0 0) 0x7fb47c071f40 con 0x7fb47c1515e0 2024-03-06T07:25:23.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.534+0000 7fb480989640 1 -- 172.21.15.72:0/719491442 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c070d50 con 0x7fb47c0cce40 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb480989640 1 --2- 172.21.15.72:0/719491442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c15b430 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0097a0 tx=0x7fb46c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b0d9480c516210 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb473fff640 1 -- 172.21.15.72:0/719491442 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c035020 con 0x7fb47c0cce40 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb473fff640 1 -- 172.21.15.72:0/719491442 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c031830 con 0x7fb47c0cce40 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb473fff640 1 -- 172.21.15.72:0/719491442 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c031c00 con 0x7fb47c0cce40 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb4833ee640 1 -- 172.21.15.72:0/719491442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 msgr2=0x7fb47c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb4833ee640 1 --2- 172.21.15.72:0/719491442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c15b430 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0097a0 tx=0x7fb46c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb4833ee640 1 -- 172.21.15.72:0/719491442 shutdown_connections 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.535+0000 7fb4833ee640 1 --2- 172.21.15.72:0/719491442 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c15b430 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.536+0000 7fb4833ee640 1 -- 172.21.15.72:0/719491442 >> 172.21.15.72:0/719491442 conn(0x7fb47c0cf8a0 msgr2=0x7fb47c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.536+0000 7fb4833ee640 1 -- 172.21.15.72:0/719491442 shutdown_connections 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.536+0000 7fb4833ee640 1 -- 172.21.15.72:0/719491442 wait complete. 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 Processor -- start 2024-03-06T07:25:23.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 -- start start 2024-03-06T07:25:23.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c1443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb47c0cd7a0 0x7fb47c0c5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb47c1515e0 0x7fb47c1593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b3f90 con 0x7fb47c0cce40 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c071310 con 0x7fb47c0cd7a0 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb4833ee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0b4110 con 0x7fb47c1515e0 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.537+0000 7fb48118a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c1443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c1443f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58848/0 (socket says 172.21.15.72:58848) 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 -- 172.21.15.72:0/3839351278 learned_addr learned my addr 172.21.15.72:0/3839351278 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb480989640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb47c0cd7a0 0x7fb47c0c5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48198b640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb47c1515e0 0x7fb47c1593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 -- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb47c1515e0 msgr2=0x7fb47c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb47c1515e0 0x7fb47c1593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 -- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb47c0cd7a0 msgr2=0x7fb47c0c5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb47c0cd7a0 0x7fb47c0c5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 -- 172.21.15.72:0/3839351278 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c1f4180 con 0x7fb47c0cce40 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48198b640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb47c1515e0 0x7fb47c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb48118a640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c1443f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb47800c430 tx=0x7fb47800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478017070 con 0x7fb47c0cce40 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb478005ce0 con 0x7fb47c0cce40 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478007310 con 0x7fb47c0cce40 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c1f4420 con 0x7fb47c0cce40 2024-03-06T07:25:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.538+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c1f49e0 con 0x7fb47c0cce40 2024-03-06T07:25:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.539+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c068840 con 0x7fb47c0cce40 2024-03-06T07:25:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.539+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb47801a070 con 0x7fb47c0cce40 2024-03-06T07:25:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.540+0000 7fb471ffb640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb46006b4d0 0x7fb46006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.540+0000 7fb480989640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb46006b4d0 0x7fb46006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.540+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb478089570 con 0x7fb47c0cce40 2024-03-06T07:25:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.540+0000 7fb480989640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb46006b4d0 0x7fb46006d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb46c036000 tx=0x7fb46c03e040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.544+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb478057da0 con 0x7fb47c0cce40 2024-03-06T07:25:23.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.757+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb47c04f330 con 0x7fb47c0cce40 2024-03-06T07:25:23.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.757+0000 7fb471ffb640 1 -- 172.21.15.72:0/3839351278 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fb47805be40 con 0x7fb47c0cce40 2024-03-06T07:25:23.759 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:23.760 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:23.760 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb46006b4d0 msgr2=0x7fb46006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb46006b4d0 0x7fb46006d990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb46c036000 tx=0x7fb46c03e040 comp rx=0 tx=0).stop 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 msgr2=0x7fb47c1443f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c1443f0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fb47800c430 tx=0x7fb47800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 shutdown_connections 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb46006b4d0 0x7fb46006d990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb47c1515e0 0x7fb47c1593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb47c0cd7a0 0x7fb47c0c5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 --2- 172.21.15.72:0/3839351278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb47c0cce40 0x7fb47c1443f0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 >> 172.21.15.72:0/3839351278 conn(0x7fb47c0cf8a0 msgr2=0x7fb47c150870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.759+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 shutdown_connections 2024-03-06T07:25:23.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.760+0000 7fb4833ee640 1 -- 172.21.15.72:0/3839351278 wait complete. 2024-03-06T07:25:23.773 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:23.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 Processor -- start 2024-03-06T07:25:23.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 -- start start 2024-03-06T07:25:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15e1d0 0x7f024c0d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f024c0cfb50 0x7f024c0d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024c0d18d0 con 0x7f024c15ddf0 2024-03-06T07:25:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024c0d27c0 con 0x7f024c0cfb50 2024-03-06T07:25:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.870+0000 7f0251266640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0b7b60 con 0x7f024c15e1d0 2024-03-06T07:25:23.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f024affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f024c15ddf0 0x7f024c0cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36736/0 (socket says 172.21.15.72:36736) 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f024affd640 1 -- 172.21.15.72:0/3192662561 learned_addr learned my addr 172.21.15.72:0/3192662561 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f02427fc640 1 --2- 172.21.15.72:0/3192662561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15e1d0 0x7f024c0d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f024b7fe640 1 --2- 172.21.15.72:0/3192662561 >> v2:172.21.15.146:3300/0 conn(0x7f024c0cfb50 0x7f024c0d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f024a7fc640 1 -- 172.21.15.72:0/3192662561 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751727516 0 0) 0x7f024c0d18d0 con 0x7f024c15ddf0 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f024a7fc640 1 -- 172.21.15.72:0/3192662561 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0238003520 con 0x7f024c15ddf0 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f02427fc640 1 -- 172.21.15.72:0/3192662561 >> v2:172.21.15.146:3300/0 conn(0x7f024c0cfb50 msgr2=0x7f024c0d0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f02427fc640 1 --2- 172.21.15.72:0/3192662561 >> v2:172.21.15.146:3300/0 conn(0x7f024c0cfb50 0x7f024c0d0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f02427fc640 1 -- 172.21.15.72:0/3192662561 >> v1:172.21.15.72:6789/0 conn(0x7f024c15ddf0 legacy=0x7f024c0cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f02427fc640 1 -- 172.21.15.72:0/3192662561 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0d1bb0 con 0x7f024c15e1d0 2024-03-06T07:25:23.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.871+0000 7f024b7fe640 1 --2- 172.21.15.72:0/3192662561 >> v2:172.21.15.146:3300/0 conn(0x7f024c0cfb50 0x7f024c0d0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:23.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.872+0000 7f02427fc640 1 --2- 172.21.15.72:0/3192662561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15e1d0 0x7f024c0d06d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f02340097a0 tx=0x7f023402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77f7c56e77dd346 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.872+0000 7f024a7fc640 1 -- 172.21.15.72:0/3192662561 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234035020 con 0x7f024c15e1d0 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.872+0000 7f024a7fc640 1 -- 172.21.15.72:0/3192662561 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0234031830 con 0x7f024c15e1d0 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.872+0000 7f024a7fc640 1 -- 172.21.15.72:0/3192662561 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234031c00 con 0x7f024c15e1d0 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.872+0000 7f0251266640 1 -- 172.21.15.72:0/3192662561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15e1d0 msgr2=0x7f024c0d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.872+0000 7f0251266640 1 --2- 172.21.15.72:0/3192662561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15e1d0 0x7f024c0d06d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f02340097a0 tx=0x7f023402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.873+0000 7f0251266640 1 -- 172.21.15.72:0/3192662561 shutdown_connections 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.873+0000 7f0251266640 1 --2- 172.21.15.72:0/3192662561 >> v2:172.21.15.146:3300/0 conn(0x7f024c0cfb50 0x7f024c0d0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.873+0000 7f0251266640 1 --2- 172.21.15.72:0/3192662561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15e1d0 0x7f024c0d06d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.873+0000 7f0251266640 1 -- 172.21.15.72:0/3192662561 >> 172.21.15.72:0/3192662561 conn(0x7f024c09cff0 msgr2=0x7f024c0d3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.873+0000 7f0251266640 1 -- 172.21.15.72:0/3192662561 shutdown_connections 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.873+0000 7f0251266640 1 -- 172.21.15.72:0/3192662561 wait complete. 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.874+0000 7f0251266640 1 Processor -- start 2024-03-06T07:25:23.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.874+0000 7f0251266640 1 -- start start 2024-03-06T07:25:23.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.874+0000 7f0251266640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f024c0cfb50 0x7f024c0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.874+0000 7f0251266640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15ddf0 0x7f024c160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0251266640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 0x7f024c15e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0251266640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0d2450 con 0x7f024c15e1d0 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0251266640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0d2170 con 0x7f024c0cfb50 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0251266640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024c0d2c50 con 0x7f024c15ddf0 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 0x7f024c15e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 0x7f024c15e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58862/0 (socket says 172.21.15.72:58862) 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 -- 172.21.15.72:0/90202127 learned_addr learned my addr 172.21.15.72:0/90202127 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f02427fc640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15ddf0 0x7f024c160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024affd640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f024c0cfb50 0x7f024c0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 -- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15ddf0 msgr2=0x7f024c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15ddf0 0x7f024c160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 -- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f024c0cfb50 msgr2=0x7f024c0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f024c0cfb50 0x7f024c0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 -- 172.21.15.72:0/90202127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024c0ccf00 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f02427fc640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15ddf0 0x7f024c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f024b7fe640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 0x7f024c15e5b0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f024400efb0 tx=0x7f024400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244018020 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0244014480 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024c0cd1e0 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0244014850 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.875+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024c0cd7a0 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.876+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f024c04f330 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.877+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0244005d10 con 0x7f024c15e1d0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.877+0000 7f0243fff640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f022c06b4d0 0x7f022c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.877+0000 7f024affd640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f022c06b4d0 0x7f022c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:23.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.877+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f024408af40 con 0x7f024c15e1d0 2024-03-06T07:25:23.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.878+0000 7f024affd640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f022c06b4d0 0x7f022c06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f023c005880 tx=0x7f023c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:23.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:23.881+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0244022030 con 0x7f024c15e1d0 2024-03-06T07:25:24.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.058+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f024c15d600 con 0x7f024c15e1d0 2024-03-06T07:25:24.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.059+0000 7f0243fff640 1 -- 172.21.15.72:0/90202127 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f0244010040 con 0x7f024c15e1d0 2024-03-06T07:25:24.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:24.061 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":71,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:53.957942+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f022c06b4d0 msgr2=0x7f022c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f022c06b4d0 0x7f022c06d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f023c005880 tx=0x7f023c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 msgr2=0x7f024c15e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 0x7f024c15e5b0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f024400efb0 tx=0x7f024400c490 comp rx=0 tx=0).stop 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 shutdown_connections 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f022c06b4d0 0x7f022c06d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f024c15e1d0 0x7f024c15e5b0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f024c15ddf0 0x7f024c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 --2- 172.21.15.72:0/90202127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f024c0cfb50 0x7f024c0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 >> 172.21.15.72:0/90202127 conn(0x7f024c09cff0 msgr2=0x7f024c0d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 shutdown_connections 2024-03-06T07:25:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.061+0000 7f0251266640 1 -- 172.21.15.72:0/90202127 wait complete. 2024-03-06T07:25:24.074 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:24:50.525301+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:24:51.269562+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:24.075 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:24.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.175+0000 7fd08a3bc640 1 Processor -- start 2024-03-06T07:25:24.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.175+0000 7fd08a3bc640 1 -- start start 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd08a3bc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd08415bb40 0x7fd0840b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd08a3bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd08415f7d0 0x7fd08406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd08a3bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08415ff90 con 0x7fd08406d5c0 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd08a3bc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08406c910 con 0x7fd08415bb40 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd08a3bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd084160270 con 0x7fd08415f7d0 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd08415bb40 0x7fd0840b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd08415bb40 0x7fd0840b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58864/0 (socket says 172.21.15.72:58864) 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 -- 172.21.15.72:0/2896478557 learned_addr learned my addr 172.21.15.72:0/2896478557 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd0837fe640 1 --2- 172.21.15.72:0/2896478557 >> v2:172.21.15.146:3300/0 conn(0x7fd08415f7d0 0x7fd08406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 -- 172.21.15.72:0/2896478557 >> v2:172.21.15.146:3300/0 conn(0x7fd08415f7d0 msgr2=0x7fd08406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 --2- 172.21.15.72:0/2896478557 >> v2:172.21.15.146:3300/0 conn(0x7fd08415f7d0 0x7fd08406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 -- 172.21.15.72:0/2896478557 >> v1:172.21.15.146:6789/0 conn(0x7fd08406d5c0 legacy=0x7fd0840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.176+0000 7fd083fff640 1 -- 172.21.15.72:0/2896478557 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd084160550 con 0x7fd08415bb40 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd083fff640 1 --2- 172.21.15.72:0/2896478557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08415bb40 0x7fd0840b4970 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd08406bfe0 tx=0x7fd07402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=117be2c972dcfe55 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd082ffd640 1 -- 172.21.15.72:0/2896478557 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074034070 con 0x7fd08415bb40 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd082ffd640 1 -- 172.21.15.72:0/2896478557 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074030860 con 0x7fd08415bb40 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd082ffd640 1 -- 172.21.15.72:0/2896478557 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074030c30 con 0x7fd08415bb40 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd08a3bc640 1 -- 172.21.15.72:0/2896478557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08415bb40 msgr2=0x7fd0840b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/2896478557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08415bb40 0x7fd0840b4970 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd08406bfe0 tx=0x7fd07402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd08a3bc640 1 -- 172.21.15.72:0/2896478557 shutdown_connections 2024-03-06T07:25:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/2896478557 >> v2:172.21.15.146:3300/0 conn(0x7fd08415f7d0 0x7fd08406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/2896478557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08415bb40 0x7fd0840b4970 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.177+0000 7fd08a3bc640 1 -- 172.21.15.72:0/2896478557 >> 172.21.15.72:0/2896478557 conn(0x7fd08406b190 msgr2=0x7fd08406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.178+0000 7fd08a3bc640 1 -- 172.21.15.72:0/2896478557 shutdown_connections 2024-03-06T07:25:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.178+0000 7fd08a3bc640 1 -- 172.21.15.72:0/2896478557 wait complete. 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 Processor -- start 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 -- start start 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 0x7fd08415e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08415bb40 0x7fd0841e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08415f7d0 0x7fd0841e6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08415bf20 con 0x7fd08406d5c0 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08415fc30 con 0x7fd08415bb40 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.179+0000 7fd08a3bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08415c0a0 con 0x7fd08415f7d0 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 0x7fd08415e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 0x7fd08415e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58868/0 (socket says 172.21.15.72:58868) 2024-03-06T07:25:24.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 -- 172.21.15.72:0/4206727369 learned_addr learned my addr 172.21.15.72:0/4206727369 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd0837fe640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08415bb40 0x7fd0841e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd088959640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08415f7d0 0x7fd0841e6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 -- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08415f7d0 msgr2=0x7fd0841e6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08415f7d0 0x7fd0841e6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 -- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08415bb40 msgr2=0x7fd0841e5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08415bb40 0x7fd0841e5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 -- 172.21.15.72:0/4206727369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0841518d0 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd083fff640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 0x7fd08415e230 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd074000c00 tx=0x7fd074004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074048070 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074030660 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0841e6d60 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074034040 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.180+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd084158ca0 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.181+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0841e6a80 con 0x7fd08406d5c0 2024-03-06T07:25:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.182+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd07404b070 con 0x7fd08406d5c0 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.182+0000 7fd080ff9640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd07006b4d0 0x7fd07006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.182+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0740ac4a0 con 0x7fd08406d5c0 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.182+0000 7fd0837fe640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd07006b4d0 0x7fd07006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.182+0000 7fd0837fe640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd07006b4d0 0x7fd07006d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd084153be0 tx=0x7fd078008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.186+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd07407adf0 con 0x7fd08406d5c0 2024-03-06T07:25:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.358+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0840732b0 con 0x7fd08406d5c0 2024-03-06T07:25:24.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.359+0000 7fd080ff9640 1 -- 172.21.15.72:0/4206727369 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd07407ee90 con 0x7fd08406d5c0 2024-03-06T07:25:24.361 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd07006b4d0 msgr2=0x7fd07006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd07006b4d0 0x7fd07006d990 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd084153be0 tx=0x7fd078008040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 msgr2=0x7fd08415e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 0x7fd08415e230 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd074000c00 tx=0x7fd074004730 comp rx=0 tx=0).stop 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 shutdown_connections 2024-03-06T07:25:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd07006b4d0 0x7fd07006d990 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd08415f7d0 0x7fd0841e6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd08415bb40 0x7fd0841e5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 --2- 172.21.15.72:0/4206727369 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd08406d5c0 0x7fd08415e230 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 >> 172.21.15.72:0/4206727369 conn(0x7fd08406b190 msgr2=0x7fd0840b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 shutdown_connections 2024-03-06T07:25:24.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.361+0000 7fd08a3bc640 1 -- 172.21.15.72:0/4206727369 wait complete. 2024-03-06T07:25:24.374 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:24.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.474+0000 7fdadf2c2640 1 Processor -- start 2024-03-06T07:25:24.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadf2c2640 1 -- start start 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadf2c2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad80d4860 0x7fdad80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadf2c2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdad80b0e30 0x7fdad80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadf2c2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad81618b0 con 0x7fdad81614d0 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadf2c2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad80d4cc0 con 0x7fdad80b0e30 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadf2c2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad80cf400 con 0x7fdad80d4860 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.475+0000 7fdadd05e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdad81614d0 0x7fdad80d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36748/0 (socket says 172.21.15.72:36748) 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadd05e640 1 -- 172.21.15.72:0/613914667 learned_addr learned my addr 172.21.15.72:0/613914667 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadc85d640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad80d4860 0x7fdad80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadd85f640 1 --2- 172.21.15.72:0/613914667 >> v2:172.21.15.146:3300/0 conn(0x7fdad80b0e30 0x7fdad80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdacffff640 1 -- 172.21.15.72:0/613914667 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115034501 0 0) 0x7fdad81618b0 con 0x7fdad81614d0 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdacffff640 1 -- 172.21.15.72:0/613914667 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdac4003520 con 0x7fdad81614d0 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadd85f640 1 -- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad80d4860 msgr2=0x7fdad80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadd85f640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad80d4860 0x7fdad80d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadd85f640 1 -- 172.21.15.72:0/613914667 >> v1:172.21.15.72:6789/0 conn(0x7fdad81614d0 legacy=0x7fdad80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadd85f640 1 -- 172.21.15.72:0/613914667 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad80cf6a0 con 0x7fdad80b0e30 2024-03-06T07:25:24.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.476+0000 7fdadc85d640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad80d4860 0x7fdad80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:24.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadd85f640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad80b12c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009d90 tx=0x7fdad002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e073612bec26f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdacffff640 1 -- 172.21.15.72:0/613914667 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0039070 con 0x7fdad80b0e30 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdacffff640 1 -- 172.21.15.72:0/613914667 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad002cc50 con 0x7fdad80b0e30 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdacffff640 1 -- 172.21.15.72:0/613914667 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0034440 con 0x7fdad80b0e30 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadf2c2640 1 -- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 msgr2=0x7fdad80b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad80b12c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdad0009d90 tx=0x7fdad002bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadf2c2640 1 -- 172.21.15.72:0/613914667 shutdown_connections 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad80b12c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/613914667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad80d4860 0x7fdad80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.477+0000 7fdadf2c2640 1 -- 172.21.15.72:0/613914667 >> 172.21.15.72:0/613914667 conn(0x7fdad80d3d40 msgr2=0x7fdad80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.478+0000 7fdadf2c2640 1 -- 172.21.15.72:0/613914667 shutdown_connections 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.478+0000 7fdadf2c2640 1 -- 172.21.15.72:0/613914667 wait complete. 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 Processor -- start 2024-03-06T07:25:24.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 -- start start 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad8153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 0x7fdad80cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad81614d0 0x7fdad80cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad80b0910 con 0x7fdad80d4860 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad80cfc60 con 0x7fdad80b0e30 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadf2c2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad80d2560 con 0x7fdad81614d0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadc85d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 0x7fdad80cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadc85d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 0x7fdad80cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58880/0 (socket says 172.21.15.72:58880) 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.479+0000 7fdadc85d640 1 -- 172.21.15.72:0/2009877935 learned_addr learned my addr 172.21.15.72:0/2009877935 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:24.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadd85f640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad81614d0 0x7fdad80cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadd05e640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad8153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadc85d640 1 -- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad81614d0 msgr2=0x7fdad80cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadc85d640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad81614d0 0x7fdad80cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadc85d640 1 -- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 msgr2=0x7fdad8153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadc85d640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad8153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadc85d640 1 -- 172.21.15.72:0/2009877935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad80b82a0 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadd05e640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad8153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadc85d640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 0x7fdad80cd430 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fdac800ec10 tx=0x7fdac800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8017070 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac800cc30 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad80caeb0 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8013470 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.480+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad80cb470 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.481+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad8068ff0 con 0x7fdad80d4860 2024-03-06T07:25:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.482+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdac801a070 con 0x7fdad80d4860 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.482+0000 7fdacdffb640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdabc06b4d0 0x7fdabc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.482+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdac808a480 con 0x7fdad80d4860 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.482+0000 7fdadd05e640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdabc06b4d0 0x7fdabc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.482+0000 7fdadd05e640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdabc06b4d0 0x7fdabc06d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007fd0 tx=0x7fdad4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.486+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdac8058cb0 con 0x7fdad80d4860 2024-03-06T07:25:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.657+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdad806bec0 con 0x7fdad80d4860 2024-03-06T07:25:24.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.658+0000 7fdacdffb640 1 -- 172.21.15.72:0/2009877935 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdac805cd50 con 0x7fdad80d4860 2024-03-06T07:25:24.660 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdabc06b4d0 msgr2=0x7fdabc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdabc06b4d0 0x7fdabc06d990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdad4007fd0 tx=0x7fdad4009040 comp rx=0 tx=0).stop 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 msgr2=0x7fdad80cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 0x7fdad80cd430 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fdac800ec10 tx=0x7fdac800c520 comp rx=0 tx=0).stop 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 shutdown_connections 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdabc06b4d0 0x7fdabc06d990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad81614d0 0x7fdad80cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdad80d4860 0x7fdad80cd430 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 --2- 172.21.15.72:0/2009877935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad80b0e30 0x7fdad8153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 >> 172.21.15.72:0/2009877935 conn(0x7fdad80d3d40 msgr2=0x7fdad80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 shutdown_connections 2024-03-06T07:25:24.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.660+0000 7fdadf2c2640 1 -- 172.21.15.72:0/2009877935 wait complete. 2024-03-06T07:25:24.673 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:24.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.775+0000 7fdecf5e2640 1 Processor -- start 2024-03-06T07:25:24.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.775+0000 7fdecf5e2640 1 -- start start 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecf5e2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdec80cfa90 0x7fdec80cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecf5e2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec8158ac0 con 0x7fdec81586e0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecf5e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec815b2a0 con 0x7fdec80b33a0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecf5e2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec815b540 con 0x7fdec80cfa90 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecd37e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdec81586e0 0x7fdec80aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36760/0 (socket says 172.21.15.72:36760) 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecd37e640 1 -- 172.21.15.72:0/1247248782 learned_addr learned my addr 172.21.15.72:0/1247248782 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdeccb7d640 1 --2- 172.21.15.72:0/1247248782 >> v2:172.21.15.146:3300/0 conn(0x7fdec80cfa90 0x7fdec80cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41130304 0 0) 0x7fdec8158ac0 con 0x7fdec81586e0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeb4003520 con 0x7fdec81586e0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309498517 0 0) 0x7fdec815b2a0 con 0x7fdec80b33a0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec8158ac0 con 0x7fdec80b33a0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20320270 0 0) 0x7fdeb4003520 con 0x7fdec81586e0 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 >> v2:172.21.15.146:3300/0 conn(0x7fdec80cfa90 msgr2=0x7fdec80cff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 --2- 172.21.15.72:0/1247248782 >> v2:172.21.15.146:3300/0 conn(0x7fdec80cfa90 0x7fdec80cff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 >> v1:172.21.15.146:6789/0 conn(0x7fdec80b33a0 legacy=0x7fdec80b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec815b820 con 0x7fdec81586e0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fdec4001670 con 0x7fdec81586e0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 >> v1:172.21.15.72:6789/0 conn(0x7fdec81586e0 legacy=0x7fdec80aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4003b90 0x7fdeb4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb4004800 0x7fdeb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb40051c0 0x7fdeb400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec815b540 con 0x7fdeb4003b90 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb400e2a0 con 0x7fdeb40051c0 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb400e4a0 con 0x7fdeb4004800 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdeccb7d640 1 --2- 172.21.15.72:0/1247248782 >> v2:172.21.15.146:3300/0 conn(0x7fdec80cfa90 0x7fdec80cff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.776+0000 7fdecd37e640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4003b90 0x7fdeb4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecdb7f640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb40051c0 0x7fdeb400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdeccb7d640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb4004800 0x7fdeb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecd37e640 1 -- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb4004800 msgr2=0x7fdeb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecd37e640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb4004800 0x7fdeb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecd37e640 1 -- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb40051c0 msgr2=0x7fdeb400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecd37e640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb40051c0 0x7fdeb400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecd37e640 1 -- 172.21.15.72:0/1247248782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb4003520 con 0x7fdeb4003b90 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecdb7f640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb40051c0 0x7fdeb400dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecdb7f640 1 -- 172.21.15.72:0/1247248782 reap_dead start 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdecd37e640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4003b90 0x7fdeb4003f90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fdec400c0e0 tx=0x7fdeb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b974f6ba769ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.777+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4007c00 con 0x7fdeb4003b90 2024-03-06T07:25:24.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec4030070 con 0x7fdeb4003b90 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdebffff640 1 -- 172.21.15.72:0/1247248782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4005d10 con 0x7fdeb4003b90 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 -- 172.21.15.72:0/1247248782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4003b90 msgr2=0x7fdeb4003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4003b90 0x7fdeb4003f90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fdec400c0e0 tx=0x7fdeb4005830 comp rx=0 tx=0).stop 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 -- 172.21.15.72:0/1247248782 shutdown_connections 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/1247248782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdeb4003b90 0x7fdeb4003f90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 -- 172.21.15.72:0/1247248782 >> 172.21.15.72:0/1247248782 conn(0x7fdec806b190 msgr2=0x7fdec806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 -- 172.21.15.72:0/1247248782 shutdown_connections 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.778+0000 7fdecf5e2640 1 -- 172.21.15.72:0/1247248782 wait complete. 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.779+0000 7fdecf5e2640 1 Processor -- start 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.779+0000 7fdecf5e2640 1 -- start start 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecf5e2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb4003b90 0x7fdec8144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecf5e2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec80cda40 0x7fdec81f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecf5e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 0x7fdec81f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecf5e2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8071350 con 0x7fdec80cdec0 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecf5e2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec815bde0 con 0x7fdeb4003b90 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecf5e2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec80714d0 con 0x7fdec80cda40 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 0x7fdec81f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 0x7fdec81f33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58892/0 (socket says 172.21.15.72:58892) 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 -- 172.21.15.72:0/433787941 learned_addr learned my addr 172.21.15.72:0/433787941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdeccb7d640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec80cda40 0x7fdec81f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecd37e640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb4003b90 0x7fdec8144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 -- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec80cda40 msgr2=0x7fdec81f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec80cda40 0x7fdec81f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 -- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb4003b90 msgr2=0x7fdec8144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb4003b90 0x7fdec8144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.780+0000 7fdecdb7f640 1 -- 172.21.15.72:0/433787941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80ced30 con 0x7fdec80cdec0 2024-03-06T07:25:24.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.781+0000 7fdecdb7f640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 0x7fdec81f33e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fdec8071130 tx=0x7fdec000eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.781+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0006b50 con 0x7fdec80cdec0 2024-03-06T07:25:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.781+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec0006d10 con 0x7fdec80cdec0 2024-03-06T07:25:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.781+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec0003970 con 0x7fdec80cdec0 2024-03-06T07:25:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.781+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec80ce490 con 0x7fdec80cdec0 2024-03-06T07:25:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.781+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec80cea50 con 0x7fdec80cdec0 2024-03-06T07:25:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.782+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec80b4820 con 0x7fdec80cdec0 2024-03-06T07:25:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.782+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdec001a020 con 0x7fdec80cdec0 2024-03-06T07:25:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.782+0000 7fdebdffb640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdeac06b500 0x7fdeac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.783+0000 7fdecd37e640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdeac06b500 0x7fdeac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.783+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdec0089740 con 0x7fdec80cdec0 2024-03-06T07:25:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.783+0000 7fdecd37e640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdeac06b500 0x7fdeac06d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdec4034000 tx=0x7fdec4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.787+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdec80b4820 con 0x7fdec80cdec0 2024-03-06T07:25:24.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.957+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdec80732b0 con 0x7fdec80cdec0 2024-03-06T07:25:24.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.958+0000 7fdebdffb640 1 -- 172.21.15.72:0/433787941 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdec00585c0 con 0x7fdec80cdec0 2024-03-06T07:25:24.960 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdeac06b500 msgr2=0x7fdeac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdeac06b500 0x7fdeac06d9c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdec4034000 tx=0x7fdec4009030 comp rx=0 tx=0).stop 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 msgr2=0x7fdec81f33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 0x7fdec81f33e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fdec8071130 tx=0x7fdec000eb40 comp rx=0 tx=0).stop 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 shutdown_connections 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdeac06b500 0x7fdeac06d9c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdec80cdec0 0x7fdec81f33e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdec80cda40 0x7fdec81f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 --2- 172.21.15.72:0/433787941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb4003b90 0x7fdec8144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 >> 172.21.15.72:0/433787941 conn(0x7fdec806b190 msgr2=0x7fdec8158cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 shutdown_connections 2024-03-06T07:25:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:24.960+0000 7fdecf5e2640 1 -- 172.21.15.72:0/433787941 wait complete. 2024-03-06T07:25:24.973 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:25.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.071+0000 7fc7500df640 1 Processor -- start 2024-03-06T07:25:25.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.071+0000 7fc7500df640 1 -- start start 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc7500df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc74815dfe0 0x7fc7480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc7500df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7480cf830 0x7fc7480d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc7500df640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7480d27f0 0x7fc7480d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc7500df640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7480d2c50 con 0x7fc7480d27f0 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc7500df640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7480b31e0 con 0x7fc7480cf830 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc7500df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b3d60 con 0x7fc74815dfe0 2024-03-06T07:25:25.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc74e67c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7480d27f0 0x7fc7480d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc74e67c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7480d27f0 0x7fc7480d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58900/0 (socket says 172.21.15.72:58900) 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc74e67c640 1 -- 172.21.15.72:0/3687971146 learned_addr learned my addr 172.21.15.72:0/3687971146 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc74d67a640 1 --2- 172.21.15.72:0/3687971146 >> v2:172.21.15.146:3300/0 conn(0x7fc7480cf830 0x7fc7480d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.072+0000 7fc74de7b640 1 --2- 172.21.15.72:0/3687971146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc74815dfe0 0x7fc7480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.073+0000 7fc74e67c640 1 -- 172.21.15.72:0/3687971146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc74815dfe0 msgr2=0x7fc7480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.073+0000 7fc74e67c640 1 --2- 172.21.15.72:0/3687971146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc74815dfe0 0x7fc7480cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.073+0000 7fc74e67c640 1 -- 172.21.15.72:0/3687971146 >> v2:172.21.15.146:3300/0 conn(0x7fc7480cf830 msgr2=0x7fc7480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.073+0000 7fc74e67c640 1 --2- 172.21.15.72:0/3687971146 >> v2:172.21.15.146:3300/0 conn(0x7fc7480cf830 0x7fc7480d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.073+0000 7fc74e67c640 1 -- 172.21.15.72:0/3687971146 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7480b3430 con 0x7fc7480d27f0 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.073+0000 7fc74e67c640 1 --2- 172.21.15.72:0/3687971146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7480d27f0 0x7fc7480d0300 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc74808cba0 tx=0x7fc74002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2710c1475dbfdee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc74ce79640 1 -- 172.21.15.72:0/3687971146 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740035020 con 0x7fc7480d27f0 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc74ce79640 1 -- 172.21.15.72:0/3687971146 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc740031870 con 0x7fc7480d27f0 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc74ce79640 1 -- 172.21.15.72:0/3687971146 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740031c40 con 0x7fc7480d27f0 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 -- 172.21.15.72:0/3687971146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7480d27f0 msgr2=0x7fc7480d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 --2- 172.21.15.72:0/3687971146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7480d27f0 0x7fc7480d0300 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fc74808cba0 tx=0x7fc74002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 -- 172.21.15.72:0/3687971146 shutdown_connections 2024-03-06T07:25:25.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 --2- 172.21.15.72:0/3687971146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7480d27f0 0x7fc7480d0300 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 --2- 172.21.15.72:0/3687971146 >> v2:172.21.15.146:3300/0 conn(0x7fc7480cf830 0x7fc7480d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 --2- 172.21.15.72:0/3687971146 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc74815dfe0 0x7fc7480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 -- 172.21.15.72:0/3687971146 >> 172.21.15.72:0/3687971146 conn(0x7fc7480d3cd0 msgr2=0x7fc7480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 -- 172.21.15.72:0/3687971146 shutdown_connections 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.074+0000 7fc7500df640 1 -- 172.21.15.72:0/3687971146 wait complete. 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.075+0000 7fc7500df640 1 Processor -- start 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.075+0000 7fc7500df640 1 -- start start 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc7500df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7480cf830 0x7fc748161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc7500df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7480d27f0 0x7fc7480b8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc7500df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 0x7fc748149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc7500df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480d1c70 con 0x7fc74815dfe0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc7500df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480b39f0 con 0x7fc7480d27f0 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc7500df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480d1df0 con 0x7fc7480cf830 2024-03-06T07:25:25.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc74e67c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 0x7fc748149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc74e67c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 0x7fc748149f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58906/0 (socket says 172.21.15.72:58906) 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc74d67a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7480d27f0 0x7fc7480b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc74e67c640 1 -- 172.21.15.72:0/4191999879 learned_addr learned my addr 172.21.15.72:0/4191999879 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.076+0000 7fc74de7b640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7480cf830 0x7fc748161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc74e67c640 1 -- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7480cf830 msgr2=0x7fc748161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc74e67c640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7480cf830 0x7fc748161190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc74e67c640 1 -- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7480d27f0 msgr2=0x7fc7480b8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc74e67c640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7480d27f0 0x7fc7480b8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc74e67c640 1 -- 172.21.15.72:0/4191999879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7480b8ee0 con 0x7fc74815dfe0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc74e67c640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 0x7fc748149f30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc74002b950 tx=0x7fc7400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740040070 con 0x7fc74815dfe0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc740030c20 con 0x7fc74815dfe0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc740035040 con 0x7fc74815dfe0 2024-03-06T07:25:25.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7480b91c0 con 0x7fc74815dfe0 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.077+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7480b9780 con 0x7fc74815dfe0 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.078+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc74804f330 con 0x7fc74815dfe0 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.078+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc740038020 con 0x7fc74815dfe0 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.079+0000 7fc73e7fc640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc72c06b4d0 0x7fc72c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.079+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7400abe50 con 0x7fc74815dfe0 2024-03-06T07:25:25.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.079+0000 7fc74de7b640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc72c06b4d0 0x7fc72c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.079+0000 7fc74de7b640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc72c06b4d0 0x7fc72c06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc744011340 tx=0x7fc74400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.083+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc74007a680 con 0x7fc74815dfe0 2024-03-06T07:25:25.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.289+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc74815d4c0 con 0x7fc74815dfe0 2024-03-06T07:25:25.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.289+0000 7fc73e7fc640 1 -- 172.21.15.72:0/4191999879 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7fc74007e720 con 0x7fc74815dfe0 2024-03-06T07:25:25.292 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:25.292 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:25.292 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc72c06b4d0 msgr2=0x7fc72c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc72c06b4d0 0x7fc72c06d990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc744011340 tx=0x7fc74400b040 comp rx=0 tx=0).stop 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 msgr2=0x7fc748149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 0x7fc748149f30 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fc74002b950 tx=0x7fc7400046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 shutdown_connections 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc72c06b4d0 0x7fc72c06d990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc74815dfe0 0x7fc748149f30 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7480d27f0 0x7fc7480b8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 --2- 172.21.15.72:0/4191999879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7480cf830 0x7fc748161190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 >> 172.21.15.72:0/4191999879 conn(0x7fc7480d3cd0 msgr2=0x7fc74806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 shutdown_connections 2024-03-06T07:25:25.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.291+0000 7fc7500df640 1 -- 172.21.15.72:0/4191999879 wait complete. 2024-03-06T07:25:25.304 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.401+0000 7f9817363640 1 Processor -- start 2024-03-06T07:25:25.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.401+0000 7f9817363640 1 -- start start 2024-03-06T07:25:25.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f9817363640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98100b4170 0x7f98100b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f9817363640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f98100d0a80 0x7f98100cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f9817363640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100d25d0 con 0x7f98100d4760 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f9817363640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100d4bf0 con 0x7f98100b4170 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f9817363640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98100ce9e0 con 0x7f98100d0a80 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f98150ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98100b4170 0x7f98100b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f98148fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f98100d4760 0x7f98100d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36776/0 (socket says 172.21.15.72:36776) 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f98148fe640 1 -- 172.21.15.72:0/2563685187 learned_addr learned my addr 172.21.15.72:0/2563685187 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.402+0000 7f9815900640 1 --2- 172.21.15.72:0/2563685187 >> v2:172.21.15.146:3300/0 conn(0x7f98100d0a80 0x7f98100cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f98150ff640 1 -- 172.21.15.72:0/2563685187 >> v2:172.21.15.146:3300/0 conn(0x7f98100d0a80 msgr2=0x7f98100cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f98150ff640 1 --2- 172.21.15.72:0/2563685187 >> v2:172.21.15.146:3300/0 conn(0x7f98100d0a80 0x7f98100cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f98150ff640 1 -- 172.21.15.72:0/2563685187 >> v1:172.21.15.72:6789/0 conn(0x7f98100d4760 legacy=0x7f98100d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f9807fff640 1 -- 172.21.15.72:0/2563685187 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733221162 0 0) 0x7f98100d25d0 con 0x7f98100d4760 2024-03-06T07:25:25.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f98150ff640 1 -- 172.21.15.72:0/2563685187 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100d2870 con 0x7f98100b4170 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f98150ff640 1 --2- 172.21.15.72:0/2563685187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100b4170 0x7f98100b6a90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f981008df00 tx=0x7f980c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8bb19af247197e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f9807fff640 1 -- 172.21.15.72:0/2563685187 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c039070 con 0x7f98100b4170 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.403+0000 7f9807fff640 1 -- 172.21.15.72:0/2563685187 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980c02cc80 con 0x7f98100b4170 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9807fff640 1 -- 172.21.15.72:0/2563685187 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c034660 con 0x7f98100b4170 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 -- 172.21.15.72:0/2563685187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100b4170 msgr2=0x7f98100b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 --2- 172.21.15.72:0/2563685187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100b4170 0x7f98100b6a90 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f981008df00 tx=0x7f980c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 -- 172.21.15.72:0/2563685187 shutdown_connections 2024-03-06T07:25:25.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 --2- 172.21.15.72:0/2563685187 >> v2:172.21.15.146:3300/0 conn(0x7f98100d0a80 0x7f98100cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 --2- 172.21.15.72:0/2563685187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100b4170 0x7f98100b6a90 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 -- 172.21.15.72:0/2563685187 >> 172.21.15.72:0/2563685187 conn(0x7f98100d3c40 msgr2=0x7f98100d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 -- 172.21.15.72:0/2563685187 shutdown_connections 2024-03-06T07:25:25.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.404+0000 7f9817363640 1 -- 172.21.15.72:0/2563685187 wait complete. 2024-03-06T07:25:25.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.405+0000 7f9817363640 1 Processor -- start 2024-03-06T07:25:25.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.405+0000 7f9817363640 1 -- start start 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9817363640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98100b4170 0x7f981015f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9817363640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98100d0a80 0x7f98100a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9817363640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 0x7f98100b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9817363640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b4dd0 con 0x7f98100a0c50 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9817363640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b4550 con 0x7f98100d0a80 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9817363640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98100b4f50 con 0x7f98100b4170 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 0x7f98100b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 0x7f98100b1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58916/0 (socket says 172.21.15.72:58916) 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 -- 172.21.15.72:0/1462397473 learned_addr learned my addr 172.21.15.72:0/1462397473 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f98150ff640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98100b4170 0x7f981015f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f98148fe640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98100d0a80 0x7f98100a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 -- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98100b4170 msgr2=0x7f981015f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98100b4170 0x7f981015f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 -- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98100d0a80 msgr2=0x7f98100a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98100d0a80 0x7f98100a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 -- 172.21.15.72:0/1462397473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98101602c0 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f98148fe640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98100d0a80 0x7f98100a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.406+0000 7f9815900640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 0x7f98100b1810 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f980800e930 tx=0x7f980800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.407+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9808017070 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.407+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f980800c980 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.407+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980800cd70 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.407+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98100b2420 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.407+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981009dfb0 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.408+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981004f330 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.408+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f980801a070 con 0x7f98100a0c50 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.409+0000 7f9805ffb640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f97f406b4d0 0x7f97f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.409+0000 7f98150ff640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f97f406b4d0 0x7f97f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.409+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f980808a520 con 0x7f98100a0c50 2024-03-06T07:25:25.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.409+0000 7f98150ff640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f97f406b4d0 0x7f97f406d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f980c036000 tx=0x7f980c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.413+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9808058e10 con 0x7f98100a0c50 2024-03-06T07:25:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.585+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98100733b0 con 0x7f98100a0c50 2024-03-06T07:25:25.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.586+0000 7f9805ffb640 1 -- 172.21.15.72:0/1462397473 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16680 (secure 0 0 0) 0x7f980805ceb0 con 0x7f98100a0c50 2024-03-06T07:25:25.588 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:25.588 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":71,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:53.957942+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.587+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f97f406b4d0 msgr2=0x7f97f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.587+0000 7f9817363640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f97f406b4d0 0x7f97f406d990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f980c036000 tx=0x7f980c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.587+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 msgr2=0x7f98100b1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.587+0000 7f9817363640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 0x7f98100b1810 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f980800e930 tx=0x7f980800edf0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 shutdown_connections 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f97f406b4d0 0x7f97f406d990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98100a0c50 0x7f98100b1810 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98100d0a80 0x7f98100a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 --2- 172.21.15.72:0/1462397473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98100b4170 0x7f981015f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 >> 172.21.15.72:0/1462397473 conn(0x7f98100d3c40 msgr2=0x7f98100ccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 shutdown_connections 2024-03-06T07:25:25.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.588+0000 7f9817363640 1 -- 172.21.15.72:0/1462397473 wait complete. 2024-03-06T07:25:25.601 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:25.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:25.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.698+0000 7f62725e9640 1 Processor -- start 2024-03-06T07:25:25.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.698+0000 7f62725e9640 1 -- start start 2024-03-06T07:25:25.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f62725e9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f626c15dfe0 0x7f626c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f62725e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c0cf830 0x7f626c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f62725e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626c0b3430 con 0x7f626c0d27f0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f62725e9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626c0d2c50 con 0x7f626c15dfe0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f62725e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0b31e0 con 0x7f626c0cf830 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f626bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f626c15dfe0 0x7f626c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f626bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f626c15dfe0 0x7f626c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58928/0 (socket says 172.21.15.72:58928) 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f626bfff640 1 -- 172.21.15.72:0/126181131 learned_addr learned my addr 172.21.15.72:0/126181131 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.699+0000 7f626b7fe640 1 --2- 172.21.15.72:0/126181131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c0cf830 0x7f626c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.700+0000 7f626bfff640 1 -- 172.21.15.72:0/126181131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c0cf830 msgr2=0x7f626c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.700+0000 7f626bfff640 1 --2- 172.21.15.72:0/126181131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c0cf830 0x7f626c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.700+0000 7f626bfff640 1 -- 172.21.15.72:0/126181131 >> v1:172.21.15.146:6789/0 conn(0x7f626c0d27f0 legacy=0x7f626c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.700+0000 7f626bfff640 1 -- 172.21.15.72:0/126181131 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c0b3710 con 0x7f626c15dfe0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.700+0000 7f626bfff640 1 --2- 172.21.15.72:0/126181131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c15dfe0 0x7f626c0cf2f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f625c009870 tx=0x7f625c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f8469205e2cd0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.700+0000 7f626affd640 1 -- 172.21.15.72:0/126181131 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c034070 con 0x7f626c15dfe0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f626affd640 1 -- 172.21.15.72:0/126181131 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f625c030840 con 0x7f626c15dfe0 2024-03-06T07:25:25.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f626affd640 1 -- 172.21.15.72:0/126181131 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625c030c10 con 0x7f626c15dfe0 2024-03-06T07:25:25.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 -- 172.21.15.72:0/126181131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c15dfe0 msgr2=0x7f626c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 --2- 172.21.15.72:0/126181131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c15dfe0 0x7f626c0cf2f0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f625c009870 tx=0x7f625c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:25.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 -- 172.21.15.72:0/126181131 shutdown_connections 2024-03-06T07:25:25.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 --2- 172.21.15.72:0/126181131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c0cf830 0x7f626c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 --2- 172.21.15.72:0/126181131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c15dfe0 0x7f626c0cf2f0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 -- 172.21.15.72:0/126181131 >> 172.21.15.72:0/126181131 conn(0x7f626c0d3cd0 msgr2=0x7f626c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 -- 172.21.15.72:0/126181131 shutdown_connections 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.701+0000 7f62725e9640 1 -- 172.21.15.72:0/126181131 wait complete. 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.702+0000 7f62725e9640 1 Processor -- start 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.702+0000 7f62725e9640 1 -- start start 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f62725e9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626c0cf830 0x7f626c0b9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f62725e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 0x7f626c0b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f62725e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c15dfe0 0x7f626c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f62725e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0d1c70 con 0x7f626c0d27f0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f62725e9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0b3d60 con 0x7f626c0cf830 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f62725e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626c0d1df0 con 0x7f626c15dfe0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 0x7f626c0b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f626b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 0x7f626c0b24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58940/0 (socket says 172.21.15.72:58940) 2024-03-06T07:25:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f626b7fe640 1 -- 172.21.15.72:0/2981960100 learned_addr learned my addr 172.21.15.72:0/2981960100 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f626bfff640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626c0cf830 0x7f626c0b9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.703+0000 7f6270b86640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c15dfe0 0x7f626c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f626b7fe640 1 -- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c15dfe0 msgr2=0x7f626c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f626b7fe640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c15dfe0 0x7f626c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f626b7fe640 1 -- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626c0cf830 msgr2=0x7f626c0b9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f626b7fe640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626c0cf830 0x7f626c0b9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f626b7fe640 1 -- 172.21.15.72:0/2981960100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c14b090 con 0x7f626c0d27f0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f626b7fe640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 0x7f626c0b24d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f626000e980 tx=0x7f626000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6260017070 con 0x7f626c0d27f0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f626000ca00 con 0x7f626c0d27f0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626000cdd0 con 0x7f626c0d27f0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626c14b370 con 0x7f626c0d27f0 2024-03-06T07:25:25.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.704+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626c0b9060 con 0x7f626c0d27f0 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.705+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626c0cf5f0 con 0x7f626c0d27f0 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.705+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f626001a070 con 0x7f626c0d27f0 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.706+0000 7f6268ff9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f625806b4d0 0x7f625806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:25.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.706+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f626008a540 con 0x7f626c0d27f0 2024-03-06T07:25:25.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.706+0000 7f626bfff640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f625806b4d0 0x7f625806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:25.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.706+0000 7f626bfff640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f625806b4d0 0x7f625806d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f625c036000 tx=0x7f625c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:25.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.710+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f626c0cf5f0 con 0x7f626c0d27f0 2024-03-06T07:25:25.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.917+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f626c0ce3c0 con 0x7f626c0d27f0 2024-03-06T07:25:25.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.918+0000 7f6268ff9640 1 -- 172.21.15.72:0/2981960100 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f626c0ce3c0 con 0x7f626c0d27f0 2024-03-06T07:25:25.920 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:25.920 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:25.920 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f625806b4d0 msgr2=0x7f625806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f625806b4d0 0x7f625806d990 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f625c036000 tx=0x7f625c03b040 comp rx=0 tx=0).stop 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 msgr2=0x7f626c0b24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 0x7f626c0b24d0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f626000e980 tx=0x7f626000ee40 comp rx=0 tx=0).stop 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 shutdown_connections 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f625806b4d0 0x7f625806d990 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f626c15dfe0 0x7f626c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f626c0d27f0 0x7f626c0b24d0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 --2- 172.21.15.72:0/2981960100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f626c0cf830 0x7f626c0b9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:25.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 >> 172.21.15.72:0/2981960100 conn(0x7f626c0d3cd0 msgr2=0x7f626c06cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:25.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 shutdown_connections 2024-03-06T07:25:25.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:25.920+0000 7f62725e9640 1 -- 172.21.15.72:0/2981960100 wait complete. 2024-03-06T07:25:25.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:26.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.031+0000 7f1dab257640 1 Processor -- start 2024-03-06T07:25:26.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.031+0000 7f1dab257640 1 -- start start 2024-03-06T07:25:26.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1dab257640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1da415bf30 0x7f1da415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1dab257640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da415f580 con 0x7f1da413dc10 2024-03-06T07:25:26.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1dab257640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da415b450 con 0x7f1da40b3060 2024-03-06T07:25:26.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1dab257640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da40cf3f0 con 0x7f1da415bf30 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da8ff3640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1da413dc10 0x7f1da40b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36782/0 (socket says 172.21.15.72:36782) 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da8ff3640 1 -- 172.21.15.72:0/74927633 learned_addr learned my addr 172.21.15.72:0/74927633 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da97f4640 1 --2- 172.21.15.72:0/74927633 >> v2:172.21.15.146:3300/0 conn(0x7f1da415bf30 0x7f1da415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da37fe640 1 -- 172.21.15.72:0/74927633 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136091039 0 0) 0x7f1da415f580 con 0x7f1da413dc10 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da37fe640 1 -- 172.21.15.72:0/74927633 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d94003520 con 0x7f1da413dc10 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da97f4640 1 -- 172.21.15.72:0/74927633 >> v1:172.21.15.146:6789/0 conn(0x7f1da40b3060 legacy=0x7f1da415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da97f4640 1 -- 172.21.15.72:0/74927633 >> v1:172.21.15.72:6789/0 conn(0x7f1da413dc10 legacy=0x7f1da40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.032+0000 7f1da97f4640 1 -- 172.21.15.72:0/74927633 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da415b730 con 0x7f1da415bf30 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.033+0000 7f1da97f4640 1 --2- 172.21.15.72:0/74927633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da415bf30 0x7f1da415ef60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009d90 tx=0x7f1d9c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c121b35a1294087 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.033+0000 7f1da37fe640 1 -- 172.21.15.72:0/74927633 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c039070 con 0x7f1da415bf30 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.033+0000 7f1da37fe640 1 -- 172.21.15.72:0/74927633 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9c02cc60 con 0x7f1da415bf30 2024-03-06T07:25:26.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.033+0000 7f1da37fe640 1 -- 172.21.15.72:0/74927633 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c034640 con 0x7f1da415bf30 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 -- 172.21.15.72:0/74927633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da415bf30 msgr2=0x7f1da415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 --2- 172.21.15.72:0/74927633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da415bf30 0x7f1da415ef60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009d90 tx=0x7f1d9c02beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 -- 172.21.15.72:0/74927633 shutdown_connections 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 --2- 172.21.15.72:0/74927633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da415bf30 0x7f1da415ef60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 -- 172.21.15.72:0/74927633 >> 172.21.15.72:0/74927633 conn(0x7f1da40cf8a0 msgr2=0x7f1da40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 -- 172.21.15.72:0/74927633 shutdown_connections 2024-03-06T07:25:26.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.034+0000 7f1dab257640 1 -- 172.21.15.72:0/74927633 wait complete. 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.035+0000 7f1dab257640 1 Processor -- start 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.035+0000 7f1dab257640 1 -- start start 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1dab257640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da40b3060 0x7f1da41547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1dab257640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1da413dc10 0x7f1da41435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1dab257640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 0x7f1da415e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1dab257640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da40718e0 con 0x7f1da415bf30 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1dab257640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da415b210 con 0x7f1da40b3060 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1dab257640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da4071a60 con 0x7f1da413dc10 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 0x7f1da415e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 0x7f1da415e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58950/0 (socket says 172.21.15.72:58950) 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 -- 172.21.15.72:0/3641706416 learned_addr learned my addr 172.21.15.72:0/3641706416 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da3fff640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1da413dc10 0x7f1da41435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da8ff3640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da40b3060 0x7f1da41547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 -- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1da413dc10 msgr2=0x7f1da41435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1da413dc10 0x7f1da41435f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 -- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da40b3060 msgr2=0x7f1da41547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da40b3060 0x7f1da41547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 -- 172.21.15.72:0/3641706416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da413d320 con 0x7f1da415bf30 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da8ff3640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da40b3060 0x7f1da41547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.036+0000 7f1da97f4640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 0x7f1da415e5b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c037040 tx=0x7f1d9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.037+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c044020 con 0x7f1da415bf30 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.037+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9c03a040 con 0x7f1da415bf30 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.037+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da41e5540 con 0x7f1da415bf30 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.037+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c039050 con 0x7f1da415bf30 2024-03-06T07:25:26.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.037+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da41e4780 con 0x7f1da415bf30 2024-03-06T07:25:26.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.038+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da40cfd10 con 0x7f1da415bf30 2024-03-06T07:25:26.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.038+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d9c003bd0 con 0x7f1da415bf30 2024-03-06T07:25:26.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.038+0000 7f1da17fa640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d8806b4d0 0x7f1d8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.038+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d9c0ac6b0 con 0x7f1da415bf30 2024-03-06T07:25:26.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.038+0000 7f1da8ff3640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d8806b4d0 0x7f1d8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.039+0000 7f1da8ff3640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d8806b4d0 0x7f1d8806d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1d980011e0 tx=0x7f1d98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.043+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1da40cfd10 con 0x7f1da415bf30 2024-03-06T07:25:26.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.249+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1da404f330 con 0x7f1da415bf30 2024-03-06T07:25:26.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.250+0000 7f1da17fa640 1 -- 172.21.15.72:0/3641706416 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4985 (secure 0 0 0) 0x7f1d9c04e060 con 0x7f1da415bf30 2024-03-06T07:25:26.252 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:26.252 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":33,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:24:52.303862+0000","modified":"2024-03-06T07:24:54.309372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6579},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6579":{"gid":6579,"name":"b","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/2032377664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":2032377664},{"type":"v1","addr":"172.21.15.146:6835","nonce":2032377664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6579,"qdb_cluster":[6579]},"id":5}]} 2024-03-06T07:25:26.252 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 33 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.251+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d8806b4d0 msgr2=0x7f1d8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.251+0000 7f1dab257640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d8806b4d0 0x7f1d8806d990 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1d980011e0 tx=0x7f1d98009040 comp rx=0 tx=0).stop 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.251+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 msgr2=0x7f1da415e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.251+0000 7f1dab257640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 0x7f1da415e5b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c037040 tx=0x7f1d9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 shutdown_connections 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d8806b4d0 0x7f1d8806d990 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1da415bf30 0x7f1da415e5b0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1da413dc10 0x7f1da41435f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 --2- 172.21.15.72:0/3641706416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1da40b3060 0x7f1da41547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 >> 172.21.15.72:0/3641706416 conn(0x7f1da40cf8a0 msgr2=0x7f1da40b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 shutdown_connections 2024-03-06T07:25:26.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.252+0000 7f1dab257640 1 -- 172.21.15.72:0/3641706416 wait complete. 2024-03-06T07:25:26.265 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:26.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.363+0000 7fd543f65640 1 Processor -- start 2024-03-06T07:25:26.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd543f65640 1 -- start start 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd543f65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c15fa10 0x7fd53c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd543f65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd53c0d16e0 0x7fd53c0d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd543f65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c0cebe0 con 0x7fd53c15bdd0 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd543f65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c0ceec0 con 0x7fd53c0d16e0 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd543f65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0cf1a0 con 0x7fd53c15fa10 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd542502640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd53c0d16e0 0x7fd53c0d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd541d01640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd53c15bdd0 0x7fd53c0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36788/0 (socket says 172.21.15.72:36788) 2024-03-06T07:25:26.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd541d01640 1 -- 172.21.15.72:0/4085410277 learned_addr learned my addr 172.21.15.72:0/4085410277 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:26.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.364+0000 7fd541500640 1 --2- 172.21.15.72:0/4085410277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c15fa10 0x7fd53c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.365+0000 7fd542502640 1 -- 172.21.15.72:0/4085410277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c15fa10 msgr2=0x7fd53c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.365+0000 7fd542502640 1 --2- 172.21.15.72:0/4085410277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c15fa10 0x7fd53c0ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.365+0000 7fd542502640 1 -- 172.21.15.72:0/4085410277 >> v1:172.21.15.72:6789/0 conn(0x7fd53c15bdd0 legacy=0x7fd53c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.365+0000 7fd542502640 1 -- 172.21.15.72:0/4085410277 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0cc8e0 con 0x7fd53c0d16e0 2024-03-06T07:25:26.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.365+0000 7fd540cff640 1 -- 172.21.15.72:0/4085410277 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559200055 0 0) 0x7fd53c0cebe0 con 0x7fd53c15bdd0 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.365+0000 7fd542502640 1 --2- 172.21.15.72:0/4085410277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c0d16e0 0x7fd53c0d1b30 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd5340097a0 tx=0x7fd53402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb624c07778f579 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd540cff640 1 -- 172.21.15.72:0/4085410277 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340315a0 con 0x7fd53c0d16e0 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd540cff640 1 -- 172.21.15.72:0/4085410277 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534037070 con 0x7fd53c0d16e0 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd540cff640 1 -- 172.21.15.72:0/4085410277 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534031960 con 0x7fd53c0d16e0 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 -- 172.21.15.72:0/4085410277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c0d16e0 msgr2=0x7fd53c0d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 --2- 172.21.15.72:0/4085410277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c0d16e0 0x7fd53c0d1b30 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fd5340097a0 tx=0x7fd53402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 -- 172.21.15.72:0/4085410277 shutdown_connections 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 --2- 172.21.15.72:0/4085410277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c0d16e0 0x7fd53c0d1b30 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 --2- 172.21.15.72:0/4085410277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c15fa10 0x7fd53c0ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 -- 172.21.15.72:0/4085410277 >> 172.21.15.72:0/4085410277 conn(0x7fd53c09a9f0 msgr2=0x7fd53c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 -- 172.21.15.72:0/4085410277 shutdown_connections 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.366+0000 7fd543f65640 1 -- 172.21.15.72:0/4085410277 wait complete. 2024-03-06T07:25:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.367+0000 7fd543f65640 1 Processor -- start 2024-03-06T07:25:26.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.367+0000 7fd543f65640 1 -- start start 2024-03-06T07:25:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd543f65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c0d16e0 0x7fd53c0cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd543f65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53c15bdd0 0x7fd53c15af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd543f65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 0x7fd53c0b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd543f65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c15c1b0 con 0x7fd53c15fa10 2024-03-06T07:25:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd543f65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0cc5d0 con 0x7fd53c15bdd0 2024-03-06T07:25:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd543f65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0ce7b0 con 0x7fd53c0d16e0 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd542502640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 0x7fd53c0b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd542502640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 0x7fd53c0b6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58980/0 (socket says 172.21.15.72:58980) 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd542502640 1 -- 172.21.15.72:0/2829105937 learned_addr learned my addr 172.21.15.72:0/2829105937 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd541d01640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c0d16e0 0x7fd53c0cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.368+0000 7fd541500640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53c15bdd0 0x7fd53c15af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd542502640 1 -- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c0d16e0 msgr2=0x7fd53c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd542502640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c0d16e0 0x7fd53c0cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd542502640 1 -- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53c15bdd0 msgr2=0x7fd53c15af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd542502640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53c15bdd0 0x7fd53c15af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd542502640 1 -- 172.21.15.72:0/2829105937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c06c3f0 con 0x7fd53c15fa10 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd542502640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 0x7fd53c0b6ba0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd5340098c0 tx=0x7fd5340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534032630 con 0x7fd53c15fa10 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5340327d0 con 0x7fd53c15fa10 2024-03-06T07:25:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c1f4910 con 0x7fd53c15fa10 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534032bc0 con 0x7fd53c15fa10 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.369+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c1f7490 con 0x7fd53c15fa10 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.370+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c04f330 con 0x7fd53c15fa10 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.370+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd53404c020 con 0x7fd53c15fa10 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.371+0000 7fd5327fc640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd52006b4d0 0x7fd52006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.371+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd53402f020 con 0x7fd53c15fa10 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.371+0000 7fd541d01640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd52006b4d0 0x7fd52006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.371+0000 7fd541d01640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd52006b4d0 0x7fd52006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd5380012e0 tx=0x7fd538001700 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.375+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd53c06c3f0 con 0x7fd53c15fa10 2024-03-06T07:25:26.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.550+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd53c0b7a40 con 0x7fd53c15fa10 2024-03-06T07:25:26.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.551+0000 7fd5327fc640 1 -- 172.21.15.72:0/2829105937 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16680 (secure 0 0 0) 0x7fd534040070 con 0x7fd53c15fa10 2024-03-06T07:25:26.553 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:26.554 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":71,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:24:53.957942+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:24:50.525301+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-06T07:24:51.269562+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd52006b4d0 msgr2=0x7fd52006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd52006b4d0 0x7fd52006d990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd5380012e0 tx=0x7fd538001700 comp rx=0 tx=0).stop 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 msgr2=0x7fd53c0b6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 0x7fd53c0b6ba0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd5340098c0 tx=0x7fd5340046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 shutdown_connections 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd52006b4d0 0x7fd52006d990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd53c15fa10 0x7fd53c0b6ba0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53c15bdd0 0x7fd53c15af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 --2- 172.21.15.72:0/2829105937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53c0d16e0 0x7fd53c0cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 >> 172.21.15.72:0/2829105937 conn(0x7fd53c09a9f0 msgr2=0x7fd53c0c55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 shutdown_connections 2024-03-06T07:25:26.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.553+0000 7fd543f65640 1 -- 172.21.15.72:0/2829105937 wait complete. 2024-03-06T07:25:26.566 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:26.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.665+0000 7f7b495bd640 1 Processor -- start 2024-03-06T07:25:26.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.665+0000 7f7b495bd640 1 -- start start 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b495bd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b440b4c00 0x7f7b440b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b495bd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b4415be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b495bd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4415d510 con 0x7f7b440b28a0 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b495bd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b440b3420 con 0x7f7b440b4c00 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b495bd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415e0b0 con 0x7f7b4415ba50 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b440b4c00 0x7f7b440b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b440b4c00 0x7f7b440b4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58982/0 (socket says 172.21.15.72:58982) 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3591078938 learned_addr learned my addr 172.21.15.72:0/3591078938 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b437fe640 1 --2- 172.21.15.72:0/3591078938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b4415be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3591078938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 msgr2=0x7f7b4415be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/3591078938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b4415be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3591078938 >> v1:172.21.15.146:6789/0 conn(0x7f7b440b28a0 legacy=0x7f7b440b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.666+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3591078938 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4415d780 con 0x7f7b440b4c00 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.667+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/3591078938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b4c00 0x7f7b440b4fe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235bd16634b2610d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.667+0000 7f7b41ffb640 1 -- 172.21.15.72:0/3591078938 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3402f840 con 0x7f7b440b4c00 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.667+0000 7f7b41ffb640 1 -- 172.21.15.72:0/3591078938 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3402f9e0 con 0x7f7b440b4c00 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.667+0000 7f7b41ffb640 1 -- 172.21.15.72:0/3591078938 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3402fdb0 con 0x7f7b440b4c00 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 -- 172.21.15.72:0/3591078938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b4c00 msgr2=0x7f7b440b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3591078938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b4c00 0x7f7b440b4fe0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3402ba20 comp rx=0 tx=0).stop 2024-03-06T07:25:26.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 -- 172.21.15.72:0/3591078938 shutdown_connections 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3591078938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b4415be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3591078938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b4c00 0x7f7b440b4fe0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 -- 172.21.15.72:0/3591078938 >> 172.21.15.72:0/3591078938 conn(0x7f7b4409a9f0 msgr2=0x7f7b4409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 -- 172.21.15.72:0/3591078938 shutdown_connections 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.668+0000 7f7b495bd640 1 -- 172.21.15.72:0/3591078938 wait complete. 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.669+0000 7f7b495bd640 1 Processor -- start 2024-03-06T07:25:26.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.669+0000 7f7b495bd640 1 -- start start 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b495bd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 0x7f7b44143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b495bd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b440b4c00 0x7f7b4413f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b495bd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b44159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b495bd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415f550 con 0x7f7b440b28a0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b495bd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415dd40 con 0x7f7b440b4c00 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b495bd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415f6d0 con 0x7f7b4415ba50 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 0x7f7b44143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 0x7f7b44143170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58998/0 (socket says 172.21.15.72:58998) 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3620765870 learned_addr learned my addr 172.21.15.72:0/3620765870 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b437fe640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b44159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b427fc640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b440b4c00 0x7f7b4413f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 msgr2=0x7f7b44159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b44159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b440b4c00 msgr2=0x7f7b4413f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b440b4c00 0x7f7b4413f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 -- 172.21.15.72:0/3620765870 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44143750 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.670+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 0x7f7b44143170 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7b3402ba60 tx=0x7f7b34004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.671+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34047070 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.671+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b34037bc0 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.671+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b44143a30 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.671+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b34042480 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.671+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b44143ff0 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.672+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4404f330 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.672+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b3404a070 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.672+0000 7f7b2b7fe640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3006b4d0 0x7f7b3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.672+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b340ac380 con 0x7f7b440b28a0 2024-03-06T07:25:26.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.672+0000 7f7b427fc640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3006b4d0 0x7f7b3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:26.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.673+0000 7f7b427fc640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3006b4d0 0x7f7b3006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7b440b5700 tx=0x7f7b38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:26.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.677+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b34033030 con 0x7f7b440b28a0 2024-03-06T07:25:26.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.884+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b440d2820 con 0x7f7b440b28a0 2024-03-06T07:25:26.887 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:25:26.885+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:26.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.888+0000 7f7b2b7fe640 1 -- 172.21.15.72:0/3620765870 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b3407ac40 con 0x7f7b440b28a0 2024-03-06T07:25:26.890 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3006b4d0 msgr2=0x7f7b3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3006b4d0 0x7f7b3006d990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7b440b5700 tx=0x7f7b38009040 comp rx=0 tx=0).stop 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 msgr2=0x7f7b44143170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 0x7f7b44143170 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7b3402ba60 tx=0x7f7b34004730 comp rx=0 tx=0).stop 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 shutdown_connections 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3006b4d0 0x7f7b3006d990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b4415ba50 0x7f7b44159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b440b4c00 0x7f7b4413f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 --2- 172.21.15.72:0/3620765870 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b440b28a0 0x7f7b44143170 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 >> 172.21.15.72:0/3620765870 conn(0x7f7b4409a9f0 msgr2=0x7f7b440b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 shutdown_connections 2024-03-06T07:25:26.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:26.890+0000 7f7b495bd640 1 -- 172.21.15.72:0/3620765870 wait complete. 2024-03-06T07:25:26.905 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:26.928 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:25:27.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.005+0000 7f90cbe9a640 1 Processor -- start 2024-03-06T07:25:27.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90cbe9a640 1 -- start start 2024-03-06T07:25:27.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90cbe9a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c406fd10 0x7f90c40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90cbe9a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90c40d0af0 0x7f90c40ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90cbe9a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90c40cd1e0 con 0x7f90c415c050 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90cbe9a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90c40cd4b0 con 0x7f90c40d0af0 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90cbe9a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c40cd790 con 0x7f90c406fd10 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90ca437640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90c40d0af0 0x7f90c40ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90c9c36640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f90c415c050 0x7f90c406f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36790/0 (socket says 172.21.15.72:36790) 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.006+0000 7f90c9c36640 1 -- 172.21.15.72:0/3213814835 learned_addr learned my addr 172.21.15.72:0/3213814835 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90c9435640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c406fd10 0x7f90c40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90ca437640 1 -- 172.21.15.72:0/3213814835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c406fd10 msgr2=0x7f90c40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90ca437640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c406fd10 0x7f90c40cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90ca437640 1 -- 172.21.15.72:0/3213814835 >> v1:172.21.15.72:6789/0 conn(0x7f90c415c050 legacy=0x7f90c406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90ca437640 1 -- 172.21.15.72:0/3213814835 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c40cda70 con 0x7f90c40d0af0 2024-03-06T07:25:27.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90c9435640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c406fd10 0x7f90c40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.007+0000 7f90ca437640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c40d0af0 0x7f90c40ccca0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8cd2108434e3e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90c8c34640 1 -- 172.21.15.72:0/3213814835 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc039070 con 0x7f90c40d0af0 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90c8c34640 1 -- 172.21.15.72:0/3213814835 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90bc02cbc0 con 0x7f90c40d0af0 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90c8c34640 1 -- 172.21.15.72:0/3213814835 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90bc034440 con 0x7f90c40d0af0 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 -- 172.21.15.72:0/3213814835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c40d0af0 msgr2=0x7f90c40ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c40d0af0 0x7f90c40ccca0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f90bc009d90 tx=0x7f90bc02be10 comp rx=0 tx=0).stop 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 -- 172.21.15.72:0/3213814835 shutdown_connections 2024-03-06T07:25:27.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c40d0af0 0x7f90c40ccca0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/3213814835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c406fd10 0x7f90c40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 -- 172.21.15.72:0/3213814835 >> 172.21.15.72:0/3213814835 conn(0x7f90c40b3800 msgr2=0x7f90c40b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 -- 172.21.15.72:0/3213814835 shutdown_connections 2024-03-06T07:25:27.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.008+0000 7f90cbe9a640 1 -- 172.21.15.72:0/3213814835 wait complete. 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.009+0000 7f90cbe9a640 1 Processor -- start 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.009+0000 7f90cbe9a640 1 -- start start 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90cbe9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 0x7f90c415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90cbe9a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90c40d0af0 0x7f90c4159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90cbe9a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c415c050 0x7f90c413f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90cbe9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c4165c30 con 0x7f90c406fd10 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90cbe9a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c4165ab0 con 0x7f90c40d0af0 2024-03-06T07:25:27.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90cbe9a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90c4165db0 con 0x7f90c415c050 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 0x7f90c415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 0x7f90c415ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59024/0 (socket says 172.21.15.72:59024) 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 -- 172.21.15.72:0/781988788 learned_addr learned my addr 172.21.15.72:0/781988788 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90ca437640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c415c050 0x7f90c413f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9435640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90c40d0af0 0x7f90c4159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 -- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c415c050 msgr2=0x7f90c413f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c415c050 0x7f90c413f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 -- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90c40d0af0 msgr2=0x7f90c4159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90c40d0af0 0x7f90c4159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.010+0000 7f90c9c36640 1 -- 172.21.15.72:0/781988788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c40b7260 con 0x7f90c406fd10 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.011+0000 7f90c9c36640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 0x7f90c415ab60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c430 tx=0x7f90c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.011+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0017070 con 0x7f90c406fd10 2024-03-06T07:25:27.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.011+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c0005ce0 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.011+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c40b7540 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.011+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c00072f0 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.011+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c415a360 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.012+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c409fe60 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.012+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90c001a070 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.013+0000 7f90ba7fc640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90a806b4d0 0x7f90a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.013+0000 7f90c9435640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90a806b4d0 0x7f90a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.013+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f90c0089610 con 0x7f90c406fd10 2024-03-06T07:25:27.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.013+0000 7f90c9435640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90a806b4d0 0x7f90a806d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f90c4165a80 tx=0x7f90b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:27.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.017+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f90c0058430 con 0x7f90c406fd10 2024-03-06T07:25:27.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.223+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f90c4068ff0 con 0x7f90c406fd10 2024-03-06T07:25:27.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.892+0000 7f90ba7fc640 1 -- 172.21.15.72:0/781988788 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f90c0057d30 con 0x7f90c406fd10 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90a806b4d0 msgr2=0x7f90a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90a806b4d0 0x7f90a806d990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f90c4165a80 tx=0x7f90b4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 msgr2=0x7f90c415ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 0x7f90c415ab60 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f90c000c430 tx=0x7f90c000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 shutdown_connections 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90a806b4d0 0x7f90a806d990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90c415c050 0x7f90c413f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90c40d0af0 0x7f90c4159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 --2- 172.21.15.72:0/781988788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90c406fd10 0x7f90c415ab60 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.894+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 >> 172.21.15.72:0/781988788 conn(0x7f90c40b3800 msgr2=0x7f90c40aeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.895+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 shutdown_connections 2024-03-06T07:25:27.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:27.895+0000 7f90cbe9a640 1 -- 172.21.15.72:0/781988788 wait complete. 2024-03-06T07:25:27.911 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:28.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.018+0000 7f4de3078640 1 Processor -- start 2024-03-06T07:25:28.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.018+0000 7f4de3078640 1 -- start start 2024-03-06T07:25:28.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4de3078640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ddc15bdd0 0x7f4ddc1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4de3078640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ddc0d0130 0x7f4ddc15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4de3078640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ddc06f420 con 0x7f4ddc0701e0 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4de3078640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc06f700 con 0x7f4ddc0d0130 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4de3078640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc06f9e0 con 0x7f4ddc15bdd0 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4ddbfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ddc0d0130 0x7f4ddc15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4ddbfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ddc0d0130 0x7f4ddc15d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59030/0 (socket says 172.21.15.72:59030) 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4ddbfff640 1 -- 172.21.15.72:0/2628335371 learned_addr learned my addr 172.21.15.72:0/2628335371 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.019+0000 7f4de0e14640 1 --2- 172.21.15.72:0/2628335371 >> v2:172.21.15.146:3300/0 conn(0x7f4ddc15bdd0 0x7f4ddc1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.020+0000 7f4ddbfff640 1 -- 172.21.15.72:0/2628335371 >> v2:172.21.15.146:3300/0 conn(0x7f4ddc15bdd0 msgr2=0x7f4ddc1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.020+0000 7f4ddbfff640 1 --2- 172.21.15.72:0/2628335371 >> v2:172.21.15.146:3300/0 conn(0x7f4ddc15bdd0 0x7f4ddc1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.020+0000 7f4ddbfff640 1 -- 172.21.15.72:0/2628335371 >> v1:172.21.15.146:6789/0 conn(0x7f4ddc0701e0 legacy=0x7f4ddc0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.020+0000 7f4ddbfff640 1 -- 172.21.15.72:0/2628335371 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc06fcc0 con 0x7f4ddc0d0130 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.020+0000 7f4ddbfff640 1 --2- 172.21.15.72:0/2628335371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0d0130 0x7f4ddc15d330 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0b4740 tx=0x7f4dd002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15b74f15947534a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.020+0000 7f4ddb7fe640 1 -- 172.21.15.72:0/2628335371 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0034070 con 0x7f4ddc0d0130 2024-03-06T07:25:28.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4ddb7fe640 1 -- 172.21.15.72:0/2628335371 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd0030860 con 0x7f4ddc0d0130 2024-03-06T07:25:28.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4ddb7fe640 1 -- 172.21.15.72:0/2628335371 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0030c30 con 0x7f4ddc0d0130 2024-03-06T07:25:28.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 -- 172.21.15.72:0/2628335371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0d0130 msgr2=0x7f4ddc15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 --2- 172.21.15.72:0/2628335371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0d0130 0x7f4ddc15d330 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc0b4740 tx=0x7f4dd002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 -- 172.21.15.72:0/2628335371 shutdown_connections 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 --2- 172.21.15.72:0/2628335371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0d0130 0x7f4ddc15d330 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 --2- 172.21.15.72:0/2628335371 >> v2:172.21.15.146:3300/0 conn(0x7f4ddc15bdd0 0x7f4ddc1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 -- 172.21.15.72:0/2628335371 >> 172.21.15.72:0/2628335371 conn(0x7f4ddc09a9f0 msgr2=0x7f4ddc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 -- 172.21.15.72:0/2628335371 shutdown_connections 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.021+0000 7f4de3078640 1 -- 172.21.15.72:0/2628335371 wait complete. 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.022+0000 7f4de3078640 1 Processor -- start 2024-03-06T07:25:28.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.022+0000 7f4de3078640 1 -- start start 2024-03-06T07:25:28.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de3078640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 0x7f4ddc06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de3078640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ddc0d0130 0x7f4ddc157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de3078640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ddc15bdd0 0x7f4ddc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de3078640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc15d9f0 con 0x7f4ddc0701e0 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de3078640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc15d870 con 0x7f4ddc15bdd0 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de3078640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0d08d0 con 0x7f4ddc0d0130 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de0e14640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 0x7f4ddc06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de0e14640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 0x7f4ddc06cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59046/0 (socket says 172.21.15.72:59046) 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de0e14640 1 -- 172.21.15.72:0/1588964510 learned_addr learned my addr 172.21.15.72:0/1588964510 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4de1615640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ddc15bdd0 0x7f4ddc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.023+0000 7f4ddbfff640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ddc0d0130 0x7f4ddc157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de0e14640 1 -- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ddc0d0130 msgr2=0x7f4ddc157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de0e14640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ddc0d0130 0x7f4ddc157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de0e14640 1 -- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ddc15bdd0 msgr2=0x7f4ddc0ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de0e14640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ddc15bdd0 0x7f4ddc0ce190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de0e14640 1 -- 172.21.15.72:0/1588964510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc06c160 con 0x7f4ddc0701e0 2024-03-06T07:25:28.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de0e14640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 0x7f4ddc06cee0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00d800 tx=0x7f4dcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00b6c0 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc019070 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00ba80 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc06c400 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.024+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc157580 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.025+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc04f330 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.025+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4dcc00bc20 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.026+0000 7f4dd97fa640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.026+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4dcc08a510 con 0x7f4ddc0701e0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.026+0000 7f4ddbfff640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:28.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.026+0000 7f4ddbfff640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0036000 tx=0x7f4dd0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:28.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.030+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4dcc058d70 con 0x7f4ddc0701e0 2024-03-06T07:25:28.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.201+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4ddc15f3b0 con 0x7f4ddc0701e0 2024-03-06T07:25:28.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.901+0000 7f4dd97fa640 1 -- 172.21.15.72:0/1588964510 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4dcc05f0c0 con 0x7f4ddc0701e0 2024-03-06T07:25:28.903 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:28.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 msgr2=0x7f4dc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0036000 tx=0x7f4dd0008030 comp rx=0 tx=0).stop 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 msgr2=0x7f4ddc06cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 0x7f4ddc06cee0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00d800 tx=0x7f4dcc00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 shutdown_connections 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4dc806b4d0 0x7f4dc806d990 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ddc15bdd0 0x7f4ddc0ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ddc0d0130 0x7f4ddc157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 --2- 172.21.15.72:0/1588964510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ddc0701e0 0x7f4ddc06cee0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 >> 172.21.15.72:0/1588964510 conn(0x7f4ddc09a9f0 msgr2=0x7f4ddc0705c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.903+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 shutdown_connections 2024-03-06T07:25:28.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:28.904+0000 7f4de3078640 1 -- 172.21.15.72:0/1588964510 wait complete. 2024-03-06T07:25:28.920 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:29.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.027+0000 7ff2c540f640 1 Processor -- start 2024-03-06T07:25:29.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.027+0000 7ff2c540f640 1 -- start start 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2c540f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 0x7ff2c00ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2c540f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c015afe0 con 0x7ff2c015ac00 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2c540f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c00cd000 con 0x7ff2c00708b0 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2c540f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00cdc10 con 0x7ff2c0070d10 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2beffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff2c015ac00 0x7ff2c006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36802/0 (socket says 172.21.15.72:36802) 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2beffd640 1 -- 172.21.15.72:0/2465557399 learned_addr learned my addr 172.21.15.72:0/2465557399 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2bf7fe640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 0x7ff2c00ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824134929 0 0) 0x7ff2c015afe0 con 0x7ff2c015ac00 2024-03-06T07:25:29.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2a8003520 con 0x7ff2c015ac00 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418276286 0 0) 0x7ff2a8003520 con 0x7ff2c015ac00 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 msgr2=0x7ff2c00ce7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.028+0000 7ff2bdffb640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 0x7ff2c00ce7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 >> v1:172.21.15.146:6789/0 conn(0x7ff2c00708b0 legacy=0x7ff2c00ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c00cd2e0 con 0x7ff2c015ac00 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff2b00019f0 con 0x7ff2c015ac00 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 >> v1:172.21.15.72:6789/0 conn(0x7ff2c015ac00 legacy=0x7ff2c006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2a80039e0 0x7ff2a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2a8004390 0x7ff2a8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 0x7ff2a8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00cdc10 con 0x7ff2a8004390 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a800baa0 con 0x7ff2a80039e0 2024-03-06T07:25:29.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a800bca0 con 0x7ff2c0070d10 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2a8004390 0x7ff2a8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2beffd640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2a80039e0 0x7ff2a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2bf7fe640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 0x7ff2a8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2be7fc640 1 -- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 msgr2=0x7ff2a8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2c0070d10 0x7ff2a8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2be7fc640 1 -- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2a80039e0 msgr2=0x7ff2a8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2a80039e0 0x7ff2a8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.029+0000 7ff2be7fc640 1 -- 172.21.15.72:0/2465557399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a8003520 con 0x7ff2a8004390 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2a8004390 0x7ff2a8004860 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4001830 tx=0x7ff2a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5f645399ed1bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4001430 con 0x7ff2a8004390 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b4033610 con 0x7ff2a8004390 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2bdffb640 1 -- 172.21.15.72:0/2465557399 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4033a00 con 0x7ff2a8004390 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2c540f640 1 -- 172.21.15.72:0/2465557399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2a8004390 msgr2=0x7ff2a8004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2c540f640 1 --2- 172.21.15.72:0/2465557399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2a8004390 0x7ff2a8004860 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4001830 tx=0x7ff2a8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2bf7fe640 1 -- 172.21.15.72:0/2465557399 reap_dead start 2024-03-06T07:25:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2c540f640 1 -- 172.21.15.72:0/2465557399 shutdown_connections 2024-03-06T07:25:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2c540f640 1 -- 172.21.15.72:0/2465557399 >> 172.21.15.72:0/2465557399 conn(0x7ff2c00b1650 msgr2=0x7ff2c00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2c540f640 1 -- 172.21.15.72:0/2465557399 shutdown_connections 2024-03-06T07:25:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.030+0000 7ff2c540f640 1 -- 172.21.15.72:0/2465557399 wait complete. 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 Processor -- start 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 -- start start 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2a8004390 0x7ff2c014bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 0x7ff2c01524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2c0152a20 0x7ff2c014f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c0071190 con 0x7ff2c014dae0 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00cd8a0 con 0x7ff2c0152a20 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.032+0000 7ff2c540f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00cf5d0 con 0x7ff2a8004390 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 0x7ff2c01524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 0x7ff2c01524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59066/0 (socket says 172.21.15.72:59066) 2024-03-06T07:25:29.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 -- 172.21.15.72:0/3328781229 learned_addr learned my addr 172.21.15.72:0/3328781229 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2beffd640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2a8004390 0x7ff2c014bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2bf7fe640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2c0152a20 0x7ff2c014f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 -- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2a8004390 msgr2=0x7ff2c014bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2a8004390 0x7ff2c014bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 -- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2c0152a20 msgr2=0x7ff2c014f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2c0152a20 0x7ff2c014f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 -- 172.21.15.72:0/3328781229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c014c4c0 con 0x7ff2c014dae0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2be7fc640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 0x7ff2c01524e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff2b402bdb0 tx=0x7ff2b4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40035f0 con 0x7ff2c014dae0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b4003790 con 0x7ff2c014dae0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c014df70 con 0x7ff2c014dae0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4003b60 con 0x7ff2c014dae0 2024-03-06T07:25:29.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.033+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c014e530 con 0x7ff2c014dae0 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.034+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c006f250 con 0x7ff2c014dae0 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.035+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2b4003d00 con 0x7ff2c014dae0 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.035+0000 7ff2a77fe640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.035+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5357+0+0 (secure 0 0 0) 0x7ff2b40abf40 con 0x7ff2c014dae0 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.035+0000 7ff2beffd640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.036+0000 7ff2beffd640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2c014cf60 tx=0x7ff2b0006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:29.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.040+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2c006f250 con 0x7ff2c014dae0 2024-03-06T07:25:29.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.209+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff2c014eb80 con 0x7ff2c014dae0 2024-03-06T07:25:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.902+0000 7ff2a77fe640 1 -- 172.21.15.72:0/3328781229 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff2b4041030 con 0x7ff2c014dae0 2024-03-06T07:25:29.904 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2ac06b4d0 msgr2=0x7ff2ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff2c014cf60 tx=0x7ff2b0006040 comp rx=0 tx=0).stop 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 msgr2=0x7ff2c01524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 0x7ff2c01524e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff2b402bdb0 tx=0x7ff2b4001230 comp rx=0 tx=0).stop 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 shutdown_connections 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2ac06b4d0 0x7ff2ac06d990 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2c0152a20 0x7ff2c014f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2c014dae0 0x7ff2c01524e0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 --2- 172.21.15.72:0/3328781229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2a8004390 0x7ff2c014bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 >> 172.21.15.72:0/3328781229 conn(0x7ff2c00b1650 msgr2=0x7ff2c00b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 shutdown_connections 2024-03-06T07:25:29.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:29.905+0000 7ff2c540f640 1 -- 172.21.15.72:0/3328781229 wait complete. 2024-03-06T07:25:29.924 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:30.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.026+0000 7f376c1d7640 1 Processor -- start 2024-03-06T07:25:30.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.026+0000 7f376c1d7640 1 -- start start 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376c1d7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f376415c050 0x7f376406f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376c1d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37640d0af0 0x7f37640cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376c1d7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37640cd4b0 con 0x7f376406fb40 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376c1d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376406ffb0 con 0x7f37640d0af0 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376c1d7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640cd790 con 0x7f376415c050 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376a774640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37640d0af0 0x7f37640cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376a774640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37640d0af0 0x7f37640cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59078/0 (socket says 172.21.15.72:59078) 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.027+0000 7f376a774640 1 -- 172.21.15.72:0/2801719170 learned_addr learned my addr 172.21.15.72:0/2801719170 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.028+0000 7f3769f73640 1 --2- 172.21.15.72:0/2801719170 >> v2:172.21.15.146:3300/0 conn(0x7f376415c050 0x7f376406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.028+0000 7f376a774640 1 -- 172.21.15.72:0/2801719170 >> v2:172.21.15.146:3300/0 conn(0x7f376415c050 msgr2=0x7f376406f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.028+0000 7f376a774640 1 --2- 172.21.15.72:0/2801719170 >> v2:172.21.15.146:3300/0 conn(0x7f376415c050 0x7f376406f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.028+0000 7f376a774640 1 -- 172.21.15.72:0/2801719170 >> v1:172.21.15.146:6789/0 conn(0x7f376406fb40 legacy=0x7f37640cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.028+0000 7f376a774640 1 -- 172.21.15.72:0/2801719170 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640cda70 con 0x7f37640d0af0 2024-03-06T07:25:30.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.028+0000 7f376a774640 1 --2- 172.21.15.72:0/2801719170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37640d0af0 0x7f37640cce70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3764070ef0 tx=0x7f375c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7de41a421d994f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f3768f71640 1 -- 172.21.15.72:0/2801719170 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c02e070 con 0x7f37640d0af0 2024-03-06T07:25:30.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f3768f71640 1 -- 172.21.15.72:0/2801719170 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c037070 con 0x7f37640d0af0 2024-03-06T07:25:30.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f3768f71640 1 -- 172.21.15.72:0/2801719170 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c031a00 con 0x7f37640d0af0 2024-03-06T07:25:30.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 -- 172.21.15.72:0/2801719170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37640d0af0 msgr2=0x7f37640cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 --2- 172.21.15.72:0/2801719170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37640d0af0 0x7f37640cce70 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f3764070ef0 tx=0x7f375c02b880 comp rx=0 tx=0).stop 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 -- 172.21.15.72:0/2801719170 shutdown_connections 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 --2- 172.21.15.72:0/2801719170 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37640d0af0 0x7f37640cce70 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 --2- 172.21.15.72:0/2801719170 >> v2:172.21.15.146:3300/0 conn(0x7f376415c050 0x7f376406f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 -- 172.21.15.72:0/2801719170 >> 172.21.15.72:0/2801719170 conn(0x7f37640b3800 msgr2=0x7f37640b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 -- 172.21.15.72:0/2801719170 shutdown_connections 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.029+0000 7f376c1d7640 1 -- 172.21.15.72:0/2801719170 wait complete. 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.030+0000 7f376c1d7640 1 Processor -- start 2024-03-06T07:25:30.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.030+0000 7f376c1d7640 1 -- start start 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376c1d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376406fb40 0x7f37641551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376c1d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37640d0af0 0x7f376415a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376c1d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 0x7f376415ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376c1d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764165c30 con 0x7f376415c050 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376c1d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764165ab0 con 0x7f376406fb40 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376c1d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3764165db0 con 0x7f37640d0af0 2024-03-06T07:25:30.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376a774640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 0x7f376415ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376a774640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 0x7f376415ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59084/0 (socket says 172.21.15.72:59084) 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f376a774640 1 -- 172.21.15.72:0/300822139 learned_addr learned my addr 172.21.15.72:0/300822139 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f3769f73640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376406fb40 0x7f37641551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.031+0000 7f3769772640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37640d0af0 0x7f376415a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376a774640 1 -- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37640d0af0 msgr2=0x7f376415a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376a774640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37640d0af0 0x7f376415a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376a774640 1 -- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376406fb40 msgr2=0x7f37641551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376a774640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376406fb40 0x7f37641551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376a774640 1 -- 172.21.15.72:0/300822139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376409f3a0 con 0x7f376415c050 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f3769772640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37640d0af0 0x7f376415a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f3769f73640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376406fb40 0x7f37641551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376a774640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 0x7f376415ade0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f375c02b8c0 tx=0x7f375c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c032670 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375c032830 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376409f680 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c032c20 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.032+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37641556e0 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.033+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37640b4650 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.033+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f375c0453f0 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.034+0000 7f375a7fc640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f374806b500 0x7f374806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.034+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f375c0abd60 con 0x7f376415c050 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.034+0000 7f3769f73640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f374806b500 0x7f374806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.034+0000 7f3769f73640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f374806b500 0x7f374806d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3764165a80 tx=0x7f376000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.038+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37640b4650 con 0x7f376415c050 2024-03-06T07:25:30.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.212+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3764068ff0 con 0x7f376415c050 2024-03-06T07:25:30.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.215+0000 7f375a7fc640 1 -- 172.21.15.72:0/300822139 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f375c040070 con 0x7f376415c050 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f374806b500 msgr2=0x7f374806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f374806b500 0x7f374806d9c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3764165a80 tx=0x7f376000b040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 msgr2=0x7f376415ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 0x7f376415ade0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f375c02b8c0 tx=0x7f375c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 shutdown_connections 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f374806b500 0x7f374806d9c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f376415c050 0x7f376415ade0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37640d0af0 0x7f376415a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 --2- 172.21.15.72:0/300822139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376406fb40 0x7f37641551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 >> 172.21.15.72:0/300822139 conn(0x7f37640b3800 msgr2=0x7f37640b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 shutdown_connections 2024-03-06T07:25:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.217+0000 7f376c1d7640 1 -- 172.21.15.72:0/300822139 wait complete. 2024-03-06T07:25:30.230 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_fragment_size_max 2024-03-06T07:25:30.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.330+0000 7f0996a2a640 1 Processor -- start 2024-03-06T07:25:30.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.330+0000 7f0996a2a640 1 -- start start 2024-03-06T07:25:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f0996a2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f0990066d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f0996a2a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09900b0db0 0x7f09900ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f0996a2a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f099006b050 con 0x7f09900ce580 2024-03-06T07:25:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f0996a2a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099006b2c0 con 0x7f09900b0db0 2024-03-06T07:25:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f0996a2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09900b0890 con 0x7f099015f4f0 2024-03-06T07:25:30.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09900b0db0 0x7f09900ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f09900b0db0 0x7f09900ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59094/0 (socket says 172.21.15.72:59094) 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 -- 172.21.15.72:0/470019479 learned_addr learned my addr 172.21.15.72:0/470019479 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098ffff640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f0990066d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 -- 172.21.15.72:0/470019479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 msgr2=0x7f0990066d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f0990066d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 -- 172.21.15.72:0/470019479 >> v1:172.21.15.146:6789/0 conn(0x7f09900ce580 legacy=0x7f0990072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098f7fe640 1 -- 172.21.15.72:0/470019479 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09900672e0 con 0x7f09900b0db0 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.331+0000 7f098ffff640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f0990066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.332+0000 7f098f7fe640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900b0db0 0x7f09900ce040 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0984009870 tx=0x7f098402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ae7a3922fe5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.332+0000 7f098effd640 1 -- 172.21.15.72:0/470019479 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984035020 con 0x7f09900b0db0 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.332+0000 7f098effd640 1 -- 172.21.15.72:0/470019479 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984031640 con 0x7f09900b0db0 2024-03-06T07:25:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.332+0000 7f098effd640 1 -- 172.21.15.72:0/470019479 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098402d040 con 0x7f09900b0db0 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.332+0000 7f0996a2a640 1 -- 172.21.15.72:0/470019479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900b0db0 msgr2=0x7f09900ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.332+0000 7f0996a2a640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900b0db0 0x7f09900ce040 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f0984009870 tx=0x7f098402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.333+0000 7f0996a2a640 1 -- 172.21.15.72:0/470019479 shutdown_connections 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.333+0000 7f0996a2a640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900b0db0 0x7f09900ce040 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.333+0000 7f0996a2a640 1 --2- 172.21.15.72:0/470019479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f0990066d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.333+0000 7f0996a2a640 1 -- 172.21.15.72:0/470019479 >> 172.21.15.72:0/470019479 conn(0x7f09900b1d70 msgr2=0x7f09900b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.333+0000 7f0996a2a640 1 -- 172.21.15.72:0/470019479 shutdown_connections 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.333+0000 7f0996a2a640 1 -- 172.21.15.72:0/470019479 wait complete. 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.334+0000 7f0996a2a640 1 Processor -- start 2024-03-06T07:25:30.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.334+0000 7f0996a2a640 1 -- start start 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09900b0db0 0x7f0990148860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 0x7f0990157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f09900caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099015f8d0 con 0x7f09900ce580 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099006b5a0 con 0x7f09900b0db0 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0990149cb0 con 0x7f099015f4f0 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 0x7f0990157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 0x7f0990157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59096/0 (socket says 172.21.15.72:59096) 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 -- 172.21.15.72:0/2243835809 learned_addr learned my addr 172.21.15.72:0/2243835809 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098ffff640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09900b0db0 0x7f0990148860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0994fc7640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f09900caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 -- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 msgr2=0x7f09900caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f09900caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 -- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09900b0db0 msgr2=0x7f0990148860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09900b0db0 0x7f0990148860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 -- 172.21.15.72:0/2243835809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09901490a0 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098f7fe640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 0x7f0990157570 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0984000c00 tx=0x7f09840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984040070 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0984039c10 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0990149380 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984035040 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.335+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0990149940 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.336+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0990149e60 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.337+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f098404b060 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.337+0000 7f098cff9640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f097c06b4d0 0x7f097c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.337+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f09840ab9c0 con 0x7f09900ce580 2024-03-06T07:25:30.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.337+0000 7f098ffff640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f097c06b4d0 0x7f097c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.337+0000 7f098ffff640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f097c06b4d0 0x7f097c06d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f098000a950 tx=0x7f0980008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.341+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f09840ade20 con 0x7f09900ce580 2024-03-06T07:25:30.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.514+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"} v 0) v1 -- 0x7f099009e3d0 con 0x7f09900ce580 2024-03-06T07:25:30.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.517+0000 7f098cff9640 1 -- 172.21.15.72:0/2243835809 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_fragment_size_max"}]=0 v25) v1 ==== 108+0+0 (secure 0 0 0) 0x7f0984031610 con 0x7f09900ce580 2024-03-06T07:25:30.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f097c06b4d0 msgr2=0x7f097c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f097c06b4d0 0x7f097c06d990 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f098000a950 tx=0x7f0980008040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 msgr2=0x7f0990157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 0x7f0990157570 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0984000c00 tx=0x7f09840046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 shutdown_connections 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f097c06b4d0 0x7f097c06d990 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f099015f4f0 0x7f09900caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09900ce580 0x7f0990157570 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 --2- 172.21.15.72:0/2243835809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09900b0db0 0x7f0990148860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 >> 172.21.15.72:0/2243835809 conn(0x7f09900b1d70 msgr2=0x7f09900b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 shutdown_connections 2024-03-06T07:25:30.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.519+0000 7f0996a2a640 1 -- 172.21.15.72:0/2243835809 wait complete. 2024-03-06T07:25:30.532 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-03-06T07:25:30.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.630+0000 7f7d8e42a640 1 Processor -- start 2024-03-06T07:25:30.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.630+0000 7f7d8e42a640 1 -- start start 2024-03-06T07:25:30.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d8e42a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d8813dc10 0x7f7d880b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d8e42a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d880b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d8e42a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8815f580 con 0x7f7d8815e850 2024-03-06T07:25:30.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d8e42a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d880cf3f0 con 0x7f7d8813dc10 2024-03-06T07:25:30.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d8e42a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8815b210 con 0x7f7d8815bf30 2024-03-06T07:25:30.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d877fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d880b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d87fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7d8813dc10 0x7f7d880b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d877fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d880b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:54948/0 (socket says 172.21.15.72:54948) 2024-03-06T07:25:30.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d8c9c7640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f7d8815e850 0x7f7d8815eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:42864/0 (socket says 172.21.15.72:42864) 2024-03-06T07:25:30.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.631+0000 7f7d877fe640 1 -- 172.21.15.72:0/1037391267 learned_addr learned my addr 172.21.15.72:0/1037391267 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d877fe640 1 -- 172.21.15.72:0/1037391267 >> v2:172.21.15.146:3300/0 conn(0x7f7d8813dc10 msgr2=0x7f7d880b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d877fe640 1 --2- 172.21.15.72:0/1037391267 >> v2:172.21.15.146:3300/0 conn(0x7f7d8813dc10 0x7f7d880b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d877fe640 1 -- 172.21.15.72:0/1037391267 >> v1:172.21.15.146:6789/0 conn(0x7f7d8815e850 legacy=0x7f7d8815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d86ffd640 1 -- 172.21.15.72:0/1037391267 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552957506 0 0) 0x7f7d8815f580 con 0x7f7d8815e850 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d877fe640 1 -- 172.21.15.72:0/1037391267 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8815b450 con 0x7f7d8815bf30 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d87fff640 1 --2- 172.21.15.72:0/1037391267 >> v2:172.21.15.146:3300/0 conn(0x7f7d8813dc10 0x7f7d880b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.632+0000 7f7d877fe640 1 --2- 172.21.15.72:0/1037391267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d880b2e90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009870 tx=0x7f7d7c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c6b7036f17c5cc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d86ffd640 1 -- 172.21.15.72:0/1037391267 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c035020 con 0x7f7d8815bf30 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d86ffd640 1 -- 172.21.15.72:0/1037391267 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c031640 con 0x7f7d8815bf30 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d86ffd640 1 -- 172.21.15.72:0/1037391267 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c02d040 con 0x7f7d8815bf30 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 -- 172.21.15.72:0/1037391267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 msgr2=0x7f7d880b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/1037391267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d880b2e90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009870 tx=0x7f7d7c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 -- 172.21.15.72:0/1037391267 shutdown_connections 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/1037391267 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d880b2e90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/1037391267 >> v2:172.21.15.146:3300/0 conn(0x7f7d8813dc10 0x7f7d880b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 -- 172.21.15.72:0/1037391267 >> 172.21.15.72:0/1037391267 conn(0x7f7d880cf8a0 msgr2=0x7f7d880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 -- 172.21.15.72:0/1037391267 shutdown_connections 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.633+0000 7f7d8e42a640 1 -- 172.21.15.72:0/1037391267 wait complete. 2024-03-06T07:25:30.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.634+0000 7f7d8e42a640 1 Processor -- start 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.634+0000 7f7d8e42a640 1 -- start start 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8e42a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d8813dc10 0x7f7d8815a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8e42a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d88135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8e42a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 0x7f7d88136520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8e42a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880cd7a0 con 0x7f7d8815e850 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8e42a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8813dff0 con 0x7f7d8813dc10 2024-03-06T07:25:30.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8e42a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d880cd920 con 0x7f7d8815bf30 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8c9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 0x7f7d88136520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8c9c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 0x7f7d88136520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59112/0 (socket says 172.21.15.72:59112) 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d8c9c7640 1 -- 172.21.15.72:0/2217573431 learned_addr learned my addr 172.21.15.72:0/2217573431 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d877fe640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d88135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.635+0000 7f7d87fff640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d8813dc10 0x7f7d8815a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8c9c7640 1 -- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 msgr2=0x7f7d88135fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8c9c7640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d88135fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8c9c7640 1 -- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d8813dc10 msgr2=0x7f7d8815a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8c9c7640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d8813dc10 0x7f7d8815a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8c9c7640 1 -- 172.21.15.72:0/2217573431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88136b80 con 0x7f7d8815e850 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d87fff640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d8813dc10 0x7f7d8815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8c9c7640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 0x7f7d88136520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000c730 tx=0x7f7d8000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80018070 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8000ce40 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d88136e60 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000a800 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.636+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d88137420 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.637+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8804f330 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.637+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d8001b070 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.638+0000 7f7d84ff9640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d7406b4d0 0x7f7d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.638+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7d80089440 con 0x7f7d8815e850 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.638+0000 7f7d87fff640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d7406b4d0 0x7f7d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:30.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.638+0000 7f7d87fff640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d7406b4d0 0x7f7d7406d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ac0 tx=0x7f7d78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:30.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.642+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d800585c0 con 0x7f7d8815e850 2024-03-06T07:25:30.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.878+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) v1 -- 0x7f7d8815ad30 con 0x7f7d8815e850 2024-03-06T07:25:30.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.915+0000 7f7d84ff9640 1 -- 172.21.15.72:0/2217573431 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v209) v1 ==== 140+0+0 (secure 0 0 0) 0x7f7d80057ec0 con 0x7f7d8815e850 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d7406b4d0 msgr2=0x7f7d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d7406b4d0 0x7f7d7406d990 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ac0 tx=0x7f7d78008040 comp rx=0 tx=0).stop 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 msgr2=0x7f7d88136520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 0x7f7d88136520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000c730 tx=0x7f7d8000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 shutdown_connections 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d7406b4d0 0x7f7d7406d990 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d8815e850 0x7f7d88136520 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d8815bf30 0x7f7d88135fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 --2- 172.21.15.72:0/2217573431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d8813dc10 0x7f7d8815a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:30.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 >> 172.21.15.72:0/2217573431 conn(0x7f7d880cf8a0 msgr2=0x7f7d8806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:30.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 shutdown_connections 2024-03-06T07:25:30.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:30.917+0000 7f7d8e42a640 1 -- 172.21.15.72:0/2217573431 wait complete. 2024-03-06T07:25:30.930 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-03-06T07:25:30.931 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-03-06T07:25:30.931 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:30.931 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:31.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.029+0000 7f6106b48640 1 Processor -- start 2024-03-06T07:25:31.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.029+0000 7f6106b48640 1 -- start start 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f6106b48640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 0x7f610006df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f6106b48640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610006e530 con 0x7f610015aae0 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f6106b48640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610006e810 con 0x7f61000cd040 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f6106b48640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610006eaf0 con 0x7f61000723f0 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f61048e4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f610015aae0 0x7f6100071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36818/0 (socket says 172.21.15.72:36818) 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f61048e4640 1 -- 172.21.15.72:0/455559573 learned_addr learned my addr 172.21.15.72:0/455559573 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f60fffff640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 0x7f610006df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788531042 0 0) 0x7f610006e530 con 0x7f610015aae0 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.030+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ec003520 con 0x7f610015aae0 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067470178 0 0) 0x7f60ec003520 con 0x7f610015aae0 2024-03-06T07:25:31.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 msgr2=0x7f610006df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 0x7f610006df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 >> v1:172.21.15.146:6789/0 conn(0x7f61000cd040 legacy=0x7f61000b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610006edd0 con 0x7f610015aae0 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f60e8001670 con 0x7f610015aae0 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 >> v1:172.21.15.72:6789/0 conn(0x7f610015aae0 legacy=0x7f6100071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60ec0039e0 0x7f60ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f60ec004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 0x7f60ec0050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610006eaf0 con 0x7f60ec004390 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec00b9c0 con 0x7f60ec0039e0 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec00bbc0 con 0x7f61000723f0 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60fffff640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f60ec004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f61048e4640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60ec0039e0 0x7f60ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f61050e5640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 0x7f60ec0050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60fffff640 1 -- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 msgr2=0x7f60ec0050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60fffff640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61000723f0 0x7f60ec0050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60fffff640 1 -- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60ec0039e0 msgr2=0x7f60ec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60fffff640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60ec0039e0 0x7f60ec003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.031+0000 7f60fffff640 1 -- 172.21.15.72:0/455559573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec003520 con 0x7f60ec004390 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f60fffff640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f60ec004860 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009ac0 tx=0x7f60ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7821c4b001947b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0039070 con 0x7f60ec004390 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f002c6b0 con 0x7f60ec004390 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f60ff7fe640 1 -- 172.21.15.72:0/455559573 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f0009690 con 0x7f60ec004390 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f6106b48640 1 -- 172.21.15.72:0/455559573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 msgr2=0x7f60ec004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f6106b48640 1 --2- 172.21.15.72:0/455559573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f60ec004860 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009ac0 tx=0x7f60ec005830 comp rx=0 tx=0).stop 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f61050e5640 1 -- 172.21.15.72:0/455559573 reap_dead start 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f6106b48640 1 -- 172.21.15.72:0/455559573 shutdown_connections 2024-03-06T07:25:31.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f6106b48640 1 -- 172.21.15.72:0/455559573 >> 172.21.15.72:0/455559573 conn(0x7f61000b1a90 msgr2=0x7f61000b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f6106b48640 1 -- 172.21.15.72:0/455559573 shutdown_connections 2024-03-06T07:25:31.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.032+0000 7f6106b48640 1 -- 172.21.15.72:0/455559573 wait complete. 2024-03-06T07:25:31.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 Processor -- start 2024-03-06T07:25:31.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 -- start start 2024-03-06T07:25:31.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f6100158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100146ec0 0x7f6100143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610015bc20 0x7f610015c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61000b52b0 con 0x7f60ec004390 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610015aec0 con 0x7f6100146ec0 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.034+0000 7f6106b48640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61000b5430 con 0x7f610015bc20 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f6100158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f6100158400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59116/0 (socket says 172.21.15.72:59116) 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 -- 172.21.15.72:0/3010911298 learned_addr learned my addr 172.21.15.72:0/3010911298 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61050e5640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610015bc20 0x7f610015c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f60fffff640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100146ec0 0x7f6100143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 -- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610015bc20 msgr2=0x7f610015c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610015bc20 0x7f610015c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 -- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100146ec0 msgr2=0x7f6100143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100146ec0 0x7f6100143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 -- 172.21.15.72:0/3010911298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610015c5e0 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f61048e4640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f6100158400 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f60e800e870 tx=0x7f60e800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e800c870 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e8018070 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e800cc70 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6100158b30 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.035+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61001590f0 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.036+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610004f330 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.037+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f60e801d460 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.037+0000 7f60fd7fa640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60e006b4d0 0x7f60e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.037+0000 7f60fffff640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60e006b4d0 0x7f60e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.037+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f60e8089cf0 con 0x7f60ec004390 2024-03-06T07:25:31.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.038+0000 7f60fffff640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60e006b4d0 0x7f60e006d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009ac0 tx=0x7f60f0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.042+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f60e808e070 con 0x7f60ec004390 2024-03-06T07:25:31.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.215+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61000b2870 con 0x7f60ec004390 2024-03-06T07:25:31.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.215+0000 7f60fd7fa640 1 -- 172.21.15.72:0/3010911298 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13035 (secure 0 0 0) 0x7f610015c5e0 con 0x7f60ec004390 2024-03-06T07:25:31.218 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:31.218 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":74,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:29.902035+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6835/2032377664":"2024-03-07T07:25:26.886227+0000","172.21.15.146:6833/2032377664":"2024-03-07T07:25:26.886227+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.217+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60e006b4d0 msgr2=0x7f60e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.217+0000 7f6106b48640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60e006b4d0 0x7f60e006d990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009ac0 tx=0x7f60f0030000 comp rx=0 tx=0).stop 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.217+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 msgr2=0x7f6100158400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.217+0000 7f6106b48640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f6100158400 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f60e800e870 tx=0x7f60e800ed30 comp rx=0 tx=0).stop 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.217+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 shutdown_connections 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.217+0000 7f6106b48640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60e006b4d0 0x7f60e006d990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.218+0000 7f6106b48640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610015bc20 0x7f610015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.218+0000 7f6106b48640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6100146ec0 0x7f6100143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.218+0000 7f6106b48640 1 --2- 172.21.15.72:0/3010911298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60ec004390 0x7f6100158400 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.218+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 >> 172.21.15.72:0/3010911298 conn(0x7f61000b1a90 msgr2=0x7f61000b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.218+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 shutdown_connections 2024-03-06T07:25:31.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.218+0000 7f6106b48640 1 -- 172.21.15.72:0/3010911298 wait complete. 2024-03-06T07:25:31.231 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:31.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.331+0000 7fe295772640 1 Processor -- start 2024-03-06T07:25:31.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.331+0000 7fe295772640 1 -- start start 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe295772640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900d4860 0x7fe2900d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe295772640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2901618b0 con 0x7fe2901614d0 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe295772640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2900cf400 con 0x7fe2900b0e30 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe295772640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2900d4cc0 con 0x7fe2900d4860 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe2901614d0 0x7fe2900d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36826/0 (socket says 172.21.15.72:36826) 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28effd640 1 -- 172.21.15.72:0/536537848 learned_addr learned my addr 172.21.15.72:0/536537848 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28e7fc640 1 --2- 172.21.15.72:0/536537848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900d4860 0x7fe2900d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436602766 0 0) 0x7fe2901618b0 con 0x7fe2901614d0 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe278003520 con 0x7fe2901614d0 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545024125 0 0) 0x7fe2900cf400 con 0x7fe2900b0e30 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2901618b0 con 0x7fe2900b0e30 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1616024112 0 0) 0x7fe278003520 con 0x7fe2901614d0 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28e7fc640 1 -- 172.21.15.72:0/536537848 >> v1:172.21.15.146:6789/0 conn(0x7fe2900b0e30 legacy=0x7fe2900b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28e7fc640 1 -- 172.21.15.72:0/536537848 >> v1:172.21.15.72:6789/0 conn(0x7fe2901614d0 legacy=0x7fe2900d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.332+0000 7fe28e7fc640 1 -- 172.21.15.72:0/536537848 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2900cf6a0 con 0x7fe2900d4860 2024-03-06T07:25:31.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.333+0000 7fe28e7fc640 1 --2- 172.21.15.72:0/536537848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900d4860 0x7fe2900d2020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe2840097d0 tx=0x7fe28402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc0c5eb9500fafea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.333+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284037070 con 0x7fe2900d4860 2024-03-06T07:25:31.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe284030510 con 0x7fe2900d4860 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe28dffb640 1 -- 172.21.15.72:0/536537848 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe280001670 con 0x7fe2900d4860 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 -- 172.21.15.72:0/536537848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900d4860 msgr2=0x7fe2900d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 --2- 172.21.15.72:0/536537848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900d4860 0x7fe2900d2020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe2840097d0 tx=0x7fe28402b800 comp rx=0 tx=0).stop 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 -- 172.21.15.72:0/536537848 shutdown_connections 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 --2- 172.21.15.72:0/536537848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900d4860 0x7fe2900d2020 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 -- 172.21.15.72:0/536537848 >> 172.21.15.72:0/536537848 conn(0x7fe2900d3d40 msgr2=0x7fe2900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 -- 172.21.15.72:0/536537848 shutdown_connections 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.334+0000 7fe295772640 1 -- 172.21.15.72:0/536537848 wait complete. 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.335+0000 7fe295772640 1 Processor -- start 2024-03-06T07:25:31.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.335+0000 7fe295772640 1 -- start start 2024-03-06T07:25:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe295772640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900b0e30 0x7fe29014aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe295772640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 0x7fe2900b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe295772640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2901614d0 0x7fe2900ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe295772640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2900b0910 con 0x7fe2900d4860 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe295772640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2900cfc60 con 0x7fe2901614d0 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe295772640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2900d2560 con 0x7fe2900b0e30 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 0x7fe2900b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 0x7fe2900b48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59126/0 (socket says 172.21.15.72:59126) 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1995861110 learned_addr learned my addr 172.21.15.72:0/1995861110 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28f7fe640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2901614d0 0x7fe2900ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28effd640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900b0e30 0x7fe29014aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900b0e30 msgr2=0x7fe29014aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900b0e30 0x7fe29014aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2901614d0 msgr2=0x7fe2900ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2901614d0 0x7fe2900ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.336+0000 7fe28e7fc640 1 -- 172.21.15.72:0/1995861110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2900b9b50 con 0x7fe2900d4860 2024-03-06T07:25:31.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.337+0000 7fe28e7fc640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 0x7fe2900b48c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe284000c00 tx=0x7fe2840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.337+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284041020 con 0x7fe2900d4860 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.337+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28403c870 con 0x7fe2900d4860 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.337+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2900b3540 con 0x7fe2900d4860 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.337+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284037040 con 0x7fe2900d4860 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.337+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2900b3b00 con 0x7fe2900d4860 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.338+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2900d3530 con 0x7fe2900d4860 2024-03-06T07:25:31.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.339+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe284044020 con 0x7fe2900d4860 2024-03-06T07:25:31.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.346+0000 7fe2777fe640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe27c06b4d0 0x7fe27c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.346+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe2840aba20 con 0x7fe2900d4860 2024-03-06T07:25:31.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.346+0000 7fe28effd640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe27c06b4d0 0x7fe27c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.347+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2900d3530 con 0x7fe2900d4860 2024-03-06T07:25:31.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.347+0000 7fe28effd640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe27c06b4d0 0x7fe27c06d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe280001380 tx=0x7fe280009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.519+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2901607a0 con 0x7fe2900d4860 2024-03-06T07:25:31.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.519+0000 7fe2777fe640 1 -- 172.21.15.72:0/1995861110 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe28404b030 con 0x7fe2900d4860 2024-03-06T07:25:31.522 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe27c06b4d0 msgr2=0x7fe27c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe27c06b4d0 0x7fe27c06d990 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe280001380 tx=0x7fe280009040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 msgr2=0x7fe2900b48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 0x7fe2900b48c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe284000c00 tx=0x7fe2840046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 shutdown_connections 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe27c06b4d0 0x7fe27c06d990 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2901614d0 0x7fe2900ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2900d4860 0x7fe2900b48c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 --2- 172.21.15.72:0/1995861110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2900b0e30 0x7fe29014aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 >> 172.21.15.72:0/1995861110 conn(0x7fe2900d3d40 msgr2=0x7fe2900cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 shutdown_connections 2024-03-06T07:25:31.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.521+0000 7fe295772640 1 -- 172.21.15.72:0/1995861110 wait complete. 2024-03-06T07:25:31.535 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:25:31.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.632+0000 7f18d9c8f640 1 Processor -- start 2024-03-06T07:25:31.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.632+0000 7f18d9c8f640 1 -- start start 2024-03-06T07:25:31.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d9c8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d415cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d9c8f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18d406afb0 0x7f18d406b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d9c8f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d415c440 con 0x7f18d41563c0 2024-03-06T07:25:31.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d9c8f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d415c790 con 0x7f18d406afb0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d9c8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d406f370 con 0x7f18d40cd380 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18d406afb0 0x7f18d406b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f18d41563c0 0x7f18d4072720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36838/0 (socket says 172.21.15.72:36838) 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d37fe640 1 -- 172.21.15.72:0/2820794709 learned_addr learned my addr 172.21.15.72:0/2820794709 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d3fff640 1 -- 172.21.15.72:0/2820794709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 msgr2=0x7f18d415cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d3fff640 1 --2- 172.21.15.72:0/2820794709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d415cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d3fff640 1 -- 172.21.15.72:0/2820794709 >> v1:172.21.15.72:6789/0 conn(0x7f18d41563c0 legacy=0x7f18d4072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d27fc640 1 -- 172.21.15.72:0/2820794709 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971809490 0 0) 0x7f18d415c440 con 0x7f18d41563c0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.633+0000 7f18d3fff640 1 -- 172.21.15.72:0/2820794709 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d406f650 con 0x7f18d406afb0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.634+0000 7f18d3fff640 1 --2- 172.21.15.72:0/2820794709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d406b440 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d90 tx=0x7f18cc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed4a29a38ad7808 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.634+0000 7f18d27fc640 1 -- 172.21.15.72:0/2820794709 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc039070 con 0x7f18d406afb0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.634+0000 7f18d27fc640 1 -- 172.21.15.72:0/2820794709 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc02cc50 con 0x7f18d406afb0 2024-03-06T07:25:31.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.634+0000 7f18d27fc640 1 -- 172.21.15.72:0/2820794709 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18cc034440 con 0x7f18d406afb0 2024-03-06T07:25:31.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 -- 172.21.15.72:0/2820794709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 msgr2=0x7f18d406b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/2820794709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d406b440 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009d90 tx=0x7f18cc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 -- 172.21.15.72:0/2820794709 shutdown_connections 2024-03-06T07:25:31.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/2820794709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d406b440 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/2820794709 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d415cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 -- 172.21.15.72:0/2820794709 >> 172.21.15.72:0/2820794709 conn(0x7f18d406c0c0 msgr2=0x7f18d406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 -- 172.21.15.72:0/2820794709 shutdown_connections 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.635+0000 7f18d9c8f640 1 -- 172.21.15.72:0/2820794709 wait complete. 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.636+0000 7f18d9c8f640 1 Processor -- start 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.636+0000 7f18d9c8f640 1 -- start start 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d9c8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d415e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d9c8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d40ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d9c8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d41563c0 0x7f18d40cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d9c8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40d0720 con 0x7f18d406afb0 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d9c8f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40d05a0 con 0x7f18d41563c0 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d9c8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d40d08a0 con 0x7f18d40cd380 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59148/0 (socket says 172.21.15.72:59148) 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 -- 172.21.15.72:0/3496815175 learned_addr learned my addr 172.21.15.72:0/3496815175 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d2ffd640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d40ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d3fff640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d41563c0 0x7f18d40cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 -- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 msgr2=0x7f18d40ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d40ca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 -- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d41563c0 msgr2=0x7f18d40cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d41563c0 0x7f18d40cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 -- 172.21.15.72:0/3496815175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d40b4a70 con 0x7f18d406afb0 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d2ffd640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d40ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d3fff640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d41563c0 0x7f18d40cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.637+0000 7f18d37fe640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d415e680 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f18c400c430 tx=0x7f18c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.638+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40074e0 con 0x7f18d406afb0 2024-03-06T07:25:31.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.638+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c40047e0 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.638+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d40cb440 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.638+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4004bd0 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.638+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d40cba00 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.639+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d40d00e0 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.639+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f18c4011070 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.639+0000 7f18bbfff640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18c006b4d0 0x7f18c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.639+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f18c4089a80 con 0x7f18d406afb0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.639+0000 7f18d2ffd640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18c006b4d0 0x7f18c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:31.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.640+0000 7f18d2ffd640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18c006b4d0 0x7f18c006d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f18d40b53c0 tx=0x7f18c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:31.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.644+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f18d40d00e0 con 0x7f18d406afb0 2024-03-06T07:25:31.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.882+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7f18d404f330 con 0x7f18d406afb0 2024-03-06T07:25:31.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.917+0000 7f18bbfff640 1 -- 172.21.15.72:0/3496815175 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v210) v1 ==== 135+0+0 (secure 0 0 0) 0x7f18c400e280 con 0x7f18d406afb0 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.918+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18c006b4d0 msgr2=0x7f18c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18c006b4d0 0x7f18c006d990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f18d40b53c0 tx=0x7f18c8008040 comp rx=0 tx=0).stop 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 msgr2=0x7f18d415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d415e680 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f18c400c430 tx=0x7f18c400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 shutdown_connections 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f18c006b4d0 0x7f18c006d990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18d41563c0 0x7f18d40cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18d40cd380 0x7f18d40ca610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 --2- 172.21.15.72:0/3496815175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18d406afb0 0x7f18d415e680 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 >> 172.21.15.72:0/3496815175 conn(0x7f18d406c0c0 msgr2=0x7f18d415adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 shutdown_connections 2024-03-06T07:25:31.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:31.919+0000 7f18d9c8f640 1 -- 172.21.15.72:0/3496815175 wait complete. 2024-03-06T07:25:31.932 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 Processor -- start 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 -- start start 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f6c0d4860 0x7f9f6c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 0x7f9f6c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c1618b0 con 0x7f9f6c1614d0 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c0d4cf0 con 0x7f9f6c0d4860 2024-03-06T07:25:32.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.034+0000 7f9f72fd6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0cf400 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 0x7f9f6c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f6bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f6c0d4860 0x7f9f6c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 0x7f9f6c0b12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:55028/0 (socket says 172.21.15.72:55028) 2024-03-06T07:25:32.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 -- 172.21.15.72:0/2327617737 learned_addr learned my addr 172.21.15.72:0/2327617737 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 -- 172.21.15.72:0/2327617737 >> v2:172.21.15.146:3300/0 conn(0x7f9f6c0d4860 msgr2=0x7f9f6c0d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 --2- 172.21.15.72:0/2327617737 >> v2:172.21.15.146:3300/0 conn(0x7f9f6c0d4860 0x7f9f6c0d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 -- 172.21.15.72:0/2327617737 >> v1:172.21.15.146:6789/0 conn(0x7f9f6c1614d0 legacy=0x7f9f6c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.035+0000 7f9f71573640 1 -- 172.21.15.72:0/2327617737 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0cf6a0 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.036+0000 7f9f71573640 1 --2- 172.21.15.72:0/2327617737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 0x7f9f6c0b12a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009870 tx=0x7f9f5c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6663e78ab891454 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.036+0000 7f9f6b7fe640 1 -- 172.21.15.72:0/2327617737 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c034070 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.036+0000 7f9f6b7fe640 1 -- 172.21.15.72:0/2327617737 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c030840 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.036+0000 7f9f6b7fe640 1 -- 172.21.15.72:0/2327617737 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c030c10 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 -- 172.21.15.72:0/2327617737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 msgr2=0x7f9f6c0b12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/2327617737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 0x7f9f6c0b12a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009870 tx=0x7f9f5c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:32.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 -- 172.21.15.72:0/2327617737 shutdown_connections 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/2327617737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c0b0e30 0x7f9f6c0b12a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/2327617737 >> v2:172.21.15.146:3300/0 conn(0x7f9f6c0d4860 0x7f9f6c0d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 -- 172.21.15.72:0/2327617737 >> 172.21.15.72:0/2327617737 conn(0x7f9f6c0d3d40 msgr2=0x7f9f6c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 -- 172.21.15.72:0/2327617737 shutdown_connections 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.037+0000 7f9f72fd6640 1 -- 172.21.15.72:0/2327617737 wait complete. 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.038+0000 7f9f72fd6640 1 Processor -- start 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.038+0000 7f9f72fd6640 1 -- start start 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f72fd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 0x7f9f6c14ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f72fd6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f6c0d4860 0x7f9f6c14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f72fd6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c1614d0 0x7f9f6c0b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f72fd6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0b0910 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f72fd6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0cfc60 con 0x7f9f6c0d4860 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f72fd6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0d2560 con 0x7f9f6c1614d0 2024-03-06T07:25:32.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 0x7f9f6c14ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 0x7f9f6c14ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59164/0 (socket says 172.21.15.72:59164) 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 -- 172.21.15.72:0/3439299152 learned_addr learned my addr 172.21.15.72:0/3439299152 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f6bfff640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f6c0d4860 0x7f9f6c14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f71573640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c1614d0 0x7f9f6c0b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 -- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c1614d0 msgr2=0x7f9f6c0b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c1614d0 0x7f9f6c0b8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 -- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f6c0d4860 msgr2=0x7f9f6c14df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f6c0d4860 0x7f9f6c14df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f70d72640 1 -- 172.21.15.72:0/3439299152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0b9110 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f6bfff640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f6c0d4860 0x7f9f6c14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.039+0000 7f9f71573640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c1614d0 0x7f9f6c0b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.040+0000 7f9f70d72640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 0x7f9f6c14ad00 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c760 tx=0x7f9f6400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.040+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64016020 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.040+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64006d50 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.040+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c0b93f0 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.040+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f64003bb0 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.040+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c0b99b0 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.041+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c068ff0 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.041+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f6400a850 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.041+0000 7f9f697fa640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f5806b4d0 0x7f9f5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.041+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9f64089ae0 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.041+0000 7f9f6bfff640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f5806b4d0 0x7f9f5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.042+0000 7f9f6bfff640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f5806b4d0 0x7f9f5806d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c1f7580 tx=0x7f9f60008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.046+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f64020020 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.217+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9f6c160680 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.220+0000 7f9f697fa640 1 -- 172.21.15.72:0/3439299152 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9f64019310 con 0x7f9f6c0b0e30 2024-03-06T07:25:32.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f5806b4d0 msgr2=0x7f9f5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f5806b4d0 0x7f9f5806d990 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c1f7580 tx=0x7f9f60008040 comp rx=0 tx=0).stop 2024-03-06T07:25:32.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 msgr2=0x7f9f6c14ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 0x7f9f6c14ad00 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f9f6400c760 tx=0x7f9f6400cc20 comp rx=0 tx=0).stop 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 shutdown_connections 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f5806b4d0 0x7f9f5806d990 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f6c1614d0 0x7f9f6c0b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f6c0d4860 0x7f9f6c14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 --2- 172.21.15.72:0/3439299152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f6c0b0e30 0x7f9f6c14ad00 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 >> 172.21.15.72:0/3439299152 conn(0x7f9f6c0d3d40 msgr2=0x7f9f6c0d1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 shutdown_connections 2024-03-06T07:25:32.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.222+0000 7f9f72fd6640 1 -- 172.21.15.72:0/3439299152 wait complete. 2024-03-06T07:25:32.235 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:32.235 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:32.272 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:32.325 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:32.326 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:32.326 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:32.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:32.341 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:25:32.341 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:32.359 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:32.359 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:32.359 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:32.397 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:32.397 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:25:32.397 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:32.416 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:32.416 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:25:32.417 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:32.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.534+0000 7f59f3288640 1 Processor -- start 2024-03-06T07:25:32.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.534+0000 7f59f3288640 1 -- start start 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.535+0000 7f59f3288640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 0x7f59ec0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.535+0000 7f59f3288640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec0b3430 con 0x7f59ec0d22b0 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.535+0000 7f59f3288640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec0b3710 con 0x7f59ec15dfe0 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.535+0000 7f59f3288640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0b31e0 con 0x7f59ec0d2710 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.535+0000 7f59f0823640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f59ec0d22b0 0x7f59ec0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38774/0 (socket says 172.21.15.72:38774) 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.535+0000 7f59f0823640 1 -- 172.21.15.72:0/2637934540 learned_addr learned my addr 172.21.15.72:0/2637934540 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613493112 0 0) 0x7f59ec0b3430 con 0x7f59ec0d22b0 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d8003520 con 0x7f59ec0d22b0 2024-03-06T07:25:32.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f1825640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 0x7f59ec0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977754018 0 0) 0x7f59d8003520 con 0x7f59ec0d22b0 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 msgr2=0x7f59ec0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 0x7f59ec0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 >> v1:172.21.15.146:6789/0 conn(0x7f59ec15dfe0 legacy=0x7f59ec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0b39f0 con 0x7f59ec0d22b0 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f59dc001670 con 0x7f59ec0d22b0 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 >> v1:172.21.15.72:6789/0 conn(0x7f59ec0d22b0 legacy=0x7f59ec0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d80039e0 0x7f59d8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 0x7f59d8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0b31e0 con 0x7f59d8004800 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800bb50 con 0x7f59d80039e0 2024-03-06T07:25:32.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d800bd50 con 0x7f59ec0d2710 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f0823640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f1825640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 0x7f59d8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f1024640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d80039e0 0x7f59d8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f0823640 1 -- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 msgr2=0x7f59d8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f0823640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec0d2710 0x7f59d8005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f0823640 1 -- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d80039e0 msgr2=0x7f59d8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f0823640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d80039e0 0x7f59d8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f0823640 1 -- 172.21.15.72:0/2637934540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8003520 con 0x7f59d8004800 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.536+0000 7f59f1024640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59d80039e0 0x7f59d8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.537+0000 7f59f0823640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c240 tx=0x7f59d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b331066fe279f15f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.537+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc007d10 con 0x7f59d8004800 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.537+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc030070 con 0x7f59d8004800 2024-03-06T07:25:32.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.537+0000 7f59e3fff640 1 -- 172.21.15.72:0/2637934540 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc02cd40 con 0x7f59d8004800 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f3288640 1 -- 172.21.15.72:0/2637934540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 msgr2=0x7f59d8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f3288640 1 --2- 172.21.15.72:0/2637934540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59d8004c80 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c240 tx=0x7f59d8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f1825640 1 -- 172.21.15.72:0/2637934540 reap_dead start 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f3288640 1 -- 172.21.15.72:0/2637934540 shutdown_connections 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f3288640 1 -- 172.21.15.72:0/2637934540 >> 172.21.15.72:0/2637934540 conn(0x7f59ec0d3cd0 msgr2=0x7f59ec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f3288640 1 -- 172.21.15.72:0/2637934540 shutdown_connections 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.538+0000 7f59f3288640 1 -- 172.21.15.72:0/2637934540 wait complete. 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.539+0000 7f59f3288640 1 Processor -- start 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.539+0000 7f59f3288640 1 -- start start 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59ec1f42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59ec0b4eb0 0x7f59ec06cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec15bdc0 0x7f59ec15c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0cfa00 con 0x7f59d8004800 2024-03-06T07:25:32.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0b3d60 con 0x7f59ec0b4eb0 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0d1c70 con 0x7f59ec15bdc0 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59ec1f42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59ec1f42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47082/0 (socket says 172.21.15.72:47082) 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 -- 172.21.15.72:0/961282950 learned_addr learned my addr 172.21.15.72:0/961282950 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1825640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec15bdc0 0x7f59ec15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f0823640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59ec0b4eb0 0x7f59ec06cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 -- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec15bdc0 msgr2=0x7f59ec15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec15bdc0 0x7f59ec15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 -- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59ec0b4eb0 msgr2=0x7f59ec06cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59ec0b4eb0 0x7f59ec06cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 -- 172.21.15.72:0/961282950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec15c780 con 0x7f59d8004800 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f0823640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59ec0b4eb0 0x7f59ec06cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f1024640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59ec1f42d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f59e800e550 tx=0x7f59e800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e80052e0 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e8018070 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec1f4b20 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.540+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e8003830 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.541+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec1f50e0 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.541+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec04f330 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.542+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f59e80039d0 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.542+0000 7f59e1ffb640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.542+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f59e8089b40 con 0x7f59d8004800 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.542+0000 7f59f0823640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.543+0000 7f59f0823640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59d006b500 0x7f59d006d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c240 tx=0x7f59dc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.547+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f59ec15c780 con 0x7f59d8004800 2024-03-06T07:25:32.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.753+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59ec1f57a0 con 0x7f59d8004800 2024-03-06T07:25:32.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.753+0000 7f59e1ffb640 1 -- 172.21.15.72:0/961282950 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f59e8058cd0 con 0x7f59d8004800 2024-03-06T07:25:32.756 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:32.756 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5375,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1258107150","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1258107150},{"type":"v1","addr":"172.21.15.146:6834","nonce":1258107150}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6106,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1246970671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":1246970671},{"type":"v1","addr":"172.21.15.146:6835","nonce":1246970671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6564,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3209472963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3209472963},{"type":"v1","addr":"172.21.15.72:6836","nonce":3209472963}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6567,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/2639759188","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":2639759188},{"type":"v1","addr":"172.21.15.72:6837","nonce":2639759188}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-03-06T07:25:32.756 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 35 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59d006b500 msgr2=0x7f59d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59d006b500 0x7f59d006d9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c240 tx=0x7f59dc009030 comp rx=0 tx=0).stop 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 msgr2=0x7f59ec1f42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59ec1f42d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f59e800e550 tx=0x7f59e800ea10 comp rx=0 tx=0).stop 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 shutdown_connections 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f59d006b500 0x7f59d006d9c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59ec15bdc0 0x7f59ec15c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59ec0b4eb0 0x7f59ec06cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 --2- 172.21.15.72:0/961282950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59d8004800 0x7f59ec1f42d0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 >> 172.21.15.72:0/961282950 conn(0x7f59ec0d3cd0 msgr2=0x7f59ec1603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 shutdown_connections 2024-03-06T07:25:32.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.755+0000 7f59f3288640 1 -- 172.21.15.72:0/961282950 wait complete. 2024-03-06T07:25:32.768 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:25:32.769 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:25:32.769 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:25:32.769 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:32.769 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:25:32.769 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:25:32.770 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:25:32.770 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:32.770 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:25:32.770 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:25:32.770 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:25:32.770 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:32.771 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:25:32.771 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:25:32.771 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:25:32.771 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:25:32.779 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:25:32.779 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:25:32.779 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:25:32.780 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:25:32.783 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:25:32.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:25:32.784 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:25:32.784 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:25:32.821 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:25:32.821 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:32.822 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:25:32.822 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:32.826 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:25:32.827 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:32.827 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:25:32.827 DEBUG:teuthology.parallel:result is None 2024-03-06T07:25:32.828 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:25:32.897 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:25:32.897 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:25:32.911 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:25:32.911 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.933+0000 7f1c32981640 1 Processor -- start 2024-03-06T07:25:32.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.933+0000 7f1c32981640 1 -- start start 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c32981640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c32981640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c2c161cf0 0x7f1c2c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c32981640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2c0d22c0 con 0x7f1c2c0b4460 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c32981640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2c161b00 con 0x7f1c2c161cf0 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c32981640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0cecc0 con 0x7f1c2c162970 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c30f1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c2c161cf0 0x7f1c2c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c30f1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c2c161cf0 0x7f1c2c162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47114/0 (socket says 172.21.15.72:47114) 2024-03-06T07:25:32.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.934+0000 7f1c30f1e640 1 -- 172.21.15.72:0/2914260731 learned_addr learned my addr 172.21.15.72:0/2914260731 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c2b7fe640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c30f1e640 1 -- 172.21.15.72:0/2914260731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 msgr2=0x7f1c2c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c30f1e640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c30f1e640 1 -- 172.21.15.72:0/2914260731 >> v1:172.21.15.146:6789/0 conn(0x7f1c2c0b4460 legacy=0x7f1c2c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c30f1e640 1 -- 172.21.15.72:0/2914260731 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0d25a0 con 0x7f1c2c161cf0 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c2b7fe640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.935+0000 7f1c30f1e640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c162180 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009e50 tx=0x7f1c2402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca12b5f4fc885796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.936+0000 7f1c2affd640 1 -- 172.21.15.72:0/2914260731 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2402caf0 con 0x7f1c2c161cf0 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.936+0000 7f1c2affd640 1 -- 172.21.15.72:0/2914260731 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2402cc90 con 0x7f1c2c161cf0 2024-03-06T07:25:32.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.936+0000 7f1c2affd640 1 -- 172.21.15.72:0/2914260731 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24034610 con 0x7f1c2c161cf0 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.936+0000 7f1c32981640 1 -- 172.21.15.72:0/2914260731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 msgr2=0x7f1c2c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.936+0000 7f1c32981640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c162180 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009e50 tx=0x7f1c2402bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.936+0000 7f1c32981640 1 -- 172.21.15.72:0/2914260731 shutdown_connections 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.937+0000 7f1c32981640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c162180 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.937+0000 7f1c32981640 1 --2- 172.21.15.72:0/2914260731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.937+0000 7f1c32981640 1 -- 172.21.15.72:0/2914260731 >> 172.21.15.72:0/2914260731 conn(0x7f1c2c0d3ce0 msgr2=0x7f1c2c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.937+0000 7f1c32981640 1 -- 172.21.15.72:0/2914260731 shutdown_connections 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.937+0000 7f1c32981640 1 -- 172.21.15.72:0/2914260731 wait complete. 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.938+0000 7f1c32981640 1 Processor -- start 2024-03-06T07:25:32.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.938+0000 7f1c32981640 1 -- start start 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c32981640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2c0b4460 0x7f1c2c155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c32981640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c0ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c32981640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c32981640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0ce950 con 0x7f1c2c161cf0 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c32981640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0d2b60 con 0x7f1c2c0b4460 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c32981640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c0b4840 con 0x7f1c2c162970 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c0ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c0ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47122/0 (socket says 172.21.15.72:47122) 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2c0b4460 0x7f1c2c155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c30f1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2bfff640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2c0b4460 0x7f1c2c155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:56070/0 (socket says 172.21.15.72:56070) 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 -- 172.21.15.72:0/2879286141 learned_addr learned my addr 172.21.15.72:0/2879286141 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 -- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 msgr2=0x7f1c2c0cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 -- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2c0b4460 msgr2=0x7f1c2c155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2c0b4460 0x7f1c2c155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c2b7fe640 1 -- 172.21.15.72:0/2879286141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c0c7940 con 0x7f1c2c161cf0 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.939+0000 7f1c30f1e640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.940+0000 7f1c2b7fe640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c0ccfd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000d800 tx=0x7f1c2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.940+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20004030 con 0x7f1c2c161cf0 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.940+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c0c7c20 con 0x7f1c2c161cf0 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.940+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c200041d0 con 0x7f1c2c161cf0 2024-03-06T07:25:32.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.940+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000b6c0 con 0x7f1c2c161cf0 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.940+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c156060 con 0x7f1c2c161cf0 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.941+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c04f330 con 0x7f1c2c161cf0 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.941+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c2000b860 con 0x7f1c2c161cf0 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.942+0000 7f1c28ff9640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.942+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1c2008a0d0 con 0x7f1c2c161cf0 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.942+0000 7f1c2bfff640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:32.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.942+0000 7f1c2bfff640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1806b4d0 0x7f1c1806d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0016e0 tx=0x7f1c1c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:32.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:32.946+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c20058b80 con 0x7f1c2c161cf0 2024-03-06T07:25:33.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.129+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1c2c06b180 con 0x7f1c2c161cf0 2024-03-06T07:25:33.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.936+0000 7f1c28ff9640 1 -- 172.21.15.72:0/2879286141 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1c2005cc20 con 0x7f1c2c161cf0 2024-03-06T07:25:33.938 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.938+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1806b4d0 msgr2=0x7f1c1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.938+0000 7f1c32981640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1806b4d0 0x7f1c1806d990 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f1c1c0016e0 tx=0x7f1c1c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.938+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 msgr2=0x7f1c2c0ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.938+0000 7f1c32981640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c0ccfd0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000d800 tx=0x7f1c2000dcc0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 shutdown_connections 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1806b4d0 0x7f1c1806d990 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c2c162970 0x7f1c2c0cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c2c161cf0 0x7f1c2c0ccfd0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 --2- 172.21.15.72:0/2879286141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c2c0b4460 0x7f1c2c155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:33.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 >> 172.21.15.72:0/2879286141 conn(0x7f1c2c0d3ce0 msgr2=0x7f1c2c0cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:33.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 shutdown_connections 2024-03-06T07:25:33.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:33.939+0000 7f1c32981640 1 -- 172.21.15.72:0/2879286141 wait complete. 2024-03-06T07:25:33.954 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:25:34.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.058+0000 7f99653e4640 1 Processor -- start 2024-03-06T07:25:34.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.058+0000 7f99653e4640 1 -- start start 2024-03-06T07:25:34.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f99653e4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9960166af0 0x7f9960168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f99653e4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f996015e380 con 0x7f99600cf850 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f99653e4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99600cdc10 con 0x7f99600d00c0 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f99653e4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996006ca90 con 0x7f9960166af0 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995e7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f99600cf850 0x7f99600cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38786/0 (socket says 172.21.15.72:38786) 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995e7fc640 1 -- 172.21.15.72:0/163569706 learned_addr learned my addr 172.21.15.72:0/163569706 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999644709 0 0) 0x7f996015e380 con 0x7f99600cf850 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9948003520 con 0x7f99600cf850 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995f7fe640 1 --2- 172.21.15.72:0/163569706 >> v2:172.21.15.146:3300/0 conn(0x7f9960166af0 0x7f9960168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281701722 0 0) 0x7f9948003520 con 0x7f99600cf850 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 >> v2:172.21.15.146:3300/0 conn(0x7f9960166af0 msgr2=0x7f9960168ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 --2- 172.21.15.72:0/163569706 >> v2:172.21.15.146:3300/0 conn(0x7f9960166af0 0x7f9960168ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 >> v1:172.21.15.146:6789/0 conn(0x7f99600d00c0 legacy=0x7f99600d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996006c160 con 0x7f99600cf850 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9954001670 con 0x7f99600cf850 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 >> v1:172.21.15.72:6789/0 conn(0x7f99600cf850 legacy=0x7f99600cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99480039b0 0x7f9948003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9948004800 0x7f9948004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99480051c0 0x7f994800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996006ca90 con 0x7f9948004800 2024-03-06T07:25:34.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994800c060 con 0x7f99480051c0 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994800c260 con 0x7f99480039b0 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.059+0000 7f995e7fc640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9948004800 0x7f9948004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995effd640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99480039b0 0x7f9948003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995f7fe640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99480051c0 0x7f994800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995e7fc640 1 -- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99480039b0 msgr2=0x7f9948003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995e7fc640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99480039b0 0x7f9948003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995e7fc640 1 -- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99480051c0 msgr2=0x7f994800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995e7fc640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99480051c0 0x7f994800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995e7fc640 1 -- 172.21.15.72:0/163569706 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948003520 con 0x7f9948004800 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995effd640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99480039b0 0x7f9948003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995f7fe640 1 -- 172.21.15.72:0/163569706 reap_dead start 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995e7fc640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9948004800 0x7f9948004c80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f995400c210 tx=0x7f9948005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afcc835a787ec22 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.060+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954007d30 con 0x7f9948004800 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9954030070 con 0x7f9948004800 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f995dffb640 1 -- 172.21.15.72:0/163569706 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9954005d70 con 0x7f9948004800 2024-03-06T07:25:34.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 -- 172.21.15.72:0/163569706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9948004800 msgr2=0x7f9948004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9948004800 0x7f9948004c80 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f995400c210 tx=0x7f9948005830 comp rx=0 tx=0).stop 2024-03-06T07:25:34.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 -- 172.21.15.72:0/163569706 shutdown_connections 2024-03-06T07:25:34.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 --2- 172.21.15.72:0/163569706 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9948004800 0x7f9948004c80 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 -- 172.21.15.72:0/163569706 >> 172.21.15.72:0/163569706 conn(0x7f99600b35a0 msgr2=0x7f99600b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 -- 172.21.15.72:0/163569706 shutdown_connections 2024-03-06T07:25:34.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.061+0000 7f99653e4640 1 -- 172.21.15.72:0/163569706 wait complete. 2024-03-06T07:25:34.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.062+0000 7f99653e4640 1 Processor -- start 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.062+0000 7f99653e4640 1 -- start start 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f99653e4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9948004800 0x7f99600c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f99653e4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601555a0 0x7f99600c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f99653e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 0x7f996006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f99653e4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99600b1030 con 0x7f996006d6c0 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f99653e4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f996006c720 con 0x7f99601555a0 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f99653e4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996015adf0 con 0x7f9948004800 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 0x7f996006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 0x7f996006db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47132/0 (socket says 172.21.15.72:47132) 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 -- 172.21.15.72:0/3967312059 learned_addr learned my addr 172.21.15.72:0/3967312059 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 -- 172.21.15.72:0/3967312059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9948004800 msgr2=0x7f99600c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9948004800 0x7f99600c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 -- 172.21.15.72:0/3967312059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601555a0 msgr2=0x7f99600c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601555a0 0x7f99600c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.063+0000 7f995f7fe640 1 -- 172.21.15.72:0/3967312059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99600c87f0 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.064+0000 7f995f7fe640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 0x7f996006db40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9960072110 tx=0x7f99580115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.064+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958011de0 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.064+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995801b070 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.064+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996015a4c0 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.064+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958003990 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.064+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996015aa80 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.065+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996004f330 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.065+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9958005d10 con 0x7f996006d6c0 2024-03-06T07:25:34.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.065+0000 7f99477fe640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f994c06b4d0 0x7f994c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.066+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f995808a2e0 con 0x7f996006d6c0 2024-03-06T07:25:34.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.066+0000 7f995effd640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f994c06b4d0 0x7f994c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.066+0000 7f995effd640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f994c06b4d0 0x7f994c06d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f99600686a0 tx=0x7f9950009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.070+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f995801d310 con 0x7f996006d6c0 2024-03-06T07:25:34.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.306+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f996015db80 con 0x7f996006d6c0 2024-03-06T07:25:34.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.306+0000 7f99477fe640 1 -- 172.21.15.72:0/3967312059 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f995805de20 con 0x7f996006d6c0 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.309 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.310 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.311 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.312 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.313 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.314 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.315 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.316 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.317 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.318 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:25:34.319 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.320 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:25:34.321 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:25:34.322 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.322 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:34.322 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:25:34.322 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f994c06b4d0 msgr2=0x7f994c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f994c06b4d0 0x7f994c06d990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f99600686a0 tx=0x7f9950009040 comp rx=0 tx=0).stop 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 msgr2=0x7f996006db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 0x7f996006db40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f9960072110 tx=0x7f99580115a0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 shutdown_connections 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f994c06b4d0 0x7f994c06d990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f996006d6c0 0x7f996006db40 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99601555a0 0x7f99600c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 --2- 172.21.15.72:0/3967312059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9948004800 0x7f99600c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 >> 172.21.15.72:0/3967312059 conn(0x7f99600b35a0 msgr2=0x7f996006fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 shutdown_connections 2024-03-06T07:25:34.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.308+0000 7f99653e4640 1 -- 172.21.15.72:0/3967312059 wait complete. 2024-03-06T07:25:34.324 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.422+0000 7efc83276640 1 Processor -- start 2024-03-06T07:25:34.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.422+0000 7efc83276640 1 -- start start 2024-03-06T07:25:34.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc83276640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc7c0b3060 0x7efc7c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc83276640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c15f580 con 0x7efc7c15bf30 2024-03-06T07:25:34.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc83276640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc7c15b450 con 0x7efc7c13dc10 2024-03-06T07:25:34.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc83276640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7c0cf3f0 con 0x7efc7c0b3060 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc81813640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efc7c15bf30 0x7efc7c15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38796/0 (socket says 172.21.15.72:38796) 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc81813640 1 -- 172.21.15.72:0/3350511248 learned_addr learned my addr 172.21.15.72:0/3350511248 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc80811640 1 --2- 172.21.15.72:0/3350511248 >> v2:172.21.15.146:3300/0 conn(0x7efc7c0b3060 0x7efc7c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403460982 0 0) 0x7efc7c15f580 con 0x7efc7c15bf30 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68003520 con 0x7efc7c15bf30 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 539287910 0 0) 0x7efc68003520 con 0x7efc7c15bf30 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 >> v2:172.21.15.146:3300/0 conn(0x7efc7c0b3060 msgr2=0x7efc7c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 --2- 172.21.15.72:0/3350511248 >> v2:172.21.15.146:3300/0 conn(0x7efc7c0b3060 0x7efc7c15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 >> v1:172.21.15.146:6789/0 conn(0x7efc7c13dc10 legacy=0x7efc7c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c15b730 con 0x7efc7c15bf30 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7efc740019f0 con 0x7efc7c15bf30 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 >> v1:172.21.15.72:6789/0 conn(0x7efc7c15bf30 legacy=0x7efc7c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc680039b0 0x7efc68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc680043b0 0x7efc68004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc68005120 0x7efc6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.423+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0cf3f0 con 0x7efc68005120 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6800c060 con 0x7efc680043b0 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6800c260 con 0x7efc680039b0 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc68005120 0x7efc6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc80811640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc680043b0 0x7efc68004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81012640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc680039b0 0x7efc68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 -- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc680039b0 msgr2=0x7efc68003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc680039b0 0x7efc68003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 -- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc680043b0 msgr2=0x7efc68004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc680043b0 0x7efc68004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 -- 172.21.15.72:0/3350511248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc68003520 con 0x7efc68005120 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81012640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc680039b0 0x7efc68003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.424+0000 7efc81813640 1 -- 172.21.15.72:0/3350511248 reap_dead start 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc81813640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc68005120 0x7efc6800b9c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7efc7400a860 tx=0x7efc68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2289fb6b978b564d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7400a480 con 0x7efc68005120 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74034630 con 0x7efc68005120 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc73fff640 1 -- 172.21.15.72:0/3350511248 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74034a00 con 0x7efc68005120 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 -- 172.21.15.72:0/3350511248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc68005120 msgr2=0x7efc6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc68005120 0x7efc6800b9c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7efc7400a860 tx=0x7efc68005830 comp rx=0 tx=0).stop 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 -- 172.21.15.72:0/3350511248 shutdown_connections 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 --2- 172.21.15.72:0/3350511248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc68005120 0x7efc6800b9c0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 -- 172.21.15.72:0/3350511248 >> 172.21.15.72:0/3350511248 conn(0x7efc7c0cf8a0 msgr2=0x7efc7c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 -- 172.21.15.72:0/3350511248 shutdown_connections 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.425+0000 7efc83276640 1 -- 172.21.15.72:0/3350511248 wait complete. 2024-03-06T07:25:34.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.426+0000 7efc83276640 1 Processor -- start 2024-03-06T07:25:34.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.426+0000 7efc83276640 1 -- start start 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc83276640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc68005120 0x7efc7c13c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc83276640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc7c156ae0 0x7efc7c13ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc83276640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 0x7efc7c0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc83276640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b2950 con 0x7efc7c0b0740 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc83276640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c15b210 con 0x7efc68005120 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc83276640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc7c0b2ad0 con 0x7efc7c156ae0 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc81813640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 0x7efc7c0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc81813640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 0x7efc7c0b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47144/0 (socket says 172.21.15.72:47144) 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc81813640 1 -- 172.21.15.72:0/1613757382 learned_addr learned my addr 172.21.15.72:0/1613757382 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc80811640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc7c156ae0 0x7efc7c13ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.427+0000 7efc81012640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc68005120 0x7efc7c13c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81813640 1 -- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc7c156ae0 msgr2=0x7efc7c13ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81813640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc7c156ae0 0x7efc7c13ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81813640 1 -- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc68005120 msgr2=0x7efc7c13c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81813640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc68005120 0x7efc7c13c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81813640 1 -- 172.21.15.72:0/1613757382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc7c157940 con 0x7efc7c0b0740 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81012640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc68005120 0x7efc7c13c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc81813640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 0x7efc7c0b0bc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7efc7402cc50 tx=0x7efc7c15b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc74039040 con 0x7efc7c0b0740 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc74043070 con 0x7efc7c0b0740 2024-03-06T07:25:34.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7c13d230 con 0x7efc7c0b0740 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7400a480 con 0x7efc7c0b0740 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.428+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7c157380 con 0x7efc7c0b0740 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.429+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7c04f330 con 0x7efc7c0b0740 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.429+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc74034630 con 0x7efc7c0b0740 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.430+0000 7efc71ffb640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.430+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efc740abb70 con 0x7efc7c0b0740 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.430+0000 7efc81012640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.430+0000 7efc81012640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc6006b4d0 0x7efc6006d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc78005580 tx=0x7efc78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.434+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc7407add0 con 0x7efc7c0b0740 2024-03-06T07:25:34.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.605+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc7c06eb00 con 0x7efc7c0b0740 2024-03-06T07:25:34.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.606+0000 7efc71ffb640 1 -- 172.21.15.72:0/1613757382 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+12904 (secure 0 0 0) 0x7efc7407a6d0 con 0x7efc7c0b0740 2024-03-06T07:25:34.608 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:34.608 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":79,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:33.934202+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:34.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc6006b4d0 msgr2=0x7efc6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc6006b4d0 0x7efc6006d990 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7efc78005580 tx=0x7efc78009040 comp rx=0 tx=0).stop 2024-03-06T07:25:34.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 msgr2=0x7efc7c0b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 0x7efc7c0b0bc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7efc7402cc50 tx=0x7efc7c15b620 comp rx=0 tx=0).stop 2024-03-06T07:25:34.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 shutdown_connections 2024-03-06T07:25:34.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc6006b4d0 0x7efc6006d990 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc7c0b0740 0x7efc7c0b0bc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc7c156ae0 0x7efc7c13ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 --2- 172.21.15.72:0/1613757382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc68005120 0x7efc7c13c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 >> 172.21.15.72:0/1613757382 conn(0x7efc7c0cf8a0 msgr2=0x7efc7c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 shutdown_connections 2024-03-06T07:25:34.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.608+0000 7efc83276640 1 -- 172.21.15.72:0/1613757382 wait complete. 2024-03-06T07:25:34.621 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:34.621 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.723+0000 7f52999a1640 1 Processor -- start 2024-03-06T07:25:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.723+0000 7f52999a1640 1 -- start start 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52999a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 0x7f52940d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52999a1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52941618b0 con 0x7f52940d1910 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52999a1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52940cf400 con 0x7f52940d2480 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52999a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52940cf6a0 con 0x7f52941614d0 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52927fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f52940d1910 0x7f52940d1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38812/0 (socket says 172.21.15.72:38812) 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52927fc640 1 -- 172.21.15.72:0/3300135145 learned_addr learned my addr 172.21.15.72:0/3300135145 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168342154 0 0) 0x7f52941618b0 con 0x7f52940d1910 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5292ffd640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 0x7f52940d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f527c003520 con 0x7f52940d1910 2024-03-06T07:25:34.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58979300 0 0) 0x7f527c003520 con 0x7f52940d1910 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 msgr2=0x7f52940d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 0x7f52940d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 >> v1:172.21.15.146:6789/0 conn(0x7f52940d2480 legacy=0x7f52940b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940cf980 con 0x7f52940d1910 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f5288001670 con 0x7f52940d1910 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 >> v1:172.21.15.72:6789/0 conn(0x7f52940d1910 legacy=0x7f52940d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 0x7f527c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f527c003f60 0x7f527c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f527c004d40 0x7f527c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940cf6a0 con 0x7f527c004d40 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c00baa0 con 0x7f527c003f60 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c00bca0 con 0x7f52941614d0 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.724+0000 7f52937fe640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f527c004d40 0x7f527c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52927fc640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f527c003f60 0x7f527c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f5292ffd640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 0x7f527c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52937fe640 1 -- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 msgr2=0x7f527c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52937fe640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f52941614d0 0x7f527c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52937fe640 1 -- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f527c003f60 msgr2=0x7f527c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52937fe640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f527c003f60 0x7f527c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52937fe640 1 -- 172.21.15.72:0/3300135145 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c003520 con 0x7f527c004d40 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.725+0000 7f52937fe640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f527c004d40 0x7f527c0051c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f528c001e20 tx=0x7f527c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e1676393119286 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c001a70 con 0x7f527c004d40 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528c003b70 con 0x7f527c004d40 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f5291ffb640 1 -- 172.21.15.72:0/3300135145 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c0050d0 con 0x7f527c004d40 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52999a1640 1 -- 172.21.15.72:0/3300135145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f527c004d40 msgr2=0x7f527c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52999a1640 1 --2- 172.21.15.72:0/3300135145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f527c004d40 0x7f527c0051c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f528c001e20 tx=0x7f527c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52937fe640 1 -- 172.21.15.72:0/3300135145 reap_dead start 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52999a1640 1 -- 172.21.15.72:0/3300135145 shutdown_connections 2024-03-06T07:25:34.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52999a1640 1 -- 172.21.15.72:0/3300135145 >> 172.21.15.72:0/3300135145 conn(0x7f52940d3d40 msgr2=0x7f52940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52999a1640 1 -- 172.21.15.72:0/3300135145 shutdown_connections 2024-03-06T07:25:34.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.726+0000 7f52999a1640 1 -- 172.21.15.72:0/3300135145 wait complete. 2024-03-06T07:25:34.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.727+0000 7f52999a1640 1 Processor -- start 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 -- start start 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 0x7f529409f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f529409fee0 0x7f529409da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529409dfd0 0x7f52940cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940d4860 con 0x7f529414e210 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52940b0910 con 0x7f529409dfd0 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52999a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52940d49e0 con 0x7f529409fee0 2024-03-06T07:25:34.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f5292ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 0x7f529409f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f5292ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 0x7f529409f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47166/0 (socket says 172.21.15.72:47166) 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f5292ffd640 1 -- 172.21.15.72:0/197257264 learned_addr learned my addr 172.21.15.72:0/197257264 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52927fc640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f529409fee0 0x7f529409da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.728+0000 7f52937fe640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529409dfd0 0x7f52940cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f5292ffd640 1 -- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f529409fee0 msgr2=0x7f529409da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f5292ffd640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f529409fee0 0x7f529409da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f5292ffd640 1 -- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529409dfd0 msgr2=0x7f52940cbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f5292ffd640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529409dfd0 0x7f52940cbe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f5292ffd640 1 -- 172.21.15.72:0/197257264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52940a0480 con 0x7f529414e210 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f52937fe640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529409dfd0 0x7f52940cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f5292ffd640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 0x7f529409f9a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5284011a00 tx=0x7f5284011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400fb50 con 0x7f529414e210 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f528400fcf0 con 0x7f529414e210 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52940a0760 con 0x7f529414e210 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528400fea0 con 0x7f529414e210 2024-03-06T07:25:34.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.729+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52940a0d20 con 0x7f529414e210 2024-03-06T07:25:34.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.730+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529409e3b0 con 0x7f529414e210 2024-03-06T07:25:34.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.730+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5284012040 con 0x7f529414e210 2024-03-06T07:25:34.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.731+0000 7f527bfff640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f528006b4d0 0x7f528006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:34.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.731+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f528408a540 con 0x7f529414e210 2024-03-06T07:25:34.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.731+0000 7f52927fc640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f528006b4d0 0x7f528006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:34.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.731+0000 7f52927fc640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f528006b4d0 0x7f528006d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52940a1ac0 tx=0x7f5288000fa0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:34.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.735+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5284019030 con 0x7f529414e210 2024-03-06T07:25:34.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.905+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52940d4150 con 0x7f529414e210 2024-03-06T07:25:34.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.906+0000 7f527bfff640 1 -- 172.21.15.72:0/197257264 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5284059710 con 0x7f529414e210 2024-03-06T07:25:34.908 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:25:34.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f528006b4d0 msgr2=0x7f528006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f528006b4d0 0x7f528006d990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f52940a1ac0 tx=0x7f5288000fa0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 msgr2=0x7f529409f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:34.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 0x7f529409f9a0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5284011a00 tx=0x7f5284011ec0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 shutdown_connections 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f528006b4d0 0x7f528006d990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f529409dfd0 0x7f52940cbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f529409fee0 0x7f529409da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 --2- 172.21.15.72:0/197257264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f529414e210 0x7f529409f9a0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 >> 172.21.15.72:0/197257264 conn(0x7f52940d3d40 msgr2=0x7f52940b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 shutdown_connections 2024-03-06T07:25:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:34.908+0000 7f52999a1640 1 -- 172.21.15.72:0/197257264 wait complete. 2024-03-06T07:25:34.921 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:25:35.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.019+0000 7feaeedbc640 1 Processor -- start 2024-03-06T07:25:35.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.019+0000 7feaeedbc640 1 -- start start 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaeedbc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0320 0x7feae80b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaeedbc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feae80d0700 0x7feae80b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaeedbc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae815c270 con 0x7feae80ccae0 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaeedbc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae80b04e0 con 0x7feae80d0700 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaeedbc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80d0e90 con 0x7feae80d0320 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feae80d0700 0x7feae80b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaed359640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7feae80ccae0 0x7feae80ccf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38816/0 (socket says 172.21.15.72:38816) 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaed359640 1 -- 172.21.15.72:0/2288062109 learned_addr learned my addr 172.21.15.72:0/2288062109 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae77fe640 1 -- 172.21.15.72:0/2288062109 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576470194 0 0) 0x7feae815c270 con 0x7feae80ccae0 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae77fe640 1 -- 172.21.15.72:0/2288062109 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead0003520 con 0x7feae80ccae0 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaecb58640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0320 0x7feae80b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae7fff640 1 -- 172.21.15.72:0/2288062109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0320 msgr2=0x7feae80b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae7fff640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0320 0x7feae80b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae7fff640 1 -- 172.21.15.72:0/2288062109 >> v1:172.21.15.72:6789/0 conn(0x7feae80ccae0 legacy=0x7feae80ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feae7fff640 1 -- 172.21.15.72:0/2288062109 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80b5ae0 con 0x7feae80d0700 2024-03-06T07:25:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.020+0000 7feaecb58640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0320 0x7feae80b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.021+0000 7feae7fff640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80d0700 0x7feae80b54b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7feadc0097a0 tx=0x7feadc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfefd347d48c6b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.021+0000 7feae77fe640 1 -- 172.21.15.72:0/2288062109 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc035020 con 0x7feae80d0700 2024-03-06T07:25:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.021+0000 7feae77fe640 1 -- 172.21.15.72:0/2288062109 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc031830 con 0x7feae80d0700 2024-03-06T07:25:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.021+0000 7feae77fe640 1 -- 172.21.15.72:0/2288062109 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc031c00 con 0x7feae80d0700 2024-03-06T07:25:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 -- 172.21.15.72:0/2288062109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80d0700 msgr2=0x7feae80b54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80d0700 0x7feae80b54b0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7feadc0097a0 tx=0x7feadc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 -- 172.21.15.72:0/2288062109 shutdown_connections 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80d0700 0x7feae80b54b0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 --2- 172.21.15.72:0/2288062109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0320 0x7feae80b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 -- 172.21.15.72:0/2288062109 >> 172.21.15.72:0/2288062109 conn(0x7feae80724d0 msgr2=0x7feae80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 -- 172.21.15.72:0/2288062109 shutdown_connections 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.022+0000 7feaeedbc640 1 -- 172.21.15.72:0/2288062109 wait complete. 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.023+0000 7feaeedbc640 1 Processor -- start 2024-03-06T07:25:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.023+0000 7feaeedbc640 1 -- start start 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaeedbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 0x7feae814f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaeedbc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feae80d0320 0x7feae80c9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaeedbc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0700 0x7feae80c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaeedbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80b60a0 con 0x7feae80ccae0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaeedbc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae80b0170 con 0x7feae80d0320 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaeedbc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae80ca790 con 0x7feae80d0700 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 0x7feae814f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 0x7feae814f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47188/0 (socket says 172.21.15.72:47188) 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 -- 172.21.15.72:0/1587231808 learned_addr learned my addr 172.21.15.72:0/1587231808 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaed359640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0700 0x7feae80c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feae7fff640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feae80d0320 0x7feae80c9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 -- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0700 msgr2=0x7feae80c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0700 0x7feae80c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 -- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feae80d0320 msgr2=0x7feae80c9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feae80d0320 0x7feae80c9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaecb58640 1 -- 172.21.15.72:0/1587231808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae8150250 con 0x7feae80ccae0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.024+0000 7feaed359640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0700 0x7feae80c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.025+0000 7feaecb58640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 0x7feae814f7f0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fead800c7f0 tx=0x7fead800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.025+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8015070 con 0x7feae80ccae0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.025+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead8007a30 con 0x7feae80ccae0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.025+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae8150530 con 0x7feae80ccae0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.025+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8007e00 con 0x7feae80ccae0 2024-03-06T07:25:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.025+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae814fd30 con 0x7feae80ccae0 2024-03-06T07:25:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.026+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae8150e60 con 0x7feae80ccae0 2024-03-06T07:25:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.026+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fead8018070 con 0x7feae80ccae0 2024-03-06T07:25:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.026+0000 7feae57fa640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fead406b4d0 0x7fead406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.026+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fead8089dd0 con 0x7feae80ccae0 2024-03-06T07:25:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.026+0000 7feae7fff640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fead406b4d0 0x7fead406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.027+0000 7feae7fff640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fead406b4d0 0x7fead406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feadc036000 tx=0x7feadc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.031+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feae8150e60 con 0x7feae80ccae0 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.204+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feae804f330 con 0x7feae80ccae0 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.205+0000 7feae57fa640 1 -- 172.21.15.72:0/1587231808 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7fead8059030 con 0x7feae80ccae0 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:25:35.207 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:25:35.208 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:35.208 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:25:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fead406b4d0 msgr2=0x7fead406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fead406b4d0 0x7fead406d990 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7feadc036000 tx=0x7feadc008030 comp rx=0 tx=0).stop 2024-03-06T07:25:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 msgr2=0x7feae814f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 0x7feae814f7f0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fead800c7f0 tx=0x7fead800cbf0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 shutdown_connections 2024-03-06T07:25:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fead406b4d0 0x7fead406d990 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feae80d0700 0x7feae80c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feae80d0320 0x7feae80c9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 --2- 172.21.15.72:0/1587231808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feae80ccae0 0x7feae814f7f0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 >> 172.21.15.72:0/1587231808 conn(0x7feae80724d0 msgr2=0x7feae80728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 shutdown_connections 2024-03-06T07:25:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.207+0000 7feaeedbc640 1 -- 172.21.15.72:0/1587231808 wait complete. 2024-03-06T07:25:35.220 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:25:35.220 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:25:35.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.321+0000 7f23a184e640 1 Processor -- start 2024-03-06T07:25:35.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.321+0000 7f23a184e640 1 -- start start 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23a184e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f239c0d4630 0x7f239c0d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23a184e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c0d2820 0x7f239c0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23a184e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c15e380 con 0x7f239c15dfa0 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23a184e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239c0d2100 con 0x7f239c0d4630 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23a184e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0cf470 con 0x7f239c0d2820 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23927fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f239c0d4630 0x7f239c0d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f239affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f239c15dfa0 0x7f239c0d18f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38826/0 (socket says 172.21.15.72:38826) 2024-03-06T07:25:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f239affd640 1 -- 172.21.15.72:0/3413705659 learned_addr learned my addr 172.21.15.72:0/3413705659 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f239a7fc640 1 -- 172.21.15.72:0/3413705659 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277578846 0 0) 0x7f239c15e380 con 0x7f239c15dfa0 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f239a7fc640 1 -- 172.21.15.72:0/3413705659 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2388003520 con 0x7f239c15dfa0 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23927fc640 1 -- 172.21.15.72:0/3413705659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c0d2820 msgr2=0x7f239c0cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23927fc640 1 --2- 172.21.15.72:0/3413705659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c0d2820 0x7f239c0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23927fc640 1 -- 172.21.15.72:0/3413705659 >> v1:172.21.15.72:6789/0 conn(0x7f239c15dfa0 legacy=0x7f239c0d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.322+0000 7f23927fc640 1 -- 172.21.15.72:0/3413705659 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0cf750 con 0x7f239c0d4630 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.323+0000 7f23927fc640 1 --2- 172.21.15.72:0/3413705659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d4630 0x7f239c0d22e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f23840097a0 tx=0x7f238402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74514d793a27512e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.323+0000 7f239a7fc640 1 -- 172.21.15.72:0/3413705659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840315a0 con 0x7f239c0d4630 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.323+0000 7f239a7fc640 1 -- 172.21.15.72:0/3413705659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384037070 con 0x7f239c0d4630 2024-03-06T07:25:35.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.323+0000 7f239a7fc640 1 -- 172.21.15.72:0/3413705659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384031960 con 0x7f239c0d4630 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 -- 172.21.15.72:0/3413705659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d4630 msgr2=0x7f239c0d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 --2- 172.21.15.72:0/3413705659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d4630 0x7f239c0d22e0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f23840097a0 tx=0x7f238402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 -- 172.21.15.72:0/3413705659 shutdown_connections 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 --2- 172.21.15.72:0/3413705659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c0d2820 0x7f239c0cb190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 --2- 172.21.15.72:0/3413705659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d4630 0x7f239c0d22e0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 -- 172.21.15.72:0/3413705659 >> 172.21.15.72:0/3413705659 conn(0x7f239c09d000 msgr2=0x7f239c0d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 -- 172.21.15.72:0/3413705659 shutdown_connections 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.324+0000 7f23a184e640 1 -- 172.21.15.72:0/3413705659 wait complete. 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.325+0000 7f23a184e640 1 Processor -- start 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.325+0000 7f23a184e640 1 -- start start 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 0x7f239c0b5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0d4630 0x7f239c09dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c15dfa0 0x7f239c1f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0cb850 con 0x7f239c0d2820 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0cb6d0 con 0x7f239c0d4630 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0cb9d0 con 0x7f239c15dfa0 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 0x7f239c0b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 0x7f239c0b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47208/0 (socket says 172.21.15.72:47208) 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 -- 172.21.15.72:0/650388300 learned_addr learned my addr 172.21.15.72:0/650388300 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 -- 172.21.15.72:0/650388300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c15dfa0 msgr2=0x7f239c1f6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c15dfa0 0x7f239c1f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 -- 172.21.15.72:0/650388300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0d4630 msgr2=0x7f239c09dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0d4630 0x7f239c09dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 -- 172.21.15.72:0/650388300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0c8100 con 0x7f239c0d2820 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f239affd640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 0x7f239c0b5080 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c390 tx=0x7f238c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c017070 con 0x7f239c0d2820 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c005ce0 con 0x7f239c0d2820 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c0a0720 con 0x7f239c0d2820 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c0072f0 con 0x7f239c0d2820 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.326+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c0c8370 con 0x7f239c0d2820 2024-03-06T07:25:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.327+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c0d2f10 con 0x7f239c0d2820 2024-03-06T07:25:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.328+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f238c01a070 con 0x7f239c0d2820 2024-03-06T07:25:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.328+0000 7f2398ff9640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.328+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f238c0892b0 con 0x7f239c0d2820 2024-03-06T07:25:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.328+0000 7f23927fc640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.329+0000 7f23927fc640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2384036000 tx=0x7f2384030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.332+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f239c0d2f10 con 0x7f239c0d2820 2024-03-06T07:25:35.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.506+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f239c04f330 con 0x7f239c0d2820 2024-03-06T07:25:35.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.948+0000 7f2398ff9640 1 -- 172.21.15.72:0/650388300 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f238c058480 con 0x7f239c0d2820 2024-03-06T07:25:35.950 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:25:35.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.951+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 msgr2=0x7f237c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.951+0000 7f23a184e640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2384036000 tx=0x7f2384030000 comp rx=0 tx=0).stop 2024-03-06T07:25:35.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.951+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 msgr2=0x7f239c0b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:35.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.951+0000 7f23a184e640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 0x7f239c0b5080 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c390 tx=0x7f238c00c850 comp rx=0 tx=0).stop 2024-03-06T07:25:35.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.952+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 shutdown_connections 2024-03-06T07:25:35.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.952+0000 7f23a184e640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f237c06b4d0 0x7f237c06d990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.952+0000 7f23a184e640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f239c15dfa0 0x7f239c1f6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.952+0000 7f23a184e640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f239c0d4630 0x7f239c09dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:35.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.952+0000 7f23a184e640 1 --2- 172.21.15.72:0/650388300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f239c0d2820 0x7f239c0b5080 secure :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c390 tx=0x7f238c00c850 comp rx=0 tx=0).stop 2024-03-06T07:25:35.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.952+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 >> 172.21.15.72:0/650388300 conn(0x7f239c09d000 msgr2=0x7f239c0ccfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:35.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.953+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 shutdown_connections 2024-03-06T07:25:35.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:35.953+0000 7f23a184e640 1 -- 172.21.15.72:0/650388300 wait complete. 2024-03-06T07:25:35.968 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:25:36.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.073+0000 7fea08e4e640 1 Processor -- start 2024-03-06T07:25:36.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.073+0000 7fea08e4e640 1 -- start start 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea08e4e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea0415bbb0 0x7fea0406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea08e4e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea0406b4f0 0x7fea0415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea08e4e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea040cea90 con 0x7fea0415ec40 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea08e4e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea04161970 con 0x7fea0406b4f0 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea08e4e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea041624f0 con 0x7fea0415bbb0 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea01d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea0406b4f0 0x7fea0415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea01d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea0406b4f0 0x7fea0415e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47216/0 (socket says 172.21.15.72:47216) 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.074+0000 7fea01d9b640 1 -- 172.21.15.72:0/712848412 learned_addr learned my addr 172.21.15.72:0/712848412 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.075+0000 7fea0259c640 1 --2- 172.21.15.72:0/712848412 >> v2:172.21.15.146:3300/0 conn(0x7fea0415bbb0 0x7fea0406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.075+0000 7fea01d9b640 1 -- 172.21.15.72:0/712848412 >> v2:172.21.15.146:3300/0 conn(0x7fea0415bbb0 msgr2=0x7fea0406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.075+0000 7fea01d9b640 1 --2- 172.21.15.72:0/712848412 >> v2:172.21.15.146:3300/0 conn(0x7fea0415bbb0 0x7fea0406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.075+0000 7fea01d9b640 1 -- 172.21.15.72:0/712848412 >> v1:172.21.15.146:6789/0 conn(0x7fea0415ec40 legacy=0x7fea040cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:36.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.075+0000 7fea01d9b640 1 -- 172.21.15.72:0/712848412 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04161bc0 con 0x7fea0406b4f0 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.075+0000 7fea01d9b640 1 --2- 172.21.15.72:0/712848412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0406b4f0 0x7fea0415e700 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fea040ce420 tx=0x7fe9f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95260df71627d2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea0159a640 1 -- 172.21.15.72:0/712848412 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8035020 con 0x7fea0406b4f0 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea0159a640 1 -- 172.21.15.72:0/712848412 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f8031870 con 0x7fea0406b4f0 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea0159a640 1 -- 172.21.15.72:0/712848412 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8031c40 con 0x7fea0406b4f0 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 -- 172.21.15.72:0/712848412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0406b4f0 msgr2=0x7fea0415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 --2- 172.21.15.72:0/712848412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0406b4f0 0x7fea0415e700 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fea040ce420 tx=0x7fe9f802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 -- 172.21.15.72:0/712848412 shutdown_connections 2024-03-06T07:25:36.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 --2- 172.21.15.72:0/712848412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0406b4f0 0x7fea0415e700 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 --2- 172.21.15.72:0/712848412 >> v2:172.21.15.146:3300/0 conn(0x7fea0415bbb0 0x7fea0406afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 -- 172.21.15.72:0/712848412 >> 172.21.15.72:0/712848412 conn(0x7fea040b2660 msgr2=0x7fea040b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 -- 172.21.15.72:0/712848412 shutdown_connections 2024-03-06T07:25:36.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.076+0000 7fea08e4e640 1 -- 172.21.15.72:0/712848412 wait complete. 2024-03-06T07:25:36.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.077+0000 7fea08e4e640 1 Processor -- start 2024-03-06T07:25:36.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.077+0000 7fea08e4e640 1 -- start start 2024-03-06T07:25:36.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea08e4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea0406b4f0 0x7fea0414b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea08e4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 0x7fea0415cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea08e4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea040cc6f0 0x7fea0415d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea08e4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea04072cb0 con 0x7fea0415ec40 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea08e4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea04162180 con 0x7fea040cc6f0 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea08e4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0415f0d0 con 0x7fea0406b4f0 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea01d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 0x7fea0415cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea01d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 0x7fea0415cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47230/0 (socket says 172.21.15.72:47230) 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea01d9b640 1 -- 172.21.15.72:0/3203714586 learned_addr learned my addr 172.21.15.72:0/3203714586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea0259c640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea0406b4f0 0x7fea0414b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.078+0000 7fea02d9d640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea040cc6f0 0x7fea0415d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea01d9b640 1 -- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea0406b4f0 msgr2=0x7fea0414b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea01d9b640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea0406b4f0 0x7fea0414b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea01d9b640 1 -- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea040cc6f0 msgr2=0x7fea0415d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea01d9b640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea040cc6f0 0x7fea0415d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea01d9b640 1 -- 172.21.15.72:0/3203714586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0415fdf0 con 0x7fea0415ec40 2024-03-06T07:25:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea02d9d640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea040cc6f0 0x7fea0415d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea0259c640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea0406b4f0 0x7fea0414b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea01d9b640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 0x7fea0415cff0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fe9f802b950 tx=0x7fe9f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8040070 con 0x7fea0415ec40 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f803a420 con 0x7fea0415ec40 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0415b110 con 0x7fea0415ec40 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8035040 con 0x7fea0415ec40 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.079+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea041594d0 con 0x7fea0415ec40 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.080+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea040733b0 con 0x7fea0415ec40 2024-03-06T07:25:36.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.080+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe9f804b060 con 0x7fea0415ec40 2024-03-06T07:25:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.081+0000 7fe9f2ffd640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe9e406b4d0 0x7fe9e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.081+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe9f80abed0 con 0x7fea0415ec40 2024-03-06T07:25:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.081+0000 7fea0259c640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe9e406b4d0 0x7fe9e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.081+0000 7fea0259c640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe9e406b4d0 0x7fe9e406d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fea04157240 tx=0x7fe9f40075b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:36.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.085+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fea040cea90 con 0x7fea0415ec40 2024-03-06T07:25:36.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.258+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fea04068ff0 con 0x7fea0415ec40 2024-03-06T07:25:36.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.953+0000 7fe9f2ffd640 1 -- 172.21.15.72:0/3203714586 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe9f807afc0 con 0x7fea0415ec40 2024-03-06T07:25:36.955 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:25:36.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe9e406b4d0 msgr2=0x7fe9e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe9e406b4d0 0x7fe9e406d990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fea04157240 tx=0x7fe9f40075b0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 msgr2=0x7fea0415cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 0x7fea0415cff0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fe9f802b950 tx=0x7fe9f80046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 shutdown_connections 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe9e406b4d0 0x7fe9e406d990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea040cc6f0 0x7fea0415d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea0415ec40 0x7fea0415cff0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 --2- 172.21.15.72:0/3203714586 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea0406b4f0 0x7fea0414b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:36.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.957+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 >> 172.21.15.72:0/3203714586 conn(0x7fea040b2660 msgr2=0x7fea04071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:36.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.958+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 shutdown_connections 2024-03-06T07:25:36.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:36.959+0000 7fea08e4e640 1 -- 172.21.15.72:0/3203714586 wait complete. 2024-03-06T07:25:36.986 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:25:37.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.089+0000 7f6f94839640 1 Processor -- start 2024-03-06T07:25:37.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.089+0000 7f6f94839640 1 -- start start 2024-03-06T07:25:37.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f94839640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 0x7f6f90071740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f94839640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f90072c40 con 0x7f6f9006d260 2024-03-06T07:25:37.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f94839640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9006caa0 con 0x7f6f9015a490 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f94839640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90071c80 con 0x7f6f900712d0 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8ed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6f9006d260 0x7f6f9006d690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38832/0 (socket says 172.21.15.72:38832) 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8ed9d640 1 -- 172.21.15.72:0/1067891725 learned_addr learned my addr 172.21.15.72:0/1067891725 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f87fff640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 0x7f6f90071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577220909 0 0) 0x7f6f90072c40 con 0x7f6f9006d260 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c003520 con 0x7f6f9006d260 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099917065 0 0) 0x7f6f7c003520 con 0x7f6f9006d260 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 msgr2=0x7f6f90071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 0x7f6f90071740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 >> v1:172.21.15.146:6789/0 conn(0x7f6f9015a490 legacy=0x7f6f90072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900b3340 con 0x7f6f9006d260 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6f880019f0 con 0x7f6f9006d260 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 >> v1:172.21.15.72:6789/0 conn(0x7f6f9006d260 legacy=0x7f6f9006d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0039e0 0x7f6f7c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 0x7f6f7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f7c004d40 0x7f6f7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90071c80 con 0x7f6f7c0039e0 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c00b9f0 con 0x7f6f7c004d40 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c00bbf0 con 0x7f6f900712d0 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.090+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0039e0 0x7f6f7c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f87fff640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 0x7f6f7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f8ed9d640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f7c004d40 0x7f6f7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f8e59c640 1 -- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 msgr2=0x7f6f7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f900712d0 0x7f6f7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f8e59c640 1 -- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f7c004d40 msgr2=0x7f6f7c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f7c004d40 0x7f6f7c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.091+0000 7f6f8e59c640 1 -- 172.21.15.72:0/1067891725 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c003520 con 0x7f6f7c0039e0 2024-03-06T07:25:37.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0039e0 0x7f6f7c003dc0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f6f80001860 tx=0x7f6f7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab4de8446267cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80038070 con 0x7f6f7c0039e0 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f80033660 con 0x7f6f7c0039e0 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f8dd9b640 1 -- 172.21.15.72:0/1067891725 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80001430 con 0x7f6f7c0039e0 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f94839640 1 -- 172.21.15.72:0/1067891725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0039e0 msgr2=0x7f6f7c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f94839640 1 --2- 172.21.15.72:0/1067891725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f7c0039e0 0x7f6f7c003dc0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f6f80001860 tx=0x7f6f7c005830 comp rx=0 tx=0).stop 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f8ed9d640 1 -- 172.21.15.72:0/1067891725 reap_dead start 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f94839640 1 -- 172.21.15.72:0/1067891725 shutdown_connections 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f94839640 1 -- 172.21.15.72:0/1067891725 >> 172.21.15.72:0/1067891725 conn(0x7f6f9006afb0 msgr2=0x7f6f9006b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f94839640 1 -- 172.21.15.72:0/1067891725 shutdown_connections 2024-03-06T07:25:37.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.092+0000 7f6f94839640 1 -- 172.21.15.72:0/1067891725 wait complete. 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.093+0000 7f6f94839640 1 Processor -- start 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 -- start start 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f7c0039e0 0x7f6f901426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 0x7f6f901f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f900ceb70 0x7f6f901f5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900b3900 con 0x7f6f900ce6f0 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9006cd40 con 0x7f6f900ceb70 2024-03-06T07:25:37.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f94839640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f900b3a80 con 0x7f6f7c0039e0 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f87fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 0x7f6f901f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f87fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 0x7f6f901f4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47242/0 (socket says 172.21.15.72:47242) 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f87fff640 1 -- 172.21.15.72:0/820299248 learned_addr learned my addr 172.21.15.72:0/820299248 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.094+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f7c0039e0 0x7f6f901426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f8ed9d640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f900ceb70 0x7f6f901f5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f87fff640 1 -- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f7c0039e0 msgr2=0x7f6f901426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f87fff640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f7c0039e0 0x7f6f901426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f87fff640 1 -- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f900ceb70 msgr2=0x7f6f901f5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f87fff640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f900ceb70 0x7f6f901f5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f87fff640 1 -- 172.21.15.72:0/820299248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90142d30 con 0x7f6f900ce6f0 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f8ed9d640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f900ceb70 0x7f6f901f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:37.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f87fff640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 0x7f6f901f4d40 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6f780119a0 tx=0x7f6f78011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7800faf0 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7800fc90 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9015d100 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78014440 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.095+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9015d6c0 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.096+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9004f330 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.096+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6f78012040 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.097+0000 7f6f86ffd640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f7006b4d0 0x7f6f7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.097+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f7006b4d0 0x7f6f7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.097+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f6f7808a7a0 con 0x7f6f900ce6f0 2024-03-06T07:25:37.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.097+0000 7f6f8e59c640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f7006b4d0 0x7f6f7006d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6f80035000 tx=0x7f6f80005040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:37.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.101+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6f78019030 con 0x7f6f900ce6f0 2024-03-06T07:25:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.308+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6f90072b50 con 0x7f6f900ce6f0 2024-03-06T07:25:37.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.955+0000 7f6f86ffd640 1 -- 172.21.15.72:0/820299248 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6f78059680 con 0x7f6f900ce6f0 2024-03-06T07:25:37.957 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 12 and data pool 13 2024-03-06T07:25:37.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f7006b4d0 msgr2=0x7f6f7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f7006b4d0 0x7f6f7006d990 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6f80035000 tx=0x7f6f80005040 comp rx=0 tx=0).stop 2024-03-06T07:25:37.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 msgr2=0x7f6f901f4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:37.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 0x7f6f901f4d40 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6f780119a0 tx=0x7f6f78011e60 comp rx=0 tx=0).stop 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 shutdown_connections 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f7006b4d0 0x7f6f7006d990 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f900ceb70 0x7f6f901f5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900ce6f0 0x7f6f901f4d40 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 --2- 172.21.15.72:0/820299248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f7c0039e0 0x7f6f901426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.959+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 >> 172.21.15.72:0/820299248 conn(0x7f6f9006afb0 msgr2=0x7f6f900cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.960+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 shutdown_connections 2024-03-06T07:25:37.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:37.960+0000 7f6f94839640 1 -- 172.21.15.72:0/820299248 wait complete. 2024-03-06T07:25:37.993 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.092+0000 7f73063d7640 1 Processor -- start 2024-03-06T07:25:38.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.092+0000 7f73063d7640 1 -- start start 2024-03-06T07:25:38.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f73063d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73001515e0 0x7f73000cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f73063d7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7300071f40 con 0x7f730015b430 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f73063d7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730015b8e0 con 0x7f73000cd7a0 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f73063d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7300070a70 con 0x7f73001515e0 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f72fffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f73001515e0 0x7f73000cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f72ff7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f730015b430 0x7f7300071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38842/0 (socket says 172.21.15.72:38842) 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.093+0000 7f72ff7fe640 1 -- 172.21.15.72:0/883304969 learned_addr learned my addr 172.21.15.72:0/883304969 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72fffff640 1 -- 172.21.15.72:0/883304969 >> v1:172.21.15.146:6789/0 conn(0x7f73000cd7a0 legacy=0x7f73000cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72fffff640 1 -- 172.21.15.72:0/883304969 >> v1:172.21.15.72:6789/0 conn(0x7f730015b430 legacy=0x7f7300071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72feffd640 1 -- 172.21.15.72:0/883304969 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757441420 0 0) 0x7f7300071f40 con 0x7f730015b430 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72fffff640 1 -- 172.21.15.72:0/883304969 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7300070d50 con 0x7f73001515e0 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72fffff640 1 --2- 172.21.15.72:0/883304969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f73000cce40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f72f00097a0 tx=0x7f72f002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7884801aa415490 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72feffd640 1 -- 172.21.15.72:0/883304969 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f002f8a0 con 0x7f73001515e0 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.094+0000 7f72feffd640 1 -- 172.21.15.72:0/883304969 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f002fa40 con 0x7f73001515e0 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f72feffd640 1 -- 172.21.15.72:0/883304969 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f002fe10 con 0x7f73001515e0 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 -- 172.21.15.72:0/883304969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 msgr2=0x7f73000cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 --2- 172.21.15.72:0/883304969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f73000cce40 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f72f00097a0 tx=0x7f72f002dae0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 -- 172.21.15.72:0/883304969 shutdown_connections 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 --2- 172.21.15.72:0/883304969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f73000cce40 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 -- 172.21.15.72:0/883304969 >> 172.21.15.72:0/883304969 conn(0x7f73000cf8a0 msgr2=0x7f73000cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 -- 172.21.15.72:0/883304969 shutdown_connections 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.095+0000 7f73063d7640 1 -- 172.21.15.72:0/883304969 wait complete. 2024-03-06T07:25:38.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.096+0000 7f73063d7640 1 Processor -- start 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.096+0000 7f73063d7640 1 -- start start 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f73063d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73000cd7a0 0x7f73001ef680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f73063d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f730015ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f73063d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f730015b430 0x7f730015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f73063d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73000cc3a0 con 0x7f73001515e0 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f73063d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7300071310 con 0x7f73000cd7a0 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f73063d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73000cc520 con 0x7f730015b430 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f730015ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f730015ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47250/0 (socket says 172.21.15.72:47250) 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 -- 172.21.15.72:0/2548006954 learned_addr learned my addr 172.21.15.72:0/2548006954 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 -- 172.21.15.72:0/2548006954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f730015b430 msgr2=0x7f730015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f730015b430 0x7f730015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 -- 172.21.15.72:0/2548006954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73000cd7a0 msgr2=0x7f73001ef680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73000cd7a0 0x7f73001ef680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.097+0000 7f72ff7fe640 1 -- 172.21.15.72:0/2548006954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730015e240 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.098+0000 7f72ff7fe640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f730015ce00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f72f400c430 tx=0x7f72f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.098+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4017070 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.098+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f4005ce0 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.098+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730015d9a0 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.098+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f4007310 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.098+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730015df60 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.099+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7300068840 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.099+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f72f401a070 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.100+0000 7f72fcff9640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72ec06b4d0 0x7f72ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.100+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72f4089590 con 0x7f73001515e0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.100+0000 7f72fffff640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72ec06b4d0 0x7f72ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.100+0000 7f72fffff640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72ec06b4d0 0x7f72ec06d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f72f0038000 tx=0x7f72f003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.104+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f72f40584a0 con 0x7f73001515e0 2024-03-06T07:25:38.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.280+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f730004f330 con 0x7f73001515e0 2024-03-06T07:25:38.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.281+0000 7f72fcff9640 1 -- 172.21.15.72:0/2548006954 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16685 (secure 0 0 0) 0x7f72f4057da0 con 0x7f73001515e0 2024-03-06T07:25:38.284 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:38.284 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":82,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:37.309874+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72ec06b4d0 msgr2=0x7f72ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72ec06b4d0 0x7f72ec06d990 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f72f0038000 tx=0x7f72f003b040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 msgr2=0x7f730015ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f730015ce00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f72f400c430 tx=0x7f72f400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 shutdown_connections 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f72ec06b4d0 0x7f72ec06d990 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f730015b430 0x7f730015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73001515e0 0x7f730015ce00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 --2- 172.21.15.72:0/2548006954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73000cd7a0 0x7f73001ef680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.283+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 >> 172.21.15.72:0/2548006954 conn(0x7f73000cf8a0 msgr2=0x7f73000cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.284+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 shutdown_connections 2024-03-06T07:25:38.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.284+0000 7f73063d7640 1 -- 172.21.15.72:0/2548006954 wait complete. 2024-03-06T07:25:38.297 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:38.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.394+0000 7f5002681640 1 Processor -- start 2024-03-06T07:25:38.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.394+0000 7f5002681640 1 -- start start 2024-03-06T07:25:38.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5002681640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ffc06da40 0x7f4ffc0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5002681640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ffc0cd7a0 0x7f4ffc15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5002681640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc0cc980 con 0x7f4ffc1424f0 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5002681640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0cd220 con 0x7f4ffc0cd7a0 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5002681640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc0ccc60 con 0x7f4ffc06da40 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5000c1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ffc0cd7a0 0x7f4ffc15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f4ffbfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4ffc1424f0 0x7f4ffc06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38850/0 (socket says 172.21.15.72:38850) 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f4ffbfff640 1 -- 172.21.15.72:0/3872995954 learned_addr learned my addr 172.21.15.72:0/3872995954 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.395+0000 7f5000c1e640 1 -- 172.21.15.72:0/3872995954 >> v2:172.21.15.146:3300/0 conn(0x7f4ffc06da40 msgr2=0x7f4ffc0cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f5000c1e640 1 --2- 172.21.15.72:0/3872995954 >> v2:172.21.15.146:3300/0 conn(0x7f4ffc06da40 0x7f4ffc0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f5000c1e640 1 -- 172.21.15.72:0/3872995954 >> v1:172.21.15.72:6789/0 conn(0x7f4ffc1424f0 legacy=0x7f4ffc06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f5000c1e640 1 -- 172.21.15.72:0/3872995954 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc0ccf40 con 0x7f4ffc0cd7a0 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f4ffaffd640 1 -- 172.21.15.72:0/3872995954 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111170198 0 0) 0x7f4ffc0cc980 con 0x7f4ffc1424f0 2024-03-06T07:25:38.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f5000c1e640 1 --2- 172.21.15.72:0/3872995954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc0cd7a0 0x7f4ffc15cef0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0b2a00 tx=0x7f4ff402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3167f5467ac22e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f4ffaffd640 1 -- 172.21.15.72:0/3872995954 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4039070 con 0x7f4ffc0cd7a0 2024-03-06T07:25:38.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f4ffaffd640 1 -- 172.21.15.72:0/3872995954 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff402cc80 con 0x7f4ffc0cd7a0 2024-03-06T07:25:38.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.396+0000 7f4ffaffd640 1 -- 172.21.15.72:0/3872995954 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4034660 con 0x7f4ffc0cd7a0 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 -- 172.21.15.72:0/3872995954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc0cd7a0 msgr2=0x7f4ffc15cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 --2- 172.21.15.72:0/3872995954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc0cd7a0 0x7f4ffc15cef0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc0b2a00 tx=0x7f4ff402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 -- 172.21.15.72:0/3872995954 shutdown_connections 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 --2- 172.21.15.72:0/3872995954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc0cd7a0 0x7f4ffc15cef0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 --2- 172.21.15.72:0/3872995954 >> v2:172.21.15.146:3300/0 conn(0x7f4ffc06da40 0x7f4ffc0cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 -- 172.21.15.72:0/3872995954 >> 172.21.15.72:0/3872995954 conn(0x7f4ffc0cf8a0 msgr2=0x7f4ffc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 -- 172.21.15.72:0/3872995954 shutdown_connections 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.397+0000 7f5002681640 1 -- 172.21.15.72:0/3872995954 wait complete. 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.398+0000 7f5002681640 1 Processor -- start 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.398+0000 7f5002681640 1 -- start start 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5002681640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 0x7f4ffc0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5002681640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ffc0cd7a0 0x7f4ffc0ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5002681640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc1424f0 0x7f4ffc0aba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5002681640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc071ce0 con 0x7f4ffc06da40 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5002681640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc071b60 con 0x7f4ffc1424f0 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5002681640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc071e60 con 0x7f4ffc0cd7a0 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 0x7f4ffc0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 0x7f4ffc0b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47274/0 (socket says 172.21.15.72:47274) 2024-03-06T07:25:38.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 -- 172.21.15.72:0/747646984 learned_addr learned my addr 172.21.15.72:0/747646984 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5000c1e640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc1424f0 0x7f4ffc0aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffb7fe640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ffc0cd7a0 0x7f4ffc0ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 -- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ffc0cd7a0 msgr2=0x7f4ffc0ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ffc0cd7a0 0x7f4ffc0ab4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 -- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc1424f0 msgr2=0x7f4ffc0aba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc1424f0 0x7f4ffc0aba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f4ffbfff640 1 -- 172.21.15.72:0/747646984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc13fe30 con 0x7f4ffc06da40 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.399+0000 7f5000c1e640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc1424f0 0x7f4ffc0aba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.400+0000 7f4ffbfff640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 0x7f4ffc0b0880 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c430 tx=0x7f4fec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.400+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0074e0 con 0x7f4ffc06da40 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.400+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec0047e0 con 0x7f4ffc06da40 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.400+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc140110 con 0x7f4ffc06da40 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.400+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec004bd0 con 0x7f4ffc06da40 2024-03-06T07:25:38.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.400+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc1406d0 con 0x7f4ffc06da40 2024-03-06T07:25:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.401+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc04f330 con 0x7f4ffc06da40 2024-03-06T07:25:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.401+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4fec011070 con 0x7f4ffc06da40 2024-03-06T07:25:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.402+0000 7f4ff8ff9640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.402+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fec089e30 con 0x7f4ffc06da40 2024-03-06T07:25:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.402+0000 7f4ffb7fe640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.402+0000 7f4ffb7fe640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4fe806b4d0 0x7f4fe806d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc141bf0 tx=0x7f4ff0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.406+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4fec058d40 con 0x7f4ffc06da40 2024-03-06T07:25:38.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.577+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ffc06e6f0 con 0x7f4ffc06da40 2024-03-06T07:25:38.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.578+0000 7f4ff8ff9640 1 -- 172.21.15.72:0/747646984 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16685 (secure 0 0 0) 0x7f4fec058640 con 0x7f4ffc06da40 2024-03-06T07:25:38.581 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:38.581 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":82,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:37.309874+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4fe806b4d0 msgr2=0x7f4fe806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4fe806b4d0 0x7f4fe806d990 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc141bf0 tx=0x7f4ff0008040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 msgr2=0x7f4ffc0b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 0x7f4ffc0b0880 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c430 tx=0x7f4fec00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 shutdown_connections 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4fe806b4d0 0x7f4fe806d990 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ffc1424f0 0x7f4ffc0aba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ffc0cd7a0 0x7f4ffc0ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 --2- 172.21.15.72:0/747646984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ffc06da40 0x7f4ffc0b0880 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.580+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 >> 172.21.15.72:0/747646984 conn(0x7f4ffc0cf8a0 msgr2=0x7f4ffc15ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.581+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 shutdown_connections 2024-03-06T07:25:38.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.581+0000 7f5002681640 1 -- 172.21.15.72:0/747646984 wait complete. 2024-03-06T07:25:38.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:25:38.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 Processor -- start 2024-03-06T07:25:38.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 -- start start 2024-03-06T07:25:38.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00d4860 0x7f9ed00d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9ed00b0e30 0x7f9ed00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed01618b0 con 0x7f9ed01614d0 2024-03-06T07:25:38.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed00d4cc0 con 0x7f9ed00b0e30 2024-03-06T07:25:38.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.695+0000 7f9ed53e4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00cf400 con 0x7f9ed00d4860 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ecf7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9ed00b0e30 0x7f9ed00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ece7fc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00d4860 0x7f9ed00d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9eceffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9ed01614d0 0x7f9ed00d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:46626/0 (socket says 172.21.15.72:46626) 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9eceffd640 1 -- 172.21.15.72:0/2727198260 learned_addr learned my addr 172.21.15.72:0/2727198260 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00d4860 msgr2=0x7f9ed00d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ecf7fe640 1 --2- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00d4860 0x7f9ed00d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/2727198260 >> v1:172.21.15.146:6789/0 conn(0x7f9ed01614d0 legacy=0x7f9ed00d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/2727198260 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed00cf6a0 con 0x7f9ed00b0e30 2024-03-06T07:25:38.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.696+0000 7f9ece7fc640 1 --2- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00d4860 0x7f9ed00d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.697+0000 7f9ecf7fe640 1 --2- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00b0e30 0x7f9ed00b12c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009d90 tx=0x7f9ec802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76523d16490d84c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.697+0000 7f9ecdffb640 1 -- 172.21.15.72:0/2727198260 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8039070 con 0x7f9ed00b0e30 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.697+0000 7f9ecdffb640 1 -- 172.21.15.72:0/2727198260 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec802cc50 con 0x7f9ed00b0e30 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.697+0000 7f9ecdffb640 1 -- 172.21.15.72:0/2727198260 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8034440 con 0x7f9ed00b0e30 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 -- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00b0e30 msgr2=0x7f9ed00b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00b0e30 0x7f9ed00b12c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009d90 tx=0x7f9ec802bea0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 -- 172.21.15.72:0/2727198260 shutdown_connections 2024-03-06T07:25:38.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00b0e30 0x7f9ed00b12c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/2727198260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00d4860 0x7f9ed00d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 -- 172.21.15.72:0/2727198260 >> 172.21.15.72:0/2727198260 conn(0x7f9ed00d3d40 msgr2=0x7f9ed00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 -- 172.21.15.72:0/2727198260 shutdown_connections 2024-03-06T07:25:38.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.698+0000 7f9ed53e4640 1 -- 172.21.15.72:0/2727198260 wait complete. 2024-03-06T07:25:38.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.699+0000 7f9ed53e4640 1 Processor -- start 2024-03-06T07:25:38.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.699+0000 7f9ed53e4640 1 -- start start 2024-03-06T07:25:38.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ed53e4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00b0e30 0x7f9ed015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ed53e4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00d4860 0x7f9ed00b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ed53e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 0x7f9ed00b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ed53e4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00b0910 con 0x7f9ed01614d0 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ed53e4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00cfc60 con 0x7f9ed00d4860 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ed53e4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed00d2560 con 0x7f9ed00b0e30 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 0x7f9ed00b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 0x7f9ed00b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47286/0 (socket says 172.21.15.72:47286) 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/3013782986 learned_addr learned my addr 172.21.15.72:0/3013782986 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9eceffd640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00b0e30 0x7f9ed015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ece7fc640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00d4860 0x7f9ed00b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00b0e30 msgr2=0x7f9ed015b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00b0e30 0x7f9ed015b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00d4860 msgr2=0x7f9ed00b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00d4860 0x7f9ed00b4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 -- 172.21.15.72:0/3013782986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed01f7b20 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ece7fc640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00d4860 0x7f9ed00b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9eceffd640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00b0e30 0x7f9ed015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.700+0000 7f9ecf7fe640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 0x7f9ed00b4f60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009d90 tx=0x7f9ec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.701+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8039050 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.701+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec8045070 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.701+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed01f88a0 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.701+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8009450 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.701+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed01f7dc0 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.702+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed0160b50 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.702+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9ec803e610 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.702+0000 7f9eb77fe640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ebc06b4d0 0x7f9ebc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.702+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ec80acce0 con 0x7f9ed01614d0 2024-03-06T07:25:38.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.702+0000 7f9eceffd640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ebc06b4d0 0x7f9ebc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:38.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.703+0000 7f9eceffd640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ebc06b4d0 0x7f9ebc06d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0000f60 tx=0x7f9ec0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:38.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.707+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ec807bae0 con 0x7f9ed01614d0 2024-03-06T07:25:38.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.914+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9ed0068ff0 con 0x7f9ed01614d0 2024-03-06T07:25:38.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.966+0000 7f9eb77fe640 1 -- 172.21.15.72:0/3013782986 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v38) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9ec807b3e0 con 0x7f9ed01614d0 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ebc06b4d0 msgr2=0x7f9ebc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ebc06b4d0 0x7f9ebc06d990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0000f60 tx=0x7f9ec0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 msgr2=0x7f9ed00b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 0x7f9ed00b4f60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009d90 tx=0x7f9ec8004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 shutdown_connections 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ebc06b4d0 0x7f9ebc06d990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ed01614d0 0x7f9ed00b4f60 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ed00d4860 0x7f9ed00b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 --2- 172.21.15.72:0/3013782986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ed00b0e30 0x7f9ed015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:38.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 >> 172.21.15.72:0/3013782986 conn(0x7f9ed00d3d40 msgr2=0x7f9ed00d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:38.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 shutdown_connections 2024-03-06T07:25:38.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:38.969+0000 7f9ed53e4640 1 -- 172.21.15.72:0/3013782986 wait complete. 2024-03-06T07:25:38.983 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:25:38.983 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:25:39.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.124+0000 7f155e415640 1 Processor -- start 2024-03-06T07:25:39.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.124+0000 7f155e415640 1 -- start start 2024-03-06T07:25:39.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f155e415640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15581614d0 0x7f15580d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f155e415640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15581618b0 con 0x7f15580d24b0 2024-03-06T07:25:39.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f155e415640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15580cf400 con 0x7f15580d1910 2024-03-06T07:25:39.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f155e415640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15580cf6a0 con 0x7f15581614d0 2024-03-06T07:25:39.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f155c9b2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f15580d24b0 0x7f15580b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38852/0 (socket says 172.21.15.72:38852) 2024-03-06T07:25:39.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f155c9b2640 1 -- 172.21.15.72:0/3273251084 learned_addr learned my addr 172.21.15.72:0/3273251084 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f1557fff640 1 --2- 172.21.15.72:0/3273251084 >> v2:172.21.15.146:3300/0 conn(0x7f15581614d0 0x7f15580d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275482836 0 0) 0x7f15581618b0 con 0x7f15580d24b0 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1540003520 con 0x7f15580d24b0 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354464298 0 0) 0x7f15580cf400 con 0x7f15580d1910 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.125+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15581618b0 con 0x7f15580d1910 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2461029499 0 0) 0x7f1540003520 con 0x7f15580d24b0 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 >> v2:172.21.15.146:3300/0 conn(0x7f15581614d0 msgr2=0x7f15580d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 --2- 172.21.15.72:0/3273251084 >> v2:172.21.15.146:3300/0 conn(0x7f15581614d0 0x7f15580d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 >> v1:172.21.15.146:6789/0 conn(0x7f15580d1910 legacy=0x7f15580d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15580cf980 con 0x7f15580d24b0 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f15500019f0 con 0x7f15580d24b0 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 >> v1:172.21.15.72:6789/0 conn(0x7f15580d24b0 legacy=0x7f15580b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1540003b40 0x7f1540003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1540004800 0x7f1540004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15400051c0 0x7f154000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580cf6a0 con 0x7f15400051c0 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154000c040 con 0x7f1540003b40 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154000c240 con 0x7f1540004800 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f155c9b2640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15400051c0 0x7f154000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f1557fff640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1540003b40 0x7f1540003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f15577fe640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1540004800 0x7f1540004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f155c9b2640 1 -- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1540004800 msgr2=0x7f1540004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f155c9b2640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1540004800 0x7f1540004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f155c9b2640 1 -- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1540003b40 msgr2=0x7f1540003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f155c9b2640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1540003b40 0x7f1540003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f155c9b2640 1 -- 172.21.15.72:0/3273251084 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1540003520 con 0x7f15400051c0 2024-03-06T07:25:39.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.126+0000 7f15577fe640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1540004800 0x7f1540004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155c9b2640 1 -- 172.21.15.72:0/3273251084 reap_dead start 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155c9b2640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15400051c0 0x7f154000b9c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f155000a820 tx=0x7f1540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f062ad65fb5f00b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000a450 con 0x7f15400051c0 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1550007310 con 0x7f15400051c0 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f1556ffd640 1 -- 172.21.15.72:0/3273251084 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1550007700 con 0x7f15400051c0 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155e415640 1 -- 172.21.15.72:0/3273251084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15400051c0 msgr2=0x7f154000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155e415640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15400051c0 0x7f154000b9c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f155000a820 tx=0x7f1540005830 comp rx=0 tx=0).stop 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155e415640 1 -- 172.21.15.72:0/3273251084 shutdown_connections 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155e415640 1 --2- 172.21.15.72:0/3273251084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15400051c0 0x7f154000b9c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.127+0000 7f155e415640 1 -- 172.21.15.72:0/3273251084 >> 172.21.15.72:0/3273251084 conn(0x7f15580d3d40 msgr2=0x7f15580d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.128+0000 7f155e415640 1 -- 172.21.15.72:0/3273251084 shutdown_connections 2024-03-06T07:25:39.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.128+0000 7f155e415640 1 -- 172.21.15.72:0/3273251084 wait complete. 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.129+0000 7f155e415640 1 Processor -- start 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.129+0000 7f155e415640 1 -- start start 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15400051c0 0x7f15580b9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 0x7f15580c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15580c90b0 0x7f155815f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580d4860 con 0x7f1558158080 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15580b0910 con 0x7f15580c90b0 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15580d49e0 con 0x7f15400051c0 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 0x7f15580c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 0x7f15580c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47308/0 (socket says 172.21.15.72:47308) 2024-03-06T07:25:39.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 -- 172.21.15.72:0/673308670 learned_addr learned my addr 172.21.15.72:0/673308670 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 -- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15400051c0 msgr2=0x7f15580b9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15400051c0 0x7f15580b9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 -- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15580c90b0 msgr2=0x7f155815f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155c9b2640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15580c90b0 0x7f155815f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15580c90b0 0x7f155815f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 -- 172.21.15.72:0/673308670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15581584e0 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f15577fe640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 0x7f15580c7870 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f154800e4e0 tx=0x7f154800e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15480052e0 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1548018070 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548003810 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.130+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15580c8010 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.131+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15580c85d0 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.132+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15580b2700 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.132+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f15480039b0 con 0x7f1558158080 2024-03-06T07:25:39.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.132+0000 7f1554ff9640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f154406b4d0 0x7f154406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:39.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.132+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1548089ed0 con 0x7f1558158080 2024-03-06T07:25:39.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.132+0000 7f1557fff640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f154406b4d0 0x7f154406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:39.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.133+0000 7f1557fff640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f154406b4d0 0x7f154406d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f154c009a40 tx=0x7f154c009320 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:39.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.141+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f15580b2700 con 0x7f1558158080 2024-03-06T07:25:39.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.351+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f15580b9c00 con 0x7f1558158080 2024-03-06T07:25:39.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.975+0000 7f1554ff9640 1 -- 172.21.15.72:0/673308670 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v39) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1548058de0 con 0x7f1558158080 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f154406b4d0 msgr2=0x7f154406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f154406b4d0 0x7f154406d990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f154c009a40 tx=0x7f154c009320 comp rx=0 tx=0).stop 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 msgr2=0x7f15580c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 0x7f15580c7870 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f154800e4e0 tx=0x7f154800e9a0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 shutdown_connections 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f154406b4d0 0x7f154406d990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15580c90b0 0x7f155815f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1558158080 0x7f15580c7870 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 --2- 172.21.15.72:0/673308670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15400051c0 0x7f15580b9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:39.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 >> 172.21.15.72:0/673308670 conn(0x7f15580d3d40 msgr2=0x7f15580b0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:39.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 shutdown_connections 2024-03-06T07:25:39.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:39.978+0000 7f155e415640 1 -- 172.21.15.72:0/673308670 wait complete. 2024-03-06T07:25:39.991 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:25:39.991 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:40.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.090+0000 7f800d7e9640 1 Processor -- start 2024-03-06T07:25:40.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.090+0000 7f800d7e9640 1 -- start start 2024-03-06T07:25:40.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f800d7e9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f80080d4760 0x7f80080b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f800d7e9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080d25d0 con 0x7f80080d0a80 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f800d7e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80080d2870 con 0x7f80080b4170 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f800d7e9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80080d4bf0 con 0x7f80080d4760 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f80077fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f80080d0a80 0x7f80080cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38860/0 (socket says 172.21.15.72:38860) 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f80077fe640 1 -- 172.21.15.72:0/1472143616 learned_addr learned my addr 172.21.15.72:0/1472143616 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.091+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843445253 0 0) 0x7f80080d25d0 con 0x7f80080d0a80 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 --2- 172.21.15.72:0/1472143616 >> v2:172.21.15.146:3300/0 conn(0x7f80080d4760 0x7f80080b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0003520 con 0x7f80080d0a80 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388450682 0 0) 0x7f7ff0003520 con 0x7f80080d0a80 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 >> v2:172.21.15.146:3300/0 conn(0x7f80080d4760 msgr2=0x7f80080b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 --2- 172.21.15.72:0/1472143616 >> v2:172.21.15.146:3300/0 conn(0x7f80080d4760 0x7f80080b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 >> v1:172.21.15.146:6789/0 conn(0x7f80080b4170 legacy=0x7f80080d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080d2b50 con 0x7f80080d0a80 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f8000001a10 con 0x7f80080d0a80 2024-03-06T07:25:40.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 >> v1:172.21.15.72:6789/0 conn(0x7f80080d0a80 legacy=0x7f80080cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff00039b0 0x7f7ff0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f7ff0004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ff0005080 0x7f7ff000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080d4bf0 con 0x7f7ff00043b0 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff000bfe0 con 0x7f7ff00039b0 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff000c1e0 con 0x7f7ff0005080 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f7ff0004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f8006ffd640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff00039b0 0x7f7ff0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80077fe640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ff0005080 0x7f7ff000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 -- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ff0005080 msgr2=0x7f7ff000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ff0005080 0x7f7ff000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 -- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff00039b0 msgr2=0x7f7ff0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ff00039b0 0x7f7ff0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80067fc640 1 -- 172.21.15.72:0/1472143616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0003520 con 0x7f7ff00043b0 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.092+0000 7f80077fe640 1 -- 172.21.15.72:0/1472143616 reap_dead start 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.093+0000 7f80067fc640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f7ff0004880 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009a90 tx=0x7f7ff0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d3acf2b8b31ad7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.093+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8039070 con 0x7f7ff00043b0 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.093+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff802c6b0 con 0x7f7ff00043b0 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.093+0000 7f8005ffb640 1 -- 172.21.15.72:0/1472143616 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8009690 con 0x7f7ff00043b0 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 -- 172.21.15.72:0/1472143616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 msgr2=0x7f7ff0004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f7ff0004880 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8009a90 tx=0x7f7ff0005830 comp rx=0 tx=0).stop 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 -- 172.21.15.72:0/1472143616 shutdown_connections 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 --2- 172.21.15.72:0/1472143616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f7ff0004880 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 -- 172.21.15.72:0/1472143616 >> 172.21.15.72:0/1472143616 conn(0x7f80080d3c40 msgr2=0x7f80080d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 -- 172.21.15.72:0/1472143616 shutdown_connections 2024-03-06T07:25:40.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.094+0000 7f800d7e9640 1 -- 172.21.15.72:0/1472143616 wait complete. 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.095+0000 7f800d7e9640 1 Processor -- start 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.095+0000 7f800d7e9640 1 -- start start 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f800d7e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f80081f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f800d7e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80080b6860 0x7f80080b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f800d7e9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80080cd1b0 0x7f80080cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f800d7e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080b4dd0 con 0x7f7ff00043b0 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f800d7e9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80080ce9e0 con 0x7f80080cd1b0 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f800d7e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80080b4f50 con 0x7f80080b6860 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f80081f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f80081f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47338/0 (socket says 172.21.15.72:47338) 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 -- 172.21.15.72:0/3523828727 learned_addr learned my addr 172.21.15.72:0/3523828727 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:40.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 -- 172.21.15.72:0/3523828727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80080b6860 msgr2=0x7f80080b6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80080b6860 0x7f80080b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 -- 172.21.15.72:0/3523828727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80080cd1b0 msgr2=0x7f80080cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80080cd1b0 0x7f80080cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 -- 172.21.15.72:0/3523828727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80080b93c0 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.096+0000 7f8006ffd640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f80081f6e20 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00f7f0 tx=0x7f7ffc00fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.097+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00fea0 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.097+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc018070 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.097+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00d6b0 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.097+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80080b96a0 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.097+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80080c83f0 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.098+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800804f330 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.098+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7ffc00d850 con 0x7f7ff00043b0 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.098+0000 7f7fef7fe640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.099+0000 7f80067fc640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.099+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ffc08a0d0 con 0x7f7ff00043b0 2024-03-06T07:25:40.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.099+0000 7f80067fc640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8036000 tx=0x7f7ff802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.103+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7ffc058f50 con 0x7f7ff00043b0 2024-03-06T07:25:40.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.318+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80080733b0 con 0x7f7ff00043b0 2024-03-06T07:25:40.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.318+0000 7f7fef7fe640 1 -- 172.21.15.72:0/3523828727 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f7ffc01be40 con 0x7f7ff00043b0 2024-03-06T07:25:40.321 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:40.321 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:40.321 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:40.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ff406b4d0 msgr2=0x7f7ff406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ff406b4d0 0x7f7ff406d990 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8036000 tx=0x7f7ff802e000 comp rx=0 tx=0).stop 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 msgr2=0x7f80081f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f80081f6e20 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00f7f0 tx=0x7f7ffc00fcb0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 shutdown_connections 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ff406b4d0 0x7f7ff406d990 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80080cd1b0 0x7f80080cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80080b6860 0x7f80080b6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 --2- 172.21.15.72:0/3523828727 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ff00043b0 0x7f80081f6e20 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 >> 172.21.15.72:0/3523828727 conn(0x7f80080d3c40 msgr2=0x7f80081611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 shutdown_connections 2024-03-06T07:25:40.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.320+0000 7f800d7e9640 1 -- 172.21.15.72:0/3523828727 wait complete. 2024-03-06T07:25:40.334 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:40.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.433+0000 7fa0ae464640 1 Processor -- start 2024-03-06T07:25:40.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.433+0000 7fa0ae464640 1 -- start start 2024-03-06T07:25:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0ae464640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0a815dfe0 0x7fa0a80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0ae464640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0ae464640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80b3430 con 0x7fa0a80cf830 2024-03-06T07:25:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0ae464640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a80b31e0 con 0x7fa0a815dfe0 2024-03-06T07:25:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0ae464640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b3d60 con 0x7fa0a80d29c0 2024-03-06T07:25:40.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa0a80cf830 0x7fa0a80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38876/0 (socket says 172.21.15.72:38876) 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a77fe640 1 -- 172.21.15.72:0/3877218738 learned_addr learned my addr 172.21.15.72:0/3877218738 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a7fff640 1 --2- 172.21.15.72:0/3877218738 >> v2:172.21.15.146:3300/0 conn(0x7fa0a815dfe0 0x7fa0a80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0aca01640 1 --2- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a6ffd640 1 -- 172.21.15.72:0/3877218738 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534342634 0 0) 0x7fa0a80b3430 con 0x7fa0a80cf830 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a6ffd640 1 -- 172.21.15.72:0/3877218738 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa090003520 con 0x7fa0a80cf830 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a7fff640 1 -- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 msgr2=0x7fa0a80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a7fff640 1 --2- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a80b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a7fff640 1 -- 172.21.15.72:0/3877218738 >> v1:172.21.15.72:6789/0 conn(0x7fa0a80cf830 legacy=0x7fa0a80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.434+0000 7fa0a7fff640 1 -- 172.21.15.72:0/3877218738 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80b3710 con 0x7fa0a815dfe0 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.435+0000 7fa0a7fff640 1 --2- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a80cf2f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097a0 tx=0x7fa09802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20900d5ca07ad6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.435+0000 7fa0a6ffd640 1 -- 172.21.15.72:0/3877218738 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098035020 con 0x7fa0a815dfe0 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.435+0000 7fa0a6ffd640 1 -- 172.21.15.72:0/3877218738 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098031830 con 0x7fa0a815dfe0 2024-03-06T07:25:40.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.435+0000 7fa0a6ffd640 1 -- 172.21.15.72:0/3877218738 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098031be0 con 0x7fa0a815dfe0 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 -- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 msgr2=0x7fa0a80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 --2- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a80cf2f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097a0 tx=0x7fa09802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 -- 172.21.15.72:0/3877218738 shutdown_connections 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 --2- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a80b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 --2- 172.21.15.72:0/3877218738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a80cf2f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 -- 172.21.15.72:0/3877218738 >> 172.21.15.72:0/3877218738 conn(0x7fa0a80d3cd0 msgr2=0x7fa0a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 -- 172.21.15.72:0/3877218738 shutdown_connections 2024-03-06T07:25:40.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.436+0000 7fa0ae464640 1 -- 172.21.15.72:0/3877218738 wait complete. 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.437+0000 7fa0ae464640 1 Processor -- start 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.437+0000 7fa0ae464640 1 -- start start 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0ae464640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 0x7fa0a80ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0ae464640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a8160330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0ae464640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a81f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0ae464640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80d1df0 con 0x7fa0a80cf830 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0ae464640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80d1c70 con 0x7fa0a815dfe0 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0ae464640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a815e3c0 con 0x7fa0a80d29c0 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 0x7fa0a80ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 0x7fa0a80ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47354/0 (socket says 172.21.15.72:47354) 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 -- 172.21.15.72:0/886975798 learned_addr learned my addr 172.21.15.72:0/886975798 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0aca01640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a81f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a77fe640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a8160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 -- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 msgr2=0x7fa0a8160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a8160330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 -- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 msgr2=0x7fa0a81f4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a81f4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a7fff640 1 -- 172.21.15.72:0/886975798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a81f4940 con 0x7fa0a80cf830 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.438+0000 7fa0a77fe640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a8160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.439+0000 7fa0a7fff640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 0x7fa0a80ca810 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fa09802b910 tx=0x7fa0980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.439+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098040070 con 0x7fa0a80cf830 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.439+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09803a910 con 0x7fa0a80cf830 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.439+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a815ec50 con 0x7fa0a80cf830 2024-03-06T07:25:40.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.439+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098035040 con 0x7fa0a80cf830 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.439+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a815f210 con 0x7fa0a80cf830 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.440+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a804f330 con 0x7fa0a80cf830 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.440+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa09804b060 con 0x7fa0a80cf830 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.441+0000 7fa0a4ff9640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa09406b500 0x7fa09406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.441+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0980ab880 con 0x7fa0a80cf830 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.441+0000 7fa0a77fe640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa09406b500 0x7fa09406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.441+0000 7fa0a77fe640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa09406b500 0x7fa09406d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa09c007fd0 tx=0x7fa09c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.445+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa09807a700 con 0x7fa0a80cf830 2024-03-06T07:25:40.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.625+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0a80d0bd0 con 0x7fa0a80cf830 2024-03-06T07:25:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.626+0000 7fa0a4ff9640 1 -- 172.21.15.72:0/886975798 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16672 (secure 0 0 0) 0x7fa09807a000 con 0x7fa0a80cf830 2024-03-06T07:25:40.629 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:40.629 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:38.966454+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa09406b500 msgr2=0x7fa09406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa09406b500 0x7fa09406d9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa09c007fd0 tx=0x7fa09c009040 comp rx=0 tx=0).stop 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 msgr2=0x7fa0a80ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 0x7fa0a80ca810 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fa09802b910 tx=0x7fa0980046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 shutdown_connections 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa09406b500 0x7fa09406d9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0a815dfe0 0x7fa0a81f4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80d29c0 0x7fa0a8160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 --2- 172.21.15.72:0/886975798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0a80cf830 0x7fa0a80ca810 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 >> 172.21.15.72:0/886975798 conn(0x7fa0a80d3cd0 msgr2=0x7fa0a80cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 shutdown_connections 2024-03-06T07:25:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.629+0000 7fa0ae464640 1 -- 172.21.15.72:0/886975798 wait complete. 2024-03-06T07:25:40.643 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:25:40.644 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.743+0000 7f3e47a6b640 1 Processor -- start 2024-03-06T07:25:40.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.743+0000 7f3e47a6b640 1 -- start start 2024-03-06T07:25:40.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.743+0000 7f3e47a6b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e400d22b0 0x7f3e400d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.743+0000 7f3e47a6b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e400b41f0 0x7f3e400b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e47a6b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400b3430 con 0x7f3e4015dfe0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e47a6b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e400d2c50 con 0x7f3e400d22b0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e47a6b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400b31e0 con 0x7f3e400b41f0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e400d22b0 0x7f3e400d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e400d22b0 0x7f3e400d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47366/0 (socket says 172.21.15.72:47366) 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 -- 172.21.15.72:0/2537718739 learned_addr learned my addr 172.21.15.72:0/2537718739 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 -- 172.21.15.72:0/2537718739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e400b41f0 msgr2=0x7f3e400b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e46008640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e400b41f0 0x7f3e400b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e400b41f0 0x7f3e400b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 -- 172.21.15.72:0/2537718739 >> v1:172.21.15.146:6789/0 conn(0x7f3e4015dfe0 legacy=0x7f3e400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e45006640 1 -- 172.21.15.72:0/2537718739 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400b3710 con 0x7f3e400d22b0 2024-03-06T07:25:40.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.744+0000 7f3e46008640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e400b41f0 0x7f3e400b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e45006640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400d22b0 0x7f3e400d2710 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009870 tx=0x7f3e3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8ef94c8ae7ed36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e44805640 1 -- 172.21.15.72:0/2537718739 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30035020 con 0x7f3e400d22b0 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e44805640 1 -- 172.21.15.72:0/2537718739 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e30031640 con 0x7f3e400d22b0 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e44805640 1 -- 172.21.15.72:0/2537718739 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3002d040 con 0x7f3e400d22b0 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 -- 172.21.15.72:0/2537718739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400d22b0 msgr2=0x7f3e400d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400d22b0 0x7f3e400d2710 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3e30009870 tx=0x7f3e3002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 -- 172.21.15.72:0/2537718739 shutdown_connections 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e400b41f0 0x7f3e400b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/2537718739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400d22b0 0x7f3e400d2710 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 -- 172.21.15.72:0/2537718739 >> 172.21.15.72:0/2537718739 conn(0x7f3e400d3cd0 msgr2=0x7f3e400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 -- 172.21.15.72:0/2537718739 shutdown_connections 2024-03-06T07:25:40.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.745+0000 7f3e47a6b640 1 -- 172.21.15.72:0/2537718739 wait complete. 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 Processor -- start 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 -- start start 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 0x7f3e40161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e400d22b0 0x7f3e400b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e4015dfe0 0x7f3e401f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400cfa00 con 0x7f3e400b41f0 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400b3d60 con 0x7f3e400d22b0 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e47a6b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e400d1c70 con 0x7f3e4015dfe0 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e45807640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 0x7f3e40161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e45807640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 0x7f3e40161260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47376/0 (socket says 172.21.15.72:47376) 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.747+0000 7f3e45807640 1 -- 172.21.15.72:0/4008096160 learned_addr learned my addr 172.21.15.72:0/4008096160 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:40.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45006640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e400d22b0 0x7f3e400b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e46008640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e4015dfe0 0x7f3e401f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45807640 1 -- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e4015dfe0 msgr2=0x7f3e401f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45807640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e4015dfe0 0x7f3e401f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45807640 1 -- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e400d22b0 msgr2=0x7f3e400b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45807640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e400d22b0 0x7f3e400b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45807640 1 -- 172.21.15.72:0/4008096160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400b6de0 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e45807640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 0x7f3e40161260 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00b3f0 tx=0x7f3e3c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c015020 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3c011830 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c011c20 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e400b97f0 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.748+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e400cc800 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.749+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4004f330 con 0x7f3e400b41f0 2024-03-06T07:25:40.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.750+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e3c018020 con 0x7f3e400b41f0 2024-03-06T07:25:40.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.750+0000 7f3e367fc640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e2406b4d0 0x7f3e2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:40.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.750+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e3c089e90 con 0x7f3e400b41f0 2024-03-06T07:25:40.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.750+0000 7f3e45006640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e2406b4d0 0x7f3e2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:40.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.751+0000 7f3e45006640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e2406b4d0 0x7f3e2406d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e30036000 tx=0x7f3e30008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:40.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.755+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e3c01f030 con 0x7f3e400b41f0 2024-03-06T07:25:40.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.929+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3e401f3780 con 0x7f3e2406b4d0 2024-03-06T07:25:40.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.935+0000 7f3e367fc640 1 -- 172.21.15.72:0/4008096160 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f3e401f3780 con 0x7f3e2406b4d0 2024-03-06T07:25:40.940 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:40.941 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e2406b4d0 msgr2=0x7f3e2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e2406b4d0 0x7f3e2406d990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f3e30036000 tx=0x7f3e30008030 comp rx=0 tx=0).stop 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 msgr2=0x7f3e40161260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 0x7f3e40161260 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00b3f0 tx=0x7f3e3c00b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 shutdown_connections 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e2406b4d0 0x7f3e2406d990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e4015dfe0 0x7f3e401f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e400d22b0 0x7f3e400b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 --2- 172.21.15.72:0/4008096160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e400b41f0 0x7f3e40161260 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 >> 172.21.15.72:0/4008096160 conn(0x7f3e400d3cd0 msgr2=0x7f3e4006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:40.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 shutdown_connections 2024-03-06T07:25:40.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:40.938+0000 7f3e47a6b640 1 -- 172.21.15.72:0/4008096160 wait complete. 2024-03-06T07:25:40.951 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-06T07:25:39.989500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258284,"kb_used_data":8840,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716436,"statfs":{"total":773094113280,"available":772829630464,"internally_reserved":0,"allocated":9052160,"data_stored":4483952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039599"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968257+0000","last_change":"2024-03-06T07:25:36.968257+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968257+0000","last_undegraded":"2024-03-06T07:25:36.968257+0000","last_fullsized":"2024-03-06T07:25:36.968257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962841+0000","last_change":"2024-03-06T07:25:36.962841+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962841+0000","last_undegraded":"2024-03-06T07:25:36.962841+0000","last_fullsized":"2024-03-06T07:25:36.962841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965522+0000","last_change":"2024-03-06T07:25:36.965522+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965522+0000","last_undegraded":"2024-03-06T07:25:36.965522+0000","last_fullsized":"2024-03-06T07:25:36.965522+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961706+0000","last_change":"2024-03-06T07:25:36.961706+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961706+0000","last_undegraded":"2024-03-06T07:25:36.961706+0000","last_fullsized":"2024-03-06T07:25:36.961706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958594+0000","last_change":"2024-03-06T07:25:36.958594+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958594+0000","last_undegraded":"2024-03-06T07:25:36.958594+0000","last_fullsized":"2024-03-06T07:25:36.958594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961812+0000","last_change":"2024-03-06T07:25:36.961812+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961812+0000","last_undegraded":"2024-03-06T07:25:36.961812+0000","last_fullsized":"2024-03-06T07:25:36.961812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958927+0000","last_change":"2024-03-06T07:25:36.958927+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958927+0000","last_undegraded":"2024-03-06T07:25:36.958927+0000","last_fullsized":"2024-03-06T07:25:36.958927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958269+0000","last_change":"2024-03-06T07:25:36.958269+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958269+0000","last_undegraded":"2024-03-06T07:25:36.958269+0000","last_fullsized":"2024-03-06T07:25:36.958269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963496+0000","last_change":"2024-03-06T07:25:36.963496+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963496+0000","last_undegraded":"2024-03-06T07:25:36.963496+0000","last_fullsized":"2024-03-06T07:25:36.963496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962145+0000","last_change":"2024-03-06T07:25:36.962145+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962145+0000","last_undegraded":"2024-03-06T07:25:36.962145+0000","last_fullsized":"2024-03-06T07:25:36.962145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.970250+0000","last_change":"2024-03-06T07:25:36.970250+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.970250+0000","last_undegraded":"2024-03-06T07:25:36.970250+0000","last_fullsized":"2024-03-06T07:25:36.970250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967316+0000","last_change":"2024-03-06T07:25:36.967316+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967316+0000","last_undegraded":"2024-03-06T07:25:36.967316+0000","last_fullsized":"2024-03-06T07:25:36.967316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961585+0000","last_change":"2024-03-06T07:25:36.961585+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961585+0000","last_undegraded":"2024-03-06T07:25:36.961585+0000","last_fullsized":"2024-03-06T07:25:36.961585+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958329+0000","last_change":"2024-03-06T07:25:36.958329+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958329+0000","last_undegraded":"2024-03-06T07:25:36.958329+0000","last_fullsized":"2024-03-06T07:25:36.958329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962098+0000","last_change":"2024-03-06T07:25:36.962098+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962098+0000","last_undegraded":"2024-03-06T07:25:36.962098+0000","last_fullsized":"2024-03-06T07:25:36.962098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962240+0000","last_change":"2024-03-06T07:25:36.962240+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962240+0000","last_undegraded":"2024-03-06T07:25:36.962240+0000","last_fullsized":"2024-03-06T07:25:36.962240+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961534+0000","last_change":"2024-03-06T07:25:36.961534+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961534+0000","last_undegraded":"2024-03-06T07:25:36.961534+0000","last_fullsized":"2024-03-06T07:25:36.961534+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966351+0000","last_change":"2024-03-06T07:25:36.966351+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966351+0000","last_undegraded":"2024-03-06T07:25:36.966351+0000","last_fullsized":"2024-03-06T07:25:36.966351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961153+0000","last_change":"2024-03-06T07:25:36.961153+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961153+0000","last_undegraded":"2024-03-06T07:25:36.961153+0000","last_fullsized":"2024-03-06T07:25:36.961153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960766+0000","last_change":"2024-03-06T07:25:36.960766+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960766+0000","last_undegraded":"2024-03-06T07:25:36.960766+0000","last_fullsized":"2024-03-06T07:25:36.960766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962960+0000","last_change":"2024-03-06T07:25:36.962960+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962960+0000","last_undegraded":"2024-03-06T07:25:36.962960+0000","last_fullsized":"2024-03-06T07:25:36.962960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964375+0000","last_change":"2024-03-06T07:25:36.964375+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964375+0000","last_undegraded":"2024-03-06T07:25:36.964375+0000","last_fullsized":"2024-03-06T07:25:36.964375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960738+0000","last_change":"2024-03-06T07:25:36.960738+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960738+0000","last_undegraded":"2024-03-06T07:25:36.960738+0000","last_fullsized":"2024-03-06T07:25:36.960738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962994+0000","last_change":"2024-03-06T07:25:36.962994+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962994+0000","last_undegraded":"2024-03-06T07:25:36.962994+0000","last_fullsized":"2024-03-06T07:25:36.962994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965583+0000","last_change":"2024-03-06T07:25:36.965583+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965583+0000","last_undegraded":"2024-03-06T07:25:36.965583+0000","last_fullsized":"2024-03-06T07:25:36.965583+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963919+0000","last_change":"2024-03-06T07:25:36.963919+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963919+0000","last_undegraded":"2024-03-06T07:25:36.963919+0000","last_fullsized":"2024-03-06T07:25:36.963919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969843+0000","last_change":"2024-03-06T07:25:36.969843+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969843+0000","last_undegraded":"2024-03-06T07:25:36.969843+0000","last_fullsized":"2024-03-06T07:25:36.969843+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962327+0000","last_change":"2024-03-06T07:25:36.962327+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962327+0000","last_undegraded":"2024-03-06T07:25:36.962327+0000","last_fullsized":"2024-03-06T07:25:36.962327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962584+0000","last_change":"2024-03-06T07:25:36.962584+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962584+0000","last_undegraded":"2024-03-06T07:25:36.962584+0000","last_fullsized":"2024-03-06T07:25:36.962584+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964171+0000","last_change":"2024-03-06T07:25:36.964171+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964171+0000","last_undegraded":"2024-03-06T07:25:36.964171+0000","last_fullsized":"2024-03-06T07:25:36.964171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963613+0000","last_change":"2024-03-06T07:25:36.963613+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963613+0000","last_undegraded":"2024-03-06T07:25:36.963613+0000","last_fullsized":"2024-03-06T07:25:36.963613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975492+0000","last_change":"2024-03-06T07:25:36.974330+0000","last_active":"2024-03-06T07:25:36.975492+0000","last_peered":"2024-03-06T07:25:36.975492+0000","last_clean":"2024-03-06T07:25:36.975492+0000","last_became_active":"2024-03-06T07:25:36.974050+0000","last_became_peered":"2024-03-06T07:25:36.974050+0000","last_unstale":"2024-03-06T07:25:36.975492+0000","last_undegraded":"2024-03-06T07:25:36.975492+0000","last_fullsized":"2024-03-06T07:25:36.975492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:15.879147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967828+0000","last_change":"2024-03-06T07:25:36.967828+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967828+0000","last_undegraded":"2024-03-06T07:25:36.967828+0000","last_fullsized":"2024-03-06T07:25:36.967828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972393+0000","last_change":"2024-03-06T07:25:36.967151+0000","last_active":"2024-03-06T07:25:36.972393+0000","last_peered":"2024-03-06T07:25:36.972393+0000","last_clean":"2024-03-06T07:25:36.972393+0000","last_became_active":"2024-03-06T07:25:36.966864+0000","last_became_peered":"2024-03-06T07:25:36.966864+0000","last_unstale":"2024-03-06T07:25:36.972393+0000","last_undegraded":"2024-03-06T07:25:36.972393+0000","last_fullsized":"2024-03-06T07:25:36.972393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969345+0000","last_change":"2024-03-06T07:25:36.969345+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969345+0000","last_undegraded":"2024-03-06T07:25:36.969345+0000","last_fullsized":"2024-03-06T07:25:36.969345+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976130+0000","last_change":"2024-03-06T07:25:36.974099+0000","last_active":"2024-03-06T07:25:36.976130+0000","last_peered":"2024-03-06T07:25:36.976130+0000","last_clean":"2024-03-06T07:25:36.976130+0000","last_became_active":"2024-03-06T07:25:36.973818+0000","last_became_peered":"2024-03-06T07:25:36.973818+0000","last_unstale":"2024-03-06T07:25:36.976130+0000","last_undegraded":"2024-03-06T07:25:36.976130+0000","last_fullsized":"2024-03-06T07:25:36.976130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:36.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965086+0000","last_change":"2024-03-06T07:25:36.965086+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965086+0000","last_undegraded":"2024-03-06T07:25:36.965086+0000","last_fullsized":"2024-03-06T07:25:36.965086+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.978822+0000","last_change":"2024-03-06T07:25:36.974705+0000","last_active":"2024-03-06T07:25:36.978822+0000","last_peered":"2024-03-06T07:25:36.978822+0000","last_clean":"2024-03-06T07:25:36.978822+0000","last_became_active":"2024-03-06T07:25:36.974418+0000","last_became_peered":"2024-03-06T07:25:36.974418+0000","last_unstale":"2024-03-06T07:25:36.978822+0000","last_undegraded":"2024-03-06T07:25:36.978822+0000","last_fullsized":"2024-03-06T07:25:36.978822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962201+0000","last_change":"2024-03-06T07:25:36.962201+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962201+0000","last_undegraded":"2024-03-06T07:25:36.962201+0000","last_fullsized":"2024-03-06T07:25:36.962201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972867+0000","last_change":"2024-03-06T07:25:36.970497+0000","last_active":"2024-03-06T07:25:36.972867+0000","last_peered":"2024-03-06T07:25:36.972867+0000","last_clean":"2024-03-06T07:25:36.972867+0000","last_became_active":"2024-03-06T07:25:36.970229+0000","last_became_peered":"2024-03-06T07:25:36.970229+0000","last_unstale":"2024-03-06T07:25:36.972867+0000","last_undegraded":"2024-03-06T07:25:36.972867+0000","last_fullsized":"2024-03-06T07:25:36.972867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962580+0000","last_change":"2024-03-06T07:25:36.962580+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962580+0000","last_undegraded":"2024-03-06T07:25:36.962580+0000","last_fullsized":"2024-03-06T07:25:36.962580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972874+0000","last_change":"2024-03-06T07:25:36.972014+0000","last_active":"2024-03-06T07:25:36.972874+0000","last_peered":"2024-03-06T07:25:36.972874+0000","last_clean":"2024-03-06T07:25:36.972874+0000","last_became_active":"2024-03-06T07:25:36.971692+0000","last_became_peered":"2024-03-06T07:25:36.971692+0000","last_unstale":"2024-03-06T07:25:36.972874+0000","last_undegraded":"2024-03-06T07:25:36.972874+0000","last_fullsized":"2024-03-06T07:25:36.972874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.918383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965503+0000","last_change":"2024-03-06T07:25:36.965503+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965503+0000","last_undegraded":"2024-03-06T07:25:36.965503+0000","last_fullsized":"2024-03-06T07:25:36.965503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974849+0000","last_change":"2024-03-06T07:25:36.969239+0000","last_active":"2024-03-06T07:25:36.974849+0000","last_peered":"2024-03-06T07:25:36.974849+0000","last_clean":"2024-03-06T07:25:36.974849+0000","last_became_active":"2024-03-06T07:25:36.968491+0000","last_became_peered":"2024-03-06T07:25:36.968491+0000","last_unstale":"2024-03-06T07:25:36.974849+0000","last_undegraded":"2024-03-06T07:25:36.974849+0000","last_fullsized":"2024-03-06T07:25:36.974849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957846+0000","last_change":"2024-03-06T07:25:36.957846+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957846+0000","last_undegraded":"2024-03-06T07:25:36.957846+0000","last_fullsized":"2024-03-06T07:25:36.957846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975832+0000","last_change":"2024-03-06T07:25:36.974784+0000","last_active":"2024-03-06T07:25:36.975832+0000","last_peered":"2024-03-06T07:25:36.975832+0000","last_clean":"2024-03-06T07:25:36.975832+0000","last_became_active":"2024-03-06T07:25:36.974534+0000","last_became_peered":"2024-03-06T07:25:36.974534+0000","last_unstale":"2024-03-06T07:25:36.975832+0000","last_undegraded":"2024-03-06T07:25:36.975832+0000","last_fullsized":"2024-03-06T07:25:36.975832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:35.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958526+0000","last_change":"2024-03-06T07:25:36.958526+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958526+0000","last_undegraded":"2024-03-06T07:25:36.958526+0000","last_fullsized":"2024-03-06T07:25:36.958526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976718+0000","last_change":"2024-03-06T07:25:36.975629+0000","last_active":"2024-03-06T07:25:36.976718+0000","last_peered":"2024-03-06T07:25:36.976718+0000","last_clean":"2024-03-06T07:25:36.976718+0000","last_became_active":"2024-03-06T07:25:36.975298+0000","last_became_peered":"2024-03-06T07:25:36.975298+0000","last_unstale":"2024-03-06T07:25:36.976718+0000","last_undegraded":"2024-03-06T07:25:36.976718+0000","last_fullsized":"2024-03-06T07:25:36.976718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957680+0000","last_change":"2024-03-06T07:25:36.957680+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957680+0000","last_undegraded":"2024-03-06T07:25:36.957680+0000","last_fullsized":"2024-03-06T07:25:36.957680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976133+0000","last_change":"2024-03-06T07:25:36.974955+0000","last_active":"2024-03-06T07:25:36.976133+0000","last_peered":"2024-03-06T07:25:36.976133+0000","last_clean":"2024-03-06T07:25:36.976133+0000","last_became_active":"2024-03-06T07:25:36.974657+0000","last_became_peered":"2024-03-06T07:25:36.974657+0000","last_unstale":"2024-03-06T07:25:36.976133+0000","last_undegraded":"2024-03-06T07:25:36.976133+0000","last_fullsized":"2024-03-06T07:25:36.976133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:33.339070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966598+0000","last_change":"2024-03-06T07:25:36.966598+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966598+0000","last_undegraded":"2024-03-06T07:25:36.966598+0000","last_fullsized":"2024-03-06T07:25:36.966598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975518+0000","last_change":"2024-03-06T07:25:36.974695+0000","last_active":"2024-03-06T07:25:36.975518+0000","last_peered":"2024-03-06T07:25:36.975518+0000","last_clean":"2024-03-06T07:25:36.975518+0000","last_became_active":"2024-03-06T07:25:36.974384+0000","last_became_peered":"2024-03-06T07:25:36.974384+0000","last_unstale":"2024-03-06T07:25:36.975518+0000","last_undegraded":"2024-03-06T07:25:36.975518+0000","last_fullsized":"2024-03-06T07:25:36.975518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:25.464178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963479+0000","last_change":"2024-03-06T07:25:36.963479+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963479+0000","last_undegraded":"2024-03-06T07:25:36.963479+0000","last_fullsized":"2024-03-06T07:25:36.963479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.965356+0000","last_change":"2024-03-06T07:25:36.964206+0000","last_active":"2024-03-06T07:25:36.965356+0000","last_peered":"2024-03-06T07:25:36.965356+0000","last_clean":"2024-03-06T07:25:36.965356+0000","last_became_active":"2024-03-06T07:25:36.963893+0000","last_became_peered":"2024-03-06T07:25:36.963893+0000","last_unstale":"2024-03-06T07:25:36.965356+0000","last_undegraded":"2024-03-06T07:25:36.965356+0000","last_fullsized":"2024-03-06T07:25:36.965356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966036+0000","last_change":"2024-03-06T07:25:36.966036+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966036+0000","last_undegraded":"2024-03-06T07:25:36.966036+0000","last_fullsized":"2024-03-06T07:25:36.966036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974005+0000","last_change":"2024-03-06T07:25:36.972698+0000","last_active":"2024-03-06T07:25:36.974005+0000","last_peered":"2024-03-06T07:25:36.974005+0000","last_clean":"2024-03-06T07:25:36.974005+0000","last_became_active":"2024-03-06T07:25:36.972395+0000","last_became_peered":"2024-03-06T07:25:36.972395+0000","last_unstale":"2024-03-06T07:25:36.974005+0000","last_undegraded":"2024-03-06T07:25:36.974005+0000","last_fullsized":"2024-03-06T07:25:36.974005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:50.352156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966888+0000","last_change":"2024-03-06T07:25:36.966888+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966888+0000","last_undegraded":"2024-03-06T07:25:36.966888+0000","last_fullsized":"2024-03-06T07:25:36.966888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973687+0000","last_change":"2024-03-06T07:25:36.968375+0000","last_active":"2024-03-06T07:25:36.973687+0000","last_peered":"2024-03-06T07:25:36.973687+0000","last_clean":"2024-03-06T07:25:36.973687+0000","last_became_active":"2024-03-06T07:25:36.967683+0000","last_became_peered":"2024-03-06T07:25:36.967683+0000","last_unstale":"2024-03-06T07:25:36.973687+0000","last_undegraded":"2024-03-06T07:25:36.973687+0000","last_fullsized":"2024-03-06T07:25:36.973687+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:10.418371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961504+0000","last_change":"2024-03-06T07:25:36.961504+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961504+0000","last_undegraded":"2024-03-06T07:25:36.961504+0000","last_fullsized":"2024-03-06T07:25:36.961504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972289+0000","last_change":"2024-03-06T07:25:36.971741+0000","last_active":"2024-03-06T07:25:36.972289+0000","last_peered":"2024-03-06T07:25:36.972289+0000","last_clean":"2024-03-06T07:25:36.972289+0000","last_became_active":"2024-03-06T07:25:36.970858+0000","last_became_peered":"2024-03-06T07:25:36.970858+0000","last_unstale":"2024-03-06T07:25:36.972289+0000","last_undegraded":"2024-03-06T07:25:36.972289+0000","last_fullsized":"2024-03-06T07:25:36.972289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961071+0000","last_change":"2024-03-06T07:25:36.961071+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961071+0000","last_undegraded":"2024-03-06T07:25:36.961071+0000","last_fullsized":"2024-03-06T07:25:36.961071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976083+0000","last_change":"2024-03-06T07:25:36.972032+0000","last_active":"2024-03-06T07:25:36.976083+0000","last_peered":"2024-03-06T07:25:36.976083+0000","last_clean":"2024-03-06T07:25:36.976083+0000","last_became_active":"2024-03-06T07:25:36.971777+0000","last_became_peered":"2024-03-06T07:25:36.971777+0000","last_unstale":"2024-03-06T07:25:36.976083+0000","last_undegraded":"2024-03-06T07:25:36.976083+0000","last_fullsized":"2024-03-06T07:25:36.976083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:32.818833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965095+0000","last_change":"2024-03-06T07:25:36.965095+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965095+0000","last_undegraded":"2024-03-06T07:25:36.965095+0000","last_fullsized":"2024-03-06T07:25:36.965095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974248+0000","last_change":"2024-03-06T07:25:36.972827+0000","last_active":"2024-03-06T07:25:36.974248+0000","last_peered":"2024-03-06T07:25:36.974248+0000","last_clean":"2024-03-06T07:25:36.974248+0000","last_became_active":"2024-03-06T07:25:36.972549+0000","last_became_peered":"2024-03-06T07:25:36.972549+0000","last_unstale":"2024-03-06T07:25:36.974248+0000","last_undegraded":"2024-03-06T07:25:36.974248+0000","last_fullsized":"2024-03-06T07:25:36.974248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:53.181415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967228+0000","last_change":"2024-03-06T07:25:36.967228+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967228+0000","last_undegraded":"2024-03-06T07:25:36.967228+0000","last_fullsized":"2024-03-06T07:25:36.967228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.978846+0000","last_change":"2024-03-06T07:25:36.974812+0000","last_active":"2024-03-06T07:25:36.978846+0000","last_peered":"2024-03-06T07:25:36.978846+0000","last_clean":"2024-03-06T07:25:36.978846+0000","last_became_active":"2024-03-06T07:25:36.974555+0000","last_became_peered":"2024-03-06T07:25:36.974555+0000","last_unstale":"2024-03-06T07:25:36.978846+0000","last_undegraded":"2024-03-06T07:25:36.978846+0000","last_fullsized":"2024-03-06T07:25:36.978846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:39.570670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962957+0000","last_change":"2024-03-06T07:25:36.962957+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962957+0000","last_undegraded":"2024-03-06T07:25:36.962957+0000","last_fullsized":"2024-03-06T07:25:36.962957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974884+0000","last_change":"2024-03-06T07:25:36.968979+0000","last_active":"2024-03-06T07:25:36.974884+0000","last_peered":"2024-03-06T07:25:36.974884+0000","last_clean":"2024-03-06T07:25:36.974884+0000","last_became_active":"2024-03-06T07:25:36.968727+0000","last_became_peered":"2024-03-06T07:25:36.968727+0000","last_unstale":"2024-03-06T07:25:36.974884+0000","last_undegraded":"2024-03-06T07:25:36.974884+0000","last_fullsized":"2024-03-06T07:25:36.974884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:18.410780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960312+0000","last_change":"2024-03-06T07:25:36.960312+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960312+0000","last_undegraded":"2024-03-06T07:25:36.960312+0000","last_fullsized":"2024-03-06T07:25:36.960312+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963596+0000","last_change":"2024-03-06T07:25:36.962047+0000","last_active":"2024-03-06T07:25:36.963596+0000","last_peered":"2024-03-06T07:25:36.963596+0000","last_clean":"2024-03-06T07:25:36.963596+0000","last_became_active":"2024-03-06T07:25:36.961774+0000","last_became_peered":"2024-03-06T07:25:36.961774+0000","last_unstale":"2024-03-06T07:25:36.963596+0000","last_undegraded":"2024-03-06T07:25:36.963596+0000","last_fullsized":"2024-03-06T07:25:36.963596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:24.570961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957718+0000","last_change":"2024-03-06T07:25:36.957718+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957718+0000","last_undegraded":"2024-03-06T07:25:36.957718+0000","last_fullsized":"2024-03-06T07:25:36.957718+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973502+0000","last_change":"2024-03-06T07:25:36.970500+0000","last_active":"2024-03-06T07:25:36.973502+0000","last_peered":"2024-03-06T07:25:36.973502+0000","last_clean":"2024-03-06T07:25:36.973502+0000","last_became_active":"2024-03-06T07:25:36.970286+0000","last_became_peered":"2024-03-06T07:25:36.970286+0000","last_unstale":"2024-03-06T07:25:36.973502+0000","last_undegraded":"2024-03-06T07:25:36.973502+0000","last_fullsized":"2024-03-06T07:25:36.973502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:03.015663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960298+0000","last_change":"2024-03-06T07:25:36.960298+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960298+0000","last_undegraded":"2024-03-06T07:25:36.960298+0000","last_fullsized":"2024-03-06T07:25:36.960298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974135+0000","last_change":"2024-03-06T07:25:36.971016+0000","last_active":"2024-03-06T07:25:36.974135+0000","last_peered":"2024-03-06T07:25:36.974135+0000","last_clean":"2024-03-06T07:25:36.974135+0000","last_became_active":"2024-03-06T07:25:36.970738+0000","last_became_peered":"2024-03-06T07:25:36.970738+0000","last_unstale":"2024-03-06T07:25:36.974135+0000","last_undegraded":"2024-03-06T07:25:36.974135+0000","last_fullsized":"2024-03-06T07:25:36.974135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:48.010645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958936+0000","last_change":"2024-03-06T07:25:36.958936+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958936+0000","last_undegraded":"2024-03-06T07:25:36.958936+0000","last_fullsized":"2024-03-06T07:25:36.958936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963460+0000","last_change":"2024-03-06T07:25:36.961782+0000","last_active":"2024-03-06T07:25:36.963460+0000","last_peered":"2024-03-06T07:25:36.963460+0000","last_clean":"2024-03-06T07:25:36.963460+0000","last_became_active":"2024-03-06T07:25:36.961534+0000","last_became_peered":"2024-03-06T07:25:36.961534+0000","last_unstale":"2024-03-06T07:25:36.963460+0000","last_undegraded":"2024-03-06T07:25:36.963460+0000","last_fullsized":"2024-03-06T07:25:36.963460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:17.024203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963202+0000","last_change":"2024-03-06T07:25:36.963202+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963202+0000","last_undegraded":"2024-03-06T07:25:36.963202+0000","last_fullsized":"2024-03-06T07:25:36.963202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970981+0000","last_change":"2024-03-06T07:25:36.968364+0000","last_active":"2024-03-06T07:25:36.970981+0000","last_peered":"2024-03-06T07:25:36.970981+0000","last_clean":"2024-03-06T07:25:36.970981+0000","last_became_active":"2024-03-06T07:25:36.968108+0000","last_became_peered":"2024-03-06T07:25:36.968108+0000","last_unstale":"2024-03-06T07:25:36.970981+0000","last_undegraded":"2024-03-06T07:25:36.970981+0000","last_fullsized":"2024-03-06T07:25:36.970981+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:35.303723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966589+0000","last_change":"2024-03-06T07:25:36.966589+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966589+0000","last_undegraded":"2024-03-06T07:25:36.966589+0000","last_fullsized":"2024-03-06T07:25:36.966589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974174+0000","last_change":"2024-03-06T07:25:36.972060+0000","last_active":"2024-03-06T07:25:36.974174+0000","last_peered":"2024-03-06T07:25:36.974174+0000","last_clean":"2024-03-06T07:25:36.974174+0000","last_became_active":"2024-03-06T07:25:36.971774+0000","last_became_peered":"2024-03-06T07:25:36.971774+0000","last_unstale":"2024-03-06T07:25:36.974174+0000","last_undegraded":"2024-03-06T07:25:36.974174+0000","last_fullsized":"2024-03-06T07:25:36.974174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:32.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968493+0000","last_change":"2024-03-06T07:25:36.968493+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968493+0000","last_undegraded":"2024-03-06T07:25:36.968493+0000","last_fullsized":"2024-03-06T07:25:36.968493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974580+0000","last_change":"2024-03-06T07:25:36.972458+0000","last_active":"2024-03-06T07:25:36.974580+0000","last_peered":"2024-03-06T07:25:36.974580+0000","last_clean":"2024-03-06T07:25:36.974580+0000","last_became_active":"2024-03-06T07:25:36.971925+0000","last_became_peered":"2024-03-06T07:25:36.971925+0000","last_unstale":"2024-03-06T07:25:36.974580+0000","last_undegraded":"2024-03-06T07:25:36.974580+0000","last_fullsized":"2024-03-06T07:25:36.974580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:13.397656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962197+0000","last_change":"2024-03-06T07:25:36.962197+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962197+0000","last_undegraded":"2024-03-06T07:25:36.962197+0000","last_fullsized":"2024-03-06T07:25:36.962197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977043+0000","last_change":"2024-03-06T07:25:36.976309+0000","last_active":"2024-03-06T07:25:36.977043+0000","last_peered":"2024-03-06T07:25:36.977043+0000","last_clean":"2024-03-06T07:25:36.977043+0000","last_became_active":"2024-03-06T07:25:36.976019+0000","last_became_peered":"2024-03-06T07:25:36.976019+0000","last_unstale":"2024-03-06T07:25:36.977043+0000","last_undegraded":"2024-03-06T07:25:36.977043+0000","last_fullsized":"2024-03-06T07:25:36.977043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957878+0000","last_change":"2024-03-06T07:25:36.957878+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957878+0000","last_undegraded":"2024-03-06T07:25:36.957878+0000","last_fullsized":"2024-03-06T07:25:36.957878+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976198+0000","last_change":"2024-03-06T07:25:36.971247+0000","last_active":"2024-03-06T07:25:36.976198+0000","last_peered":"2024-03-06T07:25:36.976198+0000","last_clean":"2024-03-06T07:25:36.976198+0000","last_became_active":"2024-03-06T07:25:36.970950+0000","last_became_peered":"2024-03-06T07:25:36.970950+0000","last_unstale":"2024-03-06T07:25:36.976198+0000","last_undegraded":"2024-03-06T07:25:36.976198+0000","last_fullsized":"2024-03-06T07:25:36.976198+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964978+0000","last_change":"2024-03-06T07:25:36.964978+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964978+0000","last_undegraded":"2024-03-06T07:25:36.964978+0000","last_fullsized":"2024-03-06T07:25:36.964978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970386+0000","last_change":"2024-03-06T07:25:36.967074+0000","last_active":"2024-03-06T07:25:36.970386+0000","last_peered":"2024-03-06T07:25:36.970386+0000","last_clean":"2024-03-06T07:25:36.970386+0000","last_became_active":"2024-03-06T07:25:36.966752+0000","last_became_peered":"2024-03-06T07:25:36.966752+0000","last_unstale":"2024-03-06T07:25:36.970386+0000","last_undegraded":"2024-03-06T07:25:36.970386+0000","last_fullsized":"2024-03-06T07:25:36.970386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965024+0000","last_change":"2024-03-06T07:25:36.965024+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965024+0000","last_undegraded":"2024-03-06T07:25:36.965024+0000","last_fullsized":"2024-03-06T07:25:36.965024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977030+0000","last_change":"2024-03-06T07:25:36.975964+0000","last_active":"2024-03-06T07:25:36.977030+0000","last_peered":"2024-03-06T07:25:36.977030+0000","last_clean":"2024-03-06T07:25:36.977030+0000","last_became_active":"2024-03-06T07:25:36.975665+0000","last_became_peered":"2024-03-06T07:25:36.975665+0000","last_unstale":"2024-03-06T07:25:36.977030+0000","last_undegraded":"2024-03-06T07:25:36.977030+0000","last_fullsized":"2024-03-06T07:25:36.977030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963945+0000","last_change":"2024-03-06T07:25:36.963945+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963945+0000","last_undegraded":"2024-03-06T07:25:36.963945+0000","last_fullsized":"2024-03-06T07:25:36.963945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976161+0000","last_change":"2024-03-06T07:25:36.974079+0000","last_active":"2024-03-06T07:25:36.976161+0000","last_peered":"2024-03-06T07:25:36.976161+0000","last_clean":"2024-03-06T07:25:36.976161+0000","last_became_active":"2024-03-06T07:25:36.973733+0000","last_became_peered":"2024-03-06T07:25:36.973733+0000","last_unstale":"2024-03-06T07:25:36.976161+0000","last_undegraded":"2024-03-06T07:25:36.976161+0000","last_fullsized":"2024-03-06T07:25:36.976161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:05.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962367+0000","last_change":"2024-03-06T07:25:36.962367+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962367+0000","last_undegraded":"2024-03-06T07:25:36.962367+0000","last_fullsized":"2024-03-06T07:25:36.962367+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975679+0000","last_change":"2024-03-06T07:25:36.974734+0000","last_active":"2024-03-06T07:25:36.975679+0000","last_peered":"2024-03-06T07:25:36.975679+0000","last_clean":"2024-03-06T07:25:36.975679+0000","last_became_active":"2024-03-06T07:25:36.974426+0000","last_became_peered":"2024-03-06T07:25:36.974426+0000","last_unstale":"2024-03-06T07:25:36.975679+0000","last_undegraded":"2024-03-06T07:25:36.975679+0000","last_fullsized":"2024-03-06T07:25:36.975679+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.967622+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:25:36.967622+0000","last_peered":"2024-03-06T07:25:36.967622+0000","last_clean":"2024-03-06T07:25:36.967622+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:25:36.967622+0000","last_undegraded":"2024-03-06T07:25:36.967622+0000","last_fullsized":"2024-03-06T07:25:36.967622+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:40.956 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:41.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.093+0000 7f3e04119640 1 Processor -- start 2024-03-06T07:25:41.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.093+0000 7f3e04119640 1 -- start start 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e04119640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3dfc0d22b0 0x7f3dfc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e04119640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3dfc0b41f0 0x7f3dfc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e04119640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dfc0b3430 con 0x7f3dfc15dfe0 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e04119640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0d2c50 con 0x7f3dfc0b41f0 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e04119640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b31e0 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e026b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3dfc0b41f0 0x7f3dfc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e026b6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3dfc0b41f0 0x7f3dfc0b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47382/0 (socket says 172.21.15.72:47382) 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.094+0000 7f3e026b6640 1 -- 172.21.15.72:0/3332413790 learned_addr learned my addr 172.21.15.72:0/3332413790 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e016b4640 1 --2- 172.21.15.72:0/3332413790 >> v2:172.21.15.146:3300/0 conn(0x7f3dfc0d22b0 0x7f3dfc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e026b6640 1 -- 172.21.15.72:0/3332413790 >> v2:172.21.15.146:3300/0 conn(0x7f3dfc0d22b0 msgr2=0x7f3dfc0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e026b6640 1 --2- 172.21.15.72:0/3332413790 >> v2:172.21.15.146:3300/0 conn(0x7f3dfc0d22b0 0x7f3dfc0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e026b6640 1 -- 172.21.15.72:0/3332413790 >> v1:172.21.15.146:6789/0 conn(0x7f3dfc15dfe0 legacy=0x7f3dfc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e026b6640 1 -- 172.21.15.72:0/3332413790 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0b3710 con 0x7f3dfc0b41f0 2024-03-06T07:25:41.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e026b6640 1 --2- 172.21.15.72:0/3332413790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0b41f0 0x7f3dfc0b4680 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009e50 tx=0x7f3df402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c7bf26428eaf18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.095+0000 7f3e00eb3640 1 -- 172.21.15.72:0/3332413790 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df402ca80 con 0x7f3dfc0b41f0 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e00eb3640 1 -- 172.21.15.72:0/3332413790 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df402cc20 con 0x7f3dfc0b41f0 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e00eb3640 1 -- 172.21.15.72:0/3332413790 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df40343f0 con 0x7f3dfc0b41f0 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 -- 172.21.15.72:0/3332413790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0b41f0 msgr2=0x7f3dfc0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 --2- 172.21.15.72:0/3332413790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0b41f0 0x7f3dfc0b4680 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f3df4009e50 tx=0x7f3df402bee0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 -- 172.21.15.72:0/3332413790 shutdown_connections 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 --2- 172.21.15.72:0/3332413790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0b41f0 0x7f3dfc0b4680 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 --2- 172.21.15.72:0/3332413790 >> v2:172.21.15.146:3300/0 conn(0x7f3dfc0d22b0 0x7f3dfc0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 -- 172.21.15.72:0/3332413790 >> 172.21.15.72:0/3332413790 conn(0x7f3dfc0d3cd0 msgr2=0x7f3dfc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 -- 172.21.15.72:0/3332413790 shutdown_connections 2024-03-06T07:25:41.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.096+0000 7f3e04119640 1 -- 172.21.15.72:0/3332413790 wait complete. 2024-03-06T07:25:41.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.097+0000 7f3e04119640 1 Processor -- start 2024-03-06T07:25:41.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.097+0000 7f3e04119640 1 -- start start 2024-03-06T07:25:41.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e04119640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc0b41f0 0x7f3dfc0b8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e04119640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 0x7f3dfc0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e04119640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc15dfe0 0x7f3dfc0b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e04119640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0cfa00 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e04119640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0b3d60 con 0x7f3dfc0b41f0 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e04119640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dfc0d1c70 con 0x7f3dfc15dfe0 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 0x7f3dfc0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 0x7f3dfc0b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47392/0 (socket says 172.21.15.72:47392) 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 -- 172.21.15.72:0/2557405086 learned_addr learned my addr 172.21.15.72:0/2557405086 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 -- 172.21.15.72:0/2557405086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc15dfe0 msgr2=0x7f3dfc0b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc15dfe0 0x7f3dfc0b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 -- 172.21.15.72:0/2557405086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc0b41f0 msgr2=0x7f3dfc0b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc0b41f0 0x7f3dfc0b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 -- 172.21.15.72:0/2557405086 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0b9be0 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.098+0000 7f3e016b4640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 0x7f3dfc0b68a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c820 tx=0x7f3dec00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.099+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec015070 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.099+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dec004240 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.099+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dfc0b9060 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.099+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dec00ce10 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.099+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dfc0b9620 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.100+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dfc04f330 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.100+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3dec007830 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.100+0000 7f3df27fc640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3de006b500 0x7f3de006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.101+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dec08a0c0 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.101+0000 7f3e01eb5640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3de006b500 0x7f3de006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.101+0000 7f3e01eb5640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3de006b500 0x7f3de006d9c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3df8006fd0 tx=0x7f3df8006bc0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.105+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3dec058f40 con 0x7f3dfc0d22b0 2024-03-06T07:25:41.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.277+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3dfc15d3e0 con 0x7f3de006b500 2024-03-06T07:25:41.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.288+0000 7f3df27fc640 1 -- 172.21.15.72:0/2557405086 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f3dfc15d3e0 con 0x7f3de006b500 2024-03-06T07:25:41.293 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:41.294 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3de006b500 msgr2=0x7f3de006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3de006b500 0x7f3de006d9c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3df8006fd0 tx=0x7f3df8006bc0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 msgr2=0x7f3dfc0b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 0x7f3dfc0b68a0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f3dec00c820 tx=0x7f3dec00cce0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 shutdown_connections 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3de006b500 0x7f3de006d9c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3dfc15dfe0 0x7f3dfc0b6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3dfc0d22b0 0x7f3dfc0b68a0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 --2- 172.21.15.72:0/2557405086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3dfc0b41f0 0x7f3dfc0b8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 >> 172.21.15.72:0/2557405086 conn(0x7f3dfc0d3cd0 msgr2=0x7f3dfc0d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 shutdown_connections 2024-03-06T07:25:41.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.292+0000 7f3e04119640 1 -- 172.21.15.72:0/2557405086 wait complete. 2024-03-06T07:25:41.305 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-06T07:25:39.989500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258284,"kb_used_data":8840,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716436,"statfs":{"total":773094113280,"available":772829630464,"internally_reserved":0,"allocated":9052160,"data_stored":4483952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039599"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968257+0000","last_change":"2024-03-06T07:25:36.968257+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968257+0000","last_undegraded":"2024-03-06T07:25:36.968257+0000","last_fullsized":"2024-03-06T07:25:36.968257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962841+0000","last_change":"2024-03-06T07:25:36.962841+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962841+0000","last_undegraded":"2024-03-06T07:25:36.962841+0000","last_fullsized":"2024-03-06T07:25:36.962841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965522+0000","last_change":"2024-03-06T07:25:36.965522+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965522+0000","last_undegraded":"2024-03-06T07:25:36.965522+0000","last_fullsized":"2024-03-06T07:25:36.965522+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961706+0000","last_change":"2024-03-06T07:25:36.961706+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961706+0000","last_undegraded":"2024-03-06T07:25:36.961706+0000","last_fullsized":"2024-03-06T07:25:36.961706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958594+0000","last_change":"2024-03-06T07:25:36.958594+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958594+0000","last_undegraded":"2024-03-06T07:25:36.958594+0000","last_fullsized":"2024-03-06T07:25:36.958594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961812+0000","last_change":"2024-03-06T07:25:36.961812+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961812+0000","last_undegraded":"2024-03-06T07:25:36.961812+0000","last_fullsized":"2024-03-06T07:25:36.961812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958927+0000","last_change":"2024-03-06T07:25:36.958927+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958927+0000","last_undegraded":"2024-03-06T07:25:36.958927+0000","last_fullsized":"2024-03-06T07:25:36.958927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958269+0000","last_change":"2024-03-06T07:25:36.958269+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958269+0000","last_undegraded":"2024-03-06T07:25:36.958269+0000","last_fullsized":"2024-03-06T07:25:36.958269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963496+0000","last_change":"2024-03-06T07:25:36.963496+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963496+0000","last_undegraded":"2024-03-06T07:25:36.963496+0000","last_fullsized":"2024-03-06T07:25:36.963496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962145+0000","last_change":"2024-03-06T07:25:36.962145+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962145+0000","last_undegraded":"2024-03-06T07:25:36.962145+0000","last_fullsized":"2024-03-06T07:25:36.962145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.970250+0000","last_change":"2024-03-06T07:25:36.970250+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.970250+0000","last_undegraded":"2024-03-06T07:25:36.970250+0000","last_fullsized":"2024-03-06T07:25:36.970250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967316+0000","last_change":"2024-03-06T07:25:36.967316+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967316+0000","last_undegraded":"2024-03-06T07:25:36.967316+0000","last_fullsized":"2024-03-06T07:25:36.967316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961585+0000","last_change":"2024-03-06T07:25:36.961585+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961585+0000","last_undegraded":"2024-03-06T07:25:36.961585+0000","last_fullsized":"2024-03-06T07:25:36.961585+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958329+0000","last_change":"2024-03-06T07:25:36.958329+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958329+0000","last_undegraded":"2024-03-06T07:25:36.958329+0000","last_fullsized":"2024-03-06T07:25:36.958329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962098+0000","last_change":"2024-03-06T07:25:36.962098+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962098+0000","last_undegraded":"2024-03-06T07:25:36.962098+0000","last_fullsized":"2024-03-06T07:25:36.962098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962240+0000","last_change":"2024-03-06T07:25:36.962240+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962240+0000","last_undegraded":"2024-03-06T07:25:36.962240+0000","last_fullsized":"2024-03-06T07:25:36.962240+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961534+0000","last_change":"2024-03-06T07:25:36.961534+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961534+0000","last_undegraded":"2024-03-06T07:25:36.961534+0000","last_fullsized":"2024-03-06T07:25:36.961534+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966351+0000","last_change":"2024-03-06T07:25:36.966351+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966351+0000","last_undegraded":"2024-03-06T07:25:36.966351+0000","last_fullsized":"2024-03-06T07:25:36.966351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961153+0000","last_change":"2024-03-06T07:25:36.961153+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961153+0000","last_undegraded":"2024-03-06T07:25:36.961153+0000","last_fullsized":"2024-03-06T07:25:36.961153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960766+0000","last_change":"2024-03-06T07:25:36.960766+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960766+0000","last_undegraded":"2024-03-06T07:25:36.960766+0000","last_fullsized":"2024-03-06T07:25:36.960766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962960+0000","last_change":"2024-03-06T07:25:36.962960+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962960+0000","last_undegraded":"2024-03-06T07:25:36.962960+0000","last_fullsized":"2024-03-06T07:25:36.962960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964375+0000","last_change":"2024-03-06T07:25:36.964375+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964375+0000","last_undegraded":"2024-03-06T07:25:36.964375+0000","last_fullsized":"2024-03-06T07:25:36.964375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960738+0000","last_change":"2024-03-06T07:25:36.960738+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960738+0000","last_undegraded":"2024-03-06T07:25:36.960738+0000","last_fullsized":"2024-03-06T07:25:36.960738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962994+0000","last_change":"2024-03-06T07:25:36.962994+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962994+0000","last_undegraded":"2024-03-06T07:25:36.962994+0000","last_fullsized":"2024-03-06T07:25:36.962994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965583+0000","last_change":"2024-03-06T07:25:36.965583+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965583+0000","last_undegraded":"2024-03-06T07:25:36.965583+0000","last_fullsized":"2024-03-06T07:25:36.965583+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963919+0000","last_change":"2024-03-06T07:25:36.963919+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963919+0000","last_undegraded":"2024-03-06T07:25:36.963919+0000","last_fullsized":"2024-03-06T07:25:36.963919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969843+0000","last_change":"2024-03-06T07:25:36.969843+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969843+0000","last_undegraded":"2024-03-06T07:25:36.969843+0000","last_fullsized":"2024-03-06T07:25:36.969843+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962327+0000","last_change":"2024-03-06T07:25:36.962327+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962327+0000","last_undegraded":"2024-03-06T07:25:36.962327+0000","last_fullsized":"2024-03-06T07:25:36.962327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962584+0000","last_change":"2024-03-06T07:25:36.962584+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962584+0000","last_undegraded":"2024-03-06T07:25:36.962584+0000","last_fullsized":"2024-03-06T07:25:36.962584+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964171+0000","last_change":"2024-03-06T07:25:36.964171+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964171+0000","last_undegraded":"2024-03-06T07:25:36.964171+0000","last_fullsized":"2024-03-06T07:25:36.964171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963613+0000","last_change":"2024-03-06T07:25:36.963613+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963613+0000","last_undegraded":"2024-03-06T07:25:36.963613+0000","last_fullsized":"2024-03-06T07:25:36.963613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975492+0000","last_change":"2024-03-06T07:25:36.974330+0000","last_active":"2024-03-06T07:25:36.975492+0000","last_peered":"2024-03-06T07:25:36.975492+0000","last_clean":"2024-03-06T07:25:36.975492+0000","last_became_active":"2024-03-06T07:25:36.974050+0000","last_became_peered":"2024-03-06T07:25:36.974050+0000","last_unstale":"2024-03-06T07:25:36.975492+0000","last_undegraded":"2024-03-06T07:25:36.975492+0000","last_fullsized":"2024-03-06T07:25:36.975492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:15.879147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967828+0000","last_change":"2024-03-06T07:25:36.967828+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967828+0000","last_undegraded":"2024-03-06T07:25:36.967828+0000","last_fullsized":"2024-03-06T07:25:36.967828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972393+0000","last_change":"2024-03-06T07:25:36.967151+0000","last_active":"2024-03-06T07:25:36.972393+0000","last_peered":"2024-03-06T07:25:36.972393+0000","last_clean":"2024-03-06T07:25:36.972393+0000","last_became_active":"2024-03-06T07:25:36.966864+0000","last_became_peered":"2024-03-06T07:25:36.966864+0000","last_unstale":"2024-03-06T07:25:36.972393+0000","last_undegraded":"2024-03-06T07:25:36.972393+0000","last_fullsized":"2024-03-06T07:25:36.972393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969345+0000","last_change":"2024-03-06T07:25:36.969345+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969345+0000","last_undegraded":"2024-03-06T07:25:36.969345+0000","last_fullsized":"2024-03-06T07:25:36.969345+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976130+0000","last_change":"2024-03-06T07:25:36.974099+0000","last_active":"2024-03-06T07:25:36.976130+0000","last_peered":"2024-03-06T07:25:36.976130+0000","last_clean":"2024-03-06T07:25:36.976130+0000","last_became_active":"2024-03-06T07:25:36.973818+0000","last_became_peered":"2024-03-06T07:25:36.973818+0000","last_unstale":"2024-03-06T07:25:36.976130+0000","last_undegraded":"2024-03-06T07:25:36.976130+0000","last_fullsized":"2024-03-06T07:25:36.976130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:36.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965086+0000","last_change":"2024-03-06T07:25:36.965086+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965086+0000","last_undegraded":"2024-03-06T07:25:36.965086+0000","last_fullsized":"2024-03-06T07:25:36.965086+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.978822+0000","last_change":"2024-03-06T07:25:36.974705+0000","last_active":"2024-03-06T07:25:36.978822+0000","last_peered":"2024-03-06T07:25:36.978822+0000","last_clean":"2024-03-06T07:25:36.978822+0000","last_became_active":"2024-03-06T07:25:36.974418+0000","last_became_peered":"2024-03-06T07:25:36.974418+0000","last_unstale":"2024-03-06T07:25:36.978822+0000","last_undegraded":"2024-03-06T07:25:36.978822+0000","last_fullsized":"2024-03-06T07:25:36.978822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962201+0000","last_change":"2024-03-06T07:25:36.962201+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962201+0000","last_undegraded":"2024-03-06T07:25:36.962201+0000","last_fullsized":"2024-03-06T07:25:36.962201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972867+0000","last_change":"2024-03-06T07:25:36.970497+0000","last_active":"2024-03-06T07:25:36.972867+0000","last_peered":"2024-03-06T07:25:36.972867+0000","last_clean":"2024-03-06T07:25:36.972867+0000","last_became_active":"2024-03-06T07:25:36.970229+0000","last_became_peered":"2024-03-06T07:25:36.970229+0000","last_unstale":"2024-03-06T07:25:36.972867+0000","last_undegraded":"2024-03-06T07:25:36.972867+0000","last_fullsized":"2024-03-06T07:25:36.972867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962580+0000","last_change":"2024-03-06T07:25:36.962580+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962580+0000","last_undegraded":"2024-03-06T07:25:36.962580+0000","last_fullsized":"2024-03-06T07:25:36.962580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972874+0000","last_change":"2024-03-06T07:25:36.972014+0000","last_active":"2024-03-06T07:25:36.972874+0000","last_peered":"2024-03-06T07:25:36.972874+0000","last_clean":"2024-03-06T07:25:36.972874+0000","last_became_active":"2024-03-06T07:25:36.971692+0000","last_became_peered":"2024-03-06T07:25:36.971692+0000","last_unstale":"2024-03-06T07:25:36.972874+0000","last_undegraded":"2024-03-06T07:25:36.972874+0000","last_fullsized":"2024-03-06T07:25:36.972874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.918383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965503+0000","last_change":"2024-03-06T07:25:36.965503+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965503+0000","last_undegraded":"2024-03-06T07:25:36.965503+0000","last_fullsized":"2024-03-06T07:25:36.965503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974849+0000","last_change":"2024-03-06T07:25:36.969239+0000","last_active":"2024-03-06T07:25:36.974849+0000","last_peered":"2024-03-06T07:25:36.974849+0000","last_clean":"2024-03-06T07:25:36.974849+0000","last_became_active":"2024-03-06T07:25:36.968491+0000","last_became_peered":"2024-03-06T07:25:36.968491+0000","last_unstale":"2024-03-06T07:25:36.974849+0000","last_undegraded":"2024-03-06T07:25:36.974849+0000","last_fullsized":"2024-03-06T07:25:36.974849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957846+0000","last_change":"2024-03-06T07:25:36.957846+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957846+0000","last_undegraded":"2024-03-06T07:25:36.957846+0000","last_fullsized":"2024-03-06T07:25:36.957846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975832+0000","last_change":"2024-03-06T07:25:36.974784+0000","last_active":"2024-03-06T07:25:36.975832+0000","last_peered":"2024-03-06T07:25:36.975832+0000","last_clean":"2024-03-06T07:25:36.975832+0000","last_became_active":"2024-03-06T07:25:36.974534+0000","last_became_peered":"2024-03-06T07:25:36.974534+0000","last_unstale":"2024-03-06T07:25:36.975832+0000","last_undegraded":"2024-03-06T07:25:36.975832+0000","last_fullsized":"2024-03-06T07:25:36.975832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:35.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958526+0000","last_change":"2024-03-06T07:25:36.958526+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958526+0000","last_undegraded":"2024-03-06T07:25:36.958526+0000","last_fullsized":"2024-03-06T07:25:36.958526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976718+0000","last_change":"2024-03-06T07:25:36.975629+0000","last_active":"2024-03-06T07:25:36.976718+0000","last_peered":"2024-03-06T07:25:36.976718+0000","last_clean":"2024-03-06T07:25:36.976718+0000","last_became_active":"2024-03-06T07:25:36.975298+0000","last_became_peered":"2024-03-06T07:25:36.975298+0000","last_unstale":"2024-03-06T07:25:36.976718+0000","last_undegraded":"2024-03-06T07:25:36.976718+0000","last_fullsized":"2024-03-06T07:25:36.976718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957680+0000","last_change":"2024-03-06T07:25:36.957680+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957680+0000","last_undegraded":"2024-03-06T07:25:36.957680+0000","last_fullsized":"2024-03-06T07:25:36.957680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976133+0000","last_change":"2024-03-06T07:25:36.974955+0000","last_active":"2024-03-06T07:25:36.976133+0000","last_peered":"2024-03-06T07:25:36.976133+0000","last_clean":"2024-03-06T07:25:36.976133+0000","last_became_active":"2024-03-06T07:25:36.974657+0000","last_became_peered":"2024-03-06T07:25:36.974657+0000","last_unstale":"2024-03-06T07:25:36.976133+0000","last_undegraded":"2024-03-06T07:25:36.976133+0000","last_fullsized":"2024-03-06T07:25:36.976133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:33.339070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966598+0000","last_change":"2024-03-06T07:25:36.966598+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966598+0000","last_undegraded":"2024-03-06T07:25:36.966598+0000","last_fullsized":"2024-03-06T07:25:36.966598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975518+0000","last_change":"2024-03-06T07:25:36.974695+0000","last_active":"2024-03-06T07:25:36.975518+0000","last_peered":"2024-03-06T07:25:36.975518+0000","last_clean":"2024-03-06T07:25:36.975518+0000","last_became_active":"2024-03-06T07:25:36.974384+0000","last_became_peered":"2024-03-06T07:25:36.974384+0000","last_unstale":"2024-03-06T07:25:36.975518+0000","last_undegraded":"2024-03-06T07:25:36.975518+0000","last_fullsized":"2024-03-06T07:25:36.975518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:25.464178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963479+0000","last_change":"2024-03-06T07:25:36.963479+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963479+0000","last_undegraded":"2024-03-06T07:25:36.963479+0000","last_fullsized":"2024-03-06T07:25:36.963479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.965356+0000","last_change":"2024-03-06T07:25:36.964206+0000","last_active":"2024-03-06T07:25:36.965356+0000","last_peered":"2024-03-06T07:25:36.965356+0000","last_clean":"2024-03-06T07:25:36.965356+0000","last_became_active":"2024-03-06T07:25:36.963893+0000","last_became_peered":"2024-03-06T07:25:36.963893+0000","last_unstale":"2024-03-06T07:25:36.965356+0000","last_undegraded":"2024-03-06T07:25:36.965356+0000","last_fullsized":"2024-03-06T07:25:36.965356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966036+0000","last_change":"2024-03-06T07:25:36.966036+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966036+0000","last_undegraded":"2024-03-06T07:25:36.966036+0000","last_fullsized":"2024-03-06T07:25:36.966036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974005+0000","last_change":"2024-03-06T07:25:36.972698+0000","last_active":"2024-03-06T07:25:36.974005+0000","last_peered":"2024-03-06T07:25:36.974005+0000","last_clean":"2024-03-06T07:25:36.974005+0000","last_became_active":"2024-03-06T07:25:36.972395+0000","last_became_peered":"2024-03-06T07:25:36.972395+0000","last_unstale":"2024-03-06T07:25:36.974005+0000","last_undegraded":"2024-03-06T07:25:36.974005+0000","last_fullsized":"2024-03-06T07:25:36.974005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:50.352156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966888+0000","last_change":"2024-03-06T07:25:36.966888+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966888+0000","last_undegraded":"2024-03-06T07:25:36.966888+0000","last_fullsized":"2024-03-06T07:25:36.966888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973687+0000","last_change":"2024-03-06T07:25:36.968375+0000","last_active":"2024-03-06T07:25:36.973687+0000","last_peered":"2024-03-06T07:25:36.973687+0000","last_clean":"2024-03-06T07:25:36.973687+0000","last_became_active":"2024-03-06T07:25:36.967683+0000","last_became_peered":"2024-03-06T07:25:36.967683+0000","last_unstale":"2024-03-06T07:25:36.973687+0000","last_undegraded":"2024-03-06T07:25:36.973687+0000","last_fullsized":"2024-03-06T07:25:36.973687+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:10.418371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961504+0000","last_change":"2024-03-06T07:25:36.961504+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961504+0000","last_undegraded":"2024-03-06T07:25:36.961504+0000","last_fullsized":"2024-03-06T07:25:36.961504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972289+0000","last_change":"2024-03-06T07:25:36.971741+0000","last_active":"2024-03-06T07:25:36.972289+0000","last_peered":"2024-03-06T07:25:36.972289+0000","last_clean":"2024-03-06T07:25:36.972289+0000","last_became_active":"2024-03-06T07:25:36.970858+0000","last_became_peered":"2024-03-06T07:25:36.970858+0000","last_unstale":"2024-03-06T07:25:36.972289+0000","last_undegraded":"2024-03-06T07:25:36.972289+0000","last_fullsized":"2024-03-06T07:25:36.972289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961071+0000","last_change":"2024-03-06T07:25:36.961071+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961071+0000","last_undegraded":"2024-03-06T07:25:36.961071+0000","last_fullsized":"2024-03-06T07:25:36.961071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976083+0000","last_change":"2024-03-06T07:25:36.972032+0000","last_active":"2024-03-06T07:25:36.976083+0000","last_peered":"2024-03-06T07:25:36.976083+0000","last_clean":"2024-03-06T07:25:36.976083+0000","last_became_active":"2024-03-06T07:25:36.971777+0000","last_became_peered":"2024-03-06T07:25:36.971777+0000","last_unstale":"2024-03-06T07:25:36.976083+0000","last_undegraded":"2024-03-06T07:25:36.976083+0000","last_fullsized":"2024-03-06T07:25:36.976083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:32.818833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965095+0000","last_change":"2024-03-06T07:25:36.965095+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965095+0000","last_undegraded":"2024-03-06T07:25:36.965095+0000","last_fullsized":"2024-03-06T07:25:36.965095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974248+0000","last_change":"2024-03-06T07:25:36.972827+0000","last_active":"2024-03-06T07:25:36.974248+0000","last_peered":"2024-03-06T07:25:36.974248+0000","last_clean":"2024-03-06T07:25:36.974248+0000","last_became_active":"2024-03-06T07:25:36.972549+0000","last_became_peered":"2024-03-06T07:25:36.972549+0000","last_unstale":"2024-03-06T07:25:36.974248+0000","last_undegraded":"2024-03-06T07:25:36.974248+0000","last_fullsized":"2024-03-06T07:25:36.974248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:53.181415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967228+0000","last_change":"2024-03-06T07:25:36.967228+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967228+0000","last_undegraded":"2024-03-06T07:25:36.967228+0000","last_fullsized":"2024-03-06T07:25:36.967228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.978846+0000","last_change":"2024-03-06T07:25:36.974812+0000","last_active":"2024-03-06T07:25:36.978846+0000","last_peered":"2024-03-06T07:25:36.978846+0000","last_clean":"2024-03-06T07:25:36.978846+0000","last_became_active":"2024-03-06T07:25:36.974555+0000","last_became_peered":"2024-03-06T07:25:36.974555+0000","last_unstale":"2024-03-06T07:25:36.978846+0000","last_undegraded":"2024-03-06T07:25:36.978846+0000","last_fullsized":"2024-03-06T07:25:36.978846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:39.570670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962957+0000","last_change":"2024-03-06T07:25:36.962957+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962957+0000","last_undegraded":"2024-03-06T07:25:36.962957+0000","last_fullsized":"2024-03-06T07:25:36.962957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974884+0000","last_change":"2024-03-06T07:25:36.968979+0000","last_active":"2024-03-06T07:25:36.974884+0000","last_peered":"2024-03-06T07:25:36.974884+0000","last_clean":"2024-03-06T07:25:36.974884+0000","last_became_active":"2024-03-06T07:25:36.968727+0000","last_became_peered":"2024-03-06T07:25:36.968727+0000","last_unstale":"2024-03-06T07:25:36.974884+0000","last_undegraded":"2024-03-06T07:25:36.974884+0000","last_fullsized":"2024-03-06T07:25:36.974884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:18.410780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960312+0000","last_change":"2024-03-06T07:25:36.960312+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960312+0000","last_undegraded":"2024-03-06T07:25:36.960312+0000","last_fullsized":"2024-03-06T07:25:36.960312+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963596+0000","last_change":"2024-03-06T07:25:36.962047+0000","last_active":"2024-03-06T07:25:36.963596+0000","last_peered":"2024-03-06T07:25:36.963596+0000","last_clean":"2024-03-06T07:25:36.963596+0000","last_became_active":"2024-03-06T07:25:36.961774+0000","last_became_peered":"2024-03-06T07:25:36.961774+0000","last_unstale":"2024-03-06T07:25:36.963596+0000","last_undegraded":"2024-03-06T07:25:36.963596+0000","last_fullsized":"2024-03-06T07:25:36.963596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:24.570961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957718+0000","last_change":"2024-03-06T07:25:36.957718+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957718+0000","last_undegraded":"2024-03-06T07:25:36.957718+0000","last_fullsized":"2024-03-06T07:25:36.957718+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973502+0000","last_change":"2024-03-06T07:25:36.970500+0000","last_active":"2024-03-06T07:25:36.973502+0000","last_peered":"2024-03-06T07:25:36.973502+0000","last_clean":"2024-03-06T07:25:36.973502+0000","last_became_active":"2024-03-06T07:25:36.970286+0000","last_became_peered":"2024-03-06T07:25:36.970286+0000","last_unstale":"2024-03-06T07:25:36.973502+0000","last_undegraded":"2024-03-06T07:25:36.973502+0000","last_fullsized":"2024-03-06T07:25:36.973502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:03.015663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960298+0000","last_change":"2024-03-06T07:25:36.960298+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960298+0000","last_undegraded":"2024-03-06T07:25:36.960298+0000","last_fullsized":"2024-03-06T07:25:36.960298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974135+0000","last_change":"2024-03-06T07:25:36.971016+0000","last_active":"2024-03-06T07:25:36.974135+0000","last_peered":"2024-03-06T07:25:36.974135+0000","last_clean":"2024-03-06T07:25:36.974135+0000","last_became_active":"2024-03-06T07:25:36.970738+0000","last_became_peered":"2024-03-06T07:25:36.970738+0000","last_unstale":"2024-03-06T07:25:36.974135+0000","last_undegraded":"2024-03-06T07:25:36.974135+0000","last_fullsized":"2024-03-06T07:25:36.974135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:48.010645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958936+0000","last_change":"2024-03-06T07:25:36.958936+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958936+0000","last_undegraded":"2024-03-06T07:25:36.958936+0000","last_fullsized":"2024-03-06T07:25:36.958936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963460+0000","last_change":"2024-03-06T07:25:36.961782+0000","last_active":"2024-03-06T07:25:36.963460+0000","last_peered":"2024-03-06T07:25:36.963460+0000","last_clean":"2024-03-06T07:25:36.963460+0000","last_became_active":"2024-03-06T07:25:36.961534+0000","last_became_peered":"2024-03-06T07:25:36.961534+0000","last_unstale":"2024-03-06T07:25:36.963460+0000","last_undegraded":"2024-03-06T07:25:36.963460+0000","last_fullsized":"2024-03-06T07:25:36.963460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:17.024203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963202+0000","last_change":"2024-03-06T07:25:36.963202+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963202+0000","last_undegraded":"2024-03-06T07:25:36.963202+0000","last_fullsized":"2024-03-06T07:25:36.963202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970981+0000","last_change":"2024-03-06T07:25:36.968364+0000","last_active":"2024-03-06T07:25:36.970981+0000","last_peered":"2024-03-06T07:25:36.970981+0000","last_clean":"2024-03-06T07:25:36.970981+0000","last_became_active":"2024-03-06T07:25:36.968108+0000","last_became_peered":"2024-03-06T07:25:36.968108+0000","last_unstale":"2024-03-06T07:25:36.970981+0000","last_undegraded":"2024-03-06T07:25:36.970981+0000","last_fullsized":"2024-03-06T07:25:36.970981+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:35.303723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966589+0000","last_change":"2024-03-06T07:25:36.966589+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966589+0000","last_undegraded":"2024-03-06T07:25:36.966589+0000","last_fullsized":"2024-03-06T07:25:36.966589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974174+0000","last_change":"2024-03-06T07:25:36.972060+0000","last_active":"2024-03-06T07:25:36.974174+0000","last_peered":"2024-03-06T07:25:36.974174+0000","last_clean":"2024-03-06T07:25:36.974174+0000","last_became_active":"2024-03-06T07:25:36.971774+0000","last_became_peered":"2024-03-06T07:25:36.971774+0000","last_unstale":"2024-03-06T07:25:36.974174+0000","last_undegraded":"2024-03-06T07:25:36.974174+0000","last_fullsized":"2024-03-06T07:25:36.974174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:32.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968493+0000","last_change":"2024-03-06T07:25:36.968493+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968493+0000","last_undegraded":"2024-03-06T07:25:36.968493+0000","last_fullsized":"2024-03-06T07:25:36.968493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974580+0000","last_change":"2024-03-06T07:25:36.972458+0000","last_active":"2024-03-06T07:25:36.974580+0000","last_peered":"2024-03-06T07:25:36.974580+0000","last_clean":"2024-03-06T07:25:36.974580+0000","last_became_active":"2024-03-06T07:25:36.971925+0000","last_became_peered":"2024-03-06T07:25:36.971925+0000","last_unstale":"2024-03-06T07:25:36.974580+0000","last_undegraded":"2024-03-06T07:25:36.974580+0000","last_fullsized":"2024-03-06T07:25:36.974580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:13.397656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962197+0000","last_change":"2024-03-06T07:25:36.962197+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962197+0000","last_undegraded":"2024-03-06T07:25:36.962197+0000","last_fullsized":"2024-03-06T07:25:36.962197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977043+0000","last_change":"2024-03-06T07:25:36.976309+0000","last_active":"2024-03-06T07:25:36.977043+0000","last_peered":"2024-03-06T07:25:36.977043+0000","last_clean":"2024-03-06T07:25:36.977043+0000","last_became_active":"2024-03-06T07:25:36.976019+0000","last_became_peered":"2024-03-06T07:25:36.976019+0000","last_unstale":"2024-03-06T07:25:36.977043+0000","last_undegraded":"2024-03-06T07:25:36.977043+0000","last_fullsized":"2024-03-06T07:25:36.977043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957878+0000","last_change":"2024-03-06T07:25:36.957878+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957878+0000","last_undegraded":"2024-03-06T07:25:36.957878+0000","last_fullsized":"2024-03-06T07:25:36.957878+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976198+0000","last_change":"2024-03-06T07:25:36.971247+0000","last_active":"2024-03-06T07:25:36.976198+0000","last_peered":"2024-03-06T07:25:36.976198+0000","last_clean":"2024-03-06T07:25:36.976198+0000","last_became_active":"2024-03-06T07:25:36.970950+0000","last_became_peered":"2024-03-06T07:25:36.970950+0000","last_unstale":"2024-03-06T07:25:36.976198+0000","last_undegraded":"2024-03-06T07:25:36.976198+0000","last_fullsized":"2024-03-06T07:25:36.976198+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964978+0000","last_change":"2024-03-06T07:25:36.964978+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964978+0000","last_undegraded":"2024-03-06T07:25:36.964978+0000","last_fullsized":"2024-03-06T07:25:36.964978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970386+0000","last_change":"2024-03-06T07:25:36.967074+0000","last_active":"2024-03-06T07:25:36.970386+0000","last_peered":"2024-03-06T07:25:36.970386+0000","last_clean":"2024-03-06T07:25:36.970386+0000","last_became_active":"2024-03-06T07:25:36.966752+0000","last_became_peered":"2024-03-06T07:25:36.966752+0000","last_unstale":"2024-03-06T07:25:36.970386+0000","last_undegraded":"2024-03-06T07:25:36.970386+0000","last_fullsized":"2024-03-06T07:25:36.970386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965024+0000","last_change":"2024-03-06T07:25:36.965024+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965024+0000","last_undegraded":"2024-03-06T07:25:36.965024+0000","last_fullsized":"2024-03-06T07:25:36.965024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977030+0000","last_change":"2024-03-06T07:25:36.975964+0000","last_active":"2024-03-06T07:25:36.977030+0000","last_peered":"2024-03-06T07:25:36.977030+0000","last_clean":"2024-03-06T07:25:36.977030+0000","last_became_active":"2024-03-06T07:25:36.975665+0000","last_became_peered":"2024-03-06T07:25:36.975665+0000","last_unstale":"2024-03-06T07:25:36.977030+0000","last_undegraded":"2024-03-06T07:25:36.977030+0000","last_fullsized":"2024-03-06T07:25:36.977030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963945+0000","last_change":"2024-03-06T07:25:36.963945+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963945+0000","last_undegraded":"2024-03-06T07:25:36.963945+0000","last_fullsized":"2024-03-06T07:25:36.963945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976161+0000","last_change":"2024-03-06T07:25:36.974079+0000","last_active":"2024-03-06T07:25:36.976161+0000","last_peered":"2024-03-06T07:25:36.976161+0000","last_clean":"2024-03-06T07:25:36.976161+0000","last_became_active":"2024-03-06T07:25:36.973733+0000","last_became_peered":"2024-03-06T07:25:36.973733+0000","last_unstale":"2024-03-06T07:25:36.976161+0000","last_undegraded":"2024-03-06T07:25:36.976161+0000","last_fullsized":"2024-03-06T07:25:36.976161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:05.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962367+0000","last_change":"2024-03-06T07:25:36.962367+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962367+0000","last_undegraded":"2024-03-06T07:25:36.962367+0000","last_fullsized":"2024-03-06T07:25:36.962367+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975679+0000","last_change":"2024-03-06T07:25:36.974734+0000","last_active":"2024-03-06T07:25:36.975679+0000","last_peered":"2024-03-06T07:25:36.975679+0000","last_clean":"2024-03-06T07:25:36.975679+0000","last_became_active":"2024-03-06T07:25:36.974426+0000","last_became_peered":"2024-03-06T07:25:36.974426+0000","last_unstale":"2024-03-06T07:25:36.975679+0000","last_undegraded":"2024-03-06T07:25:36.975679+0000","last_fullsized":"2024-03-06T07:25:36.975679+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.967622+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:25:36.967622+0000","last_peered":"2024-03-06T07:25:36.967622+0000","last_clean":"2024-03-06T07:25:36.967622+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:25:36.967622+0000","last_undegraded":"2024-03-06T07:25:36.967622+0000","last_fullsized":"2024-03-06T07:25:36.967622+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:41.309 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:41.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 Processor -- start 2024-03-06T07:25:41.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 -- start start 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f731815dfe0 0x7f73180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f73180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73180b3430 con 0x7f73180d27f0 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73180d2c50 con 0x7f731815dfe0 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.445+0000 7f731e955640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73180b31e0 con 0x7f73180cf830 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f731cef2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f73180d27f0 0x7f73180d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38892/0 (socket says 172.21.15.72:38892) 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f731cef2640 1 -- 172.21.15.72:0/4239100074 learned_addr learned my addr 172.21.15.72:0/4239100074 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341974442 0 0) 0x7f73180b3430 con 0x7f73180d27f0 2024-03-06T07:25:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7300003520 con 0x7f73180d27f0 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7317fff640 1 --2- 172.21.15.72:0/4239100074 >> v2:172.21.15.146:3300/0 conn(0x7f731815dfe0 0x7f73180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f73177fe640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f73180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3684089459 0 0) 0x7f7300003520 con 0x7f73180d27f0 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 msgr2=0x7f73180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f73180d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 >> v2:172.21.15.146:3300/0 conn(0x7f731815dfe0 msgr2=0x7f73180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 --2- 172.21.15.72:0/4239100074 >> v2:172.21.15.146:3300/0 conn(0x7f731815dfe0 0x7f73180cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73180b3710 con 0x7f73180d27f0 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f73100019f0 con 0x7f73180d27f0 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 >> v1:172.21.15.72:6789/0 conn(0x7f73180d27f0 legacy=0x7f73180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000039b0 0x7f7300003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f7300004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7300004d40 0x7f73000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73180b31e0 con 0x7f73000039b0 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730000b9c0 con 0x7f7300004d40 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730000bb40 con 0x7f73180cf830 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f7317fff640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000039b0 0x7f7300003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.446+0000 7f73177fe640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f7300004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f731cef2640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7300004d40 0x7f73000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f7317fff640 1 -- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 msgr2=0x7f7300004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f7317fff640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f7300004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f7317fff640 1 -- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7300004d40 msgr2=0x7f73000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f7317fff640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7300004d40 0x7f73000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f7317fff640 1 -- 172.21.15.72:0/4239100074 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7300003520 con 0x7f73000039b0 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f731cef2640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7300004d40 0x7f73000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f73177fe640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180cf830 0x7f7300004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.447+0000 7f7317fff640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000039b0 0x7f7300003dd0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7308009ac0 tx=0x7f7300005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cf058267abb50c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308039070 con 0x7f73000039b0 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730802c710 con 0x7f73000039b0 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f7316ffd640 1 -- 172.21.15.72:0/4239100074 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7308009690 con 0x7f73000039b0 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731e955640 1 -- 172.21.15.72:0/4239100074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000039b0 msgr2=0x7f7300003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731e955640 1 --2- 172.21.15.72:0/4239100074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73000039b0 0x7f7300003dd0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7308009ac0 tx=0x7f7300005830 comp rx=0 tx=0).stop 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731cef2640 1 -- 172.21.15.72:0/4239100074 reap_dead start 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731e955640 1 -- 172.21.15.72:0/4239100074 shutdown_connections 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731e955640 1 -- 172.21.15.72:0/4239100074 >> 172.21.15.72:0/4239100074 conn(0x7f73180d3cd0 msgr2=0x7f73180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731e955640 1 -- 172.21.15.72:0/4239100074 shutdown_connections 2024-03-06T07:25:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.448+0000 7f731e955640 1 -- 172.21.15.72:0/4239100074 wait complete. 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.450+0000 7f731e955640 1 Processor -- start 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.450+0000 7f731e955640 1 -- start start 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.451+0000 7f731e955640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180b88e0 0x7f73180cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 0x7f73180cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731815be80 0x7f731815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73180d1c70 con 0x7f73180b8cc0 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73180b3d60 con 0x7f731815be80 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73180d1df0 con 0x7f73180b88e0 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 0x7f73180cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 0x7f73180cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47404/0 (socket says 172.21.15.72:47404) 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 -- 172.21.15.72:0/4219683464 learned_addr learned my addr 172.21.15.72:0/4219683464 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:41.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f7317fff640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180b88e0 0x7f73180cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731cef2640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731815be80 0x7f731815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 -- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180b88e0 msgr2=0x7f73180cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180b88e0 0x7f73180cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 -- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731815be80 msgr2=0x7f731815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731815be80 0x7f731815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 -- 172.21.15.72:0/4219683464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731815c820 con 0x7f73180b8cc0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f73177fe640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 0x7f73180cd610 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f730c011610 tx=0x7f730c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00f8f0 con 0x7f73180b8cc0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00fa90 con 0x7f73180b8cc0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731809eb80 con 0x7f73180b8cc0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00fe40 con 0x7f73180b8cc0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.452+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731809f140 con 0x7f73180b8cc0 2024-03-06T07:25:41.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.453+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73180cf5f0 con 0x7f73180b8cc0 2024-03-06T07:25:41.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.454+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f730c015460 con 0x7f73180b8cc0 2024-03-06T07:25:41.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.454+0000 7f7314ff9640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730406b4d0 0x7f730406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.454+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f730c08ab70 con 0x7f73180b8cc0 2024-03-06T07:25:41.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.454+0000 7f7317fff640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730406b4d0 0x7f730406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.455+0000 7f7317fff640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730406b4d0 0x7f730406d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7308036000 tx=0x7f7308030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.459+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f73180cf5f0 con 0x7f73180b8cc0 2024-03-06T07:25:41.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.765+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73180671f0 con 0x7f730406b4d0 2024-03-06T07:25:41.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.771+0000 7f7314ff9640 1 -- 172.21.15.72:0/4219683464 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246752 (secure 0 0 0) 0x7f73180671f0 con 0x7f730406b4d0 2024-03-06T07:25:41.775 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:41.777 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:25:41.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.773+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730406b4d0 msgr2=0x7f730406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.773+0000 7f731e955640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730406b4d0 0x7f730406d990 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7308036000 tx=0x7f7308030000 comp rx=0 tx=0).stop 2024-03-06T07:25:41.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 msgr2=0x7f73180cd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 0x7f73180cd610 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f730c011610 tx=0x7f730c011ad0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 shutdown_connections 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f730406b4d0 0x7f730406d990 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f731815be80 0x7f731815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f73180b8cc0 0x7f73180cd610 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 --2- 172.21.15.72:0/4219683464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73180b88e0 0x7f73180cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 >> 172.21.15.72:0/4219683464 conn(0x7f73180d3cd0 msgr2=0x7f73180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 shutdown_connections 2024-03-06T07:25:41.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.774+0000 7f731e955640 1 -- 172.21.15.72:0/4219683464 wait complete. 2024-03-06T07:25:41.787 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-03-06T07:25:39.989500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258284,"kb_used_data":8840,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716436,"statfs":{"total":773094113280,"available":772829630464,"internally_reserved":0,"allocated":9052160,"data_stored":4483952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039599"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968257+0000","last_change":"2024-03-06T07:25:36.968257+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968257+0000","last_undegraded":"2024-03-06T07:25:36.968257+0000","last_fullsized":"2024-03-06T07:25:36.968257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962841+0000","last_change":"2024-03-06T07:25:36.962841+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962841+0000","last_undegraded":"2024-03-06T07:25:36.962841+0000","last_fullsized":"2024-03-06T07:25:36.962841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965522+0000","last_change":"2024-03-06T07:25:36.965522+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965522+0000","last_undegraded":"2024-03-06T07:25:36.965522+0000","last_fullsized":"2024-03-06T07:25:36.965522+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961706+0000","last_change":"2024-03-06T07:25:36.961706+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961706+0000","last_undegraded":"2024-03-06T07:25:36.961706+0000","last_fullsized":"2024-03-06T07:25:36.961706+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958594+0000","last_change":"2024-03-06T07:25:36.958594+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958594+0000","last_undegraded":"2024-03-06T07:25:36.958594+0000","last_fullsized":"2024-03-06T07:25:36.958594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961812+0000","last_change":"2024-03-06T07:25:36.961812+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961812+0000","last_undegraded":"2024-03-06T07:25:36.961812+0000","last_fullsized":"2024-03-06T07:25:36.961812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958927+0000","last_change":"2024-03-06T07:25:36.958927+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958927+0000","last_undegraded":"2024-03-06T07:25:36.958927+0000","last_fullsized":"2024-03-06T07:25:36.958927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958269+0000","last_change":"2024-03-06T07:25:36.958269+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958269+0000","last_undegraded":"2024-03-06T07:25:36.958269+0000","last_fullsized":"2024-03-06T07:25:36.958269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963496+0000","last_change":"2024-03-06T07:25:36.963496+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963496+0000","last_undegraded":"2024-03-06T07:25:36.963496+0000","last_fullsized":"2024-03-06T07:25:36.963496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962145+0000","last_change":"2024-03-06T07:25:36.962145+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962145+0000","last_undegraded":"2024-03-06T07:25:36.962145+0000","last_fullsized":"2024-03-06T07:25:36.962145+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.970250+0000","last_change":"2024-03-06T07:25:36.970250+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.970250+0000","last_undegraded":"2024-03-06T07:25:36.970250+0000","last_fullsized":"2024-03-06T07:25:36.970250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967316+0000","last_change":"2024-03-06T07:25:36.967316+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967316+0000","last_undegraded":"2024-03-06T07:25:36.967316+0000","last_fullsized":"2024-03-06T07:25:36.967316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961585+0000","last_change":"2024-03-06T07:25:36.961585+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961585+0000","last_undegraded":"2024-03-06T07:25:36.961585+0000","last_fullsized":"2024-03-06T07:25:36.961585+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958329+0000","last_change":"2024-03-06T07:25:36.958329+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958329+0000","last_undegraded":"2024-03-06T07:25:36.958329+0000","last_fullsized":"2024-03-06T07:25:36.958329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962098+0000","last_change":"2024-03-06T07:25:36.962098+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962098+0000","last_undegraded":"2024-03-06T07:25:36.962098+0000","last_fullsized":"2024-03-06T07:25:36.962098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962240+0000","last_change":"2024-03-06T07:25:36.962240+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962240+0000","last_undegraded":"2024-03-06T07:25:36.962240+0000","last_fullsized":"2024-03-06T07:25:36.962240+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961534+0000","last_change":"2024-03-06T07:25:36.961534+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961534+0000","last_undegraded":"2024-03-06T07:25:36.961534+0000","last_fullsized":"2024-03-06T07:25:36.961534+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966351+0000","last_change":"2024-03-06T07:25:36.966351+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966351+0000","last_undegraded":"2024-03-06T07:25:36.966351+0000","last_fullsized":"2024-03-06T07:25:36.966351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961153+0000","last_change":"2024-03-06T07:25:36.961153+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961153+0000","last_undegraded":"2024-03-06T07:25:36.961153+0000","last_fullsized":"2024-03-06T07:25:36.961153+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960766+0000","last_change":"2024-03-06T07:25:36.960766+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960766+0000","last_undegraded":"2024-03-06T07:25:36.960766+0000","last_fullsized":"2024-03-06T07:25:36.960766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962960+0000","last_change":"2024-03-06T07:25:36.962960+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962960+0000","last_undegraded":"2024-03-06T07:25:36.962960+0000","last_fullsized":"2024-03-06T07:25:36.962960+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964375+0000","last_change":"2024-03-06T07:25:36.964375+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964375+0000","last_undegraded":"2024-03-06T07:25:36.964375+0000","last_fullsized":"2024-03-06T07:25:36.964375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960738+0000","last_change":"2024-03-06T07:25:36.960738+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960738+0000","last_undegraded":"2024-03-06T07:25:36.960738+0000","last_fullsized":"2024-03-06T07:25:36.960738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962994+0000","last_change":"2024-03-06T07:25:36.962994+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962994+0000","last_undegraded":"2024-03-06T07:25:36.962994+0000","last_fullsized":"2024-03-06T07:25:36.962994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965583+0000","last_change":"2024-03-06T07:25:36.965583+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965583+0000","last_undegraded":"2024-03-06T07:25:36.965583+0000","last_fullsized":"2024-03-06T07:25:36.965583+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963919+0000","last_change":"2024-03-06T07:25:36.963919+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963919+0000","last_undegraded":"2024-03-06T07:25:36.963919+0000","last_fullsized":"2024-03-06T07:25:36.963919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969843+0000","last_change":"2024-03-06T07:25:36.969843+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969843+0000","last_undegraded":"2024-03-06T07:25:36.969843+0000","last_fullsized":"2024-03-06T07:25:36.969843+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962327+0000","last_change":"2024-03-06T07:25:36.962327+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962327+0000","last_undegraded":"2024-03-06T07:25:36.962327+0000","last_fullsized":"2024-03-06T07:25:36.962327+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962584+0000","last_change":"2024-03-06T07:25:36.962584+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962584+0000","last_undegraded":"2024-03-06T07:25:36.962584+0000","last_fullsized":"2024-03-06T07:25:36.962584+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964171+0000","last_change":"2024-03-06T07:25:36.964171+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964171+0000","last_undegraded":"2024-03-06T07:25:36.964171+0000","last_fullsized":"2024-03-06T07:25:36.964171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963613+0000","last_change":"2024-03-06T07:25:36.963613+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963613+0000","last_undegraded":"2024-03-06T07:25:36.963613+0000","last_fullsized":"2024-03-06T07:25:36.963613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975492+0000","last_change":"2024-03-06T07:25:36.974330+0000","last_active":"2024-03-06T07:25:36.975492+0000","last_peered":"2024-03-06T07:25:36.975492+0000","last_clean":"2024-03-06T07:25:36.975492+0000","last_became_active":"2024-03-06T07:25:36.974050+0000","last_became_peered":"2024-03-06T07:25:36.974050+0000","last_unstale":"2024-03-06T07:25:36.975492+0000","last_undegraded":"2024-03-06T07:25:36.975492+0000","last_fullsized":"2024-03-06T07:25:36.975492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:15.879147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967828+0000","last_change":"2024-03-06T07:25:36.967828+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967828+0000","last_undegraded":"2024-03-06T07:25:36.967828+0000","last_fullsized":"2024-03-06T07:25:36.967828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972393+0000","last_change":"2024-03-06T07:25:36.967151+0000","last_active":"2024-03-06T07:25:36.972393+0000","last_peered":"2024-03-06T07:25:36.972393+0000","last_clean":"2024-03-06T07:25:36.972393+0000","last_became_active":"2024-03-06T07:25:36.966864+0000","last_became_peered":"2024-03-06T07:25:36.966864+0000","last_unstale":"2024-03-06T07:25:36.972393+0000","last_undegraded":"2024-03-06T07:25:36.972393+0000","last_fullsized":"2024-03-06T07:25:36.972393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969345+0000","last_change":"2024-03-06T07:25:36.969345+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969345+0000","last_undegraded":"2024-03-06T07:25:36.969345+0000","last_fullsized":"2024-03-06T07:25:36.969345+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976130+0000","last_change":"2024-03-06T07:25:36.974099+0000","last_active":"2024-03-06T07:25:36.976130+0000","last_peered":"2024-03-06T07:25:36.976130+0000","last_clean":"2024-03-06T07:25:36.976130+0000","last_became_active":"2024-03-06T07:25:36.973818+0000","last_became_peered":"2024-03-06T07:25:36.973818+0000","last_unstale":"2024-03-06T07:25:36.976130+0000","last_undegraded":"2024-03-06T07:25:36.976130+0000","last_fullsized":"2024-03-06T07:25:36.976130+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:36.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965086+0000","last_change":"2024-03-06T07:25:36.965086+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965086+0000","last_undegraded":"2024-03-06T07:25:36.965086+0000","last_fullsized":"2024-03-06T07:25:36.965086+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.978822+0000","last_change":"2024-03-06T07:25:36.974705+0000","last_active":"2024-03-06T07:25:36.978822+0000","last_peered":"2024-03-06T07:25:36.978822+0000","last_clean":"2024-03-06T07:25:36.978822+0000","last_became_active":"2024-03-06T07:25:36.974418+0000","last_became_peered":"2024-03-06T07:25:36.974418+0000","last_unstale":"2024-03-06T07:25:36.978822+0000","last_undegraded":"2024-03-06T07:25:36.978822+0000","last_fullsized":"2024-03-06T07:25:36.978822+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962201+0000","last_change":"2024-03-06T07:25:36.962201+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962201+0000","last_undegraded":"2024-03-06T07:25:36.962201+0000","last_fullsized":"2024-03-06T07:25:36.962201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972867+0000","last_change":"2024-03-06T07:25:36.970497+0000","last_active":"2024-03-06T07:25:36.972867+0000","last_peered":"2024-03-06T07:25:36.972867+0000","last_clean":"2024-03-06T07:25:36.972867+0000","last_became_active":"2024-03-06T07:25:36.970229+0000","last_became_peered":"2024-03-06T07:25:36.970229+0000","last_unstale":"2024-03-06T07:25:36.972867+0000","last_undegraded":"2024-03-06T07:25:36.972867+0000","last_fullsized":"2024-03-06T07:25:36.972867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962580+0000","last_change":"2024-03-06T07:25:36.962580+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962580+0000","last_undegraded":"2024-03-06T07:25:36.962580+0000","last_fullsized":"2024-03-06T07:25:36.962580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972874+0000","last_change":"2024-03-06T07:25:36.972014+0000","last_active":"2024-03-06T07:25:36.972874+0000","last_peered":"2024-03-06T07:25:36.972874+0000","last_clean":"2024-03-06T07:25:36.972874+0000","last_became_active":"2024-03-06T07:25:36.971692+0000","last_became_peered":"2024-03-06T07:25:36.971692+0000","last_unstale":"2024-03-06T07:25:36.972874+0000","last_undegraded":"2024-03-06T07:25:36.972874+0000","last_fullsized":"2024-03-06T07:25:36.972874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.918383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965503+0000","last_change":"2024-03-06T07:25:36.965503+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965503+0000","last_undegraded":"2024-03-06T07:25:36.965503+0000","last_fullsized":"2024-03-06T07:25:36.965503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974849+0000","last_change":"2024-03-06T07:25:36.969239+0000","last_active":"2024-03-06T07:25:36.974849+0000","last_peered":"2024-03-06T07:25:36.974849+0000","last_clean":"2024-03-06T07:25:36.974849+0000","last_became_active":"2024-03-06T07:25:36.968491+0000","last_became_peered":"2024-03-06T07:25:36.968491+0000","last_unstale":"2024-03-06T07:25:36.974849+0000","last_undegraded":"2024-03-06T07:25:36.974849+0000","last_fullsized":"2024-03-06T07:25:36.974849+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957846+0000","last_change":"2024-03-06T07:25:36.957846+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957846+0000","last_undegraded":"2024-03-06T07:25:36.957846+0000","last_fullsized":"2024-03-06T07:25:36.957846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975832+0000","last_change":"2024-03-06T07:25:36.974784+0000","last_active":"2024-03-06T07:25:36.975832+0000","last_peered":"2024-03-06T07:25:36.975832+0000","last_clean":"2024-03-06T07:25:36.975832+0000","last_became_active":"2024-03-06T07:25:36.974534+0000","last_became_peered":"2024-03-06T07:25:36.974534+0000","last_unstale":"2024-03-06T07:25:36.975832+0000","last_undegraded":"2024-03-06T07:25:36.975832+0000","last_fullsized":"2024-03-06T07:25:36.975832+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:35.262096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958526+0000","last_change":"2024-03-06T07:25:36.958526+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958526+0000","last_undegraded":"2024-03-06T07:25:36.958526+0000","last_fullsized":"2024-03-06T07:25:36.958526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976718+0000","last_change":"2024-03-06T07:25:36.975629+0000","last_active":"2024-03-06T07:25:36.976718+0000","last_peered":"2024-03-06T07:25:36.976718+0000","last_clean":"2024-03-06T07:25:36.976718+0000","last_became_active":"2024-03-06T07:25:36.975298+0000","last_became_peered":"2024-03-06T07:25:36.975298+0000","last_unstale":"2024-03-06T07:25:36.976718+0000","last_undegraded":"2024-03-06T07:25:36.976718+0000","last_fullsized":"2024-03-06T07:25:36.976718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957680+0000","last_change":"2024-03-06T07:25:36.957680+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957680+0000","last_undegraded":"2024-03-06T07:25:36.957680+0000","last_fullsized":"2024-03-06T07:25:36.957680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976133+0000","last_change":"2024-03-06T07:25:36.974955+0000","last_active":"2024-03-06T07:25:36.976133+0000","last_peered":"2024-03-06T07:25:36.976133+0000","last_clean":"2024-03-06T07:25:36.976133+0000","last_became_active":"2024-03-06T07:25:36.974657+0000","last_became_peered":"2024-03-06T07:25:36.974657+0000","last_unstale":"2024-03-06T07:25:36.976133+0000","last_undegraded":"2024-03-06T07:25:36.976133+0000","last_fullsized":"2024-03-06T07:25:36.976133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:33.339070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966598+0000","last_change":"2024-03-06T07:25:36.966598+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966598+0000","last_undegraded":"2024-03-06T07:25:36.966598+0000","last_fullsized":"2024-03-06T07:25:36.966598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975518+0000","last_change":"2024-03-06T07:25:36.974695+0000","last_active":"2024-03-06T07:25:36.975518+0000","last_peered":"2024-03-06T07:25:36.975518+0000","last_clean":"2024-03-06T07:25:36.975518+0000","last_became_active":"2024-03-06T07:25:36.974384+0000","last_became_peered":"2024-03-06T07:25:36.974384+0000","last_unstale":"2024-03-06T07:25:36.975518+0000","last_undegraded":"2024-03-06T07:25:36.975518+0000","last_fullsized":"2024-03-06T07:25:36.975518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:25.464178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963479+0000","last_change":"2024-03-06T07:25:36.963479+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963479+0000","last_undegraded":"2024-03-06T07:25:36.963479+0000","last_fullsized":"2024-03-06T07:25:36.963479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.965356+0000","last_change":"2024-03-06T07:25:36.964206+0000","last_active":"2024-03-06T07:25:36.965356+0000","last_peered":"2024-03-06T07:25:36.965356+0000","last_clean":"2024-03-06T07:25:36.965356+0000","last_became_active":"2024-03-06T07:25:36.963893+0000","last_became_peered":"2024-03-06T07:25:36.963893+0000","last_unstale":"2024-03-06T07:25:36.965356+0000","last_undegraded":"2024-03-06T07:25:36.965356+0000","last_fullsized":"2024-03-06T07:25:36.965356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966036+0000","last_change":"2024-03-06T07:25:36.966036+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966036+0000","last_undegraded":"2024-03-06T07:25:36.966036+0000","last_fullsized":"2024-03-06T07:25:36.966036+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974005+0000","last_change":"2024-03-06T07:25:36.972698+0000","last_active":"2024-03-06T07:25:36.974005+0000","last_peered":"2024-03-06T07:25:36.974005+0000","last_clean":"2024-03-06T07:25:36.974005+0000","last_became_active":"2024-03-06T07:25:36.972395+0000","last_became_peered":"2024-03-06T07:25:36.972395+0000","last_unstale":"2024-03-06T07:25:36.974005+0000","last_undegraded":"2024-03-06T07:25:36.974005+0000","last_fullsized":"2024-03-06T07:25:36.974005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:39:50.352156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966888+0000","last_change":"2024-03-06T07:25:36.966888+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966888+0000","last_undegraded":"2024-03-06T07:25:36.966888+0000","last_fullsized":"2024-03-06T07:25:36.966888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973687+0000","last_change":"2024-03-06T07:25:36.968375+0000","last_active":"2024-03-06T07:25:36.973687+0000","last_peered":"2024-03-06T07:25:36.973687+0000","last_clean":"2024-03-06T07:25:36.973687+0000","last_became_active":"2024-03-06T07:25:36.967683+0000","last_became_peered":"2024-03-06T07:25:36.967683+0000","last_unstale":"2024-03-06T07:25:36.973687+0000","last_undegraded":"2024-03-06T07:25:36.973687+0000","last_fullsized":"2024-03-06T07:25:36.973687+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:10.418371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961504+0000","last_change":"2024-03-06T07:25:36.961504+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961504+0000","last_undegraded":"2024-03-06T07:25:36.961504+0000","last_fullsized":"2024-03-06T07:25:36.961504+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972289+0000","last_change":"2024-03-06T07:25:36.971741+0000","last_active":"2024-03-06T07:25:36.972289+0000","last_peered":"2024-03-06T07:25:36.972289+0000","last_clean":"2024-03-06T07:25:36.972289+0000","last_became_active":"2024-03-06T07:25:36.970858+0000","last_became_peered":"2024-03-06T07:25:36.970858+0000","last_unstale":"2024-03-06T07:25:36.972289+0000","last_undegraded":"2024-03-06T07:25:36.972289+0000","last_fullsized":"2024-03-06T07:25:36.972289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.961071+0000","last_change":"2024-03-06T07:25:36.961071+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.961071+0000","last_undegraded":"2024-03-06T07:25:36.961071+0000","last_fullsized":"2024-03-06T07:25:36.961071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976083+0000","last_change":"2024-03-06T07:25:36.972032+0000","last_active":"2024-03-06T07:25:36.976083+0000","last_peered":"2024-03-06T07:25:36.976083+0000","last_clean":"2024-03-06T07:25:36.976083+0000","last_became_active":"2024-03-06T07:25:36.971777+0000","last_became_peered":"2024-03-06T07:25:36.971777+0000","last_unstale":"2024-03-06T07:25:36.976083+0000","last_undegraded":"2024-03-06T07:25:36.976083+0000","last_fullsized":"2024-03-06T07:25:36.976083+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:27:32.818833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965095+0000","last_change":"2024-03-06T07:25:36.965095+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965095+0000","last_undegraded":"2024-03-06T07:25:36.965095+0000","last_fullsized":"2024-03-06T07:25:36.965095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974248+0000","last_change":"2024-03-06T07:25:36.972827+0000","last_active":"2024-03-06T07:25:36.974248+0000","last_peered":"2024-03-06T07:25:36.974248+0000","last_clean":"2024-03-06T07:25:36.974248+0000","last_became_active":"2024-03-06T07:25:36.972549+0000","last_became_peered":"2024-03-06T07:25:36.972549+0000","last_unstale":"2024-03-06T07:25:36.974248+0000","last_undegraded":"2024-03-06T07:25:36.974248+0000","last_fullsized":"2024-03-06T07:25:36.974248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:53.181415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967228+0000","last_change":"2024-03-06T07:25:36.967228+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967228+0000","last_undegraded":"2024-03-06T07:25:36.967228+0000","last_fullsized":"2024-03-06T07:25:36.967228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.978846+0000","last_change":"2024-03-06T07:25:36.974812+0000","last_active":"2024-03-06T07:25:36.978846+0000","last_peered":"2024-03-06T07:25:36.978846+0000","last_clean":"2024-03-06T07:25:36.978846+0000","last_became_active":"2024-03-06T07:25:36.974555+0000","last_became_peered":"2024-03-06T07:25:36.974555+0000","last_unstale":"2024-03-06T07:25:36.978846+0000","last_undegraded":"2024-03-06T07:25:36.978846+0000","last_fullsized":"2024-03-06T07:25:36.978846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:39.570670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962957+0000","last_change":"2024-03-06T07:25:36.962957+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962957+0000","last_undegraded":"2024-03-06T07:25:36.962957+0000","last_fullsized":"2024-03-06T07:25:36.962957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974884+0000","last_change":"2024-03-06T07:25:36.968979+0000","last_active":"2024-03-06T07:25:36.974884+0000","last_peered":"2024-03-06T07:25:36.974884+0000","last_clean":"2024-03-06T07:25:36.974884+0000","last_became_active":"2024-03-06T07:25:36.968727+0000","last_became_peered":"2024-03-06T07:25:36.968727+0000","last_unstale":"2024-03-06T07:25:36.974884+0000","last_undegraded":"2024-03-06T07:25:36.974884+0000","last_fullsized":"2024-03-06T07:25:36.974884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:13:18.410780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960312+0000","last_change":"2024-03-06T07:25:36.960312+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960312+0000","last_undegraded":"2024-03-06T07:25:36.960312+0000","last_fullsized":"2024-03-06T07:25:36.960312+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963596+0000","last_change":"2024-03-06T07:25:36.962047+0000","last_active":"2024-03-06T07:25:36.963596+0000","last_peered":"2024-03-06T07:25:36.963596+0000","last_clean":"2024-03-06T07:25:36.963596+0000","last_became_active":"2024-03-06T07:25:36.961774+0000","last_became_peered":"2024-03-06T07:25:36.961774+0000","last_unstale":"2024-03-06T07:25:36.963596+0000","last_undegraded":"2024-03-06T07:25:36.963596+0000","last_fullsized":"2024-03-06T07:25:36.963596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:24.570961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957718+0000","last_change":"2024-03-06T07:25:36.957718+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957718+0000","last_undegraded":"2024-03-06T07:25:36.957718+0000","last_fullsized":"2024-03-06T07:25:36.957718+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973502+0000","last_change":"2024-03-06T07:25:36.970500+0000","last_active":"2024-03-06T07:25:36.973502+0000","last_peered":"2024-03-06T07:25:36.973502+0000","last_clean":"2024-03-06T07:25:36.973502+0000","last_became_active":"2024-03-06T07:25:36.970286+0000","last_became_peered":"2024-03-06T07:25:36.970286+0000","last_unstale":"2024-03-06T07:25:36.973502+0000","last_undegraded":"2024-03-06T07:25:36.973502+0000","last_fullsized":"2024-03-06T07:25:36.973502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:46:03.015663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.960298+0000","last_change":"2024-03-06T07:25:36.960298+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.960298+0000","last_undegraded":"2024-03-06T07:25:36.960298+0000","last_fullsized":"2024-03-06T07:25:36.960298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974135+0000","last_change":"2024-03-06T07:25:36.971016+0000","last_active":"2024-03-06T07:25:36.974135+0000","last_peered":"2024-03-06T07:25:36.974135+0000","last_clean":"2024-03-06T07:25:36.974135+0000","last_became_active":"2024-03-06T07:25:36.970738+0000","last_became_peered":"2024-03-06T07:25:36.970738+0000","last_unstale":"2024-03-06T07:25:36.974135+0000","last_undegraded":"2024-03-06T07:25:36.974135+0000","last_fullsized":"2024-03-06T07:25:36.974135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:08:48.010645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958936+0000","last_change":"2024-03-06T07:25:36.958936+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958936+0000","last_undegraded":"2024-03-06T07:25:36.958936+0000","last_fullsized":"2024-03-06T07:25:36.958936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963460+0000","last_change":"2024-03-06T07:25:36.961782+0000","last_active":"2024-03-06T07:25:36.963460+0000","last_peered":"2024-03-06T07:25:36.963460+0000","last_clean":"2024-03-06T07:25:36.963460+0000","last_became_active":"2024-03-06T07:25:36.961534+0000","last_became_peered":"2024-03-06T07:25:36.961534+0000","last_unstale":"2024-03-06T07:25:36.963460+0000","last_undegraded":"2024-03-06T07:25:36.963460+0000","last_fullsized":"2024-03-06T07:25:36.963460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:17.024203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963202+0000","last_change":"2024-03-06T07:25:36.963202+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963202+0000","last_undegraded":"2024-03-06T07:25:36.963202+0000","last_fullsized":"2024-03-06T07:25:36.963202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970981+0000","last_change":"2024-03-06T07:25:36.968364+0000","last_active":"2024-03-06T07:25:36.970981+0000","last_peered":"2024-03-06T07:25:36.970981+0000","last_clean":"2024-03-06T07:25:36.970981+0000","last_became_active":"2024-03-06T07:25:36.968108+0000","last_became_peered":"2024-03-06T07:25:36.968108+0000","last_unstale":"2024-03-06T07:25:36.970981+0000","last_undegraded":"2024-03-06T07:25:36.970981+0000","last_fullsized":"2024-03-06T07:25:36.970981+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:35.303723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966589+0000","last_change":"2024-03-06T07:25:36.966589+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966589+0000","last_undegraded":"2024-03-06T07:25:36.966589+0000","last_fullsized":"2024-03-06T07:25:36.966589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974174+0000","last_change":"2024-03-06T07:25:36.972060+0000","last_active":"2024-03-06T07:25:36.974174+0000","last_peered":"2024-03-06T07:25:36.974174+0000","last_clean":"2024-03-06T07:25:36.974174+0000","last_became_active":"2024-03-06T07:25:36.971774+0000","last_became_peered":"2024-03-06T07:25:36.971774+0000","last_unstale":"2024-03-06T07:25:36.974174+0000","last_undegraded":"2024-03-06T07:25:36.974174+0000","last_fullsized":"2024-03-06T07:25:36.974174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:32.683690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968493+0000","last_change":"2024-03-06T07:25:36.968493+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968493+0000","last_undegraded":"2024-03-06T07:25:36.968493+0000","last_fullsized":"2024-03-06T07:25:36.968493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974580+0000","last_change":"2024-03-06T07:25:36.972458+0000","last_active":"2024-03-06T07:25:36.974580+0000","last_peered":"2024-03-06T07:25:36.974580+0000","last_clean":"2024-03-06T07:25:36.974580+0000","last_became_active":"2024-03-06T07:25:36.971925+0000","last_became_peered":"2024-03-06T07:25:36.971925+0000","last_unstale":"2024-03-06T07:25:36.974580+0000","last_undegraded":"2024-03-06T07:25:36.974580+0000","last_fullsized":"2024-03-06T07:25:36.974580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:13.397656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962197+0000","last_change":"2024-03-06T07:25:36.962197+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962197+0000","last_undegraded":"2024-03-06T07:25:36.962197+0000","last_fullsized":"2024-03-06T07:25:36.962197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977043+0000","last_change":"2024-03-06T07:25:36.976309+0000","last_active":"2024-03-06T07:25:36.977043+0000","last_peered":"2024-03-06T07:25:36.977043+0000","last_clean":"2024-03-06T07:25:36.977043+0000","last_became_active":"2024-03-06T07:25:36.976019+0000","last_became_peered":"2024-03-06T07:25:36.976019+0000","last_unstale":"2024-03-06T07:25:36.977043+0000","last_undegraded":"2024-03-06T07:25:36.977043+0000","last_fullsized":"2024-03-06T07:25:36.977043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957878+0000","last_change":"2024-03-06T07:25:36.957878+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957878+0000","last_undegraded":"2024-03-06T07:25:36.957878+0000","last_fullsized":"2024-03-06T07:25:36.957878+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976198+0000","last_change":"2024-03-06T07:25:36.971247+0000","last_active":"2024-03-06T07:25:36.976198+0000","last_peered":"2024-03-06T07:25:36.976198+0000","last_clean":"2024-03-06T07:25:36.976198+0000","last_became_active":"2024-03-06T07:25:36.970950+0000","last_became_peered":"2024-03-06T07:25:36.970950+0000","last_unstale":"2024-03-06T07:25:36.976198+0000","last_undegraded":"2024-03-06T07:25:36.976198+0000","last_fullsized":"2024-03-06T07:25:36.976198+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964978+0000","last_change":"2024-03-06T07:25:36.964978+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964978+0000","last_undegraded":"2024-03-06T07:25:36.964978+0000","last_fullsized":"2024-03-06T07:25:36.964978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970386+0000","last_change":"2024-03-06T07:25:36.967074+0000","last_active":"2024-03-06T07:25:36.970386+0000","last_peered":"2024-03-06T07:25:36.970386+0000","last_clean":"2024-03-06T07:25:36.970386+0000","last_became_active":"2024-03-06T07:25:36.966752+0000","last_became_peered":"2024-03-06T07:25:36.966752+0000","last_unstale":"2024-03-06T07:25:36.970386+0000","last_undegraded":"2024-03-06T07:25:36.970386+0000","last_fullsized":"2024-03-06T07:25:36.970386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965024+0000","last_change":"2024-03-06T07:25:36.965024+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965024+0000","last_undegraded":"2024-03-06T07:25:36.965024+0000","last_fullsized":"2024-03-06T07:25:36.965024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977030+0000","last_change":"2024-03-06T07:25:36.975964+0000","last_active":"2024-03-06T07:25:36.977030+0000","last_peered":"2024-03-06T07:25:36.977030+0000","last_clean":"2024-03-06T07:25:36.977030+0000","last_became_active":"2024-03-06T07:25:36.975665+0000","last_became_peered":"2024-03-06T07:25:36.975665+0000","last_unstale":"2024-03-06T07:25:36.977030+0000","last_undegraded":"2024-03-06T07:25:36.977030+0000","last_fullsized":"2024-03-06T07:25:36.977030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963945+0000","last_change":"2024-03-06T07:25:36.963945+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963945+0000","last_undegraded":"2024-03-06T07:25:36.963945+0000","last_fullsized":"2024-03-06T07:25:36.963945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976161+0000","last_change":"2024-03-06T07:25:36.974079+0000","last_active":"2024-03-06T07:25:36.976161+0000","last_peered":"2024-03-06T07:25:36.976161+0000","last_clean":"2024-03-06T07:25:36.976161+0000","last_became_active":"2024-03-06T07:25:36.973733+0000","last_became_peered":"2024-03-06T07:25:36.973733+0000","last_unstale":"2024-03-06T07:25:36.976161+0000","last_undegraded":"2024-03-06T07:25:36.976161+0000","last_fullsized":"2024-03-06T07:25:36.976161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:05.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962367+0000","last_change":"2024-03-06T07:25:36.962367+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962367+0000","last_undegraded":"2024-03-06T07:25:36.962367+0000","last_fullsized":"2024-03-06T07:25:36.962367+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975679+0000","last_change":"2024-03-06T07:25:36.974734+0000","last_active":"2024-03-06T07:25:36.975679+0000","last_peered":"2024-03-06T07:25:36.975679+0000","last_clean":"2024-03-06T07:25:36.975679+0000","last_became_active":"2024-03-06T07:25:36.974426+0000","last_became_peered":"2024-03-06T07:25:36.974426+0000","last_unstale":"2024-03-06T07:25:36.975679+0000","last_undegraded":"2024-03-06T07:25:36.975679+0000","last_fullsized":"2024-03-06T07:25:36.975679+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.967622+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:25:36.967622+0000","last_peered":"2024-03-06T07:25:36.967622+0000","last_clean":"2024-03-06T07:25:36.967622+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:25:36.967622+0000","last_undegraded":"2024-03-06T07:25:36.967622+0000","last_fullsized":"2024-03-06T07:25:36.967622+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574899,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3f is not active+clean 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.968257+0000', 'last_change': '2024-03-06T07:25:36.968257+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.968257+0000', 'last_undegraded': '2024-03-06T07:25:36.968257+0000', 'last_fullsized': '2024-03-06T07:25:36.968257+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3e is not active+clean 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962841+0000', 'last_change': '2024-03-06T07:25:36.962841+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962841+0000', 'last_undegraded': '2024-03-06T07:25:36.962841+0000', 'last_fullsized': '2024-03-06T07:25:36.962841+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3d is not active+clean 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.965522+0000', 'last_change': '2024-03-06T07:25:36.965522+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.965522+0000', 'last_undegraded': '2024-03-06T07:25:36.965522+0000', 'last_fullsized': '2024-03-06T07:25:36.965522+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3c is not active+clean 2024-03-06T07:25:41.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961706+0000', 'last_change': '2024-03-06T07:25:36.961706+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961706+0000', 'last_undegraded': '2024-03-06T07:25:36.961706+0000', 'last_fullsized': '2024-03-06T07:25:36.961706+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3b is not active+clean 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963000+0000', 'last_change': '2024-03-06T07:25:36.963000+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963000+0000', 'last_undegraded': '2024-03-06T07:25:36.963000+0000', 'last_fullsized': '2024-03-06T07:25:36.963000+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3a is not active+clean 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.958594+0000', 'last_change': '2024-03-06T07:25:36.958594+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.958594+0000', 'last_undegraded': '2024-03-06T07:25:36.958594+0000', 'last_fullsized': '2024-03-06T07:25:36.958594+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.39 is not active+clean 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961812+0000', 'last_change': '2024-03-06T07:25:36.961812+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961812+0000', 'last_undegraded': '2024-03-06T07:25:36.961812+0000', 'last_fullsized': '2024-03-06T07:25:36.961812+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.38 is not active+clean 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.958927+0000', 'last_change': '2024-03-06T07:25:36.958927+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.958927+0000', 'last_undegraded': '2024-03-06T07:25:36.958927+0000', 'last_fullsized': '2024-03-06T07:25:36.958927+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.37 is not active+clean 2024-03-06T07:25:41.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.958269+0000', 'last_change': '2024-03-06T07:25:36.958269+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.958269+0000', 'last_undegraded': '2024-03-06T07:25:36.958269+0000', 'last_fullsized': '2024-03-06T07:25:36.958269+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.36 is not active+clean 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963496+0000', 'last_change': '2024-03-06T07:25:36.963496+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963496+0000', 'last_undegraded': '2024-03-06T07:25:36.963496+0000', 'last_fullsized': '2024-03-06T07:25:36.963496+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.35 is not active+clean 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962145+0000', 'last_change': '2024-03-06T07:25:36.962145+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962145+0000', 'last_undegraded': '2024-03-06T07:25:36.962145+0000', 'last_fullsized': '2024-03-06T07:25:36.962145+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.34 is not active+clean 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.970250+0000', 'last_change': '2024-03-06T07:25:36.970250+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.970250+0000', 'last_undegraded': '2024-03-06T07:25:36.970250+0000', 'last_fullsized': '2024-03-06T07:25:36.970250+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.33 is not active+clean 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.967316+0000', 'last_change': '2024-03-06T07:25:36.967316+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.967316+0000', 'last_undegraded': '2024-03-06T07:25:36.967316+0000', 'last_fullsized': '2024-03-06T07:25:36.967316+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.32 is not active+clean 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961585+0000', 'last_change': '2024-03-06T07:25:36.961585+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961585+0000', 'last_undegraded': '2024-03-06T07:25:36.961585+0000', 'last_fullsized': '2024-03-06T07:25:36.961585+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.31 is not active+clean 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.958329+0000', 'last_change': '2024-03-06T07:25:36.958329+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.958329+0000', 'last_undegraded': '2024-03-06T07:25:36.958329+0000', 'last_fullsized': '2024-03-06T07:25:36.958329+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.30 is not active+clean 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962098+0000', 'last_change': '2024-03-06T07:25:36.962098+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962098+0000', 'last_undegraded': '2024-03-06T07:25:36.962098+0000', 'last_fullsized': '2024-03-06T07:25:36.962098+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2f is not active+clean 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962240+0000', 'last_change': '2024-03-06T07:25:36.962240+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962240+0000', 'last_undegraded': '2024-03-06T07:25:36.962240+0000', 'last_fullsized': '2024-03-06T07:25:36.962240+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2e is not active+clean 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961534+0000', 'last_change': '2024-03-06T07:25:36.961534+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961534+0000', 'last_undegraded': '2024-03-06T07:25:36.961534+0000', 'last_fullsized': '2024-03-06T07:25:36.961534+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2d is not active+clean 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.966351+0000', 'last_change': '2024-03-06T07:25:36.966351+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.966351+0000', 'last_undegraded': '2024-03-06T07:25:36.966351+0000', 'last_fullsized': '2024-03-06T07:25:36.966351+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2c is not active+clean 2024-03-06T07:25:41.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961153+0000', 'last_change': '2024-03-06T07:25:36.961153+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961153+0000', 'last_undegraded': '2024-03-06T07:25:36.961153+0000', 'last_fullsized': '2024-03-06T07:25:36.961153+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2b is not active+clean 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.960766+0000', 'last_change': '2024-03-06T07:25:36.960766+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.960766+0000', 'last_undegraded': '2024-03-06T07:25:36.960766+0000', 'last_fullsized': '2024-03-06T07:25:36.960766+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2a is not active+clean 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962960+0000', 'last_change': '2024-03-06T07:25:36.962960+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962960+0000', 'last_undegraded': '2024-03-06T07:25:36.962960+0000', 'last_fullsized': '2024-03-06T07:25:36.962960+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.29 is not active+clean 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.964375+0000', 'last_change': '2024-03-06T07:25:36.964375+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.964375+0000', 'last_undegraded': '2024-03-06T07:25:36.964375+0000', 'last_fullsized': '2024-03-06T07:25:36.964375+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.28 is not active+clean 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.960738+0000', 'last_change': '2024-03-06T07:25:36.960738+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.960738+0000', 'last_undegraded': '2024-03-06T07:25:36.960738+0000', 'last_fullsized': '2024-03-06T07:25:36.960738+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.27 is not active+clean 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962994+0000', 'last_change': '2024-03-06T07:25:36.962994+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962994+0000', 'last_undegraded': '2024-03-06T07:25:36.962994+0000', 'last_fullsized': '2024-03-06T07:25:36.962994+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.26 is not active+clean 2024-03-06T07:25:41.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.965583+0000', 'last_change': '2024-03-06T07:25:36.965583+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.965583+0000', 'last_undegraded': '2024-03-06T07:25:36.965583+0000', 'last_fullsized': '2024-03-06T07:25:36.965583+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.25 is not active+clean 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963919+0000', 'last_change': '2024-03-06T07:25:36.963919+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963919+0000', 'last_undegraded': '2024-03-06T07:25:36.963919+0000', 'last_fullsized': '2024-03-06T07:25:36.963919+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.24 is not active+clean 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.969843+0000', 'last_change': '2024-03-06T07:25:36.969843+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.969843+0000', 'last_undegraded': '2024-03-06T07:25:36.969843+0000', 'last_fullsized': '2024-03-06T07:25:36.969843+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.23 is not active+clean 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962327+0000', 'last_change': '2024-03-06T07:25:36.962327+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962327+0000', 'last_undegraded': '2024-03-06T07:25:36.962327+0000', 'last_fullsized': '2024-03-06T07:25:36.962327+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.22 is not active+clean 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962584+0000', 'last_change': '2024-03-06T07:25:36.962584+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962584+0000', 'last_undegraded': '2024-03-06T07:25:36.962584+0000', 'last_fullsized': '2024-03-06T07:25:36.962584+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.21 is not active+clean 2024-03-06T07:25:41.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963000+0000', 'last_change': '2024-03-06T07:25:36.963000+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963000+0000', 'last_undegraded': '2024-03-06T07:25:36.963000+0000', 'last_fullsized': '2024-03-06T07:25:36.963000+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.20 is not active+clean 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.964171+0000', 'last_change': '2024-03-06T07:25:36.964171+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.964171+0000', 'last_undegraded': '2024-03-06T07:25:36.964171+0000', 'last_fullsized': '2024-03-06T07:25:36.964171+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963613+0000', 'last_change': '2024-03-06T07:25:36.963613+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963613+0000', 'last_undegraded': '2024-03-06T07:25:36.963613+0000', 'last_fullsized': '2024-03-06T07:25:36.963613+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.967828+0000', 'last_change': '2024-03-06T07:25:36.967828+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.967828+0000', 'last_undegraded': '2024-03-06T07:25:36.967828+0000', 'last_fullsized': '2024-03-06T07:25:36.967828+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.969345+0000', 'last_change': '2024-03-06T07:25:36.969345+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.969345+0000', 'last_undegraded': '2024-03-06T07:25:36.969345+0000', 'last_fullsized': '2024-03-06T07:25:36.969345+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.965086+0000', 'last_change': '2024-03-06T07:25:36.965086+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.965086+0000', 'last_undegraded': '2024-03-06T07:25:36.965086+0000', 'last_fullsized': '2024-03-06T07:25:36.965086+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-03-06T07:25:41.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962201+0000', 'last_change': '2024-03-06T07:25:36.962201+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962201+0000', 'last_undegraded': '2024-03-06T07:25:36.962201+0000', 'last_fullsized': '2024-03-06T07:25:36.962201+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962580+0000', 'last_change': '2024-03-06T07:25:36.962580+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962580+0000', 'last_undegraded': '2024-03-06T07:25:36.962580+0000', 'last_fullsized': '2024-03-06T07:25:36.962580+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.965503+0000', 'last_change': '2024-03-06T07:25:36.965503+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.965503+0000', 'last_undegraded': '2024-03-06T07:25:36.965503+0000', 'last_fullsized': '2024-03-06T07:25:36.965503+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.19 is not active+clean 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.957846+0000', 'last_change': '2024-03-06T07:25:36.957846+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.957846+0000', 'last_undegraded': '2024-03-06T07:25:36.957846+0000', 'last_fullsized': '2024-03-06T07:25:36.957846+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.16 is not active+clean 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.958526+0000', 'last_change': '2024-03-06T07:25:36.958526+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.958526+0000', 'last_undegraded': '2024-03-06T07:25:36.958526+0000', 'last_fullsized': '2024-03-06T07:25:36.958526+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.957680+0000', 'last_change': '2024-03-06T07:25:36.957680+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.957680+0000', 'last_undegraded': '2024-03-06T07:25:36.957680+0000', 'last_fullsized': '2024-03-06T07:25:36.957680+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.14 is not active+clean 2024-03-06T07:25:41.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.966598+0000', 'last_change': '2024-03-06T07:25:36.966598+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.966598+0000', 'last_undegraded': '2024-03-06T07:25:36.966598+0000', 'last_fullsized': '2024-03-06T07:25:36.966598+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963479+0000', 'last_change': '2024-03-06T07:25:36.963479+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963479+0000', 'last_undegraded': '2024-03-06T07:25:36.963479+0000', 'last_fullsized': '2024-03-06T07:25:36.963479+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.12 is not active+clean 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.966036+0000', 'last_change': '2024-03-06T07:25:36.966036+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.966036+0000', 'last_undegraded': '2024-03-06T07:25:36.966036+0000', 'last_fullsized': '2024-03-06T07:25:36.966036+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.966888+0000', 'last_change': '2024-03-06T07:25:36.966888+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.966888+0000', 'last_undegraded': '2024-03-06T07:25:36.966888+0000', 'last_fullsized': '2024-03-06T07:25:36.966888+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961504+0000', 'last_change': '2024-03-06T07:25:36.961504+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961504+0000', 'last_undegraded': '2024-03-06T07:25:36.961504+0000', 'last_fullsized': '2024-03-06T07:25:36.961504+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-03-06T07:25:41.799 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.961071+0000', 'last_change': '2024-03-06T07:25:36.961071+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.961071+0000', 'last_undegraded': '2024-03-06T07:25:36.961071+0000', 'last_fullsized': '2024-03-06T07:25:36.961071+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.800 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-03-06T07:25:41.800 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.965095+0000', 'last_change': '2024-03-06T07:25:36.965095+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.965095+0000', 'last_undegraded': '2024-03-06T07:25:36.965095+0000', 'last_fullsized': '2024-03-06T07:25:36.965095+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.800 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-03-06T07:25:41.800 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.967228+0000', 'last_change': '2024-03-06T07:25:36.967228+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.967228+0000', 'last_undegraded': '2024-03-06T07:25:36.967228+0000', 'last_fullsized': '2024-03-06T07:25:36.967228+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.801 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-03-06T07:25:41.801 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962957+0000', 'last_change': '2024-03-06T07:25:36.962957+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962957+0000', 'last_undegraded': '2024-03-06T07:25:36.962957+0000', 'last_fullsized': '2024-03-06T07:25:36.962957+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.801 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.a is not active+clean 2024-03-06T07:25:41.801 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.960312+0000', 'last_change': '2024-03-06T07:25:36.960312+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.960312+0000', 'last_undegraded': '2024-03-06T07:25:36.960312+0000', 'last_fullsized': '2024-03-06T07:25:36.960312+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.801 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-03-06T07:25:41.802 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.957718+0000', 'last_change': '2024-03-06T07:25:36.957718+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.957718+0000', 'last_undegraded': '2024-03-06T07:25:36.957718+0000', 'last_fullsized': '2024-03-06T07:25:36.957718+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.802 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.8 is not active+clean 2024-03-06T07:25:41.802 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.960298+0000', 'last_change': '2024-03-06T07:25:36.960298+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.960298+0000', 'last_undegraded': '2024-03-06T07:25:36.960298+0000', 'last_fullsized': '2024-03-06T07:25:36.960298+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:25:41.802 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-03-06T07:25:41.802 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.958936+0000', 'last_change': '2024-03-06T07:25:36.958936+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.958936+0000', 'last_undegraded': '2024-03-06T07:25:36.958936+0000', 'last_fullsized': '2024-03-06T07:25:36.958936+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.803 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-03-06T07:25:41.803 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963202+0000', 'last_change': '2024-03-06T07:25:36.963202+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963202+0000', 'last_undegraded': '2024-03-06T07:25:36.963202+0000', 'last_fullsized': '2024-03-06T07:25:36.963202+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.803 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-03-06T07:25:41.803 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.966589+0000', 'last_change': '2024-03-06T07:25:36.966589+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.966589+0000', 'last_undegraded': '2024-03-06T07:25:36.966589+0000', 'last_fullsized': '2024-03-06T07:25:36.966589+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.803 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-03-06T07:25:41.804 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.968493+0000', 'last_change': '2024-03-06T07:25:36.968493+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.968493+0000', 'last_undegraded': '2024-03-06T07:25:36.968493+0000', 'last_fullsized': '2024-03-06T07:25:36.968493+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.804 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-03-06T07:25:41.804 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962197+0000', 'last_change': '2024-03-06T07:25:36.962197+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962197+0000', 'last_undegraded': '2024-03-06T07:25:36.962197+0000', 'last_fullsized': '2024-03-06T07:25:36.962197+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:25:41.804 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-03-06T07:25:41.805 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.957878+0000', 'last_change': '2024-03-06T07:25:36.957878+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.957878+0000', 'last_undegraded': '2024-03-06T07:25:36.957878+0000', 'last_fullsized': '2024-03-06T07:25:36.957878+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:25:41.805 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3 is not active+clean 2024-03-06T07:25:41.805 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.964978+0000', 'last_change': '2024-03-06T07:25:36.964978+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.964978+0000', 'last_undegraded': '2024-03-06T07:25:36.964978+0000', 'last_fullsized': '2024-03-06T07:25:36.964978+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:25:41.805 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-03-06T07:25:41.805 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.965024+0000', 'last_change': '2024-03-06T07:25:36.965024+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.965024+0000', 'last_undegraded': '2024-03-06T07:25:36.965024+0000', 'last_fullsized': '2024-03-06T07:25:36.965024+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:25:41.805 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-03-06T07:25:41.806 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.963945+0000', 'last_change': '2024-03-06T07:25:36.963945+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.963945+0000', 'last_undegraded': '2024-03-06T07:25:36.963945+0000', 'last_fullsized': '2024-03-06T07:25:36.963945+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:25:41.806 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-03-06T07:25:41.806 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 81, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:25:36.962367+0000', 'last_change': '2024-03-06T07:25:36.962367+0000', 'last_active': '2024-03-06T07:25:36.949776+0000', 'last_peered': '2024-03-06T07:25:36.949776+0000', 'last_clean': '2024-03-06T07:25:36.949776+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:25:36.962367+0000', 'last_undegraded': '2024-03-06T07:25:36.962367+0000', 'last_fullsized': '2024-03-06T07:25:36.962367+0000', 'mapping_epoch': 81, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 81, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'last_clean_scrub_stamp': '2024-03-06T07:25:36.949776+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:25:41.806 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:25:41.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.925+0000 7fcb63f32640 1 Processor -- start 2024-03-06T07:25:41.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 -- start start 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb5c0b4460 0x7fcb5c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb5c161cf0 0x7fcb5c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c161b00 con 0x7fcb5c0b4460 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5c0cecc0 con 0x7fcb5c161cf0 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb63f32640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c0d22c0 con 0x7fcb5c162970 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb61cce640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb5c0b4460 0x7fcb5c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb61cce640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb5c0b4460 0x7fcb5c0d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47410/0 (socket says 172.21.15.72:47410) 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.926+0000 7fcb61cce640 1 -- 172.21.15.72:0/2203357831 learned_addr learned my addr 172.21.15.72:0/2203357831 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb624cf640 1 --2- 172.21.15.72:0/2203357831 >> v2:172.21.15.146:3300/0 conn(0x7fcb5c161cf0 0x7fcb5c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb614cd640 1 --2- 172.21.15.72:0/2203357831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb61cce640 1 -- 172.21.15.72:0/2203357831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 msgr2=0x7fcb5c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb61cce640 1 --2- 172.21.15.72:0/2203357831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb61cce640 1 -- 172.21.15.72:0/2203357831 >> v2:172.21.15.146:3300/0 conn(0x7fcb5c161cf0 msgr2=0x7fcb5c162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb61cce640 1 --2- 172.21.15.72:0/2203357831 >> v2:172.21.15.146:3300/0 conn(0x7fcb5c161cf0 0x7fcb5c162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.927+0000 7fcb61cce640 1 -- 172.21.15.72:0/2203357831 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c0d25a0 con 0x7fcb5c0b4460 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb61cce640 1 --2- 172.21.15.72:0/2203357831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0d4800 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0ce2c0 tx=0x7fcb5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=138e11d55a31bded server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb60ccc640 1 -- 172.21.15.72:0/2203357831 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58035020 con 0x7fcb5c0b4460 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb60ccc640 1 -- 172.21.15.72:0/2203357831 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb58031870 con 0x7fcb5c0b4460 2024-03-06T07:25:41.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb60ccc640 1 -- 172.21.15.72:0/2203357831 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58031c40 con 0x7fcb5c0b4460 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 -- 172.21.15.72:0/2203357831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 msgr2=0x7fcb5c0d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 --2- 172.21.15.72:0/2203357831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0d4800 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0ce2c0 tx=0x7fcb5802b910 comp rx=0 tx=0).stop 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 -- 172.21.15.72:0/2203357831 shutdown_connections 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 --2- 172.21.15.72:0/2203357831 >> v2:172.21.15.146:3300/0 conn(0x7fcb5c161cf0 0x7fcb5c162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 --2- 172.21.15.72:0/2203357831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 --2- 172.21.15.72:0/2203357831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0d4800 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 -- 172.21.15.72:0/2203357831 >> 172.21.15.72:0/2203357831 conn(0x7fcb5c0d3ce0 msgr2=0x7fcb5c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 -- 172.21.15.72:0/2203357831 shutdown_connections 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.928+0000 7fcb63f32640 1 -- 172.21.15.72:0/2203357831 wait complete. 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.929+0000 7fcb63f32640 1 Processor -- start 2024-03-06T07:25:41.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 -- start start 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5c161cf0 0x7fcb5c0b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c0ce950 con 0x7fcb5c0b4460 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c0d2b60 con 0x7fcb5c161cf0 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb63f32640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5c0b3ac0 con 0x7fcb5c162970 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb61cce640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb61cce640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0cb020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47420/0 (socket says 172.21.15.72:47420) 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb61cce640 1 -- 172.21.15.72:0/3252293800 learned_addr learned my addr 172.21.15.72:0/3252293800 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb614cd640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5c161cf0 0x7fcb5c0b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.930+0000 7fcb624cf640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb61cce640 1 -- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 msgr2=0x7fcb5c0b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb61cce640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb61cce640 1 -- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5c161cf0 msgr2=0x7fcb5c0b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:41.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb61cce640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5c161cf0 0x7fcb5c0b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb61cce640 1 -- 172.21.15.72:0/3252293800 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c1542e0 con 0x7fcb5c0b4460 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb624cf640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb61cce640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0cb020 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcb58000c00 tx=0x7fcb580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58035040 con 0x7fcb5c0b4460 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb58030d00 con 0x7fcb5c0b4460 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5c0b9000 con 0x7fcb5c0b4460 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb580456a0 con 0x7fcb5c0b4460 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.931+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5c151a80 con 0x7fcb5c0b4460 2024-03-06T07:25:41.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.932+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5c04f330 con 0x7fcb5c0b4460 2024-03-06T07:25:41.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.933+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb58038020 con 0x7fcb5c0b4460 2024-03-06T07:25:41.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.933+0000 7fcb527fc640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb4006b4d0 0x7fcb4006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:41.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.933+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb580ac3f0 con 0x7fcb5c0b4460 2024-03-06T07:25:41.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.933+0000 7fcb614cd640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb4006b4d0 0x7fcb4006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:41.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.934+0000 7fcb614cd640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb4006b4d0 0x7fcb4006d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c154a50 tx=0x7fcb4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:41.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:41.937+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb5807b300 con 0x7fcb5c0b4460 2024-03-06T07:25:42.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.186+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fcb5c06b180 con 0x7fcb5c0b4460 2024-03-06T07:25:42.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.187+0000 7fcb527fc640 1 -- 172.21.15.72:0/3252293800 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1453 (secure 0 0 0) 0x7fcb5807ac00 con 0x7fcb5c0b4460 2024-03-06T07:25:42.189 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:42.189 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":256,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":83,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709696,"num_in_osds":8,"osd_in_since":1709709687,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":264482816,"bytes_avail":772829630464,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":39,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"a","status":"up:active","gid":6987}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-03-06T07:23:11.962885+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:25:42.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb4006b4d0 msgr2=0x7fcb4006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb4006b4d0 0x7fcb4006d990 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c154a50 tx=0x7fcb4c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 msgr2=0x7fcb5c0cb020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0cb020 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fcb58000c00 tx=0x7fcb580046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 shutdown_connections 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb4006b4d0 0x7fcb4006d990 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5c162970 0x7fcb5c0b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb5c161cf0 0x7fcb5c0b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 --2- 172.21.15.72:0/3252293800 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5c0b4460 0x7fcb5c0cb020 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 >> 172.21.15.72:0/3252293800 conn(0x7fcb5c0d3ce0 msgr2=0x7fcb5c0b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 shutdown_connections 2024-03-06T07:25:42.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.189+0000 7fcb63f32640 1 -- 172.21.15.72:0/3252293800 wait complete. 2024-03-06T07:25:42.203 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:25:42.203 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:42.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.302+0000 7f75d0671640 1 Processor -- start 2024-03-06T07:25:42.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.302+0000 7f75d0671640 1 -- start start 2024-03-06T07:25:42.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75d0671640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80d22b0 0x7f75c80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75d0671640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75c80b41f0 0x7f75c80b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75d0671640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c80b3430 con 0x7f75c815dfe0 2024-03-06T07:25:42.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75d0671640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c80d2c50 con 0x7f75c80b41f0 2024-03-06T07:25:42.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75d0671640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80b31e0 con 0x7f75c80d22b0 2024-03-06T07:25:42.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75cec0e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75c80b41f0 0x7f75c80b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75cdc0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80d22b0 0x7f75c80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75ce40d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f75c815dfe0 0x7f75c80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:46652/0 (socket says 172.21.15.72:46652) 2024-03-06T07:25:42.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.303+0000 7f75ce40d640 1 -- 172.21.15.72:0/133184508 learned_addr learned my addr 172.21.15.72:0/133184508 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.304+0000 7f75cd40b640 1 -- 172.21.15.72:0/133184508 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016660431 0 0) 0x7f75c80b3430 con 0x7f75c815dfe0 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.304+0000 7f75cd40b640 1 -- 172.21.15.72:0/133184508 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b0003520 con 0x7f75c815dfe0 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.304+0000 7f75cdc0c640 1 -- 172.21.15.72:0/133184508 >> v2:172.21.15.146:3300/0 conn(0x7f75c80b41f0 msgr2=0x7f75c80b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.304+0000 7f75cdc0c640 1 --2- 172.21.15.72:0/133184508 >> v2:172.21.15.146:3300/0 conn(0x7f75c80b41f0 0x7f75c80b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.304+0000 7f75cdc0c640 1 -- 172.21.15.72:0/133184508 >> v1:172.21.15.146:6789/0 conn(0x7f75c815dfe0 legacy=0x7f75c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.304+0000 7f75cdc0c640 1 -- 172.21.15.72:0/133184508 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80b3710 con 0x7f75c80d22b0 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.305+0000 7f75cdc0c640 1 --2- 172.21.15.72:0/133184508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80d22b0 0x7f75c80d2710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0097a0 tx=0x7f75bc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a2c71830eef97d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.305+0000 7f75cd40b640 1 -- 172.21.15.72:0/133184508 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc035020 con 0x7f75c80d22b0 2024-03-06T07:25:42.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.305+0000 7f75cd40b640 1 -- 172.21.15.72:0/133184508 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc031830 con 0x7f75c80d22b0 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.305+0000 7f75cd40b640 1 -- 172.21.15.72:0/133184508 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc031be0 con 0x7f75c80d22b0 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.305+0000 7f75d0671640 1 -- 172.21.15.72:0/133184508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80d22b0 msgr2=0x7f75c80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.305+0000 7f75d0671640 1 --2- 172.21.15.72:0/133184508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80d22b0 0x7f75c80d2710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0097a0 tx=0x7f75bc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.306+0000 7f75d0671640 1 -- 172.21.15.72:0/133184508 shutdown_connections 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.306+0000 7f75d0671640 1 --2- 172.21.15.72:0/133184508 >> v2:172.21.15.146:3300/0 conn(0x7f75c80b41f0 0x7f75c80b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.306+0000 7f75d0671640 1 --2- 172.21.15.72:0/133184508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80d22b0 0x7f75c80d2710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.306+0000 7f75d0671640 1 -- 172.21.15.72:0/133184508 >> 172.21.15.72:0/133184508 conn(0x7f75c80d3cd0 msgr2=0x7f75c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.306+0000 7f75d0671640 1 -- 172.21.15.72:0/133184508 shutdown_connections 2024-03-06T07:25:42.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.306+0000 7f75d0671640 1 -- 172.21.15.72:0/133184508 wait complete. 2024-03-06T07:25:42.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 Processor -- start 2024-03-06T07:25:42.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 -- start start 2024-03-06T07:25:42.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80b41f0 0x7f75c81f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 0x7f75c814f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c815dfe0 0x7f75c8158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80cfa00 con 0x7f75c80d22b0 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80b3d60 con 0x7f75c815dfe0 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.307+0000 7f75d0671640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80d1c70 con 0x7f75c80b41f0 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 0x7f75c814f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 0x7f75c814f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47434/0 (socket says 172.21.15.72:47434) 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 -- 172.21.15.72:0/1122433364 learned_addr learned my addr 172.21.15.72:0/1122433364 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75ce40d640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80b41f0 0x7f75c81f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cec0e640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c815dfe0 0x7f75c8158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 -- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80b41f0 msgr2=0x7f75c81f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80b41f0 0x7f75c81f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 -- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c815dfe0 msgr2=0x7f75c8158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c815dfe0 0x7f75c8158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 -- 172.21.15.72:0/1122433364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c814f660 con 0x7f75c80d22b0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75ce40d640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80b41f0 0x7f75c81f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75cdc0c640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 0x7f75c814f120 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f75bc02bee0 tx=0x7f75bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc035040 con 0x7f75c80d22b0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc03f070 con 0x7f75c80d22b0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc03bb40 con 0x7f75c80d22b0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c80b9a40 con 0x7f75c80d22b0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.308+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c8160100 con 0x7f75c80d22b0 2024-03-06T07:25:42.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.309+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c804f330 con 0x7f75c80d22b0 2024-03-06T07:25:42.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.310+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f75bc038020 con 0x7f75c80d22b0 2024-03-06T07:25:42.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.310+0000 7f75c6ffd640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:42.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.310+0000 7f75ce40d640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:42.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.310+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75bc0abbf0 con 0x7f75c80d22b0 2024-03-06T07:25:42.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.311+0000 7f75ce40d640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f75b406b4d0 0x7f75b406d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005c00 tx=0x7f75c0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:42.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.314+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f75c814f660 con 0x7f75c80d22b0 2024-03-06T07:25:42.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.488+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75c815d4e0 con 0x7f75b406b4d0 2024-03-06T07:25:42.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.496+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1122433364 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+248684 (secure 0 0 0) 0x7f75c815d4e0 con 0x7f75b406b4d0 2024-03-06T07:25:42.501 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:42.502 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:25:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f75b406b4d0 msgr2=0x7f75b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f75b406b4d0 0x7f75b406d990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f75c0005c00 tx=0x7f75c0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:42.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 msgr2=0x7f75c814f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 0x7f75c814f120 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f75bc02bee0 tx=0x7f75bc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 shutdown_connections 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f75b406b4d0 0x7f75b406d990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75c815dfe0 0x7f75c8158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75c80d22b0 0x7f75c814f120 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 --2- 172.21.15.72:0/1122433364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75c80b41f0 0x7f75c81f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 >> 172.21.15.72:0/1122433364 conn(0x7f75c80d3cd0 msgr2=0x7f75c806bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 shutdown_connections 2024-03-06T07:25:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:42.499+0000 7f75d0671640 1 -- 172.21.15.72:0/1122433364 wait complete. 2024-03-06T07:25:42.512 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-03-06T07:25:41.989771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590402,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":187,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258412,"kb_used_data":8968,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716308,"statfs":{"total":773094113280,"available":772829499392,"internally_reserved":0,"allocated":9183232,"data_stored":4542812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000859"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968257+0000","last_change":"2024-03-06T07:25:36.968257+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968257+0000","last_undegraded":"2024-03-06T07:25:36.968257+0000","last_fullsized":"2024-03-06T07:25:36.968257+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011873+0000","last_change":"2024-03-06T07:25:37.972532+0000","last_active":"2024-03-06T07:25:39.011873+0000","last_peered":"2024-03-06T07:25:39.011873+0000","last_clean":"2024-03-06T07:25:39.011873+0000","last_became_active":"2024-03-06T07:25:37.972199+0000","last_became_peered":"2024-03-06T07:25:37.972199+0000","last_unstale":"2024-03-06T07:25:39.011873+0000","last_undegraded":"2024-03-06T07:25:39.011873+0000","last_fullsized":"2024-03-06T07:25:39.011873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:05.865034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965522+0000","last_change":"2024-03-06T07:25:36.965522+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965522+0000","last_undegraded":"2024-03-06T07:25:36.965522+0000","last_fullsized":"2024-03-06T07:25:36.965522+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999038+0000","last_change":"2024-03-06T07:25:37.980903+0000","last_active":"2024-03-06T07:25:39.999038+0000","last_peered":"2024-03-06T07:25:39.999038+0000","last_clean":"2024-03-06T07:25:39.999038+0000","last_became_active":"2024-03-06T07:25:37.978491+0000","last_became_peered":"2024-03-06T07:25:37.978491+0000","last_unstale":"2024-03-06T07:25:39.999038+0000","last_undegraded":"2024-03-06T07:25:39.999038+0000","last_fullsized":"2024-03-06T07:25:39.999038+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:15.381106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958594+0000","last_change":"2024-03-06T07:25:36.958594+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958594+0000","last_undegraded":"2024-03-06T07:25:36.958594+0000","last_fullsized":"2024-03-06T07:25:36.958594+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998682+0000","last_change":"2024-03-06T07:25:37.972040+0000","last_active":"2024-03-06T07:25:39.998682+0000","last_peered":"2024-03-06T07:25:39.998682+0000","last_clean":"2024-03-06T07:25:39.998682+0000","last_became_active":"2024-03-06T07:25:37.971785+0000","last_became_peered":"2024-03-06T07:25:37.971785+0000","last_unstale":"2024-03-06T07:25:39.998682+0000","last_undegraded":"2024-03-06T07:25:39.998682+0000","last_fullsized":"2024-03-06T07:25:39.998682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:25.839525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958927+0000","last_change":"2024-03-06T07:25:36.958927+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958927+0000","last_undegraded":"2024-03-06T07:25:36.958927+0000","last_fullsized":"2024-03-06T07:25:36.958927+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958269+0000","last_change":"2024-03-06T07:25:36.958269+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958269+0000","last_undegraded":"2024-03-06T07:25:36.958269+0000","last_fullsized":"2024-03-06T07:25:36.958269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963496+0000","last_change":"2024-03-06T07:25:36.963496+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963496+0000","last_undegraded":"2024-03-06T07:25:36.963496+0000","last_fullsized":"2024-03-06T07:25:36.963496+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011965+0000","last_change":"2024-03-06T07:25:37.972340+0000","last_active":"2024-03-06T07:25:39.011965+0000","last_peered":"2024-03-06T07:25:39.011965+0000","last_clean":"2024-03-06T07:25:39.011965+0000","last_became_active":"2024-03-06T07:25:37.971978+0000","last_became_peered":"2024-03-06T07:25:37.971978+0000","last_unstale":"2024-03-06T07:25:39.011965+0000","last_undegraded":"2024-03-06T07:25:39.011965+0000","last_fullsized":"2024-03-06T07:25:39.011965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:32.870432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.970250+0000","last_change":"2024-03-06T07:25:36.970250+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.970250+0000","last_undegraded":"2024-03-06T07:25:36.970250+0000","last_fullsized":"2024-03-06T07:25:36.970250+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967316+0000","last_change":"2024-03-06T07:25:36.967316+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967316+0000","last_undegraded":"2024-03-06T07:25:36.967316+0000","last_fullsized":"2024-03-06T07:25:36.967316+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998882+0000","last_change":"2024-03-06T07:25:37.980695+0000","last_active":"2024-03-06T07:25:39.998882+0000","last_peered":"2024-03-06T07:25:39.998882+0000","last_clean":"2024-03-06T07:25:39.998882+0000","last_became_active":"2024-03-06T07:25:37.980394+0000","last_became_peered":"2024-03-06T07:25:37.980394+0000","last_unstale":"2024-03-06T07:25:39.998882+0000","last_undegraded":"2024-03-06T07:25:39.998882+0000","last_fullsized":"2024-03-06T07:25:39.998882+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:18.338847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958329+0000","last_change":"2024-03-06T07:25:36.958329+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958329+0000","last_undegraded":"2024-03-06T07:25:36.958329+0000","last_fullsized":"2024-03-06T07:25:36.958329+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962098+0000","last_change":"2024-03-06T07:25:36.962098+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962098+0000","last_undegraded":"2024-03-06T07:25:36.962098+0000","last_fullsized":"2024-03-06T07:25:36.962098+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962240+0000","last_change":"2024-03-06T07:25:36.962240+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962240+0000","last_undegraded":"2024-03-06T07:25:36.962240+0000","last_fullsized":"2024-03-06T07:25:36.962240+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998532+0000","last_change":"2024-03-06T07:25:37.980627+0000","last_active":"2024-03-06T07:25:39.998532+0000","last_peered":"2024-03-06T07:25:39.998532+0000","last_clean":"2024-03-06T07:25:39.998532+0000","last_became_active":"2024-03-06T07:25:37.980380+0000","last_became_peered":"2024-03-06T07:25:37.980380+0000","last_unstale":"2024-03-06T07:25:39.998532+0000","last_undegraded":"2024-03-06T07:25:39.998532+0000","last_fullsized":"2024-03-06T07:25:39.998532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:10.907996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975080+0000","last_change":"2024-03-06T07:25:37.975647+0000","last_active":"2024-03-06T07:25:38.975080+0000","last_peered":"2024-03-06T07:25:38.975080+0000","last_clean":"2024-03-06T07:25:38.975080+0000","last_became_active":"2024-03-06T07:25:37.975400+0000","last_became_peered":"2024-03-06T07:25:37.975400+0000","last_unstale":"2024-03-06T07:25:38.975080+0000","last_undegraded":"2024-03-06T07:25:38.975080+0000","last_fullsized":"2024-03-06T07:25:38.975080+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:00.380038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999087+0000","last_change":"2024-03-06T07:25:37.978431+0000","last_active":"2024-03-06T07:25:39.999087+0000","last_peered":"2024-03-06T07:25:39.999087+0000","last_clean":"2024-03-06T07:25:39.999087+0000","last_became_active":"2024-03-06T07:25:37.977962+0000","last_became_peered":"2024-03-06T07:25:37.977962+0000","last_unstale":"2024-03-06T07:25:39.999087+0000","last_undegraded":"2024-03-06T07:25:39.999087+0000","last_fullsized":"2024-03-06T07:25:39.999087+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:56.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999148+0000","last_change":"2024-03-06T07:25:37.977273+0000","last_active":"2024-03-06T07:25:39.999148+0000","last_peered":"2024-03-06T07:25:39.999148+0000","last_clean":"2024-03-06T07:25:39.999148+0000","last_became_active":"2024-03-06T07:25:37.976985+0000","last_became_peered":"2024-03-06T07:25:37.976985+0000","last_unstale":"2024-03-06T07:25:39.999148+0000","last_undegraded":"2024-03-06T07:25:39.999148+0000","last_fullsized":"2024-03-06T07:25:39.999148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:23.495818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011600+0000","last_change":"2024-03-06T07:25:37.973996+0000","last_active":"2024-03-06T07:25:39.011600+0000","last_peered":"2024-03-06T07:25:39.011600+0000","last_clean":"2024-03-06T07:25:39.011600+0000","last_became_active":"2024-03-06T07:25:37.973749+0000","last_became_peered":"2024-03-06T07:25:37.973749+0000","last_unstale":"2024-03-06T07:25:39.011600+0000","last_undegraded":"2024-03-06T07:25:39.011600+0000","last_fullsized":"2024-03-06T07:25:39.011600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:27.532403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964375+0000","last_change":"2024-03-06T07:25:36.964375+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964375+0000","last_undegraded":"2024-03-06T07:25:36.964375+0000","last_fullsized":"2024-03-06T07:25:36.964375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998403+0000","last_change":"2024-03-06T07:25:37.979581+0000","last_active":"2024-03-06T07:25:39.998403+0000","last_peered":"2024-03-06T07:25:39.998403+0000","last_clean":"2024-03-06T07:25:39.998403+0000","last_became_active":"2024-03-06T07:25:37.977877+0000","last_became_peered":"2024-03-06T07:25:37.977877+0000","last_unstale":"2024-03-06T07:25:39.998403+0000","last_undegraded":"2024-03-06T07:25:39.998403+0000","last_fullsized":"2024-03-06T07:25:39.998403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:36.309397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011763+0000","last_change":"2024-03-06T07:25:37.972168+0000","last_active":"2024-03-06T07:25:39.011763+0000","last_peered":"2024-03-06T07:25:39.011763+0000","last_clean":"2024-03-06T07:25:39.011763+0000","last_became_active":"2024-03-06T07:25:37.971899+0000","last_became_peered":"2024-03-06T07:25:37.971899+0000","last_unstale":"2024-03-06T07:25:39.011763+0000","last_undegraded":"2024-03-06T07:25:39.011763+0000","last_fullsized":"2024-03-06T07:25:39.011763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:24.475964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965583+0000","last_change":"2024-03-06T07:25:36.965583+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965583+0000","last_undegraded":"2024-03-06T07:25:36.965583+0000","last_fullsized":"2024-03-06T07:25:36.965583+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963919+0000","last_change":"2024-03-06T07:25:36.963919+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963919+0000","last_undegraded":"2024-03-06T07:25:36.963919+0000","last_fullsized":"2024-03-06T07:25:36.963919+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969843+0000","last_change":"2024-03-06T07:25:36.969843+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969843+0000","last_undegraded":"2024-03-06T07:25:36.969843+0000","last_fullsized":"2024-03-06T07:25:36.969843+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011712+0000","last_change":"2024-03-06T07:25:37.976490+0000","last_active":"2024-03-06T07:25:39.011712+0000","last_peered":"2024-03-06T07:25:39.011712+0000","last_clean":"2024-03-06T07:25:39.011712+0000","last_became_active":"2024-03-06T07:25:37.976222+0000","last_became_peered":"2024-03-06T07:25:37.976222+0000","last_unstale":"2024-03-06T07:25:39.011712+0000","last_undegraded":"2024-03-06T07:25:39.011712+0000","last_fullsized":"2024-03-06T07:25:39.011712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:11.129867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011827+0000","last_change":"2024-03-06T07:25:37.985856+0000","last_active":"2024-03-06T07:25:39.011827+0000","last_peered":"2024-03-06T07:25:39.011827+0000","last_clean":"2024-03-06T07:25:39.011827+0000","last_became_active":"2024-03-06T07:25:37.985235+0000","last_became_peered":"2024-03-06T07:25:37.985235+0000","last_unstale":"2024-03-06T07:25:39.011827+0000","last_undegraded":"2024-03-06T07:25:39.011827+0000","last_fullsized":"2024-03-06T07:25:39.011827+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:00.019595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963000+0000","last_change":"2024-03-06T07:25:36.963000+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963000+0000","last_undegraded":"2024-03-06T07:25:36.963000+0000","last_fullsized":"2024-03-06T07:25:36.963000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.964171+0000","last_change":"2024-03-06T07:25:36.964171+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.964171+0000","last_undegraded":"2024-03-06T07:25:36.964171+0000","last_fullsized":"2024-03-06T07:25:36.964171+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963613+0000","last_change":"2024-03-06T07:25:36.963613+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963613+0000","last_undegraded":"2024-03-06T07:25:36.963613+0000","last_fullsized":"2024-03-06T07:25:36.963613+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975492+0000","last_change":"2024-03-06T07:25:36.974330+0000","last_active":"2024-03-06T07:25:36.975492+0000","last_peered":"2024-03-06T07:25:36.975492+0000","last_clean":"2024-03-06T07:25:36.975492+0000","last_became_active":"2024-03-06T07:25:36.974050+0000","last_became_peered":"2024-03-06T07:25:36.974050+0000","last_unstale":"2024-03-06T07:25:36.975492+0000","last_undegraded":"2024-03-06T07:25:36.975492+0000","last_fullsized":"2024-03-06T07:25:36.975492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:15.879147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967828+0000","last_change":"2024-03-06T07:25:36.967828+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967828+0000","last_undegraded":"2024-03-06T07:25:36.967828+0000","last_fullsized":"2024-03-06T07:25:36.967828+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972393+0000","last_change":"2024-03-06T07:25:36.967151+0000","last_active":"2024-03-06T07:25:36.972393+0000","last_peered":"2024-03-06T07:25:36.972393+0000","last_clean":"2024-03-06T07:25:36.972393+0000","last_became_active":"2024-03-06T07:25:36.966864+0000","last_became_peered":"2024-03-06T07:25:36.966864+0000","last_unstale":"2024-03-06T07:25:36.972393+0000","last_undegraded":"2024-03-06T07:25:36.972393+0000","last_fullsized":"2024-03-06T07:25:36.972393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.969345+0000","last_change":"2024-03-06T07:25:36.969345+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.969345+0000","last_undegraded":"2024-03-06T07:25:36.969345+0000","last_fullsized":"2024-03-06T07:25:36.969345+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975047+0000","last_change":"2024-03-06T07:25:36.974099+0000","last_active":"2024-03-06T07:25:38.975047+0000","last_peered":"2024-03-06T07:25:38.975047+0000","last_clean":"2024-03-06T07:25:38.975047+0000","last_became_active":"2024-03-06T07:25:36.973818+0000","last_became_peered":"2024-03-06T07:25:36.973818+0000","last_unstale":"2024-03-06T07:25:38.975047+0000","last_undegraded":"2024-03-06T07:25:38.975047+0000","last_fullsized":"2024-03-06T07:25:38.975047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:10.418371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965086+0000","last_change":"2024-03-06T07:25:36.965086+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965086+0000","last_undegraded":"2024-03-06T07:25:36.965086+0000","last_fullsized":"2024-03-06T07:25:36.965086+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974974+0000","last_change":"2024-03-06T07:25:36.974705+0000","last_active":"2024-03-06T07:25:38.974974+0000","last_peered":"2024-03-06T07:25:38.974974+0000","last_clean":"2024-03-06T07:25:38.974974+0000","last_became_active":"2024-03-06T07:25:36.974418+0000","last_became_peered":"2024-03-06T07:25:36.974418+0000","last_unstale":"2024-03-06T07:25:38.974974+0000","last_undegraded":"2024-03-06T07:25:38.974974+0000","last_fullsized":"2024-03-06T07:25:38.974974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:06.544279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011817+0000","last_change":"2024-03-06T07:25:37.985947+0000","last_active":"2024-03-06T07:25:39.011817+0000","last_peered":"2024-03-06T07:25:39.011817+0000","last_clean":"2024-03-06T07:25:39.011817+0000","last_became_active":"2024-03-06T07:25:37.985621+0000","last_became_peered":"2024-03-06T07:25:37.985621+0000","last_unstale":"2024-03-06T07:25:39.011817+0000","last_undegraded":"2024-03-06T07:25:39.011817+0000","last_fullsized":"2024-03-06T07:25:39.011817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:11.219309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972867+0000","last_change":"2024-03-06T07:25:36.970497+0000","last_active":"2024-03-06T07:25:36.972867+0000","last_peered":"2024-03-06T07:25:36.972867+0000","last_clean":"2024-03-06T07:25:36.972867+0000","last_became_active":"2024-03-06T07:25:36.970229+0000","last_became_peered":"2024-03-06T07:25:36.970229+0000","last_unstale":"2024-03-06T07:25:36.972867+0000","last_undegraded":"2024-03-06T07:25:36.972867+0000","last_fullsized":"2024-03-06T07:25:36.972867+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962580+0000","last_change":"2024-03-06T07:25:36.962580+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962580+0000","last_undegraded":"2024-03-06T07:25:36.962580+0000","last_fullsized":"2024-03-06T07:25:36.962580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972874+0000","last_change":"2024-03-06T07:25:36.972014+0000","last_active":"2024-03-06T07:25:36.972874+0000","last_peered":"2024-03-06T07:25:36.972874+0000","last_clean":"2024-03-06T07:25:36.972874+0000","last_became_active":"2024-03-06T07:25:36.971692+0000","last_became_peered":"2024-03-06T07:25:36.971692+0000","last_unstale":"2024-03-06T07:25:36.972874+0000","last_undegraded":"2024-03-06T07:25:36.972874+0000","last_fullsized":"2024-03-06T07:25:36.972874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.918383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965503+0000","last_change":"2024-03-06T07:25:36.965503+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965503+0000","last_undegraded":"2024-03-06T07:25:36.965503+0000","last_fullsized":"2024-03-06T07:25:36.965503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974635+0000","last_change":"2024-03-06T07:25:36.969239+0000","last_active":"2024-03-06T07:25:38.974635+0000","last_peered":"2024-03-06T07:25:38.974635+0000","last_clean":"2024-03-06T07:25:38.974635+0000","last_became_active":"2024-03-06T07:25:36.968491+0000","last_became_peered":"2024-03-06T07:25:36.968491+0000","last_unstale":"2024-03-06T07:25:38.974635+0000","last_undegraded":"2024-03-06T07:25:38.974635+0000","last_fullsized":"2024-03-06T07:25:38.974635+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:17.024203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957846+0000","last_change":"2024-03-06T07:25:36.957846+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957846+0000","last_undegraded":"2024-03-06T07:25:36.957846+0000","last_fullsized":"2024-03-06T07:25:36.957846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011569+0000","last_change":"2024-03-06T07:25:36.974784+0000","last_active":"2024-03-06T07:25:39.011569+0000","last_peered":"2024-03-06T07:25:39.011569+0000","last_clean":"2024-03-06T07:25:39.011569+0000","last_became_active":"2024-03-06T07:25:36.974534+0000","last_became_peered":"2024-03-06T07:25:36.974534+0000","last_unstale":"2024-03-06T07:25:39.011569+0000","last_undegraded":"2024-03-06T07:25:39.011569+0000","last_fullsized":"2024-03-06T07:25:39.011569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:32.647611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958526+0000","last_change":"2024-03-06T07:25:36.958526+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958526+0000","last_undegraded":"2024-03-06T07:25:36.958526+0000","last_fullsized":"2024-03-06T07:25:36.958526+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976718+0000","last_change":"2024-03-06T07:25:36.975629+0000","last_active":"2024-03-06T07:25:36.976718+0000","last_peered":"2024-03-06T07:25:36.976718+0000","last_clean":"2024-03-06T07:25:36.976718+0000","last_became_active":"2024-03-06T07:25:36.975298+0000","last_became_peered":"2024-03-06T07:25:36.975298+0000","last_unstale":"2024-03-06T07:25:36.976718+0000","last_undegraded":"2024-03-06T07:25:36.976718+0000","last_fullsized":"2024-03-06T07:25:36.976718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957680+0000","last_change":"2024-03-06T07:25:36.957680+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957680+0000","last_undegraded":"2024-03-06T07:25:36.957680+0000","last_fullsized":"2024-03-06T07:25:36.957680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976133+0000","last_change":"2024-03-06T07:25:36.974955+0000","last_active":"2024-03-06T07:25:36.976133+0000","last_peered":"2024-03-06T07:25:36.976133+0000","last_clean":"2024-03-06T07:25:36.976133+0000","last_became_active":"2024-03-06T07:25:36.974657+0000","last_became_peered":"2024-03-06T07:25:36.974657+0000","last_unstale":"2024-03-06T07:25:36.976133+0000","last_undegraded":"2024-03-06T07:25:36.976133+0000","last_fullsized":"2024-03-06T07:25:36.976133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:29:33.339070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974957+0000","last_change":"2024-03-06T07:25:37.971266+0000","last_active":"2024-03-06T07:25:38.974957+0000","last_peered":"2024-03-06T07:25:38.974957+0000","last_clean":"2024-03-06T07:25:38.974957+0000","last_became_active":"2024-03-06T07:25:37.971011+0000","last_became_peered":"2024-03-06T07:25:37.971011+0000","last_unstale":"2024-03-06T07:25:38.974957+0000","last_undegraded":"2024-03-06T07:25:38.974957+0000","last_fullsized":"2024-03-06T07:25:38.974957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.994726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975518+0000","last_change":"2024-03-06T07:25:36.974695+0000","last_active":"2024-03-06T07:25:36.975518+0000","last_peered":"2024-03-06T07:25:36.975518+0000","last_clean":"2024-03-06T07:25:36.975518+0000","last_became_active":"2024-03-06T07:25:36.974384+0000","last_became_peered":"2024-03-06T07:25:36.974384+0000","last_unstale":"2024-03-06T07:25:36.975518+0000","last_undegraded":"2024-03-06T07:25:36.975518+0000","last_fullsized":"2024-03-06T07:25:36.975518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:25.464178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963479+0000","last_change":"2024-03-06T07:25:36.963479+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963479+0000","last_undegraded":"2024-03-06T07:25:36.963479+0000","last_fullsized":"2024-03-06T07:25:36.963479+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.965356+0000","last_change":"2024-03-06T07:25:36.964206+0000","last_active":"2024-03-06T07:25:36.965356+0000","last_peered":"2024-03-06T07:25:36.965356+0000","last_clean":"2024-03-06T07:25:36.965356+0000","last_became_active":"2024-03-06T07:25:36.963893+0000","last_became_peered":"2024-03-06T07:25:36.963893+0000","last_unstale":"2024-03-06T07:25:36.965356+0000","last_undegraded":"2024-03-06T07:25:36.965356+0000","last_fullsized":"2024-03-06T07:25:36.965356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:55:36.804851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974623+0000","last_change":"2024-03-06T07:25:37.971889+0000","last_active":"2024-03-06T07:25:38.974623+0000","last_peered":"2024-03-06T07:25:38.974623+0000","last_clean":"2024-03-06T07:25:38.974623+0000","last_became_active":"2024-03-06T07:25:37.971561+0000","last_became_peered":"2024-03-06T07:25:37.971561+0000","last_unstale":"2024-03-06T07:25:38.974623+0000","last_undegraded":"2024-03-06T07:25:38.974623+0000","last_fullsized":"2024-03-06T07:25:38.974623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:54.496647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011727+0000","last_change":"2024-03-06T07:25:36.972698+0000","last_active":"2024-03-06T07:25:39.011727+0000","last_peered":"2024-03-06T07:25:39.011727+0000","last_clean":"2024-03-06T07:25:39.011727+0000","last_became_active":"2024-03-06T07:25:36.972395+0000","last_became_peered":"2024-03-06T07:25:36.972395+0000","last_unstale":"2024-03-06T07:25:39.011727+0000","last_undegraded":"2024-03-06T07:25:39.011727+0000","last_fullsized":"2024-03-06T07:25:39.011727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:42.695940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966888+0000","last_change":"2024-03-06T07:25:36.966888+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966888+0000","last_undegraded":"2024-03-06T07:25:36.966888+0000","last_fullsized":"2024-03-06T07:25:36.966888+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.973687+0000","last_change":"2024-03-06T07:25:36.968375+0000","last_active":"2024-03-06T07:25:36.973687+0000","last_peered":"2024-03-06T07:25:36.973687+0000","last_clean":"2024-03-06T07:25:36.973687+0000","last_became_active":"2024-03-06T07:25:36.967683+0000","last_became_peered":"2024-03-06T07:25:36.967683+0000","last_unstale":"2024-03-06T07:25:36.973687+0000","last_undegraded":"2024-03-06T07:25:36.973687+0000","last_fullsized":"2024-03-06T07:25:36.973687+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:10.418371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011549+0000","last_change":"2024-03-06T07:25:37.984669+0000","last_active":"2024-03-06T07:25:39.011549+0000","last_peered":"2024-03-06T07:25:39.011549+0000","last_clean":"2024-03-06T07:25:39.011549+0000","last_became_active":"2024-03-06T07:25:37.984429+0000","last_became_peered":"2024-03-06T07:25:37.984429+0000","last_unstale":"2024-03-06T07:25:39.011549+0000","last_undegraded":"2024-03-06T07:25:39.011549+0000","last_fullsized":"2024-03-06T07:25:39.011549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:37.261068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.972289+0000","last_change":"2024-03-06T07:25:36.971741+0000","last_active":"2024-03-06T07:25:36.972289+0000","last_peered":"2024-03-06T07:25:36.972289+0000","last_clean":"2024-03-06T07:25:36.972289+0000","last_became_active":"2024-03-06T07:25:36.970858+0000","last_became_peered":"2024-03-06T07:25:36.970858+0000","last_unstale":"2024-03-06T07:25:36.972289+0000","last_undegraded":"2024-03-06T07:25:36.972289+0000","last_fullsized":"2024-03-06T07:25:36.972289+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:51:20.916375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011503+0000","last_change":"2024-03-06T07:25:37.970179+0000","last_active":"2024-03-06T07:25:39.011503+0000","last_peered":"2024-03-06T07:25:39.011503+0000","last_clean":"2024-03-06T07:25:39.011503+0000","last_became_active":"2024-03-06T07:25:37.969904+0000","last_became_peered":"2024-03-06T07:25:37.969904+0000","last_unstale":"2024-03-06T07:25:39.011503+0000","last_undegraded":"2024-03-06T07:25:39.011503+0000","last_fullsized":"2024-03-06T07:25:39.011503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:04.413441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974536+0000","last_change":"2024-03-06T07:25:36.972032+0000","last_active":"2024-03-06T07:25:38.974536+0000","last_peered":"2024-03-06T07:25:38.974536+0000","last_clean":"2024-03-06T07:25:38.974536+0000","last_became_active":"2024-03-06T07:25:36.971777+0000","last_became_peered":"2024-03-06T07:25:36.971777+0000","last_unstale":"2024-03-06T07:25:38.974536+0000","last_undegraded":"2024-03-06T07:25:38.974536+0000","last_fullsized":"2024-03-06T07:25:38.974536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.483347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965095+0000","last_change":"2024-03-06T07:25:36.965095+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965095+0000","last_undegraded":"2024-03-06T07:25:36.965095+0000","last_fullsized":"2024-03-06T07:25:36.965095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974248+0000","last_change":"2024-03-06T07:25:36.972827+0000","last_active":"2024-03-06T07:25:36.974248+0000","last_peered":"2024-03-06T07:25:36.974248+0000","last_clean":"2024-03-06T07:25:36.974248+0000","last_became_active":"2024-03-06T07:25:36.972549+0000","last_became_peered":"2024-03-06T07:25:36.972549+0000","last_unstale":"2024-03-06T07:25:36.974248+0000","last_undegraded":"2024-03-06T07:25:36.974248+0000","last_fullsized":"2024-03-06T07:25:36.974248+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:24:53.181415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.967228+0000","last_change":"2024-03-06T07:25:36.967228+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.967228+0000","last_undegraded":"2024-03-06T07:25:36.967228+0000","last_fullsized":"2024-03-06T07:25:36.967228+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975057+0000","last_change":"2024-03-06T07:25:36.974812+0000","last_active":"2024-03-06T07:25:38.975057+0000","last_peered":"2024-03-06T07:25:38.975057+0000","last_clean":"2024-03-06T07:25:38.975057+0000","last_became_active":"2024-03-06T07:25:36.974555+0000","last_became_peered":"2024-03-06T07:25:36.974555+0000","last_unstale":"2024-03-06T07:25:38.975057+0000","last_undegraded":"2024-03-06T07:25:38.975057+0000","last_fullsized":"2024-03-06T07:25:38.975057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:24.570961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962957+0000","last_change":"2024-03-06T07:25:36.962957+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962957+0000","last_undegraded":"2024-03-06T07:25:36.962957+0000","last_fullsized":"2024-03-06T07:25:36.962957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975044+0000","last_change":"2024-03-06T07:25:36.968979+0000","last_active":"2024-03-06T07:25:38.975044+0000","last_peered":"2024-03-06T07:25:38.975044+0000","last_clean":"2024-03-06T07:25:38.975044+0000","last_became_active":"2024-03-06T07:25:36.968727+0000","last_became_peered":"2024-03-06T07:25:36.968727+0000","last_unstale":"2024-03-06T07:25:38.975044+0000","last_undegraded":"2024-03-06T07:25:38.975044+0000","last_fullsized":"2024-03-06T07:25:38.975044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:21.965949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999008+0000","last_change":"2024-03-06T07:25:37.980200+0000","last_active":"2024-03-06T07:25:39.999008+0000","last_peered":"2024-03-06T07:25:39.999008+0000","last_clean":"2024-03-06T07:25:39.999008+0000","last_became_active":"2024-03-06T07:25:37.979886+0000","last_became_peered":"2024-03-06T07:25:37.979886+0000","last_unstale":"2024-03-06T07:25:39.999008+0000","last_undegraded":"2024-03-06T07:25:39.999008+0000","last_fullsized":"2024-03-06T07:25:39.999008+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:58.600920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963596+0000","last_change":"2024-03-06T07:25:36.962047+0000","last_active":"2024-03-06T07:25:36.963596+0000","last_peered":"2024-03-06T07:25:36.963596+0000","last_clean":"2024-03-06T07:25:36.963596+0000","last_became_active":"2024-03-06T07:25:36.961774+0000","last_became_peered":"2024-03-06T07:25:36.961774+0000","last_unstale":"2024-03-06T07:25:36.963596+0000","last_undegraded":"2024-03-06T07:25:36.963596+0000","last_fullsized":"2024-03-06T07:25:36.963596+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:24.570961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957718+0000","last_change":"2024-03-06T07:25:36.957718+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957718+0000","last_undegraded":"2024-03-06T07:25:36.957718+0000","last_fullsized":"2024-03-06T07:25:36.957718+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999161+0000","last_change":"2024-03-06T07:25:36.970500+0000","last_active":"2024-03-06T07:25:39.999161+0000","last_peered":"2024-03-06T07:25:39.999161+0000","last_clean":"2024-03-06T07:25:39.999161+0000","last_became_active":"2024-03-06T07:25:36.970286+0000","last_became_peered":"2024-03-06T07:25:36.970286+0000","last_unstale":"2024-03-06T07:25:39.999161+0000","last_undegraded":"2024-03-06T07:25:39.999161+0000","last_fullsized":"2024-03-06T07:25:39.999161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:36.141835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998463+0000","last_change":"2024-03-06T07:25:37.980200+0000","last_active":"2024-03-06T07:25:39.998463+0000","last_peered":"2024-03-06T07:25:39.998463+0000","last_clean":"2024-03-06T07:25:39.998463+0000","last_became_active":"2024-03-06T07:25:37.979939+0000","last_became_peered":"2024-03-06T07:25:37.979939+0000","last_unstale":"2024-03-06T07:25:39.998463+0000","last_undegraded":"2024-03-06T07:25:39.998463+0000","last_fullsized":"2024-03-06T07:25:39.998463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:33.893337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998650+0000","last_change":"2024-03-06T07:25:36.971016+0000","last_active":"2024-03-06T07:25:39.998650+0000","last_peered":"2024-03-06T07:25:39.998650+0000","last_clean":"2024-03-06T07:25:39.998650+0000","last_became_active":"2024-03-06T07:25:36.970738+0000","last_became_peered":"2024-03-06T07:25:36.970738+0000","last_unstale":"2024-03-06T07:25:39.998650+0000","last_undegraded":"2024-03-06T07:25:39.998650+0000","last_fullsized":"2024-03-06T07:25:39.998650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:16.101778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.958936+0000","last_change":"2024-03-06T07:25:36.958936+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.958936+0000","last_undegraded":"2024-03-06T07:25:36.958936+0000","last_fullsized":"2024-03-06T07:25:36.958936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.963460+0000","last_change":"2024-03-06T07:25:36.961782+0000","last_active":"2024-03-06T07:25:36.963460+0000","last_peered":"2024-03-06T07:25:36.963460+0000","last_clean":"2024-03-06T07:25:36.963460+0000","last_became_active":"2024-03-06T07:25:36.961534+0000","last_became_peered":"2024-03-06T07:25:36.961534+0000","last_unstale":"2024-03-06T07:25:36.963460+0000","last_undegraded":"2024-03-06T07:25:36.963460+0000","last_fullsized":"2024-03-06T07:25:36.963460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:17.024203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963202+0000","last_change":"2024-03-06T07:25:36.963202+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963202+0000","last_undegraded":"2024-03-06T07:25:36.963202+0000","last_fullsized":"2024-03-06T07:25:36.963202+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011756+0000","last_change":"2024-03-06T07:25:36.968364+0000","last_active":"2024-03-06T07:25:39.011756+0000","last_peered":"2024-03-06T07:25:39.011756+0000","last_clean":"2024-03-06T07:25:39.011756+0000","last_became_active":"2024-03-06T07:25:36.968108+0000","last_became_peered":"2024-03-06T07:25:36.968108+0000","last_unstale":"2024-03-06T07:25:39.011756+0000","last_undegraded":"2024-03-06T07:25:39.011756+0000","last_fullsized":"2024-03-06T07:25:39.011756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:07.154927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.966589+0000","last_change":"2024-03-06T07:25:36.966589+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.966589+0000","last_undegraded":"2024-03-06T07:25:36.966589+0000","last_fullsized":"2024-03-06T07:25:36.966589+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011846+0000","last_change":"2024-03-06T07:25:36.972060+0000","last_active":"2024-03-06T07:25:39.011846+0000","last_peered":"2024-03-06T07:25:39.011846+0000","last_clean":"2024-03-06T07:25:39.011846+0000","last_became_active":"2024-03-06T07:25:36.971774+0000","last_became_peered":"2024-03-06T07:25:36.971774+0000","last_unstale":"2024-03-06T07:25:39.011846+0000","last_undegraded":"2024-03-06T07:25:39.011846+0000","last_fullsized":"2024-03-06T07:25:39.011846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:48.807567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.968493+0000","last_change":"2024-03-06T07:25:36.968493+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.968493+0000","last_undegraded":"2024-03-06T07:25:36.968493+0000","last_fullsized":"2024-03-06T07:25:36.968493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.974580+0000","last_change":"2024-03-06T07:25:36.972458+0000","last_active":"2024-03-06T07:25:36.974580+0000","last_peered":"2024-03-06T07:25:36.974580+0000","last_clean":"2024-03-06T07:25:36.974580+0000","last_became_active":"2024-03-06T07:25:36.971925+0000","last_became_peered":"2024-03-06T07:25:36.971925+0000","last_unstale":"2024-03-06T07:25:36.974580+0000","last_undegraded":"2024-03-06T07:25:36.974580+0000","last_fullsized":"2024-03-06T07:25:36.974580+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:11:13.397656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.962197+0000","last_change":"2024-03-06T07:25:36.962197+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.962197+0000","last_undegraded":"2024-03-06T07:25:36.962197+0000","last_fullsized":"2024-03-06T07:25:36.962197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977043+0000","last_change":"2024-03-06T07:25:36.976309+0000","last_active":"2024-03-06T07:25:36.977043+0000","last_peered":"2024-03-06T07:25:36.977043+0000","last_clean":"2024-03-06T07:25:36.977043+0000","last_became_active":"2024-03-06T07:25:36.976019+0000","last_became_peered":"2024-03-06T07:25:36.976019+0000","last_unstale":"2024-03-06T07:25:36.977043+0000","last_undegraded":"2024-03-06T07:25:36.977043+0000","last_fullsized":"2024-03-06T07:25:36.977043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:27.326131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.957878+0000","last_change":"2024-03-06T07:25:36.957878+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.957878+0000","last_undegraded":"2024-03-06T07:25:36.957878+0000","last_fullsized":"2024-03-06T07:25:36.957878+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974910+0000","last_change":"2024-03-06T07:25:36.971247+0000","last_active":"2024-03-06T07:25:38.974910+0000","last_peered":"2024-03-06T07:25:38.974910+0000","last_clean":"2024-03-06T07:25:38.974910+0000","last_became_active":"2024-03-06T07:25:36.970950+0000","last_became_peered":"2024-03-06T07:25:36.970950+0000","last_unstale":"2024-03-06T07:25:38.974910+0000","last_undegraded":"2024-03-06T07:25:38.974910+0000","last_fullsized":"2024-03-06T07:25:38.974910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:50.551127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975155+0000","last_change":"2024-03-06T07:25:37.970743+0000","last_active":"2024-03-06T07:25:38.975155+0000","last_peered":"2024-03-06T07:25:38.975155+0000","last_clean":"2024-03-06T07:25:38.975155+0000","last_became_active":"2024-03-06T07:25:37.970478+0000","last_became_peered":"2024-03-06T07:25:37.970478+0000","last_unstale":"2024-03-06T07:25:38.975155+0000","last_undegraded":"2024-03-06T07:25:38.975155+0000","last_fullsized":"2024-03-06T07:25:38.975155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:54.807959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.970386+0000","last_change":"2024-03-06T07:25:36.967074+0000","last_active":"2024-03-06T07:25:36.970386+0000","last_peered":"2024-03-06T07:25:36.970386+0000","last_clean":"2024-03-06T07:25:36.970386+0000","last_became_active":"2024-03-06T07:25:36.966752+0000","last_became_peered":"2024-03-06T07:25:36.966752+0000","last_unstale":"2024-03-06T07:25:36.970386+0000","last_undegraded":"2024-03-06T07:25:36.970386+0000","last_fullsized":"2024-03-06T07:25:36.970386+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:16.076859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.965024+0000","last_change":"2024-03-06T07:25:36.965024+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.965024+0000","last_undegraded":"2024-03-06T07:25:36.965024+0000","last_fullsized":"2024-03-06T07:25:36.965024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.977030+0000","last_change":"2024-03-06T07:25:36.975964+0000","last_active":"2024-03-06T07:25:36.977030+0000","last_peered":"2024-03-06T07:25:36.977030+0000","last_clean":"2024-03-06T07:25:36.977030+0000","last_became_active":"2024-03-06T07:25:36.975665+0000","last_became_peered":"2024-03-06T07:25:36.975665+0000","last_unstale":"2024-03-06T07:25:36.977030+0000","last_undegraded":"2024-03-06T07:25:36.977030+0000","last_fullsized":"2024-03-06T07:25:36.977030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:08.250422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"creating+peering","last_fresh":"2024-03-06T07:25:36.963945+0000","last_change":"2024-03-06T07:25:36.963945+0000","last_active":"2024-03-06T07:25:36.949776+0000","last_peered":"2024-03-06T07:25:36.949776+0000","last_clean":"2024-03-06T07:25:36.949776+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:25:36.963945+0000","last_undegraded":"2024-03-06T07:25:36.963945+0000","last_fullsized":"2024-03-06T07:25:36.963945+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.976161+0000","last_change":"2024-03-06T07:25:36.974079+0000","last_active":"2024-03-06T07:25:36.976161+0000","last_peered":"2024-03-06T07:25:36.976161+0000","last_clean":"2024-03-06T07:25:36.976161+0000","last_became_active":"2024-03-06T07:25:36.973733+0000","last_became_peered":"2024-03-06T07:25:36.973733+0000","last_unstale":"2024-03-06T07:25:36.976161+0000","last_undegraded":"2024-03-06T07:25:36.976161+0000","last_fullsized":"2024-03-06T07:25:36.976161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:05.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.012099+0000","last_change":"2024-03-06T07:25:37.973362+0000","last_active":"2024-03-06T07:25:39.012099+0000","last_peered":"2024-03-06T07:25:39.012099+0000","last_clean":"2024-03-06T07:25:39.012099+0000","last_became_active":"2024-03-06T07:25:37.973060+0000","last_became_peered":"2024-03-06T07:25:37.973060+0000","last_unstale":"2024-03-06T07:25:39.012099+0000","last_undegraded":"2024-03-06T07:25:39.012099+0000","last_fullsized":"2024-03-06T07:25:39.012099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:38.450904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.975679+0000","last_change":"2024-03-06T07:25:36.974734+0000","last_active":"2024-03-06T07:25:36.975679+0000","last_peered":"2024-03-06T07:25:36.975679+0000","last_clean":"2024-03-06T07:25:36.975679+0000","last_became_active":"2024-03-06T07:25:36.974426+0000","last_became_peered":"2024-03-06T07:25:36.974426+0000","last_unstale":"2024-03-06T07:25:36.975679+0000","last_undegraded":"2024-03-06T07:25:36.975679+0000","last_fullsized":"2024-03-06T07:25:36.975679+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":282,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-06T07:25:36.967622+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:25:36.967622+0000","last_peered":"2024-03-06T07:25:36.967622+0000","last_clean":"2024-03-06T07:25:36.967622+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:25:36.967622+0000","last_undegraded":"2024-03-06T07:25:36.967622+0000","last_fullsized":"2024-03-06T07:25:36.967622+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":34,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1003270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":412902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":432834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1032192,"data_stored":432446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1022654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:43.227 INFO:tasks.ceph.mds.a.smithi072.stderr:2024-03-06T07:25:43.225+0000 7f6e7e3e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:45.517 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:25:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.616+0000 7f2f85839640 1 Processor -- start 2024-03-06T07:25:45.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.616+0000 7f2f85839640 1 -- start start 2024-03-06T07:25:45.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f85839640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f8015bdd0 0x7f2f800c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f85839640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f800d16e0 0x7f2f800d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f85839640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f800cebe0 con 0x7f2f8015fa10 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f85839640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f800ce980 con 0x7f2f8015bdd0 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f85839640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f800ceec0 con 0x7f2f800d16e0 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f8015bdd0 0x7f2f800c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f8015bdd0 0x7f2f800c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43998/0 (socket says 172.21.15.72:43998) 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 -- 172.21.15.72:0/3389757010 learned_addr learned my addr 172.21.15.72:0/3389757010 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 -- 172.21.15.72:0/3389757010 >> v2:172.21.15.146:3300/0 conn(0x7f2f800d16e0 msgr2=0x7f2f800d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 --2- 172.21.15.72:0/3389757010 >> v2:172.21.15.146:3300/0 conn(0x7f2f800d16e0 0x7f2f800d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 -- 172.21.15.72:0/3389757010 >> v1:172.21.15.146:6789/0 conn(0x7f2f8015fa10 legacy=0x7f2f800ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:45.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.617+0000 7f2f7effd640 1 -- 172.21.15.72:0/3389757010 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f800cf1a0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.618+0000 7f2f7effd640 1 --2- 172.21.15.72:0/3389757010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f800c5610 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2f8006e3c0 tx=0x7f2f7002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c409bdefa71f328 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.618+0000 7f2f7e7fc640 1 -- 172.21.15.72:0/3389757010 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7002e070 con 0x7f2f8015bdd0 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.618+0000 7f2f7e7fc640 1 -- 172.21.15.72:0/3389757010 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f70037070 con 0x7f2f8015bdd0 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.618+0000 7f2f7e7fc640 1 -- 172.21.15.72:0/3389757010 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70031a00 con 0x7f2f8015bdd0 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 -- 172.21.15.72:0/3389757010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 msgr2=0x7f2f800c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 --2- 172.21.15.72:0/3389757010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f800c5610 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2f8006e3c0 tx=0x7f2f7002b880 comp rx=0 tx=0).stop 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 -- 172.21.15.72:0/3389757010 shutdown_connections 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 --2- 172.21.15.72:0/3389757010 >> v2:172.21.15.146:3300/0 conn(0x7f2f800d16e0 0x7f2f800d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 --2- 172.21.15.72:0/3389757010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f800c5610 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 -- 172.21.15.72:0/3389757010 >> 172.21.15.72:0/3389757010 conn(0x7f2f8009a9f0 msgr2=0x7f2f8009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 -- 172.21.15.72:0/3389757010 shutdown_connections 2024-03-06T07:25:45.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.619+0000 7f2f85839640 1 -- 172.21.15.72:0/3389757010 wait complete. 2024-03-06T07:25:45.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.620+0000 7f2f85839640 1 Processor -- start 2024-03-06T07:25:45.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.620+0000 7f2f85839640 1 -- start start 2024-03-06T07:25:45.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f85839640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f800d16e0 0x7f2f8014b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f85839640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f8014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f85839640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f8015fa10 0x7f2f800c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f85839640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800cc5d0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f85839640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f800c5b50 con 0x7f2f8015fa10 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f85839640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8015c1b0 con 0x7f2f800d16e0 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f8014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f8014e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44006/0 (socket says 172.21.15.72:44006) 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 -- 172.21.15.72:0/890529722 learned_addr learned my addr 172.21.15.72:0/890529722 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 -- 172.21.15.72:0/890529722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f800d16e0 msgr2=0x7f2f8014b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f800d16e0 0x7f2f8014b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 -- 172.21.15.72:0/890529722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f8015fa10 msgr2=0x7f2f800c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f8015fa10 0x7f2f800c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 -- 172.21.15.72:0/890529722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8014eb00 con 0x7f2f8015bdd0 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.621+0000 7f2f77fff640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f8014e040 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c400 tx=0x7f2f6800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.622+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68017070 con 0x7f2f8015bdd0 2024-03-06T07:25:45.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.622+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f680035f0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.622+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8014ede0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.622+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f68003a20 con 0x7f2f8015bdd0 2024-03-06T07:25:45.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.622+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8014c4a0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.622+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8014bfb0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.623+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2f6801a070 con 0x7f2f8015bdd0 2024-03-06T07:25:45.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.623+0000 7f2f7cff9640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f6006b500 0x7f2f6006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.623+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f68089420 con 0x7f2f8015bdd0 2024-03-06T07:25:45.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.623+0000 7f2f7effd640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f6006b500 0x7f2f6006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.624+0000 7f2f7effd640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f6006b500 0x7f2f6006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2f70036000 tx=0x7f2f70030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.628+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2f680010c0 con 0x7f2f8015bdd0 2024-03-06T07:25:45.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.801+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f8004f330 con 0x7f2f6006b500 2024-03-06T07:25:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.807+0000 7f2f7cff9640 1 -- 172.21.15.72:0/890529722 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252085 (secure 0 0 0) 0x7f2f8004f330 con 0x7f2f6006b500 2024-03-06T07:25:45.812 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:45.813 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:25:45.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f6006b500 msgr2=0x7f2f6006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f6006b500 0x7f2f6006d9c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2f70036000 tx=0x7f2f70030000 comp rx=0 tx=0).stop 2024-03-06T07:25:45.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 msgr2=0x7f2f8014e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f8014e040 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800c400 tx=0x7f2f6800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 shutdown_connections 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2f6006b500 0x7f2f6006d9c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f8015fa10 0x7f2f800c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f8015bdd0 0x7f2f8014e040 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 --2- 172.21.15.72:0/890529722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f800d16e0 0x7f2f8014b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 >> 172.21.15.72:0/890529722 conn(0x7f2f8009a9f0 msgr2=0x7f2f8015eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 shutdown_connections 2024-03-06T07:25:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.810+0000 7f2f85839640 1 -- 172.21.15.72:0/890529722 wait complete. 2024-03-06T07:25:45.823 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-03-06T07:25:43.990138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":190,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258636,"kb_used_data":9192,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716084,"statfs":{"total":773094113280,"available":772829270016,"internally_reserved":0,"allocated":9412608,"data_stored":4643572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.035060"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999903+0000","last_change":"2024-03-06T07:25:37.977632+0000","last_active":"2024-03-06T07:25:39.999903+0000","last_peered":"2024-03-06T07:25:39.999903+0000","last_clean":"2024-03-06T07:25:39.999903+0000","last_became_active":"2024-03-06T07:25:37.976868+0000","last_became_peered":"2024-03-06T07:25:37.976868+0000","last_unstale":"2024-03-06T07:25:39.999903+0000","last_undegraded":"2024-03-06T07:25:39.999903+0000","last_fullsized":"2024-03-06T07:25:39.999903+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:11.424044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011873+0000","last_change":"2024-03-06T07:25:37.972532+0000","last_active":"2024-03-06T07:25:39.011873+0000","last_peered":"2024-03-06T07:25:39.011873+0000","last_clean":"2024-03-06T07:25:39.011873+0000","last_became_active":"2024-03-06T07:25:37.972199+0000","last_became_peered":"2024-03-06T07:25:37.972199+0000","last_unstale":"2024-03-06T07:25:39.011873+0000","last_undegraded":"2024-03-06T07:25:39.011873+0000","last_fullsized":"2024-03-06T07:25:39.011873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:05.865034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974774+0000","last_change":"2024-03-06T07:25:37.970392+0000","last_active":"2024-03-06T07:25:38.974774+0000","last_peered":"2024-03-06T07:25:38.974774+0000","last_clean":"2024-03-06T07:25:38.974774+0000","last_became_active":"2024-03-06T07:25:37.970133+0000","last_became_peered":"2024-03-06T07:25:37.970133+0000","last_unstale":"2024-03-06T07:25:38.974774+0000","last_undegraded":"2024-03-06T07:25:38.974774+0000","last_fullsized":"2024-03-06T07:25:38.974774+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:23:28.174836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999038+0000","last_change":"2024-03-06T07:25:37.980903+0000","last_active":"2024-03-06T07:25:39.999038+0000","last_peered":"2024-03-06T07:25:39.999038+0000","last_clean":"2024-03-06T07:25:39.999038+0000","last_became_active":"2024-03-06T07:25:37.978491+0000","last_became_peered":"2024-03-06T07:25:37.978491+0000","last_unstale":"2024-03-06T07:25:39.999038+0000","last_undegraded":"2024-03-06T07:25:39.999038+0000","last_fullsized":"2024-03-06T07:25:39.999038+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:43:15.381106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999622+0000","last_change":"2024-03-06T07:25:37.977979+0000","last_active":"2024-03-06T07:25:39.999622+0000","last_peered":"2024-03-06T07:25:39.999622+0000","last_clean":"2024-03-06T07:25:39.999622+0000","last_became_active":"2024-03-06T07:25:37.977639+0000","last_became_peered":"2024-03-06T07:25:37.977639+0000","last_unstale":"2024-03-06T07:25:39.999622+0000","last_undegraded":"2024-03-06T07:25:39.999622+0000","last_fullsized":"2024-03-06T07:25:39.999622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:18.625536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.973846+0000","last_change":"2024-03-06T07:25:37.983676+0000","last_active":"2024-03-06T07:25:38.973846+0000","last_peered":"2024-03-06T07:25:38.973846+0000","last_clean":"2024-03-06T07:25:38.973846+0000","last_became_active":"2024-03-06T07:25:37.983284+0000","last_became_peered":"2024-03-06T07:25:37.983284+0000","last_unstale":"2024-03-06T07:25:38.973846+0000","last_undegraded":"2024-03-06T07:25:38.973846+0000","last_fullsized":"2024-03-06T07:25:38.973846+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:22.971622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998682+0000","last_change":"2024-03-06T07:25:37.972040+0000","last_active":"2024-03-06T07:25:39.998682+0000","last_peered":"2024-03-06T07:25:39.998682+0000","last_clean":"2024-03-06T07:25:39.998682+0000","last_became_active":"2024-03-06T07:25:37.971785+0000","last_became_peered":"2024-03-06T07:25:37.971785+0000","last_unstale":"2024-03-06T07:25:39.998682+0000","last_undegraded":"2024-03-06T07:25:39.998682+0000","last_fullsized":"2024-03-06T07:25:39.998682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:28:25.839525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974459+0000","last_change":"2024-03-06T07:25:37.974273+0000","last_active":"2024-03-06T07:25:38.974459+0000","last_peered":"2024-03-06T07:25:38.974459+0000","last_clean":"2024-03-06T07:25:38.974459+0000","last_became_active":"2024-03-06T07:25:37.973411+0000","last_became_peered":"2024-03-06T07:25:37.973411+0000","last_unstale":"2024-03-06T07:25:38.974459+0000","last_undegraded":"2024-03-06T07:25:38.974459+0000","last_fullsized":"2024-03-06T07:25:38.974459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:26.531061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974721+0000","last_change":"2024-03-06T07:25:37.973172+0000","last_active":"2024-03-06T07:25:38.974721+0000","last_peered":"2024-03-06T07:25:38.974721+0000","last_clean":"2024-03-06T07:25:38.974721+0000","last_became_active":"2024-03-06T07:25:37.972798+0000","last_became_peered":"2024-03-06T07:25:37.972798+0000","last_unstale":"2024-03-06T07:25:38.974721+0000","last_undegraded":"2024-03-06T07:25:38.974721+0000","last_fullsized":"2024-03-06T07:25:38.974721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:54.496647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999643+0000","last_change":"2024-03-06T07:25:37.978660+0000","last_active":"2024-03-06T07:25:39.999643+0000","last_peered":"2024-03-06T07:25:39.999643+0000","last_clean":"2024-03-06T07:25:39.999643+0000","last_became_active":"2024-03-06T07:25:37.978321+0000","last_became_peered":"2024-03-06T07:25:37.978321+0000","last_unstale":"2024-03-06T07:25:39.999643+0000","last_undegraded":"2024-03-06T07:25:39.999643+0000","last_fullsized":"2024-03-06T07:25:39.999643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:18.029877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011965+0000","last_change":"2024-03-06T07:25:37.972340+0000","last_active":"2024-03-06T07:25:39.011965+0000","last_peered":"2024-03-06T07:25:39.011965+0000","last_clean":"2024-03-06T07:25:39.011965+0000","last_became_active":"2024-03-06T07:25:37.971978+0000","last_became_peered":"2024-03-06T07:25:37.971978+0000","last_unstale":"2024-03-06T07:25:39.011965+0000","last_undegraded":"2024-03-06T07:25:39.011965+0000","last_fullsized":"2024-03-06T07:25:39.011965+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:23:32.870432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999618+0000","last_change":"2024-03-06T07:25:37.979277+0000","last_active":"2024-03-06T07:25:39.999618+0000","last_peered":"2024-03-06T07:25:39.999618+0000","last_clean":"2024-03-06T07:25:39.999618+0000","last_became_active":"2024-03-06T07:25:37.978967+0000","last_became_peered":"2024-03-06T07:25:37.978967+0000","last_unstale":"2024-03-06T07:25:39.999618+0000","last_undegraded":"2024-03-06T07:25:39.999618+0000","last_fullsized":"2024-03-06T07:25:39.999618+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:08.339343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974122+0000","last_change":"2024-03-06T07:25:37.969083+0000","last_active":"2024-03-06T07:25:38.974122+0000","last_peered":"2024-03-06T07:25:38.974122+0000","last_clean":"2024-03-06T07:25:38.974122+0000","last_became_active":"2024-03-06T07:25:37.968744+0000","last_became_peered":"2024-03-06T07:25:37.968744+0000","last_unstale":"2024-03-06T07:25:38.974122+0000","last_undegraded":"2024-03-06T07:25:38.974122+0000","last_fullsized":"2024-03-06T07:25:38.974122+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:54.496647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998882+0000","last_change":"2024-03-06T07:25:37.980695+0000","last_active":"2024-03-06T07:25:39.998882+0000","last_peered":"2024-03-06T07:25:39.998882+0000","last_clean":"2024-03-06T07:25:39.998882+0000","last_became_active":"2024-03-06T07:25:37.980394+0000","last_became_peered":"2024-03-06T07:25:37.980394+0000","last_unstale":"2024-03-06T07:25:39.998882+0000","last_undegraded":"2024-03-06T07:25:39.998882+0000","last_fullsized":"2024-03-06T07:25:39.998882+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:18.338847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998584+0000","last_change":"2024-03-06T07:25:37.976163+0000","last_active":"2024-03-06T07:25:39.998584+0000","last_peered":"2024-03-06T07:25:39.998584+0000","last_clean":"2024-03-06T07:25:39.998584+0000","last_became_active":"2024-03-06T07:25:37.975940+0000","last_became_peered":"2024-03-06T07:25:37.975940+0000","last_unstale":"2024-03-06T07:25:39.998584+0000","last_undegraded":"2024-03-06T07:25:39.998584+0000","last_fullsized":"2024-03-06T07:25:39.998584+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:54.496647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999136+0000","last_change":"2024-03-06T07:25:37.970344+0000","last_active":"2024-03-06T07:25:39.999136+0000","last_peered":"2024-03-06T07:25:39.999136+0000","last_clean":"2024-03-06T07:25:39.999136+0000","last_became_active":"2024-03-06T07:25:37.970042+0000","last_became_peered":"2024-03-06T07:25:37.970042+0000","last_unstale":"2024-03-06T07:25:39.999136+0000","last_undegraded":"2024-03-06T07:25:39.999136+0000","last_fullsized":"2024-03-06T07:25:39.999136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:37.128498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999992+0000","last_change":"2024-03-06T07:25:37.972988+0000","last_active":"2024-03-06T07:25:39.999992+0000","last_peered":"2024-03-06T07:25:39.999992+0000","last_clean":"2024-03-06T07:25:39.999992+0000","last_became_active":"2024-03-06T07:25:37.972711+0000","last_became_peered":"2024-03-06T07:25:37.972711+0000","last_unstale":"2024-03-06T07:25:39.999992+0000","last_undegraded":"2024-03-06T07:25:39.999992+0000","last_fullsized":"2024-03-06T07:25:39.999992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:25.576635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998532+0000","last_change":"2024-03-06T07:25:37.980627+0000","last_active":"2024-03-06T07:25:39.998532+0000","last_peered":"2024-03-06T07:25:39.998532+0000","last_clean":"2024-03-06T07:25:39.998532+0000","last_became_active":"2024-03-06T07:25:37.980380+0000","last_became_peered":"2024-03-06T07:25:37.980380+0000","last_unstale":"2024-03-06T07:25:39.998532+0000","last_undegraded":"2024-03-06T07:25:39.998532+0000","last_fullsized":"2024-03-06T07:25:39.998532+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:10.907996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975080+0000","last_change":"2024-03-06T07:25:37.975647+0000","last_active":"2024-03-06T07:25:38.975080+0000","last_peered":"2024-03-06T07:25:38.975080+0000","last_clean":"2024-03-06T07:25:38.975080+0000","last_became_active":"2024-03-06T07:25:37.975400+0000","last_became_peered":"2024-03-06T07:25:37.975400+0000","last_unstale":"2024-03-06T07:25:38.975080+0000","last_undegraded":"2024-03-06T07:25:38.975080+0000","last_fullsized":"2024-03-06T07:25:38.975080+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:00.380038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999087+0000","last_change":"2024-03-06T07:25:37.978431+0000","last_active":"2024-03-06T07:25:39.999087+0000","last_peered":"2024-03-06T07:25:39.999087+0000","last_clean":"2024-03-06T07:25:39.999087+0000","last_became_active":"2024-03-06T07:25:37.977962+0000","last_became_peered":"2024-03-06T07:25:37.977962+0000","last_unstale":"2024-03-06T07:25:39.999087+0000","last_undegraded":"2024-03-06T07:25:39.999087+0000","last_fullsized":"2024-03-06T07:25:39.999087+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:52:56.267223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999148+0000","last_change":"2024-03-06T07:25:37.977273+0000","last_active":"2024-03-06T07:25:39.999148+0000","last_peered":"2024-03-06T07:25:39.999148+0000","last_clean":"2024-03-06T07:25:39.999148+0000","last_became_active":"2024-03-06T07:25:37.976985+0000","last_became_peered":"2024-03-06T07:25:37.976985+0000","last_unstale":"2024-03-06T07:25:39.999148+0000","last_undegraded":"2024-03-06T07:25:39.999148+0000","last_fullsized":"2024-03-06T07:25:39.999148+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:15:23.495818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011600+0000","last_change":"2024-03-06T07:25:37.973996+0000","last_active":"2024-03-06T07:25:39.011600+0000","last_peered":"2024-03-06T07:25:39.011600+0000","last_clean":"2024-03-06T07:25:39.011600+0000","last_became_active":"2024-03-06T07:25:37.973749+0000","last_became_peered":"2024-03-06T07:25:37.973749+0000","last_unstale":"2024-03-06T07:25:39.011600+0000","last_undegraded":"2024-03-06T07:25:39.011600+0000","last_fullsized":"2024-03-06T07:25:39.011600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:27.532403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999238+0000","last_change":"2024-03-06T07:25:37.977889+0000","last_active":"2024-03-06T07:25:39.999238+0000","last_peered":"2024-03-06T07:25:39.999238+0000","last_clean":"2024-03-06T07:25:39.999238+0000","last_became_active":"2024-03-06T07:25:37.977610+0000","last_became_peered":"2024-03-06T07:25:37.977610+0000","last_unstale":"2024-03-06T07:25:39.999238+0000","last_undegraded":"2024-03-06T07:25:39.999238+0000","last_fullsized":"2024-03-06T07:25:39.999238+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:18.625536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998403+0000","last_change":"2024-03-06T07:25:37.979581+0000","last_active":"2024-03-06T07:25:39.998403+0000","last_peered":"2024-03-06T07:25:39.998403+0000","last_clean":"2024-03-06T07:25:39.998403+0000","last_became_active":"2024-03-06T07:25:37.977877+0000","last_became_peered":"2024-03-06T07:25:37.977877+0000","last_unstale":"2024-03-06T07:25:39.998403+0000","last_undegraded":"2024-03-06T07:25:39.998403+0000","last_fullsized":"2024-03-06T07:25:39.998403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:52:36.309397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011763+0000","last_change":"2024-03-06T07:25:37.972168+0000","last_active":"2024-03-06T07:25:39.011763+0000","last_peered":"2024-03-06T07:25:39.011763+0000","last_clean":"2024-03-06T07:25:39.011763+0000","last_became_active":"2024-03-06T07:25:37.971899+0000","last_became_peered":"2024-03-06T07:25:37.971899+0000","last_unstale":"2024-03-06T07:25:39.011763+0000","last_undegraded":"2024-03-06T07:25:39.011763+0000","last_fullsized":"2024-03-06T07:25:39.011763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:24.475964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.973942+0000","last_change":"2024-03-06T07:25:37.979544+0000","last_active":"2024-03-06T07:25:38.973942+0000","last_peered":"2024-03-06T07:25:38.973942+0000","last_clean":"2024-03-06T07:25:38.973942+0000","last_became_active":"2024-03-06T07:25:37.979242+0000","last_became_peered":"2024-03-06T07:25:37.979242+0000","last_unstale":"2024-03-06T07:25:38.973942+0000","last_undegraded":"2024-03-06T07:25:38.973942+0000","last_fullsized":"2024-03-06T07:25:38.973942+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:26.531061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999632+0000","last_change":"2024-03-06T07:25:37.974483+0000","last_active":"2024-03-06T07:25:39.999632+0000","last_peered":"2024-03-06T07:25:39.999632+0000","last_clean":"2024-03-06T07:25:39.999632+0000","last_became_active":"2024-03-06T07:25:37.974217+0000","last_became_peered":"2024-03-06T07:25:37.974217+0000","last_unstale":"2024-03-06T07:25:39.999632+0000","last_undegraded":"2024-03-06T07:25:39.999632+0000","last_fullsized":"2024-03-06T07:25:39.999632+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:40.576344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999780+0000","last_change":"2024-03-06T07:25:37.974974+0000","last_active":"2024-03-06T07:25:39.999780+0000","last_peered":"2024-03-06T07:25:39.999780+0000","last_clean":"2024-03-06T07:25:39.999780+0000","last_became_active":"2024-03-06T07:25:37.974701+0000","last_became_peered":"2024-03-06T07:25:37.974701+0000","last_unstale":"2024-03-06T07:25:39.999780+0000","last_undegraded":"2024-03-06T07:25:39.999780+0000","last_fullsized":"2024-03-06T07:25:39.999780+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:25.576635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011712+0000","last_change":"2024-03-06T07:25:37.976490+0000","last_active":"2024-03-06T07:25:39.011712+0000","last_peered":"2024-03-06T07:25:39.011712+0000","last_clean":"2024-03-06T07:25:39.011712+0000","last_became_active":"2024-03-06T07:25:37.976222+0000","last_became_peered":"2024-03-06T07:25:37.976222+0000","last_unstale":"2024-03-06T07:25:39.011712+0000","last_undegraded":"2024-03-06T07:25:39.011712+0000","last_fullsized":"2024-03-06T07:25:39.011712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:11.129867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011827+0000","last_change":"2024-03-06T07:25:37.985856+0000","last_active":"2024-03-06T07:25:39.011827+0000","last_peered":"2024-03-06T07:25:39.011827+0000","last_clean":"2024-03-06T07:25:39.011827+0000","last_became_active":"2024-03-06T07:25:37.985235+0000","last_became_peered":"2024-03-06T07:25:37.985235+0000","last_unstale":"2024-03-06T07:25:39.011827+0000","last_undegraded":"2024-03-06T07:25:39.011827+0000","last_fullsized":"2024-03-06T07:25:39.011827+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:11:00.019595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000060+0000","last_change":"2024-03-06T07:25:37.976416+0000","last_active":"2024-03-06T07:25:40.000060+0000","last_peered":"2024-03-06T07:25:40.000060+0000","last_clean":"2024-03-06T07:25:40.000060+0000","last_became_active":"2024-03-06T07:25:37.976052+0000","last_became_peered":"2024-03-06T07:25:37.976052+0000","last_unstale":"2024-03-06T07:25:40.000060+0000","last_undegraded":"2024-03-06T07:25:40.000060+0000","last_fullsized":"2024-03-06T07:25:40.000060+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:11.424044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999243+0000","last_change":"2024-03-06T07:25:37.974438+0000","last_active":"2024-03-06T07:25:39.999243+0000","last_peered":"2024-03-06T07:25:39.999243+0000","last_clean":"2024-03-06T07:25:39.999243+0000","last_became_active":"2024-03-06T07:25:37.974186+0000","last_became_peered":"2024-03-06T07:25:37.974186+0000","last_unstale":"2024-03-06T07:25:39.999243+0000","last_undegraded":"2024-03-06T07:25:39.999243+0000","last_fullsized":"2024-03-06T07:25:39.999243+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:40.576344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999984+0000","last_change":"2024-03-06T07:25:37.970319+0000","last_active":"2024-03-06T07:25:39.999984+0000","last_peered":"2024-03-06T07:25:39.999984+0000","last_clean":"2024-03-06T07:25:39.999984+0000","last_became_active":"2024-03-06T07:25:37.970026+0000","last_became_peered":"2024-03-06T07:25:37.970026+0000","last_unstale":"2024-03-06T07:25:39.999984+0000","last_undegraded":"2024-03-06T07:25:39.999984+0000","last_fullsized":"2024-03-06T07:25:39.999984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:18.029877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000300+0000","last_change":"2024-03-06T07:25:36.974330+0000","last_active":"2024-03-06T07:25:40.000300+0000","last_peered":"2024-03-06T07:25:40.000300+0000","last_clean":"2024-03-06T07:25:40.000300+0000","last_became_active":"2024-03-06T07:25:36.974050+0000","last_became_peered":"2024-03-06T07:25:36.974050+0000","last_unstale":"2024-03-06T07:25:40.000300+0000","last_undegraded":"2024-03-06T07:25:40.000300+0000","last_fullsized":"2024-03-06T07:25:40.000300+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:23.470290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000094+0000","last_change":"2024-03-06T07:25:37.979419+0000","last_active":"2024-03-06T07:25:40.000094+0000","last_peered":"2024-03-06T07:25:40.000094+0000","last_clean":"2024-03-06T07:25:40.000094+0000","last_became_active":"2024-03-06T07:25:37.978703+0000","last_became_peered":"2024-03-06T07:25:37.978703+0000","last_unstale":"2024-03-06T07:25:40.000094+0000","last_undegraded":"2024-03-06T07:25:40.000094+0000","last_fullsized":"2024-03-06T07:25:40.000094+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:51.556800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000191+0000","last_change":"2024-03-06T07:25:36.967151+0000","last_active":"2024-03-06T07:25:40.000191+0000","last_peered":"2024-03-06T07:25:40.000191+0000","last_clean":"2024-03-06T07:25:40.000191+0000","last_became_active":"2024-03-06T07:25:36.966864+0000","last_became_peered":"2024-03-06T07:25:36.966864+0000","last_unstale":"2024-03-06T07:25:40.000191+0000","last_undegraded":"2024-03-06T07:25:40.000191+0000","last_fullsized":"2024-03-06T07:25:40.000191+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:10.124193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999935+0000","last_change":"2024-03-06T07:25:37.978443+0000","last_active":"2024-03-06T07:25:39.999935+0000","last_peered":"2024-03-06T07:25:39.999935+0000","last_clean":"2024-03-06T07:25:39.999935+0000","last_became_active":"2024-03-06T07:25:37.978168+0000","last_became_peered":"2024-03-06T07:25:37.978168+0000","last_unstale":"2024-03-06T07:25:39.999935+0000","last_undegraded":"2024-03-06T07:25:39.999935+0000","last_fullsized":"2024-03-06T07:25:39.999935+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:07.549952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975047+0000","last_change":"2024-03-06T07:25:36.974099+0000","last_active":"2024-03-06T07:25:38.975047+0000","last_peered":"2024-03-06T07:25:38.975047+0000","last_clean":"2024-03-06T07:25:38.975047+0000","last_became_active":"2024-03-06T07:25:36.973818+0000","last_became_peered":"2024-03-06T07:25:36.973818+0000","last_unstale":"2024-03-06T07:25:38.975047+0000","last_undegraded":"2024-03-06T07:25:38.975047+0000","last_fullsized":"2024-03-06T07:25:38.975047+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:10.418371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974947+0000","last_change":"2024-03-06T07:25:37.980447+0000","last_active":"2024-03-06T07:25:38.974947+0000","last_peered":"2024-03-06T07:25:38.974947+0000","last_clean":"2024-03-06T07:25:38.974947+0000","last_became_active":"2024-03-06T07:25:37.980181+0000","last_became_peered":"2024-03-06T07:25:37.980181+0000","last_unstale":"2024-03-06T07:25:38.974947+0000","last_undegraded":"2024-03-06T07:25:38.974947+0000","last_fullsized":"2024-03-06T07:25:38.974947+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:47.906225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974974+0000","last_change":"2024-03-06T07:25:36.974705+0000","last_active":"2024-03-06T07:25:38.974974+0000","last_peered":"2024-03-06T07:25:38.974974+0000","last_clean":"2024-03-06T07:25:38.974974+0000","last_became_active":"2024-03-06T07:25:36.974418+0000","last_became_peered":"2024-03-06T07:25:36.974418+0000","last_unstale":"2024-03-06T07:25:38.974974+0000","last_undegraded":"2024-03-06T07:25:38.974974+0000","last_fullsized":"2024-03-06T07:25:38.974974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:06.544279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011817+0000","last_change":"2024-03-06T07:25:37.985947+0000","last_active":"2024-03-06T07:25:39.011817+0000","last_peered":"2024-03-06T07:25:39.011817+0000","last_clean":"2024-03-06T07:25:39.011817+0000","last_became_active":"2024-03-06T07:25:37.985621+0000","last_became_peered":"2024-03-06T07:25:37.985621+0000","last_unstale":"2024-03-06T07:25:39.011817+0000","last_undegraded":"2024-03-06T07:25:39.011817+0000","last_fullsized":"2024-03-06T07:25:39.011817+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:11.219309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000280+0000","last_change":"2024-03-06T07:25:36.970497+0000","last_active":"2024-03-06T07:25:40.000280+0000","last_peered":"2024-03-06T07:25:40.000280+0000","last_clean":"2024-03-06T07:25:40.000280+0000","last_became_active":"2024-03-06T07:25:36.970229+0000","last_became_peered":"2024-03-06T07:25:36.970229+0000","last_unstale":"2024-03-06T07:25:40.000280+0000","last_undegraded":"2024-03-06T07:25:40.000280+0000","last_fullsized":"2024-03-06T07:25:40.000280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:03.407768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000093+0000","last_change":"2024-03-06T07:25:37.981703+0000","last_active":"2024-03-06T07:25:40.000093+0000","last_peered":"2024-03-06T07:25:40.000093+0000","last_clean":"2024-03-06T07:25:40.000093+0000","last_became_active":"2024-03-06T07:25:37.981402+0000","last_became_peered":"2024-03-06T07:25:37.981402+0000","last_unstale":"2024-03-06T07:25:40.000093+0000","last_undegraded":"2024-03-06T07:25:40.000093+0000","last_fullsized":"2024-03-06T07:25:40.000093+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:07.549952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999362+0000","last_change":"2024-03-06T07:25:36.972014+0000","last_active":"2024-03-06T07:25:39.999362+0000","last_peered":"2024-03-06T07:25:39.999362+0000","last_clean":"2024-03-06T07:25:39.999362+0000","last_became_active":"2024-03-06T07:25:36.971692+0000","last_became_peered":"2024-03-06T07:25:36.971692+0000","last_unstale":"2024-03-06T07:25:39.999362+0000","last_undegraded":"2024-03-06T07:25:39.999362+0000","last_fullsized":"2024-03-06T07:25:39.999362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:16.087140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.973730+0000","last_change":"2024-03-06T07:25:37.976323+0000","last_active":"2024-03-06T07:25:38.973730+0000","last_peered":"2024-03-06T07:25:38.973730+0000","last_clean":"2024-03-06T07:25:38.973730+0000","last_became_active":"2024-03-06T07:25:37.976069+0000","last_became_peered":"2024-03-06T07:25:37.976069+0000","last_unstale":"2024-03-06T07:25:38.973730+0000","last_undegraded":"2024-03-06T07:25:38.973730+0000","last_fullsized":"2024-03-06T07:25:38.973730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:14.353630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974635+0000","last_change":"2024-03-06T07:25:36.969239+0000","last_active":"2024-03-06T07:25:38.974635+0000","last_peered":"2024-03-06T07:25:38.974635+0000","last_clean":"2024-03-06T07:25:38.974635+0000","last_became_active":"2024-03-06T07:25:36.968491+0000","last_became_peered":"2024-03-06T07:25:36.968491+0000","last_unstale":"2024-03-06T07:25:38.974635+0000","last_undegraded":"2024-03-06T07:25:38.974635+0000","last_fullsized":"2024-03-06T07:25:38.974635+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:15:17.024203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998723+0000","last_change":"2024-03-06T07:25:37.976163+0000","last_active":"2024-03-06T07:25:39.998723+0000","last_peered":"2024-03-06T07:25:39.998723+0000","last_clean":"2024-03-06T07:25:39.998723+0000","last_became_active":"2024-03-06T07:25:37.975907+0000","last_became_peered":"2024-03-06T07:25:37.975907+0000","last_unstale":"2024-03-06T07:25:39.998723+0000","last_undegraded":"2024-03-06T07:25:39.998723+0000","last_fullsized":"2024-03-06T07:25:39.998723+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.994726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011569+0000","last_change":"2024-03-06T07:25:36.974784+0000","last_active":"2024-03-06T07:25:39.011569+0000","last_peered":"2024-03-06T07:25:39.011569+0000","last_clean":"2024-03-06T07:25:39.011569+0000","last_became_active":"2024-03-06T07:25:36.974534+0000","last_became_peered":"2024-03-06T07:25:36.974534+0000","last_unstale":"2024-03-06T07:25:39.011569+0000","last_undegraded":"2024-03-06T07:25:39.011569+0000","last_fullsized":"2024-03-06T07:25:39.011569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:07:32.647611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999514+0000","last_change":"2024-03-06T07:25:37.977381+0000","last_active":"2024-03-06T07:25:39.999514+0000","last_peered":"2024-03-06T07:25:39.999514+0000","last_clean":"2024-03-06T07:25:39.999514+0000","last_became_active":"2024-03-06T07:25:37.977109+0000","last_became_peered":"2024-03-06T07:25:37.977109+0000","last_unstale":"2024-03-06T07:25:39.999514+0000","last_undegraded":"2024-03-06T07:25:39.999514+0000","last_fullsized":"2024-03-06T07:25:39.999514+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:00.380038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000351+0000","last_change":"2024-03-06T07:25:36.975629+0000","last_active":"2024-03-06T07:25:40.000351+0000","last_peered":"2024-03-06T07:25:40.000351+0000","last_clean":"2024-03-06T07:25:40.000351+0000","last_became_active":"2024-03-06T07:25:36.975298+0000","last_became_peered":"2024-03-06T07:25:36.975298+0000","last_unstale":"2024-03-06T07:25:40.000351+0000","last_undegraded":"2024-03-06T07:25:40.000351+0000","last_fullsized":"2024-03-06T07:25:40.000351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:36.255395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974687+0000","last_change":"2024-03-06T07:25:37.973874+0000","last_active":"2024-03-06T07:25:38.974687+0000","last_peered":"2024-03-06T07:25:38.974687+0000","last_clean":"2024-03-06T07:25:38.974687+0000","last_became_active":"2024-03-06T07:25:37.973558+0000","last_became_peered":"2024-03-06T07:25:37.973558+0000","last_unstale":"2024-03-06T07:25:38.974687+0000","last_undegraded":"2024-03-06T07:25:38.974687+0000","last_fullsized":"2024-03-06T07:25:38.974687+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:09.797411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999654+0000","last_change":"2024-03-06T07:25:36.974955+0000","last_active":"2024-03-06T07:25:39.999654+0000","last_peered":"2024-03-06T07:25:39.999654+0000","last_clean":"2024-03-06T07:25:39.999654+0000","last_became_active":"2024-03-06T07:25:36.974657+0000","last_became_peered":"2024-03-06T07:25:36.974657+0000","last_unstale":"2024-03-06T07:25:39.999654+0000","last_undegraded":"2024-03-06T07:25:39.999654+0000","last_fullsized":"2024-03-06T07:25:39.999654+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:26.526730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974957+0000","last_change":"2024-03-06T07:25:37.971266+0000","last_active":"2024-03-06T07:25:38.974957+0000","last_peered":"2024-03-06T07:25:38.974957+0000","last_clean":"2024-03-06T07:25:38.974957+0000","last_became_active":"2024-03-06T07:25:37.971011+0000","last_became_peered":"2024-03-06T07:25:37.971011+0000","last_unstale":"2024-03-06T07:25:38.974957+0000","last_undegraded":"2024-03-06T07:25:38.974957+0000","last_fullsized":"2024-03-06T07:25:38.974957+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.994726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999313+0000","last_change":"2024-03-06T07:25:36.974695+0000","last_active":"2024-03-06T07:25:39.999313+0000","last_peered":"2024-03-06T07:25:39.999313+0000","last_clean":"2024-03-06T07:25:39.999313+0000","last_became_active":"2024-03-06T07:25:36.974384+0000","last_became_peered":"2024-03-06T07:25:36.974384+0000","last_unstale":"2024-03-06T07:25:39.999313+0000","last_undegraded":"2024-03-06T07:25:39.999313+0000","last_fullsized":"2024-03-06T07:25:39.999313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:05.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999892+0000","last_change":"2024-03-06T07:25:37.976720+0000","last_active":"2024-03-06T07:25:39.999892+0000","last_peered":"2024-03-06T07:25:39.999892+0000","last_clean":"2024-03-06T07:25:39.999892+0000","last_became_active":"2024-03-06T07:25:37.976404+0000","last_became_peered":"2024-03-06T07:25:37.976404+0000","last_unstale":"2024-03-06T07:25:39.999892+0000","last_undegraded":"2024-03-06T07:25:39.999892+0000","last_fullsized":"2024-03-06T07:25:39.999892+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:31.489020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974407+0000","last_change":"2024-03-06T07:25:36.964206+0000","last_active":"2024-03-06T07:25:38.974407+0000","last_peered":"2024-03-06T07:25:38.974407+0000","last_clean":"2024-03-06T07:25:38.974407+0000","last_became_active":"2024-03-06T07:25:36.963893+0000","last_became_peered":"2024-03-06T07:25:36.963893+0000","last_unstale":"2024-03-06T07:25:38.974407+0000","last_undegraded":"2024-03-06T07:25:38.974407+0000","last_fullsized":"2024-03-06T07:25:38.974407+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:27:36.122825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974623+0000","last_change":"2024-03-06T07:25:37.971889+0000","last_active":"2024-03-06T07:25:38.974623+0000","last_peered":"2024-03-06T07:25:38.974623+0000","last_clean":"2024-03-06T07:25:38.974623+0000","last_became_active":"2024-03-06T07:25:37.971561+0000","last_became_peered":"2024-03-06T07:25:37.971561+0000","last_unstale":"2024-03-06T07:25:38.974623+0000","last_undegraded":"2024-03-06T07:25:38.974623+0000","last_fullsized":"2024-03-06T07:25:38.974623+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:54.496647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011727+0000","last_change":"2024-03-06T07:25:36.972698+0000","last_active":"2024-03-06T07:25:39.011727+0000","last_peered":"2024-03-06T07:25:39.011727+0000","last_clean":"2024-03-06T07:25:39.011727+0000","last_became_active":"2024-03-06T07:25:36.972395+0000","last_became_peered":"2024-03-06T07:25:36.972395+0000","last_unstale":"2024-03-06T07:25:39.011727+0000","last_undegraded":"2024-03-06T07:25:39.011727+0000","last_fullsized":"2024-03-06T07:25:39.011727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:27:42.695940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974269+0000","last_change":"2024-03-06T07:25:37.975622+0000","last_active":"2024-03-06T07:25:38.974269+0000","last_peered":"2024-03-06T07:25:38.974269+0000","last_clean":"2024-03-06T07:25:38.974269+0000","last_became_active":"2024-03-06T07:25:37.975295+0000","last_became_peered":"2024-03-06T07:25:37.975295+0000","last_unstale":"2024-03-06T07:25:38.974269+0000","last_undegraded":"2024-03-06T07:25:38.974269+0000","last_fullsized":"2024-03-06T07:25:38.974269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:04:00.380038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999001+0000","last_change":"2024-03-06T07:25:36.968375+0000","last_active":"2024-03-06T07:25:39.999001+0000","last_peered":"2024-03-06T07:25:39.999001+0000","last_clean":"2024-03-06T07:25:39.999001+0000","last_became_active":"2024-03-06T07:25:36.967683+0000","last_became_peered":"2024-03-06T07:25:36.967683+0000","last_unstale":"2024-03-06T07:25:39.999001+0000","last_undegraded":"2024-03-06T07:25:39.999001+0000","last_fullsized":"2024-03-06T07:25:39.999001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:06.544279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011549+0000","last_change":"2024-03-06T07:25:37.984669+0000","last_active":"2024-03-06T07:25:39.011549+0000","last_peered":"2024-03-06T07:25:39.011549+0000","last_clean":"2024-03-06T07:25:39.011549+0000","last_became_active":"2024-03-06T07:25:37.984429+0000","last_became_peered":"2024-03-06T07:25:37.984429+0000","last_unstale":"2024-03-06T07:25:39.011549+0000","last_undegraded":"2024-03-06T07:25:39.011549+0000","last_fullsized":"2024-03-06T07:25:39.011549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:37.261068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974021+0000","last_change":"2024-03-06T07:25:36.971741+0000","last_active":"2024-03-06T07:25:38.974021+0000","last_peered":"2024-03-06T07:25:38.974021+0000","last_clean":"2024-03-06T07:25:38.974021+0000","last_became_active":"2024-03-06T07:25:36.970858+0000","last_became_peered":"2024-03-06T07:25:36.970858+0000","last_unstale":"2024-03-06T07:25:38.974021+0000","last_undegraded":"2024-03-06T07:25:38.974021+0000","last_fullsized":"2024-03-06T07:25:38.974021+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:17.619863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011503+0000","last_change":"2024-03-06T07:25:37.970179+0000","last_active":"2024-03-06T07:25:39.011503+0000","last_peered":"2024-03-06T07:25:39.011503+0000","last_clean":"2024-03-06T07:25:39.011503+0000","last_became_active":"2024-03-06T07:25:37.969904+0000","last_became_peered":"2024-03-06T07:25:37.969904+0000","last_unstale":"2024-03-06T07:25:39.011503+0000","last_undegraded":"2024-03-06T07:25:39.011503+0000","last_fullsized":"2024-03-06T07:25:39.011503+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:04.413441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974536+0000","last_change":"2024-03-06T07:25:36.972032+0000","last_active":"2024-03-06T07:25:38.974536+0000","last_peered":"2024-03-06T07:25:38.974536+0000","last_clean":"2024-03-06T07:25:38.974536+0000","last_became_active":"2024-03-06T07:25:36.971777+0000","last_became_peered":"2024-03-06T07:25:36.971777+0000","last_unstale":"2024-03-06T07:25:38.974536+0000","last_undegraded":"2024-03-06T07:25:38.974536+0000","last_fullsized":"2024-03-06T07:25:38.974536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:30.483347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974090+0000","last_change":"2024-03-06T07:25:37.971811+0000","last_active":"2024-03-06T07:25:38.974090+0000","last_peered":"2024-03-06T07:25:38.974090+0000","last_clean":"2024-03-06T07:25:38.974090+0000","last_became_active":"2024-03-06T07:25:37.971510+0000","last_became_peered":"2024-03-06T07:25:37.971510+0000","last_unstale":"2024-03-06T07:25:38.974090+0000","last_undegraded":"2024-03-06T07:25:38.974090+0000","last_fullsized":"2024-03-06T07:25:38.974090+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:09.797411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000450+0000","last_change":"2024-03-06T07:25:36.972827+0000","last_active":"2024-03-06T07:25:40.000450+0000","last_peered":"2024-03-06T07:25:40.000450+0000","last_clean":"2024-03-06T07:25:40.000450+0000","last_became_active":"2024-03-06T07:25:36.972549+0000","last_became_peered":"2024-03-06T07:25:36.972549+0000","last_unstale":"2024-03-06T07:25:40.000450+0000","last_undegraded":"2024-03-06T07:25:40.000450+0000","last_fullsized":"2024-03-06T07:25:40.000450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:10.124193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000256+0000","last_change":"2024-03-06T07:25:37.973374+0000","last_active":"2024-03-06T07:25:40.000256+0000","last_peered":"2024-03-06T07:25:40.000256+0000","last_clean":"2024-03-06T07:25:40.000256+0000","last_became_active":"2024-03-06T07:25:37.972277+0000","last_became_peered":"2024-03-06T07:25:37.972277+0000","last_unstale":"2024-03-06T07:25:40.000256+0000","last_undegraded":"2024-03-06T07:25:40.000256+0000","last_fullsized":"2024-03-06T07:25:40.000256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:39.856252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975057+0000","last_change":"2024-03-06T07:25:36.974812+0000","last_active":"2024-03-06T07:25:38.975057+0000","last_peered":"2024-03-06T07:25:38.975057+0000","last_clean":"2024-03-06T07:25:38.975057+0000","last_became_active":"2024-03-06T07:25:36.974555+0000","last_became_peered":"2024-03-06T07:25:36.974555+0000","last_unstale":"2024-03-06T07:25:38.975057+0000","last_undegraded":"2024-03-06T07:25:38.975057+0000","last_fullsized":"2024-03-06T07:25:38.975057+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:24.570961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999837+0000","last_change":"2024-03-06T07:25:37.977891+0000","last_active":"2024-03-06T07:25:39.999837+0000","last_peered":"2024-03-06T07:25:39.999837+0000","last_clean":"2024-03-06T07:25:39.999837+0000","last_became_active":"2024-03-06T07:25:37.977613+0000","last_became_peered":"2024-03-06T07:25:37.977613+0000","last_unstale":"2024-03-06T07:25:39.999837+0000","last_undegraded":"2024-03-06T07:25:39.999837+0000","last_fullsized":"2024-03-06T07:25:39.999837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:45:08.339343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975044+0000","last_change":"2024-03-06T07:25:36.968979+0000","last_active":"2024-03-06T07:25:38.975044+0000","last_peered":"2024-03-06T07:25:38.975044+0000","last_clean":"2024-03-06T07:25:38.975044+0000","last_became_active":"2024-03-06T07:25:36.968727+0000","last_became_peered":"2024-03-06T07:25:36.968727+0000","last_unstale":"2024-03-06T07:25:38.975044+0000","last_undegraded":"2024-03-06T07:25:38.975044+0000","last_fullsized":"2024-03-06T07:25:38.975044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:21.965949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999008+0000","last_change":"2024-03-06T07:25:37.980200+0000","last_active":"2024-03-06T07:25:39.999008+0000","last_peered":"2024-03-06T07:25:39.999008+0000","last_clean":"2024-03-06T07:25:39.999008+0000","last_became_active":"2024-03-06T07:25:37.979886+0000","last_became_peered":"2024-03-06T07:25:37.979886+0000","last_unstale":"2024-03-06T07:25:39.999008+0000","last_undegraded":"2024-03-06T07:25:39.999008+0000","last_fullsized":"2024-03-06T07:25:39.999008+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:24:58.600920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999694+0000","last_change":"2024-03-06T07:25:36.962047+0000","last_active":"2024-03-06T07:25:39.999694+0000","last_peered":"2024-03-06T07:25:39.999694+0000","last_clean":"2024-03-06T07:25:39.999694+0000","last_became_active":"2024-03-06T07:25:36.961774+0000","last_became_peered":"2024-03-06T07:25:36.961774+0000","last_unstale":"2024-03-06T07:25:39.999694+0000","last_undegraded":"2024-03-06T07:25:39.999694+0000","last_fullsized":"2024-03-06T07:25:39.999694+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:50.551127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974190+0000","last_change":"2024-03-06T07:25:37.974673+0000","last_active":"2024-03-06T07:25:38.974190+0000","last_peered":"2024-03-06T07:25:38.974190+0000","last_clean":"2024-03-06T07:25:38.974190+0000","last_became_active":"2024-03-06T07:25:37.974423+0000","last_became_peered":"2024-03-06T07:25:37.974423+0000","last_unstale":"2024-03-06T07:25:38.974190+0000","last_undegraded":"2024-03-06T07:25:38.974190+0000","last_fullsized":"2024-03-06T07:25:38.974190+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:42:39.856252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999161+0000","last_change":"2024-03-06T07:25:36.970500+0000","last_active":"2024-03-06T07:25:39.999161+0000","last_peered":"2024-03-06T07:25:39.999161+0000","last_clean":"2024-03-06T07:25:39.999161+0000","last_became_active":"2024-03-06T07:25:36.970286+0000","last_became_peered":"2024-03-06T07:25:36.970286+0000","last_unstale":"2024-03-06T07:25:39.999161+0000","last_undegraded":"2024-03-06T07:25:39.999161+0000","last_fullsized":"2024-03-06T07:25:39.999161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:01:36.141835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998463+0000","last_change":"2024-03-06T07:25:37.980200+0000","last_active":"2024-03-06T07:25:39.998463+0000","last_peered":"2024-03-06T07:25:39.998463+0000","last_clean":"2024-03-06T07:25:39.998463+0000","last_became_active":"2024-03-06T07:25:37.979939+0000","last_became_peered":"2024-03-06T07:25:37.979939+0000","last_unstale":"2024-03-06T07:25:39.998463+0000","last_undegraded":"2024-03-06T07:25:39.998463+0000","last_fullsized":"2024-03-06T07:25:39.998463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:53:33.893337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998650+0000","last_change":"2024-03-06T07:25:36.971016+0000","last_active":"2024-03-06T07:25:39.998650+0000","last_peered":"2024-03-06T07:25:39.998650+0000","last_clean":"2024-03-06T07:25:39.998650+0000","last_became_active":"2024-03-06T07:25:36.970738+0000","last_became_peered":"2024-03-06T07:25:36.970738+0000","last_unstale":"2024-03-06T07:25:39.998650+0000","last_undegraded":"2024-03-06T07:25:39.998650+0000","last_fullsized":"2024-03-06T07:25:39.998650+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:30:16.101778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.973902+0000","last_change":"2024-03-06T07:25:37.972187+0000","last_active":"2024-03-06T07:25:38.973902+0000","last_peered":"2024-03-06T07:25:38.973902+0000","last_clean":"2024-03-06T07:25:38.973902+0000","last_became_active":"2024-03-06T07:25:37.971900+0000","last_became_peered":"2024-03-06T07:25:37.971900+0000","last_unstale":"2024-03-06T07:25:38.973902+0000","last_undegraded":"2024-03-06T07:25:38.973902+0000","last_fullsized":"2024-03-06T07:25:38.973902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:51.556800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.998335+0000","last_change":"2024-03-06T07:25:36.961782+0000","last_active":"2024-03-06T07:25:39.998335+0000","last_peered":"2024-03-06T07:25:39.998335+0000","last_clean":"2024-03-06T07:25:39.998335+0000","last_became_active":"2024-03-06T07:25:36.961534+0000","last_became_peered":"2024-03-06T07:25:36.961534+0000","last_unstale":"2024-03-06T07:25:39.998335+0000","last_undegraded":"2024-03-06T07:25:39.998335+0000","last_fullsized":"2024-03-06T07:25:39.998335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:21.965949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000164+0000","last_change":"2024-03-06T07:25:37.966437+0000","last_active":"2024-03-06T07:25:40.000164+0000","last_peered":"2024-03-06T07:25:40.000164+0000","last_clean":"2024-03-06T07:25:40.000164+0000","last_became_active":"2024-03-06T07:25:37.965831+0000","last_became_peered":"2024-03-06T07:25:37.965831+0000","last_unstale":"2024-03-06T07:25:40.000164+0000","last_undegraded":"2024-03-06T07:25:40.000164+0000","last_fullsized":"2024-03-06T07:25:40.000164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:22.971622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011756+0000","last_change":"2024-03-06T07:25:36.968364+0000","last_active":"2024-03-06T07:25:39.011756+0000","last_peered":"2024-03-06T07:25:39.011756+0000","last_clean":"2024-03-06T07:25:39.011756+0000","last_became_active":"2024-03-06T07:25:36.968108+0000","last_became_peered":"2024-03-06T07:25:36.968108+0000","last_unstale":"2024-03-06T07:25:39.011756+0000","last_undegraded":"2024-03-06T07:25:39.011756+0000","last_fullsized":"2024-03-06T07:25:39.011756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:07.154927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974386+0000","last_change":"2024-03-06T07:25:37.970967+0000","last_active":"2024-03-06T07:25:38.974386+0000","last_peered":"2024-03-06T07:25:38.974386+0000","last_clean":"2024-03-06T07:25:38.974386+0000","last_became_active":"2024-03-06T07:25:37.970726+0000","last_became_peered":"2024-03-06T07:25:37.970726+0000","last_unstale":"2024-03-06T07:25:38.974386+0000","last_undegraded":"2024-03-06T07:25:38.974386+0000","last_fullsized":"2024-03-06T07:25:38.974386+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:54.807959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.011846+0000","last_change":"2024-03-06T07:25:36.972060+0000","last_active":"2024-03-06T07:25:39.011846+0000","last_peered":"2024-03-06T07:25:39.011846+0000","last_clean":"2024-03-06T07:25:39.011846+0000","last_became_active":"2024-03-06T07:25:36.971774+0000","last_became_peered":"2024-03-06T07:25:36.971774+0000","last_unstale":"2024-03-06T07:25:39.011846+0000","last_undegraded":"2024-03-06T07:25:39.011846+0000","last_fullsized":"2024-03-06T07:25:39.011846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:59:48.807567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000089+0000","last_change":"2024-03-06T07:25:37.984183+0000","last_active":"2024-03-06T07:25:40.000089+0000","last_peered":"2024-03-06T07:25:40.000089+0000","last_clean":"2024-03-06T07:25:40.000089+0000","last_became_active":"2024-03-06T07:25:37.983903+0000","last_became_peered":"2024-03-06T07:25:37.983903+0000","last_unstale":"2024-03-06T07:25:40.000089+0000","last_undegraded":"2024-03-06T07:25:40.000089+0000","last_fullsized":"2024-03-06T07:25:40.000089+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:22.971622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999516+0000","last_change":"2024-03-06T07:25:36.972458+0000","last_active":"2024-03-06T07:25:39.999516+0000","last_peered":"2024-03-06T07:25:39.999516+0000","last_clean":"2024-03-06T07:25:39.999516+0000","last_became_active":"2024-03-06T07:25:36.971925+0000","last_became_peered":"2024-03-06T07:25:36.971925+0000","last_unstale":"2024-03-06T07:25:39.999516+0000","last_undegraded":"2024-03-06T07:25:39.999516+0000","last_fullsized":"2024-03-06T07:25:39.999516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:16.087140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000164+0000","last_change":"2024-03-06T07:25:37.976538+0000","last_active":"2024-03-06T07:25:40.000164+0000","last_peered":"2024-03-06T07:25:40.000164+0000","last_clean":"2024-03-06T07:25:40.000164+0000","last_became_active":"2024-03-06T07:25:37.976243+0000","last_became_peered":"2024-03-06T07:25:37.976243+0000","last_unstale":"2024-03-06T07:25:40.000164+0000","last_undegraded":"2024-03-06T07:25:40.000164+0000","last_fullsized":"2024-03-06T07:25:40.000164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:51.556800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:40.000174+0000","last_change":"2024-03-06T07:25:36.976309+0000","last_active":"2024-03-06T07:25:40.000174+0000","last_peered":"2024-03-06T07:25:40.000174+0000","last_clean":"2024-03-06T07:25:40.000174+0000","last_became_active":"2024-03-06T07:25:36.976019+0000","last_became_peered":"2024-03-06T07:25:36.976019+0000","last_unstale":"2024-03-06T07:25:40.000174+0000","last_undegraded":"2024-03-06T07:25:40.000174+0000","last_fullsized":"2024-03-06T07:25:40.000174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:03.407768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974383+0000","last_change":"2024-03-06T07:25:37.973852+0000","last_active":"2024-03-06T07:25:38.974383+0000","last_peered":"2024-03-06T07:25:38.974383+0000","last_clean":"2024-03-06T07:25:38.974383+0000","last_became_active":"2024-03-06T07:25:37.973594+0000","last_became_peered":"2024-03-06T07:25:37.973594+0000","last_unstale":"2024-03-06T07:25:38.974383+0000","last_undegraded":"2024-03-06T07:25:38.974383+0000","last_fullsized":"2024-03-06T07:25:38.974383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:14.353630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974910+0000","last_change":"2024-03-06T07:25:36.971247+0000","last_active":"2024-03-06T07:25:38.974910+0000","last_peered":"2024-03-06T07:25:38.974910+0000","last_clean":"2024-03-06T07:25:38.974910+0000","last_became_active":"2024-03-06T07:25:36.970950+0000","last_became_peered":"2024-03-06T07:25:36.970950+0000","last_unstale":"2024-03-06T07:25:38.974910+0000","last_undegraded":"2024-03-06T07:25:38.974910+0000","last_fullsized":"2024-03-06T07:25:38.974910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:32:50.551127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.975155+0000","last_change":"2024-03-06T07:25:37.970743+0000","last_active":"2024-03-06T07:25:38.975155+0000","last_peered":"2024-03-06T07:25:38.975155+0000","last_clean":"2024-03-06T07:25:38.975155+0000","last_became_active":"2024-03-06T07:25:37.970478+0000","last_became_peered":"2024-03-06T07:25:37.970478+0000","last_unstale":"2024-03-06T07:25:38.975155+0000","last_undegraded":"2024-03-06T07:25:38.975155+0000","last_fullsized":"2024-03-06T07:25:38.975155+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:35:54.807959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999364+0000","last_change":"2024-03-06T07:25:36.967074+0000","last_active":"2024-03-06T07:25:39.999364+0000","last_peered":"2024-03-06T07:25:39.999364+0000","last_clean":"2024-03-06T07:25:39.999364+0000","last_became_active":"2024-03-06T07:25:36.966752+0000","last_became_peered":"2024-03-06T07:25:36.966752+0000","last_unstale":"2024-03-06T07:25:39.999364+0000","last_undegraded":"2024-03-06T07:25:39.999364+0000","last_fullsized":"2024-03-06T07:25:39.999364+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:26.526730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974232+0000","last_change":"2024-03-06T07:25:37.980122+0000","last_active":"2024-03-06T07:25:38.974232+0000","last_peered":"2024-03-06T07:25:38.974232+0000","last_clean":"2024-03-06T07:25:38.974232+0000","last_became_active":"2024-03-06T07:25:37.979450+0000","last_became_peered":"2024-03-06T07:25:37.979450+0000","last_unstale":"2024-03-06T07:25:38.974232+0000","last_undegraded":"2024-03-06T07:25:38.974232+0000","last_fullsized":"2024-03-06T07:25:38.974232+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:40:53.994726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999420+0000","last_change":"2024-03-06T07:25:36.975964+0000","last_active":"2024-03-06T07:25:39.999420+0000","last_peered":"2024-03-06T07:25:39.999420+0000","last_clean":"2024-03-06T07:25:39.999420+0000","last_became_active":"2024-03-06T07:25:36.975665+0000","last_became_peered":"2024-03-06T07:25:36.975665+0000","last_unstale":"2024-03-06T07:25:39.999420+0000","last_undegraded":"2024-03-06T07:25:39.999420+0000","last_fullsized":"2024-03-06T07:25:39.999420+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:10.918383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999636+0000","last_change":"2024-03-06T07:25:37.984497+0000","last_active":"2024-03-06T07:25:39.999636+0000","last_peered":"2024-03-06T07:25:39.999636+0000","last_clean":"2024-03-06T07:25:39.999636+0000","last_became_active":"2024-03-06T07:25:37.983866+0000","last_became_peered":"2024-03-06T07:25:37.983866+0000","last_unstale":"2024-03-06T07:25:39.999636+0000","last_undegraded":"2024-03-06T07:25:39.999636+0000","last_fullsized":"2024-03-06T07:25:39.999636+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:07:31.489020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.999180+0000","last_change":"2024-03-06T07:25:36.974079+0000","last_active":"2024-03-06T07:25:39.999180+0000","last_peered":"2024-03-06T07:25:39.999180+0000","last_clean":"2024-03-06T07:25:39.999180+0000","last_became_active":"2024-03-06T07:25:36.973733+0000","last_became_peered":"2024-03-06T07:25:36.973733+0000","last_unstale":"2024-03-06T07:25:39.999180+0000","last_undegraded":"2024-03-06T07:25:39.999180+0000","last_fullsized":"2024-03-06T07:25:39.999180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:36.255395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:39.012099+0000","last_change":"2024-03-06T07:25:37.973362+0000","last_active":"2024-03-06T07:25:39.012099+0000","last_peered":"2024-03-06T07:25:39.012099+0000","last_clean":"2024-03-06T07:25:39.012099+0000","last_became_active":"2024-03-06T07:25:37.973060+0000","last_became_peered":"2024-03-06T07:25:37.973060+0000","last_unstale":"2024-03-06T07:25:39.012099+0000","last_undegraded":"2024-03-06T07:25:39.012099+0000","last_fullsized":"2024-03-06T07:25:39.012099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:36.949776+0000","last_clean_scrub_stamp":"2024-03-06T07:25:36.949776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:28:38.450904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.974895+0000","last_change":"2024-03-06T07:25:36.974734+0000","last_active":"2024-03-06T07:25:38.974895+0000","last_peered":"2024-03-06T07:25:38.974895+0000","last_clean":"2024-03-06T07:25:38.974895+0000","last_became_active":"2024-03-06T07:25:36.974426+0000","last_became_peered":"2024-03-06T07:25:36.974426+0000","last_unstale":"2024-03-06T07:25:38.974895+0000","last_undegraded":"2024-03-06T07:25:38.974895+0000","last_fullsized":"2024-03-06T07:25:38.974895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:25:35.944103+0000","last_clean_scrub_stamp":"2024-03-06T07:25:35.944103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:29:39.570670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":287,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-06T07:25:38.973899+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:25:38.973899+0000","last_peered":"2024-03-06T07:25:38.973899+0000","last_clean":"2024-03-06T07:25:38.973899+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:25:38.973899+0000","last_undegraded":"2024-03-06T07:25:38.973899+0000","last_fullsized":"2024-03-06T07:25:38.973899+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32760,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339080,"statfs":{"total":96636764160,"available":96603217920,"internally_reserved":0,"allocated":1617920,"data_stored":1022690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574900,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1032192,"data_stored":432946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574900,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32192,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1036288,"data_stored":433886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":432252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1032192,"data_stored":433864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":432834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1032192,"data_stored":432446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574900,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1617920,"data_stored":1022654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:25:45.827 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:25:45.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:25:45.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.970+0000 7f2a74fa2640 1 Processor -- start 2024-03-06T07:25:45.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.970+0000 7f2a74fa2640 1 -- start start 2024-03-06T07:25:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.971+0000 7f2a74fa2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a700afca0 0x7f2a7006d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.971+0000 7f2a74fa2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7015aee0 con 0x7f2a7015ab00 2024-03-06T07:25:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a74fa2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a700ce790 con 0x7f2a7006f3a0 2024-03-06T07:25:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a74fa2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a70072540 con 0x7f2a700afca0 2024-03-06T07:25:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2a7015ab00 0x7f2a700cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43096/0 (socket says 172.21.15.72:43096) 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6f7fe640 1 -- 172.21.15.72:0/2824354723 learned_addr learned my addr 172.21.15.72:0/2824354723 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6effd640 1 --2- 172.21.15.72:0/2824354723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a700afca0 0x7f2a7006d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212676719 0 0) 0x7f2a7015aee0 con 0x7f2a7015ab00 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a58003520 con 0x7f2a7015ab00 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606917501 0 0) 0x7f2a700ce790 con 0x7f2a7006f3a0 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7015aee0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6effd640 1 -- 172.21.15.72:0/2824354723 >> v1:172.21.15.146:6789/0 conn(0x7f2a7006f3a0 legacy=0x7f2a7006c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6effd640 1 -- 172.21.15.72:0/2824354723 >> v1:172.21.15.72:6789/0 conn(0x7f2a7015ab00 legacy=0x7f2a700cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.972+0000 7f2a6effd640 1 -- 172.21.15.72:0/2824354723 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a700cea70 con 0x7f2a700afca0 2024-03-06T07:25:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.973+0000 7f2a6effd640 1 --2- 172.21.15.72:0/2824354723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a700afca0 0x7f2a7006d840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2a640097d0 tx=0x7f2a6402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c5cfc1f880aae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.973+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64035020 con 0x7f2a700afca0 2024-03-06T07:25:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.973+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64031800 con 0x7f2a700afca0 2024-03-06T07:25:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.973+0000 7f2a6e7fc640 1 -- 172.21.15.72:0/2824354723 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64031bd0 con 0x7f2a700afca0 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 -- 172.21.15.72:0/2824354723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a700afca0 msgr2=0x7f2a7006d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/2824354723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a700afca0 0x7f2a7006d840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2a640097d0 tx=0x7f2a6402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 -- 172.21.15.72:0/2824354723 shutdown_connections 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/2824354723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a700afca0 0x7f2a7006d840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 -- 172.21.15.72:0/2824354723 >> 172.21.15.72:0/2824354723 conn(0x7f2a700b2580 msgr2=0x7f2a700b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 -- 172.21.15.72:0/2824354723 shutdown_connections 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.974+0000 7f2a74fa2640 1 -- 172.21.15.72:0/2824354723 wait complete. 2024-03-06T07:25:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.976+0000 7f2a74fa2640 1 Processor -- start 2024-03-06T07:25:45.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.976+0000 7f2a74fa2640 1 -- start start 2024-03-06T07:25:45.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a74fa2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 0x7f2a7014b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a74fa2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a700afca0 0x7f2a70154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a74fa2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a7015ab00 0x7f2a701553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a74fa2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a700b5ce0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a74fa2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a70072b00 con 0x7f2a700afca0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a74fa2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a700ce000 con 0x7f2a7015ab00 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 0x7f2a7014b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 0x7f2a7014b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44012/0 (socket says 172.21.15.72:44012) 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6f7fe640 1 -- 172.21.15.72:0/426625239 learned_addr learned my addr 172.21.15.72:0/426625239 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6effd640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a700afca0 0x7f2a70154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6ffff640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a7015ab00 0x7f2a701553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6f7fe640 1 -- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a7015ab00 msgr2=0x7f2a701553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6f7fe640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a7015ab00 0x7f2a701553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.977+0000 7f2a6f7fe640 1 -- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a700afca0 msgr2=0x7f2a70154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a6f7fe640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a700afca0 0x7f2a70154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a6f7fe640 1 -- 172.21.15.72:0/426625239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7014b6d0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a6f7fe640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 0x7f2a7014b190 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c410 tx=0x7f2a6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a60017070 con 0x7f2a7006f3a0 2024-03-06T07:25:45.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a60005ce0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7015bf30 con 0x7f2a7006f3a0 2024-03-06T07:25:45.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a600072d0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.978+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7015c4f0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.979+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a70068f10 con 0x7f2a7006f3a0 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.980+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a6001a070 con 0x7f2a7006f3a0 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.980+0000 7f2a57fff640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a5c06b4d0 0x7f2a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.980+0000 7f2a6effd640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a5c06b4d0 0x7f2a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.980+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a600894d0 con 0x7f2a7006f3a0 2024-03-06T07:25:45.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.981+0000 7f2a6effd640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a5c06b4d0 0x7f2a5c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2a64036000 tx=0x7f2a64008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:45.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:45.988+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a60057d10 con 0x7f2a7006f3a0 2024-03-06T07:25:46.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.250+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2a70071e10 con 0x7f2a7006f3a0 2024-03-06T07:25:46.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.254+0000 7f2a57fff640 1 -- 172.21.15.72:0/426625239 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f2a6005bdb0 con 0x7f2a7006f3a0 2024-03-06T07:25:46.256 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:25:46.256 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:25:46.256 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:25:46.256 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:25:46.257 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a5c06b4d0 msgr2=0x7f2a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a5c06b4d0 0x7f2a5c06d990 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2a64036000 tx=0x7f2a64008030 comp rx=0 tx=0).stop 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 msgr2=0x7f2a7014b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 0x7f2a7014b190 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c410 tx=0x7f2a6000c8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 shutdown_connections 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a5c06b4d0 0x7f2a5c06d990 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a7015ab00 0x7f2a701553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a700afca0 0x7f2a70154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 --2- 172.21.15.72:0/426625239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a7006f3a0 0x7f2a7014b190 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 >> 172.21.15.72:0/426625239 conn(0x7f2a700b2580 msgr2=0x7f2a700b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 shutdown_connections 2024-03-06T07:25:46.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.256+0000 7f2a74fa2640 1 -- 172.21.15.72:0/426625239 wait complete. 2024-03-06T07:25:46.269 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:46.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.368+0000 7f61144ed640 1 Processor -- start 2024-03-06T07:25:46.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.368+0000 7f61144ed640 1 -- start start 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.369+0000 7f61144ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 0x7f610c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.369+0000 7f61144ed640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f610c0cf830 0x7f610c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.369+0000 7f61144ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610c0b3430 con 0x7f610c0d27f0 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.369+0000 7f61144ed640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610c0d2c50 con 0x7f610c0cf830 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.369+0000 7f61144ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0b31e0 con 0x7f610c15dfe0 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6112289640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 0x7f610c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6111a88640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f610c0cf830 0x7f610c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6112289640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 0x7f610c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:54274/0 (socket says 172.21.15.72:54274) 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6112a8a640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f610c0d27f0 0x7f610c0d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:34870/0 (socket says 172.21.15.72:34870) 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6112289640 1 -- 172.21.15.72:0/542818967 learned_addr learned my addr 172.21.15.72:0/542818967 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6111a88640 1 -- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 msgr2=0x7f610c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6111a88640 1 --2- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 0x7f610c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6111a88640 1 -- 172.21.15.72:0/542818967 >> v1:172.21.15.146:6789/0 conn(0x7f610c0d27f0 legacy=0x7f610c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6111a88640 1 -- 172.21.15.72:0/542818967 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0b3710 con 0x7f610c0cf830 2024-03-06T07:25:46.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.370+0000 7f6112289640 1 --2- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 0x7f610c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:25:46.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.371+0000 7f6111a88640 1 --2- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0cf830 0x7f610c0d22b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f61080097a0 tx=0x7f610802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66aa64dfdee138a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.371+0000 7f6111287640 1 -- 172.21.15.72:0/542818967 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108035020 con 0x7f610c0cf830 2024-03-06T07:25:46.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.371+0000 7f6111287640 1 -- 172.21.15.72:0/542818967 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6108031810 con 0x7f610c0cf830 2024-03-06T07:25:46.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.371+0000 7f6111287640 1 -- 172.21.15.72:0/542818967 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108031be0 con 0x7f610c0cf830 2024-03-06T07:25:46.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 -- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0cf830 msgr2=0x7f610c0d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 --2- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0cf830 0x7f610c0d22b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f61080097a0 tx=0x7f610802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 -- 172.21.15.72:0/542818967 shutdown_connections 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 --2- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0cf830 0x7f610c0d22b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 --2- 172.21.15.72:0/542818967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c15dfe0 0x7f610c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 -- 172.21.15.72:0/542818967 >> 172.21.15.72:0/542818967 conn(0x7f610c0d3cd0 msgr2=0x7f610c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 -- 172.21.15.72:0/542818967 shutdown_connections 2024-03-06T07:25:46.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.372+0000 7f61144ed640 1 -- 172.21.15.72:0/542818967 wait complete. 2024-03-06T07:25:46.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.373+0000 7f61144ed640 1 Processor -- start 2024-03-06T07:25:46.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.373+0000 7f61144ed640 1 -- start start 2024-03-06T07:25:46.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c0cf830 0x7f610c1f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0d27f0 0x7f610c160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 0x7f610c14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0d1c70 con 0x7f610c15dfe0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0b3d60 con 0x7f610c0d27f0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0d1df0 con 0x7f610c0cf830 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 0x7f610c14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 0x7f610c14be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44014/0 (socket says 172.21.15.72:44014) 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 -- 172.21.15.72:0/10256654 learned_addr learned my addr 172.21.15.72:0/10256654 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6111a88640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0d27f0 0x7f610c160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112289640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c0cf830 0x7f610c1f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 -- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c0cf830 msgr2=0x7f610c1f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c0cf830 0x7f610c1f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 -- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0d27f0 msgr2=0x7f610c160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0d27f0 0x7f610c160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 -- 172.21.15.72:0/10256654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c161360 con 0x7f610c15dfe0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6112a8a640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 0x7f610c14be30 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f610400c780 tx=0x7f610400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104017070 con 0x7f610c15dfe0 2024-03-06T07:25:46.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610400a640 con 0x7f610c15dfe0 2024-03-06T07:25:46.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610c0b6a50 con 0x7f610c15dfe0 2024-03-06T07:25:46.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610400a9f0 con 0x7f610c15dfe0 2024-03-06T07:25:46.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.374+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610c0b7010 con 0x7f610c15dfe0 2024-03-06T07:25:46.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.376+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610c161640 con 0x7f610c15dfe0 2024-03-06T07:25:46.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.376+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f610401a070 con 0x7f610c15dfe0 2024-03-06T07:25:46.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.376+0000 7f6102ffd640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60f006b4d0 0x7f60f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:46.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.376+0000 7f6112289640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60f006b4d0 0x7f60f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:46.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.376+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61040896a0 con 0x7f610c15dfe0 2024-03-06T07:25:46.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.377+0000 7f6112289640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60f006b4d0 0x7f60f006d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f60fc006ca0 tx=0x7f60fc009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:46.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.381+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6104057f60 con 0x7f610c15dfe0 2024-03-06T07:25:46.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.593+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f610c04f330 con 0x7f610c15dfe0 2024-03-06T07:25:46.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.593+0000 7f6102ffd640 1 -- 172.21.15.72:0/10256654 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f610405c000 con 0x7f610c15dfe0 2024-03-06T07:25:46.596 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:46.596 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:46.596 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:46.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60f006b4d0 msgr2=0x7f60f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60f006b4d0 0x7f60f006d990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f60fc006ca0 tx=0x7f60fc009240 comp rx=0 tx=0).stop 2024-03-06T07:25:46.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 msgr2=0x7f610c14be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 0x7f610c14be30 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f610400c780 tx=0x7f610400cc40 comp rx=0 tx=0).stop 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 shutdown_connections 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f60f006b4d0 0x7f60f006d990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f610c15dfe0 0x7f610c14be30 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f610c0d27f0 0x7f610c160bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 --2- 172.21.15.72:0/10256654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f610c0cf830 0x7f610c1f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 >> 172.21.15.72:0/10256654 conn(0x7f610c0d3cd0 msgr2=0x7f610c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 shutdown_connections 2024-03-06T07:25:46.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:46.596+0000 7f61144ed640 1 -- 172.21.15.72:0/10256654 wait complete. 2024-03-06T07:25:47.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 39, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:25:37.310184+0000', 'modified': '2024-03-06T07:25:39.352824+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6987}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6987': {'gid': 6987, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6836/2658331491', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2658331491}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2658331491}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6987, 'qdb_cluster': [6987]} 2024-03-06T07:25:47.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:25:47.015 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:25:47.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.141+0000 7f99a953e640 1 Processor -- start 2024-03-06T07:25:47.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.141+0000 7f99a953e640 1 -- start start 2024-03-06T07:25:47.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a953e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99a415e010 0x7f99a40cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a953e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99a40b3240 0x7f99a40b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a953e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a40b2910 0x7f99a40d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a953e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40b1f40 con 0x7f99a40b3240 2024-03-06T07:25:47.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a953e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a415e3f0 con 0x7f99a415e010 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a953e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40b1c10 con 0x7f99a40b2910 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99a40b3240 0x7f99a40b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99a40b3240 0x7f99a40b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44020/0 (socket says 172.21.15.72:44020) 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 -- 172.21.15.72:0/432396037 learned_addr learned my addr 172.21.15.72:0/432396037 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/432396037 >> v2:172.21.15.146:3300/0 conn(0x7f99a415e010 0x7f99a40cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a37fe640 1 --2- 172.21.15.72:0/432396037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a40b2910 0x7f99a40d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 -- 172.21.15.72:0/432396037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a40b2910 msgr2=0x7f99a40d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 --2- 172.21.15.72:0/432396037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a40b2910 0x7f99a40d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 -- 172.21.15.72:0/432396037 >> v2:172.21.15.146:3300/0 conn(0x7f99a415e010 msgr2=0x7f99a40cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 --2- 172.21.15.72:0/432396037 >> v2:172.21.15.146:3300/0 conn(0x7f99a415e010 0x7f99a40cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a27fc640 1 -- 172.21.15.72:0/432396037 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a406d1b0 con 0x7f99a40b3240 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.142+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/432396037 >> v2:172.21.15.146:3300/0 conn(0x7f99a415e010 0x7f99a40cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.143+0000 7f99a27fc640 1 --2- 172.21.15.72:0/432396037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a40b3240 0x7f99a40b23d0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f99a40715c0 tx=0x7f999802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b20d8ea2689111 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.143+0000 7f99a1ffb640 1 -- 172.21.15.72:0/432396037 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999802e070 con 0x7f99a40b3240 2024-03-06T07:25:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.143+0000 7f99a1ffb640 1 -- 172.21.15.72:0/432396037 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998037070 con 0x7f99a40b3240 2024-03-06T07:25:47.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.143+0000 7f99a1ffb640 1 -- 172.21.15.72:0/432396037 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998031a00 con 0x7f99a40b3240 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 -- 172.21.15.72:0/432396037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a40b3240 msgr2=0x7f99a40b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 --2- 172.21.15.72:0/432396037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a40b3240 0x7f99a40b23d0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f99a40715c0 tx=0x7f999802b880 comp rx=0 tx=0).stop 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 -- 172.21.15.72:0/432396037 shutdown_connections 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 --2- 172.21.15.72:0/432396037 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a40b2910 0x7f99a40d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 --2- 172.21.15.72:0/432396037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a40b3240 0x7f99a40b23d0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 --2- 172.21.15.72:0/432396037 >> v2:172.21.15.146:3300/0 conn(0x7f99a415e010 0x7f99a40cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 -- 172.21.15.72:0/432396037 >> 172.21.15.72:0/432396037 conn(0x7f99a409a9f0 msgr2=0x7f99a409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 -- 172.21.15.72:0/432396037 shutdown_connections 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.144+0000 7f99a953e640 1 -- 172.21.15.72:0/432396037 wait complete. 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.145+0000 7f99a953e640 1 Processor -- start 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 -- start start 2024-03-06T07:25:47.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a415e010 0x7f99a409e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 0x7f99a4152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a4152d60 0x7f99a409b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40cf310 con 0x7f99a4151250 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a406ced0 con 0x7f99a415e010 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a953e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40d2010 con 0x7f99a4152d60 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 0x7f99a4152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 0x7f99a4152820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44034/0 (socket says 172.21.15.72:44034) 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a27fc640 1 -- 172.21.15.72:0/75002229 learned_addr learned my addr 172.21.15.72:0/75002229 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a415e010 0x7f99a409e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.146+0000 7f99a37fe640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a4152d60 0x7f99a409b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a27fc640 1 -- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a4152d60 msgr2=0x7f99a409b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a27fc640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a4152d60 0x7f99a409b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a27fc640 1 -- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a415e010 msgr2=0x7f99a409e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a27fc640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a415e010 0x7f99a409e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a27fc640 1 -- 172.21.15.72:0/75002229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4151780 con 0x7f99a4151250 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a27fc640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 0x7f99a4152820 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9998000c00 tx=0x7f99980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f998b7fe640 1 -- 172.21.15.72:0/75002229 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998032880 con 0x7f99a4151250 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f998b7fe640 1 -- 172.21.15.72:0/75002229 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998032a20 con 0x7f99a4151250 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a4151a60 con 0x7f99a4151250 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f998b7fe640 1 -- 172.21.15.72:0/75002229 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998032e10 con 0x7f99a4151250 2024-03-06T07:25:47.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.147+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a4152020 con 0x7f99a4151250 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.149+0000 7f998b7fe640 1 -- 172.21.15.72:0/75002229 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99980453f0 con 0x7f99a4151250 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.149+0000 7f998b7fe640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f999006b500 0x7f999006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.149+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f999006b500 0x7f999006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.149+0000 7f998b7fe640 1 -- 172.21.15.72:0/75002229 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99980ac2e0 con 0x7f99a4151250 2024-03-06T07:25:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.150+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f999006b500 0x7f999006d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f999400aac0 tx=0x7f9994008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.150+0000 7f99ac204740 1 Processor -- start 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.150+0000 7f99ac204740 1 -- start start 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab65870 0x557b8ab65c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab698c0 0x557b8ab6b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99ac204740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 0x557b8ab95a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99ac204740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8aacf6b0 con 0x557b8ab95600 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8aa78a60 con 0x557b8ab698c0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8aacb750 con 0x557b8ab65870 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 0x557b8ab95a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 0x557b8ab95a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44050/0 (socket says 172.21.15.72:44050) 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 -- 172.21.15.72:0/3997437568 learned_addr learned my addr 172.21.15.72:0/3997437568 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a27fc640 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab698c0 0x557b8ab6b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab65870 0x557b8ab65c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 -- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab65870 msgr2=0x557b8ab65c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab65870 0x557b8ab65c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 -- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab698c0 msgr2=0x557b8ab6b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab698c0 0x557b8ab6b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a37fe640 1 -- 172.21.15.72:0/3997437568 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b8aa96f30 con 0x557b8ab95600 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.151+0000 7f99a27fc640 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab698c0 0x557b8ab6b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99a37fe640 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 0x557b8ab95a80 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f999c00f430 tx=0x7f999c00f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=139196e4a84ce146 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f9988ff9640 1 -- 172.21.15.72:0/3997437568 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c01c070 con 0x557b8ab95600 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f9988ff9640 1 -- 172.21.15.72:0/3997437568 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00d6b0 con 0x557b8ab95600 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f9988ff9640 1 -- 172.21.15.72:0/3997437568 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c00da80 con 0x557b8ab95600 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 -- 172.21.15.72:0/3997437568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 msgr2=0x557b8ab95a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 0x557b8ab95a80 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f999c00f430 tx=0x7f999c00f8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 -- 172.21.15.72:0/3997437568 shutdown_connections 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab95600 0x557b8ab95a80 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab698c0 0x557b8ab6b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 --2- 172.21.15.72:0/3997437568 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab65870 0x557b8ab65c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 -- 172.21.15.72:0/3997437568 >> 172.21.15.72:0/3997437568 conn(0x557b8ab678f0 msgr2=0x557b8ab691b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 -- 172.21.15.72:0/3997437568 shutdown_connections 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.152+0000 7f99ac204740 1 -- 172.21.15.72:0/3997437568 wait complete. 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.153+0000 7f99ac204740 1 -- start start 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.153+0000 7f99ac204740 1 Processor -- start 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8ab899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab92670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8ab69d50 con 0x557b8ab97600 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8aa8bea0 con 0x557b8abb1380 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8ab6a030 con 0x557b8ab95600 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 10 client.0 _ll_register_callbacks cb 0x557b8aa8b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab92670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab92670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44054/0 (socket says 172.21.15.72:44054) 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 -- 172.21.15.72:0/2173166271 learned_addr learned my addr 172.21.15.72:0/2173166271 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a27fc640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8ab899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 -- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 msgr2=0x557b8ab899d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8ab899d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 -- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 msgr2=0x557b8ab8b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab8b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 -- 172.21.15.72:0/2173166271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b8aa8c270 con 0x557b8ab97600 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a27fc640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99a37fe640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab92670 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f999c01a040 tx=0x7f999c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de3ca2eb44edee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f9981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f9981ffb640 1 -- 172.21.15.72:0/2173166271 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c027060 con 0x557b8ab97600 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f9981ffb640 1 -- 172.21.15.72:0/2173166271 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f999c003b90 con 0x557b8ab97600 2024-03-06T07:25:47.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.154+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f999c00d6b0 con 0x557b8ab97600 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f9981ffb640 1 -- 172.21.15.72:0/2173166271 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f999c02d050 con 0x557b8ab97600 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 10 client.7146 fetch_fsmap learned FSMap version 39 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x557b8a7e42d0 con 0x557b8ab97600 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f9981ffb640 1 -- 172.21.15.72:0/2173166271 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c00d6b0 con 0x557b8ab97600 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f9981ffb640 1 -- 172.21.15.72:0/2173166271 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 39) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f999c030020 con 0x557b8ab97600 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 10 client.7146 fetch_fsmap finished waiting for FSMap version 39 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 10 client.7146 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 20 client.7146 populate_metadata read hostname 'smithi072' 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ab8a5e0 0x557b8ab8a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 4 client.7146 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 --> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b8aadcbf0 con 0x557b8ab8a5e0 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ab8a5e0 0x557b8ab8a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f9981ffb640 10 client.7146 ms_handle_connect on v2:172.21.15.72:6834/2658331491 2024-03-06T07:25:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.155+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ab8a5e0 0x557b8ab8a9c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6f98a3215dd57ec server_cookie=c765120adb75f38d in_seq=0 out_seq=0 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f9981ffb640 1 -- 172.21.15.72:0/2173166271 <== mds.0 v2:172.21.15.72:6834/2658331491 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557b8aadcbf0 con 0x557b8ab8a5e0 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f9981ffb640 10 client.7146 handle_command_reply: tid=0 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 client.7146 shutdown 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 msgr2=0x557b8ab92670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab92670 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f999c01a040 tx=0x7f999c004ab0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 shutdown_connections 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ab8a5e0 0x557b8ab8a9c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab92670 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f9981ffb640 0 client.7146 ms_handle_reset on v2:172.21.15.72:6834/2658331491 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 --2- 172.21.15.72:0/2173166271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8ab899d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 >> 172.21.15.72:0/2173166271 conn(0x557b8ab65870 msgr2=0x557b8ab935c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 shutdown_connections 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 1 -- 172.21.15.72:0/2173166271 wait complete. 2024-03-06T07:25:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.156+0000 7f99ac204740 20 client.7146 trim_cache size 0 max 16384 2024-03-06T07:25:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 Processor -- start 2024-03-06T07:25:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 -- start start 2024-03-06T07:25:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab95600 0x557b8ab66080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab94590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8abb1380 0x557b8ab94ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41ff310 con 0x557b8ab97600 2024-03-06T07:25:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a40b6670 con 0x557b8ab95600 2024-03-06T07:25:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.178+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a4153ed0 con 0x557b8abb1380 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab94590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab94590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44068/0 (socket says 172.21.15.72:44068) 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 -- 172.21.15.72:0/3970057234 learned_addr learned my addr 172.21.15.72:0/3970057234 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 -- 172.21.15.72:0/3970057234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8abb1380 msgr2=0x557b8ab94ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8abb1380 0x557b8ab94ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 -- 172.21.15.72:0/3970057234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab95600 msgr2=0x557b8ab66080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab95600 0x557b8ab66080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 -- 172.21.15.72:0/3970057234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b8a8ae6b0 con 0x557b8ab97600 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f99a27fc640 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab94590 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f9998077070 tx=0x7f9998081040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21db43fcfd5f075 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f9983fff640 1 -- 172.21.15.72:0/3970057234 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999807ecb0 con 0x557b8ab97600 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f9983fff640 1 -- 172.21.15.72:0/3970057234 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998037040 con 0x557b8ab97600 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.179+0000 7f9983fff640 1 -- 172.21.15.72:0/3970057234 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998083270 con 0x557b8ab97600 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 -- 172.21.15.72:0/3970057234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 msgr2=0x557b8ab94590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab94590 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f9998077070 tx=0x7f9998081040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 -- 172.21.15.72:0/3970057234 shutdown_connections 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8abb1380 0x557b8ab94ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab94590 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 --2- 172.21.15.72:0/3970057234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8ab95600 0x557b8ab66080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 -- 172.21.15.72:0/3970057234 >> 172.21.15.72:0/3970057234 conn(0x557b8ab65870 msgr2=0x557b8ab8cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 -- 172.21.15.72:0/3970057234 shutdown_connections 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 -- 172.21.15.72:0/3970057234 wait complete. 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 -- start start 2024-03-06T07:25:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.180+0000 7f99ac204740 1 Processor -- start 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8abd3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab69f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab6a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8ab91e70 con 0x557b8ab97600 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8ab91610 con 0x557b8abb1380 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8ab8e320 con 0x557b8ab95600 2024-03-06T07:25:47.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99ac204740 10 client.0 _ll_register_callbacks cb 0x7f999c00db90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab69f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab69f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44076/0 (socket says 172.21.15.72:44076) 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 -- 172.21.15.72:0/2448249831 learned_addr learned my addr 172.21.15.72:0/2448249831 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a37fe640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab6a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8abd3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 -- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 msgr2=0x557b8abd3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8abd3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 -- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 msgr2=0x557b8ab6a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab6a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 -- 172.21.15.72:0/2448249831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557b8ab922b0 con 0x557b8ab97600 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f99a27fc640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab69f60 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f999802b8c0 tx=0x7f9998047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d30f1b05c09999 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.181+0000 7f9981ffb640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 1 -- 172.21.15.72:0/2448249831 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998087070 con 0x557b8ab97600 2024-03-06T07:25:47.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 1 -- 172.21.15.72:0/2448249831 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f999807fbd0 con 0x557b8ab97600 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9998037040 con 0x557b8ab97600 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 1 -- 172.21.15.72:0/2448249831 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99980aa050 con 0x557b8ab97600 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 10 client.7152 fetch_fsmap learned FSMap version 39 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x557b8ab6b2a0 con 0x557b8ab97600 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 1 -- 172.21.15.72:0/2448249831 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998037040 con 0x557b8ab97600 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 1 -- 172.21.15.72:0/2448249831 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 39) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f9998082de0 con 0x557b8ab97600 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 10 client.7152 fetch_fsmap finished waiting for FSMap version 39 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 10 client.7152 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 20 client.7152 populate_metadata read hostname 'smithi072' 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ac21670 0x557b8ac21a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 4 client.7152 mds_command: new command op to 6987 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 --> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557b8aad6f40 con 0x557b8ac21670 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ac21670 0x557b8ac21a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f99a2ffd640 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ac21670 0x557b8ac21a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daad80f57f285843 server_cookie=3c8d26703b796f6e in_seq=0 out_seq=0 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 10 client.7152 ms_handle_connect on v2:172.21.15.72:6834/2658331491 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 1 -- 172.21.15.72:0/2448249831 <== mds.0 v2:172.21.15.72:6834/2658331491 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x557b8aad6f40 con 0x557b8ac21670 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.182+0000 7f9981ffb640 10 client.7152 handle_command_reply: tid=0 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 client.7152 shutdown 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 msgr2=0x557b8ab69f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab69f60 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f999802b8c0 tx=0x7f9998047040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 shutdown_connections 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:6834/2658331491,v1:172.21.15.72:6836/2658331491] conn(0x557b8ac21670 0x557b8ac21a50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x557b8abb1380 0x557b8ab6a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x557b8ab97600 0x557b8ab69f60 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f9981ffb640 0 client.7152 ms_handle_reset on v2:172.21.15.72:6834/2658331491 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 --2- 172.21.15.72:0/2448249831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x557b8ab95600 0x557b8abd3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 >> 172.21.15.72:0/2448249831 conn(0x557b8ab65870 msgr2=0x557b8abb0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 shutdown_connections 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 1 -- 172.21.15.72:0/2448249831 wait complete. 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99ac204740 20 client.7152 trim_cache size 0 max 16384 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6987, 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:25:47.193 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 9.226853083, 2024-03-06T07:25:47.194 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 39, 2024-03-06T07:25:47.194 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 83, 2024-03-06T07:25:47.194 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 83, 2024-03-06T07:25:47.194 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 14.285575034000001 2024-03-06T07:25:47.194 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f999006b500 msgr2=0x7f999006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f999006b500 0x7f999006d9c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f999400aac0 tx=0x7f9994008040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 msgr2=0x7f99a4152820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 0x7f99a4152820 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9998000c00 tx=0x7f99980046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 shutdown_connections 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f999006b500 0x7f999006d9c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a4152d60 0x7f99a409b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99a4151250 0x7f99a4152820 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 --2- 172.21.15.72:0/75002229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a415e010 0x7f99a409e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 >> 172.21.15.72:0/75002229 conn(0x7f99a409a9f0 msgr2=0x7f99a40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 shutdown_connections 2024-03-06T07:25:47.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.183+0000 7f99a953e640 1 -- 172.21.15.72:0/75002229 wait complete. 2024-03-06T07:25:47.197 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:25:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 Processor -- start 2024-03-06T07:25:47.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 -- start start 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff90c0d22b0 0x7ff90c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff90c0b41f0 0x7ff90c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c0b3430 con 0x7ff90c15dfe0 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c0d2c50 con 0x7ff90c0d22b0 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.296+0000 7ff911fc9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c0b31e0 con 0x7ff90c0b41f0 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff90c0d22b0 0x7ff90c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff90c15dfe0 0x7ff90c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43100/0 (socket says 172.21.15.72:43100) 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90b7fe640 1 -- 172.21.15.72:0/2067846098 learned_addr learned my addr 172.21.15.72:0/2067846098 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90bfff640 1 --2- 172.21.15.72:0/2067846098 >> v2:172.21.15.146:3300/0 conn(0x7ff90c0b41f0 0x7ff90c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90affd640 1 -- 172.21.15.72:0/2067846098 >> v2:172.21.15.146:3300/0 conn(0x7ff90c0b41f0 msgr2=0x7ff90c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:47.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90affd640 1 --2- 172.21.15.72:0/2067846098 >> v2:172.21.15.146:3300/0 conn(0x7ff90c0b41f0 0x7ff90c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90affd640 1 -- 172.21.15.72:0/2067846098 >> v1:172.21.15.72:6789/0 conn(0x7ff90c15dfe0 legacy=0x7ff90c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90affd640 1 -- 172.21.15.72:0/2067846098 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c0b3710 con 0x7ff90c0d22b0 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.297+0000 7ff90a7fc640 1 -- 172.21.15.72:0/2067846098 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284100979 0 0) 0x7ff90c0b3430 con 0x7ff90c15dfe0 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff90affd640 1 --2- 172.21.15.72:0/2067846098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c0d22b0 0x7ff90c0d2710 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff9000097a0 tx=0x7ff90002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95fe2b286bca09e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff90a7fc640 1 -- 172.21.15.72:0/2067846098 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900035020 con 0x7ff90c0d22b0 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff90a7fc640 1 -- 172.21.15.72:0/2067846098 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff900031830 con 0x7ff90c0d22b0 2024-03-06T07:25:47.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff90a7fc640 1 -- 172.21.15.72:0/2067846098 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900031c00 con 0x7ff90c0d22b0 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 -- 172.21.15.72:0/2067846098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c0d22b0 msgr2=0x7ff90c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 --2- 172.21.15.72:0/2067846098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c0d22b0 0x7ff90c0d2710 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff9000097a0 tx=0x7ff90002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 -- 172.21.15.72:0/2067846098 shutdown_connections 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 --2- 172.21.15.72:0/2067846098 >> v2:172.21.15.146:3300/0 conn(0x7ff90c0b41f0 0x7ff90c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 --2- 172.21.15.72:0/2067846098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c0d22b0 0x7ff90c0d2710 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 -- 172.21.15.72:0/2067846098 >> 172.21.15.72:0/2067846098 conn(0x7ff90c0d3cd0 msgr2=0x7ff90c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 -- 172.21.15.72:0/2067846098 shutdown_connections 2024-03-06T07:25:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.298+0000 7ff911fc9640 1 -- 172.21.15.72:0/2067846098 wait complete. 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.299+0000 7ff911fc9640 1 Processor -- start 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 -- start start 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff90c0b41f0 0x7ff90c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff90c0d22b0 0x7ff90c1530f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 0x7ff90c0b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0cfa00 con 0x7ff90c15dfe0 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0b3d60 con 0x7ff90c0d22b0 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff911fc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0d1c70 con 0x7ff90c0b41f0 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff90bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 0x7ff90c0b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff90bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 0x7ff90c0b8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44082/0 (socket says 172.21.15.72:44082) 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.300+0000 7ff90bfff640 1 -- 172.21.15.72:0/1182313759 learned_addr learned my addr 172.21.15.72:0/1182313759 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90affd640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff90c0d22b0 0x7ff90c1530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90b7fe640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff90c0b41f0 0x7ff90c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90bfff640 1 -- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff90c0b41f0 msgr2=0x7ff90c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90bfff640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff90c0b41f0 0x7ff90c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90bfff640 1 -- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff90c0d22b0 msgr2=0x7ff90c1530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90bfff640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff90c0d22b0 0x7ff90c1530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90bfff640 1 -- 172.21.15.72:0/1182313759 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c15e900 con 0x7ff90c15dfe0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90affd640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff90c0d22b0 0x7ff90c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90b7fe640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff90c0b41f0 0x7ff90c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff90bfff640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 0x7ff90c0b8530 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff904007e40 tx=0x7ff90400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904014620 con 0x7ff90c15dfe0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff904019070 con 0x7ff90c15dfe0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c15eba0 con 0x7ff90c15dfe0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904014a20 con 0x7ff90c15dfe0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.301+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c15f160 con 0x7ff90c15dfe0 2024-03-06T07:25:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.302+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c04f330 con 0x7ff90c15dfe0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.303+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff904014bc0 con 0x7ff90c15dfe0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.303+0000 7ff8f3fff640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8f806b4d0 0x7ff8f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.303+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff90408ab60 con 0x7ff90c15dfe0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.303+0000 7ff90b7fe640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8f806b4d0 0x7ff8f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:47.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.304+0000 7ff90b7fe640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8f806b4d0 0x7ff8f806d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc000f60 tx=0x7ff8fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.308+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff904059960 con 0x7ff90c15dfe0 2024-03-06T07:25:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.521+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff90c0d1910 con 0x7ff90c15dfe0 2024-03-06T07:25:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.521+0000 7ff8f3fff640 1 -- 172.21.15.72:0/1182313759 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v39) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff904059260 con 0x7ff90c15dfe0 2024-03-06T07:25:47.524 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:25:47.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8f806b4d0 msgr2=0x7ff8f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8f806b4d0 0x7ff8f806d990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc000f60 tx=0x7ff8fc009040 comp rx=0 tx=0).stop 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 msgr2=0x7ff90c0b8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 0x7ff90c0b8530 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7ff904007e40 tx=0x7ff90400e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 shutdown_connections 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff8f806b4d0 0x7ff8f806d990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff90c15dfe0 0x7ff90c0b8530 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff90c0d22b0 0x7ff90c1530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 --2- 172.21.15.72:0/1182313759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff90c0b41f0 0x7ff90c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 >> 172.21.15.72:0/1182313759 conn(0x7ff90c0d3cd0 msgr2=0x7ff90c15ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 shutdown_connections 2024-03-06T07:25:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:47.523+0000 7ff911fc9640 1 -- 172.21.15.72:0/1182313759 wait complete. 2024-03-06T07:25:47.536 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:25:47.536 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:25:47.536 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:25:47.537 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:25:47.537 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:47.537 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:25:47.537 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:25:47.537 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:25:47.537 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:47.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:47.537 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.552 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42448sec preferred_lft 42448sec 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.553 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:25:47.554 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:25:47.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:47.555 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:25:47.618 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:25:47.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:47.620 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:25:47.674 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:25:47.674 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:25:47.675 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:25:47.675 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:25:47.675 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:25:47.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:47.675 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:25:47.675 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:25:47.675 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:25:47.675 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:25:47.675 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:25:47.675 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:25:47.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:47.770 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:25:47.785 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:25:47.785 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:25:47.785 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:47.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:47.785 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:25:47.852 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:25:47.852 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:47.852 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:47.906 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:25:47.906 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:25:47.906 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:47.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:25:48.027 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:25:48.083 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:48.084 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:48.085 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:48.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.086 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:25:48.141 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:25:48.141 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:25:48.183 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:25:48.212 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:25:48.267 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:25:48.268 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:25:48.269 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:25:48.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.270 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:25:48.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:25:48.284+0000 7f3fb2849480 -1 init, newargv = 0x55e083f86660 newargc=15 2024-03-06T07:25:48.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[65540]: starting ceph client 2024-03-06T07:25:48.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[65540]: starting fuse 2024-03-06T07:25:48.302 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:25:48.302 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:25:48.303 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:25:48.304 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:25:48.423 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.65540.asok 2024-03-06T07:25:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:25:48.433 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.65540.asok 2024-03-06T07:25:48.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.433 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.65540.asok status 2024-03-06T07:25:48.572 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "65540", 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 7170, 2024-03-06T07:25:48.573 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 7170 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2576857850 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2576857850 2024-03-06T07:25:48.574 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.7170 192.168.0.1:0/2576857850", 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/2576857850", 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 39, 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 83, 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:25:48.575 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:25:48.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.581 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:25:48.581 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:48.596 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:25:48.596 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:48.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.596 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:48.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.664 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-03-06T07:25:48.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.724 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-03-06T07:25:48.787 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:48.787 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:48.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:48.843 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-03-06T07:25:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:35+0 records in 2024-03-06T07:25:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:35+0 records out 2024-03-06T07:25:49.066 INFO:teuthology.orchestra.run.smithi072.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.164691 s, 223 MB/s 2024-03-06T07:25:49.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:49.067 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-03-06T07:25:49.085 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.248+0000 7f2feac5c640 1 Processor -- start 2024-03-06T07:25:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.248+0000 7f2feac5c640 1 -- start start 2024-03-06T07:25:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2feac5c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2fe40cd7a0 0x7f2fe40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2feac5c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe40cc980 con 0x7f2fe41424f0 2024-03-06T07:25:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2feac5c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe40ccc60 con 0x7f2fe406da40 2024-03-06T07:25:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2feac5c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe40cd220 con 0x7f2fe40cd7a0 2024-03-06T07:25:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe91f9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2fe40cd7a0 0x7f2fe40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe89f8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2fe41424f0 0x7f2fe406d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43108/0 (socket says 172.21.15.72:43108) 2024-03-06T07:25:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe91f9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2fe40cd7a0 0x7f2fe40cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44084/0 (socket says 172.21.15.72:44084) 2024-03-06T07:25:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe89f8640 1 -- 172.21.15.72:0/1103525511 learned_addr learned my addr 172.21.15.72:0/1103525511 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe91f9640 1 -- 172.21.15.72:0/1103525511 >> v1:172.21.15.146:6789/0 conn(0x7f2fe406da40 legacy=0x7f2fe4071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe91f9640 1 -- 172.21.15.72:0/1103525511 >> v1:172.21.15.72:6789/0 conn(0x7f2fe41424f0 legacy=0x7f2fe406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe37fe640 1 -- 172.21.15.72:0/1103525511 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459299318 0 0) 0x7f2fe40cc980 con 0x7f2fe41424f0 2024-03-06T07:25:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.249+0000 7f2fe91f9640 1 -- 172.21.15.72:0/1103525511 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40ccf40 con 0x7f2fe40cd7a0 2024-03-06T07:25:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.250+0000 7f2fe91f9640 1 --2- 172.21.15.72:0/1103525511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe40cd7a0 0x7f2fe40cc2f0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1121e23e4a91b4cb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.250+0000 7f2fe37fe640 1 -- 172.21.15.72:0/1103525511 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc039070 con 0x7f2fe40cd7a0 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.250+0000 7f2fe37fe640 1 -- 172.21.15.72:0/1103525511 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc02cc60 con 0x7f2fe40cd7a0 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.250+0000 7f2fe37fe640 1 -- 172.21.15.72:0/1103525511 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc034660 con 0x7f2fe40cd7a0 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.250+0000 7f2feac5c640 1 -- 172.21.15.72:0/1103525511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe40cd7a0 msgr2=0x7f2fe40cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.250+0000 7f2feac5c640 1 --2- 172.21.15.72:0/1103525511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe40cd7a0 0x7f2fe40cc2f0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc02beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.251+0000 7f2feac5c640 1 -- 172.21.15.72:0/1103525511 shutdown_connections 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.251+0000 7f2feac5c640 1 --2- 172.21.15.72:0/1103525511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe40cd7a0 0x7f2fe40cc2f0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.251+0000 7f2feac5c640 1 -- 172.21.15.72:0/1103525511 >> 172.21.15.72:0/1103525511 conn(0x7f2fe40cf8a0 msgr2=0x7f2fe40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.251+0000 7f2feac5c640 1 -- 172.21.15.72:0/1103525511 shutdown_connections 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.251+0000 7f2feac5c640 1 -- 172.21.15.72:0/1103525511 wait complete. 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.252+0000 7f2feac5c640 1 Processor -- start 2024-03-06T07:25:49.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.252+0000 7f2feac5c640 1 -- start start 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 0x7f2fe414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe40cd7a0 0x7f2fe41f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fe41424f0 0x7f2fe40ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe415d070 con 0x7f2fe406da40 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe415cef0 con 0x7f2fe41424f0 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe415d1f0 con 0x7f2fe40cd7a0 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 0x7f2fe414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 0x7f2fe414f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44100/0 (socket says 172.21.15.72:44100) 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 -- 172.21.15.72:0/2196398437 learned_addr learned my addr 172.21.15.72:0/2196398437 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe91f9640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fe41424f0 0x7f2fe40ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe3fff640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe40cd7a0 0x7f2fe41f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 -- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe40cd7a0 msgr2=0x7f2fe41f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe40cd7a0 0x7f2fe41f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 -- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fe41424f0 msgr2=0x7f2fe40ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fe41424f0 0x7f2fe40ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 -- 172.21.15.72:0/2196398437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe40a9f60 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe91f9640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fe41424f0 0x7f2fe40ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe89f8640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 0x7f2fe414f280 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800c430 tx=0x7f2fd800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd80074e0 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd80047e0 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8004bd0 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe40aa240 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.253+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe40aa800 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.254+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe404f330 con 0x7f2fe406da40 2024-03-06T07:25:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.255+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2fd8011070 con 0x7f2fe406da40 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.255+0000 7f2fe17fa640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.255+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fd8089ec0 con 0x7f2fe406da40 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.255+0000 7f2fe3fff640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.256+0000 7f2fe3fff640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2fc806b4d0 0x7f2fc806d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40ce170 tx=0x7f2fd0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.260+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2fd8058d40 con 0x7f2fe406da40 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.474+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fe40d0620 con 0x7f2fe406da40 2024-03-06T07:25:49.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.474+0000 7f2fe17fa640 1 -- 172.21.15.72:0/2196398437 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f2fd8058640 con 0x7f2fe406da40 2024-03-06T07:25:49.477 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:49.477 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:49.477 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.476+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2fc806b4d0 msgr2=0x7f2fc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.476+0000 7f2feac5c640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2fc806b4d0 0x7f2fc806d990 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2fe40ce170 tx=0x7f2fd0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.476+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 msgr2=0x7f2fe414f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.476+0000 7f2feac5c640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 0x7f2fe414f280 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800c430 tx=0x7f2fd800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 shutdown_connections 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2fc806b4d0 0x7f2fc806d990 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2fe41424f0 0x7f2fe40ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2fe40cd7a0 0x7f2fe41f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 --2- 172.21.15.72:0/2196398437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fe406da40 0x7f2fe414f280 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 >> 172.21.15.72:0/2196398437 conn(0x7f2fe40cf8a0 msgr2=0x7f2fe406c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 shutdown_connections 2024-03-06T07:25:49.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.477+0000 7f2feac5c640 1 -- 172.21.15.72:0/2196398437 wait complete. 2024-03-06T07:25:49.490 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-03-06T07:25:49.624 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:25:49.624 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:25:49.625 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.727+0000 7f1c290af640 1 Processor -- start 2024-03-06T07:25:49.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.727+0000 7f1c290af640 1 -- start start 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c290af640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c240b4460 0x7f1c240d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c290af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240d22c0 con 0x7f1c24162970 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c290af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240d25a0 con 0x7f1c24161cf0 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c290af640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c24161b00 con 0x7f1c240b4460 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c1bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1c24162970 0x7f1c240b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43112/0 (socket says 172.21.15.72:43112) 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c1bfff640 1 -- 172.21.15.72:0/1885271405 learned_addr learned my addr 172.21.15.72:0/1885271405 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1885271405 >> v2:172.21.15.146:3300/0 conn(0x7f1c240b4460 0x7f1c240d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383057472 0 0) 0x7f1c240d22c0 con 0x7f1c24162970 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c10003520 con 0x7f1c24162970 2024-03-06T07:25:49.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935733241 0 0) 0x7f1c240d25a0 con 0x7f1c24161cf0 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c240d22c0 con 0x7f1c24161cf0 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626039040 0 0) 0x7f1c10003520 con 0x7f1c24162970 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 >> v2:172.21.15.146:3300/0 conn(0x7f1c240b4460 msgr2=0x7f1c240d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 --2- 172.21.15.72:0/1885271405 >> v2:172.21.15.146:3300/0 conn(0x7f1c240b4460 0x7f1c240d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 >> v1:172.21.15.146:6789/0 conn(0x7f1c24161cf0 legacy=0x7f1c240cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240d2880 con 0x7f1c24162970 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1c0c001670 con 0x7f1c24162970 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 >> v1:172.21.15.72:6789/0 conn(0x7f1c24162970 legacy=0x7f1c240b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.728+0000 7f1c2259c640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c10003b40 0x7f1c10003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2259c640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10004800 0x7f1c10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2259c640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100051c0 0x7f1c1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c24161b00 con 0x7f1c10003b40 2024-03-06T07:25:49.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1000c040 con 0x7f1c100051c0 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1000c240 con 0x7f1c10004800 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c10003b40 0x7f1c10003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2359e640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100051c0 0x7f1c1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c1bfff640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10004800 0x7f1c10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c22d9d640 1 -- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10004800 msgr2=0x7f1c10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10004800 0x7f1c10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c22d9d640 1 -- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100051c0 msgr2=0x7f1c1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100051c0 0x7f1c1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c22d9d640 1 -- 172.21.15.72:0/1885271405 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c10003520 con 0x7f1c10003b40 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2359e640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c100051c0 0x7f1c1000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.729+0000 7f1c2359e640 1 -- 172.21.15.72:0/1885271405 reap_dead start 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c10003b40 0x7f1c10003f40 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009ac0 tx=0x7f1c10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dccb497e59626d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14039070 con 0x7f1c10003b40 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1402c6b0 con 0x7f1c10003b40 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c2259c640 1 -- 172.21.15.72:0/1885271405 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14009690 con 0x7f1c10003b40 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 -- 172.21.15.72:0/1885271405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c10003b40 msgr2=0x7f1c10003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c10003b40 0x7f1c10003f40 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009ac0 tx=0x7f1c10005830 comp rx=0 tx=0).stop 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 -- 172.21.15.72:0/1885271405 shutdown_connections 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 --2- 172.21.15.72:0/1885271405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c10003b40 0x7f1c10003f40 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 -- 172.21.15.72:0/1885271405 >> 172.21.15.72:0/1885271405 conn(0x7f1c240d3ce0 msgr2=0x7f1c240d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 -- 172.21.15.72:0/1885271405 shutdown_connections 2024-03-06T07:25:49.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.730+0000 7f1c290af640 1 -- 172.21.15.72:0/1885271405 wait complete. 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.732+0000 7f1c290af640 1 Processor -- start 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.732+0000 7f1c290af640 1 -- start start 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.733+0000 7f1c290af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10003b40 0x7f1c24156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.733+0000 7f1c290af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 0x7f1c240b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c290af640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c240c7810 0x7f1c240c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c290af640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240ce950 con 0x7f1c240cd230 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c290af640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240cecc0 con 0x7f1c240c7810 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c290af640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c240b4840 con 0x7f1c10003b40 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 0x7f1c240b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 0x7f1c240b7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44112/0 (socket says 172.21.15.72:44112) 2024-03-06T07:25:49.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 -- 172.21.15.72:0/1621924626 learned_addr learned my addr 172.21.15.72:0/1621924626 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c2359e640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c240c7810 0x7f1c240c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10003b40 0x7f1c24156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 -- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10003b40 msgr2=0x7f1c24156230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10003b40 0x7f1c24156230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 -- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c240c7810 msgr2=0x7f1c240c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c240c7810 0x7f1c240c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 -- 172.21.15.72:0/1621924626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240b9420 con 0x7f1c240cd230 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1bfff640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 0x7f1c240b7b70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00e760 tx=0x7f1c0c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00c8d0 con 0x7f1c240cd230 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c018070 con 0x7f1c240cd230 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00ccd0 con 0x7f1c240cd230 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c240b80b0 con 0x7f1c240cd230 2024-03-06T07:25:49.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.734+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c240b8660 con 0x7f1c240cd230 2024-03-06T07:25:49.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.735+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2404f330 con 0x7f1c240cd230 2024-03-06T07:25:49.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.736+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c0c013420 con 0x7f1c240cd230 2024-03-06T07:25:49.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.736+0000 7f1c1b7fe640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c0406b500 0x7f1c0406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:49.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.736+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c0c089da0 con 0x7f1c240cd230 2024-03-06T07:25:49.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.736+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c0406b500 0x7f1c0406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:49.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.737+0000 7f1c22d9d640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c0406b500 0x7f1c0406d9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1c14036000 tx=0x7f1c14030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:49.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.741+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c0c058c20 con 0x7f1c240cd230 2024-03-06T07:25:49.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.987+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f1c2406b180 con 0x7f1c240cd230 2024-03-06T07:25:49.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.987+0000 7f1c1b7fe640 1 -- 172.21.15.72:0/1621924626 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1651 (secure 0 0 0) 0x7f1c0c058520 con 0x7f1c240cd230 2024-03-06T07:25:49.988 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:49.988 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_bytes": 773094113280, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_avail_bytes": 772829270016, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_bytes": 264843264, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_bytes": 264843264, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_ratio": 0.00034257571678608656, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "num_osds": 8, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "num_per_pool_osds": 8, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "num_per_pool_omap_osds": 8 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "stats_by_class": { 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "ssd": { 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_bytes": 773094113280, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_avail_bytes": 772829270016, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_bytes": 264843264, 2024-03-06T07:25:49.989 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_bytes": 264843264, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "total_used_raw_ratio": 0.00034257571678608656 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "pools": [ 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "name": ".mgr", 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 1, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "stored": 590368, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "objects": 2, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "kb_used": 1160, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "bytes_used": 1187840, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "percent_used": 1.61793116149056e-06, 2024-03-06T07:25:49.990 INFO:teuthology.orchestra.run.smithi072.stdout: "max_avail": 367085486080 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephfs_metadata", 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 12, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "stored": 2410, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "objects": 22, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "kb_used": 64, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "bytes_used": 65536, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "percent_used": 8.9265306257857446e-08, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: "max_avail": 367085486080 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:25:49.991 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "name": "cephfs_data", 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 13, 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "stats": { 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "stored": 0, 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "objects": 0, 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "kb_used": 0, 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "bytes_used": 0, 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "percent_used": 0, 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: "max_avail": 367085486080 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:25:49.992 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:25:49.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c0406b500 msgr2=0x7f1c0406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c0406b500 0x7f1c0406d9c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1c14036000 tx=0x7f1c14030000 comp rx=0 tx=0).stop 2024-03-06T07:25:49.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 msgr2=0x7f1c240b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 0x7f1c240b7b70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00e760 tx=0x7f1c0c00ec20 comp rx=0 tx=0).stop 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 shutdown_connections 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c0406b500 0x7f1c0406d9c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c240c7810 0x7f1c240c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c240cd230 0x7f1c240b7b70 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 --2- 172.21.15.72:0/1621924626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c10003b40 0x7f1c24156230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 >> 172.21.15.72:0/1621924626 conn(0x7f1c240d3ce0 msgr2=0x7f1c24161250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 shutdown_connections 2024-03-06T07:25:49.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:49.989+0000 7f1c290af640 1 -- 172.21.15.72:0/1621924626 wait complete. 2024-03-06T07:25:50.002 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-03-06T07:25:50.002 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:25:50.002 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:50.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:25:50.088 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:25:50.088 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:50.089 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:50.104 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:50.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.242+0000 7f41242c2640 1 Processor -- start 2024-03-06T07:25:50.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.242+0000 7f41242c2640 1 -- start start 2024-03-06T07:25:50.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f41242c2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f411c15dfe0 0x7f411c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f41242c2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f411c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f41242c2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0b3430 con 0x7f411c0cf830 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f41242c2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411c0b31e0 con 0x7f411c15dfe0 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f41242c2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c0b3d60 con 0x7f411c0d29c0 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412185d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f411c0cf830 0x7f411c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43126/0 (socket says 172.21.15.72:43126) 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412185d640 1 -- 172.21.15.72:0/2815566391 learned_addr learned my addr 172.21.15.72:0/2815566391 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193685910 0 0) 0x7f411c0b3430 con 0x7f411c0cf830 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4108003520 con 0x7f411c0cf830 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412205e640 1 --2- 172.21.15.72:0/2815566391 >> v2:172.21.15.146:3300/0 conn(0x7f411c15dfe0 0x7f411c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412285f640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f411c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 152023673 0 0) 0x7f4108003520 con 0x7f411c0cf830 2024-03-06T07:25:50.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 msgr2=0x7f411c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f411c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 >> v2:172.21.15.146:3300/0 conn(0x7f411c15dfe0 msgr2=0x7f411c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 --2- 172.21.15.72:0/2815566391 >> v2:172.21.15.146:3300/0 conn(0x7f411c15dfe0 0x7f411c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c0b3710 con 0x7f411c0cf830 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f410c001670 con 0x7f411c0cf830 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 >> v1:172.21.15.72:6789/0 conn(0x7f411c0cf830 legacy=0x7f411c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41080039b0 0x7f4108003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f4108004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108004d40 0x7f41080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c0b31e0 con 0x7f41080039b0 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c0b3d60 con 0x7f4108004d40 2024-03-06T07:25:50.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.243+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f410800baa0 con 0x7f411c0d29c0 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412205e640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41080039b0 0x7f4108003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412285f640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108004d40 0x7f41080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412185d640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f4108004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412205e640 1 -- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 msgr2=0x7f4108004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412205e640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f4108004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412205e640 1 -- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108004d40 msgr2=0x7f41080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412205e640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108004d40 0x7f41080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412205e640 1 -- 172.21.15.72:0/2815566391 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108003520 con 0x7f41080039b0 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412185d640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c0d29c0 0x7f4108004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.244+0000 7f412285f640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4108004d40 0x7f41080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f412205e640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41080039b0 0x7f4108003dd0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f411800ba30 tx=0x7f4108005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5625e866cce341bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4118036020 con 0x7f41080039b0 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118032660 con 0x7f41080039b0 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f412105c640 1 -- 172.21.15.72:0/2815566391 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411800b600 con 0x7f41080039b0 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f41242c2640 1 -- 172.21.15.72:0/2815566391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41080039b0 msgr2=0x7f4108003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f41242c2640 1 --2- 172.21.15.72:0/2815566391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f41080039b0 0x7f4108003dd0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f411800ba30 tx=0x7f4108005830 comp rx=0 tx=0).stop 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f412285f640 1 -- 172.21.15.72:0/2815566391 reap_dead start 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f41242c2640 1 -- 172.21.15.72:0/2815566391 shutdown_connections 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f41242c2640 1 -- 172.21.15.72:0/2815566391 >> 172.21.15.72:0/2815566391 conn(0x7f411c0d3cd0 msgr2=0x7f411c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f41242c2640 1 -- 172.21.15.72:0/2815566391 shutdown_connections 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.245+0000 7f41242c2640 1 -- 172.21.15.72:0/2815566391 wait complete. 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.246+0000 7f41242c2640 1 Processor -- start 2024-03-06T07:25:50.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.246+0000 7f41242c2640 1 -- start start 2024-03-06T07:25:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f41242c2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41080039b0 0x7f411c0cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f41242c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 0x7f411c0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f41242c2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c15cc80 0x7f411c14fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f41242c2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c0d1df0 con 0x7f411c0cada0 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f41242c2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411c0d1c70 con 0x7f41080039b0 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f41242c2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411c15e3c0 con 0x7f411c15cc80 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f412185d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 0x7f411c0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f412185d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 0x7f411c0cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44140/0 (socket says 172.21.15.72:44140) 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f412185d640 1 -- 172.21.15.72:0/1597351266 learned_addr learned my addr 172.21.15.72:0/1597351266 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f412205e640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41080039b0 0x7f411c0cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.247+0000 7f412285f640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c15cc80 0x7f411c14fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412185d640 1 -- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c15cc80 msgr2=0x7f411c14fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412185d640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c15cc80 0x7f411c14fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412185d640 1 -- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41080039b0 msgr2=0x7f411c0cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412185d640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41080039b0 0x7f411c0cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412185d640 1 -- 172.21.15.72:0/1597351266 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c0cb230 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412205e640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41080039b0 0x7f411c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f412185d640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 0x7f411c0cd490 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f410c00d860 tx=0x7f410c00dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c019070 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c0074e0 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c0cb510 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0049e0 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.248+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c0cbad0 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.249+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c04f330 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.249+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f410c01c070 con 0x7f411c0cada0 2024-03-06T07:25:50.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.250+0000 7f4112ffd640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.250+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f410c089c80 con 0x7f411c0cada0 2024-03-06T07:25:50.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.250+0000 7f412205e640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.250+0000 7f412205e640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f410006b4d0 0x7f410006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f411800ba00 tx=0x7f411803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.254+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f410c058b90 con 0x7f411c0cada0 2024-03-06T07:25:50.285 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:50.287 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:25:50.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.434+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f411c0b2b90 con 0x7f411c0cada0 2024-03-06T07:25:50.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.435+0000 7f4112ffd640 1 -- 172.21.15.72:0/1597351266 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f410c058490 con 0x7f411c0cada0 2024-03-06T07:25:50.437 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:50.437 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:38.966454+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:50.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f410006b4d0 msgr2=0x7f410006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f410006b4d0 0x7f410006d990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f411800ba00 tx=0x7f411803d040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 msgr2=0x7f411c0cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 0x7f411c0cd490 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f410c00d860 tx=0x7f410c00dd20 comp rx=0 tx=0).stop 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 shutdown_connections 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f410006b4d0 0x7f410006d990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f411c15cc80 0x7f411c14fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0cada0 0x7f411c0cd490 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 --2- 172.21.15.72:0/1597351266 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41080039b0 0x7f411c0cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 >> 172.21.15.72:0/1597351266 conn(0x7f411c0d3cd0 msgr2=0x7f411c0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 shutdown_connections 2024-03-06T07:25:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.437+0000 7f41242c2640 1 -- 172.21.15.72:0/1597351266 wait complete. 2024-03-06T07:25:50.449 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:35.525319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:36.277249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:50.450 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:50.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.553+0000 7f9106f7a640 1 Processor -- start 2024-03-06T07:25:50.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.553+0000 7f9106f7a640 1 -- start start 2024-03-06T07:25:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9106f7a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91001614d0 0x7f91000d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9106f7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d1910 0x7f91000d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9106f7a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000d2740 con 0x7f91000d22b0 2024-03-06T07:25:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9106f7a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91001618b0 con 0x7f91001614d0 2024-03-06T07:25:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9106f7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91000cf400 con 0x7f91000d1910 2024-03-06T07:25:50.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91001614d0 0x7f91000d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f91001614d0 0x7f91000d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44144/0 (socket says 172.21.15.72:44144) 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 -- 172.21.15.72:0/398152588 learned_addr learned my addr 172.21.15.72:0/398152588 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9105517640 1 --1- 172.21.15.72:0/398152588 >> v1:172.21.15.72:6789/0 conn(0x7f91000d22b0 0x7f91000d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43138/0 (socket says 172.21.15.72:43138) 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 -- 172.21.15.72:0/398152588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d1910 msgr2=0x7f91000d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f90fffff640 1 --2- 172.21.15.72:0/398152588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d1910 0x7f91000d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f90ff7fe640 1 -- 172.21.15.72:0/398152588 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371707684 0 0) 0x7f90f8001490 con 0x7f91000d22b0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 --2- 172.21.15.72:0/398152588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d1910 0x7f91000d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 -- 172.21.15.72:0/398152588 >> v1:172.21.15.72:6789/0 conn(0x7f91000d22b0 legacy=0x7f91000d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.554+0000 7f9104d16640 1 -- 172.21.15.72:0/398152588 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91000cf6a0 con 0x7f91001614d0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.555+0000 7f9104d16640 1 --2- 172.21.15.72:0/398152588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000d4860 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009870 tx=0x7f90f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d826273a01b1ac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.555+0000 7f90ff7fe640 1 -- 172.21.15.72:0/398152588 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4034070 con 0x7f91001614d0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.555+0000 7f90ff7fe640 1 -- 172.21.15.72:0/398152588 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f4030840 con 0x7f91001614d0 2024-03-06T07:25:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.555+0000 7f90ff7fe640 1 -- 172.21.15.72:0/398152588 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4030c10 con 0x7f91001614d0 2024-03-06T07:25:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 -- 172.21.15.72:0/398152588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 msgr2=0x7f91000d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 --2- 172.21.15.72:0/398152588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000d4860 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009870 tx=0x7f90f402b980 comp rx=0 tx=0).stop 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 -- 172.21.15.72:0/398152588 shutdown_connections 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 --2- 172.21.15.72:0/398152588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d1910 0x7f91000d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 --2- 172.21.15.72:0/398152588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000d4860 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 -- 172.21.15.72:0/398152588 >> 172.21.15.72:0/398152588 conn(0x7f91000d3d40 msgr2=0x7f91000d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 -- 172.21.15.72:0/398152588 shutdown_connections 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.556+0000 7f9106f7a640 1 -- 172.21.15.72:0/398152588 wait complete. 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.557+0000 7f9106f7a640 1 Processor -- start 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.557+0000 7f9106f7a640 1 -- start start 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9106f7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91000d1910 0x7f91000ca510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9106f7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d22b0 0x7f9100151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9106f7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000cbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9106f7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91000b0910 con 0x7f91001614d0 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9106f7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91000cfc60 con 0x7f91000d1910 2024-03-06T07:25:50.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9106f7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91000b1e40 con 0x7f91000d22b0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000cbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000cbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44160/0 (socket says 172.21.15.72:44160) 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 -- 172.21.15.72:0/2206143625 learned_addr learned my addr 172.21.15.72:0/2206143625 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f90fffff640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d22b0 0x7f9100151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9104d16640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91000d1910 0x7f91000ca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 -- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d22b0 msgr2=0x7f9100151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d22b0 0x7f9100151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 -- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91000d1910 msgr2=0x7f91000ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91000d1910 0x7f91000ca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.558+0000 7f9105517640 1 -- 172.21.15.72:0/2206143625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91001516d0 con 0x7f91001614d0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.559+0000 7f9105517640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000cbe60 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f90f800c7f0 tx=0x7f90f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.559+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f8018070 con 0x7f91001614d0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.559+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f800a5e0 con 0x7f91001614d0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.559+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91000b9750 con 0x7f91001614d0 2024-03-06T07:25:50.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.559+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f800aa10 con 0x7f91001614d0 2024-03-06T07:25:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.559+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91000c7eb0 con 0x7f91001614d0 2024-03-06T07:25:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.560+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9100068ff0 con 0x7f91001614d0 2024-03-06T07:25:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.560+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f90f801b070 con 0x7f91001614d0 2024-03-06T07:25:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.561+0000 7f90fd7fa640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90e406b4d0 0x7f90e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.561+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90f80897e0 con 0x7f91001614d0 2024-03-06T07:25:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.561+0000 7f9104d16640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90e406b4d0 0x7f90e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.561+0000 7f9104d16640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90e406b4d0 0x7f90e406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f90f4036000 tx=0x7f90f403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.569+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f91000d2740 con 0x7f91001614d0 2024-03-06T07:25:50.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.746+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f910006bc50 con 0x7f91001614d0 2024-03-06T07:25:50.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.746+0000 7f90fd7fa640 1 -- 172.21.15.72:0/2206143625 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90f8058650 con 0x7f91001614d0 2024-03-06T07:25:50.748 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:25:50.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90e406b4d0 msgr2=0x7f90e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90e406b4d0 0x7f90e406d990 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f90f4036000 tx=0x7f90f403b040 comp rx=0 tx=0).stop 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 msgr2=0x7f91000cbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000cbe60 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f90f800c7f0 tx=0x7f90f800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 shutdown_connections 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f90e406b4d0 0x7f90e406d990 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91001614d0 0x7f91000cbe60 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91000d22b0 0x7f9100151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 --2- 172.21.15.72:0/2206143625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91000d1910 0x7f91000ca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 >> 172.21.15.72:0/2206143625 conn(0x7f91000d3d40 msgr2=0x7f91000d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 shutdown_connections 2024-03-06T07:25:50.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.748+0000 7f9106f7a640 1 -- 172.21.15.72:0/2206143625 wait complete. 2024-03-06T07:25:50.760 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:50.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.863+0000 7fb30da33640 1 Processor -- start 2024-03-06T07:25:50.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.863+0000 7fb30da33640 1 -- start start 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb30da33640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3080d4860 0x7fb3080d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb30da33640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3080b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb30da33640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3081618b0 con 0x7fb3081614d0 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb30da33640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3080d4cf0 con 0x7fb3080d4860 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb30da33640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080cf400 con 0x7fb3080b0e30 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb2fffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3080d4860 0x7fb3080d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb2fffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3080d4860 0x7fb3080d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44176/0 (socket says 172.21.15.72:44176) 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb2fffff640 1 -- 172.21.15.72:0/49917276 learned_addr learned my addr 172.21.15.72:0/49917276 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:50.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.864+0000 7fb3077fe640 1 --2- 172.21.15.72:0/49917276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3080b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb2fffff640 1 -- 172.21.15.72:0/49917276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 msgr2=0x7fb3080b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:50.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb2fffff640 1 --2- 172.21.15.72:0/49917276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3080b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb2fffff640 1 -- 172.21.15.72:0/49917276 >> v1:172.21.15.146:6789/0 conn(0x7fb3081614d0 legacy=0x7fb3080d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb2fffff640 1 -- 172.21.15.72:0/49917276 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080cf6a0 con 0x7fb3080d4860 2024-03-06T07:25:50.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb2fffff640 1 --2- 172.21.15.72:0/49917276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3080d4860 0x7fb3080d2020 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009870 tx=0x7fb2f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d22680e273dfd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb3067fc640 1 -- 172.21.15.72:0/49917276 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0034070 con 0x7fb3080d4860 2024-03-06T07:25:50.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.865+0000 7fb3067fc640 1 -- 172.21.15.72:0/49917276 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0030840 con 0x7fb3080d4860 2024-03-06T07:25:50.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb3067fc640 1 -- 172.21.15.72:0/49917276 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0030c10 con 0x7fb3080d4860 2024-03-06T07:25:50.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 -- 172.21.15.72:0/49917276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3080d4860 msgr2=0x7fb3080d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 --2- 172.21.15.72:0/49917276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3080d4860 0x7fb3080d2020 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009870 tx=0x7fb2f002b980 comp rx=0 tx=0).stop 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 -- 172.21.15.72:0/49917276 shutdown_connections 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 --2- 172.21.15.72:0/49917276 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3080b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 --2- 172.21.15.72:0/49917276 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3080d4860 0x7fb3080d2020 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 -- 172.21.15.72:0/49917276 >> 172.21.15.72:0/49917276 conn(0x7fb3080d3d40 msgr2=0x7fb3080d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 -- 172.21.15.72:0/49917276 shutdown_connections 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.866+0000 7fb30da33640 1 -- 172.21.15.72:0/49917276 wait complete. 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.867+0000 7fb30da33640 1 Processor -- start 2024-03-06T07:25:50.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 -- start start 2024-03-06T07:25:50.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3081572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3080d4860 0x7fb3080b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 0x7fb3080b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080b0910 con 0x7fb3081614d0 2024-03-06T07:25:50.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080cfc60 con 0x7fb3080d4860 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb30da33640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080d2560 con 0x7fb3080b0e30 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb3077fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 0x7fb3080b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb3077fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 0x7fb3080b6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44188/0 (socket says 172.21.15.72:44188) 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb3077fe640 1 -- 172.21.15.72:0/1271759606 learned_addr learned my addr 172.21.15.72:0/1271759606 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb2fffff640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3080d4860 0x7fb3080b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.868+0000 7fb306ffd640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3081572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb3077fe640 1 -- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 msgr2=0x7fb3081572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb3077fe640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3081572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb3077fe640 1 -- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3080d4860 msgr2=0x7fb3080b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb3077fe640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3080d4860 0x7fb3080b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb3077fe640 1 -- 172.21.15.72:0/1271759606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3080b75d0 con 0x7fb3081614d0 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb3077fe640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 0x7fb3080b6e70 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb300007e70 tx=0x7fb30000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300014620 con 0x7fb3081614d0 2024-03-06T07:25:50.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb300019070 con 0x7fb3081614d0 2024-03-06T07:25:50.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3080b78b0 con 0x7fb3081614d0 2024-03-06T07:25:50.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb300014a20 con 0x7fb3081614d0 2024-03-06T07:25:50.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.869+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3080b7e70 con 0x7fb3081614d0 2024-03-06T07:25:50.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.870+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb308068ff0 con 0x7fb3081614d0 2024-03-06T07:25:50.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.870+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb300014bc0 con 0x7fb3081614d0 2024-03-06T07:25:50.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.871+0000 7fb304ff9640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2e806b4d0 0x7fb2e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:50.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.871+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb30008a9c0 con 0x7fb3081614d0 2024-03-06T07:25:50.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.871+0000 7fb306ffd640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2e806b4d0 0x7fb2e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:50.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.871+0000 7fb306ffd640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2e806b4d0 0x7fb2e806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8000f60 tx=0x7fb2f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:50.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:50.875+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb30008ce20 con 0x7fb3081614d0 2024-03-06T07:25:51.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.052+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3080d1910 con 0x7fb3081614d0 2024-03-06T07:25:51.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.052+0000 7fb304ff9640 1 -- 172.21.15.72:0/1271759606 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb30005f070 con 0x7fb3081614d0 2024-03-06T07:25:51.054 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:25:51.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2e806b4d0 msgr2=0x7fb2e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2e806b4d0 0x7fb2e806d990 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8000f60 tx=0x7fb2f8009040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 msgr2=0x7fb3080b6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 0x7fb3080b6e70 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb300007e70 tx=0x7fb30000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 shutdown_connections 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2e806b4d0 0x7fb2e806d990 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3081614d0 0x7fb3080b6e70 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3080d4860 0x7fb3080b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 --2- 172.21.15.72:0/1271759606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3080b0e30 0x7fb3081572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 >> 172.21.15.72:0/1271759606 conn(0x7fb3080d3d40 msgr2=0x7fb3080b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 shutdown_connections 2024-03-06T07:25:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.054+0000 7fb30da33640 1 -- 172.21.15.72:0/1271759606 wait complete. 2024-03-06T07:25:51.066 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:51.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.168+0000 7fc650ab5640 1 Processor -- start 2024-03-06T07:25:51.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.168+0000 7fc650ab5640 1 -- start start 2024-03-06T07:25:51.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc650ab5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc64c15a220 0x7fc64c0cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc650ab5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0b3320 con 0x7fc64c06da00 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc650ab5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c15ec80 con 0x7fc64c0712d0 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc650ab5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c0cf610 con 0x7fc64c15a220 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc64c15a220 0x7fc64c0cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc64c06da00 0x7fc64c15e570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43150/0 (socket says 172.21.15.72:43150) 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64ad9d640 1 -- 172.21.15.72:0/2887612156 learned_addr learned my addr 172.21.15.72:0/2887612156 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64a59c640 1 -- 172.21.15.72:0/2887612156 >> v1:172.21.15.146:6789/0 conn(0x7fc64c0712d0 legacy=0x7fc64c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64a59c640 1 -- 172.21.15.72:0/2887612156 >> v1:172.21.15.72:6789/0 conn(0x7fc64c06da00 legacy=0x7fc64c15e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64a59c640 1 -- 172.21.15.72:0/2887612156 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c0728d0 con 0x7fc64c15a220 2024-03-06T07:25:51.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.170+0000 7fc64959a640 1 -- 172.21.15.72:0/2887612156 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343696704 0 0) 0x7fc64c0b3320 con 0x7fc64c06da00 2024-03-06T07:25:51.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.171+0000 7fc64a59c640 1 --2- 172.21.15.72:0/2887612156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15a220 0x7fc64c0cf0d0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc6440097a0 tx=0x7fc64402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212c125aa42242fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.171+0000 7fc64959a640 1 -- 172.21.15.72:0/2887612156 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6440315a0 con 0x7fc64c15a220 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.171+0000 7fc64959a640 1 -- 172.21.15.72:0/2887612156 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc644037070 con 0x7fc64c15a220 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.171+0000 7fc64959a640 1 -- 172.21.15.72:0/2887612156 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644031960 con 0x7fc64c15a220 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 -- 172.21.15.72:0/2887612156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15a220 msgr2=0x7fc64c0cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 --2- 172.21.15.72:0/2887612156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15a220 0x7fc64c0cf0d0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc6440097a0 tx=0x7fc64402b840 comp rx=0 tx=0).stop 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 -- 172.21.15.72:0/2887612156 shutdown_connections 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 --2- 172.21.15.72:0/2887612156 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15a220 0x7fc64c0cf0d0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 -- 172.21.15.72:0/2887612156 >> 172.21.15.72:0/2887612156 conn(0x7fc64c06b190 msgr2=0x7fc64c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 -- 172.21.15.72:0/2887612156 shutdown_connections 2024-03-06T07:25:51.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.172+0000 7fc650ab5640 1 -- 172.21.15.72:0/2887612156 wait complete. 2024-03-06T07:25:51.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.174+0000 7fc650ab5640 1 Processor -- start 2024-03-06T07:25:51.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.174+0000 7fc650ab5640 1 -- start start 2024-03-06T07:25:51.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc650ab5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc64c06da00 0x7fc64c14b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc650ab5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 0x7fc64c14b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc650ab5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c15a220 0x7fc64c0ae010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc650ab5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c072030 con 0x7fc64c0712d0 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc650ab5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c072bb0 con 0x7fc64c06da00 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc650ab5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c072530 con 0x7fc64c15a220 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 0x7fc64c14b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 0x7fc64c14b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44210/0 (socket says 172.21.15.72:44210) 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 -- 172.21.15.72:0/3459975188 learned_addr learned my addr 172.21.15.72:0/3459975188 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc64a59c640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc64c06da00 0x7fc64c14b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc64ad9d640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c15a220 0x7fc64c0ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 -- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c15a220 msgr2=0x7fc64c0ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c15a220 0x7fc64c0ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 -- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc64c06da00 msgr2=0x7fc64c14b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc64c06da00 0x7fc64c14b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 -- 172.21.15.72:0/3459975188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c146110 con 0x7fc64c0712d0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc64a59c640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc64c06da00 0x7fc64c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc64ad9d640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c15a220 0x7fc64c0ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.175+0000 7fc649d9b640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 0x7fc64c14b600 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc63400e710 tx=0x7fc63400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.176+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc634015070 con 0x7fc64c0712d0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.176+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63400ee20 con 0x7fc64c0712d0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.176+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c15b780 con 0x7fc64c0712d0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.176+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6340018e0 con 0x7fc64c0712d0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.176+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c15bd40 con 0x7fc64c0712d0 2024-03-06T07:25:51.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.177+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c0732b0 con 0x7fc64c0712d0 2024-03-06T07:25:51.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.177+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc634018070 con 0x7fc64c0712d0 2024-03-06T07:25:51.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.178+0000 7fc642ffd640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc62c06b4d0 0x7fc62c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.178+0000 7fc64a59c640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc62c06b4d0 0x7fc62c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.178+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc63408c300 con 0x7fc64c0712d0 2024-03-06T07:25:51.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.178+0000 7fc64a59c640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc62c06b4d0 0x7fc62c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc644036000 tx=0x7fc644008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.182+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc63405b210 con 0x7fc64c0712d0 2024-03-06T07:25:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.354+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc64c068ff0 con 0x7fc64c0712d0 2024-03-06T07:25:51.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.355+0000 7fc642ffd640 1 -- 172.21.15.72:0/3459975188 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc63405ab10 con 0x7fc64c0712d0 2024-03-06T07:25:51.356 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:25:51.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc62c06b4d0 msgr2=0x7fc62c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc62c06b4d0 0x7fc62c06d990 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc644036000 tx=0x7fc644008030 comp rx=0 tx=0).stop 2024-03-06T07:25:51.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 msgr2=0x7fc64c14b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 0x7fc64c14b600 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fc63400e710 tx=0x7fc63400ebd0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 shutdown_connections 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc62c06b4d0 0x7fc62c06d990 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c15a220 0x7fc64c0ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c0712d0 0x7fc64c14b600 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 --2- 172.21.15.72:0/3459975188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc64c06da00 0x7fc64c14b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 >> 172.21.15.72:0/3459975188 conn(0x7fc64c06b190 msgr2=0x7fc64c0b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 shutdown_connections 2024-03-06T07:25:51.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.357+0000 7fc650ab5640 1 -- 172.21.15.72:0/3459975188 wait complete. 2024-03-06T07:25:51.369 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.507+0000 7f4551781640 1 Processor -- start 2024-03-06T07:25:51.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.507+0000 7f4551781640 1 -- start start 2024-03-06T07:25:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f4551781640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4fe0 0x7f454c0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f4551781640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f454c15ba50 0x7f454c15be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f4551781640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454c15d510 con 0x7f454c0b4c00 2024-03-06T07:25:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f4551781640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454c15e0b0 con 0x7f454c15ba50 2024-03-06T07:25:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f4551781640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454c15d780 con 0x7f454c0b4fe0 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f454c15ba50 0x7f454c15be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f454c15ba50 0x7f454c15be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44226/0 (socket says 172.21.15.72:44226) 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 -- 172.21.15.72:0/3795019769 learned_addr learned my addr 172.21.15.72:0/3795019769 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454a7fc640 1 --2- 172.21.15.72:0/3795019769 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4fe0 0x7f454c0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 -- 172.21.15.72:0/3795019769 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4fe0 msgr2=0x7f454c0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 --2- 172.21.15.72:0/3795019769 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4fe0 0x7f454c0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 -- 172.21.15.72:0/3795019769 >> v1:172.21.15.146:6789/0 conn(0x7f454c0b4c00 legacy=0x7f454c0b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.508+0000 7f454b7fe640 1 -- 172.21.15.72:0/3795019769 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c15da60 con 0x7f454c15ba50 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.509+0000 7f454b7fe640 1 --2- 172.21.15.72:0/3795019769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c15ba50 0x7f454c15be90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f453c009870 tx=0x7f453c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4776c01c6e3d7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.509+0000 7f4549ffb640 1 -- 172.21.15.72:0/3795019769 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c034070 con 0x7f454c15ba50 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.509+0000 7f4549ffb640 1 -- 172.21.15.72:0/3795019769 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c030840 con 0x7f454c15ba50 2024-03-06T07:25:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.509+0000 7f4549ffb640 1 -- 172.21.15.72:0/3795019769 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c030c10 con 0x7f454c15ba50 2024-03-06T07:25:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 -- 172.21.15.72:0/3795019769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c15ba50 msgr2=0x7f454c15be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 --2- 172.21.15.72:0/3795019769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c15ba50 0x7f454c15be90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f453c009870 tx=0x7f453c02b980 comp rx=0 tx=0).stop 2024-03-06T07:25:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 -- 172.21.15.72:0/3795019769 shutdown_connections 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 --2- 172.21.15.72:0/3795019769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c15ba50 0x7f454c15be90 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 --2- 172.21.15.72:0/3795019769 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4fe0 0x7f454c0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 -- 172.21.15.72:0/3795019769 >> 172.21.15.72:0/3795019769 conn(0x7f454c09a9f0 msgr2=0x7f454c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 -- 172.21.15.72:0/3795019769 shutdown_connections 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.510+0000 7f4551781640 1 -- 172.21.15.72:0/3795019769 wait complete. 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.511+0000 7f4551781640 1 Processor -- start 2024-03-06T07:25:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.511+0000 7f4551781640 1 -- start start 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f4551781640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4c00 0x7f454c14f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f4551781640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 0x7f454c06c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f4551781640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c15ba50 0x7f454c1f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f4551781640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c15f6d0 con 0x7f454c0b4fe0 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f4551781640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c15f550 con 0x7f454c15ba50 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f4551781640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454c15c3d0 con 0x7f454c0b4c00 2024-03-06T07:25:51.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 0x7f454c06c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 0x7f454c06c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44238/0 (socket says 172.21.15.72:44238) 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 -- 172.21.15.72:0/3145856454 learned_addr learned my addr 172.21.15.72:0/3145856454 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454affd640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4c00 0x7f454c14f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454b7fe640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c15ba50 0x7f454c1f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 -- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4c00 msgr2=0x7f454c14f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4c00 0x7f454c14f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 -- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c15ba50 msgr2=0x7f454c1f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c15ba50 0x7f454c1f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 -- 172.21.15.72:0/3145856454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c1f4d00 con 0x7f454c0b4fe0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454b7fe640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c15ba50 0x7f454c1f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.512+0000 7f454a7fc640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 0x7f454c06c6b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f453400d830 tx=0x7f453400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.513+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453400b6c0 con 0x7f454c0b4fe0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.513+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4534019070 con 0x7f454c0b4fe0 2024-03-06T07:25:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.513+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c14ff20 con 0x7f454c0b4fe0 2024-03-06T07:25:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.513+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453400ba80 con 0x7f454c0b4fe0 2024-03-06T07:25:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.513+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c0b06e0 con 0x7f454c0b4fe0 2024-03-06T07:25:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.514+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c04f330 con 0x7f454c0b4fe0 2024-03-06T07:25:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.514+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f453400bc20 con 0x7f454c0b4fe0 2024-03-06T07:25:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.515+0000 7f45437fe640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f452c06b4d0 0x7f452c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.515+0000 7f454affd640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f452c06b4d0 0x7f452c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:51.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.515+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f453408a650 con 0x7f454c0b4fe0 2024-03-06T07:25:51.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.515+0000 7f454affd640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f452c06b4d0 0x7f452c06d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4544005520 tx=0x7f4544009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:51.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.519+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f453401c360 con 0x7f454c0b4fe0 2024-03-06T07:25:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.705+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f454c1596c0 con 0x7f454c0b4fe0 2024-03-06T07:25:51.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.706+0000 7f45437fe640 1 -- 172.21.15.72:0/3145856454 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v83) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4534059560 con 0x7f454c0b4fe0 2024-03-06T07:25:51.707 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.707+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f452c06b4d0 msgr2=0x7f452c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f452c06b4d0 0x7f452c06d990 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4544005520 tx=0x7f4544009040 comp rx=0 tx=0).stop 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 msgr2=0x7f454c06c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 0x7f454c06c6b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f453400d830 tx=0x7f453400dcf0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 shutdown_connections 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f452c06b4d0 0x7f452c06d990 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c15ba50 0x7f454c1f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f454c0b4fe0 0x7f454c06c6b0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 --2- 172.21.15.72:0/3145856454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0b4c00 0x7f454c14f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 >> 172.21.15.72:0/3145856454 conn(0x7f454c09a9f0 msgr2=0x7f454c06fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 shutdown_connections 2024-03-06T07:25:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:51.708+0000 7f4551781640 1 -- 172.21.15.72:0/3145856454 wait complete. 2024-03-06T07:25:51.719 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:25:51.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.720 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:25:51.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[65540]: fuse finished with error 0 and tester_r 0 2024-03-06T07:25:51.746 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:25:51.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.746 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:25:51.873 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:25:51.873 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:51.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:51.891 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:25:51.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.892 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:25:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:51.949 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:25:51.950 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:25:51.950 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:51.965 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:25:51.965 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:25:51.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:51.966 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:25:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:25:52.021 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:25:52.021 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:25:52.039 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:25:52.039 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:25:52.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:25:52.039 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:25:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:25:52.097 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:25:52.097 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:52.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.197+0000 7f14bc53e640 1 Processor -- start 2024-03-06T07:25:52.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.197+0000 7f14bc53e640 1 -- start start 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.198+0000 7f14bc53e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14b415dfc0 0x7f14b406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.198+0000 7f14bc53e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f14b406e480 0x7f14b406d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.198+0000 7f14bc53e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b406bb60 con 0x7f14b406d7a0 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.198+0000 7f14bc53e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b415e3a0 con 0x7f14b415dfc0 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.198+0000 7f14bc53e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b406be40 con 0x7f14b406e480 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14b415dfc0 0x7f14b406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14b415dfc0 0x7f14b406df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44240/0 (socket says 172.21.15.72:44240) 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 -- 172.21.15.72:0/827584588 learned_addr learned my addr 172.21.15.72:0/827584588 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14b9ad9640 1 --2- 172.21.15.72:0/827584588 >> v2:172.21.15.146:3300/0 conn(0x7f14b406e480 0x7f14b406d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 -- 172.21.15.72:0/827584588 >> v2:172.21.15.146:3300/0 conn(0x7f14b406e480 msgr2=0x7f14b406d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 --2- 172.21.15.72:0/827584588 >> v2:172.21.15.146:3300/0 conn(0x7f14b406e480 0x7f14b406d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 -- 172.21.15.72:0/827584588 >> v1:172.21.15.146:6789/0 conn(0x7f14b406d7a0 legacy=0x7f14b40cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.199+0000 7f14ba2da640 1 -- 172.21.15.72:0/827584588 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b406c120 con 0x7f14b415dfc0 2024-03-06T07:25:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.200+0000 7f14ba2da640 1 --2- 172.21.15.72:0/827584588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b415dfc0 0x7f14b406df40 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f14b40ccb10 tx=0x7f14b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c0d1d78d1ea520 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.200+0000 7f14b92d8640 1 -- 172.21.15.72:0/827584588 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0035020 con 0x7f14b415dfc0 2024-03-06T07:25:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.200+0000 7f14b92d8640 1 -- 172.21.15.72:0/827584588 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b0031870 con 0x7f14b415dfc0 2024-03-06T07:25:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.200+0000 7f14b92d8640 1 -- 172.21.15.72:0/827584588 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0031c40 con 0x7f14b415dfc0 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 -- 172.21.15.72:0/827584588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b415dfc0 msgr2=0x7f14b406df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 --2- 172.21.15.72:0/827584588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b415dfc0 0x7f14b406df40 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f14b40ccb10 tx=0x7f14b002b910 comp rx=0 tx=0).stop 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 -- 172.21.15.72:0/827584588 shutdown_connections 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 --2- 172.21.15.72:0/827584588 >> v2:172.21.15.146:3300/0 conn(0x7f14b406e480 0x7f14b406d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 --2- 172.21.15.72:0/827584588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b415dfc0 0x7f14b406df40 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 -- 172.21.15.72:0/827584588 >> 172.21.15.72:0/827584588 conn(0x7f14b40d00e0 msgr2=0x7f14b40d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 -- 172.21.15.72:0/827584588 shutdown_connections 2024-03-06T07:25:52.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.201+0000 7f14bc53e640 1 -- 172.21.15.72:0/827584588 wait complete. 2024-03-06T07:25:52.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.202+0000 7f14bc53e640 1 Processor -- start 2024-03-06T07:25:52.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.202+0000 7f14bc53e640 1 -- start start 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14b406d7a0 0x7f14b4157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 0x7f14b40b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14b415dfc0 0x7f14b40b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40ae320 con 0x7f14b406e480 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b406dbb0 con 0x7f14b406d7a0 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b40ae4a0 con 0x7f14b415dfc0 2024-03-06T07:25:52.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 0x7f14b40b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 0x7f14b40b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44252/0 (socket says 172.21.15.72:44252) 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 -- 172.21.15.72:0/3706023991 learned_addr learned my addr 172.21.15.72:0/3706023991 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14ba2da640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14b406d7a0 0x7f14b4157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14baadb640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14b415dfc0 0x7f14b40b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 -- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14b415dfc0 msgr2=0x7f14b40b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14b415dfc0 0x7f14b40b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 -- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14b406d7a0 msgr2=0x7f14b4157d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14b406d7a0 0x7f14b4157d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 -- 172.21.15.72:0/3706023991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b4158510 con 0x7f14b406e480 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14baadb640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14b415dfc0 0x7f14b40b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14b9ad9640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 0x7f14b40b0d00 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f14a400d5d0 tx=0x7f14a400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40125f0 con 0x7f14b406e480 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a4019070 con 0x7f14b406e480 2024-03-06T07:25:52.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a40129f0 con 0x7f14b406e480 2024-03-06T07:25:52.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.204+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b40c9490 con 0x7f14b406e480 2024-03-06T07:25:52.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.205+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b415e8b0 con 0x7f14b406e480 2024-03-06T07:25:52.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.206+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b404f330 con 0x7f14b406e480 2024-03-06T07:25:52.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.206+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14a4012b90 con 0x7f14b406e480 2024-03-06T07:25:52.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.206+0000 7f14aaffd640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f149806b4d0 0x7f149806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.206+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14a408a5a0 con 0x7f14b406e480 2024-03-06T07:25:52.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.206+0000 7f14ba2da640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f149806b4d0 0x7f149806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.207+0000 7f14ba2da640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f149806b4d0 0x7f149806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f14b0036000 tx=0x7f14b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.211+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14a4059420 con 0x7f14b406e480 2024-03-06T07:25:52.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.426+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14b4067590 con 0x7f14b406e480 2024-03-06T07:25:52.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.427+0000 7f14aaffd640 1 -- 172.21.15.72:0/3706023991 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f14a4058d20 con 0x7f14b406e480 2024-03-06T07:25:52.428 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:52.428 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:52.428 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:52.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f149806b4d0 msgr2=0x7f149806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f149806b4d0 0x7f149806d990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f14b0036000 tx=0x7f14b003c040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 msgr2=0x7f14b40b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 0x7f14b40b0d00 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f14a400d5d0 tx=0x7f14a400da90 comp rx=0 tx=0).stop 2024-03-06T07:25:52.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 shutdown_connections 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f149806b4d0 0x7f149806d990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14b415dfc0 0x7f14b40b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14b406e480 0x7f14b40b0d00 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 --2- 172.21.15.72:0/3706023991 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14b406d7a0 0x7f14b4157d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 >> 172.21.15.72:0/3706023991 conn(0x7f14b40d00e0 msgr2=0x7f14b40d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 shutdown_connections 2024-03-06T07:25:52.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.429+0000 7f14bc53e640 1 -- 172.21.15.72:0/3706023991 wait complete. 2024-03-06T07:25:52.441 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:52.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.573+0000 7f6484055640 1 Processor -- start 2024-03-06T07:25:52.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.573+0000 7f6484055640 1 -- start start 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.573+0000 7f6484055640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 0x7f647c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6484055640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f647c0b41f0 0x7f647c0b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6484055640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c0b3430 con 0x7f647c15dfe0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6484055640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c0d2c50 con 0x7f647c0b41f0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6484055640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b31e0 con 0x7f647c0d22b0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6481df1640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f647c15dfe0 0x7f647c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35952/0 (socket says 172.21.15.72:35952) 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6481df1640 1 -- 172.21.15.72:0/1290319514 learned_addr learned my addr 172.21.15.72:0/1290319514 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 0x7f647c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64825f2640 1 --2- 172.21.15.72:0/1290319514 >> v2:172.21.15.146:3300/0 conn(0x7f647c0b41f0 0x7f647c0b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512592033 0 0) 0x7f647c0b3430 con 0x7f647c15dfe0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6464003520 con 0x7f647c15dfe0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517201773 0 0) 0x7f6464003520 con 0x7f647c15dfe0 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 msgr2=0x7f647c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 0x7f647c0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 >> v2:172.21.15.146:3300/0 conn(0x7f647c0b41f0 msgr2=0x7f647c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 --2- 172.21.15.72:0/1290319514 >> v2:172.21.15.146:3300/0 conn(0x7f647c0b41f0 0x7f647c0b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c0b3710 con 0x7f647c15dfe0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f646c0019f0 con 0x7f647c15dfe0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 >> v1:172.21.15.72:6789/0 conn(0x7f647c15dfe0 legacy=0x7f647c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64640039b0 0x7f6464003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6464004800 0x7f6464004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 0x7f64640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b31e0 con 0x7f6464004800 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f646400b9c0 con 0x7f64640039b0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f646400bb40 con 0x7f647c0d22b0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6464004800 0x7f6464004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6481df1640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64640039b0 0x7f6464003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64825f2640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 0x7f64640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 -- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 msgr2=0x7f64640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0d22b0 0x7f64640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 -- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64640039b0 msgr2=0x7f6464003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64640039b0 0x7f6464003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 -- 172.21.15.72:0/1290319514 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6464003520 con 0x7f6464004800 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f64815f0640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6464004800 0x7f6464004c80 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f6470009ac0 tx=0x7f6464005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e39e8fb9f2ef76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.574+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470039070 con 0x7f6464004800 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647002c6b0 con 0x7f6464004800 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6480def640 1 -- 172.21.15.72:0/1290319514 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470009690 con 0x7f6464004800 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6484055640 1 -- 172.21.15.72:0/1290319514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6464004800 msgr2=0x7f6464004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6484055640 1 --2- 172.21.15.72:0/1290319514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6464004800 0x7f6464004c80 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f6470009ac0 tx=0x7f6464005830 comp rx=0 tx=0).stop 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f64825f2640 1 -- 172.21.15.72:0/1290319514 reap_dead start 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6484055640 1 -- 172.21.15.72:0/1290319514 shutdown_connections 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6484055640 1 -- 172.21.15.72:0/1290319514 >> 172.21.15.72:0/1290319514 conn(0x7f647c0d3cd0 msgr2=0x7f647c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6484055640 1 -- 172.21.15.72:0/1290319514 shutdown_connections 2024-03-06T07:25:52.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.575+0000 7f6484055640 1 -- 172.21.15.72:0/1290319514 wait complete. 2024-03-06T07:25:52.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.576+0000 7f6484055640 1 Processor -- start 2024-03-06T07:25:52.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.576+0000 7f6484055640 1 -- start start 2024-03-06T07:25:52.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6484055640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464004800 0x7f647c15f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6484055640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0b9600 0x7f647c0cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6484055640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 0x7f647c0ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6484055640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0cfa00 con 0x7f647c0ccac0 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6484055640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0b3d60 con 0x7f6464004800 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6484055640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0d1c70 con 0x7f647c0b9600 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f64825f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 0x7f647c0ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f64825f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 0x7f647c0ccf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42224/0 (socket says 172.21.15.72:42224) 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f64825f2640 1 -- 172.21.15.72:0/1667368273 learned_addr learned my addr 172.21.15.72:0/1667368273 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f6481df1640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464004800 0x7f647c15f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f64815f0640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0b9600 0x7f647c0cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.577+0000 7f64825f2640 1 -- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0b9600 msgr2=0x7f647c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f64825f2640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0b9600 0x7f647c0cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f64825f2640 1 -- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464004800 msgr2=0x7f647c15f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f64825f2640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464004800 0x7f647c15f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f64825f2640 1 -- 172.21.15.72:0/1667368273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c0cd480 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f6481df1640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464004800 0x7f647c15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f64815f0640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0b9600 0x7f647c0cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f64825f2640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 0x7f647c0ccf40 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f647c0cff40 tx=0x7f6474009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474003610 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647401b070 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64740039f0 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c160e20 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.578+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c160060 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.579+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c04f330 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.579+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6474003b90 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.580+0000 7f647a7fc640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f646806b500 0x7f646806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.580+0000 7f6481df1640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f646806b500 0x7f646806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.580+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6474089f30 con 0x7f647c0ccac0 2024-03-06T07:25:52.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.580+0000 7f6481df1640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f646806b500 0x7f646806d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f646c004c20 tx=0x7f646c006040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.584+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6474024020 con 0x7f647c0ccac0 2024-03-06T07:25:52.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.763+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f647c0747a0 con 0x7f647c0ccac0 2024-03-06T07:25:52.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.764+0000 7f647a7fc640 1 -- 172.21.15.72:0/1667368273 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f6474058db0 con 0x7f647c0ccac0 2024-03-06T07:25:52.766 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:52.767 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:38.966454+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f646806b500 msgr2=0x7f646806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f646806b500 0x7f646806d9c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f646c004c20 tx=0x7f646c006040 comp rx=0 tx=0).stop 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 msgr2=0x7f647c0ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 0x7f647c0ccf40 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f647c0cff40 tx=0x7f6474009f90 comp rx=0 tx=0).stop 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 shutdown_connections 2024-03-06T07:25:52.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f646806b500 0x7f646806d9c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f647c0ccac0 0x7f647c0ccf40 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f647c0b9600 0x7f647c0cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 --2- 172.21.15.72:0/1667368273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6464004800 0x7f647c15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 >> 172.21.15.72:0/1667368273 conn(0x7f647c0d3cd0 msgr2=0x7f647c0d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 shutdown_connections 2024-03-06T07:25:52.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.767+0000 7f6484055640 1 -- 172.21.15.72:0/1667368273 wait complete. 2024-03-06T07:25:52.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:25:35.525319+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:25:36.277249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:25:52.781 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:25:52.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.882+0000 7fdb98343640 1 Processor -- start 2024-03-06T07:25:52.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.883+0000 7fdb98343640 1 -- start start 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.883+0000 7fdb98343640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb901614d0 0x7fdb900d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.883+0000 7fdb98343640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb901618b0 con 0x7fdb900d24b0 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.883+0000 7fdb98343640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb900cf400 con 0x7fdb900d1910 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.883+0000 7fdb98343640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb900cf6a0 con 0x7fdb901614d0 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb968e0640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdb900d24b0 0x7fdb900b0e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35964/0 (socket says 172.21.15.72:35964) 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb968e0640 1 -- 172.21.15.72:0/897646239 learned_addr learned my addr 172.21.15.72:0/897646239 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:52.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80074703 0 0) 0x7fdb901618b0 con 0x7fdb900d24b0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78003520 con 0x7fdb900d24b0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb960df640 1 --2- 172.21.15.72:0/897646239 >> v2:172.21.15.146:3300/0 conn(0x7fdb901614d0 0x7fdb900d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766927821 0 0) 0x7fdb78003520 con 0x7fdb900d24b0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 >> v2:172.21.15.146:3300/0 conn(0x7fdb901614d0 msgr2=0x7fdb900d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 --2- 172.21.15.72:0/897646239 >> v2:172.21.15.146:3300/0 conn(0x7fdb901614d0 0x7fdb900d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 >> v1:172.21.15.146:6789/0 conn(0x7fdb900d1910 legacy=0x7fdb900d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb900cf980 con 0x7fdb900d24b0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fdb800019f0 con 0x7fdb900d24b0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 >> v1:172.21.15.72:6789/0 conn(0x7fdb900d24b0 legacy=0x7fdb900b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb780039b0 0x7fdb78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb780051c0 0x7fdb7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900cf6a0 con 0x7fdb78004800 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7800c060 con 0x7fdb780039b0 2024-03-06T07:25:52.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7800c260 con 0x7fdb780051c0 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb958de640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb960df640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb780039b0 0x7fdb78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.884+0000 7fdb968e0640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb780051c0 0x7fdb7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb958de640 1 -- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb780051c0 msgr2=0x7fdb7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb958de640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb780051c0 0x7fdb7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb958de640 1 -- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb780039b0 msgr2=0x7fdb78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb958de640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb780039b0 0x7fdb78003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb958de640 1 -- 172.21.15.72:0/897646239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb78003520 con 0x7fdb78004800 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb968e0640 1 -- 172.21.15.72:0/897646239 reap_dead start 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb958de640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb78004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdb84001720 tx=0x7fdb78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e263ff56a1ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84001370 con 0x7fdb78004800 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.885+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb840052d0 con 0x7fdb78004800 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb950dd640 1 -- 172.21.15.72:0/897646239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84003830 con 0x7fdb78004800 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 -- 172.21.15.72:0/897646239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 msgr2=0x7fdb78004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb78004c80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdb84001720 tx=0x7fdb78005830 comp rx=0 tx=0).stop 2024-03-06T07:25:52.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 -- 172.21.15.72:0/897646239 shutdown_connections 2024-03-06T07:25:52.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 --2- 172.21.15.72:0/897646239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb78004c80 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 -- 172.21.15.72:0/897646239 >> 172.21.15.72:0/897646239 conn(0x7fdb900d3d40 msgr2=0x7fdb900d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:52.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 -- 172.21.15.72:0/897646239 shutdown_connections 2024-03-06T07:25:52.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.886+0000 7fdb98343640 1 -- 172.21.15.72:0/897646239 wait complete. 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.888+0000 7fdb98343640 1 Processor -- start 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.888+0000 7fdb98343640 1 -- start start 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb98343640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb900cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb98343640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb900d0110 0x7fdb90151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb98343640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb900d0590 0x7fdb901522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb98343640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900d4860 con 0x7fdb78004800 2024-03-06T07:25:52.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb98343640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900b0910 con 0x7fdb900d0110 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb98343640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb900d49e0 con 0x7fdb900d0590 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb960df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb900cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb960df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb900cdc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42252/0 (socket says 172.21.15.72:42252) 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb960df640 1 -- 172.21.15.72:0/2368938333 learned_addr learned my addr 172.21.15.72:0/2368938333 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb958de640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb900d0110 0x7fdb90151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.889+0000 7fdb968e0640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb900d0590 0x7fdb901522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb960df640 1 -- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb900d0590 msgr2=0x7fdb901522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb960df640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb900d0590 0x7fdb901522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb960df640 1 -- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb900d0110 msgr2=0x7fdb90151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb960df640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb900d0110 0x7fdb90151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb960df640 1 -- 172.21.15.72:0/2368938333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb901f80b0 con 0x7fdb78004800 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb960df640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb900cdc60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdb88011aa0 tx=0x7fdb88011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8800fb80 con 0x7fdb78004800 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8800fd20 con 0x7fdb78004800 2024-03-06T07:25:52.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb901f7530 con 0x7fdb78004800 2024-03-06T07:25:52.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb88014660 con 0x7fdb78004800 2024-03-06T07:25:52.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.890+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb901f7af0 con 0x7fdb78004800 2024-03-06T07:25:52.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.891+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb900c71e0 con 0x7fdb78004800 2024-03-06T07:25:52.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.891+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb88012040 con 0x7fdb78004800 2024-03-06T07:25:52.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.892+0000 7fdb8effd640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb7c06b4d0 0x7fdb7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:52.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.892+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb8808a8f0 con 0x7fdb78004800 2024-03-06T07:25:52.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.892+0000 7fdb958de640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb7c06b4d0 0x7fdb7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:52.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.892+0000 7fdb958de640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb7c06b4d0 0x7fdb7c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdb84033000 tx=0x7fdb84009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:52.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:52.900+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb900c71e0 con 0x7fdb78004800 2024-03-06T07:25:53.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.079+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb90068ff0 con 0x7fdb78004800 2024-03-06T07:25:53.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.079+0000 7fdb8effd640 1 -- 172.21.15.72:0/2368938333 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdb88019030 con 0x7fdb78004800 2024-03-06T07:25:53.081 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:25:53.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb7c06b4d0 msgr2=0x7fdb7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb7c06b4d0 0x7fdb7c06d990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdb84033000 tx=0x7fdb84009030 comp rx=0 tx=0).stop 2024-03-06T07:25:53.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 msgr2=0x7fdb900cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb900cdc60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdb88011aa0 tx=0x7fdb88011f60 comp rx=0 tx=0).stop 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 shutdown_connections 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb7c06b4d0 0x7fdb7c06d990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb900d0590 0x7fdb901522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb900d0110 0x7fdb90151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 --2- 172.21.15.72:0/2368938333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb78004800 0x7fdb900cdc60 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 >> 172.21.15.72:0/2368938333 conn(0x7fdb900d3d40 msgr2=0x7fdb900b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 shutdown_connections 2024-03-06T07:25:53.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.081+0000 7fdb98343640 1 -- 172.21.15.72:0/2368938333 wait complete. 2024-03-06T07:25:53.097 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:25:53.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.207+0000 7fd905b3e640 1 Processor -- start 2024-03-06T07:25:53.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.207+0000 7fd905b3e640 1 -- start start 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd905b3e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9000cff40 0x7fd9000b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd905b3e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd900158ac0 con 0x7fd9000cfa90 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd905b3e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90015b2a0 con 0x7fd9001586e0 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd905b3e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9000d0390 con 0x7fd9000cff40 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8feffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd9000cfa90 0x7fd9000b33a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35966/0 (socket says 172.21.15.72:35966) 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8fffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd9000cff40 0x7fd9000b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8feffd640 1 -- 172.21.15.72:0/141126843 learned_addr learned my addr 172.21.15.72:0/141126843 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8fffff640 1 -- 172.21.15.72:0/141126843 >> v1:172.21.15.146:6789/0 conn(0x7fd9001586e0 legacy=0x7fd9000aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8fffff640 1 -- 172.21.15.72:0/141126843 >> v1:172.21.15.72:6789/0 conn(0x7fd9000cfa90 legacy=0x7fd9000b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8fe7fc640 1 -- 172.21.15.72:0/141126843 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564640744 0 0) 0x7fd900158ac0 con 0x7fd9000cfa90 2024-03-06T07:25:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.208+0000 7fd8fffff640 1 -- 172.21.15.72:0/141126843 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90015b540 con 0x7fd9000cff40 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd8fffff640 1 --2- 172.21.15.72:0/141126843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cff40 0x7fd9000b3ab0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009d90 tx=0x7fd8f802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7c26c783bf214a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd8fe7fc640 1 -- 172.21.15.72:0/141126843 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8039070 con 0x7fd9000cff40 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd8fe7fc640 1 -- 172.21.15.72:0/141126843 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f802cc60 con 0x7fd9000cff40 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd8fe7fc640 1 -- 172.21.15.72:0/141126843 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8034660 con 0x7fd9000cff40 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 -- 172.21.15.72:0/141126843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cff40 msgr2=0x7fd9000b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 --2- 172.21.15.72:0/141126843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cff40 0x7fd9000b3ab0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009d90 tx=0x7fd8f802beb0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 -- 172.21.15.72:0/141126843 shutdown_connections 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 --2- 172.21.15.72:0/141126843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cff40 0x7fd9000b3ab0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 -- 172.21.15.72:0/141126843 >> 172.21.15.72:0/141126843 conn(0x7fd90006b190 msgr2=0x7fd90006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 -- 172.21.15.72:0/141126843 shutdown_connections 2024-03-06T07:25:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.209+0000 7fd905b3e640 1 -- 172.21.15.72:0/141126843 wait complete. 2024-03-06T07:25:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 Processor -- start 2024-03-06T07:25:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 -- start start 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 0x7fd9001541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9000cff40 0x7fd9001e18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9001586e0 0x7fd9001e3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90015bde0 con 0x7fd9000cfa90 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90015bb00 con 0x7fd9001586e0 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.211+0000 7fd905b3e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9000af150 con 0x7fd9000cff40 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 0x7fd9001541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 0x7fd9001541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42268/0 (socket says 172.21.15.72:42268) 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 -- 172.21.15.72:0/1898868056 learned_addr learned my addr 172.21.15.72:0/1898868056 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8fffff640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9001586e0 0x7fd9001e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8feffd640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9000cff40 0x7fd9001e18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 -- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9000cff40 msgr2=0x7fd9001e18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9000cff40 0x7fd9001e18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 -- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9001586e0 msgr2=0x7fd9001e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9001586e0 0x7fd9001e3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 -- 172.21.15.72:0/1898868056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9001e1f70 con 0x7fd9000cfa90 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8ff7fe640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 0x7fd9001541d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000b3f0 tx=0x7fd8f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0015020 con 0x7fd9000cfa90 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f0011830 con 0x7fd9000cfa90 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0011c20 con 0x7fd9000cfa90 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9001e2250 con 0x7fd9000cfa90 2024-03-06T07:25:53.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.212+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9001e2810 con 0x7fd9000cfa90 2024-03-06T07:25:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.213+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9000732b0 con 0x7fd9000cfa90 2024-03-06T07:25:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.214+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd8f0018020 con 0x7fd9000cfa90 2024-03-06T07:25:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.214+0000 7fd8e7fff640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.214+0000 7fd8feffd640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.214+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8f0089e30 con 0x7fd9000cfa90 2024-03-06T07:25:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.215+0000 7fd8feffd640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd9000cbb80 tx=0x7fd8f4007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.218+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd8f001f030 con 0x7fd9000cfa90 2024-03-06T07:25:53.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.393+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd900068ff0 con 0x7fd9000cfa90 2024-03-06T07:25:53.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.394+0000 7fd8e7fff640 1 -- 172.21.15.72:0/1898868056 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd8f00172c0 con 0x7fd9000cfa90 2024-03-06T07:25:53.396 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd8ec06b4d0 msgr2=0x7fd8ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd9000cbb80 tx=0x7fd8f4007b50 comp rx=0 tx=0).stop 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 msgr2=0x7fd9001541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 0x7fd9001541d0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fd8f000b3f0 tx=0x7fd8f000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 shutdown_connections 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd8ec06b4d0 0x7fd8ec06d990 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9001586e0 0x7fd9001e3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9000cff40 0x7fd9001e18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 --2- 172.21.15.72:0/1898868056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9000cfa90 0x7fd9001541d0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 >> 172.21.15.72:0/1898868056 conn(0x7fd90006b190 msgr2=0x7fd9000b33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 shutdown_connections 2024-03-06T07:25:53.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.396+0000 7fd905b3e640 1 -- 172.21.15.72:0/1898868056 wait complete. 2024-03-06T07:25:53.409 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.511+0000 7f4457f3a640 1 Processor -- start 2024-03-06T07:25:53.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.511+0000 7f4457f3a640 1 -- start start 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4457f3a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f44501614d0 0x7f44500d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4457f3a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44500d24b0 0x7f44500b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4457f3a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44501618b0 con 0x7f44500d1910 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4457f3a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44500cf400 con 0x7f44501614d0 2024-03-06T07:25:53.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4457f3a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44500cf6a0 con 0x7f44500d24b0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4455cd6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f44501614d0 0x7f44500d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f44554d5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f44500d1910 0x7f44500d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35976/0 (socket says 172.21.15.72:35976) 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f44554d5640 1 -- 172.21.15.72:0/1056365936 learned_addr learned my addr 172.21.15.72:0/1056365936 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4455cd6640 1 -- 172.21.15.72:0/1056365936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44500d24b0 msgr2=0x7f44500b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f44564d7640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44500d24b0 0x7f44500b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4454cd4640 1 -- 172.21.15.72:0/1056365936 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934616708 0 0) 0x7f44501618b0 con 0x7f44500d1910 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.512+0000 7f4455cd6640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44500d24b0 0x7f44500b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f4455cd6640 1 -- 172.21.15.72:0/1056365936 >> v1:172.21.15.72:6789/0 conn(0x7f44500d1910 legacy=0x7f44500d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f4455cd6640 1 -- 172.21.15.72:0/1056365936 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500cf980 con 0x7f44501614d0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f44564d7640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44500d24b0 0x7f44500b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f4455cd6640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44501614d0 0x7f44500d4860 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f12f9407108d224 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f4454cd4640 1 -- 172.21.15.72:0/1056365936 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c035020 con 0x7f44501614d0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f4454cd4640 1 -- 172.21.15.72:0/1056365936 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f444c031830 con 0x7f44501614d0 2024-03-06T07:25:53.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.513+0000 7f4454cd4640 1 -- 172.21.15.72:0/1056365936 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f444c031c00 con 0x7f44501614d0 2024-03-06T07:25:53.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 -- 172.21.15.72:0/1056365936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44501614d0 msgr2=0x7f44500d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44501614d0 0x7f44500d4860 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f444c0097a0 tx=0x7f444c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 -- 172.21.15.72:0/1056365936 shutdown_connections 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44500d24b0 0x7f44500b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 --2- 172.21.15.72:0/1056365936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44501614d0 0x7f44500d4860 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 -- 172.21.15.72:0/1056365936 >> 172.21.15.72:0/1056365936 conn(0x7f44500d3d40 msgr2=0x7f44500d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 -- 172.21.15.72:0/1056365936 shutdown_connections 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.514+0000 7f4457f3a640 1 -- 172.21.15.72:0/1056365936 wait complete. 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.515+0000 7f4457f3a640 1 Processor -- start 2024-03-06T07:25:53.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.515+0000 7f4457f3a640 1 -- start start 2024-03-06T07:25:53.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4457f3a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44500d1910 0x7f44501f8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4457f3a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 0x7f4450150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4457f3a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44501614d0 0x7f44500d0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4457f3a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44500b1370 con 0x7f44500d24b0 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4457f3a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44500b0910 con 0x7f44500d1910 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4457f3a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44500b14f0 con 0x7f44501614d0 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 0x7f4450150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 0x7f4450150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42286/0 (socket says 172.21.15.72:42286) 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 -- 172.21.15.72:0/2491772279 learned_addr learned my addr 172.21.15.72:0/2491772279 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44564d7640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44501614d0 0x7f44500d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4455cd6640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44500d1910 0x7f44501f8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 -- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44501614d0 msgr2=0x7f44500d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44501614d0 0x7f44500d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 -- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44500d1910 msgr2=0x7f44501f8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44500d1910 0x7f44501f8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44554d5640 1 -- 172.21.15.72:0/2491772279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500c9f70 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f4455cd6640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44500d1910 0x7f44501f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.516+0000 7f44564d7640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44501614d0 0x7f44500d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.517+0000 7f44554d5640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 0x7f4450150d60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f444000c4d0 tx=0x7f444000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.517+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440018070 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.517+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4440007500 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.517+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44500ca7b0 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.517+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4440004a20 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.517+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44500ca1f0 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.518+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44500cf100 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.518+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f444001b070 con 0x7f44500d24b0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.519+0000 7f44467fc640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f443406b4d0 0x7f443406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.519+0000 7f4455cd6640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f443406b4d0 0x7f443406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.519+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4440089470 con 0x7f44500d24b0 2024-03-06T07:25:53.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.519+0000 7f4455cd6640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f443406b4d0 0x7f443406d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f444c036000 tx=0x7f444c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.527+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44500c9f70 con 0x7f44500d24b0 2024-03-06T07:25:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.703+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f44500c7cf0 con 0x7f44500d24b0 2024-03-06T07:25:53.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.703+0000 7f44467fc640 1 -- 172.21.15.72:0/2491772279 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4440058380 con 0x7f44500d24b0 2024-03-06T07:25:53.705 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f443406b4d0 msgr2=0x7f443406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f443406b4d0 0x7f443406d990 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f444c036000 tx=0x7f444c008030 comp rx=0 tx=0).stop 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 msgr2=0x7f4450150d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 0x7f4450150d60 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f444000c4d0 tx=0x7f444000c990 comp rx=0 tx=0).stop 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 shutdown_connections 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f443406b4d0 0x7f443406d990 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44501614d0 0x7f44500d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44500d24b0 0x7f4450150d60 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 --2- 172.21.15.72:0/2491772279 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44500d1910 0x7f44501f8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 >> 172.21.15.72:0/2491772279 conn(0x7f44500d3d40 msgr2=0x7f445006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 shutdown_connections 2024-03-06T07:25:53.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.705+0000 7f4457f3a640 1 -- 172.21.15.72:0/2491772279 wait complete. 2024-03-06T07:25:53.718 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.817+0000 7f66639c7640 1 Processor -- start 2024-03-06T07:25:53.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.817+0000 7f66639c7640 1 -- start start 2024-03-06T07:25:53.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f66639c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0cf470 0x7f665c0cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f66639c7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f665c0730d0 0x7f665c0b59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f66639c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f665c06bb70 0x7f665c06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f66639c7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c06c510 con 0x7f665c06bb70 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f66639c7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c0b4f70 con 0x7f665c0730d0 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f66639c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c15b160 con 0x7f665c0cf470 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f6661f64640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f665c06bb70 0x7f665c06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f6661f64640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f665c06bb70 0x7f665c06bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42300/0 (socket says 172.21.15.72:42300) 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f6661f64640 1 -- 172.21.15.72:0/1038343511 learned_addr learned my addr 172.21.15.72:0/1038343511 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.818+0000 7f6660f62640 1 --2- 172.21.15.72:0/1038343511 >> v2:172.21.15.146:3300/0 conn(0x7f665c0730d0 0x7f665c0b59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661763640 1 --2- 172.21.15.72:0/1038343511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0cf470 0x7f665c0cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661f64640 1 -- 172.21.15.72:0/1038343511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0cf470 msgr2=0x7f665c0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661f64640 1 --2- 172.21.15.72:0/1038343511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0cf470 0x7f665c0cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661f64640 1 -- 172.21.15.72:0/1038343511 >> v2:172.21.15.146:3300/0 conn(0x7f665c0730d0 msgr2=0x7f665c0b59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661f64640 1 --2- 172.21.15.72:0/1038343511 >> v2:172.21.15.146:3300/0 conn(0x7f665c0730d0 0x7f665c0b59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661f64640 1 -- 172.21.15.72:0/1038343511 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c0b5f00 con 0x7f665c06bb70 2024-03-06T07:25:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.819+0000 7f6661f64640 1 --2- 172.21.15.72:0/1038343511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c06bfd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f665c0721a0 tx=0x7f665402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec118d2cb85458a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f6653fff640 1 -- 172.21.15.72:0/1038343511 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665402ca70 con 0x7f665c06bb70 2024-03-06T07:25:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f6653fff640 1 -- 172.21.15.72:0/1038343511 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665402cc10 con 0x7f665c06bb70 2024-03-06T07:25:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f6653fff640 1 -- 172.21.15.72:0/1038343511 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66540343f0 con 0x7f665c06bb70 2024-03-06T07:25:53.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 -- 172.21.15.72:0/1038343511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 msgr2=0x7f665c06bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 --2- 172.21.15.72:0/1038343511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c06bfd0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f665c0721a0 tx=0x7f665402bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 -- 172.21.15.72:0/1038343511 shutdown_connections 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 --2- 172.21.15.72:0/1038343511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c06bfd0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 --2- 172.21.15.72:0/1038343511 >> v2:172.21.15.146:3300/0 conn(0x7f665c0730d0 0x7f665c0b59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 --2- 172.21.15.72:0/1038343511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0cf470 0x7f665c0cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 -- 172.21.15.72:0/1038343511 >> 172.21.15.72:0/1038343511 conn(0x7f665c0724d0 msgr2=0x7f665c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.820+0000 7f66639c7640 1 -- 172.21.15.72:0/1038343511 shutdown_connections 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.821+0000 7f66639c7640 1 -- 172.21.15.72:0/1038343511 wait complete. 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 Processor -- start 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 -- start start 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0730d0 0x7f665c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c0cf470 0x7f665c150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0b51c0 con 0x7f665c06bb70 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0b19d0 con 0x7f665c0cf470 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f66639c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0b54a0 con 0x7f665c0730d0 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f6661763640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f6661763640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42308/0 (socket says 172.21.15.72:42308) 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.822+0000 7f6661763640 1 -- 172.21.15.72:0/4008532672 learned_addr learned my addr 172.21.15.72:0/4008532672 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6660f62640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0730d0 0x7f665c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661f64640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c0cf470 0x7f665c150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661763640 1 -- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0730d0 msgr2=0x7f665c1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661763640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0730d0 0x7f665c1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661763640 1 -- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c0cf470 msgr2=0x7f665c150b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661763640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c0cf470 0x7f665c150b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661763640 1 -- 172.21.15.72:0/4008532672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c154d70 con 0x7f665c06bb70 2024-03-06T07:25:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6661763640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c154690 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f665800e9b0 tx=0x7f665800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658017070 con 0x7f665c06bb70 2024-03-06T07:25:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665800c9a0 con 0x7f665c06bb70 2024-03-06T07:25:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665800cd90 con 0x7f665c06bb70 2024-03-06T07:25:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c1f3480 con 0x7f665c06bb70 2024-03-06T07:25:53.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.823+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c1f3a40 con 0x7f665c06bb70 2024-03-06T07:25:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.824+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c04f330 con 0x7f665c06bb70 2024-03-06T07:25:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.825+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f665801a070 con 0x7f665c06bb70 2024-03-06T07:25:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.825+0000 7f6651ffb640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.825+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f665808a590 con 0x7f665c06bb70 2024-03-06T07:25:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.825+0000 7f6660f62640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:53.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.826+0000 7f6660f62640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f664006b4d0 0x7f664006d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f665c15a3f0 tx=0x7f664c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:53.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:53.830+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6658059410 con 0x7f665c06bb70 2024-03-06T07:25:54.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.047+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f665c0713c0 con 0x7f665c06bb70 2024-03-06T07:25:54.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.048+0000 7f6651ffb640 1 -- 172.21.15.72:0/4008532672 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f6658058d10 con 0x7f665c06bb70 2024-03-06T07:25:54.050 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:54.050 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:54.050 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f664006b4d0 msgr2=0x7f664006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f664006b4d0 0x7f664006d990 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f665c15a3f0 tx=0x7f664c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 msgr2=0x7f665c154690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c154690 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f665800e9b0 tx=0x7f665800ee70 comp rx=0 tx=0).stop 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 shutdown_connections 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f664006b4d0 0x7f664006d990 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c0cf470 0x7f665c150b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.049+0000 7f66639c7640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c0730d0 0x7f665c1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.050+0000 7f66639c7640 1 --2- 172.21.15.72:0/4008532672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f665c06bb70 0x7f665c154690 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.050+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 >> 172.21.15.72:0/4008532672 conn(0x7f665c0724d0 msgr2=0x7f665c159080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.050+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 shutdown_connections 2024-03-06T07:25:54.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.050+0000 7f66639c7640 1 -- 172.21.15.72:0/4008532672 wait complete. 2024-03-06T07:25:54.062 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:54.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.161+0000 7f990658a640 1 Processor -- start 2024-03-06T07:25:54.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.162+0000 7f990658a640 1 -- start start 2024-03-06T07:25:54.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f990658a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f990015bb20 0x7f99000d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f990658a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99000cfff0 0x7f99000d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f990658a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99000ce4b0 0x7f99000ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f990658a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9900161980 con 0x7f99000ce4b0 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f990658a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9900161c60 con 0x7f990015bb20 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f990658a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9900161f40 con 0x7f99000cfff0 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99000ce4b0 0x7f99000ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99000ce4b0 0x7f99000ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42312/0 (socket says 172.21.15.72:42312) 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 -- 172.21.15.72:0/153073357 learned_addr learned my addr 172.21.15.72:0/153073357 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f98fffff640 1 --2- 172.21.15.72:0/153073357 >> v2:172.21.15.146:3300/0 conn(0x7f990015bb20 0x7f99000d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f98ff7fe640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99000cfff0 0x7f99000d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 -- 172.21.15.72:0/153073357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99000cfff0 msgr2=0x7f99000d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99000cfff0 0x7f99000d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 -- 172.21.15.72:0/153073357 >> v2:172.21.15.146:3300/0 conn(0x7f990015bb20 msgr2=0x7f99000d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 --2- 172.21.15.72:0/153073357 >> v2:172.21.15.146:3300/0 conn(0x7f990015bb20 0x7f99000d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f9904b27640 1 -- 172.21.15.72:0/153073357 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9900162220 con 0x7f99000ce4b0 2024-03-06T07:25:54.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f98fffff640 1 --2- 172.21.15.72:0/153073357 >> v2:172.21.15.146:3300/0 conn(0x7f990015bb20 0x7f99000d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:54.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.163+0000 7f98ff7fe640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99000cfff0 0x7f99000d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.164+0000 7f9904b27640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99000ce940 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f99000d2bc0 tx=0x7f98f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1884787ee3cb1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.164+0000 7f98feffd640 1 -- 172.21.15.72:0/153073357 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f802ca70 con 0x7f99000ce4b0 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.164+0000 7f98feffd640 1 -- 172.21.15.72:0/153073357 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f802cc10 con 0x7f99000ce4b0 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.164+0000 7f98feffd640 1 -- 172.21.15.72:0/153073357 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f80343f0 con 0x7f99000ce4b0 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 -- 172.21.15.72:0/153073357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 msgr2=0x7f99000ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99000ce940 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f99000d2bc0 tx=0x7f98f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 -- 172.21.15.72:0/153073357 shutdown_connections 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99000ce940 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 --2- 172.21.15.72:0/153073357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99000cfff0 0x7f99000d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 --2- 172.21.15.72:0/153073357 >> v2:172.21.15.146:3300/0 conn(0x7f990015bb20 0x7f99000d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 -- 172.21.15.72:0/153073357 >> 172.21.15.72:0/153073357 conn(0x7f990009a9f0 msgr2=0x7f990009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 -- 172.21.15.72:0/153073357 shutdown_connections 2024-03-06T07:25:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.165+0000 7f990658a640 1 -- 172.21.15.72:0/153073357 wait complete. 2024-03-06T07:25:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.167+0000 7f990658a640 1 Processor -- start 2024-03-06T07:25:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.167+0000 7f990658a640 1 -- start start 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f990658a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99001f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f990658a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99000cfff0 0x7f990015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f990658a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990015bb20 0x7f99000b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f990658a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99000d24b0 con 0x7f99000ce4b0 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f990658a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99000d2330 con 0x7f99000cfff0 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f990658a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99000d1980 con 0x7f990015bb20 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f98fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99001f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f98fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99001f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42314/0 (socket says 172.21.15.72:42314) 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.168+0000 7f98fffff640 1 -- 172.21.15.72:0/886032775 learned_addr learned my addr 172.21.15.72:0/886032775 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98ff7fe640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99000cfff0 0x7f990015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f9904b27640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990015bb20 0x7f99000b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fffff640 1 -- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990015bb20 msgr2=0x7f99000b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fffff640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990015bb20 0x7f99000b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fffff640 1 -- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99000cfff0 msgr2=0x7f990015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fffff640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99000cfff0 0x7f990015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fffff640 1 -- 172.21.15.72:0/886032775 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990015c800 con 0x7f99000ce4b0 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fffff640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99001f0fb0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c820 tx=0x7f98f000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0015070 con 0x7f99000ce4b0 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f0007a30 con 0x7f99000ce4b0 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990015cae0 con 0x7f99000ce4b0 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0007de0 con 0x7f99000ce4b0 2024-03-06T07:25:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.169+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990015d0a0 con 0x7f99000ce4b0 2024-03-06T07:25:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.170+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990004f330 con 0x7f99000ce4b0 2024-03-06T07:25:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.170+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98f0018070 con 0x7f99000ce4b0 2024-03-06T07:25:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.171+0000 7f98fcff9640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98ec06b4d0 0x7f98ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.171+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98f008a0c0 con 0x7f99000ce4b0 2024-03-06T07:25:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.171+0000 7f98ff7fe640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98ec06b4d0 0x7f98ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.171+0000 7f98ff7fe640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98ec06b4d0 0x7f98ec06d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f990006e260 tx=0x7f98f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.175+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98f0058fb0 con 0x7f99000ce4b0 2024-03-06T07:25:54.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.347+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99000b5070 con 0x7f99000ce4b0 2024-03-06T07:25:54.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.348+0000 7f98fcff9640 1 -- 172.21.15.72:0/886032775 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f98f00588b0 con 0x7f99000ce4b0 2024-03-06T07:25:54.351 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:54.351 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:38.966454+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:54.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98ec06b4d0 msgr2=0x7f98ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98ec06b4d0 0x7f98ec06d990 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f990006e260 tx=0x7f98f4008040 comp rx=0 tx=0).stop 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 msgr2=0x7f99001f0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99001f0fb0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c820 tx=0x7f98f000cce0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 shutdown_connections 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98ec06b4d0 0x7f98ec06d990 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990015bb20 0x7f99000b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99000cfff0 0x7f990015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 --2- 172.21.15.72:0/886032775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99000ce4b0 0x7f99001f0fb0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 >> 172.21.15.72:0/886032775 conn(0x7f990009a9f0 msgr2=0x7f99000703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 shutdown_connections 2024-03-06T07:25:54.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.350+0000 7f990658a640 1 -- 172.21.15.72:0/886032775 wait complete. 2024-03-06T07:25:54.364 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:25:54.364 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:54.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.463+0000 7f45f62df640 1 Processor -- start 2024-03-06T07:25:54.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.463+0000 7f45f62df640 1 -- start start 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45f62df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f45f00d2710 0x7f45f00b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45f62df640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00b3430 con 0x7f45f00d22b0 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45f62df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45f00b3710 con 0x7f45f015dfe0 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45f62df640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f00b31e0 con 0x7f45f00d2710 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45ef7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f45f00d22b0 0x7f45f00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35986/0 (socket says 172.21.15.72:35986) 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45ef7fe640 1 -- 172.21.15.72:0/70251902 learned_addr learned my addr 172.21.15.72:0/70251902 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:54.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45f487c640 1 --2- 172.21.15.72:0/70251902 >> v2:172.21.15.146:3300/0 conn(0x7f45f00d2710 0x7f45f00b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213684483 0 0) 0x7f45f00b3430 con 0x7f45f00d22b0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d8003520 con 0x7f45f00d22b0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3967375311 0 0) 0x7f45d8003520 con 0x7f45f00d22b0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 >> v2:172.21.15.146:3300/0 conn(0x7f45f00d2710 msgr2=0x7f45f00b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 --2- 172.21.15.72:0/70251902 >> v2:172.21.15.146:3300/0 conn(0x7f45f00d2710 0x7f45f00b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 >> v1:172.21.15.146:6789/0 conn(0x7f45f015dfe0 legacy=0x7f45f00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f00b39f0 con 0x7f45f00d22b0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f45e4001670 con 0x7f45f00d22b0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 >> v1:172.21.15.72:6789/0 conn(0x7f45f00d22b0 legacy=0x7f45f00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80039e0 0x7f45d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d8004800 0x7f45d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.464+0000 7f45eeffd640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45d80051c0 0x7f45d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00b31e0 con 0x7f45d80051c0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d800c090 con 0x7f45d80039e0 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d800c290 con 0x7f45d8004800 2024-03-06T07:25:54.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45f487c640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45d80051c0 0x7f45d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45effff640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80039e0 0x7f45d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45ef7fe640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d8004800 0x7f45d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45f487c640 1 -- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d8004800 msgr2=0x7f45d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45f487c640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d8004800 0x7f45d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45f487c640 1 -- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80039e0 msgr2=0x7f45d8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45f487c640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80039e0 0x7f45d8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.465+0000 7f45f487c640 1 -- 172.21.15.72:0/70251902 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d8003520 con 0x7f45d80051c0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.466+0000 7f45f487c640 1 -- 172.21.15.72:0/70251902 reap_dead start 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.466+0000 7f45f487c640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45d80051c0 0x7f45d800b9c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f45e8009fc0 tx=0x7f45d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e255263a7f30b6b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.466+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8009c10 con 0x7f45d80051c0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.466+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e802cca0 con 0x7f45d80051c0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.466+0000 7f45eeffd640 1 -- 172.21.15.72:0/70251902 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e8003bd0 con 0x7f45d80051c0 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 -- 172.21.15.72:0/70251902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45d80051c0 msgr2=0x7f45d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45d80051c0 0x7f45d800b9c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f45e8009fc0 tx=0x7f45d8005830 comp rx=0 tx=0).stop 2024-03-06T07:25:54.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 -- 172.21.15.72:0/70251902 shutdown_connections 2024-03-06T07:25:54.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 --2- 172.21.15.72:0/70251902 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45d80051c0 0x7f45d800b9c0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 -- 172.21.15.72:0/70251902 >> 172.21.15.72:0/70251902 conn(0x7f45f00d3cd0 msgr2=0x7f45f00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 -- 172.21.15.72:0/70251902 shutdown_connections 2024-03-06T07:25:54.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.467+0000 7f45f62df640 1 -- 172.21.15.72:0/70251902 wait complete. 2024-03-06T07:25:54.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.468+0000 7f45f62df640 1 Processor -- start 2024-03-06T07:25:54.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.468+0000 7f45f62df640 1 -- start start 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f62df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80051c0 0x7f45f00cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f62df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 0x7f45f015b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f62df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45f00ca160 0x7f45f00ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f62df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00d1c70 con 0x7f45f015ee10 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f62df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00b3d60 con 0x7f45d80051c0 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f62df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f00d1df0 con 0x7f45f00ca160 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 0x7f45f015b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 0x7f45f015b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42344/0 (socket says 172.21.15.72:42344) 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 -- 172.21.15.72:0/2770079054 learned_addr learned my addr 172.21.15.72:0/2770079054 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45effff640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80051c0 0x7f45f00cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45f487c640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45f00ca160 0x7f45f00ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 -- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45f00ca160 msgr2=0x7f45f00ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45f00ca160 0x7f45f00ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 -- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80051c0 msgr2=0x7f45f00cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80051c0 0x7f45f00cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 -- 172.21.15.72:0/2770079054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f009ebd0 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45effff640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80051c0 0x7f45f00cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ef7fe640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 0x7f45f015b600 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f45e4007290 tx=0x7f45e4007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.469+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e4007e10 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.470+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e4018070 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.470+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f00cdb90 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.470+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e400fbc0 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.470+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f015f690 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.471+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f004f330 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.471+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f45e401c410 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.472+0000 7f45ecff9640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f45dc06b500 0x7f45dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.472+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45e4089d00 con 0x7f45f015ee10 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.472+0000 7f45effff640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f45dc06b500 0x7f45dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.472+0000 7f45effff640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f45dc06b500 0x7f45dc06d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.480+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f45f009ebd0 con 0x7f45f015ee10 2024-03-06T07:25:54.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.694+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45f00b2fc0 con 0x7f45f015ee10 2024-03-06T07:25:54.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.694+0000 7f45ecff9640 1 -- 172.21.15.72:0/2770079054 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f45e4022250 con 0x7f45f015ee10 2024-03-06T07:25:54.697 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:54.697 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:54.697 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f45dc06b500 msgr2=0x7f45dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f45dc06b500 0x7f45dc06d9c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0009040 comp rx=0 tx=0).stop 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 msgr2=0x7f45f015b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 0x7f45f015b600 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f45e4007290 tx=0x7f45e4007750 comp rx=0 tx=0).stop 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 shutdown_connections 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f45dc06b500 0x7f45dc06d9c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45f00ca160 0x7f45f00ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45f015ee10 0x7f45f015b600 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 --2- 172.21.15.72:0/2770079054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d80051c0 0x7f45f00cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 >> 172.21.15.72:0/2770079054 conn(0x7f45f00d3cd0 msgr2=0x7f45f00b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 shutdown_connections 2024-03-06T07:25:54.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.696+0000 7f45f62df640 1 -- 172.21.15.72:0/2770079054 wait complete. 2024-03-06T07:25:54.713 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:25:54.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 Processor -- start 2024-03-06T07:25:54.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 -- start start 2024-03-06T07:25:54.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f67600d22b0 0x7f67600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600b41f0 0x7f67600b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67600b3430 con 0x7f676015dfe0 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67600d2c50 con 0x7f67600d22b0 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f67667ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67600b31e0 con 0x7f67600b41f0 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.813+0000 7f675f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f67600d22b0 0x7f67600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f676015dfe0 0x7f67600cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35998/0 (socket says 172.21.15.72:35998) 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675ffff640 1 -- 172.21.15.72:0/2924585762 learned_addr learned my addr 172.21.15.72:0/2924585762 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f6764d49640 1 --2- 172.21.15.72:0/2924585762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600b41f0 0x7f67600b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675f7fe640 1 -- 172.21.15.72:0/2924585762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600b41f0 msgr2=0x7f67600b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675f7fe640 1 --2- 172.21.15.72:0/2924585762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600b41f0 0x7f67600b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675f7fe640 1 -- 172.21.15.72:0/2924585762 >> v1:172.21.15.72:6789/0 conn(0x7f676015dfe0 legacy=0x7f67600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675effd640 1 -- 172.21.15.72:0/2924585762 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775514890 0 0) 0x7f67600b3430 con 0x7f676015dfe0 2024-03-06T07:25:54.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.814+0000 7f675f7fe640 1 -- 172.21.15.72:0/2924585762 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67600b3710 con 0x7f67600d22b0 2024-03-06T07:25:54.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f675f7fe640 1 --2- 172.21.15.72:0/2924585762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67600d22b0 0x7f67600d2710 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f67540097a0 tx=0x7f675402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbf7ce3424344d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f675effd640 1 -- 172.21.15.72:0/2924585762 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754035020 con 0x7f67600d22b0 2024-03-06T07:25:54.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f675effd640 1 -- 172.21.15.72:0/2924585762 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6754031830 con 0x7f67600d22b0 2024-03-06T07:25:54.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f675effd640 1 -- 172.21.15.72:0/2924585762 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754031c00 con 0x7f67600d22b0 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 -- 172.21.15.72:0/2924585762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67600d22b0 msgr2=0x7f67600d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 --2- 172.21.15.72:0/2924585762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67600d22b0 0x7f67600d2710 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f67540097a0 tx=0x7f675402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 -- 172.21.15.72:0/2924585762 shutdown_connections 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 --2- 172.21.15.72:0/2924585762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600b41f0 0x7f67600b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 --2- 172.21.15.72:0/2924585762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67600d22b0 0x7f67600d2710 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 -- 172.21.15.72:0/2924585762 >> 172.21.15.72:0/2924585762 conn(0x7f67600d3cd0 msgr2=0x7f67600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:54.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 -- 172.21.15.72:0/2924585762 shutdown_connections 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.815+0000 7f67667ac640 1 -- 172.21.15.72:0/2924585762 wait complete. 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.816+0000 7f67667ac640 1 Processor -- start 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 -- start start 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67600b41f0 0x7f676009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600d22b0 0x7f676009f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 0x7f67600cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67600cfa00 con 0x7f676015dfe0 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67600b3d60 con 0x7f67600b41f0 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.817+0000 7f67667ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67600d1c70 con 0x7f67600d22b0 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 0x7f67600cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 0x7f67600cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42354/0 (socket says 172.21.15.72:42354) 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 -- 172.21.15.72:0/4065898675 learned_addr learned my addr 172.21.15.72:0/4065898675 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f675f7fe640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600d22b0 0x7f676009f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f675ffff640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67600b41f0 0x7f676009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 -- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600d22b0 msgr2=0x7f676009f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600d22b0 0x7f676009f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 -- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67600b41f0 msgr2=0x7f676009ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67600b41f0 0x7f676009ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 -- 172.21.15.72:0/4065898675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676009fa30 con 0x7f676015dfe0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f675f7fe640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600d22b0 0x7f676009f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f6764d49640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 0x7f67600cbdf0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f6758007e40 tx=0x7f675800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758014620 con 0x7f676015dfe0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6758019070 con 0x7f676015dfe0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676009fd10 con 0x7f676015dfe0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758014a20 con 0x7f676015dfe0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.818+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67600a02d0 con 0x7f676015dfe0 2024-03-06T07:25:54.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.819+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676004f330 con 0x7f676015dfe0 2024-03-06T07:25:54.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.820+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6758014bc0 con 0x7f676015dfe0 2024-03-06T07:25:54.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.820+0000 7f675cff9640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f674c06b4d0 0x7f674c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:54.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.820+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f675808ab70 con 0x7f676015dfe0 2024-03-06T07:25:54.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.820+0000 7f675ffff640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f674c06b4d0 0x7f674c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:54.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.820+0000 7f675ffff640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f674c06b4d0 0x7f674c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6750000f60 tx=0x7f6750009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:54.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:54.828+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6758059970 con 0x7f676015dfe0 2024-03-06T07:25:55.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.044+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67600b2c70 con 0x7f676015dfe0 2024-03-06T07:25:55.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.044+0000 7f675cff9640 1 -- 172.21.15.72:0/4065898675 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f6758059270 con 0x7f676015dfe0 2024-03-06T07:25:55.047 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:55.047 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":39,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:25:37.310184+0000","modified":"2024-03-06T07:25:39.352824+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6987},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6987":{"gid":6987,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/2658331491","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2658331491},{"type":"v1","addr":"172.21.15.72:6836","nonce":2658331491}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6987,"qdb_cluster":[6987]},"id":6}]} 2024-03-06T07:25:55.047 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 39 2024-03-06T07:25:55.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f674c06b4d0 msgr2=0x7f674c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f674c06b4d0 0x7f674c06d990 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6750000f60 tx=0x7f6750009040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 msgr2=0x7f67600cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 0x7f67600cbdf0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f6758007e40 tx=0x7f675800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 shutdown_connections 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f674c06b4d0 0x7f674c06d990 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f676015dfe0 0x7f67600cbdf0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67600d22b0 0x7f676009f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 --2- 172.21.15.72:0/4065898675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67600b41f0 0x7f676009ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 >> 172.21.15.72:0/4065898675 conn(0x7f67600d3cd0 msgr2=0x7f676006c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 shutdown_connections 2024-03-06T07:25:55.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.047+0000 7f67667ac640 1 -- 172.21.15.72:0/4065898675 wait complete. 2024-03-06T07:25:55.060 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:55.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.160+0000 7f1e5e878640 1 Processor -- start 2024-03-06T07:25:55.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.160+0000 7f1e5e878640 1 -- start start 2024-03-06T07:25:55.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5e878640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1e580d04a0 0x7f1e580ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5e878640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e58161980 con 0x7f1e580cfff0 2024-03-06T07:25:55.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5e878640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e58161c60 con 0x7f1e5815bb20 2024-03-06T07:25:55.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5e878640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e580d2500 con 0x7f1e580d04a0 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5ce15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1e580d04a0 0x7f1e580ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5ce15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1e580d04a0 0x7f1e580ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42364/0 (socket says 172.21.15.72:42364) 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e577fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1e580cfff0 0x7f1e580cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36002/0 (socket says 172.21.15.72:36002) 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5ce15640 1 -- 172.21.15.72:0/980238895 learned_addr learned my addr 172.21.15.72:0/980238895 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5ce15640 1 -- 172.21.15.72:0/980238895 >> v1:172.21.15.146:6789/0 conn(0x7f1e5815bb20 legacy=0x7f1e580d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5ce15640 1 -- 172.21.15.72:0/980238895 >> v1:172.21.15.72:6789/0 conn(0x7f1e580cfff0 legacy=0x7f1e580cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.161+0000 7f1e5ce15640 1 -- 172.21.15.72:0/980238895 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e58161f40 con 0x7f1e580d04a0 2024-03-06T07:25:55.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e5ce15640 1 --2- 172.21.15.72:0/980238895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e580d04a0 0x7f1e580ccfc0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009e50 tx=0x7f1e5002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952f8c31f47e8106 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e56ffd640 1 -- 172.21.15.72:0/980238895 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5002cb20 con 0x7f1e580d04a0 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e56ffd640 1 -- 172.21.15.72:0/980238895 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5002ccc0 con 0x7f1e580d04a0 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e56ffd640 1 -- 172.21.15.72:0/980238895 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50034610 con 0x7f1e580d04a0 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e5e878640 1 -- 172.21.15.72:0/980238895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e580d04a0 msgr2=0x7f1e580ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e5e878640 1 --2- 172.21.15.72:0/980238895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e580d04a0 0x7f1e580ccfc0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009e50 tx=0x7f1e5002bf80 comp rx=0 tx=0).stop 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e5e878640 1 -- 172.21.15.72:0/980238895 shutdown_connections 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e5e878640 1 --2- 172.21.15.72:0/980238895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e580d04a0 0x7f1e580ccfc0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.162+0000 7f1e5e878640 1 -- 172.21.15.72:0/980238895 >> 172.21.15.72:0/980238895 conn(0x7f1e5809a9f0 msgr2=0x7f1e5809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.163+0000 7f1e5e878640 1 -- 172.21.15.72:0/980238895 shutdown_connections 2024-03-06T07:25:55.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.163+0000 7f1e5e878640 1 -- 172.21.15.72:0/980238895 wait complete. 2024-03-06T07:25:55.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.164+0000 7f1e5e878640 1 Processor -- start 2024-03-06T07:25:55.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.165+0000 7f1e5e878640 1 -- start start 2024-03-06T07:25:55.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5e878640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e580cfff0 0x7f1e5815d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5e878640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e580d04a0 0x7f1e5815da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5e878640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 0x7f1e581f7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5e878640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e580d08d0 con 0x7f1e5815bb20 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5e878640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e58162500 con 0x7f1e580d04a0 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5e878640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e580d1980 con 0x7f1e580cfff0 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 0x7f1e581f7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 0x7f1e581f7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42378/0 (socket says 172.21.15.72:42378) 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 -- 172.21.15.72:0/3282482718 learned_addr learned my addr 172.21.15.72:0/3282482718 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e577fe640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e580d04a0 0x7f1e5815da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e57fff640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e580cfff0 0x7f1e5815d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 -- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e580cfff0 msgr2=0x7f1e5815d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e580cfff0 0x7f1e5815d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 -- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e580d04a0 msgr2=0x7f1e5815da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e580d04a0 0x7f1e5815da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 -- 172.21.15.72:0/3282482718 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e581e7a30 con 0x7f1e5815bb20 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e57fff640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e580cfff0 0x7f1e5815d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e5ce15640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 0x7f1e581f7b50 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f1e50036040 tx=0x7f1e50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e50043020 con 0x7f1e5815bb20 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.166+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5003e450 con 0x7f1e5815bb20 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.167+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e581e7d10 con 0x7f1e5815bb20 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.167+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5003e860 con 0x7f1e5815bb20 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.167+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e581e82d0 con 0x7f1e5815bb20 2024-03-06T07:25:55.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.167+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5804f330 con 0x7f1e5815bb20 2024-03-06T07:25:55.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.168+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e50046020 con 0x7f1e5815bb20 2024-03-06T07:25:55.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.169+0000 7f1e54ff9640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.169+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e500acb20 con 0x7f1e5815bb20 2024-03-06T07:25:55.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.169+0000 7f1e57fff640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.169+0000 7f1e57fff640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e3806b4d0 0x7f1e3806d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1e580b7a10 tx=0x7f1e48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.176+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1e5004d030 con 0x7f1e5815bb20 2024-03-06T07:25:55.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.349+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e580d1760 con 0x7f1e5815bb20 2024-03-06T07:25:55.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.350+0000 7f1e54ff9640 1 -- 172.21.15.72:0/3282482718 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16672 (secure 0 0 0) 0x7f1e5007b9a0 con 0x7f1e5815bb20 2024-03-06T07:25:55.352 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:25:55.352 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":83,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:38.966454+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:25:35.525319+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-06T07:25:36.277249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:25:55.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e3806b4d0 msgr2=0x7f1e3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e3806b4d0 0x7f1e3806d990 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1e580b7a10 tx=0x7f1e48008040 comp rx=0 tx=0).stop 2024-03-06T07:25:55.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 msgr2=0x7f1e581f7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 0x7f1e581f7b50 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f1e50036040 tx=0x7f1e50004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:55.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 shutdown_connections 2024-03-06T07:25:55.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e3806b4d0 0x7f1e3806d990 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1e5815bb20 0x7f1e581f7b50 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e580d04a0 0x7f1e5815da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 --2- 172.21.15.72:0/3282482718 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e580cfff0 0x7f1e5815d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 >> 172.21.15.72:0/3282482718 conn(0x7f1e5809a9f0 msgr2=0x7f1e5806fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 shutdown_connections 2024-03-06T07:25:55.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.352+0000 7f1e5e878640 1 -- 172.21.15.72:0/3282482718 wait complete. 2024-03-06T07:25:55.365 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:25:55.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.465+0000 7fb1172a4640 1 Processor -- start 2024-03-06T07:25:55.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.465+0000 7fb1172a4640 1 -- start start 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb1172a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1100b4460 0x7fb1100d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb1172a4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb110162970 0x7fb1100b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb1172a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb110161cf0 0x7fb110162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb1172a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb110161b00 con 0x7fb1100b4460 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb1172a4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1100cecc0 con 0x7fb110162970 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb1172a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100d22c0 con 0x7fb110161cf0 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1100b4460 0x7fb1100d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1100b4460 0x7fb1100d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42388/0 (socket says 172.21.15.72:42388) 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 -- 172.21.15.72:0/3961613698 learned_addr learned my addr 172.21.15.72:0/3961613698 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb11483f640 1 --2- 172.21.15.72:0/3961613698 >> v2:172.21.15.146:3300/0 conn(0x7fb110162970 0x7fb1100b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 -- 172.21.15.72:0/3961613698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb110161cf0 msgr2=0x7fb110162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115841640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb110161cf0 0x7fb110162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb110161cf0 0x7fb110162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 -- 172.21.15.72:0/3961613698 >> v2:172.21.15.146:3300/0 conn(0x7fb110162970 msgr2=0x7fb1100b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 --2- 172.21.15.72:0/3961613698 >> v2:172.21.15.146:3300/0 conn(0x7fb110162970 0x7fb1100b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115040640 1 -- 172.21.15.72:0/3961613698 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1100d25a0 con 0x7fb1100b4460 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb11483f640 1 --2- 172.21.15.72:0/3961613698 >> v2:172.21.15.146:3300/0 conn(0x7fb110162970 0x7fb1100b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.466+0000 7fb115841640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb110161cf0 0x7fb110162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.467+0000 7fb115040640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1100b4460 0x7fb1100d4800 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb1100d17e0 tx=0x7fb10c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2980471aa3ac5d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.467+0000 7fb107fff640 1 -- 172.21.15.72:0/3961613698 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c035020 con 0x7fb1100b4460 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.467+0000 7fb107fff640 1 -- 172.21.15.72:0/3961613698 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c02d040 con 0x7fb1100b4460 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.467+0000 7fb107fff640 1 -- 172.21.15.72:0/3961613698 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c031a80 con 0x7fb1100b4460 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.467+0000 7fb1172a4640 1 -- 172.21.15.72:0/3961613698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1100b4460 msgr2=0x7fb1100d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.467+0000 7fb1172a4640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1100b4460 0x7fb1100d4800 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fb1100d17e0 tx=0x7fb10c02b900 comp rx=0 tx=0).stop 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 -- 172.21.15.72:0/3961613698 shutdown_connections 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb110161cf0 0x7fb110162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 --2- 172.21.15.72:0/3961613698 >> v2:172.21.15.146:3300/0 conn(0x7fb110162970 0x7fb1100b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 --2- 172.21.15.72:0/3961613698 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1100b4460 0x7fb1100d4800 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 -- 172.21.15.72:0/3961613698 >> 172.21.15.72:0/3961613698 conn(0x7fb1100d3ce0 msgr2=0x7fb1100d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 -- 172.21.15.72:0/3961613698 shutdown_connections 2024-03-06T07:25:55.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.468+0000 7fb1172a4640 1 -- 172.21.15.72:0/3961613698 wait complete. 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 Processor -- start 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 -- start start 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1100b4460 0x7fb1100b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 0x7fb1100c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb110162970 0x7fb1100b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100ce950 con 0x7fb110161cf0 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100d2b60 con 0x7fb110162970 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.469+0000 7fb1172a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1100b3ac0 con 0x7fb1100b4460 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 0x7fb1100c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 0x7fb1100c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42394/0 (socket says 172.21.15.72:42394) 2024-03-06T07:25:55.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 -- 172.21.15.72:0/114658154 learned_addr learned my addr 172.21.15.72:0/114658154 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 -- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1100b4460 msgr2=0x7fb1100b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1100b4460 0x7fb1100b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 -- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb110162970 msgr2=0x7fb1100b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb115841640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb110162970 0x7fb1100b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb110162970 0x7fb1100b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 -- 172.21.15.72:0/114658154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1100c78b0 con 0x7fb110161cf0 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb11483f640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 0x7fb1100c9690 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c7f0 tx=0x7fb10000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb100015070 con 0x7fb110161cf0 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb100007a30 con 0x7fb110161cf0 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1100c7b90 con 0x7fb110161cf0 2024-03-06T07:25:55.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb100007e00 con 0x7fb110161cf0 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.470+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1100c8150 con 0x7fb110161cf0 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.471+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11008da10 con 0x7fb110161cf0 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.472+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb100018070 con 0x7fb110161cf0 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.472+0000 7fb105ffb640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0f406b4d0 0x7fb0f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.472+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb10008a120 con 0x7fb110161cf0 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.472+0000 7fb115040640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0f406b4d0 0x7fb0f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.473+0000 7fb115040640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0f406b4d0 0x7fb0f406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb10c036000 tx=0x7fb10c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.477+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb11008da10 con 0x7fb110161cf0 2024-03-06T07:25:55.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.686+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb1100c8350 con 0x7fb110161cf0 2024-03-06T07:25:55.688 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:25:55.687+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:25:55.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.688+0000 7fb105ffb640 1 -- 172.21.15.72:0/114658154 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v40) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb1100c8350 con 0x7fb110161cf0 2024-03-06T07:25:55.690 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:25:55.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0f406b4d0 msgr2=0x7fb0f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0f406b4d0 0x7fb0f406d990 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb10c036000 tx=0x7fb10c030000 comp rx=0 tx=0).stop 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 msgr2=0x7fb1100c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 0x7fb1100c9690 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c7f0 tx=0x7fb10000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 shutdown_connections 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0f406b4d0 0x7fb0f406d990 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb110162970 0x7fb1100b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb110161cf0 0x7fb1100c9690 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 --2- 172.21.15.72:0/114658154 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1100b4460 0x7fb1100b7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 >> 172.21.15.72:0/114658154 conn(0x7fb1100d3ce0 msgr2=0x7fb11006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 shutdown_connections 2024-03-06T07:25:55.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.691+0000 7fb1172a4640 1 -- 172.21.15.72:0/114658154 wait complete. 2024-03-06T07:25:55.695 INFO:tasks.ceph.mds.a.smithi072.stderr: -2777> 2024-03-06T07:25:43.225+0000 7f6e7e3e8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:25:55.706 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:25:55.722 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:25:55.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.807+0000 7f4a8881c640 1 Processor -- start 2024-03-06T07:25:55.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.807+0000 7f4a8881c640 1 -- start start 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.807+0000 7f4a8881c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a841586e0 0x7f4a840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.807+0000 7f4a8881c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840aef80 0x7f4a840cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.807+0000 7f4a8881c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a840cffd0 0x7f4a8406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a8881c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a840d03f0 con 0x7f4a840cffd0 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a8881c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a84158ac0 con 0x7f4a841586e0 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a8881c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8415b2a0 con 0x7f4a840aef80 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a840cffd0 0x7f4a8406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a840cffd0 0x7f4a8406da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42424/0 (socket says 172.21.15.72:42424) 2024-03-06T07:25:55.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 -- 172.21.15.72:0/3664076169 learned_addr learned my addr 172.21.15.72:0/3664076169 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a79d9b640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840aef80 0x7f4a840cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 -- 172.21.15.72:0/3664076169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840aef80 msgr2=0x7f4a840cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a8259c640 1 --2- 172.21.15.72:0/3664076169 >> v2:172.21.15.146:3300/0 conn(0x7f4a841586e0 0x7f4a840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840aef80 0x7f4a840cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 -- 172.21.15.72:0/3664076169 >> v2:172.21.15.146:3300/0 conn(0x7f4a841586e0 msgr2=0x7f4a840aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 --2- 172.21.15.72:0/3664076169 >> v2:172.21.15.146:3300/0 conn(0x7f4a841586e0 0x7f4a840aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a82d9d640 1 -- 172.21.15.72:0/3664076169 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8415b540 con 0x7f4a840cffd0 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a79d9b640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840aef80 0x7f4a840cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.808+0000 7f4a8259c640 1 --2- 172.21.15.72:0/3664076169 >> v2:172.21.15.146:3300/0 conn(0x7f4a841586e0 0x7f4a840aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a82d9d640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840cffd0 0x7f4a8406da20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f4a84070d10 tx=0x7f4a7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adebb9bc6631e594 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a81d9b640 1 -- 172.21.15.72:0/3664076169 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c02ca70 con 0x7f4a840cffd0 2024-03-06T07:25:55.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a81d9b640 1 -- 172.21.15.72:0/3664076169 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7c02cc10 con 0x7f4a840cffd0 2024-03-06T07:25:55.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a81d9b640 1 -- 172.21.15.72:0/3664076169 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c0343f0 con 0x7f4a840cffd0 2024-03-06T07:25:55.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 -- 172.21.15.72:0/3664076169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840cffd0 msgr2=0x7f4a8406da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840cffd0 0x7f4a8406da20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f4a84070d10 tx=0x7f4a7c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 -- 172.21.15.72:0/3664076169 shutdown_connections 2024-03-06T07:25:55.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840cffd0 0x7f4a8406da20 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 --2- 172.21.15.72:0/3664076169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840aef80 0x7f4a840cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 --2- 172.21.15.72:0/3664076169 >> v2:172.21.15.146:3300/0 conn(0x7f4a841586e0 0x7f4a840aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 -- 172.21.15.72:0/3664076169 >> 172.21.15.72:0/3664076169 conn(0x7f4a8406b190 msgr2=0x7f4a8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 -- 172.21.15.72:0/3664076169 shutdown_connections 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.809+0000 7f4a8881c640 1 -- 172.21.15.72:0/3664076169 wait complete. 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.811+0000 7f4a8881c640 1 Processor -- start 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.811+0000 7f4a8881c640 1 -- start start 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 0x7f4a8414b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840cffd0 0x7f4a8406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a841586e0 0x7f4a8406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8415bde0 con 0x7f4a840aef80 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8415bb00 con 0x7f4a841586e0 2024-03-06T07:25:55.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a84071350 con 0x7f4a840cffd0 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 0x7f4a8414b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 0x7f4a8414b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42440/0 (socket says 172.21.15.72:42440) 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 -- 172.21.15.72:0/2484905208 learned_addr learned my addr 172.21.15.72:0/2484905208 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a79d9b640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840cffd0 0x7f4a8406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a82d9d640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a841586e0 0x7f4a8406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 -- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840cffd0 msgr2=0x7f4a8406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840cffd0 0x7f4a8406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 -- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a841586e0 msgr2=0x7f4a8406e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a841586e0 0x7f4a8406e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 -- 172.21.15.72:0/2484905208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8414b860 con 0x7f4a840aef80 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8259c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 0x7f4a8414b320 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400c7f0 tx=0x7f4a7400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74015070 con 0x7f4a840aef80 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a74007820 con 0x7f4a840aef80 2024-03-06T07:25:55.817 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74007bd0 con 0x7f4a840aef80 2024-03-06T07:25:55.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8406cab0 con 0x7f4a840aef80 2024-03-06T07:25:55.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.812+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8406d070 con 0x7f4a840aef80 2024-03-06T07:25:55.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.813+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a840732b0 con 0x7f4a840aef80 2024-03-06T07:25:55.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.814+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a74018070 con 0x7f4a840aef80 2024-03-06T07:25:55.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.814+0000 7f4a7b7fe640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:55.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.814+0000 7f4a79d9b640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:55.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.814+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4a7408a220 con 0x7f4a840aef80 2024-03-06T07:25:55.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.815+0000 7f4a79d9b640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a6406b4d0 0x7f4a6406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4a841f6d30 tx=0x7f4a6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:55.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:55.819+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a74058a80 con 0x7f4a840aef80 2024-03-06T07:25:56.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.027+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4a84068ff0 con 0x7f4a840aef80 2024-03-06T07:25:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.693+0000 7f4a7b7fe640 1 -- 172.21.15.72:0/2484905208 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4a7405cb20 con 0x7f4a840aef80 2024-03-06T07:25:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a6406b4d0 msgr2=0x7f4a6406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a6406b4d0 0x7f4a6406d990 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4a841f6d30 tx=0x7f4a6c008040 comp rx=0 tx=0).stop 2024-03-06T07:25:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 msgr2=0x7f4a8414b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 0x7f4a8414b320 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400c7f0 tx=0x7f4a7400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 shutdown_connections 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a6406b4d0 0x7f4a6406d990 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a841586e0 0x7f4a8406e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a840cffd0 0x7f4a8406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 --2- 172.21.15.72:0/2484905208 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a840aef80 0x7f4a8414b320 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 >> 172.21.15.72:0/2484905208 conn(0x7f4a8406b190 msgr2=0x7f4a84156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 shutdown_connections 2024-03-06T07:25:56.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.695+0000 7f4a8881c640 1 -- 172.21.15.72:0/2484905208 wait complete. 2024-03-06T07:25:56.710 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:25:56.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.857+0000 7fb96238c640 1 Processor -- start 2024-03-06T07:25:56.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.857+0000 7fb96238c640 1 -- start start 2024-03-06T07:25:56.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb96238c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb95c15ac00 0x7fb95c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb96238c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c070d10 0x7fb95c15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb96238c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb95c15afe0 con 0x7fb95c0708b0 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb96238c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb95c0cdc10 con 0x7fb95c15ac00 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb96238c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0cd000 con 0x7fb95c070d10 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb95c15ac00 0x7fb95c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb95c0708b0 0x7fb95c15e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36006/0 (socket says 172.21.15.72:36006) 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95b7fe640 1 -- 172.21.15.72:0/375320731 learned_addr learned my addr 172.21.15.72:0/375320731 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95bfff640 1 -- 172.21.15.72:0/375320731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c070d10 msgr2=0x7fb95c15eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb960929640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c070d10 0x7fb95c15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95affd640 1 -- 172.21.15.72:0/375320731 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149186581 0 0) 0x7fb95c15afe0 con 0x7fb95c0708b0 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95bfff640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c070d10 0x7fb95c15eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95bfff640 1 -- 172.21.15.72:0/375320731 >> v1:172.21.15.72:6789/0 conn(0x7fb95c0708b0 legacy=0x7fb95c15e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb95bfff640 1 -- 172.21.15.72:0/375320731 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c0cd2e0 con 0x7fb95c15ac00 2024-03-06T07:25:56.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.858+0000 7fb960929640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c070d10 0x7fb95c15eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.859+0000 7fb95bfff640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c15ac00 0x7fb95c06c910 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0097a0 tx=0x7fb94c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf58840e3b60884 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.859+0000 7fb95affd640 1 -- 172.21.15.72:0/375320731 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c0315a0 con 0x7fb95c15ac00 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.859+0000 7fb95affd640 1 -- 172.21.15.72:0/375320731 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c037070 con 0x7fb95c15ac00 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.859+0000 7fb95affd640 1 -- 172.21.15.72:0/375320731 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c031960 con 0x7fb95c15ac00 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 -- 172.21.15.72:0/375320731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c15ac00 msgr2=0x7fb95c06c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c15ac00 0x7fb95c06c910 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0097a0 tx=0x7fb94c02b840 comp rx=0 tx=0).stop 2024-03-06T07:25:56.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 -- 172.21.15.72:0/375320731 shutdown_connections 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c070d10 0x7fb95c15eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 --2- 172.21.15.72:0/375320731 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c15ac00 0x7fb95c06c910 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 -- 172.21.15.72:0/375320731 >> 172.21.15.72:0/375320731 conn(0x7fb95c0b1650 msgr2=0x7fb95c0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 -- 172.21.15.72:0/375320731 shutdown_connections 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.860+0000 7fb96238c640 1 -- 172.21.15.72:0/375320731 wait complete. 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.861+0000 7fb96238c640 1 Processor -- start 2024-03-06T07:25:56.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.861+0000 7fb96238c640 1 -- start start 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 0x7fb95c15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c070d10 0x7fb95c156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c15ac00 0x7fb95c0a9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c071190 con 0x7fb95c0708b0 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c0cd8a0 con 0x7fb95c070d10 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95c06ce50 con 0x7fb95c15ac00 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 0x7fb95c15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 0x7fb95c15d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42466/0 (socket says 172.21.15.72:42466) 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 -- 172.21.15.72:0/718440040 learned_addr learned my addr 172.21.15.72:0/718440040 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:56.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb960929640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c15ac00 0x7fb95c0a9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95b7fe640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c070d10 0x7fb95c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 -- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c15ac00 msgr2=0x7fb95c0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c15ac00 0x7fb95c0a9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 -- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c070d10 msgr2=0x7fb95c156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c070d10 0x7fb95c156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 -- 172.21.15.72:0/718440040 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c15dfe0 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb95bfff640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 0x7fb95c15d8e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0098c0 tx=0x7fb94c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c032840 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c0329e0 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94c032dd0 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.862+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95c15ba90 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.863+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95c15c050 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.864+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95c0687d0 con 0x7fb95c0708b0 2024-03-06T07:25:56.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.864+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb94c04d3f0 con 0x7fb95c0708b0 2024-03-06T07:25:56.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.864+0000 7fb958ff9640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb94806b4d0 0x7fb94806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:56.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.864+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb94c02f030 con 0x7fb95c0708b0 2024-03-06T07:25:56.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.864+0000 7fb95b7fe640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb94806b4d0 0x7fb94806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:56.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.865+0000 7fb95b7fe640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb94806b4d0 0x7fb94806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb95c06ec60 tx=0x7fb950007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:56.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:56.869+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb94c040070 con 0x7fb95c0708b0 2024-03-06T07:25:57.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.040+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb95c04f330 con 0x7fb95c0708b0 2024-03-06T07:25:57.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.694+0000 7fb958ff9640 1 -- 172.21.15.72:0/718440040 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb94c07ac30 con 0x7fb95c0708b0 2024-03-06T07:25:57.696 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:25:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb94806b4d0 msgr2=0x7fb94806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb94806b4d0 0x7fb94806d990 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb95c06ec60 tx=0x7fb950007b50 comp rx=0 tx=0).stop 2024-03-06T07:25:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 msgr2=0x7fb95c15d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 0x7fb95c15d8e0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb94c0098c0 tx=0x7fb94c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 shutdown_connections 2024-03-06T07:25:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb94806b4d0 0x7fb94806d990 secure :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fb95c06ec60 tx=0x7fb950007b50 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb95c15ac00 0x7fb95c0a9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb95c070d10 0x7fb95c156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 --2- 172.21.15.72:0/718440040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb95c0708b0 0x7fb95c15d8e0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.698+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 >> 172.21.15.72:0/718440040 conn(0x7fb95c0b1650 msgr2=0x7fb95c1e65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.699+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 shutdown_connections 2024-03-06T07:25:57.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.700+0000 7fb96238c640 1 -- 172.21.15.72:0/718440040 wait complete. 2024-03-06T07:25:57.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:25:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.822+0000 7f37c170c640 1 Processor -- start 2024-03-06T07:25:57.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.822+0000 7f37c170c640 1 -- start start 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37c170c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37bc15dff0 0x7f37bc0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37c170c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37bc0cf8b0 con 0x7f37bc0d06b0 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37c170c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37bc0cfb30 con 0x7f37bc0d1250 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37c170c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37bc0b3320 con 0x7f37bc15dff0 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37baffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37bc15dff0 0x7f37bc0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37ba7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f37bc0d06b0 0x7f37bc0d0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36020/0 (socket says 172.21.15.72:36020) 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.823+0000 7f37ba7fc640 1 -- 172.21.15.72:0/2168542435 learned_addr learned my addr 172.21.15.72:0/2168542435 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.824+0000 7f37baffd640 1 -- 172.21.15.72:0/2168542435 >> v1:172.21.15.146:6789/0 conn(0x7f37bc0d1250 legacy=0x7f37bc0b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.824+0000 7f37baffd640 1 -- 172.21.15.72:0/2168542435 >> v1:172.21.15.72:6789/0 conn(0x7f37bc0d06b0 legacy=0x7f37bc0d0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.824+0000 7f37baffd640 1 -- 172.21.15.72:0/2168542435 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc0b3600 con 0x7f37bc15dff0 2024-03-06T07:25:57.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.824+0000 7f37b9ffb640 1 -- 172.21.15.72:0/2168542435 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576239227 0 0) 0x7f37b0001110 con 0x7f37bc0d06b0 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.824+0000 7f37baffd640 1 --2- 172.21.15.72:0/2168542435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0cf370 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009d90 tx=0x7f37ac02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2868f36162e4bda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.824+0000 7f37b9ffb640 1 -- 172.21.15.72:0/2168542435 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac039070 con 0x7f37bc15dff0 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37b9ffb640 1 -- 172.21.15.72:0/2168542435 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac02cd00 con 0x7f37bc15dff0 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37b9ffb640 1 -- 172.21.15.72:0/2168542435 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac034660 con 0x7f37bc15dff0 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 -- 172.21.15.72:0/2168542435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 msgr2=0x7f37bc0cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 --2- 172.21.15.72:0/2168542435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0cf370 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f37ac009d90 tx=0x7f37ac02bf50 comp rx=0 tx=0).stop 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 -- 172.21.15.72:0/2168542435 shutdown_connections 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 --2- 172.21.15.72:0/2168542435 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0cf370 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 -- 172.21.15.72:0/2168542435 >> 172.21.15.72:0/2168542435 conn(0x7f37bc0d3d50 msgr2=0x7f37bc0d4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 -- 172.21.15.72:0/2168542435 shutdown_connections 2024-03-06T07:25:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.825+0000 7f37c170c640 1 -- 172.21.15.72:0/2168542435 wait complete. 2024-03-06T07:25:57.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.826+0000 7f37c170c640 1 Processor -- start 2024-03-06T07:25:57.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.826+0000 7f37c170c640 1 -- start start 2024-03-06T07:25:57.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37c170c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc0d06b0 0x7f37bc14ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37c170c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0d1250 0x7f37bc0b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37c170c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37c170c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc15e550 con 0x7f37bc15dff0 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37c170c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc15e3d0 con 0x7f37bc0d06b0 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37c170c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc0b4510 con 0x7f37bc0d1250 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37bb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42476/0 (socket says 172.21.15.72:42476) 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2822867216 learned_addr learned my addr 172.21.15.72:0/2822867216 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37baffd640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc0d06b0 0x7f37bc14ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37ba7fc640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0d1250 0x7f37bc0b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0d1250 msgr2=0x7f37bc0b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.827+0000 7f37bb7fe640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0d1250 0x7f37bc0b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc0d06b0 msgr2=0x7f37bc14ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37bb7fe640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc0d06b0 0x7f37bc14ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2822867216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc0ca7a0 con 0x7f37bc15dff0 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37bb7fe640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0ca260 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c440 tx=0x7f37b400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4011790 con 0x7f37bc15dff0 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b4017070 con 0x7f37bc15dff0 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc14b220 con 0x7f37bc15dff0 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4011b70 con 0x7f37bc15dff0 2024-03-06T07:25:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.828+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc14b7e0 con 0x7f37bc15dff0 2024-03-06T07:25:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.829+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc04f330 con 0x7f37bc15dff0 2024-03-06T07:25:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.829+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37b4011d10 con 0x7f37bc15dff0 2024-03-06T07:25:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.830+0000 7f37a37fe640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.830+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f37b4089ea0 con 0x7f37bc15dff0 2024-03-06T07:25:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.830+0000 7f37baffd640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a806b4d0 0x7f37a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.830+0000 7f37baffd640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f37ac036000 tx=0x7f37ac030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:57.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:57.834+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37b4058e80 con 0x7f37bc15dff0 2024-03-06T07:25:58.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.005+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f37bc0cefe0 con 0x7f37bc15dff0 2024-03-06T07:25:58.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.699+0000 7f37a37fe640 1 -- 172.21.15.72:0/2822867216 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f37b4058780 con 0x7f37bc15dff0 2024-03-06T07:25:58.701 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:25:58.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.702+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a806b4d0 msgr2=0x7f37a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.702+0000 7f37c170c640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f37ac036000 tx=0x7f37ac030000 comp rx=0 tx=0).stop 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.702+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 msgr2=0x7f37bc0ca260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.702+0000 7f37c170c640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0ca260 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c440 tx=0x7f37b400c900 comp rx=0 tx=0).stop 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 shutdown_connections 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a806b4d0 0x7f37a806d990 secure :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f37ac036000 tx=0x7f37ac030000 comp rx=0 tx=0).stop 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15dff0 0x7f37bc0ca260 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0d1250 0x7f37bc0b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 --2- 172.21.15.72:0/2822867216 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc0d06b0 0x7f37bc14ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 >> 172.21.15.72:0/2822867216 conn(0x7f37bc0d3d50 msgr2=0x7f37bc0b0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 shutdown_connections 2024-03-06T07:25:58.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.703+0000 7f37c170c640 1 -- 172.21.15.72:0/2822867216 wait complete. 2024-03-06T07:25:58.719 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:25:58.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.820+0000 7f48ecd9a640 1 Processor -- start 2024-03-06T07:25:58.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.820+0000 7f48ecd9a640 1 -- start start 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.821+0000 7f48ecd9a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f48e80719e0 0x7f48e80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.821+0000 7f48ecd9a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e80b3320 con 0x7f48e815a220 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.821+0000 7f48ecd9a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48e80728d0 con 0x7f48e806c9f0 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.821+0000 7f48ecd9a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48e80cf610 con 0x7f48e80719e0 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.821+0000 7f48e5d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f48e80719e0 0x7f48e80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.822+0000 7f48e659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f48e815a220 0x7f48e80712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36024/0 (socket says 172.21.15.72:36024) 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.822+0000 7f48e659c640 1 -- 172.21.15.72:0/2771782796 learned_addr learned my addr 172.21.15.72:0/2771782796 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.822+0000 7f48e5d9b640 1 -- 172.21.15.72:0/2771782796 >> v1:172.21.15.146:6789/0 conn(0x7f48e806c9f0 legacy=0x7f48e806ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.822+0000 7f48e5d9b640 1 -- 172.21.15.72:0/2771782796 >> v1:172.21.15.72:6789/0 conn(0x7f48e815a220 legacy=0x7f48e80712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.822+0000 7f48e559a640 1 -- 172.21.15.72:0/2771782796 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97361111 0 0) 0x7f48e80b3320 con 0x7f48e815a220 2024-03-06T07:25:58.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.822+0000 7f48e5d9b640 1 -- 172.21.15.72:0/2771782796 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e80b3590 con 0x7f48e80719e0 2024-03-06T07:25:58.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48e5d9b640 1 --2- 172.21.15.72:0/2771782796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e80719e0 0x7f48e80cf0d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009d90 tx=0x7f48dc02cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e6f3f01799c153 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48e559a640 1 -- 172.21.15.72:0/2771782796 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc03a070 con 0x7f48e80719e0 2024-03-06T07:25:58.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48e559a640 1 -- 172.21.15.72:0/2771782796 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48dc02dce0 con 0x7f48e80719e0 2024-03-06T07:25:58.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48e559a640 1 -- 172.21.15.72:0/2771782796 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc035660 con 0x7f48e80719e0 2024-03-06T07:25:58.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 -- 172.21.15.72:0/2771782796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e80719e0 msgr2=0x7f48e80cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/2771782796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e80719e0 0x7f48e80cf0d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f48dc009d90 tx=0x7f48dc02cf30 comp rx=0 tx=0).stop 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 -- 172.21.15.72:0/2771782796 shutdown_connections 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/2771782796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e80719e0 0x7f48e80cf0d0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 -- 172.21.15.72:0/2771782796 >> 172.21.15.72:0/2771782796 conn(0x7f48e806b190 msgr2=0x7f48e806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 -- 172.21.15.72:0/2771782796 shutdown_connections 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.823+0000 7f48ecd9a640 1 -- 172.21.15.72:0/2771782796 wait complete. 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 Processor -- start 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 -- start start 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 0x7f48e8141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48e80719e0 0x7f48e81430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48e815a220 0x7f48e80ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e8072530 con 0x7f48e806c9f0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e8072030 con 0x7f48e80719e0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48ecd9a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48e80726b0 con 0x7f48e815a220 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48e659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 0x7f48e8141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48e659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 0x7f48e8141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42488/0 (socket says 172.21.15.72:42488) 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.825+0000 7f48e659c640 1 -- 172.21.15.72:0/771835880 learned_addr learned my addr 172.21.15.72:0/771835880 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48e659c640 1 -- 172.21.15.72:0/771835880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48e815a220 msgr2=0x7f48e80ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48e659c640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48e815a220 0x7f48e80ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48e659c640 1 -- 172.21.15.72:0/771835880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48e80719e0 msgr2=0x7f48e81430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48e659c640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48e80719e0 0x7f48e81430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48e659c640 1 -- 172.21.15.72:0/771835880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8142050 con 0x7f48e806c9f0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48e659c640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 0x7f48e8141950 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c460 tx=0x7f48e000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0018070 con 0x7f48e806c9f0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e0007500 con 0x7f48e806c9f0 2024-03-06T07:25:58.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e8142330 con 0x7f48e806c9f0 2024-03-06T07:25:58.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0004a20 con 0x7f48e806c9f0 2024-03-06T07:25:58.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.826+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e81428f0 con 0x7f48e806c9f0 2024-03-06T07:25:58.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.827+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e80732b0 con 0x7f48e806c9f0 2024-03-06T07:25:58.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.828+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f48e001b070 con 0x7f48e806c9f0 2024-03-06T07:25:58.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.828+0000 7f48d6ffd640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f48c806b500 0x7f48c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:58.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.828+0000 7f48e5d9b640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f48c806b500 0x7f48c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:58.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.828+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f48e0089240 con 0x7f48e806c9f0 2024-03-06T07:25:58.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.828+0000 7f48e5d9b640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f48c806b500 0x7f48c806d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f48dc037000 tx=0x7f48dc00d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:58.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:58.832+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f48e0058330 con 0x7f48e806c9f0 2024-03-06T07:25:59.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.003+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f48e8070ba0 con 0x7f48e806c9f0 2024-03-06T07:25:59.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.006+0000 7f48d6ffd640 1 -- 172.21.15.72:0/771835880 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f48e0057c30 con 0x7f48e806c9f0 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f48c806b500 msgr2=0x7f48c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f48c806b500 0x7f48c806d9c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f48dc037000 tx=0x7f48dc00d040 comp rx=0 tx=0).stop 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 msgr2=0x7f48e8141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 0x7f48e8141950 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f48e000c460 tx=0x7f48e000c920 comp rx=0 tx=0).stop 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 shutdown_connections 2024-03-06T07:25:59.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f48c806b500 0x7f48c806d9c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48e815a220 0x7f48e80ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f48e80719e0 0x7f48e81430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 --2- 172.21.15.72:0/771835880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f48e806c9f0 0x7f48e8141950 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 >> 172.21.15.72:0/771835880 conn(0x7f48e806b190 msgr2=0x7f48e80cd0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 shutdown_connections 2024-03-06T07:25:59.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.009+0000 7f48ecd9a640 1 -- 172.21.15.72:0/771835880 wait complete. 2024-03-06T07:25:59.022 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-03-06T07:25:59.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.119+0000 7fdd9c447640 1 Processor -- start 2024-03-06T07:25:59.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.119+0000 7fdd9c447640 1 -- start start 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd9c447640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdd9415dfe0 0x7fdd940cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd9c447640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940b3430 con 0x7fdd940cf830 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd9c447640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940b3710 con 0x7fdd940d29c0 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd9c447640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd940b31e0 con 0x7fdd9415dfe0 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd9a1e3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdd9415dfe0 0x7fdd940cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd999e2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdd940cf830 0x7fdd940d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36040/0 (socket says 172.21.15.72:36040) 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.120+0000 7fdd999e2640 1 -- 172.21.15.72:0/2876562519 learned_addr learned my addr 172.21.15.72:0/2876562519 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.121+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/2876562519 >> v1:172.21.15.146:6789/0 conn(0x7fdd940d29c0 legacy=0x7fdd940b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.121+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/2876562519 >> v1:172.21.15.72:6789/0 conn(0x7fdd940cf830 legacy=0x7fdd940d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.121+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/2876562519 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd940b39f0 con 0x7fdd9415dfe0 2024-03-06T07:25:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.121+0000 7fdd991e1640 1 -- 172.21.15.72:0/2876562519 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495659333 0 0) 0x7fdd940b3430 con 0x7fdd940cf830 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9a1e3640 1 --2- 172.21.15.72:0/2876562519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9415dfe0 0x7fdd940cf2f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fdd940b3360 tx=0x7fdd9002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2cc053c15cfeff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd991e1640 1 -- 172.21.15.72:0/2876562519 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90039070 con 0x7fdd9415dfe0 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd991e1640 1 -- 172.21.15.72:0/2876562519 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9002cce0 con 0x7fdd9415dfe0 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd991e1640 1 -- 172.21.15.72:0/2876562519 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90034660 con 0x7fdd9415dfe0 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 -- 172.21.15.72:0/2876562519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9415dfe0 msgr2=0x7fdd940cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 --2- 172.21.15.72:0/2876562519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9415dfe0 0x7fdd940cf2f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fdd940b3360 tx=0x7fdd9002bf30 comp rx=0 tx=0).stop 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 -- 172.21.15.72:0/2876562519 shutdown_connections 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 --2- 172.21.15.72:0/2876562519 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd9415dfe0 0x7fdd940cf2f0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 -- 172.21.15.72:0/2876562519 >> 172.21.15.72:0/2876562519 conn(0x7fdd940d3cd0 msgr2=0x7fdd940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 -- 172.21.15.72:0/2876562519 shutdown_connections 2024-03-06T07:25:59.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.122+0000 7fdd9c447640 1 -- 172.21.15.72:0/2876562519 wait complete. 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 Processor -- start 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 -- start start 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 0x7fdd940cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd940d29c0 0x7fdd940b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd9415dfe0 0x7fdd941f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9415e3c0 con 0x7fdd940cf830 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940b3d60 con 0x7fdd940d29c0 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9c447640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9415e540 con 0x7fdd9415dfe0 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9a1e3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 0x7fdd940cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9a1e3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 0x7fdd940cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42496/0 (socket says 172.21.15.72:42496) 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.124+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/4064582722 learned_addr learned my addr 172.21.15.72:0/4064582722 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/4064582722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd9415dfe0 msgr2=0x7fdd941f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9a1e3640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd9415dfe0 0x7fdd941f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/4064582722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd940d29c0 msgr2=0x7fdd940b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9a1e3640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd940d29c0 0x7fdd940b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9a1e3640 1 -- 172.21.15.72:0/4064582722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd940b81d0 con 0x7fdd940cf830 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9a1e3640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 0x7fdd940cada0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdd90037040 tx=0x7fdd90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90044020 con 0x7fdd940cf830 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9003f670 con 0x7fdd940cf830 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90039040 con 0x7fdd940cf830 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd940b84b0 con 0x7fdd940cf830 2024-03-06T07:25:59.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.125+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd940b8a70 con 0x7fdd940cf830 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.126+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9404f330 con 0x7fdd940cf830 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.126+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdd90047020 con 0x7fdd940cf830 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.127+0000 7fdd8affd640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd7806b4d0 0x7fdd7806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.127+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdd900acb20 con 0x7fdd940cf830 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.127+0000 7fdd999e2640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd7806b4d0 0x7fdd7806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.127+0000 7fdd999e2640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd7806b4d0 0x7fdd7806d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdd84007fd0 tx=0x7fdd84007900 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.131+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdd9004e030 con 0x7fdd940cf830 2024-03-06T07:25:59.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.367+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) v1 -- 0x7fdd9415d700 con 0x7fdd940cf830 2024-03-06T07:25:59.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.701+0000 7fdd8affd640 1 -- 172.21.15.72:0/4064582722 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v235) v1 ==== 132+0+0 (secure 0 0 0) 0x7fdd9007bd30 con 0x7fdd940cf830 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd7806b4d0 msgr2=0x7fdd7806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd7806b4d0 0x7fdd7806d990 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdd84007fd0 tx=0x7fdd84007900 comp rx=0 tx=0).stop 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 msgr2=0x7fdd940cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 0x7fdd940cada0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fdd90037040 tx=0x7fdd90004a40 comp rx=0 tx=0).stop 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 shutdown_connections 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdd7806b4d0 0x7fdd7806d990 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd9415dfe0 0x7fdd941f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd940d29c0 0x7fdd940b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.703+0000 7fdd9c447640 1 --2- 172.21.15.72:0/4064582722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd940cf830 0x7fdd940cada0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.704+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 >> 172.21.15.72:0/4064582722 conn(0x7fdd940d3cd0 msgr2=0x7fdd940d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.704+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 shutdown_connections 2024-03-06T07:25:59.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.704+0000 7fdd9c447640 1 -- 172.21.15.72:0/4064582722 wait complete. 2024-03-06T07:25:59.717 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-03-06T07:25:59.717 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:25:59.718 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:25:59.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.816+0000 7f857ce39640 1 Processor -- start 2024-03-06T07:25:59.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.816+0000 7f857ce39640 1 -- start start 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857ce39640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f857813dc10 0x7f85780b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857ce39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85780b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857ce39640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857815f580 con 0x7f857815bf30 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857ce39640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85780cf3f0 con 0x7f857813dc10 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857ce39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857815b210 con 0x7f857815ef60 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f8575d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f857815bf30 0x7f857815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36042/0 (socket says 172.21.15.72:36042) 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f8575d9b640 1 -- 172.21.15.72:0/259766417 learned_addr learned my addr 172.21.15.72:0/259766417 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f8576d9d640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85780b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857659c640 1 --2- 172.21.15.72:0/259766417 >> v2:172.21.15.146:3300/0 conn(0x7f857813dc10 0x7f85780b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600849763 0 0) 0x7f857815f580 con 0x7f857815bf30 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.817+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8560003520 con 0x7f857815bf30 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116651939 0 0) 0x7f8560003520 con 0x7f857815bf30 2024-03-06T07:25:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 msgr2=0x7f85780b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85780b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 >> v2:172.21.15.146:3300/0 conn(0x7f857813dc10 msgr2=0x7f85780b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 --2- 172.21.15.72:0/259766417 >> v2:172.21.15.146:3300/0 conn(0x7f857813dc10 0x7f85780b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857815b450 con 0x7f857815bf30 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f856c001670 con 0x7f857815bf30 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 >> v1:172.21.15.72:6789/0 conn(0x7f857815bf30 legacy=0x7f857815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8560003f30 0x7f8560004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f85600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780cf3f0 con 0x7f8560004d40 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857815b210 con 0x7f8560003f30 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856000ba10 con 0x7f857815ef60 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8576d9d640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f85600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857659c640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8575d9b640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8560003f30 0x7f8560004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8576d9d640 1 -- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 msgr2=0x7f85600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8576d9d640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8576d9d640 1 -- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8560003f30 msgr2=0x7f8560004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8576d9d640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8560003f30 0x7f8560004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8576d9d640 1 -- 172.21.15.72:0/259766417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560003520 con 0x7f8560004d40 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f857659c640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857815ef60 0x7f85600039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.818+0000 7f8575d9b640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8560003f30 0x7f8560004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f8576d9d640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f85600051c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8570009af0 tx=0x7f8560005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4d51177eda3f0a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570039070 con 0x7f8560004d40 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857002c6b0 con 0x7f8560004d40 2024-03-06T07:25:59.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f857559a640 1 -- 172.21.15.72:0/259766417 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570009690 con 0x7f8560004d40 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f857ce39640 1 -- 172.21.15.72:0/259766417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 msgr2=0x7f85600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f857ce39640 1 --2- 172.21.15.72:0/259766417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f85600051c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f8570009af0 tx=0x7f8560005830 comp rx=0 tx=0).stop 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.819+0000 7f8576d9d640 1 -- 172.21.15.72:0/259766417 reap_dead start 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.820+0000 7f857ce39640 1 -- 172.21.15.72:0/259766417 shutdown_connections 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.820+0000 7f857ce39640 1 -- 172.21.15.72:0/259766417 >> 172.21.15.72:0/259766417 conn(0x7f85780cf8a0 msgr2=0x7f85780cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.820+0000 7f857ce39640 1 -- 172.21.15.72:0/259766417 shutdown_connections 2024-03-06T07:25:59.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.820+0000 7f857ce39640 1 -- 172.21.15.72:0/259766417 wait complete. 2024-03-06T07:25:59.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.821+0000 7f857ce39640 1 Processor -- start 2024-03-06T07:25:59.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.821+0000 7f857ce39640 1 -- start start 2024-03-06T07:25:59.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f8578157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8578159060 0x7f85781545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857813fc10 0x7f8578140090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780b2e90 con 0x7f8560004d40 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857813dff0 con 0x7f8578159060 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85780b3010 con 0x7f857813fc10 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f8578157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f8578157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42524/0 (socket says 172.21.15.72:42524) 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 -- 172.21.15.72:0/2493108540 learned_addr learned my addr 172.21.15.72:0/2493108540 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 -- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857813fc10 msgr2=0x7f8578140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857813fc10 0x7f8578140090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f8575d9b640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8578159060 0x7f85781545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 -- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8578159060 msgr2=0x7f85781545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8578159060 0x7f85781545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:25:59.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 -- 172.21.15.72:0/2493108540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85781594c0 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857659c640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f8578157930 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8568007fb0 tx=0x7f8568010650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856801a030 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8568010e80 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8578157fc0 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8568003b90 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.822+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8578158580 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.823+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857806dcf0 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.824+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8568005d10 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.824+0000 7f8566ffd640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f855806b4d0 0x7f855806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.824+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f856808aea0 con 0x7f8560004d40 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.824+0000 7f8575d9b640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f855806b4d0 0x7f855806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:25:59.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.824+0000 7f8575d9b640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f855806b4d0 0x7f855806d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f856c000f60 tx=0x7f856c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:25:59.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:25:59.828+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f857806dcf0 con 0x7f8560004d40 2024-03-06T07:26:00.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.003+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f857804f330 con 0x7f8560004d40 2024-03-06T07:26:00.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.005+0000 7f8566ffd640 1 -- 172.21.15.72:0/2493108540 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13033 (secure 0 0 0) 0x7f8568024070 con 0x7f8560004d40 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:00.008 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":86,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:25:58.695954+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/2658331491":"2024-03-07T07:25:55.687878+0000","172.21.15.72:6834/2658331491":"2024-03-07T07:25:55.687878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f855806b4d0 msgr2=0x7f855806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f855806b4d0 0x7f855806d990 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f856c000f60 tx=0x7f856c000ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 msgr2=0x7f8578157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f8578157930 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f8568007fb0 tx=0x7f8568010650 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 shutdown_connections 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f855806b4d0 0x7f855806d990 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f857813fc10 0x7f8578140090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8578159060 0x7f85781545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 --2- 172.21.15.72:0/2493108540 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8560004d40 0x7f8578157930 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 >> 172.21.15.72:0/2493108540 conn(0x7f85780cf8a0 msgr2=0x7f857806c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 shutdown_connections 2024-03-06T07:26:00.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.009+0000 7f857ce39640 1 -- 172.21.15.72:0/2493108540 wait complete. 2024-03-06T07:26:00.023 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:00.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.125+0000 7ffb26b2b640 1 Processor -- start 2024-03-06T07:26:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.125+0000 7ffb26b2b640 1 -- start start 2024-03-06T07:26:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb26b2b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb200cff40 0x7ffb200b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb26b2b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb20158ac0 con 0x7ffb201586e0 2024-03-06T07:26:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb26b2b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2015b2a0 con 0x7ffb200cfa90 2024-03-06T07:26:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb26b2b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb200d0390 con 0x7ffb200cff40 2024-03-06T07:26:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb248c7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ffb201586e0 0x7ffb200aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36046/0 (socket says 172.21.15.72:36046) 2024-03-06T07:26:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb248c7640 1 -- 172.21.15.72:0/1061690913 learned_addr learned my addr 172.21.15.72:0/1061690913 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322961508 0 0) 0x7ffb20158ac0 con 0x7ffb201586e0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb08003520 con 0x7ffb201586e0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.126+0000 7ffb250c8640 1 --2- 172.21.15.72:0/1061690913 >> v2:172.21.15.146:3300/0 conn(0x7ffb200cff40 0x7ffb200b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2954575893 0 0) 0x7ffb08003520 con 0x7ffb201586e0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 >> v2:172.21.15.146:3300/0 conn(0x7ffb200cff40 msgr2=0x7ffb200b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 --2- 172.21.15.72:0/1061690913 >> v2:172.21.15.146:3300/0 conn(0x7ffb200cff40 0x7ffb200b3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 >> v1:172.21.15.146:6789/0 conn(0x7ffb200cfa90 legacy=0x7ffb200b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2015b540 con 0x7ffb201586e0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ffb10001670 con 0x7ffb201586e0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 >> v1:172.21.15.72:6789/0 conn(0x7ffb201586e0 legacy=0x7ffb200aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080039b0 0x7ffb08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb08004800 0x7ffb08004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb080051c0 0x7ffb0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200d0390 con 0x7ffb080051c0 2024-03-06T07:26:00.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0800c110 con 0x7ffb080039b0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0800c310 con 0x7ffb08004800 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb250c8640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb080051c0 0x7ffb0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb248c7640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080039b0 0x7ffb08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb1ffff640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb08004800 0x7ffb08004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb250c8640 1 -- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb08004800 msgr2=0x7ffb08004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb250c8640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb08004800 0x7ffb08004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb250c8640 1 -- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080039b0 msgr2=0x7ffb08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb250c8640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080039b0 0x7ffb08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.127+0000 7ffb250c8640 1 -- 172.21.15.72:0/1061690913 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb08003520 con 0x7ffb080051c0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb250c8640 1 -- 172.21.15.72:0/1061690913 reap_dead start 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb250c8640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb080051c0 0x7ffb0800b9c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ffb18007360 tx=0x7ffb08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b239e3ed247c1c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18009be0 con 0x7ffb080051c0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1802cd30 con 0x7ffb080051c0 2024-03-06T07:26:00.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb1f7fe640 1 -- 172.21.15.72:0/1061690913 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb18003bd0 con 0x7ffb080051c0 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb26b2b640 1 -- 172.21.15.72:0/1061690913 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb080051c0 msgr2=0x7ffb0800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb080051c0 0x7ffb0800b9c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7ffb18007360 tx=0x7ffb08005830 comp rx=0 tx=0).stop 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb26b2b640 1 -- 172.21.15.72:0/1061690913 shutdown_connections 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/1061690913 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb080051c0 0x7ffb0800b9c0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.128+0000 7ffb26b2b640 1 -- 172.21.15.72:0/1061690913 >> 172.21.15.72:0/1061690913 conn(0x7ffb2006b190 msgr2=0x7ffb2006b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.129+0000 7ffb26b2b640 1 -- 172.21.15.72:0/1061690913 shutdown_connections 2024-03-06T07:26:00.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.129+0000 7ffb26b2b640 1 -- 172.21.15.72:0/1061690913 wait complete. 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 Processor -- start 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 -- start start 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080051c0 0x7ffb2014ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 0x7ffb20152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2013fea0 0x7ffb20140320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015bde0 con 0x7ffb201437a0 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015bb00 con 0x7ffb080051c0 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.130+0000 7ffb26b2b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb20071350 con 0x7ffb2013fea0 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 0x7ffb20152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 0x7ffb20152b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42532/0 (socket says 172.21.15.72:42532) 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 -- 172.21.15.72:0/2313366995 learned_addr learned my addr 172.21.15.72:0/2313366995 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:00.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb248c7640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080051c0 0x7ffb2014ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb250c8640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2013fea0 0x7ffb20140320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 -- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2013fea0 msgr2=0x7ffb20140320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2013fea0 0x7ffb20140320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 -- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080051c0 msgr2=0x7ffb2014ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080051c0 0x7ffb2014ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 -- 172.21.15.72:0/2313366995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb20151350 con 0x7ffb201437a0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb248c7640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080051c0 0x7ffb2014ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb250c8640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2013fea0 0x7ffb20140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1ffff640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 0x7ffb20152b60 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400e4e0 tx=0x7ffb1400e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb140052e0 con 0x7ffb201437a0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14018070 con 0x7ffb201437a0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb20150740 con 0x7ffb201437a0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb14003830 con 0x7ffb201437a0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.131+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb20150d00 con 0x7ffb201437a0 2024-03-06T07:26:00.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.132+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb200732b0 con 0x7ffb201437a0 2024-03-06T07:26:00.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.133+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffb140039d0 con 0x7ffb201437a0 2024-03-06T07:26:00.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.133+0000 7ffb1d7fa640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0c06b500 0x7ffb0c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.133+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffb14089a40 con 0x7ffb201437a0 2024-03-06T07:26:00.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.133+0000 7ffb248c7640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0c06b500 0x7ffb0c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.134+0000 7ffb248c7640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0c06b500 0x7ffb0c06d9c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffb10001cc0 tx=0x7ffb10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.137+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffb14058c50 con 0x7ffb201437a0 2024-03-06T07:26:00.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.308+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb20068ff0 con 0x7ffb201437a0 2024-03-06T07:26:00.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.308+0000 7ffb1d7fa640 1 -- 172.21.15.72:0/2313366995 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb14058550 con 0x7ffb201437a0 2024-03-06T07:26:00.310 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:26:00.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0c06b500 msgr2=0x7ffb0c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0c06b500 0x7ffb0c06d9c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ffb10001cc0 tx=0x7ffb10009040 comp rx=0 tx=0).stop 2024-03-06T07:26:00.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 msgr2=0x7ffb20152b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 0x7ffb20152b60 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400e4e0 tx=0x7ffb1400e9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 shutdown_connections 2024-03-06T07:26:00.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffb0c06b500 0x7ffb0c06d9c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb2013fea0 0x7ffb20140320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb201437a0 0x7ffb20152b60 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 --2- 172.21.15.72:0/2313366995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb080051c0 0x7ffb2014ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 >> 172.21.15.72:0/2313366995 conn(0x7ffb2006b190 msgr2=0x7ffb201548e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 shutdown_connections 2024-03-06T07:26:00.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.310+0000 7ffb26b2b640 1 -- 172.21.15.72:0/2313366995 wait complete. 2024-03-06T07:26:00.323 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:00.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.421+0000 7fd821593640 1 Processor -- start 2024-03-06T07:26:00.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.421+0000 7fd821593640 1 -- start start 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd821593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 0x7fd81c0d47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd821593640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd81c0b2b10 0x7fd81c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd821593640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd81c0d00e0 con 0x7fd81c15e120 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd821593640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd81c0cfe00 con 0x7fd81c0b2b10 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd821593640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0d2330 con 0x7fd81c0d0dd0 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd81c15e120 0x7fd81c0d06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36054/0 (socket says 172.21.15.72:36054) 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81affd640 1 -- 172.21.15.72:0/419071303 learned_addr learned my addr 172.21.15.72:0/419071303 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81b7fe640 1 --2- 172.21.15.72:0/419071303 >> v2:172.21.15.146:3300/0 conn(0x7fd81c0b2b10 0x7fd81c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd813fff640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 0x7fd81c0d47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440852628 0 0) 0x7fd81c0d00e0 con 0x7fd81c15e120 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd808003520 con 0x7fd81c15e120 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193120472 0 0) 0x7fd808003520 con 0x7fd81c15e120 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 msgr2=0x7fd81c0d47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 0x7fd81c0d47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 >> v2:172.21.15.146:3300/0 conn(0x7fd81c0b2b10 msgr2=0x7fd81c0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 --2- 172.21.15.72:0/419071303 >> v2:172.21.15.146:3300/0 conn(0x7fd81c0b2b10 0x7fd81c0ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c0d2610 con 0x7fd81c15e120 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd80c001670 con 0x7fd81c15e120 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81b7fe640 1 --2- 172.21.15.72:0/419071303 >> v2:172.21.15.146:3300/0 conn(0x7fd81c0b2b10 0x7fd81c0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 >> v1:172.21.15.72:6789/0 conn(0x7fd81c15e120 legacy=0x7fd81c0d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 0x7fd8080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.422+0000 7fd81a7fc640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd808003f30 0x7fd808004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81a7fc640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd808004d40 0x7fd8080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0cfe00 con 0x7fd808003f30 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0d2330 con 0x7fd808004d40 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80800ba10 con 0x7fd81c0d0dd0 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd813fff640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd808003f30 0x7fd808004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81affd640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 0x7fd8080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81b7fe640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd808004d40 0x7fd8080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd813fff640 1 -- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 msgr2=0x7fd8080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd813fff640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd81c0d0dd0 0x7fd8080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd813fff640 1 -- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd808004d40 msgr2=0x7fd8080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd813fff640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd808004d40 0x7fd8080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd813fff640 1 -- 172.21.15.72:0/419071303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808003520 con 0x7fd808003f30 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.423+0000 7fd81b7fe640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd808004d40 0x7fd8080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd813fff640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd808003f30 0x7fd808004800 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fd804009af0 tx=0x7fd808005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b711a9a0f6966f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804039070 con 0x7fd808003f30 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80402c6b0 con 0x7fd808003f30 2024-03-06T07:26:00.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd81a7fc640 1 -- 172.21.15.72:0/419071303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804009690 con 0x7fd808003f30 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd821593640 1 -- 172.21.15.72:0/419071303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd808003f30 msgr2=0x7fd808004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd821593640 1 --2- 172.21.15.72:0/419071303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd808003f30 0x7fd808004800 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fd804009af0 tx=0x7fd808005830 comp rx=0 tx=0).stop 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd81b7fe640 1 -- 172.21.15.72:0/419071303 reap_dead start 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd821593640 1 -- 172.21.15.72:0/419071303 shutdown_connections 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd821593640 1 -- 172.21.15.72:0/419071303 >> 172.21.15.72:0/419071303 conn(0x7fd81c0d3cb0 msgr2=0x7fd81c0d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd821593640 1 -- 172.21.15.72:0/419071303 shutdown_connections 2024-03-06T07:26:00.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.424+0000 7fd821593640 1 -- 172.21.15.72:0/419071303 wait complete. 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 Processor -- start 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 -- start start 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd808003f30 0x7fd81c1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 0x7fd81c0cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd81c0cc460 0x7fd81c15c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0d2e10 con 0x7fd81c0c82f0 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0b1890 con 0x7fd81c0cc460 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd821593640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c0d47d0 con 0x7fd808003f30 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd813fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 0x7fd81c0cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd813fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 0x7fd81c0cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42560/0 (socket says 172.21.15.72:42560) 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd813fff640 1 -- 172.21.15.72:0/317786327 learned_addr learned my addr 172.21.15.72:0/317786327 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd81affd640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd808003f30 0x7fd81c1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.426+0000 7fd81b7fe640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd81c0cc460 0x7fd81c15c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd813fff640 1 -- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd808003f30 msgr2=0x7fd81c1592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd813fff640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd808003f30 0x7fd81c1592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd813fff640 1 -- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd81c0cc460 msgr2=0x7fd81c15c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd813fff640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd81c0cc460 0x7fd81c15c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd813fff640 1 -- 172.21.15.72:0/317786327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c1597e0 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd81b7fe640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd81c0cc460 0x7fd81c15c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd813fff640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 0x7fd81c0cad80 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd804037040 tx=0x7fd804004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804042020 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80403a040 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c0cb5d0 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804039070 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.427+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c0cbb90 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.428+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c04f330 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.428+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd804045020 con 0x7fd81c0c82f0 2024-03-06T07:26:00.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.429+0000 7fd8137fe640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.429+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd8040ab170 con 0x7fd81c0c82f0 2024-03-06T07:26:00.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.429+0000 7fd81affd640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.429+0000 7fd81affd640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd80c007d30 tx=0x7fd80c0078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.433+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd80404c070 con 0x7fd81c0c82f0 2024-03-06T07:26:00.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.672+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fd81c0d1900 con 0x7fd81c0c82f0 2024-03-06T07:26:00.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.709+0000 7fd8137fe640 1 -- 172.21.15.72:0/317786327 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v236) v1 ==== 137+0+0 (secure 0 0 0) 0x7fd81c1597e0 con 0x7fd81c0c82f0 2024-03-06T07:26:00.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 msgr2=0x7fd7fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd80c007d30 tx=0x7fd80c0078f0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 msgr2=0x7fd81c0cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 0x7fd81c0cad80 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd804037040 tx=0x7fd804004730 comp rx=0 tx=0).stop 2024-03-06T07:26:00.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 shutdown_connections 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd81c0cc460 0x7fd81c15c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81c0c82f0 0x7fd81c0cad80 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 --2- 172.21.15.72:0/317786327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd808003f30 0x7fd81c1592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 >> 172.21.15.72:0/317786327 conn(0x7fd81c0d3cb0 msgr2=0x7fd81c0d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 shutdown_connections 2024-03-06T07:26:00.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.711+0000 7fd821593640 1 -- 172.21.15.72:0/317786327 wait complete. 2024-03-06T07:26:00.724 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:00.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.824+0000 7fc3eb0c9640 1 Processor -- start 2024-03-06T07:26:00.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.824+0000 7fc3eb0c9640 1 -- start start 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3eb0c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3e4145590 0x7fc3e406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3eb0c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc3e406f6e0 0x7fc3e40cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3eb0c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e406b970 con 0x7fc3e406b4f0 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3eb0c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e4145970 con 0x7fc3e4145590 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3eb0c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e4160110 con 0x7fc3e406f6e0 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3e4145590 0x7fc3e406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3e4145590 0x7fc3e406afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42570/0 (socket says 172.21.15.72:42570) 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 -- 172.21.15.72:0/2700498444 learned_addr learned my addr 172.21.15.72:0/2700498444 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:00.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e9666640 1 --2- 172.21.15.72:0/2700498444 >> v2:172.21.15.146:3300/0 conn(0x7fc3e406f6e0 0x7fc3e40cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 -- 172.21.15.72:0/2700498444 >> v2:172.21.15.146:3300/0 conn(0x7fc3e406f6e0 msgr2=0x7fc3e40cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 --2- 172.21.15.72:0/2700498444 >> v2:172.21.15.146:3300/0 conn(0x7fc3e406f6e0 0x7fc3e40cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 -- 172.21.15.72:0/2700498444 >> v1:172.21.15.146:6789/0 conn(0x7fc3e406b4f0 legacy=0x7fc3e406efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.825+0000 7fc3e8e65640 1 -- 172.21.15.72:0/2700498444 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e415f7e0 con 0x7fc3e4145590 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.826+0000 7fc3e8e65640 1 --2- 172.21.15.72:0/2700498444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e4145590 0x7fc3e406afb0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4073170 tx=0x7fc3d802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9bf9c8abea2834 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.826+0000 7fc3e37fe640 1 -- 172.21.15.72:0/2700498444 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8035020 con 0x7fc3e4145590 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.826+0000 7fc3e37fe640 1 -- 172.21.15.72:0/2700498444 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d8031870 con 0x7fc3e4145590 2024-03-06T07:26:00.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.826+0000 7fc3e37fe640 1 -- 172.21.15.72:0/2700498444 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8031c40 con 0x7fc3e4145590 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.826+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/2700498444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e4145590 msgr2=0x7fc3e406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.826+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/2700498444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e4145590 0x7fc3e406afb0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4073170 tx=0x7fc3d802b910 comp rx=0 tx=0).stop 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.827+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/2700498444 shutdown_connections 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.827+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/2700498444 >> v2:172.21.15.146:3300/0 conn(0x7fc3e406f6e0 0x7fc3e40cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.827+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/2700498444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e4145590 0x7fc3e406afb0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.827+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/2700498444 >> 172.21.15.72:0/2700498444 conn(0x7fc3e40b2660 msgr2=0x7fc3e40b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.827+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/2700498444 shutdown_connections 2024-03-06T07:26:00.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.827+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/2700498444 wait complete. 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.828+0000 7fc3eb0c9640 1 Processor -- start 2024-03-06T07:26:00.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.828+0000 7fc3eb0c9640 1 -- start start 2024-03-06T07:26:00.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e406b4f0 0x7fc3e413d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 0x7fc3e40b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e4145590 0x7fc3e40b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e41605a0 con 0x7fc3e406f6e0 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e415fda0 con 0x7fc3e4145590 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40b4220 con 0x7fc3e406b4f0 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 0x7fc3e40b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 0x7fc3e40b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42586/0 (socket says 172.21.15.72:42586) 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 -- 172.21.15.72:0/1141873036 learned_addr learned my addr 172.21.15.72:0/1141873036 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e8e65640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e406b4f0 0x7fc3e413d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e9666640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e4145590 0x7fc3e40b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 -- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e406b4f0 msgr2=0x7fc3e413d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e406b4f0 0x7fc3e413d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 -- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e4145590 msgr2=0x7fc3e40b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e4145590 0x7fc3e40b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 -- 172.21.15.72:0/1141873036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40b4500 con 0x7fc3e406f6e0 2024-03-06T07:26:00.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e3fff640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 0x7fc3e40b50e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000c430 tx=0x7fc3d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0011790 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0017070 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0011bb0 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e40b47e0 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.829+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e40b34f0 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.830+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e40733b0 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.831+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc3d001c460 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.831+0000 7fc3e17fa640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3c806b4d0 0x7fc3c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.831+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc3d0089d30 con 0x7fc3e406f6e0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.831+0000 7fc3e8e65640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3c806b4d0 0x7fc3c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:00.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.832+0000 7fc3e8e65640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3c806b4d0 0x7fc3c806d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8036000 tx=0x7fc3d803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:00.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:00.836+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3d0058f40 con 0x7fc3e406f6e0 2024-03-06T07:26:01.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.009+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc3e4068ff0 con 0x7fc3e406f6e0 2024-03-06T07:26:01.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.014+0000 7fc3e17fa640 1 -- 172.21.15.72:0/1141873036 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc3d0058840 con 0x7fc3e406f6e0 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3c806b4d0 msgr2=0x7fc3c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3c806b4d0 0x7fc3c806d990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8036000 tx=0x7fc3d803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 msgr2=0x7fc3e40b50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 0x7fc3e40b50e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000c430 tx=0x7fc3d000c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 shutdown_connections 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc3c806b4d0 0x7fc3c806d990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3e4145590 0x7fc3e40b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3e406f6e0 0x7fc3e40b50e0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 --2- 172.21.15.72:0/1141873036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3e406b4f0 0x7fc3e413d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 >> 172.21.15.72:0/1141873036 conn(0x7fc3e40b2660 msgr2=0x7fc3e40b06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 shutdown_connections 2024-03-06T07:26:01.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.016+0000 7fc3eb0c9640 1 -- 172.21.15.72:0/1141873036 wait complete. 2024-03-06T07:26:01.029 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:01.029 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:01.082 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:01.140 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:01.140 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:26:01.140 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.155 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:01.156 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:26:01.156 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.172 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:01.172 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:26:01.172 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:01.210 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:26:01.210 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:01.229 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:01.229 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:01.229 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:01.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.348+0000 7ff49393e640 1 Processor -- start 2024-03-06T07:26:01.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.348+0000 7ff49393e640 1 -- start start 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff49393e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c06c6d0 0x7ff48c15ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff49393e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c0cd450 con 0x7ff48c0b4f70 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff49393e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c0cd730 con 0x7ff48c066d90 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff49393e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15e9f0 con 0x7ff48c06c6d0 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff4916da640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff48c0b4f70 0x7ff48c0b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36056/0 (socket says 172.21.15.72:36056) 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff4916da640 1 -- 172.21.15.72:0/4177034177 learned_addr learned my addr 172.21.15.72:0/4177034177 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545113713 0 0) 0x7ff48c0cd450 con 0x7ff48c0b4f70 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff478003520 con 0x7ff48c0b4f70 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff490ed9640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c06c6d0 0x7ff48c15ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845876269 0 0) 0x7ff478003520 con 0x7ff48c0b4f70 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c06c6d0 msgr2=0x7ff48c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.349+0000 7ff483fff640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c06c6d0 0x7ff48c15ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 >> v1:172.21.15.146:6789/0 conn(0x7ff48c066d90 legacy=0x7ff48c0ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c15e680 con 0x7ff48c0b4f70 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff488001670 con 0x7ff48c0b4f70 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 >> v1:172.21.15.72:6789/0 conn(0x7ff48c0b4f70 legacy=0x7ff48c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff478003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff478004800 0x7ff478004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff478005180 0x7ff47800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15e9f0 con 0x7ff4780039e0 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47800bf00 con 0x7ff478005180 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47800c0d0 con 0x7ff478004800 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff4916da640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff478003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff491edb640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff478005180 0x7ff47800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff490ed9640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff478004800 0x7ff478004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff4916da640 1 -- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff478004800 msgr2=0x7ff478004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff4916da640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff478004800 0x7ff478004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff4916da640 1 -- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff478005180 msgr2=0x7ff47800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff4916da640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff478005180 0x7ff47800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff4916da640 1 -- 172.21.15.72:0/4177034177 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478003520 con 0x7ff4780039e0 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff491edb640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff478005180 0x7ff47800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.350+0000 7ff490ed9640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff478004800 0x7ff478004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff4916da640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff478003e00 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c210 tx=0x7ff478005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fbc5ac58f1d973 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488007d30 con 0x7ff4780039e0 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488030070 con 0x7ff4780039e0 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff483fff640 1 -- 172.21.15.72:0/4177034177 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48802cd60 con 0x7ff4780039e0 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff49393e640 1 -- 172.21.15.72:0/4177034177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 msgr2=0x7ff478003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff49393e640 1 --2- 172.21.15.72:0/4177034177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff478003e00 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff48800c210 tx=0x7ff478005830 comp rx=0 tx=0).stop 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff491edb640 1 -- 172.21.15.72:0/4177034177 reap_dead start 2024-03-06T07:26:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff49393e640 1 -- 172.21.15.72:0/4177034177 shutdown_connections 2024-03-06T07:26:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff49393e640 1 -- 172.21.15.72:0/4177034177 >> 172.21.15.72:0/4177034177 conn(0x7ff48c0724d0 msgr2=0x7ff48c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff49393e640 1 -- 172.21.15.72:0/4177034177 shutdown_connections 2024-03-06T07:26:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.351+0000 7ff49393e640 1 -- 172.21.15.72:0/4177034177 wait complete. 2024-03-06T07:26:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 Processor -- start 2024-03-06T07:26:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 -- start start 2024-03-06T07:26:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff48c09a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48c0cddd0 0x7ff48c161cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c09da20 0x7ff48c156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15f290 con 0x7ff4780039e0 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15f110 con 0x7ff48c0cddd0 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff49393e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15f410 con 0x7ff48c09da20 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff4916da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff48c09a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff4916da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff48c09a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42600/0 (socket says 172.21.15.72:42600) 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.353+0000 7ff4916da640 1 -- 172.21.15.72:0/2382266239 learned_addr learned my addr 172.21.15.72:0/2382266239 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff491edb640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c09da20 0x7ff48c156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff490ed9640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48c0cddd0 0x7ff48c161cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff4916da640 1 -- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c09da20 msgr2=0x7ff48c156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff4916da640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c09da20 0x7ff48c156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff4916da640 1 -- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48c0cddd0 msgr2=0x7ff48c161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff4916da640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48c0cddd0 0x7ff48c161cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff4916da640 1 -- 172.21.15.72:0/2382266239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c157820 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff4916da640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff48c09a270 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff488001970 tx=0x7ff488001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488046070 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488000ed0 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488004e30 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c157a90 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.354+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c158330 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.355+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c04f330 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.356+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff488007070 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.356+0000 7ff481ffb640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff47006b4d0 0x7ff47006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.356+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff4880ab1c0 con 0x7ff4780039e0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.356+0000 7ff490ed9640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff47006b4d0 0x7ff47006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.356+0000 7ff490ed9640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff47006b4d0 0x7ff47006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0caca0 tx=0x7ff47c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.360+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff48807a420 con 0x7ff4780039e0 2024-03-06T07:26:01.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.568+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff48c157e20 con 0x7ff4780039e0 2024-03-06T07:26:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.569+0000 7ff481ffb640 1 -- 172.21.15.72:0/2382266239 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+3699 (secure 0 0 0) 0x7ff488079d20 con 0x7ff4780039e0 2024-03-06T07:26:01.571 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:01.571 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5648,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1032898007","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1032898007},{"type":"v1","addr":"172.21.15.146:6834","nonce":1032898007}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":5651,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/755205809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":755205809},{"type":"v1","addr":"172.21.15.146:6835","nonce":755205809}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/993905315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":993905315},{"type":"v1","addr":"172.21.15.72:6837","nonce":993905315}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":36},{"gid":7305,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3495915524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3495915524},{"type":"v1","addr":"172.21.15.72:6836","nonce":3495915524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41}],"filesystems":[]} 2024-03-06T07:26:01.571 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 41 2024-03-06T07:26:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff47006b4d0 msgr2=0x7ff47006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff47006b4d0 0x7ff47006d990 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0caca0 tx=0x7ff47c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 msgr2=0x7ff48c09a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff48c09a270 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff488001970 tx=0x7ff488001a40 comp rx=0 tx=0).stop 2024-03-06T07:26:01.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 shutdown_connections 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff47006b4d0 0x7ff47006d990 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff48c09da20 0x7ff48c156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff48c0cddd0 0x7ff48c161cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 --2- 172.21.15.72:0/2382266239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff4780039e0 0x7ff48c09a270 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 >> 172.21.15.72:0/2382266239 conn(0x7ff48c0724d0 msgr2=0x7ff48c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 shutdown_connections 2024-03-06T07:26:01.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.571+0000 7ff49393e640 1 -- 172.21.15.72:0/2382266239 wait complete. 2024-03-06T07:26:01.584 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:01.584 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:01.584 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:01.584 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:01.584 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:01.585 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:01.585 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:01.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:01.585 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:01.585 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:01.585 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:01.585 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:01.585 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:01.585 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:01.586 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:01.586 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:01.594 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:01.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:01.595 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:01.595 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:01.597 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:01.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:01.598 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:01.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:01.636 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:01.636 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:01.637 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:01.637 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:01.641 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:01.641 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:01.641 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:01.642 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:01.642 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:01.712 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:26:01.712 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:26:01.724 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:26:01.724 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:26:01.725 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-03-06T07:26:01.723+0000 7f255a34c640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-03-06T07:26:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.745+0000 7f5e2a316640 1 Processor -- start 2024-03-06T07:26:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.745+0000 7f5e2a316640 1 -- start start 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e2a316640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e2415e000 0x7f5e240d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e2a316640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240d4810 0x7f5e240d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e2a316640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e240d25e0 con 0x7f5e240c80e0 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e2a316640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e240c7c50 con 0x7f5e2415e000 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e2a316640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240d2880 con 0x7f5e240d4810 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e23fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e2415e000 0x7f5e240d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e288b3640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5e240c80e0 0x7f5e240c8570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36070/0 (socket says 172.21.15.72:36070) 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e288b3640 1 -- 172.21.15.72:0/1290941071 learned_addr learned my addr 172.21.15.72:0/1290941071 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e237fe640 1 --2- 172.21.15.72:0/1290941071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240d4810 0x7f5e240d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e23fff640 1 -- 172.21.15.72:0/1290941071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240d4810 msgr2=0x7f5e240d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e23fff640 1 --2- 172.21.15.72:0/1290941071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240d4810 0x7f5e240d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e23fff640 1 -- 172.21.15.72:0/1290941071 >> v1:172.21.15.72:6789/0 conn(0x7f5e240c80e0 legacy=0x7f5e240c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e23fff640 1 -- 172.21.15.72:0/1290941071 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e240d2b40 con 0x7f5e2415e000 2024-03-06T07:26:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.746+0000 7f5e22ffd640 1 -- 172.21.15.72:0/1290941071 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529615793 0 0) 0x7f5e240d25e0 con 0x7f5e240c80e0 2024-03-06T07:26:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.747+0000 7f5e23fff640 1 --2- 172.21.15.72:0/1290941071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e240d0680 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f5e140097a0 tx=0x7f5e1402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bcc1b729b4524d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.747+0000 7f5e22ffd640 1 -- 172.21.15.72:0/1290941071 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e14034070 con 0x7f5e2415e000 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.747+0000 7f5e22ffd640 1 -- 172.21.15.72:0/1290941071 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e14030610 con 0x7f5e2415e000 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.747+0000 7f5e22ffd640 1 -- 172.21.15.72:0/1290941071 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e140309e0 con 0x7f5e2415e000 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 -- 172.21.15.72:0/1290941071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 msgr2=0x7f5e240d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 --2- 172.21.15.72:0/1290941071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e240d0680 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f5e140097a0 tx=0x7f5e1402b960 comp rx=0 tx=0).stop 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 -- 172.21.15.72:0/1290941071 shutdown_connections 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 --2- 172.21.15.72:0/1290941071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240d4810 0x7f5e240d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 --2- 172.21.15.72:0/1290941071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e240d0680 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 -- 172.21.15.72:0/1290941071 >> 172.21.15.72:0/1290941071 conn(0x7f5e240d3cf0 msgr2=0x7f5e240d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 -- 172.21.15.72:0/1290941071 shutdown_connections 2024-03-06T07:26:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.748+0000 7f5e2a316640 1 -- 172.21.15.72:0/1290941071 wait complete. 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.749+0000 7f5e2a316640 1 Processor -- start 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.749+0000 7f5e2a316640 1 -- start start 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240c80e0 0x7f5e240c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d4810 0x7f5e2415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e24160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2415e3e0 con 0x7f5e2415e000 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240d22d0 con 0x7f5e240d4810 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240cf7d0 con 0x7f5e240c80e0 2024-03-06T07:26:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e24160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e24160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42648/0 (socket says 172.21.15.72:42648) 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 -- 172.21.15.72:0/2956109225 learned_addr learned my addr 172.21.15.72:0/2956109225 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 -- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240c80e0 msgr2=0x7f5e240c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e237fe640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d4810 0x7f5e2415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e23fff640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240c80e0 0x7f5e240c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240c80e0 0x7f5e240c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 -- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d4810 msgr2=0x7f5e2415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d4810 0x7f5e2415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 -- 172.21.15.72:0/2956109225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e240cdcb0 con 0x7f5e2415e000 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e237fe640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d4810 0x7f5e2415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e23fff640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240c80e0 0x7f5e240c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e288b3640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e24160730 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00c7d0 tx=0x7f5e1c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c018070 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c00a640 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e240cdf90 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c00aa70 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.750+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e240caef0 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.751+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e2404f330 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.752+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e1c01b070 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.752+0000 7f5e20ff9640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e1006b4d0 0x7f5e1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.752+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5e1c089490 con 0x7f5e2415e000 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.752+0000 7f5e23fff640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e1006b4d0 0x7f5e1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:01.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.753+0000 7f5e23fff640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e1006b4d0 0x7f5e1006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5e14036000 tx=0x7f5e14008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:01.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.756+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e1c0586a0 con 0x7f5e2415e000 2024-03-06T07:26:01.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:01.940+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5e240d09b0 con 0x7f5e2415e000 2024-03-06T07:26:02.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.723+0000 7f5e20ff9640 1 -- 172.21.15.72:0/2956109225 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v88) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5e1c057fa0 con 0x7f5e2415e000 2024-03-06T07:26:02.726 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:26:02.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e1006b4d0 msgr2=0x7f5e1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e1006b4d0 0x7f5e1006d990 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5e14036000 tx=0x7f5e14008030 comp rx=0 tx=0).stop 2024-03-06T07:26:02.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 msgr2=0x7f5e24160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e24160730 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c00c7d0 tx=0x7f5e1c00cc90 comp rx=0 tx=0).stop 2024-03-06T07:26:02.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 shutdown_connections 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e1006b4d0 0x7f5e1006d990 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e2415e000 0x7f5e24160730 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e240d4810 0x7f5e2415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 --2- 172.21.15.72:0/2956109225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e240c80e0 0x7f5e240c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 >> 172.21.15.72:0/2956109225 conn(0x7f5e240d3cf0 msgr2=0x7f5e240ce680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 shutdown_connections 2024-03-06T07:26:02.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.726+0000 7f5e2a316640 1 -- 172.21.15.72:0/2956109225 wait complete. 2024-03-06T07:26:02.741 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:02.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.839+0000 7fc06e02a640 1 Processor -- start 2024-03-06T07:26:02.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.839+0000 7fc06e02a640 1 -- start start 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.839+0000 7fc06e02a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc0680cd7a0 0x7fc0680cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.839+0000 7fc06e02a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680cc980 con 0x7fc06806da40 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.839+0000 7fc06e02a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0680ccc60 con 0x7fc0681424f0 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.839+0000 7fc06e02a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0680cd220 con 0x7fc0680cd7a0 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.840+0000 7fc067fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc0680cd7a0 0x7fc0680cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.840+0000 7fc05effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc06806da40 0x7fc068071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42506/0 (socket says 172.21.15.72:42506) 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.840+0000 7fc05effd640 1 -- 172.21.15.72:0/1443557019 learned_addr learned my addr 172.21.15.72:0/1443557019 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.840+0000 7fc067fff640 1 -- 172.21.15.72:0/1443557019 >> v1:172.21.15.146:6789/0 conn(0x7fc0681424f0 legacy=0x7fc06806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.840+0000 7fc067fff640 1 -- 172.21.15.72:0/1443557019 >> v1:172.21.15.72:6789/0 conn(0x7fc06806da40 legacy=0x7fc068071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.840+0000 7fc067fff640 1 -- 172.21.15.72:0/1443557019 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680ccf40 con 0x7fc0680cd7a0 2024-03-06T07:26:02.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc067fff640 1 --2- 172.21.15.72:0/1443557019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0680cd7a0 0x7fc0680cc2f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc060009d90 tx=0x7fc06002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c69c982f8ec6b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc066ffd640 1 -- 172.21.15.72:0/1443557019 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060039070 con 0x7fc0680cd7a0 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc066ffd640 1 -- 172.21.15.72:0/1443557019 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06002cc60 con 0x7fc0680cd7a0 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc066ffd640 1 -- 172.21.15.72:0/1443557019 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc060034660 con 0x7fc0680cd7a0 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 -- 172.21.15.72:0/1443557019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0680cd7a0 msgr2=0x7fc0680cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1443557019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0680cd7a0 0x7fc0680cc2f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc060009d90 tx=0x7fc06002beb0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 -- 172.21.15.72:0/1443557019 shutdown_connections 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1443557019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc0680cd7a0 0x7fc0680cc2f0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 -- 172.21.15.72:0/1443557019 >> 172.21.15.72:0/1443557019 conn(0x7fc0680cf8a0 msgr2=0x7fc0680cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 -- 172.21.15.72:0/1443557019 shutdown_connections 2024-03-06T07:26:02.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.841+0000 7fc06e02a640 1 -- 172.21.15.72:0/1443557019 wait complete. 2024-03-06T07:26:02.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.842+0000 7fc06e02a640 1 Processor -- start 2024-03-06T07:26:02.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 -- start start 2024-03-06T07:26:02.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 0x7fc0680b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0680cd7a0 0x7fc06813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0681424f0 0x7fc068140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06815d070 con 0x7fc06806da40 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06815cef0 con 0x7fc0680cd7a0 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc06e02a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc06815d1f0 con 0x7fc0681424f0 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc0677fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 0x7fc0680b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc0677fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 0x7fc0680b2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34468/0 (socket says 172.21.15.72:34468) 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.843+0000 7fc0677fe640 1 -- 172.21.15.72:0/1916293038 learned_addr learned my addr 172.21.15.72:0/1916293038 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc05effd640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0680cd7a0 0x7fc06813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc067fff640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0681424f0 0x7fc068140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc0677fe640 1 -- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0681424f0 msgr2=0x7fc068140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc0677fe640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0681424f0 0x7fc068140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc0677fe640 1 -- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0680cd7a0 msgr2=0x7fc06813fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:02.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc0677fe640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0680cd7a0 0x7fc06813fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc0677fe640 1 -- 172.21.15.72:0/1916293038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0680b4360 con 0x7fc06806da40 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc0677fe640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 0x7fc0680b2a50 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc05800b410 tx=0x7fc05800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058015020 con 0x7fc06806da40 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc058011830 con 0x7fc06806da40 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0680b32c0 con 0x7fc06806da40 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058011c20 con 0x7fc06806da40 2024-03-06T07:26:02.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.844+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0680b3880 con 0x7fc06806da40 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.845+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0680af3f0 con 0x7fc06806da40 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.845+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc058018020 con 0x7fc06806da40 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.846+0000 7fc064ff9640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc04806b4d0 0x7fc04806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.846+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc058089db0 con 0x7fc06806da40 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.846+0000 7fc05effd640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc04806b4d0 0x7fc04806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:02.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.846+0000 7fc05effd640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc04806b4d0 0x7fc04806d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc06815cb70 tx=0x7fc050009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:02.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:02.850+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc05801f020 con 0x7fc06806da40 2024-03-06T07:26:03.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.083+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc06804f330 con 0x7fc06806da40 2024-03-06T07:26:03.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.083+0000 7fc064ff9640 1 -- 172.21.15.72:0/1916293038 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc0580172c0 con 0x7fc06806da40 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.086 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.087 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.088 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.089 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:26:03.090 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.091 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:26:03.092 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.093 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.094 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:26:03.095 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:26:03.096 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.097 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:26:03.098 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:26:03.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc04806b4d0 msgr2=0x7fc04806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc04806b4d0 0x7fc04806d990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc06815cb70 tx=0x7fc050009040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 msgr2=0x7fc0680b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 0x7fc0680b2a50 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc05800b410 tx=0x7fc05800b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 shutdown_connections 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc04806b4d0 0x7fc04806d990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc0681424f0 0x7fc068140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc0680cd7a0 0x7fc06813fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 --2- 172.21.15.72:0/1916293038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06806da40 0x7fc0680b2a50 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 >> 172.21.15.72:0/1916293038 conn(0x7fc0680cf8a0 msgr2=0x7fc06815d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 shutdown_connections 2024-03-06T07:26:03.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.085+0000 7fc06e02a640 1 -- 172.21.15.72:0/1916293038 wait complete. 2024-03-06T07:26:03.101 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:03.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc06a684640 1 Processor -- start 2024-03-06T07:26:03.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc06a684640 1 -- start start 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc06a684640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 0x7fc0640c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc06a684640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0640cebe0 con 0x7fc06415fa10 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc06a684640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0640ceec0 con 0x7fc0640d16e0 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc06a684640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640ce980 con 0x7fc06415bdd0 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc0637fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc06415fa10 0x7fc0640ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42520/0 (socket says 172.21.15.72:42520) 2024-03-06T07:26:03.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.199+0000 7fc0637fe640 1 -- 172.21.15.72:0/1525157215 learned_addr learned my addr 172.21.15.72:0/1525157215 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc063fff640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 0x7fc0640c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713730604 0 0) 0x7fc0640cebe0 con 0x7fc06415fa10 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c003520 con 0x7fc06415fa10 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3922009772 0 0) 0x7fc04c003520 con 0x7fc06415fa10 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 msgr2=0x7fc0640c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 0x7fc0640c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 >> v1:172.21.15.146:6789/0 conn(0x7fc0640d16e0 legacy=0x7fc0640d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0640cf1a0 con 0x7fc06415fa10 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fc054001670 con 0x7fc06415fa10 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 >> v1:172.21.15.72:6789/0 conn(0x7fc06415fa10 legacy=0x7fc0640ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 0x7fc04c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc04c003f60 0x7fc04c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c004d40 0x7fc04c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640ce980 con 0x7fc04c003f60 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c00bb50 con 0x7fc04c004d40 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c00bd50 con 0x7fc06415bdd0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc04c003f60 0x7fc04c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc063fff640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 0x7fc04c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc068c21640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c004d40 0x7fc04c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc0637fe640 1 -- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 msgr2=0x7fc04c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415bdd0 0x7fc04c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc0637fe640 1 -- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c004d40 msgr2=0x7fc04c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c004d40 0x7fc04c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.200+0000 7fc0637fe640 1 -- 172.21.15.72:0/1525157215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c003520 con 0x7fc04c003f60 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.201+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc04c003f60 0x7fc04c004800 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc05400c210 tx=0x7fc04c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fe26a1713d21ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.201+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054007d30 con 0x7fc04c003f60 2024-03-06T07:26:03.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.201+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054030070 con 0x7fc04c003f60 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.201+0000 7fc062ffd640 1 -- 172.21.15.72:0/1525157215 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05402cd60 con 0x7fc04c003f60 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc06a684640 1 -- 172.21.15.72:0/1525157215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc04c003f60 msgr2=0x7fc04c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc06a684640 1 --2- 172.21.15.72:0/1525157215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc04c003f60 0x7fc04c004800 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc05400c210 tx=0x7fc04c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc068c21640 1 -- 172.21.15.72:0/1525157215 reap_dead start 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc06a684640 1 -- 172.21.15.72:0/1525157215 shutdown_connections 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc06a684640 1 -- 172.21.15.72:0/1525157215 >> 172.21.15.72:0/1525157215 conn(0x7fc06409a9f0 msgr2=0x7fc06409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc06a684640 1 -- 172.21.15.72:0/1525157215 shutdown_connections 2024-03-06T07:26:03.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.202+0000 7fc06a684640 1 -- 172.21.15.72:0/1525157215 wait complete. 2024-03-06T07:26:03.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.203+0000 7fc06a684640 1 Processor -- start 2024-03-06T07:26:03.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.203+0000 7fc06a684640 1 -- start start 2024-03-06T07:26:03.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc06a684640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c003f60 0x7fc06409c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc06a684640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 0x7fc06415c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc06a684640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415e220 0x7fc06415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc06a684640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06415c1b0 con 0x7fc06415c3c0 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc06a684640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640cc5d0 con 0x7fc04c003f60 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc06a684640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640c5610 con 0x7fc06415e220 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 0x7fc06415c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 0x7fc06415c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34484/0 (socket says 172.21.15.72:34484) 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 -- 172.21.15.72:0/1130641606 learned_addr learned my addr 172.21.15.72:0/1130641606 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc063fff640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c003f60 0x7fc06409c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc068c21640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415e220 0x7fc06415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 -- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415e220 msgr2=0x7fc06415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415e220 0x7fc06415e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 -- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c003f60 msgr2=0x7fc06409c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c003f60 0x7fc06409c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 -- 172.21.15.72:0/1130641606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc06415d180 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.204+0000 7fc0637fe640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 0x7fc06415c840 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc05400c330 tx=0x7fc054001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.205+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054045070 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.205+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc054004bf0 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.205+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc06415d460 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.205+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc054007d10 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.205+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06415da20 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.205+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc06404f330 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.206+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc054007070 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.206+0000 7fc060ff9640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc05006b500 0x7fc05006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.206+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0540aaf40 con 0x7fc06415c3c0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.206+0000 7fc063fff640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc05006b500 0x7fc05006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.207+0000 7fc063fff640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc05006b500 0x7fc05006d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc058009a10 tx=0x7fc058008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.211+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc054079ba0 con 0x7fc06415c3c0 2024-03-06T07:26:03.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.382+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0640b7a40 con 0x7fc06415c3c0 2024-03-06T07:26:03.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.383+0000 7fc060ff9640 1 -- 172.21.15.72:0/1130641606 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+12904 (secure 0 0 0) 0x7fc05407dc40 con 0x7fc06415c3c0 2024-03-06T07:26:03.385 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:03.386 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":90,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:02.735007+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:03.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc05006b500 msgr2=0x7fc05006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc05006b500 0x7fc05006d9c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc058009a10 tx=0x7fc058008040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 msgr2=0x7fc06415c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 0x7fc06415c840 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fc05400c330 tx=0x7fc054001a40 comp rx=0 tx=0).stop 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 shutdown_connections 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc05006b500 0x7fc05006d9c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc06415e220 0x7fc06415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc06415c3c0 0x7fc06415c840 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 --2- 172.21.15.72:0/1130641606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc04c003f60 0x7fc06409c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 >> 172.21.15.72:0/1130641606 conn(0x7fc06409a9f0 msgr2=0x7fc06406ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 shutdown_connections 2024-03-06T07:26:03.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.385+0000 7fc06a684640 1 -- 172.21.15.72:0/1130641606 wait complete. 2024-03-06T07:26:03.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:03.399 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:03.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.499+0000 7fef9035e640 1 Processor -- start 2024-03-06T07:26:03.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.499+0000 7fef9035e640 1 -- start start 2024-03-06T07:26:03.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.499+0000 7fef9035e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fef8815bb40 0x7fef880b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef9035e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 0x7fef880cff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef9035e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8815ff90 con 0x7fef8815f7d0 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef9035e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef8806c910 con 0x7fef8815bb40 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef9035e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef88160270 con 0x7fef880cfa90 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d8f9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fef8815f7d0 0x7fef8806d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42528/0 (socket says 172.21.15.72:42528) 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d8f9640 1 -- 172.21.15.72:0/3735498834 learned_addr learned my addr 172.21.15.72:0/3735498834 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8e8fb640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 0x7fef880cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8e0fa640 1 --2- 172.21.15.72:0/3735498834 >> v2:172.21.15.146:3300/0 conn(0x7fef8815bb40 0x7fef880b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420925055 0 0) 0x7fef8815ff90 con 0x7fef8815f7d0 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef74003520 con 0x7fef8815f7d0 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900603873 0 0) 0x7fef74003520 con 0x7fef8815f7d0 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 msgr2=0x7fef880cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 0x7fef880cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 >> v2:172.21.15.146:3300/0 conn(0x7fef8815bb40 msgr2=0x7fef880b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 --2- 172.21.15.72:0/3735498834 >> v2:172.21.15.146:3300/0 conn(0x7fef8815bb40 0x7fef880b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef88160550 con 0x7fef8815f7d0 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fef78001670 con 0x7fef8815f7d0 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 >> v1:172.21.15.72:6789/0 conn(0x7fef8815f7d0 legacy=0x7fef8806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef740039b0 0x7fef74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 0x7fef740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8806c910 con 0x7fef74004800 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88160270 con 0x7fef740039b0 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7400b9c0 con 0x7fef880cfa90 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8d8f9640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.500+0000 7fef8e0fa640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef740039b0 0x7fef74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8e8fb640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 0x7fef740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d8f9640 1 -- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 msgr2=0x7fef740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d8f9640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef880cfa90 0x7fef740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d8f9640 1 -- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef740039b0 msgr2=0x7fef74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d8f9640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef740039b0 0x7fef74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d8f9640 1 -- 172.21.15.72:0/3735498834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef74003520 con 0x7fef74004800 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8e0fa640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef740039b0 0x7fef74003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d8f9640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef74004c80 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fef78001050 tx=0x7fef74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61827379f11e3d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef78007d70 con 0x7fef74004800 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef78030070 con 0x7fef74004800 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.501+0000 7fef8d0f8640 1 -- 172.21.15.72:0/3735498834 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7802cd80 con 0x7fef74004800 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef9035e640 1 -- 172.21.15.72:0/3735498834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 msgr2=0x7fef74004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef9035e640 1 --2- 172.21.15.72:0/3735498834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef74004c80 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fef78001050 tx=0x7fef74005830 comp rx=0 tx=0).stop 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef8e8fb640 1 -- 172.21.15.72:0/3735498834 reap_dead start 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef9035e640 1 -- 172.21.15.72:0/3735498834 shutdown_connections 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef9035e640 1 -- 172.21.15.72:0/3735498834 >> 172.21.15.72:0/3735498834 conn(0x7fef8806b190 msgr2=0x7fef8806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef9035e640 1 -- 172.21.15.72:0/3735498834 shutdown_connections 2024-03-06T07:26:03.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.502+0000 7fef9035e640 1 -- 172.21.15.72:0/3735498834 wait complete. 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.503+0000 7fef9035e640 1 Processor -- start 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.503+0000 7fef9035e640 1 -- start start 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef9035e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef881f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef9035e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef88070980 0x7fef88152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef9035e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef88153360 0x7fef8815d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef9035e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8806d790 con 0x7fef74004800 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef9035e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8815fc30 con 0x7fef88153360 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef9035e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8815bf20 con 0x7fef88070980 2024-03-06T07:26:03.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef881f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef881f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34502/0 (socket says 172.21.15.72:34502) 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 -- 172.21.15.72:0/3765295730 learned_addr learned my addr 172.21.15.72:0/3765295730 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 -- 172.21.15.72:0/3765295730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef88070980 msgr2=0x7fef88152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef88070980 0x7fef88152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 -- 172.21.15.72:0/3765295730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef88153360 msgr2=0x7fef8815d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef88153360 0x7fef8815d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 -- 172.21.15.72:0/3765295730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef880adc20 con 0x7fef74004800 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.504+0000 7fef8e0fa640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef881f79f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fef840115d0 tx=0x7fef84011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.505+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84015870 con 0x7fef74004800 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.505+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef84015a10 con 0x7fef74004800 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.505+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef84015dc0 con 0x7fef74004800 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.505+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef880adf00 con 0x7fef74004800 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.505+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef880ae4c0 con 0x7fef74004800 2024-03-06T07:26:03.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.506+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef880732b0 con 0x7fef74004800 2024-03-06T07:26:03.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.506+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fef8400f460 con 0x7fef74004800 2024-03-06T07:26:03.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.506+0000 7fef7effd640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef6c06b4d0 0x7fef6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.507+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fef8408a440 con 0x7fef74004800 2024-03-06T07:26:03.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.507+0000 7fef8d8f9640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef6c06b4d0 0x7fef6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.507+0000 7fef8d8f9640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef6c06b4d0 0x7fef6c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fef78034000 tx=0x7fef7802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.511+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fef84019030 con 0x7fef74004800 2024-03-06T07:26:03.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.682+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef8806e130 con 0x7fef74004800 2024-03-06T07:26:03.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.682+0000 7fef7effd640 1 -- 172.21.15.72:0/3765295730 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fef84059610 con 0x7fef74004800 2024-03-06T07:26:03.685 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:26:03.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef6c06b4d0 msgr2=0x7fef6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef6c06b4d0 0x7fef6c06d990 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fef78034000 tx=0x7fef7802e000 comp rx=0 tx=0).stop 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 msgr2=0x7fef881f79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef881f79f0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fef840115d0 tx=0x7fef84011a90 comp rx=0 tx=0).stop 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 shutdown_connections 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fef6c06b4d0 0x7fef6c06d990 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef88153360 0x7fef8815d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef88070980 0x7fef88152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 --2- 172.21.15.72:0/3765295730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fef74004800 0x7fef881f79f0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 >> 172.21.15.72:0/3765295730 conn(0x7fef8806b190 msgr2=0x7fef880b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 shutdown_connections 2024-03-06T07:26:03.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.684+0000 7fef9035e640 1 -- 172.21.15.72:0/3765295730 wait complete. 2024-03-06T07:26:03.697 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:26:03.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.796+0000 7f27fc9bc640 1 Processor -- start 2024-03-06T07:26:03.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.796+0000 7f27fc9bc640 1 -- start start 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.796+0000 7f27fc9bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f80b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.796+0000 7f27fc9bc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27f815bf30 0x7f27f80b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.796+0000 7f27fc9bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27f815e850 0x7f27f815eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.796+0000 7f27fc9bc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f80cf3f0 con 0x7f27f815bf30 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27fc9bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f815b210 con 0x7f27f815e850 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27fc9bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f815f580 con 0x7f27f813dc10 2024-03-06T07:26:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27f815bf30 0x7f27f80b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f27f815bf30 0x7f27f80b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34508/0 (socket says 172.21.15.72:34508) 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 -- 172.21.15.72:0/2963366898 learned_addr learned my addr 172.21.15.72:0/2963366898 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27f659c640 1 --2- 172.21.15.72:0/2963366898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f80b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27f6d9d640 1 --2- 172.21.15.72:0/2963366898 >> v2:172.21.15.146:3300/0 conn(0x7f27f815e850 0x7f27f815eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 -- 172.21.15.72:0/2963366898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 msgr2=0x7f27f80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 --2- 172.21.15.72:0/2963366898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f80b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 -- 172.21.15.72:0/2963366898 >> v2:172.21.15.146:3300/0 conn(0x7f27f815e850 msgr2=0x7f27f815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 --2- 172.21.15.72:0/2963366898 >> v2:172.21.15.146:3300/0 conn(0x7f27f815e850 0x7f27f815eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27edd9b640 1 -- 172.21.15.72:0/2963366898 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f815b450 con 0x7f27f815bf30 2024-03-06T07:26:03.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.797+0000 7f27f6d9d640 1 --2- 172.21.15.72:0/2963366898 >> v2:172.21.15.146:3300/0 conn(0x7f27f815e850 0x7f27f815eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27edd9b640 1 --2- 172.21.15.72:0/2963366898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f80b2e90 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f27f806c790 tx=0x7f27e002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c480a4c2a18281c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27f5d9b640 1 -- 172.21.15.72:0/2963366898 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00317c0 con 0x7f27f815bf30 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27f5d9b640 1 -- 172.21.15.72:0/2963366898 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0037070 con 0x7f27f815bf30 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27f5d9b640 1 -- 172.21.15.72:0/2963366898 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0031b80 con 0x7f27f815bf30 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 -- 172.21.15.72:0/2963366898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 msgr2=0x7f27f80b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/2963366898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f80b2e90 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f27f806c790 tx=0x7f27e002b860 comp rx=0 tx=0).stop 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 -- 172.21.15.72:0/2963366898 shutdown_connections 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/2963366898 >> v2:172.21.15.146:3300/0 conn(0x7f27f815e850 0x7f27f815eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/2963366898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f80b2e90 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/2963366898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f80b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 -- 172.21.15.72:0/2963366898 >> 172.21.15.72:0/2963366898 conn(0x7f27f80cf8a0 msgr2=0x7f27f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 -- 172.21.15.72:0/2963366898 shutdown_connections 2024-03-06T07:26:03.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.798+0000 7f27fc9bc640 1 -- 172.21.15.72:0/2963366898 wait complete. 2024-03-06T07:26:03.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.800+0000 7f27fc9bc640 1 Processor -- start 2024-03-06T07:26:03.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.800+0000 7f27fc9bc640 1 -- start start 2024-03-06T07:26:03.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f81f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f815e850 0x7f27f81f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f815f1e0 con 0x7f27f815bf30 2024-03-06T07:26:03.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f813dff0 con 0x7f27f815e850 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f815f360 con 0x7f27f813dc10 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f81f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f81f71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34510/0 (socket says 172.21.15.72:34510) 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 -- 172.21.15.72:0/1027721689 learned_addr learned my addr 172.21.15.72:0/1027721689 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27f659c640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27f6d9d640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f815e850 0x7f27f81f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 -- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 msgr2=0x7f27f815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f815dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 -- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f815e850 msgr2=0x7f27f81f7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f815e850 0x7f27f81f7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 -- 172.21.15.72:0/1027721689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f81f7e10 con 0x7f27f815bf30 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27f6d9d640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f815e850 0x7f27f81f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27edd9b640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f81f71c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f27e002b8a0 tx=0x7f27e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0032860 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0032a20 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f81f3de0 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0032e10 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.801+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f81f43a0 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.802+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f81f3bd0 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.803+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f27e004c060 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.803+0000 7f27ef7fe640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27d806b4d0 0x7f27d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.803+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f27e00ab480 con 0x7f27f815bf30 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.803+0000 7f27f659c640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27d806b4d0 0x7f27d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:03.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.804+0000 7f27f659c640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27d806b4d0 0x7f27d806d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f27e8003610 tx=0x7f27e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:03.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.808+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f27e0040070 con 0x7f27f815bf30 2024-03-06T07:26:03.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.979+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f27f804f330 con 0x7f27f815bf30 2024-03-06T07:26:03.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.979+0000 7f27ef7fe640 1 -- 172.21.15.72:0/1027721689 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v90) v1 ==== 84+0+65 (secure 0 0 0) 0x7f27e0031e90 con 0x7f27f815bf30 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:03.982 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:26:03.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27d806b4d0 msgr2=0x7f27d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27d806b4d0 0x7f27d806d990 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f27e8003610 tx=0x7f27e8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 msgr2=0x7f27f81f71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f81f71c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f27e002b8a0 tx=0x7f27e00046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 shutdown_connections 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f27d806b4d0 0x7f27d806d990 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f815e850 0x7f27f81f7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27f815bf30 0x7f27f81f71c0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 --2- 172.21.15.72:0/1027721689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f813dc10 0x7f27f815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 >> 172.21.15.72:0/1027721689 conn(0x7f27f80cf8a0 msgr2=0x7f27f80af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 shutdown_connections 2024-03-06T07:26:03.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:03.981+0000 7f27fc9bc640 1 -- 172.21.15.72:0/1027721689 wait complete. 2024-03-06T07:26:03.994 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:26:03.994 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:26:04.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.095+0000 7fb399113640 1 Processor -- start 2024-03-06T07:26:04.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.095+0000 7fb399113640 1 -- start start 2024-03-06T07:26:04.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb399113640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb394161820 0x7fb3940d0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb399113640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3940d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb399113640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3940cf520 con 0x7fb394161440 2024-03-06T07:26:04.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb399113640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3940cf2d0 con 0x7fb394161820 2024-03-06T07:26:04.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb399113640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940b6b00 con 0x7fb3940d1300 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb392d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb394161440 0x7fb3940b4be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42532/0 (socket says 172.21.15.72:42532) 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb392d9d640 1 -- 172.21.15.72:0/625942611 learned_addr learned my addr 172.21.15.72:0/625942611 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924764080 0 0) 0x7fb3940cf520 con 0x7fb394161440 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb380003520 con 0x7fb394161440 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39359e640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3940d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb38bfff640 1 --2- 172.21.15.72:0/625942611 >> v2:172.21.15.146:3300/0 conn(0x7fb394161820 0x7fb3940d0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 876610268 0 0) 0x7fb380003520 con 0x7fb394161440 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 msgr2=0x7fb3940d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3940d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 >> v2:172.21.15.146:3300/0 conn(0x7fb394161820 msgr2=0x7fb3940d0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 --2- 172.21.15.72:0/625942611 >> v2:172.21.15.146:3300/0 conn(0x7fb394161820 0x7fb3940d0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3940cf800 con 0x7fb394161440 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fb384001690 con 0x7fb394161440 2024-03-06T07:26:04.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 >> v1:172.21.15.72:6789/0 conn(0x7fb394161440 legacy=0x7fb3940b4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3800039b0 0x7fb380003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb380004800 0x7fb380004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940cf2d0 con 0x7fb380004800 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940b6b00 con 0x7fb3800039b0 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.096+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38000ba40 con 0x7fb3940d1300 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb380004800 0x7fb380004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb392d9d640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3800039b0 0x7fb380003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb39359e640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 -- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 msgr2=0x7fb3800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 -- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3800039b0 msgr2=0x7fb380003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3800039b0 0x7fb380003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 -- 172.21.15.72:0/625942611 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb380003520 con 0x7fb380004800 2024-03-06T07:26:04.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb39359e640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3940d1300 0x7fb3800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb392d9d640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3800039b0 0x7fb380003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.097+0000 7fb38bfff640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb380004800 0x7fb380004c80 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009a90 tx=0x7fb380005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82dc0ab56d180917 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c039070 con 0x7fb380004800 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c02c6b0 con 0x7fb380004800 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb39259c640 1 -- 172.21.15.72:0/625942611 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37c009690 con 0x7fb380004800 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb399113640 1 -- 172.21.15.72:0/625942611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb380004800 msgr2=0x7fb380004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb399113640 1 --2- 172.21.15.72:0/625942611 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb380004800 0x7fb380004c80 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009a90 tx=0x7fb380005830 comp rx=0 tx=0).stop 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb39359e640 1 -- 172.21.15.72:0/625942611 reap_dead start 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb399113640 1 -- 172.21.15.72:0/625942611 shutdown_connections 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb399113640 1 -- 172.21.15.72:0/625942611 >> 172.21.15.72:0/625942611 conn(0x7fb3940d3cb0 msgr2=0x7fb3940d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb399113640 1 -- 172.21.15.72:0/625942611 shutdown_connections 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.098+0000 7fb399113640 1 -- 172.21.15.72:0/625942611 wait complete. 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.099+0000 7fb399113640 1 Processor -- start 2024-03-06T07:26:04.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 -- start start 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb380004800 0x7fb39414a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb394156bb0 0x7fb39415fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 0x7fb3940cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940d0cd0 con 0x7fb3941602b0 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940d0b50 con 0x7fb380004800 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb399113640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3940d0e50 con 0x7fb394156bb0 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb39359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 0x7fb3940cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb39359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 0x7fb3940cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34520/0 (socket says 172.21.15.72:34520) 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb39359e640 1 -- 172.21.15.72:0/987428993 learned_addr learned my addr 172.21.15.72:0/987428993 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb38bfff640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb394156bb0 0x7fb39415fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.100+0000 7fb392d9d640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb380004800 0x7fb39414a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb39359e640 1 -- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb394156bb0 msgr2=0x7fb39415fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb39359e640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb394156bb0 0x7fb39415fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb39359e640 1 -- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb380004800 msgr2=0x7fb39414a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb39359e640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb380004800 0x7fb39414a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb39359e640 1 -- 172.21.15.72:0/987428993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3940c7860 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb39359e640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 0x7fb3940cd170 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fb3940cad10 tx=0x7fb38c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c01a030 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c003990 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c003d80 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3940c7b40 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.101+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3940c8100 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.102+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3940b7000 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.102+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb38c01d070 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.103+0000 7fb38b7fe640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb37406b500 0x7fb37406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.103+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb38c08a2c0 con 0x7fb3941602b0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.103+0000 7fb392d9d640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb37406b500 0x7fb37406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.103+0000 7fb392d9d640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb37406b500 0x7fb37406d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb384001120 tx=0x7fb384007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.107+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3940b7000 con 0x7fb3941602b0 2024-03-06T07:26:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.277+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb3940c8300 con 0x7fb3941602b0 2024-03-06T07:26:04.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.726+0000 7fb38b7fe640 1 -- 172.21.15.72:0/987428993 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v91) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb3940c8300 con 0x7fb3941602b0 2024-03-06T07:26:04.728 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:26:04.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb37406b500 msgr2=0x7fb37406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb37406b500 0x7fb37406d9c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb384001120 tx=0x7fb384007040 comp rx=0 tx=0).stop 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 msgr2=0x7fb3940cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 0x7fb3940cd170 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fb3940cad10 tx=0x7fb38c010590 comp rx=0 tx=0).stop 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 shutdown_connections 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb37406b500 0x7fb37406d9c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3941602b0 0x7fb3940cd170 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb394156bb0 0x7fb39415fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 --2- 172.21.15.72:0/987428993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb380004800 0x7fb39414a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.729+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 >> 172.21.15.72:0/987428993 conn(0x7fb3940d3cb0 msgr2=0x7fb3941563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.730+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 shutdown_connections 2024-03-06T07:26:04.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.731+0000 7fb399113640 1 -- 172.21.15.72:0/987428993 wait complete. 2024-03-06T07:26:04.748 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:26:04.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.853+0000 7f2884684640 1 Processor -- start 2024-03-06T07:26:04.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.853+0000 7f2884684640 1 -- start start 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2884684640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f287c15bb50 0x7f287c1625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2884684640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f287c15f140 0x7f287c071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2884684640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c06e2c0 0x7f287c06e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2884684640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287c162160 con 0x7f287c15bb50 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2884684640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287c072870 con 0x7f287c15f140 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2884684640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c072b50 con 0x7f287c06e2c0 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882420640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f287c15bb50 0x7f287c1625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882420640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f287c15bb50 0x7f287c1625f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34528/0 (socket says 172.21.15.72:34528) 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882420640 1 -- 172.21.15.72:0/3873087010 learned_addr learned my addr 172.21.15.72:0/3873087010 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882c21640 1 --2- 172.21.15.72:0/3873087010 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c06e2c0 0x7f287c06e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2881c1f640 1 --2- 172.21.15.72:0/3873087010 >> v2:172.21.15.146:3300/0 conn(0x7f287c15f140 0x7f287c071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882420640 1 -- 172.21.15.72:0/3873087010 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c06e2c0 msgr2=0x7f287c06e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882420640 1 --2- 172.21.15.72:0/3873087010 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c06e2c0 0x7f287c06e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.854+0000 7f2882420640 1 -- 172.21.15.72:0/3873087010 >> v2:172.21.15.146:3300/0 conn(0x7f287c15f140 msgr2=0x7f287c071fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.855+0000 7f2882420640 1 --2- 172.21.15.72:0/3873087010 >> v2:172.21.15.146:3300/0 conn(0x7f287c15f140 0x7f287c071fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.855+0000 7f2882420640 1 -- 172.21.15.72:0/3873087010 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c161b10 con 0x7f287c15bb50 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.855+0000 7f2882420640 1 --2- 172.21.15.72:0/3873087010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1625f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f287c068020 tx=0x7f286c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ed6d05836e065a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.855+0000 7f288141e640 1 -- 172.21.15.72:0/3873087010 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c02e070 con 0x7f287c15bb50 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.855+0000 7f288141e640 1 -- 172.21.15.72:0/3873087010 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286c037070 con 0x7f287c15bb50 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.855+0000 7f288141e640 1 -- 172.21.15.72:0/3873087010 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286c031a00 con 0x7f287c15bb50 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 -- 172.21.15.72:0/3873087010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 msgr2=0x7f287c1625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 --2- 172.21.15.72:0/3873087010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1625f0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f287c068020 tx=0x7f286c02b880 comp rx=0 tx=0).stop 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 -- 172.21.15.72:0/3873087010 shutdown_connections 2024-03-06T07:26:04.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 --2- 172.21.15.72:0/3873087010 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c06e2c0 0x7f287c06e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 --2- 172.21.15.72:0/3873087010 >> v2:172.21.15.146:3300/0 conn(0x7f287c15f140 0x7f287c071fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 --2- 172.21.15.72:0/3873087010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1625f0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 -- 172.21.15.72:0/3873087010 >> 172.21.15.72:0/3873087010 conn(0x7f287c06b830 msgr2=0x7f287c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 -- 172.21.15.72:0/3873087010 shutdown_connections 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.856+0000 7f2884684640 1 -- 172.21.15.72:0/3873087010 wait complete. 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.857+0000 7f2884684640 1 Processor -- start 2024-03-06T07:26:04.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.857+0000 7f2884684640 1 -- start start 2024-03-06T07:26:04.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2884684640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f287c06e2c0 0x7f287c1f4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2884684640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1e78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2884684640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c15f140 0x7f287c0b1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2884684640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0ad860 con 0x7f287c15bb50 2024-03-06T07:26:04.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2884684640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0724f0 con 0x7f287c06e2c0 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2884684640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0ad9e0 con 0x7f287c15f140 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1e78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1e78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34536/0 (socket says 172.21.15.72:34536) 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 -- 172.21.15.72:0/124611159 learned_addr learned my addr 172.21.15.72:0/124611159 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2882c21640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c15f140 0x7f287c0b1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2882420640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f287c06e2c0 0x7f287c1f4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 -- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c15f140 msgr2=0x7f287c0b1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c15f140 0x7f287c0b1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 -- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f287c06e2c0 msgr2=0x7f287c1f4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f287c06e2c0 0x7f287c1f4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2881c1f640 1 -- 172.21.15.72:0/124611159 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c1f4870 con 0x7f287c15bb50 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.858+0000 7f2882c21640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c15f140 0x7f287c0b1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.859+0000 7f2881c1f640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1e78d0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f287800d5f0 tx=0x7f287800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.859+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780125f0 con 0x7f287c15bb50 2024-03-06T07:26:04.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.859+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2878019070 con 0x7f287c15bb50 2024-03-06T07:26:04.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.859+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c1f4b50 con 0x7f287c15bb50 2024-03-06T07:26:04.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.859+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780129d0 con 0x7f287c15bb50 2024-03-06T07:26:04.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.859+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c1f5110 con 0x7f287c15bb50 2024-03-06T07:26:04.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.860+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287c06e0b0 con 0x7f287c15bb50 2024-03-06T07:26:04.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.860+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2878012b70 con 0x7f287c15bb50 2024-03-06T07:26:04.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.861+0000 7f2872ffd640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f286006b500 0x7f286006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:04.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.861+0000 7f2882420640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f286006b500 0x7f286006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:04.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.861+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f287808a360 con 0x7f287c15bb50 2024-03-06T07:26:04.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.861+0000 7f2882420640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f286006b500 0x7f286006d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f286c036000 tx=0x7f286c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:04.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:04.865+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2878059450 con 0x7f287c15bb50 2024-03-06T07:26:05.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.036+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f287c068ff0 con 0x7f287c15bb50 2024-03-06T07:26:05.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.731+0000 7f2872ffd640 1 -- 172.21.15.72:0/124611159 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v92) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2878058d50 con 0x7f287c15bb50 2024-03-06T07:26:05.733 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:26:05.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.733+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f286006b500 msgr2=0x7f286006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.733+0000 7f2884684640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f286006b500 0x7f286006d9c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f286c036000 tx=0x7f286c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.733+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 msgr2=0x7f287c1e78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.733+0000 7f2884684640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1e78d0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f287800d5f0 tx=0x7f287800dab0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 shutdown_connections 2024-03-06T07:26:05.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f286006b500 0x7f286006d9c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f287c15f140 0x7f287c0b1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287c15bb50 0x7f287c1e78d0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 --2- 172.21.15.72:0/124611159 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f287c06e2c0 0x7f287c1f4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 >> 172.21.15.72:0/124611159 conn(0x7f287c06b830 msgr2=0x7f287c1606b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 shutdown_connections 2024-03-06T07:26:05.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.734+0000 7f2884684640 1 -- 172.21.15.72:0/124611159 wait complete. 2024-03-06T07:26:05.762 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:26:05.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.865+0000 7f0eba2e1640 1 Processor -- start 2024-03-06T07:26:05.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.865+0000 7f0eba2e1640 1 -- start start 2024-03-06T07:26:05.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eba2e1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb415a490 0x7f0eb4072530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eba2e1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0eb406e270 0x7f0eb4071710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eba2e1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb406caa0 con 0x7f0eb40712d0 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eba2e1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb4071c50 con 0x7f0eb406e270 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eba2e1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb40b3340 con 0x7f0eb415a490 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb887e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0eb406e270 0x7f0eb4071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb3fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb415a490 0x7f0eb4072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb37fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0eb40712d0 0x7f0eb415a870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:53472/0 (socket says 172.21.15.72:53472) 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb37fe640 1 -- 172.21.15.72:0/4293518693 learned_addr learned my addr 172.21.15.72:0/4293518693 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb887e640 1 -- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb415a490 msgr2=0x7f0eb4072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb887e640 1 --2- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb415a490 0x7f0eb4072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.866+0000 7f0eb887e640 1 -- 172.21.15.72:0/4293518693 >> v1:172.21.15.146:6789/0 conn(0x7f0eb40712d0 legacy=0x7f0eb415a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.867+0000 7f0eb887e640 1 -- 172.21.15.72:0/4293518693 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb40b3620 con 0x7f0eb406e270 2024-03-06T07:26:05.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.867+0000 7f0eb3fff640 1 --2- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb415a490 0x7f0eb4072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.867+0000 7f0eb887e640 1 --2- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb406e270 0x7f0eb4071710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009d90 tx=0x7f0eac02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef249300d7cfc8ba server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.867+0000 7f0eb2ffd640 1 -- 172.21.15.72:0/4293518693 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac039070 con 0x7f0eb406e270 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.867+0000 7f0eb2ffd640 1 -- 172.21.15.72:0/4293518693 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eac02cce0 con 0x7f0eb406e270 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.867+0000 7f0eb2ffd640 1 -- 172.21.15.72:0/4293518693 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eac034660 con 0x7f0eb406e270 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 -- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb406e270 msgr2=0x7f0eb4071710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb406e270 0x7f0eb4071710 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009d90 tx=0x7f0eac02bf30 comp rx=0 tx=0).stop 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 -- 172.21.15.72:0/4293518693 shutdown_connections 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb406e270 0x7f0eb4071710 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/4293518693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb415a490 0x7f0eb4072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 -- 172.21.15.72:0/4293518693 >> 172.21.15.72:0/4293518693 conn(0x7f0eb406afb0 msgr2=0x7f0eb406b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 -- 172.21.15.72:0/4293518693 shutdown_connections 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.868+0000 7f0eba2e1640 1 -- 172.21.15.72:0/4293518693 wait complete. 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.869+0000 7f0eba2e1640 1 Processor -- start 2024-03-06T07:26:05.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.869+0000 7f0eba2e1640 1 -- start start 2024-03-06T07:26:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eba2e1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 0x7f0eb4148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eba2e1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb40712d0 0x7f0eb415c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eba2e1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb415a490 0x7f0eb415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eba2e1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb40b3a80 con 0x7f0eb406e270 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eba2e1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb40b3900 con 0x7f0eb415a490 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eba2e1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb4071fb0 con 0x7f0eb40712d0 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 0x7f0eb4148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 0x7f0eb4148fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34538/0 (socket says 172.21.15.72:34538) 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 -- 172.21.15.72:0/2681811476 learned_addr learned my addr 172.21.15.72:0/2681811476 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb887e640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb415a490 0x7f0eb415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb37fe640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb40712d0 0x7f0eb415c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 -- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb40712d0 msgr2=0x7f0eb415c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb40712d0 0x7f0eb415c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 -- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb415a490 msgr2=0x7f0eb415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb415a490 0x7f0eb415c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb3fff640 1 -- 172.21.15.72:0/2681811476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb415d900 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb887e640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb415a490 0x7f0eb415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.870+0000 7f0eb37fe640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb40712d0 0x7f0eb415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.871+0000 7f0eb3fff640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 0x7f0eb4148fe0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400e960 tx=0x7f0ea400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.871+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4017070 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.871+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea400ca00 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.871+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb4152a40 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.871+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400cdd0 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.871+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb4153000 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.872+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb4153710 con 0x7f0eb406e270 2024-03-06T07:26:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.872+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ea401a070 con 0x7f0eb406e270 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.873+0000 7f0eb0ff9640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ea006b4d0 0x7f0ea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.873+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0ea4089ff0 con 0x7f0eb406e270 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.873+0000 7f0eb37fe640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ea006b4d0 0x7f0ea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.873+0000 7f0eb37fe640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ea006b4d0 0x7f0ea006d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0eb415dd20 tx=0x7f0ea8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:05.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:05.877+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ea4058f60 con 0x7f0eb406e270 2024-03-06T07:26:06.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.086+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0eb404f330 con 0x7f0eb406e270 2024-03-06T07:26:06.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.745+0000 7f0eb0ff9640 1 -- 172.21.15.72:0/2681811476 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v44) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0ea4058860 con 0x7f0eb406e270 2024-03-06T07:26:06.747 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 14 and data pool 15 2024-03-06T07:26:06.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ea006b4d0 msgr2=0x7f0ea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ea006b4d0 0x7f0ea006d990 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f0eb415dd20 tx=0x7f0ea8009040 comp rx=0 tx=0).stop 2024-03-06T07:26:06.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 msgr2=0x7f0eb4148fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 0x7f0eb4148fe0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f0ea400e960 tx=0x7f0ea400ee20 comp rx=0 tx=0).stop 2024-03-06T07:26:06.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 shutdown_connections 2024-03-06T07:26:06.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ea006b4d0 0x7f0ea006d990 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eb415a490 0x7f0eb415c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb40712d0 0x7f0eb415c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 --2- 172.21.15.72:0/2681811476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eb406e270 0x7f0eb4148fe0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.751+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 >> 172.21.15.72:0/2681811476 conn(0x7f0eb406afb0 msgr2=0x7f0eb4156430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.755+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 shutdown_connections 2024-03-06T07:26:06.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.757+0000 7f0eba2e1640 1 -- 172.21.15.72:0/2681811476 wait complete. 2024-03-06T07:26:06.774 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.873+0000 7f0c3e5ee640 1 Processor -- start 2024-03-06T07:26:06.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.873+0000 7f0c3e5ee640 1 -- start start 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3e5ee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c380b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3e5ee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c3815bf30 0x7f0c380b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3e5ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3815f580 con 0x7f0c3815e850 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3e5ee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c380cf3f0 con 0x7f0c3815bf30 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3e5ee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3815b210 con 0x7f0c3813dc10 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c377fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c3815bf30 0x7f0c380b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c37fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c380b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3cb8b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0c3815e850 0x7f0c3815eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:53486/0 (socket says 172.21.15.72:53486) 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.874+0000 7f0c3cb8b640 1 -- 172.21.15.72:0/2853377763 learned_addr learned my addr 172.21.15.72:0/2853377763 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:06.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.875+0000 7f0c377fe640 1 -- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 msgr2=0x7f0c380b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.875+0000 7f0c377fe640 1 --2- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c380b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.875+0000 7f0c377fe640 1 -- 172.21.15.72:0/2853377763 >> v1:172.21.15.146:6789/0 conn(0x7f0c3815e850 legacy=0x7f0c3815eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.875+0000 7f0c36ffd640 1 -- 172.21.15.72:0/2853377763 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707914386 0 0) 0x7f0c3815f580 con 0x7f0c3815e850 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.875+0000 7f0c377fe640 1 -- 172.21.15.72:0/2853377763 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3815b450 con 0x7f0c3815bf30 2024-03-06T07:26:06.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.875+0000 7f0c37fff640 1 --2- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c380b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c377fe640 1 --2- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815bf30 0x7f0c380b2e90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097a0 tx=0x7f0c2c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=737ed597c9f844a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c36ffd640 1 -- 172.21.15.72:0/2853377763 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0315a0 con 0x7f0c3815bf30 2024-03-06T07:26:06.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c36ffd640 1 -- 172.21.15.72:0/2853377763 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c037070 con 0x7f0c3815bf30 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c36ffd640 1 -- 172.21.15.72:0/2853377763 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c031960 con 0x7f0c3815bf30 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815bf30 msgr2=0x7f0c380b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815bf30 0x7f0c380b2e90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097a0 tx=0x7f0c2c02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2853377763 shutdown_connections 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815bf30 0x7f0c380b2e90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2853377763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c380b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2853377763 >> 172.21.15.72:0/2853377763 conn(0x7f0c380cf8a0 msgr2=0x7f0c380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2853377763 shutdown_connections 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.876+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2853377763 wait complete. 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.877+0000 7f0c3e5ee640 1 Processor -- start 2024-03-06T07:26:06.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 -- start start 2024-03-06T07:26:06.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c38151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 0x7f0c3806de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815e850 0x7f0c3806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c380708d0 con 0x7f0c3815bf30 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3813dff0 con 0x7f0c3815e850 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.878+0000 7f0c3e5ee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c38070a50 con 0x7f0c3813dc10 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 0x7f0c3806de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 0x7f0c3806de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34542/0 (socket says 172.21.15.72:34542) 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 -- 172.21.15.72:0/2957694117 learned_addr learned my addr 172.21.15.72:0/2957694117 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c37fff640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c38151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c3cb8b640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815e850 0x7f0c3806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 -- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 msgr2=0x7f0c38151a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c38151a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 -- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815e850 msgr2=0x7f0c3806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815e850 0x7f0c3806e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 -- 172.21.15.72:0/2957694117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c38154c90 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c377fe640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 0x7f0c3806de00 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097a0 tx=0x7f0c2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c004230 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c0043d0 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c38154f70 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c003850 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.879+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c381e4600 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.880+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3804f330 con 0x7f0c3815bf30 2024-03-06T07:26:06.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.881+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c2c0039f0 con 0x7f0c3815bf30 2024-03-06T07:26:06.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.881+0000 7f0c34ff9640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c2406b4d0 0x7f0c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:06.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.881+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c2c02f030 con 0x7f0c3815bf30 2024-03-06T07:26:06.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.881+0000 7f0c37fff640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c2406b4d0 0x7f0c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:06.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.881+0000 7f0c37fff640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c2406b4d0 0x7f0c2406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009560 tx=0x7f0c28009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:06.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:06.885+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c2c041070 con 0x7f0c3815bf30 2024-03-06T07:26:07.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.058+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c3806eb00 con 0x7f0c3815bf30 2024-03-06T07:26:07.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.059+0000 7f0c34ff9640 1 -- 172.21.15.72:0/2957694117 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16633 (secure 0 0 0) 0x7f0c2c07a820 con 0x7f0c3815bf30 2024-03-06T07:26:07.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:07.061 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:06.087866+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:07.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c2406b4d0 msgr2=0x7f0c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c2406b4d0 0x7f0c2406d990 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009560 tx=0x7f0c28009190 comp rx=0 tx=0).stop 2024-03-06T07:26:07.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 msgr2=0x7f0c3806de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 0x7f0c3806de00 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097a0 tx=0x7f0c2c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 shutdown_connections 2024-03-06T07:26:07.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0c2406b4d0 0x7f0c2406d990 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c3815e850 0x7f0c3806e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c3815bf30 0x7f0c3806de00 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 --2- 172.21.15.72:0/2957694117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c3813dc10 0x7f0c38151a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 >> 172.21.15.72:0/2957694117 conn(0x7f0c380cf8a0 msgr2=0x7f0c380cde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 shutdown_connections 2024-03-06T07:26:07.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.061+0000 7f0c3e5ee640 1 -- 172.21.15.72:0/2957694117 wait complete. 2024-03-06T07:26:07.074 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.172+0000 7f29b01dd640 1 Processor -- start 2024-03-06T07:26:07.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.172+0000 7f29b01dd640 1 -- start start 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29b01dd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f29a80d22b0 0x7f29a80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29b01dd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a80b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29b01dd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a80b3430 con 0x7f29a815dfe0 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29b01dd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a80d2c50 con 0x7f29a80d22b0 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29b01dd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80b31e0 con 0x7f29a80b41f0 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29ad778640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f29a80d22b0 0x7f29a80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29ad778640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f29a80d22b0 0x7f29a80d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34558/0 (socket says 172.21.15.72:34558) 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29ad778640 1 -- 172.21.15.72:0/2036396507 learned_addr learned my addr 172.21.15.72:0/2036396507 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.173+0000 7f29ae77a640 1 --2- 172.21.15.72:0/2036396507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a80b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.174+0000 7f29ad778640 1 -- 172.21.15.72:0/2036396507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 msgr2=0x7f29a80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.174+0000 7f29ad778640 1 --2- 172.21.15.72:0/2036396507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a80b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.174+0000 7f29ad778640 1 -- 172.21.15.72:0/2036396507 >> v1:172.21.15.146:6789/0 conn(0x7f29a815dfe0 legacy=0x7f29a80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.174+0000 7f29ad778640 1 -- 172.21.15.72:0/2036396507 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a80b3710 con 0x7f29a80d22b0 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.174+0000 7f29ad778640 1 --2- 172.21.15.72:0/2036396507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a80d22b0 0x7f29a80d2710 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f2998009870 tx=0x7f299802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2353dd4c207c8381 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29acf77640 1 -- 172.21.15.72:0/2036396507 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2998034070 con 0x7f29a80d22b0 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29acf77640 1 -- 172.21.15.72:0/2036396507 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2998030840 con 0x7f29a80d22b0 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29acf77640 1 -- 172.21.15.72:0/2036396507 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2998030c10 con 0x7f29a80d22b0 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 -- 172.21.15.72:0/2036396507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a80d22b0 msgr2=0x7f29a80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 --2- 172.21.15.72:0/2036396507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a80d22b0 0x7f29a80d2710 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f2998009870 tx=0x7f299802b980 comp rx=0 tx=0).stop 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 -- 172.21.15.72:0/2036396507 shutdown_connections 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 --2- 172.21.15.72:0/2036396507 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a80b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 --2- 172.21.15.72:0/2036396507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a80d22b0 0x7f29a80d2710 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 -- 172.21.15.72:0/2036396507 >> 172.21.15.72:0/2036396507 conn(0x7f29a80d3cd0 msgr2=0x7f29a80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 -- 172.21.15.72:0/2036396507 shutdown_connections 2024-03-06T07:26:07.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.175+0000 7f29b01dd640 1 -- 172.21.15.72:0/2036396507 wait complete. 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.176+0000 7f29b01dd640 1 Processor -- start 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.176+0000 7f29b01dd640 1 -- start start 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29b01dd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a81f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29b01dd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a80d22b0 0x7f29a815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29b01dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 0x7f29a80cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29b01dd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80cfa00 con 0x7f29a815dfe0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29b01dd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80b3d60 con 0x7f29a80d22b0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29b01dd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80d1c70 con 0x7f29a80b41f0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29ae77a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 0x7f29a80cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29ae77a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 0x7f29a80cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34564/0 (socket says 172.21.15.72:34564) 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29ae77a640 1 -- 172.21.15.72:0/243906247 learned_addr learned my addr 172.21.15.72:0/243906247 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:07.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29ad778640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a80d22b0 0x7f29a815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.177+0000 7f29adf79640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a81f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29ae77a640 1 -- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 msgr2=0x7f29a81f4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29ae77a640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a81f4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29ae77a640 1 -- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a80d22b0 msgr2=0x7f29a815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29ae77a640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a80d22b0 0x7f29a815edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29ae77a640 1 -- 172.21.15.72:0/243906247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a815f820 con 0x7f29a815dfe0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29adf79640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a81f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29ae77a640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 0x7f29a80cd220 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009dd0 tx=0x7f29a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0018020 con 0x7f29a815dfe0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a0014480 con 0x7f29a815dfe0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a80caff0 con 0x7f29a815dfe0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a0014870 con 0x7f29a815dfe0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.178+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a80cb5b0 con 0x7f29a815dfe0 2024-03-06T07:26:07.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.179+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a80cb7b0 con 0x7f29a815dfe0 2024-03-06T07:26:07.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.179+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f29a0005d10 con 0x7f29a815dfe0 2024-03-06T07:26:07.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.180+0000 7f299e7fc640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f298c06b500 0x7f298c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.180+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29a008ad10 con 0x7f29a815dfe0 2024-03-06T07:26:07.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.180+0000 7f29adf79640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f298c06b500 0x7f298c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.180+0000 7f29adf79640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f298c06b500 0x7f298c06d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f29a80b2910 tx=0x7f29a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.184+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f29a80cb7b0 con 0x7f29a815dfe0 2024-03-06T07:26:07.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.356+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29a804f330 con 0x7f29a815dfe0 2024-03-06T07:26:07.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.357+0000 7f299e7fc640 1 -- 172.21.15.72:0/243906247 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16633 (secure 0 0 0) 0x7f29a0090020 con 0x7f29a815dfe0 2024-03-06T07:26:07.360 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:07.360 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":93,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:06.087866+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f298c06b500 msgr2=0x7f298c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f298c06b500 0x7f298c06d9c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f29a80b2910 tx=0x7f29a4009040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 msgr2=0x7f29a80cd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 0x7f29a80cd220 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009dd0 tx=0x7f29a000c490 comp rx=0 tx=0).stop 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 shutdown_connections 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f298c06b500 0x7f298c06d9c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f29a815dfe0 0x7f29a80cd220 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29a80d22b0 0x7f29a815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 --2- 172.21.15.72:0/243906247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29a80b41f0 0x7f29a81f4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 >> 172.21.15.72:0/243906247 conn(0x7f29a80d3cd0 msgr2=0x7f29a80cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 shutdown_connections 2024-03-06T07:26:07.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.359+0000 7f29b01dd640 1 -- 172.21.15.72:0/243906247 wait complete. 2024-03-06T07:26:07.373 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:26:07.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.472+0000 7f30946a4640 1 Processor -- start 2024-03-06T07:26:07.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.472+0000 7f30946a4640 1 -- start start 2024-03-06T07:26:07.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f30946a4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f308c1586e0 0x7f308c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f30946a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f308c0d01a0 0x7f308c0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f30946a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308c158ac0 con 0x7f308c0aef80 2024-03-06T07:26:07.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f30946a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c15b2a0 con 0x7f308c0d01a0 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f30946a4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f308c15b540 con 0x7f308c1586e0 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f308c0d01a0 0x7f308c0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f308c0d01a0 0x7f308c0b33a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34566/0 (socket says 172.21.15.72:34566) 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 -- 172.21.15.72:0/1517621263 learned_addr learned my addr 172.21.15.72:0/1517621263 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 -- 172.21.15.72:0/1517621263 >> v2:172.21.15.146:3300/0 conn(0x7f308c1586e0 msgr2=0x7f308c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092440640 1 --2- 172.21.15.72:0/1517621263 >> v2:172.21.15.146:3300/0 conn(0x7f308c1586e0 0x7f308c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 --2- 172.21.15.72:0/1517621263 >> v2:172.21.15.146:3300/0 conn(0x7f308c1586e0 0x7f308c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 -- 172.21.15.72:0/1517621263 >> v1:172.21.15.146:6789/0 conn(0x7f308c0aef80 legacy=0x7f308c0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092c41640 1 -- 172.21.15.72:0/1517621263 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c15b820 con 0x7f308c0d01a0 2024-03-06T07:26:07.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.473+0000 7f3092440640 1 --2- 172.21.15.72:0/1517621263 >> v2:172.21.15.146:3300/0 conn(0x7f308c1586e0 0x7f308c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.474+0000 7f3092c41640 1 --2- 172.21.15.72:0/1517621263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c0b33a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f308c070d10 tx=0x7f308402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df2926d090de40 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.474+0000 7f309143e640 1 -- 172.21.15.72:0/1517621263 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f308402c9f0 con 0x7f308c0d01a0 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.474+0000 7f309143e640 1 -- 172.21.15.72:0/1517621263 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f308402cb90 con 0x7f308c0d01a0 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.474+0000 7f309143e640 1 -- 172.21.15.72:0/1517621263 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30840343f0 con 0x7f308c0d01a0 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 -- 172.21.15.72:0/1517621263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 msgr2=0x7f308c0b33a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 --2- 172.21.15.72:0/1517621263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c0b33a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f308c070d10 tx=0x7f308402be50 comp rx=0 tx=0).stop 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 -- 172.21.15.72:0/1517621263 shutdown_connections 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 --2- 172.21.15.72:0/1517621263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c0b33a0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 --2- 172.21.15.72:0/1517621263 >> v2:172.21.15.146:3300/0 conn(0x7f308c1586e0 0x7f308c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 -- 172.21.15.72:0/1517621263 >> 172.21.15.72:0/1517621263 conn(0x7f308c06b190 msgr2=0x7f308c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 -- 172.21.15.72:0/1517621263 shutdown_connections 2024-03-06T07:26:07.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.475+0000 7f30946a4640 1 -- 172.21.15.72:0/1517621263 wait complete. 2024-03-06T07:26:07.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.476+0000 7f30946a4640 1 Processor -- start 2024-03-06T07:26:07.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.476+0000 7f30946a4640 1 -- start start 2024-03-06T07:26:07.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f308c0aef80 0x7f308c151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f308c1586e0 0x7f308c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c071350 con 0x7f308c0d01a0 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308c15bde0 con 0x7f308c0aef80 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f308c0714d0 con 0x7f308c1586e0 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34578/0 (socket says 172.21.15.72:34578) 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 -- 172.21.15.72:0/2287217042 learned_addr learned my addr 172.21.15.72:0/2287217042 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3092440640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f308c0aef80 0x7f308c151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 -- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f308c1586e0 msgr2=0x7f308c0ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f308c1586e0 0x7f308c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 -- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f308c0aef80 msgr2=0x7f308c151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f308c0aef80 0x7f308c151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 -- 172.21.15.72:0/2287217042 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308c153150 con 0x7f308c0d01a0 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3091c3f640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c15a4e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f307c00c400 tx=0x7f307c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c011790 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c017070 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c011bb0 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308c153430 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.477+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308c1539f0 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.478+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308c0681e0 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.479+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f307c01c460 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.479+0000 7f3082ffd640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f307006b4d0 0x7f307006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.479+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f307c089fc0 con 0x7f308c0d01a0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.479+0000 7f3092440640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f307006b4d0 0x7f307006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.480+0000 7f3092440640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f307006b4d0 0x7f307006d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f308c152410 tx=0x7f3088008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.483+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f308c0681e0 con 0x7f308c0d01a0 2024-03-06T07:26:07.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.693+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f308c0cfa60 con 0x7f308c0d01a0 2024-03-06T07:26:07.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.754+0000 7f3082ffd640 1 -- 172.21.15.72:0/2287217042 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7f308c0cfa60 con 0x7f308c0d01a0 2024-03-06T07:26:07.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f307006b4d0 msgr2=0x7f307006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f307006b4d0 0x7f307006d990 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f308c152410 tx=0x7f3088008040 comp rx=0 tx=0).stop 2024-03-06T07:26:07.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 msgr2=0x7f308c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c15a4e0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f307c00c400 tx=0x7f307c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 shutdown_connections 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f307006b4d0 0x7f307006d990 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f308c1586e0 0x7f308c0ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f308c0d01a0 0x7f308c15a4e0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 --2- 172.21.15.72:0/2287217042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f308c0aef80 0x7f308c151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 >> 172.21.15.72:0/2287217042 conn(0x7f308c06b190 msgr2=0x7f308c156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 shutdown_connections 2024-03-06T07:26:07.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.756+0000 7f30946a4640 1 -- 172.21.15.72:0/2287217042 wait complete. 2024-03-06T07:26:07.769 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:26:07.769 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:26:07.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.869+0000 7f545359e640 1 Processor -- start 2024-03-06T07:26:07.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.869+0000 7f545359e640 1 -- start start 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f545359e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f545359e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f54540d2830 0x7f54540b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f545359e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54540d0ae0 con 0x7f54540d19d0 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f545359e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54540b3570 con 0x7f54540d2830 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f545359e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540d0dc0 con 0x7f545415e010 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f5452d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f54540d2830 0x7f54540b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f5451d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f54540d19d0 0x7f54540d2120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42542/0 (socket says 172.21.15.72:42542) 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f5451d9b640 1 -- 172.21.15.72:0/1851297295 learned_addr learned my addr 172.21.15.72:0/1851297295 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.870+0000 7f545259c640 1 --2- 172.21.15.72:0/1851297295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f5452d9d640 1 -- 172.21.15.72:0/1851297295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 msgr2=0x7f54540d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f5452d9d640 1 --2- 172.21.15.72:0/1851297295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f5452d9d640 1 -- 172.21.15.72:0/1851297295 >> v1:172.21.15.72:6789/0 conn(0x7f54540d19d0 legacy=0x7f54540d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f5452d9d640 1 -- 172.21.15.72:0/1851297295 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540b2f20 con 0x7f54540d2830 2024-03-06T07:26:07.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f545159a640 1 -- 172.21.15.72:0/1851297295 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1684562736 0 0) 0x7f54540d0ae0 con 0x7f54540d19d0 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f5452d9d640 1 --2- 172.21.15.72:0/1851297295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b3a00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f544c0097a0 tx=0x7f544c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de19b9935d699372 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.871+0000 7f545159a640 1 -- 172.21.15.72:0/1851297295 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c034070 con 0x7f54540d2830 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545159a640 1 -- 172.21.15.72:0/1851297295 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c030610 con 0x7f54540d2830 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545159a640 1 -- 172.21.15.72:0/1851297295 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c0309e0 con 0x7f54540d2830 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 -- 172.21.15.72:0/1851297295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 msgr2=0x7f54540b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 --2- 172.21.15.72:0/1851297295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b3a00 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f544c0097a0 tx=0x7f544c02b960 comp rx=0 tx=0).stop 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 -- 172.21.15.72:0/1851297295 shutdown_connections 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 --2- 172.21.15.72:0/1851297295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b3a00 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 --2- 172.21.15.72:0/1851297295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 -- 172.21.15.72:0/1851297295 >> 172.21.15.72:0/1851297295 conn(0x7f545409ce10 msgr2=0x7f54540d3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 -- 172.21.15.72:0/1851297295 shutdown_connections 2024-03-06T07:26:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.872+0000 7f545359e640 1 -- 172.21.15.72:0/1851297295 wait complete. 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.873+0000 7f545359e640 1 Processor -- start 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.873+0000 7f545359e640 1 -- start start 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540d19d0 0x7f545415a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545359e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545359e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540d0830 con 0x7f54540d2830 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545359e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54540d06b0 con 0x7f54540d19d0 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545359e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54540b3f40 con 0x7f545415e010 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f5451d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f5451d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34586/0 (socket says 172.21.15.72:34586) 2024-03-06T07:26:07.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f5451d9b640 1 -- 172.21.15.72:0/3866442328 learned_addr learned my addr 172.21.15.72:0/3866442328 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f5452d9d640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.874+0000 7f545259c640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540d19d0 0x7f545415a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5451d9b640 1 -- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 msgr2=0x7f54540cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5451d9b640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5451d9b640 1 -- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540d19d0 msgr2=0x7f545415a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5451d9b640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540d19d0 0x7f545415a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5451d9b640 1 -- 172.21.15.72:0/3866442328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54540b69b0 con 0x7f54540d2830 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5451d9b640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b9030 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f544800c4e0 tx=0x7f544800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448018070 con 0x7f54540d2830 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5448007500 con 0x7f54540d2830 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5448004a20 con 0x7f54540d2830 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54540b6c90 con 0x7f54540d2830 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.875+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54540b7250 con 0x7f54540d2830 2024-03-06T07:26:07.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.876+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5454068f10 con 0x7f54540d2830 2024-03-06T07:26:07.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.876+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f544801b070 con 0x7f54540d2830 2024-03-06T07:26:07.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.877+0000 7f5442ffd640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f543406b4d0 0x7f543406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:07.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.877+0000 7f545259c640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f543406b4d0 0x7f543406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:07.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.877+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54480894d0 con 0x7f54540d2830 2024-03-06T07:26:07.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.877+0000 7f545259c640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f543406b4d0 0x7f543406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f54540b2a20 tx=0x7f544400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:07.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:07.881+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5448057dc0 con 0x7f54540d2830 2024-03-06T07:26:08.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.091+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f54540008d0 con 0x7f54540d2830 2024-03-06T07:26:08.729 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-03-06T07:26:08.727+0000 7f2554340640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:08.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.760+0000 7f5442ffd640 1 -- 172.21.15.72:0/3866442328 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5448087e20 con 0x7f54540d2830 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f543406b4d0 msgr2=0x7f543406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f543406b4d0 0x7f543406d990 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f54540b2a20 tx=0x7f544400b040 comp rx=0 tx=0).stop 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 msgr2=0x7f54540b9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b9030 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f544800c4e0 tx=0x7f544800c9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 shutdown_connections 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f543406b4d0 0x7f543406d990 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f545415e010 0x7f54540cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54540d2830 0x7f54540b9030 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 --2- 172.21.15.72:0/3866442328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f54540d19d0 0x7f545415a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 >> 172.21.15.72:0/3866442328 conn(0x7f545409ce10 msgr2=0x7f545406c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 shutdown_connections 2024-03-06T07:26:08.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.763+0000 7f545359e640 1 -- 172.21.15.72:0/3866442328 wait complete. 2024-03-06T07:26:08.777 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:26:08.777 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:08.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 Processor -- start 2024-03-06T07:26:08.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 -- start start 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f114c1515e0 0x7f114c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f114c15b430 0x7f114c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c15b8e0 con 0x7f114c0cd7a0 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c070a70 con 0x7f114c15b430 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.877+0000 7f1152a88640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c070d50 con 0x7f114c1515e0 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f114bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f114c15b430 0x7f114c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f1151025640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f114c0cd7a0 0x7f114c0cdb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42556/0 (socket says 172.21.15.72:42556) 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f1151025640 1 -- 172.21.15.72:0/4000446010 learned_addr learned my addr 172.21.15.72:0/4000446010 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f1150824640 1 --2- 172.21.15.72:0/4000446010 >> v2:172.21.15.146:3300/0 conn(0x7f114c1515e0 0x7f114c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f114bfff640 1 -- 172.21.15.72:0/4000446010 >> v2:172.21.15.146:3300/0 conn(0x7f114c1515e0 msgr2=0x7f114c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f114bfff640 1 --2- 172.21.15.72:0/4000446010 >> v2:172.21.15.146:3300/0 conn(0x7f114c1515e0 0x7f114c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f114bfff640 1 -- 172.21.15.72:0/4000446010 >> v1:172.21.15.72:6789/0 conn(0x7f114c0cd7a0 legacy=0x7f114c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f114bfff640 1 -- 172.21.15.72:0/4000446010 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c071030 con 0x7f114c15b430 2024-03-06T07:26:08.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.878+0000 7f114b7fe640 1 -- 172.21.15.72:0/4000446010 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570792423 0 0) 0x7f114c15b8e0 con 0x7f114c0cd7a0 2024-03-06T07:26:08.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f114bfff640 1 --2- 172.21.15.72:0/4000446010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c15b430 0x7f114c071830 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f114c06eb50 tx=0x7f113802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42d317cf566af43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f114b7fe640 1 -- 172.21.15.72:0/4000446010 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138035020 con 0x7f114c15b430 2024-03-06T07:26:08.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f114b7fe640 1 -- 172.21.15.72:0/4000446010 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1138031830 con 0x7f114c15b430 2024-03-06T07:26:08.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f114b7fe640 1 -- 172.21.15.72:0/4000446010 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138031c00 con 0x7f114c15b430 2024-03-06T07:26:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 -- 172.21.15.72:0/4000446010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c15b430 msgr2=0x7f114c071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 --2- 172.21.15.72:0/4000446010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c15b430 0x7f114c071830 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f114c06eb50 tx=0x7f113802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 -- 172.21.15.72:0/4000446010 shutdown_connections 2024-03-06T07:26:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 --2- 172.21.15.72:0/4000446010 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c15b430 0x7f114c071830 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 --2- 172.21.15.72:0/4000446010 >> v2:172.21.15.146:3300/0 conn(0x7f114c1515e0 0x7f114c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 -- 172.21.15.72:0/4000446010 >> 172.21.15.72:0/4000446010 conn(0x7f114c0cf8a0 msgr2=0x7f114c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 -- 172.21.15.72:0/4000446010 shutdown_connections 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.879+0000 7f1152a88640 1 -- 172.21.15.72:0/4000446010 wait complete. 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.880+0000 7f1152a88640 1 Processor -- start 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.880+0000 7f1152a88640 1 -- start start 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1152a88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f114c0cd7a0 0x7f114c13ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1152a88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 0x7f114c13f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1152a88640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f114c15b430 0x7f114c13f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1152a88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c071ef0 con 0x7f114c1515e0 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1152a88640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c071d70 con 0x7f114c15b430 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1152a88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0b3f90 con 0x7f114c0cd7a0 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f114bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 0x7f114c13f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f114bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 0x7f114c13f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34606/0 (socket says 172.21.15.72:34606) 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f114bfff640 1 -- 172.21.15.72:0/92490295 learned_addr learned my addr 172.21.15.72:0/92490295 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1150824640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f114c0cd7a0 0x7f114c13ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.881+0000 7f1151025640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f114c15b430 0x7f114c13f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f114bfff640 1 -- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f114c0cd7a0 msgr2=0x7f114c13ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f114bfff640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f114c0cd7a0 0x7f114c13ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f114bfff640 1 -- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f114c15b430 msgr2=0x7f114c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f114bfff640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f114c15b430 0x7f114c13f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f114bfff640 1 -- 172.21.15.72:0/92490295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c13ebb0 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f114bfff640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 0x7f114c13f270 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1138009890 tx=0x7f11380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138040070 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113803e070 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c13ee90 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1138035040 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.882+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c140c50 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.883+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c068840 con 0x7f114c1515e0 2024-03-06T07:26:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.883+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1138038020 con 0x7f114c1515e0 2024-03-06T07:26:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.884+0000 7f11497fa640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.884+0000 7f1150824640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.884+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11380ab940 con 0x7f114c1515e0 2024-03-06T07:26:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.884+0000 7f1150824640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1140009b40 tx=0x7f1140009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:08.888+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1138031800 con 0x7f114c1515e0 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.094+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f114c04f330 con 0x7f114c1515e0 2024-03-06T07:26:09.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.094+0000 7f11497fa640 1 -- 172.21.15.72:0/92490295 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f113807a200 con 0x7f114c1515e0 2024-03-06T07:26:09.097 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:09.097 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":46,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1283242178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1283242178},{"type":"v1","addr":"172.21.15.146:6834","nonce":1283242178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3967731577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3967731577},{"type":"v1","addr":"172.21.15.72:6836","nonce":3967731577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7389,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/646486087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":646486087},{"type":"v1","addr":"172.21.15.72:6837","nonce":646486087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:08.092689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7404":{"gid":7404,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/209452807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":209452807},{"type":"v1","addr":"172.21.15.146:6835","nonce":209452807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7404,"qdb_cluster":[7404]},"id":7}]} 2024-03-06T07:26:09.097 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 46 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f113006b500 msgr2=0x7f113006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f113006b500 0x7f113006d9c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f1140009b40 tx=0x7f1140009190 comp rx=0 tx=0).stop 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 msgr2=0x7f114c13f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 0x7f114c13f270 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1138009890 tx=0x7f11380046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 shutdown_connections 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f113006b500 0x7f113006d9c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f114c15b430 0x7f114c13f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f114c1515e0 0x7f114c13f270 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 --2- 172.21.15.72:0/92490295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f114c0cd7a0 0x7f114c13ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.096+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 >> 172.21.15.72:0/92490295 conn(0x7f114c0cf8a0 msgr2=0x7f114c150210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.097+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 shutdown_connections 2024-03-06T07:26:09.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.097+0000 7f1152a88640 1 -- 172.21.15.72:0/92490295 wait complete. 2024-03-06T07:26:09.109 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:09.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.207+0000 7f01b5fd0640 1 Processor -- start 2024-03-06T07:26:09.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.207+0000 7f01b5fd0640 1 -- start start 2024-03-06T07:26:09.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01b5fd0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cce40 0x7f01b015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01b5fd0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01b00cd7a0 0x7f01b00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01b5fd0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0071f40 con 0x7f01b01515e0 2024-03-06T07:26:09.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01b5fd0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01b00cd2c0 con 0x7f01b00cd7a0 2024-03-06T07:26:09.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01b5fd0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0070a70 con 0x7f01b00cce40 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01b00cd7a0 0x7f01b00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01b00cd7a0 0x7f01b00cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34618/0 (socket says 172.21.15.72:34618) 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 -- 172.21.15.72:0/728135152 learned_addr learned my addr 172.21.15.72:0/728135152 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01aeffd640 1 --2- 172.21.15.72:0/728135152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cce40 0x7f01b015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 -- 172.21.15.72:0/728135152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cce40 msgr2=0x7f01b015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 --2- 172.21.15.72:0/728135152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cce40 0x7f01b015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 -- 172.21.15.72:0/728135152 >> v1:172.21.15.146:6789/0 conn(0x7f01b01515e0 legacy=0x7f01b0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01ae7fc640 1 -- 172.21.15.72:0/728135152 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291856164 0 0) 0x7f01b0071f40 con 0x7f01b01515e0 2024-03-06T07:26:09.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.208+0000 7f01affff640 1 -- 172.21.15.72:0/728135152 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0070d50 con 0x7f01b00cd7a0 2024-03-06T07:26:09.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.209+0000 7f01affff640 1 --2- 172.21.15.72:0/728135152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b00cd7a0 0x7f01b00cdbd0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009e50 tx=0x7f01a802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15837bc580582f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.209+0000 7f01ae7fc640 1 -- 172.21.15.72:0/728135152 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a802cb00 con 0x7f01b00cd7a0 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.209+0000 7f01ae7fc640 1 -- 172.21.15.72:0/728135152 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a802cca0 con 0x7f01b00cd7a0 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.209+0000 7f01ae7fc640 1 -- 172.21.15.72:0/728135152 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8034610 con 0x7f01b00cd7a0 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.209+0000 7f01b5fd0640 1 -- 172.21.15.72:0/728135152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b00cd7a0 msgr2=0x7f01b00cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.209+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/728135152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b00cd7a0 0x7f01b00cdbd0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009e50 tx=0x7f01a802bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.210+0000 7f01b5fd0640 1 -- 172.21.15.72:0/728135152 shutdown_connections 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.210+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/728135152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b00cd7a0 0x7f01b00cdbd0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.210+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/728135152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cce40 0x7f01b015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.210+0000 7f01b5fd0640 1 -- 172.21.15.72:0/728135152 >> 172.21.15.72:0/728135152 conn(0x7f01b00cf8a0 msgr2=0x7f01b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.210+0000 7f01b5fd0640 1 -- 172.21.15.72:0/728135152 shutdown_connections 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.210+0000 7f01b5fd0640 1 -- 172.21.15.72:0/728135152 wait complete. 2024-03-06T07:26:09.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.211+0000 7f01b5fd0640 1 Processor -- start 2024-03-06T07:26:09.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.211+0000 7f01b5fd0640 1 -- start start 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01b00cce40 0x7f01b0072410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cd7a0 0x7f01b0151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 0x7f01b0151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b00ce110 con 0x7f01b01515e0 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0071310 con 0x7f01b00cce40 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b00cc370 con 0x7f01b00cd7a0 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 0x7f01b0151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01aeffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cd7a0 0x7f01b0151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 0x7f01b0151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34620/0 (socket says 172.21.15.72:34620) 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 -- 172.21.15.72:0/656314610 learned_addr learned my addr 172.21.15.72:0/656314610 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01af7fe640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01b00cce40 0x7f01b0072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 -- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cd7a0 msgr2=0x7f01b0151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cd7a0 0x7f01b0151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 -- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01b00cce40 msgr2=0x7f01b0072410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01b00cce40 0x7f01b0072410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 -- 172.21.15.72:0/656314610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0147e70 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01affff640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 0x7f01b0151f60 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009e50 tx=0x7f01a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8043030 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a802ce50 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b0152870 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.212+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a8009510 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.213+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b0152e30 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.214+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b0068840 con 0x7f01b01515e0 2024-03-06T07:26:09.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.214+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f01a8046070 con 0x7f01b01515e0 2024-03-06T07:26:09.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.214+0000 7f0197fff640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f019c06b4d0 0x7f019c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.214+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01a80ac270 con 0x7f01b01515e0 2024-03-06T07:26:09.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.214+0000 7f01af7fe640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f019c06b4d0 0x7f019c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.215+0000 7f01af7fe640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f019c06b4d0 0x7f019c06d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f01a0000f60 tx=0x7f01a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.219+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f01a804d070 con 0x7f01b01515e0 2024-03-06T07:26:09.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.389+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01b00b4c30 con 0x7f01b01515e0 2024-03-06T07:26:09.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.390+0000 7f0197fff640 1 -- 172.21.15.72:0/656314610 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f01b0147e70 con 0x7f01b01515e0 2024-03-06T07:26:09.393 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:09.393 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":94,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:07.733459+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:09.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f019c06b4d0 msgr2=0x7f019c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f019c06b4d0 0x7f019c06d990 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f01a0000f60 tx=0x7f01a0009040 comp rx=0 tx=0).stop 2024-03-06T07:26:09.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 msgr2=0x7f01b0151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 0x7f01b0151f60 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f01a8009e50 tx=0x7f01a8004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 shutdown_connections 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f019c06b4d0 0x7f019c06d990 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01b01515e0 0x7f01b0151f60 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01b00cd7a0 0x7f01b0151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 --2- 172.21.15.72:0/656314610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01b00cce40 0x7f01b0072410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 >> 172.21.15.72:0/656314610 conn(0x7f01b00cf8a0 msgr2=0x7f01b01641b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 shutdown_connections 2024-03-06T07:26:09.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.392+0000 7f01b5fd0640 1 -- 172.21.15.72:0/656314610 wait complete. 2024-03-06T07:26:09.406 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:26:09.406 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:09.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.504+0000 7fba6b476640 1 Processor -- start 2024-03-06T07:26:09.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.504+0000 7fba6b476640 1 -- start start 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba6b476640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba641424f0 0x7fba640cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba6b476640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba640cd7a0 0x7fba6415cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba6b476640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba640cc980 con 0x7fba6406d330 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba6b476640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba640cd220 con 0x7fba640cd7a0 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba6b476640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640ccc60 con 0x7fba641424f0 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69a13640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fba6406d330 0x7fba64071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42566/0 (socket says 172.21.15.72:42566) 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69a13640 1 -- 172.21.15.72:0/23146124 learned_addr learned my addr 172.21.15.72:0/23146124 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69212640 1 --2- 172.21.15.72:0/23146124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba641424f0 0x7fba640cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba68a11640 1 --2- 172.21.15.72:0/23146124 >> v2:172.21.15.146:3300/0 conn(0x7fba640cd7a0 0x7fba6415cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba5bfff640 1 -- 172.21.15.72:0/23146124 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612347027 0 0) 0x7fba640cc980 con 0x7fba6406d330 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba5bfff640 1 -- 172.21.15.72:0/23146124 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50003520 con 0x7fba6406d330 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69212640 1 -- 172.21.15.72:0/23146124 >> v2:172.21.15.146:3300/0 conn(0x7fba640cd7a0 msgr2=0x7fba6415cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69212640 1 --2- 172.21.15.72:0/23146124 >> v2:172.21.15.146:3300/0 conn(0x7fba640cd7a0 0x7fba6415cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69212640 1 -- 172.21.15.72:0/23146124 >> v1:172.21.15.72:6789/0 conn(0x7fba6406d330 legacy=0x7fba64071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba69212640 1 -- 172.21.15.72:0/23146124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba640ccf40 con 0x7fba641424f0 2024-03-06T07:26:09.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.505+0000 7fba68a11640 1 --2- 172.21.15.72:0/23146124 >> v2:172.21.15.146:3300/0 conn(0x7fba640cd7a0 0x7fba6415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.506+0000 7fba69212640 1 --2- 172.21.15.72:0/23146124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba641424f0 0x7fba640cc2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fba60009d90 tx=0x7fba6002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a9fd0ad39fce0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.506+0000 7fba5bfff640 1 -- 172.21.15.72:0/23146124 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60039070 con 0x7fba641424f0 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.506+0000 7fba5bfff640 1 -- 172.21.15.72:0/23146124 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6002cc50 con 0x7fba641424f0 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.506+0000 7fba5bfff640 1 -- 172.21.15.72:0/23146124 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60034440 con 0x7fba641424f0 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 -- 172.21.15.72:0/23146124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba641424f0 msgr2=0x7fba640cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 --2- 172.21.15.72:0/23146124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba641424f0 0x7fba640cc2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fba60009d90 tx=0x7fba6002bea0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 -- 172.21.15.72:0/23146124 shutdown_connections 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 --2- 172.21.15.72:0/23146124 >> v2:172.21.15.146:3300/0 conn(0x7fba640cd7a0 0x7fba6415cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 --2- 172.21.15.72:0/23146124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba641424f0 0x7fba640cc2f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 -- 172.21.15.72:0/23146124 >> 172.21.15.72:0/23146124 conn(0x7fba640cf8a0 msgr2=0x7fba640cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 -- 172.21.15.72:0/23146124 shutdown_connections 2024-03-06T07:26:09.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.507+0000 7fba6b476640 1 -- 172.21.15.72:0/23146124 wait complete. 2024-03-06T07:26:09.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.508+0000 7fba6b476640 1 Processor -- start 2024-03-06T07:26:09.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.508+0000 7fba6b476640 1 -- start start 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba6b476640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6406d330 0x7fba641f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba6b476640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba640cd7a0 0x7fba6413d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba6b476640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 0x7fba641436a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba6b476640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406d940 con 0x7fba641424f0 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba6b476640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406d7c0 con 0x7fba6406d330 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba6b476640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6406dac0 con 0x7fba640cd7a0 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 0x7fba641436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 0x7fba641436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34628/0 (socket says 172.21.15.72:34628) 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 -- 172.21.15.72:0/738038778 learned_addr learned my addr 172.21.15.72:0/738038778 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:09.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69212640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6406d330 0x7fba641f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba68a11640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba640cd7a0 0x7fba6413d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 -- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba640cd7a0 msgr2=0x7fba6413d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba640cd7a0 0x7fba6413d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 -- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6406d330 msgr2=0x7fba641f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6406d330 0x7fba641f2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.509+0000 7fba69a13640 1 -- 172.21.15.72:0/738038778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64144040 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.510+0000 7fba69a13640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 0x7fba641436a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00c460 tx=0x7fba5c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.510+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c017070 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.510+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5c005ce0 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.510+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba64144320 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.510+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5c0072b0 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.510+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6415dff0 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.511+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6404f330 con 0x7fba641424f0 2024-03-06T07:26:09.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.511+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba5c01a070 con 0x7fba641424f0 2024-03-06T07:26:09.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.511+0000 7fba59ffb640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b4d0 0x7fba4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.512+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba5c089490 con 0x7fba641424f0 2024-03-06T07:26:09.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.512+0000 7fba69212640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b4d0 0x7fba4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.512+0000 7fba69212640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b4d0 0x7fba4806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fba60036000 tx=0x7fba60030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.516+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba5c057d50 con 0x7fba641424f0 2024-03-06T07:26:09.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.684+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fba640d0620 con 0x7fba4806b4d0 2024-03-06T07:26:09.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.690+0000 7fba59ffb640 1 -- 172.21.15.72:0/738038778 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252015 (secure 0 0 0) 0x7fba640d0620 con 0x7fba4806b4d0 2024-03-06T07:26:09.694 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:09.696 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b4d0 msgr2=0x7fba4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b4d0 0x7fba4806d990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fba60036000 tx=0x7fba60030000 comp rx=0 tx=0).stop 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 msgr2=0x7fba641436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 0x7fba641436a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00c460 tx=0x7fba5c00c920 comp rx=0 tx=0).stop 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 shutdown_connections 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b4d0 0x7fba4806d990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba641424f0 0x7fba641436a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba640cd7a0 0x7fba6413d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 --2- 172.21.15.72:0/738038778 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6406d330 0x7fba641f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 >> 172.21.15.72:0/738038778 conn(0x7fba640cf8a0 msgr2=0x7fba640cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 shutdown_connections 2024-03-06T07:26:09.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.693+0000 7fba6b476640 1 -- 172.21.15.72:0/738038778 wait complete. 2024-03-06T07:26:09.706 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-03-06T07:26:07.994087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259544,"kb_used_data":10084,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715176,"statfs":{"total":773094113280,"available":772828340224,"internally_reserved":0,"allocated":10326016,"data_stored":5053884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000597"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760795+0000","last_change":"2024-03-06T07:26:06.755635+0000","last_active":"2024-03-06T07:26:06.760795+0000","last_peered":"2024-03-06T07:26:06.760795+0000","last_clean":"2024-03-06T07:26:06.760795+0000","last_became_active":"2024-03-06T07:26:06.755367+0000","last_became_peered":"2024-03-06T07:26:06.755367+0000","last_unstale":"2024-03-06T07:26:06.760795+0000","last_undegraded":"2024-03-06T07:26:06.760795+0000","last_fullsized":"2024-03-06T07:26:06.760795+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.274410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744585+0000","last_change":"2024-03-06T07:26:06.742400+0000","last_active":"2024-03-06T07:26:06.744585+0000","last_peered":"2024-03-06T07:26:06.744585+0000","last_clean":"2024-03-06T07:26:06.744585+0000","last_became_active":"2024-03-06T07:26:06.742143+0000","last_became_peered":"2024-03-06T07:26:06.742143+0000","last_unstale":"2024-03-06T07:26:06.744585+0000","last_undegraded":"2024-03-06T07:26:06.744585+0000","last_fullsized":"2024-03-06T07:26:06.744585+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:47.608451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760821+0000","last_change":"2024-03-06T07:26:06.755309+0000","last_active":"2024-03-06T07:26:06.760821+0000","last_peered":"2024-03-06T07:26:06.760821+0000","last_clean":"2024-03-06T07:26:06.760821+0000","last_became_active":"2024-03-06T07:26:06.754779+0000","last_became_peered":"2024-03-06T07:26:06.754779+0000","last_unstale":"2024-03-06T07:26:06.760821+0000","last_undegraded":"2024-03-06T07:26:06.760821+0000","last_fullsized":"2024-03-06T07:26:06.760821+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:38.575174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758999+0000","last_change":"2024-03-06T07:26:06.747982+0000","last_active":"2024-03-06T07:26:06.758999+0000","last_peered":"2024-03-06T07:26:06.758999+0000","last_clean":"2024-03-06T07:26:06.758999+0000","last_became_active":"2024-03-06T07:26:06.747725+0000","last_became_peered":"2024-03-06T07:26:06.747725+0000","last_unstale":"2024-03-06T07:26:06.758999+0000","last_undegraded":"2024-03-06T07:26:06.758999+0000","last_fullsized":"2024-03-06T07:26:06.758999+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758412+0000","last_change":"2024-03-06T07:26:06.752903+0000","last_active":"2024-03-06T07:26:06.758412+0000","last_peered":"2024-03-06T07:26:06.758412+0000","last_clean":"2024-03-06T07:26:06.758412+0000","last_became_active":"2024-03-06T07:26:06.752570+0000","last_became_peered":"2024-03-06T07:26:06.752570+0000","last_unstale":"2024-03-06T07:26:06.758412+0000","last_undegraded":"2024-03-06T07:26:06.758412+0000","last_fullsized":"2024-03-06T07:26:06.758412+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:33.831911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.762811+0000","last_change":"2024-03-06T07:26:06.757721+0000","last_active":"2024-03-06T07:26:06.762811+0000","last_peered":"2024-03-06T07:26:06.762811+0000","last_clean":"2024-03-06T07:26:06.762811+0000","last_became_active":"2024-03-06T07:26:06.757474+0000","last_became_peered":"2024-03-06T07:26:06.757474+0000","last_unstale":"2024-03-06T07:26:06.762811+0000","last_undegraded":"2024-03-06T07:26:06.762811+0000","last_fullsized":"2024-03-06T07:26:06.762811+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:48.424079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.762842+0000","last_change":"2024-03-06T07:26:06.758597+0000","last_active":"2024-03-06T07:26:06.762842+0000","last_peered":"2024-03-06T07:26:06.762842+0000","last_clean":"2024-03-06T07:26:06.762842+0000","last_became_active":"2024-03-06T07:26:06.758350+0000","last_became_peered":"2024-03-06T07:26:06.758350+0000","last_unstale":"2024-03-06T07:26:06.762842+0000","last_undegraded":"2024-03-06T07:26:06.762842+0000","last_fullsized":"2024-03-06T07:26:06.762842+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:28.601267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.748888+0000","last_change":"2024-03-06T07:26:06.745213+0000","last_active":"2024-03-06T07:26:06.748888+0000","last_peered":"2024-03-06T07:26:06.748888+0000","last_clean":"2024-03-06T07:26:06.748888+0000","last_became_active":"2024-03-06T07:26:06.744964+0000","last_became_peered":"2024-03-06T07:26:06.744964+0000","last_unstale":"2024-03-06T07:26:06.748888+0000","last_undegraded":"2024-03-06T07:26:06.748888+0000","last_fullsized":"2024-03-06T07:26:06.748888+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.410173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751205+0000","last_change":"2024-03-06T07:26:06.748975+0000","last_active":"2024-03-06T07:26:06.751205+0000","last_peered":"2024-03-06T07:26:06.751205+0000","last_clean":"2024-03-06T07:26:06.751205+0000","last_became_active":"2024-03-06T07:26:06.748535+0000","last_became_peered":"2024-03-06T07:26:06.748535+0000","last_unstale":"2024-03-06T07:26:06.751205+0000","last_undegraded":"2024-03-06T07:26:06.751205+0000","last_fullsized":"2024-03-06T07:26:06.751205+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:45.599226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759968+0000","last_change":"2024-03-06T07:26:06.756179+0000","last_active":"2024-03-06T07:26:06.759968+0000","last_peered":"2024-03-06T07:26:06.759968+0000","last_clean":"2024-03-06T07:26:06.759968+0000","last_became_active":"2024-03-06T07:26:06.755652+0000","last_became_peered":"2024-03-06T07:26:06.755652+0000","last_unstale":"2024-03-06T07:26:06.759968+0000","last_undegraded":"2024-03-06T07:26:06.759968+0000","last_fullsized":"2024-03-06T07:26:06.759968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.563016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.756124+0000","last_change":"2024-03-06T07:26:06.753074+0000","last_active":"2024-03-06T07:26:06.756124+0000","last_peered":"2024-03-06T07:26:06.756124+0000","last_clean":"2024-03-06T07:26:06.756124+0000","last_became_active":"2024-03-06T07:26:06.752758+0000","last_became_peered":"2024-03-06T07:26:06.752758+0000","last_unstale":"2024-03-06T07:26:06.756124+0000","last_undegraded":"2024-03-06T07:26:06.756124+0000","last_fullsized":"2024-03-06T07:26:06.756124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:36.743823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766462+0000","last_change":"2024-03-06T07:26:06.755897+0000","last_active":"2024-03-06T07:26:06.766462+0000","last_peered":"2024-03-06T07:26:06.766462+0000","last_clean":"2024-03-06T07:26:06.766462+0000","last_became_active":"2024-03-06T07:26:06.755588+0000","last_became_peered":"2024-03-06T07:26:06.755588+0000","last_unstale":"2024-03-06T07:26:06.766462+0000","last_undegraded":"2024-03-06T07:26:06.766462+0000","last_fullsized":"2024-03-06T07:26:06.766462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759416+0000","last_change":"2024-03-06T07:26:06.750968+0000","last_active":"2024-03-06T07:26:06.759416+0000","last_peered":"2024-03-06T07:26:06.759416+0000","last_clean":"2024-03-06T07:26:06.759416+0000","last_became_active":"2024-03-06T07:26:06.750666+0000","last_became_peered":"2024-03-06T07:26:06.750666+0000","last_unstale":"2024-03-06T07:26:06.759416+0000","last_undegraded":"2024-03-06T07:26:06.759416+0000","last_fullsized":"2024-03-06T07:26:06.759416+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.563016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763262+0000","last_change":"2024-03-06T07:26:06.756612+0000","last_active":"2024-03-06T07:26:06.763262+0000","last_peered":"2024-03-06T07:26:06.763262+0000","last_clean":"2024-03-06T07:26:06.763262+0000","last_became_active":"2024-03-06T07:26:06.756359+0000","last_became_peered":"2024-03-06T07:26:06.756359+0000","last_unstale":"2024-03-06T07:26:06.763262+0000","last_undegraded":"2024-03-06T07:26:06.763262+0000","last_fullsized":"2024-03-06T07:26:06.763262+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:39.907630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760545+0000","last_change":"2024-03-06T07:26:06.753497+0000","last_active":"2024-03-06T07:26:06.760545+0000","last_peered":"2024-03-06T07:26:06.760545+0000","last_clean":"2024-03-06T07:26:06.760545+0000","last_became_active":"2024-03-06T07:26:06.753226+0000","last_became_peered":"2024-03-06T07:26:06.753226+0000","last_unstale":"2024-03-06T07:26:06.760545+0000","last_undegraded":"2024-03-06T07:26:06.760545+0000","last_fullsized":"2024-03-06T07:26:06.760545+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:00.479923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757934+0000","last_change":"2024-03-06T07:26:06.755415+0000","last_active":"2024-03-06T07:26:06.757934+0000","last_peered":"2024-03-06T07:26:06.757934+0000","last_clean":"2024-03-06T07:26:06.757934+0000","last_became_active":"2024-03-06T07:26:06.754809+0000","last_became_peered":"2024-03-06T07:26:06.754809+0000","last_unstale":"2024-03-06T07:26:06.757934+0000","last_undegraded":"2024-03-06T07:26:06.757934+0000","last_fullsized":"2024-03-06T07:26:06.757934+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.019146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744102+0000","last_change":"2024-03-06T07:26:06.741669+0000","last_active":"2024-03-06T07:26:06.744102+0000","last_peered":"2024-03-06T07:26:06.744102+0000","last_clean":"2024-03-06T07:26:06.744102+0000","last_became_active":"2024-03-06T07:26:06.741394+0000","last_became_peered":"2024-03-06T07:26:06.741394+0000","last_unstale":"2024-03-06T07:26:06.744102+0000","last_undegraded":"2024-03-06T07:26:06.744102+0000","last_fullsized":"2024-03-06T07:26:06.744102+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:23.552011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758936+0000","last_change":"2024-03-06T07:26:06.747177+0000","last_active":"2024-03-06T07:26:06.758936+0000","last_peered":"2024-03-06T07:26:06.758936+0000","last_clean":"2024-03-06T07:26:06.758936+0000","last_became_active":"2024-03-06T07:26:06.746884+0000","last_became_peered":"2024-03-06T07:26:06.746884+0000","last_unstale":"2024-03-06T07:26:06.758936+0000","last_undegraded":"2024-03-06T07:26:06.758936+0000","last_fullsized":"2024-03-06T07:26:06.758936+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:22.384294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763382+0000","last_change":"2024-03-06T07:26:06.757075+0000","last_active":"2024-03-06T07:26:06.763382+0000","last_peered":"2024-03-06T07:26:06.763382+0000","last_clean":"2024-03-06T07:26:06.763382+0000","last_became_active":"2024-03-06T07:26:06.756773+0000","last_became_peered":"2024-03-06T07:26:06.756773+0000","last_unstale":"2024-03-06T07:26:06.763382+0000","last_undegraded":"2024-03-06T07:26:06.763382+0000","last_fullsized":"2024-03-06T07:26:06.763382+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:39.997072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752142+0000","last_change":"2024-03-06T07:26:06.749663+0000","last_active":"2024-03-06T07:26:06.752142+0000","last_peered":"2024-03-06T07:26:06.752142+0000","last_clean":"2024-03-06T07:26:06.752142+0000","last_became_active":"2024-03-06T07:26:06.749363+0000","last_became_peered":"2024-03-06T07:26:06.749363+0000","last_unstale":"2024-03-06T07:26:06.752142+0000","last_undegraded":"2024-03-06T07:26:06.752142+0000","last_fullsized":"2024-03-06T07:26:06.752142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:53.775890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763091+0000","last_change":"2024-03-06T07:26:06.755229+0000","last_active":"2024-03-06T07:26:06.763091+0000","last_peered":"2024-03-06T07:26:06.763091+0000","last_clean":"2024-03-06T07:26:06.763091+0000","last_became_active":"2024-03-06T07:26:06.754970+0000","last_became_peered":"2024-03-06T07:26:06.754970+0000","last_unstale":"2024-03-06T07:26:06.763091+0000","last_undegraded":"2024-03-06T07:26:06.763091+0000","last_fullsized":"2024-03-06T07:26:06.763091+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:06.038831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752150+0000","last_change":"2024-03-06T07:26:06.749499+0000","last_active":"2024-03-06T07:26:06.752150+0000","last_peered":"2024-03-06T07:26:06.752150+0000","last_clean":"2024-03-06T07:26:06.752150+0000","last_became_active":"2024-03-06T07:26:06.749254+0000","last_became_peered":"2024-03-06T07:26:06.749254+0000","last_unstale":"2024-03-06T07:26:06.752150+0000","last_undegraded":"2024-03-06T07:26:06.752150+0000","last_fullsized":"2024-03-06T07:26:06.752150+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:58.564900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744132+0000","last_change":"2024-03-06T07:26:06.741957+0000","last_active":"2024-03-06T07:26:06.744132+0000","last_peered":"2024-03-06T07:26:06.744132+0000","last_clean":"2024-03-06T07:26:06.744132+0000","last_became_active":"2024-03-06T07:26:06.741698+0000","last_became_peered":"2024-03-06T07:26:06.741698+0000","last_unstale":"2024-03-06T07:26:06.744132+0000","last_undegraded":"2024-03-06T07:26:06.744132+0000","last_fullsized":"2024-03-06T07:26:06.744132+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:50.726652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757959+0000","last_change":"2024-03-06T07:26:06.755628+0000","last_active":"2024-03-06T07:26:06.757959+0000","last_peered":"2024-03-06T07:26:06.757959+0000","last_clean":"2024-03-06T07:26:06.757959+0000","last_became_active":"2024-03-06T07:26:06.755169+0000","last_became_peered":"2024-03-06T07:26:06.755169+0000","last_unstale":"2024-03-06T07:26:06.757959+0000","last_undegraded":"2024-03-06T07:26:06.757959+0000","last_fullsized":"2024-03-06T07:26:06.757959+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:17.823923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760383+0000","last_change":"2024-03-06T07:26:06.756124+0000","last_active":"2024-03-06T07:26:06.760383+0000","last_peered":"2024-03-06T07:26:06.760383+0000","last_clean":"2024-03-06T07:26:06.760383+0000","last_became_active":"2024-03-06T07:26:06.755722+0000","last_became_peered":"2024-03-06T07:26:06.755722+0000","last_unstale":"2024-03-06T07:26:06.760383+0000","last_undegraded":"2024-03-06T07:26:06.760383+0000","last_fullsized":"2024-03-06T07:26:06.760383+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:00.479923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757490+0000","last_change":"2024-03-06T07:26:06.753547+0000","last_active":"2024-03-06T07:26:06.757490+0000","last_peered":"2024-03-06T07:26:06.757490+0000","last_clean":"2024-03-06T07:26:06.757490+0000","last_became_active":"2024-03-06T07:26:06.753297+0000","last_became_peered":"2024-03-06T07:26:06.753297+0000","last_unstale":"2024-03-06T07:26:06.757490+0000","last_undegraded":"2024-03-06T07:26:06.757490+0000","last_fullsized":"2024-03-06T07:26:06.757490+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:56.246569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760663+0000","last_change":"2024-03-06T07:26:06.757035+0000","last_active":"2024-03-06T07:26:06.760663+0000","last_peered":"2024-03-06T07:26:06.760663+0000","last_clean":"2024-03-06T07:26:06.760663+0000","last_became_active":"2024-03-06T07:26:06.756775+0000","last_became_peered":"2024-03-06T07:26:06.756775+0000","last_unstale":"2024-03-06T07:26:06.760663+0000","last_undegraded":"2024-03-06T07:26:06.760663+0000","last_fullsized":"2024-03-06T07:26:06.760663+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:29.493414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766464+0000","last_change":"2024-03-06T07:26:06.763056+0000","last_active":"2024-03-06T07:26:06.766464+0000","last_peered":"2024-03-06T07:26:06.766464+0000","last_clean":"2024-03-06T07:26:06.766464+0000","last_became_active":"2024-03-06T07:26:06.762378+0000","last_became_peered":"2024-03-06T07:26:06.762378+0000","last_unstale":"2024-03-06T07:26:06.766464+0000","last_undegraded":"2024-03-06T07:26:06.766464+0000","last_fullsized":"2024-03-06T07:26:06.766464+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:53.952562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760637+0000","last_change":"2024-03-06T07:26:06.754255+0000","last_active":"2024-03-06T07:26:06.760637+0000","last_peered":"2024-03-06T07:26:06.760637+0000","last_clean":"2024-03-06T07:26:06.760637+0000","last_became_active":"2024-03-06T07:26:06.754001+0000","last_became_peered":"2024-03-06T07:26:06.754001+0000","last_unstale":"2024-03-06T07:26:06.760637+0000","last_undegraded":"2024-03-06T07:26:06.760637+0000","last_fullsized":"2024-03-06T07:26:06.760637+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:38.436587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763343+0000","last_change":"2024-03-06T07:26:06.756947+0000","last_active":"2024-03-06T07:26:06.763343+0000","last_peered":"2024-03-06T07:26:06.763343+0000","last_clean":"2024-03-06T07:26:06.763343+0000","last_became_active":"2024-03-06T07:26:06.756687+0000","last_became_peered":"2024-03-06T07:26:06.756687+0000","last_unstale":"2024-03-06T07:26:06.763343+0000","last_undegraded":"2024-03-06T07:26:06.763343+0000","last_fullsized":"2024-03-06T07:26:06.763343+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:53.253727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763198+0000","last_change":"2024-03-06T07:26:06.757991+0000","last_active":"2024-03-06T07:26:06.763198+0000","last_peered":"2024-03-06T07:26:06.763198+0000","last_clean":"2024-03-06T07:26:06.763198+0000","last_became_active":"2024-03-06T07:26:06.757736+0000","last_became_peered":"2024-03-06T07:26:06.757736+0000","last_unstale":"2024-03-06T07:26:06.763198+0000","last_undegraded":"2024-03-06T07:26:06.763198+0000","last_fullsized":"2024-03-06T07:26:06.763198+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:53.952562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757252+0000","last_change":"2024-03-06T07:26:06.753405+0000","last_active":"2024-03-06T07:26:06.757252+0000","last_peered":"2024-03-06T07:26:06.757252+0000","last_clean":"2024-03-06T07:26:06.757252+0000","last_became_active":"2024-03-06T07:26:06.753138+0000","last_became_peered":"2024-03-06T07:26:06.753138+0000","last_unstale":"2024-03-06T07:26:06.757252+0000","last_undegraded":"2024-03-06T07:26:06.757252+0000","last_fullsized":"2024-03-06T07:26:06.757252+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:08.634015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751849+0000","last_change":"2024-03-06T07:26:06.747617+0000","last_active":"2024-03-06T07:26:06.751849+0000","last_peered":"2024-03-06T07:26:06.751849+0000","last_clean":"2024-03-06T07:26:06.751849+0000","last_became_active":"2024-03-06T07:26:06.747358+0000","last_became_peered":"2024-03-06T07:26:06.747358+0000","last_unstale":"2024-03-06T07:26:06.751849+0000","last_undegraded":"2024-03-06T07:26:06.751849+0000","last_fullsized":"2024-03-06T07:26:06.751849+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:45.860295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.768968+0000","last_change":"2024-03-06T07:26:05.753683+0000","last_active":"2024-03-06T07:26:06.768968+0000","last_peered":"2024-03-06T07:26:06.768968+0000","last_clean":"2024-03-06T07:26:06.768968+0000","last_became_active":"2024-03-06T07:26:05.753411+0000","last_became_peered":"2024-03-06T07:26:05.753411+0000","last_unstale":"2024-03-06T07:26:06.768968+0000","last_undegraded":"2024-03-06T07:26:06.768968+0000","last_fullsized":"2024-03-06T07:26:06.768968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:47.419422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763166+0000","last_change":"2024-03-06T07:26:06.758626+0000","last_active":"2024-03-06T07:26:06.763166+0000","last_peered":"2024-03-06T07:26:06.763166+0000","last_clean":"2024-03-06T07:26:06.763166+0000","last_became_active":"2024-03-06T07:26:06.758377+0000","last_became_peered":"2024-03-06T07:26:06.758377+0000","last_unstale":"2024-03-06T07:26:06.763166+0000","last_undegraded":"2024-03-06T07:26:06.763166+0000","last_fullsized":"2024-03-06T07:26:06.763166+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.019146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766220+0000","last_change":"2024-03-06T07:26:05.747448+0000","last_active":"2024-03-06T07:26:06.766220+0000","last_peered":"2024-03-06T07:26:06.766220+0000","last_clean":"2024-03-06T07:26:06.766220+0000","last_became_active":"2024-03-06T07:26:05.746429+0000","last_became_peered":"2024-03-06T07:26:05.746429+0000","last_unstale":"2024-03-06T07:26:06.766220+0000","last_undegraded":"2024-03-06T07:26:06.766220+0000","last_fullsized":"2024-03-06T07:26:06.766220+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:32.112619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757223+0000","last_change":"2024-03-06T07:26:06.753040+0000","last_active":"2024-03-06T07:26:06.757223+0000","last_peered":"2024-03-06T07:26:06.757223+0000","last_clean":"2024-03-06T07:26:06.757223+0000","last_became_active":"2024-03-06T07:26:06.752712+0000","last_became_peered":"2024-03-06T07:26:06.752712+0000","last_unstale":"2024-03-06T07:26:06.757223+0000","last_undegraded":"2024-03-06T07:26:06.757223+0000","last_fullsized":"2024-03-06T07:26:06.757223+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:20.334563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.764806+0000","last_change":"2024-03-06T07:26:05.752588+0000","last_active":"2024-03-06T07:26:06.764806+0000","last_peered":"2024-03-06T07:26:06.764806+0000","last_clean":"2024-03-06T07:26:06.764806+0000","last_became_active":"2024-03-06T07:26:05.752268+0000","last_became_peered":"2024-03-06T07:26:05.752268+0000","last_unstale":"2024-03-06T07:26:06.764806+0000","last_undegraded":"2024-03-06T07:26:06.764806+0000","last_fullsized":"2024-03-06T07:26:06.764806+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:43.095761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757831+0000","last_change":"2024-03-06T07:26:06.753569+0000","last_active":"2024-03-06T07:26:06.757831+0000","last_peered":"2024-03-06T07:26:06.757831+0000","last_clean":"2024-03-06T07:26:06.757831+0000","last_became_active":"2024-03-06T07:26:06.753273+0000","last_became_peered":"2024-03-06T07:26:06.753273+0000","last_unstale":"2024-03-06T07:26:06.757831+0000","last_undegraded":"2024-03-06T07:26:06.757831+0000","last_fullsized":"2024-03-06T07:26:06.757831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:33.831911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766851+0000","last_change":"2024-03-06T07:26:05.753652+0000","last_active":"2024-03-06T07:26:06.766851+0000","last_peered":"2024-03-06T07:26:06.766851+0000","last_clean":"2024-03-06T07:26:06.766851+0000","last_became_active":"2024-03-06T07:26:05.753396+0000","last_became_peered":"2024-03-06T07:26:05.753396+0000","last_unstale":"2024-03-06T07:26:06.766851+0000","last_undegraded":"2024-03-06T07:26:06.766851+0000","last_fullsized":"2024-03-06T07:26:06.766851+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:41.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.754569+0000","last_change":"2024-03-06T07:26:06.750748+0000","last_active":"2024-03-06T07:26:06.754569+0000","last_peered":"2024-03-06T07:26:06.754569+0000","last_clean":"2024-03-06T07:26:06.754569+0000","last_became_active":"2024-03-06T07:26:06.750461+0000","last_became_peered":"2024-03-06T07:26:06.750461+0000","last_unstale":"2024-03-06T07:26:06.754569+0000","last_undegraded":"2024-03-06T07:26:06.754569+0000","last_fullsized":"2024-03-06T07:26:06.754569+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766615+0000","last_change":"2024-03-06T07:26:05.746652+0000","last_active":"2024-03-06T07:26:06.766615+0000","last_peered":"2024-03-06T07:26:06.766615+0000","last_clean":"2024-03-06T07:26:06.766615+0000","last_became_active":"2024-03-06T07:26:05.746392+0000","last_became_peered":"2024-03-06T07:26:05.746392+0000","last_unstale":"2024-03-06T07:26:06.766615+0000","last_undegraded":"2024-03-06T07:26:06.766615+0000","last_fullsized":"2024-03-06T07:26:06.766615+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:53.349741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769033+0000","last_change":"2024-03-06T07:26:06.765254+0000","last_active":"2024-03-06T07:26:06.769033+0000","last_peered":"2024-03-06T07:26:06.769033+0000","last_clean":"2024-03-06T07:26:06.769033+0000","last_became_active":"2024-03-06T07:26:06.764988+0000","last_became_peered":"2024-03-06T07:26:06.764988+0000","last_unstale":"2024-03-06T07:26:06.769033+0000","last_undegraded":"2024-03-06T07:26:06.769033+0000","last_fullsized":"2024-03-06T07:26:06.769033+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:44.100418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.764757+0000","last_change":"2024-03-06T07:26:05.752942+0000","last_active":"2024-03-06T07:26:06.764757+0000","last_peered":"2024-03-06T07:26:06.764757+0000","last_clean":"2024-03-06T07:26:06.764757+0000","last_became_active":"2024-03-06T07:26:05.752178+0000","last_became_peered":"2024-03-06T07:26:05.752178+0000","last_unstale":"2024-03-06T07:26:06.764757+0000","last_undegraded":"2024-03-06T07:26:06.764757+0000","last_fullsized":"2024-03-06T07:26:06.764757+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:27.596609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760584+0000","last_change":"2024-03-06T07:26:06.753593+0000","last_active":"2024-03-06T07:26:06.760584+0000","last_peered":"2024-03-06T07:26:06.760584+0000","last_clean":"2024-03-06T07:26:06.760584+0000","last_became_active":"2024-03-06T07:26:06.753272+0000","last_became_peered":"2024-03-06T07:26:06.753272+0000","last_unstale":"2024-03-06T07:26:06.760584+0000","last_undegraded":"2024-03-06T07:26:06.760584+0000","last_fullsized":"2024-03-06T07:26:06.760584+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:36.743823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.754078+0000","last_change":"2024-03-06T07:26:05.750175+0000","last_active":"2024-03-06T07:26:06.754078+0000","last_peered":"2024-03-06T07:26:06.754078+0000","last_clean":"2024-03-06T07:26:06.754078+0000","last_became_active":"2024-03-06T07:26:05.749834+0000","last_became_peered":"2024-03-06T07:26:05.749834+0000","last_unstale":"2024-03-06T07:26:06.754078+0000","last_undegraded":"2024-03-06T07:26:06.754078+0000","last_fullsized":"2024-03-06T07:26:06.754078+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:21.379637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763990+0000","last_change":"2024-03-06T07:26:06.759980+0000","last_active":"2024-03-06T07:26:06.763990+0000","last_peered":"2024-03-06T07:26:06.763990+0000","last_clean":"2024-03-06T07:26:06.763990+0000","last_became_active":"2024-03-06T07:26:06.759714+0000","last_became_peered":"2024-03-06T07:26:06.759714+0000","last_unstale":"2024-03-06T07:26:06.763990+0000","last_undegraded":"2024-03-06T07:26:06.763990+0000","last_fullsized":"2024-03-06T07:26:06.763990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.745241+0000","last_change":"2024-03-06T07:26:05.747778+0000","last_active":"2024-03-06T07:26:06.745241+0000","last_peered":"2024-03-06T07:26:06.745241+0000","last_clean":"2024-03-06T07:26:06.745241+0000","last_became_active":"2024-03-06T07:26:05.746734+0000","last_became_peered":"2024-03-06T07:26:05.746734+0000","last_unstale":"2024-03-06T07:26:06.745241+0000","last_undegraded":"2024-03-06T07:26:06.745241+0000","last_fullsized":"2024-03-06T07:26:06.745241+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:39.197150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752185+0000","last_change":"2024-03-06T07:26:06.750823+0000","last_active":"2024-03-06T07:26:06.752185+0000","last_peered":"2024-03-06T07:26:06.752185+0000","last_clean":"2024-03-06T07:26:06.752185+0000","last_became_active":"2024-03-06T07:26:06.750542+0000","last_became_peered":"2024-03-06T07:26:06.750542+0000","last_unstale":"2024-03-06T07:26:06.752185+0000","last_undegraded":"2024-03-06T07:26:06.752185+0000","last_fullsized":"2024-03-06T07:26:06.752185+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:33.537888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.745260+0000","last_change":"2024-03-06T07:26:05.743411+0000","last_active":"2024-03-06T07:26:06.745260+0000","last_peered":"2024-03-06T07:26:06.745260+0000","last_clean":"2024-03-06T07:26:06.745260+0000","last_became_active":"2024-03-06T07:26:05.743135+0000","last_became_peered":"2024-03-06T07:26:05.743135+0000","last_unstale":"2024-03-06T07:26:06.745260+0000","last_undegraded":"2024-03-06T07:26:06.745260+0000","last_fullsized":"2024-03-06T07:26:06.745260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:35.323058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755443+0000","last_change":"2024-03-06T07:26:06.753282+0000","last_active":"2024-03-06T07:26:06.755443+0000","last_peered":"2024-03-06T07:26:06.755443+0000","last_clean":"2024-03-06T07:26:06.755443+0000","last_became_active":"2024-03-06T07:26:06.752990+0000","last_became_peered":"2024-03-06T07:26:06.752990+0000","last_unstale":"2024-03-06T07:26:06.755443+0000","last_undegraded":"2024-03-06T07:26:06.755443+0000","last_fullsized":"2024-03-06T07:26:06.755443+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:21.523545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738001+0000","last_change":"2024-03-06T07:26:05.750497+0000","last_active":"2024-03-06T07:26:06.738001+0000","last_peered":"2024-03-06T07:26:06.738001+0000","last_clean":"2024-03-06T07:26:06.738001+0000","last_became_active":"2024-03-06T07:26:05.750222+0000","last_became_peered":"2024-03-06T07:26:05.750222+0000","last_unstale":"2024-03-06T07:26:06.738001+0000","last_undegraded":"2024-03-06T07:26:06.738001+0000","last_fullsized":"2024-03-06T07:26:06.738001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:32.827254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758307+0000","last_change":"2024-03-06T07:26:06.753505+0000","last_active":"2024-03-06T07:26:06.758307+0000","last_peered":"2024-03-06T07:26:06.758307+0000","last_clean":"2024-03-06T07:26:06.758307+0000","last_became_active":"2024-03-06T07:26:06.749927+0000","last_became_peered":"2024-03-06T07:26:06.749927+0000","last_unstale":"2024-03-06T07:26:06.758307+0000","last_undegraded":"2024-03-06T07:26:06.758307+0000","last_fullsized":"2024-03-06T07:26:06.758307+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:36.242566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769135+0000","last_change":"2024-03-06T07:26:05.745579+0000","last_active":"2024-03-06T07:26:06.769135+0000","last_peered":"2024-03-06T07:26:06.769135+0000","last_clean":"2024-03-06T07:26:06.769135+0000","last_became_active":"2024-03-06T07:26:05.745282+0000","last_became_peered":"2024-03-06T07:26:05.745282+0000","last_unstale":"2024-03-06T07:26:06.769135+0000","last_undegraded":"2024-03-06T07:26:06.769135+0000","last_fullsized":"2024-03-06T07:26:06.769135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:59.475266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758090+0000","last_change":"2024-03-06T07:26:06.756276+0000","last_active":"2024-03-06T07:26:06.758090+0000","last_peered":"2024-03-06T07:26:06.758090+0000","last_clean":"2024-03-06T07:26:06.758090+0000","last_became_active":"2024-03-06T07:26:06.755934+0000","last_became_peered":"2024-03-06T07:26:06.755934+0000","last_unstale":"2024-03-06T07:26:06.758090+0000","last_undegraded":"2024-03-06T07:26:06.758090+0000","last_fullsized":"2024-03-06T07:26:06.758090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:42.579421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.773458+0000","last_change":"2024-03-06T07:26:05.746883+0000","last_active":"2024-03-06T07:26:06.773458+0000","last_peered":"2024-03-06T07:26:06.773458+0000","last_clean":"2024-03-06T07:26:06.773458+0000","last_became_active":"2024-03-06T07:26:05.746601+0000","last_became_peered":"2024-03-06T07:26:05.746601+0000","last_unstale":"2024-03-06T07:26:06.773458+0000","last_undegraded":"2024-03-06T07:26:06.773458+0000","last_fullsized":"2024-03-06T07:26:06.773458+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:19.017740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759175+0000","last_change":"2024-03-06T07:26:06.749160+0000","last_active":"2024-03-06T07:26:06.759175+0000","last_peered":"2024-03-06T07:26:06.759175+0000","last_clean":"2024-03-06T07:26:06.759175+0000","last_became_active":"2024-03-06T07:26:06.748281+0000","last_became_peered":"2024-03-06T07:26:06.748281+0000","last_unstale":"2024-03-06T07:26:06.759175+0000","last_undegraded":"2024-03-06T07:26:06.759175+0000","last_fullsized":"2024-03-06T07:26:06.759175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766440+0000","last_change":"2024-03-06T07:26:05.749138+0000","last_active":"2024-03-06T07:26:06.766440+0000","last_peered":"2024-03-06T07:26:06.766440+0000","last_clean":"2024-03-06T07:26:06.766440+0000","last_became_active":"2024-03-06T07:26:05.748852+0000","last_became_peered":"2024-03-06T07:26:05.748852+0000","last_unstale":"2024-03-06T07:26:06.766440+0000","last_undegraded":"2024-03-06T07:26:06.766440+0000","last_fullsized":"2024-03-06T07:26:06.766440+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:54.242958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757455+0000","last_change":"2024-03-06T07:26:06.753914+0000","last_active":"2024-03-06T07:26:06.757455+0000","last_peered":"2024-03-06T07:26:06.757455+0000","last_clean":"2024-03-06T07:26:06.757455+0000","last_became_active":"2024-03-06T07:26:06.753579+0000","last_became_peered":"2024-03-06T07:26:06.753579+0000","last_unstale":"2024-03-06T07:26:06.757455+0000","last_undegraded":"2024-03-06T07:26:06.757455+0000","last_fullsized":"2024-03-06T07:26:06.757455+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:43.131393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744541+0000","last_change":"2024-03-06T07:26:05.750634+0000","last_active":"2024-03-06T07:26:06.744541+0000","last_peered":"2024-03-06T07:26:06.744541+0000","last_clean":"2024-03-06T07:26:06.744541+0000","last_became_active":"2024-03-06T07:26:05.749812+0000","last_became_peered":"2024-03-06T07:26:05.749812+0000","last_unstale":"2024-03-06T07:26:06.744541+0000","last_undegraded":"2024-03-06T07:26:06.744541+0000","last_fullsized":"2024-03-06T07:26:06.744541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:02.117850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755471+0000","last_change":"2024-03-06T07:26:06.754534+0000","last_active":"2024-03-06T07:26:06.755471+0000","last_peered":"2024-03-06T07:26:06.755471+0000","last_clean":"2024-03-06T07:26:06.755471+0000","last_became_active":"2024-03-06T07:26:06.753676+0000","last_became_peered":"2024-03-06T07:26:06.753676+0000","last_unstale":"2024-03-06T07:26:06.755471+0000","last_undegraded":"2024-03-06T07:26:06.755471+0000","last_fullsized":"2024-03-06T07:26:06.755471+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:36.242566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.743266+0000","last_change":"2024-03-06T07:26:05.746612+0000","last_active":"2024-03-06T07:26:06.743266+0000","last_peered":"2024-03-06T07:26:06.743266+0000","last_clean":"2024-03-06T07:26:06.743266+0000","last_became_active":"2024-03-06T07:26:05.746302+0000","last_became_peered":"2024-03-06T07:26:05.746302+0000","last_unstale":"2024-03-06T07:26:06.743266+0000","last_undegraded":"2024-03-06T07:26:06.743266+0000","last_fullsized":"2024-03-06T07:26:06.743266+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:15.679331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758492+0000","last_change":"2024-03-06T07:26:06.753399+0000","last_active":"2024-03-06T07:26:06.758492+0000","last_peered":"2024-03-06T07:26:06.758492+0000","last_clean":"2024-03-06T07:26:06.758492+0000","last_became_active":"2024-03-06T07:26:06.753115+0000","last_became_peered":"2024-03-06T07:26:06.753115+0000","last_unstale":"2024-03-06T07:26:06.758492+0000","last_undegraded":"2024-03-06T07:26:06.758492+0000","last_fullsized":"2024-03-06T07:26:06.758492+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:56.310166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769256+0000","last_change":"2024-03-06T07:26:05.740200+0000","last_active":"2024-03-06T07:26:06.769256+0000","last_peered":"2024-03-06T07:26:06.769256+0000","last_clean":"2024-03-06T07:26:06.769256+0000","last_became_active":"2024-03-06T07:26:05.739902+0000","last_became_peered":"2024-03-06T07:26:05.739902+0000","last_unstale":"2024-03-06T07:26:06.769256+0000","last_undegraded":"2024-03-06T07:26:06.769256+0000","last_fullsized":"2024-03-06T07:26:06.769256+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:50.744728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760404+0000","last_change":"2024-03-06T07:26:06.756866+0000","last_active":"2024-03-06T07:26:06.760404+0000","last_peered":"2024-03-06T07:26:06.760404+0000","last_clean":"2024-03-06T07:26:06.760404+0000","last_became_active":"2024-03-06T07:26:06.756523+0000","last_became_peered":"2024-03-06T07:26:06.756523+0000","last_unstale":"2024-03-06T07:26:06.760404+0000","last_undegraded":"2024-03-06T07:26:06.760404+0000","last_fullsized":"2024-03-06T07:26:06.760404+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:36.743823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769472+0000","last_change":"2024-03-06T07:26:05.755349+0000","last_active":"2024-03-06T07:26:06.769472+0000","last_peered":"2024-03-06T07:26:06.769472+0000","last_clean":"2024-03-06T07:26:06.769472+0000","last_became_active":"2024-03-06T07:26:05.755089+0000","last_became_peered":"2024-03-06T07:26:05.755089+0000","last_unstale":"2024-03-06T07:26:06.769472+0000","last_undegraded":"2024-03-06T07:26:06.769472+0000","last_fullsized":"2024-03-06T07:26:06.769472+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:56.850597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755148+0000","last_change":"2024-03-06T07:26:06.753872+0000","last_active":"2024-03-06T07:26:06.755148+0000","last_peered":"2024-03-06T07:26:06.755148+0000","last_clean":"2024-03-06T07:26:06.755148+0000","last_became_active":"2024-03-06T07:26:06.753587+0000","last_became_peered":"2024-03-06T07:26:06.753587+0000","last_unstale":"2024-03-06T07:26:06.755148+0000","last_undegraded":"2024-03-06T07:26:06.755148+0000","last_fullsized":"2024-03-06T07:26:06.755148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.768966+0000","last_change":"2024-03-06T07:26:05.745744+0000","last_active":"2024-03-06T07:26:06.768966+0000","last_peered":"2024-03-06T07:26:06.768966+0000","last_clean":"2024-03-06T07:26:06.768966+0000","last_became_active":"2024-03-06T07:26:05.745456+0000","last_became_peered":"2024-03-06T07:26:05.745456+0000","last_unstale":"2024-03-06T07:26:06.768966+0000","last_undegraded":"2024-03-06T07:26:06.768966+0000","last_fullsized":"2024-03-06T07:26:06.768966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:52.771232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.756144+0000","last_change":"2024-03-06T07:26:06.753267+0000","last_active":"2024-03-06T07:26:06.756144+0000","last_peered":"2024-03-06T07:26:06.756144+0000","last_clean":"2024-03-06T07:26:06.756144+0000","last_became_active":"2024-03-06T07:26:06.752853+0000","last_became_peered":"2024-03-06T07:26:06.752853+0000","last_unstale":"2024-03-06T07:26:06.756144+0000","last_undegraded":"2024-03-06T07:26:06.756144+0000","last_fullsized":"2024-03-06T07:26:06.756144+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:53.952562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738936+0000","last_change":"2024-03-06T07:26:05.750980+0000","last_active":"2024-03-06T07:26:06.738936+0000","last_peered":"2024-03-06T07:26:06.738936+0000","last_clean":"2024-03-06T07:26:06.738936+0000","last_became_active":"2024-03-06T07:26:05.750712+0000","last_became_peered":"2024-03-06T07:26:05.750712+0000","last_unstale":"2024-03-06T07:26:06.738936+0000","last_undegraded":"2024-03-06T07:26:06.738936+0000","last_fullsized":"2024-03-06T07:26:06.738936+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:28.488757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758564+0000","last_change":"2024-03-06T07:26:06.752363+0000","last_active":"2024-03-06T07:26:06.758564+0000","last_peered":"2024-03-06T07:26:06.758564+0000","last_clean":"2024-03-06T07:26:06.758564+0000","last_became_active":"2024-03-06T07:26:06.752110+0000","last_became_peered":"2024-03-06T07:26:06.752110+0000","last_unstale":"2024-03-06T07:26:06.758564+0000","last_undegraded":"2024-03-06T07:26:06.758564+0000","last_fullsized":"2024-03-06T07:26:06.758564+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:45.870576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.764854+0000","last_change":"2024-03-06T07:26:05.752024+0000","last_active":"2024-03-06T07:26:06.764854+0000","last_peered":"2024-03-06T07:26:06.764854+0000","last_clean":"2024-03-06T07:26:06.764854+0000","last_became_active":"2024-03-06T07:26:05.751691+0000","last_became_peered":"2024-03-06T07:26:05.751691+0000","last_unstale":"2024-03-06T07:26:06.764854+0000","last_undegraded":"2024-03-06T07:26:06.764854+0000","last_fullsized":"2024-03-06T07:26:06.764854+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:32.677065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755349+0000","last_change":"2024-03-06T07:26:06.750354+0000","last_active":"2024-03-06T07:26:06.755349+0000","last_peered":"2024-03-06T07:26:06.755349+0000","last_clean":"2024-03-06T07:26:06.755349+0000","last_became_active":"2024-03-06T07:26:06.750088+0000","last_became_peered":"2024-03-06T07:26:06.750088+0000","last_unstale":"2024-03-06T07:26:06.755349+0000","last_undegraded":"2024-03-06T07:26:06.755349+0000","last_fullsized":"2024-03-06T07:26:06.755349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:40.701820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.737246+0000","last_change":"2024-03-06T07:26:05.749049+0000","last_active":"2024-03-06T07:26:06.737246+0000","last_peered":"2024-03-06T07:26:06.737246+0000","last_clean":"2024-03-06T07:26:06.737246+0000","last_became_active":"2024-03-06T07:26:05.748819+0000","last_became_peered":"2024-03-06T07:26:05.748819+0000","last_unstale":"2024-03-06T07:26:06.737246+0000","last_undegraded":"2024-03-06T07:26:06.737246+0000","last_fullsized":"2024-03-06T07:26:06.737246+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:21.379637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751874+0000","last_change":"2024-03-06T07:26:06.747836+0000","last_active":"2024-03-06T07:26:06.751874+0000","last_peered":"2024-03-06T07:26:06.751874+0000","last_clean":"2024-03-06T07:26:06.751874+0000","last_became_active":"2024-03-06T07:26:06.747556+0000","last_became_peered":"2024-03-06T07:26:06.747556+0000","last_unstale":"2024-03-06T07:26:06.751874+0000","last_undegraded":"2024-03-06T07:26:06.751874+0000","last_fullsized":"2024-03-06T07:26:06.751874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:57.109567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.736875+0000","last_change":"2024-03-06T07:26:05.747510+0000","last_active":"2024-03-06T07:26:06.736875+0000","last_peered":"2024-03-06T07:26:06.736875+0000","last_clean":"2024-03-06T07:26:06.736875+0000","last_became_active":"2024-03-06T07:26:05.747110+0000","last_became_peered":"2024-03-06T07:26:05.747110+0000","last_unstale":"2024-03-06T07:26:06.736875+0000","last_undegraded":"2024-03-06T07:26:06.736875+0000","last_fullsized":"2024-03-06T07:26:06.736875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:32.827254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760199+0000","last_change":"2024-03-06T07:26:06.756034+0000","last_active":"2024-03-06T07:26:06.760199+0000","last_peered":"2024-03-06T07:26:06.760199+0000","last_clean":"2024-03-06T07:26:06.760199+0000","last_became_active":"2024-03-06T07:26:06.755610+0000","last_became_peered":"2024-03-06T07:26:06.755610+0000","last_unstale":"2024-03-06T07:26:06.760199+0000","last_undegraded":"2024-03-06T07:26:06.760199+0000","last_fullsized":"2024-03-06T07:26:06.760199+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738540+0000","last_change":"2024-03-06T07:26:05.752339+0000","last_active":"2024-03-06T07:26:06.738540+0000","last_peered":"2024-03-06T07:26:06.738540+0000","last_clean":"2024-03-06T07:26:06.738540+0000","last_became_active":"2024-03-06T07:26:05.752076+0000","last_became_peered":"2024-03-06T07:26:05.752076+0000","last_unstale":"2024-03-06T07:26:06.738540+0000","last_undegraded":"2024-03-06T07:26:06.738540+0000","last_fullsized":"2024-03-06T07:26:06.738540+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:16.819266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752165+0000","last_change":"2024-03-06T07:26:06.749499+0000","last_active":"2024-03-06T07:26:06.752165+0000","last_peered":"2024-03-06T07:26:06.752165+0000","last_clean":"2024-03-06T07:26:06.752165+0000","last_became_active":"2024-03-06T07:26:06.749215+0000","last_became_peered":"2024-03-06T07:26:06.749215+0000","last_unstale":"2024-03-06T07:26:06.752165+0000","last_undegraded":"2024-03-06T07:26:06.752165+0000","last_fullsized":"2024-03-06T07:26:06.752165+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.951691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766594+0000","last_change":"2024-03-06T07:26:05.745430+0000","last_active":"2024-03-06T07:26:06.766594+0000","last_peered":"2024-03-06T07:26:06.766594+0000","last_clean":"2024-03-06T07:26:06.766594+0000","last_became_active":"2024-03-06T07:26:05.745144+0000","last_became_peered":"2024-03-06T07:26:05.745144+0000","last_unstale":"2024-03-06T07:26:06.766594+0000","last_undegraded":"2024-03-06T07:26:06.766594+0000","last_fullsized":"2024-03-06T07:26:06.766594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:35.237909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760390+0000","last_change":"2024-03-06T07:26:06.754885+0000","last_active":"2024-03-06T07:26:06.760390+0000","last_peered":"2024-03-06T07:26:06.760390+0000","last_clean":"2024-03-06T07:26:06.760390+0000","last_became_active":"2024-03-06T07:26:06.754451+0000","last_became_peered":"2024-03-06T07:26:06.754451+0000","last_unstale":"2024-03-06T07:26:06.760390+0000","last_undegraded":"2024-03-06T07:26:06.760390+0000","last_fullsized":"2024-03-06T07:26:06.760390+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:55.308824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.740519+0000","last_change":"2024-03-06T07:26:05.751917+0000","last_active":"2024-03-06T07:26:06.740519+0000","last_peered":"2024-03-06T07:26:06.740519+0000","last_clean":"2024-03-06T07:26:06.740519+0000","last_became_active":"2024-03-06T07:26:05.751694+0000","last_became_peered":"2024-03-06T07:26:05.751694+0000","last_unstale":"2024-03-06T07:26:06.740519+0000","last_undegraded":"2024-03-06T07:26:06.740519+0000","last_fullsized":"2024-03-06T07:26:06.740519+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.173532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.761630+0000","last_change":"2024-03-06T07:26:06.756363+0000","last_active":"2024-03-06T07:26:06.761630+0000","last_peered":"2024-03-06T07:26:06.761630+0000","last_clean":"2024-03-06T07:26:06.761630+0000","last_became_active":"2024-03-06T07:26:06.756057+0000","last_became_peered":"2024-03-06T07:26:06.756057+0000","last_unstale":"2024-03-06T07:26:06.761630+0000","last_undegraded":"2024-03-06T07:26:06.761630+0000","last_fullsized":"2024-03-06T07:26:06.761630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:22.772489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.740287+0000","last_change":"2024-03-06T07:26:05.753999+0000","last_active":"2024-03-06T07:26:06.740287+0000","last_peered":"2024-03-06T07:26:06.740287+0000","last_clean":"2024-03-06T07:26:06.740287+0000","last_became_active":"2024-03-06T07:26:05.753734+0000","last_became_peered":"2024-03-06T07:26:05.753734+0000","last_unstale":"2024-03-06T07:26:06.740287+0000","last_undegraded":"2024-03-06T07:26:06.740287+0000","last_fullsized":"2024-03-06T07:26:06.740287+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:50.325917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759897+0000","last_change":"2024-03-06T07:26:06.754052+0000","last_active":"2024-03-06T07:26:06.759897+0000","last_peered":"2024-03-06T07:26:06.759897+0000","last_clean":"2024-03-06T07:26:06.759897+0000","last_became_active":"2024-03-06T07:26:06.753767+0000","last_became_peered":"2024-03-06T07:26:06.753767+0000","last_unstale":"2024-03-06T07:26:06.759897+0000","last_undegraded":"2024-03-06T07:26:06.759897+0000","last_fullsized":"2024-03-06T07:26:06.759897+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.743877+0000","last_change":"2024-03-06T07:26:05.748244+0000","last_active":"2024-03-06T07:26:06.743877+0000","last_peered":"2024-03-06T07:26:06.743877+0000","last_clean":"2024-03-06T07:26:06.743877+0000","last_became_active":"2024-03-06T07:26:05.747897+0000","last_became_peered":"2024-03-06T07:26:05.747897+0000","last_unstale":"2024-03-06T07:26:06.743877+0000","last_undegraded":"2024-03-06T07:26:06.743877+0000","last_fullsized":"2024-03-06T07:26:06.743877+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.754998+0000","last_change":"2024-03-06T07:26:06.751133+0000","last_active":"2024-03-06T07:26:06.754998+0000","last_peered":"2024-03-06T07:26:06.754998+0000","last_clean":"2024-03-06T07:26:06.754998+0000","last_became_active":"2024-03-06T07:26:06.749439+0000","last_became_peered":"2024-03-06T07:26:06.749439+0000","last_unstale":"2024-03-06T07:26:06.754998+0000","last_undegraded":"2024-03-06T07:26:06.754998+0000","last_fullsized":"2024-03-06T07:26:06.754998+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:35.668699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.740417+0000","last_change":"2024-03-06T07:26:05.753483+0000","last_active":"2024-03-06T07:26:06.740417+0000","last_peered":"2024-03-06T07:26:06.740417+0000","last_clean":"2024-03-06T07:26:06.740417+0000","last_became_active":"2024-03-06T07:26:05.753192+0000","last_became_peered":"2024-03-06T07:26:05.753192+0000","last_unstale":"2024-03-06T07:26:06.740417+0000","last_undegraded":"2024-03-06T07:26:06.740417+0000","last_fullsized":"2024-03-06T07:26:06.740417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:05.000463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763872+0000","last_change":"2024-03-06T07:26:06.756811+0000","last_active":"2024-03-06T07:26:06.763872+0000","last_peered":"2024-03-06T07:26:06.763872+0000","last_clean":"2024-03-06T07:26:06.763872+0000","last_became_active":"2024-03-06T07:26:06.756560+0000","last_became_peered":"2024-03-06T07:26:06.756560+0000","last_unstale":"2024-03-06T07:26:06.763872+0000","last_undegraded":"2024-03-06T07:26:06.763872+0000","last_fullsized":"2024-03-06T07:26:06.763872+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:22.384294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.741853+0000","last_change":"2024-03-06T07:26:05.749765+0000","last_active":"2024-03-06T07:26:06.741853+0000","last_peered":"2024-03-06T07:26:06.741853+0000","last_clean":"2024-03-06T07:26:06.741853+0000","last_became_active":"2024-03-06T07:26:05.749477+0000","last_became_peered":"2024-03-06T07:26:05.749477+0000","last_unstale":"2024-03-06T07:26:06.741853+0000","last_undegraded":"2024-03-06T07:26:06.741853+0000","last_fullsized":"2024-03-06T07:26:06.741853+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:44.594569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758514+0000","last_change":"2024-03-06T07:26:06.752845+0000","last_active":"2024-03-06T07:26:06.758514+0000","last_peered":"2024-03-06T07:26:06.758514+0000","last_clean":"2024-03-06T07:26:06.758514+0000","last_became_active":"2024-03-06T07:26:06.752538+0000","last_became_peered":"2024-03-06T07:26:06.752538+0000","last_unstale":"2024-03-06T07:26:06.758514+0000","last_undegraded":"2024-03-06T07:26:06.758514+0000","last_fullsized":"2024-03-06T07:26:06.758514+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:33.191204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738378+0000","last_change":"2024-03-06T07:26:05.747439+0000","last_active":"2024-03-06T07:26:06.738378+0000","last_peered":"2024-03-06T07:26:06.738378+0000","last_clean":"2024-03-06T07:26:06.738378+0000","last_became_active":"2024-03-06T07:26:05.746992+0000","last_became_peered":"2024-03-06T07:26:05.746992+0000","last_unstale":"2024-03-06T07:26:06.738378+0000","last_undegraded":"2024-03-06T07:26:06.738378+0000","last_fullsized":"2024-03-06T07:26:06.738378+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:35.237909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.768959+0000","last_change":"2024-03-06T07:26:06.763752+0000","last_active":"2024-03-06T07:26:06.768959+0000","last_peered":"2024-03-06T07:26:06.768959+0000","last_clean":"2024-03-06T07:26:06.768959+0000","last_became_active":"2024-03-06T07:26:06.763493+0000","last_became_peered":"2024-03-06T07:26:06.763493+0000","last_unstale":"2024-03-06T07:26:06.768959+0000","last_undegraded":"2024-03-06T07:26:06.768959+0000","last_fullsized":"2024-03-06T07:26:06.768959+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.019146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.741798+0000","last_change":"2024-03-06T07:26:05.748595+0000","last_active":"2024-03-06T07:26:06.741798+0000","last_peered":"2024-03-06T07:26:06.741798+0000","last_clean":"2024-03-06T07:26:06.741798+0000","last_became_active":"2024-03-06T07:26:05.748306+0000","last_became_peered":"2024-03-06T07:26:05.748306+0000","last_unstale":"2024-03-06T07:26:06.741798+0000","last_undegraded":"2024-03-06T07:26:06.741798+0000","last_fullsized":"2024-03-06T07:26:06.741798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:44.657926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751874+0000","last_change":"2024-03-06T07:26:06.747815+0000","last_active":"2024-03-06T07:26:06.751874+0000","last_peered":"2024-03-06T07:26:06.751874+0000","last_clean":"2024-03-06T07:26:06.751874+0000","last_became_active":"2024-03-06T07:26:06.747548+0000","last_became_peered":"2024-03-06T07:26:06.747548+0000","last_unstale":"2024-03-06T07:26:06.751874+0000","last_undegraded":"2024-03-06T07:26:06.751874+0000","last_fullsized":"2024-03-06T07:26:06.751874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:38.033859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.765166+0000","last_change":"2024-03-06T07:26:05.747001+0000","last_active":"2024-03-06T07:26:06.765166+0000","last_peered":"2024-03-06T07:26:06.765166+0000","last_clean":"2024-03-06T07:26:06.765166+0000","last_became_active":"2024-03-06T07:26:05.746677+0000","last_became_peered":"2024-03-06T07:26:05.746677+0000","last_unstale":"2024-03-06T07:26:06.765166+0000","last_undegraded":"2024-03-06T07:26:06.765166+0000","last_fullsized":"2024-03-06T07:26:06.765166+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:21.558359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.748033+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:26:06.748033+0000","last_peered":"2024-03-06T07:26:06.748033+0000","last_clean":"2024-03-06T07:26:06.748033+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:26:06.748033+0000","last_undegraded":"2024-03-06T07:26:06.748033+0000","last_fullsized":"2024-03-06T07:26:06.748033+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1732608,"data_stored":1072745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1150976,"data_stored":489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574905,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1748992,"data_stored":1080102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:09.709 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:26:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.843+0000 7f5c4890c640 1 Processor -- start 2024-03-06T07:26:09.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.843+0000 7f5c4890c640 1 -- start start 2024-03-06T07:26:09.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4890c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c4415dfe0 0x7f5c440cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4890c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c440cf830 0x7f5c440d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4890c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c440b3430 con 0x7f5c440d27f0 2024-03-06T07:26:09.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4890c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c440b31e0 con 0x7f5c4415dfe0 2024-03-06T07:26:09.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4890c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c440b3d60 con 0x7f5c440cf830 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c4415dfe0 0x7f5c440cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5c4415dfe0 0x7f5c440cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34640/0 (socket says 172.21.15.72:34640) 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c42d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5c440d27f0 0x7f5c440d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42574/0 (socket says 172.21.15.72:42574) 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c4259c640 1 -- 172.21.15.72:0/975188895 learned_addr learned my addr 172.21.15.72:0/975188895 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.844+0000 7f5c41d9b640 1 --2- 172.21.15.72:0/975188895 >> v2:172.21.15.146:3300/0 conn(0x7f5c440cf830 0x7f5c440d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.845+0000 7f5c4259c640 1 -- 172.21.15.72:0/975188895 >> v2:172.21.15.146:3300/0 conn(0x7f5c440cf830 msgr2=0x7f5c440d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.845+0000 7f5c4259c640 1 --2- 172.21.15.72:0/975188895 >> v2:172.21.15.146:3300/0 conn(0x7f5c440cf830 0x7f5c440d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.845+0000 7f5c4259c640 1 -- 172.21.15.72:0/975188895 >> v1:172.21.15.72:6789/0 conn(0x7f5c440d27f0 legacy=0x7f5c440d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.845+0000 7f5c4259c640 1 -- 172.21.15.72:0/975188895 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c440b3710 con 0x7f5c4415dfe0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.845+0000 7f5c4159a640 1 -- 172.21.15.72:0/975188895 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909158451 0 0) 0x7f5c440b3430 con 0x7f5c440d27f0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.845+0000 7f5c4259c640 1 --2- 172.21.15.72:0/975188895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4415dfe0 0x7f5c440cf2f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009e50 tx=0x7f5c3402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e52aa6391503ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4159a640 1 -- 172.21.15.72:0/975188895 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3402cb20 con 0x7f5c4415dfe0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4159a640 1 -- 172.21.15.72:0/975188895 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3402ccc0 con 0x7f5c4415dfe0 2024-03-06T07:26:09.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4159a640 1 -- 172.21.15.72:0/975188895 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34034610 con 0x7f5c4415dfe0 2024-03-06T07:26:09.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 -- 172.21.15.72:0/975188895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4415dfe0 msgr2=0x7f5c440cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 --2- 172.21.15.72:0/975188895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4415dfe0 0x7f5c440cf2f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009e50 tx=0x7f5c3402bf80 comp rx=0 tx=0).stop 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 -- 172.21.15.72:0/975188895 shutdown_connections 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 --2- 172.21.15.72:0/975188895 >> v2:172.21.15.146:3300/0 conn(0x7f5c440cf830 0x7f5c440d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 --2- 172.21.15.72:0/975188895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c4415dfe0 0x7f5c440cf2f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 -- 172.21.15.72:0/975188895 >> 172.21.15.72:0/975188895 conn(0x7f5c440d3cd0 msgr2=0x7f5c440d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 -- 172.21.15.72:0/975188895 shutdown_connections 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.846+0000 7f5c4890c640 1 -- 172.21.15.72:0/975188895 wait complete. 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.847+0000 7f5c4890c640 1 Processor -- start 2024-03-06T07:26:09.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 -- start start 2024-03-06T07:26:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 0x7f5c440b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c440d27f0 0x7f5c441f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c4415dfe0 0x7f5c4415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c440d1df0 con 0x7f5c440cf830 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c440d1c70 con 0x7f5c4415dfe0 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4890c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4415e3c0 con 0x7f5c440d27f0 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 0x7f5c440b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 0x7f5c440b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34646/0 (socket says 172.21.15.72:34646) 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c4259c640 1 -- 172.21.15.72:0/3594590190 learned_addr learned my addr 172.21.15.72:0/3594590190 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c41d9b640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c440d27f0 0x7f5c441f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.848+0000 7f5c42d9d640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c4415dfe0 0x7f5c4415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4259c640 1 -- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c440d27f0 msgr2=0x7f5c441f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4259c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c440d27f0 0x7f5c441f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4259c640 1 -- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c4415dfe0 msgr2=0x7f5c4415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4259c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c4415dfe0 0x7f5c4415b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4259c640 1 -- 172.21.15.72:0/3594590190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c440b95a0 con 0x7f5c440cf830 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4259c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 0x7f5c440b8bc0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009e50 tx=0x7f5c34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34043030 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3403e450 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c441f6d60 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c34009510 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.849+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c441f7320 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.850+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4404f330 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.850+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c34046070 con 0x7f5c440cf830 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.851+0000 7f5c32ffd640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c2406b4d0 0x7f5c2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:09.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.851+0000 7f5c41d9b640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c2406b4d0 0x7f5c2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:09.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.851+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c340ac5a0 con 0x7f5c440cf830 2024-03-06T07:26:09.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.851+0000 7f5c41d9b640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c2406b4d0 0x7f5c2406d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5c440d31a0 tx=0x7f5c38009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:09.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:09.855+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c3404d070 con 0x7f5c440cf830 2024-03-06T07:26:10.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.023+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c441f7560 con 0x7f5c2406b4d0 2024-03-06T07:26:10.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.029+0000 7f5c32ffd640 1 -- 172.21.15.72:0/3594590190 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252015 (secure 0 0 0) 0x7f5c441f7560 con 0x7f5c2406b4d0 2024-03-06T07:26:10.033 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:10.035 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c2406b4d0 msgr2=0x7f5c2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c2406b4d0 0x7f5c2406d990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5c440d31a0 tx=0x7f5c38009190 comp rx=0 tx=0).stop 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 msgr2=0x7f5c440b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 0x7f5c440b8bc0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009e50 tx=0x7f5c34004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 shutdown_connections 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c2406b4d0 0x7f5c2406d990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c4415dfe0 0x7f5c4415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c440d27f0 0x7f5c441f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 --2- 172.21.15.72:0/3594590190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c440cf830 0x7f5c440b8bc0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 >> 172.21.15.72:0/3594590190 conn(0x7f5c440d3cd0 msgr2=0x7f5c440d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 shutdown_connections 2024-03-06T07:26:10.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.032+0000 7f5c4890c640 1 -- 172.21.15.72:0/3594590190 wait complete. 2024-03-06T07:26:10.045 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-03-06T07:26:09.994357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":44,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259544,"kb_used_data":10084,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754715176,"statfs":{"total":773094113280,"available":772828340224,"internally_reserved":0,"allocated":10326016,"data_stored":5053884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":255315505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.266896"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760795+0000","last_change":"2024-03-06T07:26:06.755635+0000","last_active":"2024-03-06T07:26:06.760795+0000","last_peered":"2024-03-06T07:26:06.760795+0000","last_clean":"2024-03-06T07:26:06.760795+0000","last_became_active":"2024-03-06T07:26:06.755367+0000","last_became_peered":"2024-03-06T07:26:06.755367+0000","last_unstale":"2024-03-06T07:26:06.760795+0000","last_undegraded":"2024-03-06T07:26:06.760795+0000","last_fullsized":"2024-03-06T07:26:06.760795+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:23.274410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744585+0000","last_change":"2024-03-06T07:26:06.742400+0000","last_active":"2024-03-06T07:26:06.744585+0000","last_peered":"2024-03-06T07:26:06.744585+0000","last_clean":"2024-03-06T07:26:06.744585+0000","last_became_active":"2024-03-06T07:26:06.742143+0000","last_became_peered":"2024-03-06T07:26:06.742143+0000","last_unstale":"2024-03-06T07:26:06.744585+0000","last_undegraded":"2024-03-06T07:26:06.744585+0000","last_fullsized":"2024-03-06T07:26:06.744585+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:43:47.608451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760821+0000","last_change":"2024-03-06T07:26:06.755309+0000","last_active":"2024-03-06T07:26:06.760821+0000","last_peered":"2024-03-06T07:26:06.760821+0000","last_clean":"2024-03-06T07:26:06.760821+0000","last_became_active":"2024-03-06T07:26:06.754779+0000","last_became_peered":"2024-03-06T07:26:06.754779+0000","last_unstale":"2024-03-06T07:26:06.760821+0000","last_undegraded":"2024-03-06T07:26:06.760821+0000","last_fullsized":"2024-03-06T07:26:06.760821+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:39:38.575174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758999+0000","last_change":"2024-03-06T07:26:06.747982+0000","last_active":"2024-03-06T07:26:06.758999+0000","last_peered":"2024-03-06T07:26:06.758999+0000","last_clean":"2024-03-06T07:26:06.758999+0000","last_became_active":"2024-03-06T07:26:06.747725+0000","last_became_peered":"2024-03-06T07:26:06.747725+0000","last_unstale":"2024-03-06T07:26:06.758999+0000","last_undegraded":"2024-03-06T07:26:06.758999+0000","last_fullsized":"2024-03-06T07:26:06.758999+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758412+0000","last_change":"2024-03-06T07:26:06.752903+0000","last_active":"2024-03-06T07:26:06.758412+0000","last_peered":"2024-03-06T07:26:06.758412+0000","last_clean":"2024-03-06T07:26:06.758412+0000","last_became_active":"2024-03-06T07:26:06.752570+0000","last_became_peered":"2024-03-06T07:26:06.752570+0000","last_unstale":"2024-03-06T07:26:06.758412+0000","last_undegraded":"2024-03-06T07:26:06.758412+0000","last_fullsized":"2024-03-06T07:26:06.758412+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:33.831911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.762811+0000","last_change":"2024-03-06T07:26:06.757721+0000","last_active":"2024-03-06T07:26:06.762811+0000","last_peered":"2024-03-06T07:26:06.762811+0000","last_clean":"2024-03-06T07:26:06.762811+0000","last_became_active":"2024-03-06T07:26:06.757474+0000","last_became_peered":"2024-03-06T07:26:06.757474+0000","last_unstale":"2024-03-06T07:26:06.762811+0000","last_undegraded":"2024-03-06T07:26:06.762811+0000","last_fullsized":"2024-03-06T07:26:06.762811+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:48.424079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.762842+0000","last_change":"2024-03-06T07:26:06.758597+0000","last_active":"2024-03-06T07:26:06.762842+0000","last_peered":"2024-03-06T07:26:06.762842+0000","last_clean":"2024-03-06T07:26:06.762842+0000","last_became_active":"2024-03-06T07:26:06.758350+0000","last_became_peered":"2024-03-06T07:26:06.758350+0000","last_unstale":"2024-03-06T07:26:06.762842+0000","last_undegraded":"2024-03-06T07:26:06.762842+0000","last_fullsized":"2024-03-06T07:26:06.762842+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:28.601267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.748888+0000","last_change":"2024-03-06T07:26:06.745213+0000","last_active":"2024-03-06T07:26:06.748888+0000","last_peered":"2024-03-06T07:26:06.748888+0000","last_clean":"2024-03-06T07:26:06.748888+0000","last_became_active":"2024-03-06T07:26:06.744964+0000","last_became_peered":"2024-03-06T07:26:06.744964+0000","last_unstale":"2024-03-06T07:26:06.748888+0000","last_undegraded":"2024-03-06T07:26:06.748888+0000","last_fullsized":"2024-03-06T07:26:06.748888+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.410173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751205+0000","last_change":"2024-03-06T07:26:06.748975+0000","last_active":"2024-03-06T07:26:06.751205+0000","last_peered":"2024-03-06T07:26:06.751205+0000","last_clean":"2024-03-06T07:26:06.751205+0000","last_became_active":"2024-03-06T07:26:06.748535+0000","last_became_peered":"2024-03-06T07:26:06.748535+0000","last_unstale":"2024-03-06T07:26:06.751205+0000","last_undegraded":"2024-03-06T07:26:06.751205+0000","last_fullsized":"2024-03-06T07:26:06.751205+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:45.599226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759968+0000","last_change":"2024-03-06T07:26:06.756179+0000","last_active":"2024-03-06T07:26:06.759968+0000","last_peered":"2024-03-06T07:26:06.759968+0000","last_clean":"2024-03-06T07:26:06.759968+0000","last_became_active":"2024-03-06T07:26:06.755652+0000","last_became_peered":"2024-03-06T07:26:06.755652+0000","last_unstale":"2024-03-06T07:26:06.759968+0000","last_undegraded":"2024-03-06T07:26:06.759968+0000","last_fullsized":"2024-03-06T07:26:06.759968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.563016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.756124+0000","last_change":"2024-03-06T07:26:06.753074+0000","last_active":"2024-03-06T07:26:06.756124+0000","last_peered":"2024-03-06T07:26:06.756124+0000","last_clean":"2024-03-06T07:26:06.756124+0000","last_became_active":"2024-03-06T07:26:06.752758+0000","last_became_peered":"2024-03-06T07:26:06.752758+0000","last_unstale":"2024-03-06T07:26:06.756124+0000","last_undegraded":"2024-03-06T07:26:06.756124+0000","last_fullsized":"2024-03-06T07:26:06.756124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:36.743823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766462+0000","last_change":"2024-03-06T07:26:06.755897+0000","last_active":"2024-03-06T07:26:06.766462+0000","last_peered":"2024-03-06T07:26:06.766462+0000","last_clean":"2024-03-06T07:26:06.766462+0000","last_became_active":"2024-03-06T07:26:06.755588+0000","last_became_peered":"2024-03-06T07:26:06.755588+0000","last_unstale":"2024-03-06T07:26:06.766462+0000","last_undegraded":"2024-03-06T07:26:06.766462+0000","last_fullsized":"2024-03-06T07:26:06.766462+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759416+0000","last_change":"2024-03-06T07:26:06.750968+0000","last_active":"2024-03-06T07:26:06.759416+0000","last_peered":"2024-03-06T07:26:06.759416+0000","last_clean":"2024-03-06T07:26:06.759416+0000","last_became_active":"2024-03-06T07:26:06.750666+0000","last_became_peered":"2024-03-06T07:26:06.750666+0000","last_unstale":"2024-03-06T07:26:06.759416+0000","last_undegraded":"2024-03-06T07:26:06.759416+0000","last_fullsized":"2024-03-06T07:26:06.759416+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:22.563016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763262+0000","last_change":"2024-03-06T07:26:06.756612+0000","last_active":"2024-03-06T07:26:06.763262+0000","last_peered":"2024-03-06T07:26:06.763262+0000","last_clean":"2024-03-06T07:26:06.763262+0000","last_became_active":"2024-03-06T07:26:06.756359+0000","last_became_peered":"2024-03-06T07:26:06.756359+0000","last_unstale":"2024-03-06T07:26:06.763262+0000","last_undegraded":"2024-03-06T07:26:06.763262+0000","last_fullsized":"2024-03-06T07:26:06.763262+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:08:39.907630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760545+0000","last_change":"2024-03-06T07:26:06.753497+0000","last_active":"2024-03-06T07:26:06.760545+0000","last_peered":"2024-03-06T07:26:06.760545+0000","last_clean":"2024-03-06T07:26:06.760545+0000","last_became_active":"2024-03-06T07:26:06.753226+0000","last_became_peered":"2024-03-06T07:26:06.753226+0000","last_unstale":"2024-03-06T07:26:06.760545+0000","last_undegraded":"2024-03-06T07:26:06.760545+0000","last_fullsized":"2024-03-06T07:26:06.760545+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:00.479923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757934+0000","last_change":"2024-03-06T07:26:06.755415+0000","last_active":"2024-03-06T07:26:06.757934+0000","last_peered":"2024-03-06T07:26:06.757934+0000","last_clean":"2024-03-06T07:26:06.757934+0000","last_became_active":"2024-03-06T07:26:06.754809+0000","last_became_peered":"2024-03-06T07:26:06.754809+0000","last_unstale":"2024-03-06T07:26:06.757934+0000","last_undegraded":"2024-03-06T07:26:06.757934+0000","last_fullsized":"2024-03-06T07:26:06.757934+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.019146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744102+0000","last_change":"2024-03-06T07:26:06.741669+0000","last_active":"2024-03-06T07:26:06.744102+0000","last_peered":"2024-03-06T07:26:06.744102+0000","last_clean":"2024-03-06T07:26:06.744102+0000","last_became_active":"2024-03-06T07:26:06.741394+0000","last_became_peered":"2024-03-06T07:26:06.741394+0000","last_unstale":"2024-03-06T07:26:06.744102+0000","last_undegraded":"2024-03-06T07:26:06.744102+0000","last_fullsized":"2024-03-06T07:26:06.744102+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:48:23.552011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758936+0000","last_change":"2024-03-06T07:26:06.747177+0000","last_active":"2024-03-06T07:26:06.758936+0000","last_peered":"2024-03-06T07:26:06.758936+0000","last_clean":"2024-03-06T07:26:06.758936+0000","last_became_active":"2024-03-06T07:26:06.746884+0000","last_became_peered":"2024-03-06T07:26:06.746884+0000","last_unstale":"2024-03-06T07:26:06.758936+0000","last_undegraded":"2024-03-06T07:26:06.758936+0000","last_fullsized":"2024-03-06T07:26:06.758936+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:22.384294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763382+0000","last_change":"2024-03-06T07:26:06.757075+0000","last_active":"2024-03-06T07:26:06.763382+0000","last_peered":"2024-03-06T07:26:06.763382+0000","last_clean":"2024-03-06T07:26:06.763382+0000","last_became_active":"2024-03-06T07:26:06.756773+0000","last_became_peered":"2024-03-06T07:26:06.756773+0000","last_unstale":"2024-03-06T07:26:06.763382+0000","last_undegraded":"2024-03-06T07:26:06.763382+0000","last_fullsized":"2024-03-06T07:26:06.763382+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:24:39.997072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752142+0000","last_change":"2024-03-06T07:26:06.749663+0000","last_active":"2024-03-06T07:26:06.752142+0000","last_peered":"2024-03-06T07:26:06.752142+0000","last_clean":"2024-03-06T07:26:06.752142+0000","last_became_active":"2024-03-06T07:26:06.749363+0000","last_became_peered":"2024-03-06T07:26:06.749363+0000","last_unstale":"2024-03-06T07:26:06.752142+0000","last_undegraded":"2024-03-06T07:26:06.752142+0000","last_fullsized":"2024-03-06T07:26:06.752142+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:53.775890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763091+0000","last_change":"2024-03-06T07:26:06.755229+0000","last_active":"2024-03-06T07:26:06.763091+0000","last_peered":"2024-03-06T07:26:06.763091+0000","last_clean":"2024-03-06T07:26:06.763091+0000","last_became_active":"2024-03-06T07:26:06.754970+0000","last_became_peered":"2024-03-06T07:26:06.754970+0000","last_unstale":"2024-03-06T07:26:06.763091+0000","last_undegraded":"2024-03-06T07:26:06.763091+0000","last_fullsized":"2024-03-06T07:26:06.763091+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:06.038831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752150+0000","last_change":"2024-03-06T07:26:06.749499+0000","last_active":"2024-03-06T07:26:06.752150+0000","last_peered":"2024-03-06T07:26:06.752150+0000","last_clean":"2024-03-06T07:26:06.752150+0000","last_became_active":"2024-03-06T07:26:06.749254+0000","last_became_peered":"2024-03-06T07:26:06.749254+0000","last_unstale":"2024-03-06T07:26:06.752150+0000","last_undegraded":"2024-03-06T07:26:06.752150+0000","last_fullsized":"2024-03-06T07:26:06.752150+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:14:58.564900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744132+0000","last_change":"2024-03-06T07:26:06.741957+0000","last_active":"2024-03-06T07:26:06.744132+0000","last_peered":"2024-03-06T07:26:06.744132+0000","last_clean":"2024-03-06T07:26:06.744132+0000","last_became_active":"2024-03-06T07:26:06.741698+0000","last_became_peered":"2024-03-06T07:26:06.741698+0000","last_unstale":"2024-03-06T07:26:06.744132+0000","last_undegraded":"2024-03-06T07:26:06.744132+0000","last_fullsized":"2024-03-06T07:26:06.744132+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:50.726652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757959+0000","last_change":"2024-03-06T07:26:06.755628+0000","last_active":"2024-03-06T07:26:06.757959+0000","last_peered":"2024-03-06T07:26:06.757959+0000","last_clean":"2024-03-06T07:26:06.757959+0000","last_became_active":"2024-03-06T07:26:06.755169+0000","last_became_peered":"2024-03-06T07:26:06.755169+0000","last_unstale":"2024-03-06T07:26:06.757959+0000","last_undegraded":"2024-03-06T07:26:06.757959+0000","last_fullsized":"2024-03-06T07:26:06.757959+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:17.823923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760383+0000","last_change":"2024-03-06T07:26:06.756124+0000","last_active":"2024-03-06T07:26:06.760383+0000","last_peered":"2024-03-06T07:26:06.760383+0000","last_clean":"2024-03-06T07:26:06.760383+0000","last_became_active":"2024-03-06T07:26:06.755722+0000","last_became_peered":"2024-03-06T07:26:06.755722+0000","last_unstale":"2024-03-06T07:26:06.760383+0000","last_undegraded":"2024-03-06T07:26:06.760383+0000","last_fullsized":"2024-03-06T07:26:06.760383+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:37:00.479923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757490+0000","last_change":"2024-03-06T07:26:06.753547+0000","last_active":"2024-03-06T07:26:06.757490+0000","last_peered":"2024-03-06T07:26:06.757490+0000","last_clean":"2024-03-06T07:26:06.757490+0000","last_became_active":"2024-03-06T07:26:06.753297+0000","last_became_peered":"2024-03-06T07:26:06.753297+0000","last_unstale":"2024-03-06T07:26:06.757490+0000","last_undegraded":"2024-03-06T07:26:06.757490+0000","last_fullsized":"2024-03-06T07:26:06.757490+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:41:56.246569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760663+0000","last_change":"2024-03-06T07:26:06.757035+0000","last_active":"2024-03-06T07:26:06.760663+0000","last_peered":"2024-03-06T07:26:06.760663+0000","last_clean":"2024-03-06T07:26:06.760663+0000","last_became_active":"2024-03-06T07:26:06.756775+0000","last_became_peered":"2024-03-06T07:26:06.756775+0000","last_unstale":"2024-03-06T07:26:06.760663+0000","last_undegraded":"2024-03-06T07:26:06.760663+0000","last_fullsized":"2024-03-06T07:26:06.760663+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:29.493414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766464+0000","last_change":"2024-03-06T07:26:06.763056+0000","last_active":"2024-03-06T07:26:06.766464+0000","last_peered":"2024-03-06T07:26:06.766464+0000","last_clean":"2024-03-06T07:26:06.766464+0000","last_became_active":"2024-03-06T07:26:06.762378+0000","last_became_peered":"2024-03-06T07:26:06.762378+0000","last_unstale":"2024-03-06T07:26:06.766464+0000","last_undegraded":"2024-03-06T07:26:06.766464+0000","last_fullsized":"2024-03-06T07:26:06.766464+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:53.952562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760637+0000","last_change":"2024-03-06T07:26:06.754255+0000","last_active":"2024-03-06T07:26:06.760637+0000","last_peered":"2024-03-06T07:26:06.760637+0000","last_clean":"2024-03-06T07:26:06.760637+0000","last_became_active":"2024-03-06T07:26:06.754001+0000","last_became_peered":"2024-03-06T07:26:06.754001+0000","last_unstale":"2024-03-06T07:26:06.760637+0000","last_undegraded":"2024-03-06T07:26:06.760637+0000","last_fullsized":"2024-03-06T07:26:06.760637+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:35:38.436587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763343+0000","last_change":"2024-03-06T07:26:06.756947+0000","last_active":"2024-03-06T07:26:06.763343+0000","last_peered":"2024-03-06T07:26:06.763343+0000","last_clean":"2024-03-06T07:26:06.763343+0000","last_became_active":"2024-03-06T07:26:06.756687+0000","last_became_peered":"2024-03-06T07:26:06.756687+0000","last_unstale":"2024-03-06T07:26:06.763343+0000","last_undegraded":"2024-03-06T07:26:06.763343+0000","last_fullsized":"2024-03-06T07:26:06.763343+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:18:53.253727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763198+0000","last_change":"2024-03-06T07:26:06.757991+0000","last_active":"2024-03-06T07:26:06.763198+0000","last_peered":"2024-03-06T07:26:06.763198+0000","last_clean":"2024-03-06T07:26:06.763198+0000","last_became_active":"2024-03-06T07:26:06.757736+0000","last_became_peered":"2024-03-06T07:26:06.757736+0000","last_unstale":"2024-03-06T07:26:06.763198+0000","last_undegraded":"2024-03-06T07:26:06.763198+0000","last_fullsized":"2024-03-06T07:26:06.763198+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:53.952562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757252+0000","last_change":"2024-03-06T07:26:06.753405+0000","last_active":"2024-03-06T07:26:06.757252+0000","last_peered":"2024-03-06T07:26:06.757252+0000","last_clean":"2024-03-06T07:26:06.757252+0000","last_became_active":"2024-03-06T07:26:06.753138+0000","last_became_peered":"2024-03-06T07:26:06.753138+0000","last_unstale":"2024-03-06T07:26:06.757252+0000","last_undegraded":"2024-03-06T07:26:06.757252+0000","last_fullsized":"2024-03-06T07:26:06.757252+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:43:08.634015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751849+0000","last_change":"2024-03-06T07:26:06.747617+0000","last_active":"2024-03-06T07:26:06.751849+0000","last_peered":"2024-03-06T07:26:06.751849+0000","last_clean":"2024-03-06T07:26:06.751849+0000","last_became_active":"2024-03-06T07:26:06.747358+0000","last_became_peered":"2024-03-06T07:26:06.747358+0000","last_unstale":"2024-03-06T07:26:06.751849+0000","last_undegraded":"2024-03-06T07:26:06.751849+0000","last_fullsized":"2024-03-06T07:26:06.751849+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:45.860295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"93'4","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.768968+0000","last_change":"2024-03-06T07:26:05.753683+0000","last_active":"2024-03-06T07:26:06.768968+0000","last_peered":"2024-03-06T07:26:06.768968+0000","last_clean":"2024-03-06T07:26:06.768968+0000","last_became_active":"2024-03-06T07:26:05.753411+0000","last_became_peered":"2024-03-06T07:26:05.753411+0000","last_unstale":"2024-03-06T07:26:06.768968+0000","last_undegraded":"2024-03-06T07:26:06.768968+0000","last_fullsized":"2024-03-06T07:26:06.768968+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:34:47.419422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763166+0000","last_change":"2024-03-06T07:26:06.758626+0000","last_active":"2024-03-06T07:26:06.763166+0000","last_peered":"2024-03-06T07:26:06.763166+0000","last_clean":"2024-03-06T07:26:06.763166+0000","last_became_active":"2024-03-06T07:26:06.758377+0000","last_became_peered":"2024-03-06T07:26:06.758377+0000","last_unstale":"2024-03-06T07:26:06.763166+0000","last_undegraded":"2024-03-06T07:26:06.763166+0000","last_fullsized":"2024-03-06T07:26:06.763166+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.019146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766220+0000","last_change":"2024-03-06T07:26:05.747448+0000","last_active":"2024-03-06T07:26:06.766220+0000","last_peered":"2024-03-06T07:26:06.766220+0000","last_clean":"2024-03-06T07:26:06.766220+0000","last_became_active":"2024-03-06T07:26:05.746429+0000","last_became_peered":"2024-03-06T07:26:05.746429+0000","last_unstale":"2024-03-06T07:26:06.766220+0000","last_undegraded":"2024-03-06T07:26:06.766220+0000","last_fullsized":"2024-03-06T07:26:06.766220+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:32.112619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757223+0000","last_change":"2024-03-06T07:26:06.753040+0000","last_active":"2024-03-06T07:26:06.757223+0000","last_peered":"2024-03-06T07:26:06.757223+0000","last_clean":"2024-03-06T07:26:06.757223+0000","last_became_active":"2024-03-06T07:26:06.752712+0000","last_became_peered":"2024-03-06T07:26:06.752712+0000","last_unstale":"2024-03-06T07:26:06.757223+0000","last_undegraded":"2024-03-06T07:26:06.757223+0000","last_fullsized":"2024-03-06T07:26:06.757223+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:33:20.334563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.764806+0000","last_change":"2024-03-06T07:26:05.752588+0000","last_active":"2024-03-06T07:26:06.764806+0000","last_peered":"2024-03-06T07:26:06.764806+0000","last_clean":"2024-03-06T07:26:06.764806+0000","last_became_active":"2024-03-06T07:26:05.752268+0000","last_became_peered":"2024-03-06T07:26:05.752268+0000","last_unstale":"2024-03-06T07:26:06.764806+0000","last_undegraded":"2024-03-06T07:26:06.764806+0000","last_fullsized":"2024-03-06T07:26:06.764806+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:43.095761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757831+0000","last_change":"2024-03-06T07:26:06.753569+0000","last_active":"2024-03-06T07:26:06.757831+0000","last_peered":"2024-03-06T07:26:06.757831+0000","last_clean":"2024-03-06T07:26:06.757831+0000","last_became_active":"2024-03-06T07:26:06.753273+0000","last_became_peered":"2024-03-06T07:26:06.753273+0000","last_unstale":"2024-03-06T07:26:06.757831+0000","last_undegraded":"2024-03-06T07:26:06.757831+0000","last_fullsized":"2024-03-06T07:26:06.757831+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:33.831911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766851+0000","last_change":"2024-03-06T07:26:05.753652+0000","last_active":"2024-03-06T07:26:06.766851+0000","last_peered":"2024-03-06T07:26:06.766851+0000","last_clean":"2024-03-06T07:26:06.766851+0000","last_became_active":"2024-03-06T07:26:05.753396+0000","last_became_peered":"2024-03-06T07:26:05.753396+0000","last_unstale":"2024-03-06T07:26:06.766851+0000","last_undegraded":"2024-03-06T07:26:06.766851+0000","last_fullsized":"2024-03-06T07:26:06.766851+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:41.574764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.754569+0000","last_change":"2024-03-06T07:26:06.750748+0000","last_active":"2024-03-06T07:26:06.754569+0000","last_peered":"2024-03-06T07:26:06.754569+0000","last_clean":"2024-03-06T07:26:06.754569+0000","last_became_active":"2024-03-06T07:26:06.750461+0000","last_became_peered":"2024-03-06T07:26:06.750461+0000","last_unstale":"2024-03-06T07:26:06.754569+0000","last_undegraded":"2024-03-06T07:26:06.754569+0000","last_fullsized":"2024-03-06T07:26:06.754569+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766615+0000","last_change":"2024-03-06T07:26:05.746652+0000","last_active":"2024-03-06T07:26:06.766615+0000","last_peered":"2024-03-06T07:26:06.766615+0000","last_clean":"2024-03-06T07:26:06.766615+0000","last_became_active":"2024-03-06T07:26:05.746392+0000","last_became_peered":"2024-03-06T07:26:05.746392+0000","last_unstale":"2024-03-06T07:26:06.766615+0000","last_undegraded":"2024-03-06T07:26:06.766615+0000","last_fullsized":"2024-03-06T07:26:06.766615+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:44:53.349741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769033+0000","last_change":"2024-03-06T07:26:06.765254+0000","last_active":"2024-03-06T07:26:06.769033+0000","last_peered":"2024-03-06T07:26:06.769033+0000","last_clean":"2024-03-06T07:26:06.769033+0000","last_became_active":"2024-03-06T07:26:06.764988+0000","last_became_peered":"2024-03-06T07:26:06.764988+0000","last_unstale":"2024-03-06T07:26:06.769033+0000","last_undegraded":"2024-03-06T07:26:06.769033+0000","last_fullsized":"2024-03-06T07:26:06.769033+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:44.100418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.764757+0000","last_change":"2024-03-06T07:26:05.752942+0000","last_active":"2024-03-06T07:26:06.764757+0000","last_peered":"2024-03-06T07:26:06.764757+0000","last_clean":"2024-03-06T07:26:06.764757+0000","last_became_active":"2024-03-06T07:26:05.752178+0000","last_became_peered":"2024-03-06T07:26:05.752178+0000","last_unstale":"2024-03-06T07:26:06.764757+0000","last_undegraded":"2024-03-06T07:26:06.764757+0000","last_fullsized":"2024-03-06T07:26:06.764757+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:45:27.596609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760584+0000","last_change":"2024-03-06T07:26:06.753593+0000","last_active":"2024-03-06T07:26:06.760584+0000","last_peered":"2024-03-06T07:26:06.760584+0000","last_clean":"2024-03-06T07:26:06.760584+0000","last_became_active":"2024-03-06T07:26:06.753272+0000","last_became_peered":"2024-03-06T07:26:06.753272+0000","last_unstale":"2024-03-06T07:26:06.760584+0000","last_undegraded":"2024-03-06T07:26:06.760584+0000","last_fullsized":"2024-03-06T07:26:06.760584+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:36.743823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.754078+0000","last_change":"2024-03-06T07:26:05.750175+0000","last_active":"2024-03-06T07:26:06.754078+0000","last_peered":"2024-03-06T07:26:06.754078+0000","last_clean":"2024-03-06T07:26:06.754078+0000","last_became_active":"2024-03-06T07:26:05.749834+0000","last_became_peered":"2024-03-06T07:26:05.749834+0000","last_unstale":"2024-03-06T07:26:06.754078+0000","last_undegraded":"2024-03-06T07:26:06.754078+0000","last_fullsized":"2024-03-06T07:26:06.754078+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:21.379637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763990+0000","last_change":"2024-03-06T07:26:06.759980+0000","last_active":"2024-03-06T07:26:06.763990+0000","last_peered":"2024-03-06T07:26:06.763990+0000","last_clean":"2024-03-06T07:26:06.763990+0000","last_became_active":"2024-03-06T07:26:06.759714+0000","last_became_peered":"2024-03-06T07:26:06.759714+0000","last_unstale":"2024-03-06T07:26:06.763990+0000","last_undegraded":"2024-03-06T07:26:06.763990+0000","last_fullsized":"2024-03-06T07:26:06.763990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.745241+0000","last_change":"2024-03-06T07:26:05.747778+0000","last_active":"2024-03-06T07:26:06.745241+0000","last_peered":"2024-03-06T07:26:06.745241+0000","last_clean":"2024-03-06T07:26:06.745241+0000","last_became_active":"2024-03-06T07:26:05.746734+0000","last_became_peered":"2024-03-06T07:26:05.746734+0000","last_unstale":"2024-03-06T07:26:06.745241+0000","last_undegraded":"2024-03-06T07:26:06.745241+0000","last_fullsized":"2024-03-06T07:26:06.745241+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:02:39.197150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752185+0000","last_change":"2024-03-06T07:26:06.750823+0000","last_active":"2024-03-06T07:26:06.752185+0000","last_peered":"2024-03-06T07:26:06.752185+0000","last_clean":"2024-03-06T07:26:06.752185+0000","last_became_active":"2024-03-06T07:26:06.750542+0000","last_became_peered":"2024-03-06T07:26:06.750542+0000","last_unstale":"2024-03-06T07:26:06.752185+0000","last_undegraded":"2024-03-06T07:26:06.752185+0000","last_fullsized":"2024-03-06T07:26:06.752185+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:07:33.537888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.745260+0000","last_change":"2024-03-06T07:26:05.743411+0000","last_active":"2024-03-06T07:26:06.745260+0000","last_peered":"2024-03-06T07:26:06.745260+0000","last_clean":"2024-03-06T07:26:06.745260+0000","last_became_active":"2024-03-06T07:26:05.743135+0000","last_became_peered":"2024-03-06T07:26:05.743135+0000","last_unstale":"2024-03-06T07:26:06.745260+0000","last_undegraded":"2024-03-06T07:26:06.745260+0000","last_fullsized":"2024-03-06T07:26:06.745260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:44:35.323058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755443+0000","last_change":"2024-03-06T07:26:06.753282+0000","last_active":"2024-03-06T07:26:06.755443+0000","last_peered":"2024-03-06T07:26:06.755443+0000","last_clean":"2024-03-06T07:26:06.755443+0000","last_became_active":"2024-03-06T07:26:06.752990+0000","last_became_peered":"2024-03-06T07:26:06.752990+0000","last_unstale":"2024-03-06T07:26:06.755443+0000","last_undegraded":"2024-03-06T07:26:06.755443+0000","last_fullsized":"2024-03-06T07:26:06.755443+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:21.523545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738001+0000","last_change":"2024-03-06T07:26:05.750497+0000","last_active":"2024-03-06T07:26:06.738001+0000","last_peered":"2024-03-06T07:26:06.738001+0000","last_clean":"2024-03-06T07:26:06.738001+0000","last_became_active":"2024-03-06T07:26:05.750222+0000","last_became_peered":"2024-03-06T07:26:05.750222+0000","last_unstale":"2024-03-06T07:26:06.738001+0000","last_undegraded":"2024-03-06T07:26:06.738001+0000","last_fullsized":"2024-03-06T07:26:06.738001+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:32.827254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758307+0000","last_change":"2024-03-06T07:26:06.753505+0000","last_active":"2024-03-06T07:26:06.758307+0000","last_peered":"2024-03-06T07:26:06.758307+0000","last_clean":"2024-03-06T07:26:06.758307+0000","last_became_active":"2024-03-06T07:26:06.749927+0000","last_became_peered":"2024-03-06T07:26:06.749927+0000","last_unstale":"2024-03-06T07:26:06.758307+0000","last_undegraded":"2024-03-06T07:26:06.758307+0000","last_fullsized":"2024-03-06T07:26:06.758307+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:36.242566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"93'3","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769135+0000","last_change":"2024-03-06T07:26:05.745579+0000","last_active":"2024-03-06T07:26:06.769135+0000","last_peered":"2024-03-06T07:26:06.769135+0000","last_clean":"2024-03-06T07:26:06.769135+0000","last_became_active":"2024-03-06T07:26:05.745282+0000","last_became_peered":"2024-03-06T07:26:05.745282+0000","last_unstale":"2024-03-06T07:26:06.769135+0000","last_undegraded":"2024-03-06T07:26:06.769135+0000","last_fullsized":"2024-03-06T07:26:06.769135+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:36:59.475266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758090+0000","last_change":"2024-03-06T07:26:06.756276+0000","last_active":"2024-03-06T07:26:06.758090+0000","last_peered":"2024-03-06T07:26:06.758090+0000","last_clean":"2024-03-06T07:26:06.758090+0000","last_became_active":"2024-03-06T07:26:06.755934+0000","last_became_peered":"2024-03-06T07:26:06.755934+0000","last_unstale":"2024-03-06T07:26:06.758090+0000","last_undegraded":"2024-03-06T07:26:06.758090+0000","last_fullsized":"2024-03-06T07:26:06.758090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:22:42.579421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"93'6","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.773458+0000","last_change":"2024-03-06T07:26:05.746883+0000","last_active":"2024-03-06T07:26:06.773458+0000","last_peered":"2024-03-06T07:26:06.773458+0000","last_clean":"2024-03-06T07:26:06.773458+0000","last_became_active":"2024-03-06T07:26:05.746601+0000","last_became_peered":"2024-03-06T07:26:05.746601+0000","last_unstale":"2024-03-06T07:26:06.773458+0000","last_undegraded":"2024-03-06T07:26:06.773458+0000","last_fullsized":"2024-03-06T07:26:06.773458+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:19.017740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759175+0000","last_change":"2024-03-06T07:26:06.749160+0000","last_active":"2024-03-06T07:26:06.759175+0000","last_peered":"2024-03-06T07:26:06.759175+0000","last_clean":"2024-03-06T07:26:06.759175+0000","last_became_active":"2024-03-06T07:26:06.748281+0000","last_became_peered":"2024-03-06T07:26:06.748281+0000","last_unstale":"2024-03-06T07:26:06.759175+0000","last_undegraded":"2024-03-06T07:26:06.759175+0000","last_fullsized":"2024-03-06T07:26:06.759175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766440+0000","last_change":"2024-03-06T07:26:05.749138+0000","last_active":"2024-03-06T07:26:06.766440+0000","last_peered":"2024-03-06T07:26:06.766440+0000","last_clean":"2024-03-06T07:26:06.766440+0000","last_became_active":"2024-03-06T07:26:05.748852+0000","last_became_peered":"2024-03-06T07:26:05.748852+0000","last_unstale":"2024-03-06T07:26:06.766440+0000","last_undegraded":"2024-03-06T07:26:06.766440+0000","last_fullsized":"2024-03-06T07:26:06.766440+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:54:54.242958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.757455+0000","last_change":"2024-03-06T07:26:06.753914+0000","last_active":"2024-03-06T07:26:06.757455+0000","last_peered":"2024-03-06T07:26:06.757455+0000","last_clean":"2024-03-06T07:26:06.757455+0000","last_became_active":"2024-03-06T07:26:06.753579+0000","last_became_peered":"2024-03-06T07:26:06.753579+0000","last_unstale":"2024-03-06T07:26:06.757455+0000","last_undegraded":"2024-03-06T07:26:06.757455+0000","last_fullsized":"2024-03-06T07:26:06.757455+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:27:43.131393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.744541+0000","last_change":"2024-03-06T07:26:05.750634+0000","last_active":"2024-03-06T07:26:06.744541+0000","last_peered":"2024-03-06T07:26:06.744541+0000","last_clean":"2024-03-06T07:26:06.744541+0000","last_became_active":"2024-03-06T07:26:05.749812+0000","last_became_peered":"2024-03-06T07:26:05.749812+0000","last_unstale":"2024-03-06T07:26:06.744541+0000","last_undegraded":"2024-03-06T07:26:06.744541+0000","last_fullsized":"2024-03-06T07:26:06.744541+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:30:02.117850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755471+0000","last_change":"2024-03-06T07:26:06.754534+0000","last_active":"2024-03-06T07:26:06.755471+0000","last_peered":"2024-03-06T07:26:06.755471+0000","last_clean":"2024-03-06T07:26:06.755471+0000","last_became_active":"2024-03-06T07:26:06.753676+0000","last_became_peered":"2024-03-06T07:26:06.753676+0000","last_unstale":"2024-03-06T07:26:06.755471+0000","last_undegraded":"2024-03-06T07:26:06.755471+0000","last_fullsized":"2024-03-06T07:26:06.755471+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:36.242566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.743266+0000","last_change":"2024-03-06T07:26:05.746612+0000","last_active":"2024-03-06T07:26:06.743266+0000","last_peered":"2024-03-06T07:26:06.743266+0000","last_clean":"2024-03-06T07:26:06.743266+0000","last_became_active":"2024-03-06T07:26:05.746302+0000","last_became_peered":"2024-03-06T07:26:05.746302+0000","last_unstale":"2024-03-06T07:26:06.743266+0000","last_undegraded":"2024-03-06T07:26:06.743266+0000","last_fullsized":"2024-03-06T07:26:06.743266+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:57:15.679331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758492+0000","last_change":"2024-03-06T07:26:06.753399+0000","last_active":"2024-03-06T07:26:06.758492+0000","last_peered":"2024-03-06T07:26:06.758492+0000","last_clean":"2024-03-06T07:26:06.758492+0000","last_became_active":"2024-03-06T07:26:06.753115+0000","last_became_peered":"2024-03-06T07:26:06.753115+0000","last_unstale":"2024-03-06T07:26:06.758492+0000","last_undegraded":"2024-03-06T07:26:06.758492+0000","last_fullsized":"2024-03-06T07:26:06.758492+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:50:56.310166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"93'8","reported_seq":20,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769256+0000","last_change":"2024-03-06T07:26:05.740200+0000","last_active":"2024-03-06T07:26:06.769256+0000","last_peered":"2024-03-06T07:26:06.769256+0000","last_clean":"2024-03-06T07:26:06.769256+0000","last_became_active":"2024-03-06T07:26:05.739902+0000","last_became_peered":"2024-03-06T07:26:05.739902+0000","last_unstale":"2024-03-06T07:26:06.769256+0000","last_undegraded":"2024-03-06T07:26:06.769256+0000","last_fullsized":"2024-03-06T07:26:06.769256+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:48:50.744728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760404+0000","last_change":"2024-03-06T07:26:06.756866+0000","last_active":"2024-03-06T07:26:06.760404+0000","last_peered":"2024-03-06T07:26:06.760404+0000","last_clean":"2024-03-06T07:26:06.760404+0000","last_became_active":"2024-03-06T07:26:06.756523+0000","last_became_peered":"2024-03-06T07:26:06.756523+0000","last_unstale":"2024-03-06T07:26:06.760404+0000","last_undegraded":"2024-03-06T07:26:06.760404+0000","last_fullsized":"2024-03-06T07:26:06.760404+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:36.743823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.769472+0000","last_change":"2024-03-06T07:26:05.755349+0000","last_active":"2024-03-06T07:26:06.769472+0000","last_peered":"2024-03-06T07:26:06.769472+0000","last_clean":"2024-03-06T07:26:06.769472+0000","last_became_active":"2024-03-06T07:26:05.755089+0000","last_became_peered":"2024-03-06T07:26:05.755089+0000","last_unstale":"2024-03-06T07:26:06.769472+0000","last_undegraded":"2024-03-06T07:26:06.769472+0000","last_fullsized":"2024-03-06T07:26:06.769472+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:34:56.850597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755148+0000","last_change":"2024-03-06T07:26:06.753872+0000","last_active":"2024-03-06T07:26:06.755148+0000","last_peered":"2024-03-06T07:26:06.755148+0000","last_clean":"2024-03-06T07:26:06.755148+0000","last_became_active":"2024-03-06T07:26:06.753587+0000","last_became_peered":"2024-03-06T07:26:06.753587+0000","last_unstale":"2024-03-06T07:26:06.755148+0000","last_undegraded":"2024-03-06T07:26:06.755148+0000","last_fullsized":"2024-03-06T07:26:06.755148+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.768966+0000","last_change":"2024-03-06T07:26:05.745744+0000","last_active":"2024-03-06T07:26:06.768966+0000","last_peered":"2024-03-06T07:26:06.768966+0000","last_clean":"2024-03-06T07:26:06.768966+0000","last_became_active":"2024-03-06T07:26:05.745456+0000","last_became_peered":"2024-03-06T07:26:05.745456+0000","last_unstale":"2024-03-06T07:26:06.768966+0000","last_undegraded":"2024-03-06T07:26:06.768966+0000","last_fullsized":"2024-03-06T07:26:06.768966+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:52.771232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.756144+0000","last_change":"2024-03-06T07:26:06.753267+0000","last_active":"2024-03-06T07:26:06.756144+0000","last_peered":"2024-03-06T07:26:06.756144+0000","last_clean":"2024-03-06T07:26:06.756144+0000","last_became_active":"2024-03-06T07:26:06.752853+0000","last_became_peered":"2024-03-06T07:26:06.752853+0000","last_unstale":"2024-03-06T07:26:06.756144+0000","last_undegraded":"2024-03-06T07:26:06.756144+0000","last_fullsized":"2024-03-06T07:26:06.756144+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:53.952562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738936+0000","last_change":"2024-03-06T07:26:05.750980+0000","last_active":"2024-03-06T07:26:06.738936+0000","last_peered":"2024-03-06T07:26:06.738936+0000","last_clean":"2024-03-06T07:26:06.738936+0000","last_became_active":"2024-03-06T07:26:05.750712+0000","last_became_peered":"2024-03-06T07:26:05.750712+0000","last_unstale":"2024-03-06T07:26:06.738936+0000","last_undegraded":"2024-03-06T07:26:06.738936+0000","last_fullsized":"2024-03-06T07:26:06.738936+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:55:28.488757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758564+0000","last_change":"2024-03-06T07:26:06.752363+0000","last_active":"2024-03-06T07:26:06.758564+0000","last_peered":"2024-03-06T07:26:06.758564+0000","last_clean":"2024-03-06T07:26:06.758564+0000","last_became_active":"2024-03-06T07:26:06.752110+0000","last_became_peered":"2024-03-06T07:26:06.752110+0000","last_unstale":"2024-03-06T07:26:06.758564+0000","last_undegraded":"2024-03-06T07:26:06.758564+0000","last_fullsized":"2024-03-06T07:26:06.758564+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:25:45.870576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.764854+0000","last_change":"2024-03-06T07:26:05.752024+0000","last_active":"2024-03-06T07:26:06.764854+0000","last_peered":"2024-03-06T07:26:06.764854+0000","last_clean":"2024-03-06T07:26:06.764854+0000","last_became_active":"2024-03-06T07:26:05.751691+0000","last_became_peered":"2024-03-06T07:26:05.751691+0000","last_unstale":"2024-03-06T07:26:06.764854+0000","last_undegraded":"2024-03-06T07:26:06.764854+0000","last_fullsized":"2024-03-06T07:26:06.764854+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:32.677065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.755349+0000","last_change":"2024-03-06T07:26:06.750354+0000","last_active":"2024-03-06T07:26:06.755349+0000","last_peered":"2024-03-06T07:26:06.755349+0000","last_clean":"2024-03-06T07:26:06.755349+0000","last_became_active":"2024-03-06T07:26:06.750088+0000","last_became_peered":"2024-03-06T07:26:06.750088+0000","last_unstale":"2024-03-06T07:26:06.755349+0000","last_undegraded":"2024-03-06T07:26:06.755349+0000","last_fullsized":"2024-03-06T07:26:06.755349+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:01:40.701820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.737246+0000","last_change":"2024-03-06T07:26:05.749049+0000","last_active":"2024-03-06T07:26:06.737246+0000","last_peered":"2024-03-06T07:26:06.737246+0000","last_clean":"2024-03-06T07:26:06.737246+0000","last_became_active":"2024-03-06T07:26:05.748819+0000","last_became_peered":"2024-03-06T07:26:05.748819+0000","last_unstale":"2024-03-06T07:26:06.737246+0000","last_undegraded":"2024-03-06T07:26:06.737246+0000","last_fullsized":"2024-03-06T07:26:06.737246+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:21.379637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751874+0000","last_change":"2024-03-06T07:26:06.747836+0000","last_active":"2024-03-06T07:26:06.751874+0000","last_peered":"2024-03-06T07:26:06.751874+0000","last_clean":"2024-03-06T07:26:06.751874+0000","last_became_active":"2024-03-06T07:26:06.747556+0000","last_became_peered":"2024-03-06T07:26:06.747556+0000","last_unstale":"2024-03-06T07:26:06.751874+0000","last_undegraded":"2024-03-06T07:26:06.751874+0000","last_fullsized":"2024-03-06T07:26:06.751874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:04:57.109567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.736875+0000","last_change":"2024-03-06T07:26:05.747510+0000","last_active":"2024-03-06T07:26:06.736875+0000","last_peered":"2024-03-06T07:26:06.736875+0000","last_clean":"2024-03-06T07:26:06.736875+0000","last_became_active":"2024-03-06T07:26:05.747110+0000","last_became_peered":"2024-03-06T07:26:05.747110+0000","last_unstale":"2024-03-06T07:26:06.736875+0000","last_undegraded":"2024-03-06T07:26:06.736875+0000","last_fullsized":"2024-03-06T07:26:06.736875+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:03:32.827254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760199+0000","last_change":"2024-03-06T07:26:06.756034+0000","last_active":"2024-03-06T07:26:06.760199+0000","last_peered":"2024-03-06T07:26:06.760199+0000","last_clean":"2024-03-06T07:26:06.760199+0000","last_became_active":"2024-03-06T07:26:06.755610+0000","last_became_peered":"2024-03-06T07:26:06.755610+0000","last_unstale":"2024-03-06T07:26:06.760199+0000","last_undegraded":"2024-03-06T07:26:06.760199+0000","last_fullsized":"2024-03-06T07:26:06.760199+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:33.117276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738540+0000","last_change":"2024-03-06T07:26:05.752339+0000","last_active":"2024-03-06T07:26:06.738540+0000","last_peered":"2024-03-06T07:26:06.738540+0000","last_clean":"2024-03-06T07:26:06.738540+0000","last_became_active":"2024-03-06T07:26:05.752076+0000","last_became_peered":"2024-03-06T07:26:05.752076+0000","last_unstale":"2024-03-06T07:26:06.738540+0000","last_undegraded":"2024-03-06T07:26:06.738540+0000","last_fullsized":"2024-03-06T07:26:06.738540+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:16:16.819266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.752165+0000","last_change":"2024-03-06T07:26:06.749499+0000","last_active":"2024-03-06T07:26:06.752165+0000","last_peered":"2024-03-06T07:26:06.752165+0000","last_clean":"2024-03-06T07:26:06.752165+0000","last_became_active":"2024-03-06T07:26:06.749215+0000","last_became_peered":"2024-03-06T07:26:06.749215+0000","last_unstale":"2024-03-06T07:26:06.752165+0000","last_undegraded":"2024-03-06T07:26:06.752165+0000","last_fullsized":"2024-03-06T07:26:06.752165+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:58:41.951691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"93'2","reported_seq":14,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.766594+0000","last_change":"2024-03-06T07:26:05.745430+0000","last_active":"2024-03-06T07:26:06.766594+0000","last_peered":"2024-03-06T07:26:06.766594+0000","last_clean":"2024-03-06T07:26:06.766594+0000","last_became_active":"2024-03-06T07:26:05.745144+0000","last_became_peered":"2024-03-06T07:26:05.745144+0000","last_unstale":"2024-03-06T07:26:06.766594+0000","last_undegraded":"2024-03-06T07:26:06.766594+0000","last_fullsized":"2024-03-06T07:26:06.766594+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:35.237909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.760390+0000","last_change":"2024-03-06T07:26:06.754885+0000","last_active":"2024-03-06T07:26:06.760390+0000","last_peered":"2024-03-06T07:26:06.760390+0000","last_clean":"2024-03-06T07:26:06.760390+0000","last_became_active":"2024-03-06T07:26:06.754451+0000","last_became_peered":"2024-03-06T07:26:06.754451+0000","last_unstale":"2024-03-06T07:26:06.760390+0000","last_undegraded":"2024-03-06T07:26:06.760390+0000","last_fullsized":"2024-03-06T07:26:06.760390+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:08:55.308824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.740519+0000","last_change":"2024-03-06T07:26:05.751917+0000","last_active":"2024-03-06T07:26:06.740519+0000","last_peered":"2024-03-06T07:26:06.740519+0000","last_clean":"2024-03-06T07:26:06.740519+0000","last_became_active":"2024-03-06T07:26:05.751694+0000","last_became_peered":"2024-03-06T07:26:05.751694+0000","last_unstale":"2024-03-06T07:26:06.740519+0000","last_undegraded":"2024-03-06T07:26:06.740519+0000","last_fullsized":"2024-03-06T07:26:06.740519+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:05:23.173532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.761630+0000","last_change":"2024-03-06T07:26:06.756363+0000","last_active":"2024-03-06T07:26:06.761630+0000","last_peered":"2024-03-06T07:26:06.761630+0000","last_clean":"2024-03-06T07:26:06.761630+0000","last_became_active":"2024-03-06T07:26:06.756057+0000","last_became_peered":"2024-03-06T07:26:06.756057+0000","last_unstale":"2024-03-06T07:26:06.761630+0000","last_undegraded":"2024-03-06T07:26:06.761630+0000","last_fullsized":"2024-03-06T07:26:06.761630+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:41:22.772489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.740287+0000","last_change":"2024-03-06T07:26:05.753999+0000","last_active":"2024-03-06T07:26:06.740287+0000","last_peered":"2024-03-06T07:26:06.740287+0000","last_clean":"2024-03-06T07:26:06.740287+0000","last_became_active":"2024-03-06T07:26:05.753734+0000","last_became_peered":"2024-03-06T07:26:05.753734+0000","last_unstale":"2024-03-06T07:26:06.740287+0000","last_undegraded":"2024-03-06T07:26:06.740287+0000","last_fullsized":"2024-03-06T07:26:06.740287+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:51:50.325917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.759897+0000","last_change":"2024-03-06T07:26:06.754052+0000","last_active":"2024-03-06T07:26:06.759897+0000","last_peered":"2024-03-06T07:26:06.759897+0000","last_clean":"2024-03-06T07:26:06.759897+0000","last_became_active":"2024-03-06T07:26:06.753767+0000","last_became_peered":"2024-03-06T07:26:06.753767+0000","last_unstale":"2024-03-06T07:26:06.759897+0000","last_undegraded":"2024-03-06T07:26:06.759897+0000","last_fullsized":"2024-03-06T07:26:06.759897+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:29:18.804163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.743877+0000","last_change":"2024-03-06T07:26:05.748244+0000","last_active":"2024-03-06T07:26:06.743877+0000","last_peered":"2024-03-06T07:26:06.743877+0000","last_clean":"2024-03-06T07:26:06.743877+0000","last_became_active":"2024-03-06T07:26:05.747897+0000","last_became_peered":"2024-03-06T07:26:05.747897+0000","last_unstale":"2024-03-06T07:26:06.743877+0000","last_undegraded":"2024-03-06T07:26:06.743877+0000","last_fullsized":"2024-03-06T07:26:06.743877+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:12:20.518888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.754998+0000","last_change":"2024-03-06T07:26:06.751133+0000","last_active":"2024-03-06T07:26:06.754998+0000","last_peered":"2024-03-06T07:26:06.754998+0000","last_clean":"2024-03-06T07:26:06.754998+0000","last_became_active":"2024-03-06T07:26:06.749439+0000","last_became_peered":"2024-03-06T07:26:06.749439+0000","last_unstale":"2024-03-06T07:26:06.754998+0000","last_undegraded":"2024-03-06T07:26:06.754998+0000","last_fullsized":"2024-03-06T07:26:06.754998+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:22:35.668699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.740417+0000","last_change":"2024-03-06T07:26:05.753483+0000","last_active":"2024-03-06T07:26:06.740417+0000","last_peered":"2024-03-06T07:26:06.740417+0000","last_clean":"2024-03-06T07:26:06.740417+0000","last_became_active":"2024-03-06T07:26:05.753192+0000","last_became_peered":"2024-03-06T07:26:05.753192+0000","last_unstale":"2024-03-06T07:26:06.740417+0000","last_undegraded":"2024-03-06T07:26:06.740417+0000","last_fullsized":"2024-03-06T07:26:06.740417+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:05.000463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.763872+0000","last_change":"2024-03-06T07:26:06.756811+0000","last_active":"2024-03-06T07:26:06.763872+0000","last_peered":"2024-03-06T07:26:06.763872+0000","last_clean":"2024-03-06T07:26:06.763872+0000","last_became_active":"2024-03-06T07:26:06.756560+0000","last_became_peered":"2024-03-06T07:26:06.756560+0000","last_unstale":"2024-03-06T07:26:06.763872+0000","last_undegraded":"2024-03-06T07:26:06.763872+0000","last_fullsized":"2024-03-06T07:26:06.763872+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:42:22.384294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.741853+0000","last_change":"2024-03-06T07:26:05.749765+0000","last_active":"2024-03-06T07:26:06.741853+0000","last_peered":"2024-03-06T07:26:06.741853+0000","last_clean":"2024-03-06T07:26:06.741853+0000","last_became_active":"2024-03-06T07:26:05.749477+0000","last_became_peered":"2024-03-06T07:26:05.749477+0000","last_unstale":"2024-03-06T07:26:06.741853+0000","last_undegraded":"2024-03-06T07:26:06.741853+0000","last_fullsized":"2024-03-06T07:26:06.741853+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:46:44.594569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.758514+0000","last_change":"2024-03-06T07:26:06.752845+0000","last_active":"2024-03-06T07:26:06.758514+0000","last_peered":"2024-03-06T07:26:06.758514+0000","last_clean":"2024-03-06T07:26:06.758514+0000","last_became_active":"2024-03-06T07:26:06.752538+0000","last_became_peered":"2024-03-06T07:26:06.752538+0000","last_unstale":"2024-03-06T07:26:06.758514+0000","last_undegraded":"2024-03-06T07:26:06.758514+0000","last_fullsized":"2024-03-06T07:26:06.758514+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:58:33.191204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.738378+0000","last_change":"2024-03-06T07:26:05.747439+0000","last_active":"2024-03-06T07:26:06.738378+0000","last_peered":"2024-03-06T07:26:06.738378+0000","last_clean":"2024-03-06T07:26:06.738378+0000","last_became_active":"2024-03-06T07:26:05.746992+0000","last_became_peered":"2024-03-06T07:26:05.746992+0000","last_unstale":"2024-03-06T07:26:06.738378+0000","last_undegraded":"2024-03-06T07:26:06.738378+0000","last_fullsized":"2024-03-06T07:26:06.738378+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:35.237909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.768959+0000","last_change":"2024-03-06T07:26:06.763752+0000","last_active":"2024-03-06T07:26:06.768959+0000","last_peered":"2024-03-06T07:26:06.768959+0000","last_clean":"2024-03-06T07:26:06.768959+0000","last_became_active":"2024-03-06T07:26:06.763493+0000","last_became_peered":"2024-03-06T07:26:06.763493+0000","last_unstale":"2024-03-06T07:26:06.768959+0000","last_undegraded":"2024-03-06T07:26:06.768959+0000","last_fullsized":"2024-03-06T07:26:06.768959+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.019146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.741798+0000","last_change":"2024-03-06T07:26:05.748595+0000","last_active":"2024-03-06T07:26:06.741798+0000","last_peered":"2024-03-06T07:26:06.741798+0000","last_clean":"2024-03-06T07:26:06.741798+0000","last_became_active":"2024-03-06T07:26:05.748306+0000","last_became_peered":"2024-03-06T07:26:05.748306+0000","last_unstale":"2024-03-06T07:26:06.741798+0000","last_undegraded":"2024-03-06T07:26:06.741798+0000","last_fullsized":"2024-03-06T07:26:06.741798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:20:44.657926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.751874+0000","last_change":"2024-03-06T07:26:06.747815+0000","last_active":"2024-03-06T07:26:06.751874+0000","last_peered":"2024-03-06T07:26:06.751874+0000","last_clean":"2024-03-06T07:26:06.751874+0000","last_became_active":"2024-03-06T07:26:06.747548+0000","last_became_peered":"2024-03-06T07:26:06.747548+0000","last_unstale":"2024-03-06T07:26:06.751874+0000","last_undegraded":"2024-03-06T07:26:06.751874+0000","last_fullsized":"2024-03-06T07:26:06.751874+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:05.727539+0000","last_clean_scrub_stamp":"2024-03-06T07:26:05.727539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:24:38.033859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"93'1","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.765166+0000","last_change":"2024-03-06T07:26:05.747001+0000","last_active":"2024-03-06T07:26:06.765166+0000","last_peered":"2024-03-06T07:26:06.765166+0000","last_clean":"2024-03-06T07:26:06.765166+0000","last_became_active":"2024-03-06T07:26:05.746677+0000","last_became_peered":"2024-03-06T07:26:05.746677+0000","last_unstale":"2024-03-06T07:26:06.765166+0000","last_undegraded":"2024-03-06T07:26:06.765166+0000","last_fullsized":"2024-03-06T07:26:06.765166+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:26:04.722882+0000","last_clean_scrub_stamp":"2024-03-06T07:26:04.722882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:44:21.558359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-06T07:26:06.748033+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:26:06.748033+0000","last_peered":"2024-03-06T07:26:06.748033+0000","last_clean":"2024-03-06T07:26:06.748033+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:26:06.748033+0000","last_undegraded":"2024-03-06T07:26:06.748033+0000","last_fullsized":"2024-03-06T07:26:06.748033+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1732608,"data_stored":1072745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574905,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":482377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1150976,"data_stored":489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574905,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32896,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338944,"statfs":{"total":96636764160,"available":96603078656,"internally_reserved":0,"allocated":1748992,"data_stored":1080102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:26:10.081 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:26:10.081 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:26:10.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.190+0000 7f02b3f02640 1 Processor -- start 2024-03-06T07:26:10.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.190+0000 7f02b3f02640 1 -- start start 2024-03-06T07:26:10.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b3f02640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02ac1586e0 0x7f02ac0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b3f02640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02ac0d01a0 0x7f02ac0b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b3f02640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac158ac0 con 0x7f02ac0aef80 2024-03-06T07:26:10.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b3f02640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac15b2a0 con 0x7f02ac1586e0 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b3f02640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac15b540 con 0x7f02ac0d01a0 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b1c9e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02ac1586e0 0x7f02ac0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b149d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f02ac0aef80 0x7f02ac0cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42586/0 (socket says 172.21.15.72:42586) 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b149d640 1 -- 172.21.15.72:0/4119716938 learned_addr learned my addr 172.21.15.72:0/4119716938 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b249f640 1 --2- 172.21.15.72:0/4119716938 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0d01a0 0x7f02ac0b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b1c9e640 1 -- 172.21.15.72:0/4119716938 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0d01a0 msgr2=0x7f02ac0b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b1c9e640 1 --2- 172.21.15.72:0/4119716938 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0d01a0 0x7f02ac0b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b1c9e640 1 -- 172.21.15.72:0/4119716938 >> v1:172.21.15.72:6789/0 conn(0x7f02ac0aef80 legacy=0x7f02ac0cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b1c9e640 1 -- 172.21.15.72:0/4119716938 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac15b820 con 0x7f02ac1586e0 2024-03-06T07:26:10.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.191+0000 7f02b0c9c640 1 -- 172.21.15.72:0/4119716938 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781745632 0 0) 0x7f02ac158ac0 con 0x7f02ac0aef80 2024-03-06T07:26:10.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.192+0000 7f02b1c9e640 1 --2- 172.21.15.72:0/4119716938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0aea40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0cc350 tx=0x7f02a802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fc20131f4bb92a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.192+0000 7f02b0c9c640 1 -- 172.21.15.72:0/4119716938 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a80315a0 con 0x7f02ac1586e0 2024-03-06T07:26:10.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.192+0000 7f02b0c9c640 1 -- 172.21.15.72:0/4119716938 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a8037070 con 0x7f02ac1586e0 2024-03-06T07:26:10.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.192+0000 7f02b0c9c640 1 -- 172.21.15.72:0/4119716938 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a8031960 con 0x7f02ac1586e0 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 -- 172.21.15.72:0/4119716938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 msgr2=0x7f02ac0aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 --2- 172.21.15.72:0/4119716938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0aea40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f02ac0cc350 tx=0x7f02a802b840 comp rx=0 tx=0).stop 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 -- 172.21.15.72:0/4119716938 shutdown_connections 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 --2- 172.21.15.72:0/4119716938 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0d01a0 0x7f02ac0b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 --2- 172.21.15.72:0/4119716938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0aea40 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 -- 172.21.15.72:0/4119716938 >> 172.21.15.72:0/4119716938 conn(0x7f02ac06b190 msgr2=0x7f02ac06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 -- 172.21.15.72:0/4119716938 shutdown_connections 2024-03-06T07:26:10.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.193+0000 7f02b3f02640 1 -- 172.21.15.72:0/4119716938 wait complete. 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.194+0000 7f02b3f02640 1 Processor -- start 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.194+0000 7f02b3f02640 1 -- start start 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0aef80 0x7f02ac152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0d01a0 0x7f02ac0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac071350 con 0x7f02ac1586e0 2024-03-06T07:26:10.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac15bde0 con 0x7f02ac0aef80 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0714d0 con 0x7f02ac0d01a0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0ac4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34662/0 (socket says 172.21.15.72:34662) 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 -- 172.21.15.72:0/2625431608 learned_addr learned my addr 172.21.15.72:0/2625431608 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b149d640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0d01a0 0x7f02ac0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b1c9e640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0aef80 0x7f02ac152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 -- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0d01a0 msgr2=0x7f02ac0ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0d01a0 0x7f02ac0ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 -- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0aef80 msgr2=0x7f02ac152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0aef80 0x7f02ac152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 -- 172.21.15.72:0/2625431608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0cd260 con 0x7f02ac1586e0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b249f640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0ac4b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f02a400efc0 tx=0x7f02a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4018020 con 0x7f02ac1586e0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4014480 con 0x7f02ac1586e0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac0cd540 con 0x7f02ac1586e0 2024-03-06T07:26:10.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4014870 con 0x7f02ac1586e0 2024-03-06T07:26:10.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.195+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac0cdb00 con 0x7f02ac1586e0 2024-03-06T07:26:10.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.196+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac0732b0 con 0x7f02ac1586e0 2024-03-06T07:26:10.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.197+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02a4005d10 con 0x7f02ac1586e0 2024-03-06T07:26:10.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.197+0000 7f02a27fc640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029006b4d0 0x7f029006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.197+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02a408af20 con 0x7f02ac1586e0 2024-03-06T07:26:10.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.197+0000 7f02b1c9e640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029006b4d0 0x7f029006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.198+0000 7f02b1c9e640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029006b4d0 0x7f029006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f02a8036000 tx=0x7f02a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.201+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02a4022030 con 0x7f02ac1586e0 2024-03-06T07:26:10.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.439+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f02ac0ae920 con 0x7f02ac1586e0 2024-03-06T07:26:10.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.440+0000 7f02a27fc640 1 -- 172.21.15.72:0/2625431608 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f02a4010040 con 0x7f02ac1586e0 2024-03-06T07:26:10.442 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:26:10.442 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:10.443 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029006b4d0 msgr2=0x7f029006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029006b4d0 0x7f029006d990 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f02a8036000 tx=0x7f02a8008030 comp rx=0 tx=0).stop 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 msgr2=0x7f02ac0ac4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0ac4b0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f02a400efc0 tx=0x7f02a400c490 comp rx=0 tx=0).stop 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 shutdown_connections 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029006b4d0 0x7f029006d990 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02ac1586e0 0x7f02ac0ac4b0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0d01a0 0x7f02ac0ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 --2- 172.21.15.72:0/2625431608 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0aef80 0x7f02ac152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.442+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 >> 172.21.15.72:0/2625431608 conn(0x7f02ac06b190 msgr2=0x7f02ac15a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.443+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 shutdown_connections 2024-03-06T07:26:10.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.443+0000 7f02b3f02640 1 -- 172.21.15.72:0/2625431608 wait complete. 2024-03-06T07:26:10.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:10.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.553+0000 7f5c9006f640 1 Processor -- start 2024-03-06T07:26:10.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.553+0000 7f5c9006f640 1 -- start start 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c9006f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c8815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c9006f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8815f580 con 0x7f5c8813dc10 2024-03-06T07:26:10.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c9006f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8815b450 con 0x7f5c880b3060 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c9006f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880cf3f0 con 0x7f5c8815bf30 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8de0b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5c8813dc10 0x7f5c880b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42594/0 (socket says 172.21.15.72:42594) 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8de0b640 1 -- 172.21.15.72:0/3125996944 learned_addr learned my addr 172.21.15.72:0/3125996944 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8e60c640 1 --2- 172.21.15.72:0/3125996944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c8815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562627373 0 0) 0x7f5c8815f580 con 0x7f5c8813dc10 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c74003520 con 0x7f5c8813dc10 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511349512 0 0) 0x7f5c8815b450 con 0x7f5c880b3060 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8815f580 con 0x7f5c880b3060 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8e60c640 1 -- 172.21.15.72:0/3125996944 >> v1:172.21.15.146:6789/0 conn(0x7f5c880b3060 legacy=0x7f5c8815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8e60c640 1 -- 172.21.15.72:0/3125996944 >> v1:172.21.15.72:6789/0 conn(0x7f5c8813dc10 legacy=0x7f5c880b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.554+0000 7f5c8e60c640 1 -- 172.21.15.72:0/3125996944 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8815b730 con 0x7f5c8815bf30 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.555+0000 7f5c8e60c640 1 --2- 172.21.15.72:0/3125996944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c8815ef60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009dc0 tx=0x7f5c8002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec55dba84043aaf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.555+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c80039070 con 0x7f5c8815bf30 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.555+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8002cc40 con 0x7f5c8815bf30 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.555+0000 7f5c8ce09640 1 -- 172.21.15.72:0/3125996944 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c80034440 con 0x7f5c8815bf30 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 -- 172.21.15.72:0/3125996944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 msgr2=0x7f5c8815ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 --2- 172.21.15.72:0/3125996944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c8815ef60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5c80009dc0 tx=0x7f5c8002be90 comp rx=0 tx=0).stop 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 -- 172.21.15.72:0/3125996944 shutdown_connections 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 --2- 172.21.15.72:0/3125996944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c8815ef60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 -- 172.21.15.72:0/3125996944 >> 172.21.15.72:0/3125996944 conn(0x7f5c880cf8a0 msgr2=0x7f5c880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 -- 172.21.15.72:0/3125996944 shutdown_connections 2024-03-06T07:26:10.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.556+0000 7f5c9006f640 1 -- 172.21.15.72:0/3125996944 wait complete. 2024-03-06T07:26:10.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.557+0000 7f5c9006f640 1 Processor -- start 2024-03-06T07:26:10.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.557+0000 7f5c9006f640 1 -- start start 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c880b3060 0x7f5c8813d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 0x7f5c8813f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c880b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c880718e0 con 0x7f5c8813dc10 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8815b210 con 0x7f5c880b3060 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c88071a60 con 0x7f5c8815bf30 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 0x7f5c8813f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 0x7f5c8813f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34666/0 (socket says 172.21.15.72:34666) 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 -- 172.21.15.72:0/1016478600 learned_addr learned my addr 172.21.15.72:0/1016478600 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8de0b640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c880b3060 0x7f5c8813d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8e60c640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c880b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 -- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 msgr2=0x7f5c880b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c880b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 -- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c880b3060 msgr2=0x7f5c8813d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c880b3060 0x7f5c8813d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 -- 172.21.15.72:0/1016478600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8813d670 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c8d60a640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 0x7f5c8813f250 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c500 tx=0x7f5c7800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c780052d0 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c78005490 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78012870 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8813fb50 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.558+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c88140110 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.560+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8804f330 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.560+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5c78012a10 con 0x7f5c8813dc10 2024-03-06T07:26:10.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.560+0000 7f5c7e7fc640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:10.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.560+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c78089dd0 con 0x7f5c8813dc10 2024-03-06T07:26:10.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.560+0000 7f5c8de0b640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:10.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.561+0000 7f5c8de0b640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c840011e0 tx=0x7f5c84007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:10.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.565+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5c78058610 con 0x7f5c8813dc10 2024-03-06T07:26:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.772+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c880cd940 con 0x7f5c8813dc10 2024-03-06T07:26:10.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.773+0000 7f5c7e7fc640 1 -- 172.21.15.72:0/1016478600 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f5c7805c6b0 con 0x7f5c8813dc10 2024-03-06T07:26:10.775 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:10.775 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":46,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5828,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1283242178","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1283242178},{"type":"v1","addr":"172.21.15.146:6834","nonce":1283242178}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7386,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3967731577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3967731577},{"type":"v1","addr":"172.21.15.72:6836","nonce":3967731577}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7389,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/646486087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":646486087},{"type":"v1","addr":"172.21.15.72:6837","nonce":646486087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:08.092689+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7404":{"gid":7404,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/209452807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":209452807},{"type":"v1","addr":"172.21.15.146:6835","nonce":209452807}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7404,"qdb_cluster":[7404]},"id":7}]} 2024-03-06T07:26:10.775 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 46 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c6c06b4d0 msgr2=0x7f5c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5c840011e0 tx=0x7f5c84007d30 comp rx=0 tx=0).stop 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 msgr2=0x7f5c8813f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 0x7f5c8813f250 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c500 tx=0x7f5c7800c9c0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 shutdown_connections 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5c6c06b4d0 0x7f5c6c06d990 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8815bf30 0x7f5c880b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c8813dc10 0x7f5c8813f250 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 --2- 172.21.15.72:0/1016478600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c880b3060 0x7f5c8813d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 >> 172.21.15.72:0/1016478600 conn(0x7f5c880cf8a0 msgr2=0x7f5c8815e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 shutdown_connections 2024-03-06T07:26:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:10.775+0000 7f5c9006f640 1 -- 172.21.15.72:0/1016478600 wait complete. 2024-03-06T07:26:11.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:06.088200+0000', 'modified': '2024-03-06T07:26:08.092689+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7404}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7404': {'gid': 7404, 'name': 'b', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6835/209452807', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6833', 'nonce': 209452807}, {'type': 'v1', 'addr': '172.21.15.146:6835', 'nonce': 209452807}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7404, 'qdb_cluster': [7404]} 2024-03-06T07:26:11.191 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:11.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:26:11.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.292+0000 7fd7f0fee640 1 Processor -- start 2024-03-06T07:26:11.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.292+0000 7fd7f0fee640 1 -- start start 2024-03-06T07:26:11.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7f0fee640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd7ec1444c0 0x7fd7ec06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7f0fee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7f0fee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ec06d370 con 0x7fd7ec15b2d0 2024-03-06T07:26:11.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7f0fee640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ec1448a0 con 0x7fd7ec1444c0 2024-03-06T07:26:11.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7f0fee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec06d650 con 0x7fd7ec070fe0 2024-03-06T07:26:11.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd7ec1444c0 0x7fd7ec06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd7ec1444c0 0x7fd7ec06ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34674/0 (socket says 172.21.15.72:34674) 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 -- 172.21.15.72:0/909267377 learned_addr learned my addr 172.21.15.72:0/909267377 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/909267377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 -- 172.21.15.72:0/909267377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 msgr2=0x7fd7ec0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/909267377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 -- 172.21.15.72:0/909267377 >> v1:172.21.15.146:6789/0 conn(0x7fd7ec15b2d0 legacy=0x7fd7ec0708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.293+0000 7fd7ea59c640 1 -- 172.21.15.72:0/909267377 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec06d930 con 0x7fd7ec1444c0 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.294+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/909267377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec06ca00 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009e50 tx=0x7fd7dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f99504f366b5e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.294+0000 7fd7e959a640 1 -- 172.21.15.72:0/909267377 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc02cb00 con 0x7fd7ec1444c0 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.294+0000 7fd7e959a640 1 -- 172.21.15.72:0/909267377 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc02cca0 con 0x7fd7ec1444c0 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.294+0000 7fd7e959a640 1 -- 172.21.15.72:0/909267377 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc034610 con 0x7fd7ec1444c0 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.294+0000 7fd7f0fee640 1 -- 172.21.15.72:0/909267377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 msgr2=0x7fd7ec06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.294+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/909267377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec06ca00 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009e50 tx=0x7fd7dc02bf60 comp rx=0 tx=0).stop 2024-03-06T07:26:11.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.295+0000 7fd7f0fee640 1 -- 172.21.15.72:0/909267377 shutdown_connections 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.295+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/909267377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.295+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/909267377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec06ca00 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.295+0000 7fd7f0fee640 1 -- 172.21.15.72:0/909267377 >> 172.21.15.72:0/909267377 conn(0x7fd7ec0b1c80 msgr2=0x7fd7ec0cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.295+0000 7fd7f0fee640 1 -- 172.21.15.72:0/909267377 shutdown_connections 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.295+0000 7fd7f0fee640 1 -- 172.21.15.72:0/909267377 wait complete. 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.296+0000 7fd7f0fee640 1 Processor -- start 2024-03-06T07:26:11.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.296+0000 7fd7f0fee640 1 -- start start 2024-03-06T07:26:11.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec13b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7ec15b2d0 0x7fd7ec140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec06cf40 con 0x7fd7ec1444c0 2024-03-06T07:26:11.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec06def0 con 0x7fd7ec15b2d0 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec06d0c0 con 0x7fd7ec070fe0 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec13b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec13b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34688/0 (socket says 172.21.15.72:34688) 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 -- 172.21.15.72:0/2298409860 learned_addr learned my addr 172.21.15.72:0/2298409860 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7ec15b2d0 0x7fd7ec140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 -- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 msgr2=0x7fd7ec0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 -- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7ec15b2d0 msgr2=0x7fd7ec140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7ec15b2d0 0x7fd7ec140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 -- 172.21.15.72:0/2298409860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec13c960 con 0x7fd7ec1444c0 2024-03-06T07:26:11.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec13b3a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000e710 tx=0x7fd7e000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7daffd640 1 -- 172.21.15.72:0/2298409860 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00052d0 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7daffd640 1 -- 172.21.15.72:0/2298409860 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e0005470 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7daffd640 1 -- 172.21.15.72:0/2298409860 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0001ab0 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ec13d9c0 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.297+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ec13df80 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.299+0000 7fd7daffd640 1 -- 172.21.15.72:0/2298409860 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd7e00035f0 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.299+0000 7fd7daffd640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7cc06b4d0 0x7fd7cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.299+0000 7fd7daffd640 1 -- 172.21.15.72:0/2298409860 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7e008c330 con 0x7fd7ec1444c0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.299+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7cc06b4d0 0x7fd7cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.300+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7cc06b4d0 0x7fd7cc06d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc035000 tx=0x7fd7dc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.300+0000 7fd7f3bcd740 1 Processor -- start 2024-03-06T07:26:11.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.300+0000 7fd7f3bcd740 1 -- start start 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 0x55f67b028c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b02c8c0 0x55f67b02e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b058a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67af926b0 con 0x55f67b028870 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67af3ba60 con 0x55f67b058600 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f67af8e750 con 0x55f67b02c8c0 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 0x55f67b028c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 0x55f67b028c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34694/0 (socket says 172.21.15.72:34694) 2024-03-06T07:26:11.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 -- 172.21.15.72:0/2493426050 learned_addr learned my addr 172.21.15.72:0/2493426050 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b02c8c0 0x55f67b02e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b058a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 -- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b02c8c0 msgr2=0x55f67b02e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b02c8c0 0x55f67b02e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 -- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 msgr2=0x55f67b058a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b058a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.301+0000 7fd7ea59c640 1 -- 172.21.15.72:0/2493426050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f67af59f30 con 0x55f67b028870 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 0x55f67b028c50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009f40 tx=0x7fd7dc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850a840ad69dde8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7cbfff640 1 -- 172.21.15.72:0/2493426050 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc03f440 con 0x55f67b028870 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7cbfff640 1 -- 172.21.15.72:0/2493426050 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc050070 con 0x55f67b028870 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7cbfff640 1 -- 172.21.15.72:0/2493426050 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc03f7e0 con 0x55f67b028870 2024-03-06T07:26:11.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/2493426050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 msgr2=0x55f67b028c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 0x55f67b028c50 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009f40 tx=0x7fd7dc02e000 comp rx=0 tx=0).stop 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/2493426050 shutdown_connections 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b058a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b02c8c0 0x55f67b02e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/2493426050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b028870 0x55f67b028c50 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/2493426050 >> 172.21.15.72:0/2493426050 conn(0x55f67b02a8f0 msgr2=0x55f67b02c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/2493426050 shutdown_connections 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.302+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/2493426050 wait complete. 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.303+0000 7fd7f3bcd740 1 -- start start 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.303+0000 7fd7f3bcd740 1 Processor -- start 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b04e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b04f430 0x55f67b0510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b05a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67b02cd50 con 0x55f67b05a470 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67af4eea0 con 0x55f67b058600 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f67b02d030 con 0x55f67b04f430 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7f3bcd740 10 client.0 _ll_register_callbacks cb 0x55f67af4e870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b05a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b05a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34702/0 (socket says 172.21.15.72:34702) 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 -- 172.21.15.72:0/3367427202 learned_addr learned my addr 172.21.15.72:0/3367427202 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b04f430 0x55f67b0510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b04e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 -- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b04f430 msgr2=0x55f67b0510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b04f430 0x55f67b0510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 -- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 msgr2=0x55f67b04e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b04e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 -- 172.21.15.72:0/3367427202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f67af4f270 con 0x55f67b05a470 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.304+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b05a8f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4014a00 tx=0x7fd7e4014ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b7651aedc2cd43 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7c97fa640 1 -- 172.21.15.72:0/3367427202 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4012ae0 con 0x55f67b05a470 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7c97fa640 1 -- 172.21.15.72:0/3367427202 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7e4012c80 con 0x55f67b05a470 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7dc050070 con 0x55f67b05a470 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7c97fa640 1 -- 172.21.15.72:0/3367427202 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7e4012e80 con 0x55f67b05a470 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 10 client.7536 fetch_fsmap learned FSMap version 46 2024-03-06T07:26:11.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55f67aca72d0 con 0x55f67b05a470 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7c97fa640 1 -- 172.21.15.72:0/3367427202 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7dc050070 con 0x55f67b05a470 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7c97fa640 1 -- 172.21.15.72:0/3367427202 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd7e402a080 con 0x55f67b05a470 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 10 client.7536 fetch_fsmap finished waiting for FSMap version 46 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 10 client.7536 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 20 client.7536 populate_metadata read hostname 'smithi072' 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b04d3c0 0x55f67b087300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 4 client.7536 mds_command: new command op to 7404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 --> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f67af9fbf0 con 0x55f67b04d3c0 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.305+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b04d3c0 0x55f67b087300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.306+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b04d3c0 0x55f67b087300 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=deaa3d097a75bb2 server_cookie=d872bfeb49646b16 in_seq=0 out_seq=0 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.306+0000 7fd7c97fa640 10 client.7536 ms_handle_connect on v2:172.21.15.146:6833/209452807 2024-03-06T07:26:11.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.306+0000 7fd7c97fa640 1 -- 172.21.15.72:0/3367427202 <== mds.0 v2:172.21.15.146:6833/209452807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f67af9fbf0 con 0x55f67b04d3c0 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.306+0000 7fd7c97fa640 10 client.7536 handle_command_reply: tid=0 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.306+0000 7fd7f3bcd740 1 client.7536 shutdown 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 msgr2=0x55f67b05a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b05a8f0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4014a00 tx=0x7fd7e4014ec0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 shutdown_connections 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b04d3c0 0x55f67b087300 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b05a8f0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b04f430 0x55f67b0510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7c97fa640 0 client.7536 ms_handle_reset on v2:172.21.15.146:6833/209452807 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/3367427202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b058600 0x55f67b04e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 >> 172.21.15.72:0/3367427202 conn(0x55f67b028870 msgr2=0x55f67b0565c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 shutdown_connections 2024-03-06T07:26:11.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/3367427202 wait complete. 2024-03-06T07:26:11.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.307+0000 7fd7f3bcd740 20 client.7536 trim_cache size 0 max 16384 2024-03-06T07:26:11.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.323+0000 7fd7f3bcd740 1 Processor -- start 2024-03-06T07:26:11.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.323+0000 7fd7f3bcd740 1 -- start start 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b04e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b029550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c6090 con 0x55f67b05a470 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c5490 con 0x55f67b04f430 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ec0c5a90 con 0x55f67b058600 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34706/0 (socket says 172.21.15.72:34706) 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7ead9d640 1 -- 172.21.15.72:0/4290305125 learned_addr learned my addr 172.21.15.72:0/4290305125 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b029550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.324+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b04e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7ead9d640 1 -- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 msgr2=0x55f67b029550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b029550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7ead9d640 1 -- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 msgr2=0x55f67b04e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b04e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7ead9d640 1 -- 172.21.15.72:0/4290305125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f67ad716b0 con 0x55f67b05a470 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029a90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4005c60 tx=0x7fd7e4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f36d9971734d48f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7cb7fe640 1 -- 172.21.15.72:0/4290305125 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4012ce0 con 0x55f67b05a470 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7cb7fe640 1 -- 172.21.15.72:0/4290305125 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e4019070 con 0x55f67b05a470 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7cb7fe640 1 -- 172.21.15.72:0/4290305125 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4004230 con 0x55f67b05a470 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/4290305125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 msgr2=0x55f67b029a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029a90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4005c60 tx=0x7fd7e4017000 comp rx=0 tx=0).stop 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/4290305125 shutdown_connections 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029a90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b029550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/4290305125 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b04e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/4290305125 >> 172.21.15.72:0/4290305125 conn(0x55f67b028870 msgr2=0x55f67b086430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/4290305125 shutdown_connections 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.325+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/4290305125 wait complete. 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.326+0000 7fd7f3bcd740 1 -- start start 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.326+0000 7fd7f3bcd740 1 Processor -- start 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b096cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67b05b350 con 0x55f67b05a470 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f67b055e20 con 0x55f67b04f430 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f67b04d7f0 con 0x55f67b058600 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7f3bcd740 10 client.0 _ll_register_callbacks cb 0x7fd7e40145e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34710/0 (socket says 172.21.15.72:34710) 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 -- 172.21.15.72:0/1080655933 learned_addr learned my addr 172.21.15.72:0/1080655933 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b096cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 -- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 msgr2=0x55f67b096cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b096cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 -- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 msgr2=0x55f67b0cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b0cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 -- 172.21.15.72:0/1080655933 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f67b097200 con 0x55f67b05a470 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7e9d9b640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b096cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7ead9d640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029550 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4012780 tx=0x7fd7e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d098cfcf7e86a6f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.327+0000 7fd7c97fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7c97fa640 1 -- 172.21.15.72:0/1080655933 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e4004440 con 0x55f67b05a470 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7c97fa640 1 -- 172.21.15.72:0/1080655933 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7e4003610 con 0x55f67b05a470 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7e4019070 con 0x55f67b05a470 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7c97fa640 1 -- 172.21.15.72:0/1080655933 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7e4041050 con 0x55f67b05a470 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 10 client.7542 fetch_fsmap learned FSMap version 46 2024-03-06T07:26:11.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x55f67b05be00 con 0x55f67b05a470 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7c97fa640 1 -- 172.21.15.72:0/1080655933 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e4019070 con 0x55f67b05a470 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7c97fa640 1 -- 172.21.15.72:0/1080655933 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 46) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd7e4044020 con 0x55f67b05a470 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 10 client.7542 fetch_fsmap finished waiting for FSMap version 46 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 10 client.7542 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 20 client.7542 populate_metadata read hostname 'smithi072' 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b0536e0 0x55f67b0eae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 4 client.7542 mds_command: new command op to 7404 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 --> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f67b02b050 con 0x55f67b0536e0 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.328+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b0536e0 0x55f67b0eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7ea59c640 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b0536e0 0x55f67b0eae40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7753555622c3b126 server_cookie=9ab27125077d0deb in_seq=0 out_seq=0 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7c97fa640 10 client.7542 ms_handle_connect on v2:172.21.15.146:6833/209452807 2024-03-06T07:26:11.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7c97fa640 1 -- 172.21.15.72:0/1080655933 <== mds.0 v2:172.21.15.146:6833/209452807 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55f67b02b050 con 0x55f67b0536e0 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7c97fa640 10 client.7542 handle_command_reply: tid=0 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 client.7542 shutdown 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 msgr2=0x55f67b029550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029550 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4012780 tx=0x7fd7e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 shutdown_connections 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:6833/209452807,v1:172.21.15.146:6835/209452807] conn(0x55f67b0536e0 0x55f67b0eae40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55f67b05a470 0x55f67b029550 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55f67b058600 0x55f67b096cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 --2- 172.21.15.72:0/1080655933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55f67b04f430 0x55f67b0cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7c97fa640 0 client.7542 ms_handle_reset on v2:172.21.15.146:6833/209452807 2024-03-06T07:26:11.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 >> 172.21.15.72:0/1080655933 conn(0x55f67b028870 msgr2=0x55f67b08d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 shutdown_connections 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 1 -- 172.21.15.72:0/1080655933 wait complete. 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.329+0000 7fd7f3bcd740 20 client.7542 trim_cache size 0 max 16384 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 7404, 2024-03-06T07:26:11.340 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 4.5858757609999996, 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 46, 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 94, 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 94, 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 9.6047367940000008 2024-03-06T07:26:11.341 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:26:11.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7cc06b4d0 msgr2=0x7fd7cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7cc06b4d0 0x7fd7cc06d990 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc035000 tx=0x7fd7dc030000 comp rx=0 tx=0).stop 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 msgr2=0x7fd7ec13b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec13b3a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000e710 tx=0x7fd7e000ebd0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 shutdown_connections 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7cc06b4d0 0x7fd7cc06d990 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7ec15b2d0 0x7fd7ec140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd7ec1444c0 0x7fd7ec13b3a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 --2- 172.21.15.72:0/2298409860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7ec070fe0 0x7fd7ec0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 >> 172.21.15.72:0/2298409860 conn(0x7fd7ec0b1c80 msgr2=0x7fd7ec15ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 shutdown_connections 2024-03-06T07:26:11.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.330+0000 7fd7f0fee640 1 -- 172.21.15.72:0/2298409860 wait complete. 2024-03-06T07:26:11.344 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:26:11.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.441+0000 7ff7744df640 1 Processor -- start 2024-03-06T07:26:11.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.441+0000 7ff7744df640 1 -- start start 2024-03-06T07:26:11.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.442+0000 7ff7744df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c15dfe0 0x7ff76c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.442+0000 7ff7744df640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c0b3430 con 0x7ff76c0d29c0 2024-03-06T07:26:11.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.442+0000 7ff7744df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c0b3710 con 0x7ff76c0cf830 2024-03-06T07:26:11.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.442+0000 7ff7744df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0b31e0 con 0x7ff76c15dfe0 2024-03-06T07:26:11.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff77227b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c15dfe0 0x7ff76c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff772a7c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff76c0d29c0 0x7ff76c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:42602/0 (socket says 172.21.15.72:42602) 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff772a7c640 1 -- 172.21.15.72:0/3347192201 learned_addr learned my addr 172.21.15.72:0/3347192201 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628404345 0 0) 0x7ff76c0b3430 con 0x7ff76c0d29c0 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff758003520 con 0x7ff76c0d29c0 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682189243 0 0) 0x7ff76c0b3710 con 0x7ff76c0cf830 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff76c0b3430 con 0x7ff76c0cf830 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff77227b640 1 -- 172.21.15.72:0/3347192201 >> v1:172.21.15.146:6789/0 conn(0x7ff76c0cf830 legacy=0x7ff76c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff77227b640 1 -- 172.21.15.72:0/3347192201 >> v1:172.21.15.72:6789/0 conn(0x7ff76c0d29c0 legacy=0x7ff76c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.443+0000 7ff77227b640 1 -- 172.21.15.72:0/3347192201 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0b39f0 con 0x7ff76c15dfe0 2024-03-06T07:26:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff77227b640 1 --2- 172.21.15.72:0/3347192201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c15dfe0 0x7ff76c0cf2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff7640097d0 tx=0x7ff76402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c064cb33b1216d39 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764035020 con 0x7ff76c15dfe0 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff764031890 con 0x7ff76c15dfe0 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff771279640 1 -- 172.21.15.72:0/3347192201 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764031c40 con 0x7ff76c15dfe0 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff7744df640 1 -- 172.21.15.72:0/3347192201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c15dfe0 msgr2=0x7ff76c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff7744df640 1 --2- 172.21.15.72:0/3347192201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c15dfe0 0x7ff76c0cf2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff7640097d0 tx=0x7ff76402b930 comp rx=0 tx=0).stop 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff7744df640 1 -- 172.21.15.72:0/3347192201 shutdown_connections 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff7744df640 1 --2- 172.21.15.72:0/3347192201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c15dfe0 0x7ff76c0cf2f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.444+0000 7ff7744df640 1 -- 172.21.15.72:0/3347192201 >> 172.21.15.72:0/3347192201 conn(0x7ff76c0d3cd0 msgr2=0x7ff76c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.445+0000 7ff7744df640 1 -- 172.21.15.72:0/3347192201 shutdown_connections 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.445+0000 7ff7744df640 1 -- 172.21.15.72:0/3347192201 wait complete. 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 Processor -- start 2024-03-06T07:26:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 -- start start 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff76c0cf830 0x7ff76c14f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c0d29c0 0x7ff76c0b82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 0x7ff76c14f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c15e3c0 con 0x7ff76c15dfe0 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0b3d60 con 0x7ff76c0cf830 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.446+0000 7ff7744df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c15e540 con 0x7ff76c0d29c0 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 0x7ff76c14f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 0x7ff76c14f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34716/0 (socket says 172.21.15.72:34716) 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 -- 172.21.15.72:0/2520391773 learned_addr learned my addr 172.21.15.72:0/2520391773 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff771a7a640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c0d29c0 0x7ff76c0b82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff77227b640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff76c0cf830 0x7ff76c14f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 -- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c0d29c0 msgr2=0x7ff76c0b82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c0d29c0 0x7ff76c0b82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 -- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff76c0cf830 msgr2=0x7ff76c14f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff76c0cf830 0x7ff76c14f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 -- 172.21.15.72:0/2520391773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0cbf50 con 0x7ff76c15dfe0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff772a7c640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 0x7ff76c14f940 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7ff76800cba0 tx=0x7ff76800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768017070 con 0x7ff76c15dfe0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76800ace0 con 0x7ff76c15dfe0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c0cc230 con 0x7ff76c15dfe0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768012620 con 0x7ff76c15dfe0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.447+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c0cc7f0 con 0x7ff76c15dfe0 2024-03-06T07:26:11.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.448+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c04f330 con 0x7ff76c15dfe0 2024-03-06T07:26:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.449+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff76801a070 con 0x7ff76c15dfe0 2024-03-06T07:26:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.449+0000 7ff762ffd640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.449+0000 7ff77227b640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.449+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff768089f80 con 0x7ff76c15dfe0 2024-03-06T07:26:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.450+0000 7ff77227b640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff75006b4d0 0x7ff75006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff764036000 tx=0x7ff76403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:11.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.454+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff768058840 con 0x7ff76c15dfe0 2024-03-06T07:26:11.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.660+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff76c0b0b60 con 0x7ff76c15dfe0 2024-03-06T07:26:11.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.660+0000 7ff762ffd640 1 -- 172.21.15.72:0/2520391773 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff76805c8e0 con 0x7ff76c15dfe0 2024-03-06T07:26:11.662 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:26:11.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff75006b4d0 msgr2=0x7ff75006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff75006b4d0 0x7ff75006d990 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff764036000 tx=0x7ff76403c040 comp rx=0 tx=0).stop 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 msgr2=0x7ff76c14f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 0x7ff76c14f940 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7ff76800cba0 tx=0x7ff76800a430 comp rx=0 tx=0).stop 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 shutdown_connections 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff75006b4d0 0x7ff75006d990 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff76c15dfe0 0x7ff76c14f940 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff76c0d29c0 0x7ff76c0b82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 --2- 172.21.15.72:0/2520391773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff76c0cf830 0x7ff76c14f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 >> 172.21.15.72:0/2520391773 conn(0x7ff76c0d3cd0 msgr2=0x7ff76c06c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 shutdown_connections 2024-03-06T07:26:11.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:11.662+0000 7ff7744df640 1 -- 172.21.15.72:0/2520391773 wait complete. 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:26:11.675 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:11.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:11.676 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:26:11.691 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42424sec preferred_lft 42424sec 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:26:11.692 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:26:11.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:11.693 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:26:11.756 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:26:11.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:11.758 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:26:11.813 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:26:11.813 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:26:11.813 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:26:11.813 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:26:11.813 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:26:11.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:11.814 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:26:11.814 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:26:11.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:26:11.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:26:11.814 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:26:11.814 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:26:11.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:11.909 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:26:11.923 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:26:11.923 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:26:11.923 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:11.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:11.923 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:26:11.989 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:26:11.989 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:11.989 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:12.044 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:26:12.044 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:26:12.045 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:12.099 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:26:12.167 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:26:12.222 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:12.223 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:12.224 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:12.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:12.225 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:26:12.277 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:26:12.277 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:26:12.319 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:26:12.344 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:26:12.400 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:26:12.401 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:26:12.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:12.403 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:26:12.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:26:12.402+0000 7f1777a4b480 -1 init, newargv = 0x55b5248e79a0 newargc=15 2024-03-06T07:26:12.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[68328]: starting ceph client 2024-03-06T07:26:12.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[68328]: starting fuse 2024-03-06T07:26:12.418 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:26:12.419 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:26:12.419 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:26:12.420 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:26:12.539 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.68328.asok 2024-03-06T07:26:12.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:12.550 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68328.asok 2024-03-06T07:26:12.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:12.550 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68328.asok status 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "68328", 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:12.685 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 7554, 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 7554 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 486731386 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:26:12.686 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 486731386 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.7554 192.168.0.1:0/486731386", 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/486731386", 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 46, 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 94, 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:26:12.687 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:26:12.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:12.693 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:26:12.693 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:12.708 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:26:12.708 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:12.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:12.708 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:12.777 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:12.819 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:12.868 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:12.869 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.870 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:12.871 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.872 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:12.873 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.874 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:12.875 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:26:12.876 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:26:12.877 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_files = 6 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:26:12.878 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:12.879 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.880 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:12.881 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.882 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:26:12.883 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:26:12.884 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:12.885 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:12.886 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:26:12.887 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:12.888 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:12.889 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-06T07:26:12.891 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:12.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:12.980 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:26:12.980 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:12.981 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:12.982 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:12.983 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:26:12.984 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:26:12.985 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:12.986 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:12.987 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:12.988 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:26:12.989 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:26:12.990 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_files = 6 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:12.991 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.992 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.993 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.994 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:12.995 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:26:12.996 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.997 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:12.998 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:12.999 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:13.000 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:26:13.001 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-06T07:26:13.002 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:13.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:13.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.143+0000 7f55e1b93640 1 Processor -- start 2024-03-06T07:26:13.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 -- start start 2024-03-06T07:26:13.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc06cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f55dc0cc400 0x7f55dc0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55dc0cae70 0x7f55dc0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0ccd80 con 0x7f55dc0cc400 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc15b290 con 0x7f55dc0cae70 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55e1b93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0cbb30 con 0x7f55dc15aeb0 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.144+0000 7f55daffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f55dc0cc400 0x7f55dc0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f55dc0cc400 0x7f55dc0cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59652/0 (socket says 172.21.15.72:59652) 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 -- 172.21.15.72:0/2387920869 learned_addr learned my addr 172.21.15.72:0/2387920869 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55dbfff640 1 --2- 172.21.15.72:0/2387920869 >> v2:172.21.15.146:3300/0 conn(0x7f55dc0cae70 0x7f55dc0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55db7fe640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc06cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 -- 172.21.15.72:0/2387920869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 msgr2=0x7f55dc06cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc06cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 -- 172.21.15.72:0/2387920869 >> v2:172.21.15.146:3300/0 conn(0x7f55dc0cae70 msgr2=0x7f55dc0cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 --2- 172.21.15.72:0/2387920869 >> v2:172.21.15.146:3300/0 conn(0x7f55dc0cae70 0x7f55dc0cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 -- 172.21.15.72:0/2387920869 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0cbe10 con 0x7f55dc0cc400 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55dbfff640 1 --2- 172.21.15.72:0/2387920869 >> v2:172.21.15.146:3300/0 conn(0x7f55dc0cae70 0x7f55dc0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55db7fe640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc06cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.145+0000 7f55daffd640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cc400 0x7f55dc0cc840 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0ca1f0 tx=0x7f55d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6818bf2840fa6c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55da7fc640 1 -- 172.21.15.72:0/2387920869 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d0035020 con 0x7f55dc0cc400 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55da7fc640 1 -- 172.21.15.72:0/2387920869 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d0031870 con 0x7f55dc0cc400 2024-03-06T07:26:13.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55da7fc640 1 -- 172.21.15.72:0/2387920869 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d0031c40 con 0x7f55dc0cc400 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 -- 172.21.15.72:0/2387920869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cc400 msgr2=0x7f55dc0cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cc400 0x7f55dc0cc840 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0ca1f0 tx=0x7f55d002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 -- 172.21.15.72:0/2387920869 shutdown_connections 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 --2- 172.21.15.72:0/2387920869 >> v2:172.21.15.146:3300/0 conn(0x7f55dc0cae70 0x7f55dc0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cc400 0x7f55dc0cc840 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 --2- 172.21.15.72:0/2387920869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc06cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 -- 172.21.15.72:0/2387920869 >> 172.21.15.72:0/2387920869 conn(0x7f55dc0b1500 msgr2=0x7f55dc0b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 -- 172.21.15.72:0/2387920869 shutdown_connections 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.146+0000 7f55e1b93640 1 -- 172.21.15.72:0/2387920869 wait complete. 2024-03-06T07:26:13.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.148+0000 7f55e1b93640 1 Processor -- start 2024-03-06T07:26:13.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.148+0000 7f55e1b93640 1 -- start start 2024-03-06T07:26:13.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 0x7f55dc0c60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55dc0cc400 0x7f55dc1571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc0c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0714b0 con 0x7f55dc0cae70 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0711d0 con 0x7f55dc0cc400 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0cb840 con 0x7f55dc15aeb0 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 0x7f55dc0c60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 0x7f55dc0c60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59666/0 (socket says 172.21.15.72:59666) 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 -- 172.21.15.72:0/184207048 learned_addr learned my addr 172.21.15.72:0/184207048 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55dbfff640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc0c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55daffd640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55dc0cc400 0x7f55dc1571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 -- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 msgr2=0x7f55dc0c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc0c74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 -- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55dc0cc400 msgr2=0x7f55dc1571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55dc0cc400 0x7f55dc1571a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 -- 172.21.15.72:0/184207048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0c35c0 con 0x7f55dc0cae70 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55db7fe640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 0x7f55dc0c60c0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00d7d0 tx=0x7f55cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc004030 con 0x7f55dc0cae70 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc0041d0 con 0x7f55dc0cae70 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc0c38a0 con 0x7f55dc0cae70 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0096a0 con 0x7f55dc0cae70 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.149+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc0c3e60 con 0x7f55dc0cae70 2024-03-06T07:26:13.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.151+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc068fb0 con 0x7f55dc0cae70 2024-03-06T07:26:13.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.151+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f55cc009840 con 0x7f55dc0cae70 2024-03-06T07:26:13.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.151+0000 7f55c3fff640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f55c806b500 0x7f55c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.151+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55cc08a340 con 0x7f55dc0cae70 2024-03-06T07:26:13.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.151+0000 7f55daffd640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f55c806b500 0x7f55c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.152+0000 7f55daffd640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f55c806b500 0x7f55c806d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f55d0036000 tx=0x7f55d003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.156+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f55cc058b70 con 0x7f55dc0cae70 2024-03-06T07:26:13.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.325+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f55dc0b5970 con 0x7f55dc0cae70 2024-03-06T07:26:13.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.325+0000 7f55c3fff640 1 -- 172.21.15.72:0/184207048 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7f55cc05cc10 con 0x7f55dc0cae70 2024-03-06T07:26:13.327 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:26:13.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f55c806b500 msgr2=0x7f55c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f55c806b500 0x7f55c806d9c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f55d0036000 tx=0x7f55d003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 msgr2=0x7f55dc0c60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 0x7f55dc0c60c0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f55cc00d7d0 tx=0x7f55cc00dc90 comp rx=0 tx=0).stop 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 shutdown_connections 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f55c806b500 0x7f55c806d9c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55dc15aeb0 0x7f55dc0c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55dc0cc400 0x7f55dc1571a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 --2- 172.21.15.72:0/184207048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f55dc0cae70 0x7f55dc0c60c0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 >> 172.21.15.72:0/184207048 conn(0x7f55dc0b1500 msgr2=0x7f55dc0afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 shutdown_connections 2024-03-06T07:26:13.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.327+0000 7f55e1b93640 1 -- 172.21.15.72:0/184207048 wait complete. 2024-03-06T07:26:13.340 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:13.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:26:13.391 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:26:13.392 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:26:13.393 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:26:13.394 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.395 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:26:13.396 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:13.397 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:26:13.398 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:26:13.399 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:26:13.400 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:13.401 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_files = 6 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.402 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:13.403 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:13.404 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:26:13.405 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:26:13.406 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:26:13.407 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:13.408 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:13.409 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.410 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.411 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:13.412 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:13.413 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-03-06T07:26:13.415 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:13.415 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:13.415 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:13.415 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:13.415 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:13.416 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:13.416 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:13.416 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:13.465 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:13.465 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:26:13.467 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:13.467 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:26:13.467 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:13.467 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:26:13.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.608+0000 7f57225e2640 1 Processor -- start 2024-03-06T07:26:13.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.608+0000 7fa3301e9640 1 Processor -- start 2024-03-06T07:26:13.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.608+0000 7efecdb25640 1 Processor -- start 2024-03-06T07:26:13.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.608+0000 7f57225e2640 1 -- start start 2024-03-06T07:26:13.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.608+0000 7efecdb25640 1 -- start start 2024-03-06T07:26:13.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.608+0000 7fa3301e9640 1 -- start start 2024-03-06T07:26:13.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efecdb25640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efec80ce5a0 0x7efec80b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f57225e2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f571c15e150 0x7f571c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efecdb25640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec80cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f57225e2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f571c0cfba0 0x7f571c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f57225e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c0d1ea0 con 0x7f571c0d0e90 2024-03-06T07:26:13.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f57225e2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c0d1300 con 0x7f571c0cfba0 2024-03-06T07:26:13.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f57225e2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c0d2ab0 con 0x7f571c15e150 2024-03-06T07:26:13.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efecdb25640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec80ccbd0 con 0x7efec8070d70 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efecdb25640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec80b3b00 con 0x7efec80ce5a0 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efec77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efec80ce5a0 0x7efec80b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efecdb25640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec80b3de0 con 0x7efec80ce980 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa3301e9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa32815e110 0x7fa32806da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efec7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efec8070d70 0x7efec80711a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50516/0 (socket says 172.21.15.72:50516) 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efec77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efec80ce5a0 0x7efec80b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59674/0 (socket says 172.21.15.72:59674) 2024-03-06T07:26:13.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efec7fff640 1 -- 172.21.15.72:0/4206612729 learned_addr learned my addr 172.21.15.72:0/4206612729 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:13.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa3301e9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa32806f4c0 0x7fa32806af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f571b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f571c0cfba0 0x7f571c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f571b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f571c0cfba0 0x7f571c0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59688/0 (socket says 172.21.15.72:59688) 2024-03-06T07:26:13.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7efec6ffd640 1 --2- 172.21.15.72:0/4206612729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec80cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f571b7fe640 1 -- 172.21.15.72:0/2543122651 learned_addr learned my addr 172.21.15.72:0/2543122651 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa32d784640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa32806f4c0 0x7fa32806af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa32df85640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa32815e110 0x7fa32806da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa32df85640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa32815e110 0x7fa32806da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:35748/0 (socket says 172.21.15.72:35748) 2024-03-06T07:26:13.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa32df85640 1 -- 172.21.15.72:0/2781889898 learned_addr learned my addr 172.21.15.72:0/2781889898 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7fa3301e9640 1 --2- 172.21.15.72:0/2781889898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32806b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.609+0000 7f571bfff640 1 --2- 172.21.15.72:0/2543122651 >> v2:172.21.15.146:3300/0 conn(0x7f571c15e150 0x7f571c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7efec67fc640 1 -- 172.21.15.72:0/4206612729 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590313358 0 0) 0x7efec80ccbd0 con 0x7efec8070d70 2024-03-06T07:26:13.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7f571b7fe640 1 -- 172.21.15.72:0/2543122651 >> v2:172.21.15.146:3300/0 conn(0x7f571c15e150 msgr2=0x7f571c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7f571b7fe640 1 --2- 172.21.15.72:0/2543122651 >> v2:172.21.15.146:3300/0 conn(0x7f571c15e150 0x7f571c0cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7f571b7fe640 1 -- 172.21.15.72:0/2543122651 >> v1:172.21.15.146:6789/0 conn(0x7f571c0d0e90 legacy=0x7f571c0b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7f571b7fe640 1 -- 172.21.15.72:0/2543122651 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c0d2180 con 0x7f571c0cfba0 2024-03-06T07:26:13.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa32815e4f0 con 0x7fa32806f4c0 2024-03-06T07:26:13.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3280b0520 con 0x7fa32815e110 2024-03-06T07:26:13.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7efec67fc640 1 -- 172.21.15.72:0/4206612729 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb0003520 con 0x7efec8070d70 2024-03-06T07:26:13.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7efec6ffd640 1 -- 172.21.15.72:0/4206612729 >> v2:172.21.15.72:3300/0 conn(0x7efec80ce5a0 msgr2=0x7efec80b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7efec6ffd640 1 --2- 172.21.15.72:0/4206612729 >> v2:172.21.15.72:3300/0 conn(0x7efec80ce5a0 0x7efec80b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3280d0b40 con 0x7fa32806f950 2024-03-06T07:26:13.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7efec6ffd640 1 -- 172.21.15.72:0/4206612729 >> v1:172.21.15.72:6789/0 conn(0x7efec8070d70 legacy=0x7efec80711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7efec6ffd640 1 -- 172.21.15.72:0/4206612729 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec80b40c0 con 0x7efec80ce980 2024-03-06T07:26:13.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32e786640 1 --2- 172.21.15.72:0/2781889898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32806b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32e786640 1 -- 172.21.15.72:0/2781889898 >> v2:172.21.15.146:3300/0 conn(0x7fa32815e110 msgr2=0x7fa32806da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32e786640 1 --2- 172.21.15.72:0/2781889898 >> v2:172.21.15.146:3300/0 conn(0x7fa32815e110 0x7fa32806da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32e786640 1 -- 172.21.15.72:0/2781889898 >> v2:172.21.15.72:3300/0 conn(0x7fa32806f4c0 msgr2=0x7fa32806af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:13.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32e786640 1 --2- 172.21.15.72:0/2781889898 >> v2:172.21.15.72:3300/0 conn(0x7fa32806f4c0 0x7fa32806af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32e786640 1 -- 172.21.15.72:0/2781889898 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3280afc80 con 0x7fa32806f950 2024-03-06T07:26:13.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.610+0000 7fa32d784640 1 --2- 172.21.15.72:0/2781889898 >> v2:172.21.15.72:3300/0 conn(0x7fa32806f4c0 0x7fa32806af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:13.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f571b7fe640 1 --2- 172.21.15.72:0/2543122651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c0d0950 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f571c0b6af0 tx=0x7f571002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f895be177b92ca6a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7efec6ffd640 1 --2- 172.21.15.72:0/4206612729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec80cede0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efebc0097a0 tx=0x7efebc02fcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a7178d855df207 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f571affd640 1 -- 172.21.15.72:0/2543122651 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571002e070 con 0x7f571c0cfba0 2024-03-06T07:26:13.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f571affd640 1 -- 172.21.15.72:0/2543122651 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710037070 con 0x7f571c0cfba0 2024-03-06T07:26:13.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f571affd640 1 -- 172.21.15.72:0/2543122651 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710031a00 con 0x7f571c0cfba0 2024-03-06T07:26:13.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7efec67fc640 1 -- 172.21.15.72:0/4206612729 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc03d070 con 0x7efec80ce980 2024-03-06T07:26:13.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7efec67fc640 1 -- 172.21.15.72:0/4206612729 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efebc030a70 con 0x7efec80ce980 2024-03-06T07:26:13.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7efec67fc640 1 -- 172.21.15.72:0/4206612729 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efebc030e40 con 0x7efec80ce980 2024-03-06T07:26:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7fa32e786640 1 --2- 172.21.15.72:0/2781889898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32806b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa32809efb0 tx=0x7fa32002bd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7319f8e79016781 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7fa32cf83640 1 -- 172.21.15.72:0/2781889898 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320039070 con 0x7fa32806f950 2024-03-06T07:26:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7fa32cf83640 1 -- 172.21.15.72:0/2781889898 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32002cab0 con 0x7fa32806f950 2024-03-06T07:26:13.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7fa32cf83640 1 -- 172.21.15.72:0/2781889898 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa320034440 con 0x7fa32806f950 2024-03-06T07:26:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f57225e2640 1 -- 172.21.15.72:0/2543122651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 msgr2=0x7f571c0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f57225e2640 1 --2- 172.21.15.72:0/2543122651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c0d0950 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f571c0b6af0 tx=0x7f571002b880 comp rx=0 tx=0).stop 2024-03-06T07:26:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.611+0000 7f57225e2640 1 -- 172.21.15.72:0/2543122651 shutdown_connections 2024-03-06T07:26:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7f57225e2640 1 --2- 172.21.15.72:0/2543122651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c0d0950 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7f57225e2640 1 --2- 172.21.15.72:0/2543122651 >> v2:172.21.15.146:3300/0 conn(0x7f571c15e150 0x7f571c0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7f57225e2640 1 -- 172.21.15.72:0/2543122651 >> 172.21.15.72:0/2543122651 conn(0x7f571c09d230 msgr2=0x7f571c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7f57225e2640 1 -- 172.21.15.72:0/2543122651 shutdown_connections 2024-03-06T07:26:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 -- 172.21.15.72:0/4206612729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 msgr2=0x7efec80cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7f57225e2640 1 -- 172.21.15.72:0/2543122651 wait complete. 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 --2- 172.21.15.72:0/4206612729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec80cede0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efebc0097a0 tx=0x7efebc02fcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 -- 172.21.15.72:0/4206612729 shutdown_connections 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 --2- 172.21.15.72:0/4206612729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec80cede0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 --2- 172.21.15.72:0/4206612729 >> v2:172.21.15.72:3300/0 conn(0x7efec80ce5a0 0x7efec80b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 -- 172.21.15.72:0/4206612729 >> 172.21.15.72:0/4206612729 conn(0x7efec806b160 msgr2=0x7efec806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 -- 172.21.15.72:0/4206612729 shutdown_connections 2024-03-06T07:26:13.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7efecdb25640 1 -- 172.21.15.72:0/4206612729 wait complete. 2024-03-06T07:26:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 msgr2=0x7fa32806b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 --2- 172.21.15.72:0/2781889898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32806b4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa32809efb0 tx=0x7fa32002bd00 comp rx=0 tx=0).stop 2024-03-06T07:26:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 shutdown_connections 2024-03-06T07:26:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 --2- 172.21.15.72:0/2781889898 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32806b4c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 --2- 172.21.15.72:0/2781889898 >> v2:172.21.15.72:3300/0 conn(0x7fa32806f4c0 0x7fa32806af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 --2- 172.21.15.72:0/2781889898 >> v2:172.21.15.146:3300/0 conn(0x7fa32815e110 0x7fa32806da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 >> 172.21.15.72:0/2781889898 conn(0x7fa32806c090 msgr2=0x7fa32806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 shutdown_connections 2024-03-06T07:26:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.612+0000 7fa3301e9640 1 -- 172.21.15.72:0/2781889898 wait complete. 2024-03-06T07:26:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.613+0000 7f57225e2640 1 Processor -- start 2024-03-06T07:26:13.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.613+0000 7efecdb25640 1 Processor -- start 2024-03-06T07:26:13.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.613+0000 7efecdb25640 1 -- start start 2024-03-06T07:26:13.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.613+0000 7f57225e2640 1 -- start start 2024-03-06T07:26:13.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.613+0000 7fa3301e9640 1 Processor -- start 2024-03-06T07:26:13.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.613+0000 7fa3301e9640 1 -- start start 2024-03-06T07:26:13.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efecdb25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 0x7efec815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efecdb25640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec80ce5a0 0x7efec8139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efecdb25640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec815d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efecdb25640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec80cc9b0 con 0x7efec8070d70 2024-03-06T07:26:13.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f57225e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efecdb25640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec80cc830 con 0x7efec80ce5a0 2024-03-06T07:26:13.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efecdb25640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec80cceb0 con 0x7efec80ce980 2024-03-06T07:26:13.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 0x7efec815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 0x7efec815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59702/0 (socket says 172.21.15.72:59702) 2024-03-06T07:26:13.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 -- 172.21.15.72:0/764692812 learned_addr learned my addr 172.21.15.72:0/764692812 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f57225e2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571c0d0e90 0x7f571c1f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f571bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f571bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c14ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59714/0 (socket says 172.21.15.72:59714) 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f57225e2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f571c15e150 0x7f571c1f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f57225e2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0d2f40 con 0x7f571c0cfba0 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f57225e2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0d2740 con 0x7f571c15e150 2024-03-06T07:26:13.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f57225e2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0d1b30 con 0x7f571c0d0e90 2024-03-06T07:26:13.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec6ffd640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec80ce5a0 0x7efec8139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec7fff640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec815d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f571bfff640 1 -- 172.21.15.72:0/1147376508 learned_addr learned my addr 172.21.15.72:0/1147376508 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7fa3301e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa32806f4c0 0x7fa32814bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7fa3301e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32815f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7f571b7fe640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571c0d0e90 0x7f571c1f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 -- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 msgr2=0x7efec815d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec815d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 -- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec80ce5a0 msgr2=0x7efec8139e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.614+0000 7efec77fe640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec80ce5a0 0x7efec8139e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efec77fe640 1 -- 172.21.15.72:0/764692812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec8138dc0 con 0x7efec8070d70 2024-03-06T07:26:13.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32df85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa32806f4c0 0x7fa32814bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32df85640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa32806f4c0 0x7fa32814bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59720/0 (socket says 172.21.15.72:59720) 2024-03-06T07:26:13.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32df85640 1 -- 172.21.15.72:0/3006822576 learned_addr learned my addr 172.21.15.72:0/3006822576 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:13.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa32815e110 0x7fa328155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f5720b7f640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f571c15e150 0x7f571c1f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f571bfff640 1 -- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571c0d0e90 msgr2=0x7f571c1f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f571bfff640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571c0d0e90 0x7f571c1f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32806dfc0 con 0x7fa32806f4c0 2024-03-06T07:26:13.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f571bfff640 1 -- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f571c15e150 msgr2=0x7f571c1f6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f571bfff640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f571c15e150 0x7f571c1f6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f571bfff640 1 -- 172.21.15.72:0/1147376508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c14b960 con 0x7f571c0cfba0 2024-03-06T07:26:13.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f5720b7f640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f571c15e150 0x7f571c1f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:13.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3280b0240 con 0x7fa32815e110 2024-03-06T07:26:13.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efec77fe640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 0x7efec815a840 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7efeb800eb40 tx=0x7efeb800d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3280b3df0 con 0x7fa32806f950 2024-03-06T07:26:13.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32e786640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa32815e110 0x7fa328155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32d784640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32815f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f571bfff640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c14ce80 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f570c00e870 tx=0x7f570c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8010440 con 0x7efec8070d70 2024-03-06T07:26:13.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb80105e0 con 0x7efec8070d70 2024-03-06T07:26:13.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c017070 con 0x7f571c0cfba0 2024-03-06T07:26:13.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb80109d0 con 0x7efec8070d70 2024-03-06T07:26:13.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec81390a0 con 0x7efec8070d70 2024-03-06T07:26:13.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec8139660 con 0x7efec8070d70 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c00c940 con 0x7f571c0cfba0 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c00cd30 con 0x7f571c0cfba0 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c14bc40 con 0x7f571c0cfba0 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32d784640 1 -- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa32815e110 msgr2=0x7fa328155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32d784640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa32815e110 0x7fa328155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32d784640 1 -- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa32806f4c0 msgr2=0x7fa32814bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32d784640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa32806f4c0 0x7fa32814bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.615+0000 7fa32d784640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32815e870 con 0x7fa32806f950 2024-03-06T07:26:13.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.616+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c14c200 con 0x7f571c0cfba0 2024-03-06T07:26:13.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.616+0000 7fa32d784640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32815f630 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa31800e840 tx=0x7fa31800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.616+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa318004030 con 0x7fa32806f950 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.616+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3180041d0 con 0x7fa32806f950 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.616+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32815eb50 con 0x7fa32806f950 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.616+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa31800c8e0 con 0x7fa32806f950 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.617+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32815f110 con 0x7fa32806f950 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.617+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8069ee0 con 0x7efec8070d70 2024-03-06T07:26:13.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.617+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c069ee0 con 0x7f571c0cfba0 2024-03-06T07:26:13.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.617+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efeb8010b70 con 0x7efec8070d70 2024-03-06T07:26:13.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7efeaffff640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f570c01a070 con 0x7f571c0cfba0 2024-03-06T07:26:13.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efeb808ad10 con 0x7efec8070d70 2024-03-06T07:26:13.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32806da80 con 0x7fa32806f950 2024-03-06T07:26:13.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7efec6ffd640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa31800ca80 con 0x7fa32806f950 2024-03-06T07:26:13.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7f5718ff9640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f570c08a010 con 0x7f571c0cfba0 2024-03-06T07:26:13.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.618+0000 7efec6ffd640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efeb406b4d0 0x7efeb406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efebc03a000 tx=0x7efebc034000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.619+0000 7f571b7fe640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.619+0000 7fa31e7fc640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa30c06b520 0x7fa30c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:13.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.619+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa31808a2d0 con 0x7fa32806f950 2024-03-06T07:26:13.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.619+0000 7fa32df85640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa30c06b520 0x7fa30c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:13.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.619+0000 7f571b7fe640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5710036000 tx=0x7f571003d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.619+0000 7fa32df85640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa30c06b520 0x7fa30c06d9e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa3240094a0 tx=0x7fa324009240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:13.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.622+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efeb8014030 con 0x7efec8070d70 2024-03-06T07:26:13.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.623+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa32806da80 con 0x7fa32806f950 2024-03-06T07:26:13.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.627+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f570c058840 con 0x7f571c0cfba0 2024-03-06T07:26:13.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.878+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7fa32815f310 con 0x7fa32806f950 2024-03-06T07:26:13.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.879+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7efec813a3a0 con 0x7efec8070d70 2024-03-06T07:26:13.882 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:26:13.880+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:13.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.884+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f571c0b4c40 con 0x7f571c0cfba0 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.885+0000 7fa31e7fc640 1 -- 172.21.15.72:0/3006822576 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7404 v47) v1 ==== 95+0+0 (secure 0 0 0) 0x7fa32815f310 con 0x7fa32806f950 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7404 2024-03-06T07:26:13.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.886+0000 7efeaffff640 1 -- 172.21.15.72:0/764692812 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5828 v48) v1 ==== 95+0+0 (secure 0 0 0) 0x7efeb80594b0 con 0x7efec8070d70 2024-03-06T07:26:13.888 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 5828 2024-03-06T07:26:13.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.887+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa30c06b520 msgr2=0x7fa30c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.887+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa30c06b520 0x7fa30c06d9e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa3240094a0 tx=0x7fa324009240 comp rx=0 tx=0).stop 2024-03-06T07:26:13.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 msgr2=0x7fa32815f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32815f630 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa31800e840 tx=0x7fa31800ed00 comp rx=0 tx=0).stop 2024-03-06T07:26:13.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 shutdown_connections 2024-03-06T07:26:13.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa30c06b520 0x7fa30c06d9e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa32815e110 0x7fa328155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa32806f950 0x7fa32815f630 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 --2- 172.21.15.72:0/3006822576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa32806f4c0 0x7fa32814bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 >> 172.21.15.72:0/3006822576 conn(0x7fa32806c090 msgr2=0x7fa32806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 shutdown_connections 2024-03-06T07:26:13.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.888+0000 7fa3301e9640 1 -- 172.21.15.72:0/3006822576 wait complete. 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efeb406b4d0 msgr2=0x7efeb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efeb406b4d0 0x7efeb406d990 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7efebc03a000 tx=0x7efebc034000 comp rx=0 tx=0).stop 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 msgr2=0x7efec815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 0x7efec815a840 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7efeb800eb40 tx=0x7efeb800d350 comp rx=0 tx=0).stop 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 shutdown_connections 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efeb406b4d0 0x7efeb406d990 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efec80ce980 0x7efec815d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efec80ce5a0 0x7efec8139e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 --2- 172.21.15.72:0/764692812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efec8070d70 0x7efec815a840 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 >> 172.21.15.72:0/764692812 conn(0x7efec806b160 msgr2=0x7efec8071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 shutdown_connections 2024-03-06T07:26:13.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.889+0000 7efecdb25640 1 -- 172.21.15.72:0/764692812 wait complete. 2024-03-06T07:26:13.893 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:26:13.889+0000 7f7b90a4b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:13.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.891+0000 7f5718ff9640 1 -- 172.21.15.72:0/1147376508 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7389 v49) v1 ==== 95+0+0 (secure 0 0 0) 0x7f570c05c8e0 con 0x7f571c0cfba0 2024-03-06T07:26:13.893 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7389 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.893+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 msgr2=0x7f570806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.893+0000 7f57225e2640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5710036000 tx=0x7f571003d040 comp rx=0 tx=0).stop 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.893+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 msgr2=0x7f571c14ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.893+0000 7f57225e2640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c14ce80 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f570c00e870 tx=0x7f570c00ed30 comp rx=0 tx=0).stop 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 shutdown_connections 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f571c15e150 0x7f571c1f6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f571c0d0e90 0x7f571c1f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 --2- 172.21.15.72:0/1147376508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f571c0cfba0 0x7f571c14ce80 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:13.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 >> 172.21.15.72:0/1147376508 conn(0x7f571c09d230 msgr2=0x7f571c15d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:13.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.894+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 shutdown_connections 2024-03-06T07:26:13.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:13.895+0000 7f57225e2640 1 -- 172.21.15.72:0/1147376508 wait complete. 2024-03-06T07:26:13.906 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:13.907 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:13.908 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:13.908 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:13.910 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:13.910 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:13.910 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:13.911 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:13.912 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:13.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:13.955 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:13.955 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:13.986 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:26:13.986 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:26:14.030 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:26:17.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-06T07:26:19.465 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:19.466 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:26:19.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.576+0000 7fd94359e640 1 Processor -- start 2024-03-06T07:26:19.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.576+0000 7fd94359e640 1 -- start start 2024-03-06T07:26:19.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.577+0000 7fd94359e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd94415c070 0x7fd9440cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.577+0000 7fd94359e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9440b4630 con 0x7fd9440cfd40 2024-03-06T07:26:19.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.577+0000 7fd94359e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9440cdb40 con 0x7fd9440b2f30 2024-03-06T07:26:19.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.577+0000 7fd94359e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9440b1a90 con 0x7fd94415c070 2024-03-06T07:26:19.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.577+0000 7fd941d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd9440cfd40 0x7fd9440ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50522/0 (socket says 172.21.15.72:50522) 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.577+0000 7fd941d9b640 1 -- 172.21.15.72:0/1548318195 learned_addr learned my addr 172.21.15.72:0/1548318195 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94259c640 1 --2- 172.21.15.72:0/1548318195 >> v2:172.21.15.146:3300/0 conn(0x7fd94415c070 0x7fd9440cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051645287 0 0) 0x7fd938001110 con 0x7fd9440cfd40 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92c003520 con 0x7fd9440cfd40 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400965827 0 0) 0x7fd92c003520 con 0x7fd9440cfd40 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 >> v2:172.21.15.146:3300/0 conn(0x7fd94415c070 msgr2=0x7fd9440cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 --2- 172.21.15.72:0/1548318195 >> v2:172.21.15.146:3300/0 conn(0x7fd94415c070 0x7fd9440cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 >> v1:172.21.15.146:6789/0 conn(0x7fd9440b2f30 legacy=0x7fd9440ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9440b1760 con 0x7fd9440cfd40 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd938001840 con 0x7fd9440cfd40 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 >> v1:172.21.15.72:6789/0 conn(0x7fd9440cfd40 legacy=0x7fd9440ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd92c003a90 0x7fd92c003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c004800 0x7fd92c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92c005290 0x7fd92c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440b1a90 con 0x7fd92c005290 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c00c1e0 con 0x7fd92c004800 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92c00c3e0 con 0x7fd92c003a90 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd942d9d640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92c005290 0x7fd92c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd941d9b640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c004800 0x7fd92c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd94259c640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd92c003a90 0x7fd92c003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd942d9d640 1 -- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd92c003a90 msgr2=0x7fd92c003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd942d9d640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd92c003a90 0x7fd92c003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd942d9d640 1 -- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c004800 msgr2=0x7fd92c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.578+0000 7fd942d9d640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c004800 0x7fd92c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.579+0000 7fd942d9d640 1 -- 172.21.15.72:0/1548318195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c003520 con 0x7fd92c005290 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.579+0000 7fd942d9d640 1 -- 172.21.15.72:0/1548318195 reap_dead start 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.579+0000 7fd942d9d640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92c005290 0x7fd92c00b9c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd93c001750 tx=0x7fd92c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31513d92d432052 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.579+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c001370 con 0x7fd92c005290 2024-03-06T07:26:19.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.579+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c005310 con 0x7fd92c005290 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.579+0000 7fd94159a640 1 -- 172.21.15.72:0/1548318195 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c003830 con 0x7fd92c005290 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 -- 172.21.15.72:0/1548318195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92c005290 msgr2=0x7fd92c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92c005290 0x7fd92c00b9c0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd93c001750 tx=0x7fd92c005830 comp rx=0 tx=0).stop 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 -- 172.21.15.72:0/1548318195 shutdown_connections 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 --2- 172.21.15.72:0/1548318195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd92c005290 0x7fd92c00b9c0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 -- 172.21.15.72:0/1548318195 >> 172.21.15.72:0/1548318195 conn(0x7fd9440b37e0 msgr2=0x7fd9440b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 -- 172.21.15.72:0/1548318195 shutdown_connections 2024-03-06T07:26:19.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.580+0000 7fd94359e640 1 -- 172.21.15.72:0/1548318195 wait complete. 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.581+0000 7fd94359e640 1 Processor -- start 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.581+0000 7fd94359e640 1 -- start start 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94359e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c005290 0x7fd9441f3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94359e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd94406cab0 0x7fd94406cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 0x7fd9440c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94359e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440cd1b0 con 0x7fd9440c6a60 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94359e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440cd860 con 0x7fd92c005290 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94359e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9440cd330 con 0x7fd94406cab0 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 0x7fd9440c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 0x7fd9440c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59736/0 (socket says 172.21.15.72:59736) 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 -- 172.21.15.72:0/80217240 learned_addr learned my addr 172.21.15.72:0/80217240 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd94259c640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c005290 0x7fd9441f3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd941d9b640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd94406cab0 0x7fd94406cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 -- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd94406cab0 msgr2=0x7fd94406cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd94406cab0 0x7fd94406cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 -- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c005290 msgr2=0x7fd9441f3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.582+0000 7fd942d9d640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c005290 0x7fd9441f3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd942d9d640 1 -- 172.21.15.72:0/80217240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9441e5610 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd94259c640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c005290 0x7fd9441f3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd942d9d640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 0x7fd9440c6ee0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd93c000c00 tx=0x7fd93c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c041070 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93c032610 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93c001370 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9440c77f0 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.583+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9440c7db0 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.584+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9441e57e0 con 0x7fd9440c6a60 2024-03-06T07:26:19.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.584+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd93c044070 con 0x7fd9440c6a60 2024-03-06T07:26:19.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.585+0000 7fd932ffd640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd92406b4d0 0x7fd92406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.585+0000 7fd94259c640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd92406b4d0 0x7fd92406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.585+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fd93c0aae70 con 0x7fd9440c6a60 2024-03-06T07:26:19.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.585+0000 7fd94259c640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd92406b4d0 0x7fd92406d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd9440b0950 tx=0x7fd93400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.589+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd93c079600 con 0x7fd9440c6a60 2024-03-06T07:26:19.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.828+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fd94404f330 con 0x7fd9440c6a60 2024-03-06T07:26:19.830 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:26:19.828+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:19.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.830+0000 7fd932ffd640 1 -- 172.21.15.72:0/80217240 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7386 v52) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd93c07d6a0 con 0x7fd9440c6a60 2024-03-06T07:26:19.832 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7386 2024-03-06T07:26:19.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd92406b4d0 msgr2=0x7fd92406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd92406b4d0 0x7fd92406d990 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fd9440b0950 tx=0x7fd93400b040 comp rx=0 tx=0).stop 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 msgr2=0x7fd9440c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 0x7fd9440c6ee0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fd93c000c00 tx=0x7fd93c001120 comp rx=0 tx=0).stop 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 shutdown_connections 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd92406b4d0 0x7fd92406d990 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9440c6a60 0x7fd9440c6ee0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd94406cab0 0x7fd94406cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 --2- 172.21.15.72:0/80217240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92c005290 0x7fd9441f3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 >> 172.21.15.72:0/80217240 conn(0x7fd9440b37e0 msgr2=0x7fd9440b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 shutdown_connections 2024-03-06T07:26:19.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.833+0000 7fd94359e640 1 -- 172.21.15.72:0/80217240 wait complete. 2024-03-06T07:26:19.848 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:19.848 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:19.850 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:19.850 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:19.850 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:19.926 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:26:19.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.950+0000 7fd48eceb640 1 Processor -- start 2024-03-06T07:26:19.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.950+0000 7fd48eceb640 1 -- start start 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.950+0000 7fd48eceb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4880b4b90 0x7fd4880aff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.950+0000 7fd48eceb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880b3dd0 con 0x7fd48806b690 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.950+0000 7fd48eceb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880b40b0 con 0x7fd4881459b0 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.950+0000 7fd48eceb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4880b4390 con 0x7fd4880b4b90 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd487fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd48806b690 0x7fd4880af810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50540/0 (socket says 172.21.15.72:50540) 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd487fff640 1 -- 172.21.15.72:0/1395914393 learned_addr learned my addr 172.21.15.72:0/1395914393 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd48d288640 1 --2- 172.21.15.72:0/1395914393 >> v2:172.21.15.146:3300/0 conn(0x7fd4880b4b90 0x7fd4880aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd4877fe640 1 -- 172.21.15.72:0/1395914393 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242304276 0 0) 0x7fd4880b3dd0 con 0x7fd48806b690 2024-03-06T07:26:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd4877fe640 1 -- 172.21.15.72:0/1395914393 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd478003520 con 0x7fd48806b690 2024-03-06T07:26:19.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd48d288640 1 -- 172.21.15.72:0/1395914393 >> v1:172.21.15.146:6789/0 conn(0x7fd4881459b0 legacy=0x7fd48806af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd48d288640 1 -- 172.21.15.72:0/1395914393 >> v1:172.21.15.72:6789/0 conn(0x7fd48806b690 legacy=0x7fd4880af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.951+0000 7fd48d288640 1 -- 172.21.15.72:0/1395914393 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48815d620 con 0x7fd4880b4b90 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd48d288640 1 --2- 172.21.15.72:0/1395914393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4880b4b90 0x7fd4880aff20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd480009d90 tx=0x7fd48002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=620bdd3369a0b68e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd4877fe640 1 -- 172.21.15.72:0/1395914393 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480039070 con 0x7fd4880b4b90 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd4877fe640 1 -- 172.21.15.72:0/1395914393 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd48002cd00 con 0x7fd4880b4b90 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd4877fe640 1 -- 172.21.15.72:0/1395914393 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480034660 con 0x7fd4880b4b90 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd48eceb640 1 -- 172.21.15.72:0/1395914393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4880b4b90 msgr2=0x7fd4880aff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1395914393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4880b4b90 0x7fd4880aff20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd480009d90 tx=0x7fd48002bf50 comp rx=0 tx=0).stop 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd48eceb640 1 -- 172.21.15.72:0/1395914393 shutdown_connections 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1395914393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4880b4b90 0x7fd4880aff20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.952+0000 7fd48eceb640 1 -- 172.21.15.72:0/1395914393 >> 172.21.15.72:0/1395914393 conn(0x7fd48806c090 msgr2=0x7fd48806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:19.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.953+0000 7fd48eceb640 1 -- 172.21.15.72:0/1395914393 shutdown_connections 2024-03-06T07:26:19.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.953+0000 7fd48eceb640 1 -- 172.21.15.72:0/1395914393 wait complete. 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.954+0000 7fd48eceb640 1 Processor -- start 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.954+0000 7fd48eceb640 1 -- start start 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.954+0000 7fd48eceb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd48806b690 0x7fd48812b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.954+0000 7fd48eceb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4880b4b90 0x7fd48809d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.954+0000 7fd48eceb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 0x7fd48813eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48eceb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48815ddc0 con 0x7fd4881459b0 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48eceb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48815dc40 con 0x7fd48806b690 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48eceb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd48815df40 con 0x7fd4880b4b90 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 0x7fd48813eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 0x7fd48813eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59762/0 (socket says 172.21.15.72:59762) 2024-03-06T07:26:19.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 -- 172.21.15.72:0/1935790354 learned_addr learned my addr 172.21.15.72:0/1935790354 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd487fff640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4880b4b90 0x7fd48809d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48ca87640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd48806b690 0x7fd48812b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 -- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4880b4b90 msgr2=0x7fd48809d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4880b4b90 0x7fd48809d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 -- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd48806b690 msgr2=0x7fd48812b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd48806b690 0x7fd48812b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 -- 172.21.15.72:0/1935790354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48809d7f0 con 0x7fd4881459b0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48d288640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 0x7fd48813eb70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fd480009d90 tx=0x7fd480004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480039050 con 0x7fd4881459b0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd480045070 con 0x7fd4881459b0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48809dad0 con 0x7fd4881459b0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480009450 con 0x7fd4881459b0 2024-03-06T07:26:19.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.955+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48809e090 con 0x7fd4881459b0 2024-03-06T07:26:19.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.956+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488068050 con 0x7fd4881459b0 2024-03-06T07:26:19.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.957+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd48003e850 con 0x7fd4881459b0 2024-03-06T07:26:19.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.957+0000 7fd4857fa640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd46c06b500 0x7fd46c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:19.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.957+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd4800acf20 con 0x7fd4881459b0 2024-03-06T07:26:19.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.957+0000 7fd48ca87640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd46c06b500 0x7fd46c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:19.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.958+0000 7fd48ca87640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd46c06b500 0x7fd46c06d9c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd47c007fd0 tx=0x7fd47c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:19.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:19.962+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd480032e20 con 0x7fd4881459b0 2024-03-06T07:26:20.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.170+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd488145110 con 0x7fd4881459b0 2024-03-06T07:26:20.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.171+0000 7fd4857fa640 1 -- 172.21.15.72:0/1935790354 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4185 (secure 0 0 0) 0x7fd48007bca0 con 0x7fd4881459b0 2024-03-06T07:26:20.174 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:20.174 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":53,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:19.834201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:replay","state_seq":1,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 53 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.172+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd46c06b500 msgr2=0x7fd46c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.172+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd46c06b500 0x7fd46c06d9c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fd47c007fd0 tx=0x7fd47c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.172+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 msgr2=0x7fd48813eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.172+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 0x7fd48813eb70 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fd480009d90 tx=0x7fd480004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 shutdown_connections 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd46c06b500 0x7fd46c06d9c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4881459b0 0x7fd48813eb70 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4880b4b90 0x7fd48809d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 --2- 172.21.15.72:0/1935790354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd48806b690 0x7fd48812b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 >> 172.21.15.72:0/1935790354 conn(0x7fd48806c090 msgr2=0x7fd4880b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 shutdown_connections 2024-03-06T07:26:20.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:20.173+0000 7fd48eceb640 1 -- 172.21.15.72:0/1935790354 wait complete. 2024-03-06T07:26:20.374 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:20.376 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:20.588 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:06.088200+0000', 'modified': '2024-03-06T07:26:19.834201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7599': {'gid': 7599, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.72:6835/4195924898', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 4195924898}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 4195924898}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:20.588 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7599:up:replay 2024-03-06T07:26:21.589 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:21.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.698+0000 7fd5d6a4e640 1 Processor -- start 2024-03-06T07:26:21.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.698+0000 7fd5d6a4e640 1 -- start start 2024-03-06T07:26:21.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d6a4e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd5d0145900 0x7fd5d00cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d6a4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d006e160 0x7fd5d006e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d6a4e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d006d3a0 con 0x7fd5d00cf610 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d6a4e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d006d150 con 0x7fd5d0145900 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d6a4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d006d680 con 0x7fd5d006e160 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd5d0145900 0x7fd5d00cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd5d0145900 0x7fd5d00cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59770/0 (socket says 172.21.15.72:59770) 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 -- 172.21.15.72:0/1511568924 learned_addr learned my addr 172.21.15.72:0/1511568924 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d4feb640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d006e160 0x7fd5d006e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 -- 172.21.15.72:0/1511568924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d006e160 msgr2=0x7fd5d006e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d006e160 0x7fd5d006e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 -- 172.21.15.72:0/1511568924 >> v1:172.21.15.146:6789/0 conn(0x7fd5d00cf610 legacy=0x7fd5d015c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5cffff640 1 -- 172.21.15.72:0/1511568924 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d006d960 con 0x7fd5d0145900 2024-03-06T07:26:21.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.699+0000 7fd5d4feb640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d006e160 0x7fd5d006e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.700+0000 7fd5cffff640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d00cdb30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009870 tx=0x7fd5c002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e867e0db561ad0d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.700+0000 7fd5ceffd640 1 -- 172.21.15.72:0/1511568924 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0038070 con 0x7fd5d0145900 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.700+0000 7fd5ceffd640 1 -- 172.21.15.72:0/1511568924 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c0031740 con 0x7fd5d0145900 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.700+0000 7fd5ceffd640 1 -- 172.21.15.72:0/1511568924 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c002f050 con 0x7fd5d0145900 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/1511568924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 msgr2=0x7fd5d00cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d00cdb30 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009870 tx=0x7fd5c002b7e0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/1511568924 shutdown_connections 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d006e160 0x7fd5d006e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/1511568924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d00cdb30 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/1511568924 >> 172.21.15.72:0/1511568924 conn(0x7fd5d006b160 msgr2=0x7fd5d006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/1511568924 shutdown_connections 2024-03-06T07:26:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.701+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/1511568924 wait complete. 2024-03-06T07:26:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.702+0000 7fd5d6a4e640 1 Processor -- start 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.702+0000 7fd5d6a4e640 1 -- start start 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d6a4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d006e160 0x7fd5d013e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d6a4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00cf610 0x7fd5d015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d6a4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d01f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d6a4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00cfa70 con 0x7fd5d0145900 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d6a4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d015cb40 con 0x7fd5d006e160 2024-03-06T07:26:21.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d6a4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00cfbf0 con 0x7fd5d00cf610 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d01f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d01f52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59774/0 (socket says 172.21.15.72:59774) 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 -- 172.21.15.72:0/559516530 learned_addr learned my addr 172.21.15.72:0/559516530 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5cffff640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d006e160 0x7fd5d013e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5cf7fe640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00cf610 0x7fd5d015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 -- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00cf610 msgr2=0x7fd5d015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00cf610 0x7fd5d015b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 -- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d006e160 msgr2=0x7fd5d013e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d006e160 0x7fd5d013e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5d4feb640 1 -- 172.21.15.72:0/559516530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d013f0d0 con 0x7fd5d0145900 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5cffff640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d006e160 0x7fd5d013e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.703+0000 7fd5cf7fe640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00cf610 0x7fd5d015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.704+0000 7fd5d4feb640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d01f52c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800cd80 tx=0x7fd5c800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.704+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800f680 con 0x7fd5d0145900 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.704+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c800f840 con 0x7fd5d0145900 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.704+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d013f3b0 con 0x7fd5d0145900 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.704+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800fc30 con 0x7fd5d0145900 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.704+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d01e5db0 con 0x7fd5d0145900 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.705+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d0069ee0 con 0x7fd5d0145900 2024-03-06T07:26:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.705+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5c80213f0 con 0x7fd5d0145900 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.706+0000 7fd5ccff9640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5bc06b4d0 0x7fd5bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.706+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd5c808b090 con 0x7fd5d0145900 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.706+0000 7fd5cffff640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5bc06b4d0 0x7fd5bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:21.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.706+0000 7fd5cffff640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5bc06b4d0 0x7fd5bc06d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0035000 tx=0x7fd5c0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:21.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.710+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5c8014030 con 0x7fd5d0145900 2024-03-06T07:26:21.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.923+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5d00b39d0 con 0x7fd5d0145900 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.924+0000 7fd5ccff9640 1 -- 172.21.15.72:0/559516530 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4970 (secure 0 0 0) 0x7fd5c8059ea0 con 0x7fd5d0145900 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":55,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:20.844098+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:21.926 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 55 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5bc06b4d0 msgr2=0x7fd5bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5bc06b4d0 0x7fd5bc06d990 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0035000 tx=0x7fd5c0008030 comp rx=0 tx=0).stop 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 msgr2=0x7fd5d01f52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d01f52c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800cd80 tx=0x7fd5c800a650 comp rx=0 tx=0).stop 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 shutdown_connections 2024-03-06T07:26:21.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5bc06b4d0 0x7fd5bc06d990 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5d0145900 0x7fd5d01f52c0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00cf610 0x7fd5d015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 --2- 172.21.15.72:0/559516530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d006e160 0x7fd5d013e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 >> 172.21.15.72:0/559516530 conn(0x7fd5d006b160 msgr2=0x7fd5d015dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 shutdown_connections 2024-03-06T07:26:21.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:21.926+0000 7fd5d6a4e640 1 -- 172.21.15.72:0/559516530 wait complete. 2024-03-06T07:26:22.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:06.088200+0000', 'modified': '2024-03-06T07:26:20.844098+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7599': {'gid': 7599, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.72:6835/4195924898', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 4195924898}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 4195924898}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:22.341 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7599:up:rejoin 2024-03-06T07:26:23.342 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:23.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.451+0000 7f12b839a640 1 Processor -- start 2024-03-06T07:26:23.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.452+0000 7f12b839a640 1 -- start start 2024-03-06T07:26:23.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.452+0000 7f12b839a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 0x7f12b006de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.452+0000 7f12b839a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b015b0c0 con 0x7f12b015abc0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b839a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b015b3a0 con 0x7f12b0154af0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b839a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b00cf9a0 con 0x7f12b006d9e0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5935640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f12b015abc0 0x7f12b0072aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50382/0 (socket says 172.21.15.72:50382) 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5935640 1 -- 172.21.15.72:0/3847167779 learned_addr learned my addr 172.21.15.72:0/3847167779 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b6937640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 0x7f12b006de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610752773 0 0) 0x7f12b015b0c0 con 0x7f12b015abc0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c003440 con 0x7f12b015abc0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022146960 0 0) 0x7f12b015b3a0 con 0x7f12b0154af0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b015b0c0 con 0x7f12b0154af0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 122157022 0 0) 0x7f129c003440 con 0x7f12b015abc0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 msgr2=0x7f12b006de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 0x7f12b006de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 >> v1:172.21.15.146:6789/0 conn(0x7f12b0154af0 legacy=0x7f12b00cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b015b680 con 0x7f12b015abc0 2024-03-06T07:26:23.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f12a0001690 con 0x7f12b015abc0 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 >> v1:172.21.15.72:6789/0 conn(0x7f12b015abc0 legacy=0x7f12b0072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f129c003c60 0x7f129c008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 0x7f129c00ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f129c00d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c00dc10 con 0x7f129c00b2e0 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c00ddc0 con 0x7f129c003c60 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c00dfc0 con 0x7f12b006d9e0 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.453+0000 7f12b6937640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f129c00d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6136640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f129c003c60 0x7f129c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b5935640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 0x7f129c00ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6937640 1 -- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 msgr2=0x7f129c00ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6937640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b006d9e0 0x7f129c00ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6937640 1 -- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f129c003c60 msgr2=0x7f129c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6937640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f129c003c60 0x7f129c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6937640 1 -- 172.21.15.72:0/3847167779 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c003440 con 0x7f129c00b2e0 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.454+0000 7f12b6937640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f129c00d6d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009f90 tx=0x7f129c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc3855dc7c624f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac039070 con 0x7f129c00b2e0 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac02cbd0 con 0x7f129c00b2e0 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b5134640 1 -- 172.21.15.72:0/3847167779 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac009c10 con 0x7f129c00b2e0 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b839a640 1 -- 172.21.15.72:0/3847167779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 msgr2=0x7f129c00d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b839a640 1 --2- 172.21.15.72:0/3847167779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f129c00d6d0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f12ac009f90 tx=0x7f129c003730 comp rx=0 tx=0).stop 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b6937640 1 -- 172.21.15.72:0/3847167779 reap_dead start 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b839a640 1 -- 172.21.15.72:0/3847167779 shutdown_connections 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b839a640 1 -- 172.21.15.72:0/3847167779 >> 172.21.15.72:0/3847167779 conn(0x7f12b006b160 msgr2=0x7f12b006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b839a640 1 -- 172.21.15.72:0/3847167779 shutdown_connections 2024-03-06T07:26:23.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.455+0000 7f12b839a640 1 -- 172.21.15.72:0/3847167779 wait complete. 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 Processor -- start 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 -- start start 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f12b013e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b00b1dc0 0x7f12b00b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b01e3a10 0x7f12b01e3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0154ed0 con 0x7f129c00b2e0 2024-03-06T07:26:23.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0070d70 con 0x7f12b01e3a10 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b839a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b0155050 con 0x7f12b00b1dc0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b6136640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f12b013e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b6136640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f12b013e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56526/0 (socket says 172.21.15.72:56526) 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b6136640 1 -- 172.21.15.72:0/1408778163 learned_addr learned my addr 172.21.15.72:0/1408778163 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b5935640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b00b1dc0 0x7f12b00b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.457+0000 7f12b6937640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b01e3a10 0x7f12b01e3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b6136640 1 -- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b00b1dc0 msgr2=0x7f12b00b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b6136640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b00b1dc0 0x7f12b00b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b6136640 1 -- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b01e3a10 msgr2=0x7f12b01e3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b6136640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b01e3a10 0x7f12b01e3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b6136640 1 -- 172.21.15.72:0/1408778163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b01e1590 con 0x7f129c00b2e0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b6136640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f12b013e540 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f12a800f7d0 tx=0x7f12a800fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a800fe80 con 0x7f129c00b2e0 2024-03-06T07:26:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a8018070 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b01e1870 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a800d730 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.458+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b01e1e30 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.459+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b0069ee0 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.460+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f12a800d8d0 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.460+0000 7f12a6ffd640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.460+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f12a808a0a0 con 0x7f129c00b2e0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.460+0000 7f12b5935640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.460+0000 7f12b5935640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f12b01e4a10 tx=0x7f12a0001e40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:23.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.464+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f12a8058860 con 0x7f129c00b2e0 2024-03-06T07:26:23.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.672+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12b0072370 con 0x7f129c00b2e0 2024-03-06T07:26:23.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.673+0000 7f12a6ffd640 1 -- 172.21.15.72:0/1408778163 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f12a805c900 con 0x7f129c00b2e0 2024-03-06T07:26:23.675 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:23.676 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:21.846156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:clientreplay","state_seq":5,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-03-06T07:26:23.676 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 56 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 msgr2=0x7f129406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f12b01e4a10 tx=0x7f12a0001e40 comp rx=0 tx=0).stop 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 msgr2=0x7f12b013e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f12b013e540 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f12a800f7d0 tx=0x7f12a800fc90 comp rx=0 tx=0).stop 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 shutdown_connections 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f129406b4d0 0x7f129406d990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12b01e3a10 0x7f12b01e3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12b00b1dc0 0x7f12b00b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 --2- 172.21.15.72:0/1408778163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f129c00b2e0 0x7f12b013e540 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 >> 172.21.15.72:0/1408778163 conn(0x7f12b006b160 msgr2=0x7f12b0153c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 shutdown_connections 2024-03-06T07:26:23.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:23.675+0000 7f12b839a640 1 -- 172.21.15.72:0/1408778163 wait complete. 2024-03-06T07:26:24.091 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:06.088200+0000', 'modified': '2024-03-06T07:26:21.846156+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7599': {'gid': 7599, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:clientreplay', 'state_seq': 5, 'addr': '172.21.15.72:6835/4195924898', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 4195924898}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 4195924898}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:26:24.091 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7599:up:clientreplay 2024-03-06T07:26:24.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-03-06T07:26:24.908 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-03-06T07:26:24.906+0000 7f4208063640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:26:25.093 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:25.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.218+0000 7f5d205a0640 1 Processor -- start 2024-03-06T07:26:25.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.218+0000 7f5d205a0640 1 -- start start 2024-03-06T07:26:25.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.219+0000 7f5d205a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d18145900 0x7f5d180cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.219+0000 7f5d205a0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d1806e160 0x7f5d1806e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.219+0000 7f5d205a0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1806d3a0 con 0x7f5d180cf610 2024-03-06T07:26:25.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.219+0000 7f5d205a0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1806d150 con 0x7f5d1806e160 2024-03-06T07:26:25.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.219+0000 7f5d205a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1806d680 con 0x7f5d18145900 2024-03-06T07:26:25.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d1806e160 0x7f5d1806e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5d1806e160 0x7f5d1806e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:60020/0 (socket says 172.21.15.72:60020) 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/1704343191 learned_addr learned my addr 172.21.15.72:0/1704343191 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1db3b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5d180cf610 0x7f5d1815c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:43448/0 (socket says 172.21.15.72:43448) 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1e33c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d18145900 0x7f5d180cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d18145900 msgr2=0x7f5d180cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 --2- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d18145900 0x7f5d180cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/1704343191 >> v1:172.21.15.146:6789/0 conn(0x7f5d180cf610 legacy=0x7f5d1815c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/1704343191 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1806d960 con 0x7f5d1806e160 2024-03-06T07:26:25.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.220+0000 7f5d1e33c640 1 --2- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d18145900 0x7f5d180cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d1eb3d640 1 --2- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d1806e5f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009e50 tx=0x7f5d1002be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3224e7cd6149e0a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d1d33a640 1 -- 172.21.15.72:0/1704343191 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1002c9e0 con 0x7f5d1806e160 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d1d33a640 1 -- 172.21.15.72:0/1704343191 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1002cb80 con 0x7f5d1806e160 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d1d33a640 1 -- 172.21.15.72:0/1704343191 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d100343f0 con 0x7f5d1806e160 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 -- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 msgr2=0x7f5d1806e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 --2- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d1806e5f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009e50 tx=0x7f5d1002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 -- 172.21.15.72:0/1704343191 shutdown_connections 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 --2- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d1806e5f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 --2- 172.21.15.72:0/1704343191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d18145900 0x7f5d180cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 -- 172.21.15.72:0/1704343191 >> 172.21.15.72:0/1704343191 conn(0x7f5d1806b160 msgr2=0x7f5d1806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.221+0000 7f5d205a0640 1 -- 172.21.15.72:0/1704343191 shutdown_connections 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.222+0000 7f5d205a0640 1 -- 172.21.15.72:0/1704343191 wait complete. 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 Processor -- start 2024-03-06T07:26:25.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 -- start start 2024-03-06T07:26:25.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d18139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d180cf610 0x7f5d180b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 0x7f5d180b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d180cfa70 con 0x7f5d18145900 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1815cb40 con 0x7f5d1806e160 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d205a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d180cfbf0 con 0x7f5d180cf610 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d1eb3d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 0x7f5d180b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d1eb3d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 0x7f5d180b1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56528/0 (socket says 172.21.15.72:56528) 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.223+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/3028838497 learned_addr learned my addr 172.21.15.72:0/3028838497 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1e33c640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d18139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1db3b640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d180cf610 0x7f5d180b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d180cf610 msgr2=0x7f5d180b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1eb3d640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d180cf610 0x7f5d180b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 msgr2=0x7f5d18139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1eb3d640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d18139360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1eb3d640 1 -- 172.21.15.72:0/3028838497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1813ec90 con 0x7f5d18145900 2024-03-06T07:26:25.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d1eb3d640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 0x7f5d180b1240 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10043020 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1813ef70 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1003e450 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d10009510 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.224+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1814d7c0 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.225+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d18069ee0 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.226+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d10046020 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.226+0000 7f5d0effd640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.226+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5d100ac9c0 con 0x7f5d18145900 2024-03-06T07:26:25.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.226+0000 7f5d1e33c640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:25.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.227+0000 7f5d1e33c640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009560 tx=0x7f5d14009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:25.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.230+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d100aee20 con 0x7f5d18145900 2024-03-06T07:26:25.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.525+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d180b39d0 con 0x7f5d18145900 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.526+0000 7f5d0effd640 1 -- 172.21.15.72:0/3028838497 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f5d1004d030 con 0x7f5d18145900 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:25.528 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:25.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5cfc06b4d0 msgr2=0x7f5cfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f5d14009560 tx=0x7f5d14009190 comp rx=0 tx=0).stop 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 msgr2=0x7f5d180b1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 0x7f5d180b1240 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f5d10036040 tx=0x7f5d10004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 shutdown_connections 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5cfc06b4d0 0x7f5cfc06d990 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d18145900 0x7f5d180b1240 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d180cf610 0x7f5d180b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 --2- 172.21.15.72:0/3028838497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d1806e160 0x7f5d18139360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 >> 172.21.15.72:0/3028838497 conn(0x7f5d1806b160 msgr2=0x7f5d18143260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 shutdown_connections 2024-03-06T07:26:25.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:25.528+0000 7f5d205a0640 1 -- 172.21.15.72:0/3028838497 wait complete. 2024-03-06T07:26:25.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:26:06.088200+0000', 'modified': '2024-03-06T07:26:23.848449+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7599}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7599': {'gid': 7599, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.72:6835/4195924898', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 4195924898}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 4195924898}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7599, 'qdb_cluster': [7599]} 2024-03-06T07:26:25.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:26:25.944 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:26:26.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.043+0000 7fa2a5881640 1 Processor -- start 2024-03-06T07:26:26.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.043+0000 7fa2a5881640 1 -- start start 2024-03-06T07:26:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.044+0000 7fa2a5881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a015f5e0 0x7fa2a00b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.044+0000 7fa2a5881640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2a00cec60 0x7fa2a00cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.044+0000 7fa2a5881640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a00cd4d0 con 0x7fa2a00b16b0 2024-03-06T07:26:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa2a5881640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a00cf920 con 0x7fa2a00cec60 2024-03-06T07:26:26.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa2a5881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00cd7b0 con 0x7fa2a015f5e0 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2a00cec60 0x7fa2a00cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2a00cec60 0x7fa2a00cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56530/0 (socket says 172.21.15.72:56530) 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 -- 172.21.15.72:0/2634241796 learned_addr learned my addr 172.21.15.72:0/2634241796 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29effd640 1 --2- 172.21.15.72:0/2634241796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a015f5e0 0x7fa2a00b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 -- 172.21.15.72:0/2634241796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a015f5e0 msgr2=0x7fa2a00b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/2634241796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a015f5e0 0x7fa2a00b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 -- 172.21.15.72:0/2634241796 >> v1:172.21.15.146:6789/0 conn(0x7fa2a00b16b0 legacy=0x7fa2a00ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.045+0000 7fa29f7fe640 1 -- 172.21.15.72:0/2634241796 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a00cda90 con 0x7fa2a00cec60 2024-03-06T07:26:26.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.046+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/2634241796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00cec60 0x7fa2a00cfdb0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fa298009e50 tx=0x7fa29802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4693616d51763f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.046+0000 7fa29dffb640 1 -- 172.21.15.72:0/2634241796 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29802ca70 con 0x7fa2a00cec60 2024-03-06T07:26:26.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.046+0000 7fa29dffb640 1 -- 172.21.15.72:0/2634241796 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29802cc10 con 0x7fa2a00cec60 2024-03-06T07:26:26.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.046+0000 7fa29dffb640 1 -- 172.21.15.72:0/2634241796 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2980343f0 con 0x7fa2a00cec60 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 -- 172.21.15.72:0/2634241796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00cec60 msgr2=0x7fa2a00cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 --2- 172.21.15.72:0/2634241796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00cec60 0x7fa2a00cfdb0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fa298009e50 tx=0x7fa29802bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 -- 172.21.15.72:0/2634241796 shutdown_connections 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 --2- 172.21.15.72:0/2634241796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00cec60 0x7fa2a00cfdb0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 --2- 172.21.15.72:0/2634241796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a015f5e0 0x7fa2a00b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 -- 172.21.15.72:0/2634241796 >> 172.21.15.72:0/2634241796 conn(0x7fa2a00d1d50 msgr2=0x7fa2a00d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 -- 172.21.15.72:0/2634241796 shutdown_connections 2024-03-06T07:26:26.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.047+0000 7fa2a5881640 1 -- 172.21.15.72:0/2634241796 wait complete. 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.048+0000 7fa2a5881640 1 Processor -- start 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.048+0000 7fa2a5881640 1 -- start start 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 0x7fa2a014a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a00cec60 0x7fa2a014ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2a015f5e0 0x7fa2a00afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00cf270 con 0x7fa2a00b16b0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00cf0f0 con 0x7fa2a015f5e0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a00d2870 con 0x7fa2a00cec60 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 0x7fa2a014a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 0x7fa2a014a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56532/0 (socket says 172.21.15.72:56532) 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 -- 172.21.15.72:0/1539672195 learned_addr learned my addr 172.21.15.72:0/1539672195 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 -- 172.21.15.72:0/1539672195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a00cec60 msgr2=0x7fa2a014ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a00cec60 0x7fa2a014ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 -- 172.21.15.72:0/1539672195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2a015f5e0 msgr2=0x7fa2a00afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2a015f5e0 0x7fa2a00afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 -- 172.21.15.72:0/1539672195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a014a6b0 con 0x7fa2a00b16b0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa29effd640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 0x7fa2a014a170 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fa29000d600 tx=0x7fa29000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa287fff640 1 -- 172.21.15.72:0/1539672195 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900125f0 con 0x7fa2a00b16b0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa287fff640 1 -- 172.21.15.72:0/1539672195 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa290019070 con 0x7fa2a00b16b0 2024-03-06T07:26:26.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa287fff640 1 -- 172.21.15.72:0/1539672195 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900129f0 con 0x7fa2a00b16b0 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a00c8e20 con 0x7fa2a00b16b0 2024-03-06T07:26:26.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.049+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a014d550 con 0x7fa2a00b16b0 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.051+0000 7fa287fff640 1 -- 172.21.15.72:0/1539672195 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa290012b90 con 0x7fa2a00b16b0 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.051+0000 7fa287fff640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa28c06b4d0 0x7fa28c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.051+0000 7fa287fff640 1 -- 172.21.15.72:0/1539672195 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa29008a780 con 0x7fa2a00b16b0 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.051+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa28c06b4d0 0x7fa28c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.052+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa28c06b4d0 0x7fa28c06d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa294000f60 tx=0x7fa2940079d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.052+0000 7fa2a8463740 1 Processor -- start 2024-03-06T07:26:26.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.052+0000 7fa2a8463740 1 -- start start 2024-03-06T07:26:26.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56184084e870 0x56184084ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 0x561840854360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa2a8463740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618407b86b0 con 0x5618408528c0 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x561840761a60 con 0x56184087e600 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5618407b4750 con 0x56184084e870 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa29e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 0x561840854360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa29e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 0x561840854360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56540/0 (socket says 172.21.15.72:56540) 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa29e7fc640 1 -- 172.21.15.72:0/678771056 learned_addr learned my addr 172.21.15.72:0/678771056 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.053+0000 7fa29effd640 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56184084e870 0x56184084ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa29e7fc640 1 -- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56184084e870 msgr2=0x56184084ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56184084e870 0x56184084ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa29e7fc640 1 -- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 msgr2=0x56184087ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa29e7fc640 1 -- 172.21.15.72:0/678771056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56184077ff30 con 0x5618408528c0 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 0x561840854360 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa294016840 tx=0x7fa29400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7164d3cddf89601b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa2857fa640 1 -- 172.21.15.72:0/678771056 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2940162e0 con 0x5618408528c0 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa2857fa640 1 -- 172.21.15.72:0/678771056 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294005110 con 0x5618408528c0 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.054+0000 7fa2857fa640 1 -- 172.21.15.72:0/678771056 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2940138b0 con 0x5618408528c0 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 -- 172.21.15.72:0/678771056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 msgr2=0x561840854360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 0x561840854360 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fa294016840 tx=0x7fa29400f000 comp rx=0 tx=0).stop 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 -- 172.21.15.72:0/678771056 shutdown_connections 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408528c0 0x561840854360 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 --2- 172.21.15.72:0/678771056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56184084e870 0x56184084ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 -- 172.21.15.72:0/678771056 >> 172.21.15.72:0/678771056 conn(0x5618408508f0 msgr2=0x5618408521b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 -- 172.21.15.72:0/678771056 shutdown_connections 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 -- 172.21.15.72:0/678771056 wait complete. 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 -- start start 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.055+0000 7fa2a8463740 1 Processor -- start 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x561840881cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x561840873d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x561840897160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x561840852d50 con 0x5618408822d0 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x561840774ea0 con 0x56184087e600 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x561840853030 con 0x561840877220 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa2a8463740 10 client.0 _ll_register_callbacks cb 0x561840774870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa29f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x561840897160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa29f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x561840897160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56544/0 (socket says 172.21.15.72:56544) 2024-03-06T07:26:26.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa29f7fe640 1 -- 172.21.15.72:0/590399817 learned_addr learned my addr 172.21.15.72:0/590399817 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x561840873d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.056+0000 7fa29effd640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x561840881cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29f7fe640 1 -- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 msgr2=0x561840873d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x561840873d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29f7fe640 1 -- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 msgr2=0x561840881cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x561840881cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29f7fe640 1 -- 172.21.15.72:0/590399817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561840775270 con 0x5618408822d0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x561840897160 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa298009f70 tx=0x7fa298004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f6a2a950892d9b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa27e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa27e7fc640 1 -- 172.21.15.72:0/590399817 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298045070 con 0x5618408822d0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa27e7fc640 1 -- 172.21.15.72:0/590399817 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa29803f650 con 0x5618408822d0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa294005110 con 0x5618408822d0 2024-03-06T07:26:26.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa27e7fc640 1 -- 172.21.15.72:0/590399817 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa298050050 con 0x5618408822d0 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 10 client.7644 fetch_fsmap learned FSMap version 57 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x5618404cd2d0 con 0x5618408822d0 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa27e7fc640 1 -- 172.21.15.72:0/590399817 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294005110 con 0x5618408822d0 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa27e7fc640 1 -- 172.21.15.72:0/590399817 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 57) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa298053020 con 0x5618408822d0 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 10 client.7644 fetch_fsmap finished waiting for FSMap version 57 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 10 client.7644 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 20 client.7644 populate_metadata read hostname 'smithi072' 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x56184087ff30 0x5618408b8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 4 client.7644 mds_command: new command op to 7599 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 --> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618407c5bf0 con 0x56184087ff30 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.057+0000 7fa29effd640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x56184087ff30 0x5618408b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa29effd640 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x56184087ff30 0x5618408b8810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b08955194e416825 server_cookie=3a3cc0936c1be42c in_seq=0 out_seq=0 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa27e7fc640 10 client.7644 ms_handle_connect on v2:172.21.15.72:6834/4195924898 2024-03-06T07:26:26.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa27e7fc640 1 -- 172.21.15.72:0/590399817 <== mds.0 v2:172.21.15.72:6834/4195924898 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5618407c5bf0 con 0x56184087ff30 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa27e7fc640 10 client.7644 handle_command_reply: tid=0 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa2a8463740 1 client.7644 shutdown 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 msgr2=0x561840897160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa2a8463740 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x561840897160 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fa298009f70 tx=0x7fa298004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 shutdown_connections 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.058+0000 7fa2a8463740 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x56184087ff30 0x5618408b8810 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa27e7fc640 0 client.7644 ms_handle_reset on v2:172.21.15.72:6834/4195924898 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x561840897160 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x561840873d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 1 --2- 172.21.15.72:0/590399817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x561840881cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 >> 172.21.15.72:0/590399817 conn(0x56184084e870 msgr2=0x56184084d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 shutdown_connections 2024-03-06T07:26:26.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 1 -- 172.21.15.72:0/590399817 wait complete. 2024-03-06T07:26:26.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.059+0000 7fa2a8463740 20 client.7644 trim_cache size 0 max 16384 2024-03-06T07:26:26.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.075+0000 7fa2a8463740 1 Processor -- start 2024-03-06T07:26:26.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.075+0000 7fa2a8463740 1 -- start start 2024-03-06T07:26:26.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.076+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561840877220 0x561840881cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.076+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 0x5618408bdca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.076+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5618408822d0 0x5618408ac070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.076+0000 7fa2a8463740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a01fd410 con 0x56184087e600 2024-03-06T07:26:26.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.076+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a01fc840 con 0x561840877220 2024-03-06T07:26:26.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.076+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a01fce10 con 0x5618408822d0 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 0x5618408bdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 0x5618408bdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56548/0 (socket says 172.21.15.72:56548) 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 -- 172.21.15.72:0/842072232 learned_addr learned my addr 172.21.15.72:0/842072232 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5618408822d0 0x5618408ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29effd640 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561840877220 0x561840881cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 -- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5618408822d0 msgr2=0x5618408ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5618408822d0 0x5618408ac070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 -- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561840877220 msgr2=0x561840881cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561840877220 0x561840881cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 -- 172.21.15.72:0/842072232 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618405976b0 con 0x56184087e600 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 0x5618408bdca0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa294016f50 tx=0x7fa294011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8000f21b7c2c1ba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa284ff9640 1 -- 172.21.15.72:0/842072232 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2940162e0 con 0x56184087e600 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa284ff9640 1 -- 172.21.15.72:0/842072232 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294016480 con 0x56184087e600 2024-03-06T07:26:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.077+0000 7fa284ff9640 1 -- 172.21.15.72:0/842072232 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2940052d0 con 0x56184087e600 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 -- 172.21.15.72:0/842072232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 msgr2=0x5618408bdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 0x5618408bdca0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fa294016f50 tx=0x7fa294011000 comp rx=0 tx=0).stop 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 -- 172.21.15.72:0/842072232 shutdown_connections 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5618408822d0 0x5618408ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56184087e600 0x5618408bdca0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 --2- 172.21.15.72:0/842072232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561840877220 0x561840881cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 -- 172.21.15.72:0/842072232 >> 172.21.15.72:0/842072232 conn(0x56184084e870 msgr2=0x56184089aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 -- 172.21.15.72:0/842072232 shutdown_connections 2024-03-06T07:26:26.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.078+0000 7fa2a8463740 1 -- 172.21.15.72:0/842072232 wait complete. 2024-03-06T07:26:26.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.079+0000 7fa2a8463740 1 -- start start 2024-03-06T07:26:26.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.079+0000 7fa2a8463740 1 Processor -- start 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x56184087d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x5618409092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x561840897c30 con 0x5618408822d0 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618408811f0 con 0x56184087e600 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56184087fa50 con 0x561840877220 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa2a8463740 10 client.0 _ll_register_callbacks cb 0x7fa29802cc10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x5618409092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x5618409092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56552/0 (socket says 172.21.15.72:56552) 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 -- 172.21.15.72:0/749432239 learned_addr learned my addr 172.21.15.72:0/749432239 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29effd640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x56184087d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 -- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 msgr2=0x56184087d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x56184087d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 -- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 msgr2=0x56184087da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29f7fe640 1 -- 172.21.15.72:0/749432239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56184090a280 con 0x5618408822d0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.080+0000 7fa29e7fc640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa29f7fe640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x5618409092e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa298009f70 tx=0x7fa298009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7374c71221daaf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa27e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa27e7fc640 1 -- 172.21.15.72:0/749432239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298045040 con 0x5618408822d0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa27e7fc640 1 -- 172.21.15.72:0/749432239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa29804b030 con 0x5618408822d0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa294016480 con 0x5618408822d0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa27e7fc640 1 -- 172.21.15.72:0/749432239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa298061050 con 0x5618408822d0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 10 client.7650 fetch_fsmap learned FSMap version 57 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x56184087df80 con 0x5618408822d0 2024-03-06T07:26:26.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa27e7fc640 1 -- 172.21.15.72:0/749432239 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa294016480 con 0x5618408822d0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa27e7fc640 1 -- 172.21.15.72:0/749432239 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 57) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fa298064020 con 0x5618408822d0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 10 client.7650 fetch_fsmap finished waiting for FSMap version 57 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 10 client.7650 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 20 client.7650 populate_metadata read hostname 'smithi072' 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x561840909cc0 0x56184090c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 4 client.7650 mds_command: new command op to 7599 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 --> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56184084f630 con 0x561840909cc0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.081+0000 7fa29effd640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x561840909cc0 0x56184090c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa27e7fc640 10 client.7650 ms_handle_connect on v2:172.21.15.72:6834/4195924898 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa29effd640 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x561840909cc0 0x56184090c8d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d1e1cd493771d28 server_cookie=b50ba5ca4b3752ef in_seq=0 out_seq=0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa27e7fc640 1 -- 172.21.15.72:0/749432239 <== mds.0 v2:172.21.15.72:6834/4195924898 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x56184084f630 con 0x561840909cc0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa27e7fc640 10 client.7650 handle_command_reply: tid=0 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 client.7650 shutdown 2024-03-06T07:26:26.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 msgr2=0x5618409092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x5618409092e0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fa298009f70 tx=0x7fa298009000 comp rx=0 tx=0).stop 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 shutdown_connections 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:6834/4195924898,v1:172.21.15.72:6835/4195924898] conn(0x561840909cc0 0x56184090c8d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5618408822d0 0x5618409092e0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa27e7fc640 0 client.7650 ms_handle_reset on v2:172.21.15.72:6834/4195924898 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56184087e600 0x56184087da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 --2- 172.21.15.72:0/749432239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561840877220 0x56184087d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 >> 172.21.15.72:0/749432239 conn(0x56184084e870 msgr2=0x561840881710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 shutdown_connections 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 1 -- 172.21.15.72:0/749432239 wait complete. 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.082+0000 7fa2a8463740 20 client.7650 trim_cache size 0 max 16384 2024-03-06T07:26:26.090 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 7599, 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 6.247773435, 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 57, 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 98, 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 98, 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 12.051994347000001 2024-03-06T07:26:26.091 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa28c06b4d0 msgr2=0x7fa28c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa28c06b4d0 0x7fa28c06d990 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa294000f60 tx=0x7fa2940079d0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 msgr2=0x7fa2a014a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 0x7fa2a014a170 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fa29000d600 tx=0x7fa29000dac0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 shutdown_connections 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa28c06b4d0 0x7fa28c06d990 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2a015f5e0 0x7fa2a00afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2a00cec60 0x7fa2a014ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 --2- 172.21.15.72:0/1539672195 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2a00b16b0 0x7fa2a014a170 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 >> 172.21.15.72:0/1539672195 conn(0x7fa2a00d1d50 msgr2=0x7fa2a00d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 shutdown_connections 2024-03-06T07:26:26.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:26.083+0000 7fa2a5881640 1 -- 172.21.15.72:0/1539672195 wait complete. 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> subdir = "delete_me" 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> size_unit = 1024 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> file_multiplier = 200 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-03-06T07:26:26.096 DEBUG:teuthology.orchestra.run.smithi072:> for i in range(0, file_multiplier): 2024-03-06T07:26:26.097 DEBUG:teuthology.orchestra.run.smithi072:> for size in range(0, 16*size_unit, size_unit): 2024-03-06T07:26:26.097 DEBUG:teuthology.orchestra.run.smithi072:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-03-06T07:26:26.097 DEBUG:teuthology.orchestra.run.smithi072:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-03-06T07:26:26.097 DEBUG:teuthology.orchestra.run.smithi072:> f.write(size * '"'"'x'"'"') 2024-03-06T07:26:26.097 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:26:32.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:26:32.367 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config get mds_max_purge_ops 2024-03-06T07:26:32.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:32.368 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-03-06T07:26:32.571 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_max_purge_ops":"100000000"} 2024-03-06T07:26:32.571 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-03-06T07:26:32.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_files 2024-03-06T07:26:32.684 INFO:teuthology.orchestra.run.smithi146.stdout:{"mds_max_purge_files":"6"} 2024-03-06T07:26:32.685 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-03-06T07:26:32.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:32.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.788+0000 7fbc6795c640 1 Processor -- start 2024-03-06T07:26:32.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.788+0000 7fbc6795c640 1 -- start start 2024-03-06T07:26:32.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc6795c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbc600d0950 0x7fbc600d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc6795c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600d1ea0 con 0x7fbc6015e150 2024-03-06T07:26:32.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc6795c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600d2180 con 0x7fbc600b9060 2024-03-06T07:26:32.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc6795c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc600d12f0 con 0x7fbc600d0950 2024-03-06T07:26:32.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc656f8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbc6015e150 0x7fbc600cf660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36960/0 (socket says 172.21.15.72:36960) 2024-03-06T07:26:32.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc656f8640 1 -- 172.21.15.72:0/1747947341 learned_addr learned my addr 172.21.15.72:0/1747947341 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:32.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc64ef7640 1 --2- 172.21.15.72:0/1747947341 >> v2:172.21.15.72:3300/0 conn(0x7fbc600d0950 0x7fbc600d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc64ef7640 1 -- 172.21.15.72:0/1747947341 >> v1:172.21.15.146:6789/0 conn(0x7fbc600b9060 legacy=0x7fbc600b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:32.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc64ef7640 1 -- 172.21.15.72:0/1747947341 >> v1:172.21.15.72:6789/0 conn(0x7fbc6015e150 legacy=0x7fbc600cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.790+0000 7fbc64ef7640 1 -- 172.21.15.72:0/1747947341 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600d2460 con 0x7fbc600d0950 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.791+0000 7fbc64ef7640 1 --2- 172.21.15.72:0/1747947341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc600d0950 0x7fbc600d0db0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097a0 tx=0x7fbc5002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31e097725896651 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.791+0000 7fbc57fff640 1 -- 172.21.15.72:0/1747947341 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc500315a0 con 0x7fbc600d0950 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.791+0000 7fbc57fff640 1 -- 172.21.15.72:0/1747947341 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50037070 con 0x7fbc600d0950 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.791+0000 7fbc57fff640 1 -- 172.21.15.72:0/1747947341 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50031960 con 0x7fbc600d0950 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 -- 172.21.15.72:0/1747947341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc600d0950 msgr2=0x7fbc600d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 --2- 172.21.15.72:0/1747947341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc600d0950 0x7fbc600d0db0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097a0 tx=0x7fbc5002b840 comp rx=0 tx=0).stop 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 -- 172.21.15.72:0/1747947341 shutdown_connections 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 --2- 172.21.15.72:0/1747947341 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc600d0950 0x7fbc600d0db0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 -- 172.21.15.72:0/1747947341 >> 172.21.15.72:0/1747947341 conn(0x7fbc6009d230 msgr2=0x7fbc600d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:32.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 -- 172.21.15.72:0/1747947341 shutdown_connections 2024-03-06T07:26:32.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.792+0000 7fbc6795c640 1 -- 172.21.15.72:0/1747947341 wait complete. 2024-03-06T07:26:32.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.794+0000 7fbc6795c640 1 Processor -- start 2024-03-06T07:26:32.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.794+0000 7fbc6795c640 1 -- start start 2024-03-06T07:26:32.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc6795c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc600b9060 0x7fbc6014c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc6795c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc600d0950 0x7fbc601608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc6795c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 0x7fbc600cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc6795c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600d2f40 con 0x7fbc6015e150 2024-03-06T07:26:32.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc6795c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600d2ab0 con 0x7fbc600d0950 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc6795c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600b7dc0 con 0x7fbc600b9060 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 0x7fbc600cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 0x7fbc600cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57796/0 (socket says 172.21.15.72:57796) 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 -- 172.21.15.72:0/997716765 learned_addr learned my addr 172.21.15.72:0/997716765 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc64ef7640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc600d0950 0x7fbc601608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc656f8640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc600b9060 0x7fbc6014c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 -- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc600b9060 msgr2=0x7fbc6014c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc600b9060 0x7fbc6014c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 -- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc600d0950 msgr2=0x7fbc601608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc600d0950 0x7fbc601608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc65ef9640 1 -- 172.21.15.72:0/997716765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600c8c90 con 0x7fbc6015e150 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.795+0000 7fbc656f8640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc600b9060 0x7fbc6014c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.796+0000 7fbc65ef9640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 0x7fbc600cd040 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c780 tx=0x7fbc5800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.796+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58004b60 con 0x7fbc6015e150 2024-03-06T07:26:32.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.796+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc58004d00 con 0x7fbc6015e150 2024-03-06T07:26:32.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.796+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5800aa90 con 0x7fbc6015e150 2024-03-06T07:26:32.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.796+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc600c7bf0 con 0x7fbc6015e150 2024-03-06T07:26:32.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.796+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc600c81b0 con 0x7fbc6015e150 2024-03-06T07:26:32.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.797+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60069ee0 con 0x7fbc6015e150 2024-03-06T07:26:32.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.798+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc5800ac30 con 0x7fbc6015e150 2024-03-06T07:26:32.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.798+0000 7fbc55ffb640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4406b500 0x7fbc4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:32.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.799+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbc5808a510 con 0x7fbc6015e150 2024-03-06T07:26:32.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.799+0000 7fbc656f8640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4406b500 0x7fbc4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:32.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.799+0000 7fbc656f8640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4406b500 0x7fbc4406d9c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbc600cde90 tx=0x7fbc5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:32.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:32.807+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc58020070 con 0x7fbc6015e150 2024-03-06T07:26:33.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.075+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc600d4670 con 0x7fbc6015e150 2024-03-06T07:26:33.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.076+0000 7fbc55ffb640 1 -- 172.21.15.72:0/997716765 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fbc58059290 con 0x7fbc6015e150 2024-03-06T07:26:33.078 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:33.078 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:33.078 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:33.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4406b500 msgr2=0x7fbc4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4406b500 0x7fbc4406d9c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbc600cde90 tx=0x7fbc5c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:33.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 msgr2=0x7fbc600cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:33.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 0x7fbc600cd040 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c780 tx=0x7fbc5800cc40 comp rx=0 tx=0).stop 2024-03-06T07:26:33.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 shutdown_connections 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4406b500 0x7fbc4406d9c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc6015e150 0x7fbc600cd040 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc600d0950 0x7fbc601608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 --2- 172.21.15.72:0/997716765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc600b9060 0x7fbc6014c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 >> 172.21.15.72:0/997716765 conn(0x7fbc6009d230 msgr2=0x7fbc600b9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 shutdown_connections 2024-03-06T07:26:33.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:33.078+0000 7fbc6795c640 1 -- 172.21.15.72:0/997716765 wait complete. 2024-03-06T07:26:33.092 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:33.240 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":22,"msgr_send_messages":22,"msgr_recv_bytes":14845,"msgr_send_bytes":4222861,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015647504,"msgr_running_send_time":0.008524326,"msgr_running_recv_time":0.004101780,"msgr_running_fast_dispatch_time":0.001957818,"msgr_send_messages_queue_lat":{"avgcount":21,"sum":0.007728097,"avgtime":0.000368004},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4308,"msgr_send_messages":10209,"msgr_recv_bytes":1493529,"msgr_send_bytes":10395961,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.435359611,"msgr_running_send_time":0.228875813,"msgr_running_recv_time":0.250888338,"msgr_running_fast_dispatch_time":0.004076005,"msgr_send_messages_queue_lat":{"avgcount":10208,"sum":0.108802562,"avgtime":0.000010658},"msgr_handle_ack_lat":{"avgcount":14348,"sum":0.002166298,"avgtime":0.000000150},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92400},"AsyncMessenger::Worker-2":{"msgr_recv_messages":31,"msgr_send_messages":26,"msgr_recv_bytes":111714,"msgr_send_bytes":10208,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.004023855,"msgr_running_send_time":0.000759255,"msgr_running_recv_time":0.003576608,"msgr_running_fast_dispatch_time":0.000418290,"msgr_send_messages_queue_lat":{"avgcount":25,"sum":0.003799986,"avgtime":0.000151999},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107936,"msgr_send_encrypted_bytes":7296},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":818,"complete_latency":{"avgcount":74,"sum":0.953629354,"avgtime":0.012886883}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000151768,"avgtime":0.000037942}},"mds":{"request":3381,"reply":3378,"reply_latency":{"avgcount":3378,"sum":3.429257892,"avgtime":0.001015174},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3211,"inodes_top":0,"inodes_bottom":3211,"inodes_pin_tail":0,"inodes_pinned":3211,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3387,"traverse_hit":3386,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3100,"root_rbytes":1231872,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":162,"ceph_cap_op_grant":3236,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":876,"handle_client_caps_dirty":790,"handle_client_cap_release":0,"process_request_cap_release":3539},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":169,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":169,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4235,"evex":0,"evtrm":0,"ev":4238,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":12702171,"rdpos":4195244,"jlat":{"avgcount":11,"sum":0.053646211,"avgtime":0.004876928},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3380,"dn+":3380,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":82284,"heap":248164},"mds_server":{"dispatch_client_request":3384,"dispatch_server_request":0,"handle_client_request":3381,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":169,"sum":0.214867838,"avgtime":0.001271407},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3230,"average_load":3230,"avg_session_uptime":819,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":569521,"buffer_anon_items":658,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":4924464,"mds_co_items":24621,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":64,"op_send_bytes":8743420,"op_resend":0,"op_reply":64,"op_latency":{"avgcount":64,"sum":0.152996713,"avgtime":0.002390573},"op_inflight":0,"oplen_avg":{"avgcount":64,"sum":119},"op":64,"op_r":36,"op_w":28,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":11,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":260,"max":104857600,"get_started":0,"get":4357,"get_sum":1225241,"get_or_fail_fail":0,"get_or_fail_success":4357,"take":0,"take_sum":0,"put":4356,"put_sum":1224981,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":8744537,"put":64,"put_sum":8744537,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":64,"put":64,"put_sum":64,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":119259,"max":3758096384,"get_started":0,"get":4235,"get_sum":8506486,"get_or_fail_fail":0,"get_or_fail_success":4235,"take":0,"take_sum":0,"put":11,"put_sum":8387227,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:33.241 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 14845, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 22, "msgr_running_fast_dispatch_time": 0.001957818, "msgr_running_recv_time": 0.00410178, "msgr_running_send_time": 0.008524326, "msgr_running_total_time": 0.015647504, "msgr_send_bytes": 4222861, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 22, "msgr_send_messages_queue_lat": { "avgcount": 21, "avgtime": 0.000368004, "sum": 0.007728097 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 14348, "avgtime": 1.5e-07, "sum": 0.002166298 }, "msgr_recv_bytes": 1493529, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 4308, "msgr_running_fast_dispatch_time": 0.004076005, "msgr_running_recv_time": 0.250888338, "msgr_running_send_time": 0.228875813, "msgr_running_total_time": 0.435359611, "msgr_send_bytes": 10395961, "msgr_send_encrypted_bytes": 92400, "msgr_send_messages": 10209, "msgr_send_messages_queue_lat": { "avgcount": 10208, "avgtime": 1.0658e-05, "sum": 0.108802562 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 111714, "msgr_recv_encrypted_bytes": 107936, "msgr_recv_messages": 31, "msgr_running_fast_dispatch_time": 0.00041829, "msgr_running_recv_time": 0.003576608, "msgr_running_send_time": 0.000759255, "msgr_running_total_time": 0.004023855, "msgr_send_bytes": 10208, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 26, "msgr_send_messages_queue_lat": { "avgcount": 25, "avgtime": 0.000151999, "sum": 0.003799986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.012886883, "sum": 0.953629354 }, "queue_len": 818 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.7942e-05, "sum": 0.000151768 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3236, "ceph_cap_op_revoke": 162, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 876, "handle_client_caps_dirty": 790, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3211, "inodes_bottom": 3211, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3211, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3539, "q": 0, "reply": 3378, "reply_latency": { "avgcount": 3378, "avgtime": 0.001015174, "sum": 3.429257892 }, "request": 3381, "root_rbytes": 1231872, "root_rfiles": 3100, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3387, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3386, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 169, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 169, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 4238, "evadd": 4235, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 11, "avgtime": 0.004876928, "sum": 0.053646211 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 12702171 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3380, "dn+": 3380, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 82284 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3384, "dispatch_server_request": 0, "handle_client_request": 3381, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 169, "avgtime": 0.001271407, "sum": 0.214867838 } }, "mds_sessions": { "average_load": 3230, "avg_session_uptime": 819, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3230 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 569521, "buffer_anon_items": 658, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 4924464, "mds_co_items": 24621, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 15, "op": 64, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 64, "avgtime": 0.002390573, "sum": 0.152996713 }, "op_pg": 0, "op_r": 36, "op_reply": 64, "op_resend": 0, "op_rmw": 0, "op_send": 64, "op_send_bytes": 8743420, "op_w": 28, "oplen_avg": { "avgcount": 64, "sum": 119 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 11, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4357, "get_or_fail_fail": 0, "get_or_fail_success": 4357, "get_started": 0, "get_sum": 1225241, "max": 104857600, "put": 4356, "put_sum": 1224981, "take": 0, "take_sum": 0, "val": 260, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 64, "put_sum": 8744537, "take": 64, "take_sum": 8744537, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 64, "put_sum": 64, "take": 64, "take_sum": 64, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 4235, "get_or_fail_fail": 0, "get_or_fail_success": 4235, "get_started": 0, "get_sum": 8506486, "max": 3758096384, "put": 11, "put_sum": 8387227, "take": 0, "take_sum": 0, "val": 119259, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:33.242 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/169, 0/169 2024-03-06T07:26:34.242 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:34.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.372+0000 7f601fd02640 1 Processor -- start 2024-03-06T07:26:34.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.372+0000 7f601fd02640 1 -- start start 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601fd02640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f601815e150 0x7f60180cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601fd02640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601fd02640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60180d1ea0 con 0x7f60180cfba0 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601fd02640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60180d2ab0 con 0x7f601815e150 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601fd02640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60180d2180 con 0x7f60180d1060 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601da9e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f601815e150 0x7f60180cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601d29d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f60180cfba0 0x7f60180d0950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36972/0 (socket says 172.21.15.72:36972) 2024-03-06T07:26:34.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601d29d640 1 -- 172.21.15.72:0/789000903 learned_addr learned my addr 172.21.15.72:0/789000903 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601e29f640 1 --2- 172.21.15.72:0/789000903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601da9e640 1 -- 172.21.15.72:0/789000903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 msgr2=0x7f60180b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601da9e640 1 --2- 172.21.15.72:0/789000903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601da9e640 1 -- 172.21.15.72:0/789000903 >> v1:172.21.15.72:6789/0 conn(0x7f60180cfba0 legacy=0x7f60180d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.374+0000 7f601da9e640 1 -- 172.21.15.72:0/789000903 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60180d2460 con 0x7f601815e150 2024-03-06T07:26:34.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.375+0000 7f601da9e640 1 --2- 172.21.15.72:0/789000903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180cf660 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f60140097a0 tx=0x7f601402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b71ef499c7d43e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.375+0000 7f601ca9c640 1 -- 172.21.15.72:0/789000903 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60140315a0 con 0x7f601815e150 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.375+0000 7f601ca9c640 1 -- 172.21.15.72:0/789000903 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6014037070 con 0x7f601815e150 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.375+0000 7f601ca9c640 1 -- 172.21.15.72:0/789000903 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014031960 con 0x7f601815e150 2024-03-06T07:26:34.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 -- 172.21.15.72:0/789000903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 msgr2=0x7f60180cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 --2- 172.21.15.72:0/789000903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180cf660 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f60140097a0 tx=0x7f601402b840 comp rx=0 tx=0).stop 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 -- 172.21.15.72:0/789000903 shutdown_connections 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 --2- 172.21.15.72:0/789000903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 --2- 172.21.15.72:0/789000903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180cf660 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 -- 172.21.15.72:0/789000903 >> 172.21.15.72:0/789000903 conn(0x7f601809d230 msgr2=0x7f60180d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 -- 172.21.15.72:0/789000903 shutdown_connections 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.376+0000 7f601fd02640 1 -- 172.21.15.72:0/789000903 wait complete. 2024-03-06T07:26:34.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.377+0000 7f601fd02640 1 Processor -- start 2024-03-06T07:26:34.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.377+0000 7f601fd02640 1 -- start start 2024-03-06T07:26:34.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.378+0000 7f601fd02640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180cfba0 0x7f60180a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.378+0000 7f601fd02640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.378+0000 7f601fd02640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180b7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601fd02640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180d1b30 con 0x7f601815e150 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601fd02640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180d2f40 con 0x7f60180cfba0 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601fd02640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601815e530 con 0x7f60180d1060 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57814/0 (socket says 172.21.15.72:57814) 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 -- 172.21.15.72:0/2173847275 learned_addr learned my addr 172.21.15.72:0/2173847275 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601da9e640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180cfba0 0x7f60180a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601d29d640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 -- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 msgr2=0x7f60180cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 -- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180cfba0 msgr2=0x7f60180a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180cfba0 0x7f60180a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 -- 172.21.15.72:0/2173847275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601806e5e0 con 0x7f601815e150 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601e29f640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180b7250 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6010007e60 tx=0x7f601000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010014620 con 0x7f601815e150 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6010019070 con 0x7f601815e150 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010014a00 con 0x7f601815e150 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6018071d50 con 0x7f601815e150 2024-03-06T07:26:34.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.379+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60180a19d0 con 0x7f601815e150 2024-03-06T07:26:34.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.381+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018069ee0 con 0x7f601815e150 2024-03-06T07:26:34.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.381+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6010014ba0 con 0x7f601815e150 2024-03-06T07:26:34.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.382+0000 7f600e7fc640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ffc06b500 0x7f5ffc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:34.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.382+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f601008ab50 con 0x7f601815e150 2024-03-06T07:26:34.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.382+0000 7f601da9e640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ffc06b500 0x7f5ffc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:34.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.382+0000 7f601da9e640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ffc06b500 0x7f5ffc06d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6014036000 tx=0x7f601403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:34.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.389+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6010059960 con 0x7f601815e150 2024-03-06T07:26:34.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.710+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60180d0c10 con 0x7f601815e150 2024-03-06T07:26:34.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.710+0000 7f600e7fc640 1 -- 172.21.15.72:0/2173847275 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f6010059260 con 0x7f601815e150 2024-03-06T07:26:34.713 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:34.713 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:34.713 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.712+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ffc06b500 msgr2=0x7f5ffc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ffc06b500 0x7f5ffc06d9c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6014036000 tx=0x7f601403d040 comp rx=0 tx=0).stop 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 msgr2=0x7f60180b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180b7250 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6010007e60 tx=0x7f601000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 shutdown_connections 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5ffc06b500 0x7f5ffc06d9c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f601815e150 0x7f60180b7250 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180d1060 0x7f60180cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 --2- 172.21.15.72:0/2173847275 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60180cfba0 0x7f60180a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 >> 172.21.15.72:0/2173847275 conn(0x7f601809d230 msgr2=0x7f60180b8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 shutdown_connections 2024-03-06T07:26:34.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:34.713+0000 7f601fd02640 1 -- 172.21.15.72:0/2173847275 wait complete. 2024-03-06T07:26:34.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:34.857 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":22,"msgr_send_messages":22,"msgr_recv_bytes":14845,"msgr_send_bytes":4222861,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.015647504,"msgr_running_send_time":0.008524326,"msgr_running_recv_time":0.004101780,"msgr_running_fast_dispatch_time":0.001957818,"msgr_send_messages_queue_lat":{"avgcount":21,"sum":0.007728097,"avgtime":0.000368004},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":5971,"msgr_send_messages":13263,"msgr_recv_bytes":2089988,"msgr_send_bytes":13492214,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.559845445,"msgr_running_send_time":0.300604472,"msgr_running_recv_time":0.310650628,"msgr_running_fast_dispatch_time":0.005271391,"msgr_send_messages_queue_lat":{"avgcount":13262,"sum":0.148067557,"avgtime":0.000011164},"msgr_handle_ack_lat":{"avgcount":18728,"sum":0.002913902,"avgtime":0.000000155},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":92400},"AsyncMessenger::Worker-2":{"msgr_recv_messages":31,"msgr_send_messages":26,"msgr_recv_bytes":111714,"msgr_send_bytes":10208,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.004023855,"msgr_running_send_time":0.000759255,"msgr_running_recv_time":0.003576608,"msgr_running_fast_dispatch_time":0.000418290,"msgr_send_messages_queue_lat":{"avgcount":25,"sum":0.003799986,"avgtime":0.000151999},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":107936,"msgr_send_encrypted_bytes":7296},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":893,"complete_latency":{"avgcount":77,"sum":1.721635385,"avgtime":0.022358901}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000151768,"avgtime":0.000037942}},"mds":{"request":3777,"reply":3775,"reply_latency":{"avgcount":3775,"sum":4.463255916,"avgtime":0.001182319},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3776,"inodes_top":0,"inodes_bottom":3776,"inodes_pin_tail":0,"inodes_pinned":3776,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3783,"traverse_hit":3782,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3057,"root_rbytes":5236736,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":466,"ceph_cap_op_grant":4121,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2057,"handle_client_caps_dirty":1865,"handle_client_cap_release":0,"process_request_cap_release":4331},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":565,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":565,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":5812,"evex":0,"evtrm":0,"ev":5815,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":9,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16258953,"rdpos":4195244,"jlat":{"avgcount":12,"sum":0.064335778,"avgtime":0.005361314},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3776,"dn+":3776,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":86380,"heap":248164},"mds_server":{"dispatch_client_request":3780,"dispatch_server_request":0,"handle_client_request":3777,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":565,"sum":1.247354702,"avgtime":0.002207707},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3576,"average_load":3576,"avg_session_uptime":824,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":12702191,"buffer_anon_items":3391,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12646926,"mds_co_items":82875,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":66,"op_send_bytes":10753008,"op_resend":0,"op_reply":66,"op_latency":{"avgcount":66,"sum":0.163996476,"avgtime":0.002484795},"op_inflight":0,"oplen_avg":{"avgcount":66,"sum":123},"op":66,"op_r":36,"op_w":30,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":12,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":17,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":21356,"max":104857600,"get_started":0,"get":6020,"get_sum":1672030,"get_or_fail_fail":0,"get_or_fail_success":6020,"take":0,"take_sum":0,"put":5938,"put_sum":1650674,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":10754125,"put":66,"put_sum":10754125,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":66,"take_sum":66,"put":66,"put_sum":66,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":1679519,"max":3758096384,"get_started":0,"get":5813,"get_sum":12065202,"get_or_fail_fail":0,"get_or_fail_success":5813,"take":0,"take_sum":0,"put":12,"put_sum":10385683,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:34.858 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 14845, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 22, "msgr_running_fast_dispatch_time": 0.001957818, "msgr_running_recv_time": 0.00410178, "msgr_running_send_time": 0.008524326, "msgr_running_total_time": 0.015647504, "msgr_send_bytes": 4222861, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 22, "msgr_send_messages_queue_lat": { "avgcount": 21, "avgtime": 0.000368004, "sum": 0.007728097 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 18728, "avgtime": 1.55e-07, "sum": 0.002913902 }, "msgr_recv_bytes": 2089988, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 5971, "msgr_running_fast_dispatch_time": 0.005271391, "msgr_running_recv_time": 0.310650628, "msgr_running_send_time": 0.300604472, "msgr_running_total_time": 0.559845445, "msgr_send_bytes": 13492214, "msgr_send_encrypted_bytes": 92400, "msgr_send_messages": 13263, "msgr_send_messages_queue_lat": { "avgcount": 13262, "avgtime": 1.1164e-05, "sum": 0.148067557 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 111714, "msgr_recv_encrypted_bytes": 107936, "msgr_recv_messages": 31, "msgr_running_fast_dispatch_time": 0.00041829, "msgr_running_recv_time": 0.003576608, "msgr_running_send_time": 0.000759255, "msgr_running_total_time": 0.004023855, "msgr_send_bytes": 10208, "msgr_send_encrypted_bytes": 7296, "msgr_send_messages": 26, "msgr_send_messages_queue_lat": { "avgcount": 25, "avgtime": 0.000151999, "sum": 0.003799986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 77, "avgtime": 0.022358901, "sum": 1.721635385 }, "queue_len": 893 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.7942e-05, "sum": 0.000151768 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4121, "ceph_cap_op_revoke": 466, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2057, "handle_client_caps_dirty": 1865, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3776, "inodes_bottom": 3776, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3776, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 4331, "q": 0, "reply": 3775, "reply_latency": { "avgcount": 3775, "avgtime": 0.001182319, "sum": 4.463255916 }, "request": 3777, "root_rbytes": 5236736, "root_rfiles": 3057, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3783, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3782, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 565, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 565, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5815, "evadd": 5812, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 12, "avgtime": 0.005361314, "sum": 0.064335778 }, "rdpos": 4195244, "replayed": 3, "seg": 9, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 16258953 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3776, "dn+": 3776, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 86380 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3780, "dispatch_server_request": 0, "handle_client_request": 3777, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 565, "avgtime": 0.002207707, "sum": 1.247354702 } }, "mds_sessions": { "average_load": 3576, "avg_session_uptime": 824, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3576 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 12702191, "buffer_anon_items": 3391, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 12646926, "mds_co_items": 82875, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 17, "op": 66, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 66, "avgtime": 0.002484795, "sum": 0.163996476 }, "op_pg": 0, "op_r": 36, "op_reply": 66, "op_resend": 0, "op_rmw": 0, "op_send": 66, "op_send_bytes": 10753008, "op_w": 30, "oplen_avg": { "avgcount": 66, "sum": 123 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 12, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6020, "get_or_fail_fail": 0, "get_or_fail_success": 6020, "get_started": 0, "get_sum": 1672030, "max": 104857600, "put": 5938, "put_sum": 1650674, "take": 0, "take_sum": 0, "val": 21356, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 66, "put_sum": 10754125, "take": 66, "take_sum": 10754125, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 66, "put_sum": 66, "take": 66, "take_sum": 66, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 5813, "get_or_fail_fail": 0, "get_or_fail_success": 5813, "get_started": 0, "get_sum": 12065202, "max": 3758096384, "put": 12, "put_sum": 10385683, "take": 0, "take_sum": 0, "val": 1679519, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:34.859 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/565, 0/565 2024-03-06T07:26:35.860 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:35.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.993+0000 7f1c58f16640 1 Processor -- start 2024-03-06T07:26:35.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.994+0000 7f1c58f16640 1 -- start start 2024-03-06T07:26:35.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c58f16640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f4c0 0x7f1c5406af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c58f16640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c5406f920 0x7f1c5406b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c58f16640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5415e4f0 con 0x7f1c5415e110 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c58f16640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c540b0520 con 0x7f1c5406f920 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c58f16640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540d0b40 con 0x7f1c5406f4c0 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c5406f920 0x7f1c5406b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c5406f920 0x7f1c5406b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57820/0 (socket says 172.21.15.72:57820) 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 -- 172.21.15.72:0/718837577 learned_addr learned my addr 172.21.15.72:0/718837577 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c4bfff640 1 --2- 172.21.15.72:0/718837577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f4c0 0x7f1c5406af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 -- 172.21.15.72:0/718837577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f4c0 msgr2=0x7f1c5406af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 --2- 172.21.15.72:0/718837577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f4c0 0x7f1c5406af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 -- 172.21.15.72:0/718837577 >> v1:172.21.15.146:6789/0 conn(0x7f1c5415e110 legacy=0x7f1c5406da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:35.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.996+0000 7f1c52d9d640 1 -- 172.21.15.72:0/718837577 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c540afc80 con 0x7f1c5406f920 2024-03-06T07:26:35.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c52d9d640 1 --2- 172.21.15.72:0/718837577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f920 0x7f1c5406b4c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009e50 tx=0x7f1c4c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899345bda14479f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:35.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c51d9b640 1 -- 172.21.15.72:0/718837577 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c02c9e0 con 0x7f1c5406f920 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c51d9b640 1 -- 172.21.15.72:0/718837577 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4c02cb80 con 0x7f1c5406f920 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c51d9b640 1 -- 172.21.15.72:0/718837577 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4c0343f0 con 0x7f1c5406f920 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 -- 172.21.15.72:0/718837577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f920 msgr2=0x7f1c5406b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 --2- 172.21.15.72:0/718837577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f920 0x7f1c5406b4c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f1c4c009e50 tx=0x7f1c4c02be40 comp rx=0 tx=0).stop 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 -- 172.21.15.72:0/718837577 shutdown_connections 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 --2- 172.21.15.72:0/718837577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f920 0x7f1c5406b4c0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 --2- 172.21.15.72:0/718837577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f4c0 0x7f1c5406af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 -- 172.21.15.72:0/718837577 >> 172.21.15.72:0/718837577 conn(0x7f1c5406c090 msgr2=0x7f1c5406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 -- 172.21.15.72:0/718837577 shutdown_connections 2024-03-06T07:26:36.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.997+0000 7f1c58f16640 1 -- 172.21.15.72:0/718837577 wait complete. 2024-03-06T07:26:36.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.999+0000 7f1c58f16640 1 Processor -- start 2024-03-06T07:26:36.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:35.999+0000 7f1c58f16640 1 -- start start 2024-03-06T07:26:36.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c58f16640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 0x7f1c540cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c58f16640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f920 0x7f1c540ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c58f16640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c5415e110 0x7f1c54152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c58f16640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540b3df0 con 0x7f1c5406f4c0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c58f16640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540b0240 con 0x7f1c5415e110 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c58f16640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540b3f70 con 0x7f1c5406f920 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 0x7f1c540cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 0x7f1c540cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57836/0 (socket says 172.21.15.72:57836) 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 -- 172.21.15.72:0/4253763525 learned_addr learned my addr 172.21.15.72:0/4253763525 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c52d9d640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c5415e110 0x7f1c54152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c4bfff640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f920 0x7f1c540ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 -- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f920 msgr2=0x7f1c540ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f920 0x7f1c540ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 -- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c5415e110 msgr2=0x7f1c54152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c5415e110 0x7f1c54152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c5259c640 1 -- 172.21.15.72:0/4253763525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c540cdb60 con 0x7f1c5406f4c0 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.000+0000 7f1c4bfff640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f920 0x7f1c540ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:36.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.001+0000 7f1c5259c640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 0x7f1c540cd260 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c3c0 tx=0x7f1c4400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.001+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c440052d0 con 0x7f1c5406f4c0 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.001+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c44005490 con 0x7f1c5406f4c0 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.001+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c540cde40 con 0x7f1c5406f4c0 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.001+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c44007260 con 0x7f1c5406f4c0 2024-03-06T07:26:36.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.001+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c540ce400 con 0x7f1c5406f4c0 2024-03-06T07:26:36.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.002+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c54068050 con 0x7f1c5406f4c0 2024-03-06T07:26:36.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.004+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c44007400 con 0x7f1c5406f4c0 2024-03-06T07:26:36.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.004+0000 7f1c4affd640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:36.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.004+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1c44089710 con 0x7f1c5406f4c0 2024-03-06T07:26:36.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.004+0000 7f1c4bfff640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:36.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.004+0000 7f1c4bfff640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009ac0 tx=0x7f1c3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:36.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.009+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c5415e4f0 con 0x7f1c5406f4c0 2024-03-06T07:26:36.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.344+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c540724e0 con 0x7f1c5406f4c0 2024-03-06T07:26:36.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.345+0000 7f1c4affd640 1 -- 172.21.15.72:0/4253763525 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f1c44058520 con 0x7f1c5406f4c0 2024-03-06T07:26:36.346 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:36.346 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:36.346 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 msgr2=0x7f1c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c009ac0 tx=0x7f1c3c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 msgr2=0x7f1c540cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 0x7f1c540cd260 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f1c4400c3c0 tx=0x7f1c4400c880 comp rx=0 tx=0).stop 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 shutdown_connections 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c5415e110 0x7f1c54152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c5406f920 0x7f1c540ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 --2- 172.21.15.72:0/4253763525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c5406f4c0 0x7f1c540cd260 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:36.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 >> 172.21.15.72:0/4253763525 conn(0x7f1c5406c090 msgr2=0x7f1c5406da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:36.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 shutdown_connections 2024-03-06T07:26:36.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:36.349+0000 7f1c58f16640 1 -- 172.21.15.72:0/4253763525 wait complete. 2024-03-06T07:26:36.368 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:36.513 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":148,"msgr_send_messages":148,"msgr_recv_bytes":46833,"msgr_send_bytes":4263921,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.029870976,"msgr_running_send_time":0.016358695,"msgr_running_recv_time":0.006459177,"msgr_running_fast_dispatch_time":0.005409353,"msgr_send_messages_queue_lat":{"avgcount":147,"sum":0.021304135,"avgtime":0.000144926},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7286,"msgr_send_messages":16596,"msgr_recv_bytes":2570495,"msgr_send_bytes":18262050,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.704485673,"msgr_running_send_time":0.387396819,"msgr_running_recv_time":0.371545181,"msgr_running_fast_dispatch_time":0.009057131,"msgr_send_messages_queue_lat":{"avgcount":16595,"sum":0.212989399,"avgtime":0.000012834},"msgr_handle_ack_lat":{"avgcount":22589,"sum":0.003643459,"avgtime":0.000000161},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94192},"AsyncMessenger::Worker-2":{"msgr_recv_messages":126,"msgr_send_messages":121,"msgr_recv_bytes":163969,"msgr_send_bytes":69763,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.018920380,"msgr_running_send_time":0.009463554,"msgr_running_recv_time":0.005955272,"msgr_running_fast_dispatch_time":0.003558736,"msgr_send_messages_queue_lat":{"avgcount":120,"sum":0.014815982,"avgtime":0.000123466},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108208,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":85,"sum":4.139643810,"avgtime":0.048701691}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":445,"sum":0.409028388,"avgtime":0.000919164}},"mds":{"request":4355,"reply":4352,"reply_latency":{"avgcount":4352,"sum":6.063338110,"avgtime":0.001393230},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4254,"inodes_top":0,"inodes_bottom":4254,"inodes_pin_tail":0,"inodes_pinned":4254,"inodes_expired":0,"inodes_with_caps":2923,"caps":2923,"subtrees":2,"traverse":4361,"traverse_hit":4360,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2622,"root_rbytes":11584512,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":727,"ceph_cap_op_grant":5043,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2794,"handle_client_caps_dirty":2483,"handle_client_cap_release":1,"process_request_cap_release":5487},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":864,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1143,"strays_enqueued":279,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":7410,"evex":0,"evtrm":0,"ev":7413,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":19706740,"rdpos":4195244,"jlat":{"avgcount":15,"sum":0.702695039,"avgtime":0.046846335},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4353,"dn+":4353,"dn-":0,"cap":2923,"cap+":3202,"cap-":279,"rss":87148,"heap":248164},"mds_server":{"dispatch_client_request":4358,"dispatch_server_request":0,"handle_client_request":4355,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1142,"sum":2.847436896,"avgtime":0.002493377},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4067,"average_load":4067,"avg_session_uptime":824,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":12702191,"buffer_anon_items":3391,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12646926,"mds_co_items":82875,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":365,"op_send_bytes":14392633,"op_resend":0,"op_reply":365,"op_latency":{"avgcount":365,"sum":0.776983324,"avgtime":0.002128721},"op_inflight":0,"oplen_avg":{"avgcount":365,"sum":426},"op":365,"op_r":39,"op_w":326,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":18,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":293,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":21,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":279,"pq_executed":279,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":556,"max":104857600,"get_started":0,"get":7556,"get_sum":2098488,"get_or_fail_fail":0,"get_or_fail_success":7556,"take":0,"take_sum":0,"put":7554,"put_sum":2097932,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":365,"take_sum":14421929,"put":365,"put_sum":14421929,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":365,"take_sum":365,"put":365,"put_sum":365,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":279,"get_sum":28179,"get_or_fail_fail":0,"get_or_fail_success":279,"take":0,"take_sum":0,"put":3,"put_sum":28179,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":1533119,"max":3758096384,"get_started":0,"get":7410,"get_sum":15511055,"get_or_fail_fail":0,"get_or_fail_success":7410,"take":0,"take_sum":0,"put":15,"put_sum":13977936,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:36.515 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 46833, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 148, "msgr_running_fast_dispatch_time": 0.005409353, "msgr_running_recv_time": 0.006459177, "msgr_running_send_time": 0.016358695, "msgr_running_total_time": 0.029870976, "msgr_send_bytes": 4263921, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 148, "msgr_send_messages_queue_lat": { "avgcount": 147, "avgtime": 0.000144926, "sum": 0.021304135 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 22589, "avgtime": 1.61e-07, "sum": 0.003643459 }, "msgr_recv_bytes": 2570495, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7286, "msgr_running_fast_dispatch_time": 0.009057131, "msgr_running_recv_time": 0.371545181, "msgr_running_send_time": 0.387396819, "msgr_running_total_time": 0.704485673, "msgr_send_bytes": 18262050, "msgr_send_encrypted_bytes": 94192, "msgr_send_messages": 16596, "msgr_send_messages_queue_lat": { "avgcount": 16595, "avgtime": 1.2834e-05, "sum": 0.212989399 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 163969, "msgr_recv_encrypted_bytes": 108208, "msgr_recv_messages": 126, "msgr_running_fast_dispatch_time": 0.003558736, "msgr_running_recv_time": 0.005955272, "msgr_running_send_time": 0.009463554, "msgr_running_total_time": 0.01892038, "msgr_send_bytes": 69763, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 121, "msgr_send_messages_queue_lat": { "avgcount": 120, "avgtime": 0.000123466, "sum": 0.014815982 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 85, "avgtime": 0.048701691, "sum": 4.13964381 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 445, "avgtime": 0.000919164, "sum": 0.409028388 }, "queue_len": 0 }, "mds": { "caps": 2923, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5043, "ceph_cap_op_revoke": 727, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 1, "handle_client_caps": 2794, "handle_client_caps_dirty": 2483, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4254, "inodes_bottom": 4254, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4254, "inodes_top": 0, "inodes_with_caps": 2923, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 5487, "q": 0, "reply": 4352, "reply_latency": { "avgcount": 4352, "avgtime": 0.00139323, "sum": 6.06333811 }, "request": 4355, "root_rbytes": 11584512, "root_rfiles": 2622, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4361, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4360, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 864, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1143, "strays_enqueued": 279, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7413, "evadd": 7410, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 15, "avgtime": 0.046846335, "sum": 0.702695039 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 19706740 }, "mds_mem": { "cap": 2923, "cap+": 3202, "cap-": 279, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4353, "dn+": 4353, "dn-": 0, "heap": 248164, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 87148 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4358, "dispatch_server_request": 0, "handle_client_request": 4355, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1142, "avgtime": 0.002493377, "sum": 2.847436896 } }, "mds_sessions": { "average_load": 4067, "avg_session_uptime": 824, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4067 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 12702191, "buffer_anon_items": 3391, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 12646926, "mds_co_items": 82875, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 21, "op": 365, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 365, "avgtime": 0.002128721, "sum": 0.776983324 }, "op_pg": 0, "op_r": 39, "op_reply": 365, "op_resend": 0, "op_rmw": 0, "op_send": 365, "op_send_bytes": 14392633, "op_w": 326, "oplen_avg": { "avgcount": 365, "sum": 426 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 293, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 18, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 279, "pq_executed_ops": 279, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7556, "get_or_fail_fail": 0, "get_or_fail_success": 7556, "get_started": 0, "get_sum": 2098488, "max": 104857600, "put": 7554, "put_sum": 2097932, "take": 0, "take_sum": 0, "val": 556, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 365, "put_sum": 14421929, "take": 365, "take_sum": 14421929, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 365, "put_sum": 365, "take": 365, "take_sum": 365, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 279, "get_or_fail_fail": 0, "get_or_fail_success": 279, "get_started": 0, "get_sum": 28179, "max": 3758096384, "put": 3, "put_sum": 28179, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 7410, "get_or_fail_fail": 0, "get_or_fail_success": 7410, "get_started": 0, "get_sum": 15511055, "max": 3758096384, "put": 15, "put_sum": 13977936, "take": 0, "take_sum": 0, "val": 1533119, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:36.515 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/864, 279/1143 2024-03-06T07:26:37.516 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:37.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.646+0000 7ff028481640 1 Processor -- start 2024-03-06T07:26:37.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.646+0000 7ff028481640 1 -- start start 2024-03-06T07:26:37.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff028481640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff020166d60 0x7ff020169150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:37.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff028481640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0200cc580 con 0x7ff02006c750 2024-03-06T07:26:37.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff028481640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0200cc860 con 0x7ff0200b12a0 2024-03-06T07:26:37.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff028481640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0200cc330 con 0x7ff020166d60 2024-03-06T07:26:37.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff026a1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff020166d60 0x7ff020169150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff026a1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff020166d60 0x7ff020169150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57852/0 (socket says 172.21.15.72:57852) 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff026a1e640 1 -- 172.21.15.72:0/2050546589 learned_addr learned my addr 172.21.15.72:0/2050546589 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff02621d640 1 --1- 172.21.15.72:0/2050546589 >> v1:172.21.15.72:6789/0 conn(0x7ff02006c750 0x7ff02006cb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36986/0 (socket says 172.21.15.72:36986) 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff026a1e640 1 -- 172.21.15.72:0/2050546589 >> v1:172.21.15.146:6789/0 conn(0x7ff0200b12a0 legacy=0x7ff0200d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff026a1e640 1 -- 172.21.15.72:0/2050546589 >> v1:172.21.15.72:6789/0 conn(0x7ff02006c750 legacy=0x7ff02006cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff026a1e640 1 -- 172.21.15.72:0/2050546589 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200ccb40 con 0x7ff020166d60 2024-03-06T07:26:37.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.647+0000 7ff02521b640 1 -- 172.21.15.72:0/2050546589 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587506249 0 0) 0x7ff0200cc580 con 0x7ff02006c750 2024-03-06T07:26:37.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.648+0000 7ff026a1e640 1 --2- 172.21.15.72:0/2050546589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff020166d60 0x7ff020169150 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff018009e50 tx=0x7ff01802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a841f014e0ecdeda server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:37.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.648+0000 7ff02521b640 1 -- 172.21.15.72:0/2050546589 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01802cb10 con 0x7ff020166d60 2024-03-06T07:26:37.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.648+0000 7ff02521b640 1 -- 172.21.15.72:0/2050546589 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01802ccb0 con 0x7ff020166d60 2024-03-06T07:26:37.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.648+0000 7ff02521b640 1 -- 172.21.15.72:0/2050546589 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01802ce60 con 0x7ff020166d60 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 -- 172.21.15.72:0/2050546589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff020166d60 msgr2=0x7ff020169150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 --2- 172.21.15.72:0/2050546589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff020166d60 0x7ff020169150 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff018009e50 tx=0x7ff01802bf70 comp rx=0 tx=0).stop 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 -- 172.21.15.72:0/2050546589 shutdown_connections 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 --2- 172.21.15.72:0/2050546589 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff020166d60 0x7ff020169150 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 -- 172.21.15.72:0/2050546589 >> 172.21.15.72:0/2050546589 conn(0x7ff0200b3850 msgr2=0x7ff0200b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 -- 172.21.15.72:0/2050546589 shutdown_connections 2024-03-06T07:26:37.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.649+0000 7ff028481640 1 -- 172.21.15.72:0/2050546589 wait complete. 2024-03-06T07:26:37.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.651+0000 7ff028481640 1 Processor -- start 2024-03-06T07:26:37.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.651+0000 7ff028481640 1 -- start start 2024-03-06T07:26:37.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff028481640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff02006c750 0x7ff020158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:37.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff028481640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 0x7ff0200b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff028481640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff020166d60 0x7ff020070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff028481640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200b44e0 con 0x7ff0200b12a0 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff028481640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200d1140 con 0x7ff02006c750 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff028481640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200b4660 con 0x7ff020166d60 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 0x7ff0200b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 0x7ff0200b5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57854/0 (socket says 172.21.15.72:57854) 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 -- 172.21.15.72:0/3175692089 learned_addr learned my addr 172.21.15.72:0/3175692089 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff02621d640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff02006c750 0x7ff020158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff026a1e640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff020166d60 0x7ff020070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 -- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff020166d60 msgr2=0x7ff020070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff020166d60 0x7ff020070250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 -- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff02006c750 msgr2=0x7ff020158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff02006c750 0x7ff020158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff025a1c640 1 -- 172.21.15.72:0/3175692089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff020158690 con 0x7ff0200b12a0 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.652+0000 7ff026a1e640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff020166d60 0x7ff020070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff02621d640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff02006c750 0x7ff020158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff025a1c640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 0x7ff0200b5a10 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff01000c400 tx=0x7ff01000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010017070 con 0x7ff0200b12a0 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0100052d0 con 0x7ff0200b12a0 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0200b6dc0 con 0x7ff0200b12a0 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010003830 con 0x7ff0200b12a0 2024-03-06T07:26:37.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.653+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0200b6800 con 0x7ff0200b12a0 2024-03-06T07:26:37.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.654+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0200cfb60 con 0x7ff0200b12a0 2024-03-06T07:26:37.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.655+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff01001a070 con 0x7ff0200b12a0 2024-03-06T07:26:37.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.656+0000 7ff016ffd640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff00406b4d0 0x7ff00406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:37.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.656+0000 7ff02621d640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff00406b4d0 0x7ff00406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:37.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.656+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff010089680 con 0x7ff0200b12a0 2024-03-06T07:26:37.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.656+0000 7ff02621d640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff00406b4d0 0x7ff00406d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff01c007fd0 tx=0x7ff01c007de0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:37.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.662+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff0200cfb60 con 0x7ff0200b12a0 2024-03-06T07:26:37.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.976+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff020069ee0 con 0x7ff0200b12a0 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.977+0000 7ff016ffd640 1 -- 172.21.15.72:0/3175692089 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7ff020158690 con 0x7ff0200b12a0 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:37.978 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff00406b4d0 msgr2=0x7ff00406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff00406b4d0 0x7ff00406d990 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff01c007fd0 tx=0x7ff01c007de0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 msgr2=0x7ff0200b5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 0x7ff0200b5a10 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff01000c400 tx=0x7ff01000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 shutdown_connections 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff00406b4d0 0x7ff00406d990 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff020166d60 0x7ff020070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff0200b12a0 0x7ff0200b5a10 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 --2- 172.21.15.72:0/3175692089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff02006c750 0x7ff020158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 >> 172.21.15.72:0/3175692089 conn(0x7ff0200b3850 msgr2=0x7ff0200d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 shutdown_connections 2024-03-06T07:26:37.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:37.980+0000 7ff028481640 1 -- 172.21.15.72:0/3175692089 wait complete. 2024-03-06T07:26:37.995 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:38.280 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":292,"msgr_send_messages":292,"msgr_recv_bytes":83401,"msgr_send_bytes":4310857,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.050202139,"msgr_running_send_time":0.029516830,"msgr_running_recv_time":0.009520360,"msgr_running_fast_dispatch_time":0.008489318,"msgr_send_messages_queue_lat":{"avgcount":291,"sum":0.029448731,"avgtime":0.000101198},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8646,"msgr_send_messages":19022,"msgr_recv_bytes":3078562,"msgr_send_bytes":21965567,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.821946566,"msgr_running_send_time":0.456443961,"msgr_running_recv_time":0.427239264,"msgr_running_fast_dispatch_time":0.011814107,"msgr_send_messages_queue_lat":{"avgcount":19021,"sum":0.271254334,"avgtime":0.000014260},"msgr_handle_ack_lat":{"avgcount":25735,"sum":0.004173042,"avgtime":0.000000162},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94192},"AsyncMessenger::Worker-2":{"msgr_recv_messages":214,"msgr_send_messages":209,"msgr_recv_bytes":245504,"msgr_send_bytes":157724,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.028476433,"msgr_running_send_time":0.014038820,"msgr_running_recv_time":0.007807007,"msgr_running_fast_dispatch_time":0.005897600,"msgr_send_messages_queue_lat":{"avgcount":208,"sum":0.020950256,"avgtime":0.000100722},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108208,"msgr_send_encrypted_bytes":7920},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":90,"sum":5.276923258,"avgtime":0.058632480}},"finisher-PurgeQueue":{"queue_len":261,"complete_latency":{"avgcount":905,"sum":0.526897891,"avgtime":0.000582207}},"mds":{"request":5037,"reply":5034,"reply_latency":{"avgcount":5034,"sum":7.207488360,"avgtime":0.001431761},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4730,"inodes_top":0,"inodes_bottom":4730,"inodes_pin_tail":0,"inodes_pinned":4730,"inodes_expired":0,"inodes_with_caps":2336,"caps":2336,"subtrees":2,"traverse":5043,"traverse_hit":5042,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2072,"root_rbytes":11799552,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":836,"ceph_cap_op_grant":5625,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3397,"handle_client_caps_dirty":3000,"handle_client_cap_release":3,"process_request_cap_release":6851},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1096,"num_strays_delayed":0,"num_strays_enqueuing":138,"strays_created":1824,"strays_enqueued":728,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":9146,"evex":0,"evtrm":0,"ev":9149,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":23318939,"rdpos":4195244,"jlat":{"avgcount":16,"sum":0.718624387,"avgtime":0.044914024},"replayed":3},"mds_mem":{"ino":2909,"ino+":3214,"ino-":305,"dir":13,"dir+":13,"dir-":0,"dn":4730,"dn+":5035,"dn-":305,"cap":2336,"cap+":3202,"cap-":866,"rss":100844,"heap":248164},"mds_server":{"dispatch_client_request":5040,"dispatch_server_request":0,"handle_client_request":5037,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1824,"sum":3.991587146,"avgtime":0.002188370},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4670,"average_load":4670,"avg_session_uptime":824,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":12702191,"buffer_anon_items":3391,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":12646926,"mds_co_items":82875,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":669,"op_send_bytes":17260590,"op_resend":0,"op_reply":669,"op_latency":{"avgcount":669,"sum":1.292972269,"avgtime":0.001932693},"op_inflight":0,"oplen_avg":{"avgcount":669,"sum":733},"op":669,"op_r":45,"op_w":624,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":17,"osdop_write":25,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":581,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":24,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":1,"pq_executing_ops_high_water":6,"pq_executing":1,"pq_executing_high_water":6,"pq_executed_ops":565,"pq_executed":565,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":298,"max":104857600,"get_started":0,"get":9148,"get_sum":2581378,"get_or_fail_fail":0,"get_or_fail_success":9148,"take":0,"take_sum":0,"put":9147,"put_sum":2581080,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":669,"take_sum":17349173,"put":669,"put_sum":17349173,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":669,"take_sum":669,"put":669,"put_sum":669,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":866,"get_sum":87466,"get_or_fail_fail":0,"get_or_fail_success":866,"take":0,"take_sum":0,"put":9,"put_sum":87466,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":2347419,"max":3758096384,"get_started":0,"get":9146,"get_sum":19123254,"get_or_fail_fail":0,"get_or_fail_success":9146,"take":0,"take_sum":0,"put":16,"put_sum":16775835,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:38.281 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 83401, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 292, "msgr_running_fast_dispatch_time": 0.008489318, "msgr_running_recv_time": 0.00952036, "msgr_running_send_time": 0.02951683, "msgr_running_total_time": 0.050202139, "msgr_send_bytes": 4310857, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 292, "msgr_send_messages_queue_lat": { "avgcount": 291, "avgtime": 0.000101198, "sum": 0.029448731 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 25735, "avgtime": 1.62e-07, "sum": 0.004173042 }, "msgr_recv_bytes": 3078562, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 8646, "msgr_running_fast_dispatch_time": 0.011814107, "msgr_running_recv_time": 0.427239264, "msgr_running_send_time": 0.456443961, "msgr_running_total_time": 0.821946566, "msgr_send_bytes": 21965567, "msgr_send_encrypted_bytes": 94192, "msgr_send_messages": 19022, "msgr_send_messages_queue_lat": { "avgcount": 19021, "avgtime": 1.426e-05, "sum": 0.271254334 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 245504, "msgr_recv_encrypted_bytes": 108208, "msgr_recv_messages": 214, "msgr_running_fast_dispatch_time": 0.0058976, "msgr_running_recv_time": 0.007807007, "msgr_running_send_time": 0.01403882, "msgr_running_total_time": 0.028476433, "msgr_send_bytes": 157724, "msgr_send_encrypted_bytes": 7920, "msgr_send_messages": 209, "msgr_send_messages_queue_lat": { "avgcount": 208, "avgtime": 0.000100722, "sum": 0.020950256 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 90, "avgtime": 0.05863248, "sum": 5.276923258 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 905, "avgtime": 0.000582207, "sum": 0.526897891 }, "queue_len": 261 }, "mds": { "caps": 2336, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5625, "ceph_cap_op_revoke": 836, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 3, "handle_client_caps": 3397, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4730, "inodes_bottom": 4730, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4730, "inodes_top": 0, "inodes_with_caps": 2336, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 6851, "q": 0, "reply": 5034, "reply_latency": { "avgcount": 5034, "avgtime": 0.001431761, "sum": 7.20748836 }, "request": 5037, "root_rbytes": 11799552, "root_rfiles": 2072, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5043, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5042, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1096, "num_strays_delayed": 0, "num_strays_enqueuing": 138, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1824, "strays_enqueued": 728, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 9149, "evadd": 9146, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 16, "avgtime": 0.044914024, "sum": 0.718624387 }, "rdpos": 4195244, "replayed": 3, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 23318939 }, "mds_mem": { "cap": 2336, "cap+": 3202, "cap-": 866, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4730, "dn+": 5035, "dn-": 305, "heap": 248164, "ino": 2909, "ino+": 3214, "ino-": 305, "rss": 100844 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5040, "dispatch_server_request": 0, "handle_client_request": 5037, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1824, "avgtime": 0.00218837, "sum": 3.991587146 } }, "mds_sessions": { "average_load": 4670, "avg_session_uptime": 824, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4670 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 12702191, "buffer_anon_items": 3391, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 12646926, "mds_co_items": 82875, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 24, "op": 669, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 669, "avgtime": 0.001932693, "sum": 1.292972269 }, "op_pg": 0, "op_r": 45, "op_reply": 669, "op_resend": 0, "op_rmw": 0, "op_send": 669, "op_send_bytes": 17260590, "op_w": 624, "oplen_avg": { "avgcount": 669, "sum": 733 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 581, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 17, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 25, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 565, "pq_executed_ops": 565, "pq_executing": 1, "pq_executing_high_water": 6, "pq_executing_ops": 1, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9148, "get_or_fail_fail": 0, "get_or_fail_success": 9148, "get_started": 0, "get_sum": 2581378, "max": 104857600, "put": 9147, "put_sum": 2581080, "take": 0, "take_sum": 0, "val": 298, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 669, "put_sum": 17349173, "take": 669, "take_sum": 17349173, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 669, "put_sum": 669, "take": 669, "take_sum": 669, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 866, "get_or_fail_fail": 0, "get_or_fail_success": 866, "get_started": 0, "get_sum": 87466, "max": 3758096384, "put": 9, "put_sum": 87466, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 9146, "get_or_fail_fail": 0, "get_or_fail_success": 9146, "get_started": 0, "get_sum": 19123254, "max": 3758096384, "put": 16, "put_sum": 16775835, "take": 0, "take_sum": 0, "val": 2347419, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:38.281 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 1/1096, 565/1824 2024-03-06T07:26:39.282 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:39.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.414+0000 7fc9fe193640 1 Processor -- start 2024-03-06T07:26:39.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.415+0000 7fc9fe193640 1 -- start start 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.415+0000 7fc9fe193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80b4b40 0x7fc9f806dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.415+0000 7fc9fe193640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9f80af930 0x7fc9f8164930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.415+0000 7fc9fe193640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f806c370 con 0x7fc9f815b380 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.415+0000 7fc9fe193640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9f806ee40 con 0x7fc9f80af930 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.415+0000 7fc9fe193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f806e5a0 con 0x7fc9f80b4b40 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9f80af930 0x7fc9f8164930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc9f815b380 0x7fc9f815b760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37000/0 (socket says 172.21.15.72:37000) 2024-03-06T07:26:39.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9f80af930 0x7fc9f8164930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57868/0 (socket says 172.21.15.72:57868) 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f77fe640 1 -- 172.21.15.72:0/2447920858 learned_addr learned my addr 172.21.15.72:0/2447920858 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f6ffd640 1 --2- 172.21.15.72:0/2447920858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80b4b40 0x7fc9f806dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f7fff640 1 -- 172.21.15.72:0/2447920858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80b4b40 msgr2=0x7fc9f806dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f7fff640 1 --2- 172.21.15.72:0/2447920858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80b4b40 0x7fc9f806dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f7fff640 1 -- 172.21.15.72:0/2447920858 >> v1:172.21.15.72:6789/0 conn(0x7fc9f815b380 legacy=0x7fc9f815b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.416+0000 7fc9f7fff640 1 -- 172.21.15.72:0/2447920858 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f806e880 con 0x7fc9f80af930 2024-03-06T07:26:39.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.417+0000 7fc9f7fff640 1 --2- 172.21.15.72:0/2447920858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f80af930 0x7fc9f8164930 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009d90 tx=0x7fc9f0030220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37dce713b01ac462 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:39.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.417+0000 7fc9f67fc640 1 -- 172.21.15.72:0/2447920858 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f003d070 con 0x7fc9f80af930 2024-03-06T07:26:39.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.417+0000 7fc9f67fc640 1 -- 172.21.15.72:0/2447920858 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f0038440 con 0x7fc9f80af930 2024-03-06T07:26:39.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.417+0000 7fc9f67fc640 1 -- 172.21.15.72:0/2447920858 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0038810 con 0x7fc9f80af930 2024-03-06T07:26:39.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 -- 172.21.15.72:0/2447920858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f80af930 msgr2=0x7fc9f8164930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 --2- 172.21.15.72:0/2447920858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f80af930 0x7fc9f8164930 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009d90 tx=0x7fc9f0030220 comp rx=0 tx=0).stop 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 -- 172.21.15.72:0/2447920858 shutdown_connections 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 --2- 172.21.15.72:0/2447920858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f80af930 0x7fc9f8164930 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 --2- 172.21.15.72:0/2447920858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80b4b40 0x7fc9f806dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 -- 172.21.15.72:0/2447920858 >> 172.21.15.72:0/2447920858 conn(0x7fc9f80d0470 msgr2=0x7fc9f80d0880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 -- 172.21.15.72:0/2447920858 shutdown_connections 2024-03-06T07:26:39.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.418+0000 7fc9fe193640 1 -- 172.21.15.72:0/2447920858 wait complete. 2024-03-06T07:26:39.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.419+0000 7fc9fe193640 1 Processor -- start 2024-03-06T07:26:39.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.420+0000 7fc9fe193640 1 -- start start 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.420+0000 7fc9fe193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80af930 0x7fc9f80cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.420+0000 7fc9fe193640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80b4b40 0x7fc9f8147f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9fe193640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 0x7fc9f80a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9fe193640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f815be70 con 0x7fc9f815b380 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9fe193640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f80b3610 con 0x7fc9f80b4b40 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9fe193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f815bff0 con 0x7fc9f80af930 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 0x7fc9f80a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 0x7fc9f80a9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57876/0 (socket says 172.21.15.72:57876) 2024-03-06T07:26:39.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 -- 172.21.15.72:0/227250024 learned_addr learned my addr 172.21.15.72:0/227250024 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f77fe640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80af930 0x7fc9f80cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f6ffd640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80b4b40 0x7fc9f8147f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 -- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80af930 msgr2=0x7fc9f80cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80af930 0x7fc9f80cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 -- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80b4b40 msgr2=0x7fc9f8147f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80b4b40 0x7fc9f8147f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 -- 172.21.15.72:0/227250024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f80aa3a0 con 0x7fc9f815b380 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9f7fff640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 0x7fc9f80a9e60 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009d90 tx=0x7fc9f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0048020 con 0x7fc9f815b380 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f00368f0 con 0x7fc9f815b380 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f80aa650 con 0x7fc9f815b380 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.421+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f003d040 con 0x7fc9f815b380 2024-03-06T07:26:39.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.422+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f80aac10 con 0x7fc9f815b380 2024-03-06T07:26:39.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.423+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9f8068050 con 0x7fc9f815b380 2024-03-06T07:26:39.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.423+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9f004b020 con 0x7fc9f815b380 2024-03-06T07:26:39.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.424+0000 7fc9dffff640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9e406b4d0 0x7fc9e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:39.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.424+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc9f00b1150 con 0x7fc9f815b380 2024-03-06T07:26:39.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.424+0000 7fc9f77fe640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9e406b4d0 0x7fc9e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:39.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.425+0000 7fc9f77fe640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9e406b4d0 0x7fc9e406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007fd0 tx=0x7fc9e8007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:39.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.429+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9f0052030 con 0x7fc9f815b380 2024-03-06T07:26:39.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.732+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9f806d650 con 0x7fc9f815b380 2024-03-06T07:26:39.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.733+0000 7fc9dffff640 1 -- 172.21.15.72:0/227250024 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fc9f0038e20 con 0x7fc9f815b380 2024-03-06T07:26:39.735 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:39.735 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:39.736 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9e406b4d0 msgr2=0x7fc9e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9e406b4d0 0x7fc9e406d990 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007fd0 tx=0x7fc9e8007e80 comp rx=0 tx=0).stop 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 msgr2=0x7fc9f80a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 0x7fc9f80a9e60 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009d90 tx=0x7fc9f0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 shutdown_connections 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9e406b4d0 0x7fc9e406d990 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9f815b380 0x7fc9f80a9e60 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9f80b4b40 0x7fc9f8147f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 --2- 172.21.15.72:0/227250024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9f80af930 0x7fc9f80cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:39.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 >> 172.21.15.72:0/227250024 conn(0x7fc9f80d0470 msgr2=0x7fc9f815de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:39.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 shutdown_connections 2024-03-06T07:26:39.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:39.736+0000 7fc9fe193640 1 -- 172.21.15.72:0/227250024 wait complete. 2024-03-06T07:26:39.750 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:39.898 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":558,"msgr_send_messages":558,"msgr_recv_bytes":150957,"msgr_send_bytes":4397565,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.075984708,"msgr_running_send_time":0.042133442,"msgr_running_recv_time":0.014590783,"msgr_running_fast_dispatch_time":0.014858013,"msgr_send_messages_queue_lat":{"avgcount":557,"sum":0.038822094,"avgtime":0.000069698},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9744,"msgr_send_messages":21187,"msgr_recv_bytes":3489184,"msgr_send_bytes":27015490,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.939886733,"msgr_running_send_time":0.523697858,"msgr_running_recv_time":0.480340535,"msgr_running_fast_dispatch_time":0.017799886,"msgr_send_messages_queue_lat":{"avgcount":21186,"sum":0.360279971,"avgtime":0.000017005},"msgr_handle_ack_lat":{"avgcount":28186,"sum":0.004597601,"avgtime":0.000000163},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":94192},"AsyncMessenger::Worker-2":{"msgr_recv_messages":391,"msgr_send_messages":386,"msgr_recv_bytes":321524,"msgr_send_bytes":246948,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.046612445,"msgr_running_send_time":0.021976134,"msgr_running_recv_time":0.012515020,"msgr_running_fast_dispatch_time":0.009847612,"msgr_send_messages_queue_lat":{"avgcount":385,"sum":0.030710419,"avgtime":0.000079767},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108480,"msgr_send_encrypted_bytes":8544},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":2081,"complete_latency":{"avgcount":94,"sum":5.277372689,"avgtime":0.056142262}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1937,"sum":0.918372781,"avgtime":0.000474121}},"mds":{"request":5950,"reply":5947,"reply_latency":{"avgcount":5947,"sum":8.549260708,"avgtime":0.001437575},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5383,"inodes_top":0,"inodes_bottom":5383,"inodes_pin_tail":0,"inodes_pinned":5383,"inodes_expired":0,"inodes_with_caps":1902,"caps":1902,"subtrees":2,"traverse":5955,"traverse_hit":5954,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1603,"root_rbytes":11104256,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":836,"ceph_cap_op_grant":6106,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3397,"handle_client_caps_dirty":3000,"handle_client_cap_release":5,"process_request_cap_release":8676},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1563,"num_strays_delayed":126,"num_strays_enqueuing":0,"strays_created":2737,"strays_enqueued":1174,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":10508,"evex":0,"evtrm":0,"ev":10511,"evexg":0,"evexd":0,"segadd":13,"segex":0,"segtrm":0,"seg":15,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":26501584,"rdpos":4195244,"jlat":{"avgcount":18,"sum":0.759049561,"avgtime":0.042169420},"replayed":3},"mds_mem":{"ino":2649,"ino+":3214,"ino-":565,"dir":13,"dir+":13,"dir-":0,"dn":5383,"dn+":5948,"dn-":565,"cap":1902,"cap+":3202,"cap-":1300,"rss":106620,"heap":248164},"mds_server":{"dispatch_client_request":5952,"dispatch_server_request":0,"handle_client_request":5950,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2737,"sum":5.333359494,"avgtime":0.001948615},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5477,"average_load":5477,"avg_session_uptime":829,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":29982667,"buffer_anon_items":8409,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14763033,"mds_co_items":151760,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1292,"op_send_bytes":21486427,"op_resend":0,"op_reply":1292,"op_latency":{"avgcount":1292,"sum":2.380948966,"avgtime":0.001842839},"op_inflight":0,"oplen_avg":{"avgcount":1292,"sum":1358},"op":1292,"op_r":48,"op_w":1244,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":20,"osdop_write":31,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1190,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":27,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1174,"pq_executed":1174,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":297,"max":104857600,"get_started":0,"get":10689,"get_sum":2996834,"get_or_fail_fail":0,"get_or_fail_success":10689,"take":0,"take_sum":0,"put":10688,"put_sum":2996537,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1292,"take_sum":21606118,"put":1292,"put_sum":21606118,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1292,"take_sum":1292,"put":1292,"put_sum":1292,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1174,"get_sum":118574,"get_or_fail_fail":0,"get_or_fail_success":1174,"take":0,"take_sum":0,"put":12,"put_sum":118574,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":1335760,"max":3758096384,"get_started":0,"get":10508,"get_sum":22305899,"get_or_fail_fail":0,"get_or_fail_success":10508,"take":0,"take_sum":0,"put":19,"put_sum":20970139,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:39.900 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 150957, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 558, "msgr_running_fast_dispatch_time": 0.014858013, "msgr_running_recv_time": 0.014590783, "msgr_running_send_time": 0.042133442, "msgr_running_total_time": 0.075984708, "msgr_send_bytes": 4397565, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 558, "msgr_send_messages_queue_lat": { "avgcount": 557, "avgtime": 6.9698e-05, "sum": 0.038822094 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 28186, "avgtime": 1.63e-07, "sum": 0.004597601 }, "msgr_recv_bytes": 3489184, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9744, "msgr_running_fast_dispatch_time": 0.017799886, "msgr_running_recv_time": 0.480340535, "msgr_running_send_time": 0.523697858, "msgr_running_total_time": 0.939886733, "msgr_send_bytes": 27015490, "msgr_send_encrypted_bytes": 94192, "msgr_send_messages": 21187, "msgr_send_messages_queue_lat": { "avgcount": 21186, "avgtime": 1.7005e-05, "sum": 0.360279971 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 321524, "msgr_recv_encrypted_bytes": 108480, "msgr_recv_messages": 391, "msgr_running_fast_dispatch_time": 0.009847612, "msgr_running_recv_time": 0.01251502, "msgr_running_send_time": 0.021976134, "msgr_running_total_time": 0.046612445, "msgr_send_bytes": 246948, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 386, "msgr_send_messages_queue_lat": { "avgcount": 385, "avgtime": 7.9767e-05, "sum": 0.030710419 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 94, "avgtime": 0.056142262, "sum": 5.277372689 }, "queue_len": 2081 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1937, "avgtime": 0.000474121, "sum": 0.918372781 }, "queue_len": 0 }, "mds": { "caps": 1902, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6106, "ceph_cap_op_revoke": 836, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 5, "handle_client_caps": 3397, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5383, "inodes_bottom": 5383, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5383, "inodes_top": 0, "inodes_with_caps": 1902, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 8676, "q": 0, "reply": 5947, "reply_latency": { "avgcount": 5947, "avgtime": 0.001437575, "sum": 8.549260708 }, "request": 5950, "root_rbytes": 11104256, "root_rfiles": 1603, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5955, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5954, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1563, "num_strays_delayed": 126, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2737, "strays_enqueued": 1174, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 10511, "evadd": 10508, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 18, "avgtime": 0.04216942, "sum": 0.759049561 }, "rdpos": 4195244, "replayed": 3, "seg": 15, "segadd": 13, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 26501584 }, "mds_mem": { "cap": 1902, "cap+": 3202, "cap-": 1300, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5383, "dn+": 5948, "dn-": 565, "heap": 248164, "ino": 2649, "ino+": 3214, "ino-": 565, "rss": 106620 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5952, "dispatch_server_request": 0, "handle_client_request": 5950, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2737, "avgtime": 0.001948615, "sum": 5.333359494 } }, "mds_sessions": { "average_load": 5477, "avg_session_uptime": 829, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5477 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 29982667, "buffer_anon_items": 8409, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14763033, "mds_co_items": 151760, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 27, "op": 1292, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1292, "avgtime": 0.001842839, "sum": 2.380948966 }, "op_pg": 0, "op_r": 48, "op_reply": 1292, "op_resend": 0, "op_rmw": 0, "op_send": 1292, "op_send_bytes": 21486427, "op_w": 1244, "oplen_avg": { "avgcount": 1292, "sum": 1358 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1190, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 20, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 31, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1174, "pq_executed_ops": 1174, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10689, "get_or_fail_fail": 0, "get_or_fail_success": 10689, "get_started": 0, "get_sum": 2996834, "max": 104857600, "put": 10688, "put_sum": 2996537, "take": 0, "take_sum": 0, "val": 297, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1292, "put_sum": 21606118, "take": 1292, "take_sum": 21606118, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1292, "put_sum": 1292, "take": 1292, "take_sum": 1292, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1174, "get_or_fail_fail": 0, "get_or_fail_success": 1174, "get_started": 0, "get_sum": 118574, "max": 3758096384, "put": 12, "put_sum": 118574, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 10508, "get_or_fail_fail": 0, "get_or_fail_success": 10508, "get_started": 0, "get_sum": 22305899, "max": 3758096384, "put": 19, "put_sum": 20970139, "take": 0, "take_sum": 0, "val": 1335760, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:39.900 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1563, 1174/2737 2024-03-06T07:26:40.901 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:41.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.016+0000 7fd5ef913640 1 Processor -- start 2024-03-06T07:26:41.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.016+0000 7fd5ef913640 1 -- start start 2024-03-06T07:26:41.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ef913640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e815e150 0x7fd5e80cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ef913640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5e80cfba0 0x7fd5e80d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ef913640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd5e80d0e90 0x7fd5e80b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ef913640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e80d2ab0 con 0x7fd5e80d0e90 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ef913640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e80d1ea0 con 0x7fd5e80cfba0 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ef913640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80d2180 con 0x7fd5e815e150 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd5e80d0e90 0x7fd5e80b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd5e80d0e90 0x7fd5e80b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57890/0 (socket says 172.21.15.72:57890) 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 -- 172.21.15.72:0/1416382936 learned_addr learned my addr 172.21.15.72:0/1416382936 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5eceae640 1 --2- 172.21.15.72:0/1416382936 >> v2:172.21.15.146:3300/0 conn(0x7fd5e80cfba0 0x7fd5e80d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ed6af640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e815e150 0x7fd5e80cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 -- 172.21.15.72:0/1416382936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e815e150 msgr2=0x7fd5e80cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e815e150 0x7fd5e80cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 -- 172.21.15.72:0/1416382936 >> v2:172.21.15.146:3300/0 conn(0x7fd5e80cfba0 msgr2=0x7fd5e80d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 --2- 172.21.15.72:0/1416382936 >> v2:172.21.15.146:3300/0 conn(0x7fd5e80cfba0 0x7fd5e80d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5edeb0640 1 -- 172.21.15.72:0/1416382936 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e80d2460 con 0x7fd5e80d0e90 2024-03-06T07:26:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5eceae640 1 --2- 172.21.15.72:0/1416382936 >> v2:172.21.15.146:3300/0 conn(0x7fd5e80cfba0 0x7fd5e80d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.017+0000 7fd5ed6af640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e815e150 0x7fd5e80cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.018+0000 7fd5edeb0640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80d0e90 0x7fd5e80b9060 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd5e808c920 tx=0x7fd5e002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940070c581d80cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.018+0000 7fd5dffff640 1 -- 172.21.15.72:0/1416382936 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e002c9e0 con 0x7fd5e80d0e90 2024-03-06T07:26:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.018+0000 7fd5dffff640 1 -- 172.21.15.72:0/1416382936 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e002cb80 con 0x7fd5e80d0e90 2024-03-06T07:26:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.018+0000 7fd5dffff640 1 -- 172.21.15.72:0/1416382936 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00343f0 con 0x7fd5e80d0e90 2024-03-06T07:26:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 -- 172.21.15.72:0/1416382936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80d0e90 msgr2=0x7fd5e80b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80d0e90 0x7fd5e80b9060 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd5e808c920 tx=0x7fd5e002be40 comp rx=0 tx=0).stop 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 -- 172.21.15.72:0/1416382936 shutdown_connections 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80d0e90 0x7fd5e80b9060 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 --2- 172.21.15.72:0/1416382936 >> v2:172.21.15.146:3300/0 conn(0x7fd5e80cfba0 0x7fd5e80d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 --2- 172.21.15.72:0/1416382936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e815e150 0x7fd5e80cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 -- 172.21.15.72:0/1416382936 >> 172.21.15.72:0/1416382936 conn(0x7fd5e809d230 msgr2=0x7fd5e80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 -- 172.21.15.72:0/1416382936 shutdown_connections 2024-03-06T07:26:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.019+0000 7fd5ef913640 1 -- 172.21.15.72:0/1416382936 wait complete. 2024-03-06T07:26:41.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.021+0000 7fd5ef913640 1 Processor -- start 2024-03-06T07:26:41.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.021+0000 7fd5ef913640 1 -- start start 2024-03-06T07:26:41.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ef913640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 0x7fd5e8150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ef913640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e80d0e90 0x7fd5e809dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ef913640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e815e150 0x7fd5e809e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ef913640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80d1b30 con 0x7fd5e80cfba0 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ef913640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80d2f40 con 0x7fd5e815e150 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ef913640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e815e530 con 0x7fd5e80d0e90 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 0x7fd5e8150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 0x7fd5e8150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57894/0 (socket says 172.21.15.72:57894) 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 -- 172.21.15.72:0/3094547909 learned_addr learned my addr 172.21.15.72:0/3094547909 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5edeb0640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e815e150 0x7fd5e809e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5eceae640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e80d0e90 0x7fd5e809dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 -- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e80d0e90 msgr2=0x7fd5e809dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e80d0e90 0x7fd5e809dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 -- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e815e150 msgr2=0x7fd5e809e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e815e150 0x7fd5e809e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 -- 172.21.15.72:0/3094547909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e80a09a0 con 0x7fd5e80cfba0 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.022+0000 7fd5ed6af640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 0x7fd5e8150330 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c820 tx=0x7fd5e400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.023+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4015070 con 0x7fd5e80cfba0 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.023+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e4007a30 con 0x7fd5e80cfba0 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.023+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4007e00 con 0x7fd5e80cfba0 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.023+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e80a0c80 con 0x7fd5e80cfba0 2024-03-06T07:26:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.023+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e80a1240 con 0x7fd5e80cfba0 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.024+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e8069ee0 con 0x7fd5e80cfba0 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.024+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd5e4018070 con 0x7fd5e80cfba0 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.025+0000 7fd5ddffb640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5cc06b4d0 0x7fd5cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.025+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd5e408a1f0 con 0x7fd5e80cfba0 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.025+0000 7fd5eceae640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5cc06b4d0 0x7fd5cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.025+0000 7fd5eceae640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5cc06b4d0 0x7fd5cc06d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd5e80c73b0 tx=0x7fd5d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.033+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd5e4058f70 con 0x7fd5e80cfba0 2024-03-06T07:26:41.257 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:41.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.355+0000 7fd304888640 1 Processor -- start 2024-03-06T07:26:41.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.355+0000 7fd304888640 1 -- start start 2024-03-06T07:26:41.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd304888640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e150 0x7fd3000cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd304888640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3000d1060 0x7fd3000b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd304888640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3000d1ea0 con 0x7fd3000cfba0 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd304888640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3000d2ab0 con 0x7fd3000d1060 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd304888640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000d2180 con 0x7fd30015e150 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3000d1060 0x7fd3000b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd3000d1060 0x7fd3000b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57900/0 (socket says 172.21.15.72:57900) 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 -- 172.21.15.72:0/3178903818 learned_addr learned my addr 172.21.15.72:0/3178903818 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fe59c640 1 --2- 172.21.15.72:0/3178903818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e150 0x7fd3000cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 -- 172.21.15.72:0/3178903818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e150 msgr2=0x7fd3000cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 --2- 172.21.15.72:0/3178903818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e150 0x7fd3000cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 -- 172.21.15.72:0/3178903818 >> v1:172.21.15.146:6789/0 conn(0x7fd3000cfba0 legacy=0x7fd3000d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.356+0000 7fd2fed9d640 1 -- 172.21.15.72:0/3178903818 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3000d2460 con 0x7fd3000d1060 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.357+0000 7fd2fed9d640 1 --2- 172.21.15.72:0/3178903818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3000d1060 0x7fd3000b9060 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009e50 tx=0x7fd2f802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8015a378191d830 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.357+0000 7fd2fd59a640 1 -- 172.21.15.72:0/3178903818 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f802ca70 con 0x7fd3000d1060 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.357+0000 7fd2fd59a640 1 -- 172.21.15.72:0/3178903818 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f802cc10 con 0x7fd3000d1060 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.357+0000 7fd2fd59a640 1 -- 172.21.15.72:0/3178903818 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f80343f0 con 0x7fd3000d1060 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 -- 172.21.15.72:0/3178903818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3000d1060 msgr2=0x7fd3000b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 --2- 172.21.15.72:0/3178903818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3000d1060 0x7fd3000b9060 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009e50 tx=0x7fd2f802bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 -- 172.21.15.72:0/3178903818 shutdown_connections 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 --2- 172.21.15.72:0/3178903818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3000d1060 0x7fd3000b9060 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 --2- 172.21.15.72:0/3178903818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd30015e150 0x7fd3000cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 -- 172.21.15.72:0/3178903818 >> 172.21.15.72:0/3178903818 conn(0x7fd30009d230 msgr2=0x7fd3000d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 -- 172.21.15.72:0/3178903818 shutdown_connections 2024-03-06T07:26:41.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.358+0000 7fd304888640 1 -- 172.21.15.72:0/3178903818 wait complete. 2024-03-06T07:26:41.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.360+0000 7fd304888640 1 Processor -- start 2024-03-06T07:26:41.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.360+0000 7fd304888640 1 -- start start 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000cfba0 0x7fd300159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000d1060 0x7fd30015b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 0x7fd30015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000d1b30 con 0x7fd30015e150 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3000d2f40 con 0x7fd3000cfba0 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30015e530 con 0x7fd3000d1060 2024-03-06T07:26:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 0x7fd30015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 0x7fd30015b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57914/0 (socket says 172.21.15.72:57914) 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 -- 172.21.15.72:0/2975309645 learned_addr learned my addr 172.21.15.72:0/2975309645 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fe59c640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000cfba0 0x7fd300159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fdd9b640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000d1060 0x7fd30015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 -- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000d1060 msgr2=0x7fd30015b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000d1060 0x7fd30015b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 -- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000cfba0 msgr2=0x7fd300159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000cfba0 0x7fd300159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 -- 172.21.15.72:0/2975309645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30015a4f0 con 0x7fd30015e150 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fe59c640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000cfba0 0x7fd300159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2fed9d640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 0x7fd30015b7a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009f70 tx=0x7fd2f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8043020 con 0x7fd30015e150 2024-03-06T07:26:41.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f803e450 con 0x7fd30015e150 2024-03-06T07:26:41.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30015a7d0 con 0x7fd30015e150 2024-03-06T07:26:41.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.362+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f8009510 con 0x7fd30015e150 2024-03-06T07:26:41.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.363+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3000b7380 con 0x7fd30015e150 2024-03-06T07:26:41.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.364+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd300069ee0 con 0x7fd30015e150 2024-03-06T07:26:41.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.364+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2f8046020 con 0x7fd30015e150 2024-03-06T07:26:41.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.364+0000 7fd2eeffd640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:41.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.365+0000 7fd2fe59c640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:41.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.365+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd2f80ac980 con 0x7fd30015e150 2024-03-06T07:26:41.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.365+0000 7fd2fe59c640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009af0 tx=0x7fd2f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:41.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.370+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5e80d4670 con 0x7fd5e80cfba0 2024-03-06T07:26:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.373+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2f804d030 con 0x7fd30015e150 2024-03-06T07:26:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.373+0000 7fd5ddffb640 1 -- 172.21.15.72:0/3094547909 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fd5e4058870 con 0x7fd5e80cfba0 2024-03-06T07:26:41.375 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:41.375 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5cc06b4d0 msgr2=0x7fd5cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5cc06b4d0 0x7fd5cc06d990 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd5e80c73b0 tx=0x7fd5d8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 msgr2=0x7fd5e8150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 0x7fd5e8150330 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fd5e400c820 tx=0x7fd5e400cc20 comp rx=0 tx=0).stop 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 shutdown_connections 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5cc06b4d0 0x7fd5cc06d990 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e815e150 0x7fd5e809e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e80d0e90 0x7fd5e809dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 --2- 172.21.15.72:0/3094547909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5e80cfba0 0x7fd5e8150330 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 >> 172.21.15.72:0/3094547909 conn(0x7fd5e809d230 msgr2=0x7fd5e8160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 shutdown_connections 2024-03-06T07:26:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.375+0000 7fd5ef913640 1 -- 172.21.15.72:0/3094547909 wait complete. 2024-03-06T07:26:41.388 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:41.506 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":736,"msgr_send_messages":736,"msgr_recv_bytes":196169,"msgr_send_bytes":4455593,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.097285496,"msgr_running_send_time":0.053200562,"msgr_running_recv_time":0.018246552,"msgr_running_fast_dispatch_time":0.020101849,"msgr_send_messages_queue_lat":{"avgcount":735,"sum":0.047427598,"avgtime":0.000064527},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10334,"msgr_send_messages":24747,"msgr_recv_bytes":3708298,"msgr_send_bytes":28015579,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.045081184,"msgr_running_send_time":0.591069007,"msgr_running_recv_time":0.514757720,"msgr_running_fast_dispatch_time":0.020886098,"msgr_send_messages_queue_lat":{"avgcount":24746,"sum":0.470286597,"avgtime":0.000019004},"msgr_handle_ack_lat":{"avgcount":30598,"sum":0.005015839,"avgtime":0.000000163},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95984},"AsyncMessenger::Worker-2":{"msgr_recv_messages":543,"msgr_send_messages":538,"msgr_recv_bytes":404149,"msgr_send_bytes":680659,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.074554323,"msgr_running_send_time":0.038648878,"msgr_running_recv_time":0.015805469,"msgr_running_fast_dispatch_time":0.015523043,"msgr_send_messages_queue_lat":{"avgcount":537,"sum":0.040935933,"avgtime":0.000076230},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108480,"msgr_send_encrypted_bytes":8544},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":98,"sum":7.473357707,"avgtime":0.076258752}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":2600,"sum":1.197732103,"avgtime":0.000460666}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.676088480,"avgtime":0.001509294},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":5112,"inodes_top":0,"inodes_bottom":5112,"inodes_pin_tail":0,"inodes_pinned":5112,"inodes_expired":0,"inodes_with_caps":1591,"caps":1591,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":132,"root_rbytes":1028096,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":838,"ceph_cap_op_grant":7592,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3399,"handle_client_caps_dirty":3000,"handle_client_cap_release":6,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1590,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":1611,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11410,"evex":0,"evtrm":0,"ev":11413,"evexg":0,"evexd":0,"segadd":14,"segex":0,"segtrm":0,"seg":16,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28242440,"rdpos":4195244,"jlat":{"avgcount":20,"sum":2.437884606,"avgtime":0.121894230},"replayed":3},"mds_mem":{"ino":1914,"ino+":3214,"ino-":1300,"dir":13,"dir+":13,"dir-":0,"dn":5112,"dn+":6412,"dn-":1300,"cap":1591,"cap+":3202,"cap-":1611,"rss":113660,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.534121606,"avgtime":0.534121606},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.926065660,"avgtime":0.001851895},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5850,"average_load":5850,"avg_session_uptime":829,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":29982667,"buffer_anon_items":8409,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14763033,"mds_co_items":151760,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1745,"op_send_bytes":24417750,"op_resend":0,"op_reply":1745,"op_latency":{"avgcount":1745,"sum":3.198931440,"avgtime":0.001833198},"op_inflight":0,"oplen_avg":{"avgcount":1745,"sum":1813},"op":1745,"op_r":54,"op_w":1691,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":26,"osdop_write":38,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1627,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":29,"omap_rd":48,"omap_del":9},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1611,"pq_executed":1611,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11609,"get_sum":3260985,"get_or_fail_fail":0,"get_or_fail_success":11609,"take":0,"take_sum":0,"put":11609,"put_sum":3260985,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1745,"take_sum":24581578,"put":1745,"put_sum":24581578,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1745,"take_sum":1745,"put":1745,"put_sum":1745,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1611,"get_sum":162711,"get_or_fail_fail":0,"get_or_fail_success":1611,"take":0,"take_sum":0,"put":18,"put_sum":162711,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":189835,"max":3758096384,"get_started":0,"get":11410,"get_sum":24046755,"get_or_fail_fail":0,"get_or_fail_success":11410,"take":0,"take_sum":0,"put":20,"put_sum":23856920,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:41.507 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 196169, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 736, "msgr_running_fast_dispatch_time": 0.020101849, "msgr_running_recv_time": 0.018246552, "msgr_running_send_time": 0.053200562, "msgr_running_total_time": 0.097285496, "msgr_send_bytes": 4455593, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 736, "msgr_send_messages_queue_lat": { "avgcount": 735, "avgtime": 6.4527e-05, "sum": 0.047427598 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30598, "avgtime": 1.63e-07, "sum": 0.005015839 }, "msgr_recv_bytes": 3708298, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10334, "msgr_running_fast_dispatch_time": 0.020886098, "msgr_running_recv_time": 0.51475772, "msgr_running_send_time": 0.591069007, "msgr_running_total_time": 1.045081184, "msgr_send_bytes": 28015579, "msgr_send_encrypted_bytes": 95984, "msgr_send_messages": 24747, "msgr_send_messages_queue_lat": { "avgcount": 24746, "avgtime": 1.9004e-05, "sum": 0.470286597 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 404149, "msgr_recv_encrypted_bytes": 108480, "msgr_recv_messages": 543, "msgr_running_fast_dispatch_time": 0.015523043, "msgr_running_recv_time": 0.015805469, "msgr_running_send_time": 0.038648878, "msgr_running_total_time": 0.074554323, "msgr_send_bytes": 680659, "msgr_send_encrypted_bytes": 8544, "msgr_send_messages": 538, "msgr_send_messages_queue_lat": { "avgcount": 537, "avgtime": 7.623e-05, "sum": 0.040935933 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 98, "avgtime": 0.076258752, "sum": 7.473357707 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2600, "avgtime": 0.000460666, "sum": 1.197732103 }, "queue_len": 0 }, "mds": { "caps": 1591, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7592, "ceph_cap_op_revoke": 838, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 6, "handle_client_caps": 3399, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 5112, "inodes_bottom": 5112, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 5112, "inodes_top": 0, "inodes_with_caps": 1591, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001509294, "sum": 9.67608848 }, "request": 6413, "root_rbytes": 1028096, "root_rfiles": 132, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1590, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 1611, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11413, "evadd": 11410, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 20, "avgtime": 0.12189423, "sum": 2.437884606 }, "rdpos": 4195244, "replayed": 3, "seg": 16, "segadd": 14, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28242440 }, "mds_mem": { "cap": 1591, "cap+": 3202, "cap-": 1611, "dir": 13, "dir+": 13, "dir-": 0, "dn": 5112, "dn+": 6412, "dn-": 1300, "heap": 248164, "ino": 1914, "ino+": 3214, "ino-": 1300, "rss": 113660 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.534121606, "sum": 0.534121606 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001851895, "sum": 5.92606566 } }, "mds_sessions": { "average_load": 5850, "avg_session_uptime": 829, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5850 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 29982667, "buffer_anon_items": 8409, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14763033, "mds_co_items": 151760, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 9, "omap_rd": 48, "omap_wr": 29, "op": 1745, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1745, "avgtime": 0.001833198, "sum": 3.19893144 }, "op_pg": 0, "op_r": 54, "op_reply": 1745, "op_resend": 0, "op_rmw": 0, "op_send": 1745, "op_send_bytes": 24417750, "op_w": 1691, "oplen_avg": { "avgcount": 1745, "sum": 1813 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1627, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 26, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 38, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1611, "pq_executed_ops": 1611, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11609, "get_or_fail_fail": 0, "get_or_fail_success": 11609, "get_started": 0, "get_sum": 3260985, "max": 104857600, "put": 11609, "put_sum": 3260985, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1745, "put_sum": 24581578, "take": 1745, "take_sum": 24581578, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1745, "put_sum": 1745, "take": 1745, "take_sum": 1745, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1611, "get_or_fail_fail": 0, "get_or_fail_success": 1611, "get_started": 0, "get_sum": 162711, "max": 3758096384, "put": 18, "put_sum": 162711, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 11410, "get_or_fail_fail": 0, "get_or_fail_success": 11410, "get_started": 0, "get_sum": 24046755, "max": 3758096384, "put": 20, "put_sum": 23856920, "take": 0, "take_sum": 0, "val": 189835, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:41.507 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1590, 1611/3201 2024-03-06T07:26:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.671+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3000b9e20 con 0x7fd30015e150 2024-03-06T07:26:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.672+0000 7fd2eeffd640 1 -- 172.21.15.72:0/2975309645 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fd2f807b790 con 0x7fd30015e150 2024-03-06T07:26:41.674 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:41.674 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2e006b4d0 msgr2=0x7fd2e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2e006b4d0 0x7fd2e006d990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009af0 tx=0x7fd2f0008040 comp rx=0 tx=0).stop 2024-03-06T07:26:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 msgr2=0x7fd30015b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 0x7fd30015b7a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fd2f8009f70 tx=0x7fd2f8004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 shutdown_connections 2024-03-06T07:26:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2e006b4d0 0x7fd2e006d990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd30015e150 0x7fd30015b7a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3000d1060 0x7fd30015b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 --2- 172.21.15.72:0/2975309645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3000cfba0 0x7fd300159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 >> 172.21.15.72:0/2975309645 conn(0x7fd30009d230 msgr2=0x7fd30006d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 shutdown_connections 2024-03-06T07:26:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:41.674+0000 7fd304888640 1 -- 172.21.15.72:0/2975309645 wait complete. 2024-03-06T07:26:41.687 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:26:42.070 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:26:42.071 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:26:42.508 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:42.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.608+0000 7f7bbfb79640 1 Processor -- start 2024-03-06T07:26:42.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.608+0000 7f7bbfb79640 1 -- start start 2024-03-06T07:26:42.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbfb79640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb8161670 0x7f7bb8161e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbfb79640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7bb80b2da0 0x7f7bb81623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbfb79640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bb80cc680 0x7f7bb80ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbfb79640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb80b3590 con 0x7f7bb80cc680 2024-03-06T07:26:42.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbfb79640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bb8162c70 con 0x7f7bb80b2da0 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbfb79640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb806cd80 con 0x7f7bb8161670 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bb80cc680 0x7f7bb80ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bb80cc680 0x7f7bb80ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58272/0 (socket says 172.21.15.72:58272) 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 -- 172.21.15.72:0/3179438063 learned_addr learned my addr 172.21.15.72:0/3179438063 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbd114640 1 --2- 172.21.15.72:0/3179438063 >> v2:172.21.15.146:3300/0 conn(0x7f7bb80b2da0 0x7f7bb81623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbd915640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb8161670 0x7f7bb8161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 -- 172.21.15.72:0/3179438063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb8161670 msgr2=0x7f7bb8161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb8161670 0x7f7bb8161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 -- 172.21.15.72:0/3179438063 >> v2:172.21.15.146:3300/0 conn(0x7f7bb80b2da0 msgr2=0x7f7bb81623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 --2- 172.21.15.72:0/3179438063 >> v2:172.21.15.146:3300/0 conn(0x7f7bb80b2da0 0x7f7bb81623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbe116640 1 -- 172.21.15.72:0/3179438063 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb806d060 con 0x7f7bb80cc680 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.609+0000 7f7bbd915640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb8161670 0x7f7bb8161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.610+0000 7f7bbe116640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb80cc680 0x7f7bb80ccb10 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8071c80 tx=0x7f7bb002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91c3933537162fd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.610+0000 7f7bbc913640 1 -- 172.21.15.72:0/3179438063 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb002ca70 con 0x7f7bb80cc680 2024-03-06T07:26:42.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.610+0000 7f7bbc913640 1 -- 172.21.15.72:0/3179438063 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb002cc10 con 0x7f7bb80cc680 2024-03-06T07:26:42.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.610+0000 7f7bbc913640 1 -- 172.21.15.72:0/3179438063 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb00343f0 con 0x7f7bb80cc680 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 -- 172.21.15.72:0/3179438063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb80cc680 msgr2=0x7f7bb80ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb80cc680 0x7f7bb80ccb10 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8071c80 tx=0x7f7bb002bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 -- 172.21.15.72:0/3179438063 shutdown_connections 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb80cc680 0x7f7bb80ccb10 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/3179438063 >> v2:172.21.15.146:3300/0 conn(0x7f7bb80b2da0 0x7f7bb81623a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/3179438063 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb8161670 0x7f7bb8161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 -- 172.21.15.72:0/3179438063 >> 172.21.15.72:0/3179438063 conn(0x7f7bb80d1d50 msgr2=0x7f7bb80d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 -- 172.21.15.72:0/3179438063 shutdown_connections 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.611+0000 7f7bbfb79640 1 -- 172.21.15.72:0/3179438063 wait complete. 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.612+0000 7f7bbfb79640 1 Processor -- start 2024-03-06T07:26:42.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.612+0000 7f7bbfb79640 1 -- start start 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb80b2da0 0x7f7bb81f5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bb80cc680 0x7f7bb815b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 0x7f7bb815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb806d340 con 0x7f7bb8161670 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb81628e0 con 0x7f7bb80cc680 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bb806d4c0 con 0x7f7bb80b2da0 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 0x7f7bb815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 0x7f7bb815d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58280/0 (socket says 172.21.15.72:58280) 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 -- 172.21.15.72:0/4085764171 learned_addr learned my addr 172.21.15.72:0/4085764171 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 -- 172.21.15.72:0/4085764171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb80b2da0 msgr2=0x7f7bb81f5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb80b2da0 0x7f7bb81f5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 -- 172.21.15.72:0/4085764171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bb80cc680 msgr2=0x7f7bb815b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bb80cc680 0x7f7bb815b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 -- 172.21.15.72:0/4085764171 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb80b4ba0 con 0x7f7bb8161670 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbe116640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 0x7f7bb815d850 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f7bb002c4e0 tx=0x7f7bb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0043020 con 0x7f7bb8161670 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb003e450 con 0x7f7bb8161670 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb80b4e40 con 0x7f7bb8161670 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0009510 con 0x7f7bb8161670 2024-03-06T07:26:42.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.614+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb80b5400 con 0x7f7bb8161670 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.615+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bb8068050 con 0x7f7bb8161670 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.616+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7bb0046020 con 0x7f7bb8161670 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.616+0000 7f7bb67fc640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ba406b500 0x7f7ba406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.616+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7bb00ac7b0 con 0x7f7bb8161670 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.616+0000 7f7bbd915640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ba406b500 0x7f7ba406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:42.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.617+0000 7f7bbd915640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ba406b500 0x7f7ba406d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8006b10 tx=0x7f7ba8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:42.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.625+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7bb004d070 con 0x7f7bb8161670 2024-03-06T07:26:42.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.840+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bb80b7cd0 con 0x7f7bb8161670 2024-03-06T07:26:42.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.840+0000 7f7bb67fc640 1 -- 172.21.15.72:0/4085764171 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f7bb007b5c0 con 0x7f7bb8161670 2024-03-06T07:26:42.842 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:42.843 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:42.843 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:42.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ba406b500 msgr2=0x7f7ba406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ba406b500 0x7f7ba406d9c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7ba8006b10 tx=0x7f7ba8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:42.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 msgr2=0x7f7bb815d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 0x7f7bb815d850 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f7bb002c4e0 tx=0x7f7bb0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 shutdown_connections 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7ba406b500 0x7f7ba406d9c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bb8161670 0x7f7bb815d850 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bb80cc680 0x7f7bb815b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 --2- 172.21.15.72:0/4085764171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bb80b2da0 0x7f7bb81f5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 >> 172.21.15.72:0/4085764171 conn(0x7f7bb80d1d50 msgr2=0x7f7bb80ccaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 shutdown_connections 2024-03-06T07:26:42.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:42.842+0000 7f7bbfb79640 1 -- 172.21.15.72:0/4085764171 wait complete. 2024-03-06T07:26:42.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:42.984 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1424,"msgr_send_messages":1424,"msgr_recv_bytes":428408,"msgr_send_bytes":5698966,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.143194879,"msgr_running_send_time":0.073634912,"msgr_running_recv_time":0.022151282,"msgr_running_fast_dispatch_time":0.040470968,"msgr_send_messages_queue_lat":{"avgcount":1423,"sum":1.330879263,"avgtime":0.000935263},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10780,"msgr_send_messages":25190,"msgr_recv_bytes":3895741,"msgr_send_bytes":28274483,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.093886108,"msgr_running_send_time":0.605985589,"msgr_running_recv_time":0.517587509,"msgr_running_fast_dispatch_time":0.044744094,"msgr_send_messages_queue_lat":{"avgcount":25189,"sum":0.915899822,"avgtime":0.000036361},"msgr_handle_ack_lat":{"avgcount":30603,"sum":0.005016494,"avgtime":0.000000163},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95984},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1047,"msgr_send_messages":1042,"msgr_recv_bytes":573868,"msgr_send_bytes":1152318,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.107222291,"msgr_running_send_time":0.046179350,"msgr_running_recv_time":0.018769624,"msgr_running_fast_dispatch_time":0.037237196,"msgr_send_messages_queue_lat":{"avgcount":1041,"sum":0.579673933,"avgtime":0.000556843},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108752,"msgr_send_encrypted_bytes":9168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":122,"sum":7.695738426,"avgtime":0.063079823}},"finisher-PurgeQueue":{"queue_len":3,"complete_latency":{"avgcount":2601,"sum":1.197817236,"avgtime":0.000460521}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.676088480,"avgtime":0.001509294},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":4801,"inodes_top":0,"inodes_bottom":4801,"inodes_pin_tail":0,"inodes_pinned":4801,"inodes_expired":0,"inodes_with_caps":1001,"caps":1001,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":838,"ceph_cap_op_grant":7592,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3399,"handle_client_caps_dirty":3000,"handle_client_cap_release":8,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1590,"num_strays_delayed":1590,"num_strays_enqueuing":846,"strays_created":3201,"strays_enqueued":1611,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":11413,"evex":11413,"evtrm":11413,"ev":11415,"evexg":0,"evexd":11412,"segadd":15,"segex":16,"segtrm":16,"seg":1,"segmjr":1,"segexg":0,"segexd":15,"expos":28242440,"wrpos":28246121,"rdpos":4195244,"jlat":{"avgcount":23,"sum":2.495590211,"avgtime":0.108503922},"replayed":3},"mds_mem":{"ino":1603,"ino+":3214,"ino-":1611,"dir":13,"dir+":13,"dir-":0,"dn":1601,"dn+":6412,"dn-":4811,"cap":2,"cap+":3202,"cap-":3200,"rss":116220,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.534121606,"avgtime":0.534121606},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.926065660,"avgtime":0.001851895},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5850,"average_load":5850,"avg_session_uptime":829,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":29982667,"buffer_anon_items":8409,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":14763033,"mds_co_items":151760,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3378,"op_send_bytes":25736405,"op_resend":0,"op_reply":3378,"op_latency":{"avgcount":3378,"sum":142.102955706,"avgtime":0.042067186},"op_inflight":0,"oplen_avg":{"avgcount":3378,"sum":6670},"op":3378,"op_r":55,"op_w":3323,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1602,"osdop_read":27,"osdop_write":43,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1632,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3204,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":51,"omap_rd":48,"omap_del":12},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1611,"pq_executed":1611,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13247,"get_sum":3702914,"get_or_fail_fail":0,"get_or_fail_success":13247,"take":0,"take_sum":0,"put":13247,"put_sum":3702914,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3378,"take_sum":25900334,"put":3378,"put_sum":25900334,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3378,"take_sum":3378,"put":3378,"put_sum":3378,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":80497,"max":3758096384,"get_started":0,"get":2464,"get_sum":248864,"get_or_fail_fail":0,"get_or_fail_success":2464,"take":0,"take_sum":0,"put":20,"put_sum":168367,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":0,"max":3758096384,"get_started":0,"get":11413,"get_sum":24050436,"get_or_fail_fail":0,"get_or_fail_success":11413,"take":0,"take_sum":0,"put":23,"put_sum":24050436,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:42.986 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 428408, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 1424, "msgr_running_fast_dispatch_time": 0.040470968, "msgr_running_recv_time": 0.022151282, "msgr_running_send_time": 0.073634912, "msgr_running_total_time": 0.143194879, "msgr_send_bytes": 5698966, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1424, "msgr_send_messages_queue_lat": { "avgcount": 1423, "avgtime": 0.000935263, "sum": 1.330879263 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30603, "avgtime": 1.63e-07, "sum": 0.005016494 }, "msgr_recv_bytes": 3895741, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 10780, "msgr_running_fast_dispatch_time": 0.044744094, "msgr_running_recv_time": 0.517587509, "msgr_running_send_time": 0.605985589, "msgr_running_total_time": 1.093886108, "msgr_send_bytes": 28274483, "msgr_send_encrypted_bytes": 95984, "msgr_send_messages": 25190, "msgr_send_messages_queue_lat": { "avgcount": 25189, "avgtime": 3.6361e-05, "sum": 0.915899822 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 573868, "msgr_recv_encrypted_bytes": 108752, "msgr_recv_messages": 1047, "msgr_running_fast_dispatch_time": 0.037237196, "msgr_running_recv_time": 0.018769624, "msgr_running_send_time": 0.04617935, "msgr_running_total_time": 0.107222291, "msgr_send_bytes": 1152318, "msgr_send_encrypted_bytes": 9168, "msgr_send_messages": 1042, "msgr_send_messages_queue_lat": { "avgcount": 1041, "avgtime": 0.000556843, "sum": 0.579673933 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 122, "avgtime": 0.063079823, "sum": 7.695738426 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 2601, "avgtime": 0.000460521, "sum": 1.197817236 }, "queue_len": 3 }, "mds": { "caps": 1001, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7592, "ceph_cap_op_revoke": 838, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 3399, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4801, "inodes_bottom": 4801, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4801, "inodes_top": 0, "inodes_with_caps": 1001, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001509294, "sum": 9.67608848 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1590, "num_strays_delayed": 1590, "num_strays_enqueuing": 846, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 1611, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 11415, "evadd": 11413, "evex": 11413, "evexd": 11412, "evexg": 0, "evlrg": 0, "evtrm": 11413, "expos": 28242440, "jlat": { "avgcount": 23, "avgtime": 0.108503922, "sum": 2.495590211 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 15, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 28246121 }, "mds_mem": { "cap": 2, "cap+": 3202, "cap-": 3200, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1601, "dn+": 6412, "dn-": 4811, "heap": 248164, "ino": 1603, "ino+": 3214, "ino-": 1611, "rss": 116220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.534121606, "sum": 0.534121606 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001851895, "sum": 5.92606566 } }, "mds_sessions": { "average_load": 5850, "avg_session_uptime": 829, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5850 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 29982667, "buffer_anon_items": 8409, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 14763033, "mds_co_items": 151760, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 12, "omap_rd": 48, "omap_wr": 51, "op": 3378, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3378, "avgtime": 0.042067186, "sum": 142.102955706 }, "op_pg": 0, "op_r": 55, "op_reply": 3378, "op_resend": 0, "op_rmw": 0, "op_send": 3378, "op_send_bytes": 25736405, "op_w": 3323, "oplen_avg": { "avgcount": 3378, "sum": 6670 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1602, "osdop_delete": 1632, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 27, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3204, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 43, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 1611, "pq_executed_ops": 1611, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13247, "get_or_fail_fail": 0, "get_or_fail_success": 13247, "get_started": 0, "get_sum": 3702914, "max": 104857600, "put": 13247, "put_sum": 3702914, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3378, "put_sum": 25900334, "take": 3378, "take_sum": 25900334, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3378, "put_sum": 3378, "take": 3378, "take_sum": 3378, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2464, "get_or_fail_fail": 0, "get_or_fail_success": 2464, "get_started": 0, "get_sum": 248864, "max": 3758096384, "put": 20, "put_sum": 168367, "take": 0, "take_sum": 0, "val": 80497, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 11413, "get_or_fail_fail": 0, "get_or_fail_success": 11413, "get_started": 0, "get_sum": 24050436, "max": 3758096384, "put": 23, "put_sum": 24050436, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:42.986 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1590, 1611/3201 2024-03-06T07:26:43.987 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.108+0000 7fe55318c640 1 Processor -- start 2024-03-06T07:26:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.108+0000 7fe55318c640 1 -- start start 2024-03-06T07:26:44.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.109+0000 7fe55318c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe54c163950 0x7fe54c165d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.109+0000 7fe55318c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0b5300 con 0x7fe54c0ce5a0 2024-03-06T07:26:44.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.109+0000 7fe55318c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c0ccbd0 con 0x7fe54c0ce980 2024-03-06T07:26:44.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.109+0000 7fe55318c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c0b3b00 con 0x7fe54c163950 2024-03-06T07:26:44.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.110+0000 7fe550f28640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe54c0ce5a0 0x7fe54c0b4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45542/0 (socket says 172.21.15.72:45542) 2024-03-06T07:26:44.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.110+0000 7fe550f28640 1 -- 172.21.15.72:0/1405085099 learned_addr learned my addr 172.21.15.72:0/1405085099 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:44.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.110+0000 7fe551729640 1 --2- 172.21.15.72:0/1405085099 >> v2:172.21.15.72:3300/0 conn(0x7fe54c163950 0x7fe54c165d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.110+0000 7fe551729640 1 -- 172.21.15.72:0/1405085099 >> v1:172.21.15.146:6789/0 conn(0x7fe54c0ce980 legacy=0x7fe54c0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.110+0000 7fe551729640 1 -- 172.21.15.72:0/1405085099 >> v1:172.21.15.72:6789/0 conn(0x7fe54c0ce5a0 legacy=0x7fe54c0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.110+0000 7fe551729640 1 -- 172.21.15.72:0/1405085099 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c0b3de0 con 0x7fe54c163950 2024-03-06T07:26:44.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe551729640 1 --2- 172.21.15.72:0/1405085099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c165d40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fe544009d90 tx=0x7fe54402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af2a58654da0673 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe54b7fe640 1 -- 172.21.15.72:0/1405085099 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544039070 con 0x7fe54c163950 2024-03-06T07:26:44.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe54b7fe640 1 -- 172.21.15.72:0/1405085099 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54402cbd0 con 0x7fe54c163950 2024-03-06T07:26:44.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe54b7fe640 1 -- 172.21.15.72:0/1405085099 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544034440 con 0x7fe54c163950 2024-03-06T07:26:44.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe55318c640 1 -- 172.21.15.72:0/1405085099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 msgr2=0x7fe54c165d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe55318c640 1 --2- 172.21.15.72:0/1405085099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c165d40 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fe544009d90 tx=0x7fe54402be20 comp rx=0 tx=0).stop 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe55318c640 1 -- 172.21.15.72:0/1405085099 shutdown_connections 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe55318c640 1 --2- 172.21.15.72:0/1405085099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c165d40 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.111+0000 7fe55318c640 1 -- 172.21.15.72:0/1405085099 >> 172.21.15.72:0/1405085099 conn(0x7fe54c06b160 msgr2=0x7fe54c06b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.112+0000 7fe55318c640 1 -- 172.21.15.72:0/1405085099 shutdown_connections 2024-03-06T07:26:44.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.112+0000 7fe55318c640 1 -- 172.21.15.72:0/1405085099 wait complete. 2024-03-06T07:26:44.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 Processor -- start 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 -- start start 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe54c0ce5a0 0x7fe54c13aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe54c0ce980 0x7fe54c15be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c0cc9b0 con 0x7fe54c163950 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c0cc830 con 0x7fe54c0ce980 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.113+0000 7fe55318c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c0cceb0 con 0x7fe54c0ce5a0 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c0716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58300/0 (socket says 172.21.15.72:58300) 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 -- 172.21.15.72:0/875536026 learned_addr learned my addr 172.21.15.72:0/875536026 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe54bfff640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe54c0ce980 0x7fe54c15be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe550f28640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe54c0ce5a0 0x7fe54c13aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 -- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe54c0ce5a0 msgr2=0x7fe54c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe54c0ce5a0 0x7fe54c13aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 -- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe54c0ce980 msgr2=0x7fe54c15be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe54c0ce980 0x7fe54c15be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 -- 172.21.15.72:0/875536026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c162ad0 con 0x7fe54c163950 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe550f28640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe54c0ce5a0 0x7fe54c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe551729640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c0716a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe544009d90 tx=0x7fe544004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544039050 con 0x7fe54c163950 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54403a040 con 0x7fe54c163950 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c162db0 con 0x7fe54c163950 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54403e610 con 0x7fe54c163950 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.114+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c163370 con 0x7fe54c163950 2024-03-06T07:26:44.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.115+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c069ee0 con 0x7fe54c163950 2024-03-06T07:26:44.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.116+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe544047020 con 0x7fe54c163950 2024-03-06T07:26:44.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.117+0000 7fe5497fa640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe53006b4d0 0x7fe53006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.117+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe5440ac5f0 con 0x7fe54c163950 2024-03-06T07:26:44.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.117+0000 7fe550f28640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe53006b4d0 0x7fe53006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.117+0000 7fe550f28640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe53006b4d0 0x7fe53006d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe540000f60 tx=0x7fe540009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.121+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe54404e020 con 0x7fe54c163950 2024-03-06T07:26:44.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.330+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe54c142310 con 0x7fe54c163950 2024-03-06T07:26:44.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.330+0000 7fe5497fa640 1 -- 172.21.15.72:0/875536026 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fe54407b4c0 con 0x7fe54c163950 2024-03-06T07:26:44.332 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:44.332 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:44.333 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:44.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe53006b4d0 msgr2=0x7fe53006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe53006b4d0 0x7fe53006d990 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe540000f60 tx=0x7fe540009040 comp rx=0 tx=0).stop 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 msgr2=0x7fe54c0716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c0716a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe544009d90 tx=0x7fe544004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 shutdown_connections 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe53006b4d0 0x7fe53006d990 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54c163950 0x7fe54c0716a0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe54c0ce980 0x7fe54c15be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 --2- 172.21.15.72:0/875536026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe54c0ce5a0 0x7fe54c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 >> 172.21.15.72:0/875536026 conn(0x7fe54c06b160 msgr2=0x7fe54c1430c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 shutdown_connections 2024-03-06T07:26:44.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.332+0000 7fe55318c640 1 -- 172.21.15.72:0/875536026 wait complete. 2024-03-06T07:26:44.345 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:44.462 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2100,"msgr_send_messages":2100,"msgr_recv_bytes":600112,"msgr_send_bytes":5919342,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.189959802,"msgr_running_send_time":0.092108638,"msgr_running_recv_time":0.033763693,"msgr_running_fast_dispatch_time":0.053941615,"msgr_send_messages_queue_lat":{"avgcount":2099,"sum":1.349713478,"avgtime":0.000643026},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11204,"msgr_send_messages":25612,"msgr_recv_bytes":4003507,"msgr_send_bytes":28412164,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.122365512,"msgr_running_send_time":0.616570953,"msgr_running_recv_time":0.524606542,"msgr_running_fast_dispatch_time":0.053050678,"msgr_send_messages_queue_lat":{"avgcount":25611,"sum":0.927176988,"avgtime":0.000036202},"msgr_handle_ack_lat":{"avgcount":30605,"sum":0.005016650,"avgtime":0.000000163},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":95984},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1548,"msgr_send_messages":1543,"msgr_recv_bytes":861547,"msgr_send_bytes":2425699,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.146261681,"msgr_running_send_time":0.062800815,"msgr_running_recv_time":0.026686291,"msgr_running_fast_dispatch_time":0.048808065,"msgr_send_messages_queue_lat":{"avgcount":1542,"sum":0.597046341,"avgtime":0.000387189},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108752,"msgr_send_encrypted_bytes":9168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":125,"sum":7.755799914,"avgtime":0.062046399}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5377,"sum":1.352303971,"avgtime":0.000251497}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.676088480,"avgtime":0.001509294},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":1601,"inodes_top":0,"inodes_bottom":1601,"inodes_pin_tail":0,"inodes_pinned":1601,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":838,"ceph_cap_op_grant":7592,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3399,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13004,"evex":11413,"evtrm":11413,"ev":1594,"evexg":0,"evexd":11412,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28242440,"wrpos":29201126,"rdpos":4195244,"jlat":{"avgcount":24,"sum":2.504433730,"avgtime":0.104351405},"replayed":3},"mds_mem":{"ino":1603,"ino+":3214,"ino-":1611,"dir":13,"dir+":13,"dir-":0,"dn":1601,"dn+":6412,"dn-":4811,"cap":1,"cap+":3202,"cap-":3201,"rss":116220,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.534121606,"avgtime":0.534121606},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.926065660,"avgtime":0.001851895},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5850,"average_load":5850,"avg_session_uptime":834,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13045105,"buffer_anon_items":3279,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5088678,"mds_co_items":84457,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4977,"op_send_bytes":26846599,"op_resend":0,"op_reply":4977,"op_latency":{"avgcount":4977,"sum":144.477904832,"avgtime":0.029029114},"op_inflight":0,"oplen_avg":{"avgcount":4977,"sum":8270},"op":4977,"op_r":58,"op_w":4919,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1602,"osdop_read":30,"osdop_write":46,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3204,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":52,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14848,"get_sum":4125973,"get_or_fail_fail":0,"get_or_fail_success":14848,"take":0,"take_sum":0,"put":14848,"put_sum":4125973,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4977,"take_sum":27171017,"put":4977,"put_sum":27171017,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4977,"take_sum":4977,"put":4977,"put_sum":4977,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":22,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":0,"max":3758096384,"get_started":0,"get":13004,"get_sum":25005441,"get_or_fail_fail":0,"get_or_fail_success":13004,"take":0,"take_sum":0,"put":24,"put_sum":25005441,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:44.464 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 600112, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 2100, "msgr_running_fast_dispatch_time": 0.053941615, "msgr_running_recv_time": 0.033763693, "msgr_running_send_time": 0.092108638, "msgr_running_total_time": 0.189959802, "msgr_send_bytes": 5919342, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 2100, "msgr_send_messages_queue_lat": { "avgcount": 2099, "avgtime": 0.000643026, "sum": 1.349713478 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30605, "avgtime": 1.63e-07, "sum": 0.00501665 }, "msgr_recv_bytes": 4003507, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11204, "msgr_running_fast_dispatch_time": 0.053050678, "msgr_running_recv_time": 0.524606542, "msgr_running_send_time": 0.616570953, "msgr_running_total_time": 1.122365512, "msgr_send_bytes": 28412164, "msgr_send_encrypted_bytes": 95984, "msgr_send_messages": 25612, "msgr_send_messages_queue_lat": { "avgcount": 25611, "avgtime": 3.6202e-05, "sum": 0.927176988 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 861547, "msgr_recv_encrypted_bytes": 108752, "msgr_recv_messages": 1548, "msgr_running_fast_dispatch_time": 0.048808065, "msgr_running_recv_time": 0.026686291, "msgr_running_send_time": 0.062800815, "msgr_running_total_time": 0.146261681, "msgr_send_bytes": 2425699, "msgr_send_encrypted_bytes": 9168, "msgr_send_messages": 1543, "msgr_send_messages_queue_lat": { "avgcount": 1542, "avgtime": 0.000387189, "sum": 0.597046341 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 125, "avgtime": 0.062046399, "sum": 7.755799914 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5377, "avgtime": 0.000251497, "sum": 1.352303971 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7592, "ceph_cap_op_revoke": 838, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3399, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1601, "inodes_bottom": 1601, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1601, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001509294, "sum": 9.67608848 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1594, "evadd": 13004, "evex": 11413, "evexd": 11412, "evexg": 0, "evlrg": 0, "evtrm": 11413, "expos": 28242440, "jlat": { "avgcount": 24, "avgtime": 0.104351405, "sum": 2.50443373 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 29201126 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1601, "dn+": 6412, "dn-": 4811, "heap": 248164, "ino": 1603, "ino+": 3214, "ino-": 1611, "rss": 116220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.534121606, "sum": 0.534121606 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001851895, "sum": 5.92606566 } }, "mds_sessions": { "average_load": 5850, "avg_session_uptime": 834, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5850 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13045105, "buffer_anon_items": 3279, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5088678, "mds_co_items": 84457, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 52, "op": 4977, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4977, "avgtime": 0.029029114, "sum": 144.477904832 }, "op_pg": 0, "op_r": 58, "op_reply": 4977, "op_resend": 0, "op_rmw": 0, "op_send": 4977, "op_send_bytes": 26846599, "op_w": 4919, "oplen_avg": { "avgcount": 4977, "sum": 8270 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1602, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3204, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 46, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14848, "get_or_fail_fail": 0, "get_or_fail_success": 14848, "get_started": 0, "get_sum": 4125973, "max": 104857600, "put": 14848, "put_sum": 4125973, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4977, "put_sum": 27171017, "take": 4977, "take_sum": 27171017, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4977, "put_sum": 4977, "take": 4977, "take_sum": 4977, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 22, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 13004, "get_or_fail_fail": 0, "get_or_fail_success": 13004, "get_started": 0, "get_sum": 25005441, "max": 3758096384, "put": 24, "put_sum": 25005441, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:44.464 INFO:tasks.cephfs.test_strays:Complete purge in 7 seconds 2024-03-06T07:26:44.465 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:44.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.563+0000 7ff553484640 1 Processor -- start 2024-03-06T07:26:44.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.564+0000 7ff553484640 1 -- start start 2024-03-06T07:26:44.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.564+0000 7ff553484640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff54c0b65e0 0x7ff54c1624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.564+0000 7ff553484640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c0d12d0 0x7ff54c0d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff553484640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff54c0cf2f0 con 0x7ff54c15e310 2024-03-06T07:26:44.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff553484640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff54c0b5140 con 0x7ff54c0b65e0 2024-03-06T07:26:44.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff553484640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c0cf8b0 con 0x7ff54c0d12d0 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff54c0b65e0 0x7ff54c1624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff54c0b65e0 0x7ff54c1624a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58314/0 (socket says 172.21.15.72:58314) 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 -- 172.21.15.72:0/3828380806 learned_addr learned my addr 172.21.15.72:0/3828380806 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff551220640 1 --1- 172.21.15.72:0/3828380806 >> v1:172.21.15.72:6789/0 conn(0x7ff54c15e310 0x7ff54c161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45550/0 (socket says 172.21.15.72:45550) 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 -- 172.21.15.72:0/3828380806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c0d12d0 msgr2=0x7ff54c0d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 --2- 172.21.15.72:0/3828380806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c0d12d0 0x7ff54c0d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 -- 172.21.15.72:0/3828380806 >> v1:172.21.15.72:6789/0 conn(0x7ff54c15e310 legacy=0x7ff54c161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff54bfff640 1 -- 172.21.15.72:0/3828380806 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042313236 0 0) 0x7ff54c0cf2f0 con 0x7ff54c15e310 2024-03-06T07:26:44.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.565+0000 7ff550a1f640 1 -- 172.21.15.72:0/3828380806 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c0cf5d0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff550a1f640 1 --2- 172.21.15.72:0/3828380806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c1624a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff540009870 tx=0x7ff54002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f668dbcee44572 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff54bfff640 1 -- 172.21.15.72:0/3828380806 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff540035020 con 0x7ff54c0b65e0 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff54bfff640 1 -- 172.21.15.72:0/3828380806 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff540031640 con 0x7ff54c0b65e0 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff54bfff640 1 -- 172.21.15.72:0/3828380806 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff54002d040 con 0x7ff54c0b65e0 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 -- 172.21.15.72:0/3828380806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 msgr2=0x7ff54c1624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 --2- 172.21.15.72:0/3828380806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c1624a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7ff540009870 tx=0x7ff54002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 -- 172.21.15.72:0/3828380806 shutdown_connections 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 --2- 172.21.15.72:0/3828380806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c0d12d0 0x7ff54c0d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 --2- 172.21.15.72:0/3828380806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c1624a0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 -- 172.21.15.72:0/3828380806 >> 172.21.15.72:0/3828380806 conn(0x7ff54c09aa40 msgr2=0x7ff54c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.566+0000 7ff553484640 1 -- 172.21.15.72:0/3828380806 shutdown_connections 2024-03-06T07:26:44.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.567+0000 7ff553484640 1 -- 172.21.15.72:0/3828380806 wait complete. 2024-03-06T07:26:44.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.568+0000 7ff553484640 1 Processor -- start 2024-03-06T07:26:44.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.568+0000 7ff553484640 1 -- start start 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.569+0000 7ff553484640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c14eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.569+0000 7ff553484640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0d12d0 0x7ff54c152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff553484640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c15e310 0x7ff54c1f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff553484640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c09b7c0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff553484640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c09b640 con 0x7ff54c0d12d0 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff553484640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff54c1629e0 con 0x7ff54c15e310 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c14eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c14eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58320/0 (socket says 172.21.15.72:58320) 2024-03-06T07:26:44.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 -- 172.21.15.72:0/672647380 learned_addr learned my addr 172.21.15.72:0/672647380 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff550a1f640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0d12d0 0x7ff54c152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551a21640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c15e310 0x7ff54c1f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 -- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c15e310 msgr2=0x7ff54c1f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c15e310 0x7ff54c1f43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 -- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0d12d0 msgr2=0x7ff54c152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0d12d0 0x7ff54c152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 -- 172.21.15.72:0/672647380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff54c1f4b50 con 0x7ff54c0b65e0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551a21640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c15e310 0x7ff54c1f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff551220640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c14eb30 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00c7c0 tx=0x7ff53c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c018070 con 0x7ff54c0b65e0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff53c00a640 con 0x7ff54c0b65e0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff53c00aa30 con 0x7ff54c0b65e0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.570+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff54c1f4e30 con 0x7ff54c0b65e0 2024-03-06T07:26:44.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.571+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff54c1f53f0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.571+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff54c06d9a0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.572+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff53c01b070 con 0x7ff54c0b65e0 2024-03-06T07:26:44.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.572+0000 7ff549ffb640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff53806b4d0 0x7ff53806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:44.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.572+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff53c089930 con 0x7ff54c0b65e0 2024-03-06T07:26:44.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.572+0000 7ff550a1f640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff53806b4d0 0x7ff53806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:44.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.573+0000 7ff550a1f640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff53806b4d0 0x7ff53806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff540036000 tx=0x7ff54003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:44.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.576+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff54c06d9a0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.793+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff54c069ee0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.794+0000 7ff549ffb640 1 -- 172.21.15.72:0/672647380 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7ff53c0586b0 con 0x7ff54c0b65e0 2024-03-06T07:26:44.796 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:44.796 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:44.796 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff53806b4d0 msgr2=0x7ff53806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff53806b4d0 0x7ff53806d990 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff540036000 tx=0x7ff54003c040 comp rx=0 tx=0).stop 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 msgr2=0x7ff54c14eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c14eb30 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7ff53c00c7c0 tx=0x7ff53c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 shutdown_connections 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff53806b4d0 0x7ff53806d990 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff54c15e310 0x7ff54c1f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff54c0d12d0 0x7ff54c152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.796+0000 7ff553484640 1 --2- 172.21.15.72:0/672647380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff54c0b65e0 0x7ff54c14eb30 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:44.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.797+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 >> 172.21.15.72:0/672647380 conn(0x7ff54c09aa40 msgr2=0x7ff54c158f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:44.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.797+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 shutdown_connections 2024-03-06T07:26:44.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:44.797+0000 7ff553484640 1 -- 172.21.15.72:0/672647380 wait complete. 2024-03-06T07:26:44.809 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-03-06T07:26:44.925 INFO:teuthology.orchestra.run.smithi072.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":2100,"msgr_send_messages":2100,"msgr_recv_bytes":600112,"msgr_send_bytes":5919342,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.189959802,"msgr_running_send_time":0.092108638,"msgr_running_recv_time":0.033763693,"msgr_running_fast_dispatch_time":0.053941615,"msgr_send_messages_queue_lat":{"avgcount":2099,"sum":1.349713478,"avgtime":0.000643026},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000245,"avgtime":0.000000122},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":11205,"msgr_send_messages":25613,"msgr_recv_bytes":4003925,"msgr_send_bytes":28413956,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":1.122543323,"msgr_running_send_time":0.616674096,"msgr_running_recv_time":0.524632060,"msgr_running_fast_dispatch_time":0.053090613,"msgr_send_messages_queue_lat":{"avgcount":25612,"sum":0.927201425,"avgtime":0.000036201},"msgr_handle_ack_lat":{"avgcount":30606,"sum":0.005016774,"avgtime":0.000000163},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":97776},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1548,"msgr_send_messages":1543,"msgr_recv_bytes":861547,"msgr_send_bytes":2425699,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.146261681,"msgr_running_send_time":0.062800815,"msgr_running_recv_time":0.026686291,"msgr_running_fast_dispatch_time":0.048808065,"msgr_send_messages_queue_lat":{"avgcount":1542,"sum":0.597046341,"avgtime":0.000387189},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":108752,"msgr_send_encrypted_bytes":9168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":125,"sum":7.755799914,"avgtime":0.062046399}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5377,"sum":1.352303971,"avgtime":0.000251497}},"mds":{"request":6413,"reply":6411,"reply_latency":{"avgcount":6411,"sum":9.676088480,"avgtime":0.001509294},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":1601,"inodes_top":0,"inodes_bottom":1601,"inodes_pin_tail":0,"inodes_pinned":1601,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":6419,"traverse_hit":6418,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":838,"ceph_cap_op_grant":7592,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3399,"handle_client_caps_dirty":3000,"handle_client_cap_release":9,"process_request_cap_release":9603},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":13004,"evex":11413,"evtrm":11413,"ev":1594,"evexg":0,"evexd":11412,"segadd":16,"segex":16,"segtrm":16,"seg":2,"segmjr":1,"segexg":0,"segexd":15,"expos":28242440,"wrpos":29201126,"rdpos":4195244,"jlat":{"avgcount":24,"sum":2.504433730,"avgtime":0.104351405},"replayed":3},"mds_mem":{"ino":1603,"ino+":3214,"ino-":1611,"dir":13,"dir+":13,"dir-":0,"dn":1601,"dn+":6412,"dn-":4811,"cap":1,"cap+":3202,"cap-":3201,"rss":116220,"heap":248164},"mds_server":{"dispatch_client_request":6419,"dispatch_server_request":0,"handle_client_request":6413,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":2.870230849,"avgtime":0.000896947},"req_getattr_latency":{"avgcount":6,"sum":0.129824902,"avgtime":0.021637483},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003005074,"avgtime":0.003005074},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":3,"sum":0.212840389,"avgtime":0.070946796},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.534121606,"avgtime":0.534121606},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":5.926065660,"avgtime":0.001851895},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5850,"average_load":5850,"avg_session_uptime":834,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13045105,"buffer_anon_items":3279,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5800,"osdmap_items":94,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5088678,"mds_co_items":84457,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":4977,"op_send_bytes":26846599,"op_resend":0,"op_reply":4977,"op_latency":{"avgcount":4977,"sum":144.477904832,"avgtime":0.029029114},"op_inflight":0,"oplen_avg":{"avgcount":4977,"sum":8270},"op":4977,"op_r":58,"op_w":4919,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":1602,"osdop_read":30,"osdop_write":46,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3222,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":3204,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":52,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14849,"get_sum":4126301,"get_or_fail_fail":0,"get_or_fail_success":14849,"take":0,"take_sum":0,"put":14849,"put_sum":4126301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4977,"take_sum":27171017,"put":4977,"put_sum":27171017,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":4977,"take_sum":4977,"put":4977,"put_sum":4977,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":22,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x55fbc5198a00":{"val":0,"max":3758096384,"get_started":0,"get":13004,"get_sum":25005441,"get_or_fail_fail":0,"get_or_fail_success":13004,"take":0,"take_sum":0,"put":24,"put_sum":25005441,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-03-06T07:26:44.926 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.22e-07, "sum": 2.45e-07 }, "msgr_recv_bytes": 600112, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 2100, "msgr_running_fast_dispatch_time": 0.053941615, "msgr_running_recv_time": 0.033763693, "msgr_running_send_time": 0.092108638, "msgr_running_total_time": 0.189959802, "msgr_send_bytes": 5919342, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 2100, "msgr_send_messages_queue_lat": { "avgcount": 2099, "avgtime": 0.000643026, "sum": 1.349713478 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 30606, "avgtime": 1.63e-07, "sum": 0.005016774 }, "msgr_recv_bytes": 4003925, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 11205, "msgr_running_fast_dispatch_time": 0.053090613, "msgr_running_recv_time": 0.52463206, "msgr_running_send_time": 0.616674096, "msgr_running_total_time": 1.122543323, "msgr_send_bytes": 28413956, "msgr_send_encrypted_bytes": 97776, "msgr_send_messages": 25613, "msgr_send_messages_queue_lat": { "avgcount": 25612, "avgtime": 3.6201e-05, "sum": 0.927201425 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 861547, "msgr_recv_encrypted_bytes": 108752, "msgr_recv_messages": 1548, "msgr_running_fast_dispatch_time": 0.048808065, "msgr_running_recv_time": 0.026686291, "msgr_running_send_time": 0.062800815, "msgr_running_total_time": 0.146261681, "msgr_send_bytes": 2425699, "msgr_send_encrypted_bytes": 9168, "msgr_send_messages": 1543, "msgr_send_messages_queue_lat": { "avgcount": 1542, "avgtime": 0.000387189, "sum": 0.597046341 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 125, "avgtime": 0.062046399, "sum": 7.755799914 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5377, "avgtime": 0.000251497, "sum": 1.352303971 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7592, "ceph_cap_op_revoke": 838, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 9, "handle_client_caps": 3399, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 1601, "inodes_bottom": 1601, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 1601, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 9603, "q": 0, "reply": 6411, "reply_latency": { "avgcount": 6411, "avgtime": 0.001509294, "sum": 9.67608848 }, "request": 6413, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6419, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6418, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 1594, "evadd": 13004, "evex": 11413, "evexd": 11412, "evexg": 0, "evlrg": 0, "evtrm": 11413, "expos": 28242440, "jlat": { "avgcount": 24, "avgtime": 0.104351405, "sum": 2.50443373 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 16, "segex": 16, "segexd": 15, "segexg": 0, "segmjr": 1, "segtrm": 16, "wrpos": 29201126 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 13, "dir+": 13, "dir-": 0, "dn": 1601, "dn+": 6412, "dn-": 4811, "heap": 248164, "ino": 1603, "ino+": 3214, "ino-": 1611, "rss": 116220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6419, "dispatch_server_request": 0, "handle_client_request": 6413, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.000896947, "sum": 2.870230849 }, "req_getattr_latency": { "avgcount": 6, "avgtime": 0.021637483, "sum": 0.129824902 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003005074, "sum": 0.003005074 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 3, "avgtime": 0.070946796, "sum": 0.212840389 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.534121606, "sum": 0.534121606 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001851895, "sum": 5.92606566 } }, "mds_sessions": { "average_load": 5850, "avg_session_uptime": 834, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5850 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13045105, "buffer_anon_items": 3279, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 5088678, "mds_co_items": 84457, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5800, "osdmap_items": 94, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 52, "op": 4977, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 4977, "avgtime": 0.029029114, "sum": 144.477904832 }, "op_pg": 0, "op_r": 58, "op_reply": 4977, "op_resend": 0, "op_rmw": 0, "op_send": 4977, "op_send_bytes": 26846599, "op_w": 4919, "oplen_avg": { "avgcount": 4977, "sum": 8270 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 1602, "osdop_delete": 3222, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 30, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 3204, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 46, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14849, "get_or_fail_fail": 0, "get_or_fail_success": 14849, "get_started": 0, "get_sum": 4126301, "max": 104857600, "put": 14849, "put_sum": 4126301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 4977, "put_sum": 27171017, "take": 4977, "take_sum": 27171017, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 4977, "put_sum": 4977, "take": 4977, "take_sum": 4977, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 22, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x55fbc5198a00": { "get": 13004, "get_or_fail_fail": 0, "get_or_fail_success": 13004, "get_started": 0, "get_sum": 25005441, "max": 3758096384, "put": 24, "put_sum": 25005441, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-03-06T07:26:44.927 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:44.927 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:44.965 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:45.017 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:45.017 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:26:45.017 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:45.033 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:45.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.171+0000 7f1b5a29a640 1 Processor -- start 2024-03-06T07:26:45.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.172+0000 7f1b5a29a640 1 -- start start 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b5a29a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b540d0950 0x7f1b540d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b5a29a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b540b94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b5a29a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b540d1ea0 con 0x7f1b5415e150 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b5a29a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b540d12f0 con 0x7f1b540d0950 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b5a29a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540d2ab0 con 0x7f1b540b9060 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b540d0950 0x7f1b540d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b540d0950 0x7f1b540d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58328/0 (socket says 172.21.15.72:58328) 2024-03-06T07:26:45.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 -- 172.21.15.72:0/3439230015 learned_addr learned my addr 172.21.15.72:0/3439230015 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 -- 172.21.15.72:0/3439230015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 msgr2=0x7f1b540b94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 --2- 172.21.15.72:0/3439230015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b540b94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 -- 172.21.15.72:0/3439230015 >> v1:172.21.15.146:6789/0 conn(0x7f1b5415e150 legacy=0x7f1b540cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.173+0000 7f1b537fe640 1 -- 172.21.15.72:0/3439230015 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b540d2180 con 0x7f1b540d0950 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.174+0000 7f1b537fe640 1 --2- 172.21.15.72:0/3439230015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540d0db0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009870 tx=0x7f1b4402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48707e040101ca0f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.174+0000 7f1b52ffd640 1 -- 172.21.15.72:0/3439230015 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440317c0 con 0x7f1b540d0950 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.174+0000 7f1b52ffd640 1 -- 172.21.15.72:0/3439230015 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44037070 con 0x7f1b540d0950 2024-03-06T07:26:45.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.174+0000 7f1b52ffd640 1 -- 172.21.15.72:0/3439230015 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44031b80 con 0x7f1b540d0950 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 -- 172.21.15.72:0/3439230015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 msgr2=0x7f1b540d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/3439230015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540d0db0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009870 tx=0x7f1b4402b860 comp rx=0 tx=0).stop 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 -- 172.21.15.72:0/3439230015 shutdown_connections 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/3439230015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b540b94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/3439230015 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540d0db0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 -- 172.21.15.72:0/3439230015 >> 172.21.15.72:0/3439230015 conn(0x7f1b5409d230 msgr2=0x7f1b540d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 -- 172.21.15.72:0/3439230015 shutdown_connections 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.175+0000 7f1b5a29a640 1 -- 172.21.15.72:0/3439230015 wait complete. 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.176+0000 7f1b5a29a640 1 Processor -- start 2024-03-06T07:26:45.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.176+0000 7f1b5a29a640 1 -- start start 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b5a29a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b5415a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b5a29a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b5a29a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b5415e150 0x7f1b540ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b5a29a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540d2f40 con 0x7f1b540d0950 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b5a29a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540d2740 con 0x7f1b5415e150 2024-03-06T07:26:45.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b5a29a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b540d1b30 con 0x7f1b540b9060 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58344/0 (socket says 172.21.15.72:58344) 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 -- 172.21.15.72:0/542935683 learned_addr learned my addr 172.21.15.72:0/542935683 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b58837640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b5415e150 0x7f1b540ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b53fff640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b5415a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 -- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 msgr2=0x7f1b5415a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b5415a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 -- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b5415e150 msgr2=0x7f1b540ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b5415e150 0x7f1b540ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 -- 172.21.15.72:0/542935683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5415aa10 con 0x7f1b540d0950 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b53fff640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b5415a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.177+0000 7f1b537fe640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540cca90 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009870 tx=0x7f1b440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.178+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44032860 con 0x7f1b540d0950 2024-03-06T07:26:45.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.178+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b44032a00 con 0x7f1b540d0950 2024-03-06T07:26:45.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.178+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b540cd390 con 0x7f1b540d0950 2024-03-06T07:26:45.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.178+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44032df0 con 0x7f1b540d0950 2024-03-06T07:26:45.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.178+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b540c7920 con 0x7f1b540d0950 2024-03-06T07:26:45.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.179+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b54069ee0 con 0x7f1b540d0950 2024-03-06T07:26:45.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.179+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b4404c060 con 0x7f1b540d0950 2024-03-06T07:26:45.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.180+0000 7f1b50ff9640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b3406b4d0 0x7f1b3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.180+0000 7f1b53fff640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b3406b4d0 0x7f1b3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.180+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1b440abb90 con 0x7f1b540d0950 2024-03-06T07:26:45.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.180+0000 7f1b53fff640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b3406b4d0 0x7f1b3406d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00a810 tx=0x7f1b3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.184+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b5415aa10 con 0x7f1b540d0950 2024-03-06T07:26:45.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.357+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b540d4670 con 0x7f1b540d0950 2024-03-06T07:26:45.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.358+0000 7f1b50ff9640 1 -- 172.21.15.72:0/542935683 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17007 (secure 0 0 0) 0x7f1b44040070 con 0x7f1b540d0950 2024-03-06T07:26:45.360 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:45.360 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":98,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:19.829591+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6835/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6836/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6835/209452807":"2024-03-07T07:26:13.880837+0000","172.21.15.72:6834/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6833/209452807":"2024-03-07T07:26:13.880837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b3406b4d0 msgr2=0x7f1b3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b3406b4d0 0x7f1b3406d990 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00a810 tx=0x7f1b3c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 msgr2=0x7f1b540cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540cca90 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009870 tx=0x7f1b440046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 shutdown_connections 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1b3406b4d0 0x7f1b3406d990 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b5415e150 0x7f1b540ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b540d0950 0x7f1b540cca90 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 --2- 172.21.15.72:0/542935683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b540b9060 0x7f1b5415a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 >> 172.21.15.72:0/542935683 conn(0x7f1b5409d230 msgr2=0x7f1b5406cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 shutdown_connections 2024-03-06T07:26:45.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.360+0000 7f1b5a29a640 1 -- 172.21.15.72:0/542935683 wait complete. 2024-03-06T07:26:45.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:04.296462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:05.055015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:45.373 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:45.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.474+0000 7f424029a640 1 Processor -- start 2024-03-06T07:26:45.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.474+0000 7f424029a640 1 -- start start 2024-03-06T07:26:45.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f424029a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f423815e150 0x7f42380cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f424029a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380cfba0 0x7f42380d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f424029a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42380d1ea0 con 0x7f42380d0e90 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f424029a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42380d2ab0 con 0x7f423815e150 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f424029a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42380d2180 con 0x7f42380cfba0 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f423e837640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f42380d0e90 0x7f42380b9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45562/0 (socket says 172.21.15.72:45562) 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.475+0000 7f423e837640 1 -- 172.21.15.72:0/2029787730 learned_addr learned my addr 172.21.15.72:0/2029787730 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423e036640 1 --2- 172.21.15.72:0/2029787730 >> v2:172.21.15.146:3300/0 conn(0x7f423815e150 0x7f42380cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423d835640 1 --2- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380cfba0 0x7f42380d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423d034640 1 -- 172.21.15.72:0/2029787730 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082797209 0 0) 0x7f42380d1ea0 con 0x7f42380d0e90 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423d034640 1 -- 172.21.15.72:0/2029787730 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220003520 con 0x7f42380d0e90 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423d034640 1 -- 172.21.15.72:0/2029787730 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 197157892 0 0) 0x7f4220003520 con 0x7f42380d0e90 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423e036640 1 -- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380cfba0 msgr2=0x7f42380d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423e036640 1 --2- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380cfba0 0x7f42380d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423e036640 1 -- 172.21.15.72:0/2029787730 >> v1:172.21.15.72:6789/0 conn(0x7f42380d0e90 legacy=0x7f42380b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.476+0000 7f423e036640 1 -- 172.21.15.72:0/2029787730 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380d2460 con 0x7f423815e150 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.477+0000 7f423e036640 1 --2- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f423815e150 0x7f42380cf660 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f42280097a0 tx=0x7f422802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=464fc5f0255a5618 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.477+0000 7f423d034640 1 -- 172.21.15.72:0/2029787730 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228037070 con 0x7f423815e150 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.477+0000 7f423d034640 1 -- 172.21.15.72:0/2029787730 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228030510 con 0x7f423815e150 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.477+0000 7f423d034640 1 -- 172.21.15.72:0/2029787730 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42300019f0 con 0x7f423815e150 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 -- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f423815e150 msgr2=0x7f42380cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 --2- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f423815e150 0x7f42380cf660 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f42280097a0 tx=0x7f422802b710 comp rx=0 tx=0).stop 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 -- 172.21.15.72:0/2029787730 shutdown_connections 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 --2- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380cfba0 0x7f42380d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 --2- 172.21.15.72:0/2029787730 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f423815e150 0x7f42380cf660 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 -- 172.21.15.72:0/2029787730 >> 172.21.15.72:0/2029787730 conn(0x7f423809d230 msgr2=0x7f42380d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 -- 172.21.15.72:0/2029787730 shutdown_connections 2024-03-06T07:26:45.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.478+0000 7f424029a640 1 -- 172.21.15.72:0/2029787730 wait complete. 2024-03-06T07:26:45.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.479+0000 7f424029a640 1 Processor -- start 2024-03-06T07:26:45.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.479+0000 7f424029a640 1 -- start start 2024-03-06T07:26:45.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42380cfba0 0x7f42380b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380d0e90 0x7f42381f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 0x7f42380c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380d1b30 con 0x7f423815e150 2024-03-06T07:26:45.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42380d2f40 con 0x7f42380cfba0 2024-03-06T07:26:45.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f423815e530 con 0x7f42380d0e90 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 0x7f42380c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 0x7f42380c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58356/0 (socket says 172.21.15.72:58356) 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 -- 172.21.15.72:0/912547099 learned_addr learned my addr 172.21.15.72:0/912547099 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e036640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42380cfba0 0x7f42380b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423d835640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380d0e90 0x7f42381f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 -- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380d0e90 msgr2=0x7f42381f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380d0e90 0x7f42381f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 -- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42380cfba0 msgr2=0x7f42380b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42380cfba0 0x7f42380b6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 -- 172.21.15.72:0/912547099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42380b9540 con 0x7f423815e150 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e036640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42380cfba0 0x7f42380b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f423e837640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 0x7f42380c9990 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f4230001890 tx=0x7f423000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230012400 con 0x7f423815e150 2024-03-06T07:26:45.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42300125c0 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380b9820 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42300129b0 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.480+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42380c9470 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.481+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42380c9670 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.482+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4230012b50 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.482+0000 7f4236ffd640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f422406b500 0x7f422406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.482+0000 7f423e036640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f422406b500 0x7f422406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.482+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f423008abd0 con 0x7f423815e150 2024-03-06T07:26:45.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.483+0000 7f423e036640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f422406b500 0x7f422406d9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4228034000 tx=0x7f4228008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.486+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42380c9670 con 0x7f423815e150 2024-03-06T07:26:45.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.664+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42380d3ee0 con 0x7f423815e150 2024-03-06T07:26:45.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.664+0000 7f4236ffd640 1 -- 172.21.15.72:0/912547099 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4230059950 con 0x7f423815e150 2024-03-06T07:26:45.666 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:26:45.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f422406b500 msgr2=0x7f422406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f422406b500 0x7f422406d9c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f4228034000 tx=0x7f4228008030 comp rx=0 tx=0).stop 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 msgr2=0x7f42380c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 0x7f42380c9990 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f4230001890 tx=0x7f423000a430 comp rx=0 tx=0).stop 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 shutdown_connections 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f422406b500 0x7f422406d9c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f423815e150 0x7f42380c9990 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42380d0e90 0x7f42381f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 --2- 172.21.15.72:0/912547099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42380cfba0 0x7f42380b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 >> 172.21.15.72:0/912547099 conn(0x7f423809d230 msgr2=0x7f42380cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 shutdown_connections 2024-03-06T07:26:45.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.666+0000 7f424029a640 1 -- 172.21.15.72:0/912547099 wait complete. 2024-03-06T07:26:45.678 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:45.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.782+0000 7f5dc644e640 1 Processor -- start 2024-03-06T07:26:45.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.782+0000 7f5dc644e640 1 -- start start 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dc644e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5dc00b5010 0x7f5dc00d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dc644e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00d25a0 0x7f5dc00d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dc644e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc0161e60 con 0x7f5dc0161a80 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dc644e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc00b1250 con 0x7f5dc00b5010 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dc644e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00b3e30 con 0x7f5dc00d25a0 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5dc00b5010 0x7f5dc00d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5dc00b5010 0x7f5dc00d1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58362/0 (socket says 172.21.15.72:58362) 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/1717387056 learned_addr learned my addr 172.21.15.72:0/1717387056 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbffff640 1 --1- 172.21.15.72:0/1717387056 >> v1:172.21.15.72:6789/0 conn(0x7f5dc0161a80 0x7f5dc00b0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45572/0 (socket says 172.21.15.72:45572) 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dc49eb640 1 --2- 172.21.15.72:0/1717387056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00d25a0 0x7f5dc00d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/1717387056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00d25a0 msgr2=0x7f5dc00d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:45.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 --2- 172.21.15.72:0/1717387056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00d25a0 0x7f5dc00d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/1717387056 >> v1:172.21.15.72:6789/0 conn(0x7f5dc0161a80 legacy=0x7f5dc00b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.783+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/1717387056 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00d1470 con 0x7f5dc00b5010 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dbf7fe640 1 --2- 172.21.15.72:0/1717387056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00b5010 0x7f5dc00d1da0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009870 tx=0x7f5db402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf58d3e2641d8c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dbeffd640 1 -- 172.21.15.72:0/1717387056 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4035020 con 0x7f5dc00b5010 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dbeffd640 1 -- 172.21.15.72:0/1717387056 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db4031640 con 0x7f5dc00b5010 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dbeffd640 1 -- 172.21.15.72:0/1717387056 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db402d040 con 0x7f5dc00b5010 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 -- 172.21.15.72:0/1717387056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00b5010 msgr2=0x7f5dc00d1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 --2- 172.21.15.72:0/1717387056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00b5010 0x7f5dc00d1da0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f5db4009870 tx=0x7f5db402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 -- 172.21.15.72:0/1717387056 shutdown_connections 2024-03-06T07:26:45.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 --2- 172.21.15.72:0/1717387056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00d25a0 0x7f5dc00d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 --2- 172.21.15.72:0/1717387056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00b5010 0x7f5dc00d1da0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 -- 172.21.15.72:0/1717387056 >> 172.21.15.72:0/1717387056 conn(0x7f5dc00d3fc0 msgr2=0x7f5dc00d43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 -- 172.21.15.72:0/1717387056 shutdown_connections 2024-03-06T07:26:45.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.785+0000 7f5dc644e640 1 -- 172.21.15.72:0/1717387056 wait complete. 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.786+0000 7f5dc644e640 1 Processor -- start 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 -- start start 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00b5010 0x7f5dc015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 0x7f5dc00ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dc0161a80 0x7f5dc00cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00d1090 con 0x7f5dc00d25a0 2024-03-06T07:26:45.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00b3ac0 con 0x7f5dc0161a80 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dc644e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00d1210 con 0x7f5dc00b5010 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dbf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 0x7f5dc00ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dbf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 0x7f5dc00ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58366/0 (socket says 172.21.15.72:58366) 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.787+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/2005313863 learned_addr learned my addr 172.21.15.72:0/2005313863 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbffff640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00b5010 0x7f5dc015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dc49eb640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dc0161a80 0x7f5dc00cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00b5010 msgr2=0x7f5dc015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbf7fe640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00b5010 0x7f5dc015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dc0161a80 msgr2=0x7f5dc00cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbf7fe640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dc0161a80 0x7f5dc00cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbf7fe640 1 -- 172.21.15.72:0/2005313863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00cbfc0 con 0x7f5dc00d25a0 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbf7fe640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 0x7f5dc00ccba0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f5db4000c00 tx=0x7f5db40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4040070 con 0x7f5dc00d25a0 2024-03-06T07:26:45.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db40399f0 con 0x7f5dc00d25a0 2024-03-06T07:26:45.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc00b6be0 con 0x7f5dc00d25a0 2024-03-06T07:26:45.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db4035040 con 0x7f5dc00d25a0 2024-03-06T07:26:45.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.788+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc00b71a0 con 0x7f5dc00d25a0 2024-03-06T07:26:45.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.789+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc0068050 con 0x7f5dc00d25a0 2024-03-06T07:26:45.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.790+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5db404b060 con 0x7f5dc00d25a0 2024-03-06T07:26:45.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.790+0000 7f5dbcff9640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5dac06b4d0 0x7f5dac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:45.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.790+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5db40abc80 con 0x7f5dc00d25a0 2024-03-06T07:26:45.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.790+0000 7f5dbffff640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5dac06b4d0 0x7f5dac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:45.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.790+0000 7f5dbffff640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5dac06b4d0 0x7f5dac06d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5db0000f60 tx=0x7f5db0007ce0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:45.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.794+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5db407aa00 con 0x7f5dc00d25a0 2024-03-06T07:26:45.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.971+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5dc00d2380 con 0x7f5dc00d25a0 2024-03-06T07:26:45.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.972+0000 7f5dbcff9640 1 -- 172.21.15.72:0/2005313863 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5db407a300 con 0x7f5dc00d25a0 2024-03-06T07:26:45.974 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5dac06b4d0 msgr2=0x7f5dac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5dac06b4d0 0x7f5dac06d990 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f5db0000f60 tx=0x7f5db0007ce0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 msgr2=0x7f5dc00ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 0x7f5dc00ccba0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f5db4000c00 tx=0x7f5db40046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 shutdown_connections 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5dac06b4d0 0x7f5dac06d990 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dc0161a80 0x7f5dc00cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dc00d25a0 0x7f5dc00ccba0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 --2- 172.21.15.72:0/2005313863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dc00b5010 0x7f5dc015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 >> 172.21.15.72:0/2005313863 conn(0x7f5dc00d3fc0 msgr2=0x7f5dc00d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 shutdown_connections 2024-03-06T07:26:45.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:45.974+0000 7f5dc644e640 1 -- 172.21.15.72:0/2005313863 wait complete. 2024-03-06T07:26:45.990 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:46.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.116+0000 7ffbda29a640 1 Processor -- start 2024-03-06T07:26:46.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.116+0000 7ffbda29a640 1 -- start start 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbda29a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffbd415f5e0 0x7ffbd40b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbda29a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffbd40b16b0 0x7ffbd40ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbda29a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d2870 0x7ffbd40cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbda29a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd40cf920 con 0x7ffbd415f5e0 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbda29a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd40cd4d0 con 0x7ffbd40b16b0 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbda29a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40cd7b0 con 0x7ffbd40d2870 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbd3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffbd415f5e0 0x7ffbd40b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbd3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffbd415f5e0 0x7ffbd40b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58374/0 (socket says 172.21.15.72:58374) 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.117+0000 7ffbd3fff640 1 -- 172.21.15.72:0/1207660958 learned_addr learned my addr 172.21.15.72:0/1207660958 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:46.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd37fe640 1 --2- 172.21.15.72:0/1207660958 >> v2:172.21.15.146:3300/0 conn(0x7ffbd40b16b0 0x7ffbd40ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd8837640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d2870 0x7ffbd40cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd3fff640 1 -- 172.21.15.72:0/1207660958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d2870 msgr2=0x7ffbd40cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d2870 0x7ffbd40cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd3fff640 1 -- 172.21.15.72:0/1207660958 >> v2:172.21.15.146:3300/0 conn(0x7ffbd40b16b0 msgr2=0x7ffbd40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1207660958 >> v2:172.21.15.146:3300/0 conn(0x7ffbd40b16b0 0x7ffbd40ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd3fff640 1 -- 172.21.15.72:0/1207660958 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd40cda90 con 0x7ffbd415f5e0 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd37fe640 1 --2- 172.21.15.72:0/1207660958 >> v2:172.21.15.146:3300/0 conn(0x7ffbd40b16b0 0x7ffbd40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.118+0000 7ffbd8837640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d2870 0x7ffbd40cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd415f5e0 0x7ffbd40b2bf0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4071dc0 tx=0x7ffbc402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839d489f07a2ad82 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbd2ffd640 1 -- 172.21.15.72:0/1207660958 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc402e070 con 0x7ffbd415f5e0 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbd2ffd640 1 -- 172.21.15.72:0/1207660958 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc4037070 con 0x7ffbd415f5e0 2024-03-06T07:26:46.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbd2ffd640 1 -- 172.21.15.72:0/1207660958 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4031a00 con 0x7ffbd415f5e0 2024-03-06T07:26:46.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 -- 172.21.15.72:0/1207660958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd415f5e0 msgr2=0x7ffbd40b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd415f5e0 0x7ffbd40b2bf0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4071dc0 tx=0x7ffbc402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 -- 172.21.15.72:0/1207660958 shutdown_connections 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40d2870 0x7ffbd40cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1207660958 >> v2:172.21.15.146:3300/0 conn(0x7ffbd40b16b0 0x7ffbd40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1207660958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd415f5e0 0x7ffbd40b2bf0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.119+0000 7ffbda29a640 1 -- 172.21.15.72:0/1207660958 >> 172.21.15.72:0/1207660958 conn(0x7ffbd40d1d50 msgr2=0x7ffbd40d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.120+0000 7ffbda29a640 1 -- 172.21.15.72:0/1207660958 shutdown_connections 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.120+0000 7ffbda29a640 1 -- 172.21.15.72:0/1207660958 wait complete. 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.121+0000 7ffbda29a640 1 Processor -- start 2024-03-06T07:26:46.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.122+0000 7ffbda29a640 1 -- start start 2024-03-06T07:26:46.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40b16b0 0x7ffbd40aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 0x7ffbd414f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd415f5e0 0x7ffbd414f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40cf270 con 0x7ffbd40d2870 2024-03-06T07:26:46.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd40cf0f0 con 0x7ffbd415f5e0 2024-03-06T07:26:46.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd415fcd0 con 0x7ffbd40b16b0 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 0x7ffbd414f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 0x7ffbd414f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58380/0 (socket says 172.21.15.72:58380) 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 -- 172.21.15.72:0/1489232623 learned_addr learned my addr 172.21.15.72:0/1489232623 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40b16b0 0x7ffbd40aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd8837640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd415f5e0 0x7ffbd414f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 -- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40b16b0 msgr2=0x7ffbd40aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40b16b0 0x7ffbd40aecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 -- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd415f5e0 msgr2=0x7ffbd414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd415f5e0 0x7ffbd414f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 -- 172.21.15.72:0/1489232623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd41f5220 con 0x7ffbd40d2870 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40b16b0 0x7ffbd40aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd8837640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd415f5e0 0x7ffbd414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:46.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd37fe640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 0x7ffbd414f170 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c7c0 tx=0x7ffbc800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8015070 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc8007820 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.123+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd41f5500 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.124+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8007bf0 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.124+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd41f5ac0 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.125+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd4068050 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.125+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffbc8018070 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.125+0000 7ffbd0ff9640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffbc006b4d0 0x7ffbc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.125+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ffbc808a150 con 0x7ffbd40d2870 2024-03-06T07:26:46.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.125+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffbc006b4d0 0x7ffbc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.126+0000 7ffbd3fff640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffbc006b4d0 0x7ffbc006d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4036000 tx=0x7ffbc403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.129+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffbc8058f60 con 0x7ffbd40d2870 2024-03-06T07:26:46.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.305+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbd40b7cd0 con 0x7ffbd40d2870 2024-03-06T07:26:46.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.306+0000 7ffbd0ff9640 1 -- 172.21.15.72:0/1489232623 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffbc8058860 con 0x7ffbd40d2870 2024-03-06T07:26:46.308 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.307+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffbc006b4d0 msgr2=0x7ffbc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.307+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffbc006b4d0 0x7ffbc006d990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4036000 tx=0x7ffbc403d040 comp rx=0 tx=0).stop 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.307+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 msgr2=0x7ffbd414f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.307+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 0x7ffbd414f170 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800c7c0 tx=0x7ffbc800cc80 comp rx=0 tx=0).stop 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 shutdown_connections 2024-03-06T07:26:46.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffbc006b4d0 0x7ffbc006d990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbd415f5e0 0x7ffbd414f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffbd40d2870 0x7ffbd414f170 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 --2- 172.21.15.72:0/1489232623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbd40b16b0 0x7ffbd40aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 >> 172.21.15.72:0/1489232623 conn(0x7ffbd40d1d50 msgr2=0x7ffbd40ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 shutdown_connections 2024-03-06T07:26:46.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.308+0000 7ffbda29a640 1 -- 172.21.15.72:0/1489232623 wait complete. 2024-03-06T07:26:46.321 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:26:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.420+0000 7f90414d6640 1 Processor -- start 2024-03-06T07:26:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.420+0000 7f90414d6640 1 -- start start 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f90414d6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 0x7f903c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f90414d6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f903c0ae350 0x7f903c0ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f90414d6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903c0b37b0 con 0x7f903c15e370 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f90414d6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903c15e750 con 0x7f903c0ae350 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f90414d6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0b31f0 con 0x7f903c0b4d20 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f903affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f903c15e370 0x7f903c06e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45574/0 (socket says 172.21.15.72:45574) 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f903affd640 1 -- 172.21.15.72:0/3650873885 learned_addr learned my addr 172.21.15.72:0/3650873885 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:46.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f903a7fc640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 0x7f903c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018586013 0 0) 0x7f903c0b37b0 con 0x7f903c15e370 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f903b7fe640 1 --2- 172.21.15.72:0/3650873885 >> v2:172.21.15.146:3300/0 conn(0x7f903c0ae350 0x7f903c0ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.421+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9024003520 con 0x7f903c15e370 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1134815540 0 0) 0x7f9024003520 con 0x7f903c15e370 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 msgr2=0x7f903c06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 0x7f903c06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 >> v2:172.21.15.146:3300/0 conn(0x7f903c0ae350 msgr2=0x7f903c0ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 --2- 172.21.15.72:0/3650873885 >> v2:172.21.15.146:3300/0 conn(0x7f903c0ae350 0x7f903c0ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c0b34d0 con 0x7f903c15e370 2024-03-06T07:26:46.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9030001670 con 0x7f903c15e370 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 >> v1:172.21.15.72:6789/0 conn(0x7f903c15e370 legacy=0x7f903c06e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90240039b0 0x7f9024003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024004800 0x7f9024004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 0x7f90240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c15e750 con 0x7f9024004800 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0b31f0 con 0x7f90240039b0 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f902400b9c0 con 0x7f903c0b4d20 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903a7fc640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024004800 0x7f9024004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903affd640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90240039b0 0x7f9024003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903b7fe640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 0x7f90240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903a7fc640 1 -- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 msgr2=0x7f90240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903a7fc640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f903c0b4d20 0x7f90240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903a7fc640 1 -- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90240039b0 msgr2=0x7f9024003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:46.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903a7fc640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90240039b0 0x7f9024003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.422+0000 7f903a7fc640 1 -- 172.21.15.72:0/3650873885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024003520 con 0x7f9024004800 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.423+0000 7f903a7fc640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024004800 0x7f9024004c80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f902c009af0 tx=0x7f9024005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66b594ae9bc9d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.423+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c039070 con 0x7f9024004800 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.423+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c02c6b0 con 0x7f9024004800 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.423+0000 7f9039ffb640 1 -- 172.21.15.72:0/3650873885 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c009690 con 0x7f9024004800 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f90414d6640 1 -- 172.21.15.72:0/3650873885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024004800 msgr2=0x7f9024004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f90414d6640 1 --2- 172.21.15.72:0/3650873885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9024004800 0x7f9024004c80 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f902c009af0 tx=0x7f9024005830 comp rx=0 tx=0).stop 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f903b7fe640 1 -- 172.21.15.72:0/3650873885 reap_dead start 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f90414d6640 1 -- 172.21.15.72:0/3650873885 shutdown_connections 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f90414d6640 1 -- 172.21.15.72:0/3650873885 >> 172.21.15.72:0/3650873885 conn(0x7f903c0b1dc0 msgr2=0x7f903c0b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f90414d6640 1 -- 172.21.15.72:0/3650873885 shutdown_connections 2024-03-06T07:26:46.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.424+0000 7f90414d6640 1 -- 172.21.15.72:0/3650873885 wait complete. 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.425+0000 7f90414d6640 1 Processor -- start 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.425+0000 7f90414d6640 1 -- start start 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f90414d6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9024004800 0x7f903c1463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f90414d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 0x7f903c1501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f90414d6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903c09b790 0x7f903c09bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f90414d6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0b29c0 con 0x7f903c14fd30 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f90414d6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0ade30 con 0x7f903c09b790 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f90414d6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903c0b2b40 con 0x7f9024004800 2024-03-06T07:26:46.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 0x7f903c1501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 0x7f903c1501b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58402/0 (socket says 172.21.15.72:58402) 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 -- 172.21.15.72:0/170021720 learned_addr learned my addr 172.21.15.72:0/170021720 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903b7fe640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903c09b790 0x7f903c09bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903affd640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9024004800 0x7f903c1463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 -- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9024004800 msgr2=0x7f903c1463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9024004800 0x7f903c1463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 -- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903c09b790 msgr2=0x7f903c09bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903c09b790 0x7f903c09bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 -- 172.21.15.72:0/170021720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c14f4f0 con 0x7f903c14fd30 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.426+0000 7f903a7fc640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 0x7f903c1501b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f902c037040 tx=0x7f902c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.427+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c042020 con 0x7f903c14fd30 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.427+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902c03a040 con 0x7f903c14fd30 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.427+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902c039070 con 0x7f903c14fd30 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.427+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c1508d0 con 0x7f903c14fd30 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.427+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c150e90 con 0x7f903c14fd30 2024-03-06T07:26:46.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.428+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c069ee0 con 0x7f903c14fd30 2024-03-06T07:26:46.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.428+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f902c045020 con 0x7f903c14fd30 2024-03-06T07:26:46.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.429+0000 7f90237fe640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:46.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.429+0000 7f903affd640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:46.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.429+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f902c0ab830 con 0x7f903c14fd30 2024-03-06T07:26:46.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.429+0000 7f903affd640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f902806b4d0 0x7f902806d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9030001eb0 tx=0x7f9030007970 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:46.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.433+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f902c04c030 con 0x7f903c14fd30 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.622+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f903c06bd70 con 0x7f903c14fd30 2024-03-06T07:26:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.622+0000 7f90237fe640 1 -- 172.21.15.72:0/170021720 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v98) v1 ==== 80+0+364 (secure 0 0 0) 0x7f902c079ff0 con 0x7f903c14fd30 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6837/646486087 2024-03-07T07:26:13.890692+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6835/646486087 2024-03-07T07:26:13.890692+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6836/3967731577 2024-03-07T07:26:19.829563+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.146:6835/209452807 2024-03-07T07:26:13.880837+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6834/3967731577 2024-03-07T07:26:19.829563+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.146:6833/209452807 2024-03-07T07:26:13.880837+0000 2024-03-06T07:26:46.625 INFO:teuthology.orchestra.run.smithi072.stderr:listed 6 entries 2024-03-06T07:26:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f902806b4d0 msgr2=0x7f902806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f902806b4d0 0x7f902806d990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9030001eb0 tx=0x7f9030007970 comp rx=0 tx=0).stop 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 msgr2=0x7f903c1501b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 0x7f903c1501b0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f902c037040 tx=0x7f902c004730 comp rx=0 tx=0).stop 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 shutdown_connections 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f902806b4d0 0x7f902806d990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f903c09b790 0x7f903c09bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f903c14fd30 0x7f903c1501b0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 --2- 172.21.15.72:0/170021720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9024004800 0x7f903c1463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 >> 172.21.15.72:0/170021720 conn(0x7f903c0b1dc0 msgr2=0x7f903c0ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 shutdown_connections 2024-03-06T07:26:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:46.624+0000 7f90414d6640 1 -- 172.21.15.72:0/170021720 wait complete. 2024-03-06T07:26:46.637 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:26:46.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:46.637 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:26:46.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[68328]: fuse finished with error 0 and tester_r 0 2024-03-06T07:26:46.664 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:26:46.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:46.664 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:26:46.831 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:26:46.831 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:46.848 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:46.849 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:26:46.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:46.849 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:26:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:46.904 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:26:46.905 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:26:46.905 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:46.921 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:46.921 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:26:46.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:46.921 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:26:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:46.975 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:26:46.976 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:26:46.976 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:46.992 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:46.992 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:26:46.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:26:46.992 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:26:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:26:47.048 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:26:47.049 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:47.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.172+0000 7f477ade9640 1 Processor -- start 2024-03-06T07:26:47.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.172+0000 7f477ade9640 1 -- start start 2024-03-06T07:26:47.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f477ade9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f47740b3cc0 0x7f47740afa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f477ade9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f47740affa0 0x7f477415e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f477ade9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477406b190 con 0x7f4774158570 2024-03-06T07:26:47.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f477ade9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47740b0410 con 0x7f47740b3cc0 2024-03-06T07:26:47.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f477ade9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477406b470 con 0x7f47740affa0 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f47740b3cc0 0x7f47740afa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f47740b3cc0 0x7f47740afa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58410/0 (socket says 172.21.15.72:58410) 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 -- 172.21.15.72:0/3398299577 learned_addr learned my addr 172.21.15.72:0/3398299577 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4779386640 1 --2- 172.21.15.72:0/3398299577 >> v2:172.21.15.146:3300/0 conn(0x7f47740affa0 0x7f477415e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 -- 172.21.15.72:0/3398299577 >> v2:172.21.15.146:3300/0 conn(0x7f47740affa0 msgr2=0x7f477415e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 --2- 172.21.15.72:0/3398299577 >> v2:172.21.15.146:3300/0 conn(0x7f47740affa0 0x7f477415e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 -- 172.21.15.72:0/3398299577 >> v1:172.21.15.146:6789/0 conn(0x7f4774158570 legacy=0x7f477406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4773fff640 1 -- 172.21.15.72:0/3398299577 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477406b750 con 0x7f47740b3cc0 2024-03-06T07:26:47.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.173+0000 7f4779386640 1 --2- 172.21.15.72:0/3398299577 >> v2:172.21.15.146:3300/0 conn(0x7f47740affa0 0x7f477415e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.174+0000 7f4773fff640 1 --2- 172.21.15.72:0/3398299577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47740b3cc0 0x7f47740afa60 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f4774071b30 tx=0x7f476002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac6dd70a0cf5062 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.174+0000 7f47737fe640 1 -- 172.21.15.72:0/3398299577 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760034070 con 0x7f47740b3cc0 2024-03-06T07:26:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.174+0000 7f47737fe640 1 -- 172.21.15.72:0/3398299577 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4760030860 con 0x7f47740b3cc0 2024-03-06T07:26:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.174+0000 7f47737fe640 1 -- 172.21.15.72:0/3398299577 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4760030c30 con 0x7f47740b3cc0 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 -- 172.21.15.72:0/3398299577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47740b3cc0 msgr2=0x7f47740afa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 --2- 172.21.15.72:0/3398299577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47740b3cc0 0x7f47740afa60 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f4774071b30 tx=0x7f476002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 -- 172.21.15.72:0/3398299577 shutdown_connections 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 --2- 172.21.15.72:0/3398299577 >> v2:172.21.15.146:3300/0 conn(0x7f47740affa0 0x7f477415e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 --2- 172.21.15.72:0/3398299577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47740b3cc0 0x7f47740afa60 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 -- 172.21.15.72:0/3398299577 >> 172.21.15.72:0/3398299577 conn(0x7f47740b29c0 msgr2=0x7f47740b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 -- 172.21.15.72:0/3398299577 shutdown_connections 2024-03-06T07:26:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.175+0000 7f477ade9640 1 -- 172.21.15.72:0/3398299577 wait complete. 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.176+0000 7f477ade9640 1 Processor -- start 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.176+0000 7f477ade9640 1 -- start start 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f477ade9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740affa0 0x7f4774150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f477ade9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740b3cc0 0x7f477406ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f477ade9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 0x7f4774141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f477ade9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f477415eb60 con 0x7f4774158570 2024-03-06T07:26:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f477ade9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4774158950 con 0x7f47740affa0 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f477ade9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477415ece0 con 0x7f47740b3cc0 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 0x7f4774141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 0x7f4774141870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58412/0 (socket says 172.21.15.72:58412) 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 -- 172.21.15.72:0/3184012987 learned_addr learned my addr 172.21.15.72:0/3184012987 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4778b85640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740affa0 0x7f4774150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4773fff640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740b3cc0 0x7f477406ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 -- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740b3cc0 msgr2=0x7f477406ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740b3cc0 0x7f477406ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 -- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740affa0 msgr2=0x7f4774150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740affa0 0x7f4774150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 -- 172.21.15.72:0/3184012987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477406f200 con 0x7f4774158570 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4773fff640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740b3cc0 0x7f477406ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.177+0000 7f4779386640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 0x7f4774141870 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f476c009fa0 tx=0x7f476c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.178+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c01a070 con 0x7f4774158570 2024-03-06T07:26:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.178+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476c003bb0 con 0x7f4774158570 2024-03-06T07:26:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.178+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4774151140 con 0x7f4774158570 2024-03-06T07:26:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.178+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c004030 con 0x7f4774158570 2024-03-06T07:26:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.178+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4774151700 con 0x7f4774158570 2024-03-06T07:26:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.179+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4774069ee0 con 0x7f4774158570 2024-03-06T07:26:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.179+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f476c01d070 con 0x7f4774158570 2024-03-06T07:26:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.180+0000 7f47717fa640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f475806b4d0 0x7f475806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.180+0000 7f4778b85640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f475806b4d0 0x7f475806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.180+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f476c08a640 con 0x7f4774158570 2024-03-06T07:26:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.180+0000 7f4778b85640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f475806b4d0 0x7f475806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4768007fd0 tx=0x7f4768009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.184+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f476c010070 con 0x7f4774158570 2024-03-06T07:26:47.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.398+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4774160ef0 con 0x7f4774158570 2024-03-06T07:26:47.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.399+0000 7f47717fa640 1 -- 172.21.15.72:0/3184012987 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f476c0593c0 con 0x7f4774158570 2024-03-06T07:26:47.401 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:47.401 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:47.401 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:47.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f475806b4d0 msgr2=0x7f475806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f475806b4d0 0x7f475806d990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f4768007fd0 tx=0x7f4768009040 comp rx=0 tx=0).stop 2024-03-06T07:26:47.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 msgr2=0x7f4774141870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 0x7f4774141870 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f476c009fa0 tx=0x7f476c00d560 comp rx=0 tx=0).stop 2024-03-06T07:26:47.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 shutdown_connections 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f475806b4d0 0x7f475806d990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4774158570 0x7f4774141870 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47740b3cc0 0x7f477406ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 --2- 172.21.15.72:0/3184012987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47740affa0 0x7f4774150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 >> 172.21.15.72:0/3184012987 conn(0x7f47740b29c0 msgr2=0x7f47740b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 shutdown_connections 2024-03-06T07:26:47.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.401+0000 7f477ade9640 1 -- 172.21.15.72:0/3184012987 wait complete. 2024-03-06T07:26:47.414 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:47.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.513+0000 7f9f63cb8640 1 Processor -- start 2024-03-06T07:26:47.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.514+0000 7f9f63cb8640 1 -- start start 2024-03-06T07:26:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.514+0000 7f9f63cb8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f5c0b2da0 0x7f9f5c162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.514+0000 7f9f63cb8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c162c70 con 0x7f9f5c161670 2024-03-06T07:26:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.514+0000 7f9f63cb8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5c06cd80 con 0x7f9f5c0cd690 2024-03-06T07:26:47.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.514+0000 7f9f63cb8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5c0b3590 con 0x7f9f5c0b2da0 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f61a54640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9f5c161670 0x7f9f5c161e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45578/0 (socket says 172.21.15.72:45578) 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f61a54640 1 -- 172.21.15.72:0/2526618960 learned_addr learned my addr 172.21.15.72:0/2526618960 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f62255640 1 --2- 172.21.15.72:0/2526618960 >> v2:172.21.15.146:3300/0 conn(0x7f9f5c0b2da0 0x7f9f5c162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701392945 0 0) 0x7f9f5c162c70 con 0x7f9f5c161670 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f48003520 con 0x7f9f5c161670 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2999911698 0 0) 0x7f9f48003520 con 0x7f9f5c161670 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 >> v2:172.21.15.146:3300/0 conn(0x7f9f5c0b2da0 msgr2=0x7f9f5c162570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 --2- 172.21.15.72:0/2526618960 >> v2:172.21.15.146:3300/0 conn(0x7f9f5c0b2da0 0x7f9f5c162570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 >> v1:172.21.15.146:6789/0 conn(0x7f9f5c0cd690 legacy=0x7f9f5c0cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c06d060 con 0x7f9f5c161670 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9f58001670 con 0x7f9f5c161670 2024-03-06T07:26:47.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f62255640 1 --2- 172.21.15.72:0/2526618960 >> v2:172.21.15.146:3300/0 conn(0x7f9f5c0b2da0 0x7f9f5c162570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 >> v1:172.21.15.72:6789/0 conn(0x7f9f5c161670 legacy=0x7f9f5c161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f480039b0 0x7f9f48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f48004800 0x7f9f48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4800c030 con 0x7f9f480051c0 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4800c1e0 con 0x7f9f48004800 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4800c3e0 con 0x7f9f480039b0 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f62255640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f61a54640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f480039b0 0x7f9f48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.515+0000 7f9f61253640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f48004800 0x7f9f48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 -- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f480039b0 msgr2=0x7f9f48003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f480039b0 0x7f9f48003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 -- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f48004800 msgr2=0x7f9f48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f48004800 0x7f9f48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 -- 172.21.15.72:0/2526618960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f48003520 con 0x7f9f480051c0 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f61253640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f48004800 0x7f9f48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 -- 172.21.15.72:0/2526618960 reap_dead start 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f62255640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f4800b9c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400e350 tx=0x7f9f48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552357853201fb05 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f54009de0 con 0x7f9f480051c0 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5402ce40 con 0x7f9f480051c0 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.516+0000 7f9f60a52640 1 -- 172.21.15.72:0/2526618960 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f54003bd0 con 0x7f9f480051c0 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 -- 172.21.15.72:0/2526618960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 msgr2=0x7f9f4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f4800b9c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f9f5400e350 tx=0x7f9f48005830 comp rx=0 tx=0).stop 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 -- 172.21.15.72:0/2526618960 shutdown_connections 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/2526618960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f4800b9c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 -- 172.21.15.72:0/2526618960 >> 172.21.15.72:0/2526618960 conn(0x7f9f5c0d1d50 msgr2=0x7f9f5c0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 -- 172.21.15.72:0/2526618960 shutdown_connections 2024-03-06T07:26:47.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.517+0000 7f9f63cb8640 1 -- 172.21.15.72:0/2526618960 wait complete. 2024-03-06T07:26:47.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.518+0000 7f9f63cb8640 1 Processor -- start 2024-03-06T07:26:47.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.518+0000 7f9f63cb8640 1 -- start start 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f63cb8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f5c151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f63cb8640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f5c0b5ec0 0x7f9f5c160640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f63cb8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5c15d100 0x7f9f5c15d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f63cb8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c06d4c0 con 0x7f9f480051c0 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f63cb8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c06d340 con 0x7f9f5c15d100 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f63cb8640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5c161a50 con 0x7f9f5c0b5ec0 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f5c151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f5c151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58440/0 (socket says 172.21.15.72:58440) 2024-03-06T07:26:47.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 -- 172.21.15.72:0/1768815984 learned_addr learned my addr 172.21.15.72:0/1768815984 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f62255640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5c15d100 0x7f9f5c15d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61253640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f5c0b5ec0 0x7f9f5c160640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 -- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f5c0b5ec0 msgr2=0x7f9f5c160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f5c0b5ec0 0x7f9f5c160640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 -- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5c15d100 msgr2=0x7f9f5c15d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5c15d100 0x7f9f5c15d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 -- 172.21.15.72:0/1768815984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c0b65a0 con 0x7f9f480051c0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f62255640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5c15d100 0x7f9f5c15d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.519+0000 7f9f61a54640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f5c151b70 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f9f580011e0 tx=0x7f9f58007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.520+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5800f960 con 0x7f9f480051c0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.520+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58018070 con 0x7f9f480051c0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.520+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5800fd60 con 0x7f9f480051c0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.520+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f5c0b6880 con 0x7f9f480051c0 2024-03-06T07:26:47.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.520+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f5c0b6e40 con 0x7f9f480051c0 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.521+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5c068050 con 0x7f9f480051c0 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.521+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f5801d460 con 0x7f9f480051c0 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.522+0000 7f9f527fc640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f4006b500 0x7f9f4006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.522+0000 7f9f61253640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f4006b500 0x7f9f4006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.522+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9f5808a260 con 0x7f9f480051c0 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.522+0000 7f9f61253640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f4006b500 0x7f9f4006d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c000c00 tx=0x7f9f4c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.526+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f58059070 con 0x7f9f480051c0 2024-03-06T07:26:47.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.700+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f5c0caee0 con 0x7f9f480051c0 2024-03-06T07:26:47.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.701+0000 7f9f527fc640 1 -- 172.21.15.72:0/1768815984 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17007 (secure 0 0 0) 0x7f9f58058970 con 0x7f9f480051c0 2024-03-06T07:26:47.704 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:47.704 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":98,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:19.829591+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6835/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6836/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6835/209452807":"2024-03-07T07:26:13.880837+0000","172.21.15.72:6834/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6833/209452807":"2024-03-07T07:26:13.880837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f4006b500 msgr2=0x7f9f4006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f4006b500 0x7f9f4006d9c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c000c00 tx=0x7f9f4c009040 comp rx=0 tx=0).stop 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 msgr2=0x7f9f5c151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f5c151b70 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f9f580011e0 tx=0x7f9f58007720 comp rx=0 tx=0).stop 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 shutdown_connections 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f4006b500 0x7f9f4006d9c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f5c15d100 0x7f9f5c15d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f5c0b5ec0 0x7f9f5c160640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 --2- 172.21.15.72:0/1768815984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f480051c0 0x7f9f5c151b70 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 >> 172.21.15.72:0/1768815984 conn(0x7f9f5c0d1d50 msgr2=0x7f9f5c0b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.703+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 shutdown_connections 2024-03-06T07:26:47.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.704+0000 7f9f63cb8640 1 -- 172.21.15.72:0/1768815984 wait complete. 2024-03-06T07:26:47.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:26:04.296462+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:26:05.055015+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:47.717 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:47.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.822+0000 7f3c9eedf640 1 Processor -- start 2024-03-06T07:26:47.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.822+0000 7f3c9eedf640 1 -- start start 2024-03-06T07:26:47.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9eedf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c9815e150 0x7f3c980cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9eedf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3c980cfba0 0x7f3c980d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9eedf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c980b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9eedf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980d2ab0 con 0x7f3c980cfba0 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9eedf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c980d1ea0 con 0x7f3c9815e150 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9eedf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980d2180 con 0x7f3c980d0e90 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3c980cfba0 0x7f3c980d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3c980cfba0 0x7f3c980d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58456/0 (socket says 172.21.15.72:58456) 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 -- 172.21.15.72:0/2094209600 learned_addr learned my addr 172.21.15.72:0/2094209600 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9cc7b640 1 --2- 172.21.15.72:0/2094209600 >> v2:172.21.15.146:3300/0 conn(0x7f3c9815e150 0x7f3c980cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9d47c640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c980b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 -- 172.21.15.72:0/2094209600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 msgr2=0x7f3c980b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c980b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 -- 172.21.15.72:0/2094209600 >> v2:172.21.15.146:3300/0 conn(0x7f3c9815e150 msgr2=0x7f3c980cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 --2- 172.21.15.72:0/2094209600 >> v2:172.21.15.146:3300/0 conn(0x7f3c9815e150 0x7f3c980cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c97fff640 1 -- 172.21.15.72:0/2094209600 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980d2460 con 0x7f3c980cfba0 2024-03-06T07:26:47.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9cc7b640 1 --2- 172.21.15.72:0/2094209600 >> v2:172.21.15.146:3300/0 conn(0x7f3c9815e150 0x7f3c980cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:47.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.823+0000 7f3c9d47c640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c980b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:47.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.824+0000 7f3c97fff640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c980cfba0 0x7f3c980d0950 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f3c980b6af0 tx=0x7f3c8402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1063773122bed9a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.824+0000 7f3c977fe640 1 -- 172.21.15.72:0/2094209600 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c8402e070 con 0x7f3c980cfba0 2024-03-06T07:26:47.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.824+0000 7f3c977fe640 1 -- 172.21.15.72:0/2094209600 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c84037070 con 0x7f3c980cfba0 2024-03-06T07:26:47.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.824+0000 7f3c977fe640 1 -- 172.21.15.72:0/2094209600 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84031a00 con 0x7f3c980cfba0 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2094209600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c980cfba0 msgr2=0x7f3c980d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c980cfba0 0x7f3c980d0950 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f3c980b6af0 tx=0x7f3c8402b880 comp rx=0 tx=0).stop 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2094209600 shutdown_connections 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c980b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2094209600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c980cfba0 0x7f3c980d0950 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2094209600 >> v2:172.21.15.146:3300/0 conn(0x7f3c9815e150 0x7f3c980cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2094209600 >> 172.21.15.72:0/2094209600 conn(0x7f3c9809d230 msgr2=0x7f3c980d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2094209600 shutdown_connections 2024-03-06T07:26:47.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.825+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2094209600 wait complete. 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.826+0000 7f3c9eedf640 1 Processor -- start 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.826+0000 7f3c9eedf640 1 -- start start 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9eedf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c980cfba0 0x7f3c980a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9eedf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c9815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9eedf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 0x7f3c98160b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9eedf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980d1b30 con 0x7f3c9815e150 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9eedf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c980d2f40 con 0x7f3c980cfba0 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9eedf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9815e530 con 0x7f3c980d0e90 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 0x7f3c98160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 0x7f3c98160b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58460/0 (socket says 172.21.15.72:58460) 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 -- 172.21.15.72:0/2234931370 learned_addr learned my addr 172.21.15.72:0/2234931370 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9cc7b640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c980cfba0 0x7f3c980a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c97fff640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c9815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 -- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 msgr2=0x7f3c9815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c9815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 -- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c980cfba0 msgr2=0x7f3c980a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c980cfba0 0x7f3c980a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 -- 172.21.15.72:0/2234931370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c980a0f20 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9cc7b640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c980cfba0 0x7f3c980a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.827+0000 7f3c9d47c640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 0x7f3c98160b80 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000cf20 tx=0x7f3c9000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.828+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9001a070 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.828+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c9000d840 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.828+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c980a1200 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.828+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000dc30 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.828+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c980a17c0 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.829+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c98069ee0 con 0x7f3c9815e150 2024-03-06T07:26:47.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.829+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3c9001d070 con 0x7f3c9815e150 2024-03-06T07:26:47.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.830+0000 7f3c957fa640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:47.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.830+0000 7f3c9cc7b640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:47.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.830+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3c9008abc0 con 0x7f3c9815e150 2024-03-06T07:26:47.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.830+0000 7f3c9cc7b640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00a910 tx=0x7f3c8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:47.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:47.834+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3c90059940 con 0x7f3c9815e150 2024-03-06T07:26:48.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.011+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3c980d4670 con 0x7f3c9815e150 2024-03-06T07:26:48.014 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.012+0000 7f3c957fa640 1 -- 172.21.15.72:0/2234931370 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3c90059240 con 0x7f3c9815e150 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.013+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3c7c06b4d0 msgr2=0x7f3c7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.013+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00a910 tx=0x7f3c8c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.013+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 msgr2=0x7f3c98160b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.013+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 0x7f3c98160b80 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000cf20 tx=0x7f3c9000a7d0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 shutdown_connections 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3c7c06b4d0 0x7f3c7c06d990 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3c9815e150 0x7f3c98160b80 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c980d0e90 0x7f3c9815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 --2- 172.21.15.72:0/2234931370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c980cfba0 0x7f3c980a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 >> 172.21.15.72:0/2234931370 conn(0x7f3c9809d230 msgr2=0x7f3c9806d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 shutdown_connections 2024-03-06T07:26:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.014+0000 7f3c9eedf640 1 -- 172.21.15.72:0/2234931370 wait complete. 2024-03-06T07:26:48.113 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:26:48.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.214+0000 7fcea100e640 1 Processor -- start 2024-03-06T07:26:48.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.214+0000 7fcea100e640 1 -- start start 2024-03-06T07:26:48.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fcea100e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fce9c15f5e0 0x7fce9c0b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fcea100e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce9c0b16b0 0x7fce9c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fcea100e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c0d2870 0x7fce9c0cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fcea100e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce9c0cf920 con 0x7fce9c15f5e0 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fcea100e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce9c0cd4d0 con 0x7fce9c0b16b0 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fcea100e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c0cd7b0 con 0x7fce9c0d2870 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fce9c15f5e0 0x7fce9c0b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fce9c15f5e0 0x7fce9c0b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58476/0 (socket says 172.21.15.72:58476) 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 -- 172.21.15.72:0/2540992367 learned_addr learned my addr 172.21.15.72:0/2540992367 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce93fff640 1 --2- 172.21.15.72:0/2540992367 >> v2:172.21.15.146:3300/0 conn(0x7fce9c0b16b0 0x7fce9c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 -- 172.21.15.72:0/2540992367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c0d2870 msgr2=0x7fce9c0cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9b59e640 1 --2- 172.21.15.72:0/2540992367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c0d2870 0x7fce9c0cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 --2- 172.21.15.72:0/2540992367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c0d2870 0x7fce9c0cea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 -- 172.21.15.72:0/2540992367 >> v2:172.21.15.146:3300/0 conn(0x7fce9c0b16b0 msgr2=0x7fce9c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 --2- 172.21.15.72:0/2540992367 >> v2:172.21.15.146:3300/0 conn(0x7fce9c0b16b0 0x7fce9c0ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce9ad9d640 1 -- 172.21.15.72:0/2540992367 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0cda90 con 0x7fce9c15f5e0 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.215+0000 7fce93fff640 1 --2- 172.21.15.72:0/2540992367 >> v2:172.21.15.146:3300/0 conn(0x7fce9c0b16b0 0x7fce9c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:48.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.216+0000 7fce9ad9d640 1 --2- 172.21.15.72:0/2540992367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c15f5e0 0x7fce9c0b2bf0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fce9c071dc0 tx=0x7fce8c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ae14edc30de47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.216+0000 7fce9a59c640 1 -- 172.21.15.72:0/2540992367 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c02e070 con 0x7fce9c15f5e0 2024-03-06T07:26:48.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.216+0000 7fce9a59c640 1 -- 172.21.15.72:0/2540992367 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c037070 con 0x7fce9c15f5e0 2024-03-06T07:26:48.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.216+0000 7fce9a59c640 1 -- 172.21.15.72:0/2540992367 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c031a00 con 0x7fce9c15f5e0 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 -- 172.21.15.72:0/2540992367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c15f5e0 msgr2=0x7fce9c0b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 --2- 172.21.15.72:0/2540992367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c15f5e0 0x7fce9c0b2bf0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fce9c071dc0 tx=0x7fce8c02b880 comp rx=0 tx=0).stop 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 -- 172.21.15.72:0/2540992367 shutdown_connections 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 --2- 172.21.15.72:0/2540992367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c0d2870 0x7fce9c0cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 --2- 172.21.15.72:0/2540992367 >> v2:172.21.15.146:3300/0 conn(0x7fce9c0b16b0 0x7fce9c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 --2- 172.21.15.72:0/2540992367 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c15f5e0 0x7fce9c0b2bf0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 -- 172.21.15.72:0/2540992367 >> 172.21.15.72:0/2540992367 conn(0x7fce9c0d1d50 msgr2=0x7fce9c0d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 -- 172.21.15.72:0/2540992367 shutdown_connections 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.217+0000 7fcea100e640 1 -- 172.21.15.72:0/2540992367 wait complete. 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.219+0000 7fcea100e640 1 Processor -- start 2024-03-06T07:26:48.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.219+0000 7fcea100e640 1 -- start start 2024-03-06T07:26:48.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fcea100e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 0x7fce9c0b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fcea100e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce9c0d2870 0x7fce9c14d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fcea100e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c15f5e0 0x7fce9c1f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fcea100e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c0cf270 con 0x7fce9c0b16b0 2024-03-06T07:26:48.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fcea100e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c0cf0f0 con 0x7fce9c0d2870 2024-03-06T07:26:48.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fcea100e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9c15fcd0 con 0x7fce9c15f5e0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 0x7fce9c0b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 0x7fce9c0b55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58490/0 (socket says 172.21.15.72:58490) 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 -- 172.21.15.72:0/3545376258 learned_addr learned my addr 172.21.15.72:0/3545376258 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 -- 172.21.15.72:0/3545376258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c15f5e0 msgr2=0x7fce9c1f1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c15f5e0 0x7fce9c1f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 -- 172.21.15.72:0/3545376258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce9c0d2870 msgr2=0x7fce9c14d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce9c0d2870 0x7fce9c14d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 -- 172.21.15.72:0/3545376258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c0b5b30 con 0x7fce9c0b16b0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce9ad9d640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 0x7fce9c0b55f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fce8c000c00 tx=0x7fce8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.220+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c032880 con 0x7fce9c0b16b0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.221+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c032a20 con 0x7fce9c0b16b0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.221+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9c1f2600 con 0x7fce9c0b16b0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.221+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c032e10 con 0x7fce9c0b16b0 2024-03-06T07:26:48.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.221+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce9c0b4df0 con 0x7fce9c0b16b0 2024-03-06T07:26:48.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.222+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9c068050 con 0x7fce9c0b16b0 2024-03-06T07:26:48.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.222+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fce8c0453f0 con 0x7fce9c0b16b0 2024-03-06T07:26:48.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.222+0000 7fce937fe640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fce7c06b4d0 0x7fce7c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.223+0000 7fce93fff640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fce7c06b4d0 0x7fce7c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.223+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fce8c0ac2d0 con 0x7fce9c0b16b0 2024-03-06T07:26:48.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.223+0000 7fce93fff640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fce7c06b4d0 0x7fce7c06d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fce9c15b500 tx=0x7fce84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.230+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fce8c040070 con 0x7fce9c0b16b0 2024-03-06T07:26:48.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.409+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce9c0b7cd0 con 0x7fce9c0b16b0 2024-03-06T07:26:48.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.410+0000 7fce937fe640 1 -- 172.21.15.72:0/3545376258 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7fce8c07b170 con 0x7fce9c0b16b0 2024-03-06T07:26:48.412 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:26:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fce7c06b4d0 msgr2=0x7fce7c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fce7c06b4d0 0x7fce7c06d990 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fce9c15b500 tx=0x7fce84008040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 msgr2=0x7fce9c0b55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 0x7fce9c0b55f0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fce8c000c00 tx=0x7fce8c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 shutdown_connections 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fce7c06b4d0 0x7fce7c06d990 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce9c15f5e0 0x7fce9c1f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce9c0d2870 0x7fce9c14d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 --2- 172.21.15.72:0/3545376258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fce9c0b16b0 0x7fce9c0b55f0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 >> 172.21.15.72:0/3545376258 conn(0x7fce9c0d1d50 msgr2=0x7fce9c0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 shutdown_connections 2024-03-06T07:26:48.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.412+0000 7fcea100e640 1 -- 172.21.15.72:0/3545376258 wait complete. 2024-03-06T07:26:48.426 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:26:48.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.528+0000 7fe076d35640 1 Processor -- start 2024-03-06T07:26:48.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.528+0000 7fe076d35640 1 -- start start 2024-03-06T07:26:48.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.529+0000 7fe076d35640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 0x7fe0700cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.529+0000 7fe076d35640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0700cf880 0x7fe07006af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.529+0000 7fe076d35640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0700cfec0 con 0x7fe07006b4c0 2024-03-06T07:26:48.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.529+0000 7fe076d35640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07006b930 con 0x7fe0700cf880 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.529+0000 7fe076d35640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700d01a0 con 0x7fe070145700 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe074ad1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 0x7fe0700cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0700cf880 0x7fe07006af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0700cf880 0x7fe07006af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:38278/0 (socket says 172.21.15.72:38278) 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe074ad1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 0x7fe0700cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:39708/0 (socket says 172.21.15.72:39708) 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 -- 172.21.15.72:0/1554662567 learned_addr learned my addr 172.21.15.72:0/1554662567 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe0752d2640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe07006b4c0 0x7fe070071c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57660/0 (socket says 172.21.15.72:57660) 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 -- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 msgr2=0x7fe0700cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 --2- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 0x7fe0700cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 -- 172.21.15.72:0/1554662567 >> v1:172.21.15.146:6789/0 conn(0x7fe07006b4c0 legacy=0x7fe070071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe06ffff640 1 -- 172.21.15.72:0/1554662567 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0700d0480 con 0x7fe0700cf880 2024-03-06T07:26:48.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.530+0000 7fe074ad1640 1 --2- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 0x7fe0700cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:48.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.531+0000 7fe06ffff640 1 --2- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0700cf880 0x7fe07006af80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009870 tx=0x7fe05c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71365d558b7b7796 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.531+0000 7fe06f7fe640 1 -- 172.21.15.72:0/1554662567 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c034070 con 0x7fe0700cf880 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.531+0000 7fe06f7fe640 1 -- 172.21.15.72:0/1554662567 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c030840 con 0x7fe0700cf880 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.531+0000 7fe06f7fe640 1 -- 172.21.15.72:0/1554662567 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c030c10 con 0x7fe0700cf880 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 -- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0700cf880 msgr2=0x7fe07006af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 --2- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0700cf880 0x7fe07006af80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fe05c009870 tx=0x7fe05c02b980 comp rx=0 tx=0).stop 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 -- 172.21.15.72:0/1554662567 shutdown_connections 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 --2- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0700cf880 0x7fe07006af80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 --2- 172.21.15.72:0/1554662567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe070145700 0x7fe0700cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 -- 172.21.15.72:0/1554662567 >> 172.21.15.72:0/1554662567 conn(0x7fe0700b2890 msgr2=0x7fe0700b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 -- 172.21.15.72:0/1554662567 shutdown_connections 2024-03-06T07:26:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.532+0000 7fe076d35640 1 -- 172.21.15.72:0/1554662567 wait complete. 2024-03-06T07:26:48.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.533+0000 7fe076d35640 1 Processor -- start 2024-03-06T07:26:48.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.534+0000 7fe076d35640 1 -- start start 2024-03-06T07:26:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe076d35640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe07006b4c0 0x7fe07013d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe076d35640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 0x7fe0701f1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe076d35640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe070145700 0x7fe07015d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe076d35640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07015fc40 con 0x7fe0700cf880 2024-03-06T07:26:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe076d35640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070145ae0 con 0x7fe070145700 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe076d35640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700b5280 con 0x7fe07006b4c0 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 0x7fe0701f1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 0x7fe0701f1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58500/0 (socket says 172.21.15.72:58500) 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 -- 172.21.15.72:0/3239506153 learned_addr learned my addr 172.21.15.72:0/3239506153 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe074ad1640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe07006b4c0 0x7fe07013d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe0752d2640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe070145700 0x7fe07015d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 -- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe07006b4c0 msgr2=0x7fe07013d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe07006b4c0 0x7fe07013d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 -- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe070145700 msgr2=0x7fe07015d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe070145700 0x7fe07015d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe06ffff640 1 -- 172.21.15.72:0/3239506153 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07013e1b0 con 0x7fe0700cf880 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe074ad1640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe07006b4c0 0x7fe07013d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.535+0000 7fe0752d2640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe070145700 0x7fe07015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.536+0000 7fe06ffff640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 0x7fe0701f1250 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe05c000c00 tx=0x7fe05c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.536+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c048070 con 0x7fe0700cf880 2024-03-06T07:26:48.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.536+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05c030640 con 0x7fe0700cf880 2024-03-06T07:26:48.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.536+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07013e490 con 0x7fe0700cf880 2024-03-06T07:26:48.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.536+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c034040 con 0x7fe0700cf880 2024-03-06T07:26:48.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.536+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07013ea50 con 0x7fe0700cf880 2024-03-06T07:26:48.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.537+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070068050 con 0x7fe0700cf880 2024-03-06T07:26:48.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.538+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe05c04b070 con 0x7fe0700cf880 2024-03-06T07:26:48.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.538+0000 7fe06d7fa640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe05406b4d0 0x7fe05406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.538+0000 7fe074ad1640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe05406b4d0 0x7fe05406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.538+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe05c0ac2d0 con 0x7fe0700cf880 2024-03-06T07:26:48.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.538+0000 7fe074ad1640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe05406b4d0 0x7fe05406d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe064009af0 tx=0x7fe064008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.542+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe05c07b170 con 0x7fe0700cf880 2024-03-06T07:26:48.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.797+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0700b3c90 con 0x7fe0700cf880 2024-03-06T07:26:48.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.797+0000 7fe06d7fa640 1 -- 172.21.15.72:0/3239506153 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe05c07aa70 con 0x7fe0700cf880 2024-03-06T07:26:48.799 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe05406b4d0 msgr2=0x7fe05406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe05406b4d0 0x7fe05406d990 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe064009af0 tx=0x7fe064008040 comp rx=0 tx=0).stop 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 msgr2=0x7fe0701f1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 0x7fe0701f1250 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe05c000c00 tx=0x7fe05c004730 comp rx=0 tx=0).stop 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 shutdown_connections 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe05406b4d0 0x7fe05406d990 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe070145700 0x7fe07015d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0700cf880 0x7fe0701f1250 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 --2- 172.21.15.72:0/3239506153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe07006b4c0 0x7fe07013d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 >> 172.21.15.72:0/3239506153 conn(0x7fe0700b2890 msgr2=0x7fe0700b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 shutdown_connections 2024-03-06T07:26:48.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.799+0000 7fe076d35640 1 -- 172.21.15.72:0/3239506153 wait complete. 2024-03-06T07:26:48.812 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:48.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.922+0000 7f24348df640 1 Processor -- start 2024-03-06T07:26:48.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.922+0000 7f24348df640 1 -- start start 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f24348df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f243015f5e0 0x7f24300b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f24348df640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24300cd4d0 con 0x7f24300b16b0 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f24348df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24300cd7b0 con 0x7f24300cec60 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f24348df640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24300cf920 con 0x7f243015f5e0 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f2425d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f24300b16b0 0x7f24300ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45594/0 (socket says 172.21.15.72:45594) 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f2425d9b640 1 -- 172.21.15.72:0/2041647233 learned_addr learned my addr 172.21.15.72:0/2041647233 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f242e59c640 1 --2- 172.21.15.72:0/2041647233 >> v2:172.21.15.146:3300/0 conn(0x7f243015f5e0 0x7f24300b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.923+0000 7f242dd9b640 1 -- 172.21.15.72:0/2041647233 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417067720 0 0) 0x7f24300cd4d0 con 0x7f24300b16b0 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.924+0000 7f242dd9b640 1 -- 172.21.15.72:0/2041647233 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c003520 con 0x7f24300b16b0 2024-03-06T07:26:48.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.924+0000 7f242e59c640 1 -- 172.21.15.72:0/2041647233 >> v1:172.21.15.146:6789/0 conn(0x7f24300cec60 legacy=0x7f24300cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.924+0000 7f242dd9b640 1 -- 172.21.15.72:0/2041647233 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072914704 0 0) 0x7f24300cd7b0 con 0x7f24300cec60 2024-03-06T07:26:48.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.924+0000 7f242e59c640 1 -- 172.21.15.72:0/2041647233 >> v1:172.21.15.72:6789/0 conn(0x7f24300b16b0 legacy=0x7f24300ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.924+0000 7f242e59c640 1 -- 172.21.15.72:0/2041647233 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24300cda90 con 0x7f243015f5e0 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.924+0000 7f242e59c640 1 --2- 172.21.15.72:0/2041647233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243015f5e0 0x7f24300b2bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f24200097a0 tx=0x7f242002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc9c9bd949087a18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f242dd9b640 1 -- 172.21.15.72:0/2041647233 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420035020 con 0x7f243015f5e0 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f242dd9b640 1 -- 172.21.15.72:0/2041647233 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420031830 con 0x7f243015f5e0 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f242dd9b640 1 -- 172.21.15.72:0/2041647233 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420031be0 con 0x7f243015f5e0 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 -- 172.21.15.72:0/2041647233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243015f5e0 msgr2=0x7f24300b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 --2- 172.21.15.72:0/2041647233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243015f5e0 0x7f24300b2bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f24200097a0 tx=0x7f242002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 -- 172.21.15.72:0/2041647233 shutdown_connections 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 --2- 172.21.15.72:0/2041647233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243015f5e0 0x7f24300b2bf0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 -- 172.21.15.72:0/2041647233 >> 172.21.15.72:0/2041647233 conn(0x7f24300d1d50 msgr2=0x7f24300d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:48.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 -- 172.21.15.72:0/2041647233 shutdown_connections 2024-03-06T07:26:48.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.925+0000 7f24348df640 1 -- 172.21.15.72:0/2041647233 wait complete. 2024-03-06T07:26:48.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 Processor -- start 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 -- start start 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 0x7f24300b76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24300cec60 0x7f24300b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243015f5e0 0x7f24301f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300cf270 con 0x7f24300b16b0 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300cf0f0 con 0x7f24300cec60 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f24348df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24300d2870 con 0x7f243015f5e0 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f242e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 0x7f24300b76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f242e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 0x7f24300b76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58506/0 (socket says 172.21.15.72:58506) 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.927+0000 7f242e59c640 1 -- 172.21.15.72:0/297212389 learned_addr learned my addr 172.21.15.72:0/297212389 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242ed9d640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243015f5e0 0x7f24301f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f2425d9b640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24300cec60 0x7f24300b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242e59c640 1 -- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243015f5e0 msgr2=0x7f24301f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242e59c640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243015f5e0 0x7f24301f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242e59c640 1 -- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24300cec60 msgr2=0x7f24300b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242e59c640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24300cec60 0x7f24300b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242e59c640 1 -- 172.21.15.72:0/297212389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243015e940 con 0x7f24300b16b0 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f2425d9b640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24300cec60 0x7f24300b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f242e59c640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 0x7f24300b76e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f242002b910 tx=0x7f24200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242003a8b0 con 0x7f24300b16b0 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242003aa50 con 0x7f24300b16b0 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242003ae40 con 0x7f24300b16b0 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243015ec20 con 0x7f24300b16b0 2024-03-06T07:26:48.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.928+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243015e430 con 0x7f24300b16b0 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.929+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2430068050 con 0x7f24300b16b0 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.930+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f242004b060 con 0x7f24300b16b0 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.930+0000 7f24277fe640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241006b4d0 0x7f241006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.930+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f24200abf70 con 0x7f24300b16b0 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.930+0000 7f2425d9b640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241006b4d0 0x7f241006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:48.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.930+0000 7f2425d9b640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241006b4d0 0x7f241006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f24301574f0 tx=0x7f2418009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:48.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:48.934+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f242003f070 con 0x7f24300b16b0 2024-03-06T07:26:49.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.149+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24300b7cd0 con 0x7f24300b16b0 2024-03-06T07:26:49.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.150+0000 7f24277fe640 1 -- 172.21.15.72:0/297212389 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f2420031800 con 0x7f24300b16b0 2024-03-06T07:26:49.152 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:49.152 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:49.153 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:49.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241006b4d0 msgr2=0x7f241006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241006b4d0 0x7f241006d990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f24301574f0 tx=0x7f2418009040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 msgr2=0x7f24300b76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 0x7f24300b76e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f242002b910 tx=0x7f24200046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 shutdown_connections 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241006b4d0 0x7f241006d990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243015f5e0 0x7f24301f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24300cec60 0x7f24300b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 --2- 172.21.15.72:0/297212389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24300b16b0 0x7f24300b76e0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 >> 172.21.15.72:0/297212389 conn(0x7f24300d1d50 msgr2=0x7f243006f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 shutdown_connections 2024-03-06T07:26:49.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.153+0000 7f24348df640 1 -- 172.21.15.72:0/297212389 wait complete. 2024-03-06T07:26:49.166 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:49.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.264+0000 7f0670c9d640 1 Processor -- start 2024-03-06T07:26:49.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.264+0000 7f0670c9d640 1 -- start start 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f0670c9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 0x7f066c0cf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f0670c9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f066c0d1060 0x7f066c0b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f0670c9d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066c0d1ea0 con 0x7f066c0cfba0 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f0670c9d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066c0d2ab0 con 0x7f066c0d1060 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f0670c9d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0d2180 con 0x7f066c15e150 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f066ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f066c0d1060 0x7f066c0b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f066ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f066c0d1060 0x7f066c0b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58510/0 (socket says 172.21.15.72:58510) 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f066ad9d640 1 -- 172.21.15.72:0/64999548 learned_addr learned my addr 172.21.15.72:0/64999548 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f066ad9d640 1 -- 172.21.15.72:0/64999548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 msgr2=0x7f066c0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f066ad9d640 1 --2- 172.21.15.72:0/64999548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 0x7f066c0cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.265+0000 7f066ad9d640 1 -- 172.21.15.72:0/64999548 >> v1:172.21.15.146:6789/0 conn(0x7f066c0cfba0 legacy=0x7f066c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.266+0000 7f066ad9d640 1 -- 172.21.15.72:0/64999548 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0d2460 con 0x7f066c0d1060 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.266+0000 7f066ad9d640 1 --2- 172.21.15.72:0/64999548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0d1060 0x7f066c0b9060 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0664009e50 tx=0x7f066402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c364285abd0dd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.266+0000 7f0669d9b640 1 -- 172.21.15.72:0/64999548 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066402c9e0 con 0x7f066c0d1060 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.266+0000 7f0669d9b640 1 -- 172.21.15.72:0/64999548 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066402cb80 con 0x7f066c0d1060 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.266+0000 7f0669d9b640 1 -- 172.21.15.72:0/64999548 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06640343f0 con 0x7f066c0d1060 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 -- 172.21.15.72:0/64999548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0d1060 msgr2=0x7f066c0b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 --2- 172.21.15.72:0/64999548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0d1060 0x7f066c0b9060 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0664009e50 tx=0x7f066402be40 comp rx=0 tx=0).stop 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 -- 172.21.15.72:0/64999548 shutdown_connections 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 --2- 172.21.15.72:0/64999548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0d1060 0x7f066c0b9060 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 --2- 172.21.15.72:0/64999548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 0x7f066c0cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 -- 172.21.15.72:0/64999548 >> 172.21.15.72:0/64999548 conn(0x7f066c09d230 msgr2=0x7f066c0d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 -- 172.21.15.72:0/64999548 shutdown_connections 2024-03-06T07:26:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.267+0000 7f0670c9d640 1 -- 172.21.15.72:0/64999548 wait complete. 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.268+0000 7f0670c9d640 1 Processor -- start 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.268+0000 7f0670c9d640 1 -- start start 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0670c9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 0x7f066c148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0670c9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f066c0d1060 0x7f066c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0670c9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 0x7f066c0cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0670c9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0d1b30 con 0x7f066c0cfba0 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0670c9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066c0d2f40 con 0x7f066c0d1060 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0670c9d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f066c15e530 con 0x7f066c15e150 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 0x7f066c148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 0x7f066c148cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58522/0 (socket says 172.21.15.72:58522) 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 -- 172.21.15.72:0/685753615 learned_addr learned my addr 172.21.15.72:0/685753615 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f0663fff640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f066c0d1060 0x7f066c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 -- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 msgr2=0x7f066c0cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 0x7f066c0cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 -- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f066c0d1060 msgr2=0x7f066c0ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f066c0d1060 0x7f066c0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.269+0000 7f066a59c640 1 -- 172.21.15.72:0/685753615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c1f4c70 con 0x7f066c0cfba0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.270+0000 7f066a59c640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 0x7f066c148cd0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f065c00b410 tx=0x7f065c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.270+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c015020 con 0x7f066c0cfba0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.270+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c011850 con 0x7f066c0cfba0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.270+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f066c14b6e0 con 0x7f066c0cfba0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.270+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c011c40 con 0x7f066c0cfba0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.270+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f066c14c1c0 con 0x7f066c0cfba0 2024-03-06T07:26:49.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.271+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f066c14c480 con 0x7f066c0cfba0 2024-03-06T07:26:49.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.271+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f065c018020 con 0x7f066c0cfba0 2024-03-06T07:26:49.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.271+0000 7f0662ffd640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f064c06b4d0 0x7f064c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.272+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f065c08a1d0 con 0x7f066c0cfba0 2024-03-06T07:26:49.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.272+0000 7f0663fff640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f064c06b4d0 0x7f064c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.272+0000 7f0663fff640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f064c06b4d0 0x7f064c06d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f066c1f50b0 tx=0x7f0654009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.280+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f065c01f020 con 0x7f066c0cfba0 2024-03-06T07:26:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.457+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f066c14b4e0 con 0x7f066c0cfba0 2024-03-06T07:26:49.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.458+0000 7f0662ffd640 1 -- 172.21.15.72:0/685753615 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17007 (secure 0 0 0) 0x7f065c0172c0 con 0x7f066c0cfba0 2024-03-06T07:26:49.460 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:49.460 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":98,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:19.829591+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6835/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6836/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6835/209452807":"2024-03-07T07:26:13.880837+0000","172.21.15.72:6834/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6833/209452807":"2024-03-07T07:26:13.880837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f064c06b4d0 msgr2=0x7f064c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f064c06b4d0 0x7f064c06d990 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f066c1f50b0 tx=0x7f0654009040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 msgr2=0x7f066c148cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 0x7f066c148cd0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f065c00b410 tx=0x7f065c00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 shutdown_connections 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f064c06b4d0 0x7f064c06d990 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f066c15e150 0x7f066c0cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f066c0d1060 0x7f066c0ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 --2- 172.21.15.72:0/685753615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f066c0cfba0 0x7f066c148cd0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 >> 172.21.15.72:0/685753615 conn(0x7f066c09d230 msgr2=0x7f066c15fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 shutdown_connections 2024-03-06T07:26:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.460+0000 7f0670c9d640 1 -- 172.21.15.72:0/685753615 wait complete. 2024-03-06T07:26:49.473 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:26:49.474 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:49.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.572+0000 7f892fb5c640 1 Processor -- start 2024-03-06T07:26:49.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.572+0000 7f892fb5c640 1 -- start start 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892fb5c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89281587f0 0x7f892815f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892fb5c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f892806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892fb5c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892815f5b0 con 0x7f892806c4d0 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892fb5c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280722d0 con 0x7f89281587f0 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892fb5c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89280ae7f0 con 0x7f89280ae2c0 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89281587f0 0x7f892815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89281587f0 0x7f892815f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58538/0 (socket says 172.21.15.72:58538) 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 -- 172.21.15.72:0/2776728384 learned_addr learned my addr 172.21.15.72:0/2776728384 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d0f7640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f892806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 -- 172.21.15.72:0/2776728384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 msgr2=0x7f892806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f892806bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 -- 172.21.15.72:0/2776728384 >> v1:172.21.15.146:6789/0 conn(0x7f892806c4d0 legacy=0x7f8928161c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d8f8640 1 -- 172.21.15.72:0/2776728384 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892815f9f0 con 0x7f89281587f0 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.574+0000 7f892d0f7640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f892806bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.575+0000 7f892d8f8640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89281587f0 0x7f892815f070 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f8924009870 tx=0x7f892402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14183504324e9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.575+0000 7f892c8f6640 1 -- 172.21.15.72:0/2776728384 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924034070 con 0x7f89281587f0 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.575+0000 7f892c8f6640 1 -- 172.21.15.72:0/2776728384 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924030840 con 0x7f89281587f0 2024-03-06T07:26:49.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.575+0000 7f892c8f6640 1 -- 172.21.15.72:0/2776728384 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924030c10 con 0x7f89281587f0 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 -- 172.21.15.72:0/2776728384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89281587f0 msgr2=0x7f892815f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89281587f0 0x7f892815f070 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f8924009870 tx=0x7f892402b980 comp rx=0 tx=0).stop 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 -- 172.21.15.72:0/2776728384 shutdown_connections 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f892806bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 --2- 172.21.15.72:0/2776728384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89281587f0 0x7f892815f070 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 -- 172.21.15.72:0/2776728384 >> 172.21.15.72:0/2776728384 conn(0x7f8928071510 msgr2=0x7f8928071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 -- 172.21.15.72:0/2776728384 shutdown_connections 2024-03-06T07:26:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.576+0000 7f892fb5c640 1 -- 172.21.15.72:0/2776728384 wait complete. 2024-03-06T07:26:49.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.578+0000 7f892fb5c640 1 Processor -- start 2024-03-06T07:26:49.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.578+0000 7f892fb5c640 1 -- start start 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892fb5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 0x7f89280cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892fb5c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f89281408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892fb5c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89281587f0 0x7f89280cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892fb5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89281614e0 con 0x7f892806c4d0 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892fb5c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892815ffb0 con 0x7f89281587f0 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892fb5c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8928161660 con 0x7f89280ae2c0 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 0x7f89280cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 0x7f89280cca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58542/0 (socket says 172.21.15.72:58542) 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 -- 172.21.15.72:0/3617137121 learned_addr learned my addr 172.21.15.72:0/3617137121 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 -- 172.21.15.72:0/3617137121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 msgr2=0x7f89281408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f89281408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 -- 172.21.15.72:0/3617137121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89281587f0 msgr2=0x7f89280cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89281587f0 0x7f89280cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 -- 172.21.15.72:0/3617137121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8928141130 con 0x7f892806c4d0 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.579+0000 7f892d8f8640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 0x7f89280cca10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8924000c00 tx=0x7f8924004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.580+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924048070 con 0x7f892806c4d0 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.580+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8924030640 con 0x7f892806c4d0 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.580+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8928141410 con 0x7f892806c4d0 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.580+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8924034040 con 0x7f892806c4d0 2024-03-06T07:26:49.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.580+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89281419d0 con 0x7f892806c4d0 2024-03-06T07:26:49.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.581+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8928068050 con 0x7f892806c4d0 2024-03-06T07:26:49.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.581+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f892404b070 con 0x7f892806c4d0 2024-03-06T07:26:49.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.581+0000 7f891e7fc640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:49.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.581+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f89240ac390 con 0x7f892806c4d0 2024-03-06T07:26:49.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.581+0000 7f892d0f7640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:49.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.582+0000 7f892d0f7640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f890c06b4d0 0x7f890c06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8918006fd0 tx=0x7f8918008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:49.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.586+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f892407b2c0 con 0x7f892806c4d0 2024-03-06T07:26:49.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.936+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8928067200 con 0x7f892806c4d0 2024-03-06T07:26:49.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.936+0000 7f891e7fc640 1 -- 172.21.15.72:0/3617137121 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f892407abc0 con 0x7f892806c4d0 2024-03-06T07:26:49.938 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:49.939 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:49.939 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:49.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f890c06b4d0 msgr2=0x7f890c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f890c06b4d0 0x7f890c06d990 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8918006fd0 tx=0x7f8918008040 comp rx=0 tx=0).stop 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 msgr2=0x7f89280cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 0x7f89280cca10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f8924000c00 tx=0x7f8924004730 comp rx=0 tx=0).stop 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 shutdown_connections 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f890c06b4d0 0x7f890c06d990 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89281587f0 0x7f89280cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89280ae2c0 0x7f89281408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 --2- 172.21.15.72:0/3617137121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f892806c4d0 0x7f89280cca10 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 >> 172.21.15.72:0/3617137121 conn(0x7f8928071510 msgr2=0x7f89280b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 shutdown_connections 2024-03-06T07:26:49.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:49.938+0000 7f892fb5c640 1 -- 172.21.15.72:0/3617137121 wait complete. 2024-03-06T07:26:49.952 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:50.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.050+0000 7fe2ef650640 1 Processor -- start 2024-03-06T07:26:50.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2ef650640 1 -- start start 2024-03-06T07:26:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2ef650640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe2e80d0950 0x7fe2e80d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2ef650640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e80d1ea0 con 0x7fe2e80b9060 2024-03-06T07:26:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2ef650640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e80d2180 con 0x7fe2e815e150 2024-03-06T07:26:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2ef650640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e80d12f0 con 0x7fe2e80d0950 2024-03-06T07:26:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2edbed640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe2e80b9060 0x7fe2e80b94d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45610/0 (socket says 172.21.15.72:45610) 2024-03-06T07:26:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.051+0000 7fe2edbed640 1 -- 172.21.15.72:0/3721229601 learned_addr learned my addr 172.21.15.72:0/3721229601 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:50.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30441786 0 0) 0x7fe2e80d1ea0 con 0x7fe2e80b9060 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ecbeb640 1 --2- 172.21.15.72:0/3721229601 >> v2:172.21.15.146:3300/0 conn(0x7fe2e80d0950 0x7fe2e80d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d4003520 con 0x7fe2e80b9060 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2643319262 0 0) 0x7fe2d4003520 con 0x7fe2e80b9060 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 >> v2:172.21.15.146:3300/0 conn(0x7fe2e80d0950 msgr2=0x7fe2e80d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 --2- 172.21.15.72:0/3721229601 >> v2:172.21.15.146:3300/0 conn(0x7fe2e80d0950 0x7fe2e80d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 >> v1:172.21.15.146:6789/0 conn(0x7fe2e815e150 legacy=0x7fe2e80cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e80d2460 con 0x7fe2e80b9060 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe2e4001670 con 0x7fe2e80b9060 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 >> v1:172.21.15.72:6789/0 conn(0x7fe2e80b9060 legacy=0x7fe2e80b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2d40039b0 0x7fe2d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40051c0 0x7fe2d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d400c130 con 0x7fe2d40039b0 2024-03-06T07:26:50.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d400c2e0 con 0x7fe2d40051c0 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d400c4e0 con 0x7fe2d4004800 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2d40039b0 0x7fe2d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2edbed640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40051c0 0x7fe2d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ecbeb640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ed3ec640 1 -- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2d4004800 msgr2=0x7fe2d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ed3ec640 1 -- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40051c0 msgr2=0x7fe2d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.052+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40051c0 0x7fe2d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2ed3ec640 1 -- 172.21.15.72:0/3721229601 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d4003520 con 0x7fe2d40039b0 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2edbed640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40051c0 0x7fe2d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2edbed640 1 -- 172.21.15.72:0/3721229601 reap_dead start 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2d40039b0 0x7fe2d4003dd0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0001e50 tx=0x7fe2d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541df20b2384f50f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0001a70 con 0x7fe2d40039b0 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e0003b70 con 0x7fe2d40039b0 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.053+0000 7fe2dbfff640 1 -- 172.21.15.72:0/3721229601 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e00050d0 con 0x7fe2d40039b0 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 -- 172.21.15.72:0/3721229601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2d40039b0 msgr2=0x7fe2d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2d40039b0 0x7fe2d4003dd0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0001e50 tx=0x7fe2d4005830 comp rx=0 tx=0).stop 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 -- 172.21.15.72:0/3721229601 shutdown_connections 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 --2- 172.21.15.72:0/3721229601 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2d40039b0 0x7fe2d4003dd0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 -- 172.21.15.72:0/3721229601 >> 172.21.15.72:0/3721229601 conn(0x7fe2e809d230 msgr2=0x7fe2e80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 -- 172.21.15.72:0/3721229601 shutdown_connections 2024-03-06T07:26:50.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.054+0000 7fe2ef650640 1 -- 172.21.15.72:0/3721229601 wait complete. 2024-03-06T07:26:50.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.056+0000 7fe2ef650640 1 Processor -- start 2024-03-06T07:26:50.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.056+0000 7fe2ef650640 1 -- start start 2024-03-06T07:26:50.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ef650640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40039b0 0x7fe2e81e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ef650640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2e80c94a0 0x7fe2e80c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ef650640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 0x7fe2e8160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ef650640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80d2f40 con 0x7fe2e815fff0 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ef650640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80d2ab0 con 0x7fe2d40039b0 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ef650640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e80d1b30 con 0x7fe2e80c94a0 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 0x7fe2e8160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 0x7fe2e8160470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58562/0 (socket says 172.21.15.72:58562) 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 -- 172.21.15.72:0/2821023422 learned_addr learned my addr 172.21.15.72:0/2821023422 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40039b0 0x7fe2e81e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ecbeb640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2e80c94a0 0x7fe2e80c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 -- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2e80c94a0 msgr2=0x7fe2e80c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2e80c94a0 0x7fe2e80c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 -- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40039b0 msgr2=0x7fe2e81e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40039b0 0x7fe2e81e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 -- 172.21.15.72:0/2821023422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e81f7ae0 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40039b0 0x7fe2e81e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2edbed640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 0x7fe2e8160470 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe2e808c8a0 tx=0x7fe2e400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400c590 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.057+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e4018070 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.058+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e80c9e60 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.058+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400c950 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.058+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e81f8030 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.059+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e8069ee0 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.059+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe2e400caf0 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.060+0000 7fe2d9ffb640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.060+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe2e4089ad0 con 0x7fe2e815fff0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.060+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.060+0000 7fe2ed3ec640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0034000 tx=0x7fe2e0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.068+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2e80c9e60 con 0x7fe2e815fff0 2024-03-06T07:26:50.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.279+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2e80ce400 con 0x7fe2e815fff0 2024-03-06T07:26:50.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.279+0000 7fe2d9ffb640 1 -- 172.21.15.72:0/2821023422 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4977 (secure 0 0 0) 0x7fe2e4022250 con 0x7fe2e815fff0 2024-03-06T07:26:50.281 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:50.282 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:26:06.088200+0000","modified":"2024-03-06T07:26:23.848449+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7599},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7599":{"gid":7599,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":6,"addr":"172.21.15.72:6835/4195924898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4195924898},{"type":"v1","addr":"172.21.15.72:6835","nonce":4195924898}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7599,"qdb_cluster":[7599]},"id":7}]} 2024-03-06T07:26:50.282 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 57 2024-03-06T07:26:50.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe2cc06b500 msgr2=0x7fe2cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0034000 tx=0x7fe2e0009030 comp rx=0 tx=0).stop 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 msgr2=0x7fe2e8160470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 0x7fe2e8160470 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fe2e808c8a0 tx=0x7fe2e400eb90 comp rx=0 tx=0).stop 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 shutdown_connections 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe2cc06b500 0x7fe2cc06d9c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2e815fff0 0x7fe2e8160470 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2e80c94a0 0x7fe2e80c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 --2- 172.21.15.72:0/2821023422 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2d40039b0 0x7fe2e81e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 >> 172.21.15.72:0/2821023422 conn(0x7fe2e809d230 msgr2=0x7fe2e80b9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 shutdown_connections 2024-03-06T07:26:50.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.282+0000 7fe2ef650640 1 -- 172.21.15.72:0/2821023422 wait complete. 2024-03-06T07:26:50.296 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:50.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.394+0000 7faf7bb8a640 1 Processor -- start 2024-03-06T07:26:50.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7bb8a640 1 -- start start 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7bb8a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faf740d0950 0x7faf740d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7bb8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faf740b9060 0x7faf740b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7bb8a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf740d1ea0 con 0x7faf7415e150 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7bb8a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf740d12f0 con 0x7faf740b9060 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7bb8a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf740d2ab0 con 0x7faf740d0950 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.395+0000 7faf7a127640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faf740b9060 0x7faf740b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faf740b9060 0x7faf740b94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58578/0 (socket says 172.21.15.72:58578) 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 -- 172.21.15.72:0/544203573 learned_addr learned my addr 172.21.15.72:0/544203573 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:50.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf79125640 1 --2- 172.21.15.72:0/544203573 >> v2:172.21.15.146:3300/0 conn(0x7faf740d0950 0x7faf740d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 -- 172.21.15.72:0/544203573 >> v2:172.21.15.146:3300/0 conn(0x7faf740d0950 msgr2=0x7faf740d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 --2- 172.21.15.72:0/544203573 >> v2:172.21.15.146:3300/0 conn(0x7faf740d0950 0x7faf740d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 -- 172.21.15.72:0/544203573 >> v1:172.21.15.146:6789/0 conn(0x7faf7415e150 legacy=0x7faf740cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 -- 172.21.15.72:0/544203573 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf740d2180 con 0x7faf740b9060 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.396+0000 7faf7a127640 1 --2- 172.21.15.72:0/544203573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf740b94d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7faf740b6b50 tx=0x7faf6c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b0f26fb938300a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf78924640 1 -- 172.21.15.72:0/544203573 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c02c9f0 con 0x7faf740b9060 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf78924640 1 -- 172.21.15.72:0/544203573 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c02cb90 con 0x7faf740b9060 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf78924640 1 -- 172.21.15.72:0/544203573 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0343f0 con 0x7faf740b9060 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 -- 172.21.15.72:0/544203573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 msgr2=0x7faf740b94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/544203573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf740b94d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7faf740b6b50 tx=0x7faf6c02be50 comp rx=0 tx=0).stop 2024-03-06T07:26:50.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 -- 172.21.15.72:0/544203573 shutdown_connections 2024-03-06T07:26:50.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/544203573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf740b94d0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/544203573 >> v2:172.21.15.146:3300/0 conn(0x7faf740d0950 0x7faf740d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 -- 172.21.15.72:0/544203573 >> 172.21.15.72:0/544203573 conn(0x7faf7409d230 msgr2=0x7faf740d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 -- 172.21.15.72:0/544203573 shutdown_connections 2024-03-06T07:26:50.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.397+0000 7faf7bb8a640 1 -- 172.21.15.72:0/544203573 wait complete. 2024-03-06T07:26:50.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 Processor -- start 2024-03-06T07:26:50.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 -- start start 2024-03-06T07:26:50.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf741533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf740d0950 0x7faf740b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7415e150 0x7faf740c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf740d2f40 con 0x7faf740b9060 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf740d2740 con 0x7faf740d0950 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf7bb8a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf740d1b30 con 0x7faf7415e150 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf79926640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf741533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf79926640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf741533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58590/0 (socket says 172.21.15.72:58590) 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.399+0000 7faf79926640 1 -- 172.21.15.72:0/3302569522 learned_addr learned my addr 172.21.15.72:0/3302569522 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79125640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf740d0950 0x7faf740b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf7a127640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7415e150 0x7faf740c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79926640 1 -- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7415e150 msgr2=0x7faf740c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79926640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7415e150 0x7faf740c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79926640 1 -- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf740d0950 msgr2=0x7faf740b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79926640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf740d0950 0x7faf740b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79926640 1 -- 172.21.15.72:0/3302569522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf741f42e0 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79125640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf740d0950 0x7faf740b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf79926640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf741533f0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b3f0 tx=0x7faf7000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70015020 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf70011830 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70011c20 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf740b36c0 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.400+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf740b3c80 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.401+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf74069ee0 con 0x7faf740b9060 2024-03-06T07:26:50.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.402+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faf70018020 con 0x7faf740b9060 2024-03-06T07:26:50.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.402+0000 7faf6a7fc640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faf5806b4d0 0x7faf5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.402+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7faf70089e90 con 0x7faf740b9060 2024-03-06T07:26:50.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.402+0000 7faf79125640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faf5806b4d0 0x7faf5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.403+0000 7faf79125640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faf5806b4d0 0x7faf5806d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faf7415cab0 tx=0x7faf64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.406+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faf7001f030 con 0x7faf740b9060 2024-03-06T07:26:50.454 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:50.456 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:26:50.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.591+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf740d4670 con 0x7faf740b9060 2024-03-06T07:26:50.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.592+0000 7faf6a7fc640 1 -- 172.21.15.72:0/3302569522 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17007 (secure 0 0 0) 0x7faf700172c0 con 0x7faf740b9060 2024-03-06T07:26:50.595 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:50.595 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":98,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:19.829591+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:26:04.296462+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-06T07:26:05.055015+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6835/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6836/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6835/209452807":"2024-03-07T07:26:13.880837+0000","172.21.15.72:6834/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6833/209452807":"2024-03-07T07:26:13.880837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:50.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faf5806b4d0 msgr2=0x7faf5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faf5806b4d0 0x7faf5806d990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7faf7415cab0 tx=0x7faf64008040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 msgr2=0x7faf741533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf741533f0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b3f0 tx=0x7faf7000b8b0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 shutdown_connections 2024-03-06T07:26:50.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faf5806b4d0 0x7faf5806d990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf7415e150 0x7faf740c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf740d0950 0x7faf740b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 --2- 172.21.15.72:0/3302569522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faf740b9060 0x7faf741533f0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 >> 172.21.15.72:0/3302569522 conn(0x7faf7409d230 msgr2=0x7faf740b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 shutdown_connections 2024-03-06T07:26:50.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.594+0000 7faf7bb8a640 1 -- 172.21.15.72:0/3302569522 wait complete. 2024-03-06T07:26:50.608 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:26:50.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.707+0000 7fb1edbbc640 1 Processor -- start 2024-03-06T07:26:50.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.707+0000 7fb1edbbc640 1 -- start start 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1edbbc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1e80d0950 0x7fb1e80d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1edbbc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1e80b9060 0x7fb1e80b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1edbbc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80d1ea0 con 0x7fb1e815e150 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1edbbc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e80d12f0 con 0x7fb1e80b9060 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1edbbc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1e80d2ab0 con 0x7fb1e80d0950 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1e80b9060 0x7fb1e80b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1e80b9060 0x7fb1e80b94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58594/0 (socket says 172.21.15.72:58594) 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2375610725 learned_addr learned my addr 172.21.15.72:0/2375610725 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2375610725 >> v2:172.21.15.146:3300/0 conn(0x7fb1e80d0950 msgr2=0x7fb1e80d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e6ffd640 1 --2- 172.21.15.72:0/2375610725 >> v2:172.21.15.146:3300/0 conn(0x7fb1e80d0950 0x7fb1e80d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 --2- 172.21.15.72:0/2375610725 >> v2:172.21.15.146:3300/0 conn(0x7fb1e80d0950 0x7fb1e80d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2375610725 >> v1:172.21.15.146:6789/0 conn(0x7fb1e815e150 legacy=0x7fb1e80cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.709+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2375610725 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80d2180 con 0x7fb1e80b9060 2024-03-06T07:26:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1e7fff640 1 --2- 172.21.15.72:0/2375610725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e80b9060 0x7fb1e80b94d0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80b6b50 tx=0x7fb1e002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cc4102f0521e9d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1e67fc640 1 -- 172.21.15.72:0/2375610725 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e002c9f0 con 0x7fb1e80b9060 2024-03-06T07:26:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1e67fc640 1 -- 172.21.15.72:0/2375610725 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e002cb90 con 0x7fb1e80b9060 2024-03-06T07:26:50.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1e67fc640 1 -- 172.21.15.72:0/2375610725 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00343f0 con 0x7fb1e80b9060 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2375610725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e80b9060 msgr2=0x7fb1e80b94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2375610725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e80b9060 0x7fb1e80b94d0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80b6b50 tx=0x7fb1e002be50 comp rx=0 tx=0).stop 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2375610725 shutdown_connections 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2375610725 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e80b9060 0x7fb1e80b94d0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2375610725 >> v2:172.21.15.146:3300/0 conn(0x7fb1e80d0950 0x7fb1e80d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.710+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2375610725 >> 172.21.15.72:0/2375610725 conn(0x7fb1e809d230 msgr2=0x7fb1e80d3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.711+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2375610725 shutdown_connections 2024-03-06T07:26:50.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.711+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2375610725 wait complete. 2024-03-06T07:26:50.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.712+0000 7fb1edbbc640 1 Processor -- start 2024-03-06T07:26:50.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.712+0000 7fb1edbbc640 1 -- start start 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e80b9060 0x7fb1e815a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e80d0950 0x7fb1e80cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 0x7fb1e80cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80d2f40 con 0x7fb1e815e150 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80d2740 con 0x7fb1e80b9060 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80d1b30 con 0x7fb1e80d0950 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 0x7fb1e80cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 0x7fb1e80cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58606/0 (socket says 172.21.15.72:58606) 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2925293592 learned_addr learned my addr 172.21.15.72:0/2925293592 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:50.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e80d0950 msgr2=0x7fb1e80cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e80d0950 0x7fb1e80cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e80b9060 msgr2=0x7fb1e815a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e77fe640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e80b9060 0x7fb1e815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e80b9060 0x7fb1e815a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 -- 172.21.15.72:0/2925293592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80cc400 con 0x7fb1e815e150 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1e7fff640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 0x7fb1e80cd360 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0036040 tx=0x7fb1e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0043020 con 0x7fb1e815e150 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e003e450 con 0x7fb1e815e150 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e80cc6e0 con 0x7fb1e815e150 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.714+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e003e840 con 0x7fb1e815e150 2024-03-06T07:26:50.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.715+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e80c7840 con 0x7fb1e815e150 2024-03-06T07:26:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.716+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb1e0005d10 con 0x7fb1e815e150 2024-03-06T07:26:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.716+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e8069ee0 con 0x7fb1e815e150 2024-03-06T07:26:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.716+0000 7fb1cffff640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1d406b500 0x7fb1d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.716+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb1e00ace80 con 0x7fb1e815e150 2024-03-06T07:26:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.716+0000 7fb1e77fe640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1d406b500 0x7fb1d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:50.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.717+0000 7fb1e77fe640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1d406b500 0x7fb1d406d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800a730 tx=0x7fb1d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:50.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.720+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1e007bc00 con 0x7fb1e815e150 2024-03-06T07:26:50.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.936+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb1e80d4670 con 0x7fb1e815e150 2024-03-06T07:26:50.940 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:26:50.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.938+0000 7fb1cffff640 1 -- 172.21.15.72:0/2925293592 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb1e007b500 con 0x7fb1e815e150 2024-03-06T07:26:50.972 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:26:50.936+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:26:50.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1d406b500 msgr2=0x7fb1d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1d406b500 0x7fb1d406d9c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800a730 tx=0x7fb1d8008040 comp rx=0 tx=0).stop 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 msgr2=0x7fb1e80cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 0x7fb1e80cd360 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0036040 tx=0x7fb1e0004a40 comp rx=0 tx=0).stop 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 shutdown_connections 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb1d406b500 0x7fb1d406d9c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1e815e150 0x7fb1e80cd360 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1e80d0950 0x7fb1e80cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 --2- 172.21.15.72:0/2925293592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1e80b9060 0x7fb1e815a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 >> 172.21.15.72:0/2925293592 conn(0x7fb1e809d230 msgr2=0x7fb1e80cf4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 shutdown_connections 2024-03-06T07:26:50.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:50.940+0000 7fb1edbbc640 1 -- 172.21.15.72:0/2925293592 wait complete. 2024-03-06T07:26:50.985 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:26:51.015 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:26:51.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.085+0000 7fbdcd181640 1 Processor -- start 2024-03-06T07:26:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.085+0000 7fbdcd181640 1 -- start start 2024-03-06T07:26:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdcd181640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbdc80ce5a0 0x7fbdc80b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdcd181640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc80ce980 0x7fbdc80cede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdcd181640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdc80ccbd0 con 0x7fbdc8070d70 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdcd181640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdc80b3b00 con 0x7fbdc80ce5a0 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdcd181640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80b3de0 con 0x7fbdc80ce980 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdc6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbdc80ce5a0 0x7fbdc80b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdc6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbdc80ce5a0 0x7fbdc80b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58632/0 (socket says 172.21.15.72:58632) 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.086+0000 7fbdc6d9d640 1 -- 172.21.15.72:0/2650921288 learned_addr learned my addr 172.21.15.72:0/2650921288 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdc6d9d640 1 -- 172.21.15.72:0/2650921288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc80ce980 msgr2=0x7fbdc80cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdbe59c640 1 --2- 172.21.15.72:0/2650921288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc80ce980 0x7fbdc80cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdc6d9d640 1 --2- 172.21.15.72:0/2650921288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc80ce980 0x7fbdc80cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdc6d9d640 1 -- 172.21.15.72:0/2650921288 >> v1:172.21.15.146:6789/0 conn(0x7fbdc8070d70 legacy=0x7fbdc80711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdc6d9d640 1 -- 172.21.15.72:0/2650921288 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80b40c0 con 0x7fbdc80ce5a0 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdc6d9d640 1 --2- 172.21.15.72:0/2650921288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80b4bf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009870 tx=0x7fbdb802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7602d94592345c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.087+0000 7fbdc659c640 1 -- 172.21.15.72:0/2650921288 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8035020 con 0x7fbdc80ce5a0 2024-03-06T07:26:51.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdc659c640 1 -- 172.21.15.72:0/2650921288 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdb8031640 con 0x7fbdc80ce5a0 2024-03-06T07:26:51.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdc659c640 1 -- 172.21.15.72:0/2650921288 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb802d040 con 0x7fbdc80ce5a0 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 -- 172.21.15.72:0/2650921288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce5a0 msgr2=0x7fbdc80b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 --2- 172.21.15.72:0/2650921288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80b4bf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8009870 tx=0x7fbdb802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 -- 172.21.15.72:0/2650921288 shutdown_connections 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 --2- 172.21.15.72:0/2650921288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc80ce980 0x7fbdc80cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 --2- 172.21.15.72:0/2650921288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80b4bf0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 -- 172.21.15.72:0/2650921288 >> 172.21.15.72:0/2650921288 conn(0x7fbdc806b160 msgr2=0x7fbdc806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 -- 172.21.15.72:0/2650921288 shutdown_connections 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.088+0000 7fbdcd181640 1 -- 172.21.15.72:0/2650921288 wait complete. 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.089+0000 7fbdcd181640 1 Processor -- start 2024-03-06T07:26:51.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 -- start start 2024-03-06T07:26:51.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc8070d70 0x7fbdc81560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 0x7fbdc80c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80cc9b0 con 0x7fbdc80ce980 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80cc830 con 0x7fbdc80ce5a0 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdcd181640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc80cceb0 con 0x7fbdc8070d70 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdc759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 0x7fbdc80c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdc759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 0x7fbdc80c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58648/0 (socket says 172.21.15.72:58648) 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdc759e640 1 -- 172.21.15.72:0/458188962 learned_addr learned my addr 172.21.15.72:0/458188962 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdc6d9d640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc8070d70 0x7fbdc81560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.090+0000 7fbdbe59c640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdc759e640 1 -- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc8070d70 msgr2=0x7fbdc81560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdc759e640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc8070d70 0x7fbdc81560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdc759e640 1 -- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdc80ce5a0 msgr2=0x7fbdc80c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdc759e640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdc759e640 1 -- 172.21.15.72:0/458188962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80c67a0 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdc759e640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 0x7fbdc80c7780 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c7b0 tx=0x7fbdc000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0004b60 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc0004d00 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc80c8740 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000aa90 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.091+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc80c8d00 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.092+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc8069ee0 con 0x7fbdc80ce980 2024-03-06T07:26:51.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.093+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbdc000ac30 con 0x7fbdc80ce980 2024-03-06T07:26:51.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.093+0000 7fbdbffff640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbda806b4d0 0x7fbda806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:51.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.093+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6053+0+0 (secure 0 0 0) 0x7fbdc008a470 con 0x7fbdc80ce980 2024-03-06T07:26:51.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.093+0000 7fbdc6d9d640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbda806b4d0 0x7fbda806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:51.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.093+0000 7fbdc6d9d640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbda806b4d0 0x7fbda806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8036000 tx=0x7fbdb803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:51.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.097+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbdc0020070 con 0x7fbdc80ce980 2024-03-06T07:26:51.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.309+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbdc80b36d0 con 0x7fbdc80ce980 2024-03-06T07:26:51.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.939+0000 7fbdbffff640 1 -- 172.21.15.72:0/458188962 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbdc0058b40 con 0x7fbdc80ce980 2024-03-06T07:26:51.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbda806b4d0 msgr2=0x7fbda806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbda806b4d0 0x7fbda806d990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8036000 tx=0x7fbdb803c040 comp rx=0 tx=0).stop 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 msgr2=0x7fbdc80c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 0x7fbdc80c7780 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000c7b0 tx=0x7fbdc000cc70 comp rx=0 tx=0).stop 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 shutdown_connections 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbda806b4d0 0x7fbda806d990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdc80ce980 0x7fbdc80c7780 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdc80ce5a0 0x7fbdc80c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 --2- 172.21.15.72:0/458188962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdc8070d70 0x7fbdc81560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 >> 172.21.15.72:0/458188962 conn(0x7fbdc806b160 msgr2=0x7fbdc815c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 shutdown_connections 2024-03-06T07:26:51.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:51.942+0000 7fbdcd181640 1 -- 172.21.15.72:0/458188962 wait complete. 2024-03-06T07:26:51.958 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:26:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 Processor -- start 2024-03-06T07:26:52.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 -- start start 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d940c92b0 0x7f8d940d1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8d940ceb60 0x7f8d940ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d940d2860 con 0x7f8d9415e1d0 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d940d2b40 con 0x7f8d940c92b0 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.066+0000 7f8d999e2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d940d2e20 con 0x7f8d940ceb60 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d8bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8d940c92b0 0x7f8d940d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d92ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8d9415e1d0 0x7f8d940d0940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45636/0 (socket says 172.21.15.72:45636) 2024-03-06T07:26:52.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d92ffd640 1 -- 172.21.15.72:0/3909251613 learned_addr learned my addr 172.21.15.72:0/3909251613 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:52.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3909251613 >> v2:172.21.15.146:3300/0 conn(0x7f8d940ceb60 msgr2=0x7f8d940ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:52.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/3909251613 >> v2:172.21.15.146:3300/0 conn(0x7f8d940ceb60 0x7f8d940ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3909251613 >> v1:172.21.15.72:6789/0 conn(0x7f8d9415e1d0 legacy=0x7f8d940d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d927fc640 1 -- 172.21.15.72:0/3909251613 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177681889 0 0) 0x7f8d940d2860 con 0x7f8d9415e1d0 2024-03-06T07:26:52.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.067+0000 7f8d8bfff640 1 -- 172.21.15.72:0/3909251613 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940c7aa0 con 0x7f8d940c92b0 2024-03-06T07:26:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/3909251613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940d1050 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8d940d4f90 tx=0x7f8d7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ba9a385adc8948 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d927fc640 1 -- 172.21.15.72:0/3909251613 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c0315a0 con 0x7f8d940c92b0 2024-03-06T07:26:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d927fc640 1 -- 172.21.15.72:0/3909251613 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c037070 con 0x7f8d940c92b0 2024-03-06T07:26:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d927fc640 1 -- 172.21.15.72:0/3909251613 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c031960 con 0x7f8d940c92b0 2024-03-06T07:26:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 -- 172.21.15.72:0/3909251613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 msgr2=0x7f8d940d1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 --2- 172.21.15.72:0/3909251613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940d1050 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f8d940d4f90 tx=0x7f8d7c02b840 comp rx=0 tx=0).stop 2024-03-06T07:26:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 -- 172.21.15.72:0/3909251613 shutdown_connections 2024-03-06T07:26:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 --2- 172.21.15.72:0/3909251613 >> v2:172.21.15.146:3300/0 conn(0x7f8d940ceb60 0x7f8d940ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 --2- 172.21.15.72:0/3909251613 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940d1050 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 -- 172.21.15.72:0/3909251613 >> 172.21.15.72:0/3909251613 conn(0x7f8d940d3f50 msgr2=0x7f8d940d4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.068+0000 7f8d999e2640 1 -- 172.21.15.72:0/3909251613 shutdown_connections 2024-03-06T07:26:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.069+0000 7f8d999e2640 1 -- 172.21.15.72:0/3909251613 wait complete. 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 Processor -- start 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 -- start start 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d940ceb60 0x7f8d940b8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d9415e1d0 0x7f8d941f3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940d2530 con 0x7f8d940c92b0 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9415e5b0 con 0x7f8d9415e1d0 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d999e2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d940d1590 con 0x7f8d940ceb60 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d92ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58670/0 (socket says 172.21.15.72:58670) 2024-03-06T07:26:52.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.070+0000 7f8d92ffd640 1 -- 172.21.15.72:0/1202369784 learned_addr learned my addr 172.21.15.72:0/1202369784 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d937fe640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d9415e1d0 0x7f8d941f3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d940ceb60 0x7f8d940b8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d92ffd640 1 -- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d940ceb60 msgr2=0x7f8d940b8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d92ffd640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d940ceb60 0x7f8d940b8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d92ffd640 1 -- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d9415e1d0 msgr2=0x7f8d941f3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d92ffd640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d9415e1d0 0x7f8d941f3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d92ffd640 1 -- 172.21.15.72:0/1202369784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940b0e10 con 0x7f8d940c92b0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d940ceb60 0x7f8d940b8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d92ffd640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940b2e80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400c430 tx=0x7f8d8400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d840074e0 con 0x7f8d940c92b0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d840047e0 con 0x7f8d940c92b0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d941f4b30 con 0x7f8d940c92b0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d84004bd0 con 0x7f8d940c92b0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.071+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d940cc350 con 0x7f8d940c92b0 2024-03-06T07:26:52.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.072+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d94069ee0 con 0x7f8d940c92b0 2024-03-06T07:26:52.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.072+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8d84011070 con 0x7f8d940c92b0 2024-03-06T07:26:52.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.073+0000 7f8d90ff9640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7406b4d0 0x7f8d7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:52.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.073+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6053+0+0 (secure 0 0 0) 0x7f8d84089f40 con 0x7f8d940c92b0 2024-03-06T07:26:52.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.073+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7406b4d0 0x7f8d7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:52.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.073+0000 7f8d8bfff640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7406b4d0 0x7f8d7406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c036000 tx=0x7f8d7c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:52.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.077+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8d941f4b30 con 0x7f8d940c92b0 2024-03-06T07:26:52.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:52.247+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8d940ceff0 con 0x7f8d940c92b0 2024-03-06T07:26:53.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.110+0000 7f8d90ff9640 1 -- 172.21.15.72:0/1202369784 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8d84058cf0 con 0x7f8d940c92b0 2024-03-06T07:26:53.112 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:26:53.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7406b4d0 msgr2=0x7f8d7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7406b4d0 0x7f8d7406d990 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c036000 tx=0x7f8d7c008030 comp rx=0 tx=0).stop 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 msgr2=0x7f8d940b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940b2e80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8d8400c430 tx=0x7f8d8400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 shutdown_connections 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8d7406b4d0 0x7f8d7406d990 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d9415e1d0 0x7f8d941f3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d940ceb60 0x7f8d940b8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 --2- 172.21.15.72:0/1202369784 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8d940c92b0 0x7f8d940b2e80 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.115+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 >> 172.21.15.72:0/1202369784 conn(0x7f8d940d3f50 msgr2=0x7f8d940cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.116+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 shutdown_connections 2024-03-06T07:26:53.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.116+0000 7f8d999e2640 1 -- 172.21.15.72:0/1202369784 wait complete. 2024-03-06T07:26:53.132 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:26:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.243+0000 7fd1e8066640 1 Processor -- start 2024-03-06T07:26:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.244+0000 7fd1e8066640 1 -- start start 2024-03-06T07:26:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.244+0000 7fd1e8066640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd1e00cd690 0x7fd1e0162570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.244+0000 7fd1e8066640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1e00b2da0 0x7fd1e00cff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.244+0000 7fd1e8066640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e0162c70 con 0x7fd1e0161670 2024-03-06T07:26:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.244+0000 7fd1e8066640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e00b3590 con 0x7fd1e00b2da0 2024-03-06T07:26:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.244+0000 7fd1e8066640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1e006cd80 con 0x7fd1e00cd690 2024-03-06T07:26:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e5e02640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd1e0161670 0x7fd1e0161e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58242/0 (socket says 172.21.15.72:58242) 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e5e02640 1 -- 172.21.15.72:0/1214746076 learned_addr learned my addr 172.21.15.72:0/1214746076 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e6603640 1 --2- 172.21.15.72:0/1214746076 >> v2:172.21.15.72:3300/0 conn(0x7fd1e00b2da0 0x7fd1e00cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e5601640 1 --2- 172.21.15.72:0/1214746076 >> v2:172.21.15.146:3300/0 conn(0x7fd1e00cd690 0x7fd1e0162570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e6603640 1 -- 172.21.15.72:0/1214746076 >> v2:172.21.15.146:3300/0 conn(0x7fd1e00cd690 msgr2=0x7fd1e0162570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e6603640 1 --2- 172.21.15.72:0/1214746076 >> v2:172.21.15.146:3300/0 conn(0x7fd1e00cd690 0x7fd1e0162570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e6603640 1 -- 172.21.15.72:0/1214746076 >> v1:172.21.15.72:6789/0 conn(0x7fd1e0161670 legacy=0x7fd1e0161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e4e00640 1 -- 172.21.15.72:0/1214746076 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447400857 0 0) 0x7fd1e0162c70 con 0x7fd1e0161670 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e6603640 1 -- 172.21.15.72:0/1214746076 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e006d060 con 0x7fd1e00b2da0 2024-03-06T07:26:53.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.245+0000 7fd1e5601640 1 --2- 172.21.15.72:0/1214746076 >> v2:172.21.15.146:3300/0 conn(0x7fd1e00cd690 0x7fd1e0162570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:53.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.246+0000 7fd1e6603640 1 --2- 172.21.15.72:0/1214746076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e00b2da0 0x7fd1e00cff50 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80097a0 tx=0x7fd1d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1ef72eaf71e4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.246+0000 7fd1e4e00640 1 -- 172.21.15.72:0/1214746076 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8035020 con 0x7fd1e00b2da0 2024-03-06T07:26:53.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.246+0000 7fd1e4e00640 1 -- 172.21.15.72:0/1214746076 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d8031830 con 0x7fd1e00b2da0 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.246+0000 7fd1e4e00640 1 -- 172.21.15.72:0/1214746076 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8031c00 con 0x7fd1e00b2da0 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 -- 172.21.15.72:0/1214746076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e00b2da0 msgr2=0x7fd1e00cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 --2- 172.21.15.72:0/1214746076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e00b2da0 0x7fd1e00cff50 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80097a0 tx=0x7fd1d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 -- 172.21.15.72:0/1214746076 shutdown_connections 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 --2- 172.21.15.72:0/1214746076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e00b2da0 0x7fd1e00cff50 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 --2- 172.21.15.72:0/1214746076 >> v2:172.21.15.146:3300/0 conn(0x7fd1e00cd690 0x7fd1e0162570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 -- 172.21.15.72:0/1214746076 >> 172.21.15.72:0/1214746076 conn(0x7fd1e00d1d50 msgr2=0x7fd1e00d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 -- 172.21.15.72:0/1214746076 shutdown_connections 2024-03-06T07:26:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.247+0000 7fd1e8066640 1 -- 172.21.15.72:0/1214746076 wait complete. 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.248+0000 7fd1e8066640 1 Processor -- start 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.248+0000 7fd1e8066640 1 -- start start 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1e00b2da0 0x7fd1e01f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1e00cd690 0x7fd1e00c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 0x7fd1e01e9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e006d4c0 con 0x7fd1e0161670 2024-03-06T07:26:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e006d340 con 0x7fd1e00b2da0 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e0161a50 con 0x7fd1e00cd690 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 0x7fd1e01e9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 0x7fd1e01e9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48084/0 (socket says 172.21.15.72:48084) 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 -- 172.21.15.72:0/683464876 learned_addr learned my addr 172.21.15.72:0/683464876 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e5e02640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1e00b2da0 0x7fd1e01f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e5601640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1e00cd690 0x7fd1e00c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 -- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1e00cd690 msgr2=0x7fd1e00c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1e00cd690 0x7fd1e00c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 -- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1e00b2da0 msgr2=0x7fd1e01f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1e00b2da0 0x7fd1e01f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 -- 172.21.15.72:0/683464876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e01e9f90 con 0x7fd1e0161670 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e5601640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1e00cd690 0x7fd1e00c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e6603640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 0x7fd1e01e9a50 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8034040 tx=0x7fd1d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8040070 con 0x7fd1e0161670 2024-03-06T07:26:53.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1d80398e0 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e01f47a0 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8035040 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.249+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e01f4d60 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.250+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e0162990 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.251+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd1d804b060 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.251+0000 7fd1d67fc640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1c406b4d0 0x7fd1c406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.251+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5615+0+0 (secure 0 0 0) 0x7fd1d80aba80 con 0x7fd1e0161670 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.251+0000 7fd1e5e02640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1c406b4d0 0x7fd1c406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:53.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.252+0000 7fd1e5e02640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1c406b4d0 0x7fd1c406d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0012e0 tx=0x7fd1dc007ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:53.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.255+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1e0162990 con 0x7fd1e0161670 2024-03-06T07:26:53.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:53.426+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd1e0068050 con 0x7fd1e0161670 2024-03-06T07:26:54.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.115+0000 7fd1d67fc640 1 -- 172.21.15.72:0/683464876 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd1d807a960 con 0x7fd1e0161670 2024-03-06T07:26:54.117 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1c406b4d0 msgr2=0x7fd1c406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1c406b4d0 0x7fd1c406d990 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0012e0 tx=0x7fd1dc007ed0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 msgr2=0x7fd1e01e9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 0x7fd1e01e9a50 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8034040 tx=0x7fd1d80046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 shutdown_connections 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd1c406b4d0 0x7fd1c406d990 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1e0161670 0x7fd1e01e9a50 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1e00cd690 0x7fd1e00c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 --2- 172.21.15.72:0/683464876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1e00b2da0 0x7fd1e01f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 >> 172.21.15.72:0/683464876 conn(0x7fd1e00d1d50 msgr2=0x7fd1e015ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 shutdown_connections 2024-03-06T07:26:54.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.118+0000 7fd1e8066640 1 -- 172.21.15.72:0/683464876 wait complete. 2024-03-06T07:26:54.138 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:54.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:54.191 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.192 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:54.193 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:26:54.194 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:26:54.195 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:26:54.196 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:26:54.197 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.198 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:26:54.199 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:54.199 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:54.199 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.199 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.199 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:54.199 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.200 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:26:54.201 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:26:54.202 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.203 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_files = 6 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.204 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:26:54.205 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.206 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:54.207 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.208 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:54.209 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:26:54.210 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:54.211 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.212 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:26:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.212 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:54.213 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.214 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.215 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:26:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:26:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-06T07:26:54.217 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_ops = 100000000 2024-03-06T07:26:54.219 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:54.261 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.310 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:54.311 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.312 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:54.313 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.314 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:26:54.315 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:54.316 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:54.317 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:26:54.318 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:26:54.319 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:26:54.320 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi072.stdout: mds_max_purge_files = 6 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:26:54.321 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.322 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.323 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:54.324 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.325 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:26:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:26:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:54.325 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:26:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:26:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.326 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:26:54.327 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:26:54.328 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:26:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:26:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:26:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:26:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:26:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:26:54.329 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:26:54.330 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:54.331 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:54.332 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:26:54.333 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.334 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.335 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.336 INFO:teuthology.orchestra.run.smithi146.stdout: mds_max_purge_files = 6 2024-03-06T07:26:54.338 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:54.380 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-06T07:26:54.428 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-06T07:26:54.428 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-06T07:26:54.428 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:54.428 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-06T07:26:54.428 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:54.429 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.430 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.431 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-06T07:26:54.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: bdev async discard = True 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: bdev enable discard = True 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore allocator = bitmap 2024-03-06T07:26:54.433 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 1/20 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 1/20 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 4/10 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.434 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-06T07:26:54.435 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.436 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.437 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-03-06T07:26:54.438 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-03-06T07:26:54.439 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.440 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-03-06T07:26:54.441 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.72,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = 0927e84a-942e-4592-98d9-697c1cf6243f 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-03-06T07:26:54.442 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-03-06T07:26:54.443 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-03-06T07:26:54.444 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-03-06T07:26:54.445 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.446 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-06T07:26:54.447 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-06T07:26:54.448 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-03-06T07:26:54.449 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-03-06T07:26:54.450 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-03-06T07:26:54.452 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:26:54.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.593+0000 7f0eb1cc9640 1 Processor -- start 2024-03-06T07:26:54.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.593+0000 7f0eb1cc9640 1 -- start start 2024-03-06T07:26:54.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eb1cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac066d90 0x7f0eac0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eb1cc9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0eac0b5410 0x7f0eac0ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eb1cc9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac06b710 con 0x7f0eac0cf9a0 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eb1cc9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eac0b61c0 con 0x7f0eac0b5410 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eb1cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac06b150 con 0x7f0eac066d90 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0eac0b5410 0x7f0eac0ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0eac0b5410 0x7f0eac0ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48086/0 (socket says 172.21.15.72:48086) 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 -- 172.21.15.72:0/27850817 learned_addr learned my addr 172.21.15.72:0/27850817 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 -- 172.21.15.72:0/27850817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac066d90 msgr2=0x7f0eac0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 --2- 172.21.15.72:0/27850817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac066d90 0x7f0eac0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 -- 172.21.15.72:0/27850817 >> v1:172.21.15.146:6789/0 conn(0x7f0eac0cf9a0 legacy=0x7f0eac0b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.594+0000 7f0eabfff640 1 -- 172.21.15.72:0/27850817 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac06b430 con 0x7f0eac0b5410 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.595+0000 7f0eabfff640 1 --2- 172.21.15.72:0/27850817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0ce040 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009e50 tx=0x7f0ea402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8a62099c2def5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.595+0000 7f0eaa7fc640 1 -- 172.21.15.72:0/27850817 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea402ca70 con 0x7f0eac0b5410 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.595+0000 7f0eaa7fc640 1 -- 172.21.15.72:0/27850817 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea402cc10 con 0x7f0eac0b5410 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.595+0000 7f0eaa7fc640 1 -- 172.21.15.72:0/27850817 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40343f0 con 0x7f0eac0b5410 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/27850817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 msgr2=0x7f0eac0ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/27850817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0ce040 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009e50 tx=0x7f0ea402bed0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/27850817 shutdown_connections 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/27850817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0ce040 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/27850817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac066d90 0x7f0eac0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/27850817 >> 172.21.15.72:0/27850817 conn(0x7f0eac0b1d70 msgr2=0x7f0eac0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/27850817 shutdown_connections 2024-03-06T07:26:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.596+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/27850817 wait complete. 2024-03-06T07:26:54.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.597+0000 7f0eb1cc9640 1 Processor -- start 2024-03-06T07:26:54.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.597+0000 7f0eb1cc9640 1 -- start start 2024-03-06T07:26:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac066d90 0x7f0eac1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac0cf9a0 0x7f0eac0ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0cfd80 con 0x7f0eac0b5410 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0b0c50 con 0x7f0eac066d90 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac0cff00 con 0x7f0eac0cf9a0 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0caac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48088/0 (socket says 172.21.15.72:48088) 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 -- 172.21.15.72:0/2236538220 learned_addr learned my addr 172.21.15.72:0/2236538220 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 -- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac0cf9a0 msgr2=0x7f0eac0ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac0cf9a0 0x7f0eac0ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eab7fe640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac066d90 0x7f0eac1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 -- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac066d90 msgr2=0x7f0eac1f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac066d90 0x7f0eac1f4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 -- 172.21.15.72:0/2236538220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0cb310 con 0x7f0eac0b5410 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eaaffd640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0caac0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000b450 tx=0x7f0ea000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0015020 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea0011890 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac0cb5f0 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0011c80 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.598+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac0cbbb0 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.599+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac09e3d0 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.600+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0ea0018020 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.600+0000 7f0e93fff640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e9806b4d0 0x7f0e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.600+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f0ea008a200 con 0x7f0eac0b5410 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.600+0000 7f0eab7fe640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e9806b4d0 0x7f0e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.601+0000 7f0eab7fe640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e9806b4d0 0x7f0e9806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00a730 tx=0x7f0e9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.605+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0ea001f020 con 0x7f0eac0b5410 2024-03-06T07:26:54.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.780+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0eac0cbfd0 con 0x7f0eac0b5410 2024-03-06T07:26:54.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.783+0000 7f0e93fff640 1 -- 172.21.15.72:0/2236538220 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0ea000d040 con 0x7f0eac0b5410 2024-03-06T07:26:54.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e9806b4d0 msgr2=0x7f0e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e9806b4d0 0x7f0e9806d990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c00a730 tx=0x7f0e9c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 msgr2=0x7f0eac0caac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0caac0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f0ea000b450 tx=0x7f0ea000b910 comp rx=0 tx=0).stop 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 shutdown_connections 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e9806b4d0 0x7f0e9806d990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eac0cf9a0 0x7f0eac0ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0eac0b5410 0x7f0eac0caac0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 --2- 172.21.15.72:0/2236538220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0eac066d90 0x7f0eac1f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 >> 172.21.15.72:0/2236538220 conn(0x7f0eac0b1d70 msgr2=0x7f0eac06e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 shutdown_connections 2024-03-06T07:26:54.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.785+0000 7f0eb1cc9640 1 -- 172.21.15.72:0/2236538220 wait complete. 2024-03-06T07:26:54.798 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-03-06T07:26:54.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.900+0000 7fe44bea4640 1 Processor -- start 2024-03-06T07:26:54.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.900+0000 7fe44bea4640 1 -- start start 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44bea4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe444164b90 0x7fe444166f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44bea4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe444160630 con 0x7fe444144f10 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44bea4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440cf3a0 con 0x7fe4440cbfb0 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44bea4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4440cf680 con 0x7fe444164b90 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44a441640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe444164b90 0x7fe444166f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44943f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe444144f10 0x7fe44415ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58252/0 (socket says 172.21.15.72:58252) 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44943f640 1 -- 172.21.15.72:0/39191557 learned_addr learned my addr 172.21.15.72:0/39191557 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44a441640 1 -- 172.21.15.72:0/39191557 >> v1:172.21.15.146:6789/0 conn(0x7fe4440cbfb0 legacy=0x7fe44415f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44a441640 1 -- 172.21.15.72:0/39191557 >> v1:172.21.15.72:6789/0 conn(0x7fe444144f10 legacy=0x7fe44415ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe44a441640 1 -- 172.21.15.72:0/39191557 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4440b47a0 con 0x7fe444164b90 2024-03-06T07:26:54.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.901+0000 7fe448c3e640 1 -- 172.21.15.72:0/39191557 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756687939 0 0) 0x7fe444160630 con 0x7fe444144f10 2024-03-06T07:26:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe44a441640 1 --2- 172.21.15.72:0/39191557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444164b90 0x7fe444166f80 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009dc0 tx=0x7fe43c02be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60dd8e9f603ddb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe448c3e640 1 -- 172.21.15.72:0/39191557 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c039070 con 0x7fe444164b90 2024-03-06T07:26:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe448c3e640 1 -- 172.21.15.72:0/39191557 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43c02cbb0 con 0x7fe444164b90 2024-03-06T07:26:54.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe448c3e640 1 -- 172.21.15.72:0/39191557 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c034440 con 0x7fe444164b90 2024-03-06T07:26:54.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe44bea4640 1 -- 172.21.15.72:0/39191557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444164b90 msgr2=0x7fe444166f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe44bea4640 1 --2- 172.21.15.72:0/39191557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444164b90 0x7fe444166f80 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009dc0 tx=0x7fe43c02be00 comp rx=0 tx=0).stop 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe44bea4640 1 -- 172.21.15.72:0/39191557 shutdown_connections 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe44bea4640 1 --2- 172.21.15.72:0/39191557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444164b90 0x7fe444166f80 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.902+0000 7fe44bea4640 1 -- 172.21.15.72:0/39191557 >> 172.21.15.72:0/39191557 conn(0x7fe44406b3d0 msgr2=0x7fe44406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.903+0000 7fe44bea4640 1 -- 172.21.15.72:0/39191557 shutdown_connections 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.903+0000 7fe44bea4640 1 -- 172.21.15.72:0/39191557 wait complete. 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 Processor -- start 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 -- start start 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4440cbfb0 0x7fe44415b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 0x7fe444140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe444164b90 0x7fe4441408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440cc390 con 0x7fe444144f10 2024-03-06T07:26:54.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440cf090 con 0x7fe444164b90 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44bea4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440cc510 con 0x7fe4440cbfb0 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44943f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 0x7fe444140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44943f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 0x7fe444140390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48114/0 (socket says 172.21.15.72:48114) 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.904+0000 7fe44943f640 1 -- 172.21.15.72:0/497015393 learned_addr learned my addr 172.21.15.72:0/497015393 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44a441640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe444164b90 0x7fe4441408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe449c40640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4440cbfb0 0x7fe44415b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44943f640 1 -- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4440cbfb0 msgr2=0x7fe44415b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44943f640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4440cbfb0 0x7fe44415b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44943f640 1 -- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe444164b90 msgr2=0x7fe4441408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44943f640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe444164b90 0x7fe4441408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44943f640 1 -- 172.21.15.72:0/497015393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44413c480 con 0x7fe444144f10 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44943f640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 0x7fe444140390 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fe44000c4d0 tx=0x7fe44000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440018070 con 0x7fe444144f10 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe440007500 con 0x7fe444144f10 2024-03-06T07:26:54.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440004a20 con 0x7fe444144f10 2024-03-06T07:26:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44413c760 con 0x7fe444144f10 2024-03-06T07:26:54.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.905+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44413cd20 con 0x7fe444144f10 2024-03-06T07:26:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.906+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe44404f330 con 0x7fe444144f10 2024-03-06T07:26:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.907+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe44001b070 con 0x7fe444144f10 2024-03-06T07:26:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.907+0000 7fe43a7fc640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe42806b500 0x7fe42806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.907+0000 7fe449c40640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe42806b500 0x7fe42806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.907+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fe4400891f0 con 0x7fe444144f10 2024-03-06T07:26:54.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.908+0000 7fe449c40640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe42806b500 0x7fe42806d9c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe434004f60 tx=0x7fe434007e30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:54.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:54.911+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe440057ca0 con 0x7fe444144f10 2024-03-06T07:26:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:55.149+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) v1 -- 0x7fe4440b3410 con 0x7fe444144f10 2024-03-06T07:26:56.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.122+0000 7fe43a7fc640 1 -- 172.21.15.72:0/497015393 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v284) v1 ==== 134+0+0 (secure 0 0 0) 0x7fe44005bd40 con 0x7fe444144f10 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe42806b500 msgr2=0x7fe42806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe42806b500 0x7fe42806d9c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe434004f60 tx=0x7fe434007e30 comp rx=0 tx=0).stop 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 msgr2=0x7fe444140390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 0x7fe444140390 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fe44000c4d0 tx=0x7fe44000c990 comp rx=0 tx=0).stop 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 shutdown_connections 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe42806b500 0x7fe42806d9c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe444164b90 0x7fe4441408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe444144f10 0x7fe444140390 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 --2- 172.21.15.72:0/497015393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4440cbfb0 0x7fe44415b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.124+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 >> 172.21.15.72:0/497015393 conn(0x7fe44406b3d0 msgr2=0x7fe4440b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.125+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 shutdown_connections 2024-03-06T07:26:56.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.125+0000 7fe44bea4640 1 -- 172.21.15.72:0/497015393 wait complete. 2024-03-06T07:26:56.140 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-03-06T07:26:56.140 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-03-06T07:26:56.141 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:26:56.141 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:56.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.244+0000 7fbd97781640 1 Processor -- start 2024-03-06T07:26:56.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 -- start start 2024-03-06T07:26:56.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd9015dfe0 0x7fbd900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbd900cf830 0x7fbd900d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd900b31e0 con 0x7fbd900cf830 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd900b3d60 con 0x7fbd9015dfe0 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.245+0000 7fbd97781640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900b3430 con 0x7fbd900d27f0 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbd900cf830 0x7fbd900d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbd900cf830 0x7fbd900d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48116/0 (socket says 172.21.15.72:48116) 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 -- 172.21.15.72:0/4099888571 learned_addr learned my addr 172.21.15.72:0/4099888571 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd9551d640 1 --2- 172.21.15.72:0/4099888571 >> v2:172.21.15.146:3300/0 conn(0x7fbd9015dfe0 0x7fbd900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd95d1e640 1 --2- 172.21.15.72:0/4099888571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 -- 172.21.15.72:0/4099888571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 msgr2=0x7fbd900d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 --2- 172.21.15.72:0/4099888571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 -- 172.21.15.72:0/4099888571 >> v2:172.21.15.146:3300/0 conn(0x7fbd9015dfe0 msgr2=0x7fbd900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 --2- 172.21.15.72:0/4099888571 >> v2:172.21.15.146:3300/0 conn(0x7fbd9015dfe0 0x7fbd900cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.246+0000 7fbd94d1c640 1 -- 172.21.15.72:0/4099888571 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900b3710 con 0x7fbd900cf830 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd94d1c640 1 --2- 172.21.15.72:0/4099888571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd900cf830 0x7fbd900d22b0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbd90067990 tx=0x7fbd8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a64768529b206d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd87fff640 1 -- 172.21.15.72:0/4099888571 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd80035020 con 0x7fbd900cf830 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd87fff640 1 -- 172.21.15.72:0/4099888571 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd80031870 con 0x7fbd900cf830 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd87fff640 1 -- 172.21.15.72:0/4099888571 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd80031c40 con 0x7fbd900cf830 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 -- 172.21.15.72:0/4099888571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd900cf830 msgr2=0x7fbd900d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 --2- 172.21.15.72:0/4099888571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd900cf830 0x7fbd900d22b0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fbd90067990 tx=0x7fbd8002b910 comp rx=0 tx=0).stop 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 -- 172.21.15.72:0/4099888571 shutdown_connections 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 --2- 172.21.15.72:0/4099888571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 --2- 172.21.15.72:0/4099888571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd900cf830 0x7fbd900d22b0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 --2- 172.21.15.72:0/4099888571 >> v2:172.21.15.146:3300/0 conn(0x7fbd9015dfe0 0x7fbd900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 -- 172.21.15.72:0/4099888571 >> 172.21.15.72:0/4099888571 conn(0x7fbd900d3cd0 msgr2=0x7fbd900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 -- 172.21.15.72:0/4099888571 shutdown_connections 2024-03-06T07:26:56.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.247+0000 7fbd97781640 1 -- 172.21.15.72:0/4099888571 wait complete. 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 Processor -- start 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 -- start start 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900cf830 0x7fbd9015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 0x7fbd900b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900d1df0 con 0x7fbd9015dfe0 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd900d1c70 con 0x7fbd900cf830 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd97781640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9015e3c0 con 0x7fbd900d27f0 2024-03-06T07:26:56.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd95d1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 0x7fbd900b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd95d1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 0x7fbd900b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48132/0 (socket says 172.21.15.72:48132) 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.249+0000 7fbd95d1e640 1 -- 172.21.15.72:0/2931212333 learned_addr learned my addr 172.21.15.72:0/2931212333 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd94d1c640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd9551d640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900cf830 0x7fbd9015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd95d1e640 1 -- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 msgr2=0x7fbd900b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd95d1e640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd95d1e640 1 -- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900cf830 msgr2=0x7fbd9015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd95d1e640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900cf830 0x7fbd9015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd95d1e640 1 -- 172.21.15.72:0/2931212333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900cd0f0 con 0x7fbd9015dfe0 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd9551d640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900cf830 0x7fbd9015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd95d1e640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 0x7fbd900b6e10 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00d800 tx=0x7fbd8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c018020 con 0x7fbd9015dfe0 2024-03-06T07:26:56.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c0096c0 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c009a90 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd900b7440 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.250+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd900caf40 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.251+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9004f330 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.252+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd8c01b020 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.252+0000 7fbd85ffb640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd7406b4d0 0x7fbd7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.252+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fbd8c08a240 con 0x7fbd9015dfe0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.252+0000 7fbd9551d640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd7406b4d0 0x7fbd7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.252+0000 7fbd9551d640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd7406b4d0 0x7fbd7406d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbd900ccb70 tx=0x7fbd8800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.256+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd8c022030 con 0x7fbd9015dfe0 2024-03-06T07:26:56.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.427+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd900ce9b0 con 0x7fbd9015dfe0 2024-03-06T07:26:56.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.428+0000 7fbd85ffb640 1 -- 172.21.15.72:0/2931212333 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+13422 (secure 0 0 0) 0x7fbd8c0592b0 con 0x7fbd9015dfe0 2024-03-06T07:26:56.430 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:56.430 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":101,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:54.111991+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6834/4195924898":"2024-03-07T07:26:50.937486+0000","172.21.15.72:6837/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6835/4195924898":"2024-03-07T07:26:50.937486+0000","172.21.15.72:6835/646486087":"2024-03-07T07:26:13.890692+0000","172.21.15.72:6836/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6835/209452807":"2024-03-07T07:26:13.880837+0000","172.21.15.72:6834/3967731577":"2024-03-07T07:26:19.829563+0000","172.21.15.146:6833/209452807":"2024-03-07T07:26:13.880837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd7406b4d0 msgr2=0x7fbd7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd7406b4d0 0x7fbd7406d990 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbd900ccb70 tx=0x7fbd8800b040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 msgr2=0x7fbd900b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 0x7fbd900b6e10 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c00d800 tx=0x7fbd8c00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 shutdown_connections 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd7406b4d0 0x7fbd7406d990 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd9015dfe0 0x7fbd900b6e10 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd900d27f0 0x7fbd900b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 --2- 172.21.15.72:0/2931212333 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd900cf830 0x7fbd9015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 >> 172.21.15.72:0/2931212333 conn(0x7fbd900d3cd0 msgr2=0x7fbd9006bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 shutdown_connections 2024-03-06T07:26:56.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.430+0000 7fbd97781640 1 -- 172.21.15.72:0/2931212333 wait complete. 2024-03-06T07:26:56.443 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:56.443 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.543+0000 7f59790c3640 1 Processor -- start 2024-03-06T07:26:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.543+0000 7f59790c3640 1 -- start start 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f59790c3640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f59740d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f59790c3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59740d14a0 0x7f597415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f59790c3640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f597415eb00 0x7f59740cae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f59790c3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597415c050 con 0x7f59740d14a0 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f59790c3640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59740d0060 con 0x7f597415eb00 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f59790c3640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740cfa50 con 0x7f59740d10c0 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59740d14a0 0x7f597415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59740d14a0 0x7f597415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48136/0 (socket says 172.21.15.72:48136) 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 -- 172.21.15.72:0/2766629408 learned_addr learned my addr 172.21.15.72:0/2766629408 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f5972d9d640 1 --2- 172.21.15.72:0/2766629408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f59740d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597359e640 1 --2- 172.21.15.72:0/2766629408 >> v2:172.21.15.146:3300/0 conn(0x7f597415eb00 0x7f59740cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 -- 172.21.15.72:0/2766629408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 msgr2=0x7f59740d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 --2- 172.21.15.72:0/2766629408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f59740d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 -- 172.21.15.72:0/2766629408 >> v2:172.21.15.146:3300/0 conn(0x7f597415eb00 msgr2=0x7f59740cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 --2- 172.21.15.72:0/2766629408 >> v2:172.21.15.146:3300/0 conn(0x7f597415eb00 0x7f59740cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.544+0000 7f597259c640 1 -- 172.21.15.72:0/2766629408 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740d0a30 con 0x7f59740d14a0 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.545+0000 7f597259c640 1 --2- 172.21.15.72:0/2766629408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59740d14a0 0x7f597415e5c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f5964009870 tx=0x7f596402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df1e68d70e97af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.545+0000 7f5971d9b640 1 -- 172.21.15.72:0/2766629408 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5964035020 con 0x7f59740d14a0 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.545+0000 7f5971d9b640 1 -- 172.21.15.72:0/2766629408 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5964031640 con 0x7f59740d14a0 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.545+0000 7f5971d9b640 1 -- 172.21.15.72:0/2766629408 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596402d040 con 0x7f59740d14a0 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 -- 172.21.15.72:0/2766629408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59740d14a0 msgr2=0x7f597415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 --2- 172.21.15.72:0/2766629408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59740d14a0 0x7f597415e5c0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f5964009870 tx=0x7f596402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 -- 172.21.15.72:0/2766629408 shutdown_connections 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 --2- 172.21.15.72:0/2766629408 >> v2:172.21.15.146:3300/0 conn(0x7f597415eb00 0x7f59740cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 --2- 172.21.15.72:0/2766629408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59740d14a0 0x7f597415e5c0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 --2- 172.21.15.72:0/2766629408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f59740d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 -- 172.21.15.72:0/2766629408 >> 172.21.15.72:0/2766629408 conn(0x7f59740b3800 msgr2=0x7f59740b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 -- 172.21.15.72:0/2766629408 shutdown_connections 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.546+0000 7f59790c3640 1 -- 172.21.15.72:0/2766629408 wait complete. 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.547+0000 7f59790c3640 1 Processor -- start 2024-03-06T07:26:56.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.547+0000 7f59790c3640 1 -- start start 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f59790c3640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f5974152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f59790c3640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59740d14a0 0x7f59740b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f59790c3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 0x7f59740b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f59790c3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597415f440 con 0x7f597415eb00 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f59790c3640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597415f160 con 0x7f59740d14a0 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f59790c3640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59740cfcf0 con 0x7f59740d10c0 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 0x7f59740b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 0x7f59740b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48152/0 (socket says 172.21.15.72:48152) 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 -- 172.21.15.72:0/3586385354 learned_addr learned my addr 172.21.15.72:0/3586385354 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:56.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f5972d9d640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f5974152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597259c640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59740d14a0 0x7f59740b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 -- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 msgr2=0x7f5974152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f5974152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 -- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59740d14a0 msgr2=0x7f59740b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59740d14a0 0x7f59740b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.548+0000 7f597359e640 1 -- 172.21.15.72:0/3586385354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740b5850 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.549+0000 7f597359e640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 0x7f59740b0970 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f596c007e70 tx=0x7f596c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.549+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c014620 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.549+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c019070 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.549+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c014a20 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.549+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59740b5b30 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.549+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59740b60f0 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.550+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597409fe60 con 0x7f597415eb00 2024-03-06T07:26:56.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.550+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f596c014bc0 con 0x7f597415eb00 2024-03-06T07:26:56.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.551+0000 7f595b7fe640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f596006b4d0 0x7f596006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.551+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f596c08a880 con 0x7f597415eb00 2024-03-06T07:26:56.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.551+0000 7f5972d9d640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f596006b4d0 0x7f596006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.551+0000 7f5972d9d640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f596006b4d0 0x7f596006d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5968006fd0 tx=0x7f5968008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.555+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f596c059980 con 0x7f597415eb00 2024-03-06T07:26:56.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.726+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59740d0c30 con 0x7f597415eb00 2024-03-06T07:26:56.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.727+0000 7f595b7fe640 1 -- 172.21.15.72:0/3586385354 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f596c059280 con 0x7f597415eb00 2024-03-06T07:26:56.729 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f596006b4d0 msgr2=0x7f596006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f596006b4d0 0x7f596006d990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5968006fd0 tx=0x7f5968008040 comp rx=0 tx=0).stop 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 msgr2=0x7f59740b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 0x7f59740b0970 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f596c007e70 tx=0x7f596c00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 shutdown_connections 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f596006b4d0 0x7f596006d990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f597415eb00 0x7f59740b0970 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f59740d14a0 0x7f59740b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 --2- 172.21.15.72:0/3586385354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59740d10c0 0x7f5974152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 >> 172.21.15.72:0/3586385354 conn(0x7f59740b3800 msgr2=0x7f597415f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 shutdown_connections 2024-03-06T07:26:56.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.729+0000 7f59790c3640 1 -- 172.21.15.72:0/3586385354 wait complete. 2024-03-06T07:26:56.742 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:26:56.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.841+0000 7faa29bb5640 1 Processor -- start 2024-03-06T07:26:56.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.841+0000 7faa29bb5640 1 -- start start 2024-03-06T07:26:56.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa29bb5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa2413dc10 0x7faa240b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa29bb5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2415f580 con 0x7faa2415bf30 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa29bb5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2415b450 con 0x7faa2415ef60 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa29bb5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa240cf3f0 con 0x7faa2413dc10 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa22ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7faa2415bf30 0x7faa2415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58260/0 (socket says 172.21.15.72:58260) 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa22ffd640 1 -- 172.21.15.72:0/553108985 learned_addr learned my addr 172.21.15.72:0/553108985 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa237fe640 1 --2- 172.21.15.72:0/553108985 >> v2:172.21.15.146:3300/0 conn(0x7faa2413dc10 0x7faa240b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa227fc640 1 -- 172.21.15.72:0/553108985 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253164572 0 0) 0x7faa2415f580 con 0x7faa2415bf30 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa227fc640 1 -- 172.21.15.72:0/553108985 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa0c003520 con 0x7faa2415bf30 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa237fe640 1 -- 172.21.15.72:0/553108985 >> v1:172.21.15.146:6789/0 conn(0x7faa2415ef60 legacy=0x7faa240708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa237fe640 1 -- 172.21.15.72:0/553108985 >> v1:172.21.15.72:6789/0 conn(0x7faa2415bf30 legacy=0x7faa2415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa227fc640 1 -- 172.21.15.72:0/553108985 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2886677819 0 0) 0x7faa0c003520 con 0x7faa2415bf30 2024-03-06T07:26:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.842+0000 7faa237fe640 1 -- 172.21.15.72:0/553108985 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2415b730 con 0x7faa2413dc10 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.843+0000 7faa237fe640 1 --2- 172.21.15.72:0/553108985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2413dc10 0x7faa240b2950 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faa180097a0 tx=0x7faa1802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e856d09815d9d465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.843+0000 7faa227fc640 1 -- 172.21.15.72:0/553108985 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18037070 con 0x7faa2413dc10 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.843+0000 7faa227fc640 1 -- 172.21.15.72:0/553108985 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18030510 con 0x7faa2413dc10 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.843+0000 7faa227fc640 1 -- 172.21.15.72:0/553108985 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14001670 con 0x7faa2413dc10 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 -- 172.21.15.72:0/553108985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2413dc10 msgr2=0x7faa240b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 --2- 172.21.15.72:0/553108985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2413dc10 0x7faa240b2950 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faa180097a0 tx=0x7faa1802b790 comp rx=0 tx=0).stop 2024-03-06T07:26:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 -- 172.21.15.72:0/553108985 shutdown_connections 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 --2- 172.21.15.72:0/553108985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2413dc10 0x7faa240b2950 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 -- 172.21.15.72:0/553108985 >> 172.21.15.72:0/553108985 conn(0x7faa240cf8a0 msgr2=0x7faa240cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 -- 172.21.15.72:0/553108985 shutdown_connections 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.844+0000 7faa29bb5640 1 -- 172.21.15.72:0/553108985 wait complete. 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.845+0000 7faa29bb5640 1 Processor -- start 2024-03-06T07:26:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.845+0000 7faa29bb5640 1 -- start start 2024-03-06T07:26:56.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 0x7faa2413e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2415bf30 0x7faa24154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa2415ef60 0x7faa24143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa240b2e90 con 0x7faa2413dc10 2024-03-06T07:26:56.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2415b210 con 0x7faa2415bf30 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa240b3010 con 0x7faa2415ef60 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 0x7faa2413e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 0x7faa2413e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48158/0 (socket says 172.21.15.72:48158) 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 -- 172.21.15.72:0/3714492851 learned_addr learned my addr 172.21.15.72:0/3714492851 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa22ffd640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2415bf30 0x7faa24154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa23fff640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa2415ef60 0x7faa24143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 -- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa2415ef60 msgr2=0x7faa24143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa2415ef60 0x7faa24143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 -- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2415bf30 msgr2=0x7faa24154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2415bf30 0x7faa24154150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 -- 172.21.15.72:0/3714492851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24141260 con 0x7faa2413dc10 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa237fe640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 0x7faa2413e430 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7faa1802b7d0 tx=0x7faa180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18042070 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1803e870 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18037040 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.846+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa24140220 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.847+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa24140a60 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.847+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2404f330 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.848+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa18045070 con 0x7faa2413dc10 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.848+0000 7faa0bfff640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa1006b500 0x7faa1006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.848+0000 7faa22ffd640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa1006b500 0x7faa1006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:56.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.848+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7faa180ab900 con 0x7faa2413dc10 2024-03-06T07:26:56.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.849+0000 7faa22ffd640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa1006b500 0x7faa1006d9c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faa2415d730 tx=0x7faa14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:56.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:56.853+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa1804c070 con 0x7faa2413dc10 2024-03-06T07:26:57.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.093+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7faa2406eb00 con 0x7faa2413dc10 2024-03-06T07:26:57.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.123+0000 7faa0bfff640 1 -- 172.21.15.72:0/3714492851 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v285) v1 ==== 145+0+0 (secure 0 0 0) 0x7faa24140220 con 0x7faa2413dc10 2024-03-06T07:26:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa1006b500 msgr2=0x7faa1006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa1006b500 0x7faa1006d9c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faa2415d730 tx=0x7faa14009040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 msgr2=0x7faa2413e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 0x7faa2413e430 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7faa1802b7d0 tx=0x7faa180046c0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 shutdown_connections 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa1006b500 0x7faa1006d9c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa2415ef60 0x7faa24143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa2415bf30 0x7faa24154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 --2- 172.21.15.72:0/3714492851 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa2413dc10 0x7faa2413e430 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 >> 172.21.15.72:0/3714492851 conn(0x7faa240cf8a0 msgr2=0x7faa2415de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 shutdown_connections 2024-03-06T07:26:57.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.126+0000 7faa29bb5640 1 -- 172.21.15.72:0/3714492851 wait complete. 2024-03-06T07:26:57.139 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.241+0000 7ff1bf155640 1 Processor -- start 2024-03-06T07:26:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.241+0000 7ff1bf155640 1 -- start start 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.241+0000 7ff1bf155640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff1b80af870 0x7ff1b80b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.241+0000 7ff1bf155640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1b815be00 0x7ff1b815c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.241+0000 7ff1bf155640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b806be30 con 0x7ff1b815b240 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.241+0000 7ff1bf155640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b80cf980 con 0x7ff1b815be00 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bf155640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b815c7d0 con 0x7ff1b80af870 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1b815be00 0x7ff1b815c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff1b815be00 0x7ff1b815c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48166/0 (socket says 172.21.15.72:48166) 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 -- 172.21.15.72:0/1412298638 learned_addr learned my addr 172.21.15.72:0/1412298638 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 -- 172.21.15.72:0/1412298638 >> v2:172.21.15.146:3300/0 conn(0x7ff1b80af870 msgr2=0x7ff1b80b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1b7fff640 1 --2- 172.21.15.72:0/1412298638 >> v2:172.21.15.146:3300/0 conn(0x7ff1b80af870 0x7ff1b80b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 --2- 172.21.15.72:0/1412298638 >> v2:172.21.15.146:3300/0 conn(0x7ff1b80af870 0x7ff1b80b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 -- 172.21.15.72:0/1412298638 >> v1:172.21.15.146:6789/0 conn(0x7ff1b815b240 legacy=0x7ff1b806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.242+0000 7ff1bd6f2640 1 -- 172.21.15.72:0/1412298638 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b815b620 con 0x7ff1b815be00 2024-03-06T07:26:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bd6f2640 1 --2- 172.21.15.72:0/1412298638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b815be00 0x7ff1b815c290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8072420 tx=0x7ff1b002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee9d2a83e6339d0d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1b77fe640 1 -- 172.21.15.72:0/1412298638 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b002ca80 con 0x7ff1b815be00 2024-03-06T07:26:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1b77fe640 1 -- 172.21.15.72:0/1412298638 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b002cc20 con 0x7ff1b815be00 2024-03-06T07:26:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1b77fe640 1 -- 172.21.15.72:0/1412298638 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b00343f0 con 0x7ff1b815be00 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 -- 172.21.15.72:0/1412298638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b815be00 msgr2=0x7ff1b815c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1412298638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b815be00 0x7ff1b815c290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8072420 tx=0x7ff1b002bee0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 -- 172.21.15.72:0/1412298638 shutdown_connections 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1412298638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b815be00 0x7ff1b815c290 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1412298638 >> v2:172.21.15.146:3300/0 conn(0x7ff1b80af870 0x7ff1b80b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 -- 172.21.15.72:0/1412298638 >> 172.21.15.72:0/1412298638 conn(0x7ff1b80b1d70 msgr2=0x7ff1b80b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 -- 172.21.15.72:0/1412298638 shutdown_connections 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.243+0000 7ff1bf155640 1 -- 172.21.15.72:0/1412298638 wait complete. 2024-03-06T07:26:57.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 Processor -- start 2024-03-06T07:26:57.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 -- start start 2024-03-06T07:26:57.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 0x7ff1b8158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b815b240 0x7ff1b8158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b815be00 0x7ff1b80c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b815b8e0 con 0x7ff1b80af870 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b8073450 con 0x7ff1b815b240 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bf155640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b806b460 con 0x7ff1b815be00 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.245+0000 7ff1bcef1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 0x7ff1b8158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 0x7ff1b8158aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48182/0 (socket says 172.21.15.72:48182) 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 -- 172.21.15.72:0/1509025512 learned_addr learned my addr 172.21.15.72:0/1509025512 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1b7fff640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b815b240 0x7ff1b8158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bd6f2640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b815be00 0x7ff1b80c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 -- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b815be00 msgr2=0x7ff1b80c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b815be00 0x7ff1b80c7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 -- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b815b240 msgr2=0x7ff1b8158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b815b240 0x7ff1b8158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 -- 172.21.15.72:0/1509025512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b8157a00 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1b7fff640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b815b240 0x7ff1b8158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bcef1640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 0x7ff1b8158aa0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c430 tx=0x7ff1ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac011790 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac017070 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b8157ce0 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac011bb0 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.246+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b81582a0 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.247+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b809e3d0 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.248+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1ac01c460 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.248+0000 7ff1b57fa640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff19c06b4d0 0x7ff19c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.248+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7ff1ac089ef0 con 0x7ff1b80af870 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.248+0000 7ff1b7fff640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff19c06b4d0 0x7ff19c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.248+0000 7ff1b7fff640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff19c06b4d0 0x7ff19c06d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff1b81f4640 tx=0x7ff1a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.252+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1ac058f60 con 0x7ff1b80af870 2024-03-06T07:26:57.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.422+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff1b80b6330 con 0x7ff1b80af870 2024-03-06T07:26:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.426+0000 7ff1b57fa640 1 -- 172.21.15.72:0/1509025512 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff1ac058860 con 0x7ff1b80af870 2024-03-06T07:26:57.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff19c06b4d0 msgr2=0x7ff19c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff19c06b4d0 0x7ff19c06d990 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff1b81f4640 tx=0x7ff1a4008040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 msgr2=0x7ff1b8158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 0x7ff1b8158aa0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac00c430 tx=0x7ff1ac00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 shutdown_connections 2024-03-06T07:26:57.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff19c06b4d0 0x7ff19c06d990 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1b815be00 0x7ff1b80c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1b815b240 0x7ff1b8158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 --2- 172.21.15.72:0/1509025512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff1b80af870 0x7ff1b8158aa0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 >> 172.21.15.72:0/1509025512 conn(0x7ff1b80b1d70 msgr2=0x7ff1b806d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 shutdown_connections 2024-03-06T07:26:57.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.428+0000 7ff1bf155640 1 -- 172.21.15.72:0/1509025512 wait complete. 2024-03-06T07:26:57.441 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:26:57.441 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:57.479 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:26:57.528 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:26:57.528 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:26:57.528 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.544 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:57.544 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:26:57.544 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.561 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:57.561 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:26:57.562 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:26:57.599 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:26:57.599 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:26:57.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:26:57.618 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:26:57.619 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:26:57.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.736+0000 7f9aa7cfb640 1 Processor -- start 2024-03-06T07:26:57.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.736+0000 7f9aa7cfb640 1 -- start start 2024-03-06T07:26:57.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa7cfb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa7cfb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9aa015bf30 0x7f9aa00b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa7cfb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9aa015e850 0x7f9aa015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa7cfb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa00cf3f0 con 0x7f9aa015bf30 2024-03-06T07:26:57.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa7cfb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa015b210 con 0x7f9aa015e850 2024-03-06T07:26:57.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa7cfb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa015f580 con 0x7f9aa013dc10 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9aa015bf30 0x7f9aa00b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9aa015bf30 0x7f9aa00b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48184/0 (socket says 172.21.15.72:48184) 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 -- 172.21.15.72:0/3495097025 learned_addr learned my addr 172.21.15.72:0/3495097025 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5a97640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa6298640 1 --2- 172.21.15.72:0/3495097025 >> v2:172.21.15.146:3300/0 conn(0x7f9aa015e850 0x7f9aa015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 -- 172.21.15.72:0/3495097025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 msgr2=0x7f9aa00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 -- 172.21.15.72:0/3495097025 >> v2:172.21.15.146:3300/0 conn(0x7f9aa015e850 msgr2=0x7f9aa015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 --2- 172.21.15.72:0/3495097025 >> v2:172.21.15.146:3300/0 conn(0x7f9aa015e850 0x7f9aa015eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5296640 1 -- 172.21.15.72:0/3495097025 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa015b450 con 0x7f9aa015bf30 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa6298640 1 --2- 172.21.15.72:0/3495097025 >> v2:172.21.15.146:3300/0 conn(0x7f9aa015e850 0x7f9aa015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.737+0000 7f9aa5a97640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.738+0000 7f9aa5296640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015bf30 0x7f9aa00b2e90 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9aa006c790 tx=0x7f9a9002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5950ff7bc427b00f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.738+0000 7f9aa4a95640 1 -- 172.21.15.72:0/3495097025 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a900317c0 con 0x7f9aa015bf30 2024-03-06T07:26:57.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.738+0000 7f9aa4a95640 1 -- 172.21.15.72:0/3495097025 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a90037070 con 0x7f9aa015bf30 2024-03-06T07:26:57.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.738+0000 7f9aa4a95640 1 -- 172.21.15.72:0/3495097025 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90031b80 con 0x7f9aa015bf30 2024-03-06T07:26:57.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/3495097025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015bf30 msgr2=0x7f9aa00b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015bf30 0x7f9aa00b2e90 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9aa006c790 tx=0x7f9a9002b860 comp rx=0 tx=0).stop 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/3495097025 shutdown_connections 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/3495097025 >> v2:172.21.15.146:3300/0 conn(0x7f9aa015e850 0x7f9aa015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015bf30 0x7f9aa00b2e90 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/3495097025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/3495097025 >> 172.21.15.72:0/3495097025 conn(0x7f9aa00cf8a0 msgr2=0x7f9aa00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/3495097025 shutdown_connections 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.739+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/3495097025 wait complete. 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.740+0000 7f9aa7cfb640 1 Processor -- start 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.740+0000 7f9aa7cfb640 1 -- start start 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa015bf30 0x7f9aa0139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 0x7f9aa0159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa015f1e0 con 0x7f9aa015e850 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa013dff0 con 0x7f9aa015bf30 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa015f360 con 0x7f9aa013dc10 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 0x7f9aa0159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 0x7f9aa0159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48188/0 (socket says 172.21.15.72:48188) 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 -- 172.21.15.72:0/310086270 learned_addr learned my addr 172.21.15.72:0/310086270 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa5a97640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa5296640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa015bf30 0x7f9aa0139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 -- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 msgr2=0x7f9aa00ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 -- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa015bf30 msgr2=0x7f9aa0139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa015bf30 0x7f9aa0139240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 -- 172.21.15.72:0/310086270 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa01e6040 con 0x7f9aa015e850 2024-03-06T07:26:57.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa6298640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 0x7f9aa0159d30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f9a98007ef0 tx=0x7f9a9800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9801a070 con 0x7f9aa015e850 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a98003b90 con 0x7f9aa015e850 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a98004030 con 0x7f9aa015e850 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa013ae70 con 0x7f9aa015e850 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.741+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa013b430 con 0x7f9aa015e850 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.743+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa004f330 con 0x7f9aa015e850 2024-03-06T07:26:57.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.743+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a9801d070 con 0x7f9aa015e850 2024-03-06T07:26:57.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.743+0000 7f9a967fc640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b4d0 0x7f9a8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:57.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.743+0000 7f9aa5a97640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b4d0 0x7f9a8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:57.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.743+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f9a9808a170 con 0x7f9aa015e850 2024-03-06T07:26:57.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.744+0000 7f9aa5a97640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b4d0 0x7f9a8406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:57.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.748+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a98010070 con 0x7f9aa015e850 2024-03-06T07:26:57.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.958+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9aa006eb00 con 0x7f9aa015e850 2024-03-06T07:26:57.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.959+0000 7f9a967fc640 1 -- 172.21.15.72:0/310086270 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f9a98058c20 con 0x7f9aa015e850 2024-03-06T07:26:57.961 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:57.961 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5966,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/479496418","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":479496418},{"type":"v1","addr":"172.21.15.146:6835","nonce":479496418}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6562,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/1807964996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":1807964996},{"type":"v1","addr":"172.21.15.146:6834","nonce":1807964996}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7611,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/76908069","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":76908069},{"type":"v1","addr":"172.21.15.72:6837","nonce":76908069}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":54},{"gid":7830,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/2553074430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2553074430},{"type":"v1","addr":"172.21.15.72:6835","nonce":2553074430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[]} 2024-03-06T07:26:57.961 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 59 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b4d0 msgr2=0x7f9a8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b4d0 0x7f9a8406d990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c008040 comp rx=0 tx=0).stop 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 msgr2=0x7f9aa0159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 0x7f9aa0159d30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f9a98007ef0 tx=0x7f9a9800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 shutdown_connections 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a8406b4d0 0x7f9a8406d990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9aa015e850 0x7f9aa0159d30 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9aa015bf30 0x7f9aa0139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 --2- 172.21.15.72:0/310086270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9aa013dc10 0x7f9aa00ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:57.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 >> 172.21.15.72:0/310086270 conn(0x7f9aa00cf8a0 msgr2=0x7f9aa013c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:57.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 shutdown_connections 2024-03-06T07:26:57.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:57.961+0000 7f9aa7cfb640 1 -- 172.21.15.72:0/310086270 wait complete. 2024-03-06T07:26:57.974 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:26:57.974 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:26:57.974 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:26:57.975 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:57.975 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:26:57.975 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:26:57.975 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:26:57.975 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:57.975 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:26:57.975 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:26:57.976 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:26:57.976 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:57.976 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:26:57.976 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:26:57.976 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:26:57.976 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:26:57.983 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:26:57.983 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:26:57.984 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:26:57.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:26:57.987 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:26:57.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:26:57.988 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:26:57.988 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:26:58.026 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:26:58.027 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:58.027 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:26:58.027 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:58.030 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:26:58.030 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:58.030 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:26:58.031 DEBUG:teuthology.parallel:result is None 2024-03-06T07:26:58.031 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:26:58.102 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:26:58.103 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:26:58.112 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:26:58.112 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:26:58.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.135+0000 7f2eeedac640 1 Processor -- start 2024-03-06T07:26:58.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.135+0000 7f2eeedac640 1 -- start start 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eeedac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2ee81599d0 0x7f2ee80ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eeedac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee815f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eeedac640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ee80ce710 0x7f2ee815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eeedac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee8159db0 con 0x7f2ee81599d0 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eeedac640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee80b4370 con 0x7f2ee80ce710 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eeedac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80b3a40 con 0x7f2ee80b4800 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2ee81599d0 0x7f2ee80ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2ee81599d0 0x7f2ee80ce1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48232/0 (socket says 172.21.15.72:48232) 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 -- 172.21.15.72:0/3798947473 learned_addr learned my addr 172.21.15.72:0/3798947473 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2ee7fff640 1 --2- 172.21.15.72:0/3798947473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee815f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eed349640 1 --2- 172.21.15.72:0/3798947473 >> v2:172.21.15.146:3300/0 conn(0x7f2ee80ce710 0x7f2ee815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 -- 172.21.15.72:0/3798947473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 msgr2=0x7f2ee815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 --2- 172.21.15.72:0/3798947473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee815f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 -- 172.21.15.72:0/3798947473 >> v2:172.21.15.146:3300/0 conn(0x7f2ee80ce710 msgr2=0x7f2ee815fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:58.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 --2- 172.21.15.72:0/3798947473 >> v2:172.21.15.146:3300/0 conn(0x7f2ee80ce710 0x7f2ee815fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.136+0000 7f2eecb48640 1 -- 172.21.15.72:0/3798947473 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee80b3d20 con 0x7f2ee81599d0 2024-03-06T07:26:58.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.137+0000 7f2eecb48640 1 --2- 172.21.15.72:0/3798947473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee80ce1d0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80cfff0 tx=0x7f2edc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d51708ca6230f7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.139+0000 7f2ee77fe640 1 -- 172.21.15.72:0/3798947473 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc035020 con 0x7f2ee81599d0 2024-03-06T07:26:58.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.139+0000 7f2ee77fe640 1 -- 172.21.15.72:0/3798947473 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc031870 con 0x7f2ee81599d0 2024-03-06T07:26:58.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.139+0000 7f2ee77fe640 1 -- 172.21.15.72:0/3798947473 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc031c40 con 0x7f2ee81599d0 2024-03-06T07:26:58.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 -- 172.21.15.72:0/3798947473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 msgr2=0x7f2ee80ce1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 --2- 172.21.15.72:0/3798947473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee80ce1d0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80cfff0 tx=0x7f2edc02b910 comp rx=0 tx=0).stop 2024-03-06T07:26:58.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 -- 172.21.15.72:0/3798947473 shutdown_connections 2024-03-06T07:26:58.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 --2- 172.21.15.72:0/3798947473 >> v2:172.21.15.146:3300/0 conn(0x7f2ee80ce710 0x7f2ee815fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 --2- 172.21.15.72:0/3798947473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee815f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 --2- 172.21.15.72:0/3798947473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee80ce1d0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 -- 172.21.15.72:0/3798947473 >> 172.21.15.72:0/3798947473 conn(0x7f2ee80b1c80 msgr2=0x7f2ee80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:58.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 -- 172.21.15.72:0/3798947473 shutdown_connections 2024-03-06T07:26:58.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.140+0000 7f2eeedac640 1 -- 172.21.15.72:0/3798947473 wait complete. 2024-03-06T07:26:58.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.141+0000 7f2eeedac640 1 Processor -- start 2024-03-06T07:26:58.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.141+0000 7f2eeedac640 1 -- start start 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eeedac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eeedac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee80ce710 0x7f2ee8156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eeedac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee815c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eeedac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8160810 con 0x7f2ee81599d0 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eeedac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee8160690 con 0x7f2ee80ce710 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eeedac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee80b3680 con 0x7f2ee80b4800 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee815c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee815c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48234/0 (socket says 172.21.15.72:48234) 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 -- 172.21.15.72:0/2681105720 learned_addr learned my addr 172.21.15.72:0/2681105720 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2ee7fff640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee80ce710 0x7f2ee8156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eecb48640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 -- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 msgr2=0x7f2ee814d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee814d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 -- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee80ce710 msgr2=0x7f2ee8156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee80ce710 0x7f2ee8156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eed349640 1 -- 172.21.15.72:0/2681105720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee806d6a0 con 0x7f2ee81599d0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.142+0000 7f2eecb48640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee814d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.143+0000 7f2eed349640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee815c350 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009dd0 tx=0x7f2ee000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.143+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0018020 con 0x7f2ee81599d0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.143+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee0014480 con 0x7f2ee81599d0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.143+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee80ada00 con 0x7f2ee81599d0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.143+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0014870 con 0x7f2ee81599d0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.143+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee80adfc0 con 0x7f2ee81599d0 2024-03-06T07:26:58.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.144+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee804f330 con 0x7f2ee81599d0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.144+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2ee0005d10 con 0x7f2ee81599d0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.144+0000 7f2ee57fa640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2ecc06b4d0 0x7f2ecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.145+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f2ee008abb0 con 0x7f2ee81599d0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.145+0000 7f2eecb48640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2ecc06b4d0 0x7f2ecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:58.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.145+0000 7f2eecb48640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2ecc06b4d0 0x7f2ecc06d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2edc036000 tx=0x7f2edc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:58.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.149+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2ee0022070 con 0x7f2ee81599d0 2024-03-06T07:26:58.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:58.334+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2ee80ae190 con 0x7f2ee81599d0 2024-03-06T07:26:59.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.141+0000 7f2ee57fa640 1 -- 172.21.15.72:0/2681105720 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v106) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2ee0010040 con 0x7f2ee81599d0 2024-03-06T07:26:59.143 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2ecc06b4d0 msgr2=0x7f2ecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2ecc06b4d0 0x7f2ecc06d990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2edc036000 tx=0x7f2edc03c040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 msgr2=0x7f2ee815c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee815c350 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009dd0 tx=0x7f2ee000c490 comp rx=0 tx=0).stop 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 shutdown_connections 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2ecc06b4d0 0x7f2ecc06d990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ee81599d0 0x7f2ee815c350 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ee80ce710 0x7f2ee8156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 --2- 172.21.15.72:0/2681105720 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ee80b4800 0x7f2ee814d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.143+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 >> 172.21.15.72:0/2681105720 conn(0x7f2ee80b1c80 msgr2=0x7f2ee8158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.144+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 shutdown_connections 2024-03-06T07:26:59.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.144+0000 7f2eeedac640 1 -- 172.21.15.72:0/2681105720 wait complete. 2024-03-06T07:26:59.158 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:26:59.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.258+0000 7f351545c640 1 Processor -- start 2024-03-06T07:26:59.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.258+0000 7f351545c640 1 -- start start 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f351545c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f35100d2710 0x7f35100b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f351545c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35100b3430 con 0x7f35100d22b0 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f351545c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35100b3710 con 0x7f351015dfe0 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f351545c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35100b31e0 con 0x7f35100d2710 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f3507fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f35100d22b0 0x7f35100b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58270/0 (socket says 172.21.15.72:58270) 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f3507fff640 1 -- 172.21.15.72:0/673799993 learned_addr learned my addr 172.21.15.72:0/673799993 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831811893 0 0) 0x7f35100b3430 con 0x7f35100d22b0 2024-03-06T07:26:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34fc003520 con 0x7f35100d22b0 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350f7fe640 1 --2- 172.21.15.72:0/673799993 >> v2:172.21.15.146:3300/0 conn(0x7f35100d2710 0x7f35100b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781023221 0 0) 0x7f34fc003520 con 0x7f35100d22b0 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 >> v2:172.21.15.146:3300/0 conn(0x7f35100d2710 msgr2=0x7f35100b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 --2- 172.21.15.72:0/673799993 >> v2:172.21.15.146:3300/0 conn(0x7f35100d2710 0x7f35100b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 >> v1:172.21.15.146:6789/0 conn(0x7f351015dfe0 legacy=0x7f35100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35100b39f0 con 0x7f35100d22b0 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f34f8001670 con 0x7f35100d22b0 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 >> v1:172.21.15.72:6789/0 conn(0x7f35100d22b0 legacy=0x7f35100b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34fc0039e0 0x7f34fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc005210 0x7f34fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100b31e0 con 0x7f34fc004800 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc00c140 con 0x7f34fc0039e0 2024-03-06T07:26:59.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.259+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34fc00c340 con 0x7f34fc005210 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f3507fff640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f350effd640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34fc0039e0 0x7f34fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f350f7fe640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc005210 0x7f34fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f3507fff640 1 -- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc005210 msgr2=0x7f34fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f3507fff640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc005210 0x7f34fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f3507fff640 1 -- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34fc0039e0 msgr2=0x7f34fc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f3507fff640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34fc0039e0 0x7f34fc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f3507fff640 1 -- 172.21.15.72:0/673799993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc003520 con 0x7f34fc004800 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.260+0000 7f350f7fe640 1 -- 172.21.15.72:0/673799993 reap_dead start 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f3507fff640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f34f8007fd0 tx=0x7f34fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58f565fb4a593b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8007ce0 con 0x7f34fc004800 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f8030070 con 0x7f34fc004800 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f350e7fc640 1 -- 172.21.15.72:0/673799993 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8005d10 con 0x7f34fc004800 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 -- 172.21.15.72:0/673799993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34fc004800 msgr2=0x7f34fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f34f8007fd0 tx=0x7f34fc005830 comp rx=0 tx=0).stop 2024-03-06T07:26:59.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 -- 172.21.15.72:0/673799993 shutdown_connections 2024-03-06T07:26:59.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 --2- 172.21.15.72:0/673799993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34fc004800 0x7f34fc004c80 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 -- 172.21.15.72:0/673799993 >> 172.21.15.72:0/673799993 conn(0x7f35100d3cd0 msgr2=0x7f35100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 -- 172.21.15.72:0/673799993 shutdown_connections 2024-03-06T07:26:59.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.261+0000 7f351545c640 1 -- 172.21.15.72:0/673799993 wait complete. 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.262+0000 7f351545c640 1 Processor -- start 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 -- start start 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc004800 0x7f351015bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35101f3d50 0x7f3510160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 0x7f3510160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100d1c70 con 0x7f35101f41d0 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35100b3d60 con 0x7f35101f3d50 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f351545c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35100d1df0 con 0x7f34fc004800 2024-03-06T07:26:59.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f350f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 0x7f3510160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f350f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 0x7f3510160bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48248/0 (socket says 172.21.15.72:48248) 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f350f7fe640 1 -- 172.21.15.72:0/2635610176 learned_addr learned my addr 172.21.15.72:0/2635610176 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f3507fff640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35101f3d50 0x7f3510160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.263+0000 7f350effd640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc004800 0x7f351015bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f350f7fe640 1 -- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc004800 msgr2=0x7f351015bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f350f7fe640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc004800 0x7f351015bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f350f7fe640 1 -- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35101f3d50 msgr2=0x7f3510160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f350f7fe640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35101f3d50 0x7f3510160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f350f7fe640 1 -- 172.21.15.72:0/2635610176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35101f47a0 con 0x7f35101f41d0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f3507fff640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35101f3d50 0x7f3510160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f350f7fe640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 0x7f3510160bd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f3510161bf0 tx=0x7f3508007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508014410 con 0x7f35101f41d0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3508019070 con 0x7f35101f41d0 2024-03-06T07:26:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35101f4a80 con 0x7f35101f41d0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3508014810 con 0x7f35101f41d0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.264+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35100b88e0 con 0x7f35101f41d0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.265+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35100d1a90 con 0x7f35101f41d0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.265+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f35080149b0 con 0x7f35101f41d0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.266+0000 7f35077fe640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f34f006b4d0 0x7f34f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.266+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f350808a3f0 con 0x7f35101f41d0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.266+0000 7f350effd640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f34f006b4d0 0x7f34f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.266+0000 7f350effd640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f34f006b4d0 0x7f34f006d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3500000c00 tx=0x7f3500009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.270+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f35100d1a90 con 0x7f35101f41d0 2024-03-06T07:26:59.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.506+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f351004f330 con 0x7f35101f41d0 2024-03-06T07:26:59.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.506+0000 7f35077fe640 1 -- 172.21.15.72:0/2635610176 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3508059650 con 0x7f35101f41d0 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:59.509 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.510 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.511 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.513 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:26:59.514 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.515 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:26:59.516 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.517 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.519 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:26:59.520 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.521 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:26:59.522 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:26:59.522 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:26:59.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f34f006b4d0 msgr2=0x7f34f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f34f006b4d0 0x7f34f006d990 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3500000c00 tx=0x7f3500009040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 msgr2=0x7f3510160bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 0x7f3510160bd0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f3510161bf0 tx=0x7f3508007700 comp rx=0 tx=0).stop 2024-03-06T07:26:59.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 shutdown_connections 2024-03-06T07:26:59.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f34f006b4d0 0x7f34f006d990 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35101f41d0 0x7f3510160bd0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f35101f3d50 0x7f3510160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 --2- 172.21.15.72:0/2635610176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34fc004800 0x7f351015bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 >> 172.21.15.72:0/2635610176 conn(0x7f35100d3cd0 msgr2=0x7f35100d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 shutdown_connections 2024-03-06T07:26:59.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.508+0000 7f351545c640 1 -- 172.21.15.72:0/2635610176 wait complete. 2024-03-06T07:26:59.524 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:26:59.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.623+0000 7efccd98a640 1 Processor -- start 2024-03-06T07:26:59.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.623+0000 7efccd98a640 1 -- start start 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.623+0000 7efccd98a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcc80d22b0 0x7efcc80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.623+0000 7efccd98a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80b3430 con 0x7efcc815dfe0 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.623+0000 7efccd98a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80b3710 con 0x7efcc80b41f0 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.623+0000 7efccd98a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc80d2c50 con 0x7efcc80d22b0 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efcc815dfe0 0x7efcc80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58272/0 (socket says 172.21.15.72:58272) 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc6ffd640 1 -- 172.21.15.72:0/189185878 learned_addr learned my addr 172.21.15.72:0/189185878 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc67fc640 1 --2- 172.21.15.72:0/189185878 >> v2:172.21.15.146:3300/0 conn(0x7efcc80d22b0 0x7efcc80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc5ffb640 1 -- 172.21.15.72:0/189185878 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924093063 0 0) 0x7efcc80b3430 con 0x7efcc815dfe0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc5ffb640 1 -- 172.21.15.72:0/189185878 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb0003520 con 0x7efcc815dfe0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc67fc640 1 -- 172.21.15.72:0/189185878 >> v1:172.21.15.146:6789/0 conn(0x7efcc80b41f0 legacy=0x7efcc80b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc5ffb640 1 -- 172.21.15.72:0/189185878 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848588937 0 0) 0x7efcc80b3710 con 0x7efcc80b41f0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc67fc640 1 -- 172.21.15.72:0/189185878 >> v1:172.21.15.72:6789/0 conn(0x7efcc815dfe0 legacy=0x7efcc80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.624+0000 7efcc67fc640 1 -- 172.21.15.72:0/189185878 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc80b39f0 con 0x7efcc80d22b0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.625+0000 7efcc67fc640 1 --2- 172.21.15.72:0/189185878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc80d2710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0097d0 tx=0x7efcbc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50826cd983681c8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.625+0000 7efcc5ffb640 1 -- 172.21.15.72:0/189185878 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc034070 con 0x7efcc80d22b0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.625+0000 7efcc5ffb640 1 -- 172.21.15.72:0/189185878 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcbc030610 con 0x7efcc80d22b0 2024-03-06T07:26:59.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.625+0000 7efcc5ffb640 1 -- 172.21.15.72:0/189185878 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcbc0309e0 con 0x7efcc80d22b0 2024-03-06T07:26:59.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 -- 172.21.15.72:0/189185878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 msgr2=0x7efcc80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 --2- 172.21.15.72:0/189185878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc80d2710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0097d0 tx=0x7efcbc02b960 comp rx=0 tx=0).stop 2024-03-06T07:26:59.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 -- 172.21.15.72:0/189185878 shutdown_connections 2024-03-06T07:26:59.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 --2- 172.21.15.72:0/189185878 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc80d2710 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 -- 172.21.15.72:0/189185878 >> 172.21.15.72:0/189185878 conn(0x7efcc80d3cd0 msgr2=0x7efcc80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 -- 172.21.15.72:0/189185878 shutdown_connections 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.626+0000 7efccd98a640 1 -- 172.21.15.72:0/189185878 wait complete. 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.627+0000 7efccd98a640 1 Processor -- start 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.627+0000 7efccd98a640 1 -- start start 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 0x7efcc81f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc815dfe0 0x7efcc81f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80b3d60 con 0x7efcc80b41f0 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80b31e0 con 0x7efcc80d22b0 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80d1c70 con 0x7efcc815dfe0 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 0x7efcc81f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 0x7efcc81f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48258/0 (socket says 172.21.15.72:48258) 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 -- 172.21.15.72:0/259232123 learned_addr learned my addr 172.21.15.72:0/259232123 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:59.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc67fc640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc77fe640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc815dfe0 0x7efcc81f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 -- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc815dfe0 msgr2=0x7efcc81f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc815dfe0 0x7efcc81f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 -- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 msgr2=0x7efcc815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 -- 172.21.15.72:0/259232123 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc81f7e80 con 0x7efcc80b41f0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc67fc640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcc6ffd640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 0x7efcc81f7940 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7efcb800c390 tx=0x7efcb800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb8017070 con 0x7efcc80b41f0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb8005ce0 con 0x7efcc80b41f0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb80072f0 con 0x7efcc80b41f0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc81f4390 con 0x7efcc80b41f0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.628+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc81f7140 con 0x7efcc80b41f0 2024-03-06T07:26:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.629+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc804f330 con 0x7efcc80b41f0 2024-03-06T07:26:59.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.630+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efcb801a070 con 0x7efcc80b41f0 2024-03-06T07:26:59.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.630+0000 7efcaffff640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb406b4d0 0x7efcb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.630+0000 7efcc67fc640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb406b4d0 0x7efcb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.630+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efcb8089220 con 0x7efcc80b41f0 2024-03-06T07:26:59.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.631+0000 7efcc67fc640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb406b4d0 0x7efcb406d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efcbc036000 tx=0x7efcbc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.635+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efcc81f7e80 con 0x7efcc80b41f0 2024-03-06T07:26:59.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.806+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcc806c000 con 0x7efcc80b41f0 2024-03-06T07:26:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.807+0000 7efcaffff640 1 -- 172.21.15.72:0/259232123 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+12905 (secure 0 0 0) 0x7efcb8058480 con 0x7efcc80b41f0 2024-03-06T07:26:59.809 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:26:59.809 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":106,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:26:59.138827+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:26:59.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb406b4d0 msgr2=0x7efcb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb406b4d0 0x7efcb406d990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efcbc036000 tx=0x7efcbc03b040 comp rx=0 tx=0).stop 2024-03-06T07:26:59.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 msgr2=0x7efcc81f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 0x7efcc81f7940 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7efcb800c390 tx=0x7efcb800c850 comp rx=0 tx=0).stop 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 shutdown_connections 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efcb406b4d0 0x7efcb406d990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcc815dfe0 0x7efcc81f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcc80d22b0 0x7efcc815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 --2- 172.21.15.72:0/259232123 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcc80b41f0 0x7efcc81f7940 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 >> 172.21.15.72:0/259232123 conn(0x7efcc80d3cd0 msgr2=0x7efcc80d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 shutdown_connections 2024-03-06T07:26:59.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.809+0000 7efccd98a640 1 -- 172.21.15.72:0/259232123 wait complete. 2024-03-06T07:26:59.822 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:26:59.822 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:26:59.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.923+0000 7f2ba64c0640 1 Processor -- start 2024-03-06T07:26:59.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2ba64c0640 1 -- start start 2024-03-06T07:26:59.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2ba64c0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 0x7f2ba00b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2ba64c0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ba00cea50 0x7f2ba00ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2ba64c0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba006d030 con 0x7f2ba006cab0 2024-03-06T07:26:59.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2ba64c0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba00cf1c0 con 0x7f2ba00cea50 2024-03-06T07:26:59.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2ba64c0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba00cc760 con 0x7f2ba015def0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2b9f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2ba006cab0 0x7f2ba00ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58276/0 (socket says 172.21.15.72:58276) 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.924+0000 7f2b9f7fe640 1 -- 172.21.15.72:0/1433132920 learned_addr learned my addr 172.21.15.72:0/1433132920 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2ba4a5d640 1 --2- 172.21.15.72:0/1433132920 >> v2:172.21.15.146:3300/0 conn(0x7f2ba00cea50 0x7f2ba00ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 0x7f2ba00b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287818708 0 0) 0x7f2ba006d030 con 0x7f2ba006cab0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b88003520 con 0x7f2ba006cab0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2720266140 0 0) 0x7f2b88003520 con 0x7f2ba006cab0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 msgr2=0x7f2ba00b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 0x7f2ba00b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 >> v2:172.21.15.146:3300/0 conn(0x7f2ba00cea50 msgr2=0x7f2ba00ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 --2- 172.21.15.72:0/1433132920 >> v2:172.21.15.146:3300/0 conn(0x7f2ba00cea50 0x7f2ba00ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba006d310 con 0x7f2ba006cab0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2b94001670 con 0x7f2ba006cab0 2024-03-06T07:26:59.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 >> v1:172.21.15.72:6789/0 conn(0x7f2ba006cab0 legacy=0x7f2ba00ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b880039b0 0x7f2b88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 0x7f2b88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b88004d40 0x7f2b880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba00cc760 con 0x7f2b880039b0 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8800b9c0 con 0x7f2b88004d40 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8800bb40 con 0x7f2ba015def0 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2ba4a5d640 1 --2- 172.21.15.72:0/1433132920 >> v2:172.21.15.146:3300/0 conn(0x7f2ba00cea50 0x7f2ba00ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b880039b0 0x7f2b88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9f7fe640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 0x7f2b88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2ba4a5d640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b88004d40 0x7f2b880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 -- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 msgr2=0x7f2b88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ba015def0 0x7f2b88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 -- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b88004d40 msgr2=0x7f2b880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:26:59.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b88004d40 0x7f2b880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2b9ffff640 1 -- 172.21.15.72:0/1433132920 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b88003520 con 0x7f2b880039b0 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.925+0000 7f2ba4a5d640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b88004d40 0x7f2b880051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.926+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b880039b0 0x7f2b88003dd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009ac0 tx=0x7f2b88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87d5753f7ef0b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.926+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90039070 con 0x7f2b880039b0 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.926+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9002c6b0 con 0x7f2b880039b0 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.926+0000 7f2b9effd640 1 -- 172.21.15.72:0/1433132920 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b90009690 con 0x7f2b880039b0 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba64c0640 1 -- 172.21.15.72:0/1433132920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b880039b0 msgr2=0x7f2b88003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/1433132920 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b880039b0 0x7f2b88003dd0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f2b90009ac0 tx=0x7f2b88005830 comp rx=0 tx=0).stop 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba4a5d640 1 -- 172.21.15.72:0/1433132920 reap_dead start 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba64c0640 1 -- 172.21.15.72:0/1433132920 shutdown_connections 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba64c0640 1 -- 172.21.15.72:0/1433132920 >> 172.21.15.72:0/1433132920 conn(0x7f2ba00b3690 msgr2=0x7f2ba00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba64c0640 1 -- 172.21.15.72:0/1433132920 shutdown_connections 2024-03-06T07:26:59.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.927+0000 7f2ba64c0640 1 -- 172.21.15.72:0/1433132920 wait complete. 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.928+0000 7f2ba64c0640 1 Processor -- start 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.928+0000 7f2ba64c0640 1 -- start start 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b880039b0 0x7f2ba00b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 0x7f2ba014faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ba01605e0 0x7f2ba014ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba015e2d0 con 0x7f2ba01601a0 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba00cc3f0 con 0x7f2ba01605e0 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba015e450 con 0x7f2b880039b0 2024-03-06T07:26:59.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 0x7f2ba014faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 0x7f2ba014faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48276/0 (socket says 172.21.15.72:48276) 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 -- 172.21.15.72:0/3217129822 learned_addr learned my addr 172.21.15.72:0/3217129822 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba4a5d640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ba01605e0 0x7f2ba014ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b880039b0 0x7f2ba00b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 -- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b880039b0 msgr2=0x7f2ba00b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b880039b0 0x7f2ba00b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 -- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ba01605e0 msgr2=0x7f2ba014ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ba01605e0 0x7f2ba014ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 -- 172.21.15.72:0/3217129822 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba00b6120 con 0x7f2ba01601a0 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba4a5d640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ba01605e0 0x7f2ba014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b880039b0 0x7f2ba00b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9f7fe640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 0x7f2ba014faa0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e780 tx=0x7f2b9400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400c8d0 con 0x7f2ba01601a0 2024-03-06T07:26:59.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b94018070 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9400ccd0 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.929+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba00b6400 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.930+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba00b69c0 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.930+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba009fcf0 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.931+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b9401d460 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.931+0000 7f2b9cff9640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b8c06b4d0 0x7f2b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.931+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2b94089e40 con 0x7f2ba01601a0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.931+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b8c06b4d0 0x7f2b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.932+0000 7f2b9ffff640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b8c06b4d0 0x7f2b8c06d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2b90036000 tx=0x7f2b90030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:26:59.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:26:59.936+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b94059010 con 0x7f2ba01601a0 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.107+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2ba0068ff0 con 0x7f2ba01601a0 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.108+0000 7f2b9cff9640 1 -- 172.21.15.72:0/3217129822 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b94058910 con 0x7f2ba01601a0 2024-03-06T07:27:00.110 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b8c06b4d0 msgr2=0x7f2b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b8c06b4d0 0x7f2b8c06d990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2b90036000 tx=0x7f2b90030000 comp rx=0 tx=0).stop 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 msgr2=0x7f2ba014faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 0x7f2ba014faa0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f2b9400e780 tx=0x7f2b9400ec40 comp rx=0 tx=0).stop 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 shutdown_connections 2024-03-06T07:27:00.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b8c06b4d0 0x7f2b8c06d990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ba01605e0 0x7f2ba014ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2ba01601a0 0x7f2ba014faa0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 --2- 172.21.15.72:0/3217129822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b880039b0 0x7f2ba00b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 >> 172.21.15.72:0/3217129822 conn(0x7f2ba00b3690 msgr2=0x7f2ba006f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 shutdown_connections 2024-03-06T07:27:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.110+0000 7f2ba64c0640 1 -- 172.21.15.72:0/3217129822 wait complete. 2024-03-06T07:27:00.123 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:27:00.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.220+0000 7f0b12502640 1 Processor -- start 2024-03-06T07:27:00.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.220+0000 7f0b12502640 1 -- start start 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b12502640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 0x7f0b0c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b12502640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c071f40 con 0x7f0b0c0cd7a0 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b12502640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c15b8b0 con 0x7f0b0c15b430 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b12502640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c070a70 con 0x7f0b0c1515e0 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b10a9f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0b0c0cd7a0 0x7f0b0c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58282/0 (socket says 172.21.15.72:58282) 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b10a9f640 1 -- 172.21.15.72:0/4194547782 learned_addr learned my addr 172.21.15.72:0/4194547782 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b0bfff640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 0x7f0b0c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.221+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508998856 0 0) 0x7f0b0c071f40 con 0x7f0b0c0cd7a0 2024-03-06T07:27:00.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af4004630 con 0x7f0b0c0cd7a0 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692495802 0 0) 0x7f0af4004630 con 0x7f0b0c0cd7a0 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 msgr2=0x7f0b0c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 0x7f0b0c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 >> v1:172.21.15.146:6789/0 conn(0x7f0b0c15b430 legacy=0x7f0b0c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c070d50 con 0x7f0b0c0cd7a0 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f0b040019f0 con 0x7f0b0c0cd7a0 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 >> v1:172.21.15.72:6789/0 conn(0x7f0b0c0cd7a0 legacy=0x7f0b0c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 0x7f0af4004af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0af4005910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4005e50 0x7f0af40062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c070a70 con 0x7f0af4005070 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af400cbb0 con 0x7f0af4005e50 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af400cdb0 con 0x7f0b0c1515e0 2024-03-06T07:27:00.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0af4005910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0bfff640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 0x7f0af4004af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b10a9f640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4005e50 0x7f0af40062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0b7fe640 1 -- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 msgr2=0x7f0af4004af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c1515e0 0x7f0af4004af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0b7fe640 1 -- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4005e50 msgr2=0x7f0af40062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0af4005e50 0x7f0af40062d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.222+0000 7f0b0b7fe640 1 -- 172.21.15.72:0/4194547782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4004630 con 0x7f0af4005070 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.223+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0af4005910 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0b00001750 tx=0x7f0af4006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c204b56c7e364e5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.223+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00001370 con 0x7f0af4005070 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.223+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b000052d0 con 0x7f0af4005070 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.223+0000 7f0b0affd640 1 -- 172.21.15.72:0/4194547782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00003830 con 0x7f0af4005070 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b12502640 1 -- 172.21.15.72:0/4194547782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 msgr2=0x7f0af4005910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b12502640 1 --2- 172.21.15.72:0/4194547782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0af4005910 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f0b00001750 tx=0x7f0af4006940 comp rx=0 tx=0).stop 2024-03-06T07:27:00.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b10a9f640 1 -- 172.21.15.72:0/4194547782 reap_dead start 2024-03-06T07:27:00.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b12502640 1 -- 172.21.15.72:0/4194547782 shutdown_connections 2024-03-06T07:27:00.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b12502640 1 -- 172.21.15.72:0/4194547782 >> 172.21.15.72:0/4194547782 conn(0x7f0b0c0cf8a0 msgr2=0x7f0b0c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b12502640 1 -- 172.21.15.72:0/4194547782 shutdown_connections 2024-03-06T07:27:00.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.224+0000 7f0b12502640 1 -- 172.21.15.72:0/4194547782 wait complete. 2024-03-06T07:27:00.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.225+0000 7f0b12502640 1 Processor -- start 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.225+0000 7f0b12502640 1 -- start start 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0b0c153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c143260 0x7f0b0c142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b0c1522c0 0x7f0b0c06dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0cce40 con 0x7f0af4005070 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c071310 con 0x7f0b0c1522c0 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0ccfc0 con 0x7f0b0c143260 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0b0c153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0b0c153be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48302/0 (socket says 172.21.15.72:48302) 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 -- 172.21.15.72:0/1886377286 learned_addr learned my addr 172.21.15.72:0/1886377286 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b10a9f640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b0c1522c0 0x7f0b0c06dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c143260 0x7f0b0c142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 -- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c143260 msgr2=0x7f0b0c142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c143260 0x7f0b0c142b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 -- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b0c1522c0 msgr2=0x7f0b0c06dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b0c1522c0 0x7f0b0c06dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 -- 172.21.15.72:0/1886377286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c143060 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b0bfff640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0b0c153be0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0119d0 tx=0x7f0afc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc00faf0 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc00fc90 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c1541c0 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc014440 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.226+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c151ac0 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.227+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c068840 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.228+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0afc012040 con 0x7f0af4005070 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.228+0000 7f0b08ff9640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0af806b4d0 0x7f0af806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.228+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0af806b4d0 0x7f0af806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.228+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0afc08a430 con 0x7f0af4005070 2024-03-06T07:27:00.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.229+0000 7f0b0b7fe640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0af806b4d0 0x7f0af806d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0b00033000 tx=0x7f0b00009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.233+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0b0c143060 con 0x7f0af4005070 2024-03-06T07:27:00.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.409+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0b0c0b4c30 con 0x7f0af4005070 2024-03-06T07:27:00.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.410+0000 7f0b08ff9640 1 -- 172.21.15.72:0/1886377286 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0afc019030 con 0x7f0af4005070 2024-03-06T07:27:00.412 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:00.412 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:27:00.412 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:27:00.412 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:27:00.413 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:27:00.413 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:27:00.413 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:27:00.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0af806b4d0 msgr2=0x7f0af806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0af806b4d0 0x7f0af806d990 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0b00033000 tx=0x7f0b00009030 comp rx=0 tx=0).stop 2024-03-06T07:27:00.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 msgr2=0x7f0b0c153be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0b0c153be0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0afc0119d0 tx=0x7f0afc011e90 comp rx=0 tx=0).stop 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 shutdown_connections 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0af806b4d0 0x7f0af806d990 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0b0c1522c0 0x7f0b0c06dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0b0c143260 0x7f0b0c142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 --2- 172.21.15.72:0/1886377286 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0af4005070 0x7f0b0c153be0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 >> 172.21.15.72:0/1886377286 conn(0x7f0b0c0cf8a0 msgr2=0x7f0b0c0ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 shutdown_connections 2024-03-06T07:27:00.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.412+0000 7f0b12502640 1 -- 172.21.15.72:0/1886377286 wait complete. 2024-03-06T07:27:00.425 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:27:00.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:27:00.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.528+0000 7f8f4fcb5640 1 Processor -- start 2024-03-06T07:27:00.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.528+0000 7f8f4fcb5640 1 -- start start 2024-03-06T07:27:00.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4fcb5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f4815e050 0x7f8f480cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4fcb5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8f480ce170 0x7f8f480cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4fcb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f4806b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4fcb5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f4815e430 con 0x7f8f480ce170 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4fcb5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f480cd440 con 0x7f8f4815e050 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4fcb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480cdce0 con 0x7f8f4806afb0 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8f480ce170 0x7f8f480cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8f480ce170 0x7f8f480cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48316/0 (socket says 172.21.15.72:48316) 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 -- 172.21.15.72:0/300076333 learned_addr learned my addr 172.21.15.72:0/300076333 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4da51640 1 --2- 172.21.15.72:0/300076333 >> v2:172.21.15.146:3300/0 conn(0x7f8f4815e050 0x7f8f480cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4e252640 1 --2- 172.21.15.72:0/300076333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f4806b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 -- 172.21.15.72:0/300076333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 msgr2=0x7f8f4806b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 --2- 172.21.15.72:0/300076333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f4806b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 -- 172.21.15.72:0/300076333 >> v2:172.21.15.146:3300/0 conn(0x7f8f4815e050 msgr2=0x7f8f480cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 --2- 172.21.15.72:0/300076333 >> v2:172.21.15.146:3300/0 conn(0x7f8f4815e050 0x7f8f480cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.529+0000 7f8f4d250640 1 -- 172.21.15.72:0/300076333 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480d1860 con 0x7f8f480ce170 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.530+0000 7f8f4d250640 1 --2- 172.21.15.72:0/300076333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f480ce170 0x7f8f480cf5f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f8f480ce570 tx=0x7f8f3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b0782cd083da1d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.530+0000 7f8f4ca4f640 1 -- 172.21.15.72:0/300076333 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38035020 con 0x7f8f480ce170 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.530+0000 7f8f4ca4f640 1 -- 172.21.15.72:0/300076333 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f38031870 con 0x7f8f480ce170 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.530+0000 7f8f4ca4f640 1 -- 172.21.15.72:0/300076333 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38031c40 con 0x7f8f480ce170 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.530+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/300076333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f480ce170 msgr2=0x7f8f480cf5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.530+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/300076333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f480ce170 0x7f8f480cf5f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f8f480ce570 tx=0x7f8f3802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/300076333 shutdown_connections 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/300076333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f4806b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/300076333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f480ce170 0x7f8f480cf5f0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/300076333 >> v2:172.21.15.146:3300/0 conn(0x7f8f4815e050 0x7f8f480cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/300076333 >> 172.21.15.72:0/300076333 conn(0x7f8f480d0a30 msgr2=0x7f8f480d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:00.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/300076333 shutdown_connections 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.531+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/300076333 wait complete. 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 Processor -- start 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 -- start start 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f480aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f480ce170 0x7f8f481565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 0x7f8f481f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480cd970 con 0x7f8f4815e050 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480cd690 con 0x7f8f480ce170 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.532+0000 7f8f4fcb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f480719a0 con 0x7f8f4806afb0 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 0x7f8f481f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 0x7f8f481f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48330/0 (socket says 172.21.15.72:48330) 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 -- 172.21.15.72:0/3272687259 learned_addr learned my addr 172.21.15.72:0/3272687259 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4d250640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f480ce170 0x7f8f481565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4da51640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f480aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 -- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 msgr2=0x7f8f480aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f480aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 -- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f480ce170 msgr2=0x7f8f481565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f480ce170 0x7f8f481565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 -- 172.21.15.72:0/3272687259 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4815e7c0 con 0x7f8f4815e050 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4da51640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f480aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4e252640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 0x7f8f481f6f60 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f8f40007e60 tx=0x7f8f4000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40014620 con 0x7f8f4815e050 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f40019070 con 0x7f8f4815e050 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4815eaa0 con 0x7f8f4815e050 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f40014a20 con 0x7f8f4815e050 2024-03-06T07:27:00.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.533+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4815fee0 con 0x7f8f4815e050 2024-03-06T07:27:00.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.534+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4804f330 con 0x7f8f4815e050 2024-03-06T07:27:00.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.535+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f40014bc0 con 0x7f8f4815e050 2024-03-06T07:27:00.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.535+0000 7f8f3e7fc640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:00.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.535+0000 7f8f4da51640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:00.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.535+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8f4008a630 con 0x7f8f4815e050 2024-03-06T07:27:00.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.535+0000 7f8f4da51640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8f4815cfe0 tx=0x7f8f44008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:00.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.539+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f40059800 con 0x7f8f4815e050 2024-03-06T07:27:00.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:00.711+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8f48072bc0 con 0x7f8f4815e050 2024-03-06T07:27:01.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.156+0000 7f8f3e7fc640 1 -- 172.21.15.72:0/3272687259 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v107) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8f40059100 con 0x7f8f4815e050 2024-03-06T07:27:01.159 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.158+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b500 msgr2=0x7f8f2c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.158+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8f4815cfe0 tx=0x7f8f44008040 comp rx=0 tx=0).stop 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.158+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 msgr2=0x7f8f481f6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.158+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 0x7f8f481f6f60 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f8f40007e60 tx=0x7f8f4000e4b0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 shutdown_connections 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f2c06b500 0x7f8f2c06d9c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8f4815e050 0x7f8f481f6f60 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f480ce170 0x7f8f481565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 --2- 172.21.15.72:0/3272687259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f4806afb0 0x7f8f480aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 >> 172.21.15.72:0/3272687259 conn(0x7f8f480d0a30 msgr2=0x7f8f480b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 shutdown_connections 2024-03-06T07:27:01.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.159+0000 7f8f4fcb5640 1 -- 172.21.15.72:0/3272687259 wait complete. 2024-03-06T07:27:01.175 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:27:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 Processor -- start 2024-03-06T07:27:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 -- start start 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb7c80d4860 0x7fb7c80d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7c80b0e30 0x7fb7c80b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7c81618b0 con 0x7fb7c81614d0 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80d4cf0 con 0x7fb7c80b0e30 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.280+0000 7fb7cd3f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7c80cf400 con 0x7fb7c80d4860 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7c80b0e30 0x7fb7c80b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7c80b0e30 0x7fb7c80b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48340/0 (socket says 172.21.15.72:48340) 2024-03-06T07:27:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 -- 172.21.15.72:0/2120219416 learned_addr learned my addr 172.21.15.72:0/2120219416 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 -- 172.21.15.72:0/2120219416 >> v2:172.21.15.146:3300/0 conn(0x7fb7c80d4860 msgr2=0x7fb7c80d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 --2- 172.21.15.72:0/2120219416 >> v2:172.21.15.146:3300/0 conn(0x7fb7c80d4860 0x7fb7c80d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 -- 172.21.15.72:0/2120219416 >> v1:172.21.15.146:6789/0 conn(0x7fb7c81614d0 legacy=0x7fb7c80d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.281+0000 7fb7c77fe640 1 -- 172.21.15.72:0/2120219416 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80cf6a0 con 0x7fb7c80b0e30 2024-03-06T07:27:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7c77fe640 1 --2- 172.21.15.72:0/2120219416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c80b0e30 0x7fb7c80b1260 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80b2a70 tx=0x7fb7c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf14a6bf6d4f96c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7c67fc640 1 -- 172.21.15.72:0/2120219416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c002ca70 con 0x7fb7c80b0e30 2024-03-06T07:27:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7c67fc640 1 -- 172.21.15.72:0/2120219416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c002cc10 con 0x7fb7c80b0e30 2024-03-06T07:27:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7c67fc640 1 -- 172.21.15.72:0/2120219416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c00343f0 con 0x7fb7c80b0e30 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/2120219416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c80b0e30 msgr2=0x7fb7c80b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/2120219416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c80b0e30 0x7fb7c80b1260 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fb7c80b2a70 tx=0x7fb7c002bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/2120219416 shutdown_connections 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/2120219416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c80b0e30 0x7fb7c80b1260 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/2120219416 >> v2:172.21.15.146:3300/0 conn(0x7fb7c80d4860 0x7fb7c80d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/2120219416 >> 172.21.15.72:0/2120219416 conn(0x7fb7c80d3d40 msgr2=0x7fb7c80d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/2120219416 shutdown_connections 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.282+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/2120219416 wait complete. 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.283+0000 7fb7cd3f7640 1 Processor -- start 2024-03-06T07:27:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 -- start start 2024-03-06T07:27:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c80b0e30 0x7fb7c81f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c80d4860 0x7fb7c815e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 0x7fb7c80b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80b0910 con 0x7fb7c81614d0 2024-03-06T07:27:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80cfc60 con 0x7fb7c80d4860 2024-03-06T07:27:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7cd3f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7c80d2560 con 0x7fb7c80b0e30 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7c77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 0x7fb7c80b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7c77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 0x7fb7c80b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48346/0 (socket says 172.21.15.72:48346) 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.284+0000 7fb7c77fe640 1 -- 172.21.15.72:0/1379308140 learned_addr learned my addr 172.21.15.72:0/1379308140 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c6ffd640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c80b0e30 0x7fb7c81f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7be7fc640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c80d4860 0x7fb7c815e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c77fe640 1 -- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c80b0e30 msgr2=0x7fb7c81f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c77fe640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c80b0e30 0x7fb7c81f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c77fe640 1 -- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c80d4860 msgr2=0x7fb7c815e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c77fe640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c80d4860 0x7fb7c815e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c77fe640 1 -- 172.21.15.72:0/1379308140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c80c7a50 con 0x7fb7c81614d0 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7be7fc640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c80d4860 0x7fb7c815e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7c77fe640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 0x7fb7c80b3ac0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0036040 tx=0x7fb7c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0043020 con 0x7fb7c81614d0 2024-03-06T07:27:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7c003e450 con 0x7fb7c81614d0 2024-03-06T07:27:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7c80c7d30 con 0x7fb7c81614d0 2024-03-06T07:27:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c003e820 con 0x7fb7c81614d0 2024-03-06T07:27:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.285+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7c80c82f0 con 0x7fb7c81614d0 2024-03-06T07:27:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.286+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7c8068ff0 con 0x7fb7c81614d0 2024-03-06T07:27:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.287+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7c0005d10 con 0x7fb7c81614d0 2024-03-06T07:27:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.287+0000 7fb7bffff640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7a806b500 0x7fb7a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.287+0000 7fb7c6ffd640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7a806b500 0x7fb7a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.287+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb7c00aca10 con 0x7fb7c81614d0 2024-03-06T07:27:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.288+0000 7fb7c6ffd640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7a806b500 0x7fb7a806d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8007fd0 tx=0x7fb7b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:01.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.292+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7c004d070 con 0x7fb7c81614d0 2024-03-06T07:27:01.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:01.464+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb7c80d3360 con 0x7fb7c81614d0 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.162+0000 7fb7bffff640 1 -- 172.21.15.72:0/1379308140 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v108) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb7c007ba80 con 0x7fb7c81614d0 2024-03-06T07:27:02.164 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.164+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7a806b500 msgr2=0x7fb7a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.164+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7a806b500 0x7fb7a806d9c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb7b8007fd0 tx=0x7fb7b8009040 comp rx=0 tx=0).stop 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 msgr2=0x7fb7c80b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 0x7fb7c80b3ac0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0036040 tx=0x7fb7c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 shutdown_connections 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7a806b500 0x7fb7a806d9c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7c81614d0 0x7fb7c80b3ac0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7c80d4860 0x7fb7c815e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 --2- 172.21.15.72:0/1379308140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7c80b0e30 0x7fb7c81f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 >> 172.21.15.72:0/1379308140 conn(0x7fb7c80d3d40 msgr2=0x7fb7c806bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.165+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 shutdown_connections 2024-03-06T07:27:02.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.166+0000 7fb7cd3f7640 1 -- 172.21.15.72:0/1379308140 wait complete. 2024-03-06T07:27:02.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:27:02.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.292+0000 7f02b1d81640 1 Processor -- start 2024-03-06T07:27:02.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.292+0000 7f02b1d81640 1 -- start start 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.293+0000 7f02b1d81640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02ac0cf1f0 0x7f02ac0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.293+0000 7f02b1d81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0ceb30 con 0x7f02ac161980 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.293+0000 7f02b1d81640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0ce570 con 0x7f02ac06b6c0 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.293+0000 7f02b1d81640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac0c6ff0 con 0x7f02ac0cf1f0 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.293+0000 7f02ab7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f02ac161980 0x7f02ac06afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58288/0 (socket says 172.21.15.72:58288) 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.293+0000 7f02ab7fe640 1 -- 172.21.15.72:0/3124835581 learned_addr learned my addr 172.21.15.72:0/3124835581 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02abfff640 1 --2- 172.21.15.72:0/3124835581 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0cf1f0 0x7f02ac0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576148597 0 0) 0x7f029c001110 con 0x7f02ac161980 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294003520 con 0x7f02ac161980 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864608149 0 0) 0x7f02ac0ce570 con 0x7f02ac06b6c0 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f029c001110 con 0x7f02ac06b6c0 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085266859 0 0) 0x7f0294003520 con 0x7f02ac161980 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0cf1f0 msgr2=0x7f02ac0c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 --2- 172.21.15.72:0/3124835581 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0cf1f0 0x7f02ac0c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 >> v1:172.21.15.146:6789/0 conn(0x7f02ac06b6c0 legacy=0x7f02ac06e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0ce850 con 0x7f02ac161980 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f029c001840 con 0x7f02ac161980 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02abfff640 1 --2- 172.21.15.72:0/3124835581 >> v2:172.21.15.146:3300/0 conn(0x7f02ac0cf1f0 0x7f02ac0c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 >> v1:172.21.15.72:6789/0 conn(0x7f02ac161980 legacy=0x7f02ac06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f0294003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0294004800 0x7f0294004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0294005230 0x7f029400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029400c160 con 0x7f0294003b40 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029400c310 con 0x7f0294005230 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029400c510 con 0x7f0294004800 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f0294003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02aaffd640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0294004800 0x7f0294004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02abfff640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0294005230 0x7f029400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02ab7fe640 1 -- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0294004800 msgr2=0x7f0294004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0294004800 0x7f0294004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02ab7fe640 1 -- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0294005230 msgr2=0x7f029400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0294005230 0x7f029400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02abfff640 1 -- 172.21.15.72:0/3124835581 reap_dead start 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.294+0000 7f02ab7fe640 1 -- 172.21.15.72:0/3124835581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0294003520 con 0x7f0294003b40 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.295+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f0294003f40 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f029c00c4c0 tx=0x7f0294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57419941ca64732e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.295+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c00c0e0 con 0x7f0294003b40 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.295+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c030070 con 0x7f0294003b40 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.295+0000 7f02aa7fc640 1 -- 172.21.15.72:0/3124835581 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c0058d0 con 0x7f0294003b40 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 -- 172.21.15.72:0/3124835581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 msgr2=0x7f0294003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f0294003f40 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f029c00c4c0 tx=0x7f0294005830 comp rx=0 tx=0).stop 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 -- 172.21.15.72:0/3124835581 shutdown_connections 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 --2- 172.21.15.72:0/3124835581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f0294003f40 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 -- 172.21.15.72:0/3124835581 >> 172.21.15.72:0/3124835581 conn(0x7f02ac0cd420 msgr2=0x7f02ac0cd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 -- 172.21.15.72:0/3124835581 shutdown_connections 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.296+0000 7f02b1d81640 1 -- 172.21.15.72:0/3124835581 wait complete. 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.297+0000 7f02b1d81640 1 Processor -- start 2024-03-06T07:27:02.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.297+0000 7f02b1d81640 1 -- start start 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02b1d81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f02ac0c9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02b1d81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0c8480 0x7f02ac0aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02b1d81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0b3b90 0x7f02ac0b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02b1d81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0c6c80 con 0x7f0294003b40 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02b1d81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0cf640 con 0x7f02ac0b3b90 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02b1d81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0ce090 con 0x7f02ac0c8480 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f02ac0c9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f02ac0c9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48350/0 (socket says 172.21.15.72:48350) 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 -- 172.21.15.72:0/2073583511 learned_addr learned my addr 172.21.15.72:0/2073583511 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02aaffd640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0c8480 0x7f02ac0aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02abfff640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0b3b90 0x7f02ac0b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 -- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0c8480 msgr2=0x7f02ac0aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0c8480 0x7f02ac0aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 -- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0b3b90 msgr2=0x7f02ac0b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0b3b90 0x7f02ac0b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 -- 172.21.15.72:0/2073583511 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac0ca370 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02abfff640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0b3b90 0x7f02ac0b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.298+0000 7f02ab7fe640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f02ac0c9d00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f029c0010e0 tx=0x7f02ac0c6f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.299+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c004bf0 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.299+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f029c004d90 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.299+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f029c007270 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.299+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac0ca650 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.299+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac0cac10 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.300+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac04f330 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.300+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f029c007410 con 0x7f0294003b40 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.300+0000 7f0293fff640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029806b500 0x7f029806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:02.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.300+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f029c0ab980 con 0x7f0294003b40 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.300+0000 7f02aaffd640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029806b500 0x7f029806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.301+0000 7f02aaffd640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029806b500 0x7f029806d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f02a00010e0 tx=0x7f02a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:02.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.305+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02ac0ceb30 con 0x7f0294003b40 2024-03-06T07:27:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:02.514+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f02ac06f740 con 0x7f0294003b40 2024-03-06T07:27:03.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.163+0000 7f0293fff640 1 -- 172.21.15.72:0/2073583511 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v61) v1 ==== 174+0+0 (secure 0 0 0) 0x7f029c07a870 con 0x7f0294003b40 2024-03-06T07:27:03.166 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 16 and data pool 17 2024-03-06T07:27:03.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.169+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029806b500 msgr2=0x7f029806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.169+0000 7f02b1d81640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029806b500 0x7f029806d9c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f02a00010e0 tx=0x7f02a0009040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.169+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 msgr2=0x7f02ac0c9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.169+0000 7f02b1d81640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f02ac0c9d00 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f029c0010e0 tx=0x7f02ac0c6f60 comp rx=0 tx=0).stop 2024-03-06T07:27:03.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.170+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 shutdown_connections 2024-03-06T07:27:03.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.170+0000 7f02b1d81640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f029806b500 0x7f029806d9c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.170+0000 7f02b1d81640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02ac0b3b90 0x7f02ac0b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.170+0000 7f02b1d81640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02ac0c8480 0x7f02ac0aeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.170+0000 7f02b1d81640 1 --2- 172.21.15.72:0/2073583511 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0294003b40 0x7f02ac0c9d00 secure :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f029c0010e0 tx=0x7f02ac0c6f60 comp rx=0 tx=0).stop 2024-03-06T07:27:03.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.170+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 >> 172.21.15.72:0/2073583511 conn(0x7f02ac0cd420 msgr2=0x7f02ac0cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.176+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 shutdown_connections 2024-03-06T07:27:03.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.176+0000 7f02b1d81640 1 -- 172.21.15.72:0/2073583511 wait complete. 2024-03-06T07:27:03.207 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:03.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.308+0000 7fbc627f6640 1 Processor -- start 2024-03-06T07:27:03.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.308+0000 7fbc627f6640 1 -- start start 2024-03-06T07:27:03.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc627f6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c1515e0 0x7fbc5c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc627f6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbc5c15b430 0x7fbc5c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc627f6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c15b8b0 con 0x7fbc5c0cd7a0 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc627f6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c070a70 con 0x7fbc5c15b430 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc627f6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c070d50 con 0x7fbc5c1515e0 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc60d93640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbc5c0cd7a0 0x7fbc5c0cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59756/0 (socket says 172.21.15.72:59756) 2024-03-06T07:27:03.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc60d93640 1 -- 172.21.15.72:0/1206534239 learned_addr learned my addr 172.21.15.72:0/1206534239 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5b7fe640 1 --2- 172.21.15.72:0/1206534239 >> v2:172.21.15.146:3300/0 conn(0x7fbc5c15b430 0x7fbc5c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5bfff640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c1515e0 0x7fbc5c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5affd640 1 -- 172.21.15.72:0/1206534239 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431350267 0 0) 0x7fbc5c15b8b0 con 0x7fbc5c0cd7a0 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5affd640 1 -- 172.21.15.72:0/1206534239 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc44004630 con 0x7fbc5c0cd7a0 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c1515e0 msgr2=0x7fbc5c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5b7fe640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c1515e0 0x7fbc5c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5affd640 1 -- 172.21.15.72:0/1206534239 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1691797601 0 0) 0x7fbc44004630 con 0x7fbc5c0cd7a0 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1206534239 >> v1:172.21.15.72:6789/0 conn(0x7fbc5c0cd7a0 legacy=0x7fbc5c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1206534239 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c071030 con 0x7fbc5c15b430 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.309+0000 7fbc5bfff640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c1515e0 0x7fbc5c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.310+0000 7fbc5b7fe640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c071830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097a0 tx=0x7fbc5002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e784d31be49027d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.310+0000 7fbc5affd640 1 -- 172.21.15.72:0/1206534239 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50037070 con 0x7fbc5c15b430 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.310+0000 7fbc5affd640 1 -- 172.21.15.72:0/1206534239 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc50030510 con 0x7fbc5c15b430 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.310+0000 7fbc5affd640 1 -- 172.21.15.72:0/1206534239 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc540019f0 con 0x7fbc5c15b430 2024-03-06T07:27:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 -- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 msgr2=0x7fbc5c071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c071830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbc500097a0 tx=0x7fbc5002b790 comp rx=0 tx=0).stop 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 -- 172.21.15.72:0/1206534239 shutdown_connections 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c071830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1206534239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c1515e0 0x7fbc5c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 -- 172.21.15.72:0/1206534239 >> 172.21.15.72:0/1206534239 conn(0x7fbc5c0cf8a0 msgr2=0x7fbc5c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 -- 172.21.15.72:0/1206534239 shutdown_connections 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.311+0000 7fbc627f6640 1 -- 172.21.15.72:0/1206534239 wait complete. 2024-03-06T07:27:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.312+0000 7fbc627f6640 1 Processor -- start 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.312+0000 7fbc627f6640 1 -- start start 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc627f6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c0cd7a0 0x7fbc5c1561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc627f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 0x7fbc5c1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc627f6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c14c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc627f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c071ef0 con 0x7fbc5c1515e0 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc627f6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c071d70 con 0x7fbc5c15b430 2024-03-06T07:27:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc627f6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0cc3a0 con 0x7fbc5c0cd7a0 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 0x7fbc5c1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 0x7fbc5c1572c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34008/0 (socket says 172.21.15.72:34008) 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1906371874 learned_addr learned my addr 172.21.15.72:0/1906371874 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc60d93640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c14c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5bfff640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c0cd7a0 0x7fbc5c1561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c0cd7a0 msgr2=0x7fbc5c1561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c0cd7a0 0x7fbc5c1561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 msgr2=0x7fbc5c14c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c14c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.313+0000 7fbc5b7fe640 1 -- 172.21.15.72:0/1906371874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c156700 con 0x7fbc5c1515e0 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.314+0000 7fbc5b7fe640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 0x7fbc5c1572c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fbc5002b7d0 tx=0x7fbc500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.314+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50041020 con 0x7fbc5c1515e0 2024-03-06T07:27:03.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.314+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5003c870 con 0x7fbc5c1515e0 2024-03-06T07:27:03.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.314+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c157b10 con 0x7fbc5c1515e0 2024-03-06T07:27:03.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.314+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50037040 con 0x7fbc5c1515e0 2024-03-06T07:27:03.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.314+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c14b8c0 con 0x7fbc5c1515e0 2024-03-06T07:27:03.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.315+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c068840 con 0x7fbc5c1515e0 2024-03-06T07:27:03.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.315+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbc50044020 con 0x7fbc5c1515e0 2024-03-06T07:27:03.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.316+0000 7fbc58ff9640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4806b4d0 0x7fbc4806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.316+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc500aba70 con 0x7fbc5c1515e0 2024-03-06T07:27:03.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.316+0000 7fbc5bfff640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4806b4d0 0x7fbc4806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.316+0000 7fbc5bfff640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4806b4d0 0x7fbc4806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009560 tx=0x7fbc4c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.320+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbc5004b030 con 0x7fbc5c1515e0 2024-03-06T07:27:03.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.492+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc5c04f330 con 0x7fbc5c1515e0 2024-03-06T07:27:03.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.493+0000 7fbc58ff9640 1 -- 172.21.15.72:0/1906371874 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16652 (secure 0 0 0) 0x7fbc5007a330 con 0x7fbc5c1515e0 2024-03-06T07:27:03.497 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:03.497 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":109,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:02.515441+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4806b4d0 msgr2=0x7fbc4806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4806b4d0 0x7fbc4806d990 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009560 tx=0x7fbc4c009190 comp rx=0 tx=0).stop 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 msgr2=0x7fbc5c1572c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 0x7fbc5c1572c0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fbc5002b7d0 tx=0x7fbc500046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 shutdown_connections 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbc4806b4d0 0x7fbc4806d990 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc5c15b430 0x7fbc5c14c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc5c1515e0 0x7fbc5c1572c0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 --2- 172.21.15.72:0/1906371874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc5c0cd7a0 0x7fbc5c1561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 >> 172.21.15.72:0/1906371874 conn(0x7fbc5c0cf8a0 msgr2=0x7fbc5c14d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 shutdown_connections 2024-03-06T07:27:03.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.496+0000 7fbc627f6640 1 -- 172.21.15.72:0/1906371874 wait complete. 2024-03-06T07:27:03.509 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:03.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.607+0000 7f2a35da1640 1 Processor -- start 2024-03-06T07:27:03.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.607+0000 7f2a35da1640 1 -- start start 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a35da1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2a3006c6d0 0x7f2a3015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a35da1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a300ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a35da1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a300cd450 con 0x7f2a300b4f70 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a35da1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3015e9f0 con 0x7f2a3006c6d0 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a35da1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300cda10 con 0x7f2a30066d90 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2a300b4f70 0x7f2a300b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59766/0 (socket says 172.21.15.72:59766) 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2f7fe640 1 -- 172.21.15.72:0/3887840499 learned_addr learned my addr 172.21.15.72:0/3887840499 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2effd640 1 --2- 172.21.15.72:0/3887840499 >> v2:172.21.15.146:3300/0 conn(0x7f2a3006c6d0 0x7f2a3015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280048719 0 0) 0x7f2a300cd450 con 0x7f2a300b4f70 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a18003520 con 0x7f2a300b4f70 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a300ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 370670405 0 0) 0x7f2a18003520 con 0x7f2a300b4f70 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 msgr2=0x7f2a300ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a300ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 >> v2:172.21.15.146:3300/0 conn(0x7f2a3006c6d0 msgr2=0x7f2a3015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 --2- 172.21.15.72:0/3887840499 >> v2:172.21.15.146:3300/0 conn(0x7f2a3006c6d0 0x7f2a3015ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a300cd730 con 0x7f2a300b4f70 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2a20001670 con 0x7f2a300b4f70 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a300ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.608+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 >> v1:172.21.15.72:6789/0 conn(0x7f2a300b4f70 legacy=0x7f2a300b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2e7fc640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180039b0 0x7f2a18003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2e7fc640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2e7fc640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a18004d40 0x7f2a180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3015e9f0 con 0x7f2a18004d40 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300cda10 con 0x7f2a180039b0 2024-03-06T07:27:03.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1800b9c0 con 0x7f2a30066d90 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a18004d40 0x7f2a180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2f7fe640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180039b0 0x7f2a18003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2effd640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2ffff640 1 -- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 msgr2=0x7f2a18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30066d90 0x7f2a18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2ffff640 1 -- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180039b0 msgr2=0x7f2a18003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180039b0 0x7f2a18003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.609+0000 7f2a2ffff640 1 -- 172.21.15.72:0/3887840499 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a18003520 con 0x7f2a18004d40 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a18004d40 0x7f2a180051c0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2a280073b0 tx=0x7f2a18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7482c62c9c316e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28039070 con 0x7f2a18004d40 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2802cd30 con 0x7f2a18004d40 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a2e7fc640 1 -- 172.21.15.72:0/3887840499 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28009c80 con 0x7f2a18004d40 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a35da1640 1 -- 172.21.15.72:0/3887840499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a18004d40 msgr2=0x7f2a180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a35da1640 1 --2- 172.21.15.72:0/3887840499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a18004d40 0x7f2a180051c0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f2a280073b0 tx=0x7f2a18005830 comp rx=0 tx=0).stop 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a2ffff640 1 -- 172.21.15.72:0/3887840499 reap_dead start 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a35da1640 1 -- 172.21.15.72:0/3887840499 shutdown_connections 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a35da1640 1 -- 172.21.15.72:0/3887840499 >> 172.21.15.72:0/3887840499 conn(0x7f2a300724d0 msgr2=0x7f2a300728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a35da1640 1 -- 172.21.15.72:0/3887840499 shutdown_connections 2024-03-06T07:27:03.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.610+0000 7f2a35da1640 1 -- 172.21.15.72:0/3887840499 wait complete. 2024-03-06T07:27:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.611+0000 7f2a35da1640 1 Processor -- start 2024-03-06T07:27:03.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 -- start start 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30145530 0x7f2a3015f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a301f4c00 0x7f2a300cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 0x7f2a301593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300ccb50 con 0x7f2a30158f50 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3015f3c0 con 0x7f2a301f4c00 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a35da1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300cccd0 con 0x7f2a30145530 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a2ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 0x7f2a301593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a2ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 0x7f2a301593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34016/0 (socket says 172.21.15.72:34016) 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.612+0000 7f2a2ffff640 1 -- 172.21.15.72:0/4010151850 learned_addr learned my addr 172.21.15.72:0/4010151850 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2f7fe640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30145530 0x7f2a3015f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2effd640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a301f4c00 0x7f2a300cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2ffff640 1 -- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30145530 msgr2=0x7f2a3015f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30145530 0x7f2a3015f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2ffff640 1 -- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a301f4c00 msgr2=0x7f2a300cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a301f4c00 0x7f2a300cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2ffff640 1 -- 172.21.15.72:0/4010151850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a300ce540 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a2ffff640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 0x7f2a301593e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2a280073b0 tx=0x7f2a28004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28039030 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2803a040 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a300ce820 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a28009c80 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.613+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a300cf7c0 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.614+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3004f330 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.614+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2a28045020 con 0x7f2a30158f50 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.615+0000 7f2a17fff640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a1c06b500 0x7f2a1c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.615+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a280aba70 con 0x7f2a30158f50 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.615+0000 7f2a2f7fe640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a1c06b500 0x7f2a1c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.615+0000 7f2a2f7fe640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a1c06b500 0x7f2a1c06d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2a20004840 tx=0x7f2a20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.619+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2a2807a8f0 con 0x7f2a30158f50 2024-03-06T07:27:03.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.793+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a300cf990 con 0x7f2a30158f50 2024-03-06T07:27:03.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.794+0000 7f2a17fff640 1 -- 172.21.15.72:0/4010151850 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f2a2807a1f0 con 0x7f2a30158f50 2024-03-06T07:27:03.796 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:03.797 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":109,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:02.515441+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:03.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a1c06b500 msgr2=0x7f2a1c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a1c06b500 0x7f2a1c06d9c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f2a20004840 tx=0x7f2a20009040 comp rx=0 tx=0).stop 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 msgr2=0x7f2a301593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 0x7f2a301593e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f2a280073b0 tx=0x7f2a28004ab0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 shutdown_connections 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2a1c06b500 0x7f2a1c06d9c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a30158f50 0x7f2a301593e0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a301f4c00 0x7f2a300cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 --2- 172.21.15.72:0/4010151850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a30145530 0x7f2a3015f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 >> 172.21.15.72:0/4010151850 conn(0x7f2a300724d0 msgr2=0x7f2a3006bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 shutdown_connections 2024-03-06T07:27:03.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.796+0000 7f2a35da1640 1 -- 172.21.15.72:0/4010151850 wait complete. 2024-03-06T07:27:03.810 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:27:03.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.910+0000 7f1c4d535640 1 Processor -- start 2024-03-06T07:27:03.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.910+0000 7f1c4d535640 1 -- start start 2024-03-06T07:27:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c4d535640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c480d4860 0x7f1c480d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c4d535640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c481618b0 con 0x7f1c480b0e30 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c4d535640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c480cf400 con 0x7f1c481614d0 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c4d535640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c480d4cf0 con 0x7f1c480d4860 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c477fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1c480b0e30 0x7f1c480b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59780/0 (socket says 172.21.15.72:59780) 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c477fe640 1 -- 172.21.15.72:0/588272686 learned_addr learned my addr 172.21.15.72:0/588272686 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.911+0000 7f1c467fc640 1 --2- 172.21.15.72:0/588272686 >> v2:172.21.15.146:3300/0 conn(0x7f1c480d4860 0x7f1c480d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366295839 0 0) 0x7f1c481618b0 con 0x7f1c480b0e30 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c30003520 con 0x7f1c480b0e30 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2014249980 0 0) 0x7f1c30003520 con 0x7f1c480b0e30 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 >> v2:172.21.15.146:3300/0 conn(0x7f1c480d4860 msgr2=0x7f1c480d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 --2- 172.21.15.72:0/588272686 >> v2:172.21.15.146:3300/0 conn(0x7f1c480d4860 0x7f1c480d2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 >> v1:172.21.15.146:6789/0 conn(0x7f1c481614d0 legacy=0x7f1c480d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c480cf6a0 con 0x7f1c480b0e30 2024-03-06T07:27:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1c40001690 con 0x7f1c480b0e30 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c467fc640 1 --2- 172.21.15.72:0/588272686 >> v2:172.21.15.146:3300/0 conn(0x7f1c480d4860 0x7f1c480d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 >> v1:172.21.15.72:6789/0 conn(0x7f1c480b0e30 legacy=0x7f1c480b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300039b0 0x7f1c30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c30004800 0x7f1c30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300051c0 0x7f1c3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480d4cf0 con 0x7f1c300051c0 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3000c060 con 0x7f1c300039b0 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3000c260 con 0x7f1c30004800 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c477fe640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300051c0 0x7f1c3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c46ffd640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300039b0 0x7f1c30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c467fc640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c30004800 0x7f1c30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c477fe640 1 -- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c30004800 msgr2=0x7f1c30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c477fe640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c30004800 0x7f1c30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c477fe640 1 -- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300039b0 msgr2=0x7f1c30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c477fe640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300039b0 0x7f1c30003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.912+0000 7f1c477fe640 1 -- 172.21.15.72:0/588272686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30003520 con 0x7f1c300051c0 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.913+0000 7f1c477fe640 1 -- 172.21.15.72:0/588272686 reap_dead start 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.913+0000 7f1c477fe640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300051c0 0x7f1c3000b9c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000c210 tx=0x7f1c30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e8e950ecb8a31d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.913+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40007d60 con 0x7f1c300051c0 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.913+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40030070 con 0x7f1c300051c0 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.913+0000 7f1c45ffb640 1 -- 172.21.15.72:0/588272686 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4002cd40 con 0x7f1c300051c0 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 -- 172.21.15.72:0/588272686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300051c0 msgr2=0x7f1c3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300051c0 0x7f1c3000b9c0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000c210 tx=0x7f1c30005830 comp rx=0 tx=0).stop 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 -- 172.21.15.72:0/588272686 shutdown_connections 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 --2- 172.21.15.72:0/588272686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300051c0 0x7f1c3000b9c0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 -- 172.21.15.72:0/588272686 >> 172.21.15.72:0/588272686 conn(0x7f1c480d3d40 msgr2=0x7f1c480d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 -- 172.21.15.72:0/588272686 shutdown_connections 2024-03-06T07:27:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.914+0000 7f1c4d535640 1 -- 172.21.15.72:0/588272686 wait complete. 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.915+0000 7f1c4d535640 1 Processor -- start 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.915+0000 7f1c4d535640 1 -- start start 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c4d535640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300051c0 0x7f1c48156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c4d535640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c4815d630 0x7f1c480b3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c4d535640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 0x7f1c480b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c4d535640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480b0910 con 0x7f1c480b45f0 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c4d535640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480cfc60 con 0x7f1c4815d630 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c4d535640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c480d1910 con 0x7f1c300051c0 2024-03-06T07:27:03.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 0x7f1c480b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 0x7f1c480b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34022/0 (socket says 172.21.15.72:34022) 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 -- 172.21.15.72:0/4005256947 learned_addr learned my addr 172.21.15.72:0/4005256947 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c467fc640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c4815d630 0x7f1c480b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c46ffd640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300051c0 0x7f1c48156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 -- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300051c0 msgr2=0x7f1c48156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300051c0 0x7f1c48156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 -- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c4815d630 msgr2=0x7f1c480b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c4815d630 0x7f1c480b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c477fe640 1 -- 172.21.15.72:0/4005256947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4815e3a0 con 0x7f1c480b45f0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.916+0000 7f1c467fc640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c4815d630 0x7f1c480b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.917+0000 7f1c477fe640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 0x7f1c480b3a80 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1c40000f60 tx=0x7f1c40001a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.917+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40007d60 con 0x7f1c480b45f0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.917+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40004c10 con 0x7f1c480b45f0 2024-03-06T07:27:03.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.917+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c480b7540 con 0x7f1c480b45f0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.917+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40007070 con 0x7f1c480b45f0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.917+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c48157960 con 0x7f1c480b45f0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.918+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c48068ff0 con 0x7f1c480b45f0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.918+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c40047060 con 0x7f1c480b45f0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.919+0000 7f1c2f7fe640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.919+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c400ab040 con 0x7f1c480b45f0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.919+0000 7f1c46ffd640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:03.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.919+0000 7f1c46ffd640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1c38000f80 tx=0x7f1c38009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:03.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:03.923+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c40079ec0 con 0x7f1c480b45f0 2024-03-06T07:27:04.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.131+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1c480b55b0 con 0x7f1c480b45f0 2024-03-06T07:27:04.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.180+0000 7f1c2f7fe640 1 -- 172.21.15.72:0/4005256947 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1c400797c0 con 0x7f1c480b45f0 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 msgr2=0x7f1c3406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1c38000f80 tx=0x7f1c38009040 comp rx=0 tx=0).stop 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 msgr2=0x7f1c480b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 0x7f1c480b3a80 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f1c40000f60 tx=0x7f1c40001a60 comp rx=0 tx=0).stop 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 shutdown_connections 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c3406b4d0 0x7f1c3406d990 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c480b45f0 0x7f1c480b3a80 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c4815d630 0x7f1c480b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 --2- 172.21.15.72:0/4005256947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300051c0 0x7f1c48156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.182+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 >> 172.21.15.72:0/4005256947 conn(0x7f1c480d3d40 msgr2=0x7f1c4806c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.183+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 shutdown_connections 2024-03-06T07:27:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.183+0000 7f1c4d535640 1 -- 172.21.15.72:0/4005256947 wait complete. 2024-03-06T07:27:04.195 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:27:04.196 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:27:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.296+0000 7fe588e2a640 1 Processor -- start 2024-03-06T07:27:04.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.296+0000 7fe588e2a640 1 -- start start 2024-03-06T07:27:04.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe588e2a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe58415bbb0 0x7fe58406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe588e2a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5840cea90 con 0x7fe58415ee10 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe588e2a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe584161bc0 con 0x7fe58406b4f0 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe588e2a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe584161970 con 0x7fe58415bbb0 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe58259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe58415bbb0 0x7fe58406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe582d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe58415ee10 0x7fe584071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59788/0 (socket says 172.21.15.72:59788) 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.297+0000 7fe582d9d640 1 -- 172.21.15.72:0/1106121878 learned_addr learned my addr 172.21.15.72:0/1106121878 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58259c640 1 -- 172.21.15.72:0/1106121878 >> v1:172.21.15.146:6789/0 conn(0x7fe58406b4f0 legacy=0x7fe58415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58259c640 1 -- 172.21.15.72:0/1106121878 >> v1:172.21.15.72:6789/0 conn(0x7fe58415ee10 legacy=0x7fe584071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58259c640 1 -- 172.21.15.72:0/1106121878 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584161ea0 con 0x7fe58415bbb0 2024-03-06T07:27:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58259c640 1 --2- 172.21.15.72:0/1106121878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe58415bbb0 0x7fe58406afb0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe574009d90 tx=0x7fe57402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c6e5ffab31f74e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58159a640 1 -- 172.21.15.72:0/1106121878 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574039070 con 0x7fe58415bbb0 2024-03-06T07:27:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58159a640 1 -- 172.21.15.72:0/1106121878 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57402cd00 con 0x7fe58415bbb0 2024-03-06T07:27:04.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.298+0000 7fe58159a640 1 -- 172.21.15.72:0/1106121878 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe574034660 con 0x7fe58415bbb0 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 -- 172.21.15.72:0/1106121878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe58415bbb0 msgr2=0x7fe58406afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 --2- 172.21.15.72:0/1106121878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe58415bbb0 0x7fe58406afb0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fe574009d90 tx=0x7fe57402bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 -- 172.21.15.72:0/1106121878 shutdown_connections 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 --2- 172.21.15.72:0/1106121878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe58415bbb0 0x7fe58406afb0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 -- 172.21.15.72:0/1106121878 >> 172.21.15.72:0/1106121878 conn(0x7fe5840b2660 msgr2=0x7fe5840b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 -- 172.21.15.72:0/1106121878 shutdown_connections 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.299+0000 7fe588e2a640 1 -- 172.21.15.72:0/1106121878 wait complete. 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.300+0000 7fe588e2a640 1 Processor -- start 2024-03-06T07:27:04.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.300+0000 7fe588e2a640 1 -- start start 2024-03-06T07:27:04.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe588e2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe58406b4f0 0x7fe5840cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe588e2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe58415ee10 0x7fe58406e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe588e2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 0x7fe58414c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe588e2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5840723b0 con 0x7fe584159680 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe588e2a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5841624f0 con 0x7fe58415ee10 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe588e2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe584072530 con 0x7fe58406b4f0 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 0x7fe58414c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 0x7fe58414c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34052/0 (socket says 172.21.15.72:34052) 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 -- 172.21.15.72:0/2652440001 learned_addr learned my addr 172.21.15.72:0/2652440001 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe581d9b640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe58415ee10 0x7fe58406e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe58259c640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe58406b4f0 0x7fe5840cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 -- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe58406b4f0 msgr2=0x7fe5840cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe58406b4f0 0x7fe5840cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 -- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe58415ee10 msgr2=0x7fe58406e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe58415ee10 0x7fe58406e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.301+0000 7fe582d9d640 1 -- 172.21.15.72:0/2652440001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584159c20 con 0x7fe584159680 2024-03-06T07:27:04.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.302+0000 7fe582d9d640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 0x7fe58414c500 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00c370 tx=0x7fe57c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.302+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c017070 con 0x7fe584159680 2024-03-06T07:27:04.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.302+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe57c005ce0 con 0x7fe584159680 2024-03-06T07:27:04.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.302+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0072d0 con 0x7fe584159680 2024-03-06T07:27:04.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.302+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe584159f00 con 0x7fe584159680 2024-03-06T07:27:04.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.302+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe584154ab0 con 0x7fe584159680 2024-03-06T07:27:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.303+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5840733b0 con 0x7fe584159680 2024-03-06T07:27:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.303+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe57c01a070 con 0x7fe584159680 2024-03-06T07:27:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.304+0000 7fe572ffd640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe56406b4d0 0x7fe56406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.304+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe57c0893b0 con 0x7fe584159680 2024-03-06T07:27:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.304+0000 7fe58259c640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe56406b4d0 0x7fe56406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:04.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.304+0000 7fe58259c640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe56406b4d0 0x7fe56406d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe574036000 tx=0x7fe574030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:04.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.308+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe5840cea90 con 0x7fe584159680 2024-03-06T07:27:04.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:04.516+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe58406f960 con 0x7fe584159680 2024-03-06T07:27:05.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.189+0000 7fe572ffd640 1 -- 172.21.15.72:0/2652440001 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v63) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe57c057c70 con 0x7fe584159680 2024-03-06T07:27:05.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe56406b4d0 msgr2=0x7fe56406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe56406b4d0 0x7fe56406d990 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe574036000 tx=0x7fe574030000 comp rx=0 tx=0).stop 2024-03-06T07:27:05.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 msgr2=0x7fe58414c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 0x7fe58414c500 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00c370 tx=0x7fe57c00c830 comp rx=0 tx=0).stop 2024-03-06T07:27:05.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 shutdown_connections 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe56406b4d0 0x7fe56406d990 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe584159680 0x7fe58414c500 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe58415ee10 0x7fe58406e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 --2- 172.21.15.72:0/2652440001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe58406b4f0 0x7fe5840cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 >> 172.21.15.72:0/2652440001 conn(0x7fe5840b2660 msgr2=0x7fe584071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 shutdown_connections 2024-03-06T07:27:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.191+0000 7fe588e2a640 1 -- 172.21.15.72:0/2652440001 wait complete. 2024-03-06T07:27:05.205 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:27:05.205 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:05.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.305+0000 7f2e6802a640 1 Processor -- start 2024-03-06T07:27:05.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.305+0000 7f2e6802a640 1 -- start start 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e6802a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e600ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e6802a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e600d0910 con 0x7f2e600cfff0 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e6802a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e60161980 con 0x7f2e6015bb20 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e6802a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e600d2500 con 0x7f2e600d04a0 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e655c5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2e600cfff0 0x7f2e600cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59800/0 (socket says 172.21.15.72:59800) 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e655c5640 1 -- 172.21.15.72:0/3989546303 learned_addr learned my addr 172.21.15.72:0/3989546303 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e665c7640 1 --2- 172.21.15.72:0/3989546303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e600ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761882100 0 0) 0x7f2e600d0910 con 0x7f2e600cfff0 2024-03-06T07:27:05.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.306+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e4c003520 con 0x7f2e600cfff0 2024-03-06T07:27:05.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320295387 0 0) 0x7f2e60161980 con 0x7f2e6015bb20 2024-03-06T07:27:05.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e600d0910 con 0x7f2e6015bb20 2024-03-06T07:27:05.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e665c7640 1 -- 172.21.15.72:0/3989546303 >> v1:172.21.15.146:6789/0 conn(0x7f2e6015bb20 legacy=0x7f2e600d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2344168203 0 0) 0x7f2e4c003520 con 0x7f2e600cfff0 2024-03-06T07:27:05.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e665c7640 1 -- 172.21.15.72:0/3989546303 >> v1:172.21.15.72:6789/0 conn(0x7f2e600cfff0 legacy=0x7f2e600cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e665c7640 1 -- 172.21.15.72:0/3989546303 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60161c60 con 0x7f2e600d04a0 2024-03-06T07:27:05.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.307+0000 7f2e665c7640 1 --2- 172.21.15.72:0/3989546303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e600ccfc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009dc0 tx=0x7f2e5802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1975db62ab7eb8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58039070 con 0x7f2e600d04a0 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e5802c910 con 0x7f2e600d04a0 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e64dc4640 1 -- 172.21.15.72:0/3989546303 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e50001670 con 0x7f2e600d04a0 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 -- 172.21.15.72:0/3989546303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 msgr2=0x7f2e600ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 --2- 172.21.15.72:0/3989546303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e600ccfc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009dc0 tx=0x7f2e5802bde0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 -- 172.21.15.72:0/3989546303 shutdown_connections 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 --2- 172.21.15.72:0/3989546303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e600ccfc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 -- 172.21.15.72:0/3989546303 >> 172.21.15.72:0/3989546303 conn(0x7f2e6009a9f0 msgr2=0x7f2e6009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 -- 172.21.15.72:0/3989546303 shutdown_connections 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.308+0000 7f2e6802a640 1 -- 172.21.15.72:0/3989546303 wait complete. 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.309+0000 7f2e6802a640 1 Processor -- start 2024-03-06T07:27:05.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.309+0000 7f2e6802a640 1 -- start start 2024-03-06T07:27:05.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e6802a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e600cfff0 0x7f2e6015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e6802a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e6006fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e6802a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 0x7f2e60070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e6802a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e60162500 con 0x7f2e6015bb20 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e6802a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e60162220 con 0x7f2e600cfff0 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e6802a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e600d1980 con 0x7f2e600d04a0 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e665c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 0x7f2e60070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e655c5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e6006fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e65dc6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e600cfff0 0x7f2e6015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e665c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 0x7f2e60070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34064/0 (socket says 172.21.15.72:34064) 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e65dc6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e600cfff0 0x7f2e6015aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:56062/0 (socket says 172.21.15.72:56062) 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e665c7640 1 -- 172.21.15.72:0/2975842052 learned_addr learned my addr 172.21.15.72:0/2975842052 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.310+0000 7f2e655c5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e6006fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:41318/0 (socket says 172.21.15.72:41318) 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e665c7640 1 -- 172.21.15.72:0/2975842052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 msgr2=0x7f2e6006fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e665c7640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e6006fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e665c7640 1 -- 172.21.15.72:0/2975842052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e600cfff0 msgr2=0x7f2e6015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e665c7640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e600cfff0 0x7f2e6015aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e665c7640 1 -- 172.21.15.72:0/2975842052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6015b560 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e665c7640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 0x7f2e60070170 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009ee0 tx=0x7f2e58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58039050 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e58045070 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e600c7790 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58009450 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.311+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e600c7d50 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.312+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e601e9270 con 0x7f2e6015bb20 2024-03-06T07:27:05.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.312+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2e58040690 con 0x7f2e6015bb20 2024-03-06T07:27:05.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.313+0000 7f2e567fc640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2e4406b4d0 0x7f2e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.313+0000 7f2e65dc6640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2e4406b4d0 0x7f2e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.313+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e580ac860 con 0x7f2e6015bb20 2024-03-06T07:27:05.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.313+0000 7f2e65dc6640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2e4406b4d0 0x7f2e4406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0011e0 tx=0x7f2e5c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.317+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2e6015b560 con 0x7f2e6015bb20 2024-03-06T07:27:05.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.525+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e6004f330 con 0x7f2e6015bb20 2024-03-06T07:27:05.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.525+0000 7f2e567fc640 1 -- 172.21.15.72:0/2975842052 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f2e580ac020 con 0x7f2e6015bb20 2024-03-06T07:27:05.528 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:05.528 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2347391904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2347391904},{"type":"v1","addr":"172.21.15.146:6834","nonce":2347391904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/1152083148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1152083148},{"type":"v1","addr":"172.21.15.72:6835","nonce":1152083148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3618393393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":3618393393},{"type":"v1","addr":"172.21.15.72:6837","nonce":3618393393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:04.517794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/118157918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":118157918},{"type":"v1","addr":"172.21.15.146:6835","nonce":118157918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7926,"qdb_cluster":[7926]},"id":8}]} 2024-03-06T07:27:05.528 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 63 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2e4406b4d0 msgr2=0x7f2e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2e4406b4d0 0x7f2e4406d990 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0011e0 tx=0x7f2e5c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 msgr2=0x7f2e60070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 0x7f2e60070170 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f2e58009ee0 tx=0x7f2e58004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 shutdown_connections 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2e4406b4d0 0x7f2e4406d990 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e6015bb20 0x7f2e60070170 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e600d04a0 0x7f2e6006fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 --2- 172.21.15.72:0/2975842052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e600cfff0 0x7f2e6015aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 >> 172.21.15.72:0/2975842052 conn(0x7f2e6009a9f0 msgr2=0x7f2e600cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 shutdown_connections 2024-03-06T07:27:05.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.527+0000 7f2e6802a640 1 -- 172.21.15.72:0/2975842052 wait complete. 2024-03-06T07:27:05.541 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:05.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.639+0000 7f0c13793640 1 Processor -- start 2024-03-06T07:27:05.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c13793640 1 -- start start 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c13793640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c0c0b4f70 0x7f0c0c0b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c13793640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0c0c066d90 0x7f0c0c0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c13793640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c0b5890 con 0x7f0c0c06c6d0 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c13793640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c15e9f0 con 0x7f0c0c066d90 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c13793640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c0cda10 con 0x7f0c0c0b4f70 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.640+0000 7f0c11d30640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0c0c066d90 0x7f0c0c0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0c0c066d90 0x7f0c0c0ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34072/0 (socket says 172.21.15.72:34072) 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 -- 172.21.15.72:0/3818957352 learned_addr learned my addr 172.21.15.72:0/3818957352 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c1152f640 1 --2- 172.21.15.72:0/3818957352 >> v2:172.21.15.146:3300/0 conn(0x7f0c0c0b4f70 0x7f0c0c0b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 -- 172.21.15.72:0/3818957352 >> v2:172.21.15.146:3300/0 conn(0x7f0c0c0b4f70 msgr2=0x7f0c0c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 --2- 172.21.15.72:0/3818957352 >> v2:172.21.15.146:3300/0 conn(0x7f0c0c0b4f70 0x7f0c0c0b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 -- 172.21.15.72:0/3818957352 >> v1:172.21.15.146:6789/0 conn(0x7f0c0c06c6d0 legacy=0x7f0c0c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 -- 172.21.15.72:0/3818957352 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0cd450 con 0x7f0c0c066d90 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.641+0000 7f0c11d30640 1 --2- 172.21.15.72:0/3818957352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0ccb50 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c06cac0 tx=0x7f0c0402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7822ccc508d9c36b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c03fff640 1 -- 172.21.15.72:0/3818957352 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0402cb20 con 0x7f0c0c066d90 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c03fff640 1 -- 172.21.15.72:0/3818957352 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0402ccc0 con 0x7f0c0c066d90 2024-03-06T07:27:05.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c03fff640 1 -- 172.21.15.72:0/3818957352 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04034610 con 0x7f0c0c066d90 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 -- 172.21.15.72:0/3818957352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c066d90 msgr2=0x7f0c0c0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 --2- 172.21.15.72:0/3818957352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0ccb50 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c06cac0 tx=0x7f0c0402bf80 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 -- 172.21.15.72:0/3818957352 shutdown_connections 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 --2- 172.21.15.72:0/3818957352 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0ccb50 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 --2- 172.21.15.72:0/3818957352 >> v2:172.21.15.146:3300/0 conn(0x7f0c0c0b4f70 0x7f0c0c0b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 -- 172.21.15.72:0/3818957352 >> 172.21.15.72:0/3818957352 conn(0x7f0c0c0724d0 msgr2=0x7f0c0c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 -- 172.21.15.72:0/3818957352 shutdown_connections 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.642+0000 7f0c13793640 1 -- 172.21.15.72:0/3818957352 wait complete. 2024-03-06T07:27:05.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.643+0000 7f0c13793640 1 Processor -- start 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.643+0000 7f0c13793640 1 -- start start 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c13793640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c13793640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0b4f70 0x7f0c0c14b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c13793640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 0x7f0c0c0c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c13793640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0cd090 con 0x7f0c0c1551f0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c13793640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c15e680 con 0x7f0c0c066d90 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c13793640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0cd210 con 0x7f0c0c0b4f70 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c11d30640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 0x7f0c0c0c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c11d30640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 0x7f0c0c0c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34088/0 (socket says 172.21.15.72:34088) 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c11d30640 1 -- 172.21.15.72:0/204853574 learned_addr learned my addr 172.21.15.72:0/204853574 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c1152f640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c10d2e640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0b4f70 0x7f0c0c14b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c11d30640 1 -- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0b4f70 msgr2=0x7f0c0c14b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.644+0000 7f0c11d30640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0b4f70 0x7f0c0c14b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c11d30640 1 -- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c066d90 msgr2=0x7f0c0c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c11d30640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c11d30640 1 -- 172.21.15.72:0/204853574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0cb7f0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c11d30640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 0x7f0c0c0c9e10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009f40 tx=0x7f0c04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04043020 con 0x7f0c0c1551f0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0403e450 con 0x7f0c0c1551f0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04009510 con 0x7f0c0c1551f0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c0cbad0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.645+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c0c98f0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.646+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c0c9af0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.646+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0c04005d10 con 0x7f0c0c1551f0 2024-03-06T07:27:05.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.647+0000 7f0c01ffb640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bf006b500 0x7f0bf006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.647+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c040ac4b0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.647+0000 7f0c1152f640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bf006b500 0x7f0bf006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.647+0000 7f0c1152f640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bf006b500 0x7f0bf006d9c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c155d90 tx=0x7f0c08009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.651+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0c0c0c9af0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.822+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c0c04f330 con 0x7f0c0c1551f0 2024-03-06T07:27:05.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.823+0000 7f0c01ffb640 1 -- 172.21.15.72:0/204853574 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f0c040810c0 con 0x7f0c0c1551f0 2024-03-06T07:27:05.826 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:05.826 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":110,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:04.164794+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bf006b500 msgr2=0x7f0bf006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bf006b500 0x7f0bf006d9c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c155d90 tx=0x7f0c08009190 comp rx=0 tx=0).stop 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 msgr2=0x7f0c0c0c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 0x7f0c0c0c9e10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009f40 tx=0x7f0c04004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 shutdown_connections 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0bf006b500 0x7f0bf006d9c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0c0c1551f0 0x7f0c0c0c9e10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0c0b4f70 0x7f0c0c14b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 --2- 172.21.15.72:0/204853574 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c0c066d90 0x7f0c0c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 >> 172.21.15.72:0/204853574 conn(0x7f0c0c0724d0 msgr2=0x7f0c0c0728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 shutdown_connections 2024-03-06T07:27:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.825+0000 7f0c13793640 1 -- 172.21.15.72:0/204853574 wait complete. 2024-03-06T07:27:05.838 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:27:05.839 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:05.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.936+0000 7fcb3ae88640 1 Processor -- start 2024-03-06T07:27:05.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.936+0000 7fcb3ae88640 1 -- start start 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb3ae88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb340b4170 0x7fcb340b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb3ae88640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb340d4760 0x7fcb340d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb3ae88640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb340d25d0 con 0x7fcb340b4dd0 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb3ae88640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb340d4bf0 con 0x7fcb340d4760 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb3ae88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340ce9e0 con 0x7fcb340b4170 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb33fff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb340d4760 0x7fcb340d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb38c24640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb340b4170 0x7fcb340b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb39425640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fcb340b4dd0 0x7fcb340cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:41732/0 (socket says 172.21.15.72:41732) 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb39425640 1 -- 172.21.15.72:0/933803565 learned_addr learned my addr 172.21.15.72:0/933803565 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb33fff640 1 -- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb340b4170 msgr2=0x7fcb340b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb33fff640 1 --2- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb340b4170 0x7fcb340b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.937+0000 7fcb33fff640 1 -- 172.21.15.72:0/933803565 >> v1:172.21.15.146:6789/0 conn(0x7fcb340b4dd0 legacy=0x7fcb340cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.938+0000 7fcb33fff640 1 -- 172.21.15.72:0/933803565 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb340d2870 con 0x7fcb340d4760 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.938+0000 7fcb337fe640 1 -- 172.21.15.72:0/933803565 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312813044 0 0) 0x7fcb340d25d0 con 0x7fcb340b4dd0 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.938+0000 7fcb33fff640 1 --2- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340d4760 0x7fcb340d0370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcb280097a0 tx=0x7fcb2802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d294cef604ef1d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.938+0000 7fcb337fe640 1 -- 172.21.15.72:0/933803565 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28035020 con 0x7fcb340d4760 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.938+0000 7fcb337fe640 1 -- 172.21.15.72:0/933803565 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28031830 con 0x7fcb340d4760 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.938+0000 7fcb337fe640 1 -- 172.21.15.72:0/933803565 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28031c00 con 0x7fcb340d4760 2024-03-06T07:27:05.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 -- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340d4760 msgr2=0x7fcb340d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340d4760 0x7fcb340d0370 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcb280097a0 tx=0x7fcb2802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 -- 172.21.15.72:0/933803565 shutdown_connections 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340d4760 0x7fcb340d0370 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/933803565 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb340b4170 0x7fcb340b6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 -- 172.21.15.72:0/933803565 >> 172.21.15.72:0/933803565 conn(0x7fcb340d3c40 msgr2=0x7fcb340d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 -- 172.21.15.72:0/933803565 shutdown_connections 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.939+0000 7fcb3ae88640 1 -- 172.21.15.72:0/933803565 wait complete. 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.940+0000 7fcb3ae88640 1 Processor -- start 2024-03-06T07:27:05.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.940+0000 7fcb3ae88640 1 -- start start 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb3ae88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 0x7fcb34152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb3ae88640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340b4dd0 0x7fcb340c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb3ae88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb341f3b90 0x7fcb341f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb3ae88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340b47d0 con 0x7fcb340b4170 2024-03-06T07:27:05.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb3ae88640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340b4550 con 0x7fcb340b4dd0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb3ae88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb340b4950 con 0x7fcb341f3b90 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 0x7fcb34152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 0x7fcb34152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34098/0 (socket says 172.21.15.72:34098) 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 -- 172.21.15.72:0/1762373361 learned_addr learned my addr 172.21.15.72:0/1762373361 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb39425640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb341f3b90 0x7fcb341f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb33fff640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340b4dd0 0x7fcb340c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 -- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb341f3b90 msgr2=0x7fcb341f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb341f3b90 0x7fcb341f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 -- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340b4dd0 msgr2=0x7fcb340c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340b4dd0 0x7fcb340c9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.941+0000 7fcb38c24640 1 -- 172.21.15.72:0/1762373361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3415bdc0 con 0x7fcb340b4170 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.942+0000 7fcb38c24640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 0x7fcb34152150 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400e9b0 tx=0x7fcb2400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.942+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24017070 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.942+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2400c9a0 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.942+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3415c0a0 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.942+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2400cd90 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.942+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3415c660 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.943+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3404f330 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.943+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb2401a070 con 0x7fcb340b4170 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.944+0000 7fcb317fa640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb2006b4d0 0x7fcb2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.944+0000 7fcb33fff640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb2006b4d0 0x7fcb2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:05.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.944+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb2408a140 con 0x7fcb340b4170 2024-03-06T07:27:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.944+0000 7fcb33fff640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb2006b4d0 0x7fcb2006d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcb28036000 tx=0x7fcb2803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:05.948+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb24058970 con 0x7fcb340b4170 2024-03-06T07:27:06.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.116+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb340733b0 con 0x7fcb2006b4d0 2024-03-06T07:27:06.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.122+0000 7fcb317fa640 1 -- 172.21.15.72:0/1762373361 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240245 (secure 0 0 0) 0x7fcb28082a90 con 0x7fcb2006b4d0 2024-03-06T07:27:06.126 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:06.128 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:27:06.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb2006b4d0 msgr2=0x7fcb2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb2006b4d0 0x7fcb2006d990 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fcb28036000 tx=0x7fcb2803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 msgr2=0x7fcb34152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 0x7fcb34152150 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7fcb2400e9b0 tx=0x7fcb2400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 shutdown_connections 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb2006b4d0 0x7fcb2006d990 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb341f3b90 0x7fcb341f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb340b4dd0 0x7fcb340c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 --2- 172.21.15.72:0/1762373361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb340b4170 0x7fcb34152150 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 >> 172.21.15.72:0/1762373361 conn(0x7fcb340d3c40 msgr2=0x7fcb340b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 shutdown_connections 2024-03-06T07:27:06.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.125+0000 7fcb3ae88640 1 -- 172.21.15.72:0/1762373361 wait complete. 2024-03-06T07:27:06.138 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:06.005395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323416,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651304,"statfs":{"total":773094113280,"available":772762935296,"internally_reserved":0,"allocated":11837440,"data_stored":5749768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":319213105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.846724"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160432+0000","last_change":"2024-03-06T07:27:01.160432+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160432+0000","last_undegraded":"2024-03-06T07:27:01.160432+0000","last_fullsized":"2024-03-06T07:27:01.160432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162404+0000","last_change":"2024-03-06T07:27:01.162404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162404+0000","last_undegraded":"2024-03-06T07:27:01.162404+0000","last_fullsized":"2024-03-06T07:27:01.162404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164196+0000","last_change":"2024-03-06T07:27:01.164196+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164196+0000","last_undegraded":"2024-03-06T07:27:01.164196+0000","last_fullsized":"2024-03-06T07:27:01.164196+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163484+0000","last_change":"2024-03-06T07:27:01.163484+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163484+0000","last_undegraded":"2024-03-06T07:27:01.163484+0000","last_fullsized":"2024-03-06T07:27:01.163484+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161185+0000","last_change":"2024-03-06T07:27:01.161185+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161185+0000","last_undegraded":"2024-03-06T07:27:01.161185+0000","last_fullsized":"2024-03-06T07:27:01.161185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163456+0000","last_change":"2024-03-06T07:27:01.163456+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163456+0000","last_undegraded":"2024-03-06T07:27:01.163456+0000","last_fullsized":"2024-03-06T07:27:01.163456+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161376+0000","last_change":"2024-03-06T07:27:01.161376+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161376+0000","last_undegraded":"2024-03-06T07:27:01.161376+0000","last_fullsized":"2024-03-06T07:27:01.161376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163990+0000","last_change":"2024-03-06T07:27:01.163990+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163990+0000","last_undegraded":"2024-03-06T07:27:01.163990+0000","last_fullsized":"2024-03-06T07:27:01.163990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163845+0000","last_change":"2024-03-06T07:27:01.163845+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163845+0000","last_undegraded":"2024-03-06T07:27:01.163845+0000","last_fullsized":"2024-03-06T07:27:01.163845+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163280+0000","last_change":"2024-03-06T07:27:01.163280+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163280+0000","last_undegraded":"2024-03-06T07:27:01.163280+0000","last_fullsized":"2024-03-06T07:27:01.163280+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162738+0000","last_change":"2024-03-06T07:27:01.162738+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162738+0000","last_undegraded":"2024-03-06T07:27:01.162738+0000","last_fullsized":"2024-03-06T07:27:01.162738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163027+0000","last_change":"2024-03-06T07:27:01.163027+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163027+0000","last_undegraded":"2024-03-06T07:27:01.163027+0000","last_fullsized":"2024-03-06T07:27:01.163027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161781+0000","last_change":"2024-03-06T07:27:01.161781+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161781+0000","last_undegraded":"2024-03-06T07:27:01.161781+0000","last_fullsized":"2024-03-06T07:27:01.161781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165818+0000","last_change":"2024-03-06T07:27:01.165818+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165818+0000","last_undegraded":"2024-03-06T07:27:01.165818+0000","last_fullsized":"2024-03-06T07:27:01.165818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164299+0000","last_change":"2024-03-06T07:27:01.164299+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164299+0000","last_undegraded":"2024-03-06T07:27:01.164299+0000","last_fullsized":"2024-03-06T07:27:01.164299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162317+0000","last_change":"2024-03-06T07:27:01.162317+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162317+0000","last_undegraded":"2024-03-06T07:27:01.162317+0000","last_fullsized":"2024-03-06T07:27:01.162317+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164672+0000","last_change":"2024-03-06T07:27:01.164672+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164672+0000","last_undegraded":"2024-03-06T07:27:01.164672+0000","last_fullsized":"2024-03-06T07:27:01.164672+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165839+0000","last_change":"2024-03-06T07:27:01.165839+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165839+0000","last_undegraded":"2024-03-06T07:27:01.165839+0000","last_fullsized":"2024-03-06T07:27:01.165839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160694+0000","last_change":"2024-03-06T07:27:01.160694+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160694+0000","last_undegraded":"2024-03-06T07:27:01.160694+0000","last_fullsized":"2024-03-06T07:27:01.160694+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161404+0000","last_change":"2024-03-06T07:27:01.161404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161404+0000","last_undegraded":"2024-03-06T07:27:01.161404+0000","last_fullsized":"2024-03-06T07:27:01.161404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163113+0000","last_change":"2024-03-06T07:27:01.163113+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163113+0000","last_undegraded":"2024-03-06T07:27:01.163113+0000","last_fullsized":"2024-03-06T07:27:01.163113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161201+0000","last_change":"2024-03-06T07:27:01.161201+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161201+0000","last_undegraded":"2024-03-06T07:27:01.161201+0000","last_fullsized":"2024-03-06T07:27:01.161201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163646+0000","last_change":"2024-03-06T07:27:01.163646+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163646+0000","last_undegraded":"2024-03-06T07:27:01.163646+0000","last_fullsized":"2024-03-06T07:27:01.163646+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163830+0000","last_change":"2024-03-06T07:27:01.163830+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163830+0000","last_undegraded":"2024-03-06T07:27:01.163830+0000","last_fullsized":"2024-03-06T07:27:01.163830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165803+0000","last_change":"2024-03-06T07:27:01.165803+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165803+0000","last_undegraded":"2024-03-06T07:27:01.165803+0000","last_fullsized":"2024-03-06T07:27:01.165803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162339+0000","last_change":"2024-03-06T07:27:01.162339+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162339+0000","last_undegraded":"2024-03-06T07:27:01.162339+0000","last_fullsized":"2024-03-06T07:27:01.162339+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163702+0000","last_change":"2024-03-06T07:27:01.163702+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163702+0000","last_undegraded":"2024-03-06T07:27:01.163702+0000","last_fullsized":"2024-03-06T07:27:01.163702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160425+0000","last_change":"2024-03-06T07:27:01.160425+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160425+0000","last_undegraded":"2024-03-06T07:27:01.160425+0000","last_fullsized":"2024-03-06T07:27:01.160425+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160424+0000","last_change":"2024-03-06T07:27:01.160424+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160424+0000","last_undegraded":"2024-03-06T07:27:01.160424+0000","last_fullsized":"2024-03-06T07:27:01.160424+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163123+0000","last_change":"2024-03-06T07:27:01.163123+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163123+0000","last_undegraded":"2024-03-06T07:27:01.163123+0000","last_fullsized":"2024-03-06T07:27:01.163123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163302+0000","last_change":"2024-03-06T07:27:01.163302+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163302+0000","last_undegraded":"2024-03-06T07:27:01.163302+0000","last_fullsized":"2024-03-06T07:27:01.163302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160423+0000","last_change":"2024-03-06T07:27:01.160423+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160423+0000","last_undegraded":"2024-03-06T07:27:01.160423+0000","last_fullsized":"2024-03-06T07:27:01.160423+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":327,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:01.162219+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:27:01.162219+0000","last_peered":"2024-03-06T07:27:01.162219+0000","last_clean":"2024-03-06T07:27:01.162219+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:27:01.162219+0000","last_undegraded":"2024-03-06T07:27:01.162219+0000","last_fullsized":"2024-03-06T07:27:01.162219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:06.142 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:06.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.278+0000 7f00aedd0640 1 Processor -- start 2024-03-06T07:27:06.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.278+0000 7f00aedd0640 1 -- start start 2024-03-06T07:27:06.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00aedd0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f00a80cce40 0x7f00a815b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00aedd0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a80cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00aedd0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a8071f40 con 0x7f00a81515e0 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00aedd0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a80cd2f0 con 0x7f00a80cce40 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00aedd0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a8070a70 con 0x7f00a80cd7a0 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00a7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f00a80cce40 0x7f00a815b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00acb6c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f00a81515e0 0x7f00a8071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59814/0 (socket says 172.21.15.72:59814) 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00acb6c640 1 -- 172.21.15.72:0/1475487418 learned_addr learned my addr 172.21.15.72:0/1475487418 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00ad36d640 1 --2- 172.21.15.72:0/1475487418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a80cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00a7fff640 1 -- 172.21.15.72:0/1475487418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 msgr2=0x7f00a80cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00a7fff640 1 --2- 172.21.15.72:0/1475487418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a80cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00a7fff640 1 -- 172.21.15.72:0/1475487418 >> v1:172.21.15.72:6789/0 conn(0x7f00a81515e0 legacy=0x7f00a8071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00a7fff640 1 -- 172.21.15.72:0/1475487418 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a8070d50 con 0x7f00a80cce40 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.279+0000 7f00a77fe640 1 -- 172.21.15.72:0/1475487418 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554147804 0 0) 0x7f00a8071f40 con 0x7f00a81515e0 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00a7fff640 1 --2- 172.21.15.72:0/1475487418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a815b430 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f00940097a0 tx=0x7f009402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b06d6da0fed6f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00a77fe640 1 -- 172.21.15.72:0/1475487418 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094035020 con 0x7f00a80cce40 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00a77fe640 1 -- 172.21.15.72:0/1475487418 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094031830 con 0x7f00a80cce40 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00a77fe640 1 -- 172.21.15.72:0/1475487418 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094031c00 con 0x7f00a80cce40 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00aedd0640 1 -- 172.21.15.72:0/1475487418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 msgr2=0x7f00a815b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00aedd0640 1 --2- 172.21.15.72:0/1475487418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a815b430 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f00940097a0 tx=0x7f009402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00aedd0640 1 -- 172.21.15.72:0/1475487418 shutdown_connections 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00aedd0640 1 --2- 172.21.15.72:0/1475487418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a80cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00aedd0640 1 --2- 172.21.15.72:0/1475487418 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a815b430 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.280+0000 7f00aedd0640 1 -- 172.21.15.72:0/1475487418 >> 172.21.15.72:0/1475487418 conn(0x7f00a80cf8a0 msgr2=0x7f00a80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.281+0000 7f00aedd0640 1 -- 172.21.15.72:0/1475487418 shutdown_connections 2024-03-06T07:27:06.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.281+0000 7f00aedd0640 1 -- 172.21.15.72:0/1475487418 wait complete. 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 Processor -- start 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 -- start start 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a8144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a81515e0 0x7f00a815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a80ce0f0 con 0x7f00a80cce40 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a8071310 con 0x7f00a81515e0 2024-03-06T07:27:06.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.282+0000 7f00aedd0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a80cc340 con 0x7f00a80cd7a0 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a8144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a8144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34128/0 (socket says 172.21.15.72:34128) 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 -- 172.21.15.72:0/3303963376 learned_addr learned my addr 172.21.15.72:0/3303963376 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00a7fff640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00ad36d640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a81515e0 0x7f00a815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 -- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 msgr2=0x7f00a806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 -- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a81515e0 msgr2=0x7f00a815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a81515e0 0x7f00a815f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 -- 172.21.15.72:0/3303963376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a814bd90 con 0x7f00a80cce40 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00acb6c640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a8144e90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f009c00c400 tx=0x7f009c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c017070 con 0x7f00a80cce40 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c005ce0 con 0x7f00a80cce40 2024-03-06T07:27:06.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a814c070 con 0x7f00a80cce40 2024-03-06T07:27:06.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c007310 con 0x7f00a80cce40 2024-03-06T07:27:06.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.283+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a81453d0 con 0x7f00a80cce40 2024-03-06T07:27:06.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.284+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a8068840 con 0x7f00a80cce40 2024-03-06T07:27:06.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.285+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f009c01a070 con 0x7f00a80cce40 2024-03-06T07:27:06.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.285+0000 7f00a57fa640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.285+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f009c089590 con 0x7f00a80cce40 2024-03-06T07:27:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.285+0000 7f00a7fff640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.286+0000 7f00a7fff640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f008c06b4d0 0x7f008c06d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0094036000 tx=0x7f0094008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.289+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f009c057dc0 con 0x7f00a80cce40 2024-03-06T07:27:06.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.457+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00a806e7d0 con 0x7f008c06b4d0 2024-03-06T07:27:06.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.463+0000 7f00a57fa640 1 -- 172.21.15.72:0/3303963376 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240245 (secure 0 0 0) 0x7f0094046440 con 0x7f008c06b4d0 2024-03-06T07:27:06.467 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:06.468 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f008c06b4d0 msgr2=0x7f008c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f008c06b4d0 0x7f008c06d990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0094036000 tx=0x7f0094008030 comp rx=0 tx=0).stop 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 msgr2=0x7f00a8144e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a8144e90 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f009c00c400 tx=0x7f009c00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 shutdown_connections 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f008c06b4d0 0x7f008c06d990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a81515e0 0x7f00a815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cd7a0 0x7f00a806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 --2- 172.21.15.72:0/3303963376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00a80cce40 0x7f00a8144e90 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 >> 172.21.15.72:0/3303963376 conn(0x7f00a80cf8a0 msgr2=0x7f00a80720d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 shutdown_connections 2024-03-06T07:27:06.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.466+0000 7f00aedd0640 1 -- 172.21.15.72:0/3303963376 wait complete. 2024-03-06T07:27:06.479 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:06.005395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323416,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651304,"statfs":{"total":773094113280,"available":772762935296,"internally_reserved":0,"allocated":11837440,"data_stored":5749768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":319213105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.846724"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160432+0000","last_change":"2024-03-06T07:27:01.160432+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160432+0000","last_undegraded":"2024-03-06T07:27:01.160432+0000","last_fullsized":"2024-03-06T07:27:01.160432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162404+0000","last_change":"2024-03-06T07:27:01.162404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162404+0000","last_undegraded":"2024-03-06T07:27:01.162404+0000","last_fullsized":"2024-03-06T07:27:01.162404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164196+0000","last_change":"2024-03-06T07:27:01.164196+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164196+0000","last_undegraded":"2024-03-06T07:27:01.164196+0000","last_fullsized":"2024-03-06T07:27:01.164196+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163484+0000","last_change":"2024-03-06T07:27:01.163484+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163484+0000","last_undegraded":"2024-03-06T07:27:01.163484+0000","last_fullsized":"2024-03-06T07:27:01.163484+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161185+0000","last_change":"2024-03-06T07:27:01.161185+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161185+0000","last_undegraded":"2024-03-06T07:27:01.161185+0000","last_fullsized":"2024-03-06T07:27:01.161185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163456+0000","last_change":"2024-03-06T07:27:01.163456+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163456+0000","last_undegraded":"2024-03-06T07:27:01.163456+0000","last_fullsized":"2024-03-06T07:27:01.163456+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161376+0000","last_change":"2024-03-06T07:27:01.161376+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161376+0000","last_undegraded":"2024-03-06T07:27:01.161376+0000","last_fullsized":"2024-03-06T07:27:01.161376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163990+0000","last_change":"2024-03-06T07:27:01.163990+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163990+0000","last_undegraded":"2024-03-06T07:27:01.163990+0000","last_fullsized":"2024-03-06T07:27:01.163990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163845+0000","last_change":"2024-03-06T07:27:01.163845+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163845+0000","last_undegraded":"2024-03-06T07:27:01.163845+0000","last_fullsized":"2024-03-06T07:27:01.163845+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163280+0000","last_change":"2024-03-06T07:27:01.163280+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163280+0000","last_undegraded":"2024-03-06T07:27:01.163280+0000","last_fullsized":"2024-03-06T07:27:01.163280+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162738+0000","last_change":"2024-03-06T07:27:01.162738+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162738+0000","last_undegraded":"2024-03-06T07:27:01.162738+0000","last_fullsized":"2024-03-06T07:27:01.162738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163027+0000","last_change":"2024-03-06T07:27:01.163027+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163027+0000","last_undegraded":"2024-03-06T07:27:01.163027+0000","last_fullsized":"2024-03-06T07:27:01.163027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161781+0000","last_change":"2024-03-06T07:27:01.161781+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161781+0000","last_undegraded":"2024-03-06T07:27:01.161781+0000","last_fullsized":"2024-03-06T07:27:01.161781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165818+0000","last_change":"2024-03-06T07:27:01.165818+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165818+0000","last_undegraded":"2024-03-06T07:27:01.165818+0000","last_fullsized":"2024-03-06T07:27:01.165818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164299+0000","last_change":"2024-03-06T07:27:01.164299+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164299+0000","last_undegraded":"2024-03-06T07:27:01.164299+0000","last_fullsized":"2024-03-06T07:27:01.164299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162317+0000","last_change":"2024-03-06T07:27:01.162317+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162317+0000","last_undegraded":"2024-03-06T07:27:01.162317+0000","last_fullsized":"2024-03-06T07:27:01.162317+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164672+0000","last_change":"2024-03-06T07:27:01.164672+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164672+0000","last_undegraded":"2024-03-06T07:27:01.164672+0000","last_fullsized":"2024-03-06T07:27:01.164672+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165839+0000","last_change":"2024-03-06T07:27:01.165839+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165839+0000","last_undegraded":"2024-03-06T07:27:01.165839+0000","last_fullsized":"2024-03-06T07:27:01.165839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160694+0000","last_change":"2024-03-06T07:27:01.160694+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160694+0000","last_undegraded":"2024-03-06T07:27:01.160694+0000","last_fullsized":"2024-03-06T07:27:01.160694+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161404+0000","last_change":"2024-03-06T07:27:01.161404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161404+0000","last_undegraded":"2024-03-06T07:27:01.161404+0000","last_fullsized":"2024-03-06T07:27:01.161404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163113+0000","last_change":"2024-03-06T07:27:01.163113+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163113+0000","last_undegraded":"2024-03-06T07:27:01.163113+0000","last_fullsized":"2024-03-06T07:27:01.163113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161201+0000","last_change":"2024-03-06T07:27:01.161201+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161201+0000","last_undegraded":"2024-03-06T07:27:01.161201+0000","last_fullsized":"2024-03-06T07:27:01.161201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163646+0000","last_change":"2024-03-06T07:27:01.163646+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163646+0000","last_undegraded":"2024-03-06T07:27:01.163646+0000","last_fullsized":"2024-03-06T07:27:01.163646+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163830+0000","last_change":"2024-03-06T07:27:01.163830+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163830+0000","last_undegraded":"2024-03-06T07:27:01.163830+0000","last_fullsized":"2024-03-06T07:27:01.163830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165803+0000","last_change":"2024-03-06T07:27:01.165803+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165803+0000","last_undegraded":"2024-03-06T07:27:01.165803+0000","last_fullsized":"2024-03-06T07:27:01.165803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162339+0000","last_change":"2024-03-06T07:27:01.162339+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162339+0000","last_undegraded":"2024-03-06T07:27:01.162339+0000","last_fullsized":"2024-03-06T07:27:01.162339+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163702+0000","last_change":"2024-03-06T07:27:01.163702+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163702+0000","last_undegraded":"2024-03-06T07:27:01.163702+0000","last_fullsized":"2024-03-06T07:27:01.163702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160425+0000","last_change":"2024-03-06T07:27:01.160425+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160425+0000","last_undegraded":"2024-03-06T07:27:01.160425+0000","last_fullsized":"2024-03-06T07:27:01.160425+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160424+0000","last_change":"2024-03-06T07:27:01.160424+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160424+0000","last_undegraded":"2024-03-06T07:27:01.160424+0000","last_fullsized":"2024-03-06T07:27:01.160424+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163123+0000","last_change":"2024-03-06T07:27:01.163123+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163123+0000","last_undegraded":"2024-03-06T07:27:01.163123+0000","last_fullsized":"2024-03-06T07:27:01.163123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163302+0000","last_change":"2024-03-06T07:27:01.163302+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163302+0000","last_undegraded":"2024-03-06T07:27:01.163302+0000","last_fullsized":"2024-03-06T07:27:01.163302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160423+0000","last_change":"2024-03-06T07:27:01.160423+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160423+0000","last_undegraded":"2024-03-06T07:27:01.160423+0000","last_fullsized":"2024-03-06T07:27:01.160423+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":327,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:01.162219+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:27:01.162219+0000","last_peered":"2024-03-06T07:27:01.162219+0000","last_clean":"2024-03-06T07:27:01.162219+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:27:01.162219+0000","last_undegraded":"2024-03-06T07:27:01.162219+0000","last_fullsized":"2024-03-06T07:27:01.162219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:06.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:06.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.617+0000 7efc31c64640 1 Processor -- start 2024-03-06T07:27:06.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc31c64640 1 -- start start 2024-03-06T07:27:06.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc31c64640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc2c15dfe0 0x7efc2c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc31c64640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc2c0cf830 0x7efc2c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc31c64640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc2c0b3430 con 0x7efc2c0d27f0 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc31c64640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc2c0b31e0 con 0x7efc2c15dfe0 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc31c64640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc2c0b3d60 con 0x7efc2c0cf830 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc2b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc2c15dfe0 0x7efc2c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc2b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc2c15dfe0 0x7efc2c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34142/0 (socket says 172.21.15.72:34142) 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.618+0000 7efc2b7fe640 1 -- 172.21.15.72:0/1106690903 learned_addr learned my addr 172.21.15.72:0/1106690903 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.619+0000 7efc22ffd640 1 --2- 172.21.15.72:0/1106690903 >> v2:172.21.15.146:3300/0 conn(0x7efc2c0cf830 0x7efc2c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.619+0000 7efc2b7fe640 1 -- 172.21.15.72:0/1106690903 >> v2:172.21.15.146:3300/0 conn(0x7efc2c0cf830 msgr2=0x7efc2c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.619+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/1106690903 >> v2:172.21.15.146:3300/0 conn(0x7efc2c0cf830 0x7efc2c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.619+0000 7efc2b7fe640 1 -- 172.21.15.72:0/1106690903 >> v1:172.21.15.146:6789/0 conn(0x7efc2c0d27f0 legacy=0x7efc2c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.619+0000 7efc2b7fe640 1 -- 172.21.15.72:0/1106690903 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c0b3710 con 0x7efc2c15dfe0 2024-03-06T07:27:06.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.619+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/1106690903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cf2f0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7efc1c009870 tx=0x7efc1c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3f46439541eaf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc2affd640 1 -- 172.21.15.72:0/1106690903 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c034070 con 0x7efc2c15dfe0 2024-03-06T07:27:06.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc2affd640 1 -- 172.21.15.72:0/1106690903 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc1c030860 con 0x7efc2c15dfe0 2024-03-06T07:27:06.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc2affd640 1 -- 172.21.15.72:0/1106690903 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc1c030c30 con 0x7efc2c15dfe0 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 -- 172.21.15.72:0/1106690903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 msgr2=0x7efc2c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 --2- 172.21.15.72:0/1106690903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cf2f0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7efc1c009870 tx=0x7efc1c02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 -- 172.21.15.72:0/1106690903 shutdown_connections 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 --2- 172.21.15.72:0/1106690903 >> v2:172.21.15.146:3300/0 conn(0x7efc2c0cf830 0x7efc2c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 --2- 172.21.15.72:0/1106690903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cf2f0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 -- 172.21.15.72:0/1106690903 >> 172.21.15.72:0/1106690903 conn(0x7efc2c0d3cd0 msgr2=0x7efc2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 -- 172.21.15.72:0/1106690903 shutdown_connections 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.620+0000 7efc31c64640 1 -- 172.21.15.72:0/1106690903 wait complete. 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.621+0000 7efc31c64640 1 Processor -- start 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 -- start start 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc2c0cf830 0x7efc2c15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc2c0d27f0 0x7efc2c0cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c0d1df0 con 0x7efc2c15dfe0 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c0d1c70 con 0x7efc2c0cf830 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc31c64640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc2c15e3c0 con 0x7efc2c0d27f0 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc2bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc2bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34152/0 (socket says 172.21.15.72:34152) 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc2bfff640 1 -- 172.21.15.72:0/2436352098 learned_addr learned my addr 172.21.15.72:0/2436352098 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc2c0cf830 0x7efc2c15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.622+0000 7efc22ffd640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc2c0d27f0 0x7efc2c0cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc2bfff640 1 -- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc2c0d27f0 msgr2=0x7efc2c0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc2bfff640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc2c0d27f0 0x7efc2c0cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc2bfff640 1 -- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc2c0cf830 msgr2=0x7efc2c15c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc2bfff640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc2c0cf830 0x7efc2c15c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc2bfff640 1 -- 172.21.15.72:0/2436352098 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc2c15fa70 con 0x7efc2c15dfe0 2024-03-06T07:27:06.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc2bfff640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cd850 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7efc2400c7a0 tx=0x7efc2400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc24003b70 con 0x7efc2c15dfe0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc24003d30 con 0x7efc2c15dfe0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2400aa90 con 0x7efc2c15dfe0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc2c15fd50 con 0x7efc2c15dfe0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.623+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc2c160310 con 0x7efc2c15dfe0 2024-03-06T07:27:06.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.624+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc2c160740 con 0x7efc2c15dfe0 2024-03-06T07:27:06.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.624+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc2400ac30 con 0x7efc2c15dfe0 2024-03-06T07:27:06.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.625+0000 7efc28ff9640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc0c06b4d0 0x7efc0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.625+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc24089f70 con 0x7efc2c15dfe0 2024-03-06T07:27:06.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.625+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc0c06b4d0 0x7efc0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.625+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc0c06b4d0 0x7efc0c06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7efc1c036000 tx=0x7efc1c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.629+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc24058830 con 0x7efc2c15dfe0 2024-03-06T07:27:06.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.797+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc2c04f330 con 0x7efc0c06b4d0 2024-03-06T07:27:06.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.803+0000 7efc28ff9640 1 -- 172.21.15.72:0/2436352098 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240245 (secure 0 0 0) 0x7efc2c04f330 con 0x7efc0c06b4d0 2024-03-06T07:27:06.807 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:06.808 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc0c06b4d0 msgr2=0x7efc0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc0c06b4d0 0x7efc0c06d990 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7efc1c036000 tx=0x7efc1c008030 comp rx=0 tx=0).stop 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 msgr2=0x7efc2c0cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cd850 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7efc2400c7a0 tx=0x7efc2400cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 shutdown_connections 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc0c06b4d0 0x7efc0c06d990 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15dfe0 0x7efc2c0cd850 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc2c0d27f0 0x7efc2c0cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 --2- 172.21.15.72:0/2436352098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc2c0cf830 0x7efc2c15c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 >> 172.21.15.72:0/2436352098 conn(0x7efc2c0d3cd0 msgr2=0x7efc2c06c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 shutdown_connections 2024-03-06T07:27:06.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.806+0000 7efc31c64640 1 -- 172.21.15.72:0/2436352098 wait complete. 2024-03-06T07:27:06.819 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:06.005395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323416,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651304,"statfs":{"total":773094113280,"available":772762935296,"internally_reserved":0,"allocated":11837440,"data_stored":5749768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":319213105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.846724"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160432+0000","last_change":"2024-03-06T07:27:01.160432+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160432+0000","last_undegraded":"2024-03-06T07:27:01.160432+0000","last_fullsized":"2024-03-06T07:27:01.160432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162404+0000","last_change":"2024-03-06T07:27:01.162404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162404+0000","last_undegraded":"2024-03-06T07:27:01.162404+0000","last_fullsized":"2024-03-06T07:27:01.162404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164196+0000","last_change":"2024-03-06T07:27:01.164196+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164196+0000","last_undegraded":"2024-03-06T07:27:01.164196+0000","last_fullsized":"2024-03-06T07:27:01.164196+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163484+0000","last_change":"2024-03-06T07:27:01.163484+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163484+0000","last_undegraded":"2024-03-06T07:27:01.163484+0000","last_fullsized":"2024-03-06T07:27:01.163484+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161185+0000","last_change":"2024-03-06T07:27:01.161185+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161185+0000","last_undegraded":"2024-03-06T07:27:01.161185+0000","last_fullsized":"2024-03-06T07:27:01.161185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163456+0000","last_change":"2024-03-06T07:27:01.163456+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163456+0000","last_undegraded":"2024-03-06T07:27:01.163456+0000","last_fullsized":"2024-03-06T07:27:01.163456+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161376+0000","last_change":"2024-03-06T07:27:01.161376+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161376+0000","last_undegraded":"2024-03-06T07:27:01.161376+0000","last_fullsized":"2024-03-06T07:27:01.161376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163990+0000","last_change":"2024-03-06T07:27:01.163990+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163990+0000","last_undegraded":"2024-03-06T07:27:01.163990+0000","last_fullsized":"2024-03-06T07:27:01.163990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163845+0000","last_change":"2024-03-06T07:27:01.163845+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163845+0000","last_undegraded":"2024-03-06T07:27:01.163845+0000","last_fullsized":"2024-03-06T07:27:01.163845+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163280+0000","last_change":"2024-03-06T07:27:01.163280+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163280+0000","last_undegraded":"2024-03-06T07:27:01.163280+0000","last_fullsized":"2024-03-06T07:27:01.163280+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162738+0000","last_change":"2024-03-06T07:27:01.162738+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162738+0000","last_undegraded":"2024-03-06T07:27:01.162738+0000","last_fullsized":"2024-03-06T07:27:01.162738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163027+0000","last_change":"2024-03-06T07:27:01.163027+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163027+0000","last_undegraded":"2024-03-06T07:27:01.163027+0000","last_fullsized":"2024-03-06T07:27:01.163027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161781+0000","last_change":"2024-03-06T07:27:01.161781+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161781+0000","last_undegraded":"2024-03-06T07:27:01.161781+0000","last_fullsized":"2024-03-06T07:27:01.161781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165818+0000","last_change":"2024-03-06T07:27:01.165818+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165818+0000","last_undegraded":"2024-03-06T07:27:01.165818+0000","last_fullsized":"2024-03-06T07:27:01.165818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164299+0000","last_change":"2024-03-06T07:27:01.164299+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164299+0000","last_undegraded":"2024-03-06T07:27:01.164299+0000","last_fullsized":"2024-03-06T07:27:01.164299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162317+0000","last_change":"2024-03-06T07:27:01.162317+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162317+0000","last_undegraded":"2024-03-06T07:27:01.162317+0000","last_fullsized":"2024-03-06T07:27:01.162317+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164672+0000","last_change":"2024-03-06T07:27:01.164672+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164672+0000","last_undegraded":"2024-03-06T07:27:01.164672+0000","last_fullsized":"2024-03-06T07:27:01.164672+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165839+0000","last_change":"2024-03-06T07:27:01.165839+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165839+0000","last_undegraded":"2024-03-06T07:27:01.165839+0000","last_fullsized":"2024-03-06T07:27:01.165839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160694+0000","last_change":"2024-03-06T07:27:01.160694+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160694+0000","last_undegraded":"2024-03-06T07:27:01.160694+0000","last_fullsized":"2024-03-06T07:27:01.160694+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161404+0000","last_change":"2024-03-06T07:27:01.161404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161404+0000","last_undegraded":"2024-03-06T07:27:01.161404+0000","last_fullsized":"2024-03-06T07:27:01.161404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163113+0000","last_change":"2024-03-06T07:27:01.163113+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163113+0000","last_undegraded":"2024-03-06T07:27:01.163113+0000","last_fullsized":"2024-03-06T07:27:01.163113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161201+0000","last_change":"2024-03-06T07:27:01.161201+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161201+0000","last_undegraded":"2024-03-06T07:27:01.161201+0000","last_fullsized":"2024-03-06T07:27:01.161201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163646+0000","last_change":"2024-03-06T07:27:01.163646+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163646+0000","last_undegraded":"2024-03-06T07:27:01.163646+0000","last_fullsized":"2024-03-06T07:27:01.163646+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163830+0000","last_change":"2024-03-06T07:27:01.163830+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163830+0000","last_undegraded":"2024-03-06T07:27:01.163830+0000","last_fullsized":"2024-03-06T07:27:01.163830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165803+0000","last_change":"2024-03-06T07:27:01.165803+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165803+0000","last_undegraded":"2024-03-06T07:27:01.165803+0000","last_fullsized":"2024-03-06T07:27:01.165803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162339+0000","last_change":"2024-03-06T07:27:01.162339+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162339+0000","last_undegraded":"2024-03-06T07:27:01.162339+0000","last_fullsized":"2024-03-06T07:27:01.162339+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163702+0000","last_change":"2024-03-06T07:27:01.163702+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163702+0000","last_undegraded":"2024-03-06T07:27:01.163702+0000","last_fullsized":"2024-03-06T07:27:01.163702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160425+0000","last_change":"2024-03-06T07:27:01.160425+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160425+0000","last_undegraded":"2024-03-06T07:27:01.160425+0000","last_fullsized":"2024-03-06T07:27:01.160425+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160424+0000","last_change":"2024-03-06T07:27:01.160424+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160424+0000","last_undegraded":"2024-03-06T07:27:01.160424+0000","last_fullsized":"2024-03-06T07:27:01.160424+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163123+0000","last_change":"2024-03-06T07:27:01.163123+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163123+0000","last_undegraded":"2024-03-06T07:27:01.163123+0000","last_fullsized":"2024-03-06T07:27:01.163123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163302+0000","last_change":"2024-03-06T07:27:01.163302+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163302+0000","last_undegraded":"2024-03-06T07:27:01.163302+0000","last_fullsized":"2024-03-06T07:27:01.163302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160423+0000","last_change":"2024-03-06T07:27:01.160423+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160423+0000","last_undegraded":"2024-03-06T07:27:01.160423+0000","last_fullsized":"2024-03-06T07:27:01.160423+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":327,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:01.162219+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:27:01.162219+0000","last_peered":"2024-03-06T07:27:01.162219+0000","last_clean":"2024-03-06T07:27:01.162219+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:27:01.162219+0000","last_undegraded":"2024-03-06T07:27:01.162219+0000","last_fullsized":"2024-03-06T07:27:01.162219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:06.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3f is not active+clean 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3e is not active+clean 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3d is not active+clean 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3c is not active+clean 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3b is not active+clean 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3a is not active+clean 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.39 is not active+clean 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.38 is not active+clean 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.37 is not active+clean 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.36 is not active+clean 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.35 is not active+clean 2024-03-06T07:27:06.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.34 is not active+clean 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.33 is not active+clean 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.32 is not active+clean 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.31 is not active+clean 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.30 is not active+clean 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2f is not active+clean 2024-03-06T07:27:06.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2e is not active+clean 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2d is not active+clean 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2c is not active+clean 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2b is not active+clean 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2a is not active+clean 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.29 is not active+clean 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.28 is not active+clean 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.27 is not active+clean 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.26 is not active+clean 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.25 is not active+clean 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.24 is not active+clean 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.23 is not active+clean 2024-03-06T07:27:06.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.22 is not active+clean 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.21 is not active+clean 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.20 is not active+clean 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1e is not active+clean 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1f is not active+clean 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.160432+0000', 'last_change': '2024-03-06T07:27:01.160432+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.160432+0000', 'last_undegraded': '2024-03-06T07:27:01.160432+0000', 'last_fullsized': '2024-03-06T07:27:01.160432+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:06.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1f is not active+clean 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1e is not active+clean 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.162404+0000', 'last_change': '2024-03-06T07:27:01.162404+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.162404+0000', 'last_undegraded': '2024-03-06T07:27:01.162404+0000', 'last_fullsized': '2024-03-06T07:27:01.162404+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1c is not active+clean 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1d is not active+clean 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.164196+0000', 'last_change': '2024-03-06T07:27:01.164196+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.164196+0000', 'last_undegraded': '2024-03-06T07:27:01.164196+0000', 'last_fullsized': '2024-03-06T07:27:01.164196+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1d is not active+clean 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1c is not active+clean 2024-03-06T07:27:06.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163484+0000', 'last_change': '2024-03-06T07:27:01.163484+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163484+0000', 'last_undegraded': '2024-03-06T07:27:01.163484+0000', 'last_fullsized': '2024-03-06T07:27:01.163484+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1a is not active+clean 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1b is not active+clean 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.161185+0000', 'last_change': '2024-03-06T07:27:01.161185+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.161185+0000', 'last_undegraded': '2024-03-06T07:27:01.161185+0000', 'last_fullsized': '2024-03-06T07:27:01.161185+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1b is not active+clean 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1a is not active+clean 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163456+0000', 'last_change': '2024-03-06T07:27:01.163456+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163456+0000', 'last_undegraded': '2024-03-06T07:27:01.163456+0000', 'last_fullsized': '2024-03-06T07:27:01.163456+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.18 is not active+clean 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.19 is not active+clean 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.161376+0000', 'last_change': '2024-03-06T07:27:01.161376+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.161376+0000', 'last_undegraded': '2024-03-06T07:27:01.161376+0000', 'last_fullsized': '2024-03-06T07:27:01.161376+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.19 is not active+clean 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.18 is not active+clean 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163990+0000', 'last_change': '2024-03-06T07:27:01.163990+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163990+0000', 'last_undegraded': '2024-03-06T07:27:01.163990+0000', 'last_fullsized': '2024-03-06T07:27:01.163990+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.16 is not active+clean 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.17 is not active+clean 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163845+0000', 'last_change': '2024-03-06T07:27:01.163845+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163845+0000', 'last_undegraded': '2024-03-06T07:27:01.163845+0000', 'last_fullsized': '2024-03-06T07:27:01.163845+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.17 is not active+clean 2024-03-06T07:27:06.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.16 is not active+clean 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163280+0000', 'last_change': '2024-03-06T07:27:01.163280+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163280+0000', 'last_undegraded': '2024-03-06T07:27:01.163280+0000', 'last_fullsized': '2024-03-06T07:27:01.163280+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.14 is not active+clean 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.15 is not active+clean 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.162738+0000', 'last_change': '2024-03-06T07:27:01.162738+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.162738+0000', 'last_undegraded': '2024-03-06T07:27:01.162738+0000', 'last_fullsized': '2024-03-06T07:27:01.162738+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.15 is not active+clean 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.14 is not active+clean 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163027+0000', 'last_change': '2024-03-06T07:27:01.163027+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163027+0000', 'last_undegraded': '2024-03-06T07:27:01.163027+0000', 'last_fullsized': '2024-03-06T07:27:01.163027+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.832 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.12 is not active+clean 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.13 is not active+clean 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.161781+0000', 'last_change': '2024-03-06T07:27:01.161781+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.161781+0000', 'last_undegraded': '2024-03-06T07:27:01.161781+0000', 'last_fullsized': '2024-03-06T07:27:01.161781+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.13 is not active+clean 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.12 is not active+clean 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.165818+0000', 'last_change': '2024-03-06T07:27:01.165818+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.165818+0000', 'last_undegraded': '2024-03-06T07:27:01.165818+0000', 'last_fullsized': '2024-03-06T07:27:01.165818+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.11 is not active+clean 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.10 is not active+clean 2024-03-06T07:27:06.833 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.164299+0000', 'last_change': '2024-03-06T07:27:01.164299+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.164299+0000', 'last_undegraded': '2024-03-06T07:27:01.164299+0000', 'last_fullsized': '2024-03-06T07:27:01.164299+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.e is not active+clean 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.f is not active+clean 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.162317+0000', 'last_change': '2024-03-06T07:27:01.162317+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.162317+0000', 'last_undegraded': '2024-03-06T07:27:01.162317+0000', 'last_fullsized': '2024-03-06T07:27:01.162317+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.f is not active+clean 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.e is not active+clean 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.164672+0000', 'last_change': '2024-03-06T07:27:01.164672+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.164672+0000', 'last_undegraded': '2024-03-06T07:27:01.164672+0000', 'last_fullsized': '2024-03-06T07:27:01.164672+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.c is not active+clean 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.d is not active+clean 2024-03-06T07:27:06.834 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.165839+0000', 'last_change': '2024-03-06T07:27:01.165839+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.165839+0000', 'last_undegraded': '2024-03-06T07:27:01.165839+0000', 'last_fullsized': '2024-03-06T07:27:01.165839+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.d is not active+clean 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.c is not active+clean 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.160694+0000', 'last_change': '2024-03-06T07:27:01.160694+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.160694+0000', 'last_undegraded': '2024-03-06T07:27:01.160694+0000', 'last_fullsized': '2024-03-06T07:27:01.160694+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.a is not active+clean 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.b is not active+clean 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.161404+0000', 'last_change': '2024-03-06T07:27:01.161404+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.161404+0000', 'last_undegraded': '2024-03-06T07:27:01.161404+0000', 'last_fullsized': '2024-03-06T07:27:01.161404+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.b is not active+clean 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.a is not active+clean 2024-03-06T07:27:06.835 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163113+0000', 'last_change': '2024-03-06T07:27:01.163113+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163113+0000', 'last_undegraded': '2024-03-06T07:27:01.163113+0000', 'last_fullsized': '2024-03-06T07:27:01.163113+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.8 is not active+clean 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.9 is not active+clean 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.161201+0000', 'last_change': '2024-03-06T07:27:01.161201+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.161201+0000', 'last_undegraded': '2024-03-06T07:27:01.161201+0000', 'last_fullsized': '2024-03-06T07:27:01.161201+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.9 is not active+clean 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.8 is not active+clean 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163646+0000', 'last_change': '2024-03-06T07:27:01.163646+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163646+0000', 'last_undegraded': '2024-03-06T07:27:01.163646+0000', 'last_fullsized': '2024-03-06T07:27:01.163646+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.6 is not active+clean 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.836 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.7 is not active+clean 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163830+0000', 'last_change': '2024-03-06T07:27:01.163830+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163830+0000', 'last_undegraded': '2024-03-06T07:27:01.163830+0000', 'last_fullsized': '2024-03-06T07:27:01.163830+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.7 is not active+clean 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.6 is not active+clean 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.165803+0000', 'last_change': '2024-03-06T07:27:01.165803+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.165803+0000', 'last_undegraded': '2024-03-06T07:27:01.165803+0000', 'last_fullsized': '2024-03-06T07:27:01.165803+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.4 is not active+clean 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.5 is not active+clean 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.162339+0000', 'last_change': '2024-03-06T07:27:01.162339+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.162339+0000', 'last_undegraded': '2024-03-06T07:27:01.162339+0000', 'last_fullsized': '2024-03-06T07:27:01.162339+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.5 is not active+clean 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.837 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.4 is not active+clean 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163702+0000', 'last_change': '2024-03-06T07:27:01.163702+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163702+0000', 'last_undegraded': '2024-03-06T07:27:01.163702+0000', 'last_fullsized': '2024-03-06T07:27:01.163702+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.2 is not active+clean 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.3 is not active+clean 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.160425+0000', 'last_change': '2024-03-06T07:27:01.160425+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.160425+0000', 'last_undegraded': '2024-03-06T07:27:01.160425+0000', 'last_fullsized': '2024-03-06T07:27:01.160425+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.3 is not active+clean 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.2 is not active+clean 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.160424+0000', 'last_change': '2024-03-06T07:27:01.160424+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.160424+0000', 'last_undegraded': '2024-03-06T07:27:01.160424+0000', 'last_fullsized': '2024-03-06T07:27:01.160424+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.0 is not active+clean 2024-03-06T07:27:06.838 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.1 is not active+clean 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163123+0000', 'last_change': '2024-03-06T07:27:01.163123+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163123+0000', 'last_undegraded': '2024-03-06T07:27:01.163123+0000', 'last_fullsized': '2024-03-06T07:27:01.163123+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.1 is not active+clean 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.0 is not active+clean 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.163302+0000', 'last_change': '2024-03-06T07:27:01.163302+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.163302+0000', 'last_undegraded': '2024-03-06T07:27:01.163302+0000', 'last_fullsized': '2024-03-06T07:27:01.163302+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:PG 17.10 is not active+clean 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-06T07:27:02.158127+0000', 'last_change': '2024-03-06T07:27:02.158127+0000', 'last_active': '2024-03-06T07:27:02.158127+0000', 'last_peered': '2024-03-06T07:27:02.158127+0000', 'last_clean': '2024-03-06T07:27:02.158127+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:02.158127+0000', 'last_undegraded': '2024-03-06T07:27:02.158127+0000', 'last_fullsized': '2024-03-06T07:27:02.158127+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:02.158127+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:PG 16.11 is not active+clean 2024-03-06T07:27:06.839 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '16.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 107, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:27:01.160423+0000', 'last_change': '2024-03-06T07:27:01.160423+0000', 'last_active': '2024-03-06T07:27:01.149204+0000', 'last_peered': '2024-03-06T07:27:01.149204+0000', 'last_clean': '2024-03-06T07:27:01.149204+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:27:01.160423+0000', 'last_undegraded': '2024-03-06T07:27:01.160423+0000', 'last_fullsized': '2024-03-06T07:27:01.160423+0000', 'mapping_epoch': 107, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 107, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'last_clean_scrub_stamp': '2024-03-06T07:27:01.149204+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:27:06.840 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:27:06.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.955+0000 7f07c6f81640 1 Processor -- start 2024-03-06T07:27:06.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.955+0000 7f07c6f81640 1 -- start start 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c6f81640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07c00b4460 0x7f07c00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c6f81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c6f81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c00d22c0 con 0x7f07c0162970 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c6f81640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c0161b00 con 0x7f07c00b4460 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c6f81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00cecc0 con 0x7f07c0161cf0 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c4d1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07c00b4460 0x7f07c00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07bffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f07c0162970 0x7f07c00b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59826/0 (socket says 172.21.15.72:59826) 2024-03-06T07:27:06.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07bffff640 1 -- 172.21.15.72:0/2444296124 learned_addr learned my addr 172.21.15.72:0/2444296124 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.956+0000 7f07c551e640 1 --2- 172.21.15.72:0/2444296124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.957+0000 7f07c4d1d640 1 -- 172.21.15.72:0/2444296124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 msgr2=0x7f07c0162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.957+0000 7f07c4d1d640 1 --2- 172.21.15.72:0/2444296124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.957+0000 7f07c4d1d640 1 -- 172.21.15.72:0/2444296124 >> v1:172.21.15.72:6789/0 conn(0x7f07c0162970 legacy=0x7f07c00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.957+0000 7f07c4d1d640 1 -- 172.21.15.72:0/2444296124 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c00d25a0 con 0x7f07c00b4460 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.957+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2444296124 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203103874 0 0) 0x7f07c00d22c0 con 0x7f07c0162970 2024-03-06T07:27:06.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.957+0000 7f07c4d1d640 1 --2- 172.21.15.72:0/2444296124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c00d4800 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7eaa584dcb559f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2444296124 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0035020 con 0x7f07c00b4460 2024-03-06T07:27:06.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2444296124 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0031830 con 0x7f07c00b4460 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07bf7fe640 1 -- 172.21.15.72:0/2444296124 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0031c00 con 0x7f07c00b4460 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 -- 172.21.15.72:0/2444296124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 msgr2=0x7f07c00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 --2- 172.21.15.72:0/2444296124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c00d4800 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 -- 172.21.15.72:0/2444296124 shutdown_connections 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 --2- 172.21.15.72:0/2444296124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 --2- 172.21.15.72:0/2444296124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c00d4800 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 -- 172.21.15.72:0/2444296124 >> 172.21.15.72:0/2444296124 conn(0x7f07c00d3ce0 msgr2=0x7f07c00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 -- 172.21.15.72:0/2444296124 shutdown_connections 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.958+0000 7f07c6f81640 1 -- 172.21.15.72:0/2444296124 wait complete. 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.959+0000 7f07c6f81640 1 Processor -- start 2024-03-06T07:27:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.959+0000 7f07c6f81640 1 -- start start 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c6f81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c0158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c6f81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c6f81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c0162970 0x7f07c00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c6f81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00ce950 con 0x7f07c00b4460 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c6f81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00d2b60 con 0x7f07c0162970 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c6f81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00b4840 con 0x7f07c0161cf0 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c4d1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c0158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c4d1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c0158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34168/0 (socket says 172.21.15.72:34168) 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c4d1d640 1 -- 172.21.15.72:0/1630020993 learned_addr learned my addr 172.21.15.72:0/1630020993 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07c551e640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c0162970 0x7f07c00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.960+0000 7f07bffff640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c4d1d640 1 -- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 msgr2=0x7f07c0155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c4d1d640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c4d1d640 1 -- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c0162970 msgr2=0x7f07c00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c4d1d640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c0162970 0x7f07c00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c4d1d640 1 -- 172.21.15.72:0/1630020993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c00cd900 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c4d1d640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c0158810 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f07b00098c0 tx=0x7f07b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0040070 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b003e070 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0035040 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c0157770 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.961+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c0157d30 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.962+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c00671c0 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.962+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07b0038020 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.963+0000 7f07bd7fa640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07ac06b4d0 0x7f07ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.963+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07b00ab920 con 0x7f07c00b4460 2024-03-06T07:27:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.963+0000 7f07bffff640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07ac06b4d0 0x7f07ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:06.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.963+0000 7f07bffff640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07ac06b4d0 0x7f07ac06d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f07c00b9520 tx=0x7f07b4007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:06.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:06.967+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07c00671c0 con 0x7f07c00b4460 2024-03-06T07:27:07.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.207+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f07c004f330 con 0x7f07c00b4460 2024-03-06T07:27:07.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.208+0000 7f07bd7fa640 1 -- 172.21.15.72:0/1630020993 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1529 (secure 0 0 0) 0x7f07b0031800 con 0x7f07c00b4460 2024-03-06T07:27:07.210 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:07.210 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":110,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709696,"num_in_osds":8,"osd_in_since":1709709687,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":331177984,"bytes_avail":772762935296,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":63,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"b","status":"up:active","gid":7926}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-06T07:26:19.996494+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-06T07:27:07.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07ac06b4d0 msgr2=0x7f07ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07ac06b4d0 0x7f07ac06d990 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f07c00b9520 tx=0x7f07b4007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:07.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 msgr2=0x7f07c0158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c0158810 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f07b00098c0 tx=0x7f07b00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 shutdown_connections 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07ac06b4d0 0x7f07ac06d990 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c0162970 0x7f07c00cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c0161cf0 0x7f07c0155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 --2- 172.21.15.72:0/1630020993 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07c00b4460 0x7f07c0158810 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 >> 172.21.15.72:0/1630020993 conn(0x7f07c00d3ce0 msgr2=0x7f07c00d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 shutdown_connections 2024-03-06T07:27:07.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.210+0000 7f07c6f81640 1 -- 172.21.15.72:0/1630020993 wait complete. 2024-03-06T07:27:07.223 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:27:07.223 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.322+0000 7f59484fb640 1 Processor -- start 2024-03-06T07:27:07.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.322+0000 7f59484fb640 1 -- start start 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f59484fb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f594013dc10 0x7f59400b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f59484fb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f594015bf30 0x7f59400b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f59484fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594015e850 0x7f594015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f59484fb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400cf3f0 con 0x7f594013dc10 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f59484fb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594015b210 con 0x7f594015bf30 2024-03-06T07:27:07.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f59484fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594015f580 con 0x7f594015e850 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f594013dc10 0x7f59400b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f594013dc10 0x7f59400b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34176/0 (socket says 172.21.15.72:34176) 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 -- 172.21.15.72:0/1107609659 learned_addr learned my addr 172.21.15.72:0/1107609659 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5945a96640 1 --2- 172.21.15.72:0/1107609659 >> v2:172.21.15.146:3300/0 conn(0x7f594015bf30 0x7f59400b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946a98640 1 --2- 172.21.15.72:0/1107609659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594015e850 0x7f594015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 -- 172.21.15.72:0/1107609659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594015e850 msgr2=0x7f594015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 --2- 172.21.15.72:0/1107609659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594015e850 0x7f594015eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 -- 172.21.15.72:0/1107609659 >> v2:172.21.15.146:3300/0 conn(0x7f594015bf30 msgr2=0x7f59400b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 --2- 172.21.15.72:0/1107609659 >> v2:172.21.15.146:3300/0 conn(0x7f594015bf30 0x7f59400b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.323+0000 7f5946297640 1 -- 172.21.15.72:0/1107609659 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594015b450 con 0x7f594013dc10 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.324+0000 7f5946297640 1 --2- 172.21.15.72:0/1107609659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594013dc10 0x7f59400b2950 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f594006c790 tx=0x7f593c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989a511c941f6fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.324+0000 7f5945295640 1 -- 172.21.15.72:0/1107609659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c02e070 con 0x7f594013dc10 2024-03-06T07:27:07.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.324+0000 7f5945295640 1 -- 172.21.15.72:0/1107609659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c037070 con 0x7f594013dc10 2024-03-06T07:27:07.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.324+0000 7f5945295640 1 -- 172.21.15.72:0/1107609659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c031a00 con 0x7f594013dc10 2024-03-06T07:27:07.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 -- 172.21.15.72:0/1107609659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594013dc10 msgr2=0x7f59400b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 --2- 172.21.15.72:0/1107609659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594013dc10 0x7f59400b2950 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f594006c790 tx=0x7f593c02b880 comp rx=0 tx=0).stop 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 -- 172.21.15.72:0/1107609659 shutdown_connections 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 --2- 172.21.15.72:0/1107609659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594015e850 0x7f594015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 --2- 172.21.15.72:0/1107609659 >> v2:172.21.15.146:3300/0 conn(0x7f594015bf30 0x7f59400b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 --2- 172.21.15.72:0/1107609659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594013dc10 0x7f59400b2950 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 -- 172.21.15.72:0/1107609659 >> 172.21.15.72:0/1107609659 conn(0x7f59400cf8a0 msgr2=0x7f59400cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 -- 172.21.15.72:0/1107609659 shutdown_connections 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.325+0000 7f59484fb640 1 -- 172.21.15.72:0/1107609659 wait complete. 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.326+0000 7f59484fb640 1 Processor -- start 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.326+0000 7f59484fb640 1 -- start start 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594013dc10 0x7f5940140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594015bf30 0x7f59400b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 0x7f59401f4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594015f1e0 con 0x7f594015e850 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594013dff0 con 0x7f594015bf30 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594015f360 con 0x7f594013dc10 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 0x7f59401f4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 0x7f59401f4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34188/0 (socket says 172.21.15.72:34188) 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 -- 172.21.15.72:0/491824478 learned_addr learned my addr 172.21.15.72:0/491824478 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5945a96640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594015bf30 0x7f59400b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946297640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594013dc10 0x7f5940140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 -- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594013dc10 msgr2=0x7f5940140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594013dc10 0x7f5940140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 -- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594015bf30 msgr2=0x7f59400b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594015bf30 0x7f59400b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 -- 172.21.15.72:0/491824478 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59401f57c0 con 0x7f594015e850 2024-03-06T07:27:07.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5946a98640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 0x7f59401f4780 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f593800efc0 tx=0x7f593800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938018020 con 0x7f594015e850 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5938014480 con 0x7f594015e850 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59401f5aa0 con 0x7f594015e850 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938014870 con 0x7f594015e850 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.327+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59401f6060 con 0x7f594015e850 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.328+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f594004f330 con 0x7f594015e850 2024-03-06T07:27:07.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.329+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5938005d10 con 0x7f594015e850 2024-03-06T07:27:07.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.329+0000 7f5936ffd640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592406b4d0 0x7f592406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:07.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.329+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f593808abe0 con 0x7f594015e850 2024-03-06T07:27:07.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.329+0000 7f5946297640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592406b4d0 0x7f592406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:07.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.330+0000 7f5946297640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592406b4d0 0x7f592406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f593c036000 tx=0x7f593c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:07.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.333+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5938022070 con 0x7f594015e850 2024-03-06T07:27:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.502+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f594006eb00 con 0x7f592406b4d0 2024-03-06T07:27:07.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.507+0000 7f5936ffd640 1 -- 172.21.15.72:0/491824478 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240245 (secure 0 0 0) 0x7f594006eb00 con 0x7f592406b4d0 2024-03-06T07:27:07.512 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:07.513 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592406b4d0 msgr2=0x7f592406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592406b4d0 0x7f592406d990 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f593c036000 tx=0x7f593c03d040 comp rx=0 tx=0).stop 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 msgr2=0x7f59401f4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 0x7f59401f4780 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f593800efc0 tx=0x7f593800c490 comp rx=0 tx=0).stop 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 shutdown_connections 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f592406b4d0 0x7f592406d990 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f594015e850 0x7f59401f4780 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f594015bf30 0x7f59400b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 --2- 172.21.15.72:0/491824478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f594013dc10 0x7f5940140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:07.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 >> 172.21.15.72:0/491824478 conn(0x7f59400cf8a0 msgr2=0x7f594013be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:07.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 shutdown_connections 2024-03-06T07:27:07.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:07.510+0000 7f59484fb640 1 -- 172.21.15.72:0/491824478 wait complete. 2024-03-06T07:27:07.523 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-03-06T07:27:06.005395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323416,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754651304,"statfs":{"total":773094113280,"available":772762935296,"internally_reserved":0,"allocated":11837440,"data_stored":5749768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":319213105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.846724"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160432+0000","last_change":"2024-03-06T07:27:01.160432+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160432+0000","last_undegraded":"2024-03-06T07:27:01.160432+0000","last_fullsized":"2024-03-06T07:27:01.160432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162404+0000","last_change":"2024-03-06T07:27:01.162404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162404+0000","last_undegraded":"2024-03-06T07:27:01.162404+0000","last_fullsized":"2024-03-06T07:27:01.162404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164196+0000","last_change":"2024-03-06T07:27:01.164196+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164196+0000","last_undegraded":"2024-03-06T07:27:01.164196+0000","last_fullsized":"2024-03-06T07:27:01.164196+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163484+0000","last_change":"2024-03-06T07:27:01.163484+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163484+0000","last_undegraded":"2024-03-06T07:27:01.163484+0000","last_fullsized":"2024-03-06T07:27:01.163484+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161185+0000","last_change":"2024-03-06T07:27:01.161185+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161185+0000","last_undegraded":"2024-03-06T07:27:01.161185+0000","last_fullsized":"2024-03-06T07:27:01.161185+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163456+0000","last_change":"2024-03-06T07:27:01.163456+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163456+0000","last_undegraded":"2024-03-06T07:27:01.163456+0000","last_fullsized":"2024-03-06T07:27:01.163456+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161376+0000","last_change":"2024-03-06T07:27:01.161376+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161376+0000","last_undegraded":"2024-03-06T07:27:01.161376+0000","last_fullsized":"2024-03-06T07:27:01.161376+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163990+0000","last_change":"2024-03-06T07:27:01.163990+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163990+0000","last_undegraded":"2024-03-06T07:27:01.163990+0000","last_fullsized":"2024-03-06T07:27:01.163990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163845+0000","last_change":"2024-03-06T07:27:01.163845+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163845+0000","last_undegraded":"2024-03-06T07:27:01.163845+0000","last_fullsized":"2024-03-06T07:27:01.163845+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163280+0000","last_change":"2024-03-06T07:27:01.163280+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163280+0000","last_undegraded":"2024-03-06T07:27:01.163280+0000","last_fullsized":"2024-03-06T07:27:01.163280+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162738+0000","last_change":"2024-03-06T07:27:01.162738+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162738+0000","last_undegraded":"2024-03-06T07:27:01.162738+0000","last_fullsized":"2024-03-06T07:27:01.162738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163027+0000","last_change":"2024-03-06T07:27:01.163027+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163027+0000","last_undegraded":"2024-03-06T07:27:01.163027+0000","last_fullsized":"2024-03-06T07:27:01.163027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161781+0000","last_change":"2024-03-06T07:27:01.161781+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161781+0000","last_undegraded":"2024-03-06T07:27:01.161781+0000","last_fullsized":"2024-03-06T07:27:01.161781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165818+0000","last_change":"2024-03-06T07:27:01.165818+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165818+0000","last_undegraded":"2024-03-06T07:27:01.165818+0000","last_fullsized":"2024-03-06T07:27:01.165818+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164299+0000","last_change":"2024-03-06T07:27:01.164299+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164299+0000","last_undegraded":"2024-03-06T07:27:01.164299+0000","last_fullsized":"2024-03-06T07:27:01.164299+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162317+0000","last_change":"2024-03-06T07:27:01.162317+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162317+0000","last_undegraded":"2024-03-06T07:27:01.162317+0000","last_fullsized":"2024-03-06T07:27:01.162317+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.164672+0000","last_change":"2024-03-06T07:27:01.164672+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.164672+0000","last_undegraded":"2024-03-06T07:27:01.164672+0000","last_fullsized":"2024-03-06T07:27:01.164672+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165839+0000","last_change":"2024-03-06T07:27:01.165839+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165839+0000","last_undegraded":"2024-03-06T07:27:01.165839+0000","last_fullsized":"2024-03-06T07:27:01.165839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160694+0000","last_change":"2024-03-06T07:27:01.160694+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160694+0000","last_undegraded":"2024-03-06T07:27:01.160694+0000","last_fullsized":"2024-03-06T07:27:01.160694+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161404+0000","last_change":"2024-03-06T07:27:01.161404+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161404+0000","last_undegraded":"2024-03-06T07:27:01.161404+0000","last_fullsized":"2024-03-06T07:27:01.161404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163113+0000","last_change":"2024-03-06T07:27:01.163113+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163113+0000","last_undegraded":"2024-03-06T07:27:01.163113+0000","last_fullsized":"2024-03-06T07:27:01.163113+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.161201+0000","last_change":"2024-03-06T07:27:01.161201+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.161201+0000","last_undegraded":"2024-03-06T07:27:01.161201+0000","last_fullsized":"2024-03-06T07:27:01.161201+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163646+0000","last_change":"2024-03-06T07:27:01.163646+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163646+0000","last_undegraded":"2024-03-06T07:27:01.163646+0000","last_fullsized":"2024-03-06T07:27:01.163646+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163830+0000","last_change":"2024-03-06T07:27:01.163830+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163830+0000","last_undegraded":"2024-03-06T07:27:01.163830+0000","last_fullsized":"2024-03-06T07:27:01.163830+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.165803+0000","last_change":"2024-03-06T07:27:01.165803+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.165803+0000","last_undegraded":"2024-03-06T07:27:01.165803+0000","last_fullsized":"2024-03-06T07:27:01.165803+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.162339+0000","last_change":"2024-03-06T07:27:01.162339+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.162339+0000","last_undegraded":"2024-03-06T07:27:01.162339+0000","last_fullsized":"2024-03-06T07:27:01.162339+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163702+0000","last_change":"2024-03-06T07:27:01.163702+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163702+0000","last_undegraded":"2024-03-06T07:27:01.163702+0000","last_fullsized":"2024-03-06T07:27:01.163702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160425+0000","last_change":"2024-03-06T07:27:01.160425+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160425+0000","last_undegraded":"2024-03-06T07:27:01.160425+0000","last_fullsized":"2024-03-06T07:27:01.160425+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160424+0000","last_change":"2024-03-06T07:27:01.160424+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160424+0000","last_undegraded":"2024-03-06T07:27:01.160424+0000","last_fullsized":"2024-03-06T07:27:01.160424+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163123+0000","last_change":"2024-03-06T07:27:01.163123+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163123+0000","last_undegraded":"2024-03-06T07:27:01.163123+0000","last_fullsized":"2024-03-06T07:27:01.163123+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.163302+0000","last_change":"2024-03-06T07:27:01.163302+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.163302+0000","last_undegraded":"2024-03-06T07:27:01.163302+0000","last_fullsized":"2024-03-06T07:27:01.163302+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-06T07:27:02.158127+0000","last_change":"2024-03-06T07:27:02.158127+0000","last_active":"2024-03-06T07:27:02.158127+0000","last_peered":"2024-03-06T07:27:02.158127+0000","last_clean":"2024-03-06T07:27:02.158127+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:02.158127+0000","last_undegraded":"2024-03-06T07:27:02.158127+0000","last_fullsized":"2024-03-06T07:27:02.158127+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-03-06T07:27:01.160423+0000","last_change":"2024-03-06T07:27:01.160423+0000","last_active":"2024-03-06T07:27:01.149204+0000","last_peered":"2024-03-06T07:27:01.149204+0000","last_clean":"2024-03-06T07:27:01.149204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:27:01.160423+0000","last_undegraded":"2024-03-06T07:27:01.160423+0000","last_fullsized":"2024-03-06T07:27:01.160423+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":327,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-06T07:27:01.162219+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:27:01.162219+0000","last_peered":"2024-03-06T07:27:01.162219+0000","last_clean":"2024-03-06T07:27:01.162219+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:27:01.162219+0000","last_undegraded":"2024-03-06T07:27:01.162219+0000","last_fullsized":"2024-03-06T07:27:01.162219+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1331200,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":1925120,"data_stored":1161497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:08.436 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-03-06T07:27:08.435+0000 7f2261140640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:10.528 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:27:10.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.638+0000 7fb123555640 1 Processor -- start 2024-03-06T07:27:10.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.638+0000 7fb123555640 1 -- start start 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb123555640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb11c066d90 0x7fb11c0ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb123555640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0cd450 con 0x7fb11c0b4f70 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb123555640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0cd730 con 0x7fb11c06c6d0 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb123555640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c15e9f0 con 0x7fb11c066d90 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb121af2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb11c066d90 0x7fb11c0ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb1212f1640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb11c0b4f70 0x7fb11c0b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59842/0 (socket says 172.21.15.72:59842) 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.639+0000 7fb1212f1640 1 -- 172.21.15.72:0/2741221178 learned_addr learned my addr 172.21.15.72:0/2741221178 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.640+0000 7fb121af2640 1 -- 172.21.15.72:0/2741221178 >> v1:172.21.15.146:6789/0 conn(0x7fb11c06c6d0 legacy=0x7fb11c15ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.640+0000 7fb121af2640 1 -- 172.21.15.72:0/2741221178 >> v1:172.21.15.72:6789/0 conn(0x7fb11c0b4f70 legacy=0x7fb11c0b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.640+0000 7fb113fff640 1 -- 172.21.15.72:0/2741221178 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794969751 0 0) 0x7fb11c0cd450 con 0x7fb11c0b4f70 2024-03-06T07:27:10.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.640+0000 7fb121af2640 1 -- 172.21.15.72:0/2741221178 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c15e680 con 0x7fb11c066d90 2024-03-06T07:27:10.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.640+0000 7fb121af2640 1 --2- 172.21.15.72:0/2741221178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c0ccb50 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fb114009d90 tx=0x7fb11402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e98a0255212d28f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb113fff640 1 -- 172.21.15.72:0/2741221178 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114039070 con 0x7fb11c066d90 2024-03-06T07:27:10.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb113fff640 1 -- 172.21.15.72:0/2741221178 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11402cd00 con 0x7fb11c066d90 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb113fff640 1 -- 172.21.15.72:0/2741221178 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114034660 con 0x7fb11c066d90 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 -- 172.21.15.72:0/2741221178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 msgr2=0x7fb11c0ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 --2- 172.21.15.72:0/2741221178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c0ccb50 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fb114009d90 tx=0x7fb11402bf50 comp rx=0 tx=0).stop 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 -- 172.21.15.72:0/2741221178 shutdown_connections 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 --2- 172.21.15.72:0/2741221178 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c0ccb50 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 -- 172.21.15.72:0/2741221178 >> 172.21.15.72:0/2741221178 conn(0x7fb11c0724d0 msgr2=0x7fb11c0728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 -- 172.21.15.72:0/2741221178 shutdown_connections 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.641+0000 7fb123555640 1 -- 172.21.15.72:0/2741221178 wait complete. 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.642+0000 7fb123555640 1 Processor -- start 2024-03-06T07:27:10.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.642+0000 7fb123555640 1 -- start start 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb123555640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c15a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb123555640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0b4f70 0x7fb11c14b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb123555640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c14b680 0x7fb11c1f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb123555640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c161ad0 con 0x7fb11c066d90 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb123555640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0cda10 con 0x7fb11c0b4f70 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb123555640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c161c50 con 0x7fb11c14b680 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c15a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c15a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34202/0 (socket says 172.21.15.72:34202) 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 -- 172.21.15.72:0/1601132314 learned_addr learned my addr 172.21.15.72:0/1601132314 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb120af0640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0b4f70 0x7fb11c14b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb121af2640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c14b680 0x7fb11c1f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 -- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c14b680 msgr2=0x7fb11c1f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c14b680 0x7fb11c1f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 -- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0b4f70 msgr2=0x7fb11c14b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0b4f70 0x7fb11c14b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.643+0000 7fb1212f1640 1 -- 172.21.15.72:0/1601132314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c1f82f0 con 0x7fb11c066d90 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb120af0640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0b4f70 0x7fb11c14b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb1212f1640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c15a400 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fb11800c3f0 tx=0x7fb11800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1180074e0 con 0x7fb11c066d90 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1180047e0 con 0x7fb11c066d90 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c0cde90 con 0x7fb11c066d90 2024-03-06T07:27:10.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118004c10 con 0x7fb11c066d90 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.644+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c0ce450 con 0x7fb11c066d90 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.645+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c04f330 con 0x7fb11c066d90 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.645+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb118011070 con 0x7fb11c066d90 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.646+0000 7fb111ffb640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10006b4d0 0x7fb10006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.646+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb118089e00 con 0x7fb11c066d90 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.646+0000 7fb120af0640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10006b4d0 0x7fb10006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.646+0000 7fb120af0640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10006b4d0 0x7fb10006d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb10c007fd0 tx=0x7fb10c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.650+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb11801a2b0 con 0x7fb11c066d90 2024-03-06T07:27:10.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.819+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb11c15bc00 con 0x7fb10006b4d0 2024-03-06T07:27:10.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.824+0000 7fb111ffb640 1 -- 172.21.15.72:0/1601132314 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7fb11c15bc00 con 0x7fb10006b4d0 2024-03-06T07:27:10.829 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:10.830 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10006b4d0 msgr2=0x7fb10006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10006b4d0 0x7fb10006d990 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb10c007fd0 tx=0x7fb10c009040 comp rx=0 tx=0).stop 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 msgr2=0x7fb11c15a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c15a400 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fb11800c3f0 tx=0x7fb11800c8b0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 shutdown_connections 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10006b4d0 0x7fb10006d990 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c14b680 0x7fb11c1f7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0b4f70 0x7fb11c14b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 --2- 172.21.15.72:0/1601132314 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb11c066d90 0x7fb11c15a400 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 >> 172.21.15.72:0/1601132314 conn(0x7fb11c0724d0 msgr2=0x7fb11c1611a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 shutdown_connections 2024-03-06T07:27:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.828+0000 7fb123555640 1 -- 172.21.15.72:0/1601132314 wait complete. 2024-03-06T07:27:10.841 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-03-06T07:27:10.006398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":323788,"kb_used_data":11912,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754650932,"statfs":{"total":773094113280,"available":772762554368,"internally_reserved":0,"allocated":12197888,"data_stored":5909388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":319213105},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.842769"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168392+0000","last_change":"2024-03-06T07:27:03.175661+0000","last_active":"2024-03-06T07:27:04.168392+0000","last_peered":"2024-03-06T07:27:04.168392+0000","last_clean":"2024-03-06T07:27:04.168392+0000","last_became_active":"2024-03-06T07:27:03.175363+0000","last_became_peered":"2024-03-06T07:27:03.175363+0000","last_unstale":"2024-03-06T07:27:04.168392+0000","last_undegraded":"2024-03-06T07:27:04.168392+0000","last_fullsized":"2024-03-06T07:27:04.168392+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:33:41.476918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700434+0000","last_change":"2024-03-06T07:27:03.183006+0000","last_active":"2024-03-06T07:27:04.700434+0000","last_peered":"2024-03-06T07:27:04.700434+0000","last_clean":"2024-03-06T07:27:04.700434+0000","last_became_active":"2024-03-06T07:27:03.182668+0000","last_became_peered":"2024-03-06T07:27:03.182668+0000","last_unstale":"2024-03-06T07:27:04.700434+0000","last_undegraded":"2024-03-06T07:27:04.700434+0000","last_fullsized":"2024-03-06T07:27:04.700434+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.700672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170183+0000","last_change":"2024-03-06T07:27:03.182465+0000","last_active":"2024-03-06T07:27:04.170183+0000","last_peered":"2024-03-06T07:27:04.170183+0000","last_clean":"2024-03-06T07:27:04.170183+0000","last_became_active":"2024-03-06T07:27:03.182156+0000","last_became_peered":"2024-03-06T07:27:03.182156+0000","last_unstale":"2024-03-06T07:27:04.170183+0000","last_undegraded":"2024-03-06T07:27:04.170183+0000","last_fullsized":"2024-03-06T07:27:04.170183+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:15.184189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168336+0000","last_change":"2024-03-06T07:27:03.180875+0000","last_active":"2024-03-06T07:27:04.168336+0000","last_peered":"2024-03-06T07:27:04.168336+0000","last_clean":"2024-03-06T07:27:04.168336+0000","last_became_active":"2024-03-06T07:27:03.180620+0000","last_became_peered":"2024-03-06T07:27:03.180620+0000","last_unstale":"2024-03-06T07:27:04.168336+0000","last_undegraded":"2024-03-06T07:27:04.168336+0000","last_fullsized":"2024-03-06T07:27:04.168336+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:30:32.420867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919988+0000","last_change":"2024-03-06T07:27:03.190995+0000","last_active":"2024-03-06T07:27:04.919988+0000","last_peered":"2024-03-06T07:27:04.919988+0000","last_clean":"2024-03-06T07:27:04.919988+0000","last_became_active":"2024-03-06T07:27:03.190724+0000","last_became_peered":"2024-03-06T07:27:03.190724+0000","last_unstale":"2024-03-06T07:27:04.919988+0000","last_undegraded":"2024-03-06T07:27:04.919988+0000","last_fullsized":"2024-03-06T07:27:04.919988+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:19:51.449734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419649+0000","last_change":"2024-03-06T07:27:03.180875+0000","last_active":"2024-03-06T07:27:04.419649+0000","last_peered":"2024-03-06T07:27:04.419649+0000","last_clean":"2024-03-06T07:27:04.419649+0000","last_became_active":"2024-03-06T07:27:03.180568+0000","last_became_peered":"2024-03-06T07:27:03.180568+0000","last_unstale":"2024-03-06T07:27:04.419649+0000","last_undegraded":"2024-03-06T07:27:04.419649+0000","last_fullsized":"2024-03-06T07:27:04.419649+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:36:34.867175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169654+0000","last_change":"2024-03-06T07:27:03.183483+0000","last_active":"2024-03-06T07:27:04.169654+0000","last_peered":"2024-03-06T07:27:04.169654+0000","last_clean":"2024-03-06T07:27:04.169654+0000","last_became_active":"2024-03-06T07:27:03.183174+0000","last_became_peered":"2024-03-06T07:27:03.183174+0000","last_unstale":"2024-03-06T07:27:04.169654+0000","last_undegraded":"2024-03-06T07:27:04.169654+0000","last_fullsized":"2024-03-06T07:27:04.169654+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:57:29.916006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170640+0000","last_change":"2024-03-06T07:27:03.191256+0000","last_active":"2024-03-06T07:27:04.170640+0000","last_peered":"2024-03-06T07:27:04.170640+0000","last_clean":"2024-03-06T07:27:04.170640+0000","last_became_active":"2024-03-06T07:27:03.190390+0000","last_became_peered":"2024-03-06T07:27:03.190390+0000","last_unstale":"2024-03-06T07:27:04.170640+0000","last_undegraded":"2024-03-06T07:27:04.170640+0000","last_fullsized":"2024-03-06T07:27:04.170640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:20.502655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168522+0000","last_change":"2024-03-06T07:27:03.183198+0000","last_active":"2024-03-06T07:27:04.168522+0000","last_peered":"2024-03-06T07:27:04.168522+0000","last_clean":"2024-03-06T07:27:04.168522+0000","last_became_active":"2024-03-06T07:27:03.182922+0000","last_became_peered":"2024-03-06T07:27:03.182922+0000","last_unstale":"2024-03-06T07:27:04.168522+0000","last_undegraded":"2024-03-06T07:27:04.168522+0000","last_fullsized":"2024-03-06T07:27:04.168522+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:18.631865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168384+0000","last_change":"2024-03-06T07:27:03.187764+0000","last_active":"2024-03-06T07:27:04.168384+0000","last_peered":"2024-03-06T07:27:04.168384+0000","last_clean":"2024-03-06T07:27:04.168384+0000","last_became_active":"2024-03-06T07:27:03.187413+0000","last_became_peered":"2024-03-06T07:27:03.187413+0000","last_unstale":"2024-03-06T07:27:04.168384+0000","last_undegraded":"2024-03-06T07:27:04.168384+0000","last_fullsized":"2024-03-06T07:27:04.168384+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:20:04.309181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700455+0000","last_change":"2024-03-06T07:27:03.182685+0000","last_active":"2024-03-06T07:27:04.700455+0000","last_peered":"2024-03-06T07:27:04.700455+0000","last_clean":"2024-03-06T07:27:04.700455+0000","last_became_active":"2024-03-06T07:27:03.182401+0000","last_became_peered":"2024-03-06T07:27:03.182401+0000","last_unstale":"2024-03-06T07:27:04.700455+0000","last_undegraded":"2024-03-06T07:27:04.700455+0000","last_fullsized":"2024-03-06T07:27:04.700455+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:51.408341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168500+0000","last_change":"2024-03-06T07:27:03.179582+0000","last_active":"2024-03-06T07:27:04.168500+0000","last_peered":"2024-03-06T07:27:04.168500+0000","last_clean":"2024-03-06T07:27:04.168500+0000","last_became_active":"2024-03-06T07:27:03.179162+0000","last_became_peered":"2024-03-06T07:27:03.179162+0000","last_unstale":"2024-03-06T07:27:04.168500+0000","last_undegraded":"2024-03-06T07:27:04.168500+0000","last_fullsized":"2024-03-06T07:27:04.168500+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:57:10.066159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170890+0000","last_change":"2024-03-06T07:27:03.187967+0000","last_active":"2024-03-06T07:27:04.170890+0000","last_peered":"2024-03-06T07:27:04.170890+0000","last_clean":"2024-03-06T07:27:04.170890+0000","last_became_active":"2024-03-06T07:27:03.187447+0000","last_became_peered":"2024-03-06T07:27:03.187447+0000","last_unstale":"2024-03-06T07:27:04.170890+0000","last_undegraded":"2024-03-06T07:27:04.170890+0000","last_fullsized":"2024-03-06T07:27:04.170890+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:53.211683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170191+0000","last_change":"2024-03-06T07:27:03.183297+0000","last_active":"2024-03-06T07:27:04.170191+0000","last_peered":"2024-03-06T07:27:04.170191+0000","last_clean":"2024-03-06T07:27:04.170191+0000","last_became_active":"2024-03-06T07:27:03.183044+0000","last_became_peered":"2024-03-06T07:27:03.183044+0000","last_unstale":"2024-03-06T07:27:04.170191+0000","last_undegraded":"2024-03-06T07:27:04.170191+0000","last_fullsized":"2024-03-06T07:27:04.170191+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:27.665314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419604+0000","last_change":"2024-03-06T07:27:03.188531+0000","last_active":"2024-03-06T07:27:04.419604+0000","last_peered":"2024-03-06T07:27:04.419604+0000","last_clean":"2024-03-06T07:27:04.419604+0000","last_became_active":"2024-03-06T07:27:03.188278+0000","last_became_peered":"2024-03-06T07:27:03.188278+0000","last_unstale":"2024-03-06T07:27:04.419604+0000","last_undegraded":"2024-03-06T07:27:04.419604+0000","last_fullsized":"2024-03-06T07:27:04.419604+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:42:52.677157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168049+0000","last_change":"2024-03-06T07:27:03.191208+0000","last_active":"2024-03-06T07:27:04.168049+0000","last_peered":"2024-03-06T07:27:04.168049+0000","last_clean":"2024-03-06T07:27:04.168049+0000","last_became_active":"2024-03-06T07:27:03.190931+0000","last_became_peered":"2024-03-06T07:27:03.190931+0000","last_unstale":"2024-03-06T07:27:04.168049+0000","last_undegraded":"2024-03-06T07:27:04.168049+0000","last_fullsized":"2024-03-06T07:27:04.168049+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:03:06.567363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700534+0000","last_change":"2024-03-06T07:27:03.184932+0000","last_active":"2024-03-06T07:27:04.700534+0000","last_peered":"2024-03-06T07:27:04.700534+0000","last_clean":"2024-03-06T07:27:04.700534+0000","last_became_active":"2024-03-06T07:27:03.182627+0000","last_became_peered":"2024-03-06T07:27:03.182627+0000","last_unstale":"2024-03-06T07:27:04.700534+0000","last_undegraded":"2024-03-06T07:27:04.700534+0000","last_fullsized":"2024-03-06T07:27:04.700534+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:51.511001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919978+0000","last_change":"2024-03-06T07:27:03.184424+0000","last_active":"2024-03-06T07:27:04.919978+0000","last_peered":"2024-03-06T07:27:04.919978+0000","last_clean":"2024-03-06T07:27:04.919978+0000","last_became_active":"2024-03-06T07:27:03.183941+0000","last_became_peered":"2024-03-06T07:27:03.183941+0000","last_unstale":"2024-03-06T07:27:04.919978+0000","last_undegraded":"2024-03-06T07:27:04.919978+0000","last_fullsized":"2024-03-06T07:27:04.919978+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:27.333733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170865+0000","last_change":"2024-03-06T07:27:03.189601+0000","last_active":"2024-03-06T07:27:04.170865+0000","last_peered":"2024-03-06T07:27:04.170865+0000","last_clean":"2024-03-06T07:27:04.170865+0000","last_became_active":"2024-03-06T07:27:03.187331+0000","last_became_peered":"2024-03-06T07:27:03.187331+0000","last_unstale":"2024-03-06T07:27:04.170865+0000","last_undegraded":"2024-03-06T07:27:04.170865+0000","last_fullsized":"2024-03-06T07:27:04.170865+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:09:41.927371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700575+0000","last_change":"2024-03-06T07:27:03.190033+0000","last_active":"2024-03-06T07:27:04.700575+0000","last_peered":"2024-03-06T07:27:04.700575+0000","last_clean":"2024-03-06T07:27:04.700575+0000","last_became_active":"2024-03-06T07:27:03.189743+0000","last_became_peered":"2024-03-06T07:27:03.189743+0000","last_unstale":"2024-03-06T07:27:04.700575+0000","last_undegraded":"2024-03-06T07:27:04.700575+0000","last_fullsized":"2024-03-06T07:27:04.700575+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:09:41.927371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700460+0000","last_change":"2024-03-06T07:27:03.191345+0000","last_active":"2024-03-06T07:27:04.700460+0000","last_peered":"2024-03-06T07:27:04.700460+0000","last_clean":"2024-03-06T07:27:04.700460+0000","last_became_active":"2024-03-06T07:27:03.190693+0000","last_became_peered":"2024-03-06T07:27:03.190693+0000","last_unstale":"2024-03-06T07:27:04.700460+0000","last_undegraded":"2024-03-06T07:27:04.700460+0000","last_fullsized":"2024-03-06T07:27:04.700460+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:20.502655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920179+0000","last_change":"2024-03-06T07:27:03.175622+0000","last_active":"2024-03-06T07:27:04.920179+0000","last_peered":"2024-03-06T07:27:04.920179+0000","last_clean":"2024-03-06T07:27:04.920179+0000","last_became_active":"2024-03-06T07:27:03.175347+0000","last_became_peered":"2024-03-06T07:27:03.175347+0000","last_unstale":"2024-03-06T07:27:04.920179+0000","last_undegraded":"2024-03-06T07:27:04.920179+0000","last_fullsized":"2024-03-06T07:27:04.920179+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:27.665314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700185+0000","last_change":"2024-03-06T07:27:03.189271+0000","last_active":"2024-03-06T07:27:04.700185+0000","last_peered":"2024-03-06T07:27:04.700185+0000","last_clean":"2024-03-06T07:27:04.700185+0000","last_became_active":"2024-03-06T07:27:03.188971+0000","last_became_peered":"2024-03-06T07:27:03.188971+0000","last_unstale":"2024-03-06T07:27:04.700185+0000","last_undegraded":"2024-03-06T07:27:04.700185+0000","last_fullsized":"2024-03-06T07:27:04.700185+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.784403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919953+0000","last_change":"2024-03-06T07:27:03.193081+0000","last_active":"2024-03-06T07:27:04.919953+0000","last_peered":"2024-03-06T07:27:04.919953+0000","last_clean":"2024-03-06T07:27:04.919953+0000","last_became_active":"2024-03-06T07:27:03.192562+0000","last_became_peered":"2024-03-06T07:27:03.192562+0000","last_unstale":"2024-03-06T07:27:04.919953+0000","last_undegraded":"2024-03-06T07:27:04.919953+0000","last_fullsized":"2024-03-06T07:27:04.919953+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:48.018038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700593+0000","last_change":"2024-03-06T07:27:03.184426+0000","last_active":"2024-03-06T07:27:04.700593+0000","last_peered":"2024-03-06T07:27:04.700593+0000","last_clean":"2024-03-06T07:27:04.700593+0000","last_became_active":"2024-03-06T07:27:03.182202+0000","last_became_peered":"2024-03-06T07:27:03.182202+0000","last_unstale":"2024-03-06T07:27:04.700593+0000","last_undegraded":"2024-03-06T07:27:04.700593+0000","last_fullsized":"2024-03-06T07:27:04.700593+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:53.211683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168390+0000","last_change":"2024-03-06T07:27:03.179172+0000","last_active":"2024-03-06T07:27:04.168390+0000","last_peered":"2024-03-06T07:27:04.168390+0000","last_clean":"2024-03-06T07:27:04.168390+0000","last_became_active":"2024-03-06T07:27:03.178914+0000","last_became_peered":"2024-03-06T07:27:03.178914+0000","last_unstale":"2024-03-06T07:27:04.168390+0000","last_undegraded":"2024-03-06T07:27:04.168390+0000","last_fullsized":"2024-03-06T07:27:04.168390+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:37:13.283244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419863+0000","last_change":"2024-03-06T07:27:03.187558+0000","last_active":"2024-03-06T07:27:04.419863+0000","last_peered":"2024-03-06T07:27:04.419863+0000","last_clean":"2024-03-06T07:27:04.419863+0000","last_became_active":"2024-03-06T07:27:03.187284+0000","last_became_peered":"2024-03-06T07:27:03.187284+0000","last_unstale":"2024-03-06T07:27:04.419863+0000","last_undegraded":"2024-03-06T07:27:04.419863+0000","last_fullsized":"2024-03-06T07:27:04.419863+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:14:02.971547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920292+0000","last_change":"2024-03-06T07:27:03.180165+0000","last_active":"2024-03-06T07:27:04.920292+0000","last_peered":"2024-03-06T07:27:04.920292+0000","last_clean":"2024-03-06T07:27:04.920292+0000","last_became_active":"2024-03-06T07:27:03.179863+0000","last_became_peered":"2024-03-06T07:27:03.179863+0000","last_unstale":"2024-03-06T07:27:04.920292+0000","last_undegraded":"2024-03-06T07:27:04.920292+0000","last_fullsized":"2024-03-06T07:27:04.920292+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:51.408341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700234+0000","last_change":"2024-03-06T07:27:03.191594+0000","last_active":"2024-03-06T07:27:04.700234+0000","last_peered":"2024-03-06T07:27:04.700234+0000","last_clean":"2024-03-06T07:27:04.700234+0000","last_became_active":"2024-03-06T07:27:03.191343+0000","last_became_peered":"2024-03-06T07:27:03.191343+0000","last_unstale":"2024-03-06T07:27:04.700234+0000","last_undegraded":"2024-03-06T07:27:04.700234+0000","last_fullsized":"2024-03-06T07:27:04.700234+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:40:48.018038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169721+0000","last_change":"2024-03-06T07:27:03.185238+0000","last_active":"2024-03-06T07:27:04.169721+0000","last_peered":"2024-03-06T07:27:04.169721+0000","last_clean":"2024-03-06T07:27:04.169721+0000","last_became_active":"2024-03-06T07:27:03.184930+0000","last_became_peered":"2024-03-06T07:27:03.184930+0000","last_unstale":"2024-03-06T07:27:04.169721+0000","last_undegraded":"2024-03-06T07:27:04.169721+0000","last_fullsized":"2024-03-06T07:27:04.169721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:02.230894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168021+0000","last_change":"2024-03-06T07:27:03.185092+0000","last_active":"2024-03-06T07:27:04.168021+0000","last_peered":"2024-03-06T07:27:04.168021+0000","last_clean":"2024-03-06T07:27:04.168021+0000","last_became_active":"2024-03-06T07:27:03.184808+0000","last_became_peered":"2024-03-06T07:27:03.184808+0000","last_unstale":"2024-03-06T07:27:04.168021+0000","last_undegraded":"2024-03-06T07:27:04.168021+0000","last_fullsized":"2024-03-06T07:27:04.168021+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:22:16.486474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419589+0000","last_change":"2024-03-06T07:27:03.188420+0000","last_active":"2024-03-06T07:27:04.419589+0000","last_peered":"2024-03-06T07:27:04.419589+0000","last_clean":"2024-03-06T07:27:04.419589+0000","last_became_active":"2024-03-06T07:27:03.188163+0000","last_became_peered":"2024-03-06T07:27:03.188163+0000","last_unstale":"2024-03-06T07:27:04.419589+0000","last_undegraded":"2024-03-06T07:27:04.419589+0000","last_fullsized":"2024-03-06T07:27:04.419589+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:08:29.968476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419827+0000","last_change":"2024-03-06T07:27:03.186717+0000","last_active":"2024-03-06T07:27:04.419827+0000","last_peered":"2024-03-06T07:27:04.419827+0000","last_clean":"2024-03-06T07:27:04.419827+0000","last_became_active":"2024-03-06T07:27:03.186419+0000","last_became_peered":"2024-03-06T07:27:03.186419+0000","last_unstale":"2024-03-06T07:27:04.419827+0000","last_undegraded":"2024-03-06T07:27:04.419827+0000","last_fullsized":"2024-03-06T07:27:04.419827+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:29:08.473163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169820+0000","last_change":"2024-03-06T07:27:02.177359+0000","last_active":"2024-03-06T07:27:04.169820+0000","last_peered":"2024-03-06T07:27:04.169820+0000","last_clean":"2024-03-06T07:27:04.169820+0000","last_became_active":"2024-03-06T07:27:02.177058+0000","last_became_peered":"2024-03-06T07:27:02.177058+0000","last_unstale":"2024-03-06T07:27:04.169820+0000","last_undegraded":"2024-03-06T07:27:04.169820+0000","last_fullsized":"2024-03-06T07:27:04.169820+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:40.956451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920234+0000","last_change":"2024-03-06T07:27:03.185626+0000","last_active":"2024-03-06T07:27:04.920234+0000","last_peered":"2024-03-06T07:27:04.920234+0000","last_clean":"2024-03-06T07:27:04.920234+0000","last_became_active":"2024-03-06T07:27:03.184986+0000","last_became_peered":"2024-03-06T07:27:03.184986+0000","last_unstale":"2024-03-06T07:27:04.920234+0000","last_undegraded":"2024-03-06T07:27:04.920234+0000","last_fullsized":"2024-03-06T07:27:04.920234+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.185077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170946+0000","last_change":"2024-03-06T07:27:02.177385+0000","last_active":"2024-03-06T07:27:04.170946+0000","last_peered":"2024-03-06T07:27:04.170946+0000","last_clean":"2024-03-06T07:27:04.170946+0000","last_became_active":"2024-03-06T07:27:02.177078+0000","last_became_peered":"2024-03-06T07:27:02.177078+0000","last_unstale":"2024-03-06T07:27:04.170946+0000","last_undegraded":"2024-03-06T07:27:04.170946+0000","last_fullsized":"2024-03-06T07:27:04.170946+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:59.083237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419820+0000","last_change":"2024-03-06T07:27:03.181982+0000","last_active":"2024-03-06T07:27:04.419820+0000","last_peered":"2024-03-06T07:27:04.419820+0000","last_clean":"2024-03-06T07:27:04.419820+0000","last_became_active":"2024-03-06T07:27:03.181657+0000","last_became_peered":"2024-03-06T07:27:03.181657+0000","last_unstale":"2024-03-06T07:27:04.419820+0000","last_undegraded":"2024-03-06T07:27:04.419820+0000","last_fullsized":"2024-03-06T07:27:04.419820+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:18.016806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919975+0000","last_change":"2024-03-06T07:27:02.181591+0000","last_active":"2024-03-06T07:27:04.919975+0000","last_peered":"2024-03-06T07:27:04.919975+0000","last_clean":"2024-03-06T07:27:04.919975+0000","last_became_active":"2024-03-06T07:27:02.181328+0000","last_became_peered":"2024-03-06T07:27:02.181328+0000","last_unstale":"2024-03-06T07:27:04.919975+0000","last_undegraded":"2024-03-06T07:27:04.919975+0000","last_fullsized":"2024-03-06T07:27:04.919975+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:47:41.020891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.171172+0000","last_change":"2024-03-06T07:27:03.188937+0000","last_active":"2024-03-06T07:27:04.171172+0000","last_peered":"2024-03-06T07:27:04.171172+0000","last_clean":"2024-03-06T07:27:04.171172+0000","last_became_active":"2024-03-06T07:27:03.188404+0000","last_became_peered":"2024-03-06T07:27:03.188404+0000","last_unstale":"2024-03-06T07:27:04.171172+0000","last_undegraded":"2024-03-06T07:27:04.171172+0000","last_fullsized":"2024-03-06T07:27:04.171172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:11:48.242407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919721+0000","last_change":"2024-03-06T07:27:02.178616+0000","last_active":"2024-03-06T07:27:04.919721+0000","last_peered":"2024-03-06T07:27:04.919721+0000","last_clean":"2024-03-06T07:27:04.919721+0000","last_became_active":"2024-03-06T07:27:02.178349+0000","last_became_peered":"2024-03-06T07:27:02.178349+0000","last_unstale":"2024-03-06T07:27:04.919721+0000","last_undegraded":"2024-03-06T07:27:04.919721+0000","last_fullsized":"2024-03-06T07:27:04.919721+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:43:17.805958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168094+0000","last_change":"2024-03-06T07:27:03.183165+0000","last_active":"2024-03-06T07:27:04.168094+0000","last_peered":"2024-03-06T07:27:04.168094+0000","last_clean":"2024-03-06T07:27:04.168094+0000","last_became_active":"2024-03-06T07:27:03.182887+0000","last_became_peered":"2024-03-06T07:27:03.182887+0000","last_unstale":"2024-03-06T07:27:04.168094+0000","last_undegraded":"2024-03-06T07:27:04.168094+0000","last_fullsized":"2024-03-06T07:27:04.168094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:38:21.698026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169657+0000","last_change":"2024-03-06T07:27:02.172039+0000","last_active":"2024-03-06T07:27:04.169657+0000","last_peered":"2024-03-06T07:27:04.169657+0000","last_clean":"2024-03-06T07:27:04.169657+0000","last_became_active":"2024-03-06T07:27:02.171738+0000","last_became_peered":"2024-03-06T07:27:02.171738+0000","last_unstale":"2024-03-06T07:27:04.169657+0000","last_undegraded":"2024-03-06T07:27:04.169657+0000","last_fullsized":"2024-03-06T07:27:04.169657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:45:24.120974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700284+0000","last_change":"2024-03-06T07:27:03.189678+0000","last_active":"2024-03-06T07:27:04.700284+0000","last_peered":"2024-03-06T07:27:04.700284+0000","last_clean":"2024-03-06T07:27:04.700284+0000","last_became_active":"2024-03-06T07:27:03.189408+0000","last_became_peered":"2024-03-06T07:27:03.189408+0000","last_unstale":"2024-03-06T07:27:04.700284+0000","last_undegraded":"2024-03-06T07:27:04.700284+0000","last_fullsized":"2024-03-06T07:27:04.700284+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:15.184189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920395+0000","last_change":"2024-03-06T07:27:02.174386+0000","last_active":"2024-03-06T07:27:04.920395+0000","last_peered":"2024-03-06T07:27:04.920395+0000","last_clean":"2024-03-06T07:27:04.920395+0000","last_became_active":"2024-03-06T07:27:02.174130+0000","last_became_peered":"2024-03-06T07:27:02.174130+0000","last_unstale":"2024-03-06T07:27:04.920395+0000","last_undegraded":"2024-03-06T07:27:04.920395+0000","last_fullsized":"2024-03-06T07:27:04.920395+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:56:58.025274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.167968+0000","last_change":"2024-03-06T07:27:03.179361+0000","last_active":"2024-03-06T07:27:04.167968+0000","last_peered":"2024-03-06T07:27:04.167968+0000","last_clean":"2024-03-06T07:27:04.167968+0000","last_became_active":"2024-03-06T07:27:03.179077+0000","last_became_peered":"2024-03-06T07:27:03.179077+0000","last_unstale":"2024-03-06T07:27:04.167968+0000","last_undegraded":"2024-03-06T07:27:04.167968+0000","last_fullsized":"2024-03-06T07:27:04.167968+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:21.388691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700111+0000","last_change":"2024-03-06T07:27:02.186412+0000","last_active":"2024-03-06T07:27:04.700111+0000","last_peered":"2024-03-06T07:27:04.700111+0000","last_clean":"2024-03-06T07:27:04.700111+0000","last_became_active":"2024-03-06T07:27:02.186155+0000","last_became_peered":"2024-03-06T07:27:02.186155+0000","last_unstale":"2024-03-06T07:27:04.700111+0000","last_undegraded":"2024-03-06T07:27:04.700111+0000","last_fullsized":"2024-03-06T07:27:04.700111+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:03:40.956451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169572+0000","last_change":"2024-03-06T07:27:03.183550+0000","last_active":"2024-03-06T07:27:04.169572+0000","last_peered":"2024-03-06T07:27:04.169572+0000","last_clean":"2024-03-06T07:27:04.169572+0000","last_became_active":"2024-03-06T07:27:03.183283+0000","last_became_peered":"2024-03-06T07:27:03.183283+0000","last_unstale":"2024-03-06T07:27:04.169572+0000","last_undegraded":"2024-03-06T07:27:04.169572+0000","last_fullsized":"2024-03-06T07:27:04.169572+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:05:41.730717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168027+0000","last_change":"2024-03-06T07:27:02.178071+0000","last_active":"2024-03-06T07:27:04.168027+0000","last_peered":"2024-03-06T07:27:04.168027+0000","last_clean":"2024-03-06T07:27:04.168027+0000","last_became_active":"2024-03-06T07:27:02.177199+0000","last_became_peered":"2024-03-06T07:27:02.177199+0000","last_unstale":"2024-03-06T07:27:04.168027+0000","last_undegraded":"2024-03-06T07:27:04.168027+0000","last_fullsized":"2024-03-06T07:27:04.168027+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:05:06.497845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170190+0000","last_change":"2024-03-06T07:27:03.179731+0000","last_active":"2024-03-06T07:27:04.170190+0000","last_peered":"2024-03-06T07:27:04.170190+0000","last_clean":"2024-03-06T07:27:04.170190+0000","last_became_active":"2024-03-06T07:27:03.179468+0000","last_became_peered":"2024-03-06T07:27:03.179468+0000","last_unstale":"2024-03-06T07:27:04.170190+0000","last_undegraded":"2024-03-06T07:27:04.170190+0000","last_fullsized":"2024-03-06T07:27:04.170190+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:30.193587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919764+0000","last_change":"2024-03-06T07:27:02.178725+0000","last_active":"2024-03-06T07:27:04.919764+0000","last_peered":"2024-03-06T07:27:04.919764+0000","last_clean":"2024-03-06T07:27:04.919764+0000","last_became_active":"2024-03-06T07:27:02.178465+0000","last_became_peered":"2024-03-06T07:27:02.178465+0000","last_unstale":"2024-03-06T07:27:04.919764+0000","last_undegraded":"2024-03-06T07:27:04.919764+0000","last_fullsized":"2024-03-06T07:27:04.919764+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:33:49.197554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920281+0000","last_change":"2024-03-06T07:27:03.180861+0000","last_active":"2024-03-06T07:27:04.920281+0000","last_peered":"2024-03-06T07:27:04.920281+0000","last_clean":"2024-03-06T07:27:04.920281+0000","last_became_active":"2024-03-06T07:27:03.180527+0000","last_became_peered":"2024-03-06T07:27:03.180527+0000","last_unstale":"2024-03-06T07:27:04.920281+0000","last_undegraded":"2024-03-06T07:27:04.920281+0000","last_fullsized":"2024-03-06T07:27:04.920281+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:10:45.784403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920140+0000","last_change":"2024-03-06T07:27:02.181515+0000","last_active":"2024-03-06T07:27:04.920140+0000","last_peered":"2024-03-06T07:27:04.920140+0000","last_clean":"2024-03-06T07:27:04.920140+0000","last_became_active":"2024-03-06T07:27:02.181268+0000","last_became_peered":"2024-03-06T07:27:02.181268+0000","last_unstale":"2024-03-06T07:27:04.920140+0000","last_undegraded":"2024-03-06T07:27:04.920140+0000","last_fullsized":"2024-03-06T07:27:04.920140+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:29.181474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920347+0000","last_change":"2024-03-06T07:27:03.181413+0000","last_active":"2024-03-06T07:27:04.920347+0000","last_peered":"2024-03-06T07:27:04.920347+0000","last_clean":"2024-03-06T07:27:04.920347+0000","last_became_active":"2024-03-06T07:27:03.180238+0000","last_became_peered":"2024-03-06T07:27:03.180238+0000","last_unstale":"2024-03-06T07:27:04.920347+0000","last_undegraded":"2024-03-06T07:27:04.920347+0000","last_fullsized":"2024-03-06T07:27:04.920347+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:09:41.927371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"109'3","reported_seq":17,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168204+0000","last_change":"2024-03-06T07:27:02.183552+0000","last_active":"2024-03-06T07:27:04.168204+0000","last_peered":"2024-03-06T07:27:04.168204+0000","last_clean":"2024-03-06T07:27:04.168204+0000","last_became_active":"2024-03-06T07:27:02.183294+0000","last_became_peered":"2024-03-06T07:27:02.183294+0000","last_unstale":"2024-03-06T07:27:04.168204+0000","last_undegraded":"2024-03-06T07:27:04.168204+0000","last_fullsized":"2024-03-06T07:27:04.168204+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:44:51.277531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169945+0000","last_change":"2024-03-06T07:27:03.193100+0000","last_active":"2024-03-06T07:27:04.169945+0000","last_peered":"2024-03-06T07:27:04.169945+0000","last_clean":"2024-03-06T07:27:04.169945+0000","last_became_active":"2024-03-06T07:27:03.192576+0000","last_became_peered":"2024-03-06T07:27:03.192576+0000","last_unstale":"2024-03-06T07:27:04.169945+0000","last_undegraded":"2024-03-06T07:27:04.169945+0000","last_fullsized":"2024-03-06T07:27:04.169945+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:31:27.333733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"109'6","reported_seq":20,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168555+0000","last_change":"2024-03-06T07:27:02.183182+0000","last_active":"2024-03-06T07:27:04.168555+0000","last_peered":"2024-03-06T07:27:04.168555+0000","last_clean":"2024-03-06T07:27:04.168555+0000","last_became_active":"2024-03-06T07:27:02.182919+0000","last_became_peered":"2024-03-06T07:27:02.182919+0000","last_unstale":"2024-03-06T07:27:04.168555+0000","last_undegraded":"2024-03-06T07:27:04.168555+0000","last_fullsized":"2024-03-06T07:27:04.168555+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:59:13.482600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920340+0000","last_change":"2024-03-06T07:27:03.185774+0000","last_active":"2024-03-06T07:27:04.920340+0000","last_peered":"2024-03-06T07:27:04.920340+0000","last_clean":"2024-03-06T07:27:04.920340+0000","last_became_active":"2024-03-06T07:27:03.185348+0000","last_became_peered":"2024-03-06T07:27:03.185348+0000","last_unstale":"2024-03-06T07:27:04.920340+0000","last_undegraded":"2024-03-06T07:27:04.920340+0000","last_fullsized":"2024-03-06T07:27:04.920340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:51.511001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419727+0000","last_change":"2024-03-06T07:27:02.181469+0000","last_active":"2024-03-06T07:27:04.419727+0000","last_peered":"2024-03-06T07:27:04.419727+0000","last_clean":"2024-03-06T07:27:04.419727+0000","last_became_active":"2024-03-06T07:27:02.180763+0000","last_became_peered":"2024-03-06T07:27:02.180763+0000","last_unstale":"2024-03-06T07:27:04.419727+0000","last_undegraded":"2024-03-06T07:27:04.419727+0000","last_fullsized":"2024-03-06T07:27:04.419727+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:03:39.522083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920495+0000","last_change":"2024-03-06T07:27:03.197139+0000","last_active":"2024-03-06T07:27:04.920495+0000","last_peered":"2024-03-06T07:27:04.920495+0000","last_clean":"2024-03-06T07:27:04.920495+0000","last_became_active":"2024-03-06T07:27:03.196841+0000","last_became_peered":"2024-03-06T07:27:03.196841+0000","last_unstale":"2024-03-06T07:27:04.920495+0000","last_undegraded":"2024-03-06T07:27:04.920495+0000","last_fullsized":"2024-03-06T07:27:04.920495+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:26:53.211683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919879+0000","last_change":"2024-03-06T07:27:02.178815+0000","last_active":"2024-03-06T07:27:04.919879+0000","last_peered":"2024-03-06T07:27:04.919879+0000","last_clean":"2024-03-06T07:27:04.919879+0000","last_became_active":"2024-03-06T07:27:02.178537+0000","last_became_peered":"2024-03-06T07:27:02.178537+0000","last_unstale":"2024-03-06T07:27:04.919879+0000","last_undegraded":"2024-03-06T07:27:04.919879+0000","last_fullsized":"2024-03-06T07:27:04.919879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:45.928910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920188+0000","last_change":"2024-03-06T07:27:03.188456+0000","last_active":"2024-03-06T07:27:04.920188+0000","last_peered":"2024-03-06T07:27:04.920188+0000","last_clean":"2024-03-06T07:27:04.920188+0000","last_became_active":"2024-03-06T07:27:03.187882+0000","last_became_peered":"2024-03-06T07:27:03.187882+0000","last_unstale":"2024-03-06T07:27:04.920188+0000","last_undegraded":"2024-03-06T07:27:04.920188+0000","last_fullsized":"2024-03-06T07:27:04.920188+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.700672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":22,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919556+0000","last_change":"2024-03-06T07:27:02.179811+0000","last_active":"2024-03-06T07:27:04.919556+0000","last_peered":"2024-03-06T07:27:04.919556+0000","last_clean":"2024-03-06T07:27:04.919556+0000","last_became_active":"2024-03-06T07:27:02.179556+0000","last_became_peered":"2024-03-06T07:27:02.179556+0000","last_unstale":"2024-03-06T07:27:04.919556+0000","last_undegraded":"2024-03-06T07:27:04.919556+0000","last_fullsized":"2024-03-06T07:27:04.919556+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:31.664231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919933+0000","last_change":"2024-03-06T07:27:03.194557+0000","last_active":"2024-03-06T07:27:04.919933+0000","last_peered":"2024-03-06T07:27:04.919933+0000","last_clean":"2024-03-06T07:27:04.919933+0000","last_became_active":"2024-03-06T07:27:03.194171+0000","last_became_peered":"2024-03-06T07:27:03.194171+0000","last_unstale":"2024-03-06T07:27:04.919933+0000","last_undegraded":"2024-03-06T07:27:04.919933+0000","last_fullsized":"2024-03-06T07:27:04.919933+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:50.383150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168571+0000","last_change":"2024-03-06T07:27:02.179499+0000","last_active":"2024-03-06T07:27:04.168571+0000","last_peered":"2024-03-06T07:27:04.168571+0000","last_clean":"2024-03-06T07:27:04.168571+0000","last_became_active":"2024-03-06T07:27:02.179215+0000","last_became_peered":"2024-03-06T07:27:02.179215+0000","last_unstale":"2024-03-06T07:27:04.168571+0000","last_undegraded":"2024-03-06T07:27:04.168571+0000","last_fullsized":"2024-03-06T07:27:04.168571+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:21:18.044751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419893+0000","last_change":"2024-03-06T07:27:03.184452+0000","last_active":"2024-03-06T07:27:04.419893+0000","last_peered":"2024-03-06T07:27:04.419893+0000","last_clean":"2024-03-06T07:27:04.419893+0000","last_became_active":"2024-03-06T07:27:03.184229+0000","last_became_peered":"2024-03-06T07:27:03.184229+0000","last_unstale":"2024-03-06T07:27:04.419893+0000","last_undegraded":"2024-03-06T07:27:04.419893+0000","last_fullsized":"2024-03-06T07:27:04.419893+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:09:48.531813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919809+0000","last_change":"2024-03-06T07:27:02.186570+0000","last_active":"2024-03-06T07:27:04.919809+0000","last_peered":"2024-03-06T07:27:04.919809+0000","last_clean":"2024-03-06T07:27:04.919809+0000","last_became_active":"2024-03-06T07:27:02.186324+0000","last_became_peered":"2024-03-06T07:27:02.186324+0000","last_unstale":"2024-03-06T07:27:04.919809+0000","last_undegraded":"2024-03-06T07:27:04.919809+0000","last_fullsized":"2024-03-06T07:27:04.919809+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:29.253576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170604+0000","last_change":"2024-03-06T07:27:03.189397+0000","last_active":"2024-03-06T07:27:04.170604+0000","last_peered":"2024-03-06T07:27:04.170604+0000","last_clean":"2024-03-06T07:27:04.170604+0000","last_became_active":"2024-03-06T07:27:03.188447+0000","last_became_peered":"2024-03-06T07:27:03.188447+0000","last_unstale":"2024-03-06T07:27:04.170604+0000","last_undegraded":"2024-03-06T07:27:04.170604+0000","last_fullsized":"2024-03-06T07:27:04.170604+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:53:51.408341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919925+0000","last_change":"2024-03-06T07:27:02.179329+0000","last_active":"2024-03-06T07:27:04.919925+0000","last_peered":"2024-03-06T07:27:04.919925+0000","last_clean":"2024-03-06T07:27:04.919925+0000","last_became_active":"2024-03-06T07:27:02.179062+0000","last_became_peered":"2024-03-06T07:27:02.179062+0000","last_unstale":"2024-03-06T07:27:04.919925+0000","last_undegraded":"2024-03-06T07:27:04.919925+0000","last_fullsized":"2024-03-06T07:27:04.919925+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:41:29.181474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.171172+0000","last_change":"2024-03-06T07:27:03.187039+0000","last_active":"2024-03-06T07:27:04.171172+0000","last_peered":"2024-03-06T07:27:04.171172+0000","last_clean":"2024-03-06T07:27:04.171172+0000","last_became_active":"2024-03-06T07:27:03.186771+0000","last_became_peered":"2024-03-06T07:27:03.186771+0000","last_unstale":"2024-03-06T07:27:04.171172+0000","last_undegraded":"2024-03-06T07:27:04.171172+0000","last_fullsized":"2024-03-06T07:27:04.171172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:13:54.025082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169559+0000","last_change":"2024-03-06T07:27:02.179399+0000","last_active":"2024-03-06T07:27:04.169559+0000","last_peered":"2024-03-06T07:27:04.169559+0000","last_clean":"2024-03-06T07:27:04.169559+0000","last_became_active":"2024-03-06T07:27:02.179126+0000","last_became_peered":"2024-03-06T07:27:02.179126+0000","last_unstale":"2024-03-06T07:27:04.169559+0000","last_undegraded":"2024-03-06T07:27:04.169559+0000","last_fullsized":"2024-03-06T07:27:04.169559+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:14.912627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170351+0000","last_change":"2024-03-06T07:27:03.186307+0000","last_active":"2024-03-06T07:27:04.170351+0000","last_peered":"2024-03-06T07:27:04.170351+0000","last_clean":"2024-03-06T07:27:04.170351+0000","last_became_active":"2024-03-06T07:27:03.182171+0000","last_became_peered":"2024-03-06T07:27:03.182171+0000","last_unstale":"2024-03-06T07:27:04.170351+0000","last_undegraded":"2024-03-06T07:27:04.170351+0000","last_fullsized":"2024-03-06T07:27:04.170351+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:23.700672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419793+0000","last_change":"2024-03-06T07:27:02.171852+0000","last_active":"2024-03-06T07:27:04.419793+0000","last_peered":"2024-03-06T07:27:04.419793+0000","last_clean":"2024-03-06T07:27:04.419793+0000","last_became_active":"2024-03-06T07:27:02.171594+0000","last_became_peered":"2024-03-06T07:27:02.171594+0000","last_unstale":"2024-03-06T07:27:04.419793+0000","last_undegraded":"2024-03-06T07:27:04.419793+0000","last_fullsized":"2024-03-06T07:27:04.419793+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:52:46.752239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168094+0000","last_change":"2024-03-06T07:27:03.187553+0000","last_active":"2024-03-06T07:27:04.168094+0000","last_peered":"2024-03-06T07:27:04.168094+0000","last_clean":"2024-03-06T07:27:04.168094+0000","last_became_active":"2024-03-06T07:27:03.187286+0000","last_became_peered":"2024-03-06T07:27:03.187286+0000","last_unstale":"2024-03-06T07:27:04.168094+0000","last_undegraded":"2024-03-06T07:27:04.168094+0000","last_fullsized":"2024-03-06T07:27:04.168094+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:33:12.442227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919646+0000","last_change":"2024-03-06T07:27:02.182365+0000","last_active":"2024-03-06T07:27:04.919646+0000","last_peered":"2024-03-06T07:27:04.919646+0000","last_clean":"2024-03-06T07:27:04.919646+0000","last_became_active":"2024-03-06T07:27:02.181434+0000","last_became_peered":"2024-03-06T07:27:02.181434+0000","last_unstale":"2024-03-06T07:27:04.919646+0000","last_undegraded":"2024-03-06T07:27:04.919646+0000","last_fullsized":"2024-03-06T07:27:04.919646+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:30:14.225827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.171035+0000","last_change":"2024-03-06T07:27:03.194515+0000","last_active":"2024-03-06T07:27:04.171035+0000","last_peered":"2024-03-06T07:27:04.171035+0000","last_clean":"2024-03-06T07:27:04.171035+0000","last_became_active":"2024-03-06T07:27:03.194242+0000","last_became_peered":"2024-03-06T07:27:03.194242+0000","last_unstale":"2024-03-06T07:27:04.171035+0000","last_undegraded":"2024-03-06T07:27:04.171035+0000","last_fullsized":"2024-03-06T07:27:04.171035+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:43:07.369680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.419524+0000","last_change":"2024-03-06T07:27:02.172834+0000","last_active":"2024-03-06T07:27:04.419524+0000","last_peered":"2024-03-06T07:27:04.419524+0000","last_clean":"2024-03-06T07:27:04.419524+0000","last_became_active":"2024-03-06T07:27:02.172590+0000","last_became_peered":"2024-03-06T07:27:02.172590+0000","last_unstale":"2024-03-06T07:27:04.419524+0000","last_undegraded":"2024-03-06T07:27:04.419524+0000","last_fullsized":"2024-03-06T07:27:04.419524+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:46:24.022931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920307+0000","last_change":"2024-03-06T07:27:03.190469+0000","last_active":"2024-03-06T07:27:04.920307+0000","last_peered":"2024-03-06T07:27:04.920307+0000","last_clean":"2024-03-06T07:27:04.920307+0000","last_became_active":"2024-03-06T07:27:03.190192+0000","last_became_peered":"2024-03-06T07:27:03.190192+0000","last_unstale":"2024-03-06T07:27:04.920307+0000","last_undegraded":"2024-03-06T07:27:04.920307+0000","last_fullsized":"2024-03-06T07:27:04.920307+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:17:20.502655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168102+0000","last_change":"2024-03-06T07:27:02.178292+0000","last_active":"2024-03-06T07:27:04.168102+0000","last_peered":"2024-03-06T07:27:04.168102+0000","last_clean":"2024-03-06T07:27:04.168102+0000","last_became_active":"2024-03-06T07:27:02.177823+0000","last_became_peered":"2024-03-06T07:27:02.177823+0000","last_unstale":"2024-03-06T07:27:04.168102+0000","last_undegraded":"2024-03-06T07:27:04.168102+0000","last_fullsized":"2024-03-06T07:27:04.168102+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:06:50.012502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919937+0000","last_change":"2024-03-06T07:27:03.189828+0000","last_active":"2024-03-06T07:27:04.919937+0000","last_peered":"2024-03-06T07:27:04.919937+0000","last_clean":"2024-03-06T07:27:04.919937+0000","last_became_active":"2024-03-06T07:27:03.189551+0000","last_became_peered":"2024-03-06T07:27:03.189551+0000","last_unstale":"2024-03-06T07:27:04.919937+0000","last_undegraded":"2024-03-06T07:27:04.919937+0000","last_fullsized":"2024-03-06T07:27:04.919937+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:27:02.230894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920135+0000","last_change":"2024-03-06T07:27:02.179659+0000","last_active":"2024-03-06T07:27:04.920135+0000","last_peered":"2024-03-06T07:27:04.920135+0000","last_clean":"2024-03-06T07:27:04.920135+0000","last_became_active":"2024-03-06T07:27:02.179407+0000","last_became_peered":"2024-03-06T07:27:02.179407+0000","last_unstale":"2024-03-06T07:27:04.920135+0000","last_undegraded":"2024-03-06T07:27:04.920135+0000","last_fullsized":"2024-03-06T07:27:04.920135+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:06:45.928910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168536+0000","last_change":"2024-03-06T07:27:03.181165+0000","last_active":"2024-03-06T07:27:04.168536+0000","last_peered":"2024-03-06T07:27:04.168536+0000","last_clean":"2024-03-06T07:27:04.168536+0000","last_became_active":"2024-03-06T07:27:03.180897+0000","last_became_peered":"2024-03-06T07:27:03.180897+0000","last_unstale":"2024-03-06T07:27:04.168536+0000","last_undegraded":"2024-03-06T07:27:04.168536+0000","last_fullsized":"2024-03-06T07:27:04.168536+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:13:30.340196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"110'1","reported_seq":15,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920071+0000","last_change":"2024-03-06T07:27:02.178008+0000","last_active":"2024-03-06T07:27:04.920071+0000","last_peered":"2024-03-06T07:27:04.920071+0000","last_clean":"2024-03-06T07:27:04.920071+0000","last_became_active":"2024-03-06T07:27:02.177729+0000","last_became_peered":"2024-03-06T07:27:02.177729+0000","last_unstale":"2024-03-06T07:27:04.920071+0000","last_undegraded":"2024-03-06T07:27:04.920071+0000","last_fullsized":"2024-03-06T07:27:04.920071+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:59.083237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168646+0000","last_change":"2024-03-06T07:27:03.177824+0000","last_active":"2024-03-06T07:27:04.168646+0000","last_peered":"2024-03-06T07:27:04.168646+0000","last_clean":"2024-03-06T07:27:04.168646+0000","last_became_active":"2024-03-06T07:27:03.177407+0000","last_became_peered":"2024-03-06T07:27:03.177407+0000","last_unstale":"2024-03-06T07:27:04.168646+0000","last_undegraded":"2024-03-06T07:27:04.168646+0000","last_fullsized":"2024-03-06T07:27:04.168646+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:35:24.465522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168240+0000","last_change":"2024-03-06T07:27:02.184617+0000","last_active":"2024-03-06T07:27:04.168240+0000","last_peered":"2024-03-06T07:27:04.168240+0000","last_clean":"2024-03-06T07:27:04.168240+0000","last_became_active":"2024-03-06T07:27:02.184346+0000","last_became_peered":"2024-03-06T07:27:02.184346+0000","last_unstale":"2024-03-06T07:27:04.168240+0000","last_undegraded":"2024-03-06T07:27:04.168240+0000","last_fullsized":"2024-03-06T07:27:04.168240+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:04:06.906181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919995+0000","last_change":"2024-03-06T07:27:03.180582+0000","last_active":"2024-03-06T07:27:04.919995+0000","last_peered":"2024-03-06T07:27:04.919995+0000","last_clean":"2024-03-06T07:27:04.919995+0000","last_became_active":"2024-03-06T07:27:03.180296+0000","last_became_peered":"2024-03-06T07:27:03.180296+0000","last_unstale":"2024-03-06T07:27:04.919995+0000","last_undegraded":"2024-03-06T07:27:04.919995+0000","last_fullsized":"2024-03-06T07:27:04.919995+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:30.193587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.920352+0000","last_change":"2024-03-06T07:27:02.180981+0000","last_active":"2024-03-06T07:27:04.920352+0000","last_peered":"2024-03-06T07:27:04.920352+0000","last_clean":"2024-03-06T07:27:04.920352+0000","last_became_active":"2024-03-06T07:27:02.180736+0000","last_became_peered":"2024-03-06T07:27:02.180736+0000","last_unstale":"2024-03-06T07:27:04.920352+0000","last_undegraded":"2024-03-06T07:27:04.920352+0000","last_fullsized":"2024-03-06T07:27:04.920352+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:20:59.083237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700173+0000","last_change":"2024-03-06T07:27:03.188968+0000","last_active":"2024-03-06T07:27:04.700173+0000","last_peered":"2024-03-06T07:27:04.700173+0000","last_clean":"2024-03-06T07:27:04.700173+0000","last_became_active":"2024-03-06T07:27:03.188583+0000","last_became_peered":"2024-03-06T07:27:03.188583+0000","last_unstale":"2024-03-06T07:27:04.700173+0000","last_undegraded":"2024-03-06T07:27:04.700173+0000","last_fullsized":"2024-03-06T07:27:04.700173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:46:30.185077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169784+0000","last_change":"2024-03-06T07:27:02.183933+0000","last_active":"2024-03-06T07:27:04.169784+0000","last_peered":"2024-03-06T07:27:04.169784+0000","last_clean":"2024-03-06T07:27:04.169784+0000","last_became_active":"2024-03-06T07:27:02.183628+0000","last_became_peered":"2024-03-06T07:27:02.183628+0000","last_unstale":"2024-03-06T07:27:04.169784+0000","last_undegraded":"2024-03-06T07:27:04.169784+0000","last_fullsized":"2024-03-06T07:27:04.169784+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:17:15.039247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700425+0000","last_change":"2024-03-06T07:27:03.187024+0000","last_active":"2024-03-06T07:27:04.700425+0000","last_peered":"2024-03-06T07:27:04.700425+0000","last_clean":"2024-03-06T07:27:04.700425+0000","last_became_active":"2024-03-06T07:27:03.186729+0000","last_became_peered":"2024-03-06T07:27:03.186729+0000","last_unstale":"2024-03-06T07:27:04.700425+0000","last_undegraded":"2024-03-06T07:27:04.700425+0000","last_fullsized":"2024-03-06T07:27:04.700425+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:28:27.665314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.700144+0000","last_change":"2024-03-06T07:27:02.184302+0000","last_active":"2024-03-06T07:27:04.700144+0000","last_peered":"2024-03-06T07:27:04.700144+0000","last_clean":"2024-03-06T07:27:04.700144+0000","last_became_active":"2024-03-06T07:27:02.184008+0000","last_became_peered":"2024-03-06T07:27:02.184008+0000","last_unstale":"2024-03-06T07:27:04.700144+0000","last_undegraded":"2024-03-06T07:27:04.700144+0000","last_fullsized":"2024-03-06T07:27:04.700144+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:06:14.912627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170825+0000","last_change":"2024-03-06T07:27:03.190648+0000","last_active":"2024-03-06T07:27:04.170825+0000","last_peered":"2024-03-06T07:27:04.170825+0000","last_clean":"2024-03-06T07:27:04.170825+0000","last_became_active":"2024-03-06T07:27:03.189934+0000","last_became_peered":"2024-03-06T07:27:03.189934+0000","last_unstale":"2024-03-06T07:27:04.170825+0000","last_undegraded":"2024-03-06T07:27:04.170825+0000","last_fullsized":"2024-03-06T07:27:04.170825+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:54:51.511001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919575+0000","last_change":"2024-03-06T07:27:02.181634+0000","last_active":"2024-03-06T07:27:04.919575+0000","last_peered":"2024-03-06T07:27:04.919575+0000","last_clean":"2024-03-06T07:27:04.919575+0000","last_became_active":"2024-03-06T07:27:02.181397+0000","last_became_peered":"2024-03-06T07:27:02.181397+0000","last_unstale":"2024-03-06T07:27:04.919575+0000","last_undegraded":"2024-03-06T07:27:04.919575+0000","last_fullsized":"2024-03-06T07:27:04.919575+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:13:16.945210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168025+0000","last_change":"2024-03-06T07:27:03.177665+0000","last_active":"2024-03-06T07:27:04.168025+0000","last_peered":"2024-03-06T07:27:04.168025+0000","last_clean":"2024-03-06T07:27:04.168025+0000","last_became_active":"2024-03-06T07:27:03.177294+0000","last_became_peered":"2024-03-06T07:27:03.177294+0000","last_unstale":"2024-03-06T07:27:04.168025+0000","last_undegraded":"2024-03-06T07:27:04.168025+0000","last_fullsized":"2024-03-06T07:27:04.168025+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:45:46.336607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.168205+0000","last_change":"2024-03-06T07:27:02.180103+0000","last_active":"2024-03-06T07:27:04.168205+0000","last_peered":"2024-03-06T07:27:04.168205+0000","last_clean":"2024-03-06T07:27:04.168205+0000","last_became_active":"2024-03-06T07:27:02.177732+0000","last_became_peered":"2024-03-06T07:27:02.177732+0000","last_unstale":"2024-03-06T07:27:04.168205+0000","last_undegraded":"2024-03-06T07:27:04.168205+0000","last_fullsized":"2024-03-06T07:27:04.168205+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:05:15.548587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.919994+0000","last_change":"2024-03-06T07:27:03.192837+0000","last_active":"2024-03-06T07:27:04.919994+0000","last_peered":"2024-03-06T07:27:04.919994+0000","last_clean":"2024-03-06T07:27:04.919994+0000","last_became_active":"2024-03-06T07:27:03.191968+0000","last_became_peered":"2024-03-06T07:27:03.191968+0000","last_unstale":"2024-03-06T07:27:04.919994+0000","last_undegraded":"2024-03-06T07:27:04.919994+0000","last_fullsized":"2024-03-06T07:27:04.919994+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:02.158127+0000","last_clean_scrub_stamp":"2024-03-06T07:27:02.158127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:21:15.184189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.169646+0000","last_change":"2024-03-06T07:27:02.179139+0000","last_active":"2024-03-06T07:27:04.169646+0000","last_peered":"2024-03-06T07:27:04.169646+0000","last_clean":"2024-03-06T07:27:04.169646+0000","last_became_active":"2024-03-06T07:27:02.178813+0000","last_became_peered":"2024-03-06T07:27:02.178813+0000","last_unstale":"2024-03-06T07:27:04.169646+0000","last_undegraded":"2024-03-06T07:27:04.169646+0000","last_fullsized":"2024-03-06T07:27:04.169646+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:27:01.149204+0000","last_clean_scrub_stamp":"2024-03-06T07:27:01.149204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:28:09.791610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":333,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-06T07:27:04.170970+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:27:04.170970+0000","last_peered":"2024-03-06T07:27:04.170970+0000","last_clean":"2024-03-06T07:27:04.170970+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:27:04.170970+0000","last_undegraded":"2024-03-06T07:27:04.170970+0000","last_fullsized":"2024-03-06T07:27:04.170970+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37268,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334572,"statfs":{"total":96636764160,"available":96598601728,"internally_reserved":0,"allocated":1970176,"data_stored":1181463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574917,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40844,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330996,"statfs":{"total":96636764160,"available":96594939904,"internally_reserved":0,"allocated":1372160,"data_stored":590549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40848,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330992,"statfs":{"total":96636764160,"available":96594935808,"internally_reserved":0,"allocated":1376256,"data_stored":591083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574917,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1368064,"data_stored":590479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40856,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330984,"statfs":{"total":96636764160,"available":96594927616,"internally_reserved":0,"allocated":1384448,"data_stored":592161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40840,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331000,"statfs":{"total":96636764160,"available":96594944000,"internally_reserved":0,"allocated":1368064,"data_stored":590479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40848,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330992,"statfs":{"total":96636764160,"available":96594935808,"internally_reserved":0,"allocated":1376256,"data_stored":590603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41444,"kb_used_data":1936,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330396,"statfs":{"total":96636764160,"available":96594325504,"internally_reserved":0,"allocated":1982464,"data_stored":1182571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:27:10.844 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:27:10.845 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:27:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.987+0000 7fe6538b5640 1 Processor -- start 2024-03-06T07:27:10.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.987+0000 7fe6538b5640 1 -- start start 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe6538b5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe64c1614d0 0x7fe64c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe6538b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c0d1910 0x7fe64c0d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe6538b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe64c0d22b0 0x7fe64c0d06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe6538b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64c0d2740 con 0x7fe64c0d22b0 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe6538b5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64c1618b0 con 0x7fe64c1614d0 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe6538b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0cf400 con 0x7fe64c0d1910 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe64c0d22b0 0x7fe64c0d06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe64c0d22b0 0x7fe64c0d06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34216/0 (socket says 172.21.15.72:34216) 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 -- 172.21.15.72:0/1703605572 learned_addr learned my addr 172.21.15.72:0/1703605572 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651651640 1 --2- 172.21.15.72:0/1703605572 >> v2:172.21.15.146:3300/0 conn(0x7fe64c1614d0 0x7fe64c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe650e50640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c0d1910 0x7fe64c0d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 -- 172.21.15.72:0/1703605572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c0d1910 msgr2=0x7fe64c0d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c0d1910 0x7fe64c0d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 -- 172.21.15.72:0/1703605572 >> v2:172.21.15.146:3300/0 conn(0x7fe64c1614d0 msgr2=0x7fe64c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:10.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 --2- 172.21.15.72:0/1703605572 >> v2:172.21.15.146:3300/0 conn(0x7fe64c1614d0 0x7fe64c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe651e52640 1 -- 172.21.15.72:0/1703605572 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c0cf6a0 con 0x7fe64c0d22b0 2024-03-06T07:27:10.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.988+0000 7fe650e50640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c0d1910 0x7fe64c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:10.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.989+0000 7fe651e52640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c0d06a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe64c08c470 tx=0x7fe64402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1ce35552b02f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.989+0000 7fe643fff640 1 -- 172.21.15.72:0/1703605572 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe64402ca70 con 0x7fe64c0d22b0 2024-03-06T07:27:10.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.989+0000 7fe643fff640 1 -- 172.21.15.72:0/1703605572 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe64402cc10 con 0x7fe64c0d22b0 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.989+0000 7fe643fff640 1 -- 172.21.15.72:0/1703605572 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6440343f0 con 0x7fe64c0d22b0 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 -- 172.21.15.72:0/1703605572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 msgr2=0x7fe64c0d06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c0d06a0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fe64c08c470 tx=0x7fe64402bed0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 -- 172.21.15.72:0/1703605572 shutdown_connections 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c0d06a0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1703605572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c0d1910 0x7fe64c0d1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1703605572 >> v2:172.21.15.146:3300/0 conn(0x7fe64c1614d0 0x7fe64c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 -- 172.21.15.72:0/1703605572 >> 172.21.15.72:0/1703605572 conn(0x7fe64c0d3d40 msgr2=0x7fe64c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 -- 172.21.15.72:0/1703605572 shutdown_connections 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.990+0000 7fe6538b5640 1 -- 172.21.15.72:0/1703605572 wait complete. 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.991+0000 7fe6538b5640 1 Processor -- start 2024-03-06T07:27:10.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.991+0000 7fe6538b5640 1 -- start start 2024-03-06T07:27:10.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe6538b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe64c0d1910 0x7fe64c1f0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe6538b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c1f4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe6538b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c1614d0 0x7fe64c14fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe6538b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0b0910 con 0x7fe64c0d22b0 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe6538b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0cfc60 con 0x7fe64c0d1910 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe6538b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0d0be0 con 0x7fe64c1614d0 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c1f4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c1f4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34230/0 (socket says 172.21.15.72:34230) 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 -- 172.21.15.72:0/1220330473 learned_addr learned my addr 172.21.15.72:0/1220330473 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe651651640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe64c0d1910 0x7fe64c1f0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe651e52640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c1614d0 0x7fe64c14fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 -- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c1614d0 msgr2=0x7fe64c14fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c1614d0 0x7fe64c14fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 -- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe64c0d1910 msgr2=0x7fe64c1f0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:10.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe64c0d1910 0x7fe64c1f0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 -- 172.21.15.72:0/1220330473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c1f3b50 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.992+0000 7fe650e50640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c1f4330 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00e760 tx=0x7fe63c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.993+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00f040 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.993+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c004230 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.993+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64c150950 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.993+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c00ee70 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.993+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c1f3e10 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.994+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64c1f3970 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.994+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe63c00c6d0 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.994+0000 7fe641ffb640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe63006b4d0 0x7fe63006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.995+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe63c089e30 con 0x7fe64c0d22b0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.995+0000 7fe651651640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe63006b4d0 0x7fe63006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:10.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.995+0000 7fe651651640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe63006b4d0 0x7fe63006d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe648009a20 tx=0x7fe648009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:10.999+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe64c1f3970 con 0x7fe64c0d22b0 2024-03-06T07:27:11.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.239+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe64c0a1c90 con 0x7fe64c0d22b0 2024-03-06T07:27:11.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.241+0000 7fe641ffb640 1 -- 172.21.15.72:0/1220330473 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe63c017020 con 0x7fe64c0d22b0 2024-03-06T07:27:11.243 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:27:11.243 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:27:11.243 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:27:11.243 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:27:11.243 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:27:11.243 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe63006b4d0 msgr2=0x7fe63006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe63006b4d0 0x7fe63006d990 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe648009a20 tx=0x7fe648009190 comp rx=0 tx=0).stop 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 msgr2=0x7fe64c1f4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c1f4330 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fe63c00e760 tx=0x7fe63c00ec20 comp rx=0 tx=0).stop 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 shutdown_connections 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe63006b4d0 0x7fe63006d990 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe64c1614d0 0x7fe64c14fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe64c0d22b0 0x7fe64c1f4330 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 --2- 172.21.15.72:0/1220330473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe64c0d1910 0x7fe64c1f0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 >> 172.21.15.72:0/1220330473 conn(0x7fe64c0d3d40 msgr2=0x7fe64c0b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 shutdown_connections 2024-03-06T07:27:11.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.243+0000 7fe6538b5640 1 -- 172.21.15.72:0/1220330473 wait complete. 2024-03-06T07:27:11.256 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.354+0000 7f4588881640 1 Processor -- start 2024-03-06T07:27:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.354+0000 7f4588881640 1 -- start start 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4588881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f458415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4588881640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4584071f40 con 0x7f45840cd7a0 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4588881640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45840cd2c0 con 0x7f45841515e0 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4588881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4584070a70 con 0x7f45840cce40 2024-03-06T07:27:11.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4582d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f45840cd7a0 0x7f45840cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59852/0 (socket says 172.21.15.72:59852) 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4582d9d640 1 -- 172.21.15.72:0/30425925 learned_addr learned my addr 172.21.15.72:0/30425925 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f457bfff640 1 --2- 172.21.15.72:0/30425925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f458415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094993173 0 0) 0x7f4584071f40 con 0x7f45840cd7a0 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4570004630 con 0x7f45840cd7a0 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218188412 0 0) 0x7f4574001110 con 0x7f45841515e0 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4584071f40 con 0x7f45841515e0 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f457bfff640 1 -- 172.21.15.72:0/30425925 >> v1:172.21.15.146:6789/0 conn(0x7f45841515e0 legacy=0x7f4584071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f457bfff640 1 -- 172.21.15.72:0/30425925 >> v1:172.21.15.72:6789/0 conn(0x7f45840cd7a0 legacy=0x7f45840cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.355+0000 7f457bfff640 1 -- 172.21.15.72:0/30425925 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584070d50 con 0x7f45840cce40 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.356+0000 7f457bfff640 1 --2- 172.21.15.72:0/30425925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f458415b430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f456c0097d0 tx=0x7f456c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a614c51ea2f6902 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.356+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c035020 con 0x7f45840cce40 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.356+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c031890 con 0x7f45840cce40 2024-03-06T07:27:11.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.356+0000 7f4581d9b640 1 -- 172.21.15.72:0/30425925 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c031c60 con 0x7f45840cce40 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 -- 172.21.15.72:0/30425925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 msgr2=0x7f458415b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 --2- 172.21.15.72:0/30425925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f458415b430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f456c0097d0 tx=0x7f456c02b930 comp rx=0 tx=0).stop 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 -- 172.21.15.72:0/30425925 shutdown_connections 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 --2- 172.21.15.72:0/30425925 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f458415b430 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 -- 172.21.15.72:0/30425925 >> 172.21.15.72:0/30425925 conn(0x7f45840cf8a0 msgr2=0x7f45840cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 -- 172.21.15.72:0/30425925 shutdown_connections 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.357+0000 7f4588881640 1 -- 172.21.15.72:0/30425925 wait complete. 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.358+0000 7f4588881640 1 Processor -- start 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.358+0000 7f4588881640 1 -- start start 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4588881640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f45840cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4588881640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 0x7f458415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4588881640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45841515e0 0x7f45840c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4588881640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45840cc3a0 con 0x7f45840cd7a0 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4588881640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4584071310 con 0x7f45841515e0 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4588881640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45840cc520 con 0x7f45840cce40 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 0x7f458415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 0x7f458415cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34232/0 (socket says 172.21.15.72:34232) 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 -- 172.21.15.72:0/1460906242 learned_addr learned my addr 172.21.15.72:0/1460906242 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f4582d9d640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45841515e0 0x7f45840c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f458259c640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f45840cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 -- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 msgr2=0x7f45840cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f45840cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 -- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45841515e0 msgr2=0x7f45840c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45841515e0 0x7f45840c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.359+0000 7f457bfff640 1 -- 172.21.15.72:0/1460906242 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458415d420 con 0x7f45840cd7a0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.360+0000 7f457bfff640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 0x7f458415cee0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f456c0098f0 tx=0x7f456c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.360+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c035040 con 0x7f45840cd7a0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.360+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f456c030d20 con 0x7f45840cd7a0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.360+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456c006e10 con 0x7f45840cd7a0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.360+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458415be40 con 0x7f45840cd7a0 2024-03-06T07:27:11.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.360+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458415c400 con 0x7f45840cd7a0 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.361+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4584068840 con 0x7f45840cd7a0 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.361+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f456c038020 con 0x7f45840cd7a0 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.361+0000 7f457affd640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f456406b4d0 0x7f456406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.362+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f456c0ac730 con 0x7f45840cd7a0 2024-03-06T07:27:11.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.362+0000 7f458259c640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f456406b4d0 0x7f456406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:11.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.362+0000 7f458259c640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f456406b4d0 0x7f456406d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f45740012b0 tx=0x7f4574009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:11.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.366+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f456c02d040 con 0x7f45840cd7a0 2024-03-06T07:27:11.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.572+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458404f330 con 0x7f45840cd7a0 2024-03-06T07:27:11.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.572+0000 7f457affd640 1 -- 172.21.15.72:0/1460906242 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f456c07af60 con 0x7f45840cd7a0 2024-03-06T07:27:11.575 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:11.575 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2347391904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2347391904},{"type":"v1","addr":"172.21.15.146:6834","nonce":2347391904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/1152083148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1152083148},{"type":"v1","addr":"172.21.15.72:6835","nonce":1152083148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3618393393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":3618393393},{"type":"v1","addr":"172.21.15.72:6837","nonce":3618393393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:04.517794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/118157918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":118157918},{"type":"v1","addr":"172.21.15.146:6835","nonce":118157918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7926,"qdb_cluster":[7926]},"id":8}]} 2024-03-06T07:27:11.575 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 63 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f456406b4d0 msgr2=0x7f456406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f456406b4d0 0x7f456406d990 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f45740012b0 tx=0x7f4574009040 comp rx=0 tx=0).stop 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 msgr2=0x7f458415cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 0x7f458415cee0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f456c0098f0 tx=0x7f456c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 shutdown_connections 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f456406b4d0 0x7f456406d990 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45841515e0 0x7f45840c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f45840cd7a0 0x7f458415cee0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 --2- 172.21.15.72:0/1460906242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45840cce40 0x7f45840cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 >> 172.21.15.72:0/1460906242 conn(0x7f45840cf8a0 msgr2=0x7f458414e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 shutdown_connections 2024-03-06T07:27:11.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:11.574+0000 7f4588881640 1 -- 172.21.15.72:0/1460906242 wait complete. 2024-03-06T07:27:11.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:02.515750+0000', 'modified': '2024-03-06T07:27:04.517794+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7926}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7926': {'gid': 7926, 'name': 'b', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6835/118157918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6833', 'nonce': 118157918}, {'type': 'v1', 'addr': '172.21.15.146:6835', 'nonce': 118157918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7926, 'qdb_cluster': [7926]} 2024-03-06T07:27:11.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:11.991 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-06T07:27:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.092+0000 7fdbac3b5640 1 Processor -- start 2024-03-06T07:27:12.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.092+0000 7fdbac3b5640 1 -- start start 2024-03-06T07:27:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbac3b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdba41444c0 0x7fdba406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbac3b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbac3b5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdba4070e10 0x7fdba415e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbac3b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba41448a0 con 0x7fdba41444c0 2024-03-06T07:27:12.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbac3b5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba4071260 con 0x7fdba4070e10 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbac3b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba406d370 con 0x7fdba415b2d0 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdba41444c0 0x7fdba406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdba41444c0 0x7fdba406ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34234/0 (socket says 172.21.15.72:34234) 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 -- 172.21.15.72:0/4195509884 learned_addr learned my addr 172.21.15.72:0/4195509884 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa952640 1 --2- 172.21.15.72:0/4195509884 >> v2:172.21.15.146:3300/0 conn(0x7fdba4070e10 0x7fdba415e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdba9950640 1 --2- 172.21.15.72:0/4195509884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 -- 172.21.15.72:0/4195509884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 msgr2=0x7fdba40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 --2- 172.21.15.72:0/4195509884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 -- 172.21.15.72:0/4195509884 >> v2:172.21.15.146:3300/0 conn(0x7fdba4070e10 msgr2=0x7fdba415e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 --2- 172.21.15.72:0/4195509884 >> v2:172.21.15.146:3300/0 conn(0x7fdba4070e10 0x7fdba415e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.093+0000 7fdbaa151640 1 -- 172.21.15.72:0/4195509884 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba406d650 con 0x7fdba41444c0 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdbaa151640 1 --2- 172.21.15.72:0/4195509884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba41444c0 0x7fdba406ca00 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fdba40cf510 tx=0x7fdba002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da7b843e5c87cea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdba914f640 1 -- 172.21.15.72:0/4195509884 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0035020 con 0x7fdba41444c0 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdba914f640 1 -- 172.21.15.72:0/4195509884 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0031870 con 0x7fdba41444c0 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdba914f640 1 -- 172.21.15.72:0/4195509884 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0031c40 con 0x7fdba41444c0 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdbac3b5640 1 -- 172.21.15.72:0/4195509884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba41444c0 msgr2=0x7fdba406ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/4195509884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba41444c0 0x7fdba406ca00 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fdba40cf510 tx=0x7fdba002b910 comp rx=0 tx=0).stop 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdbac3b5640 1 -- 172.21.15.72:0/4195509884 shutdown_connections 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.094+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/4195509884 >> v2:172.21.15.146:3300/0 conn(0x7fdba4070e10 0x7fdba415e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.095+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/4195509884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.095+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/4195509884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba41444c0 0x7fdba406ca00 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.095+0000 7fdbac3b5640 1 -- 172.21.15.72:0/4195509884 >> 172.21.15.72:0/4195509884 conn(0x7fdba40b1c80 msgr2=0x7fdba40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.095+0000 7fdbac3b5640 1 -- 172.21.15.72:0/4195509884 shutdown_connections 2024-03-06T07:27:12.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.095+0000 7fdbac3b5640 1 -- 172.21.15.72:0/4195509884 wait complete. 2024-03-06T07:27:12.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 Processor -- start 2024-03-06T07:27:12.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 -- start start 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 0x7fdba415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdba41444c0 0x7fdba40c7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba406def0 con 0x7fdba4070e10 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba406dc10 con 0x7fdba41444c0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbac3b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba406cf40 con 0x7fdba415b2d0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.096+0000 7fdbaa151640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 0x7fdba415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 0x7fdba415a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34244/0 (socket says 172.21.15.72:34244) 2024-03-06T07:27:12.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 -- 172.21.15.72:0/87882981 learned_addr learned my addr 172.21.15.72:0/87882981 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdba9950640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdba41444c0 0x7fdba40c7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa952640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 -- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 msgr2=0x7fdba40cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 -- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdba41444c0 msgr2=0x7fdba40c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdba41444c0 0x7fdba40c7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 -- 172.21.15.72:0/87882981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba415a9a0 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbaa151640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 0x7fdba415a460 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fdba0000c00 tx=0x7fdba00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdb9affd640 1 -- 172.21.15.72:0/87882981 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0035040 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdb9affd640 1 -- 172.21.15.72:0/87882981 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba0030d00 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdb9affd640 1 -- 172.21.15.72:0/87882981 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba00456c0 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba40c8390 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.097+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba40c8950 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.099+0000 7fdb9affd640 1 -- 172.21.15.72:0/87882981 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdba0038020 con 0x7fdba4070e10 2024-03-06T07:27:12.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.099+0000 7fdb9affd640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.099+0000 7fdb9affd640 1 -- 172.21.15.72:0/87882981 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdba00ac3f0 con 0x7fdba4070e10 2024-03-06T07:27:12.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.099+0000 7fdba9950640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.099+0000 7fdba9950640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb8806b4d0 0x7fdb8806d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdba40ccfd0 tx=0x7fdb94009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.100+0000 7fdbaef9f740 1 Processor -- start 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.100+0000 7fdbaef9f740 1 -- start start 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbf8b870 0x560ebbf8bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbf8f8c0 0x560ebbf91360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebbef56b0 con 0x560ebbfbb600 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebbe9ea60 con 0x560ebbf8f8c0 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ebbef1750 con 0x560ebbf8b870 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34258/0 (socket says 172.21.15.72:34258) 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaa952640 1 -- 172.21.15.72:0/744385478 learned_addr learned my addr 172.21.15.72:0/744385478 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdba9950640 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbf8f8c0 0x560ebbf91360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaa151640 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbf8b870 0x560ebbf8bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaa952640 1 -- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbf8b870 msgr2=0x560ebbf8bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.101+0000 7fdbaa952640 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbf8b870 0x560ebbf8bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdbaa952640 1 -- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbf8f8c0 msgr2=0x560ebbf91360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdbaa952640 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbf8f8c0 0x560ebbf91360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdbaa952640 1 -- 172.21.15.72:0/744385478 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ebbebcf30 con 0x560ebbfbb600 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdba9950640 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbf8f8c0 0x560ebbf91360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdbaa952640 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbba80 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c011260 tx=0x7fdb9c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f0cc1124084bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdb87fff640 1 -- 172.21.15.72:0/744385478 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c01a070 con 0x560ebbfbb600 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdb87fff640 1 -- 172.21.15.72:0/744385478 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c016860 con 0x560ebbfbb600 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdb87fff640 1 -- 172.21.15.72:0/744385478 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c016c50 con 0x560ebbfbb600 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdbaef9f740 1 -- 172.21.15.72:0/744385478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 msgr2=0x560ebbfbba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.102+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbba80 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c011260 tx=0x7fdb9c00b040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 -- 172.21.15.72:0/744385478 shutdown_connections 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbba80 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbf8f8c0 0x560ebbf91360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/744385478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbf8b870 0x560ebbf8bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 -- 172.21.15.72:0/744385478 >> 172.21.15.72:0/744385478 conn(0x560ebbf8d8f0 msgr2=0x560ebbf8f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 -- 172.21.15.72:0/744385478 shutdown_connections 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 -- 172.21.15.72:0/744385478 wait complete. 2024-03-06T07:27:12.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 -- start start 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.103+0000 7fdbaef9f740 1 Processor -- start 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfbb600 0x560ebbfafc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbfbd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 0x560ebbfb5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebbf8fd50 con 0x560ebbfb43f0 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebbeb1ea0 con 0x560ebbfb1660 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ebbf90030 con 0x560ebbfbb600 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaef9f740 10 client.0 _ll_register_callbacks cb 0x560ebbeb1870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 0x560ebbfb5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 0x560ebbfb5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34264/0 (socket says 172.21.15.72:34264) 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaa952640 1 -- 172.21.15.72:0/1627412464 learned_addr learned my addr 172.21.15.72:0/1627412464 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdba9950640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbfbd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.104+0000 7fdbaa151640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfbb600 0x560ebbfafc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaa952640 1 -- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfbb600 msgr2=0x560ebbfafc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaa952640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfbb600 0x560ebbfafc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaa952640 1 -- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 msgr2=0x560ebbfbd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaa952640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbfbd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaa952640 1 -- 172.21.15.72:0/1627412464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ebbeb2270 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaa952640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 0x560ebbfb5e10 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c016f70 tx=0x7fdb9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c790b6e2bd6e222a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdb857fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdb857fa640 1 -- 172.21.15.72:0/1627412464 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c01a050 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdb857fa640 1 -- 172.21.15.72:0/1627412464 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb9c003990 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb9c016860 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdb857fa640 1 -- 172.21.15.72:0/1627412464 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb9c02f050 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 10 client.8079 fetch_fsmap learned FSMap version 63 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x560ebbc0a2d0 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdb857fa640 1 -- 172.21.15.72:0/1627412464 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c016860 con 0x560ebbfb43f0 2024-03-06T07:27:12.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdb857fa640 1 -- 172.21.15.72:0/1627412464 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 63) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdb9c032020 con 0x560ebbfb43f0 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 10 client.8079 fetch_fsmap finished waiting for FSMap version 63 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 10 client.8079 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 20 client.8079 populate_metadata read hostname 'smithi072' 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbfbcec0 0x560ebbff5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 4 client.8079 mds_command: new command op to 7926 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.105+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 --> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ebbf02bf0 con 0x560ebbfbcec0 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.106+0000 7fdbaa151640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbfbcec0 0x560ebbff5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.106+0000 7fdbaa151640 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbfbcec0 0x560ebbff5210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb66718577b7e4b0 server_cookie=2fe17dd1ba914778 in_seq=0 out_seq=0 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.106+0000 7fdb857fa640 10 client.8079 ms_handle_connect on v2:172.21.15.146:6833/118157918 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdb857fa640 1 -- 172.21.15.72:0/1627412464 <== mds.0 v2:172.21.15.146:6833/118157918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560ebbf02bf0 con 0x560ebbfbcec0 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdb857fa640 10 client.8079 handle_command_reply: tid=0 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 client.8079 shutdown 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 msgr2=0x560ebbfb5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 0x560ebbfb5e10 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c016f70 tx=0x7fdb9c004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 shutdown_connections 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfb43f0 0x560ebbfb5e10 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbfbd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbfbcec0 0x560ebbff5210 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/1627412464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfbb600 0x560ebbfafc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdb857fa640 0 client.8079 ms_handle_reset on v2:172.21.15.146:6833/118157918 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 >> 172.21.15.72:0/1627412464 conn(0x560ebbf8b870 msgr2=0x560ebbfb95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 shutdown_connections 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 1 -- 172.21.15.72:0/1627412464 wait complete. 2024-03-06T07:27:12.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.107+0000 7fdbaef9f740 20 client.8079 trim_cache size 0 max 16384 2024-03-06T07:27:12.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.124+0000 7fdbaef9f740 1 Processor -- start 2024-03-06T07:27:12.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.124+0000 7fdbaef9f740 1 -- start start 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbf86d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb43f0 0x560ebbfbd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfba850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba41fd0d0 con 0x560ebbfbb600 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4142ba0 con 0x560ebbfb1660 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba41431a0 con 0x560ebbfb43f0 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfba850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34278/0 (socket says 172.21.15.72:34278) 2024-03-06T07:27:12.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 -- 172.21.15.72:0/2014590812 learned_addr learned my addr 172.21.15.72:0/2014590812 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa151640 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbf86d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdba9950640 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb43f0 0x560ebbfbd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 -- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb43f0 msgr2=0x560ebbfbd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb43f0 0x560ebbfbd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 -- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 msgr2=0x560ebbf86d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbf86d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 -- 172.21.15.72:0/2014590812 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ebbcd46b0 con 0x560ebbfbb600 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.125+0000 7fdbaa952640 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfba850 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c011ff0 tx=0x7fdb9c021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c193708ae738be95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdb877fe640 1 -- 172.21.15.72:0/2014590812 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c01a070 con 0x560ebbfbb600 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdb877fe640 1 -- 172.21.15.72:0/2014590812 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c023030 con 0x560ebbfbb600 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdb877fe640 1 -- 172.21.15.72:0/2014590812 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c016860 con 0x560ebbfbb600 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2014590812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 msgr2=0x560ebbfba850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfba850 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c011ff0 tx=0x7fdb9c021070 comp rx=0 tx=0).stop 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2014590812 shutdown_connections 2024-03-06T07:27:12.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfba850 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb43f0 0x560ebbfbd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2014590812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb1660 0x560ebbf86d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2014590812 >> 172.21.15.72:0/2014590812 conn(0x560ebbf8b870 msgr2=0x560ebbfb5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2014590812 shutdown_connections 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.126+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2014590812 wait complete. 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.127+0000 7fdbaef9f740 1 -- start start 2024-03-06T07:27:12.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.127+0000 7fdbaef9f740 1 Processor -- start 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb1660 0x560ebc049280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb43f0 0x560ebbfba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebbfbe180 con 0x560ebbfbb600 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebbfb0b80 con 0x560ebbfb43f0 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x560ebbfb4870 con 0x560ebbfb1660 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 10 client.0 _ll_register_callbacks cb 0x7fdb9c009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbaad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34292/0 (socket says 172.21.15.72:34292) 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 -- 172.21.15.72:0/2750400573 learned_addr learned my addr 172.21.15.72:0/2750400573 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa151640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb1660 0x560ebc049280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdba9950640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb43f0 0x560ebbfba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 -- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb1660 msgr2=0x560ebc049280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb1660 0x560ebc049280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 -- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb43f0 msgr2=0x560ebbfba590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb43f0 0x560ebbfba590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 -- 172.21.15.72:0/2750400573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ebbf86e30 con 0x560ebbfbb600 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaa952640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbaad0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c003de0 tx=0x7fdb9c00d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9c1824857c03e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdb857fa640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdb857fa640 1 -- 172.21.15.72:0/2750400573 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c042060 con 0x560ebbfbb600 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdb857fa640 1 -- 172.21.15.72:0/2750400573 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb9c004110 con 0x560ebbfbb600 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.128+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb9c023030 con 0x560ebbfbb600 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdb857fa640 1 -- 172.21.15.72:0/2750400573 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb9c01a050 con 0x560ebbfbb600 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 10 client.8085 fetch_fsmap learned FSMap version 63 2024-03-06T07:27:12.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x560ebbfebf40 con 0x560ebbfbb600 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdb857fa640 1 -- 172.21.15.72:0/2750400573 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c023030 con 0x560ebbfbb600 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdb857fa640 1 -- 172.21.15.72:0/2750400573 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 63) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fdb9c032050 con 0x560ebbfbb600 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 10 client.8085 fetch_fsmap finished waiting for FSMap version 63 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 10 client.8085 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 20 client.8085 populate_metadata read hostname 'smithi072' 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbff9e40 0x560ebbffa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 4 client.8085 mds_command: new command op to 7926 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 --> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560ebbefcf40 con 0x560ebbff9e40 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaa151640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbff9e40 0x560ebbffa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdb857fa640 10 client.8085 ms_handle_connect on v2:172.21.15.146:6833/118157918 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaa151640 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbff9e40 0x560ebbffa220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3814c11ad4d36ce server_cookie=1c5468d5e20c2e92 in_seq=0 out_seq=0 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdb857fa640 1 -- 172.21.15.72:0/2750400573 <== mds.0 v2:172.21.15.146:6833/118157918 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x560ebbefcf40 con 0x560ebbff9e40 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdb857fa640 10 client.8085 handle_command_reply: tid=0 2024-03-06T07:27:12.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.129+0000 7fdbaef9f740 1 client.8085 shutdown 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 msgr2=0x560ebbfbaad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbaad0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c003de0 tx=0x7fdb9c00d040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 shutdown_connections 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560ebbfbb600 0x560ebbfbaad0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x560ebbfb43f0 0x560ebbfba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:6833/118157918,v1:172.21.15.146:6835/118157918] conn(0x560ebbff9e40 0x560ebbffa220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 --2- 172.21.15.72:0/2750400573 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x560ebbfb1660 0x560ebc049280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 >> 172.21.15.72:0/2750400573 conn(0x560ebbf8b870 msgr2=0x560ebbfb53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdb857fa640 0 client.8085 ms_handle_reset on v2:172.21.15.146:6833/118157918 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 shutdown_connections 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 1 -- 172.21.15.72:0/2750400573 wait complete. 2024-03-06T07:27:12.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbaef9f740 20 client.8085 trim_cache size 0 max 16384 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 7926, 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 8.9654449029999999, 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 63, 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 110, 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 110, 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 14.018009198 2024-03-06T07:27:12.139 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:27:12.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb8806b4d0 msgr2=0x7fdb8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb8806b4d0 0x7fdb8806d990 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fdba40ccfd0 tx=0x7fdb94009190 comp rx=0 tx=0).stop 2024-03-06T07:27:12.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 msgr2=0x7fdba415a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 0x7fdba415a460 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fdba0000c00 tx=0x7fdba00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 shutdown_connections 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb8806b4d0 0x7fdb8806d990 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdba415b2d0 0x7fdba40cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdba41444c0 0x7fdba40c7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 --2- 172.21.15.72:0/87882981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdba4070e10 0x7fdba415a460 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 >> 172.21.15.72:0/87882981 conn(0x7fdba40b1c80 msgr2=0x7fdba413ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 shutdown_connections 2024-03-06T07:27:12.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.130+0000 7fdbac3b5640 1 -- 172.21.15.72:0/87882981 wait complete. 2024-03-06T07:27:12.143 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:12.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 Processor -- start 2024-03-06T07:27:12.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 -- start start 2024-03-06T07:27:12.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6ee01515e0 0x7f6ee00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6ee00cd7a0 0x7f6ee00cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee0071f40 con 0x7f6ee015b430 2024-03-06T07:27:12.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee015b8e0 con 0x7f6ee01515e0 2024-03-06T07:27:12.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.241+0000 7f6ee6dee640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee0070a70 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6ee4b8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6ee01515e0 0x7f6ee00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6edffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6ee015b430 0x7f6ee0071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59858/0 (socket says 172.21.15.72:59858) 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6edffff640 1 -- 172.21.15.72:0/473636153 learned_addr learned my addr 172.21.15.72:0/473636153 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6ee538b640 1 --2- 172.21.15.72:0/473636153 >> v2:172.21.15.146:3300/0 conn(0x7f6ee00cd7a0 0x7f6ee00cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/473636153 >> v2:172.21.15.146:3300/0 conn(0x7f6ee00cd7a0 msgr2=0x7f6ee00cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6ee4b8a640 1 --2- 172.21.15.72:0/473636153 >> v2:172.21.15.146:3300/0 conn(0x7f6ee00cd7a0 0x7f6ee00cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/473636153 >> v1:172.21.15.72:6789/0 conn(0x7f6ee015b430 legacy=0x7f6ee0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.242+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/473636153 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0070d50 con 0x7f6ee01515e0 2024-03-06T07:27:12.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.243+0000 7f6ee4b8a640 1 --2- 172.21.15.72:0/473636153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee01515e0 0x7f6ee00cce40 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00cc340 tx=0x7f6ed402cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec73f2172e698453 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.243+0000 7f6edf7fe640 1 -- 172.21.15.72:0/473636153 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed403a070 con 0x7f6ee01515e0 2024-03-06T07:27:12.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.243+0000 7f6edf7fe640 1 -- 172.21.15.72:0/473636153 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed402db50 con 0x7f6ee01515e0 2024-03-06T07:27:12.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.243+0000 7f6edf7fe640 1 -- 172.21.15.72:0/473636153 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed400b040 con 0x7f6ee01515e0 2024-03-06T07:27:12.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 -- 172.21.15.72:0/473636153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee01515e0 msgr2=0x7f6ee00cce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/473636153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee01515e0 0x7f6ee00cce40 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00cc340 tx=0x7f6ed402cda0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 -- 172.21.15.72:0/473636153 shutdown_connections 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/473636153 >> v2:172.21.15.146:3300/0 conn(0x7f6ee00cd7a0 0x7f6ee00cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/473636153 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee01515e0 0x7f6ee00cce40 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 -- 172.21.15.72:0/473636153 >> 172.21.15.72:0/473636153 conn(0x7f6ee00cf8a0 msgr2=0x7f6ee00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 -- 172.21.15.72:0/473636153 shutdown_connections 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.244+0000 7f6ee6dee640 1 -- 172.21.15.72:0/473636153 wait complete. 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.245+0000 7f6ee6dee640 1 Processor -- start 2024-03-06T07:27:12.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.245+0000 7f6ee6dee640 1 -- start start 2024-03-06T07:27:12.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee6dee640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 0x7f6ee014a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee6dee640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ee01515e0 0x7f6ee0149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee6dee640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee015b430 0x7f6ee014e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee6dee640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00ce0c0 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee6dee640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0071310 con 0x7f6ee015b430 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee6dee640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00cc3a0 con 0x7f6ee01515e0 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 0x7f6ee014a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 0x7f6ee014a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34314/0 (socket says 172.21.15.72:34314) 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/1410255134 learned_addr learned my addr 172.21.15.72:0/1410255134 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee538b640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee015b430 0x7f6ee014e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6edffff640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ee01515e0 0x7f6ee0149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ee01515e0 msgr2=0x7f6ee0149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ee01515e0 0x7f6ee0149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee015b430 msgr2=0x7f6ee014e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee015b430 0x7f6ee014e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee4b8a640 1 -- 172.21.15.72:0/1410255134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee0151ac0 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.246+0000 7f6ee538b640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee015b430 0x7f6ee014e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.247+0000 7f6ee4b8a640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 0x7f6ee014a690 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400aa30 tx=0x7f6ed40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.247+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed403a040 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.247+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed4045070 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.247+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee015d8e0 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.247+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4006e10 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.247+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0151d60 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.248+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee0068840 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.248+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6ed4040680 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.249+0000 7f6edd7fa640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6ec406b4d0 0x7f6ec406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.249+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ed40ad770 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.249+0000 7f6edffff640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6ec406b4d0 0x7f6ec406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:12.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.249+0000 7f6edffff640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6ec406b4d0 0x7f6ec406d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0150ee0 tx=0x7f6ecc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:12.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.253+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6ed407c5f0 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.460+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6ee004f330 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.460+0000 7f6edd7fa640 1 -- 172.21.15.72:0/1410255134 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6ed407bef0 con 0x7f6ee00cd7a0 2024-03-06T07:27:12.462 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6ec406b4d0 msgr2=0x7f6ec406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6ec406b4d0 0x7f6ec406d990 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0150ee0 tx=0x7f6ecc009040 comp rx=0 tx=0).stop 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 msgr2=0x7f6ee014a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 0x7f6ee014a690 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400aa30 tx=0x7f6ed40046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 shutdown_connections 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6ec406b4d0 0x7f6ec406d990 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ee015b430 0x7f6ee014e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ee01515e0 0x7f6ee0149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 --2- 172.21.15.72:0/1410255134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ee00cd7a0 0x7f6ee014a690 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 >> 172.21.15.72:0/1410255134 conn(0x7f6ee00cf8a0 msgr2=0x7f6ee015aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 shutdown_connections 2024-03-06T07:27:12.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:12.462+0000 7f6ee6dee640 1 -- 172.21.15.72:0/1410255134 wait complete. 2024-03-06T07:27:12.475 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:12.475 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:12.475 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:27:12.475 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:12.475 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:12.475 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:12.476 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:12.476 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:12.476 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:12.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.476 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:27:12.491 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:27:12.492 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42363sec preferred_lft 42363sec 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:12.493 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:12.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.494 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:12.557 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:27:12.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.558 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:27:12.614 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:27:12.614 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:12.614 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:12.615 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:12.615 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:12.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.615 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:27:12.615 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:27:12.615 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:12.615 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:12.615 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:12.615 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:27:12.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.710 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:27:12.725 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:12.725 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:12.725 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:12.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:12.725 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:27:12.791 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:12.791 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:12.791 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:12.846 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:12.846 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:12.846 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:12.901 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:27:12.967 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:27:13.022 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:13.023 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:13.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.025 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:27:13.077 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:13.077 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:13.120 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:27:13.146 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:13.202 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:27:13.202 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:27:13.203 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:27:13.204 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:13.205 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:13.205 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:13.205 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:13.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.205 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:27:13.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:27:13.203+0000 7febe5a60480 -1 init, newargv = 0x559c92deb7a0 newargc=15 2024-03-06T07:27:13.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[72507]: starting ceph client 2024-03-06T07:27:13.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[72507]: starting fuse 2024-03-06T07:27:13.220 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:27:13.220 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:27:13.221 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:13.222 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:27:13.340 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.72507.asok 2024-03-06T07:27:13.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:13.350 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72507.asok 2024-03-06T07:27:13.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.350 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72507.asok status 2024-03-06T07:27:13.486 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "72507", 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 8097, 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:27:13.487 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 8097 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1825565399 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1825565399 2024-03-06T07:27:13.488 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.8097 192.168.0.1:0/1825565399", 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/1825565399", 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 63, 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 110, 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:27:13.489 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:27:13.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.495 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:27:13.495 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:13.510 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:27:13.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:13.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.510 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:13.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.578 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-03-06T07:27:13.637 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:27:13.637 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:13.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.692 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-03-06T07:27:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:8+0 records in 2024-03-06T07:27:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:8+0 records out 2024-03-06T07:27:13.807 INFO:teuthology.orchestra.run.smithi072.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0571902 s, 147 MB/s 2024-03-06T07:27:13.807 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:27:13.807 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:27:13.808 DEBUG:teuthology.orchestra.run.smithi072:> import stat 2024-03-06T07:27:13.808 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:13.808 DEBUG:teuthology.orchestra.run.smithi072:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-03-06T07:27:13.808 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:27:13.872 INFO:teuthology.orchestra.run.smithi072.stdout:1099511627777 2024-03-06T07:27:13.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:13.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.881 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-03-06T07:27:13.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:13.938 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-03-06T07:27:13.997 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:27:13.997 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:27:13.997 DEBUG:teuthology.orchestra.run.smithi072:> import stat 2024-03-06T07:27:13.997 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:13.998 DEBUG:teuthology.orchestra.run.smithi072:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-03-06T07:27:13.998 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:27:14.103 INFO:teuthology.orchestra.run.smithi072.stdout:1099511627777 2024-03-06T07:27:14.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:14.111 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:14.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.249+0000 7f04e0c81640 1 Processor -- start 2024-03-06T07:27:14.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.249+0000 7f04e0c81640 1 -- start start 2024-03-06T07:27:14.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04e0c81640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f04dc15fa10 0x7f04dc0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04e0c81640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f04dc0d16e0 0x7f04dc0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04e0c81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0cebe0 con 0x7f04dc15bdd0 2024-03-06T07:27:14.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04e0c81640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc0ceec0 con 0x7f04dc15fa10 2024-03-06T07:27:14.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04e0c81640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04dc0cf1a0 con 0x7f04dc0d16e0 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04da59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f04dc15bdd0 0x7f04dc0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45564/0 (socket says 172.21.15.72:45564) 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04d9d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f04dc15fa10 0x7f04dc0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04da59c640 1 -- 172.21.15.72:0/447392775 learned_addr learned my addr 172.21.15.72:0/447392775 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.250+0000 7f04dad9d640 1 --2- 172.21.15.72:0/447392775 >> v2:172.21.15.146:3300/0 conn(0x7f04dc0d16e0 0x7f04dc0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d9d9b640 1 -- 172.21.15.72:0/447392775 >> v2:172.21.15.146:3300/0 conn(0x7f04dc0d16e0 msgr2=0x7f04dc0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d9d9b640 1 --2- 172.21.15.72:0/447392775 >> v2:172.21.15.146:3300/0 conn(0x7f04dc0d16e0 0x7f04dc0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d9d9b640 1 -- 172.21.15.72:0/447392775 >> v1:172.21.15.72:6789/0 conn(0x7f04dc15bdd0 legacy=0x7f04dc0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d9d9b640 1 -- 172.21.15.72:0/447392775 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0cc8e0 con 0x7f04dc15fa10 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d959a640 1 -- 172.21.15.72:0/447392775 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434677269 0 0) 0x7f04dc0cebe0 con 0x7f04dc15bdd0 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d9d9b640 1 --2- 172.21.15.72:0/447392775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc0ce270 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0c5520 tx=0x7f04d002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feddff4c45d3c92e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.251+0000 7f04d959a640 1 -- 172.21.15.72:0/447392775 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0039070 con 0x7f04dc15fa10 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04d959a640 1 -- 172.21.15.72:0/447392775 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d002cbf0 con 0x7f04dc15fa10 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04d959a640 1 -- 172.21.15.72:0/447392775 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04d0034440 con 0x7f04dc15fa10 2024-03-06T07:27:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 -- 172.21.15.72:0/447392775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 msgr2=0x7f04dc0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 --2- 172.21.15.72:0/447392775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc0ce270 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0c5520 tx=0x7f04d002be40 comp rx=0 tx=0).stop 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 -- 172.21.15.72:0/447392775 shutdown_connections 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 --2- 172.21.15.72:0/447392775 >> v2:172.21.15.146:3300/0 conn(0x7f04dc0d16e0 0x7f04dc0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 --2- 172.21.15.72:0/447392775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc0ce270 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 -- 172.21.15.72:0/447392775 >> 172.21.15.72:0/447392775 conn(0x7f04dc09a9f0 msgr2=0x7f04dc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 -- 172.21.15.72:0/447392775 shutdown_connections 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.252+0000 7f04e0c81640 1 -- 172.21.15.72:0/447392775 wait complete. 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.253+0000 7f04e0c81640 1 Processor -- start 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.253+0000 7f04e0c81640 1 -- start start 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04e0c81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04dc0d16e0 0x7f04dc0b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04e0c81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04dc15bdd0 0x7f04dc0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04e0c81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc06d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04e0c81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0d2030 con 0x7f04dc15fa10 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04e0c81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc15c1b0 con 0x7f04dc15bdd0 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04e0c81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc0ce7b0 con 0x7f04dc0d16e0 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc06d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc06d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60636/0 (socket says 172.21.15.72:60636) 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 -- 172.21.15.72:0/217233234 learned_addr learned my addr 172.21.15.72:0/217233234 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04d9d9b640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04dc15bdd0 0x7f04dc0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04da59c640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04dc0d16e0 0x7f04dc0b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 -- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04dc0d16e0 msgr2=0x7f04dc0b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04dc0d16e0 0x7f04dc0b6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 -- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04dc15bdd0 msgr2=0x7f04dc0c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04dc15bdd0 0x7f04dc0c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.254+0000 7f04dad9d640 1 -- 172.21.15.72:0/217233234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0b73d0 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.255+0000 7f04dad9d640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc06d090 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00e930 tx=0x7f04cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.255+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc017070 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.255+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04cc00c980 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.255+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc0c9f50 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.255+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00cd70 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.255+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc06cb70 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.256+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc0ca5d0 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.256+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f04cc01a070 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.257+0000 7f04caffd640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f04bc06b4d0 0x7f04bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.257+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f04cc08a250 con 0x7f04dc15fa10 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.257+0000 7f04da59c640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f04bc06b4d0 0x7f04bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.257+0000 7f04da59c640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f04bc06b4d0 0x7f04bc06d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f04d4000f60 tx=0x7f04d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.261+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f04dc0b73d0 con 0x7f04dc15fa10 2024-03-06T07:27:14.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.465+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04dc04f330 con 0x7f04dc15fa10 2024-03-06T07:27:14.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.465+0000 7f04caffd640 1 -- 172.21.15.72:0/217233234 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f04cc058a30 con 0x7f04dc15fa10 2024-03-06T07:27:14.468 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:14.468 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2347391904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2347391904},{"type":"v1","addr":"172.21.15.146:6834","nonce":2347391904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/1152083148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1152083148},{"type":"v1","addr":"172.21.15.72:6835","nonce":1152083148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3618393393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":3618393393},{"type":"v1","addr":"172.21.15.72:6837","nonce":3618393393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:04.517794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/118157918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":118157918},{"type":"v1","addr":"172.21.15.146:6835","nonce":118157918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7926,"qdb_cluster":[7926]},"id":8}]} 2024-03-06T07:27:14.468 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 63 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f04bc06b4d0 msgr2=0x7f04bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f04bc06b4d0 0x7f04bc06d990 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f04d4000f60 tx=0x7f04d4009040 comp rx=0 tx=0).stop 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 msgr2=0x7f04dc06d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc06d090 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00e930 tx=0x7f04cc00edf0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 shutdown_connections 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f04bc06b4d0 0x7f04bc06d990 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04dc15fa10 0x7f04dc06d090 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f04dc15bdd0 0x7f04dc0c6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 --2- 172.21.15.72:0/217233234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f04dc0d16e0 0x7f04dc0b6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 >> 172.21.15.72:0/217233234 conn(0x7f04dc09a9f0 msgr2=0x7f04dc0cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 shutdown_connections 2024-03-06T07:27:14.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.467+0000 7f04e0c81640 1 -- 172.21.15.72:0/217233234 wait complete. 2024-03-06T07:27:14.480 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:27:14.612 INFO:teuthology.orchestra.run.smithi146.stdout:{"message":"","return_code":0} 2024-03-06T07:27:14.613 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:14.613 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:27:14.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.652+0000 7f0e1f64bf40 1 Processor -- start 2024-03-06T07:27:14.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.652+0000 7f0e1f64bf40 1 -- start start 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1f64bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 0x5592def4c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1f64bf40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb93f0 0x5592def45300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1f64bf40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592dee9f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1f64bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592def2e9f0 con 0x5592deeb8fe0 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1f64bf40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592dee1ab70 con 0x5592def4e520 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1f64bf40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592def2ec90 con 0x5592deeb93f0 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 0x5592def4c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 0x5592def4c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60656/0 (socket says 172.21.15.72:60656) 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 -- 172.21.15.72:0/4002939921 learned_addr learned my addr 172.21.15.72:0/4002939921 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1d1a9640 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592dee9f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e17fff640 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb93f0 0x5592def45300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 -- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb93f0 msgr2=0x5592def45300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb93f0 0x5592def45300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 -- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 msgr2=0x5592dee9f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592dee9f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 -- 172.21.15.72:0/4002939921 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592def2d090 con 0x5592deeb8fe0 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1d1a9640 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592dee9f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.653+0000 7f0e1c9a8640 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 0x5592def4c980 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0e1001e320 tx=0x7f0e10051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8613b6ded8490ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e16ffd640 1 -- 172.21.15.72:0/4002939921 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10067070 con 0x5592deeb8fe0 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e16ffd640 1 -- 172.21.15.72:0/4002939921 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e10050cc0 con 0x5592deeb8fe0 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e16ffd640 1 -- 172.21.15.72:0/4002939921 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e10058640 con 0x5592deeb8fe0 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/4002939921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 msgr2=0x5592def4c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 0x5592def4c980 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f0e1001e320 tx=0x7f0e10051000 comp rx=0 tx=0).stop 2024-03-06T07:27:14.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/4002939921 shutdown_connections 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592dee9f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb93f0 0x5592def45300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/4002939921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb8fe0 0x5592def4c980 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/4002939921 >> 172.21.15.72:0/4002939921 conn(0x5592dee728b0 msgr2=0x5592deea0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/4002939921 shutdown_connections 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.654+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/4002939921 wait complete. 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 Processor -- start 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 -- start start 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb8fe0 0x5592deebead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb93f0 0x5592def445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592deebb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592def2d390 con 0x5592deeb93f0 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592def2b640 con 0x5592def4e520 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e1f64bf40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5592def2ba80 con 0x5592deeb8fe0 2024-03-06T07:27:14.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.655+0000 7f0e17fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb93f0 0x5592def445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb93f0 0x5592def445f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60670/0 (socket says 172.21.15.72:60670) 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 -- 172.21.15.72:0/3676802465 learned_addr learned my addr 172.21.15.72:0/3676802465 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e1c9a8640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb8fe0 0x5592deebead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e1d1a9640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592deebb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 -- 172.21.15.72:0/3676802465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb8fe0 msgr2=0x5592deebead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5592deeb8fe0 0x5592deebead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 -- 172.21.15.72:0/3676802465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 msgr2=0x5592deebb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5592def4e520 0x5592deebb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 -- 172.21.15.72:0/3676802465 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592deebdfa0 con 0x5592deeb93f0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e17fff640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb93f0 0x5592def445f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009e90 tx=0x7f0e08009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e15ffb640 1 -- 172.21.15.72:0/3676802465 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e080143f0 con 0x5592deeb93f0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e15ffb640 1 -- 172.21.15.72:0/3676802465 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08019070 con 0x5592deeb93f0 2024-03-06T07:27:14.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5592def44b30 con 0x5592deeb93f0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e15ffb640 1 -- 172.21.15.72:0/3676802465 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08014790 con 0x5592deeb93f0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.656+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5592deebe520 con 0x5592deeb93f0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.657+0000 7f0e15ffb640 1 -- 172.21.15.72:0/3676802465 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0e08014930 con 0x5592deeb93f0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e15ffb640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e00069eb0 0x7f0e0006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e15ffb640 1 -- 172.21.15.72:0/3676802465 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e08088b60 con 0x5592deeb93f0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e1c9a8640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e00069eb0 0x7f0e0006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x5592defe1900 0x5592defe1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e110) v9 -- 0x5592defe2220 con 0x5592defe1900 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e1d1a9640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x5592defe1900 0x5592defe1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e1c9a8640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e00069eb0 0x7f0e0006c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0e10064000 tx=0x7f0e10053000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.658+0000 7f0e1d1a9640 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x5592defe1900 0x5592defe1ce0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1d1a9640 1 -- 172.21.15.72:0/3676802465 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7f0e18003450 con 0x5592defe1900 2024-03-06T07:27:14.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x5592defe1900 msgr2=0x5592defe1ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x5592defe1900 0x5592defe1ce0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e00069eb0 msgr2=0x7f0e0006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0e00069eb0 0x7f0e0006c370 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0e10064000 tx=0x7f0e10053000 comp rx=0 tx=0).stop 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb93f0 msgr2=0x5592def445f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 --2- 172.21.15.72:0/3676802465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5592deeb93f0 0x5592def445f0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009e90 tx=0x7f0e08009f80 comp rx=0 tx=0).stop 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1d1a9640 1 -- 172.21.15.72:0/3676802465 reap_dead start 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 shutdown_connections 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 >> 172.21.15.72:0/3676802465 conn(0x5592dee728b0 msgr2=0x5592def45620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 shutdown_connections 2024-03-06T07:27:14.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:14.659+0000 7f0e1f64bf40 1 -- 172.21.15.72:0/3676802465 wait complete. 2024-03-06T07:27:14.709 INFO:teuthology.orchestra.run.smithi072.stdout:Wfile_a 2024-03-06T07:27:14.709 INFO:teuthology.orchestra.run.smithi072.stdout:dir_1 2024-03-06T07:27:14.709 DEBUG:teuthology.orchestra.run.smithi072:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:27:14.843 INFO:teuthology.orchestra.run.smithi072.stdout: "ino": 1099511627777, 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "ancestors": [ 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1099511627776, 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "file_a", 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 10 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1, 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "dir_1", 2024-03-06T07:27:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 17 2024-03-06T07:27:14.845 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:27:14.845 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-06T07:27:14.845 INFO:teuthology.orchestra.run.smithi072.stdout: "pool": 17, 2024-03-06T07:27:14.845 INFO:teuthology.orchestra.run.smithi072.stdout: "old_pools": [] 2024-03-06T07:27:14.845 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:27:14.845 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:14.861 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:27:14.861 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:14.877 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:15.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.016+0000 7f242d815640 1 Processor -- start 2024-03-06T07:27:15.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.016+0000 7f242d815640 1 -- start start 2024-03-06T07:27:15.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f242d815640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f242815dfe0 0x7f24280cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f242d815640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280b3430 con 0x7f24280cf830 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f242d815640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24280b3710 con 0x7f24280d29c0 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f242d815640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24280b31e0 con 0x7f242815dfe0 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f24267fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f24280cf830 0x7f24280d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45582/0 (socket says 172.21.15.72:45582) 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f24267fc640 1 -- 172.21.15.72:0/3412121997 learned_addr learned my addr 172.21.15.72:0/3412121997 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f2426ffd640 1 --2- 172.21.15.72:0/3412121997 >> v2:172.21.15.146:3300/0 conn(0x7f242815dfe0 0x7f24280cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f2425ffb640 1 -- 172.21.15.72:0/3412121997 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131434825 0 0) 0x7f24280b3430 con 0x7f24280cf830 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f2425ffb640 1 -- 172.21.15.72:0/3412121997 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2410003520 con 0x7f24280cf830 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f2426ffd640 1 -- 172.21.15.72:0/3412121997 >> v1:172.21.15.146:6789/0 conn(0x7f24280d29c0 legacy=0x7f24280b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f2426ffd640 1 -- 172.21.15.72:0/3412121997 >> v1:172.21.15.72:6789/0 conn(0x7f24280cf830 legacy=0x7f24280d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.017+0000 7f2426ffd640 1 -- 172.21.15.72:0/3412121997 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24280b39f0 con 0x7f242815dfe0 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.018+0000 7f2426ffd640 1 --2- 172.21.15.72:0/3412121997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242815dfe0 0x7f24280cf2f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f24280b3360 tx=0x7f242002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efde9dc3c118aed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.018+0000 7f2425ffb640 1 -- 172.21.15.72:0/3412121997 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420039070 con 0x7f242815dfe0 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.018+0000 7f2425ffb640 1 -- 172.21.15.72:0/3412121997 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242002cce0 con 0x7f242815dfe0 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.018+0000 7f2425ffb640 1 -- 172.21.15.72:0/3412121997 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420034640 con 0x7f242815dfe0 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 -- 172.21.15.72:0/3412121997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242815dfe0 msgr2=0x7f24280cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 --2- 172.21.15.72:0/3412121997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242815dfe0 0x7f24280cf2f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f24280b3360 tx=0x7f242002bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 -- 172.21.15.72:0/3412121997 shutdown_connections 2024-03-06T07:27:15.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 --2- 172.21.15.72:0/3412121997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f242815dfe0 0x7f24280cf2f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 -- 172.21.15.72:0/3412121997 >> 172.21.15.72:0/3412121997 conn(0x7f24280d3cd0 msgr2=0x7f24280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 -- 172.21.15.72:0/3412121997 shutdown_connections 2024-03-06T07:27:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.019+0000 7f242d815640 1 -- 172.21.15.72:0/3412121997 wait complete. 2024-03-06T07:27:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.020+0000 7f242d815640 1 Processor -- start 2024-03-06T07:27:15.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.020+0000 7f242d815640 1 -- start start 2024-03-06T07:27:15.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f242d815640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24280cf830 0x7f242815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f242d815640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280d29c0 0x7f24281f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f242d815640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 0x7f24280b8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f242d815640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242815e3c0 con 0x7f242815dfe0 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f242d815640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b3d60 con 0x7f24280d29c0 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f242d815640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f242815e540 con 0x7f24280cf830 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 0x7f24280b8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 0x7f24280b8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60678/0 (socket says 172.21.15.72:60678) 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 -- 172.21.15.72:0/3388440081 learned_addr learned my addr 172.21.15.72:0/3388440081 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24267fc640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280d29c0 0x7f24281f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f2426ffd640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24280cf830 0x7f242815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 -- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24280cf830 msgr2=0x7f242815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24280cf830 0x7f242815f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 -- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280d29c0 msgr2=0x7f24281f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280d29c0 0x7f24281f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 -- 172.21.15.72:0/3388440081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242815fea0 con 0x7f242815dfe0 2024-03-06T07:27:15.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.021+0000 7f24277fe640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 0x7f24280b8dd0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f241800c3a0 tx=0x7f241800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.022+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418017070 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.022+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24180052d0 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.022+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2428160180 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.022+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418003830 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.022+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2428160740 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.022+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242804f330 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.023+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f241801a070 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.023+0000 7f240ffff640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241406b4d0 0x7f241406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.024+0000 7f2426ffd640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241406b4d0 0x7f241406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.024+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2418089440 con 0x7f242815dfe0 2024-03-06T07:27:15.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.024+0000 7f2426ffd640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241406b4d0 0x7f241406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2420036000 tx=0x7f2420030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.028+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2418057dc0 con 0x7f242815dfe0 2024-03-06T07:27:15.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.199+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24280b2c30 con 0x7f242815dfe0 2024-03-06T07:27:15.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.200+0000 7f240ffff640 1 -- 172.21.15.72:0/3388440081 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f241805be60 con 0x7f242815dfe0 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:15.203 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":110,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:04.164794+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241406b4d0 msgr2=0x7f241406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241406b4d0 0x7f241406d990 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f2420036000 tx=0x7f2420030000 comp rx=0 tx=0).stop 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 msgr2=0x7f24280b8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 0x7f24280b8dd0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f241800c3a0 tx=0x7f241800c860 comp rx=0 tx=0).stop 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 shutdown_connections 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f241406b4d0 0x7f241406d990 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f242815dfe0 0x7f24280b8dd0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24280d29c0 0x7f24281f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 --2- 172.21.15.72:0/3388440081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24280cf830 0x7f242815f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.202+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 >> 172.21.15.72:0/3388440081 conn(0x7f24280d3cd0 msgr2=0x7f24280d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.203+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 shutdown_connections 2024-03-06T07:27:15.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.203+0000 7f242d815640 1 -- 172.21.15.72:0/3388440081 wait complete. 2024-03-06T07:27:15.216 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:00.730111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:01.483409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:15.237 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:15.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.337+0000 7f1b0f415640 1 Processor -- start 2024-03-06T07:27:15.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.338+0000 7f1b0f415640 1 -- start start 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.338+0000 7f1b0f415640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b080cf0d0 0x7f1b0815a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.338+0000 7f1b0f415640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b080712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.338+0000 7f1b0f415640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b080b3320 con 0x7f1b080cfef0 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.338+0000 7f1b0f415640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b0815a760 con 0x7f1b080cf0d0 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.338+0000 7f1b0f415640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080728d0 con 0x7f1b080cfa90 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0d1b1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b080cf0d0 0x7f1b0815a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0c9b0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b080712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0d9b2640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1b080cfef0 0x7f1b0815de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57222/0 (socket says 172.21.15.72:57222) 2024-03-06T07:27:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0d9b2640 1 -- 172.21.15.72:0/2704599456 learned_addr learned my addr 172.21.15.72:0/2704599456 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0c9b0640 1 -- 172.21.15.72:0/2704599456 >> v2:172.21.15.146:3300/0 conn(0x7f1b080cf0d0 msgr2=0x7f1b0815a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0c9b0640 1 --2- 172.21.15.72:0/2704599456 >> v2:172.21.15.146:3300/0 conn(0x7f1b080cf0d0 0x7f1b0815a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0c9b0640 1 -- 172.21.15.72:0/2704599456 >> v1:172.21.15.146:6789/0 conn(0x7f1b080cfef0 legacy=0x7f1b0815de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1afffff640 1 -- 172.21.15.72:0/2704599456 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536195288 0 0) 0x7f1b080b3320 con 0x7f1b080cfef0 2024-03-06T07:27:15.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.339+0000 7f1b0c9b0640 1 -- 172.21.15.72:0/2704599456 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b080b3590 con 0x7f1b080cfa90 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0c9b0640 1 --2- 172.21.15.72:0/2704599456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b080712d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1af80097a0 tx=0x7f1af802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49743240314e9871 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1afffff640 1 -- 172.21.15.72:0/2704599456 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af80315a0 con 0x7f1b080cfa90 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1afffff640 1 -- 172.21.15.72:0/2704599456 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af8037070 con 0x7f1b080cfa90 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1afffff640 1 -- 172.21.15.72:0/2704599456 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8031960 con 0x7f1b080cfa90 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0f415640 1 -- 172.21.15.72:0/2704599456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 msgr2=0x7f1b080712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2704599456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b080712d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1af80097a0 tx=0x7f1af802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:15.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0f415640 1 -- 172.21.15.72:0/2704599456 shutdown_connections 2024-03-06T07:27:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2704599456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b080712d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2704599456 >> v2:172.21.15.146:3300/0 conn(0x7f1b080cf0d0 0x7f1b0815a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.340+0000 7f1b0f415640 1 -- 172.21.15.72:0/2704599456 >> 172.21.15.72:0/2704599456 conn(0x7f1b0806b190 msgr2=0x7f1b0806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.341+0000 7f1b0f415640 1 -- 172.21.15.72:0/2704599456 shutdown_connections 2024-03-06T07:27:15.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.341+0000 7f1b0f415640 1 -- 172.21.15.72:0/2704599456 wait complete. 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 Processor -- start 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 -- start start 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b080cf0d0 0x7f1b081522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b08147360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 0x7f1b081478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08072030 con 0x7f1b080cfef0 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b080cf4b0 con 0x7f1b080cf0d0 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0f415640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08072530 con 0x7f1b080cfa90 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0d9b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 0x7f1b081478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0d9b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 0x7f1b081478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60686/0 (socket says 172.21.15.72:60686) 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.342+0000 7f1b0d9b2640 1 -- 172.21.15.72:0/2201840239 learned_addr learned my addr 172.21.15.72:0/2201840239 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d1b1640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b080cf0d0 0x7f1b081522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0c9b0640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b08147360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d9b2640 1 -- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 msgr2=0x7f1b08147360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d9b2640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b08147360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d9b2640 1 -- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b080cf0d0 msgr2=0x7f1b081522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d9b2640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b080cf0d0 0x7f1b081522f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d9b2640 1 -- 172.21.15.72:0/2201840239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b081507c0 con 0x7f1b080cfef0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d1b1640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b080cf0d0 0x7f1b081522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0d9b2640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 0x7f1b081478a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400c780 tx=0x7f1b0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b04018070 con 0x7f1b080cfef0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0400a640 con 0x7f1b080cfef0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b08150aa0 con 0x7f1b080cfef0 2024-03-06T07:27:15.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400aa70 con 0x7f1b080cfef0 2024-03-06T07:27:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.343+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b08151060 con 0x7f1b080cfef0 2024-03-06T07:27:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.344+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b080732b0 con 0x7f1b080cfef0 2024-03-06T07:27:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.345+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1b0400ac10 con 0x7f1b080cfef0 2024-03-06T07:27:15.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.345+0000 7f1afdffb640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1aec06b4d0 0x7f1aec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.345+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b04089840 con 0x7f1b080cfef0 2024-03-06T07:27:15.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.345+0000 7f1b0d1b1640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1aec06b4d0 0x7f1aec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.345+0000 7f1b0d1b1640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1aec06b4d0 0x7f1aec06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009e90 tx=0x7f1b00009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.349+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1b04057ff0 con 0x7f1b080cfef0 2024-03-06T07:27:15.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.520+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b08068ff0 con 0x7f1b080cfef0 2024-03-06T07:27:15.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.521+0000 7f1afdffb640 1 -- 172.21.15.72:0/2201840239 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b0405c090 con 0x7f1b080cfef0 2024-03-06T07:27:15.523 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1aec06b4d0 msgr2=0x7f1aec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1aec06b4d0 0x7f1aec06d990 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009e90 tx=0x7f1b00009190 comp rx=0 tx=0).stop 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 msgr2=0x7f1b081478a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 0x7f1b081478a0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400c780 tx=0x7f1b0400cc40 comp rx=0 tx=0).stop 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 shutdown_connections 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1aec06b4d0 0x7f1aec06d990 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b080cfef0 0x7f1b081478a0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b080cfa90 0x7f1b08147360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 --2- 172.21.15.72:0/2201840239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b080cf0d0 0x7f1b081522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 >> 172.21.15.72:0/2201840239 conn(0x7f1b0806b190 msgr2=0x7f1b080cc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 shutdown_connections 2024-03-06T07:27:15.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.523+0000 7f1b0f415640 1 -- 172.21.15.72:0/2201840239 wait complete. 2024-03-06T07:27:15.536 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:15.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.636+0000 7fda8a09a640 1 Processor -- start 2024-03-06T07:27:15.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.636+0000 7fda8a09a640 1 -- start start 2024-03-06T07:27:15.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda8a09a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda841586e0 0x7fda840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda8a09a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda840aef80 0x7fda840cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda8a09a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fda840cffd0 0x7fda8406da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda8a09a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda840d03f0 con 0x7fda840aef80 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda8a09a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda84158ac0 con 0x7fda840cffd0 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda8a09a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8415b2a0 con 0x7fda841586e0 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda840aef80 0x7fda840cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda840aef80 0x7fda840cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60702/0 (socket says 172.21.15.72:60702) 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 -- 172.21.15.72:0/2488005107 learned_addr learned my addr 172.21.15.72:0/2488005107 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda837fe640 1 --2- 172.21.15.72:0/2488005107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda841586e0 0x7fda840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda83fff640 1 --2- 172.21.15.72:0/2488005107 >> v2:172.21.15.146:3300/0 conn(0x7fda840cffd0 0x7fda8406da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 -- 172.21.15.72:0/2488005107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda841586e0 msgr2=0x7fda840aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 --2- 172.21.15.72:0/2488005107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda841586e0 0x7fda840aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 -- 172.21.15.72:0/2488005107 >> v2:172.21.15.146:3300/0 conn(0x7fda840cffd0 msgr2=0x7fda8406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 --2- 172.21.15.72:0/2488005107 >> v2:172.21.15.146:3300/0 conn(0x7fda840cffd0 0x7fda8406da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.637+0000 7fda82ffd640 1 -- 172.21.15.72:0/2488005107 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8415b540 con 0x7fda840aef80 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.638+0000 7fda82ffd640 1 --2- 172.21.15.72:0/2488005107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda840aef80 0x7fda840cfa90 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fda84072e00 tx=0x7fda7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3bb8a2eebefcabb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.638+0000 7fda827fc640 1 -- 172.21.15.72:0/2488005107 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78035020 con 0x7fda840aef80 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.638+0000 7fda827fc640 1 -- 172.21.15.72:0/2488005107 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda78031870 con 0x7fda840aef80 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.638+0000 7fda827fc640 1 -- 172.21.15.72:0/2488005107 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78031c40 con 0x7fda840aef80 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 -- 172.21.15.72:0/2488005107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda840aef80 msgr2=0x7fda840cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 --2- 172.21.15.72:0/2488005107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda840aef80 0x7fda840cfa90 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fda84072e00 tx=0x7fda7802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 -- 172.21.15.72:0/2488005107 shutdown_connections 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 --2- 172.21.15.72:0/2488005107 >> v2:172.21.15.146:3300/0 conn(0x7fda840cffd0 0x7fda8406da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 --2- 172.21.15.72:0/2488005107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda840aef80 0x7fda840cfa90 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 --2- 172.21.15.72:0/2488005107 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda841586e0 0x7fda840aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 -- 172.21.15.72:0/2488005107 >> 172.21.15.72:0/2488005107 conn(0x7fda8406b190 msgr2=0x7fda8406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 -- 172.21.15.72:0/2488005107 shutdown_connections 2024-03-06T07:27:15.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.639+0000 7fda8a09a640 1 -- 172.21.15.72:0/2488005107 wait complete. 2024-03-06T07:27:15.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.640+0000 7fda8a09a640 1 Processor -- start 2024-03-06T07:27:15.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.640+0000 7fda8a09a640 1 -- start start 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda8a09a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda840aef80 0x7fda84156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda8a09a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda840cffd0 0x7fda84145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda8a09a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 0x7fda84145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda8a09a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8415bde0 con 0x7fda841586e0 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda8a09a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8415bb00 con 0x7fda840aef80 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda8a09a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda84071350 con 0x7fda840cffd0 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 0x7fda84145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 0x7fda84145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60712/0 (socket says 172.21.15.72:60712) 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 -- 172.21.15.72:0/351206538 learned_addr learned my addr 172.21.15.72:0/351206538 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda82ffd640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda840cffd0 0x7fda84145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda837fe640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda840aef80 0x7fda84156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 -- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda840cffd0 msgr2=0x7fda84145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda840cffd0 0x7fda84145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 -- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda840aef80 msgr2=0x7fda84156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda840aef80 0x7fda84156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 -- 172.21.15.72:0/351206538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84148a60 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.641+0000 7fda83fff640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 0x7fda84145560 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00eed0 tx=0x7fda7c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.642+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c018020 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.642+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c014400 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.642+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda84145e60 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.642+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0147f0 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.642+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda84146420 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.643+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8415c250 con 0x7fda841586e0 2024-03-06T07:27:15.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.643+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fda7c005d10 con 0x7fda841586e0 2024-03-06T07:27:15.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.643+0000 7fda6bfff640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda7006b500 0x7fda7006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.644+0000 7fda837fe640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda7006b500 0x7fda7006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.644+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda7c08a780 con 0x7fda841586e0 2024-03-06T07:27:15.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.644+0000 7fda837fe640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda7006b500 0x7fda7006d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fda840acc50 tx=0x7fda74009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.648+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fda8415c250 con 0x7fda841586e0 2024-03-06T07:27:15.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.819+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda840732b0 con 0x7fda841586e0 2024-03-06T07:27:15.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.820+0000 7fda6bfff640 1 -- 172.21.15.72:0/351206538 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda7c022070 con 0x7fda841586e0 2024-03-06T07:27:15.822 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.821+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda7006b500 msgr2=0x7fda7006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.821+0000 7fda8a09a640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda7006b500 0x7fda7006d9c0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fda840acc50 tx=0x7fda74009190 comp rx=0 tx=0).stop 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.821+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 msgr2=0x7fda84145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.821+0000 7fda8a09a640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 0x7fda84145560 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00eed0 tx=0x7fda7c00b480 comp rx=0 tx=0).stop 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 shutdown_connections 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fda7006b500 0x7fda7006d9c0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda841586e0 0x7fda84145560 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fda840cffd0 0x7fda84145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 --2- 172.21.15.72:0/351206538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fda840aef80 0x7fda84156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 >> 172.21.15.72:0/351206538 conn(0x7fda8406b190 msgr2=0x7fda84159bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 shutdown_connections 2024-03-06T07:27:15.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.822+0000 7fda8a09a640 1 -- 172.21.15.72:0/351206538 wait complete. 2024-03-06T07:27:15.835 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:15.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.934+0000 7f19cb908640 1 Processor -- start 2024-03-06T07:27:15.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.935+0000 7f19cb908640 1 -- start start 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.935+0000 7f19cb908640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c415a220 0x7f19c40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.935+0000 7f19cb908640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19c406da00 0x7f19c4071730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19cb908640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c40b3320 con 0x7f19c40712d0 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19cb908640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c40cf610 con 0x7f19c406da00 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19cb908640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4071c70 con 0x7f19c415a220 2024-03-06T07:27:15.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c9ea5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19c406da00 0x7f19c4071730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c8ea3640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f19c40712d0 0x7f19c415de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45598/0 (socket says 172.21.15.72:45598) 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c8ea3640 1 -- 172.21.15.72:0/312852049 learned_addr learned my addr 172.21.15.72:0/312852049 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c96a4640 1 --2- 172.21.15.72:0/312852049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c415a220 0x7f19c40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c9ea5640 1 -- 172.21.15.72:0/312852049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c415a220 msgr2=0x7f19c40cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c9ea5640 1 --2- 172.21.15.72:0/312852049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c415a220 0x7f19c40cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c9ea5640 1 -- 172.21.15.72:0/312852049 >> v1:172.21.15.72:6789/0 conn(0x7f19c40712d0 legacy=0x7f19c415de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19c9ea5640 1 -- 172.21.15.72:0/312852049 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c40728d0 con 0x7f19c406da00 2024-03-06T07:27:15.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.936+0000 7f19b7fff640 1 -- 172.21.15.72:0/312852049 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112885570 0 0) 0x7f19c40b3320 con 0x7f19c40712d0 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19c9ea5640 1 --2- 172.21.15.72:0/312852049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c406da00 0x7f19c4071730 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009d90 tx=0x7f19bc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7679b91288e7429 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19b7fff640 1 -- 172.21.15.72:0/312852049 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc039070 con 0x7f19c406da00 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19b7fff640 1 -- 172.21.15.72:0/312852049 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc02cbc0 con 0x7f19c406da00 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19b7fff640 1 -- 172.21.15.72:0/312852049 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc034440 con 0x7f19c406da00 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19cb908640 1 -- 172.21.15.72:0/312852049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c406da00 msgr2=0x7f19c4071730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19cb908640 1 --2- 172.21.15.72:0/312852049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c406da00 0x7f19c4071730 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009d90 tx=0x7f19bc02be10 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19cb908640 1 -- 172.21.15.72:0/312852049 shutdown_connections 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19cb908640 1 --2- 172.21.15.72:0/312852049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c406da00 0x7f19c4071730 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19cb908640 1 --2- 172.21.15.72:0/312852049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c415a220 0x7f19c40cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.937+0000 7f19cb908640 1 -- 172.21.15.72:0/312852049 >> 172.21.15.72:0/312852049 conn(0x7f19c406b190 msgr2=0x7f19c406b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.938+0000 7f19cb908640 1 -- 172.21.15.72:0/312852049 shutdown_connections 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.938+0000 7f19cb908640 1 -- 172.21.15.72:0/312852049 wait complete. 2024-03-06T07:27:15.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 Processor -- start 2024-03-06T07:27:15.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 -- start start 2024-03-06T07:27:15.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c406da00 0x7f19c415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c40712d0 0x7f19c414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 0x7f19c4154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4072030 con 0x7f19c415a220 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4072bb0 con 0x7f19c406da00 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19cb908640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c4072530 con 0x7f19c40712d0 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19c9ea5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 0x7f19c4154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.939+0000 7f19c9ea5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 0x7f19c4154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60722/0 (socket says 172.21.15.72:60722) 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 -- 172.21.15.72:0/1515150244 learned_addr learned my addr 172.21.15.72:0/1515150244 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c96a4640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c406da00 0x7f19c415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c8ea3640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c40712d0 0x7f19c414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 -- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c40712d0 msgr2=0x7f19c414b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c40712d0 0x7f19c414b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 -- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c406da00 msgr2=0x7f19c415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c406da00 0x7f19c415e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 -- 172.21.15.72:0/1515150244 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c4153eb0 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19c9ea5640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 0x7f19c4154f50 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009d90 tx=0x7f19bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc039050 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc045070 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c4154190 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc009450 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.940+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c4154750 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.941+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c40732b0 con 0x7f19c415a220 2024-03-06T07:27:15.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.942+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f19bc03e850 con 0x7f19c415a220 2024-03-06T07:27:15.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.942+0000 7f19b5ffb640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f19a806b4d0 0x7f19a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:15.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.942+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19bc0ad160 con 0x7f19c415a220 2024-03-06T07:27:15.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.942+0000 7f19c96a4640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f19a806b4d0 0x7f19a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:15.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.943+0000 7f19c96a4640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f19a806b4d0 0x7f19a806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009aa0 tx=0x7f19b8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:15.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:15.947+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f19bc07ba20 con 0x7f19c415a220 2024-03-06T07:27:16.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.119+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19c4068ff0 con 0x7f19c415a220 2024-03-06T07:27:16.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.119+0000 7f19b5ffb640 1 -- 172.21.15.72:0/1515150244 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19bc07fac0 con 0x7f19c415a220 2024-03-06T07:27:16.121 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f19a806b4d0 msgr2=0x7f19a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f19a806b4d0 0x7f19a806d990 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f19b8009aa0 tx=0x7f19b8009190 comp rx=0 tx=0).stop 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 msgr2=0x7f19c4154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 0x7f19c4154f50 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009d90 tx=0x7f19bc004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 shutdown_connections 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f19a806b4d0 0x7f19a806d990 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19c415a220 0x7f19c4154f50 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c40712d0 0x7f19c414b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 --2- 172.21.15.72:0/1515150244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c406da00 0x7f19c415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 >> 172.21.15.72:0/1515150244 conn(0x7f19c406b190 msgr2=0x7f19c40b1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 shutdown_connections 2024-03-06T07:27:16.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.121+0000 7f19cb908640 1 -- 172.21.15.72:0/1515150244 wait complete. 2024-03-06T07:27:16.134 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.233+0000 7fd927e4e640 1 Processor -- start 2024-03-06T07:27:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.233+0000 7fd927e4e640 1 -- start start 2024-03-06T07:27:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd927e4e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd92015b380 0x7fd9200708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd927e4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd9200cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd927e4e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92006cd60 con 0x7fd920158880 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd927e4e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92006d030 con 0x7fd92015b380 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd927e4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92006d310 con 0x7fd920070e10 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9253e9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd92015b380 0x7fd9200708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd925bea640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd920158880 0x7fd92006c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57234/0 (socket says 172.21.15.72:57234) 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd925bea640 1 -- 172.21.15.72:0/3676369131 learned_addr learned my addr 172.21.15.72:0/3676369131 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9263eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd9200cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9253e9640 1 -- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 msgr2=0x7fd9200cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9253e9640 1 --2- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd9200cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9253e9640 1 -- 172.21.15.72:0/3676369131 >> v1:172.21.15.146:6789/0 conn(0x7fd920158880 legacy=0x7fd92006c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9253e9640 1 -- 172.21.15.72:0/3676369131 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92006d5f0 con 0x7fd92015b380 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd924be8640 1 -- 172.21.15.72:0/3676369131 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661838565 0 0) 0x7fd92006cd60 con 0x7fd920158880 2024-03-06T07:27:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.234+0000 7fd9263eb640 1 --2- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd9200cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:16.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.235+0000 7fd9253e9640 1 --2- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9200708d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd9100097a0 tx=0x7fd91002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6659eb6f19798c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.235+0000 7fd924be8640 1 -- 172.21.15.72:0/3676369131 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910035020 con 0x7fd92015b380 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.235+0000 7fd924be8640 1 -- 172.21.15.72:0/3676369131 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910031830 con 0x7fd92015b380 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.235+0000 7fd924be8640 1 -- 172.21.15.72:0/3676369131 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910031c00 con 0x7fd92015b380 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 -- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 msgr2=0x7fd9200708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 --2- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9200708d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd9100097a0 tx=0x7fd91002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 -- 172.21.15.72:0/3676369131 shutdown_connections 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 --2- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd9200cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 --2- 172.21.15.72:0/3676369131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9200708d0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 -- 172.21.15.72:0/3676369131 >> 172.21.15.72:0/3676369131 conn(0x7fd9200b1c80 msgr2=0x7fd9200cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 -- 172.21.15.72:0/3676369131 shutdown_connections 2024-03-06T07:27:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.236+0000 7fd927e4e640 1 -- 172.21.15.72:0/3676369131 wait complete. 2024-03-06T07:27:16.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.237+0000 7fd927e4e640 1 Processor -- start 2024-03-06T07:27:16.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.237+0000 7fd927e4e640 1 -- start start 2024-03-06T07:27:16.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd927e4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd920156f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd927e4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 0x7fd920158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd927e4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9201e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd927e4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cdd00 con 0x7fd920158880 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd927e4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92006dbb0 con 0x7fd92015b380 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd927e4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9200cde80 con 0x7fd920070e10 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 0x7fd920158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 0x7fd920158fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60724/0 (socket says 172.21.15.72:60724) 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 -- 172.21.15.72:0/1009990245 learned_addr learned my addr 172.21.15.72:0/1009990245 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9263eb640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9201e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd925bea640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd920156f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 -- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 msgr2=0x7fd920156f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd920156f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 -- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 msgr2=0x7fd9201e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9201e4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 -- 172.21.15.72:0/1009990245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92015a420 con 0x7fd920158880 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9253e9640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 0x7fd920158fb0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fd9100097a0 tx=0x7fd9100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.238+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910035040 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.239+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910004230 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.239+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92015a700 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.239+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91003a640 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.239+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9200ada00 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.240+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92004f330 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.240+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd910038020 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.240+0000 7fd9167fc640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.240+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9100abe20 con 0x7fd920158880 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.240+0000 7fd925bea640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.241+0000 7fd925bea640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd91c000f60 tx=0x7fd91c007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.245+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd91007aca0 con 0x7fd920158880 2024-03-06T07:27:16.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.430+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd92015c590 con 0x7fd920158880 2024-03-06T07:27:16.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.430+0000 7fd9167fc640 1 -- 172.21.15.72:0/1009990245 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd91007a5a0 con 0x7fd920158880 2024-03-06T07:27:16.432 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:27:16.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 msgr2=0x7fd90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd91c000f60 tx=0x7fd91c007d50 comp rx=0 tx=0).stop 2024-03-06T07:27:16.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 msgr2=0x7fd920158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 0x7fd920158fb0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fd9100097a0 tx=0x7fd9100046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 shutdown_connections 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd90406b4d0 0x7fd90406d990 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd92015b380 0x7fd9201e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd920158880 0x7fd920158fb0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 --2- 172.21.15.72:0/1009990245 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd920070e10 0x7fd920156f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 >> 172.21.15.72:0/1009990245 conn(0x7fd9200b1c80 msgr2=0x7fd9200b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 shutdown_connections 2024-03-06T07:27:16.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.432+0000 7fd927e4e640 1 -- 172.21.15.72:0/1009990245 wait complete. 2024-03-06T07:27:16.445 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:27:16.445 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:27:16.460 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:27:16.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.597+0000 7f3466c9a640 1 Processor -- start 2024-03-06T07:27:16.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.597+0000 7f3466c9a640 1 -- start start 2024-03-06T07:27:16.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3466c9a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34600b4f70 0x7f34600b5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3466c9a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f346006c6d0 0x7f346015ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3466c9a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600b5890 con 0x7f3460066d90 2024-03-06T07:27:16.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3466c9a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346015e9f0 con 0x7f346006c6d0 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3466c9a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34600cda10 con 0x7f34600b4f70 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f346006c6d0 0x7f346015ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f346006c6d0 0x7f346015ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60740/0 (socket says 172.21.15.72:60740) 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 -- 172.21.15.72:0/539249261 learned_addr learned my addr 172.21.15.72:0/539249261 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3464a36640 1 --2- 172.21.15.72:0/539249261 >> v2:172.21.15.146:3300/0 conn(0x7f34600b4f70 0x7f34600b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 -- 172.21.15.72:0/539249261 >> v2:172.21.15.146:3300/0 conn(0x7f34600b4f70 msgr2=0x7f34600b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 --2- 172.21.15.72:0/539249261 >> v2:172.21.15.146:3300/0 conn(0x7f34600b4f70 0x7f34600b5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 -- 172.21.15.72:0/539249261 >> v1:172.21.15.146:6789/0 conn(0x7f3460066d90 legacy=0x7f34600ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f345ffff640 1 -- 172.21.15.72:0/539249261 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600cd450 con 0x7f346006c6d0 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.598+0000 7f3464a36640 1 --2- 172.21.15.72:0/539249261 >> v2:172.21.15.146:3300/0 conn(0x7f34600b4f70 0x7f34600b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.599+0000 7f345ffff640 1 --2- 172.21.15.72:0/539249261 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f346006c6d0 0x7f346015ee80 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f346006cac0 tx=0x7f345402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a57ebcf7b17b1ee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.599+0000 7f345f7fe640 1 -- 172.21.15.72:0/539249261 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454034070 con 0x7f346006c6d0 2024-03-06T07:27:16.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.599+0000 7f345f7fe640 1 -- 172.21.15.72:0/539249261 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454030860 con 0x7f346006c6d0 2024-03-06T07:27:16.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.599+0000 7f345f7fe640 1 -- 172.21.15.72:0/539249261 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454030c30 con 0x7f346006c6d0 2024-03-06T07:27:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 -- 172.21.15.72:0/539249261 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f346006c6d0 msgr2=0x7f346015ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 --2- 172.21.15.72:0/539249261 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f346006c6d0 0x7f346015ee80 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f346006cac0 tx=0x7f345402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 -- 172.21.15.72:0/539249261 shutdown_connections 2024-03-06T07:27:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 --2- 172.21.15.72:0/539249261 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f346006c6d0 0x7f346015ee80 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 --2- 172.21.15.72:0/539249261 >> v2:172.21.15.146:3300/0 conn(0x7f34600b4f70 0x7f34600b5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 -- 172.21.15.72:0/539249261 >> 172.21.15.72:0/539249261 conn(0x7f34600724d0 msgr2=0x7f34600728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 -- 172.21.15.72:0/539249261 shutdown_connections 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.600+0000 7f3466c9a640 1 -- 172.21.15.72:0/539249261 wait complete. 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.601+0000 7f3466c9a640 1 Processor -- start 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.601+0000 7f3466c9a640 1 -- start start 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3466c9a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3460066d90 0x7f34600cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3466c9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 0x7f34600cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3466c9a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3460153df0 0x7f34600ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3466c9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346015f3c0 con 0x7f34600b4f70 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3466c9a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346015e680 con 0x7f3460066d90 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3466c9a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3460161ad0 con 0x7f3460153df0 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 0x7f34600cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 0x7f34600cbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60752/0 (socket says 172.21.15.72:60752) 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 -- 172.21.15.72:0/1131201116 learned_addr learned my addr 172.21.15.72:0/1131201116 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:16.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3464a36640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3460066d90 0x7f34600cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f3465237640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3460153df0 0x7f34600ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 -- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3460153df0 msgr2=0x7f34600ad5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3460153df0 0x7f34600ad5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 -- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3460066d90 msgr2=0x7f34600cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3460066d90 0x7f34600cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.602+0000 7f345ffff640 1 -- 172.21.15.72:0/1131201116 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600aebb0 con 0x7f34600b4f70 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.603+0000 7f345ffff640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 0x7f34600cbe10 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f345402b9e0 tx=0x7f3454004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.603+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454048070 con 0x7f34600b4f70 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.603+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454030660 con 0x7f34600b4f70 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.603+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454034040 con 0x7f34600b4f70 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.603+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34600ae310 con 0x7f34600b4f70 2024-03-06T07:27:16.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.603+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34600ae8d0 con 0x7f34600b4f70 2024-03-06T07:27:16.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.604+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346004f330 con 0x7f34600b4f70 2024-03-06T07:27:16.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.604+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3454037070 con 0x7f34600b4f70 2024-03-06T07:27:16.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.605+0000 7f345d7fa640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f344c06b4d0 0x7f344c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.605+0000 7f3464a36640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f344c06b4d0 0x7f344c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.605+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34540ac090 con 0x7f34600b4f70 2024-03-06T07:27:16.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.605+0000 7f3464a36640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f344c06b4d0 0x7f344c06d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3450006fd0 tx=0x7f3450008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.609+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f345407a9e0 con 0x7f34600b4f70 2024-03-06T07:27:16.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.781+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34600b1860 con 0x7f34600b4f70 2024-03-06T07:27:16.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.782+0000 7f345d7fa640 1 -- 172.21.15.72:0/1131201116 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f345407ea80 con 0x7f34600b4f70 2024-03-06T07:27:16.784 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:16.784 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":110,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:04.164794+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:27:16.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f344c06b4d0 msgr2=0x7f344c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f344c06b4d0 0x7f344c06d990 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3450006fd0 tx=0x7f3450008040 comp rx=0 tx=0).stop 2024-03-06T07:27:16.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 msgr2=0x7f34600cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 0x7f34600cbe10 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f345402b9e0 tx=0x7f3454004730 comp rx=0 tx=0).stop 2024-03-06T07:27:16.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 shutdown_connections 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f344c06b4d0 0x7f344c06d990 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3460153df0 0x7f34600ad5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34600b4f70 0x7f34600cbe10 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 --2- 172.21.15.72:0/1131201116 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3460066d90 0x7f34600cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 >> 172.21.15.72:0/1131201116 conn(0x7f34600724d0 msgr2=0x7f346006d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 shutdown_connections 2024-03-06T07:27:16.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.784+0000 7f3466c9a640 1 -- 172.21.15.72:0/1131201116 wait complete. 2024-03-06T07:27:16.797 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:00.730111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:01.483409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:27:16.797 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:27:16.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.898+0000 7fb64a1a0640 1 Processor -- start 2024-03-06T07:27:16.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.898+0000 7fb64a1a0640 1 -- start start 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64a1a0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb6440b3fe0 0x7fb6440cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64a1a0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb64415f7e0 0x7fb64415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64a1a0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb644160130 con 0x7fb64415bd40 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64a1a0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6440ccdb0 con 0x7fb6440b3fe0 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64a1a0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb64415c120 con 0x7fb64415f7e0 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb6440b3fe0 0x7fb6440cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb6440b3fe0 0x7fb6440cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60768/0 (socket says 172.21.15.72:60768) 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 -- 172.21.15.72:0/3073179504 learned_addr learned my addr 172.21.15.72:0/3073179504 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64999f640 1 --2- 172.21.15.72:0/3073179504 >> v2:172.21.15.146:3300/0 conn(0x7fb64415f7e0 0x7fb64415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 -- 172.21.15.72:0/3073179504 >> v2:172.21.15.146:3300/0 conn(0x7fb64415f7e0 msgr2=0x7fb64415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 --2- 172.21.15.72:0/3073179504 >> v2:172.21.15.146:3300/0 conn(0x7fb64415f7e0 0x7fb64415fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 -- 172.21.15.72:0/3073179504 >> v1:172.21.15.146:6789/0 conn(0x7fb64415bd40 legacy=0x7fb6440cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.899+0000 7fb64899d640 1 -- 172.21.15.72:0/3073179504 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440d06c0 con 0x7fb6440b3fe0 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.900+0000 7fb64899d640 1 --2- 172.21.15.72:0/3073179504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440cd240 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb6440b7830 tx=0x7fb63402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=575500153d9e66e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.900+0000 7fb63bfff640 1 -- 172.21.15.72:0/3073179504 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634034070 con 0x7fb6440b3fe0 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.900+0000 7fb63bfff640 1 -- 172.21.15.72:0/3073179504 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb634030860 con 0x7fb6440b3fe0 2024-03-06T07:27:16.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.900+0000 7fb63bfff640 1 -- 172.21.15.72:0/3073179504 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb634030c30 con 0x7fb6440b3fe0 2024-03-06T07:27:16.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.900+0000 7fb64a1a0640 1 -- 172.21.15.72:0/3073179504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 msgr2=0x7fb6440cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.900+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/3073179504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440cd240 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb6440b7830 tx=0x7fb63402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.901+0000 7fb64a1a0640 1 -- 172.21.15.72:0/3073179504 shutdown_connections 2024-03-06T07:27:16.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.901+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/3073179504 >> v2:172.21.15.146:3300/0 conn(0x7fb64415f7e0 0x7fb64415fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.901+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/3073179504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440cd240 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.901+0000 7fb64a1a0640 1 -- 172.21.15.72:0/3073179504 >> 172.21.15.72:0/3073179504 conn(0x7fb6440b3450 msgr2=0x7fb6440b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.901+0000 7fb64a1a0640 1 -- 172.21.15.72:0/3073179504 shutdown_connections 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.901+0000 7fb64a1a0640 1 -- 172.21.15.72:0/3073179504 wait complete. 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 Processor -- start 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 -- start start 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb64415bd40 0x7fb6440caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb64415f7e0 0x7fb6440cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cd780 con 0x7fb6440b3fe0 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cca40 con 0x7fb64415bd40 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.902+0000 7fb64a1a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6440cc3a0 con 0x7fb64415f7e0 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60772/0 (socket says 172.21.15.72:60772) 2024-03-06T07:27:16.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 -- 172.21.15.72:0/2134287691 learned_addr learned my addr 172.21.15.72:0/2134287691 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64899d640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb64415bd40 0x7fb6440caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64999f640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb64415f7e0 0x7fb6440cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 -- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb64415f7e0 msgr2=0x7fb6440cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb64415f7e0 0x7fb6440cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 -- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb64415bd40 msgr2=0x7fb6440caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb64415bd40 0x7fb6440caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 -- 172.21.15.72:0/2134287691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6440ca430 con 0x7fb6440b3fe0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64919e640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440c9ef0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c3c0 tx=0x7fb64000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6400052d0 con 0x7fb6440b3fe0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb640005490 con 0x7fb6440b3fe0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6441f78a0 con 0x7fb6440b3fe0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb640007260 con 0x7fb6440b3fe0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.903+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6441f7e60 con 0x7fb6440b3fe0 2024-03-06T07:27:16.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.904+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb644068f10 con 0x7fb6440b3fe0 2024-03-06T07:27:16.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.905+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb640007400 con 0x7fb6440b3fe0 2024-03-06T07:27:16.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.905+0000 7fb639ffb640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb62806b4d0 0x7fb62806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:16.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.905+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb640089630 con 0x7fb6440b3fe0 2024-03-06T07:27:16.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.905+0000 7fb64899d640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb62806b4d0 0x7fb62806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:16.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.905+0000 7fb64899d640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb62806b4d0 0x7fb62806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb634036000 tx=0x7fb63403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:16.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:16.909+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb644160130 con 0x7fb6440b3fe0 2024-03-06T07:27:17.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.081+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6440b7880 con 0x7fb6440b3fe0 2024-03-06T07:27:17.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.081+0000 7fb639ffb640 1 -- 172.21.15.72:0/2134287691 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb64005ce20 con 0x7fb6440b3fe0 2024-03-06T07:27:17.083 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:27:17.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb62806b4d0 msgr2=0x7fb62806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb62806b4d0 0x7fb62806d990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fb634036000 tx=0x7fb63403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 msgr2=0x7fb6440c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440c9ef0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb64000c3c0 tx=0x7fb64000c880 comp rx=0 tx=0).stop 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 shutdown_connections 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb62806b4d0 0x7fb62806d990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb64415f7e0 0x7fb6440cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb64415bd40 0x7fb6440caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 --2- 172.21.15.72:0/2134287691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb6440b3fe0 0x7fb6440c9ef0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 >> 172.21.15.72:0/2134287691 conn(0x7fb6440b3450 msgr2=0x7fb6440cf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 shutdown_connections 2024-03-06T07:27:17.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.083+0000 7fb64a1a0640 1 -- 172.21.15.72:0/2134287691 wait complete. 2024-03-06T07:27:17.096 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:27:17.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.197+0000 7f16a9cf4640 1 Processor -- start 2024-03-06T07:27:17.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.197+0000 7f16a9cf4640 1 -- start start 2024-03-06T07:27:17.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a9cf4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16a415bdd0 0x7f16a40b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a9cf4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f16a40ce300 0x7f16a40ce790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a9cf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a40cecd0 0x7f16a40cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a9cf4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a40cf4f0 con 0x7f16a40ce300 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a9cf4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a40d0850 con 0x7f16a415bdd0 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a9cf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40d0240 con 0x7f16a40cecd0 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f16a40ce300 0x7f16a40ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f16a40ce300 0x7f16a40ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60780/0 (socket says 172.21.15.72:60780) 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 -- 172.21.15.72:0/1333500719 learned_addr learned my addr 172.21.15.72:0/1333500719 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a37fe640 1 --2- 172.21.15.72:0/1333500719 >> v2:172.21.15.146:3300/0 conn(0x7f16a415bdd0 0x7f16a40b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a3fff640 1 --2- 172.21.15.72:0/1333500719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a40cecd0 0x7f16a40cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 -- 172.21.15.72:0/1333500719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a40cecd0 msgr2=0x7f16a40cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 --2- 172.21.15.72:0/1333500719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a40cecd0 0x7f16a40cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 -- 172.21.15.72:0/1333500719 >> v2:172.21.15.146:3300/0 conn(0x7f16a415bdd0 msgr2=0x7f16a40b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 --2- 172.21.15.72:0/1333500719 >> v2:172.21.15.146:3300/0 conn(0x7f16a415bdd0 0x7f16a40b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.198+0000 7f16a2ffd640 1 -- 172.21.15.72:0/1333500719 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a40cf210 con 0x7f16a40ce300 2024-03-06T07:27:17.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.199+0000 7f16a2ffd640 1 --2- 172.21.15.72:0/1333500719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a40ce790 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f16a4071cc0 tx=0x7f169802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88a345ef0c38120 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.199+0000 7f16a27fc640 1 -- 172.21.15.72:0/1333500719 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698035020 con 0x7f16a40ce300 2024-03-06T07:27:17.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.199+0000 7f16a27fc640 1 -- 172.21.15.72:0/1333500719 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1698031870 con 0x7f16a40ce300 2024-03-06T07:27:17.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.199+0000 7f16a27fc640 1 -- 172.21.15.72:0/1333500719 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698031c40 con 0x7f16a40ce300 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1333500719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 msgr2=0x7f16a40ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1333500719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a40ce790 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f16a4071cc0 tx=0x7f169802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1333500719 shutdown_connections 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1333500719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a40cecd0 0x7f16a40cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1333500719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a40ce790 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1333500719 >> v2:172.21.15.146:3300/0 conn(0x7f16a415bdd0 0x7f16a40b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1333500719 >> 172.21.15.72:0/1333500719 conn(0x7f16a40b3690 msgr2=0x7f16a40b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1333500719 shutdown_connections 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.200+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1333500719 wait complete. 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.201+0000 7f16a9cf4640 1 Processor -- start 2024-03-06T07:27:17.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.201+0000 7f16a9cf4640 1 -- start start 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a9cf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a9cf4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a40cecd0 0x7f16a4159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a9cf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a415bdd0 0x7f16a415d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a9cf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40b10c0 con 0x7f16a40ce300 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a9cf4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a415c1b0 con 0x7f16a40cecd0 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a9cf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a40b1240 con 0x7f16a415bdd0 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a41598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60782/0 (socket says 172.21.15.72:60782) 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 -- 172.21.15.72:0/1442460792 learned_addr learned my addr 172.21.15.72:0/1442460792 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a2ffd640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a40cecd0 0x7f16a4159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a3fff640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a415bdd0 0x7f16a415d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 -- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a415bdd0 msgr2=0x7f16a415d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a415bdd0 0x7f16a415d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 -- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a40cecd0 msgr2=0x7f16a4159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a40cecd0 0x7f16a4159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 -- 172.21.15.72:0/1442460792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a40b48b0 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a3fff640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a415bdd0 0x7f16a415d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.202+0000 7f16a37fe640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a41598d0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f169400e9b0 tx=0x7f169400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.203+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694017070 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.203+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169400c9a0 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.203+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169400cd90 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.203+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a40b4b90 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.203+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a40b5150 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.204+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a409fcf0 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.204+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f169401a070 con 0x7f16a40ce300 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.204+0000 7f168bfff640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f169006b500 0x7f169006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.204+0000 7f16a2ffd640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f169006b500 0x7f169006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.204+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f169408a3f0 con 0x7f16a40ce300 2024-03-06T07:27:17.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.205+0000 7f16a2ffd640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f169006b500 0x7f169006d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1698036000 tx=0x7f169803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.209+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1694058cb0 con 0x7f16a40ce300 2024-03-06T07:27:17.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.383+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f16a4068ff0 con 0x7f16a40ce300 2024-03-06T07:27:17.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.383+0000 7f168bfff640 1 -- 172.21.15.72:0/1442460792 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) v1 ==== 104+0+11 (secure 0 0 0) 0x7f169405cd50 con 0x7f16a40ce300 2024-03-06T07:27:17.385 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:27:17.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f169006b500 msgr2=0x7f169006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f169006b500 0x7f169006d9c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f1698036000 tx=0x7f169803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 msgr2=0x7f16a41598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a41598d0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f169400e9b0 tx=0x7f169400ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 shutdown_connections 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f169006b500 0x7f169006d9c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a415bdd0 0x7f16a415d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a40cecd0 0x7f16a4159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 --2- 172.21.15.72:0/1442460792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f16a40ce300 0x7f16a41598d0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 >> 172.21.15.72:0/1442460792 conn(0x7f16a40b3690 msgr2=0x7f16a40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 shutdown_connections 2024-03-06T07:27:17.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.385+0000 7f16a9cf4640 1 -- 172.21.15.72:0/1442460792 wait complete. 2024-03-06T07:27:17.398 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:27:17.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.499+0000 7fc361508640 1 Processor -- start 2024-03-06T07:27:17.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.499+0000 7fc361508640 1 -- start start 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc361508640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc35c0d4860 0x7fc35c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc361508640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c1618b0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc361508640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c0cf400 con 0x7fc35c1614d0 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc361508640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35c0d4cf0 con 0x7fc35c0d4860 2024-03-06T07:27:17.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc35a7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc35c0d4860 0x7fc35c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc35b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc35c0b0e30 0x7fc35c0b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45606/0 (socket says 172.21.15.72:45606) 2024-03-06T07:27:17.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc35b7fe640 1 -- 172.21.15.72:0/3689037884 learned_addr learned my addr 172.21.15.72:0/3689037884 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:17.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc35a7fc640 1 -- 172.21.15.72:0/3689037884 >> v1:172.21.15.146:6789/0 conn(0x7fc35c1614d0 legacy=0x7fc35c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc35a7fc640 1 -- 172.21.15.72:0/3689037884 >> v1:172.21.15.72:6789/0 conn(0x7fc35c0b0e30 legacy=0x7fc35c0b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc359ffb640 1 -- 172.21.15.72:0/3689037884 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305991447 0 0) 0x7fc35c1618b0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.500+0000 7fc35a7fc640 1 -- 172.21.15.72:0/3689037884 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0cf6a0 con 0x7fc35c0d4860 2024-03-06T07:27:17.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.501+0000 7fc35a7fc640 1 --2- 172.21.15.72:0/3689037884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0d4860 0x7fc35c0d2020 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fc3440097a0 tx=0x7fc34402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af038f29e94ca0da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.501+0000 7fc359ffb640 1 -- 172.21.15.72:0/3689037884 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344034070 con 0x7fc35c0d4860 2024-03-06T07:27:17.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.501+0000 7fc359ffb640 1 -- 172.21.15.72:0/3689037884 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344030610 con 0x7fc35c0d4860 2024-03-06T07:27:17.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.501+0000 7fc359ffb640 1 -- 172.21.15.72:0/3689037884 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3440309e0 con 0x7fc35c0d4860 2024-03-06T07:27:17.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 -- 172.21.15.72:0/3689037884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0d4860 msgr2=0x7fc35c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 --2- 172.21.15.72:0/3689037884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0d4860 0x7fc35c0d2020 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fc3440097a0 tx=0x7fc34402b960 comp rx=0 tx=0).stop 2024-03-06T07:27:17.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 -- 172.21.15.72:0/3689037884 shutdown_connections 2024-03-06T07:27:17.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 --2- 172.21.15.72:0/3689037884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0d4860 0x7fc35c0d2020 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 -- 172.21.15.72:0/3689037884 >> 172.21.15.72:0/3689037884 conn(0x7fc35c0d3d40 msgr2=0x7fc35c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 -- 172.21.15.72:0/3689037884 shutdown_connections 2024-03-06T07:27:17.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.502+0000 7fc361508640 1 -- 172.21.15.72:0/3689037884 wait complete. 2024-03-06T07:27:17.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.503+0000 7fc361508640 1 Processor -- start 2024-03-06T07:27:17.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.503+0000 7fc361508640 1 -- start start 2024-03-06T07:27:17.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 0x7fc35c0cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc35c0d4860 0x7fc35c156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc35c1614d0 0x7fc35c1f7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0b0910 con 0x7fc35c0b0e30 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0cfc60 con 0x7fc35c1614d0 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0d2560 con 0x7fc35c0d4860 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 0x7fc35c0cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 0x7fc35c0cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60802/0 (socket says 172.21.15.72:60802) 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 -- 172.21.15.72:0/1423110883 learned_addr learned my addr 172.21.15.72:0/1423110883 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35b7fe640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc35c1614d0 0x7fc35c1f7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35a7fc640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc35c0d4860 0x7fc35c156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 -- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc35c0d4860 msgr2=0x7fc35c156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc35c0d4860 0x7fc35c156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 -- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc35c1614d0 msgr2=0x7fc35c1f7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc35c1614d0 0x7fc35c1f7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 -- 172.21.15.72:0/1423110883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0ca020 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35b7fe640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc35c1614d0 0x7fc35c1f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc35affd640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 0x7fc35c0cb7d0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c760 tx=0x7fc34c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c003b70 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c003d30 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c0cbee0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c00aa90 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.504+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c0cc4a0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.505+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c068ff0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.506+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc34c00ac30 con 0x7fc35c0b0e30 2024-03-06T07:27:17.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.506+0000 7fc3537fe640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.506+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc34c08a090 con 0x7fc35c0b0e30 2024-03-06T07:27:17.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.506+0000 7fc35a7fc640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.506+0000 7fc35a7fc640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc33c06b4d0 0x7fc33c06d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc344036000 tx=0x7fc34403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.510+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc34c058fa0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.680+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc35c0d0af0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.681+0000 7fc3537fe640 1 -- 172.21.15.72:0/1423110883 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc34c0588a0 con 0x7fc35c0b0e30 2024-03-06T07:27:17.683 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:27:17.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc33c06b4d0 msgr2=0x7fc33c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc33c06b4d0 0x7fc33c06d990 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc344036000 tx=0x7fc34403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 msgr2=0x7fc35c0cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 0x7fc35c0cb7d0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c760 tx=0x7fc34c00cc20 comp rx=0 tx=0).stop 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 shutdown_connections 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc33c06b4d0 0x7fc33c06d990 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc35c1614d0 0x7fc35c1f7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc35c0d4860 0x7fc35c156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 --2- 172.21.15.72:0/1423110883 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc35c0b0e30 0x7fc35c0cb7d0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 >> 172.21.15.72:0/1423110883 conn(0x7fc35c0d3d40 msgr2=0x7fc35c0d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 shutdown_connections 2024-03-06T07:27:17.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.683+0000 7fc361508640 1 -- 172.21.15.72:0/1423110883 wait complete. 2024-03-06T07:27:17.696 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.796+0000 7fcba5650640 1 Processor -- start 2024-03-06T07:27:17.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.796+0000 7fcba5650640 1 -- start start 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcba5650640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcba00b4460 0x7fcba00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcba5650640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0162970 0x7fcba00b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcba5650640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba00d22c0 con 0x7fcba0161cf0 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcba5650640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba0161b00 con 0x7fcba00b4460 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcba5650640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00cecc0 con 0x7fcba0162970 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcba00b4460 0x7fcba00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcba00b4460 0x7fcba00d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60806/0 (socket says 172.21.15.72:60806) 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 -- 172.21.15.72:0/2301136441 learned_addr learned my addr 172.21.15.72:0/2301136441 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb967fc640 1 --2- 172.21.15.72:0/2301136441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0162970 0x7fcba00b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 -- 172.21.15.72:0/2301136441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0162970 msgr2=0x7fcba00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 --2- 172.21.15.72:0/2301136441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0162970 0x7fcba00b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 -- 172.21.15.72:0/2301136441 >> v1:172.21.15.146:6789/0 conn(0x7fcba0161cf0 legacy=0x7fcba00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.797+0000 7fcb9effd640 1 -- 172.21.15.72:0/2301136441 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba00d25a0 con 0x7fcba00b4460 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.798+0000 7fcb9effd640 1 --2- 172.21.15.72:0/2301136441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba00b4460 0x7fcba00d4800 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009870 tx=0x7fcb9002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8da6dcccd53b171a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.798+0000 7fcb9e7fc640 1 -- 172.21.15.72:0/2301136441 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90034070 con 0x7fcba00b4460 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.798+0000 7fcb9e7fc640 1 -- 172.21.15.72:0/2301136441 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb90030840 con 0x7fcba00b4460 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.798+0000 7fcb9e7fc640 1 -- 172.21.15.72:0/2301136441 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90030c10 con 0x7fcba00b4460 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.798+0000 7fcba5650640 1 -- 172.21.15.72:0/2301136441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba00b4460 msgr2=0x7fcba00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.798+0000 7fcba5650640 1 --2- 172.21.15.72:0/2301136441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba00b4460 0x7fcba00d4800 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009870 tx=0x7fcb9002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.799+0000 7fcba5650640 1 -- 172.21.15.72:0/2301136441 shutdown_connections 2024-03-06T07:27:17.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.799+0000 7fcba5650640 1 --2- 172.21.15.72:0/2301136441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba0162970 0x7fcba00b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.799+0000 7fcba5650640 1 --2- 172.21.15.72:0/2301136441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba00b4460 0x7fcba00d4800 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.799+0000 7fcba5650640 1 -- 172.21.15.72:0/2301136441 >> 172.21.15.72:0/2301136441 conn(0x7fcba00d3ce0 msgr2=0x7fcba00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.799+0000 7fcba5650640 1 -- 172.21.15.72:0/2301136441 shutdown_connections 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.799+0000 7fcba5650640 1 -- 172.21.15.72:0/2301136441 wait complete. 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.800+0000 7fcba5650640 1 Processor -- start 2024-03-06T07:27:17.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.800+0000 7fcba5650640 1 -- start start 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba00b4460 0x7fcba015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 0x7fcba01e8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba0162970 0x7fcba01e8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00ce950 con 0x7fcba0161cf0 2024-03-06T07:27:17.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00d2b60 con 0x7fcba0162970 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba00b3ac0 con 0x7fcba00b4460 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 0x7fcba01e8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 0x7fcba01e8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60816/0 (socket says 172.21.15.72:60816) 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 -- 172.21.15.72:0/1298966741 learned_addr learned my addr 172.21.15.72:0/1298966741 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb9f7fe640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba0162970 0x7fcba01e8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb9effd640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba00b4460 0x7fcba015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 -- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba00b4460 msgr2=0x7fcba015ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba00b4460 0x7fcba015ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 -- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba0162970 msgr2=0x7fcba01e8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba0162970 0x7fcba01e8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 -- 172.21.15.72:0/1298966741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba01e8f80 con 0x7fcba0161cf0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb967fc640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 0x7fcba01e8500 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e9b0 tx=0x7fcb8800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb88017070 con 0x7fcba0161cf0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8800c9a0 con 0x7fcba0161cf0 2024-03-06T07:27:17.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba01e9230 con 0x7fcba0161cf0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8800cd70 con 0x7fcba0161cf0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.801+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba01e97f0 con 0x7fcba0161cf0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.802+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba004f330 con 0x7fcba0161cf0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.803+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb8801a070 con 0x7fcba0161cf0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.803+0000 7fcb97fff640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb8006b4d0 0x7fcb8006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.803+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb8808a130 con 0x7fcba0161cf0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.803+0000 7fcb9effd640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb8006b4d0 0x7fcb8006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:17.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.803+0000 7fcb9effd640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb8006b4d0 0x7fcb8006d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcb90036000 tx=0x7fcb9003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:17.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.807+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb88058960 con 0x7fcba0161cf0 2024-03-06T07:27:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.990+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcba01e9ea0 con 0x7fcba0161cf0 2024-03-06T07:27:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.991+0000 7fcb97fff640 1 -- 172.21.15.72:0/1298966741 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v110) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcb8805ca00 con 0x7fcba0161cf0 2024-03-06T07:27:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb8006b4d0 msgr2=0x7fcb8006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb8006b4d0 0x7fcb8006d990 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fcb90036000 tx=0x7fcb9003b040 comp rx=0 tx=0).stop 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 msgr2=0x7fcba01e8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 0x7fcba01e8500 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcb8800e9b0 tx=0x7fcb8800ee70 comp rx=0 tx=0).stop 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 shutdown_connections 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb8006b4d0 0x7fcb8006d990 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcba0162970 0x7fcba01e8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba0161cf0 0x7fcba01e8500 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 --2- 172.21.15.72:0/1298966741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcba00b4460 0x7fcba015ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 >> 172.21.15.72:0/1298966741 conn(0x7fcba00d3ce0 msgr2=0x7fcba00d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 shutdown_connections 2024-03-06T07:27:17.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:17.993+0000 7fcba5650640 1 -- 172.21.15.72:0/1298966741 wait complete. 2024-03-06T07:27:18.006 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:27:18.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:18.006 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:18.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[72507]: fuse finished with error 0 and tester_r 0 2024-03-06T07:27:18.034 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:21.180 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:21.183 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:24.136 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:27:24.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:24.136 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:27:24.155 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:24.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.302+0000 7fdb28871640 1 Processor -- start 2024-03-06T07:27:24.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.302+0000 7fdb28871640 1 -- start start 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.303+0000 7fdb28871640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdb2415dfe0 0x7fdb240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.303+0000 7fdb28871640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240b3430 con 0x7fdb240d29c0 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.303+0000 7fdb28871640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240b3710 con 0x7fdb240cf830 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.303+0000 7fdb28871640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb240b31e0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb22d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdb240d29c0 0x7fdb240b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38788/0 (socket says 172.21.15.72:38788) 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb22d9d640 1 -- 172.21.15.72:0/370072907 learned_addr learned my addr 172.21.15.72:0/370072907 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb2259c640 1 --2- 172.21.15.72:0/370072907 >> v2:172.21.15.146:3300/0 conn(0x7fdb2415dfe0 0x7fdb240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb2159a640 1 -- 172.21.15.72:0/370072907 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899896592 0 0) 0x7fdb240b3430 con 0x7fdb240d29c0 2024-03-06T07:27:24.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb2159a640 1 -- 172.21.15.72:0/370072907 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb0c003520 con 0x7fdb240d29c0 2024-03-06T07:27:24.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb2259c640 1 -- 172.21.15.72:0/370072907 >> v1:172.21.15.146:6789/0 conn(0x7fdb240cf830 legacy=0x7fdb240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb2259c640 1 -- 172.21.15.72:0/370072907 >> v1:172.21.15.72:6789/0 conn(0x7fdb240d29c0 legacy=0x7fdb240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.304+0000 7fdb2259c640 1 -- 172.21.15.72:0/370072907 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240b39f0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.305+0000 7fdb2259c640 1 --2- 172.21.15.72:0/370072907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cf2f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdb240b3360 tx=0x7fdb1402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d63050c0a34e8da server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.305+0000 7fdb2159a640 1 -- 172.21.15.72:0/370072907 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb14034070 con 0x7fdb2415dfe0 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.305+0000 7fdb2159a640 1 -- 172.21.15.72:0/370072907 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb14030610 con 0x7fdb2415dfe0 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.305+0000 7fdb2159a640 1 -- 172.21.15.72:0/370072907 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb140309e0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.305+0000 7fdb28871640 1 -- 172.21.15.72:0/370072907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb2415dfe0 msgr2=0x7fdb240cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.305+0000 7fdb28871640 1 --2- 172.21.15.72:0/370072907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cf2f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdb240b3360 tx=0x7fdb1402b960 comp rx=0 tx=0).stop 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.306+0000 7fdb28871640 1 -- 172.21.15.72:0/370072907 shutdown_connections 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.306+0000 7fdb28871640 1 --2- 172.21.15.72:0/370072907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cf2f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.306+0000 7fdb28871640 1 -- 172.21.15.72:0/370072907 >> 172.21.15.72:0/370072907 conn(0x7fdb240d3cd0 msgr2=0x7fdb240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.306+0000 7fdb28871640 1 -- 172.21.15.72:0/370072907 shutdown_connections 2024-03-06T07:27:24.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.306+0000 7fdb28871640 1 -- 172.21.15.72:0/370072907 wait complete. 2024-03-06T07:27:24.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.307+0000 7fdb28871640 1 Processor -- start 2024-03-06T07:27:24.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.307+0000 7fdb28871640 1 -- start start 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb28871640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb240cf830 0x7fdb241f43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb28871640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb240d29c0 0x7fdb241500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb28871640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb28871640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2415e3c0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb28871640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb240b3d60 con 0x7fdb240cf830 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb28871640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2415e540 con 0x7fdb240d29c0 2024-03-06T07:27:24.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:50918/0 (socket says 172.21.15.72:50918) 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 -- 172.21.15.72:0/802691269 learned_addr learned my addr 172.21.15.72:0/802691269 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb2259c640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb240cf830 0x7fdb241f43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb21d9b640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb240d29c0 0x7fdb241500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 -- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb240d29c0 msgr2=0x7fdb241500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb240d29c0 0x7fdb241500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 -- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb240cf830 msgr2=0x7fdb241f43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb240cf830 0x7fdb241f43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 -- 172.21.15.72:0/802691269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240b6c00 con 0x7fdb2415dfe0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb21d9b640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb240d29c0 0x7fdb241500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb22d9d640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cd2e0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00cbb0 tx=0x7fdb1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c017070 con 0x7fdb2415dfe0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c00ad40 con 0x7fdb2415dfe0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.308+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c012600 con 0x7fdb2415dfe0 2024-03-06T07:27:24.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.309+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb240b6ee0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.309+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb240b74a0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.310+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb2404f330 con 0x7fdb2415dfe0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.310+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdb1c01a070 con 0x7fdb2415dfe0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.310+0000 7fdb12ffd640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb0406b4d0 0x7fdb0406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.310+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb1c089f30 con 0x7fdb2415dfe0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.310+0000 7fdb2259c640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb0406b4d0 0x7fdb0406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.311+0000 7fdb2259c640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb0406b4d0 0x7fdb0406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdb14036000 tx=0x7fdb1403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.315+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdb1c0586e0 con 0x7fdb2415dfe0 2024-03-06T07:27:24.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.521+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb240cf380 con 0x7fdb2415dfe0 2024-03-06T07:27:24.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.522+0000 7fdb12ffd640 1 -- 172.21.15.72:0/802691269 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4982 (secure 0 0 0) 0x7fdb1c05c780 con 0x7fdb2415dfe0 2024-03-06T07:27:24.523 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:24.523 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":63,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6805,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2347391904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2347391904},{"type":"v1","addr":"172.21.15.146:6834","nonce":2347391904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/1152083148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1152083148},{"type":"v1","addr":"172.21.15.72:6835","nonce":1152083148}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7911,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3618393393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":3618393393},{"type":"v1","addr":"172.21.15.72:6837","nonce":3618393393}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:04.517794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7926},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7926":{"gid":7926,"name":"b","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6835/118157918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":118157918},{"type":"v1","addr":"172.21.15.146:6835","nonce":118157918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7926,"qdb_cluster":[7926]},"id":8}]} 2024-03-06T07:27:24.523 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 63 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb0406b4d0 msgr2=0x7fdb0406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb0406b4d0 0x7fdb0406d990 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdb14036000 tx=0x7fdb1403b040 comp rx=0 tx=0).stop 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 msgr2=0x7fdb240cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cd2e0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00cbb0 tx=0x7fdb1c00a430 comp rx=0 tx=0).stop 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 shutdown_connections 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdb0406b4d0 0x7fdb0406d990 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb2415dfe0 0x7fdb240cd2e0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdb240d29c0 0x7fdb241500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 --2- 172.21.15.72:0/802691269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdb240cf830 0x7fdb241f43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 >> 172.21.15.72:0/802691269 conn(0x7fdb240d3cd0 msgr2=0x7fdb240b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 shutdown_connections 2024-03-06T07:27:24.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.524+0000 7fdb28871640 1 -- 172.21.15.72:0/802691269 wait complete. 2024-03-06T07:27:24.536 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-03-06T07:27:24.661 INFO:teuthology.orchestra.run.smithi146.stdout:{"message":"","return_code":0} 2024-03-06T07:27:24.662 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:27:24.662 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:27:24.662 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:24.663 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:27:24.663 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:24.663 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:27:24.663 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:24.663 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:27:24.663 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:27:24.671 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:27:24.672 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-03-06T07:27:24.672 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:27:24.673 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-03-06T07:27:24.674 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:27:24.674 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:27:24.674 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:27:24.675 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:27:24.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f7cd87d0640 1 Processor -- start 2024-03-06T07:27:24.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f7cd87d0640 1 -- start start 2024-03-06T07:27:24.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f69fe15e640 1 Processor -- start 2024-03-06T07:27:24.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f9d2f25c640 1 Processor -- start 2024-03-06T07:27:24.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f9d2f25c640 1 -- start start 2024-03-06T07:27:24.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f69fe15e640 1 -- start start 2024-03-06T07:27:24.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f05dd9bc640 1 Processor -- start 2024-03-06T07:27:24.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.817+0000 7f05dd9bc640 1 -- start start 2024-03-06T07:27:24.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f7cd87d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7cd0162970 0x7f7cd00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f69fe15e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69f80d06f0 0x7f69f81669c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d2f25c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d280ccb50 0x7f9d2815a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f7cd87d0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd00d22c0 con 0x7f7cd00b4460 2024-03-06T07:27:24.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f7cd87d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd00d25a0 con 0x7f7cd0161cf0 2024-03-06T07:27:24.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f7cd87d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cd0161b00 con 0x7f7cd0162970 2024-03-06T07:27:24.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d2f25c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d280ccf30 con 0x7f9d280709f0 2024-03-06T07:27:24.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d2f25c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2815afd0 con 0x7f9d280b4ac0 2024-03-06T07:27:24.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d2f25c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d2815e110 con 0x7f9d280ccb50 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f69fe15e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80b51d0 0x7f69f8166f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d27fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9d280709f0 0x7f9d280b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38802/0 (socket says 172.21.15.72:38802) 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d27fff640 1 -- 172.21.15.72:0/905025615 learned_addr learned my addr 172.21.15.72:0/905025615 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:24.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f9d2cff8640 1 --2- 172.21.15.72:0/905025615 >> v2:172.21.15.72:3300/0 conn(0x7f9d280ccb50 0x7f9d2815a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f05dd9bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f05d806b6c0 0x7f05d806fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f05dd9bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d80d1500 0x7f05d806d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f05dd9bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d80cd4c0 con 0x7f05d8145760 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f05dd9bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d80cfeb0 con 0x7f05d806b6c0 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f05dd9bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80cda40 con 0x7f05d80d1500 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f7cd656c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7cd00b4460 0x7f7cd00b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38790/0 (socket says 172.21.15.72:38790) 2024-03-06T07:27:24.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f7cd656c640 1 -- 172.21.15.72:0/4232011246 learned_addr learned my addr 172.21.15.72:0/4232011246 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f69fe15e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f80d4c10 con 0x7f69f80d0ad0 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f69fe15e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f80b4cb0 con 0x7f69f80d06f0 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.818+0000 7f69fe15e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80cfd30 con 0x7f69f80b51d0 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69ef7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f69f80d06f0 0x7f69f81669c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d67fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f05d806b6c0 0x7f05d806fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d6ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f05d8145760 0x7f05d806afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:34078/0 (socket says 172.21.15.72:34078) 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d77fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d80d1500 0x7f05d806d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d6ffd640 1 -- 172.21.15.72:0/277961067 learned_addr learned my addr 172.21.15.72:0/277961067 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:24.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f9d2cff8640 1 -- 172.21.15.72:0/905025615 >> v1:172.21.15.146:6789/0 conn(0x7f9d280b4ac0 legacy=0x7f9d2815e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69f7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f69f80d0ad0 0x7f69f80b6b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38804/0 (socket says 172.21.15.72:38804) 2024-03-06T07:27:24.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69f7fff640 1 -- 172.21.15.72:0/506103462 learned_addr learned my addr 172.21.15.72:0/506103462 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:24.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69f77fe640 1 --2- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80b51d0 0x7f69f8166f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f9d2cff8640 1 -- 172.21.15.72:0/905025615 >> v1:172.21.15.72:6789/0 conn(0x7f9d280709f0 legacy=0x7f9d280b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd5d6b640 1 --2- 172.21.15.72:0/4232011246 >> v2:172.21.15.146:3300/0 conn(0x7f7cd0162970 0x7f7cd00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd556a640 1 -- 172.21.15.72:0/4232011246 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059801669 0 0) 0x7f7cd00d22c0 con 0x7f7cd00b4460 2024-03-06T07:27:24.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f9d2cff8640 1 -- 172.21.15.72:0/905025615 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2815b2b0 con 0x7f9d280ccb50 2024-03-06T07:27:24.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd556a640 1 -- 172.21.15.72:0/4232011246 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc003520 con 0x7f7cd00b4460 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69f6ffd640 1 -- 172.21.15.72:0/506103462 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005868503 0 0) 0x7f69f80d4c10 con 0x7f69f80d0ad0 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69f6ffd640 1 -- 172.21.15.72:0/506103462 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e4003520 con 0x7f69f80d0ad0 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69ef7fe640 1 -- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80b51d0 msgr2=0x7f69f8166f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69ef7fe640 1 --2- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80b51d0 0x7f69f8166f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69ef7fe640 1 -- 172.21.15.72:0/506103462 >> v1:172.21.15.72:6789/0 conn(0x7f69f80d0ad0 legacy=0x7f69f80b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f69ef7fe640 1 -- 172.21.15.72:0/506103462 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f80d4930 con 0x7f69f80d06f0 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd5d6b640 1 -- 172.21.15.72:0/4232011246 >> v1:172.21.15.146:6789/0 conn(0x7f7cd0161cf0 legacy=0x7f7cd00cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd556a640 1 -- 172.21.15.72:0/4232011246 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494454681 0 0) 0x7f7cbc003520 con 0x7f7cd00b4460 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd5d6b640 1 -- 172.21.15.72:0/4232011246 >> v1:172.21.15.72:6789/0 conn(0x7f7cd00b4460 legacy=0x7f7cd00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f7cd5d6b640 1 -- 172.21.15.72:0/4232011246 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd00d2880 con 0x7f7cd0162970 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d77fe640 1 -- 172.21.15.72:0/277961067 >> v2:172.21.15.146:3300/0 conn(0x7f05d806b6c0 msgr2=0x7f05d806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d77fe640 1 --2- 172.21.15.72:0/277961067 >> v2:172.21.15.146:3300/0 conn(0x7f05d806b6c0 0x7f05d806fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d77fe640 1 -- 172.21.15.72:0/277961067 >> v1:172.21.15.146:6789/0 conn(0x7f05d8145760 legacy=0x7f05d806afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.819+0000 7f05d77fe640 1 -- 172.21.15.72:0/277961067 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d80d0340 con 0x7f05d80d1500 2024-03-06T07:27:24.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f9d2cff8640 1 --2- 172.21.15.72:0/905025615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d2815a9f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00a8c0 tx=0x7f9d1c02c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9202dc95abdd1c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f9d277fe640 1 -- 172.21.15.72:0/905025615 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c030810 con 0x7f9d280ccb50 2024-03-06T07:27:24.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f9d277fe640 1 -- 172.21.15.72:0/905025615 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1c0309b0 con 0x7f9d280ccb50 2024-03-06T07:27:24.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f9d277fe640 1 -- 172.21.15.72:0/905025615 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c030d80 con 0x7f9d280ccb50 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f05d77fe640 1 --2- 172.21.15.72:0/277961067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d80d1500 0x7f05d806d7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009d90 tx=0x7f05d002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f125c32d87ce7d25 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f05d5ffb640 1 -- 172.21.15.72:0/277961067 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0039070 con 0x7f05d80d1500 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f69ef7fe640 1 --2- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80d06f0 0x7f69f81669c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f69e80097a0 tx=0x7f69e802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d543e0876187a8ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f05d5ffb640 1 -- 172.21.15.72:0/277961067 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05d002cce0 con 0x7f05d80d1500 2024-03-06T07:27:24.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f05d5ffb640 1 -- 172.21.15.72:0/277961067 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05d0034660 con 0x7f05d80d1500 2024-03-06T07:27:24.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f7cd5d6b640 1 --2- 172.21.15.72:0/4232011246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd00d4800 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00097a0 tx=0x7f7cc002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3fe2d39b8a0d8a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f69f6ffd640 1 -- 172.21.15.72:0/506103462 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e80315a0 con 0x7f69f80d06f0 2024-03-06T07:27:24.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f69f6ffd640 1 -- 172.21.15.72:0/506103462 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8037070 con 0x7f69f80d06f0 2024-03-06T07:27:24.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f69f6ffd640 1 -- 172.21.15.72:0/506103462 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8031940 con 0x7f69f80d06f0 2024-03-06T07:27:24.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f7cd556a640 1 -- 172.21.15.72:0/4232011246 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0038070 con 0x7f7cd0162970 2024-03-06T07:27:24.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f7cd556a640 1 -- 172.21.15.72:0/4232011246 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0031500 con 0x7f7cd0162970 2024-03-06T07:27:24.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.820+0000 7f7cd556a640 1 -- 172.21.15.72:0/4232011246 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc001670 con 0x7f7cd0162970 2024-03-06T07:27:24.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 -- 172.21.15.72:0/905025615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 msgr2=0x7f9d2815a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/905025615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d2815a9f0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00a8c0 tx=0x7f9d1c02c9f0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 -- 172.21.15.72:0/905025615 shutdown_connections 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/905025615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d2815a9f0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 -- 172.21.15.72:0/905025615 >> 172.21.15.72:0/905025615 conn(0x7f9d280b1c80 msgr2=0x7f9d280cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 -- 172.21.15.72:0/905025615 shutdown_connections 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f9d2f25c640 1 -- 172.21.15.72:0/905025615 wait complete. 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 -- 172.21.15.72:0/277961067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d80d1500 msgr2=0x7f05d806d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 --2- 172.21.15.72:0/277961067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d80d1500 0x7f05d806d7a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009d90 tx=0x7f05d002bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:24.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 -- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80d06f0 msgr2=0x7f69f81669c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 --2- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80d06f0 0x7f69f81669c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f69e80097a0 tx=0x7f69e802b840 comp rx=0 tx=0).stop 2024-03-06T07:27:24.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 -- 172.21.15.72:0/506103462 shutdown_connections 2024-03-06T07:27:24.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 -- 172.21.15.72:0/4232011246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 msgr2=0x7f7cd00d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/4232011246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd00d4800 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00097a0 tx=0x7f7cc002b820 comp rx=0 tx=0).stop 2024-03-06T07:27:24.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 -- 172.21.15.72:0/277961067 shutdown_connections 2024-03-06T07:27:24.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 --2- 172.21.15.72:0/277961067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d80d1500 0x7f05d806d7a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 --2- 172.21.15.72:0/277961067 >> v2:172.21.15.146:3300/0 conn(0x7f05d806b6c0 0x7f05d806fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 --2- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80b51d0 0x7f69f8166f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 --2- 172.21.15.72:0/506103462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80d06f0 0x7f69f81669c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 -- 172.21.15.72:0/506103462 >> 172.21.15.72:0/506103462 conn(0x7f69f80d3ce0 msgr2=0x7f69f80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 -- 172.21.15.72:0/277961067 >> 172.21.15.72:0/277961067 conn(0x7f05d80d0a10 msgr2=0x7f05d80d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 -- 172.21.15.72:0/506103462 shutdown_connections 2024-03-06T07:27:24.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 -- 172.21.15.72:0/4232011246 shutdown_connections 2024-03-06T07:27:24.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 -- 172.21.15.72:0/277961067 shutdown_connections 2024-03-06T07:27:24.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/4232011246 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd00d4800 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 -- 172.21.15.72:0/4232011246 >> 172.21.15.72:0/4232011246 conn(0x7f7cd00d3ce0 msgr2=0x7f7cd00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:24.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f69fe15e640 1 -- 172.21.15.72:0/506103462 wait complete. 2024-03-06T07:27:24.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f05dd9bc640 1 -- 172.21.15.72:0/277961067 wait complete. 2024-03-06T07:27:24.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 -- 172.21.15.72:0/4232011246 shutdown_connections 2024-03-06T07:27:24.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.821+0000 7f7cd87d0640 1 -- 172.21.15.72:0/4232011246 wait complete. 2024-03-06T07:27:24.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.822+0000 7f69fe15e640 1 Processor -- start 2024-03-06T07:27:24.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f9d2f25c640 1 Processor -- start 2024-03-06T07:27:24.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 -- start start 2024-03-06T07:27:24.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f9d2f25c640 1 -- start start 2024-03-06T07:27:24.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f7cd87d0640 1 Processor -- start 2024-03-06T07:27:24.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f7cd87d0640 1 -- start start 2024-03-06T07:27:24.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 0x7f69f80cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80d06f0 0x7f69f80cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69f80d0ad0 0x7f69f81fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80c6ff0 con 0x7f69f80d0ad0 2024-03-06T07:27:24.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f05dd9bc640 1 Processor -- start 2024-03-06T07:27:24.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80cf810 con 0x7f69f80b51d0 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.823+0000 7f69fe15e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f80c7170 con 0x7f69f80d06f0 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 0x7f69f80cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 0x7f69f80cbe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:34970/0 (socket says 172.21.15.72:34970) 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69f80d0ad0 0x7f69f81fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 -- 172.21.15.72:0/2007682087 learned_addr learned my addr 172.21.15.72:0/2007682087 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69f77fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80d06f0 0x7f69f80cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f05dd9bc640 1 -- start start 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd87d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7cd00b4460 0x7f7cd01f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd87d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cd0161cf0 0x7f7cd00b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 -- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80d06f0 msgr2=0x7f69f80cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80d06f0 0x7f69f80cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 -- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69f80d0ad0 msgr2=0x7f69f81fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69f80d0ad0 0x7f69f81fb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 -- 172.21.15.72:0/2007682087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f80cca10 con 0x7f69f80b51d0 2024-03-06T07:27:24.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69f7fff640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69f80d0ad0 0x7f69f81fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd87d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd87d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00ce950 con 0x7f7cd0161cf0 2024-03-06T07:27:24.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69f77fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80d06f0 0x7f69f80cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd656c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7cd00b4460 0x7f7cd01f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd87d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00cecc0 con 0x7f7cd0162970 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd87d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd00b4840 con 0x7f7cd00b4460 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd656c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7cd00b4460 0x7f7cd01f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:33080/0 (socket says 172.21.15.72:33080) 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd656c640 1 -- 172.21.15.72:0/2938213909 learned_addr learned my addr 172.21.15.72:0/2938213909 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd5d6b640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cd0161cf0 0x7f7cd00b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f9d2f25c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 0x7f9d28151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f9d2f25c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d280b4ac0 0x7f9d28150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd6d6d640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f69ef7fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 0x7f69f80cbe00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f69e8034040 tx=0x7f69e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f9d2f25c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d28157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd5d6b640 1 -- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7cd00b4460 msgr2=0x7f7cd01f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd5d6b640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7cd00b4460 0x7f7cd01f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd5d6b640 1 -- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 msgr2=0x7f7cd015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd5d6b640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd015c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd5d6b640 1 -- 172.21.15.72:0/2938213909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd01f4290 con 0x7f7cd0161cf0 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f9d2cff8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 0x7f9d28151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f9d2cff8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 0x7f9d28151eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:34992/0 (socket says 172.21.15.72:34992) 2024-03-06T07:27:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f7cd6d6d640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:24.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.824+0000 7f9d2cff8640 1 -- 172.21.15.72:0/1848510714 learned_addr learned my addr 172.21.15.72:0/1848510714 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:24.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d27fff640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d280b4ac0 0x7f9d28150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2d7f9640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d28157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8032430 con 0x7f69f80b51d0 2024-03-06T07:27:24.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d280b3060 con 0x7f9d280ccb50 2024-03-06T07:27:24.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2815b8b0 con 0x7f9d280709f0 2024-03-06T07:27:24.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d281509a0 con 0x7f9d280b4ac0 2024-03-06T07:27:24.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e80325d0 con 0x7f69f80b51d0 2024-03-06T07:27:24.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f80cccf0 con 0x7f69f80b51d0 2024-03-06T07:27:24.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f7cd5d6b640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cd0161cf0 0x7f7cd00b7420 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0033000 tx=0x7f7cc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05dd9bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05d806b6c0 0x7f05d8135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0038040 con 0x7f7cd0161cf0 2024-03-06T07:27:24.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd01f4570 con 0x7f7cd0161cf0 2024-03-06T07:27:24.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0004230 con 0x7f7cd0161cf0 2024-03-06T07:27:24.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8032980 con 0x7f69f80b51d0 2024-03-06T07:27:24.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2cff8640 1 -- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d280b4ac0 msgr2=0x7f9d28150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2cff8640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d280b4ac0 0x7f9d28150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2cff8640 1 -- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 msgr2=0x7f9d28157a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2cff8640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d28157a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2cff8640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d281524f0 con 0x7f9d280709f0 2024-03-06T07:27:24.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f80cd2b0 con 0x7f69f80b51d0 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05dd9bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80d1500 0x7f05d813acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05dd9bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d8145760 0x7f05d81434a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05d806b6c0 0x7f05d8135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05dd9bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d806be80 con 0x7f05d80d1500 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05dd9bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d80cfb40 con 0x7f05d806b6c0 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05d6ffd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05d806b6c0 0x7f05d8135cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:34994/0 (socket says 172.21.15.72:34994) 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05d6ffd640 1 -- 172.21.15.72:0/1520835908 learned_addr learned my addr 172.21.15.72:0/1520835908 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:24.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05dd9bc640 1 -- 172.21.15.72:0/1520835908 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d806dce0 con 0x7f05d8145760 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f9d2cff8640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 0x7f9d28151eb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00a9e0 tx=0x7f9d1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f05d67fc640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80d1500 0x7f05d813acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d77fe640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d8145760 0x7f05d81434a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d67fc640 1 -- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d8145760 msgr2=0x7f05d81434a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d67fc640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d8145760 0x7f05d81434a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d67fc640 1 -- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05d806b6c0 msgr2=0x7f05d8135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:24.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d67fc640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05d806b6c0 0x7f05d8135cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d67fc640 1 -- 172.21.15.72:0/1520835908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d81f1d90 con 0x7f05d80d1500 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d77fe640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d8145760 0x7f05d81434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05d67fc640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80d1500 0x7f05d813acb0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00e820 tx=0x7f05cc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc017070 con 0x7f05d80d1500 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc00c940 con 0x7f05d80d1500 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc00cd10 con 0x7f05d80d1500 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f05dd9bc640 1 -- 172.21.15.72:0/1520835908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d813b690 con 0x7f05d80d1500 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c046070 con 0x7f9d280709f0 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1c038610 con 0x7f9d280709f0 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d281527d0 con 0x7f9d280709f0 2024-03-06T07:27:24.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d28155670 con 0x7f9d280709f0 2024-03-06T07:27:24.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f815d880 con 0x7f69f80b51d0 2024-03-06T07:27:24.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc003d730 con 0x7f7cd0161cf0 2024-03-06T07:27:24.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.825+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd00c7940 con 0x7f7cd0161cf0 2024-03-06T07:27:24.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.826+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd004f330 con 0x7f7cd0161cf0 2024-03-06T07:27:24.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c0389e0 con 0x7f9d280709f0 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7cc003b070 con 0x7f7cd0161cf0 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9d1c049070 con 0x7f9d280709f0 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f05dd9bc640 1 -- 172.21.15.72:0/1520835908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d813bc50 con 0x7f05d80d1500 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f69e8032b20 con 0x7f69f80b51d0 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f69f4ff9640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f69d806b500 0x7f69d806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.827+0000 7f69f77fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f69d806b500 0x7f69d806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b0005290 con 0x7f05d80d1500 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f9d257fa640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d0c06b520 0x7f9d0c06d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f7cc6ffd640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7cb406b4d0 0x7f7cb406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7cc00abc50 con 0x7f7cd0161cf0 2024-03-06T07:27:24.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f69f77fe640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f69d806b500 0x7f69d806d9c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f69f815d670 tx=0x7f69e0009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e802f030 con 0x7f69f80b51d0 2024-03-06T07:27:24.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d1c0ab790 con 0x7f9d280709f0 2024-03-06T07:27:24.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f7cd656c640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7cb406b4d0 0x7f7cb406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.828+0000 7f9d27fff640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d0c06b520 0x7f9d0c06d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.829+0000 7f9d27fff640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d0c06b520 0x7f9d0c06d9e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9d28155140 tx=0x7f9d14007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.829+0000 7f7cd656c640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7cb406b4d0 0x7f7cb406d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc001eb0 tx=0x7f7ccc001170 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.829+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d280719b0 con 0x7f9d280709f0 2024-03-06T07:27:24.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.829+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f05cc01a070 con 0x7f05d80d1500 2024-03-06T07:27:24.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.830+0000 7f05bffff640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05c406b5d0 0x7f05c406da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:24.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.830+0000 7f05d6ffd640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05c406b5d0 0x7f05c406da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:24.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.830+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05cc089bf0 con 0x7f05d80d1500 2024-03-06T07:27:24.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.830+0000 7f05d6ffd640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05c406b5d0 0x7f05c406da90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005880 tx=0x7f05c8005a90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:24.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.832+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f69f815d880 con 0x7f69f80b51d0 2024-03-06T07:27:24.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.834+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7cc0042020 con 0x7f7cd0161cf0 2024-03-06T07:27:24.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.837+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9d280719b0 con 0x7f9d280709f0 2024-03-06T07:27:24.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:24.838+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f05cc058490 con 0x7f05d80d1500 2024-03-06T07:27:24.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-06T07:27:25.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.085+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f7cd006b180 con 0x7f7cd0161cf0 2024-03-06T07:27:25.086 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:27:25.086+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.087+0000 7f7cc6ffd640 1 -- 172.21.15.72:0/2938213909 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7926 v64) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7cc0030070 con 0x7f7cd0161cf0 2024-03-06T07:27:25.088 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7926 2024-03-06T07:27:25.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.088+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f9d2804f330 con 0x7f9d280709f0 2024-03-06T07:27:25.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.090+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7cb406b4d0 msgr2=0x7f7cb406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.090+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7cb406b4d0 0x7f7cb406d990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc001eb0 tx=0x7f7ccc001170 comp rx=0 tx=0).stop 2024-03-06T07:27:25.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.090+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cd0161cf0 msgr2=0x7f7cd00b7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.090+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cd0161cf0 0x7f7cd00b7420 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0033000 tx=0x7f7cc00046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 shutdown_connections 2024-03-06T07:27:25.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7cb406b4d0 0x7f7cb406d990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7cd0162970 0x7f7cd015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7cd0161cf0 0x7f7cd00b7420 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 --2- 172.21.15.72:0/2938213909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7cd00b4460 0x7f7cd01f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 >> 172.21.15.72:0/2938213909 conn(0x7f7cd00d3ce0 msgr2=0x7f7cd00d1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 shutdown_connections 2024-03-06T07:27:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.091+0000 7f7cd87d0640 1 -- 172.21.15.72:0/2938213909 wait complete. 2024-03-06T07:27:25.095 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:27:25.094+0000 7f7b90a4b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:25.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.098+0000 7f9d257fa640 1 -- 172.21.15.72:0/1848510714 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7911 v66) v1 ==== 95+0+0 (secure 0 0 0) 0x7f9d1c034030 con 0x7f9d280709f0 2024-03-06T07:27:25.099 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7911 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.101+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d0c06b520 msgr2=0x7f9d0c06d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.101+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d0c06b520 0x7f9d0c06d9e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9d28155140 tx=0x7f9d14007e80 comp rx=0 tx=0).stop 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.101+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 msgr2=0x7f9d28151eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.101+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 0x7f9d28151eb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00a9e0 tx=0x7f9d1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 shutdown_connections 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9d0c06b520 0x7f9d0c06d9e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d280ccb50 0x7f9d28157a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d280b4ac0 0x7f9d28150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 --2- 172.21.15.72:0/1848510714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d280709f0 0x7f9d28151eb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 >> 172.21.15.72:0/1848510714 conn(0x7f9d280b1c80 msgr2=0x7f9d2806da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 shutdown_connections 2024-03-06T07:27:25.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.102+0000 7f9d2f25c640 1 -- 172.21.15.72:0/1848510714 wait complete. 2024-03-06T07:27:25.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.104+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f69f804f330 con 0x7f69f80b51d0 2024-03-06T07:27:25.109 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:27:25.109 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:27:25.111 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:27:25.111 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:25.114 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:27:25.114 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:27:25.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.114+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f05b0005580 con 0x7f05d80d1500 2024-03-06T07:27:25.116 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:27:25.116 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:25.187 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:27:25.190 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:27:25.270 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:27:25.269+0000 7f7b90a4b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:27:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.272+0000 7f05bffff640 1 -- 172.21.15.72:0/1520835908 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7908 v68) v1 ==== 95+0+0 (secure 0 0 0) 0x7f05cc05c530 con 0x7f05d80d1500 2024-03-06T07:27:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7908 2024-03-06T07:27:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05c406b5d0 msgr2=0x7f05c406da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05c406b5d0 0x7f05c406da90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f05c8005880 tx=0x7f05c8005a90 comp rx=0 tx=0).stop 2024-03-06T07:27:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80d1500 msgr2=0x7f05d813acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80d1500 0x7f05d813acb0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00e820 tx=0x7f05cc00ece0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 shutdown_connections 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f05c406b5d0 0x7f05c406da90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05d8145760 0x7f05d81434a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f05d80d1500 0x7f05d813acb0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 --2- 172.21.15.72:0/1520835908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05d806b6c0 0x7f05d8135cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 >> 172.21.15.72:0/1520835908 conn(0x7f05d80d0a10 msgr2=0x7f05d8144990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f69f4ff9640 1 -- 172.21.15.72:0/2007682087 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6805 v68) v1 ==== 95+0+0 (secure 0 0 0) 0x7f69e8040070 con 0x7f69f80b51d0 2024-03-06T07:27:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 shutdown_connections 2024-03-06T07:27:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.275+0000 7f05bdffb640 1 -- 172.21.15.72:0/1520835908 wait complete. 2024-03-06T07:27:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 6805 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f69d806b500 msgr2=0x7f69d806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f69d806b500 0x7f69d806d9c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f69f815d670 tx=0x7f69e0009190 comp rx=0 tx=0).stop 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 msgr2=0x7f69f80cbe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 0x7f69f80cbe00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f69e8034040 tx=0x7f69e80046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 shutdown_connections 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f69d806b500 0x7f69d806d9c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69f80d0ad0 0x7f69f81fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69f80d06f0 0x7f69f80cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 --2- 172.21.15.72:0/2007682087 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f69f80b51d0 0x7f69f80cbe00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.278+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 >> 172.21.15.72:0/2007682087 conn(0x7f69f80d3ce0 msgr2=0x7f69f815bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.279+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 shutdown_connections 2024-03-06T07:27:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.279+0000 7f69fe15e640 1 -- 172.21.15.72:0/2007682087 wait complete. 2024-03-06T07:27:25.294 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:27:25.294 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:27:25.296 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:27:25.296 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:27:25.298 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:27:25.298 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:25.337 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:27:25.337 DEBUG:teuthology.parallel:result is None 2024-03-06T07:27:25.338 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:25.373 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:27:25.413 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:27:25.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.437+0000 7f2bf3b3e640 1 Processor -- start 2024-03-06T07:27:25.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.438+0000 7f2bf3b3e640 1 -- start start 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.438+0000 7f2bf3b3e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2bec0cf850 0x7f2bec0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.438+0000 7f2bf3b3e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2bec06cf20 0x7f2bec06d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf3b3e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bec15e380 con 0x7f2bec0d00c0 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf3b3e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bec06ca90 con 0x7f2bec06cf20 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf3b3e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bec0cdc10 con 0x7f2bec0cf850 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2bec06cf20 0x7f2bec06d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2bec06cf20 0x7f2bec06d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51022/0 (socket says 172.21.15.72:51022) 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 -- 172.21.15.72:0/3886335302 learned_addr learned my addr 172.21.15.72:0/3886335302 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 -- 172.21.15.72:0/3886335302 >> v2:172.21.15.146:3300/0 conn(0x7f2bec0cf850 msgr2=0x7f2bec0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 --2- 172.21.15.72:0/3886335302 >> v2:172.21.15.146:3300/0 conn(0x7f2bec0cf850 0x7f2bec0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 -- 172.21.15.72:0/3886335302 >> v1:172.21.15.146:6789/0 conn(0x7f2bec0d00c0 legacy=0x7f2bec0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:25.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.439+0000 7f2bf20db640 1 -- 172.21.15.72:0/3886335302 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec06c160 con 0x7f2bec06cf20 2024-03-06T07:27:25.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf20db640 1 --2- 172.21.15.72:0/3886335302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec06cf20 0x7f2bec06d3b0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f2bec071de0 tx=0x7f2be402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af7ba97481785a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf08d8640 1 -- 172.21.15.72:0/3886335302 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be402ca80 con 0x7f2bec06cf20 2024-03-06T07:27:25.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf08d8640 1 -- 172.21.15.72:0/3886335302 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be402cc20 con 0x7f2bec06cf20 2024-03-06T07:27:25.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf08d8640 1 -- 172.21.15.72:0/3886335302 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be40343f0 con 0x7f2bec06cf20 2024-03-06T07:27:25.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/3886335302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec06cf20 msgr2=0x7f2bec06d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/3886335302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec06cf20 0x7f2bec06d3b0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f2bec071de0 tx=0x7f2be402bee0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/3886335302 shutdown_connections 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/3886335302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec06cf20 0x7f2bec06d3b0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/3886335302 >> v2:172.21.15.146:3300/0 conn(0x7f2bec0cf850 0x7f2bec0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/3886335302 >> 172.21.15.72:0/3886335302 conn(0x7f2bec0b35a0 msgr2=0x7f2bec0b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/3886335302 shutdown_connections 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.440+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/3886335302 wait complete. 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.441+0000 7f2bf3b3e640 1 Processor -- start 2024-03-06T07:27:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 -- start start 2024-03-06T07:27:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bec06cf20 0x7f2bec0ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bec0cf850 0x7f2bec0afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 0x7f2bec0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0cd900 con 0x7f2bec0d00c0 2024-03-06T07:27:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec06c720 con 0x7f2bec06cf20 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf3b3e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec0b1030 con 0x7f2bec0cf850 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf20db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 0x7f2bec0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf20db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 0x7f2bec0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51032/0 (socket says 172.21.15.72:51032) 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf20db640 1 -- 172.21.15.72:0/4157435162 learned_addr learned my addr 172.21.15.72:0/4157435162 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf18da640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bec06cf20 0x7f2bec0ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.442+0000 7f2bf10d9640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bec0cf850 0x7f2bec0afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf20db640 1 -- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bec0cf850 msgr2=0x7f2bec0afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf20db640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bec0cf850 0x7f2bec0afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf20db640 1 -- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bec06cf20 msgr2=0x7f2bec0ca3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf20db640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bec06cf20 0x7f2bec0ca3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf20db640 1 -- 172.21.15.72:0/4157435162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec0c90b0 con 0x7f2bec0d00c0 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf18da640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bec06cf20 0x7f2bec0ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf20db640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 0x7f2bec0b0000 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009f40 tx=0x7f2be4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4043020 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be403e450 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec0c9390 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be4009510 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.443+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec0c9950 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.444+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec04f330 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.444+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2be4046020 con 0x7f2bec0d00c0 2024-03-06T07:27:25.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.445+0000 7f2bde7fc640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bd006b500 0x7f2bd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:25.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.445+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2be40ac7c0 con 0x7f2bec0d00c0 2024-03-06T07:27:25.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.445+0000 7f2bf18da640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bd006b500 0x7f2bd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:25.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.445+0000 7f2bf18da640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bd006b500 0x7f2bd006d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2be0006fd0 tx=0x7f2be0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:25.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.449+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2be404d070 con 0x7f2bec0d00c0 2024-03-06T07:27:25.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.656+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bec0c85d0 con 0x7f2bec0d00c0 2024-03-06T07:27:25.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.657+0000 7f2bde7fc640 1 -- 172.21.15.72:0/4157435162 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+3392 (secure 0 0 0) 0x7f2be407b540 con 0x7f2bec0d00c0 2024-03-06T07:27:25.658 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:25.658 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":69,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:25.278047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:replay","state_seq":1,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:25.658 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 69 2024-03-06T07:27:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bd006b500 msgr2=0x7f2bd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bd006b500 0x7f2bd006d9c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2be0006fd0 tx=0x7f2be0009060 comp rx=0 tx=0).stop 2024-03-06T07:27:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 msgr2=0x7f2bec0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 0x7f2bec0b0000 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009f40 tx=0x7f2be4004a40 comp rx=0 tx=0).stop 2024-03-06T07:27:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 shutdown_connections 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bd006b500 0x7f2bd006d9c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bec0d00c0 0x7f2bec0b0000 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bec0cf850 0x7f2bec0afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 --2- 172.21.15.72:0/4157435162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bec06cf20 0x7f2bec0ca3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 >> 172.21.15.72:0/4157435162 conn(0x7f2bec0b35a0 msgr2=0x7f2bec06d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 shutdown_connections 2024-03-06T07:27:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:25.659+0000 7f2bf3b3e640 1 -- 172.21.15.72:0/4157435162 wait complete. 2024-03-06T07:27:26.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:02.515750+0000', 'modified': '2024-03-06T07:27:25.278047+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8217': {'gid': 8217, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.72:6835/3088662828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3088662828}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 3088662828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:26.073 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8217:up:replay 2024-03-06T07:27:27.074 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:27.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.185+0000 7f5042ff7640 1 Processor -- start 2024-03-06T07:27:27.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.185+0000 7f5042ff7640 1 -- start start 2024-03-06T07:27:27.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f5042ff7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f5042ff7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b3430 con 0x7f503c15dfe0 2024-03-06T07:27:27.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f5040d93640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f503c15dfe0 0x7f503c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38806/0 (socket says 172.21.15.72:38806) 2024-03-06T07:27:27.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f5042ff7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c0b3710 con 0x7f503c0b41f0 2024-03-06T07:27:27.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f5040d93640 1 -- 172.21.15.72:0/659876340 learned_addr learned my addr 172.21.15.72:0/659876340 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:27.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f5042ff7640 1 -- 172.21.15.72:0/659876340 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0d2c50 con 0x7f503c0d22b0 2024-03-06T07:27:27.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.186+0000 7f503bfff640 1 --2- 172.21.15.72:0/659876340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712498298 0 0) 0x7f503c0b3430 con 0x7f503c15dfe0 2024-03-06T07:27:27.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c003520 con 0x7f503c15dfe0 2024-03-06T07:27:27.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615117035 0 0) 0x7f503c0b3710 con 0x7f503c0b41f0 2024-03-06T07:27:27.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c0b3430 con 0x7f503c0b41f0 2024-03-06T07:27:27.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503bfff640 1 -- 172.21.15.72:0/659876340 >> v1:172.21.15.146:6789/0 conn(0x7f503c0b41f0 legacy=0x7f503c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503bfff640 1 -- 172.21.15.72:0/659876340 >> v1:172.21.15.72:6789/0 conn(0x7f503c15dfe0 legacy=0x7f503c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.187+0000 7f503bfff640 1 -- 172.21.15.72:0/659876340 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c0b39f0 con 0x7f503c0d22b0 2024-03-06T07:27:27.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.188+0000 7f503bfff640 1 --2- 172.21.15.72:0/659876340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c0d2710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50280097d0 tx=0x7f502802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8686d86fe3f2c7e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.188+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028034070 con 0x7f503c0d22b0 2024-03-06T07:27:27.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.188+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5028030800 con 0x7f503c0d22b0 2024-03-06T07:27:27.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.188+0000 7f503b7fe640 1 -- 172.21.15.72:0/659876340 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028030bb0 con 0x7f503c0d22b0 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 -- 172.21.15.72:0/659876340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 msgr2=0x7f503c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 --2- 172.21.15.72:0/659876340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c0d2710 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f50280097d0 tx=0x7f502802b940 comp rx=0 tx=0).stop 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 -- 172.21.15.72:0/659876340 shutdown_connections 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 --2- 172.21.15.72:0/659876340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c0d2710 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 -- 172.21.15.72:0/659876340 >> 172.21.15.72:0/659876340 conn(0x7f503c0d3cd0 msgr2=0x7f503c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 -- 172.21.15.72:0/659876340 shutdown_connections 2024-03-06T07:27:27.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.189+0000 7f5042ff7640 1 -- 172.21.15.72:0/659876340 wait complete. 2024-03-06T07:27:27.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.190+0000 7f5042ff7640 1 Processor -- start 2024-03-06T07:27:27.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.190+0000 7f5042ff7640 1 -- start start 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5042ff7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f503c0b41f0 0x7f503c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5042ff7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c14c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5042ff7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 0x7f503c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5042ff7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0b3d60 con 0x7f503c15dfe0 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5042ff7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0b31e0 con 0x7f503c0b41f0 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5042ff7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c0d1c70 con 0x7f503c0d22b0 2024-03-06T07:27:27.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5041594640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 0x7f503c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5041594640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 0x7f503c160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51042/0 (socket says 172.21.15.72:51042) 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5041594640 1 -- 172.21.15.72:0/1442486473 learned_addr learned my addr 172.21.15.72:0/1442486473 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f503bfff640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c14c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.191+0000 7f5040d93640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f503c0b41f0 0x7f503c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5041594640 1 -- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 msgr2=0x7f503c14c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5041594640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c14c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5041594640 1 -- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f503c0b41f0 msgr2=0x7f503c0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5041594640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f503c0b41f0 0x7f503c0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5041594640 1 -- 172.21.15.72:0/1442486473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c15ff60 con 0x7f503c15dfe0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5041594640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 0x7f503c160d20 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f503400c7a0 tx=0x7f503400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034003b70 con 0x7f503c15dfe0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5034003d30 con 0x7f503c15dfe0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c160240 con 0x7f503c15dfe0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503400aa90 con 0x7f503c15dfe0 2024-03-06T07:27:27.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.192+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c160800 con 0x7f503c15dfe0 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.193+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c04f330 con 0x7f503c15dfe0 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.194+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f503400ac30 con 0x7f503c15dfe0 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.194+0000 7f50397fa640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f502006b4d0 0x7f502006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:27.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.194+0000 7f5040d93640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f502006b4d0 0x7f502006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:27.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.194+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f503408a080 con 0x7f503c15dfe0 2024-03-06T07:27:27.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.195+0000 7f5040d93640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f502006b4d0 0x7f502006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f50300011e0 tx=0x7f5030007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:27.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.199+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5034058840 con 0x7f503c15dfe0 2024-03-06T07:27:27.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.408+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503c0b0c10 con 0x7f503c15dfe0 2024-03-06T07:27:27.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.409+0000 7f50397fa640 1 -- 172.21.15.72:0/1442486473 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f503405c8e0 con 0x7f503c15dfe0 2024-03-06T07:27:27.411 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:27.411 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":71,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:26.286794+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-03-06T07:27:27.411 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 71 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.410+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f502006b4d0 msgr2=0x7f502006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.410+0000 7f5042ff7640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f502006b4d0 0x7f502006d990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f50300011e0 tx=0x7f5030007d30 comp rx=0 tx=0).stop 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 msgr2=0x7f503c160d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 0x7f503c160d20 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f503400c7a0 tx=0x7f503400cc60 comp rx=0 tx=0).stop 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 shutdown_connections 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f502006b4d0 0x7f502006d990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15dfe0 0x7f503c160d20 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f503c0d22b0 0x7f503c14c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 --2- 172.21.15.72:0/1442486473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f503c0b41f0 0x7f503c0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:27.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 >> 172.21.15.72:0/1442486473 conn(0x7f503c0d3cd0 msgr2=0x7f503c0d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:27.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 shutdown_connections 2024-03-06T07:27:27.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:27.411+0000 7f5042ff7640 1 -- 172.21.15.72:0/1442486473 wait complete. 2024-03-06T07:27:27.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:02.515750+0000', 'modified': '2024-03-06T07:27:26.286794+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8217': {'gid': 8217, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.72:6835/3088662828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3088662828}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 3088662828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-03-06T07:27:27.826 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8217:up:rejoin 2024-03-06T07:27:28.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.926+0000 7fc76e6df640 1 Processor -- start 2024-03-06T07:27:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.926+0000 7fc76e6df640 1 -- start start 2024-03-06T07:27:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc76e6df640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7680b4170 0x7fc7680b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc76e6df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7680d0a80 0x7fc7680cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc76e6df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7680d25d0 con 0x7fc7680d4760 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc76e6df640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680d4bf0 con 0x7fc7680b4170 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc76e6df640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7680ce9e0 con 0x7fc7680d0a80 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7680b4170 0x7fc7680b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc7680b4170 0x7fc7680b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51058/0 (socket says 172.21.15.72:51058) 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 -- 172.21.15.72:0/2331826032 learned_addr learned my addr 172.21.15.72:0/2331826032 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 -- 172.21.15.72:0/2331826032 >> v2:172.21.15.146:3300/0 conn(0x7fc7680d0a80 msgr2=0x7fc7680cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 --2- 172.21.15.72:0/2331826032 >> v2:172.21.15.146:3300/0 conn(0x7fc7680d0a80 0x7fc7680cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 -- 172.21.15.72:0/2331826032 >> v1:172.21.15.146:6789/0 conn(0x7fc7680d4760 legacy=0x7fc7680d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.927+0000 7fc767fff640 1 -- 172.21.15.72:0/2331826032 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680d2870 con 0x7fc7680b4170 2024-03-06T07:27:28.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.928+0000 7fc767fff640 1 --2- 172.21.15.72:0/2331826032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b4170 0x7fc7680b6a90 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fc76808df00 tx=0x7fc75802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffbd2f7dbc1af36 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.928+0000 7fc766ffd640 1 -- 172.21.15.72:0/2331826032 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758035020 con 0x7fc7680b4170 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.928+0000 7fc766ffd640 1 -- 172.21.15.72:0/2331826032 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc758031870 con 0x7fc7680b4170 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.928+0000 7fc766ffd640 1 -- 172.21.15.72:0/2331826032 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758031c40 con 0x7fc7680b4170 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 -- 172.21.15.72:0/2331826032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b4170 msgr2=0x7fc7680b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 --2- 172.21.15.72:0/2331826032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b4170 0x7fc7680b6a90 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fc76808df00 tx=0x7fc75802b910 comp rx=0 tx=0).stop 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 -- 172.21.15.72:0/2331826032 shutdown_connections 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 --2- 172.21.15.72:0/2331826032 >> v2:172.21.15.146:3300/0 conn(0x7fc7680d0a80 0x7fc7680cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 --2- 172.21.15.72:0/2331826032 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b4170 0x7fc7680b6a90 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 -- 172.21.15.72:0/2331826032 >> 172.21.15.72:0/2331826032 conn(0x7fc7680d3c40 msgr2=0x7fc7680d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 -- 172.21.15.72:0/2331826032 shutdown_connections 2024-03-06T07:27:28.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.929+0000 7fc76e6df640 1 -- 172.21.15.72:0/2331826032 wait complete. 2024-03-06T07:27:28.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.930+0000 7fc76e6df640 1 Processor -- start 2024-03-06T07:27:28.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.930+0000 7fc76e6df640 1 -- start start 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76e6df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7680b4170 0x7fc76809f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76e6df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7680d0a80 0x7fc7680b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76e6df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 0x7fc76809fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76e6df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680b4dd0 con 0x7fc7680b0da0 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76e6df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680b4550 con 0x7fc7680b4170 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76e6df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7680b4f50 con 0x7fc7680d0a80 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 0x7fc76809fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 0x7fc76809fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51068/0 (socket says 172.21.15.72:51068) 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 -- 172.21.15.72:0/3146170576 learned_addr learned my addr 172.21.15.72:0/3146170576 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:28.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc7677fe640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7680d0a80 0x7fc7680b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc767fff640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7680b4170 0x7fc76809f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 -- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7680d0a80 msgr2=0x7fc7680b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7680d0a80 0x7fc7680b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 -- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7680b4170 msgr2=0x7fc76809f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7680b4170 0x7fc76809f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 -- 172.21.15.72:0/3146170576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7680b1320 con 0x7fc7680b0da0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.931+0000 7fc76cc7c640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 0x7fc76809fd70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b9a0 tx=0x7fc76000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.932+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760019070 con 0x7fc7680b0da0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.932+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76000cbc0 con 0x7fc7680b0da0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.932+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7680b1600 con 0x7fc7680b0da0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.932+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760014440 con 0x7fc7680b0da0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.932+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7680b1bc0 con 0x7fc7680b0da0 2024-03-06T07:27:28.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.933+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7680d2ec0 con 0x7fc7680b0da0 2024-03-06T07:27:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.933+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc76001c070 con 0x7fc7680b0da0 2024-03-06T07:27:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.933+0000 7fc764ff9640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.933+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc760089fc0 con 0x7fc7680b0da0 2024-03-06T07:27:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.934+0000 7fc767fff640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:28.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.934+0000 7fc767fff640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc75406b4d0 0x7fc75406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc758036000 tx=0x7fc75803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:28.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:28.942+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7680d2ec0 con 0x7fc7680b0da0 2024-03-06T07:27:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.158+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc76804f330 con 0x7fc7680b0da0 2024-03-06T07:27:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.158+0000 7fc764ff9640 1 -- 172.21.15.72:0/3146170576 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fc760058dd0 con 0x7fc7680b0da0 2024-03-06T07:27:29.160 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:29.160 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:27:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc75406b4d0 msgr2=0x7fc75406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc75406b4d0 0x7fc75406d990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fc758036000 tx=0x7fc75803c040 comp rx=0 tx=0).stop 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 msgr2=0x7fc76809fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 0x7fc76809fd70 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fc76000b9a0 tx=0x7fc76000be60 comp rx=0 tx=0).stop 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 shutdown_connections 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc75406b4d0 0x7fc75406d990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7680b0da0 0x7fc76809fd70 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7680d0a80 0x7fc7680b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 --2- 172.21.15.72:0/3146170576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7680b4170 0x7fc76809f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 >> 172.21.15.72:0/3146170576 conn(0x7fc7680d3c40 msgr2=0x7fc7680cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 shutdown_connections 2024-03-06T07:27:29.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.161+0000 7fc76e6df640 1 -- 172.21.15.72:0/3146170576 wait complete. 2024-03-06T07:27:29.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:27:02.515750+0000', 'modified': '2024-03-06T07:27:28.290731+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8217}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8217': {'gid': 8217, 'name': 'c', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.72:6835/3088662828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3088662828}, {'type': 'v1', 'addr': '172.21.15.72:6835', 'nonce': 3088662828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8217, 'qdb_cluster': [8217]} 2024-03-06T07:27:29.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:27:29.580 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-06T07:27:29.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.681+0000 7f57ee9bc640 1 Processor -- start 2024-03-06T07:27:29.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.681+0000 7f57ee9bc640 1 -- start start 2024-03-06T07:27:29.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.682+0000 7f57ee9bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e80b4c00 0x7f57e80b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ee9bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f57e815ba50 0x7f57e815be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ee9bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e815d510 con 0x7f57e80b28a0 2024-03-06T07:27:29.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ee9bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e815e0b0 con 0x7f57e815ba50 2024-03-06T07:27:29.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ee9bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e80b33f0 con 0x7f57e80b4c00 2024-03-06T07:27:29.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ecf59640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f57e815ba50 0x7f57e815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57e77fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f57e80b28a0 0x7f57e80b2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:34114/0 (socket says 172.21.15.72:34114) 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57e77fe640 1 -- 172.21.15.72:0/4186827265 learned_addr learned my addr 172.21.15.72:0/4186827265 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e80b4c00 0x7f57e80b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ecf59640 1 -- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e80b4c00 msgr2=0x7f57e80b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ecf59640 1 --2- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e80b4c00 0x7f57e80b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ecf59640 1 -- 172.21.15.72:0/4186827265 >> v1:172.21.15.146:6789/0 conn(0x7f57e80b28a0 legacy=0x7f57e80b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57ecf59640 1 -- 172.21.15.72:0/4186827265 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e815d780 con 0x7f57e815ba50 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57e6ffd640 1 -- 172.21.15.72:0/4186827265 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221359122 0 0) 0x7f57e815d510 con 0x7f57e80b28a0 2024-03-06T07:27:29.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.683+0000 7f57e7fff640 1 --2- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e80b4c00 0x7f57e80b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.684+0000 7f57ecf59640 1 --2- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e815ba50 0x7f57e815be90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009d90 tx=0x7f57e002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b5c3c141b6feaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.684+0000 7f57e6ffd640 1 -- 172.21.15.72:0/4186827265 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0039070 con 0x7f57e815ba50 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.684+0000 7f57e6ffd640 1 -- 172.21.15.72:0/4186827265 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e002cce0 con 0x7f57e815ba50 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.684+0000 7f57e6ffd640 1 -- 172.21.15.72:0/4186827265 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0034660 con 0x7f57e815ba50 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 -- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e815ba50 msgr2=0x7f57e815be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e815ba50 0x7f57e815be90 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f57e0009d90 tx=0x7f57e002bf30 comp rx=0 tx=0).stop 2024-03-06T07:27:29.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 -- 172.21.15.72:0/4186827265 shutdown_connections 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e815ba50 0x7f57e815be90 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/4186827265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e80b4c00 0x7f57e80b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 -- 172.21.15.72:0/4186827265 >> 172.21.15.72:0/4186827265 conn(0x7f57e809a9f0 msgr2=0x7f57e809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 -- 172.21.15.72:0/4186827265 shutdown_connections 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.685+0000 7f57ee9bc640 1 -- 172.21.15.72:0/4186827265 wait complete. 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.686+0000 7f57ee9bc640 1 Processor -- start 2024-03-06T07:27:29.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.686+0000 7f57ee9bc640 1 -- start start 2024-03-06T07:27:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ee9bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e80b28a0 0x7f57e809e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ee9bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 0x7f57e8149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ee9bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e815ba50 0x7f57e80afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ee9bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e815f550 con 0x7f57e80b4c00 2024-03-06T07:27:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ee9bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e815dd40 con 0x7f57e80b28a0 2024-03-06T07:27:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ee9bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e815f6d0 con 0x7f57e815ba50 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 0x7f57e8149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 0x7f57e8149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51070/0 (socket says 172.21.15.72:51070) 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 -- 172.21.15.72:0/1122819288 learned_addr learned my addr 172.21.15.72:0/1122819288 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 -- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e815ba50 msgr2=0x7f57e80afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57ecf59640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e815ba50 0x7f57e80afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e7fff640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e80b28a0 0x7f57e809e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e815ba50 0x7f57e80afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 -- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e80b28a0 msgr2=0x7f57e809e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e80b28a0 0x7f57e809e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 -- 172.21.15.72:0/1122819288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e809ebe0 con 0x7f57e80b4c00 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.687+0000 7f57e77fe640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 0x7f57e8149440 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c500 tx=0x7f57dc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.688+0000 7f57e4ff9640 1 -- 172.21.15.72:0/1122819288 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc018070 con 0x7f57e80b4c00 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.688+0000 7f57e4ff9640 1 -- 172.21.15.72:0/1122819288 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc007580 con 0x7f57e80b4c00 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.688+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e809eec0 con 0x7f57e80b4c00 2024-03-06T07:27:29.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.688+0000 7f57e4ff9640 1 -- 172.21.15.72:0/1122819288 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004a20 con 0x7f57e80b4c00 2024-03-06T07:27:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.688+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e809f480 con 0x7f57e80b4c00 2024-03-06T07:27:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.690+0000 7f57e4ff9640 1 -- 172.21.15.72:0/1122819288 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f57dc01b070 con 0x7f57e80b4c00 2024-03-06T07:27:29.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.690+0000 7f57e4ff9640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57d406b4d0 0x7f57d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.690+0000 7f57e4ff9640 1 -- 172.21.15.72:0/1122819288 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f57dc089590 con 0x7f57e80b4c00 2024-03-06T07:27:29.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.690+0000 7f57e7fff640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57d406b4d0 0x7f57d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.690+0000 7f57e7fff640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57d406b4d0 0x7f57d406d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009560 tx=0x7f57d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 Processor -- start 2024-03-06T07:27:29.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 -- start start 2024-03-06T07:27:29.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b20289e870 0x55b20289ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 0x55b2028a4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b2028cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2028086b0 con 0x55b2028a28c0 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2027b1a60 con 0x55b20289e870 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b202804750 con 0x55b2028ce600 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 0x55b2028a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 0x55b2028a4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51078/0 (socket says 172.21.15.72:51078) 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57e77fe640 1 -- 172.21.15.72:0/46823348 learned_addr learned my addr 172.21.15.72:0/46823348 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.691+0000 7f57e7fff640 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b20289e870 0x55b20289ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57ecf59640 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b2028cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57e77fe640 1 -- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 msgr2=0x55b2028cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57e77fe640 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b2028cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57e77fe640 1 -- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b20289e870 msgr2=0x55b20289ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57e77fe640 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b20289e870 0x55b20289ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57e77fe640 1 -- 172.21.15.72:0/46823348 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2027cff30 con 0x55b2028a28c0 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57e77fe640 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 0x55b2028a4360 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f57dc087a70 tx=0x7f57dc05bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86e9065e3ddc073 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57cdffb640 1 -- 172.21.15.72:0/46823348 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc05e0c0 con 0x55b2028a28c0 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57cdffb640 1 -- 172.21.15.72:0/46823348 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc055fa0 con 0x55b2028a28c0 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.692+0000 7f57cdffb640 1 -- 172.21.15.72:0/46823348 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004bc0 con 0x55b2028a28c0 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 -- 172.21.15.72:0/46823348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 msgr2=0x55b2028a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 0x55b2028a4360 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f57dc087a70 tx=0x7f57dc05bdf0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 -- 172.21.15.72:0/46823348 shutdown_connections 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b2028cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028a28c0 0x55b2028a4360 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 --2- 172.21.15.72:0/46823348 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b20289e870 0x55b20289ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 -- 172.21.15.72:0/46823348 >> 172.21.15.72:0/46823348 conn(0x55b2028a08f0 msgr2=0x55b2028a21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 -- 172.21.15.72:0/46823348 shutdown_connections 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 -- 172.21.15.72:0/46823348 wait complete. 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 -- start start 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.693+0000 7f57f15a5740 1 Processor -- start 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b2028e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028ea120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028b50b0 0x55b2028b54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2028a2d50 con 0x55b2028e9d40 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2027c4ea0 con 0x55b2028ce600 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2028a3030 con 0x55b2028b50b0 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57f15a5740 10 client.0 _ll_register_callbacks cb 0x55b2027c4870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028ea120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57e77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028ea120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51082/0 (socket says 172.21.15.72:51082) 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57e77fe640 1 -- 172.21.15.72:0/2949172915 learned_addr learned my addr 172.21.15.72:0/2949172915 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57ecf59640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028b50b0 0x55b2028b54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57e7fff640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b2028e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.694+0000 7f57e77fe640 1 -- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028b50b0 msgr2=0x55b2028b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57e77fe640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028b50b0 0x55b2028b54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57e77fe640 1 -- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 msgr2=0x55b2028e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57e77fe640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b2028e5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57e77fe640 1 -- 172.21.15.72:0/2949172915 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2027c5270 con 0x55b2028e9d40 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57ecf59640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028b50b0 0x55b2028b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57e77fe640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028ea120 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f57dc060570 tx=0x7f57dc05a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5697646300230d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57c6ffd640 1 -- 172.21.15.72:0/2949172915 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc087060 con 0x55b2028e9d40 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57c6ffd640 1 -- 172.21.15.72:0/2949172915 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f57dc0890c0 con 0x55b2028e9d40 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57dc055fa0 con 0x55b2028e9d40 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57c6ffd640 1 -- 172.21.15.72:0/2949172915 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57dc053f40 con 0x55b2028e9d40 2024-03-06T07:27:29.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 10 client.8253 fetch_fsmap learned FSMap version 72 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x55b20251d2d0 con 0x55b2028e9d40 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57c6ffd640 1 -- 172.21.15.72:0/2949172915 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc055fa0 con 0x55b2028e9d40 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57c6ffd640 1 -- 172.21.15.72:0/2949172915 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f57dc0570f0 con 0x55b2028e9d40 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 10 client.8253 fetch_fsmap finished waiting for FSMap version 72 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 10 client.8253 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 20 client.8253 populate_metadata read hostname 'smithi072' 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028b75c0 0x55b2028b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 4 client.8253 mds_command: new command op to 8217 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 --> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b202815bf0 con 0x55b2028b75c0 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.695+0000 7f57e7fff640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028b75c0 0x55b2028b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.696+0000 7f57c6ffd640 10 client.8253 ms_handle_connect on v2:172.21.15.72:6834/3088662828 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.696+0000 7f57e7fff640 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028b75c0 0x55b2028b79a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=798802d2bac48ee6 server_cookie=834e53e0099230fb in_seq=0 out_seq=0 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.696+0000 7f57c6ffd640 1 -- 172.21.15.72:0/2949172915 <== mds.0 v2:172.21.15.72:6834/3088662828 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b202815bf0 con 0x55b2028b75c0 2024-03-06T07:27:29.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.696+0000 7f57c6ffd640 10 client.8253 handle_command_reply: tid=0 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.696+0000 7f57f15a5740 1 client.8253 shutdown 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 msgr2=0x55b2028ea120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028ea120 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f57dc060570 tx=0x7f57dc05a040 comp rx=0 tx=0).stop 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 shutdown_connections 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028b75c0 0x55b2028b79a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028b50b0 0x55b2028b54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028ea120 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57c6ffd640 0 client.8253 ms_handle_reset on v2:172.21.15.72:6834/3088662828 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 --2- 172.21.15.72:0/2949172915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b2028e5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 >> 172.21.15.72:0/2949172915 conn(0x55b20289e870 msgr2=0x55b20289d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 shutdown_connections 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 1 -- 172.21.15.72:0/2949172915 wait complete. 2024-03-06T07:27:29.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.697+0000 7f57f15a5740 20 client.8253 trim_cache size 0 max 16384 2024-03-06T07:27:29.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.728+0000 7f57f15a5740 1 Processor -- start 2024-03-06T07:27:29.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.728+0000 7f57f15a5740 1 -- start start 2024-03-06T07:27:29.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028b50b0 0x55b2028c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b20289d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57f15a5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e81ff2f0 con 0x55b2028e9d40 2024-03-06T07:27:29.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e81fe6f0 con 0x55b2028b50b0 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e81fecf0 con 0x55b2028ce600 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028a31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51096/0 (socket says 172.21.15.72:51096) 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 -- 172.21.15.72:0/813339539 learned_addr learned my addr 172.21.15.72:0/813339539 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57e7fff640 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028b50b0 0x55b2028c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57e77fe640 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b20289d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 -- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 msgr2=0x55b20289d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b20289d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 -- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028b50b0 msgr2=0x55b2028c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028b50b0 0x55b2028c8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 -- 172.21.15.72:0/813339539 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2025e76b0 con 0x55b2028e9d40 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57e7fff640 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028b50b0 0x55b2028c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57ecf59640 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028a31b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f57e002c540 tx=0x7f57e002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c0f3869e68027e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57cd7fa640 1 -- 172.21.15.72:0/813339539 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0045070 con 0x55b2028e9d40 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57cd7fa640 1 -- 172.21.15.72:0/813339539 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0041440 con 0x55b2028e9d40 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.730+0000 7f57cd7fa640 1 -- 172.21.15.72:0/813339539 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0039040 con 0x55b2028e9d40 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 -- 172.21.15.72:0/813339539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 msgr2=0x55b2028a31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028a31b0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f57e002c540 tx=0x7f57e002e000 comp rx=0 tx=0).stop 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 -- 172.21.15.72:0/813339539 shutdown_connections 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028e9d40 0x55b2028a31b0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028ce600 0x55b20289d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 --2- 172.21.15.72:0/813339539 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028b50b0 0x55b2028c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 -- 172.21.15.72:0/813339539 >> 172.21.15.72:0/813339539 conn(0x55b20289e870 msgr2=0x55b2028d1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 -- 172.21.15.72:0/813339539 shutdown_connections 2024-03-06T07:27:29.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.731+0000 7f57f15a5740 1 -- 172.21.15.72:0/813339539 wait complete. 2024-03-06T07:27:29.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.732+0000 7f57f15a5740 1 -- start start 2024-03-06T07:27:29.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.732+0000 7f57f15a5740 1 Processor -- start 2024-03-06T07:27:29.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 0x55b2028b6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b202958d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028e9d40 0x55b2029592a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2028a2d50 con 0x55b2028b50b0 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b2028a3030 con 0x55b2028ce600 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b2028e6470 con 0x55b2028e9d40 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57f15a5740 10 client.0 _ll_register_callbacks cb 0x7f57dc057ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 0x55b2028b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57e7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 0x55b2028b6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51102/0 (socket says 172.21.15.72:51102) 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57e7fff640 1 -- 172.21.15.72:0/4002912904 learned_addr learned my addr 172.21.15.72:0/4002912904 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.733+0000 7f57e77fe640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b202958d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57ecf59640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028e9d40 0x55b2029592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57e7fff640 1 -- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028e9d40 msgr2=0x55b2029592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57e7fff640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028e9d40 0x55b2029592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57e7fff640 1 -- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 msgr2=0x55b202958d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57e7fff640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b202958d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57e7fff640 1 -- 172.21.15.72:0/4002912904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b2028b6840 con 0x55b2028b50b0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57e7fff640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 0x55b2028b6300 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x55b2028e3780 tx=0x7f57d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca49487df94ace68 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57c6ffd640 1 -- 172.21.15.72:0/4002912904 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d8030060 con 0x55b2028b50b0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57c6ffd640 1 -- 172.21.15.72:0/4002912904 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f57d802b720 con 0x55b2028b50b0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57e0041440 con 0x55b2028b50b0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57c6ffd640 1 -- 172.21.15.72:0/4002912904 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57d8036050 con 0x55b2028b50b0 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 10 client.8259 fetch_fsmap learned FSMap version 72 2024-03-06T07:27:29.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x55b2028e61e0 con 0x55b2028b50b0 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57c6ffd640 1 -- 172.21.15.72:0/4002912904 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0041440 con 0x55b2028b50b0 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57c6ffd640 1 -- 172.21.15.72:0/4002912904 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 72) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f57d8039020 con 0x55b2028b50b0 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 10 client.8259 fetch_fsmap finished waiting for FSMap version 72 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 10 client.8259 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 20 client.8259 populate_metadata read hostname 'smithi072' 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028e4250 0x55b2028c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 4 client.8259 mds_command: new command op to 8217 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.734+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 --> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b20280ff40 con 0x55b2028e4250 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57e77fe640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028e4250 0x55b2028c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57c6ffd640 10 client.8259 ms_handle_connect on v2:172.21.15.72:6834/3088662828 2024-03-06T07:27:29.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57e77fe640 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028e4250 0x55b2028c7a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb1831f19690651f server_cookie=1899a00db94fc8ef in_seq=0 out_seq=0 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57c6ffd640 1 -- 172.21.15.72:0/4002912904 <== mds.0 v2:172.21.15.72:6834/3088662828 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55b20280ff40 con 0x55b2028e4250 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57c6ffd640 10 client.8259 handle_command_reply: tid=0 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 client.8259 shutdown 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 msgr2=0x55b2028b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 0x55b2028b6300 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x55b2028e3780 tx=0x7f57d8004730 comp rx=0 tx=0).stop 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 shutdown_connections 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:6834/3088662828,v1:172.21.15.72:6835/3088662828] conn(0x55b2028e4250 0x55b2028c7a20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55b2028e9d40 0x55b2029592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57c6ffd640 0 client.8259 ms_handle_reset on v2:172.21.15.72:6834/3088662828 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55b2028ce600 0x55b202958d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 --2- 172.21.15.72:0/4002912904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55b2028b50b0 0x55b2028b6300 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 >> 172.21.15.72:0/4002912904 conn(0x55b20289e870 msgr2=0x55b20289d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 shutdown_connections 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 1 -- 172.21.15.72:0/4002912904 wait complete. 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.735+0000 7f57f15a5740 20 client.8259 trim_cache size 0 max 16384 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 8217, 2024-03-06T07:27:29.746 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 4.4534260720000001, 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 72, 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 114, 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 114, 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 4.5459593500000004 2024-03-06T07:27:29.747 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57d406b4d0 msgr2=0x7f57d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57d406b4d0 0x7f57d406d990 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f57d8009560 tx=0x7f57d8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 msgr2=0x7f57e8149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 0x7f57e8149440 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c500 tx=0x7f57dc00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 shutdown_connections 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f57d406b4d0 0x7f57d406d990 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57e815ba50 0x7f57e80afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57e80b4c00 0x7f57e8149440 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 --2- 172.21.15.72:0/1122819288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f57e80b28a0 0x7f57e809e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 >> 172.21.15.72:0/1122819288 conn(0x7f57e809a9f0 msgr2=0x7f57e815afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 shutdown_connections 2024-03-06T07:27:29.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.736+0000 7f57ee9bc640 1 -- 172.21.15.72:0/1122819288 wait complete. 2024-03-06T07:27:29.750 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:27:29.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.849+0000 7f306f478640 1 Processor -- start 2024-03-06T07:27:29.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.849+0000 7f306f478640 1 -- start start 2024-03-06T07:27:29.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306f478640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f30680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306f478640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30680d29c0 0x7f30680b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306f478640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680b3430 con 0x7f30680cf830 2024-03-06T07:27:29.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306f478640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680b31e0 con 0x7f30680d29c0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306f478640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30680b3d60 con 0x7f306815dfe0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30680d29c0 0x7f30680b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30680d29c0 0x7f30680b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51108/0 (socket says 172.21.15.72:51108) 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 -- 172.21.15.72:0/1087649852 learned_addr learned my addr 172.21.15.72:0/1087649852 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306d214640 1 --2- 172.21.15.72:0/1087649852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f30680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 -- 172.21.15.72:0/1087649852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 msgr2=0x7f30680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 --2- 172.21.15.72:0/1087649852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f30680cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 -- 172.21.15.72:0/1087649852 >> v1:172.21.15.146:6789/0 conn(0x7f30680cf830 legacy=0x7f30680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.850+0000 7f306da15640 1 -- 172.21.15.72:0/1087649852 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680b3710 con 0x7f30680d29c0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.851+0000 7f306da15640 1 --2- 172.21.15.72:0/1087649852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680b41f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f3060009e50 tx=0x7f306002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3db8c8a967bbc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.851+0000 7f305bfff640 1 -- 172.21.15.72:0/1087649852 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306002cb00 con 0x7f30680d29c0 2024-03-06T07:27:29.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.851+0000 7f305bfff640 1 -- 172.21.15.72:0/1087649852 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306002cca0 con 0x7f30680d29c0 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.851+0000 7f305bfff640 1 -- 172.21.15.72:0/1087649852 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060034610 con 0x7f30680d29c0 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 -- 172.21.15.72:0/1087649852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 msgr2=0x7f30680b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 --2- 172.21.15.72:0/1087649852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680b41f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f3060009e50 tx=0x7f306002bf60 comp rx=0 tx=0).stop 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 -- 172.21.15.72:0/1087649852 shutdown_connections 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 --2- 172.21.15.72:0/1087649852 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680b41f0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 --2- 172.21.15.72:0/1087649852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f30680cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 -- 172.21.15.72:0/1087649852 >> 172.21.15.72:0/1087649852 conn(0x7f30680d3cd0 msgr2=0x7f30680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 -- 172.21.15.72:0/1087649852 shutdown_connections 2024-03-06T07:27:29.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.852+0000 7f306f478640 1 -- 172.21.15.72:0/1087649852 wait complete. 2024-03-06T07:27:29.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.853+0000 7f306f478640 1 Processor -- start 2024-03-06T07:27:29.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.853+0000 7f306f478640 1 -- start start 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306f478640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680cf830 0x7f30681612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306f478640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306f478640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f306815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306f478640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680d1df0 con 0x7f30680d29c0 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306f478640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680d1c70 con 0x7f30680cf830 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306f478640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306815e3c0 con 0x7f306815dfe0 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51116/0 (socket says 172.21.15.72:51116) 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 -- 172.21.15.72:0/643787645 learned_addr learned my addr 172.21.15.72:0/643787645 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306da15640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f306815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306d214640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680cf830 0x7f30681612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 -- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 msgr2=0x7f306815fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f306815fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 -- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680cf830 msgr2=0x7f30681612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680cf830 0x7f30681612a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 -- 172.21.15.72:0/643787645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068160200 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.854+0000 7f306ca13640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680cdd00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f306400c440 tx=0x7f3064001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.855+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30640052d0 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.855+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3064005470 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.855+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30681604e0 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.855+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064007260 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.855+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3068160aa0 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.856+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306804f330 con 0x7f30680d29c0 2024-03-06T07:27:29.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.856+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3064007400 con 0x7f30680d29c0 2024-03-06T07:27:29.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.857+0000 7f3059ffb640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:29.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.857+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3064089780 con 0x7f30680d29c0 2024-03-06T07:27:29.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.857+0000 7f306d214640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:29.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.857+0000 7f306d214640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f305c006fd0 tx=0x7f305c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:29.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:29.861+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3064058590 con 0x7f30680d29c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.076+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f30680cef00 con 0x7f30680d29c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.076+0000 7f3059ffb640 1 -- 172.21.15.72:0/643787645 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v72) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3064057e90 con 0x7f30680d29c0 2024-03-06T07:27:30.078 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 msgr2=0x7f304c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f305c006fd0 tx=0x7f305c008040 comp rx=0 tx=0).stop 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 msgr2=0x7f30680cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680cdd00 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f306400c440 tx=0x7f3064001e70 comp rx=0 tx=0).stop 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 shutdown_connections 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304c06b4d0 0x7f304c06d990 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f306815dfe0 0x7f306815fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30680d29c0 0x7f30680cdd00 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 --2- 172.21.15.72:0/643787645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30680cf830 0x7f30681612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:30.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 >> 172.21.15.72:0/643787645 conn(0x7f30680d3cd0 msgr2=0x7f306806cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:30.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 shutdown_connections 2024-03-06T07:27:30.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:30.079+0000 7f306f478640 1 -- 172.21.15.72:0/643787645 wait complete. 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:27:30.092 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:30.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.092 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:30.108 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42345sec preferred_lft 42345sec 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.109 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:30.110 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:27:30.111 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:27:30.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.111 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:27:30.174 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:27:30.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.175 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:27:30.230 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:27:30.230 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:27:30.230 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:27:30.231 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:27:30.231 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:27:30.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.231 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:27:30.231 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:27:30.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:27:30.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:27:30.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:27:30.231 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:27:30.285 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-03-06T07:27:30.283+0000 7f2f0ad0a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:27:30.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.328 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:27:30.384 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:27:30.384 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:27:30.384 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:30.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.384 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:27:30.451 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:27:30.451 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:30.451 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:30.505 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:27:30.506 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:27:30.506 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:30.560 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:27:30.628 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:30.685 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:27:30.685 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:27:30.686 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:30.687 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:30.688 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:30.688 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:30.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.688 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:27:30.741 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:27:30.741 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:27:30.784 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:27:30.813 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:27:30.870 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:27:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:27:30.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:30.872 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:27:30.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:27:30.868+0000 7fdf20c93480 -1 init, newargv = 0x5564fdd386a0 newargc=15 2024-03-06T07:27:30.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[74138]: starting ceph client 2024-03-06T07:27:30.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[74138]: starting fuse 2024-03-06T07:27:30.888 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:27:30.889 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:27:30.889 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:27:30.890 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:27:30.891 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:27:31.020 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.74138.asok 2024-03-06T07:27:31.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:27:31.030 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.74138.asok 2024-03-06T07:27:31.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:31.031 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.74138.asok status 2024-03-06T07:27:31.167 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "74138", 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:27:31.168 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 8271, 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 8271 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 999352257 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:27:31.169 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 999352257 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.8271 192.168.0.1:0/999352257", 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/999352257", 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 72, 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 114, 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:27:31.170 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:27:31.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:31.176 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:27:31.176 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:31.191 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:27:31.191 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:31.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:31.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:27:31.257 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:27:31.257 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-03-06T07:27:31.318 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:31.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.458+0000 7f4f068a4640 1 Processor -- start 2024-03-06T07:27:31.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.458+0000 7f4f068a4640 1 -- start start 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4f068a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4f000d22b0 0x7f4f000d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4f068a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f000b41f0 0x7f4f000b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4f068a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f000b3430 con 0x7f4f0015dfe0 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4f068a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f000d2c50 con 0x7f4f000d22b0 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4f068a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000b31e0 con 0x7f4f000b41f0 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4eff7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4f000d22b0 0x7f4f000d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4eff7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4f000d22b0 0x7f4f000d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51132/0 (socket says 172.21.15.72:51132) 2024-03-06T07:27:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.459+0000 7f4eff7fe640 1 -- 172.21.15.72:0/2242728660 learned_addr learned my addr 172.21.15.72:0/2242728660 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4eff7fe640 1 -- 172.21.15.72:0/2242728660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f000b41f0 msgr2=0x7f4f000b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4f04e41640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f000b41f0 0x7f4f000b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4eff7fe640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f000b41f0 0x7f4f000b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4eff7fe640 1 -- 172.21.15.72:0/2242728660 >> v1:172.21.15.72:6789/0 conn(0x7f4f0015dfe0 legacy=0x7f4f000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4eff7fe640 1 -- 172.21.15.72:0/2242728660 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f000b3710 con 0x7f4f000d22b0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4efeffd640 1 -- 172.21.15.72:0/2242728660 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96651431 0 0) 0x7f4f000b3430 con 0x7f4f0015dfe0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4f04e41640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f000b41f0 0x7f4f000b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4eff7fe640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000d22b0 0x7f4f000d2710 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009870 tx=0x7f4ef002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514fc52252f0fd7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.460+0000 7f4efeffd640 1 -- 172.21.15.72:0/2242728660 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0034070 con 0x7f4f000d22b0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4efeffd640 1 -- 172.21.15.72:0/2242728660 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0030840 con 0x7f4f000d22b0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4efeffd640 1 -- 172.21.15.72:0/2242728660 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0030c10 con 0x7f4f000d22b0 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 -- 172.21.15.72:0/2242728660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000d22b0 msgr2=0x7f4f000d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000d22b0 0x7f4f000d2710 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009870 tx=0x7f4ef002b980 comp rx=0 tx=0).stop 2024-03-06T07:27:31.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 -- 172.21.15.72:0/2242728660 shutdown_connections 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f000b41f0 0x7f4f000b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 --2- 172.21.15.72:0/2242728660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000d22b0 0x7f4f000d2710 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 -- 172.21.15.72:0/2242728660 >> 172.21.15.72:0/2242728660 conn(0x7f4f000d3cd0 msgr2=0x7f4f000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 -- 172.21.15.72:0/2242728660 shutdown_connections 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.461+0000 7f4f068a4640 1 -- 172.21.15.72:0/2242728660 wait complete. 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.463+0000 7f4f068a4640 1 Processor -- start 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.463+0000 7f4f068a4640 1 -- start start 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.464+0000 7f4f068a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 0x7f4f0015b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.464+0000 7f4f068a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f000d22b0 0x7f4f001603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.464+0000 7f4f068a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0015dfe0 0x7f4f00160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.464+0000 7f4f068a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000cfa00 con 0x7f4f000b41f0 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.464+0000 7f4f068a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000b3d60 con 0x7f4f000d22b0 2024-03-06T07:27:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4f068a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f000d1c70 con 0x7f4f0015dfe0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 0x7f4f0015b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 0x7f4f0015b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51138/0 (socket says 172.21.15.72:51138) 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 -- 172.21.15.72:0/481303675 learned_addr learned my addr 172.21.15.72:0/481303675 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4eff7fe640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f000d22b0 0x7f4f001603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4f04e41640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0015dfe0 0x7f4f00160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 -- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0015dfe0 msgr2=0x7f4f00160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0015dfe0 0x7f4f00160910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 -- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f000d22b0 msgr2=0x7f4f001603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f000d22b0 0x7f4f001603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 -- 172.21.15.72:0/481303675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f001f3a00 con 0x7f4f000b41f0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efffff640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 0x7f4f0015b350 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c410 tx=0x7f4ef400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4017070 con 0x7f4f000b41f0 2024-03-06T07:27:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef40035f0 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f001f3c90 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef40039e0 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.465+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f001f4250 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.466+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0004f330 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.467+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ef401a070 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.467+0000 7f4efcff9640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eec06b4d0 0x7f4eec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.467+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4ef4089720 con 0x7f4f000b41f0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.467+0000 7f4eff7fe640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eec06b4d0 0x7f4eec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:31.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.468+0000 7f4eff7fe640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eec06b4d0 0x7f4eec06d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0036000 tx=0x7f4ef0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:31.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.472+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ef40584f0 con 0x7f4f000b41f0 2024-03-06T07:27:31.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-06T07:27:31.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.864+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f0006bc50 con 0x7f4f000b41f0 2024-03-06T07:27:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.865+0000 7f4efcff9640 1 -- 172.21.15.72:0/481303675 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f4ef4057df0 con 0x7f4f000b41f0 2024-03-06T07:27:31.867 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:31.867 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:27:31.867 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eec06b4d0 msgr2=0x7f4eec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eec06b4d0 0x7f4eec06d990 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0036000 tx=0x7f4ef0008030 comp rx=0 tx=0).stop 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 msgr2=0x7f4f0015b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 0x7f4f0015b350 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f4ef400c410 tx=0x7f4ef400c8d0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 shutdown_connections 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4eec06b4d0 0x7f4eec06d990 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0015dfe0 0x7f4f00160910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f000d22b0 0x7f4f001603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 --2- 172.21.15.72:0/481303675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4f000b41f0 0x7f4f0015b350 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 >> 172.21.15.72:0/481303675 conn(0x7f4f000d3cd0 msgr2=0x7f4f000b0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 shutdown_connections 2024-03-06T07:27:31.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:31.867+0000 7f4f068a4640 1 -- 172.21.15.72:0/481303675 wait complete. 2024-03-06T07:27:31.881 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:27:32.003 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:27:32.003 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:27:32.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:27:32.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.105+0000 7f070475c640 1 Processor -- start 2024-03-06T07:27:32.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.105+0000 7f070475c640 1 -- start start 2024-03-06T07:27:32.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f070475c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f06fc0d22b0 0x7f06fc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f070475c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f070475c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06fc0b3430 con 0x7f06fc15dfe0 2024-03-06T07:27:32.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f070475c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06fc0d2c50 con 0x7f06fc0d22b0 2024-03-06T07:27:32.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f070475c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0b31e0 con 0x7f06fc0b41f0 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f06fc0d22b0 0x7f06fc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f06fc0d22b0 0x7f06fc0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51150/0 (socket says 172.21.15.72:51150) 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 -- 172.21.15.72:0/2891893795 learned_addr learned my addr 172.21.15.72:0/2891893795 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 -- 172.21.15.72:0/2891893795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 msgr2=0x7f06fc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 --2- 172.21.15.72:0/2891893795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 -- 172.21.15.72:0/2891893795 >> v1:172.21.15.146:6789/0 conn(0x7f06fc15dfe0 legacy=0x7f06fc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:27:32.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.106+0000 7f0701cf7640 1 -- 172.21.15.72:0/2891893795 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc0b3710 con 0x7f06fc0d22b0 2024-03-06T07:27:32.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.107+0000 7f0701cf7640 1 --2- 172.21.15.72:0/2891893795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0d2710 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009870 tx=0x7f06ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1aa7181f35dcd8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.107+0000 7f07014f6640 1 -- 172.21.15.72:0/2891893795 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec035020 con 0x7f06fc0d22b0 2024-03-06T07:27:32.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.107+0000 7f07014f6640 1 -- 172.21.15.72:0/2891893795 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec031640 con 0x7f06fc0d22b0 2024-03-06T07:27:32.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.107+0000 7f07014f6640 1 -- 172.21.15.72:0/2891893795 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec02d040 con 0x7f06fc0d22b0 2024-03-06T07:27:32.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.107+0000 7f070475c640 1 -- 172.21.15.72:0/2891893795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 msgr2=0x7f06fc0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.107+0000 7f070475c640 1 --2- 172.21.15.72:0/2891893795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0d2710 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f06ec009870 tx=0x7f06ec02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.108+0000 7f070475c640 1 -- 172.21.15.72:0/2891893795 shutdown_connections 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.108+0000 7f070475c640 1 --2- 172.21.15.72:0/2891893795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.108+0000 7f070475c640 1 --2- 172.21.15.72:0/2891893795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0d2710 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.108+0000 7f070475c640 1 -- 172.21.15.72:0/2891893795 >> 172.21.15.72:0/2891893795 conn(0x7f06fc0d3cd0 msgr2=0x7f06fc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.108+0000 7f070475c640 1 -- 172.21.15.72:0/2891893795 shutdown_connections 2024-03-06T07:27:32.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.108+0000 7f070475c640 1 -- 172.21.15.72:0/2891893795 wait complete. 2024-03-06T07:27:32.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 Processor -- start 2024-03-06T07:27:32.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 -- start start 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06fc15dfe0 0x7f06fc14eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0cfa00 con 0x7f06fc0d22b0 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0b3d60 con 0x7f06fc15dfe0 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f070475c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc0d1c70 con 0x7f06fc0b41f0 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.109+0000 7f0701cf7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51164/0 (socket says 172.21.15.72:51164) 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 -- 172.21.15.72:0/3082831143 learned_addr learned my addr 172.21.15.72:0/3082831143 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f07024f8640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 -- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 msgr2=0x7f06fc0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0702cf9640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06fc15dfe0 0x7f06fc14eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 -- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06fc15dfe0 msgr2=0x7f06fc14eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06fc15dfe0 0x7f06fc14eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 -- 172.21.15.72:0/3082831143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc0cc910 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f07024f8640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0702cf9640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06fc15dfe0 0x7f06fc14eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f0701cf7640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0cd9b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f06ec02b930 tx=0x7f06ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec03ac60 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec03ae20 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc0ccbf0 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec044630 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.110+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06fc0cd1b0 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.111+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06fc04f330 con 0x7f06fc0d22b0 2024-03-06T07:27:32.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.112+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06ec038020 con 0x7f06fc0d22b0 2024-03-06T07:27:32.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.112+0000 7f06f2ffd640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06e006b4d0 0x7f06e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:27:32.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.112+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f06ec0ac0c0 con 0x7f06fc0d22b0 2024-03-06T07:27:32.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.112+0000 7f07024f8640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06e006b4d0 0x7f06e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:27:32.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.112+0000 7f07024f8640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06e006b4d0 0x7f06e006d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f06f800a810 tx=0x7f06f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:27:32.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.116+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06ec03f070 con 0x7f06fc0d22b0 2024-03-06T07:27:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.361+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06fc0ce9b0 con 0x7f06fc0d22b0 2024-03-06T07:27:32.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.361+0000 7f06f2ffd640 1 -- 172.21.15.72:0/3082831143 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f06ec07afb0 con 0x7f06fc0d22b0 2024-03-06T07:27:32.364 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:27:32.364 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:27:32.364 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06e006b4d0 msgr2=0x7f06e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06e006b4d0 0x7f06e006d990 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f06f800a810 tx=0x7f06f8008040 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 msgr2=0x7f06fc0cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0cd9b0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f06ec02b930 tx=0x7f06ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 shutdown_connections 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06e006b4d0 0x7f06e006d990 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06fc15dfe0 0x7f06fc14eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06fc0d22b0 0x7f06fc0cd9b0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 --2- 172.21.15.72:0/3082831143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06fc0b41f0 0x7f06fc0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 >> 172.21.15.72:0/3082831143 conn(0x7f06fc0d3cd0 msgr2=0x7f06fc06caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 shutdown_connections 2024-03-06T07:27:32.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:27:32.363+0000 7f070475c640 1 -- 172.21.15.72:0/3082831143 wait complete. 2024-03-06T07:27:32.376 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:27:32.536 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:27:32.536 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:27:51.245 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:27:51.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:02.538 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.577+0000 7f1fdc15ff40 1 Processor -- start 2024-03-06T07:28:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.577+0000 7f1fdc15ff40 1 -- start start 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fdc15ff40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fdc15ff40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511aaddab0 0x56511ab67ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fdc15ff40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 0x56511aae0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fdc15ff40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511aae33b0 con 0x56511ab6aee0 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fdc15ff40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511aa40b70 con 0x56511ab652c0 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fdc15ff40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56511aae3650 con 0x56511aaddab0 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 0x56511aae0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 0x56511aae0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54476/0 (socket says 172.21.15.72:54476) 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 -- 172.21.15.72:0/1306955297 learned_addr learned my addr 172.21.15.72:0/1306955297 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:02.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fd9461640 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511aaddab0 0x56511ab67ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 -- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511aaddab0 msgr2=0x56511ab67ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511aaddab0 0x56511ab67ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 -- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 msgr2=0x56511aadd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 -- 172.21.15.72:0/1306955297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56511ab68bc0 con 0x56511ab6aee0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.578+0000 7f1fda463640 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 0x56511aae0cc0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x56511aadb9c0 tx=0x7f1fd4005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8b2858f3e1f5fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fcbfff640 1 -- 172.21.15.72:0/1306955297 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4012400 con 0x56511ab6aee0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fcbfff640 1 -- 172.21.15.72:0/1306955297 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd4018070 con 0x56511ab6aee0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fcbfff640 1 -- 172.21.15.72:0/1306955297 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd40127c0 con 0x56511ab6aee0 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/1306955297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 msgr2=0x56511aae0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 0x56511aae0cc0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x56511aadb9c0 tx=0x7f1fd4005d80 comp rx=0 tx=0).stop 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/1306955297 shutdown_connections 2024-03-06T07:28:02.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511ab6aee0 0x56511aae0cc0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511aaddab0 0x56511ab67ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/1306955297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/1306955297 >> 172.21.15.72:0/1306955297 conn(0x56511aac4780 msgr2=0x56511aadc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/1306955297 shutdown_connections 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.579+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/1306955297 wait complete. 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 Processor -- start 2024-03-06T07:28:02.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 -- start start 2024-03-06T07:28:02.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511ab6aee0 0x56511ab50e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 0x56511aac9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511aadcbd0 con 0x56511aac8ba0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511aac97a0 con 0x56511ab652c0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.580+0000 7f1fdc15ff40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x56511aac9a40 con 0x56511ab6aee0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 0x56511aac9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 0x56511aac9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54490/0 (socket says 172.21.15.72:54490) 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 learned_addr learned my addr 172.21.15.72:0/3282611343 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511ab6aee0 0x56511ab50e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511ab6aee0 msgr2=0x56511ab50e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x56511ab6aee0 0x56511ab50e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 msgr2=0x56511aadf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x56511ab652c0 0x56511aadf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56511ab669b0 con 0x56511aac8ba0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 0x56511aac9020 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4015040 tx=0x7f1fd4003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fcaffd640 1 -- 172.21.15.72:0/3282611343 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd401c6a0 con 0x56511aac8ba0 2024-03-06T07:28:02.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fcaffd640 1 -- 172.21.15.72:0/3282611343 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd4021070 con 0x56511aac8ba0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56511ab66c90 con 0x56511aac8ba0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fcaffd640 1 -- 172.21.15.72:0/3282611343 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd401ca80 con 0x56511aac8ba0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.581+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56511ab67250 con 0x56511aac8ba0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.583+0000 7f1fcaffd640 1 -- 172.21.15.72:0/3282611343 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fd4011070 con 0x56511aac8ba0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.583+0000 7f1fcaffd640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fbc069ee0 0x7f1fbc06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.583+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fbc069ee0 0x7f1fbc06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.583+0000 7f1fcaffd640 1 -- 172.21.15.72:0/3282611343 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f1fd408a1e0 con 0x56511aac8ba0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.583+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x56511ab4eed0 0x56511ab4f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.583+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.584+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fbc069ee0 0x7f1fbc06c3a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc003130 tx=0x7f1fcc0014b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.584+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x56511ab4eed0 0x56511ab4f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.584+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x56511ab4eed0 0x56511ab4f2b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 1 ==== osd_op_reply(1 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f1fd005f950 0x7f1fd0006440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f1fd005f950 0x7f1fd0006440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f1fd005f950 0x7f1fd0006440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 1 ==== osd_op_reply(2 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc00da90 con 0x7f1fd005f950 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f1fcc0183e0 0x7f1fcc018830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.585+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f1fcc0183e0 0x7f1fcc018830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.586+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f1fcc0183e0 0x7f1fcc018830 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.586+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 1 ==== osd_op_reply(3 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.586+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1fd005e3d0 0x7f1fd005e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.586+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.586+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1fd005e3d0 0x7f1fd005e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1fd005e3d0 0x7f1fd005e810 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== osd_op_reply(4 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 2 ==== osd_op_reply(5 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 2 ==== osd_op_reply(6 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.587+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.588+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 2 ==== osd_op_reply(7 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.588+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.588+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 3 ==== osd_op_reply(8 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.588+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.589+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 3 ==== osd_op_reply(9 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.589+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.589+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 3 ==== osd_op_reply(10 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.589+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.589+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 4 ==== osd_op_reply(11 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.589+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.590+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 4 ==== osd_op_reply(12 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.590+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.590+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 4 ==== osd_op_reply(13 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.590+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1fd0070040 0x7f1fd0070930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.590+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.590+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1fd0070040 0x7f1fd0070930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.591+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1fd0070040 0x7f1fd0070930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.591+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 1 ==== osd_op_reply(14 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc00da90 con 0x7f1fd0070040 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.591+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.591+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 2 ==== osd_op_reply(15 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc00da90 con 0x7f1fd005f950 2024-03-06T07:28:02.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.591+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.592+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 5 ==== osd_op_reply(16 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.592+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.592+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 2 ==== osd_op_reply(17 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc00da90 con 0x7f1fd0070040 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.592+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f1fcc02b040 0x7f1fcc02a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.592+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc02b040 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.592+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f1fcc02b040 0x7f1fcc02a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.593+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f1fcc02b040 0x7f1fcc02a3b0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.593+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.5 v2:172.21.15.146:6800/2545401618 1 ==== osd_op_reply(18 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc02b040 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.593+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.593+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 5 ==== osd_op_reply(19 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.593+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 5 ==== osd_op_reply(20 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 6 ==== osd_op_reply(21 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 6 ==== osd_op_reply(22 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f1fd007f3a0 0x7f1fd007f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.594+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd007f3a0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.595+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f1fd007f3a0 0x7f1fd007f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.595+0000 7f1fda463640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f1fd007f3a0 0x7f1fd007f820 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.595+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.2 v2:172.21.15.72:6817/3054266462 1 ==== osd_op_reply(23 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd007f3a0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.595+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.595+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 3 ==== osd_op_reply(24 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc00da90 con 0x7f1fd005f950 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.595+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f1fcc00b360 0x7f1fcc00b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.596+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc00b360 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.596+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f1fcc00b360 0x7f1fcc00b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.596+0000 7f1fd9c62640 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f1fcc00b360 0x7f1fcc00b7e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.597+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.6 v2:172.21.15.146:6808/1113968375 1 ==== osd_op_reply(25 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fcc00b360 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.597+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.597+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 6 ==== osd_op_reply(26 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.597+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc02b040 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.597+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.5 v2:172.21.15.146:6800/2545401618 2 ==== osd_op_reply(27 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc02b040 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.597+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.598+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 7 ==== osd_op_reply(28 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.598+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd007f3a0 2024-03-06T07:28:02.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.598+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.2 v2:172.21.15.72:6817/3054266462 2 ==== osd_op_reply(29 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd007f3a0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.598+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc00b360 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.598+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.6 v2:172.21.15.146:6808/1113968375 2 ==== osd_op_reply(30 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fcc00b360 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.598+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 4 ==== osd_op_reply(31 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd005f950 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 3 ==== osd_op_reply(32 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 8 ==== osd_op_reply(33 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 9 ==== osd_op_reply(34 [pgnls start_epoch 116 out=49b] v110'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.599+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.600+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 7 ==== osd_op_reply(35 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.600+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.600+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 5 ==== osd_op_reply(36 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd005f950 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.600+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.600+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 8 ==== osd_op_reply(37 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.600+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.601+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 4 ==== osd_op_reply(38 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.601+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc02b040 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.601+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.5 v2:172.21.15.146:6800/2545401618 3 ==== osd_op_reply(39 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc02b040 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.601+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc02b040 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.601+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.5 v2:172.21.15.146:6800/2545401618 4 ==== osd_op_reply(40 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc02b040 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.601+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc00b360 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.602+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.6 v2:172.21.15.146:6808/1113968375 3 ==== osd_op_reply(41 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fcc00b360 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.602+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.602+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 6 ==== osd_op_reply(42 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd005f950 2024-03-06T07:28:02.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.602+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc02b040 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.602+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.5 v2:172.21.15.146:6800/2545401618 5 ==== osd_op_reply(43 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc02b040 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.602+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.603+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 5 ==== osd_op_reply(44 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.603+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.603+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 7 ==== osd_op_reply(45 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.603+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.603+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 8 ==== osd_op_reply(46 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.603+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.604+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 9 ==== osd_op_reply(47 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.604+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.604+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 10 ==== osd_op_reply(48 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.604+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.604+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 6 ==== osd_op_reply(49 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.604+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.605+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 7 ==== osd_op_reply(50 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.605+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc0183e0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.605+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.3 v2:172.21.15.72:6808/4245662075 9 ==== osd_op_reply(51 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x7f1fcc0183e0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.605+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x56511ab4eed0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.605+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 <== osd.7 v2:172.21.15.146:6816/1306818419 11 ==== osd_op_reply(52 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd005a040 con 0x56511ab4eed0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.605+0000 7f1fd9461640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.606+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 10 ==== osd_op_reply(53 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.606+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.606+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 8 ==== osd_op_reply(54 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.606+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.606+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 9 ==== osd_op_reply(55 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.606+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd007f3a0 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.2 v2:172.21.15.72:6817/3054266462 3 ==== osd_op_reply(56 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd007f3a0 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 11 ==== osd_op_reply(57 [pgnls start_epoch 116 out=81b] v110'8 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 10 ==== osd_op_reply(58 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc00b360 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.607+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.6 v2:172.21.15.146:6808/1113968375 4 ==== osd_op_reply(59 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fcc00b360 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.608+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.608+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 12 ==== osd_op_reply(60 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.608+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005f950 2024-03-06T07:28:02.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.608+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.1 v2:172.21.15.72:6800/2878132174 7 ==== osd_op_reply(61 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd005f950 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.608+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd0070040 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.4 v2:172.21.15.146:6820/3084745005 11 ==== osd_op_reply(62 [pgnls start_epoch 116 out=81b] v110'9 uv1 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fd0070040 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fcc00b360 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 <== osd.6 v2:172.21.15.146:6808/1113968375 5 ==== osd_op_reply(63 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fcc039040 con 0x7f1fcc00b360 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fd9c62640 1 -- 172.21.15.72:0/3282611343 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x56511ac14900 con 0x7f1fd005e3d0 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 <== osd.0 v2:172.21.15.72:6816/4178159476 13 ==== osd_op_reply(64 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f1fd402f070 con 0x7f1fd005e3d0 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stdout:10000000001.00000000 2024-03-06T07:28:02.615 INFO:teuthology.orchestra.run.smithi072.stdout:10000000001.00000001 2024-03-06T07:28:02.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1fd005e3d0 msgr2=0x7f1fd005e810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1fd005e3d0 0x7f1fd005e810 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f1fd005f950 msgr2=0x7f1fd0006440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f1fd005f950 0x7f1fd0006440 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f1fd007f3a0 msgr2=0x7f1fd007f820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f1fd007f3a0 0x7f1fd007f820 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f1fcc0183e0 msgr2=0x7f1fcc018830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f1fcc0183e0 0x7f1fcc018830 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 reap_dead start 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1fd0070040 msgr2=0x7f1fd0070930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1fd0070040 0x7f1fd0070930 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f1fcc02b040 msgr2=0x7f1fcc02a3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f1fcc02b040 0x7f1fcc02a3b0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f1fcc00b360 msgr2=0x7f1fcc00b7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f1fcc00b360 0x7f1fcc00b7e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x56511ab4eed0 msgr2=0x56511ab4f2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x56511ab4eed0 0x56511ab4f2b0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.609+0000 7f1fda463640 1 -- 172.21.15.72:0/3282611343 reap_dead start 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fbc069ee0 msgr2=0x7f1fbc06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fbc069ee0 0x7f1fbc06c3a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc003130 tx=0x7f1fcc0014b0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 msgr2=0x56511aac9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 0x56511aac9020 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4015040 tx=0x7f1fd4003330 comp rx=0 tx=0).stop 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 shutdown_connections 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fbc069ee0 0x7f1fbc06c3a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 --2- 172.21.15.72:0/3282611343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56511aac8ba0 0x56511aac9020 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 >> 172.21.15.72:0/3282611343 conn(0x56511aac4780 msgr2=0x56511aa97b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 shutdown_connections 2024-03-06T07:28:02.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.610+0000 7f1fdc15ff40 1 -- 172.21.15.72:0/3282611343 wait complete. 2024-03-06T07:28:02.620 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-03-06T07:28:02.620 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:02.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.758+0000 7f9087239640 1 Processor -- start 2024-03-06T07:28:02.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.758+0000 7f9087239640 1 -- start start 2024-03-06T07:28:02.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9087239640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f90801424f0 0x7f90800cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9087239640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90800cd7a0 0x7f908015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9087239640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90800cc980 con 0x7f908006d330 2024-03-06T07:28:02.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9087239640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90800cd220 con 0x7f90800cd7a0 2024-03-06T07:28:02.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9087239640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90800ccc60 con 0x7f90801424f0 2024-03-06T07:28:02.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90800cd7a0 0x7f908015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f90800cd7a0 0x7f908015cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54496/0 (socket says 172.21.15.72:54496) 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 -- 172.21.15.72:0/3419215183 learned_addr learned my addr 172.21.15.72:0/3419215183 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9084fd5640 1 --2- 172.21.15.72:0/3419215183 >> v2:172.21.15.146:3300/0 conn(0x7f90801424f0 0x7f90800cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 -- 172.21.15.72:0/3419215183 >> v2:172.21.15.146:3300/0 conn(0x7f90801424f0 msgr2=0x7f90800cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 --2- 172.21.15.72:0/3419215183 >> v2:172.21.15.146:3300/0 conn(0x7f90801424f0 0x7f90800cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 -- 172.21.15.72:0/3419215183 >> v1:172.21.15.146:6789/0 conn(0x7f908006d330 legacy=0x7f9080071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9077fff640 1 -- 172.21.15.72:0/3419215183 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90800ccf40 con 0x7f90800cd7a0 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.759+0000 7f9084fd5640 1 --2- 172.21.15.72:0/3419215183 >> v2:172.21.15.146:3300/0 conn(0x7f90801424f0 0x7f90800cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.760+0000 7f9077fff640 1 --2- 172.21.15.72:0/3419215183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800cd7a0 0x7f908015cef0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f908006c6d0 tx=0x7f907002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20cc6877148bbc57 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.760+0000 7f90777fe640 1 -- 172.21.15.72:0/3419215183 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070034070 con 0x7f90800cd7a0 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.760+0000 7f90777fe640 1 -- 172.21.15.72:0/3419215183 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9070030860 con 0x7f90800cd7a0 2024-03-06T07:28:02.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.760+0000 7f90777fe640 1 -- 172.21.15.72:0/3419215183 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070030c30 con 0x7f90800cd7a0 2024-03-06T07:28:02.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 -- 172.21.15.72:0/3419215183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800cd7a0 msgr2=0x7f908015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 --2- 172.21.15.72:0/3419215183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800cd7a0 0x7f908015cef0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f908006c6d0 tx=0x7f907002b9a0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 -- 172.21.15.72:0/3419215183 shutdown_connections 2024-03-06T07:28:02.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 --2- 172.21.15.72:0/3419215183 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90800cd7a0 0x7f908015cef0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 --2- 172.21.15.72:0/3419215183 >> v2:172.21.15.146:3300/0 conn(0x7f90801424f0 0x7f90800cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 -- 172.21.15.72:0/3419215183 >> 172.21.15.72:0/3419215183 conn(0x7f90800cf8a0 msgr2=0x7f90800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 -- 172.21.15.72:0/3419215183 shutdown_connections 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.761+0000 7f9087239640 1 -- 172.21.15.72:0/3419215183 wait complete. 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.762+0000 7f9087239640 1 Processor -- start 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.762+0000 7f9087239640 1 -- start start 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9087239640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 0x7f908013f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9087239640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90800cd7a0 0x7f90801532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9087239640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90801424f0 0x7f9080153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9087239640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908006d920 con 0x7f908006d330 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9087239640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f908006d7a0 con 0x7f90800cd7a0 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9087239640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f908006daa0 con 0x7f90801424f0 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 0x7f908013f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 0x7f908013f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54504/0 (socket says 172.21.15.72:54504) 2024-03-06T07:28:02.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 -- 172.21.15.72:0/2800780426 learned_addr learned my addr 172.21.15.72:0/2800780426 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9077fff640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90800cd7a0 0x7f90801532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f90857d6640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90801424f0 0x7f9080153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 -- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90801424f0 msgr2=0x7f9080153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90801424f0 0x7f9080153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 -- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90800cd7a0 msgr2=0x7f90801532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90800cd7a0 0x7f90801532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 -- 172.21.15.72:0/2800780426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90801f6090 con 0x7f908006d330 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.763+0000 7f9084fd5640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 0x7f908013f230 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f907c00d7d0 tx=0x7f907c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.764+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c004030 con 0x7f908006d330 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.764+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c0041d0 con 0x7f908006d330 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.764+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90801f52d0 con 0x7f908006d330 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.764+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c0096a0 con 0x7f908006d330 2024-03-06T07:28:02.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.764+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90801f5890 con 0x7f908006d330 2024-03-06T07:28:02.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.765+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f908004f330 con 0x7f908006d330 2024-03-06T07:28:02.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.765+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f907c009840 con 0x7f908006d330 2024-03-06T07:28:02.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.766+0000 7f90757fa640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906c06b4d0 0x7f906c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:02.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.766+0000 7f9077fff640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906c06b4d0 0x7f906c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:02.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.766+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f907c08a3e0 con 0x7f908006d330 2024-03-06T07:28:02.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.766+0000 7f9077fff640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906c06b4d0 0x7f906c06d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9070036000 tx=0x7f9070008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:02.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.770+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f907c058b60 con 0x7f908006d330 2024-03-06T07:28:02.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.987+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90800cde40 con 0x7f908006d330 2024-03-06T07:28:02.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.987+0000 7f90757fa640 1 -- 172.21.15.72:0/2800780426 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f907c05cc00 con 0x7f908006d330 2024-03-06T07:28:02.989 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:02.990 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:02.990 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906c06b4d0 msgr2=0x7f906c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906c06b4d0 0x7f906c06d990 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9070036000 tx=0x7f9070008030 comp rx=0 tx=0).stop 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 msgr2=0x7f908013f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 0x7f908013f230 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f907c00d7d0 tx=0x7f907c00dc90 comp rx=0 tx=0).stop 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 shutdown_connections 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f906c06b4d0 0x7f906c06d990 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90801424f0 0x7f9080153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90800cd7a0 0x7f90801532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 --2- 172.21.15.72:0/2800780426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f908006d330 0x7f908013f230 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 >> 172.21.15.72:0/2800780426 conn(0x7f90800cf8a0 msgr2=0x7f908015f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 shutdown_connections 2024-03-06T07:28:02.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:02.989+0000 7f9087239640 1 -- 172.21.15.72:0/2800780426 wait complete. 2024-03-06T07:28:03.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:03.146 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:03.147 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:03.147 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:03.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.246+0000 7f4c93215640 1 Processor -- start 2024-03-06T07:28:03.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c93215640 1 -- start start 2024-03-06T07:28:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c93215640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c1515e0 0x7f4c8c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c93215640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c8c0cd7a0 0x7f4c8c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c93215640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c071f40 con 0x7f4c8c15b430 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c93215640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c15b8b0 con 0x7f4c8c0cd7a0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c93215640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c070a70 con 0x7f4c8c1515e0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c917b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c8c0cd7a0 0x7f4c8c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c917b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c8c0cd7a0 0x7f4c8c0cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54506/0 (socket says 172.21.15.72:54506) 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.247+0000 7f4c917b2640 1 -- 172.21.15.72:0/743392932 learned_addr learned my addr 172.21.15.72:0/743392932 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c90fb1640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c1515e0 0x7f4c8c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c917b2640 1 -- 172.21.15.72:0/743392932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c1515e0 msgr2=0x7f4c8c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c917b2640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c1515e0 0x7f4c8c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c917b2640 1 -- 172.21.15.72:0/743392932 >> v1:172.21.15.146:6789/0 conn(0x7f4c8c15b430 legacy=0x7f4c8c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c917b2640 1 -- 172.21.15.72:0/743392932 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c070d50 con 0x7f4c8c0cd7a0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c90fb1640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c1515e0 0x7f4c8c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.248+0000 7f4c917b2640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c0cdbd0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009870 tx=0x7f4c7c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8633c6dec9bdad04 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c8b7fe640 1 -- 172.21.15.72:0/743392932 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c035020 con 0x7f4c8c0cd7a0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c8b7fe640 1 -- 172.21.15.72:0/743392932 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c031640 con 0x7f4c8c0cd7a0 2024-03-06T07:28:03.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c8b7fe640 1 -- 172.21.15.72:0/743392932 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c02d040 con 0x7f4c8c0cd7a0 2024-03-06T07:28:03.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 -- 172.21.15.72:0/743392932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c0cd7a0 msgr2=0x7f4c8c0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c0cdbd0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009870 tx=0x7f4c7c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 -- 172.21.15.72:0/743392932 shutdown_connections 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c0cdbd0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 --2- 172.21.15.72:0/743392932 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c1515e0 0x7f4c8c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 -- 172.21.15.72:0/743392932 >> 172.21.15.72:0/743392932 conn(0x7f4c8c0cf8a0 msgr2=0x7f4c8c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 -- 172.21.15.72:0/743392932 shutdown_connections 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.249+0000 7f4c93215640 1 -- 172.21.15.72:0/743392932 wait complete. 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.251+0000 7f4c93215640 1 Processor -- start 2024-03-06T07:28:03.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.251+0000 7f4c93215640 1 -- start start 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c93215640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c93215640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 0x7f4c8c0c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c93215640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c15b430 0x7f4c8c14b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c93215640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0ce110 con 0x7f4c8c1515e0 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c93215640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c071310 con 0x7f4c8c0cd7a0 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c93215640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0b3f90 con 0x7f4c8c15b430 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.252+0000 7f4c8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 0x7f4c8c0c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 0x7f4c8c0c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54508/0 (socket says 172.21.15.72:54508) 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 -- 172.21.15.72:0/2210912547 learned_addr learned my addr 172.21.15.72:0/2210912547 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c90fb1640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c917b2640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c15b430 0x7f4c8c14b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 -- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c15b430 msgr2=0x7f4c8c14b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c15b430 0x7f4c8c14b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 -- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8c0cd7a0 msgr2=0x7f4c8c150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c150460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 -- 172.21.15.72:0/2210912547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c0c64b0 con 0x7f4c8c1515e0 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c8bfff640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 0x7f4c8c0c59f0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d990 tx=0x7f4c7400de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7401b070 con 0x7f4c8c1515e0 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7400ed60 con 0x7f4c8c1515e0 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c0c6790 con 0x7f4c8c1515e0 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74016680 con 0x7f4c8c1515e0 2024-03-06T07:28:03.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.253+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c0c6d50 con 0x7f4c8c1515e0 2024-03-06T07:28:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.254+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c0ca180 con 0x7f4c8c1515e0 2024-03-06T07:28:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.255+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c7401e070 con 0x7f4c8c1515e0 2024-03-06T07:28:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.255+0000 7f4c897fa640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.255+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4c7408c000 con 0x7f4c8c1515e0 2024-03-06T07:28:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.255+0000 7f4c90fb1640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.255+0000 7f4c90fb1640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4c840075b0 tx=0x7f4c84007540 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.259+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c7405add0 con 0x7f4c8c1515e0 2024-03-06T07:28:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.475+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c8c068840 con 0x7f4c8c1515e0 2024-03-06T07:28:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.476+0000 7f4c897fa640 1 -- 172.21.15.72:0/2210912547 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f4c7405a6d0 con 0x7f4c8c1515e0 2024-03-06T07:28:03.479 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:03.479 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:03.479 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 msgr2=0x7f4c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4c840075b0 tx=0x7f4c84007540 comp rx=0 tx=0).stop 2024-03-06T07:28:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 msgr2=0x7f4c8c0c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 0x7f4c8c0c59f0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400d990 tx=0x7f4c7400de50 comp rx=0 tx=0).stop 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 shutdown_connections 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8c15b430 0x7f4c8c14b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8c1515e0 0x7f4c8c0c59f0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 --2- 172.21.15.72:0/2210912547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8c0cd7a0 0x7f4c8c150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 >> 172.21.15.72:0/2210912547 conn(0x7f4c8c0cf8a0 msgr2=0x7f4c8c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 shutdown_connections 2024-03-06T07:28:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.478+0000 7f4c93215640 1 -- 172.21.15.72:0/2210912547 wait complete. 2024-03-06T07:28:03.494 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:03.622 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:28:03.622 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:03.622 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.662+0000 7ff4a0237f40 1 Processor -- start 2024-03-06T07:28:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.662+0000 7ff4a0237f40 1 -- start start 2024-03-06T07:28:03.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff4a0237f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b7de590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff4a0237f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b726d80 0x55cf5b753c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff4a0237f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b7e3570 0x55cf5b7e06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff4a0237f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf5b727190 con 0x55cf5b7246c0 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff4a0237f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf5b6ccb70 con 0x55cf5b7e3570 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff4a0237f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf5b725c80 con 0x55cf5b726d80 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.663+0000 7ff49d5c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b7de590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b7de590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54528/0 (socket says 172.21.15.72:54528) 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 -- 172.21.15.72:0/319801710 learned_addr learned my addr 172.21.15.72:0/319801710 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49ddc3640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b7e3570 0x55cf5b7e06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff497fff640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b726d80 0x55cf5b753c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 -- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b726d80 msgr2=0x55cf5b753c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b726d80 0x55cf5b753c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 -- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b7e3570 msgr2=0x55cf5b7e06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b7e3570 0x55cf5b7e06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 -- 172.21.15.72:0/319801710 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf5b764c00 con 0x55cf5b7246c0 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff497fff640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b726d80 0x55cf5b753c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49ddc3640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b7e3570 0x55cf5b7e06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff49d5c2640 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b7de590 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff488058040 tx=0x7ff48804f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867b8a341770e485 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff4977fe640 1 -- 172.21.15.72:0/319801710 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48805a040 con 0x55cf5b7246c0 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff4977fe640 1 -- 172.21.15.72:0/319801710 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488066070 con 0x55cf5b7246c0 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.664+0000 7ff4977fe640 1 -- 172.21.15.72:0/319801710 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48804dd50 con 0x55cf5b7246c0 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 -- 172.21.15.72:0/319801710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 msgr2=0x55cf5b7de590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b7de590 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff488058040 tx=0x7ff48804f000 comp rx=0 tx=0).stop 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 -- 172.21.15.72:0/319801710 shutdown_connections 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b7e3570 0x55cf5b7e06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b726d80 0x55cf5b753c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/319801710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b7de590 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 -- 172.21.15.72:0/319801710 >> 172.21.15.72:0/319801710 conn(0x55cf5b751350 msgr2=0x55cf5b754970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 -- 172.21.15.72:0/319801710 shutdown_connections 2024-03-06T07:28:03.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 -- 172.21.15.72:0/319801710 wait complete. 2024-03-06T07:28:03.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 Processor -- start 2024-03-06T07:28:03.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.665+0000 7ff4a0237f40 1 -- start start 2024-03-06T07:28:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff4a0237f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b76e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff4a0237f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b726d80 0x55cf5b754590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff4a0237f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b7e3570 0x55cf5b754ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff4a0237f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf5b7dead0 con 0x55cf5b7246c0 2024-03-06T07:28:03.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff4a0237f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf5b74f570 con 0x55cf5b726d80 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff4a0237f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf5b7f9d20 con 0x55cf5b7e3570 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b76e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b76e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54544/0 (socket says 172.21.15.72:54544) 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 -- 172.21.15.72:0/692831231 learned_addr learned my addr 172.21.15.72:0/692831231 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff497fff640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b726d80 0x55cf5b754590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49ddc3640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b7e3570 0x55cf5b754ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 -- 172.21.15.72:0/692831231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b7e3570 msgr2=0x55cf5b754ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55cf5b7e3570 0x55cf5b754ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 -- 172.21.15.72:0/692831231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b726d80 msgr2=0x55cf5b754590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55cf5b726d80 0x55cf5b754590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 -- 172.21.15.72:0/692831231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf5b7e0fc0 con 0x55cf5b7246c0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.666+0000 7ff49d5c2640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b76e010 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff488063040 tx=0x7ff488002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.667+0000 7ff4967fc640 1 -- 172.21.15.72:0/692831231 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48805a040 con 0x55cf5b7246c0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.667+0000 7ff4967fc640 1 -- 172.21.15.72:0/692831231 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488073070 con 0x55cf5b7246c0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.667+0000 7ff4967fc640 1 -- 172.21.15.72:0/692831231 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488055440 con 0x55cf5b7246c0 2024-03-06T07:28:03.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.667+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cf5b7de170 con 0x55cf5b7246c0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.667+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf5b7e0200 con 0x55cf5b7246c0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.668+0000 7ff4967fc640 1 -- 172.21.15.72:0/692831231 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff4880555e0 con 0x55cf5b7246c0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.668+0000 7ff4967fc640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff484069eb0 0x7ff48406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.668+0000 7ff497fff640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff484069eb0 0x7ff48406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.668+0000 7ff4967fc640 1 -- 172.21.15.72:0/692831231 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ff4880ce4d0 con 0x55cf5b7246c0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55cf5b768f70 0x55cf5b769350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x55cf5b769890 con 0x55cf5b768f70 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff49ddc3640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55cf5b768f70 0x55cf5b769350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff497fff640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff484069eb0 0x7ff48406c370 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55cf5b7e26c0 tx=0x7ff490005730 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff49ddc3640 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55cf5b768f70 0x55cf5b769350 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff49ddc3640 1 -- 172.21.15.72:0/692831231 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+99 (crc 0 0 0) 0x7ff498011040 con 0x55cf5b768f70 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55cf5b768f70 msgr2=0x55cf5b769350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.669+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55cf5b768f70 0x55cf5b769350 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff484069eb0 msgr2=0x7ff48406c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff484069eb0 0x7ff48406c370 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x55cf5b7e26c0 tx=0x7ff490005730 comp rx=0 tx=0).stop 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 msgr2=0x55cf5b76e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 --2- 172.21.15.72:0/692831231 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cf5b7246c0 0x55cf5b76e010 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff488063040 tx=0x7ff488002fb0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff49ddc3640 1 -- 172.21.15.72:0/692831231 reap_dead start 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 shutdown_connections 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 >> 172.21.15.72:0/692831231 conn(0x55cf5b751350 msgr2=0x55cf5b889310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 shutdown_connections 2024-03-06T07:28:03.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.670+0000 7ff4a0237f40 1 -- 172.21.15.72:0/692831231 wait complete. 2024-03-06T07:28:03.677 INFO:teuthology.orchestra.run.smithi072.stdout:] 10000000001 stray0 2024-03-06T07:28:03.677 DEBUG:teuthology.orchestra.run.smithi072:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:03.763 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:28:03.763 INFO:teuthology.orchestra.run.smithi072.stdout: "ino": 1099511627777, 2024-03-06T07:28:03.763 INFO:teuthology.orchestra.run.smithi072.stdout: "ancestors": [ 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1536, 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "10000000001", 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 11 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 256, 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "stray0", 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 5 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "pool": 17, 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout: "old_pools": [] 2024-03-06T07:28:03.764 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:28:03.765 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:03.766 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:03.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.905+0000 7f2d820fb640 1 Processor -- start 2024-03-06T07:28:03.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.905+0000 7f2d820fb640 1 -- start start 2024-03-06T07:28:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d820fb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2d7c15dfe0 0x7f2d7c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d820fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c0cf830 0x7f2d7c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d820fb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2d7c0d27f0 0x7f2d7c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d820fb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0d2c50 con 0x7f2d7c0d27f0 2024-03-06T07:28:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d820fb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d7c0b31e0 con 0x7f2d7c15dfe0 2024-03-06T07:28:03.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d820fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0b3d60 con 0x7f2d7c0cf830 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2d7c0d27f0 0x7f2d7c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2d7c0d27f0 0x7f2d7c0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54556/0 (socket says 172.21.15.72:54556) 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 -- 172.21.15.72:0/2460428849 learned_addr learned my addr 172.21.15.72:0/2460428849 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 -- 172.21.15.72:0/2460428849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c0cf830 msgr2=0x7f2d7c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 --2- 172.21.15.72:0/2460428849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c0cf830 0x7f2d7c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 -- 172.21.15.72:0/2460428849 >> v2:172.21.15.146:3300/0 conn(0x7f2d7c15dfe0 msgr2=0x7f2d7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 --2- 172.21.15.72:0/2460428849 >> v2:172.21.15.146:3300/0 conn(0x7f2d7c15dfe0 0x7f2d7c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.906+0000 7f2d7bfff640 1 -- 172.21.15.72:0/2460428849 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c0b3430 con 0x7f2d7c0d27f0 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.907+0000 7f2d7bfff640 1 --2- 172.21.15.72:0/2460428849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c0d0300 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c067990 tx=0x7f2d7402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d3fbd32b247b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.907+0000 7f2d7a7fc640 1 -- 172.21.15.72:0/2460428849 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7402ca70 con 0x7f2d7c0d27f0 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.907+0000 7f2d7a7fc640 1 -- 172.21.15.72:0/2460428849 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7402cc10 con 0x7f2d7c0d27f0 2024-03-06T07:28:03.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.907+0000 7f2d7a7fc640 1 -- 172.21.15.72:0/2460428849 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740343f0 con 0x7f2d7c0d27f0 2024-03-06T07:28:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 -- 172.21.15.72:0/2460428849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0d27f0 msgr2=0x7f2d7c0d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2460428849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c0d0300 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c067990 tx=0x7f2d7402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 -- 172.21.15.72:0/2460428849 shutdown_connections 2024-03-06T07:28:03.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2460428849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c0d0300 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2460428849 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c0cf830 0x7f2d7c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2460428849 >> v2:172.21.15.146:3300/0 conn(0x7f2d7c15dfe0 0x7f2d7c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 -- 172.21.15.72:0/2460428849 >> 172.21.15.72:0/2460428849 conn(0x7f2d7c0d3cd0 msgr2=0x7f2d7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 -- 172.21.15.72:0/2460428849 shutdown_connections 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.908+0000 7f2d820fb640 1 -- 172.21.15.72:0/2460428849 wait complete. 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.909+0000 7f2d820fb640 1 Processor -- start 2024-03-06T07:28:03.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.909+0000 7f2d820fb640 1 -- start start 2024-03-06T07:28:03.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d820fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 0x7f2d7c158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d820fb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c15fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d820fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c15dfe0 0x7f2d7c15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d820fb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0d1c70 con 0x7f2d7c0cf830 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d820fb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0b39f0 con 0x7f2d7c0d27f0 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d820fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c0d1df0 con 0x7f2d7c15dfe0 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 0x7f2d7c158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 0x7f2d7c158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54568/0 (socket says 172.21.15.72:54568) 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/2355606175 learned_addr learned my addr 172.21.15.72:0/2355606175 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7bfff640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c15dfe0 0x7f2d7c15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7affd640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c15fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c15dfe0 msgr2=0x7f2d7c15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c15dfe0 0x7f2d7c15b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d7c0d27f0 msgr2=0x7f2d7c15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:03.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c15fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7b7fe640 1 -- 172.21.15.72:0/2355606175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c0ccbf0 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.910+0000 7f2d7affd640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c15fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.911+0000 7f2d7b7fe640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 0x7f2d7c158140 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00b410 tx=0x7f2d6c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.911+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c015020 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.911+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c011850 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.911+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c0cbb50 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.911+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c011c40 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.911+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c0cc3f0 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.912+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c04f330 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.912+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d6c018020 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.912+0000 7f2d63fff640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d6806b4d0 0x7f2d6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.913+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f2d6c08a270 con 0x7f2d7c0cf830 2024-03-06T07:28:03.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.913+0000 7f2d7affd640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d6806b4d0 0x7f2d6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:03.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.913+0000 7f2d7affd640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d6806b4d0 0x7f2d6806d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2d70006fd0 tx=0x7f2d70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:03.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:03.917+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d6c01f020 con 0x7f2d7c0cf830 2024-03-06T07:28:04.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.133+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d7c0ceb30 con 0x7f2d7c0cf830 2024-03-06T07:28:04.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.133+0000 7f2d63fff640 1 -- 172.21.15.72:0/2355606175 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f2d6c0172c0 con 0x7f2d7c0cf830 2024-03-06T07:28:04.135 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:04.136 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:04.136 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:04.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d6806b4d0 msgr2=0x7f2d6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d6806b4d0 0x7f2d6806d990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2d70006fd0 tx=0x7f2d70008040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 msgr2=0x7f2d7c158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 0x7f2d7c158140 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c00b410 tx=0x7f2d6c00b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 shutdown_connections 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d6806b4d0 0x7f2d6806d990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d7c15dfe0 0x7f2d7c15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d7c0d27f0 0x7f2d7c15fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 --2- 172.21.15.72:0/2355606175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d7c0cf830 0x7f2d7c158140 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 >> 172.21.15.72:0/2355606175 conn(0x7f2d7c0d3cd0 msgr2=0x7f2d7c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 shutdown_connections 2024-03-06T07:28:04.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.135+0000 7f2d820fb640 1 -- 172.21.15.72:0/2355606175 wait complete. 2024-03-06T07:28:04.152 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:04.269 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-03-06T07:28:04.269 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-03-06T07:28:04.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:04.270 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-03-06T07:28:04.294 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:04.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.437+0000 7f4a1c548640 1 Processor -- start 2024-03-06T07:28:04.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.437+0000 7f4a1c548640 1 -- start start 2024-03-06T07:28:04.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1c548640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a140d2710 0x7f4a140b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1c548640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a140b3430 con 0x7f4a1415dfe0 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1c548640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a140b3710 con 0x7f4a140d22b0 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1c548640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a140b31e0 con 0x7f4a140d2710 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1aae5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a140d2710 0x7f4a140b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1a2e4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4a1415dfe0 0x7f4a140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54892/0 (socket says 172.21.15.72:54892) 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1a2e4640 1 -- 172.21.15.72:0/235007455 learned_addr learned my addr 172.21.15.72:0/235007455 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1aae5640 1 -- 172.21.15.72:0/235007455 >> v1:172.21.15.146:6789/0 conn(0x7f4a140d22b0 legacy=0x7f4a140b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1aae5640 1 -- 172.21.15.72:0/235007455 >> v1:172.21.15.72:6789/0 conn(0x7f4a1415dfe0 legacy=0x7f4a140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a192e2640 1 -- 172.21.15.72:0/235007455 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243879121 0 0) 0x7f4a140b3430 con 0x7f4a1415dfe0 2024-03-06T07:28:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.438+0000 7f4a1aae5640 1 -- 172.21.15.72:0/235007455 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a140b39f0 con 0x7f4a140d2710 2024-03-06T07:28:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.439+0000 7f4a1aae5640 1 --2- 172.21.15.72:0/235007455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140b4900 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009dc0 tx=0x7f4a0c02be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53ca6af8cd75809 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.439+0000 7f4a192e2640 1 -- 172.21.15.72:0/235007455 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c039070 con 0x7f4a140d2710 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a192e2640 1 -- 172.21.15.72:0/235007455 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c02cc40 con 0x7f4a140d2710 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a192e2640 1 -- 172.21.15.72:0/235007455 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c034440 con 0x7f4a140d2710 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 -- 172.21.15.72:0/235007455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 msgr2=0x7f4a140b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 --2- 172.21.15.72:0/235007455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140b4900 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c009dc0 tx=0x7f4a0c02be90 comp rx=0 tx=0).stop 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 -- 172.21.15.72:0/235007455 shutdown_connections 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 --2- 172.21.15.72:0/235007455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140b4900 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 -- 172.21.15.72:0/235007455 >> 172.21.15.72:0/235007455 conn(0x7f4a140d3cd0 msgr2=0x7f4a140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 -- 172.21.15.72:0/235007455 shutdown_connections 2024-03-06T07:28:04.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.440+0000 7f4a1c548640 1 -- 172.21.15.72:0/235007455 wait complete. 2024-03-06T07:28:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.441+0000 7f4a1c548640 1 Processor -- start 2024-03-06T07:28:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.441+0000 7f4a1c548640 1 -- start start 2024-03-06T07:28:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1c548640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a140d22b0 0x7f4a140b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1c548640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1c548640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1415dfe0 0x7f4a14159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1c548640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140cfa00 con 0x7f4a140d2710 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1c548640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140b3d60 con 0x7f4a1415dfe0 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1c548640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140d1c70 con 0x7f4a140d22b0 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54572/0 (socket says 172.21.15.72:54572) 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 -- 172.21.15.72:0/1086798967 learned_addr learned my addr 172.21.15.72:0/1086798967 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1aae5640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1415dfe0 0x7f4a14159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1a2e4640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a140d22b0 0x7f4a140b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 -- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a140d22b0 msgr2=0x7f4a140b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a140d22b0 0x7f4a140b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 -- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1415dfe0 msgr2=0x7f4a14159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1415dfe0 0x7f4a14159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a19ae3640 1 -- 172.21.15.72:0/1086798967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a141599b0 con 0x7f4a140d2710 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1aae5640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1415dfe0 0x7f4a14159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.442+0000 7f4a1a2e4640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a140d22b0 0x7f4a140b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.443+0000 7f4a19ae3640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140ca690 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c460 tx=0x7f4a0400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.443+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04016020 con 0x7f4a140d2710 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.443+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a04012870 con 0x7f4a140d2710 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.443+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a04012c60 con 0x7f4a140d2710 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.443+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a140b9220 con 0x7f4a140d2710 2024-03-06T07:28:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.443+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a1415f370 con 0x7f4a140d2710 2024-03-06T07:28:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.444+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a140cabd0 con 0x7f4a140d2710 2024-03-06T07:28:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.444+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a04019020 con 0x7f4a140d2710 2024-03-06T07:28:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.445+0000 7f4a0affd640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f49f806b4d0 0x7f49f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.445+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f4a04089d30 con 0x7f4a140d2710 2024-03-06T07:28:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.445+0000 7f4a1a2e4640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f49f806b4d0 0x7f49f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.445+0000 7f4a1a2e4640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f49f806b4d0 0x7f49f806d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4a10000f60 tx=0x7f4a10009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.453+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a04020070 con 0x7f4a140d2710 2024-03-06T07:28:04.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.667+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a1404f330 con 0x7f4a140d2710 2024-03-06T07:28:04.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.667+0000 7f4a0affd640 1 -- 172.21.15.72:0/1086798967 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f4a04058b00 con 0x7f4a140d2710 2024-03-06T07:28:04.670 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:04.670 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:04.670 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f49f806b4d0 msgr2=0x7f49f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f49f806b4d0 0x7f49f806d990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f4a10000f60 tx=0x7f4a10009040 comp rx=0 tx=0).stop 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 msgr2=0x7f4a140ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140ca690 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c460 tx=0x7f4a0400c920 comp rx=0 tx=0).stop 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 shutdown_connections 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f49f806b4d0 0x7f49f806d990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1415dfe0 0x7f4a14159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a140d2710 0x7f4a140ca690 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 --2- 172.21.15.72:0/1086798967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a140d22b0 0x7f4a140b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 >> 172.21.15.72:0/1086798967 conn(0x7f4a140d3cd0 msgr2=0x7f4a1406bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 shutdown_connections 2024-03-06T07:28:04.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.669+0000 7f4a1c548640 1 -- 172.21.15.72:0/1086798967 wait complete. 2024-03-06T07:28:04.682 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:04.799 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:04.800 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:04.800 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.898+0000 7eff40377640 1 Processor -- start 2024-03-06T07:28:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.898+0000 7eff40377640 1 -- start start 2024-03-06T07:28:04.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff40377640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7eff380d22b0 0x7eff380d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff40377640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380b3430 con 0x7eff3815dfe0 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff40377640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff380b3710 con 0x7eff380b41f0 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff40377640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff380d2c50 con 0x7eff380d22b0 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d912640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7eff380d22b0 0x7eff380d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d912640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7eff380d22b0 0x7eff380d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54576/0 (socket says 172.21.15.72:54576) 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d912640 1 -- 172.21.15.72:0/716234248 learned_addr learned my addr 172.21.15.72:0/716234248 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3e113640 1 --1- 172.21.15.72:0/716234248 >> v1:172.21.15.72:6789/0 conn(0x7eff3815dfe0 0x7eff380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54898/0 (socket says 172.21.15.72:54898) 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d912640 1 -- 172.21.15.72:0/716234248 >> v1:172.21.15.146:6789/0 conn(0x7eff380b41f0 legacy=0x7eff380b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d912640 1 -- 172.21.15.72:0/716234248 >> v1:172.21.15.72:6789/0 conn(0x7eff3815dfe0 legacy=0x7eff380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d111640 1 -- 172.21.15.72:0/716234248 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549705889 0 0) 0x7eff380b3430 con 0x7eff3815dfe0 2024-03-06T07:28:04.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.899+0000 7eff3d912640 1 -- 172.21.15.72:0/716234248 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380b39f0 con 0x7eff380d22b0 2024-03-06T07:28:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.900+0000 7eff3d912640 1 --2- 172.21.15.72:0/716234248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380d22b0 0x7eff380d2710 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7eff340098a0 tx=0x7eff3402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c5ac92b0742b17 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.900+0000 7eff3d111640 1 -- 172.21.15.72:0/716234248 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34034070 con 0x7eff380d22b0 2024-03-06T07:28:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.900+0000 7eff3d111640 1 -- 172.21.15.72:0/716234248 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff34030840 con 0x7eff380d22b0 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.900+0000 7eff3d111640 1 -- 172.21.15.72:0/716234248 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34030c10 con 0x7eff380d22b0 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 -- 172.21.15.72:0/716234248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380d22b0 msgr2=0x7eff380d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 --2- 172.21.15.72:0/716234248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380d22b0 0x7eff380d2710 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7eff340098a0 tx=0x7eff3402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 -- 172.21.15.72:0/716234248 shutdown_connections 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 --2- 172.21.15.72:0/716234248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380d22b0 0x7eff380d2710 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 -- 172.21.15.72:0/716234248 >> 172.21.15.72:0/716234248 conn(0x7eff380d3cd0 msgr2=0x7eff380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 -- 172.21.15.72:0/716234248 shutdown_connections 2024-03-06T07:28:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.901+0000 7eff40377640 1 -- 172.21.15.72:0/716234248 wait complete. 2024-03-06T07:28:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.902+0000 7eff40377640 1 Processor -- start 2024-03-06T07:28:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.902+0000 7eff40377640 1 -- start start 2024-03-06T07:28:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 0x7eff380b75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380d22b0 0x7eff380b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff3815dfe0 0x7eff3814f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b3d60 con 0x7eff380b41f0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff380b31e0 con 0x7eff3815dfe0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff380d1c70 con 0x7eff380d22b0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 0x7eff380b75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 0x7eff380b75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54586/0 (socket says 172.21.15.72:54586) 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 -- 172.21.15.72:0/463741984 learned_addr learned my addr 172.21.15.72:0/463741984 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 -- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380d22b0 msgr2=0x7eff380b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380d22b0 0x7eff380b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 -- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff3815dfe0 msgr2=0x7eff3814f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e914640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff3815dfe0 0x7eff3814f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff3815dfe0 0x7eff3814f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 -- 172.21.15.72:0/463741984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380b7cc0 con 0x7eff380b41f0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff3e113640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 0x7eff380b75c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00c430 tx=0x7eff2c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0074e0 con 0x7eff380b41f0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c0047e0 con 0x7eff380b41f0 2024-03-06T07:28:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff380b7fa0 con 0x7eff380b41f0 2024-03-06T07:28:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c004bd0 con 0x7eff380b41f0 2024-03-06T07:28:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.903+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3815fed0 con 0x7eff380b41f0 2024-03-06T07:28:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.904+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3804f330 con 0x7eff380b41f0 2024-03-06T07:28:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.905+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7eff2c011070 con 0x7eff380b41f0 2024-03-06T07:28:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.906+0000 7eff2affd640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7eff1c06b4d0 0x7eff1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.906+0000 7eff3d912640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7eff1c06b4d0 0x7eff1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.906+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7eff2c08a000 con 0x7eff380b41f0 2024-03-06T07:28:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.906+0000 7eff3d912640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7eff1c06b4d0 0x7eff1c06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7eff34036000 tx=0x7eff34030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:04.910+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7eff2c058d40 con 0x7eff380b41f0 2024-03-06T07:28:05.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.120+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff3814f5d0 con 0x7eff380b41f0 2024-03-06T07:28:05.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.121+0000 7eff2affd640 1 -- 172.21.15.72:0/463741984 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7eff2c058640 con 0x7eff380b41f0 2024-03-06T07:28:05.123 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:05.123 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:05.123 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:05.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7eff1c06b4d0 msgr2=0x7eff1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7eff1c06b4d0 0x7eff1c06d990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7eff34036000 tx=0x7eff34030000 comp rx=0 tx=0).stop 2024-03-06T07:28:05.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 msgr2=0x7eff380b75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 0x7eff380b75c0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00c430 tx=0x7eff2c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 shutdown_connections 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7eff1c06b4d0 0x7eff1c06d990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff3815dfe0 0x7eff3814f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff380d22b0 0x7eff380b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 --2- 172.21.15.72:0/463741984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff380b41f0 0x7eff380b75c0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 >> 172.21.15.72:0/463741984 conn(0x7eff380d3cd0 msgr2=0x7eff3815d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 shutdown_connections 2024-03-06T07:28:05.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.123+0000 7eff40377640 1 -- 172.21.15.72:0/463741984 wait complete. 2024-03-06T07:28:05.136 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:05.253 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-03-06T07:28:05.253 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-03-06T07:28:05.253 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:05.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.352+0000 7fede3cc9640 1 Processor -- start 2024-03-06T07:28:05.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.352+0000 7fede3cc9640 1 -- start start 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede3cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc0d4920 0x7feddc0cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede3cc9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feddc1625a0 0x7feddc15dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede3cc9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0ceac0 con 0x7feddc1621c0 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede3cc9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc0cf690 con 0x7feddc1625a0 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede3cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0d2dc0 con 0x7feddc0d4920 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede1a65640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7feddc1621c0 0x7feddc0d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54906/0 (socket says 172.21.15.72:54906) 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede1a65640 1 -- 172.21.15.72:0/1226454827 learned_addr learned my addr 172.21.15.72:0/1226454827 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede1264640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc0d4920 0x7feddc0cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.353+0000 7fede2266640 1 --2- 172.21.15.72:0/1226454827 >> v2:172.21.15.146:3300/0 conn(0x7feddc1625a0 0x7feddc15dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.354+0000 7fede2266640 1 -- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc0d4920 msgr2=0x7feddc0cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.354+0000 7fede2266640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc0d4920 0x7feddc0cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.354+0000 7fede2266640 1 -- 172.21.15.72:0/1226454827 >> v1:172.21.15.72:6789/0 conn(0x7feddc1621c0 legacy=0x7feddc0d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.354+0000 7fede2266640 1 -- 172.21.15.72:0/1226454827 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc0ced60 con 0x7feddc1625a0 2024-03-06T07:28:05.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.354+0000 7fede1264640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc0d4920 0x7feddc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:05.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.355+0000 7fede2266640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc1625a0 0x7feddc15dd40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009d90 tx=0x7fedd402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5be11d45c10ead14 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.355+0000 7fede0a63640 1 -- 172.21.15.72:0/1226454827 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4039070 con 0x7feddc1625a0 2024-03-06T07:28:05.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.355+0000 7fede0a63640 1 -- 172.21.15.72:0/1226454827 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd402cc50 con 0x7feddc1625a0 2024-03-06T07:28:05.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.355+0000 7fede0a63640 1 -- 172.21.15.72:0/1226454827 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4034440 con 0x7feddc1625a0 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 -- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc1625a0 msgr2=0x7feddc15dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc1625a0 0x7feddc15dd40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009d90 tx=0x7fedd402bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 -- 172.21.15.72:0/1226454827 shutdown_connections 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc1625a0 0x7feddc15dd40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 --2- 172.21.15.72:0/1226454827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc0d4920 0x7feddc0cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 -- 172.21.15.72:0/1226454827 >> 172.21.15.72:0/1226454827 conn(0x7feddc0d3d10 msgr2=0x7feddc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 -- 172.21.15.72:0/1226454827 shutdown_connections 2024-03-06T07:28:05.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.356+0000 7fede3cc9640 1 -- 172.21.15.72:0/1226454827 wait complete. 2024-03-06T07:28:05.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.358+0000 7fede3cc9640 1 Processor -- start 2024-03-06T07:28:05.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.358+0000 7fede3cc9640 1 -- start start 2024-03-06T07:28:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede3cc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc0d4920 0x7feddc0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede3cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc1621c0 0x7feddc0b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede3cc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 0x7feddc0b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede3cc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc15e280 con 0x7feddc1625a0 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede3cc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc0cf320 con 0x7feddc0d4920 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede3cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc15e400 con 0x7feddc1621c0 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede2266640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 0x7feddc0b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede2266640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 0x7feddc0b85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54594/0 (socket says 172.21.15.72:54594) 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.359+0000 7fede2266640 1 -- 172.21.15.72:0/3009333689 learned_addr learned my addr 172.21.15.72:0/3009333689 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede1a65640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc0d4920 0x7feddc0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede1264640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc1621c0 0x7feddc0b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede2266640 1 -- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc1621c0 msgr2=0x7feddc0b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede2266640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc1621c0 0x7feddc0b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede2266640 1 -- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc0d4920 msgr2=0x7feddc0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede2266640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc0d4920 0x7feddc0c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede2266640 1 -- 172.21.15.72:0/3009333689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc1e8e40 con 0x7feddc1625a0 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede2266640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 0x7feddc0b85e0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009d90 tx=0x7fedd4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4044020 con 0x7feddc1625a0 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd402ca50 con 0x7feddc1625a0 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc0cb710 con 0x7feddc1625a0 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4039040 con 0x7feddc1625a0 2024-03-06T07:28:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.360+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc0cbcd0 con 0x7feddc1625a0 2024-03-06T07:28:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.361+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc04f330 con 0x7feddc1625a0 2024-03-06T07:28:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.362+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedd4047020 con 0x7feddc1625a0 2024-03-06T07:28:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.362+0000 7fedd27fc640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.362+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fedd40ac9c0 con 0x7feddc1625a0 2024-03-06T07:28:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.362+0000 7fede1a65640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.363+0000 7fede1a65640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fedc006b4d0 0x7fedc006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fedd8007fd0 tx=0x7fedd8007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.366+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedd40aee20 con 0x7feddc1625a0 2024-03-06T07:28:05.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.582+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feddc0cbea0 con 0x7feddc1625a0 2024-03-06T07:28:05.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.583+0000 7fedd27fc640 1 -- 172.21.15.72:0/3009333689 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fedd404e030 con 0x7feddc1625a0 2024-03-06T07:28:05.585 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:05.585 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:05.585 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fedc006b4d0 msgr2=0x7fedc006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fedc006b4d0 0x7fedc006d990 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fedd8007fd0 tx=0x7fedd8007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 msgr2=0x7feddc0b85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 0x7feddc0b85e0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009d90 tx=0x7fedd4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 shutdown_connections 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fedc006b4d0 0x7fedc006d990 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feddc1625a0 0x7feddc0b85e0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feddc1621c0 0x7feddc0b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.584+0000 7fede3cc9640 1 --2- 172.21.15.72:0/3009333689 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feddc0d4920 0x7feddc0c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.585+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 >> 172.21.15.72:0/3009333689 conn(0x7feddc0d3d10 msgr2=0x7feddc15c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.585+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 shutdown_connections 2024-03-06T07:28:05.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.585+0000 7fede3cc9640 1 -- 172.21.15.72:0/3009333689 wait complete. 2024-03-06T07:28:05.598 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:05.721 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:28:05.721 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:05.721 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:05.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 Processor -- start 2024-03-06T07:28:05.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 -- start start 2024-03-06T07:28:05.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb8012ec0 0x558eb80df980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb80e7e00 0x558eb80dfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eb7fbab70 con 0x558eb8043310 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eb7fbead0 con 0x558eb80e7e00 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcdba35f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558eb805bd40 con 0x558eb8012ec0 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcd8d78640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcd8d78640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54606/0 (socket says 172.21.15.72:54606) 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.762+0000 7fdcd8d78640 1 -- 172.21.15.72:0/2378308772 learned_addr learned my addr 172.21.15.72:0/2378308772 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd8d78640 1 -- 172.21.15.72:0/2378308772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb8012ec0 msgr2=0x558eb80df980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd8d78640 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb8012ec0 0x558eb80df980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd8d78640 1 -- 172.21.15.72:0/2378308772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb80e7e00 msgr2=0x558eb80dfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd8d78640 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb80e7e00 0x558eb80dfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd8d78640 1 -- 172.21.15.72:0/2378308772 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558eb80e73f0 con 0x558eb8043310 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd8d78640 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e05b0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fdccc002fb0 tx=0x7fdccc00e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62cf75aeb967d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd2ffd640 1 -- 172.21.15.72:0/2378308772 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc001ac0 con 0x558eb8043310 2024-03-06T07:28:05.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd2ffd640 1 -- 172.21.15.72:0/2378308772 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc001c60 con 0x558eb8043310 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.763+0000 7fdcd2ffd640 1 -- 172.21.15.72:0/2378308772 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc002250 con 0x558eb8043310 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 -- 172.21.15.72:0/2378308772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 msgr2=0x558eb80e05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e05b0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fdccc002fb0 tx=0x7fdccc00e760 comp rx=0 tx=0).stop 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 -- 172.21.15.72:0/2378308772 shutdown_connections 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb80e7e00 0x558eb80dfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb8012ec0 0x558eb80df980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/2378308772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e05b0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 -- 172.21.15.72:0/2378308772 >> 172.21.15.72:0/2378308772 conn(0x558eb80128b0 msgr2=0x558eb80ec120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 -- 172.21.15.72:0/2378308772 shutdown_connections 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.764+0000 7fdcdba35f40 1 -- 172.21.15.72:0/2378308772 wait complete. 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.765+0000 7fdcdba35f40 1 Processor -- start 2024-03-06T07:28:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.765+0000 7fdcdba35f40 1 -- start start 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb8012ec0 0x558eb80e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb80e7e00 0x558eb80e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eb80eb820 con 0x558eb8043310 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eb80e7660 con 0x558eb8012ec0 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558eb80e0af0 con 0x558eb80e7e00 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54618/0 (socket says 172.21.15.72:54618) 2024-03-06T07:28:05.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 -- 172.21.15.72:0/3606737134 learned_addr learned my addr 172.21.15.72:0/3606737134 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 -- 172.21.15.72:0/3606737134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb80e7e00 msgr2=0x558eb80e6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558eb80e7e00 0x558eb80e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 -- 172.21.15.72:0/3606737134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb8012ec0 msgr2=0x558eb80e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558eb8012ec0 0x558eb80e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 -- 172.21.15.72:0/3606737134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558eb80d2470 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd3fff640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e68c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4001820 tx=0x7fdcc400e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd1ffb640 1 -- 172.21.15.72:0/3606737134 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc400f920 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcd1ffb640 1 -- 172.21.15.72:0/3606737134 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc4013030 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.766+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558eb80d2750 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.767+0000 7fdcd1ffb640 1 -- 172.21.15.72:0/3606737134 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc400fd00 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.767+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558eb80d2d10 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.768+0000 7fdcd1ffb640 1 -- 172.21.15.72:0/3606737134 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdcc40183e0 con 0x558eb8043310 2024-03-06T07:28:05.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.768+0000 7fdcd1ffb640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcbc069ee0 0x7fdcbc06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.768+0000 7fdcd1ffb640 1 -- 172.21.15.72:0/3606737134 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fdcc4089e00 con 0x558eb8043310 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.768+0000 7fdcd8d78640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcbc069ee0 0x7fdcbc06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.768+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x558eb80cf480 0x558eb80d3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.768+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x558eb80cf880 con 0x558eb80cf480 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcd9579640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x558eb80cf480 0x558eb80d3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcd8d78640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcbc069ee0 0x7fdcbc06c3a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e7a0 tx=0x7fdccc00b000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcd9579640 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x558eb80cf480 0x558eb80d3510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcd9579640 1 -- 172.21.15.72:0/3606737134 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fdcd4005040 con 0x558eb80cf480 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x558eb80cf480 msgr2=0x558eb80d3510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x558eb80cf480 0x558eb80d3510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcbc069ee0 msgr2=0x7fdcbc06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdcbc069ee0 0x7fdcbc06c3a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fdccc00e7a0 tx=0x7fdccc00b000 comp rx=0 tx=0).stop 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 msgr2=0x558eb80e68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:05.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 --2- 172.21.15.72:0/3606737134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x558eb8043310 0x558eb80e68c0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4001820 tx=0x7fdcc400e870 comp rx=0 tx=0).stop 2024-03-06T07:28:05.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcd9579640 1 -- 172.21.15.72:0/3606737134 reap_dead start 2024-03-06T07:28:05.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 shutdown_connections 2024-03-06T07:28:05.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 >> 172.21.15.72:0/3606737134 conn(0x558eb80128b0 msgr2=0x558eb80ec020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:05.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 shutdown_connections 2024-03-06T07:28:05.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:05.769+0000 7fdcdba35f40 1 -- 172.21.15.72:0/3606737134 wait complete. 2024-03-06T07:28:05.776 INFO:teuthology.orchestra.run.smithi072.stdout:W�file_cdir_2! 2024-03-06T07:28:05.776 DEBUG:teuthology.orchestra.run.smithi072:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi072.stdout: "ino": 1099511627777, 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi072.stdout: "ancestors": [ 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1099511628278, 2024-03-06T07:28:05.876 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "file_c", 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 15 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1, 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "dir_2", 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 33 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: "pool": 17, 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: "old_pools": [] 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:28:05.877 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:05.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.880 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-03-06T07:28:05.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:05.940 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-03-06T07:28:05.999 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:06.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.137+0000 7f5f7b57a640 1 Processor -- start 2024-03-06T07:28:06.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.137+0000 7f5f7b57a640 1 -- start start 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f7b57a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5f740cfff0 0x7f5f740d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f7b57a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f74161980 con 0x7f5f7415bb20 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f7b57a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f74161c60 con 0x7f5f740cc8b0 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f7b57a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f740d2500 con 0x7f5f740cfff0 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f78b15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5f740cfff0 0x7f5f740d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f78b15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5f740cfff0 0x7f5f740d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54622/0 (socket says 172.21.15.72:54622) 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f79316640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5f7415bb20 0x7f5f740d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54916/0 (socket says 172.21.15.72:54916) 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.138+0000 7f5f78b15640 1 -- 172.21.15.72:0/3319785230 learned_addr learned my addr 172.21.15.72:0/3319785230 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f78b15640 1 -- 172.21.15.72:0/3319785230 >> v1:172.21.15.146:6789/0 conn(0x7f5f740cc8b0 legacy=0x7f5f740ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f78b15640 1 -- 172.21.15.72:0/3319785230 >> v1:172.21.15.72:6789/0 conn(0x7f5f7415bb20 legacy=0x7f5f740d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f6bfff640 1 -- 172.21.15.72:0/3319785230 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350852994 0 0) 0x7f5f74161980 con 0x7f5f7415bb20 2024-03-06T07:28:06.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f78b15640 1 -- 172.21.15.72:0/3319785230 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f74161f40 con 0x7f5f740cfff0 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f78b15640 1 --2- 172.21.15.72:0/3319785230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f740d04a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009870 tx=0x7f5f6402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bef516e09da38c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f6bfff640 1 -- 172.21.15.72:0/3319785230 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6402f840 con 0x7f5f740cfff0 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f6bfff640 1 -- 172.21.15.72:0/3319785230 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6402f9e0 con 0x7f5f740cfff0 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.139+0000 7f5f6bfff640 1 -- 172.21.15.72:0/3319785230 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6402fdb0 con 0x7f5f740cfff0 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 -- 172.21.15.72:0/3319785230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 msgr2=0x7f5f740d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/3319785230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f740d04a0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009870 tx=0x7f5f6402ba20 comp rx=0 tx=0).stop 2024-03-06T07:28:06.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 -- 172.21.15.72:0/3319785230 shutdown_connections 2024-03-06T07:28:06.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/3319785230 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f740d04a0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 -- 172.21.15.72:0/3319785230 >> 172.21.15.72:0/3319785230 conn(0x7f5f7409a9f0 msgr2=0x7f5f7409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 -- 172.21.15.72:0/3319785230 shutdown_connections 2024-03-06T07:28:06.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.140+0000 7f5f7b57a640 1 -- 172.21.15.72:0/3319785230 wait complete. 2024-03-06T07:28:06.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.141+0000 7f5f7b57a640 1 Processor -- start 2024-03-06T07:28:06.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.141+0000 7f5f7b57a640 1 -- start start 2024-03-06T07:28:06.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f7b57a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f740cc8b0 0x7f5f740cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f7b57a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f74150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f7b57a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f7415bb20 0x7f5f741510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f7b57a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740d1980 con 0x7f5f740cfff0 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f7b57a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f74162500 con 0x7f5f7415bb20 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f7b57a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740d1b00 con 0x7f5f740cc8b0 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f74150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f74150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54626/0 (socket says 172.21.15.72:54626) 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 -- 172.21.15.72:0/2144010433 learned_addr learned my addr 172.21.15.72:0/2144010433 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f79b17640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f7415bb20 0x7f5f741510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f79316640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f740cc8b0 0x7f5f740cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 -- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f740cc8b0 msgr2=0x7f5f740cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f740cc8b0 0x7f5f740cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 -- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f7415bb20 msgr2=0x7f5f741510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f7415bb20 0x7f5f741510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f78b15640 1 -- 172.21.15.72:0/2144010433 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740c5720 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.142+0000 7f5f79316640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f740cc8b0 0x7f5f740cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.143+0000 7f5f78b15640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f74150b90 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009870 tx=0x7f5f64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.143+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f64041400 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.143+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f64046070 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.143+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f64041800 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.143+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f740c5a00 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.143+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f740c5fc0 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.144+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7404f330 con 0x7f5f740cfff0 2024-03-06T07:28:06.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.144+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5f64040070 con 0x7f5f740cfff0 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.145+0000 7f5f69ffb640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5f5806b4d0 0x7f5f5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.145+0000 7f5f79316640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5f5806b4d0 0x7f5f5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.145+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f5f640acae0 con 0x7f5f740cfff0 2024-03-06T07:28:06.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.145+0000 7f5f79316640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5f5806b4d0 0x7f5f5806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5f70000f60 tx=0x7f5f70009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:06.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.149+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5f64033030 con 0x7f5f740cfff0 2024-03-06T07:28:06.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.371+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f7406f0e0 con 0x7f5f740cfff0 2024-03-06T07:28:06.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.371+0000 7f5f69ffb640 1 -- 172.21.15.72:0/2144010433 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f5f6407b8b0 con 0x7f5f740cfff0 2024-03-06T07:28:06.374 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:06.374 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:06.374 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.373+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5f5806b4d0 msgr2=0x7f5f5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.373+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5f5806b4d0 0x7f5f5806d990 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f5f70000f60 tx=0x7f5f70009040 comp rx=0 tx=0).stop 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.373+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 msgr2=0x7f5f74150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.373+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f74150b90 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009870 tx=0x7f5f64004730 comp rx=0 tx=0).stop 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 shutdown_connections 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5f5806b4d0 0x7f5f5806d990 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f7415bb20 0x7f5f741510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f740cfff0 0x7f5f74150b90 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 --2- 172.21.15.72:0/2144010433 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f740cc8b0 0x7f5f740cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:06.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 >> 172.21.15.72:0/2144010433 conn(0x7f5f7409a9f0 msgr2=0x7f5f740b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:06.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 shutdown_connections 2024-03-06T07:28:06.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:06.374+0000 7f5f7b57a640 1 -- 172.21.15.72:0/2144010433 wait complete. 2024-03-06T07:28:06.387 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:06.502 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:28:06.503 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:28:06.503 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-03-06T07:28:11.504 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:11.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.616+0000 7f63c393a640 1 Processor -- start 2024-03-06T07:28:11.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.616+0000 7f63c393a640 1 -- start start 2024-03-06T07:28:11.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.616+0000 7f63c393a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63bc1424f0 0x7f63bc0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.616+0000 7f63c393a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc0cd7a0 0x7f63bc15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.616+0000 7f63c393a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0cc980 con 0x7f63bc06d330 2024-03-06T07:28:11.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c393a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc0cd220 con 0x7f63bc1424f0 2024-03-06T07:28:11.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c393a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0ccc60 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c16d6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63bc1424f0 0x7f63bc0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c1ed7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f63bc06d330 0x7f63bc071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54932/0 (socket says 172.21.15.72:54932) 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c1ed7640 1 -- 172.21.15.72:0/1174688889 learned_addr learned my addr 172.21.15.72:0/1174688889 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c0ed5640 1 --2- 172.21.15.72:0/1174688889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc0cd7a0 0x7f63bc15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c16d6640 1 -- 172.21.15.72:0/1174688889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc0cd7a0 msgr2=0x7f63bc15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c16d6640 1 --2- 172.21.15.72:0/1174688889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc0cd7a0 0x7f63bc15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c16d6640 1 -- 172.21.15.72:0/1174688889 >> v1:172.21.15.72:6789/0 conn(0x7f63bc06d330 legacy=0x7f63bc071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63b3fff640 1 -- 172.21.15.72:0/1174688889 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082921046 0 0) 0x7f63bc0cc980 con 0x7f63bc06d330 2024-03-06T07:28:11.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.617+0000 7f63c16d6640 1 -- 172.21.15.72:0/1174688889 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0ccf40 con 0x7f63bc1424f0 2024-03-06T07:28:11.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.618+0000 7f63c16d6640 1 --2- 172.21.15.72:0/1174688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc1424f0 0x7f63bc0cc2f0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f63b40097a0 tx=0x7f63b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7946291df5540150 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.618+0000 7f63b3fff640 1 -- 172.21.15.72:0/1174688889 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4035020 con 0x7f63bc1424f0 2024-03-06T07:28:11.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.618+0000 7f63b3fff640 1 -- 172.21.15.72:0/1174688889 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4031830 con 0x7f63bc1424f0 2024-03-06T07:28:11.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.618+0000 7f63b3fff640 1 -- 172.21.15.72:0/1174688889 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4031c00 con 0x7f63bc1424f0 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 -- 172.21.15.72:0/1174688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc1424f0 msgr2=0x7f63bc0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 --2- 172.21.15.72:0/1174688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc1424f0 0x7f63bc0cc2f0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f63b40097a0 tx=0x7f63b402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 -- 172.21.15.72:0/1174688889 shutdown_connections 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 --2- 172.21.15.72:0/1174688889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc0cd7a0 0x7f63bc15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 --2- 172.21.15.72:0/1174688889 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc1424f0 0x7f63bc0cc2f0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 -- 172.21.15.72:0/1174688889 >> 172.21.15.72:0/1174688889 conn(0x7f63bc0cf8a0 msgr2=0x7f63bc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 -- 172.21.15.72:0/1174688889 shutdown_connections 2024-03-06T07:28:11.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.619+0000 7f63c393a640 1 -- 172.21.15.72:0/1174688889 wait complete. 2024-03-06T07:28:11.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.620+0000 7f63c393a640 1 Processor -- start 2024-03-06T07:28:11.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.620+0000 7f63c393a640 1 -- start start 2024-03-06T07:28:11.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c393a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc06d330 0x7f63bc143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c393a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 0x7f63bc15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c393a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63bc1424f0 0x7f63bc1f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c393a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc06d940 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c393a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc06d7c0 con 0x7f63bc1424f0 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c393a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc06dac0 con 0x7f63bc06d330 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 0x7f63bc15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 0x7f63bc15ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54646/0 (socket says 172.21.15.72:54646) 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 -- 172.21.15.72:0/1875389126 learned_addr learned my addr 172.21.15.72:0/1875389126 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c16d6640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc06d330 0x7f63bc143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c1ed7640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63bc1424f0 0x7f63bc1f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 -- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc06d330 msgr2=0x7f63bc143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc06d330 0x7f63bc143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 -- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63bc1424f0 msgr2=0x7f63bc1f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63bc1424f0 0x7f63bc1f1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 -- 172.21.15.72:0/1875389126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0a9950 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63c0ed5640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 0x7f63bc15ac90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00d800 tx=0x7f63ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac00b660 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.621+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac019070 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.622+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc1445a0 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.622+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac00ba20 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.622+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc144b60 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.623+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc0b2870 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.623+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f63ac00bbc0 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.624+0000 7f63b1ffb640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63a006b4d0 0x7f63a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.624+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f63ac08a6f0 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.624+0000 7f63c16d6640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63a006b4d0 0x7f63a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.624+0000 7f63c16d6640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63a006b4d0 0x7f63a006d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f63b4036000 tx=0x7f63b403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.628+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63bc0b2870 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.839+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63bc04f330 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.839+0000 7f63b1ffb640 1 -- 172.21.15.72:0/1875389126 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f63ac0594c0 con 0x7f63bc0cd7a0 2024-03-06T07:28:11.840 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:11.841 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:11.841 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:11.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63a006b4d0 msgr2=0x7f63a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63a006b4d0 0x7f63a006d990 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f63b4036000 tx=0x7f63b403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 msgr2=0x7f63bc15ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 0x7f63bc15ac90 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f63ac00d800 tx=0x7f63ac00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 shutdown_connections 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63a006b4d0 0x7f63a006d990 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63bc1424f0 0x7f63bc1f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63bc0cd7a0 0x7f63bc15ac90 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 --2- 172.21.15.72:0/1875389126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63bc06d330 0x7f63bc143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 >> 172.21.15.72:0/1875389126 conn(0x7f63bc0cf8a0 msgr2=0x7f63bc06b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 shutdown_connections 2024-03-06T07:28:11.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:11.841+0000 7f63c393a640 1 -- 172.21.15.72:0/1875389126 wait complete. 2024-03-06T07:28:11.853 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:28:11.969 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"num_strays":0}} 2024-03-06T07:28:11.970 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-03-06T07:28:11.970 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 Processor -- start 2024-03-06T07:28:12.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 -- start start 2024-03-06T07:28:12.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59900b3530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5990071960 0x7f5990071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f599006cd50 0x7f599006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59900722e0 con 0x7f599006cd50 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59900b2800 con 0x7f5990071960 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.071+0000 7f5994fa0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59900cf460 con 0x7f599015a1e0 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f599006cd50 0x7f599006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f599006cd50 0x7f599006d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54652/0 (socket says 172.21.15.72:54652) 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 -- 172.21.15.72:0/3448474005 learned_addr learned my addr 172.21.15.72:0/3448474005 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598f7fe640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59900b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598effd640 1 --2- 172.21.15.72:0/3448474005 >> v2:172.21.15.146:3300/0 conn(0x7f5990071960 0x7f5990071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 -- 172.21.15.72:0/3448474005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 msgr2=0x7f59900b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59900b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 -- 172.21.15.72:0/3448474005 >> v2:172.21.15.146:3300/0 conn(0x7f5990071960 msgr2=0x7f5990071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 --2- 172.21.15.72:0/3448474005 >> v2:172.21.15.146:3300/0 conn(0x7f5990071960 0x7f5990071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598ffff640 1 -- 172.21.15.72:0/3448474005 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5990072500 con 0x7f599006cd50 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598f7fe640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59900b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.072+0000 7f598effd640 1 --2- 172.21.15.72:0/3448474005 >> v2:172.21.15.146:3300/0 conn(0x7f5990071960 0x7f5990071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:12.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f598ffff640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f599006cd50 0x7f599006d1e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f599006e880 tx=0x7f598802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f30b5a1914215b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f598e7fc640 1 -- 172.21.15.72:0/3448474005 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988035020 con 0x7f599006cd50 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f598e7fc640 1 -- 172.21.15.72:0/3448474005 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5988031870 con 0x7f599006cd50 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f598e7fc640 1 -- 172.21.15.72:0/3448474005 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988031c40 con 0x7f599006cd50 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f5994fa0640 1 -- 172.21.15.72:0/3448474005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f599006cd50 msgr2=0x7f599006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f5994fa0640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f599006cd50 0x7f599006d1e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f599006e880 tx=0x7f598802b910 comp rx=0 tx=0).stop 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f5994fa0640 1 -- 172.21.15.72:0/3448474005 shutdown_connections 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.073+0000 7f5994fa0640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f599006cd50 0x7f599006d1e0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.074+0000 7f5994fa0640 1 --2- 172.21.15.72:0/3448474005 >> v2:172.21.15.146:3300/0 conn(0x7f5990071960 0x7f5990071da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.074+0000 7f5994fa0640 1 --2- 172.21.15.72:0/3448474005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59900b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.074+0000 7f5994fa0640 1 -- 172.21.15.72:0/3448474005 >> 172.21.15.72:0/3448474005 conn(0x7f59900b16e0 msgr2=0x7f59900b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.074+0000 7f5994fa0640 1 -- 172.21.15.72:0/3448474005 shutdown_connections 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.074+0000 7f5994fa0640 1 -- 172.21.15.72:0/3448474005 wait complete. 2024-03-06T07:28:12.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.075+0000 7f5994fa0640 1 Processor -- start 2024-03-06T07:28:12.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.075+0000 7f5994fa0640 1 -- start start 2024-03-06T07:28:12.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f599006cd50 0x7f59901572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 0x7f5990158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59901f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59900b3010 con 0x7f5990071960 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59900b2d30 con 0x7f599006cd50 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f599006c4c0 con 0x7f599015a1e0 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 0x7f5990158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 0x7f5990158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54662/0 (socket says 172.21.15.72:54662) 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 -- 172.21.15.72:0/681961524 learned_addr learned my addr 172.21.15.72:0/681961524 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598ffff640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59901f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598f7fe640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f599006cd50 0x7f59901572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 -- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 msgr2=0x7f59901f74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59901f74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 -- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f599006cd50 msgr2=0x7f59901572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f599006cd50 0x7f59901572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 -- 172.21.15.72:0/681961524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59901594a0 con 0x7f5990071960 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f598effd640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 0x7f5990158c30 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f598400e810 tx=0x7f598400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598400f040 con 0x7f5990071960 2024-03-06T07:28:12.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5984004030 con 0x7f5990071960 2024-03-06T07:28:12.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5990159780 con 0x7f5990071960 2024-03-06T07:28:12.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984004400 con 0x7f5990071960 2024-03-06T07:28:12.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.076+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59901f6ff0 con 0x7f5990071960 2024-03-06T07:28:12.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.077+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5990068700 con 0x7f5990071960 2024-03-06T07:28:12.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.078+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f598400c6d0 con 0x7f5990071960 2024-03-06T07:28:12.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.078+0000 7f5977fff640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f597c06b500 0x7f597c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.078+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f5984089ff0 con 0x7f5990071960 2024-03-06T07:28:12.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.078+0000 7f598f7fe640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f597c06b500 0x7f597c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.079+0000 7f598f7fe640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f597c06b500 0x7f597c06d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5980003fd0 tx=0x7f5980003ce0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.082+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5984017020 con 0x7f5990071960 2024-03-06T07:28:12.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.296+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59900ce0f0 con 0x7f5990071960 2024-03-06T07:28:12.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.297+0000 7f5977fff640 1 -- 172.21.15.72:0/681961524 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f5984058d40 con 0x7f5990071960 2024-03-06T07:28:12.298 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:12.298 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:12.298 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:12.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f597c06b500 msgr2=0x7f597c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f597c06b500 0x7f597c06d9c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5980003fd0 tx=0x7f5980003ce0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 msgr2=0x7f5990158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 0x7f5990158c30 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f598400e810 tx=0x7f598400ecd0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 shutdown_connections 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f597c06b500 0x7f597c06d9c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f599015a1e0 0x7f59901f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5990071960 0x7f5990158c30 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 --2- 172.21.15.72:0/681961524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f599006cd50 0x7f59901572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 >> 172.21.15.72:0/681961524 conn(0x7f59900b16e0 msgr2=0x7f59900b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 shutdown_connections 2024-03-06T07:28:12.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.299+0000 7f5994fa0640 1 -- 172.21.15.72:0/681961524 wait complete. 2024-03-06T07:28:12.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-03-06T07:28:12.428 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-03-06T07:28:12.429 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-03-06T07:28:12.429 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.529+0000 7fa4c4a84640 1 Processor -- start 2024-03-06T07:28:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.529+0000 7fa4c4a84640 1 -- start start 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4c4a84640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa4c015bb20 0x7fa4c00d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4c4a84640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa4c00cfff0 0x7fa4c00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4c4a84640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c0161980 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4c4a84640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c0161c60 con 0x7fa4c00cfff0 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4c4a84640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c0161f40 con 0x7fa4c015bb20 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4b7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa4c00cfff0 0x7fa4c00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4bed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa4c00ce4b0 0x7fa4c00ce8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36912/0 (socket says 172.21.15.72:36912) 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.530+0000 7fa4bed9d640 1 -- 172.21.15.72:0/3434202507 learned_addr learned my addr 172.21.15.72:0/3434202507 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4be59c640 1 --2- 172.21.15.72:0/3434202507 >> v2:172.21.15.146:3300/0 conn(0x7fa4c015bb20 0x7fa4c00d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4b7fff640 1 -- 172.21.15.72:0/3434202507 >> v2:172.21.15.146:3300/0 conn(0x7fa4c015bb20 msgr2=0x7fa4c00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4b7fff640 1 --2- 172.21.15.72:0/3434202507 >> v2:172.21.15.146:3300/0 conn(0x7fa4c015bb20 0x7fa4c00d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4b7fff640 1 -- 172.21.15.72:0/3434202507 >> v1:172.21.15.72:6789/0 conn(0x7fa4c00ce4b0 legacy=0x7fa4c00ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4b7fff640 1 -- 172.21.15.72:0/3434202507 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c0162220 con 0x7fa4c00cfff0 2024-03-06T07:28:12.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4bdd9b640 1 -- 172.21.15.72:0/3434202507 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551619704 0 0) 0x7fa4c0161980 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4be59c640 1 --2- 172.21.15.72:0/3434202507 >> v2:172.21.15.146:3300/0 conn(0x7fa4c015bb20 0x7fa4c00d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4b7fff640 1 --2- 172.21.15.72:0/3434202507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00d04a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00d2bc0 tx=0x7fa4a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916f1a26e481e4ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.531+0000 7fa4bdd9b640 1 -- 172.21.15.72:0/3434202507 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8035020 con 0x7fa4c00cfff0 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4bdd9b640 1 -- 172.21.15.72:0/3434202507 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a8031830 con 0x7fa4c00cfff0 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4bdd9b640 1 -- 172.21.15.72:0/3434202507 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8031c00 con 0x7fa4c00cfff0 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 -- 172.21.15.72:0/3434202507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00cfff0 msgr2=0x7fa4c00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/3434202507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00d04a0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fa4c00d2bc0 tx=0x7fa4a802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 -- 172.21.15.72:0/3434202507 shutdown_connections 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/3434202507 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00d04a0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/3434202507 >> v2:172.21.15.146:3300/0 conn(0x7fa4c015bb20 0x7fa4c00d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 -- 172.21.15.72:0/3434202507 >> 172.21.15.72:0/3434202507 conn(0x7fa4c009a9f0 msgr2=0x7fa4c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 -- 172.21.15.72:0/3434202507 shutdown_connections 2024-03-06T07:28:12.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.532+0000 7fa4c4a84640 1 -- 172.21.15.72:0/3434202507 wait complete. 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 Processor -- start 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 -- start start 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 0x7fa4c0155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4c015bb20 0x7fa4c00c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00d24b0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00d2330 con 0x7fa4c00cfff0 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4c4a84640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c00d1980 con 0x7fa4c015bb20 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4be59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 0x7fa4c0155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.534+0000 7fa4be59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 0x7fa4c0155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47070/0 (socket says 172.21.15.72:47070) 2024-03-06T07:28:12.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 -- 172.21.15.72:0/1637036598 learned_addr learned my addr 172.21.15.72:0/1637036598 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4bed9d640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4c015bb20 0x7fa4c00c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4b7fff640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 -- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4c015bb20 msgr2=0x7fa4c00c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4c015bb20 0x7fa4c00c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 -- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4c00cfff0 msgr2=0x7fa4c00c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 -- 172.21.15.72:0/1637036598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c00ca510 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4be59c640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 0x7fa4c0155000 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000c820 tx=0x7fa4b000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0015070 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0007a30 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0007de0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c00ca7f0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.535+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c00cadb0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.537+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c004f330 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.537+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa4b0018070 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.537+0000 7fa4b6ffd640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4a006b4d0 0x7fa4a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.537+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fa4b008a1e0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.537+0000 7fa4b7fff640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4a006b4d0 0x7fa4a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.538+0000 7fa4b7fff640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4a006b4d0 0x7fa4a006d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8036000 tx=0x7fa4a8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.541+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa4b0058fb0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.754+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4c00b1820 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.754+0000 7fa4b6ffd640 1 -- 172.21.15.72:0/1637036598 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fa4b00588b0 con 0x7fa4c00ce4b0 2024-03-06T07:28:12.756 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:12.757 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:12.757 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4a006b4d0 msgr2=0x7fa4a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4a006b4d0 0x7fa4a006d990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8036000 tx=0x7fa4a8008030 comp rx=0 tx=0).stop 2024-03-06T07:28:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 msgr2=0x7fa4c0155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 0x7fa4c0155000 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fa4b000c820 tx=0x7fa4b000cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 shutdown_connections 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa4a006b4d0 0x7fa4a006d990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4c015bb20 0x7fa4c00c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4c00cfff0 0x7fa4c00c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 --2- 172.21.15.72:0/1637036598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4c00ce4b0 0x7fa4c0155000 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 >> 172.21.15.72:0/1637036598 conn(0x7fa4c009a9f0 msgr2=0x7fa4c00afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 shutdown_connections 2024-03-06T07:28:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.757+0000 7fa4c4a84640 1 -- 172.21.15.72:0/1637036598 wait complete. 2024-03-06T07:28:12.770 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:28:12.899 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:28:12.899 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:28:12.899 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-03-06T07:28:12.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.941+0000 7fc265cf9f40 1 Processor -- start 2024-03-06T07:28:12.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.941+0000 7fc265cf9f40 1 -- start start 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc265cf9f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 0x55bb56ecb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc265cf9f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f7f830 0x55bb56f98060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc265cf9f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 0x55bb56f985a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc265cf9f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb56e73b70 con 0x55bb56f877a0 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc265cf9f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb56e77ad0 con 0x7fc24c002480 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc265cf9f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb56ecf6c0 con 0x55bb56f7f830 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 0x55bb56f985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 0x55bb56f985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47082/0 (socket says 172.21.15.72:47082) 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 -- 172.21.15.72:0/1075274676 learned_addr learned my addr 172.21.15.72:0/1075274676 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 -- 172.21.15.72:0/1075274676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f7f830 msgr2=0x55bb56f98060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f7f830 0x55bb56f98060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 -- 172.21.15.72:0/1075274676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 msgr2=0x55bb56ecb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 0x55bb56ecb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 -- 172.21.15.72:0/1075274676 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb56ecc5a0 con 0x55bb56f877a0 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.942+0000 7fc263ffd640 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 0x55bb56f985a0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x55bb56ed0de0 tx=0x7fc25c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3aa59bebb5815c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc261ff9640 1 -- 172.21.15.72:0/1075274676 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c013700 con 0x55bb56f877a0 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc261ff9640 1 -- 172.21.15.72:0/1075274676 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c01a070 con 0x55bb56f877a0 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc261ff9640 1 -- 172.21.15.72:0/1075274676 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c013aa0 con 0x55bb56f877a0 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 -- 172.21.15.72:0/1075274676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 msgr2=0x55bb56f985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 0x55bb56f985a0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x55bb56ed0de0 tx=0x7fc25c005d80 comp rx=0 tx=0).stop 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 -- 172.21.15.72:0/1075274676 shutdown_connections 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f877a0 0x55bb56f985a0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f7f830 0x55bb56f98060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/1075274676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 0x55bb56ecb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 -- 172.21.15.72:0/1075274676 >> 172.21.15.72:0/1075274676 conn(0x55bb56ef7790 msgr2=0x55bb56f9ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 -- 172.21.15.72:0/1075274676 shutdown_connections 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.943+0000 7fc265cf9f40 1 -- 172.21.15.72:0/1075274676 wait complete. 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 Processor -- start 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 -- start start 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f7f830 0x55bb56f7f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f877a0 0x55bb56f81f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 0x55bb56f82450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb56ecc300 con 0x55bb56f7f830 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb56f88350 con 0x7fc24c002480 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc265cf9f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bb56f885b0 con 0x55bb56f877a0 2024-03-06T07:28:12.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc2637fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f7f830 0x55bb56f7f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.944+0000 7fc2637fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f7f830 0x55bb56f7f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47088/0 (socket says 172.21.15.72:47088) 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 -- 172.21.15.72:0/2591351482 learned_addr learned my addr 172.21.15.72:0/2591351482 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc262ffb640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f877a0 0x55bb56f81f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc263ffd640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 0x55bb56f82450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 -- 172.21.15.72:0/2591351482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f877a0 msgr2=0x55bb56f81f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55bb56f877a0 0x55bb56f81f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 -- 172.21.15.72:0/2591351482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 msgr2=0x55bb56f82450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc24c002480 0x55bb56f82450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 -- 172.21.15.72:0/2591351482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb56f800b0 con 0x55bb56f7f830 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc2637fc640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f7f830 0x55bb56f7f010 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fc25801c270 tx=0x7fc25804cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc260ff7640 1 -- 172.21.15.72:0/2591351482 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258063020 con 0x55bb56f7f830 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc260ff7640 1 -- 172.21.15.72:0/2591351482 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25804ce50 con 0x55bb56f7f830 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bb56f80390 con 0x55bb56f7f830 2024-03-06T07:28:12.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc260ff7640 1 -- 172.21.15.72:0/2591351482 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258009610 con 0x55bb56f7f830 2024-03-06T07:28:12.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.945+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb56f80950 con 0x55bb56f7f830 2024-03-06T07:28:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.947+0000 7fc260ff7640 1 -- 172.21.15.72:0/2591351482 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc258066020 con 0x55bb56f7f830 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.947+0000 7fc260ff7640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc250069eb0 0x7fc25006c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.947+0000 7fc260ff7640 1 -- 172.21.15.72:0/2591351482 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fc2580caf70 con 0x55bb56f7f830 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc262ffb640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc250069eb0 0x7fc25006c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55bb56f9b150 0x55bb56f99640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116) v9 -- 0x55bb56f7fc40 con 0x55bb56f9b150 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc263ffd640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55bb56f9b150 0x55bb56f99640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc262ffb640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc250069eb0 0x7fc25006c370 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55bb56f9adb0 tx=0x7fc25400d340 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc263ffd640 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55bb56f9b150 0x55bb56f99640 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc263ffd640 1 -- 172.21.15.72:0/2591351482 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+93 (crc 0 0 0) 0x7fc25c0139a0 con 0x55bb56f9b150 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55bb56f9b150 msgr2=0x55bb56f99640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.948+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x55bb56f9b150 0x55bb56f99640 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc250069eb0 msgr2=0x7fc25006c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc250069eb0 0x7fc25006c370 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55bb56f9adb0 tx=0x7fc25400d340 comp rx=0 tx=0).stop 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f7f830 msgr2=0x55bb56f7f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 --2- 172.21.15.72:0/2591351482 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55bb56f7f830 0x55bb56f7f010 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fc25801c270 tx=0x7fc25804cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc263ffd640 1 -- 172.21.15.72:0/2591351482 reap_dead start 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 shutdown_connections 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 >> 172.21.15.72:0/2591351482 conn(0x55bb56ef7790 msgr2=0x55bb56ecb860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 shutdown_connections 2024-03-06T07:28:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:12.949+0000 7fc265cf9f40 1 -- 172.21.15.72:0/2591351482 wait complete. 2024-03-06T07:28:12.956 INFO:teuthology.orchestra.run.smithi072.stdout:W�file_ddir_2, 2024-03-06T07:28:12.956 DEBUG:teuthology.orchestra.run.smithi072:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-03-06T07:28:13.053 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:28:13.053 INFO:teuthology.orchestra.run.smithi072.stdout: "ino": 1099511627777, 2024-03-06T07:28:13.053 INFO:teuthology.orchestra.run.smithi072.stdout: "ancestors": [ 2024-03-06T07:28:13.053 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:13.053 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1099511628278, 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "file_d", 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 25 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "dirino": 1, 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "dname": "dir_2", 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "version": 44 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "pool": 17, 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: "old_pools": [] 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:28:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:13.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:13.057 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-03-06T07:28:13.115 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.271+0000 7fde1350f640 1 Processor -- start 2024-03-06T07:28:13.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.271+0000 7fde1350f640 1 -- start start 2024-03-06T07:28:13.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde1350f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde0c13dc10 0x7fde0c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde1350f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde1350f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c15f580 con 0x7fde0c15e850 2024-03-06T07:28:13.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde1350f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c0cf3f0 con 0x7fde0c13dc10 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde1350f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c15b210 con 0x7fde0c15bf30 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde0c13dc10 0x7fde0c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fde0c13dc10 0x7fde0c0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47098/0 (socket says 172.21.15.72:47098) 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 -- 172.21.15.72:0/3708469410 learned_addr learned my addr 172.21.15.72:0/3708469410 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde10aaa640 1 --2- 172.21.15.72:0/3708469410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 -- 172.21.15.72:0/3708469410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 msgr2=0x7fde0c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 --2- 172.21.15.72:0/3708469410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 -- 172.21.15.72:0/3708469410 >> v1:172.21.15.146:6789/0 conn(0x7fde0c15e850 legacy=0x7fde0c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.272+0000 7fde112ab640 1 -- 172.21.15.72:0/3708469410 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c15b450 con 0x7fde0c13dc10 2024-03-06T07:28:13.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde112ab640 1 --2- 172.21.15.72:0/3708469410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c0b2950 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fde08009870 tx=0x7fde0802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e26ccaf01f4f33c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde03fff640 1 -- 172.21.15.72:0/3708469410 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08035020 con 0x7fde0c13dc10 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde03fff640 1 -- 172.21.15.72:0/3708469410 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde08031640 con 0x7fde0c13dc10 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde03fff640 1 -- 172.21.15.72:0/3708469410 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0802d040 con 0x7fde0c13dc10 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde1350f640 1 -- 172.21.15.72:0/3708469410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 msgr2=0x7fde0c0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde1350f640 1 --2- 172.21.15.72:0/3708469410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c0b2950 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fde08009870 tx=0x7fde0802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde1350f640 1 -- 172.21.15.72:0/3708469410 shutdown_connections 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde1350f640 1 --2- 172.21.15.72:0/3708469410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde1350f640 1 --2- 172.21.15.72:0/3708469410 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c0b2950 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.273+0000 7fde1350f640 1 -- 172.21.15.72:0/3708469410 >> 172.21.15.72:0/3708469410 conn(0x7fde0c0cf8a0 msgr2=0x7fde0c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.274+0000 7fde1350f640 1 -- 172.21.15.72:0/3708469410 shutdown_connections 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.274+0000 7fde1350f640 1 -- 172.21.15.72:0/3708469410 wait complete. 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 Processor -- start 2024-03-06T07:28:13.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 -- start start 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c1393d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c15e850 0x7fde0c0d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0708d0 con 0x7fde0c13dc10 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c13dff0 con 0x7fde0c15e850 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde1350f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c070a50 con 0x7fde0c15bf30 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.275+0000 7fde112ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c1393d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c1393d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47106/0 (socket says 172.21.15.72:47106) 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 -- 172.21.15.72:0/700532598 learned_addr learned my addr 172.21.15.72:0/700532598 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde11aac640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c15e850 0x7fde0c0d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde10aaa640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 -- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 msgr2=0x7fde0c0cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 -- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c15e850 msgr2=0x7fde0c0d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c15e850 0x7fde0c0d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 -- 172.21.15.72:0/700532598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c15a0f0 con 0x7fde0c13dc10 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde112ab640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c1393d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fde08000c00 tx=0x7fde080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08040070 con 0x7fde0c13dc10 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde080399f0 con 0x7fde0c13dc10 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08035040 con 0x7fde0c13dc10 2024-03-06T07:28:13.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c159e10 con 0x7fde0c13dc10 2024-03-06T07:28:13.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.276+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c155ca0 con 0x7fde0c13dc10 2024-03-06T07:28:13.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.277+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c04f330 con 0x7fde0c13dc10 2024-03-06T07:28:13.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.278+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde0804b060 con 0x7fde0c13dc10 2024-03-06T07:28:13.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.279+0000 7fde01ffb640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddf006b4d0 0x7fddf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:13.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.279+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fde080abce0 con 0x7fde0c13dc10 2024-03-06T07:28:13.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.279+0000 7fde10aaa640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddf006b4d0 0x7fddf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:13.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.279+0000 7fde10aaa640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddf006b4d0 0x7fddf006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fde0c1f4fb0 tx=0x7fddfc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:13.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.283+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde0807aa20 con 0x7fde0c13dc10 2024-03-06T07:28:13.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.499+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde0c06eb00 con 0x7fde0c13dc10 2024-03-06T07:28:13.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.499+0000 7fde01ffb640 1 -- 172.21.15.72:0/700532598 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fde0807a320 con 0x7fde0c13dc10 2024-03-06T07:28:13.502 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:13.502 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:13.502 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddf006b4d0 msgr2=0x7fddf006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddf006b4d0 0x7fddf006d990 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fde0c1f4fb0 tx=0x7fddfc008040 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 msgr2=0x7fde0c1393d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c1393d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fde08000c00 tx=0x7fde080046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 shutdown_connections 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddf006b4d0 0x7fddf006d990 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c15e850 0x7fde0c0d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c15bf30 0x7fde0c0cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 --2- 172.21.15.72:0/700532598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde0c13dc10 0x7fde0c1393d0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 >> 172.21.15.72:0/700532598 conn(0x7fde0c0cf8a0 msgr2=0x7fde0c0cda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.501+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 shutdown_connections 2024-03-06T07:28:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:13.502+0000 7fde1350f640 1 -- 172.21.15.72:0/700532598 wait complete. 2024-03-06T07:28:13.514 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:13.629 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:28:13.630 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:28:13.630 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-03-06T07:28:18.631 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:18.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.730+0000 7f16a44c9640 1 Processor -- start 2024-03-06T07:28:18.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.730+0000 7f16a44c9640 1 -- start start 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a44c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f169c15dfe0 0x7f169c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a44c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0cf830 0x7f169c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a44c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f169c0d27f0 0x7f169c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a44c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c0d2c50 con 0x7f169c15dfe0 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a44c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169c0b31e0 con 0x7f169c0d27f0 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a44c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b3d60 con 0x7f169c0cf830 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a2265640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f169c15dfe0 0x7f169c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a2265640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f169c15dfe0 0x7f169c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47118/0 (socket says 172.21.15.72:47118) 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a2265640 1 -- 172.21.15.72:0/531167471 learned_addr learned my addr 172.21.15.72:0/531167471 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a2a66640 1 --2- 172.21.15.72:0/531167471 >> v2:172.21.15.146:3300/0 conn(0x7f169c0d27f0 0x7f169c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.731+0000 7f16a1a64640 1 --2- 172.21.15.72:0/531167471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0cf830 0x7f169c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a2265640 1 -- 172.21.15.72:0/531167471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0cf830 msgr2=0x7f169c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a2265640 1 --2- 172.21.15.72:0/531167471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0cf830 0x7f169c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a2265640 1 -- 172.21.15.72:0/531167471 >> v2:172.21.15.146:3300/0 conn(0x7f169c0d27f0 msgr2=0x7f169c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a2265640 1 --2- 172.21.15.72:0/531167471 >> v2:172.21.15.146:3300/0 conn(0x7f169c0d27f0 0x7f169c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a2265640 1 -- 172.21.15.72:0/531167471 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0b3430 con 0x7f169c15dfe0 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a2265640 1 --2- 172.21.15.72:0/531167471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c0cf2f0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f169c067ad0 tx=0x7f168c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2953b320c6dc1695 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a1263640 1 -- 172.21.15.72:0/531167471 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c035020 con 0x7f169c15dfe0 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a1263640 1 -- 172.21.15.72:0/531167471 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c031870 con 0x7f169c15dfe0 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.732+0000 7f16a1263640 1 -- 172.21.15.72:0/531167471 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168c031c40 con 0x7f169c15dfe0 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 -- 172.21.15.72:0/531167471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 msgr2=0x7f169c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 --2- 172.21.15.72:0/531167471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c0cf2f0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f169c067ad0 tx=0x7f168c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 -- 172.21.15.72:0/531167471 shutdown_connections 2024-03-06T07:28:18.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 --2- 172.21.15.72:0/531167471 >> v2:172.21.15.146:3300/0 conn(0x7f169c0d27f0 0x7f169c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 --2- 172.21.15.72:0/531167471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0cf830 0x7f169c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 --2- 172.21.15.72:0/531167471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c0cf2f0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 -- 172.21.15.72:0/531167471 >> 172.21.15.72:0/531167471 conn(0x7f169c0d3cd0 msgr2=0x7f169c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 -- 172.21.15.72:0/531167471 shutdown_connections 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.733+0000 7f16a44c9640 1 -- 172.21.15.72:0/531167471 wait complete. 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.734+0000 7f16a44c9640 1 Processor -- start 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.734+0000 7f16a44c9640 1 -- start start 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a44c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0cf830 0x7f169c1f4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a44c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0d27f0 0x7f169c160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a44c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a44c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0d1c70 con 0x7f169c15dfe0 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a44c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0b39f0 con 0x7f169c0cf830 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a44c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169c0d1df0 con 0x7f169c0d27f0 2024-03-06T07:28:18.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c15fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47122/0 (socket says 172.21.15.72:47122) 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 -- 172.21.15.72:0/1043680192 learned_addr learned my addr 172.21.15.72:0/1043680192 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a1a64640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0d27f0 0x7f169c160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2265640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0cf830 0x7f169c1f4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 -- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0d27f0 msgr2=0x7f169c160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0d27f0 0x7f169c160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 -- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0cf830 msgr2=0x7f169c1f4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0cf830 0x7f169c1f4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 -- 172.21.15.72:0/1043680192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0b9740 con 0x7f169c15dfe0 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a1a64640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0d27f0 0x7f169c160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2265640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0cf830 0x7f169c1f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.735+0000 7f16a2a66640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c15fcb0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f169400efc0 tx=0x7f169400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.736+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694018020 con 0x7f169c15dfe0 2024-03-06T07:28:18.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.736+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1694014480 con 0x7f169c15dfe0 2024-03-06T07:28:18.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.736+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169c0b9a20 con 0x7f169c15dfe0 2024-03-06T07:28:18.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.736+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694014870 con 0x7f169c15dfe0 2024-03-06T07:28:18.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.736+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f169c0cceb0 con 0x7f169c15dfe0 2024-03-06T07:28:18.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.737+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169c0b5200 con 0x7f169c15dfe0 2024-03-06T07:28:18.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.737+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1694005d10 con 0x7f169c15dfe0 2024-03-06T07:28:18.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.738+0000 7f1692ffd640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f168006b4d0 0x7f168006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:18.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.738+0000 7f16a2265640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f168006b4d0 0x7f168006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:18.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.738+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f169408b1c0 con 0x7f169c15dfe0 2024-03-06T07:28:18.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.738+0000 7f16a2265640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f168006b4d0 0x7f168006d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f168c036000 tx=0x7f168c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:18.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.742+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f169c0b5200 con 0x7f169c15dfe0 2024-03-06T07:28:18.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.957+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f169c0cd0b0 con 0x7f169c15dfe0 2024-03-06T07:28:18.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.958+0000 7f1692ffd640 1 -- 172.21.15.72:0/1043680192 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f169c0cd0b0 con 0x7f169c15dfe0 2024-03-06T07:28:18.960 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:18.960 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:18.961 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f168006b4d0 msgr2=0x7f168006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f168006b4d0 0x7f168006d990 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f168c036000 tx=0x7f168c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:18.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 msgr2=0x7f169c15fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c15fcb0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f169400efc0 tx=0x7f169400c490 comp rx=0 tx=0).stop 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 shutdown_connections 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f168006b4d0 0x7f168006d990 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f169c15dfe0 0x7f169c15fcb0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f169c0d27f0 0x7f169c160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 --2- 172.21.15.72:0/1043680192 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f169c0cf830 0x7f169c1f4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 >> 172.21.15.72:0/1043680192 conn(0x7f169c0d3cd0 msgr2=0x7f169c06cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 shutdown_connections 2024-03-06T07:28:18.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:18.960+0000 7f16a44c9640 1 -- 172.21.15.72:0/1043680192 wait complete. 2024-03-06T07:28:18.973 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:19.089 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:19.089 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:19.089 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.188+0000 7fcf56177640 1 Processor -- start 2024-03-06T07:28:19.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.188+0000 7fcf56177640 1 -- start start 2024-03-06T07:28:19.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.189+0000 7fcf56177640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500d22b0 0x7fcf500d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.189+0000 7fcf56177640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcf500b41f0 0x7fcf500b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.189+0000 7fcf56177640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500b3430 con 0x7fcf5015dfe0 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.189+0000 7fcf56177640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf500d2c50 con 0x7fcf500b41f0 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.189+0000 7fcf56177640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500b31e0 con 0x7fcf500d22b0 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcf500b41f0 0x7fcf500b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcf5015dfe0 0x7fcf500cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36930/0 (socket says 172.21.15.72:36930) 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4f7fe640 1 -- 172.21.15.72:0/3509773975 learned_addr learned my addr 172.21.15.72:0/3509773975 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4effd640 1 --2- 172.21.15.72:0/3509773975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500d22b0 0x7fcf500d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4ffff640 1 -- 172.21.15.72:0/3509773975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500d22b0 msgr2=0x7fcf500d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4ffff640 1 --2- 172.21.15.72:0/3509773975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500d22b0 0x7fcf500d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4ffff640 1 -- 172.21.15.72:0/3509773975 >> v1:172.21.15.72:6789/0 conn(0x7fcf5015dfe0 legacy=0x7fcf500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4e7fc640 1 -- 172.21.15.72:0/3509773975 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630169502 0 0) 0x7fcf500b3430 con 0x7fcf5015dfe0 2024-03-06T07:28:19.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.190+0000 7fcf4ffff640 1 -- 172.21.15.72:0/3509773975 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500b3710 con 0x7fcf500b41f0 2024-03-06T07:28:19.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.191+0000 7fcf4ffff640 1 --2- 172.21.15.72:0/3509773975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500b41f0 0x7fcf500b4650 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009d90 tx=0x7fcf4802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786ec3d67e3d2348 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.191+0000 7fcf4e7fc640 1 -- 172.21.15.72:0/3509773975 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48039070 con 0x7fcf500b41f0 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.191+0000 7fcf4e7fc640 1 -- 172.21.15.72:0/3509773975 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4802cc50 con 0x7fcf500b41f0 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.191+0000 7fcf4e7fc640 1 -- 172.21.15.72:0/3509773975 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf48034440 con 0x7fcf500b41f0 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 -- 172.21.15.72:0/3509773975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500b41f0 msgr2=0x7fcf500b4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 --2- 172.21.15.72:0/3509773975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500b41f0 0x7fcf500b4650 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fcf48009d90 tx=0x7fcf4802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 -- 172.21.15.72:0/3509773975 shutdown_connections 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 --2- 172.21.15.72:0/3509773975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500b41f0 0x7fcf500b4650 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 --2- 172.21.15.72:0/3509773975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500d22b0 0x7fcf500d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 -- 172.21.15.72:0/3509773975 >> 172.21.15.72:0/3509773975 conn(0x7fcf500d3cd0 msgr2=0x7fcf500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 -- 172.21.15.72:0/3509773975 shutdown_connections 2024-03-06T07:28:19.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.192+0000 7fcf56177640 1 -- 172.21.15.72:0/3509773975 wait complete. 2024-03-06T07:28:19.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.193+0000 7fcf56177640 1 Processor -- start 2024-03-06T07:28:19.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.193+0000 7fcf56177640 1 -- start start 2024-03-06T07:28:19.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf56177640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500b41f0 0x7fcf501f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf56177640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 0x7fcf501522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf56177640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf5015dfe0 0x7fcf501f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf56177640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500cfa00 con 0x7fcf500d22b0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf56177640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500b3d60 con 0x7fcf5015dfe0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf56177640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf500d1c70 con 0x7fcf500b41f0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 0x7fcf501522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 0x7fcf501522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47128/0 (socket says 172.21.15.72:47128) 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 -- 172.21.15.72:0/1884497782 learned_addr learned my addr 172.21.15.72:0/1884497782 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4ffff640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf5015dfe0 0x7fcf501f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4f7fe640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500b41f0 0x7fcf501f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 -- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500b41f0 msgr2=0x7fcf501f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500b41f0 0x7fcf501f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 -- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf5015dfe0 msgr2=0x7fcf501f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf5015dfe0 0x7fcf501f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 -- 172.21.15.72:0/1884497782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500b7a20 con 0x7fcf500d22b0 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4ffff640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf5015dfe0 0x7fcf501f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:19.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.194+0000 7fcf4effd640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 0x7fcf501522b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800d830 tx=0x7fcf3800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.195+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3800b660 con 0x7fcf500d22b0 2024-03-06T07:28:19.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.195+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf38019070 con 0x7fcf500d22b0 2024-03-06T07:28:19.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.195+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3800ba20 con 0x7fcf500d22b0 2024-03-06T07:28:19.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.195+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf500cbf70 con 0x7fcf500d22b0 2024-03-06T07:28:19.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.195+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf500cc530 con 0x7fcf500d22b0 2024-03-06T07:28:19.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.196+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5004f330 con 0x7fcf500d22b0 2024-03-06T07:28:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.196+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf3801e460 con 0x7fcf500d22b0 2024-03-06T07:28:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.196+0000 7fcf47fff640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf3006b4d0 0x7fcf3006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.197+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fcf3808a9c0 con 0x7fcf500d22b0 2024-03-06T07:28:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.197+0000 7fcf4f7fe640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf3006b4d0 0x7fcf3006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.197+0000 7fcf4f7fe640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf3006b4d0 0x7fcf3006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcf40000f60 tx=0x7fcf40007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.205+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf3808ce20 con 0x7fcf500d22b0 2024-03-06T07:28:19.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.421+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf500ce910 con 0x7fcf500d22b0 2024-03-06T07:28:19.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.422+0000 7fcf47fff640 1 -- 172.21.15.72:0/1884497782 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fcf38059700 con 0x7fcf500d22b0 2024-03-06T07:28:19.424 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:19.424 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:19.425 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf3006b4d0 msgr2=0x7fcf3006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf3006b4d0 0x7fcf3006d990 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcf40000f60 tx=0x7fcf40007c00 comp rx=0 tx=0).stop 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 msgr2=0x7fcf501522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 0x7fcf501522b0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fcf3800d830 tx=0x7fcf3800dcf0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 shutdown_connections 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf3006b4d0 0x7fcf3006d990 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf5015dfe0 0x7fcf501f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf500d22b0 0x7fcf501522b0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 --2- 172.21.15.72:0/1884497782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf500b41f0 0x7fcf501f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 >> 172.21.15.72:0/1884497782 conn(0x7fcf500d3cd0 msgr2=0x7fcf5006cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 shutdown_connections 2024-03-06T07:28:19.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.424+0000 7fcf56177640 1 -- 172.21.15.72:0/1884497782 wait complete. 2024-03-06T07:28:19.437 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-03-06T07:28:19.553 INFO:teuthology.orchestra.run.smithi072.stdout:{"purge_queue":{"pq_executed":1}} 2024-03-06T07:28:19.554 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-03-06T07:28:19.554 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:19.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.652+0000 7f7e5e271640 1 Processor -- start 2024-03-06T07:28:19.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.652+0000 7f7e5e271640 1 -- start start 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5e271640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 0x7f7e580cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5e271640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e580cf830 0x7f7e580d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5e271640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e580b3430 con 0x7f7e580d27f0 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5e271640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e580d2c50 con 0x7f7e580cf830 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5e271640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580b31e0 con 0x7f7e5815dfe0 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5c80e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e580d27f0 0x7f7e580d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36934/0 (socket says 172.21.15.72:36934) 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e5c80e640 1 -- 172.21.15.72:0/446758036 learned_addr learned my addr 172.21.15.72:0/446758036 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576028344 0 0) 0x7f7e580b3430 con 0x7f7e580d27f0 2024-03-06T07:28:19.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e57fff640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 0x7f7e580cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e40003520 con 0x7f7e580d27f0 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.653+0000 7f7e577fe640 1 --2- 172.21.15.72:0/446758036 >> v2:172.21.15.146:3300/0 conn(0x7f7e580cf830 0x7f7e580d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432571225 0 0) 0x7f7e40003520 con 0x7f7e580d27f0 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 msgr2=0x7f7e580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 0x7f7e580cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 >> v2:172.21.15.146:3300/0 conn(0x7f7e580cf830 msgr2=0x7f7e580d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 --2- 172.21.15.72:0/446758036 >> v2:172.21.15.146:3300/0 conn(0x7f7e580cf830 0x7f7e580d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580b3710 con 0x7f7e580d27f0 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f7e500019f0 con 0x7f7e580d27f0 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 >> v1:172.21.15.72:6789/0 conn(0x7f7e580d27f0 legacy=0x7f7e580d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 0x7f7e400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e40003f30 0x7f7e40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e40004d40 0x7f7e400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580b31e0 con 0x7f7e40004d40 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4000b9c0 con 0x7f7e40003f30 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4000bb40 con 0x7f7e5815dfe0 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e5c80e640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e40004d40 0x7f7e400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e577fe640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e40003f30 0x7f7e40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e57fff640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 0x7f7e400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e5c80e640 1 -- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 msgr2=0x7f7e400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e5c80e640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e5815dfe0 0x7f7e400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e5c80e640 1 -- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e40003f30 msgr2=0x7f7e40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e5c80e640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e40003f30 0x7f7e40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.654+0000 7f7e5c80e640 1 -- 172.21.15.72:0/446758036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40003520 con 0x7f7e40004d40 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.655+0000 7f7e5c80e640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e40004d40 0x7f7e400051c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000a920 tx=0x7f7e40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6471b00336229df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.655+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000a4c0 con 0x7f7e40004d40 2024-03-06T07:28:19.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.655+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e50034650 con 0x7f7e40004d40 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.655+0000 7f7e56ffd640 1 -- 172.21.15.72:0/446758036 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e50034a80 con 0x7f7e40004d40 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5e271640 1 -- 172.21.15.72:0/446758036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e40004d40 msgr2=0x7f7e400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5e271640 1 --2- 172.21.15.72:0/446758036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e40004d40 0x7f7e400051c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000a920 tx=0x7f7e40005830 comp rx=0 tx=0).stop 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5c80e640 1 -- 172.21.15.72:0/446758036 reap_dead start 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5e271640 1 -- 172.21.15.72:0/446758036 shutdown_connections 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5e271640 1 -- 172.21.15.72:0/446758036 >> 172.21.15.72:0/446758036 conn(0x7f7e580d3cd0 msgr2=0x7f7e580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5e271640 1 -- 172.21.15.72:0/446758036 shutdown_connections 2024-03-06T07:28:19.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.656+0000 7f7e5e271640 1 -- 172.21.15.72:0/446758036 wait complete. 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.657+0000 7f7e5e271640 1 Processor -- start 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.657+0000 7f7e5e271640 1 -- start start 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5e271640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e40004d40 0x7f7e5814e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5e271640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 0x7f7e5815f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5e271640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5815acd0 0x7f7e580b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5e271640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580d1c70 con 0x7f7e5815a850 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5e271640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580b3d60 con 0x7f7e5815acd0 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5e271640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580d1df0 con 0x7f7e40004d40 2024-03-06T07:28:19.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 0x7f7e5815f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 0x7f7e5815f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47134/0 (socket says 172.21.15.72:47134) 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 -- 172.21.15.72:0/652263954 learned_addr learned my addr 172.21.15.72:0/652263954 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e57fff640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e40004d40 0x7f7e5814e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e5c80e640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5815acd0 0x7f7e580b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 -- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e40004d40 msgr2=0x7f7e5814e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e40004d40 0x7f7e5814e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 -- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5815acd0 msgr2=0x7f7e580b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5815acd0 0x7f7e580b8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 -- 172.21.15.72:0/652263954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580b7a70 con 0x7f7e5815a850 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.658+0000 7f7e577fe640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 0x7f7e5815f8e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c011960 tx=0x7f7e4c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.659+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c019070 con 0x7f7e5815a850 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.659+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c004240 con 0x7f7e5815a850 2024-03-06T07:28:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.659+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e580b8150 con 0x7f7e5815a850 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.659+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c00f7c0 con 0x7f7e5815a850 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.659+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e580b8710 con 0x7f7e5815a850 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.660+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e580ce3c0 con 0x7f7e5815a850 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.660+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e4c01c070 con 0x7f7e5815a850 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.660+0000 7f7e54ff9640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e4406b4d0 0x7f7e4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.660+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f7e4c08a6e0 con 0x7f7e5815a850 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.661+0000 7f7e57fff640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e4406b4d0 0x7f7e4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:19.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.661+0000 7f7e57fff640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e4406b4d0 0x7f7e4406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009bc0 tx=0x7f7e48009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:19.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.665+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e580ce3c0 con 0x7f7e5815a850 2024-03-06T07:28:19.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.879+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e5804f330 con 0x7f7e5815a850 2024-03-06T07:28:19.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.879+0000 7f7e54ff9640 1 -- 172.21.15.72:0/652263954 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f7e4c058dd0 con 0x7f7e5815a850 2024-03-06T07:28:19.881 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:19.881 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:19.882 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:19.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e4406b4d0 msgr2=0x7f7e4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e4406b4d0 0x7f7e4406d990 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009bc0 tx=0x7f7e48009190 comp rx=0 tx=0).stop 2024-03-06T07:28:19.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 msgr2=0x7f7e5815f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 0x7f7e5815f8e0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c011960 tx=0x7f7e4c011d60 comp rx=0 tx=0).stop 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 shutdown_connections 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e4406b4d0 0x7f7e4406d990 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e5815acd0 0x7f7e580b8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e5815a850 0x7f7e5815f8e0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 --2- 172.21.15.72:0/652263954 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e40004d40 0x7f7e5814e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 >> 172.21.15.72:0/652263954 conn(0x7f7e580d3cd0 msgr2=0x7f7e580cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 shutdown_connections 2024-03-06T07:28:19.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:19.881+0000 7f7e5e271640 1 -- 172.21.15.72:0/652263954 wait complete. 2024-03-06T07:28:19.894 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-03-06T07:28:20.010 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0}} 2024-03-06T07:28:20.010 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2 } } 2024-03-06T07:28:20.011 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.109+0000 7f1525193640 1 Processor -- start 2024-03-06T07:28:20.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.109+0000 7f1525193640 1 -- start start 2024-03-06T07:28:20.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f1525193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15200cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f1525193640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f15200cf830 0x7f15200d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f1525193640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15200d27f0 0x7f15200d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f1525193640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200d2c50 con 0x7f15200cf830 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f1525193640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15200b31e0 con 0x7f15200d27f0 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f1525193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15200b3d60 con 0x7f152015dfe0 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.110+0000 7f151e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f15200cf830 0x7f15200d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f15200cf830 0x7f15200d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47140/0 (socket says 172.21.15.72:47140) 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 -- 172.21.15.72:0/905302503 learned_addr learned my addr 172.21.15.72:0/905302503 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 -- 172.21.15.72:0/905302503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 msgr2=0x7f15200cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151f59e640 1 --2- 172.21.15.72:0/905302503 >> v2:172.21.15.146:3300/0 conn(0x7f15200d27f0 0x7f15200d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151ed9d640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15200cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15200cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 -- 172.21.15.72:0/905302503 >> v2:172.21.15.146:3300/0 conn(0x7f15200d27f0 msgr2=0x7f15200d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 --2- 172.21.15.72:0/905302503 >> v2:172.21.15.146:3300/0 conn(0x7f15200d27f0 0x7f15200d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151e59c640 1 -- 172.21.15.72:0/905302503 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15200b3430 con 0x7f15200cf830 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151f59e640 1 --2- 172.21.15.72:0/905302503 >> v2:172.21.15.146:3300/0 conn(0x7f15200d27f0 0x7f15200d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.111+0000 7f151ed9d640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15200cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.112+0000 7f151e59c640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200cf830 0x7f15200d22b0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f1520067ad0 tx=0x7f151402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2438162b2b8f11e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.112+0000 7f151dd9b640 1 -- 172.21.15.72:0/905302503 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514035020 con 0x7f15200cf830 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.112+0000 7f151dd9b640 1 -- 172.21.15.72:0/905302503 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1514031870 con 0x7f15200cf830 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.112+0000 7f151dd9b640 1 -- 172.21.15.72:0/905302503 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514031c40 con 0x7f15200cf830 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.112+0000 7f1525193640 1 -- 172.21.15.72:0/905302503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200cf830 msgr2=0x7f15200d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.112+0000 7f1525193640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200cf830 0x7f15200d22b0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f1520067ad0 tx=0x7f151402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 -- 172.21.15.72:0/905302503 shutdown_connections 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 --2- 172.21.15.72:0/905302503 >> v2:172.21.15.146:3300/0 conn(0x7f15200d27f0 0x7f15200d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200cf830 0x7f15200d22b0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 --2- 172.21.15.72:0/905302503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15200cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 -- 172.21.15.72:0/905302503 >> 172.21.15.72:0/905302503 conn(0x7f15200d3cd0 msgr2=0x7f15200d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 -- 172.21.15.72:0/905302503 shutdown_connections 2024-03-06T07:28:20.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.113+0000 7f1525193640 1 -- 172.21.15.72:0/905302503 wait complete. 2024-03-06T07:28:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.114+0000 7f1525193640 1 Processor -- start 2024-03-06T07:28:20.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.114+0000 7f1525193640 1 -- start start 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f1525193640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15200cf830 0x7f15201602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f1525193640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 0x7f15201f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f1525193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15201f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f1525193640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200d1c70 con 0x7f15200d27f0 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f1525193640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15200b39f0 con 0x7f15200cf830 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f1525193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15200d1df0 con 0x7f152015dfe0 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f151e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 0x7f15201f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f151e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 0x7f15201f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47146/0 (socket says 172.21.15.72:47146) 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.115+0000 7f151e59c640 1 -- 172.21.15.72:0/4095749140 learned_addr learned my addr 172.21.15.72:0/4095749140 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151f59e640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15201f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151ed9d640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15200cf830 0x7f15201602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151e59c640 1 -- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 msgr2=0x7f15201f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151e59c640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15201f73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151e59c640 1 -- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15200cf830 msgr2=0x7f15201602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151e59c640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15200cf830 0x7f15201602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151e59c640 1 -- 172.21.15.72:0/4095749140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1520161660 con 0x7f15200d27f0 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151ed9d640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15200cf830 0x7f15201602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f151e59c640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 0x7f15201f6ea0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f15140090b0 tx=0x7f15140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514040070 con 0x7f15200d27f0 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f151403a420 con 0x7f15200d27f0 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152015e450 con 0x7f15200d27f0 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1514035040 con 0x7f15200d27f0 2024-03-06T07:28:20.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.116+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1520160b80 con 0x7f15200d27f0 2024-03-06T07:28:20.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.117+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152004f330 con 0x7f15200d27f0 2024-03-06T07:28:20.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.117+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f151404b060 con 0x7f15200d27f0 2024-03-06T07:28:20.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.118+0000 7f15077fe640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f150c06b4d0 0x7f150c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.118+0000 7f151ed9d640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f150c06b4d0 0x7f150c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.118+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f15140ac490 con 0x7f15200d27f0 2024-03-06T07:28:20.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.118+0000 7f151ed9d640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f150c06b4d0 0x7f150c06d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f151000a990 tx=0x7f1510008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.122+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f151407b1d0 con 0x7f15200d27f0 2024-03-06T07:28:20.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.338+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15200b54c0 con 0x7f15200d27f0 2024-03-06T07:28:20.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.338+0000 7f15077fe640 1 -- 172.21.15.72:0/4095749140 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f151407aad0 con 0x7f15200d27f0 2024-03-06T07:28:20.341 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:20.341 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:20.341 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f150c06b4d0 msgr2=0x7f150c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f150c06b4d0 0x7f150c06d990 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f151000a990 tx=0x7f1510008040 comp rx=0 tx=0).stop 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 msgr2=0x7f15201f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 0x7f15201f6ea0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f15140090b0 tx=0x7f15140046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 shutdown_connections 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f150c06b4d0 0x7f150c06d990 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f152015dfe0 0x7f15201f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f15200d27f0 0x7f15201f6ea0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 --2- 172.21.15.72:0/4095749140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15200cf830 0x7f15201602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.340+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 >> 172.21.15.72:0/4095749140 conn(0x7f15200d3cd0 msgr2=0x7f15200cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.341+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 shutdown_connections 2024-03-06T07:28:20.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.341+0000 7f1525193640 1 -- 172.21.15.72:0/4095749140 wait complete. 2024-03-06T07:28:20.354 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-03-06T07:28:20.472 INFO:teuthology.orchestra.run.smithi072.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-03-06T07:28:20.473 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-03-06T07:28:20.473 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.511+0000 7f89ec647f40 1 Processor -- start 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.511+0000 7f89ec647f40 1 -- start start 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ec647f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89d4002850 0x55da336c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ec647f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 0x55da336f7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ec647f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da336f33b0 0x55da336f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ec647f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da33713180 con 0x55da3379e760 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ec647f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da33670b70 con 0x7f89d4002850 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ec647f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da33713300 con 0x55da336f33b0 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 0x55da336f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 0x55da336f7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47166/0 (socket says 172.21.15.72:47166) 2024-03-06T07:28:20.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 -- 172.21.15.72:0/569731228 learned_addr learned my addr 172.21.15.72:0/569731228 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9988640 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89d4002850 0x55da336c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89ea189640 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da336f33b0 0x55da336f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 -- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da336f33b0 msgr2=0x55da336f7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da336f33b0 0x55da336f7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 -- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89d4002850 msgr2=0x55da336c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89d4002850 0x55da336c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 -- 172.21.15.72:0/569731228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da33712720 con 0x55da3379e760 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9988640 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89d4002850 0x55da336c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89e9187640 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 0x55da336f7680 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f89dc01d080 tx=0x7f89dc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa4fee2a22240ca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89dbfff640 1 -- 172.21.15.72:0/569731228 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc061070 con 0x55da3379e760 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89dbfff640 1 -- 172.21.15.72:0/569731228 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89dc0529f0 con 0x55da3379e760 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.512+0000 7f89dbfff640 1 -- 172.21.15.72:0/569731228 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc057040 con 0x55da3379e760 2024-03-06T07:28:20.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 -- 172.21.15.72:0/569731228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 msgr2=0x55da336f7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 0x55da336f7680 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f89dc01d080 tx=0x7f89dc002fb0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 -- 172.21.15.72:0/569731228 shutdown_connections 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da336f33b0 0x55da336f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55da3379e760 0x55da336f7680 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 --2- 172.21.15.72:0/569731228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89d4002850 0x55da336c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 -- 172.21.15.72:0/569731228 >> 172.21.15.72:0/569731228 conn(0x55da336f5340 msgr2=0x55da336f6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 -- 172.21.15.72:0/569731228 shutdown_connections 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 -- 172.21.15.72:0/569731228 wait complete. 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 Processor -- start 2024-03-06T07:28:20.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.513+0000 7f89ec647f40 1 -- start start 2024-03-06T07:28:20.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ec647f40 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55da336f33b0 0x55da33784d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ec647f40 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da3379e760 0x55da3377d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ec647f40 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89d4002850 0x55da3370f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ec647f40 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da336ca520 con 0x7f89d4002850 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ec647f40 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da336f8ea0 con 0x55da336f33b0 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ec647f40 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55da336f90c0 con 0x55da3379e760 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ea189640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89d4002850 0x55da3370f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ea189640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89d4002850 0x55da3370f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47174/0 (socket says 172.21.15.72:47174) 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 learned_addr learned my addr 172.21.15.72:0/3358406746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da3379e760 0x55da3377d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.514+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55da336f33b0 0x55da33784d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da3379e760 msgr2=0x55da3377d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55da3379e760 0x55da3377d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55da336f33b0 msgr2=0x55da33784d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55da336f33b0 0x55da33784d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da33781510 con 0x7f89d4002850 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89d4002850 0x55da3370f940 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f89e4005fd0 tx=0x7f89e400a990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89daffd640 1 -- 172.21.15.72:0/3358406746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4018030 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89daffd640 1 -- 172.21.15.72:0/3358406746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e4014660 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55da3377dd50 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89daffd640 1 -- 172.21.15.72:0/3358406746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4014850 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.515+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da3379ab60 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.517+0000 7f89daffd640 1 -- 172.21.15.72:0/3358406746 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89e401b030 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.517+0000 7f89daffd640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89cc069ee0 0x7f89cc06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.517+0000 7f89daffd640 1 -- 172.21.15.72:0/3358406746 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f89e4089210 con 0x7f89d4002850 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.517+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89cc069ee0 0x7f89cc06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.517+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x55da33787bc0 0x55da3379a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.517+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.518+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89cc069ee0 0x7f89cc06c3a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f89e0001050 tx=0x7f89e0000fe0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.518+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x55da33787bc0 0x55da3379a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.518+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x55da33787bc0 0x55da3379a4a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 1 ==== osd_op_reply(1 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f89dc068040 0x7f89dc006440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f89dc068040 0x7f89dc006440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f89dc068040 0x7f89dc006440 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 1 ==== osd_op_reply(2 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc068040 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f89e001c040 0x7f89e001c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.519+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f89e001c040 0x7f89e001c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f89e001c040 0x7f89e001c420 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 1 ==== osd_op_reply(3 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f89dc05c500 0x7f89dc06a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f89dc05c500 0x7f89dc06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.520+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f89dc05c500 0x7f89dc06a940 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.521+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== osd_op_reply(4 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.521+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.521+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 2 ==== osd_op_reply(5 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.521+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.521+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 2 ==== osd_op_reply(6 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.521+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.522+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 2 ==== osd_op_reply(7 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.522+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.522+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 3 ==== osd_op_reply(8 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.522+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.522+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 3 ==== osd_op_reply(9 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.522+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.523+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 3 ==== osd_op_reply(10 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.523+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.523+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 4 ==== osd_op_reply(11 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.523+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.523+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 4 ==== osd_op_reply(12 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.523+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.524+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 4 ==== osd_op_reply(13 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.524+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f89dc066920 0x7f89dc0773f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.524+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.524+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f89dc066920 0x7f89dc0773f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.524+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f89dc066920 0x7f89dc0773f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 1 ==== osd_op_reply(14 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 2 ==== osd_op_reply(15 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc068040 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 5 ==== osd_op_reply(16 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 2 ==== osd_op_reply(17 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f89e002e040 0x7f89e0019900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.525+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e002e040 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.526+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f89e002e040 0x7f89e0019900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.526+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f89e002e040 0x7f89e0019900 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.526+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.5 v2:172.21.15.146:6800/2545401618 1 ==== osd_op_reply(18 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e002e040 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.526+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.527+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 5 ==== osd_op_reply(19 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.527+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.527+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 5 ==== osd_op_reply(20 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.527+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.527+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 6 ==== osd_op_reply(21 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.527+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 6 ==== osd_op_reply(22 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89e9187640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f89dc08a940 0x7f89dc0893c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc08a940 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f89dc08a940 0x7f89dc0893c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89ea189640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f89dc08a940 0x7f89dc0893c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.2 v2:172.21.15.72:6817/3054266462 1 ==== osd_op_reply(23 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc08a940 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.528+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 3 ==== osd_op_reply(24 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc068040 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.529+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f89e0030940 0x7f89e000d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.529+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e0030940 2024-03-06T07:28:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.529+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f89e0030940 0x7f89e000d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.529+0000 7f89e9988640 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f89e0030940 0x7f89e000d3e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.529+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.6 v2:172.21.15.146:6808/1113968375 1 ==== osd_op_reply(25 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89e0030940 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.529+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.530+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 6 ==== osd_op_reply(26 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.530+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e002e040 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.530+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.5 v2:172.21.15.146:6800/2545401618 2 ==== osd_op_reply(27 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e002e040 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.530+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.530+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 7 ==== osd_op_reply(28 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.530+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc08a940 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.531+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.2 v2:172.21.15.72:6817/3054266462 2 ==== osd_op_reply(29 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc08a940 2024-03-06T07:28:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.531+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e0030940 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.531+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.6 v2:172.21.15.146:6808/1113968375 2 ==== osd_op_reply(30 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89e0030940 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.531+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.531+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 4 ==== osd_op_reply(31 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc068040 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.531+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 3 ==== osd_op_reply(32 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 8 ==== osd_op_reply(33 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 9 ==== osd_op_reply(34 [pgnls start_epoch 116 out=49b] v110'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 7 ==== osd_op_reply(35 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.532+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.533+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 5 ==== osd_op_reply(36 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc068040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.533+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.533+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 8 ==== osd_op_reply(37 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.533+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.533+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 4 ==== osd_op_reply(38 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.533+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e002e040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.534+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.5 v2:172.21.15.146:6800/2545401618 3 ==== osd_op_reply(39 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e002e040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.534+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e002e040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.534+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.5 v2:172.21.15.146:6800/2545401618 4 ==== osd_op_reply(40 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e002e040 2024-03-06T07:28:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.534+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e0030940 2024-03-06T07:28:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.534+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.6 v2:172.21.15.146:6808/1113968375 3 ==== osd_op_reply(41 [pgnls start_epoch 116 out=49b] v110'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89e0030940 2024-03-06T07:28:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.534+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.535+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 6 ==== osd_op_reply(42 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc068040 2024-03-06T07:28:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.535+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e002e040 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.535+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.5 v2:172.21.15.146:6800/2545401618 5 ==== osd_op_reply(43 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e002e040 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.535+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.535+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 5 ==== osd_op_reply(44 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.535+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.536+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 7 ==== osd_op_reply(45 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.536+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.536+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 8 ==== osd_op_reply(46 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.536+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.536+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 9 ==== osd_op_reply(47 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.536+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 10 ==== osd_op_reply(48 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 6 ==== osd_op_reply(49 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 7 ==== osd_op_reply(50 [pgnls start_epoch 116 out=49b] v110'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e001c040 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.3 v2:172.21.15.72:6808/4245662075 9 ==== osd_op_reply(51 [pgnls start_epoch 116 out=49b] v110'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x7f89e001c040 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.537+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x55da33787bc0 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.538+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 <== osd.7 v2:172.21.15.146:6816/1306818419 11 ==== osd_op_reply(52 [pgnls start_epoch 116 out=49b] v110'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89dc052ba0 con 0x55da33787bc0 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.538+0000 7f89e9187640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.538+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 10 ==== osd_op_reply(53 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.538+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.538+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 8 ==== osd_op_reply(54 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.538+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.539+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 9 ==== osd_op_reply(55 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.539+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc08a940 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.539+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.2 v2:172.21.15.72:6817/3054266462 3 ==== osd_op_reply(56 [pgnls start_epoch 116 out=49b] v110'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc08a940 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.539+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.539+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 11 ==== osd_op_reply(57 [pgnls start_epoch 116 out=49b] v116'12 uv12 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.539+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 10 ==== osd_op_reply(58 [pgnls start_epoch 116 out=49b] v110'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0010a80 con 0x7f89dc066920 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e0030940 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.6 v2:172.21.15.146:6808/1113968375 4 ==== osd_op_reply(59 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0050040 con 0x7f89e0030940 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 12 ==== osd_op_reply(60 [pgnls start_epoch 116 out=49b] v110'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc068040 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.1 v2:172.21.15.72:6800/2878132174 7 ==== osd_op_reply(61 [pgnls start_epoch 116 out=49b] v110'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0050040 con 0x7f89dc068040 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.540+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc066920 2024-03-06T07:28:20.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.4 v2:172.21.15.146:6820/3084745005 11 ==== osd_op_reply(62 [pgnls start_epoch 116 out=49b] v116'10 uv10 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0050040 con 0x7f89dc066920 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89e0030940 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 <== osd.6 v2:172.21.15.146:6808/1113968375 5 ==== osd_op_reply(63 [pgnls start_epoch 116 out=49b] v110'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e0050040 con 0x7f89e0030940 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89e9988640 1 -- 172.21.15.72:0/3358406746 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 116 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e116) v9 -- 0x55da337836e0 con 0x7f89dc05c500 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 <== osd.0 v2:172.21.15.72:6816/4178159476 13 ==== osd_op_reply(64 [pgnls start_epoch 116 out=49b] v110'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x7f89e4024070 con 0x7f89dc05c500 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f89dc05c500 msgr2=0x7f89dc06a940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f89dc05c500 0x7f89dc06a940 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f89dc068040 msgr2=0x7f89dc006440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f89dc068040 0x7f89dc006440 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f89dc08a940 msgr2=0x7f89dc0893c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f89dc08a940 0x7f89dc0893c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f89e001c040 msgr2=0x7f89e001c420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7f89e001c040 0x7f89e001c420 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f89dc066920 msgr2=0x7f89dc0773f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f89dc066920 0x7f89dc0773f0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.541+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 reap_dead start 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f89e002e040 msgr2=0x7f89e0019900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 reap_dead start 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7f89e002e040 0x7f89e0019900 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f89e0030940 msgr2=0x7f89e000d3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f89e0030940 0x7f89e000d3e0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x55da33787bc0 msgr2=0x55da3379a4a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x55da33787bc0 0x55da3379a4a0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89cc069ee0 msgr2=0x7f89cc06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89cc069ee0 0x7f89cc06c3a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f89e0001050 tx=0x7f89e0000fe0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89d4002850 msgr2=0x55da3370f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 --2- 172.21.15.72:0/3358406746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89d4002850 0x55da3370f940 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f89e4005fd0 tx=0x7f89e400a990 comp rx=0 tx=0).stop 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ea189640 1 -- 172.21.15.72:0/3358406746 reap_dead start 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 shutdown_connections 2024-03-06T07:28:20.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 >> 172.21.15.72:0/3358406746 conn(0x55da336f5340 msgr2=0x55da336f7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 shutdown_connections 2024-03-06T07:28:20.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.542+0000 7f89ec647f40 1 -- 172.21.15.72:0/3358406746 wait complete. 2024-03-06T07:28:20.555 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-03-06T07:28:20.555 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:20.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.688+0000 7fc93fb56640 1 Processor -- start 2024-03-06T07:28:20.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.689+0000 7fc93fb56640 1 -- start start 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.689+0000 7fc93fb56640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9380d0a80 0x7fc9380b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.689+0000 7fc93fb56640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380d25d0 con 0x7fc9380b4170 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.689+0000 7fc93fb56640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9380d2870 con 0x7fc9380d4760 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.689+0000 7fc93fb56640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9380d4bf0 con 0x7fc9380d0a80 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93e0f3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9380d0a80 0x7fc9380b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93e0f3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9380d0a80 0x7fc9380b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47180/0 (socket says 172.21.15.72:47180) 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93e0f3640 1 -- 172.21.15.72:0/30252084 learned_addr learned my addr 172.21.15.72:0/30252084 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:20.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93d8f2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc9380b4170 0x7fc9380d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36942/0 (socket says 172.21.15.72:36942) 2024-03-06T07:28:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93e0f3640 1 -- 172.21.15.72:0/30252084 >> v1:172.21.15.146:6789/0 conn(0x7fc9380d4760 legacy=0x7fc9380cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93e0f3640 1 -- 172.21.15.72:0/30252084 >> v1:172.21.15.72:6789/0 conn(0x7fc9380b4170 legacy=0x7fc9380d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93c8f0640 1 -- 172.21.15.72:0/30252084 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797180322 0 0) 0x7fc9380d25d0 con 0x7fc9380b4170 2024-03-06T07:28:20.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.690+0000 7fc93e0f3640 1 -- 172.21.15.72:0/30252084 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9380d2b50 con 0x7fc9380d0a80 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.691+0000 7fc93e0f3640 1 --2- 172.21.15.72:0/30252084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b6a90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc930009e50 tx=0x7fc93002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985db5f87f372b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.691+0000 7fc93c8f0640 1 -- 172.21.15.72:0/30252084 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93002cb20 con 0x7fc9380d0a80 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.691+0000 7fc93c8f0640 1 -- 172.21.15.72:0/30252084 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93002ccc0 con 0x7fc9380d0a80 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.691+0000 7fc93c8f0640 1 -- 172.21.15.72:0/30252084 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930034610 con 0x7fc9380d0a80 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 -- 172.21.15.72:0/30252084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 msgr2=0x7fc9380b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 --2- 172.21.15.72:0/30252084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b6a90 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fc930009e50 tx=0x7fc93002bf80 comp rx=0 tx=0).stop 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 -- 172.21.15.72:0/30252084 shutdown_connections 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 --2- 172.21.15.72:0/30252084 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b6a90 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 -- 172.21.15.72:0/30252084 >> 172.21.15.72:0/30252084 conn(0x7fc9380d3c40 msgr2=0x7fc9380d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 -- 172.21.15.72:0/30252084 shutdown_connections 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.692+0000 7fc93fb56640 1 -- 172.21.15.72:0/30252084 wait complete. 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.694+0000 7fc93fb56640 1 Processor -- start 2024-03-06T07:28:20.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.694+0000 7fc93fb56640 1 -- start start 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93fb56640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9380b4170 0x7fc93814bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93fb56640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93fb56640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93814f060 0x7fc9380b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93fb56640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380b4dd0 con 0x7fc9380d0a80 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93fb56640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380ce9e0 con 0x7fc9380b4170 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93fb56640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9380b4f50 con 0x7fc93814f060 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47184/0 (socket says 172.21.15.72:47184) 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 -- 172.21.15.72:0/1980593875 learned_addr learned my addr 172.21.15.72:0/1980593875 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93e0f3640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93814f060 0x7fc9380b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d8f2640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9380b4170 0x7fc93814bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 -- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93814f060 msgr2=0x7fc9380b8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93814f060 0x7fc9380b8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 -- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9380b4170 msgr2=0x7fc93814bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9380b4170 0x7fc93814bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 -- 172.21.15.72:0/1980593875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9381f7f40 con 0x7fc9380d0a80 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.695+0000 7fc93d0f1640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b88e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fc92800b3c0 tx=0x7fc92800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.696+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928011830 con 0x7fc9380d0a80 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.696+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc928016070 con 0x7fc9380d0a80 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.696+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93814e580 con 0x7fc9380d0a80 2024-03-06T07:28:20.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.696+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928011c50 con 0x7fc9380d0a80 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.696+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93814eb40 con 0x7fc9380d0a80 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.696+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93804f330 con 0x7fc9380d0a80 2024-03-06T07:28:20.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.697+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc92801b460 con 0x7fc9380d0a80 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.698+0000 7fc92e7fc640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc91c06b500 0x7fc91c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.698+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fc928089e10 con 0x7fc9380d0a80 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.698+0000 7fc93d8f2640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc91c06b500 0x7fc91c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:20.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.698+0000 7fc93d8f2640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc91c06b500 0x7fc91c06d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc934000f60 tx=0x7fc934007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:20.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.702+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc928020020 con 0x7fc9380d0a80 2024-03-06T07:28:20.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.913+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93814ed10 con 0x7fc9380d0a80 2024-03-06T07:28:20.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.914+0000 7fc92e7fc640 1 -- 172.21.15.72:0/1980593875 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fc928058b50 con 0x7fc9380d0a80 2024-03-06T07:28:20.916 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:20.916 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:20.916 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:20.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc91c06b500 msgr2=0x7fc91c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc91c06b500 0x7fc91c06d9c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc934000f60 tx=0x7fc934007b50 comp rx=0 tx=0).stop 2024-03-06T07:28:20.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 msgr2=0x7fc9380b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:20.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b88e0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fc92800b3c0 tx=0x7fc92800b880 comp rx=0 tx=0).stop 2024-03-06T07:28:20.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 shutdown_connections 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc91c06b500 0x7fc91c06d9c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93814f060 0x7fc9380b8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9380d0a80 0x7fc9380b88e0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 --2- 172.21.15.72:0/1980593875 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9380b4170 0x7fc93814bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 >> 172.21.15.72:0/1980593875 conn(0x7fc9380d3c40 msgr2=0x7fc9380b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 shutdown_connections 2024-03-06T07:28:20.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:20.916+0000 7fc93fb56640 1 -- 172.21.15.72:0/1980593875 wait complete. 2024-03-06T07:28:20.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:28:21.049 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_created":3}} 2024-03-06T07:28:21.049 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-03-06T07:28:21.050 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:21.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.148+0000 7f0804b93640 1 Processor -- start 2024-03-06T07:28:21.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.148+0000 7f0804b93640 1 -- start start 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f0804b93640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f080015bdd0 0x7f08000c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f0804b93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f08000ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f0804b93640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08000cebe0 con 0x7f08000d16e0 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f0804b93640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08000ceec0 con 0x7f080015bdd0 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f0804b93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000cf1a0 con 0x7f080015fa10 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f07fe59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f080015bdd0 0x7f08000c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f07fe59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f080015bdd0 0x7f08000c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47188/0 (socket says 172.21.15.72:47188) 2024-03-06T07:28:21.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.149+0000 7f07fe59c640 1 -- 172.21.15.72:0/4157470176 learned_addr learned my addr 172.21.15.72:0/4157470176 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:21.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.150+0000 7f07fdd9b640 1 --2- 172.21.15.72:0/4157470176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f08000ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.150+0000 7f07fe59c640 1 -- 172.21.15.72:0/4157470176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 msgr2=0x7f08000ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.150+0000 7f07fe59c640 1 --2- 172.21.15.72:0/4157470176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f08000ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.150+0000 7f07fe59c640 1 -- 172.21.15.72:0/4157470176 >> v1:172.21.15.146:6789/0 conn(0x7f08000d16e0 legacy=0x7f08000d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.150+0000 7f07fe59c640 1 -- 172.21.15.72:0/4157470176 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000cc8e0 con 0x7f080015bdd0 2024-03-06T07:28:21.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.151+0000 7f07fe59c640 1 --2- 172.21.15.72:0/4157470176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08000c5610 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009870 tx=0x7f07f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb22e018f0e6bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.151+0000 7f07fd59a640 1 -- 172.21.15.72:0/4157470176 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4035020 con 0x7f080015bdd0 2024-03-06T07:28:21.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.151+0000 7f07fd59a640 1 -- 172.21.15.72:0/4157470176 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f4031640 con 0x7f080015bdd0 2024-03-06T07:28:21.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.151+0000 7f07fd59a640 1 -- 172.21.15.72:0/4157470176 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f402d040 con 0x7f080015bdd0 2024-03-06T07:28:21.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 -- 172.21.15.72:0/4157470176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 msgr2=0x7f08000c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 --2- 172.21.15.72:0/4157470176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08000c5610 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009870 tx=0x7f07f402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 -- 172.21.15.72:0/4157470176 shutdown_connections 2024-03-06T07:28:21.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 --2- 172.21.15.72:0/4157470176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f08000ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 --2- 172.21.15.72:0/4157470176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08000c5610 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 -- 172.21.15.72:0/4157470176 >> 172.21.15.72:0/4157470176 conn(0x7f080009a9f0 msgr2=0x7f080009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 -- 172.21.15.72:0/4157470176 shutdown_connections 2024-03-06T07:28:21.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.152+0000 7f0804b93640 1 -- 172.21.15.72:0/4157470176 wait complete. 2024-03-06T07:28:21.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.153+0000 7f0804b93640 1 Processor -- start 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.153+0000 7f0804b93640 1 -- start start 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f0804b93640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08000d16e0 0x7f080015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f0804b93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08001e79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f0804b93640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f080015e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f0804b93640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080015c1b0 con 0x7f080015bdd0 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f0804b93640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08000cc5d0 con 0x7f08000d16e0 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f0804b93640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08000ce7b0 con 0x7f080015fa10 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08001e79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08001e79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47194/0 (socket says 172.21.15.72:47194) 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 -- 172.21.15.72:0/3405706205 learned_addr learned my addr 172.21.15.72:0/3405706205 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fed9d640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f080015e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fe59c640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08000d16e0 0x7f080015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 -- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 msgr2=0x7f080015e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f080015e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 -- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08000d16e0 msgr2=0x7f080015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08000d16e0 0x7f080015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 -- 172.21.15.72:0/3405706205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08000b5140 con 0x7f080015bdd0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07fdd9b640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08001e79c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f07f000e870 tx=0x7f07f000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.154+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f0017070 con 0x7f080015bdd0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.155+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f000c940 con 0x7f080015bdd0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.155+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f000cd30 con 0x7f080015bdd0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.155+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080015d120 con 0x7f080015bdd0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.155+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080015ce40 con 0x7f080015bdd0 2024-03-06T07:28:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.156+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080004f330 con 0x7f080015bdd0 2024-03-06T07:28:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.156+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07f001a070 con 0x7f080015bdd0 2024-03-06T07:28:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.156+0000 7f07eeffd640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07e006b4d0 0x7f07e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.157+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f07f008a190 con 0x7f080015bdd0 2024-03-06T07:28:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.157+0000 7f07fe59c640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07e006b4d0 0x7f07e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.157+0000 7f07fe59c640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07e006b4d0 0x7f07e006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f07f4036000 tx=0x7f07f403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.161+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07f0058f60 con 0x7f080015bdd0 2024-03-06T07:28:21.372 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:21.375 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:21.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.376+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08000b7a40 con 0x7f080015bdd0 2024-03-06T07:28:21.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.376+0000 7f07eeffd640 1 -- 172.21.15.72:0/3405706205 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f07f0058860 con 0x7f080015bdd0 2024-03-06T07:28:21.379 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:21.379 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:21.379 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:21.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07e006b4d0 msgr2=0x7f07e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07e006b4d0 0x7f07e006d990 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f07f4036000 tx=0x7f07f403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 msgr2=0x7f08001e79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08001e79c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f07f000e870 tx=0x7f07f000ed30 comp rx=0 tx=0).stop 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 shutdown_connections 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07e006b4d0 0x7f07e006d990 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f080015fa10 0x7f080015e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f080015bdd0 0x7f08001e79c0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 --2- 172.21.15.72:0/3405706205 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08000d16e0 0x7f080015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 >> 172.21.15.72:0/3405706205 conn(0x7f080009a9f0 msgr2=0x7f08000cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 shutdown_connections 2024-03-06T07:28:21.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.378+0000 7f0804b93640 1 -- 172.21.15.72:0/3405706205 wait complete. 2024-03-06T07:28:21.392 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:28:21.549 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-03-06T07:28:21.549 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-03-06T07:28:21.550 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:21.550 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:21.597 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:21.645 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:21.645 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:28:21.645 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:21.661 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:21.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.800+0000 7fc558d02640 1 Processor -- start 2024-03-06T07:28:21.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.800+0000 7fc558d02640 1 -- start start 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc558d02640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540d22b0 0x7fc5540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc558d02640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5540b41f0 0x7fc5540b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc558d02640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5540b3430 con 0x7fc55415dfe0 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc558d02640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5540d2c50 con 0x7fc5540b41f0 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc558d02640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5540b31e0 con 0x7fc5540d22b0 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc552d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5540b41f0 0x7fc5540b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc549d9b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540d22b0 0x7fc5540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc55259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fc55415dfe0 0x7fc5540cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:49524/0 (socket says 172.21.15.72:49524) 2024-03-06T07:28:21.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc55259c640 1 -- 172.21.15.72:0/2725761017 learned_addr learned my addr 172.21.15.72:0/2725761017 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:21.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc549d9b640 1 -- 172.21.15.72:0/2725761017 >> v2:172.21.15.146:3300/0 conn(0x7fc5540b41f0 msgr2=0x7fc5540b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:21.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc549d9b640 1 --2- 172.21.15.72:0/2725761017 >> v2:172.21.15.146:3300/0 conn(0x7fc5540b41f0 0x7fc5540b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.801+0000 7fc549d9b640 1 -- 172.21.15.72:0/2725761017 >> v1:172.21.15.146:6789/0 conn(0x7fc55415dfe0 legacy=0x7fc5540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.802+0000 7fc551d9b640 1 -- 172.21.15.72:0/2725761017 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481276316 0 0) 0x7fc5540b3430 con 0x7fc55415dfe0 2024-03-06T07:28:21.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.802+0000 7fc549d9b640 1 -- 172.21.15.72:0/2725761017 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540b3710 con 0x7fc5540d22b0 2024-03-06T07:28:21.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.802+0000 7fc552d9d640 1 --2- 172.21.15.72:0/2725761017 >> v2:172.21.15.146:3300/0 conn(0x7fc5540b41f0 0x7fc5540b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc549d9b640 1 --2- 172.21.15.72:0/2725761017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540d22b0 0x7fc5540d2710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc53c0097a0 tx=0x7fc53c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21fbf221bfbbc857 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc551d9b640 1 -- 172.21.15.72:0/2725761017 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c035020 con 0x7fc5540d22b0 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc551d9b640 1 -- 172.21.15.72:0/2725761017 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c031830 con 0x7fc5540d22b0 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc551d9b640 1 -- 172.21.15.72:0/2725761017 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c031c00 con 0x7fc5540d22b0 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc558d02640 1 -- 172.21.15.72:0/2725761017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540d22b0 msgr2=0x7fc5540d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc558d02640 1 --2- 172.21.15.72:0/2725761017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540d22b0 0x7fc5540d2710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc53c0097a0 tx=0x7fc53c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc558d02640 1 -- 172.21.15.72:0/2725761017 shutdown_connections 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc558d02640 1 --2- 172.21.15.72:0/2725761017 >> v2:172.21.15.146:3300/0 conn(0x7fc5540b41f0 0x7fc5540b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc558d02640 1 --2- 172.21.15.72:0/2725761017 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540d22b0 0x7fc5540d2710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.803+0000 7fc558d02640 1 -- 172.21.15.72:0/2725761017 >> 172.21.15.72:0/2725761017 conn(0x7fc5540d3cd0 msgr2=0x7fc5540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.804+0000 7fc558d02640 1 -- 172.21.15.72:0/2725761017 shutdown_connections 2024-03-06T07:28:21.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.804+0000 7fc558d02640 1 -- 172.21.15.72:0/2725761017 wait complete. 2024-03-06T07:28:21.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.805+0000 7fc558d02640 1 Processor -- start 2024-03-06T07:28:21.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.805+0000 7fc558d02640 1 -- start start 2024-03-06T07:28:21.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc558d02640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540b41f0 0x7fc5540b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc558d02640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5540d22b0 0x7fc5540b84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc558d02640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 0x7fc5540b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc558d02640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5540cfa00 con 0x7fc55415dfe0 2024-03-06T07:28:21.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc558d02640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5540b3d60 con 0x7fc5540d22b0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc558d02640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5540d1c70 con 0x7fc5540b41f0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 0x7fc5540b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 0x7fc5540b89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47204/0 (socket says 172.21.15.72:47204) 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 -- 172.21.15.72:0/2454287360 learned_addr learned my addr 172.21.15.72:0/2454287360 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc55259c640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540b41f0 0x7fc5540b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc549d9b640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5540d22b0 0x7fc5540b84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 -- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540b41f0 msgr2=0x7fc5540b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540b41f0 0x7fc5540b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 -- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5540d22b0 msgr2=0x7fc5540b84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5540d22b0 0x7fc5540b84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 -- 172.21.15.72:0/2454287360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5540cd940 con 0x7fc55415dfe0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.806+0000 7fc552d9d640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 0x7fc5540b89e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009e60 tx=0x7fc54c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.807+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c018030 con 0x7fc55415dfe0 2024-03-06T07:28:21.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.807+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc54c00cdf0 con 0x7fc55415dfe0 2024-03-06T07:28:21.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.807+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5540cc8a0 con 0x7fc55415dfe0 2024-03-06T07:28:21.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.807+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c003bb0 con 0x7fc55415dfe0 2024-03-06T07:28:21.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.807+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5540cce60 con 0x7fc55415dfe0 2024-03-06T07:28:21.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.808+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55404f330 con 0x7fc55415dfe0 2024-03-06T07:28:21.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.808+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc54c01b070 con 0x7fc55415dfe0 2024-03-06T07:28:21.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.809+0000 7fc54b7fe640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc53406b4d0 0x7fc53406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:21.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.809+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fc54c08ac30 con 0x7fc55415dfe0 2024-03-06T07:28:21.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.809+0000 7fc55259c640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc53406b4d0 0x7fc53406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:21.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.809+0000 7fc55259c640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc53406b4d0 0x7fc53406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc544000f60 tx=0x7fc544007d50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:21.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.813+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc54c010040 con 0x7fc55415dfe0 2024-03-06T07:28:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.988+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc55415d2e0 con 0x7fc55415dfe0 2024-03-06T07:28:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.989+0000 7fc54b7fe640 1 -- 172.21.15.72:0/2454287360 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17182 (secure 0 0 0) 0x7fc54c0592a0 con 0x7fc55415dfe0 2024-03-06T07:28:21.992 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:21.992 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":116,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:32.298669+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1a","mappings":[{"from":0,"to":2}]},{"pgid":"17.25","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6835/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6834/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6837/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.72:6836/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.146:6835/118157918":"2024-03-07T07:27:25.085724+0000","172.21.15.146:6833/118157918":"2024-03-07T07:27:25.085724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc53406b4d0 msgr2=0x7fc53406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc53406b4d0 0x7fc53406d990 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc544000f60 tx=0x7fc544007d50 comp rx=0 tx=0).stop 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 msgr2=0x7fc5540b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 0x7fc5540b89e0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009e60 tx=0x7fc54c00c490 comp rx=0 tx=0).stop 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 shutdown_connections 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc53406b4d0 0x7fc53406d990 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc55415dfe0 0x7fc5540b89e0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5540d22b0 0x7fc5540b84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 --2- 172.21.15.72:0/2454287360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5540b41f0 0x7fc5540b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 >> 172.21.15.72:0/2454287360 conn(0x7fc5540d3cd0 msgr2=0x7fc5540d08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 shutdown_connections 2024-03-06T07:28:21.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:21.992+0000 7fc558d02640 1 -- 172.21.15.72:0/2454287360 wait complete. 2024-03-06T07:28:22.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:00.730111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:01.483409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:22.008 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:22.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.110+0000 7f26261c9640 1 Processor -- start 2024-03-06T07:28:22.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.110+0000 7f26261c9640 1 -- start start 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.111+0000 7f26261c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 0x7f26200cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.111+0000 7f26261c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f26200d0af0 0x7f26200ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.111+0000 7f26261c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26200cd1e0 con 0x7f262015c050 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.111+0000 7f26261c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26200cd4b0 con 0x7f26200d0af0 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.111+0000 7f26261c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26200cd790 con 0x7f262006fd10 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.111+0000 7f2617fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 0x7f26200cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f261ffff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f26200d0af0 0x7f26200ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f2617fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 0x7f26200cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:36890/0 (socket says 172.21.15.72:36890) 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f2617fff640 1 -- 172.21.15.72:0/1514378523 learned_addr learned my addr 172.21.15.72:0/1514378523 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:22.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f261f7fe640 1 --1- 172.21.15.72:0/1514378523 >> v1:172.21.15.146:6789/0 conn(0x7f262015c050 0x7f262006f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:49538/0 (socket says 172.21.15.72:49538) 2024-03-06T07:28:22.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f2617fff640 1 -- 172.21.15.72:0/1514378523 >> v2:172.21.15.146:3300/0 conn(0x7f26200d0af0 msgr2=0x7f26200ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f2617fff640 1 --2- 172.21.15.72:0/1514378523 >> v2:172.21.15.146:3300/0 conn(0x7f26200d0af0 0x7f26200ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f2617fff640 1 -- 172.21.15.72:0/1514378523 >> v1:172.21.15.146:6789/0 conn(0x7f262015c050 legacy=0x7f262006f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.112+0000 7f2617fff640 1 -- 172.21.15.72:0/1514378523 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26200cda70 con 0x7f262006fd10 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f2617fff640 1 --2- 172.21.15.72:0/1514378523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 0x7f26200cc760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2608009870 tx=0x7f260802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e51f6b214f0c2fb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f261effd640 1 -- 172.21.15.72:0/1514378523 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2608035020 con 0x7f262006fd10 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f261effd640 1 -- 172.21.15.72:0/1514378523 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2608031640 con 0x7f262006fd10 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f261effd640 1 -- 172.21.15.72:0/1514378523 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260802d040 con 0x7f262006fd10 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 -- 172.21.15.72:0/1514378523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 msgr2=0x7f26200cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 --2- 172.21.15.72:0/1514378523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 0x7f26200cc760 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2608009870 tx=0x7f260802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 -- 172.21.15.72:0/1514378523 shutdown_connections 2024-03-06T07:28:22.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 --2- 172.21.15.72:0/1514378523 >> v2:172.21.15.146:3300/0 conn(0x7f26200d0af0 0x7f26200ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 --2- 172.21.15.72:0/1514378523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f262006fd10 0x7f26200cc760 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 -- 172.21.15.72:0/1514378523 >> 172.21.15.72:0/1514378523 conn(0x7f26200b3800 msgr2=0x7f26200b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 -- 172.21.15.72:0/1514378523 shutdown_connections 2024-03-06T07:28:22.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.113+0000 7f26261c9640 1 -- 172.21.15.72:0/1514378523 wait complete. 2024-03-06T07:28:22.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 Processor -- start 2024-03-06T07:28:22.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 -- start start 2024-03-06T07:28:22.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 0x7f26200b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26200d0af0 0x7f262015a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f262015c050 0x7f2620154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620165c30 con 0x7f262006fd10 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2620165ab0 con 0x7f262015c050 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f26261c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2620165db0 con 0x7f26200d0af0 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 0x7f26200b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 0x7f26200b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47220/0 (socket says 172.21.15.72:47220) 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.115+0000 7f261f7fe640 1 -- 172.21.15.72:0/3189002477 learned_addr learned my addr 172.21.15.72:0/3189002477 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261ffff640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f262015c050 0x7f2620154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f2617fff640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26200d0af0 0x7f262015a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261f7fe640 1 -- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26200d0af0 msgr2=0x7f262015a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261f7fe640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26200d0af0 0x7f262015a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261f7fe640 1 -- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f262015c050 msgr2=0x7f2620154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261f7fe640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f262015c050 0x7f2620154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261f7fe640 1 -- 172.21.15.72:0/3189002477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262015ab10 con 0x7f262006fd10 2024-03-06T07:28:22.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261f7fe640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 0x7f26200b6790 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f261000c400 tx=0x7f261000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100074e0 con 0x7f262006fd10 2024-03-06T07:28:22.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2610007680 con 0x7f262006fd10 2024-03-06T07:28:22.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262015adf0 con 0x7f262006fd10 2024-03-06T07:28:22.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610004a00 con 0x7f262006fd10 2024-03-06T07:28:22.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.116+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262009cca0 con 0x7f262006fd10 2024-03-06T07:28:22.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.117+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262009fe60 con 0x7f262006fd10 2024-03-06T07:28:22.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.118+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f261001a020 con 0x7f262006fd10 2024-03-06T07:28:22.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.118+0000 7f261cff9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f260006b500 0x7f260006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.118+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f2610089960 con 0x7f262006fd10 2024-03-06T07:28:22.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.118+0000 7f2617fff640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f260006b500 0x7f260006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.119+0000 7f2617fff640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f260006b500 0x7f260006d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2608036000 tx=0x7f260803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.123+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f26100580e0 con 0x7f262006fd10 2024-03-06T07:28:22.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.297+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2620155c20 con 0x7f262006fd10 2024-03-06T07:28:22.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.298+0000 7f261cff9640 1 -- 172.21.15.72:0/3189002477 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f261005c180 con 0x7f262006fd10 2024-03-06T07:28:22.300 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:28:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f260006b500 msgr2=0x7f260006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f260006b500 0x7f260006d9c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f2608036000 tx=0x7f260803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 msgr2=0x7f26200b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 0x7f26200b6790 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f261000c400 tx=0x7f261000c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 shutdown_connections 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f260006b500 0x7f260006d9c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f262015c050 0x7f2620154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f26200d0af0 0x7f262015a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 --2- 172.21.15.72:0/3189002477 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f262006fd10 0x7f26200b6790 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 >> 172.21.15.72:0/3189002477 conn(0x7f26200b3800 msgr2=0x7f2620155050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 shutdown_connections 2024-03-06T07:28:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.300+0000 7f26261c9640 1 -- 172.21.15.72:0/3189002477 wait complete. 2024-03-06T07:28:22.313 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:22.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.417+0000 7f4435ba0640 1 Processor -- start 2024-03-06T07:28:22.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.417+0000 7f4435ba0640 1 -- start start 2024-03-06T07:28:22.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f4435ba0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f443015e010 0x7f44300d0170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f4435ba0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f44300d2830 0x7f44300b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f4435ba0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300d0ae0 con 0x7f44300d19d0 2024-03-06T07:28:22.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f4435ba0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300b3570 con 0x7f44300d2830 2024-03-06T07:28:22.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f4435ba0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44300d0dc0 con 0x7f443015e010 2024-03-06T07:28:22.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f443539f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f44300d2830 0x7f44300b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.418+0000 7f443539f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f44300d2830 0x7f44300b3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47228/0 (socket says 172.21.15.72:47228) 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.419+0000 7f443539f640 1 -- 172.21.15.72:0/2095444040 learned_addr learned my addr 172.21.15.72:0/2095444040 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.419+0000 7f4434b9e640 1 --2- 172.21.15.72:0/2095444040 >> v2:172.21.15.146:3300/0 conn(0x7f443015e010 0x7f44300d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.419+0000 7f443539f640 1 -- 172.21.15.72:0/2095444040 >> v2:172.21.15.146:3300/0 conn(0x7f443015e010 msgr2=0x7f44300d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.419+0000 7f443539f640 1 --2- 172.21.15.72:0/2095444040 >> v2:172.21.15.146:3300/0 conn(0x7f443015e010 0x7f44300d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.419+0000 7f443539f640 1 -- 172.21.15.72:0/2095444040 >> v1:172.21.15.146:6789/0 conn(0x7f44300d19d0 legacy=0x7f44300d2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.419+0000 7f443539f640 1 -- 172.21.15.72:0/2095444040 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300b2f20 con 0x7f44300d2830 2024-03-06T07:28:22.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f443539f640 1 --2- 172.21.15.72:0/2095444040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d2830 0x7f44300b3a00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f443006b3c0 tx=0x7f442802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22c51f22ec2507f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f442f7fe640 1 -- 172.21.15.72:0/2095444040 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442802cbb0 con 0x7f44300d2830 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f442f7fe640 1 -- 172.21.15.72:0/2095444040 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442802cd50 con 0x7f44300d2830 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f442f7fe640 1 -- 172.21.15.72:0/2095444040 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428034610 con 0x7f44300d2830 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f4435ba0640 1 -- 172.21.15.72:0/2095444040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d2830 msgr2=0x7f44300b3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f4435ba0640 1 --2- 172.21.15.72:0/2095444040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d2830 0x7f44300b3a00 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f443006b3c0 tx=0x7f442802c010 comp rx=0 tx=0).stop 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f4435ba0640 1 -- 172.21.15.72:0/2095444040 shutdown_connections 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f4435ba0640 1 --2- 172.21.15.72:0/2095444040 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d2830 0x7f44300b3a00 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.420+0000 7f4435ba0640 1 --2- 172.21.15.72:0/2095444040 >> v2:172.21.15.146:3300/0 conn(0x7f443015e010 0x7f44300d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.421+0000 7f4435ba0640 1 -- 172.21.15.72:0/2095444040 >> 172.21.15.72:0/2095444040 conn(0x7f443009ce10 msgr2=0x7f44300d3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.421+0000 7f4435ba0640 1 -- 172.21.15.72:0/2095444040 shutdown_connections 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.421+0000 7f4435ba0640 1 -- 172.21.15.72:0/2095444040 wait complete. 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.422+0000 7f4435ba0640 1 Processor -- start 2024-03-06T07:28:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.422+0000 7f4435ba0640 1 -- start start 2024-03-06T07:28:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.422+0000 7f4435ba0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 0x7f443015e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.422+0000 7f4435ba0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44300d2830 0x7f44300cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4435ba0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f443015e010 0x7f44301f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4435ba0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44300d0830 con 0x7f44300d19d0 2024-03-06T07:28:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4435ba0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44300d06b0 con 0x7f443015e010 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4435ba0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44300b3f40 con 0x7f44300d2830 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 0x7f443015e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 0x7f443015e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47230/0 (socket says 172.21.15.72:47230) 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 -- 172.21.15.72:0/923007524 learned_addr learned my addr 172.21.15.72:0/923007524 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f443539f640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f443015e010 0x7f44301f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f442ffff640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44300d2830 0x7f44300cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 -- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44300d2830 msgr2=0x7f44300cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44300d2830 0x7f44300cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 -- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f443015e010 msgr2=0x7f44301f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f443015e010 0x7f44301f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 -- 172.21.15.72:0/923007524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430160970 con 0x7f44300d19d0 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f443539f640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f443015e010 0x7f44301f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:22.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4434b9e640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 0x7f443015e460 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f442400d620 tx=0x7f442400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400b620 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4424019070 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430160c50 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400ba00 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.423+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4430161210 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.424+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4430068f10 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.425+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f442400bba0 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.425+0000 7f442d7fa640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f441406b4d0 0x7f441406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.425+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f442408a6f0 con 0x7f44300d19d0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.425+0000 7f442ffff640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f441406b4d0 0x7f441406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.426+0000 7f442ffff640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f441406b4d0 0x7f441406d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f44300b4630 tx=0x7f441c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.429+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f44240594c0 con 0x7f44300d19d0 2024-03-06T07:28:22.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.606+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44300b6440 con 0x7f44300d19d0 2024-03-06T07:28:22.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.607+0000 7f442d7fa640 1 -- 172.21.15.72:0/923007524 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4424058dc0 con 0x7f44300d19d0 2024-03-06T07:28:22.609 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:28:22.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f441406b4d0 msgr2=0x7f441406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f441406b4d0 0x7f441406d990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f44300b4630 tx=0x7f441c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 msgr2=0x7f443015e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 0x7f443015e460 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f442400d620 tx=0x7f442400dae0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 shutdown_connections 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f441406b4d0 0x7f441406d990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f443015e010 0x7f44301f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44300d2830 0x7f44300cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 --2- 172.21.15.72:0/923007524 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44300d19d0 0x7f443015e460 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 >> 172.21.15.72:0/923007524 conn(0x7f443009ce10 msgr2=0x7f443006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 shutdown_connections 2024-03-06T07:28:22.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.609+0000 7f4435ba0640 1 -- 172.21.15.72:0/923007524 wait complete. 2024-03-06T07:28:22.622 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:22.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.723+0000 7fada69f7640 1 Processor -- start 2024-03-06T07:28:22.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.723+0000 7fada69f7640 1 -- start start 2024-03-06T07:28:22.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada69f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fada01614d0 0x7fada00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada69f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 0x7fada00d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada69f7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada00d2740 con 0x7fada00d22b0 2024-03-06T07:28:22.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada69f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada01618b0 con 0x7fada01614d0 2024-03-06T07:28:22.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada69f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cf400 con 0x7fada00d1910 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada4f94640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fada00d22b0 0x7fada00d06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55354/0 (socket says 172.21.15.72:55354) 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fada4f94640 1 -- 172.21.15.72:0/4143084523 learned_addr learned my addr 172.21.15.72:0/4143084523 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fad9f7fe640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 0x7fada00d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fad9ffff640 1 --2- 172.21.15.72:0/4143084523 >> v2:172.21.15.146:3300/0 conn(0x7fada01614d0 0x7fada00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620182067 0 0) 0x7fad98001490 con 0x7fada00d22b0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.724+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad88003520 con 0x7fada00d22b0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3773595420 0 0) 0x7fad88003520 con 0x7fada00d22b0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 msgr2=0x7fada00d1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 0x7fada00d1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 >> v2:172.21.15.146:3300/0 conn(0x7fada01614d0 msgr2=0x7fada00d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 --2- 172.21.15.72:0/4143084523 >> v2:172.21.15.146:3300/0 conn(0x7fada01614d0 0x7fada00d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada00cf6a0 con 0x7fada00d22b0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fad98001bc0 con 0x7fada00d22b0 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 >> v1:172.21.15.72:6789/0 conn(0x7fada00d22b0 legacy=0x7fada00d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fad88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 0x7fad88004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad88004d40 0x7fad880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada01618b0 con 0x7fad880039b0 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cf400 con 0x7fad88004d40 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8800b9c0 con 0x7fada00d1910 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9ffff640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fad88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9f7fe640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 0x7fad88004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fada4f94640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad88004d40 0x7fad880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9ffff640 1 -- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 msgr2=0x7fad88004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9ffff640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00d1910 0x7fad88004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9ffff640 1 -- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad88004d40 msgr2=0x7fad880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9ffff640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad88004d40 0x7fad880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.725+0000 7fad9ffff640 1 -- 172.21.15.72:0/4143084523 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad88003520 con 0x7fad880039b0 2024-03-06T07:28:22.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.726+0000 7fad9ffff640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fad88003dd0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fad90009af0 tx=0x7fad88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbb82d7382d357 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.726+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90039070 con 0x7fad880039b0 2024-03-06T07:28:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.726+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9002c710 con 0x7fad880039b0 2024-03-06T07:28:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.726+0000 7fad9effd640 1 -- 172.21.15.72:0/4143084523 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90009690 con 0x7fad880039b0 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada69f7640 1 -- 172.21.15.72:0/4143084523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 msgr2=0x7fad88003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada69f7640 1 --2- 172.21.15.72:0/4143084523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fad88003dd0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fad90009af0 tx=0x7fad88005830 comp rx=0 tx=0).stop 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada4f94640 1 -- 172.21.15.72:0/4143084523 reap_dead start 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada69f7640 1 -- 172.21.15.72:0/4143084523 shutdown_connections 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada69f7640 1 -- 172.21.15.72:0/4143084523 >> 172.21.15.72:0/4143084523 conn(0x7fada00d3d40 msgr2=0x7fada00d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada69f7640 1 -- 172.21.15.72:0/4143084523 shutdown_connections 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.727+0000 7fada69f7640 1 -- 172.21.15.72:0/4143084523 wait complete. 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.728+0000 7fada69f7640 1 Processor -- start 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.728+0000 7fada69f7640 1 -- start start 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fada69f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fada00b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fada69f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada015a340 0x7fada00b7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fada69f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00b7aa0 0x7fada015c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fada69f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada00b0910 con 0x7fad880039b0 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fada69f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada00cfc60 con 0x7fada015a340 2024-03-06T07:28:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fada69f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada00b1e40 con 0x7fada00b7aa0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fada00b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fada00b6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45554/0 (socket says 172.21.15.72:45554) 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 -- 172.21.15.72:0/3421871948 learned_addr learned my addr 172.21.15.72:0/3421871948 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 -- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00b7aa0 msgr2=0x7fada015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00b7aa0 0x7fada015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 -- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada015a340 msgr2=0x7fada00b7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9f7fe640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada015a340 0x7fada00b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada015a340 0x7fada00b7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 -- 172.21.15.72:0/3421871948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada00b9420 con 0x7fad880039b0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.729+0000 7fad9ffff640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fada00b6bd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fad90009af0 tx=0x7fad90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.730+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90042020 con 0x7fad880039b0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.730+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad9003a040 con 0x7fad880039b0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.730+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada00b9700 con 0x7fad880039b0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.730+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90039070 con 0x7fad880039b0 2024-03-06T07:28:22.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.730+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada00b82b0 con 0x7fad880039b0 2024-03-06T07:28:22.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.731+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada00b7ec0 con 0x7fad880039b0 2024-03-06T07:28:22.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.731+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad90045020 con 0x7fad880039b0 2024-03-06T07:28:22.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.732+0000 7fad9cff9640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad8c06b500 0x7fad8c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:22.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.732+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fad900ab740 con 0x7fad880039b0 2024-03-06T07:28:22.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.732+0000 7fad9f7fe640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad8c06b500 0x7fad8c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:22.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.732+0000 7fad9f7fe640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad8c06b500 0x7fad8c06d9c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fada00b8580 tx=0x7fad94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.736+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad9004c070 con 0x7fad880039b0 2024-03-06T07:28:22.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.909+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fada00c7db0 con 0x7fad880039b0 2024-03-06T07:28:22.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.910+0000 7fad9cff9640 1 -- 172.21.15.72:0/3421871948 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad9007a510 con 0x7fad880039b0 2024-03-06T07:28:22.912 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad8c06b500 msgr2=0x7fad8c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad8c06b500 0x7fad8c06d9c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fada00b8580 tx=0x7fad94008040 comp rx=0 tx=0).stop 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 msgr2=0x7fada00b6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fada00b6bd0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fad90009af0 tx=0x7fad90004730 comp rx=0 tx=0).stop 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 shutdown_connections 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad8c06b500 0x7fad8c06d9c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada00b7aa0 0x7fada015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada015a340 0x7fada00b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 --2- 172.21.15.72:0/3421871948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad880039b0 0x7fada00b6bd0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:22.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 >> 172.21.15.72:0/3421871948 conn(0x7fada00d3d40 msgr2=0x7fada00b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:22.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 shutdown_connections 2024-03-06T07:28:22.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:22.912+0000 7fada69f7640 1 -- 172.21.15.72:0/3421871948 wait complete. 2024-03-06T07:28:22.925 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:28:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.043+0000 7f14d9081640 1 Processor -- start 2024-03-06T07:28:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.043+0000 7f14d9081640 1 -- start start 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d9081640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 0x7f14d40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d9081640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f14d4161cf0 0x7f14d4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d9081640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14d40d22c0 con 0x7f14d40b4460 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d9081640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14d4161b00 con 0x7f14d4161cf0 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d9081640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40cecc0 con 0x7f14d4162970 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d2d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f14d40b4460 0x7f14d40b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55358/0 (socket says 172.21.15.72:55358) 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d2d9d640 1 -- 172.21.15.72:0/2956070217 learned_addr learned my addr 172.21.15.72:0/2956070217 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866537727 0 0) 0x7f14d40d22c0 con 0x7f14d40b4460 2024-03-06T07:28:23.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c0003520 con 0x7f14d40b4460 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d259c640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 0x7f14d40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d359e640 1 --2- 172.21.15.72:0/2956070217 >> v2:172.21.15.146:3300/0 conn(0x7f14d4161cf0 0x7f14d4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 473741530 0 0) 0x7f14c0003520 con 0x7f14d40b4460 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 msgr2=0x7f14d40d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 0x7f14d40d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 >> v2:172.21.15.146:3300/0 conn(0x7f14d4161cf0 msgr2=0x7f14d4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 --2- 172.21.15.72:0/2956070217 >> v2:172.21.15.146:3300/0 conn(0x7f14d4161cf0 0x7f14d4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d40d25a0 con 0x7f14d40b4460 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f14c4001670 con 0x7f14d40b4460 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 >> v1:172.21.15.72:6789/0 conn(0x7f14d40b4460 legacy=0x7f14d40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 0x7f14c00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c0003fe0 0x7f14c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d4161b00 con 0x7f14c0003fe0 2024-03-06T07:28:23.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40cecc0 con 0x7f14c0004d40 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c000bac0 con 0x7f14d4162970 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d259c640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c0003fe0 0x7f14c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.045+0000 7f14d2d9d640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 0x7f14c00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d359e640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d259c640 1 -- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 msgr2=0x7f14c00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d259c640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14d4162970 0x7f14c00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d259c640 1 -- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14c0004d40 msgr2=0x7f14c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d259c640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d259c640 1 -- 172.21.15.72:0/2956070217 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0003520 con 0x7f14c0003fe0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.046+0000 7f14d359e640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14c0004d40 0x7f14c00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d259c640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c0003fe0 0x7f14c0004800 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00ba00 tx=0x7f14c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c106f6e65c60f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc036020 con 0x7f14c0003fe0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc032620 con 0x7f14c0003fe0 2024-03-06T07:28:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d1d9b640 1 -- 172.21.15.72:0/2956070217 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc00b600 con 0x7f14c0003fe0 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d9081640 1 -- 172.21.15.72:0/2956070217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c0003fe0 msgr2=0x7f14c0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d9081640 1 --2- 172.21.15.72:0/2956070217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14c0003fe0 0x7f14c0004800 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00ba00 tx=0x7f14c0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d359e640 1 -- 172.21.15.72:0/2956070217 reap_dead start 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d9081640 1 -- 172.21.15.72:0/2956070217 shutdown_connections 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.047+0000 7f14d9081640 1 -- 172.21.15.72:0/2956070217 >> 172.21.15.72:0/2956070217 conn(0x7f14d40d3ce0 msgr2=0x7f14d40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.048+0000 7f14d9081640 1 -- 172.21.15.72:0/2956070217 shutdown_connections 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.048+0000 7f14d9081640 1 -- 172.21.15.72:0/2956070217 wait complete. 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.049+0000 7f14d9081640 1 Processor -- start 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.049+0000 7f14d9081640 1 -- start start 2024-03-06T07:28:23.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d9081640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14c0003fe0 0x7f14d4152240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d9081640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 0x7f14d40b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d9081640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14d40cda40 0x7f14d40cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d9081640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40ce950 con 0x7f14d40cd5c0 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d9081640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40d2b60 con 0x7f14d40cda40 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d9081640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14d40b4840 con 0x7f14c0003fe0 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 0x7f14d40b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 0x7f14d40b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45576/0 (socket says 172.21.15.72:45576) 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 -- 172.21.15.72:0/638451895 learned_addr learned my addr 172.21.15.72:0/638451895 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d359e640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14d40cda40 0x7f14d40cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 -- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14c0003fe0 msgr2=0x7f14d4152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d2d9d640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14c0003fe0 0x7f14d4152240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14c0003fe0 0x7f14d4152240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 -- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14d40cda40 msgr2=0x7f14d40cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14d40cda40 0x7f14d40cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 -- 172.21.15.72:0/638451895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d40c8b40 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d359e640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14d40cda40 0x7f14d40cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d2d9d640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14c0003fe0 0x7f14d4152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.050+0000 7f14d259c640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 0x7f14d40b6900 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00baf0 tx=0x7f14bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.051+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc036040 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.051+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc032e40 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.051+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14d40c8e20 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.051+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0476c0 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.051+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14d40c93e0 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.052+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14d404f330 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.052+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f14bc039020 con 0x7f14d40cd5c0 2024-03-06T07:28:23.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.052+0000 7f14cb7fe640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.052+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f14bc0ac710 con 0x7f14d40cd5c0 2024-03-06T07:28:23.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.052+0000 7f14d2d9d640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.053+0000 7f14d2d9d640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f14c4000fd0 tx=0x7f14c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.057+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f14bc07b3c0 con 0x7f14d40cd5c0 2024-03-06T07:28:23.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.304+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f14d406b180 con 0x7f14d40cd5c0 2024-03-06T07:28:23.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.304+0000 7f14cb7fe640 1 -- 172.21.15.72:0/638451895 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v116) v1 ==== 80+0+366 (secure 0 0 0) 0x7f14bc07acc0 con 0x7f14d40cd5c0 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6835/1152083148 2024-03-07T07:27:25.269515+0000 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6834/1152083148 2024-03-07T07:27:25.269515+0000 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6837/3618393393 2024-03-07T07:27:25.094353+0000 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6836/3618393393 2024-03-07T07:27:25.094353+0000 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.146:6835/118157918 2024-03-07T07:27:25.085724+0000 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.146:6833/118157918 2024-03-07T07:27:25.085724+0000 2024-03-06T07:28:23.307 INFO:teuthology.orchestra.run.smithi072.stderr:listed 6 entries 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14b406b4d0 msgr2=0x7f14b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14b406b4d0 0x7f14b406d990 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f14c4000fd0 tx=0x7f14c4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 msgr2=0x7f14d40b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 0x7f14d40b6900 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f14bc00baf0 tx=0x7f14bc004730 comp rx=0 tx=0).stop 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 shutdown_connections 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f14b406b4d0 0x7f14b406d990 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f14d40cda40 0x7f14d40cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d40cd5c0 0x7f14d40b6900 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 --2- 172.21.15.72:0/638451895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f14c0003fe0 0x7f14d4152240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 >> 172.21.15.72:0/638451895 conn(0x7f14d40d3ce0 msgr2=0x7f14d40b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 shutdown_connections 2024-03-06T07:28:23.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.307+0000 7f14d9081640 1 -- 172.21.15.72:0/638451895 wait complete. 2024-03-06T07:28:23.320 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:28:23.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.320 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:23.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[74138]: fuse finished with error 0 and tester_r 0 2024-03-06T07:28:23.352 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:28:23.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.352 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:28:23.515 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:28:23.515 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:23.534 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:23.534 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:28:23.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.534 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:28:23.592 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:28:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:23.593 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:28:23.593 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:23.608 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:23.609 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:28:23.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.609 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:28:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:23.663 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:28:23.663 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:28:23.663 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:23.681 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:23.681 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:28:23.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:23.682 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:28:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:28:23.739 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:28:23.739 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:23.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.839+0000 7f688538c640 1 Processor -- start 2024-03-06T07:28:23.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.839+0000 7f688538c640 1 -- start start 2024-03-06T07:28:23.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f688538c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68800d22b0 0x7f68800d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f688538c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f68800b41f0 0x7f68800b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f688538c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800b3430 con 0x7f688015dfe0 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f688538c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68800d2c50 con 0x7f68800b41f0 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f688538c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68800b31e0 con 0x7f68800d22b0 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f68800b41f0 0x7f68800b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f688015dfe0 0x7f68800cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55366/0 (socket says 172.21.15.72:55366) 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687effd640 1 -- 172.21.15.72:0/715352963 learned_addr learned my addr 172.21.15.72:0/715352963 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687e7fc640 1 --2- 172.21.15.72:0/715352963 >> v2:172.21.15.146:3300/0 conn(0x7f68800d22b0 0x7f68800d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687f7fe640 1 -- 172.21.15.72:0/715352963 >> v2:172.21.15.146:3300/0 conn(0x7f68800d22b0 msgr2=0x7f68800d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687f7fe640 1 --2- 172.21.15.72:0/715352963 >> v2:172.21.15.146:3300/0 conn(0x7f68800d22b0 0x7f68800d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.840+0000 7f687f7fe640 1 -- 172.21.15.72:0/715352963 >> v1:172.21.15.72:6789/0 conn(0x7f688015dfe0 legacy=0x7f68800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.841+0000 7f687dffb640 1 -- 172.21.15.72:0/715352963 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171795689 0 0) 0x7f68800b3430 con 0x7f688015dfe0 2024-03-06T07:28:23.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.841+0000 7f687f7fe640 1 -- 172.21.15.72:0/715352963 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800b3710 con 0x7f68800b41f0 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f687f7fe640 1 --2- 172.21.15.72:0/715352963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800b41f0 0x7f68800b4680 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f6878009d90 tx=0x7f687802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e484c24bfde5901 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f687dffb640 1 -- 172.21.15.72:0/715352963 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878039070 con 0x7f68800b41f0 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f687dffb640 1 -- 172.21.15.72:0/715352963 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687802cc60 con 0x7f68800b41f0 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f687dffb640 1 -- 172.21.15.72:0/715352963 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878034660 con 0x7f68800b41f0 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 -- 172.21.15.72:0/715352963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800b41f0 msgr2=0x7f68800b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 --2- 172.21.15.72:0/715352963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800b41f0 0x7f68800b4680 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f6878009d90 tx=0x7f687802beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 -- 172.21.15.72:0/715352963 shutdown_connections 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 --2- 172.21.15.72:0/715352963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800b41f0 0x7f68800b4680 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 --2- 172.21.15.72:0/715352963 >> v2:172.21.15.146:3300/0 conn(0x7f68800d22b0 0x7f68800d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 -- 172.21.15.72:0/715352963 >> 172.21.15.72:0/715352963 conn(0x7f68800d3cd0 msgr2=0x7f68800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 -- 172.21.15.72:0/715352963 shutdown_connections 2024-03-06T07:28:23.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.842+0000 7f688538c640 1 -- 172.21.15.72:0/715352963 wait complete. 2024-03-06T07:28:23.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 Processor -- start 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 -- start start 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68800b41f0 0x7f688015ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 0x7f68800cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f688015dfe0 0x7f68800b7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800cfa00 con 0x7f68800d22b0 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800b3d60 con 0x7f68800b41f0 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f688538c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68800d1c70 con 0x7f688015dfe0 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.844+0000 7f687e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 0x7f68800cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 0x7f68800cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45588/0 (socket says 172.21.15.72:45588) 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 -- 172.21.15.72:0/3247665675 learned_addr learned my addr 172.21.15.72:0/3247665675 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687f7fe640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f688015dfe0 0x7f68800b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687effd640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68800b41f0 0x7f688015ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 -- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f688015dfe0 msgr2=0x7f68800b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f688015dfe0 0x7f68800b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 -- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68800b41f0 msgr2=0x7f688015ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68800b41f0 0x7f688015ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 -- 172.21.15.72:0/3247665675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800cc820 con 0x7f68800d22b0 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687effd640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68800b41f0 0x7f688015ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f687e7fc640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 0x7f68800cd8c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f687000c7f0 tx=0x7f687000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870015070 con 0x7f68800d22b0 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6870004240 con 0x7f68800d22b0 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68800ccb00 con 0x7f68800d22b0 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6870007620 con 0x7f68800d22b0 2024-03-06T07:28:23.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.845+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68800cd3a0 con 0x7f68800d22b0 2024-03-06T07:28:23.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.846+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800cccd0 con 0x7f68800d22b0 2024-03-06T07:28:23.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.847+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f68700077c0 con 0x7f68800d22b0 2024-03-06T07:28:23.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.847+0000 7f68677fe640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f686c06b4d0 0x7f686c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:23.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.847+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f687008a5b0 con 0x7f68800d22b0 2024-03-06T07:28:23.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.847+0000 7f687effd640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f686c06b4d0 0x7f686c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:23.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.847+0000 7f687effd640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f686c06b4d0 0x7f686c06d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6874000f60 tx=0x7f6874009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:23.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:23.851+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f68700592f0 con 0x7f68800d22b0 2024-03-06T07:28:24.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.059+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f688004f330 con 0x7f68800d22b0 2024-03-06T07:28:24.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.059+0000 7f68677fe640 1 -- 172.21.15.72:0/3247665675 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f6870058bf0 con 0x7f68800d22b0 2024-03-06T07:28:24.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:24.062 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:24.062 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f686c06b4d0 msgr2=0x7f686c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f686c06b4d0 0x7f686c06d990 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f6874000f60 tx=0x7f6874009040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 msgr2=0x7f68800cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 0x7f68800cd8c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f687000c7f0 tx=0x7f687000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 shutdown_connections 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f686c06b4d0 0x7f686c06d990 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f688015dfe0 0x7f68800b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68800d22b0 0x7f68800cd8c0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 --2- 172.21.15.72:0/3247665675 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68800b41f0 0x7f688015ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 >> 172.21.15.72:0/3247665675 conn(0x7f68800d3cd0 msgr2=0x7f68800d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 shutdown_connections 2024-03-06T07:28:24.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.061+0000 7f688538c640 1 -- 172.21.15.72:0/3247665675 wait complete. 2024-03-06T07:28:24.075 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:24.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.172+0000 7f231deed640 1 Processor -- start 2024-03-06T07:28:24.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.172+0000 7f231deed640 1 -- start start 2024-03-06T07:28:24.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f231deed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f231815dfe0 0x7f23180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f231deed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180b3430 con 0x7f23180cf830 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f231deed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23180b3710 con 0x7f23180d29c0 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f231deed640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23180b31e0 con 0x7f231815dfe0 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f23177fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f231815dfe0 0x7f23180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f2316ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f23180cf830 0x7f23180d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55368/0 (socket says 172.21.15.72:55368) 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f2316ffd640 1 -- 172.21.15.72:0/3777227965 learned_addr learned my addr 172.21.15.72:0/3777227965 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f23177fe640 1 -- 172.21.15.72:0/3777227965 >> v1:172.21.15.146:6789/0 conn(0x7f23180d29c0 legacy=0x7f23180b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f23177fe640 1 -- 172.21.15.72:0/3777227965 >> v1:172.21.15.72:6789/0 conn(0x7f23180cf830 legacy=0x7f23180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f23177fe640 1 -- 172.21.15.72:0/3777227965 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180b39f0 con 0x7f231815dfe0 2024-03-06T07:28:24.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.173+0000 7f23167fc640 1 -- 172.21.15.72:0/3777227965 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350230885 0 0) 0x7f23180b3430 con 0x7f23180cf830 2024-03-06T07:28:24.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.174+0000 7f23177fe640 1 --2- 172.21.15.72:0/3777227965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180cf2f0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f23180b3360 tx=0x7f230802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80f4d44c8f6ad36f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.174+0000 7f23167fc640 1 -- 172.21.15.72:0/3777227965 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308035020 con 0x7f231815dfe0 2024-03-06T07:28:24.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.174+0000 7f23167fc640 1 -- 172.21.15.72:0/3777227965 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2308031830 con 0x7f231815dfe0 2024-03-06T07:28:24.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.174+0000 7f23167fc640 1 -- 172.21.15.72:0/3777227965 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308031c00 con 0x7f231815dfe0 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 -- 172.21.15.72:0/3777227965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 msgr2=0x7f23180cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 --2- 172.21.15.72:0/3777227965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180cf2f0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f23180b3360 tx=0x7f230802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 -- 172.21.15.72:0/3777227965 shutdown_connections 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 --2- 172.21.15.72:0/3777227965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180cf2f0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 -- 172.21.15.72:0/3777227965 >> 172.21.15.72:0/3777227965 conn(0x7f23180d3cd0 msgr2=0x7f23180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 -- 172.21.15.72:0/3777227965 shutdown_connections 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.175+0000 7f231deed640 1 -- 172.21.15.72:0/3777227965 wait complete. 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.176+0000 7f231deed640 1 Processor -- start 2024-03-06T07:28:24.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.176+0000 7f231deed640 1 -- start start 2024-03-06T07:28:24.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f231deed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23180cf830 0x7f2318150230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f231deed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f23180d29c0 0x7f231809fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f231deed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f231deed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231815e3c0 con 0x7f231815dfe0 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f231deed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23180b3d60 con 0x7f23180cf830 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f231deed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231815e540 con 0x7f23180d29c0 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45612/0 (socket says 172.21.15.72:45612) 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 -- 172.21.15.72:0/4170115763 learned_addr learned my addr 172.21.15.72:0/4170115763 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f23177fe640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23180cf830 0x7f2318150230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2316ffd640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f23180d29c0 0x7f231809fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 -- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f23180d29c0 msgr2=0x7f231809fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f23180d29c0 0x7f231809fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 -- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23180cf830 msgr2=0x7f2318150230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.177+0000 7f2317fff640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23180cf830 0x7f2318150230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f2317fff640 1 -- 172.21.15.72:0/4170115763 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23180a0850 con 0x7f231815dfe0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f23177fe640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23180cf830 0x7f2318150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f2316ffd640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f23180d29c0 0x7f231809fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f2317fff640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180a0010 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f231000c700 tx=0x7f231000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310016020 con 0x7f231815dfe0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231000aa50 con 0x7f231815dfe0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23180a0b30 con 0x7f231815dfe0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231000ae40 con 0x7f231815dfe0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.178+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23180a10f0 con 0x7f231815dfe0 2024-03-06T07:28:24.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.179+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231804f330 con 0x7f231815dfe0 2024-03-06T07:28:24.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.179+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2310019020 con 0x7f231815dfe0 2024-03-06T07:28:24.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.180+0000 7f22fffff640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f230406b4d0 0x7f230406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.180+0000 7f23177fe640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f230406b4d0 0x7f230406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.180+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f231008a5f0 con 0x7f231815dfe0 2024-03-06T07:28:24.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.180+0000 7f23177fe640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f230406b4d0 0x7f230406d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2308036000 tx=0x7f230803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.184+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2310020020 con 0x7f231815dfe0 2024-03-06T07:28:24.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.360+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23180b4e50 con 0x7f231815dfe0 2024-03-06T07:28:24.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.361+0000 7f22fffff640 1 -- 172.21.15.72:0/4170115763 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17182 (secure 0 0 0) 0x7f2310058cd0 con 0x7f231815dfe0 2024-03-06T07:28:24.363 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:24.363 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":116,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:32.298669+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1a","mappings":[{"from":0,"to":2}]},{"pgid":"17.25","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6835/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6834/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6837/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.72:6836/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.146:6835/118157918":"2024-03-07T07:27:25.085724+0000","172.21.15.146:6833/118157918":"2024-03-07T07:27:25.085724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:24.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f230406b4d0 msgr2=0x7f230406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f230406b4d0 0x7f230406d990 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2308036000 tx=0x7f230803c040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 msgr2=0x7f23180a0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180a0010 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f231000c700 tx=0x7f231000cbc0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 shutdown_connections 2024-03-06T07:28:24.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f230406b4d0 0x7f230406d990 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f231815dfe0 0x7f23180a0010 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f23180d29c0 0x7f231809fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 --2- 172.21.15.72:0/4170115763 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23180cf830 0x7f2318150230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 >> 172.21.15.72:0/4170115763 conn(0x7f23180d3cd0 msgr2=0x7f231806cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 shutdown_connections 2024-03-06T07:28:24.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.363+0000 7f231deed640 1 -- 172.21.15.72:0/4170115763 wait complete. 2024-03-06T07:28:24.377 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:27:00.730111+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:27:01.483409+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:24.377 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 Processor -- start 2024-03-06T07:28:24.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 -- start start 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63800d4860 0x7f63800d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63800b0e30 0x7f63800b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63801618b0 con 0x7f63801614d0 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63800d4cf0 con 0x7f63800d4860 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.483+0000 7f638517a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63800cf400 con 0x7f63800b0e30 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f63800d4860 0x7f63800d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637ed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f63801614d0 0x7f63800d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55378/0 (socket says 172.21.15.72:55378) 2024-03-06T07:28:24.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637ed9d640 1 -- 172.21.15.72:0/3636431175 learned_addr learned my addr 172.21.15.72:0/3636431175 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637f59e640 1 --2- 172.21.15.72:0/3636431175 >> v2:172.21.15.146:3300/0 conn(0x7f63800b0e30 0x7f63800b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637e59c640 1 -- 172.21.15.72:0/3636431175 >> v2:172.21.15.146:3300/0 conn(0x7f63800b0e30 msgr2=0x7f63800b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637e59c640 1 --2- 172.21.15.72:0/3636431175 >> v2:172.21.15.146:3300/0 conn(0x7f63800b0e30 0x7f63800b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637e59c640 1 -- 172.21.15.72:0/3636431175 >> v1:172.21.15.72:6789/0 conn(0x7f63801614d0 legacy=0x7f63800d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637e59c640 1 -- 172.21.15.72:0/3636431175 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800cf6a0 con 0x7f63800d4860 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637dd9b640 1 -- 172.21.15.72:0/3636431175 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081621370 0 0) 0x7f63801618b0 con 0x7f63801614d0 2024-03-06T07:28:24.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.484+0000 7f637f59e640 1 --2- 172.21.15.72:0/3636431175 >> v2:172.21.15.146:3300/0 conn(0x7f63800b0e30 0x7f63800b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f637e59c640 1 --2- 172.21.15.72:0/3636431175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63800d4860 0x7f63800d2020 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f63800b2a70 tx=0x7f637402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9447bc0d95b62d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f637dd9b640 1 -- 172.21.15.72:0/3636431175 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6374035020 con 0x7f63800d4860 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f637dd9b640 1 -- 172.21.15.72:0/3636431175 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6374031830 con 0x7f63800d4860 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f637dd9b640 1 -- 172.21.15.72:0/3636431175 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6374031c00 con 0x7f63800d4860 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 -- 172.21.15.72:0/3636431175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63800d4860 msgr2=0x7f63800d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 --2- 172.21.15.72:0/3636431175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63800d4860 0x7f63800d2020 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f63800b2a70 tx=0x7f637402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 -- 172.21.15.72:0/3636431175 shutdown_connections 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 --2- 172.21.15.72:0/3636431175 >> v2:172.21.15.146:3300/0 conn(0x7f63800b0e30 0x7f63800b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 --2- 172.21.15.72:0/3636431175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63800d4860 0x7f63800d2020 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 -- 172.21.15.72:0/3636431175 >> 172.21.15.72:0/3636431175 conn(0x7f63800d3d40 msgr2=0x7f63800d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 -- 172.21.15.72:0/3636431175 shutdown_connections 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.485+0000 7f638517a640 1 -- 172.21.15.72:0/3636431175 wait complete. 2024-03-06T07:28:24.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 Processor -- start 2024-03-06T07:28:24.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 -- start start 2024-03-06T07:28:24.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63800b0e30 0x7f6380152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63800d4860 0x7f63800b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 0x7f6380151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63800b0910 con 0x7f63801614d0 2024-03-06T07:28:24.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63800cfc60 con 0x7f63800b0e30 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.487+0000 7f638517a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63800d2560 con 0x7f63800d4860 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 0x7f6380151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 0x7f6380151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45618/0 (socket says 172.21.15.72:45618) 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 -- 172.21.15.72:0/771209248 learned_addr learned my addr 172.21.15.72:0/771209248 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637ed9d640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63800b0e30 0x7f6380152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 -- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63800d4860 msgr2=0x7f63800b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63800d4860 0x7f63800b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 -- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63800b0e30 msgr2=0x7f6380152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63800b0e30 0x7f6380152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 -- 172.21.15.72:0/771209248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63800c90b0 con 0x7f63801614d0 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f637f59e640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 0x7f6380151e20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f637000e810 tx=0x7f637000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637000f040 con 0x7f63801614d0 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6370004030 con 0x7f63801614d0 2024-03-06T07:28:24.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63700043e0 con 0x7f63801614d0 2024-03-06T07:28:24.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63800cc770 con 0x7f63801614d0 2024-03-06T07:28:24.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.488+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63800ccd30 con 0x7f63801614d0 2024-03-06T07:28:24.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.489+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6380068ff0 con 0x7f63801614d0 2024-03-06T07:28:24.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.490+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f637000c6d0 con 0x7f63801614d0 2024-03-06T07:28:24.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.490+0000 7f63677fe640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f636c06b4d0 0x7f636c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.490+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f637008a010 con 0x7f63801614d0 2024-03-06T07:28:24.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.490+0000 7f637ed9d640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f636c06b4d0 0x7f636c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.491+0000 7f637ed9d640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f636c06b4d0 0x7f636c06d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f63780012e0 tx=0x7f6378009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.494+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f63800c90b0 con 0x7f63801614d0 2024-03-06T07:28:24.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.670+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63800b1ec0 con 0x7f63801614d0 2024-03-06T07:28:24.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.670+0000 7f63677fe640 1 -- 172.21.15.72:0/771209248 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6370017020 con 0x7f63801614d0 2024-03-06T07:28:24.673 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f636c06b4d0 msgr2=0x7f636c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f636c06b4d0 0x7f636c06d990 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f63780012e0 tx=0x7f6378009040 comp rx=0 tx=0).stop 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 msgr2=0x7f6380151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 0x7f6380151e20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f637000e810 tx=0x7f637000ecd0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 shutdown_connections 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f636c06b4d0 0x7f636c06d990 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f63801614d0 0x7f6380151e20 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63800d4860 0x7f63800b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 --2- 172.21.15.72:0/771209248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63800b0e30 0x7f6380152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 >> 172.21.15.72:0/771209248 conn(0x7f63800d3d40 msgr2=0x7f63800cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 shutdown_connections 2024-03-06T07:28:24.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.672+0000 7f638517a640 1 -- 172.21.15.72:0/771209248 wait complete. 2024-03-06T07:28:24.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:28:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.787+0000 7ff8aea1c640 1 Processor -- start 2024-03-06T07:28:24.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.787+0000 7ff8aea1c640 1 -- start start 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8aea1c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8a80d10c0 0x7ff8a80d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8aea1c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 0x7ff8a815e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8aea1c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a80cfa50 con 0x7ff8a815eb00 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8aea1c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a815c050 con 0x7ff8a80d10c0 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8aea1c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80d0060 con 0x7ff8a80d14a0 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8acfb9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff8a815eb00 0x7ff8a80cae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55390/0 (socket says 172.21.15.72:55390) 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8acfb9640 1 -- 172.21.15.72:0/2844518770 learned_addr learned my addr 172.21.15.72:0/2844518770 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479633464 0 0) 0x7ff8a80cfa50 con 0x7ff8a815eb00 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff890003520 con 0x7ff8a815eb00 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a77fe640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 0x7ff8a815e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a7fff640 1 --2- 172.21.15.72:0/2844518770 >> v2:172.21.15.146:3300/0 conn(0x7ff8a80d10c0 0x7ff8a80d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400703703 0 0) 0x7ff890003520 con 0x7ff8a815eb00 2024-03-06T07:28:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 msgr2=0x7ff8a815e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 0x7ff8a815e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 >> v2:172.21.15.146:3300/0 conn(0x7ff8a80d10c0 msgr2=0x7ff8a80d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 --2- 172.21.15.72:0/2844518770 >> v2:172.21.15.146:3300/0 conn(0x7ff8a80d10c0 0x7ff8a80d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80d0a30 con 0x7ff8a815eb00 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff8a00019f0 con 0x7ff8a815eb00 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 >> v1:172.21.15.72:6789/0 conn(0x7ff8a815eb00 legacy=0x7ff8a80cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.788+0000 7ff8a6ffd640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8900039b0 0x7ff890003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a6ffd640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 0x7ff890004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a6ffd640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff890004d40 0x7ff8900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80d0060 con 0x7ff890004d40 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89000b9c0 con 0x7ff8900039b0 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89000bb40 con 0x7ff8a80d14a0 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8acfb9640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff890004d40 0x7ff8900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a7fff640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8900039b0 0x7ff890003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8a77fe640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 0x7ff890004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8acfb9640 1 -- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 msgr2=0x7ff890004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8acfb9640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a80d14a0 0x7ff890004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8acfb9640 1 -- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8900039b0 msgr2=0x7ff890003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8acfb9640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8900039b0 0x7ff890003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.789+0000 7ff8acfb9640 1 -- 172.21.15.72:0/2844518770 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890003520 con 0x7ff890004d40 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8acfb9640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff890004d40 0x7ff8900051c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000a850 tx=0x7ff890005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2756598e8f01198 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000a4a0 con 0x7ff890004d40 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a0034650 con 0x7ff890004d40 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8a6ffd640 1 -- 172.21.15.72:0/2844518770 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0034a40 con 0x7ff890004d40 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8aea1c640 1 -- 172.21.15.72:0/2844518770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff890004d40 msgr2=0x7ff8900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/2844518770 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff890004d40 0x7ff8900051c0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000a850 tx=0x7ff890005830 comp rx=0 tx=0).stop 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8acfb9640 1 -- 172.21.15.72:0/2844518770 reap_dead start 2024-03-06T07:28:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8aea1c640 1 -- 172.21.15.72:0/2844518770 shutdown_connections 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8aea1c640 1 -- 172.21.15.72:0/2844518770 >> 172.21.15.72:0/2844518770 conn(0x7ff8a80b3800 msgr2=0x7ff8a80b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8aea1c640 1 -- 172.21.15.72:0/2844518770 shutdown_connections 2024-03-06T07:28:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.790+0000 7ff8aea1c640 1 -- 172.21.15.72:0/2844518770 wait complete. 2024-03-06T07:28:24.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.792+0000 7ff8aea1c640 1 Processor -- start 2024-03-06T07:28:24.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.792+0000 7ff8aea1c640 1 -- start start 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 0x7ff8a80b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a80cd660 0x7ff8a80ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a814fb20 0x7ff8a80ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a815f440 con 0x7ff8a814f740 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a815f160 con 0x7ff8a80cd660 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80cfcf0 con 0x7ff8a814fb20 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 0x7ff8a80b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 0x7ff8a80b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45644/0 (socket says 172.21.15.72:45644) 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 -- 172.21.15.72:0/1301341733 learned_addr learned my addr 172.21.15.72:0/1301341733 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 -- 172.21.15.72:0/1301341733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a814fb20 msgr2=0x7ff8a80ce5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a814fb20 0x7ff8a80ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 -- 172.21.15.72:0/1301341733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a80cd660 msgr2=0x7ff8a80ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a80cd660 0x7ff8a80ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 -- 172.21.15.72:0/1301341733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80b5260 con 0x7ff8a814f740 2024-03-06T07:28:24.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a7fff640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 0x7ff8a80b48f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff8980118e0 tx=0x7ff898011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89800faf0 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89800fc90 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a80b5540 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898014440 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.794+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a80b5b00 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.795+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a809fe60 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.796+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff898012040 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.796+0000 7ff8a4ff9640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff89406b4d0 0x7ff89406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.796+0000 7ff8a77fe640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff89406b4d0 0x7ff89406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.796+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7ff89808ab00 con 0x7ff8a814f740 2024-03-06T07:28:24.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.797+0000 7ff8a77fe640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff89406b4d0 0x7ff89406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff8a815d380 tx=0x7ff89c0095d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:24.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.801+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff898019030 con 0x7ff8a814f740 2024-03-06T07:28:25.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.998+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff8a80d0c30 con 0x7ff8a814f740 2024-03-06T07:28:25.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:24.999+0000 7ff8a4ff9640 1 -- 172.21.15.72:0/1301341733 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8980597b0 con 0x7ff8a814f740 2024-03-06T07:28:25.001 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.000+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff89406b4d0 msgr2=0x7ff89406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.000+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff89406b4d0 0x7ff89406d990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7ff8a815d380 tx=0x7ff89c0095d0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.000+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 msgr2=0x7ff8a80b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 0x7ff8a80b48f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff8980118e0 tx=0x7ff898011da0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 shutdown_connections 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff89406b4d0 0x7ff89406d990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a814fb20 0x7ff8a80ce5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a80cd660 0x7ff8a80ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 --2- 172.21.15.72:0/1301341733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a814f740 0x7ff8a80b48f0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 >> 172.21.15.72:0/1301341733 conn(0x7ff8a80b3800 msgr2=0x7ff8a80b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 shutdown_connections 2024-03-06T07:28:25.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.001+0000 7ff8aea1c640 1 -- 172.21.15.72:0/1301341733 wait complete. 2024-03-06T07:28:25.014 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:28:25.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb8c15640 1 Processor -- start 2024-03-06T07:28:25.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb8c15640 1 -- start start 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb8c15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9bb40d4860 0x7f9bb40d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb8c15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb41618b0 con 0x7f9bb40b0e30 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb8c15640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb40cf400 con 0x7f9bb41614d0 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb8c15640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb40d4cf0 con 0x7f9bb40d4860 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.117+0000 7f9bb2d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9bb40b0e30 0x7f9bb40b1260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55394/0 (socket says 172.21.15.72:55394) 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9bb2d9d640 1 -- 172.21.15.72:0/1419087983 learned_addr learned my addr 172.21.15.72:0/1419087983 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9babfff640 1 --2- 172.21.15.72:0/1419087983 >> v2:172.21.15.146:3300/0 conn(0x7f9bb40d4860 0x7f9bb40d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9bb1d9b640 1 -- 172.21.15.72:0/1419087983 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296584687 0 0) 0x7f9bb41618b0 con 0x7f9bb40b0e30 2024-03-06T07:28:25.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9bb1d9b640 1 -- 172.21.15.72:0/1419087983 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba0003520 con 0x7f9bb40b0e30 2024-03-06T07:28:25.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9babfff640 1 -- 172.21.15.72:0/1419087983 >> v1:172.21.15.146:6789/0 conn(0x7f9bb41614d0 legacy=0x7f9bb40d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9babfff640 1 -- 172.21.15.72:0/1419087983 >> v1:172.21.15.72:6789/0 conn(0x7f9bb40b0e30 legacy=0x7f9bb40b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.118+0000 7f9babfff640 1 -- 172.21.15.72:0/1419087983 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40cf6a0 con 0x7f9bb40d4860 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.119+0000 7f9babfff640 1 --2- 172.21.15.72:0/1419087983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40d4860 0x7f9bb40d2020 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0097a0 tx=0x7f9b9c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af8df5ce63d47a75 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.119+0000 7f9bb1d9b640 1 -- 172.21.15.72:0/1419087983 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c034070 con 0x7f9bb40d4860 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.119+0000 7f9bb1d9b640 1 -- 172.21.15.72:0/1419087983 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c030610 con 0x7f9bb40d4860 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.119+0000 7f9bb1d9b640 1 -- 172.21.15.72:0/1419087983 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c0309c0 con 0x7f9bb40d4860 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.119+0000 7f9bb8c15640 1 -- 172.21.15.72:0/1419087983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40d4860 msgr2=0x7f9bb40d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.119+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/1419087983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40d4860 0x7f9bb40d2020 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0097a0 tx=0x7f9b9c02b960 comp rx=0 tx=0).stop 2024-03-06T07:28:25.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.120+0000 7f9bb8c15640 1 -- 172.21.15.72:0/1419087983 shutdown_connections 2024-03-06T07:28:25.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.120+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/1419087983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40d4860 0x7f9bb40d2020 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.120+0000 7f9bb8c15640 1 -- 172.21.15.72:0/1419087983 >> 172.21.15.72:0/1419087983 conn(0x7f9bb40d3d40 msgr2=0x7f9bb40d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.120+0000 7f9bb8c15640 1 -- 172.21.15.72:0/1419087983 shutdown_connections 2024-03-06T07:28:25.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.120+0000 7f9bb8c15640 1 -- 172.21.15.72:0/1419087983 wait complete. 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.121+0000 7f9bb8c15640 1 Processor -- start 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.121+0000 7f9bb8c15640 1 -- start start 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb8c15640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40b0e30 0x7f9bb40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb8c15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 0x7f9bb40b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb8c15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bb41614d0 0x7f9bb40ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb8c15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40b0910 con 0x7f9bb40d4860 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb8c15640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40cfc60 con 0x7f9bb40b0e30 2024-03-06T07:28:25.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb8c15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb40d2560 con 0x7f9bb41614d0 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 0x7f9bb40b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 0x7f9bb40b7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45646/0 (socket says 172.21.15.72:45646) 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 -- 172.21.15.72:0/2853098632 learned_addr learned my addr 172.21.15.72:0/2853098632 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb259c640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40b0e30 0x7f9bb40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9bb2d9d640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bb41614d0 0x7f9bb40ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 -- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bb41614d0 msgr2=0x7f9bb40ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bb41614d0 0x7f9bb40ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 -- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40b0e30 msgr2=0x7f9bb40cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40b0e30 0x7f9bb40cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 -- 172.21.15.72:0/2853098632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb40b7fa0 con 0x7f9bb40d4860 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9babfff640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 0x7f9bb40b7a60 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c000c00 tx=0x7f9b9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.122+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c034040 con 0x7f9bb40d4860 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.123+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb40cb3f0 con 0x7f9bb40d4860 2024-03-06T07:28:25.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.123+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9c030410 con 0x7f9bb40d4860 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.123+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9c004230 con 0x7f9bb40d4860 2024-03-06T07:28:25.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.123+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb40cb9b0 con 0x7f9bb40d4860 2024-03-06T07:28:25.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.124+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb4068ff0 con 0x7f9bb40d4860 2024-03-06T07:28:25.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.124+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b9c04c060 con 0x7f9bb40d4860 2024-03-06T07:28:25.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.125+0000 7f9baaffd640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.125+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f9b9c0abed0 con 0x7f9bb40d4860 2024-03-06T07:28:25.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.125+0000 7f9bb259c640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.125+0000 7f9bb259c640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4007fd0 tx=0x7f9ba4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.129+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b9c07ad40 con 0x7f9bb40d4860 2024-03-06T07:28:25.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.306+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9bb40d1cd0 con 0x7f9bb40d4860 2024-03-06T07:28:25.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.306+0000 7f9baaffd640 1 -- 172.21.15.72:0/2853098632 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9b9c07a640 con 0x7f9bb40d4860 2024-03-06T07:28:25.309 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b9406b4d0 msgr2=0x7f9b9406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b9406b4d0 0x7f9b9406d990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9ba4007fd0 tx=0x7f9ba4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 msgr2=0x7f9bb40b7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 0x7f9bb40b7a60 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c000c00 tx=0x7f9b9c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 shutdown_connections 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b9406b4d0 0x7f9b9406d990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bb41614d0 0x7f9bb40ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bb40d4860 0x7f9bb40b7a60 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 --2- 172.21.15.72:0/2853098632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bb40b0e30 0x7f9bb40cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 >> 172.21.15.72:0/2853098632 conn(0x7f9bb40d3d40 msgr2=0x7f9bb40d4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 shutdown_connections 2024-03-06T07:28:25.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.308+0000 7f9bb8c15640 1 -- 172.21.15.72:0/2853098632 wait complete. 2024-03-06T07:28:25.322 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:25.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.420+0000 7fcb6aa37640 1 Processor -- start 2024-03-06T07:28:25.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.420+0000 7fcb6aa37640 1 -- start start 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb6aa37640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcb6415bdd0 0x7fcb640c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb6aa37640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb640d16e0 0x7fcb640d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb6aa37640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb640cebe0 con 0x7fcb6415fa10 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb6aa37640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb640ce980 con 0x7fcb640d16e0 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb6aa37640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb640ceec0 con 0x7fcb6415bdd0 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb68fd4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb640d16e0 0x7fcb640d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb68fd4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb640d16e0 0x7fcb640d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45650/0 (socket says 172.21.15.72:45650) 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.421+0000 7fcb68fd4640 1 -- 172.21.15.72:0/3756048994 learned_addr learned my addr 172.21.15.72:0/3756048994 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb63fff640 1 --2- 172.21.15.72:0/3756048994 >> v2:172.21.15.146:3300/0 conn(0x7fcb6415bdd0 0x7fcb640c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb68fd4640 1 -- 172.21.15.72:0/3756048994 >> v2:172.21.15.146:3300/0 conn(0x7fcb6415bdd0 msgr2=0x7fcb640c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb68fd4640 1 --2- 172.21.15.72:0/3756048994 >> v2:172.21.15.146:3300/0 conn(0x7fcb6415bdd0 0x7fcb640c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb68fd4640 1 -- 172.21.15.72:0/3756048994 >> v1:172.21.15.146:6789/0 conn(0x7fcb6415fa10 legacy=0x7fcb640ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb68fd4640 1 -- 172.21.15.72:0/3756048994 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb640cf1a0 con 0x7fcb640d16e0 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb68fd4640 1 --2- 172.21.15.72:0/3756048994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640d1af0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fcb6406e3c0 tx=0x7fcb5c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f77283a1208f8ef server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.422+0000 7fcb62ffd640 1 -- 172.21.15.72:0/3756048994 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c02ca80 con 0x7fcb640d16e0 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb62ffd640 1 -- 172.21.15.72:0/3756048994 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c02cc20 con 0x7fcb640d16e0 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb62ffd640 1 -- 172.21.15.72:0/3756048994 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0343f0 con 0x7fcb640d16e0 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 -- 172.21.15.72:0/3756048994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 msgr2=0x7fcb640d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/3756048994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640d1af0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fcb6406e3c0 tx=0x7fcb5c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 -- 172.21.15.72:0/3756048994 shutdown_connections 2024-03-06T07:28:25.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/3756048994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640d1af0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/3756048994 >> v2:172.21.15.146:3300/0 conn(0x7fcb6415bdd0 0x7fcb640c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 -- 172.21.15.72:0/3756048994 >> 172.21.15.72:0/3756048994 conn(0x7fcb6409a9f0 msgr2=0x7fcb6409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 -- 172.21.15.72:0/3756048994 shutdown_connections 2024-03-06T07:28:25.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.423+0000 7fcb6aa37640 1 -- 172.21.15.72:0/3756048994 wait complete. 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.424+0000 7fcb6aa37640 1 Processor -- start 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.424+0000 7fcb6aa37640 1 -- start start 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb6aa37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb6aa37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb6415bdd0 0x7fcb64143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb6aa37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb6415fa10 0x7fcb641442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb6aa37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640cc5d0 con 0x7fcb640d16e0 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb6aa37640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb640c5b50 con 0x7fcb6415fa10 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb6aa37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6415c1b0 con 0x7fcb6415bdd0 2024-03-06T07:28:25.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45658/0 (socket says 172.21.15.72:45658) 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb63fff640 1 -- 172.21.15.72:0/2152773277 learned_addr learned my addr 172.21.15.72:0/2152773277 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb68fd4640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb6415fa10 0x7fcb641442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.425+0000 7fcb637fe640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb6415bdd0 0x7fcb64143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb63fff640 1 -- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb6415bdd0 msgr2=0x7fcb64143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb63fff640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb6415bdd0 0x7fcb64143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb63fff640 1 -- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb6415fa10 msgr2=0x7fcb641442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb63fff640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb6415fa10 0x7fcb641442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb63fff640 1 -- 172.21.15.72:0/2152773277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6415e040 con 0x7fcb640d16e0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb68fd4640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb6415fa10 0x7fcb641442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb63fff640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640b46d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400d5f0 tx=0x7fcb5400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb540125f0 con 0x7fcb640d16e0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb54019070 con 0x7fcb640d16e0 2024-03-06T07:28:25.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6415e320 con 0x7fcb640d16e0 2024-03-06T07:28:25.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb540129d0 con 0x7fcb640d16e0 2024-03-06T07:28:25.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.426+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6415e8e0 con 0x7fcb640d16e0 2024-03-06T07:28:25.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.427+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6404f330 con 0x7fcb640d16e0 2024-03-06T07:28:25.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.427+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb54012b70 con 0x7fcb640d16e0 2024-03-06T07:28:25.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.428+0000 7fcb60ff9640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5006b500 0x7fcb5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.428+0000 7fcb637fe640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5006b500 0x7fcb5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.428+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fcb5408a680 con 0x7fcb640d16e0 2024-03-06T07:28:25.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.428+0000 7fcb637fe640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5006b500 0x7fcb5006d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcb641e82a0 tx=0x7fcb58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.432+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb54059450 con 0x7fcb640d16e0 2024-03-06T07:28:25.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.719+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6415ed20 con 0x7fcb640d16e0 2024-03-06T07:28:25.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.719+0000 7fcb60ff9640 1 -- 172.21.15.72:0/2152773277 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7fcb54058d50 con 0x7fcb640d16e0 2024-03-06T07:28:25.721 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:25.721 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:25.722 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5006b500 msgr2=0x7fcb5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5006b500 0x7fcb5006d9c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fcb641e82a0 tx=0x7fcb58009040 comp rx=0 tx=0).stop 2024-03-06T07:28:25.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 msgr2=0x7fcb640b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640b46d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400d5f0 tx=0x7fcb5400dab0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 shutdown_connections 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5006b500 0x7fcb5006d9c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb6415fa10 0x7fcb641442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb6415bdd0 0x7fcb64143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 --2- 172.21.15.72:0/2152773277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb640d16e0 0x7fcb640b46d0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 >> 172.21.15.72:0/2152773277 conn(0x7fcb6409a9f0 msgr2=0x7fcb640c5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 shutdown_connections 2024-03-06T07:28:25.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.721+0000 7fcb6aa37640 1 -- 172.21.15.72:0/2152773277 wait complete. 2024-03-06T07:28:25.734 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:25.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.834+0000 7f071e593640 1 Processor -- start 2024-03-06T07:28:25.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.834+0000 7f071e593640 1 -- start start 2024-03-06T07:28:25.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071e593640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f071815bf40 0x7f07180cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071e593640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f071806baa0 0x7f07180cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071e593640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07180afe40 con 0x7f071806afb0 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071e593640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180cd680 con 0x7f071806baa0 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071e593640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180b0120 con 0x7f071815bf40 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071cb30640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f071806baa0 0x7f07180cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f0717fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f071806afb0 0x7f071806b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55406/0 (socket says 172.21.15.72:55406) 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f0717fff640 1 -- 172.21.15.72:0/1611232735 learned_addr learned my addr 172.21.15.72:0/1611232735 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f07177fe640 1 --2- 172.21.15.72:0/1611232735 >> v2:172.21.15.146:3300/0 conn(0x7f071815bf40 0x7f07180cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071cb30640 1 -- 172.21.15.72:0/1611232735 >> v2:172.21.15.146:3300/0 conn(0x7f071815bf40 msgr2=0x7f07180cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071cb30640 1 --2- 172.21.15.72:0/1611232735 >> v2:172.21.15.146:3300/0 conn(0x7f071815bf40 0x7f07180cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071cb30640 1 -- 172.21.15.72:0/1611232735 >> v1:172.21.15.72:6789/0 conn(0x7f071806afb0 legacy=0x7f071806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f071cb30640 1 -- 172.21.15.72:0/1611232735 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180cd310 con 0x7f071806baa0 2024-03-06T07:28:25.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.835+0000 7f07177fe640 1 --2- 172.21.15.72:0/1611232735 >> v2:172.21.15.146:3300/0 conn(0x7f071815bf40 0x7f07180cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:25.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.836+0000 7f071cb30640 1 --2- 172.21.15.72:0/1611232735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f07180cf3a0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f071804fa70 tx=0x7f071002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e7ce1e697efdaa server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.836+0000 7f0716ffd640 1 -- 172.21.15.72:0/1611232735 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710039070 con 0x7f071806baa0 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.836+0000 7f0716ffd640 1 -- 172.21.15.72:0/1611232735 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071002cc80 con 0x7f071806baa0 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.836+0000 7f0716ffd640 1 -- 172.21.15.72:0/1611232735 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710034660 con 0x7f071806baa0 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 -- 172.21.15.72:0/1611232735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 msgr2=0x7f07180cf3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 --2- 172.21.15.72:0/1611232735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f07180cf3a0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f071804fa70 tx=0x7f071002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 -- 172.21.15.72:0/1611232735 shutdown_connections 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 --2- 172.21.15.72:0/1611232735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f07180cf3a0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 --2- 172.21.15.72:0/1611232735 >> v2:172.21.15.146:3300/0 conn(0x7f071815bf40 0x7f07180cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 -- 172.21.15.72:0/1611232735 >> 172.21.15.72:0/1611232735 conn(0x7f07180d0960 msgr2=0x7f07180d0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 -- 172.21.15.72:0/1611232735 shutdown_connections 2024-03-06T07:28:25.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.837+0000 7f071e593640 1 -- 172.21.15.72:0/1611232735 wait complete. 2024-03-06T07:28:25.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.838+0000 7f071e593640 1 Processor -- start 2024-03-06T07:28:25.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.839+0000 7f071e593640 1 -- start start 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071e593640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f071806afb0 0x7f0718142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071e593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f0718145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071e593640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071815bf40 0x7f07180a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071e593640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071806f510 con 0x7f071806baa0 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071e593640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071806f390 con 0x7f071815bf40 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071e593640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071806f690 con 0x7f071806afb0 2024-03-06T07:28:25.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f0718145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f0718145be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45662/0 (socket says 172.21.15.72:45662) 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 -- 172.21.15.72:0/3326883916 learned_addr learned my addr 172.21.15.72:0/3326883916 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f0717fff640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f071806afb0 0x7f0718142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071cb30640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071815bf40 0x7f07180a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 -- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f071806afb0 msgr2=0x7f0718142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f071806afb0 0x7f0718142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 -- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071815bf40 msgr2=0x7f07180a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071815bf40 0x7f07180a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 -- 172.21.15.72:0/3326883916 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180a9a00 con 0x7f071806baa0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f071cb30640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071815bf40 0x7f07180a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.840+0000 7f07177fe640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f0718145be0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f070c00c7c0 tx=0x7f070c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.841+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c015070 con 0x7f071806baa0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.841+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070c007820 con 0x7f071806baa0 2024-03-06T07:28:25.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.841+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071815d3c0 con 0x7f071806baa0 2024-03-06T07:28:25.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.841+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f070c007bf0 con 0x7f071806baa0 2024-03-06T07:28:25.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.841+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071815d960 con 0x7f071806baa0 2024-03-06T07:28:25.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.842+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071804f330 con 0x7f071806baa0 2024-03-06T07:28:25.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.842+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f070c018070 con 0x7f071806baa0 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.842+0000 7f0714ff9640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f070406b4d0 0x7f070406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.842+0000 7f0717fff640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f070406b4d0 0x7f070406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.842+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f070c08a140 con 0x7f071806baa0 2024-03-06T07:28:25.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.843+0000 7f0717fff640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f070406b4d0 0x7f070406d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0708000f60 tx=0x7f0708009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:25.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:25.847+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07180a9a00 con 0x7f071806baa0 2024-03-06T07:28:26.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.024+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07180729d0 con 0x7f071806baa0 2024-03-06T07:28:26.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.025+0000 7f0714ff9640 1 -- 172.21.15.72:0/3326883916 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17182 (secure 0 0 0) 0x7f070c058f10 con 0x7f071806baa0 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:26.028 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":116,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:32.298669+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1a","mappings":[{"from":0,"to":2}]},{"pgid":"17.25","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6835/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6834/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6837/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.72:6836/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.146:6835/118157918":"2024-03-07T07:27:25.085724+0000","172.21.15.146:6833/118157918":"2024-03-07T07:27:25.085724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f070406b4d0 msgr2=0x7f070406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f070406b4d0 0x7f070406d990 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f0708000f60 tx=0x7f0708009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 msgr2=0x7f0718145be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f0718145be0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f070c00c7c0 tx=0x7f070c00cc80 comp rx=0 tx=0).stop 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 shutdown_connections 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f070406b4d0 0x7f070406d990 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f071815bf40 0x7f07180a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f071806baa0 0x7f0718145be0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 --2- 172.21.15.72:0/3326883916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f071806afb0 0x7f0718142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.027+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 >> 172.21.15.72:0/3326883916 conn(0x7f07180d0960 msgr2=0x7f07180b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.028+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 shutdown_connections 2024-03-06T07:28:26.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.028+0000 7f071e593640 1 -- 172.21.15.72:0/3326883916 wait complete. 2024-03-06T07:28:26.041 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:28:26.041 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.140+0000 7f6dd1f65640 1 Processor -- start 2024-03-06T07:28:26.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.140+0000 7f6dd1f65640 1 -- start start 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.141+0000 7f6dd1f65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 0x7f6dcc0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.141+0000 7f6dd1f65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc0b3430 con 0x7f6dcc15dfe0 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.141+0000 7f6dd1f65640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc0b3710 con 0x7f6dcc0d22b0 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.141+0000 7f6dd1f65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0b31e0 con 0x7f6dcc0d2710 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.141+0000 7f6dcb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6dcc15dfe0 0x7f6dcc0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55414/0 (socket says 172.21.15.72:55414) 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.141+0000 7f6dcb7fe640 1 -- 172.21.15.72:0/2038508810 learned_addr learned my addr 172.21.15.72:0/2038508810 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681115408 0 0) 0x7f6dcc0b3430 con 0x7f6dcc15dfe0 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db8003520 con 0x7f6dcc15dfe0 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 0x7f6dcc0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882447746 0 0) 0x7f6db8003520 con 0x7f6dcc15dfe0 2024-03-06T07:28:26.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 msgr2=0x7f6dcc0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 0x7f6dcc0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 >> v1:172.21.15.146:6789/0 conn(0x7f6dcc0d22b0 legacy=0x7f6dcc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0b39f0 con 0x7f6dcc15dfe0 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6dbc001670 con 0x7f6dcc15dfe0 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 >> v1:172.21.15.72:6789/0 conn(0x7f6dcc15dfe0 legacy=0x7f6dcc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 0x7f6db8003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6db8003f80 0x7f6db8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6db80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0b31e0 con 0x7f6db8004d40 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db800bad0 con 0x7f6db8003f80 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db800bcd0 con 0x7f6dcc0d2710 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6db80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcb7fe640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 0x7f6db8003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dc3fff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6db8003f80 0x7f6db8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 -- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 msgr2=0x7f6db8003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc0d2710 0x7f6db8003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 -- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6db8003f80 msgr2=0x7f6db8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6db8003f80 0x7f6db8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dcbfff640 1 -- 172.21.15.72:0/2038508810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8003520 con 0x7f6db8004d40 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.142+0000 7f6dc3fff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6db8003f80 0x7f6db8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.143+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6db80051c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009f90 tx=0x7f6db8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512cc2015e3a81b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.143+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4039070 con 0x7f6db8004d40 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.143+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc402cbd0 con 0x7f6db8004d40 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.143+0000 7f6dcaffd640 1 -- 172.21.15.72:0/2038508810 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc4009c10 con 0x7f6db8004d40 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2038508810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 msgr2=0x7f6db80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2038508810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6db80051c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f6dc4009f90 tx=0x7f6db8005830 comp rx=0 tx=0).stop 2024-03-06T07:28:26.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dcbfff640 1 -- 172.21.15.72:0/2038508810 reap_dead start 2024-03-06T07:28:26.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2038508810 shutdown_connections 2024-03-06T07:28:26.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2038508810 >> 172.21.15.72:0/2038508810 conn(0x7f6dcc0d3cd0 msgr2=0x7f6dcc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2038508810 shutdown_connections 2024-03-06T07:28:26.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.144+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2038508810 wait complete. 2024-03-06T07:28:26.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.145+0000 7f6dd1f65640 1 Processor -- start 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.145+0000 7f6dd1f65640 1 -- start start 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6dcc0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dcc161010 0x7f6dcc151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc15ed20 0x7f6dcc15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0cfa00 con 0x7f6db8004d40 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0b3d60 con 0x7f6dcc161010 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0d1c70 con 0x7f6dcc15ed20 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6dcc0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6dcc0b8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45674/0 (socket says 172.21.15.72:45674) 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 -- 172.21.15.72:0/2001168407 learned_addr learned my addr 172.21.15.72:0/2001168407 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:26.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc15ed20 0x7f6dcc15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dc3fff640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dcc161010 0x7f6dcc151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 -- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc15ed20 msgr2=0x7f6dcc15f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc15ed20 0x7f6dcc15f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 -- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dcc161010 msgr2=0x7f6dcc151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dcc161010 0x7f6dcc151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 -- 172.21.15.72:0/2001168407 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0b9100 con 0x7f6db8004d40 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcbfff640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc15ed20 0x7f6dcc15f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dc3fff640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dcc161010 0x7f6dcc151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dcb7fe640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6dcc0b8bc0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc007290 tx=0x7f6dbc007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc00f9c0 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc018070 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc0b93a0 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.146+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc00fda0 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.147+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc0b9960 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.148+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc0b9ba0 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.148+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6dbc01d460 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.148+0000 7f6dc8ff9640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6dac06b500 0x7f6dac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.148+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f6dbc08a370 con 0x7f6db8004d40 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.148+0000 7f6dc3fff640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6dac06b500 0x7f6dac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.149+0000 7f6dc3fff640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6dac06b500 0x7f6dac06d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6db4007fd0 tx=0x7f6db4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.153+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6dbc059140 con 0x7f6db8004d40 2024-03-06T07:28:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.368+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dcc04f330 con 0x7f6db8004d40 2024-03-06T07:28:26.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.369+0000 7f6dc8ff9640 1 -- 172.21.15.72:0/2001168407 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f6dbc058a40 con 0x7f6db8004d40 2024-03-06T07:28:26.371 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:26.371 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:26.371 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6dac06b500 msgr2=0x7f6dac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6dac06b500 0x7f6dac06d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6db4007fd0 tx=0x7f6db4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 msgr2=0x7f6dcc0b8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6dcc0b8bc0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc007290 tx=0x7f6dbc007750 comp rx=0 tx=0).stop 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 shutdown_connections 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6dac06b500 0x7f6dac06d9c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6dcc15ed20 0x7f6dcc15f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6dcc161010 0x7f6dcc151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 --2- 172.21.15.72:0/2001168407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6db8004d40 0x7f6dcc0b8bc0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 >> 172.21.15.72:0/2001168407 conn(0x7f6dcc0d3cd0 msgr2=0x7f6dcc06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 shutdown_connections 2024-03-06T07:28:26.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.371+0000 7f6dd1f65640 1 -- 172.21.15.72:0/2001168407 wait complete. 2024-03-06T07:28:26.384 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:26.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.483+0000 7f5bbbd79640 1 Processor -- start 2024-03-06T07:28:26.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.483+0000 7f5bbbd79640 1 -- start start 2024-03-06T07:28:26.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bbbd79640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bb415dfe0 0x7f5bb40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bbbd79640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 0x7f5bb40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bbbd79640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40b3430 con 0x7f5bb40cf830 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bbbd79640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb40b31e0 con 0x7f5bb415dfe0 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bbbd79640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40b3d60 con 0x7f5bb40d29c0 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bb415dfe0 0x7f5bb40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bba316640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 0x7f5bb40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bb415dfe0 0x7f5bb40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:33960/0 (socket says 172.21.15.72:33960) 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bba316640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 0x7f5bb40b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:40750/0 (socket says 172.21.15.72:40750) 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 -- 172.21.15.72:0/439050828 learned_addr learned my addr 172.21.15.72:0/439050828 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9314640 1 --1- 172.21.15.72:0/439050828 >> v1:172.21.15.146:6789/0 conn(0x7f5bb40cf830 0x7f5bb40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:56122/0 (socket says 172.21.15.72:56122) 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 -- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 msgr2=0x7f5bb40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 --2- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 0x7f5bb40b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 -- 172.21.15.72:0/439050828 >> v1:172.21.15.146:6789/0 conn(0x7f5bb40cf830 legacy=0x7f5bb40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb9b15640 1 -- 172.21.15.72:0/439050828 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40b3710 con 0x7f5bb415dfe0 2024-03-06T07:28:26.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bb8b13640 1 -- 172.21.15.72:0/439050828 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151432797 0 0) 0x7f5bb40b3430 con 0x7f5bb40cf830 2024-03-06T07:28:26.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.485+0000 7f5bba316640 1 --2- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 0x7f5bb40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.486+0000 7f5bb9b15640 1 --2- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb415dfe0 0x7f5bb40cf2f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009870 tx=0x7f5ba402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b443b80442c044e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.486+0000 7f5bb8b13640 1 -- 172.21.15.72:0/439050828 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4034070 con 0x7f5bb415dfe0 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.486+0000 7f5bb8b13640 1 -- 172.21.15.72:0/439050828 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba4030840 con 0x7f5bb415dfe0 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.486+0000 7f5bb8b13640 1 -- 172.21.15.72:0/439050828 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4030c10 con 0x7f5bb415dfe0 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 -- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb415dfe0 msgr2=0x7f5bb40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb415dfe0 0x7f5bb40cf2f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009870 tx=0x7f5ba402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 -- 172.21.15.72:0/439050828 shutdown_connections 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb40d29c0 0x7f5bb40b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/439050828 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb415dfe0 0x7f5bb40cf2f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 -- 172.21.15.72:0/439050828 >> 172.21.15.72:0/439050828 conn(0x7f5bb40d3cd0 msgr2=0x7f5bb40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 -- 172.21.15.72:0/439050828 shutdown_connections 2024-03-06T07:28:26.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.487+0000 7f5bbbd79640 1 -- 172.21.15.72:0/439050828 wait complete. 2024-03-06T07:28:26.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.488+0000 7f5bbbd79640 1 Processor -- start 2024-03-06T07:28:26.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.488+0000 7f5bbbd79640 1 -- start start 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bbbd79640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 0x7f5bb40b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bbbd79640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb40d29c0 0x7f5bb40b8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bbbd79640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb415dfe0 0x7f5bb414c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bbbd79640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40d1df0 con 0x7f5bb40cf830 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bbbd79640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40d1c70 con 0x7f5bb40d29c0 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bbbd79640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb415e3c0 con 0x7f5bb415dfe0 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 0x7f5bb40b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 0x7f5bb40b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45678/0 (socket says 172.21.15.72:45678) 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 -- 172.21.15.72:0/1182164444 learned_addr learned my addr 172.21.15.72:0/1182164444 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bba316640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb415dfe0 0x7f5bb414c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 -- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb415dfe0 msgr2=0x7f5bb414c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb415dfe0 0x7f5bb414c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9314640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb40d29c0 0x7f5bb40b8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 -- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb40d29c0 msgr2=0x7f5bb40b8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb40d29c0 0x7f5bb40b8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 -- 172.21.15.72:0/1182164444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40b8cf0 con 0x7f5bb40cf830 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.489+0000 7f5bb9b15640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 0x7f5bb40b4c60 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4000c00 tx=0x7f5ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.490+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4034040 con 0x7f5bb40cf830 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.490+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba4039d90 con 0x7f5bb40cf830 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.490+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb40b8fd0 con 0x7f5bb40cf830 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.490+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4043630 con 0x7f5bb40cf830 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.490+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb40b9590 con 0x7f5bb40cf830 2024-03-06T07:28:26.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.491+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb404f330 con 0x7f5bb40cf830 2024-03-06T07:28:26.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.492+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ba404a060 con 0x7f5bb40cf830 2024-03-06T07:28:26.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.492+0000 7f5baa7fc640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b9806b4d0 0x7f5b9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.492+0000 7f5bb9314640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b9806b4d0 0x7f5b9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.492+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f5ba40ac380 con 0x7f5bb40cf830 2024-03-06T07:28:26.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.492+0000 7f5bb9314640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b9806b4d0 0x7f5b9806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0007fd0 tx=0x7f5bb0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.496+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ba403e070 con 0x7f5bb40cf830 2024-03-06T07:28:26.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.710+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bb40ca330 con 0x7f5bb40cf830 2024-03-06T07:28:26.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.711+0000 7f5baa7fc640 1 -- 172.21.15.72:0/1182164444 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+4981 (secure 0 0 0) 0x7f5ba402d330 con 0x7f5bb40cf830 2024-03-06T07:28:26.713 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:26.713 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":72,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:27:02.515750+0000","modified":"2024-03-06T07:27:28.290731+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8217},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8217":{"gid":8217,"name":"c","rank":0,"incarnation":69,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6835/3088662828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3088662828},{"type":"v1","addr":"172.21.15.72:6835","nonce":3088662828}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8217,"qdb_cluster":[8217]},"id":8}]} 2024-03-06T07:28:26.713 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 72 2024-03-06T07:28:26.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b9806b4d0 msgr2=0x7f5b9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b9806b4d0 0x7f5b9806d990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0007fd0 tx=0x7f5bb0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:26.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 msgr2=0x7f5bb40b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 0x7f5bb40b4c60 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4000c00 tx=0x7f5ba40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 shutdown_connections 2024-03-06T07:28:26.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b9806b4d0 0x7f5b9806d990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bb415dfe0 0x7f5bb414c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bb40d29c0 0x7f5bb40b8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 --2- 172.21.15.72:0/1182164444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5bb40cf830 0x7f5bb40b4c60 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 >> 172.21.15.72:0/1182164444 conn(0x7f5bb40d3cd0 msgr2=0x7f5bb406ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 shutdown_connections 2024-03-06T07:28:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.713+0000 7f5bbbd79640 1 -- 172.21.15.72:0/1182164444 wait complete. 2024-03-06T07:28:26.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.824+0000 7fe22eb1e640 1 Processor -- start 2024-03-06T07:28:26.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 -- start start 2024-03-06T07:28:26.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2281515e0 0x7fe2280cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe22815b430 0x7fe228071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2280cd7a0 0x7fe2280cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22815b8b0 con 0x7fe2280cd7a0 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe228070a70 con 0x7fe22815b430 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22eb1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe228070d50 con 0x7fe2281515e0 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.825+0000 7fe22d0bb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2280cd7a0 0x7fe2280cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2280cd7a0 0x7fe2280cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45690/0 (socket says 172.21.15.72:45690) 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 -- 172.21.15.72:0/694878012 learned_addr learned my addr 172.21.15.72:0/694878012 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22c8ba640 1 --2- 172.21.15.72:0/694878012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2281515e0 0x7fe2280cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe227fff640 1 --2- 172.21.15.72:0/694878012 >> v2:172.21.15.146:3300/0 conn(0x7fe22815b430 0x7fe228071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 -- 172.21.15.72:0/694878012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2281515e0 msgr2=0x7fe2280cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 --2- 172.21.15.72:0/694878012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2281515e0 0x7fe2280cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 -- 172.21.15.72:0/694878012 >> v2:172.21.15.146:3300/0 conn(0x7fe22815b430 msgr2=0x7fe228071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 --2- 172.21.15.72:0/694878012 >> v2:172.21.15.146:3300/0 conn(0x7fe22815b430 0x7fe228071830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe22d0bb640 1 -- 172.21.15.72:0/694878012 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe228071030 con 0x7fe2280cd7a0 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.826+0000 7fe227fff640 1 --2- 172.21.15.72:0/694878012 >> v2:172.21.15.146:3300/0 conn(0x7fe22815b430 0x7fe228071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:26.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22d0bb640 1 --2- 172.21.15.72:0/694878012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2280cd7a0 0x7fe2280cdbd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe228073220 tx=0x7fe22002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c511104cb249363e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe2277fe640 1 -- 172.21.15.72:0/694878012 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe22002ea80 con 0x7fe2280cd7a0 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe2277fe640 1 -- 172.21.15.72:0/694878012 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe22002ec20 con 0x7fe2280cd7a0 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe2277fe640 1 -- 172.21.15.72:0/694878012 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220009600 con 0x7fe2280cd7a0 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 -- 172.21.15.72:0/694878012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2280cd7a0 msgr2=0x7fe2280cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/694878012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2280cd7a0 0x7fe2280cdbd0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fe228073220 tx=0x7fe22002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 -- 172.21.15.72:0/694878012 shutdown_connections 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/694878012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2280cd7a0 0x7fe2280cdbd0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/694878012 >> v2:172.21.15.146:3300/0 conn(0x7fe22815b430 0x7fe228071830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/694878012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2281515e0 0x7fe2280cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 -- 172.21.15.72:0/694878012 >> 172.21.15.72:0/694878012 conn(0x7fe2280cf8a0 msgr2=0x7fe2280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 -- 172.21.15.72:0/694878012 shutdown_connections 2024-03-06T07:28:26.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.827+0000 7fe22eb1e640 1 -- 172.21.15.72:0/694878012 wait complete. 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.828+0000 7fe22eb1e640 1 Processor -- start 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.828+0000 7fe22eb1e640 1 -- start start 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22eb1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2280cd7a0 0x7fe22813ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22eb1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 0x7fe228072360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22eb1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe22815b430 0x7fe2280c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22eb1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe228071d70 con 0x7fe2281515e0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22eb1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2280ce110 con 0x7fe22815b430 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22eb1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe228071ef0 con 0x7fe2280cd7a0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe227fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 0x7fe228072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe227fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 0x7fe228072360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45700/0 (socket says 172.21.15.72:45700) 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe227fff640 1 -- 172.21.15.72:0/3885709176 learned_addr learned my addr 172.21.15.72:0/3885709176 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.829+0000 7fe22c8ba640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2280cd7a0 0x7fe22813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe22d0bb640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe22815b430 0x7fe2280c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe227fff640 1 -- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2280cd7a0 msgr2=0x7fe22813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe227fff640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2280cd7a0 0x7fe22813ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe227fff640 1 -- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe22815b430 msgr2=0x7fe2280c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe227fff640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe22815b430 0x7fe2280c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe227fff640 1 -- 172.21.15.72:0/3885709176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2280c9c80 con 0x7fe2281515e0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe227fff640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 0x7fe228072360 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c820 tx=0x7fe21800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218015070 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe218007a30 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe218007e00 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22813e840 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.830+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2280ca1e0 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.831+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe228068840 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.832+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe218018070 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.832+0000 7fe2257fa640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe20806b4d0 0x7fe20806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.832+0000 7fe22c8ba640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe20806b4d0 0x7fe20806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.832+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fe21808a1d0 con 0x7fe2281515e0 2024-03-06T07:28:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.833+0000 7fe22c8ba640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe20806b4d0 0x7fe20806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe210003610 tx=0x7fe210008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:26.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:26.836+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2280c9c80 con 0x7fe2281515e0 2024-03-06T07:28:27.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.024+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2280682b0 con 0x7fe2281515e0 2024-03-06T07:28:27.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.025+0000 7fe2257fa640 1 -- 172.21.15.72:0/3885709176 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+17182 (secure 0 0 0) 0x7fe2280682b0 con 0x7fe2281515e0 2024-03-06T07:28:27.027 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:27.027 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":116,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:27:32.298669+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:27:00.730111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-06T07:27:01.483409+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.1a","mappings":[{"from":0,"to":2}]},{"pgid":"17.25","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6835/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6834/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6837/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.72:6836/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.146:6835/118157918":"2024-03-07T07:27:25.085724+0000","172.21.15.146:6833/118157918":"2024-03-07T07:27:25.085724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe20806b4d0 msgr2=0x7fe20806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe20806b4d0 0x7fe20806d990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe210003610 tx=0x7fe210008040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 msgr2=0x7fe228072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 0x7fe228072360 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fe21800c820 tx=0x7fe21800cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 shutdown_connections 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe20806b4d0 0x7fe20806d990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe22815b430 0x7fe2280c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2281515e0 0x7fe228072360 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 --2- 172.21.15.72:0/3885709176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2280cd7a0 0x7fe22813ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 >> 172.21.15.72:0/3885709176 conn(0x7fe2280cf8a0 msgr2=0x7fe22814dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 shutdown_connections 2024-03-06T07:28:27.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.027+0000 7fe22eb1e640 1 -- 172.21.15.72:0/3885709176 wait complete. 2024-03-06T07:28:27.041 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:28:27.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.141+0000 7fc4a6fe2640 1 Processor -- start 2024-03-06T07:28:27.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.141+0000 7fc4a6fe2640 1 -- start start 2024-03-06T07:28:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a6fe2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0162970 0x7fc4a00d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a6fe2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc4a0161cf0 0x7fc4a0162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a6fe2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a00d22c0 con 0x7fc4a00b4460 2024-03-06T07:28:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a6fe2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a0161b00 con 0x7fc4a0161cf0 2024-03-06T07:28:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a6fe2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00cecc0 con 0x7fc4a0162970 2024-03-06T07:28:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a557f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc4a0161cf0 0x7fc4a0162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a4d7e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc4a00b4460 0x7fc4a00b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55422/0 (socket says 172.21.15.72:55422) 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a4d7e640 1 -- 172.21.15.72:0/653521241 learned_addr learned my addr 172.21.15.72:0/653521241 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc49ffff640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0162970 0x7fc4a00d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a557f640 1 -- 172.21.15.72:0/653521241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0162970 msgr2=0x7fc4a00d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a557f640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0162970 0x7fc4a00d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a557f640 1 -- 172.21.15.72:0/653521241 >> v1:172.21.15.72:6789/0 conn(0x7fc4a00b4460 legacy=0x7fc4a00b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc4a557f640 1 -- 172.21.15.72:0/653521241 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a00d25a0 con 0x7fc4a0161cf0 2024-03-06T07:28:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.142+0000 7fc49ffff640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0162970 0x7fc4a00d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.143+0000 7fc4a557f640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0161cf0 0x7fc4a0162180 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d90 tx=0x7fc49802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54eebf6738f6a823 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.143+0000 7fc49f7fe640 1 -- 172.21.15.72:0/653521241 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498039070 con 0x7fc4a0161cf0 2024-03-06T07:28:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.143+0000 7fc49f7fe640 1 -- 172.21.15.72:0/653521241 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49802cc50 con 0x7fc4a0161cf0 2024-03-06T07:28:27.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.143+0000 7fc49f7fe640 1 -- 172.21.15.72:0/653521241 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498034440 con 0x7fc4a0161cf0 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/653521241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0161cf0 msgr2=0x7fc4a0162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0161cf0 0x7fc4a0162180 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d90 tx=0x7fc49802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/653521241 shutdown_connections 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0161cf0 0x7fc4a0162180 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/653521241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0162970 0x7fc4a00d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/653521241 >> 172.21.15.72:0/653521241 conn(0x7fc4a00d3ce0 msgr2=0x7fc4a00d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/653521241 shutdown_connections 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.144+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/653521241 wait complete. 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.145+0000 7fc4a6fe2640 1 Processor -- start 2024-03-06T07:28:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.146+0000 7fc4a6fe2640 1 -- start start 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a6fe2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a00b4460 0x7fc4a00c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a6fe2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0161cf0 0x7fc4a01559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a6fe2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 0x7fc4a0153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a6fe2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00ce950 con 0x7fc4a0162970 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a6fe2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00d2b60 con 0x7fc4a00b4460 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a6fe2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a00b4840 con 0x7fc4a0161cf0 2024-03-06T07:28:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 0x7fc4a0153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 0x7fc4a0153e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45710/0 (socket says 172.21.15.72:45710) 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 -- 172.21.15.72:0/619824882 learned_addr learned my addr 172.21.15.72:0/619824882 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc49ffff640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0161cf0 0x7fc4a01559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a4d7e640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a00b4460 0x7fc4a00c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 -- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0161cf0 msgr2=0x7fc4a01559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0161cf0 0x7fc4a01559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 -- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a00b4460 msgr2=0x7fc4a00c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a00b4460 0x7fc4a00c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.147+0000 7fc4a557f640 1 -- 172.21.15.72:0/619824882 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a0154920 con 0x7fc4a0162970 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.148+0000 7fc4a557f640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 0x7fc4a0153e50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d90 tx=0x7fc498004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.148+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc498039050 con 0x7fc4a0162970 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.148+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc498045070 con 0x7fc4a0162970 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.148+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a0154c00 con 0x7fc4a0162970 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.148+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49803e610 con 0x7fc4a0162970 2024-03-06T07:28:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.148+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01551c0 con 0x7fc4a0162970 2024-03-06T07:28:27.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.149+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a004f330 con 0x7fc4a0162970 2024-03-06T07:28:27.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.149+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc49803e7b0 con 0x7fc4a0162970 2024-03-06T07:28:27.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.150+0000 7fc49d7fa640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc48406b4d0 0x7fc48406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.150+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fc4980ace60 con 0x7fc4a0162970 2024-03-06T07:28:27.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.150+0000 7fc4a4d7e640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc48406b4d0 0x7fc48406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.150+0000 7fc4a4d7e640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc48406b4d0 0x7fc48406d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc494000f60 tx=0x7fc494009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.154+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc498032e20 con 0x7fc4a0162970 2024-03-06T07:28:27.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.450+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc4a006b180 con 0x7fc4a0162970 2024-03-06T07:28:27.452 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:28:27.450+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:28:27.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.451+0000 7fc49d7fa640 1 -- 172.21.15.72:0/619824882 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc49807bcc0 con 0x7fc4a0162970 2024-03-06T07:28:27.453 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:28:27.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.454+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc48406b4d0 msgr2=0x7fc48406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc48406b4d0 0x7fc48406d990 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc494000f60 tx=0x7fc494009040 comp rx=0 tx=0).stop 2024-03-06T07:28:27.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 msgr2=0x7fc4a0153e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 0x7fc4a0153e50 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fc498009d90 tx=0x7fc498004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 shutdown_connections 2024-03-06T07:28:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc48406b4d0 0x7fc48406d990 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4a0162970 0x7fc4a0153e50 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a0161cf0 0x7fc4a01559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 --2- 172.21.15.72:0/619824882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a00b4460 0x7fc4a00c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 >> 172.21.15.72:0/619824882 conn(0x7fc4a00d3ce0 msgr2=0x7fc4a00d40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 shutdown_connections 2024-03-06T07:28:27.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.455+0000 7fc4a6fe2640 1 -- 172.21.15.72:0/619824882 wait complete. 2024-03-06T07:28:27.460 INFO:tasks.ceph.mds.c.smithi072.stderr: -8786> 2024-03-06T07:27:30.283+0000 7f2f0ad0a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:27.469 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:28:27.494 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.570+0000 7fa2b4f81640 1 Processor -- start 2024-03-06T07:28:27.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.570+0000 7fa2b4f81640 1 -- start start 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2b4f81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 0x7fa2b015ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2b4f81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b006bd30 con 0x7fa2b00b2920 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2b4f81640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b00726f0 con 0x7fa2b015e120 2024-03-06T07:28:27.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2b4f81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00729d0 con 0x7fa2b00731d0 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2a7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa2b00b2920 0x7fa2b00b4ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55440/0 (socket says 172.21.15.72:55440) 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2a7fff640 1 -- 172.21.15.72:0/1005924945 learned_addr learned my addr 172.21.15.72:0/1005924945 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2aed9d640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 0x7fa2b015ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274533699 0 0) 0x7fa2b006bd30 con 0x7fa2b00b2920 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c005730 con 0x7fa2b00b2920 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201544943 0 0) 0x7fa29c005730 con 0x7fa2b00b2920 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 msgr2=0x7fa2b015ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 0x7fa2b015ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 >> v1:172.21.15.146:6789/0 conn(0x7fa2b015e120 legacy=0x7fa2b015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b0072cb0 con 0x7fa2b00b2920 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fa298001670 con 0x7fa2b00b2920 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 >> v1:172.21.15.72:6789/0 conn(0x7fa2b00b2920 legacy=0x7fa2b00b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa29c005bf0 0x7fa29c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 0x7fa29c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa29c006f50 0x7fa29c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b00729d0 con 0x7fa29c005bf0 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c00dc80 con 0x7fa29c006f50 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c00de80 con 0x7fa2b00731d0 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa29c005bf0 0x7fa29c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.571+0000 7fa2a7fff640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 0x7fa29c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2aed9d640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa29c006f50 0x7fa29c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2ae59c640 1 -- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 msgr2=0x7fa29c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2b00731d0 0x7fa29c006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2ae59c640 1 -- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa29c006f50 msgr2=0x7fa29c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa29c006f50 0x7fa29c0073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2ae59c640 1 -- 172.21.15.72:0/1005924945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c005730 con 0x7fa29c005bf0 2024-03-06T07:28:27.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa29c005bf0 0x7fa29c006010 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0001720 tx=0x7fa29c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d715c8c1d20abb31 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0001370 con 0x7fa29c005bf0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a00052d0 con 0x7fa29c005bf0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.572+0000 7fa2add9b640 1 -- 172.21.15.72:0/1005924945 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0003830 con 0x7fa29c005bf0 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2b4f81640 1 -- 172.21.15.72:0/1005924945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa29c005bf0 msgr2=0x7fa29c006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/1005924945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa29c005bf0 0x7fa29c006010 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0001720 tx=0x7fa29c007a40 comp rx=0 tx=0).stop 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2aed9d640 1 -- 172.21.15.72:0/1005924945 reap_dead start 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2b4f81640 1 -- 172.21.15.72:0/1005924945 shutdown_connections 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2b4f81640 1 -- 172.21.15.72:0/1005924945 >> 172.21.15.72:0/1005924945 conn(0x7fa2b00b1d70 msgr2=0x7fa2b00b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2b4f81640 1 -- 172.21.15.72:0/1005924945 shutdown_connections 2024-03-06T07:28:27.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.573+0000 7fa2b4f81640 1 -- 172.21.15.72:0/1005924945 wait complete. 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.574+0000 7fa2b4f81640 1 Processor -- start 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.574+0000 7fa2b4f81640 1 -- start start 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2b4f81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa29c005bf0 0x7fa2b00b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2b4f81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2b00cba70 0x7fa2b00cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2b4f81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 0x7fa2b0159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2b4f81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b0158520 con 0x7fa2b0158ed0 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2b4f81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b01583a0 con 0x7fa2b00cba70 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2b4f81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b015ec10 con 0x7fa29c005bf0 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 0x7fa2b0159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 0x7fa2b0159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45734/0 (socket says 172.21.15.72:45734) 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 -- 172.21.15.72:0/4088658792 learned_addr learned my addr 172.21.15.72:0/4088658792 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa29c005bf0 0x7fa2b00b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2a7fff640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2b00cba70 0x7fa2b00cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 -- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa29c005bf0 msgr2=0x7fa2b00b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa29c005bf0 0x7fa2b00b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 -- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2b00cba70 msgr2=0x7fa2b00cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2b00cba70 0x7fa2b00cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.575+0000 7fa2aed9d640 1 -- 172.21.15.72:0/4088658792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b00cc4f0 con 0x7fa2b0158ed0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.576+0000 7fa2aed9d640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 0x7fa2b0159350 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00707a0 tx=0x7fa2a80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.576+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8011e10 con 0x7fa2b0158ed0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.576+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a801b070 con 0x7fa2b0158ed0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.576+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b00cc7d0 con 0x7fa2b0158ed0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.576+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8003990 con 0x7fa2b0158ed0 2024-03-06T07:28:27.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.576+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b0155d40 con 0x7fa2b0158ed0 2024-03-06T07:28:27.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.577+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b009e3d0 con 0x7fa2b0158ed0 2024-03-06T07:28:27.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.577+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa2a8003b50 con 0x7fa2b0158ed0 2024-03-06T07:28:27.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.577+0000 7fa2a6ffd640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa29006b500 0x7fa29006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:27.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.578+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fa2a808aab0 con 0x7fa2b0158ed0 2024-03-06T07:28:27.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.578+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa29006b500 0x7fa29006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:27.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.578+0000 7fa2ae59c640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa29006b500 0x7fa29006d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0033000 tx=0x7fa2a0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:27.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.582+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa2a8059830 con 0x7fa2b0158ed0 2024-03-06T07:28:27.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:27.788+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa2b00cc9a0 con 0x7fa2b0158ed0 2024-03-06T07:28:28.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.458+0000 7fa2a6ffd640 1 -- 172.21.15.72:0/4088658792 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa2a8059130 con 0x7fa2b0158ed0 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa29006b500 msgr2=0x7fa29006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa29006b500 0x7fa29006d9c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0033000 tx=0x7fa2a0009030 comp rx=0 tx=0).stop 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 msgr2=0x7fa2b0159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 0x7fa2b0159350 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00707a0 tx=0x7fa2a80115a0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 shutdown_connections 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa29006b500 0x7fa29006d9c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2b0158ed0 0x7fa2b0159350 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2b00cba70 0x7fa2b00cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 --2- 172.21.15.72:0/4088658792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa29c005bf0 0x7fa2b00b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.460+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 >> 172.21.15.72:0/4088658792 conn(0x7fa2b00b1d70 msgr2=0x7fa2b006e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.461+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 shutdown_connections 2024-03-06T07:28:28.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.461+0000 7fa2b4f81640 1 -- 172.21.15.72:0/4088658792 wait complete. 2024-03-06T07:28:28.474 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:28:28.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.617+0000 7fceb9a65640 1 Processor -- start 2024-03-06T07:28:28.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.617+0000 7fceb9a65640 1 -- start start 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb9a65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fceb415a490 0x7fceb406df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb9a65640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fceb40b3060 0x7fceb415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb9a65640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb406e4d0 con 0x7fceb40b3fd0 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb9a65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb40b4490 con 0x7fceb415a490 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb9a65640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb40cf4a0 con 0x7fceb40b3060 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fceb415a490 0x7fceb406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fceb415a490 0x7fceb406df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45746/0 (socket says 172.21.15.72:45746) 2024-03-06T07:28:28.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 -- 172.21.15.72:0/1963630447 learned_addr learned my addr 172.21.15.72:0/1963630447 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 -- 172.21.15.72:0/1963630447 >> v2:172.21.15.146:3300/0 conn(0x7fceb40b3060 msgr2=0x7fceb415dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 --2- 172.21.15.72:0/1963630447 >> v2:172.21.15.146:3300/0 conn(0x7fceb40b3060 0x7fceb415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 -- 172.21.15.72:0/1963630447 >> v1:172.21.15.146:6789/0 conn(0x7fceb40b3fd0 legacy=0x7fceb40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.618+0000 7fceb37fe640 1 -- 172.21.15.72:0/1963630447 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb40b4750 con 0x7fceb415a490 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.619+0000 7fceb37fe640 1 --2- 172.21.15.72:0/1963630447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb406df90 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fceb4070840 tx=0x7fcea402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8cfb891b7a7d59 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.619+0000 7fceb27fc640 1 -- 172.21.15.72:0/1963630447 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4035020 con 0x7fceb415a490 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.619+0000 7fceb27fc640 1 -- 172.21.15.72:0/1963630447 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4031870 con 0x7fceb415a490 2024-03-06T07:28:28.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.619+0000 7fceb27fc640 1 -- 172.21.15.72:0/1963630447 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4031c40 con 0x7fceb415a490 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 -- 172.21.15.72:0/1963630447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 msgr2=0x7fceb406df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 --2- 172.21.15.72:0/1963630447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb406df90 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fceb4070840 tx=0x7fcea402b910 comp rx=0 tx=0).stop 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 -- 172.21.15.72:0/1963630447 shutdown_connections 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 --2- 172.21.15.72:0/1963630447 >> v2:172.21.15.146:3300/0 conn(0x7fceb40b3060 0x7fceb415dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 --2- 172.21.15.72:0/1963630447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb406df90 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 -- 172.21.15.72:0/1963630447 >> 172.21.15.72:0/1963630447 conn(0x7fceb40b1650 msgr2=0x7fceb40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:28.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 -- 172.21.15.72:0/1963630447 shutdown_connections 2024-03-06T07:28:28.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.620+0000 7fceb9a65640 1 -- 172.21.15.72:0/1963630447 wait complete. 2024-03-06T07:28:28.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.621+0000 7fceb9a65640 1 Processor -- start 2024-03-06T07:28:28.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.621+0000 7fceb9a65640 1 -- start start 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb9a65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fceb40b3060 0x7fceb41f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb9a65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fceb40b3fd0 0x7fceb415ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb9a65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb415cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb9a65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb40cf0e0 con 0x7fceb415a490 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb9a65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb40b3490 con 0x7fceb40b3060 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb9a65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb40cf260 con 0x7fceb40b3fd0 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb415cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb415cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45750/0 (socket says 172.21.15.72:45750) 2024-03-06T07:28:28.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 -- 172.21.15.72:0/2193351535 learned_addr learned my addr 172.21.15.72:0/2193351535 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb37fe640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fceb40b3060 0x7fceb41f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb2ffd640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fceb40b3fd0 0x7fceb415ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 -- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fceb40b3fd0 msgr2=0x7fceb415ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fceb40b3fd0 0x7fceb415ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 -- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fceb40b3060 msgr2=0x7fceb41f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fceb40b3060 0x7fceb41f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 -- 172.21.15.72:0/2193351535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb41f5690 con 0x7fceb415a490 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb37fe640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fceb40b3060 0x7fceb41f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fceb3fff640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb415cfc0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fceac00b9a0 tx=0x7fceac00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.622+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac019070 con 0x7fceb415a490 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.623+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceac00cbc0 con 0x7fceb415a490 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.623+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb41f5970 con 0x7fceb415a490 2024-03-06T07:28:28.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.623+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceac014440 con 0x7fceb415a490 2024-03-06T07:28:28.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.623+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb41f5f30 con 0x7fceb415a490 2024-03-06T07:28:28.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.624+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb41f6130 con 0x7fceb415a490 2024-03-06T07:28:28.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.624+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fceac01c070 con 0x7fceb415a490 2024-03-06T07:28:28.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.625+0000 7fce9bfff640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcea006b4d0 0x7fcea006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:28.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.625+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6111+0+0 (secure 0 0 0) 0x7fceac08a110 con 0x7fceb415a490 2024-03-06T07:28:28.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.625+0000 7fceb37fe640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcea006b4d0 0x7fcea006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:28.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.625+0000 7fceb37fe640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcea006b4d0 0x7fcea006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcea4036000 tx=0x7fcea403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:28.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.629+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fceb41f6130 con 0x7fceb415a490 2024-03-06T07:28:28.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:28.802+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fceb40cbbd0 con 0x7fceb415a490 2024-03-06T07:28:29.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.460+0000 7fce9bfff640 1 -- 172.21.15.72:0/2193351535 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v118) v1 ==== 181+0+0 (secure 0 0 0) 0x7fceac058e00 con 0x7fceb415a490 2024-03-06T07:28:29.462 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:28:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.463+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcea006b4d0 msgr2=0x7fcea006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcea006b4d0 0x7fcea006d990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcea4036000 tx=0x7fcea403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 msgr2=0x7fceb415cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb415cfc0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fceac00b9a0 tx=0x7fceac00be60 comp rx=0 tx=0).stop 2024-03-06T07:28:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 shutdown_connections 2024-03-06T07:28:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcea006b4d0 0x7fcea006d990 secure :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcea4036000 tx=0x7fcea403c040 comp rx=0 tx=0).stop 2024-03-06T07:28:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fceb415a490 0x7fceb415cfc0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fceb40b3fd0 0x7fceb415ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 --2- 172.21.15.72:0/2193351535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fceb40b3060 0x7fceb41f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.464+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 >> 172.21.15.72:0/2193351535 conn(0x7fceb40b1650 msgr2=0x7fceb40cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.465+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 shutdown_connections 2024-03-06T07:28:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.465+0000 7fceb9a65640 1 -- 172.21.15.72:0/2193351535 wait complete. 2024-03-06T07:28:29.482 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.586+0000 7f0387a78640 1 Processor -- start 2024-03-06T07:28:29.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.586+0000 7f0387a78640 1 -- start start 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0387a78640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f03800708b0 0x7f0380070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0387a78640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038015afe0 con 0x7f038015ac00 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0387a78640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03800cd000 con 0x7f03800ce0a0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0387a78640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0380071250 con 0x7f03800708b0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0385013640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f03800708b0 0x7f0380070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0385814640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f038015ac00 0x7f038006c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55446/0 (socket says 172.21.15.72:55446) 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0385814640 1 -- 172.21.15.72:0/3873687431 learned_addr learned my addr 172.21.15.72:0/3873687431 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0385013640 1 -- 172.21.15.72:0/3873687431 >> v1:172.21.15.146:6789/0 conn(0x7f03800ce0a0 legacy=0x7f03800ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0385013640 1 -- 172.21.15.72:0/3873687431 >> v1:172.21.15.72:6789/0 conn(0x7f038015ac00 legacy=0x7f038006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.587+0000 7f0385013640 1 -- 172.21.15.72:0/3873687431 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03800cd2e0 con 0x7f03800708b0 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.588+0000 7f0385013640 1 --2- 172.21.15.72:0/3873687431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03800708b0 0x7f0380070d10 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f03700097a0 tx=0x7f037002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3b0a29b544dc23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.588+0000 7f0384812640 1 -- 172.21.15.72:0/3873687431 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03700315a0 con 0x7f03800708b0 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.588+0000 7f0384812640 1 -- 172.21.15.72:0/3873687431 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0370037070 con 0x7f03800708b0 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.588+0000 7f0384812640 1 -- 172.21.15.72:0/3873687431 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0370031960 con 0x7f03800708b0 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 -- 172.21.15.72:0/3873687431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03800708b0 msgr2=0x7f0380070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 --2- 172.21.15.72:0/3873687431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03800708b0 0x7f0380070d10 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f03700097a0 tx=0x7f037002b840 comp rx=0 tx=0).stop 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 -- 172.21.15.72:0/3873687431 shutdown_connections 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 --2- 172.21.15.72:0/3873687431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03800708b0 0x7f0380070d10 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 -- 172.21.15.72:0/3873687431 >> 172.21.15.72:0/3873687431 conn(0x7f03800b1650 msgr2=0x7f03800b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 -- 172.21.15.72:0/3873687431 shutdown_connections 2024-03-06T07:28:29.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.589+0000 7f0387a78640 1 -- 172.21.15.72:0/3873687431 wait complete. 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.590+0000 7f0387a78640 1 Processor -- start 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.590+0000 7f0387a78640 1 -- start start 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.590+0000 7f0387a78640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03800708b0 0x7f03800aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.590+0000 7f0387a78640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03800ce0a0 0x7f0380151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.590+0000 7f0387a78640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 0x7f03801f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0387a78640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03800cd8a0 con 0x7f038015ac00 2024-03-06T07:28:29.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0387a78640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03800cdc10 con 0x7f03800708b0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0387a78640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03800cf5d0 con 0x7f03800ce0a0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 0x7f03801f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 0x7f03801f4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45766/0 (socket says 172.21.15.72:45766) 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 -- 172.21.15.72:0/1825304441 learned_addr learned my addr 172.21.15.72:0/1825304441 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0385814640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03800708b0 0x7f03800aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0385013640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03800ce0a0 0x7f0380151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 -- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03800ce0a0 msgr2=0x7f0380151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03800ce0a0 0x7f0380151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 -- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03800708b0 msgr2=0x7f03800aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03800708b0 0x7f03800aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 -- 172.21.15.72:0/1825304441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0380157990 con 0x7f038015ac00 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0386015640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 0x7f03801f4c00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f037800c780 tx=0x7f037800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0378017070 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0378003da0 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03801f5750 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03780052b0 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.591+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0380157c50 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.592+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038004f330 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.593+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f037801a070 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.593+0000 7f03767fc640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f036406b4d0 0x7f036406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.593+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f03780896e0 con 0x7f038015ac00 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.593+0000 7f0385814640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f036406b4d0 0x7f036406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:29.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.594+0000 7f0385814640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f036406b4d0 0x7f036406d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f037c007fd0 tx=0x7f037c007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:29.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.597+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0378057fc0 con 0x7f038015ac00 2024-03-06T07:28:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:29.768+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0380072550 con 0x7f038015ac00 2024-03-06T07:28:30.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.467+0000 7f03767fc640 1 -- 172.21.15.72:0/1825304441 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v119) v1 ==== 169+0+0 (secure 0 0 0) 0x7f037805c060 con 0x7f038015ac00 2024-03-06T07:28:30.469 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f036406b4d0 msgr2=0x7f036406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f036406b4d0 0x7f036406d990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f037c007fd0 tx=0x7f037c007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 msgr2=0x7f03801f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 0x7f03801f4c00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f037800c780 tx=0x7f037800cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 shutdown_connections 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f036406b4d0 0x7f036406d990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f038015ac00 0x7f03801f4c00 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03800ce0a0 0x7f0380151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 --2- 172.21.15.72:0/1825304441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03800708b0 0x7f03800aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.470+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 >> 172.21.15.72:0/1825304441 conn(0x7f03800b1650 msgr2=0x7f03800b09e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.471+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 shutdown_connections 2024-03-06T07:28:30.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.471+0000 7f0387a78640 1 -- 172.21.15.72:0/1825304441 wait complete. 2024-03-06T07:28:30.485 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.595+0000 7f480e11e640 1 Processor -- start 2024-03-06T07:28:30.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.595+0000 7f480e11e640 1 -- start start 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f480e11e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f48080b4970 0x7f480806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f480e11e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f480806d5c0 0x7f48080cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f480e11e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f480815ff90 con 0x7f480815bb40 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f480e11e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48080b4dd0 con 0x7f480806d5c0 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f480e11e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f480806c910 con 0x7f48080b4970 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f480806d5c0 0x7f48080cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f480806d5c0 0x7f48080cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45782/0 (socket says 172.21.15.72:45782) 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 -- 172.21.15.72:0/3384660659 learned_addr learned my addr 172.21.15.72:0/3384660659 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4806ffd640 1 --2- 172.21.15.72:0/3384660659 >> v2:172.21.15.146:3300/0 conn(0x7f48080b4970 0x7f480806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 -- 172.21.15.72:0/3384660659 >> v2:172.21.15.146:3300/0 conn(0x7f48080b4970 msgr2=0x7f480806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 --2- 172.21.15.72:0/3384660659 >> v2:172.21.15.146:3300/0 conn(0x7f48080b4970 0x7f480806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 -- 172.21.15.72:0/3384660659 >> v1:172.21.15.146:6789/0 conn(0x7f480815bb40 legacy=0x7f480815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.596+0000 7f4807fff640 1 -- 172.21.15.72:0/3384660659 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808160270 con 0x7f480806d5c0 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.597+0000 7f4807fff640 1 --2- 172.21.15.72:0/3384660659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480806d5c0 0x7f48080cfa90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f480806bfe0 tx=0x7f480002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c7eb24df1dfd034 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.597+0000 7f48067fc640 1 -- 172.21.15.72:0/3384660659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480002ca80 con 0x7f480806d5c0 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.597+0000 7f48067fc640 1 -- 172.21.15.72:0/3384660659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480002cc20 con 0x7f480806d5c0 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.597+0000 7f48067fc640 1 -- 172.21.15.72:0/3384660659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000343f0 con 0x7f480806d5c0 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 -- 172.21.15.72:0/3384660659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480806d5c0 msgr2=0x7f48080cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 --2- 172.21.15.72:0/3384660659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480806d5c0 0x7f48080cfa90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f480806bfe0 tx=0x7f480002bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 -- 172.21.15.72:0/3384660659 shutdown_connections 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 --2- 172.21.15.72:0/3384660659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480806d5c0 0x7f48080cfa90 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 --2- 172.21.15.72:0/3384660659 >> v2:172.21.15.146:3300/0 conn(0x7f48080b4970 0x7f480806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 -- 172.21.15.72:0/3384660659 >> 172.21.15.72:0/3384660659 conn(0x7f480806b190 msgr2=0x7f480806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 -- 172.21.15.72:0/3384660659 shutdown_connections 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.598+0000 7f480e11e640 1 -- 172.21.15.72:0/3384660659 wait complete. 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.599+0000 7f480e11e640 1 Processor -- start 2024-03-06T07:28:30.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.599+0000 7f480e11e640 1 -- start start 2024-03-06T07:28:30.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f480806d5c0 0x7f48080b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48080b4970 0x7f480814ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 0x7f48080af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480815bf20 con 0x7f480815bb40 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480806cb60 con 0x7f480806d5c0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480815c0a0 con 0x7f48080b4970 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 0x7f48080af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 0x7f48080af4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45788/0 (socket says 172.21.15.72:45788) 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 -- 172.21.15.72:0/3037273760 learned_addr learned my addr 172.21.15.72:0/3037273760 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f48077fe640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f480806d5c0 0x7f48080b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4806ffd640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48080b4970 0x7f480814ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 -- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48080b4970 msgr2=0x7f480814ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48080b4970 0x7f480814ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 -- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f480806d5c0 msgr2=0x7f48080b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f480806d5c0 0x7f48080b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 -- 172.21.15.72:0/3037273760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808157110 con 0x7f480815bb40 2024-03-06T07:28:30.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f4807fff640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 0x7f48080af4e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f4800009f40 tx=0x7f4800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800043020 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480003e450 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48081573f0 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4800009510 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.600+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480814b7e0 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.602+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48080732b0 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.602+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4800046020 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.602+0000 7f47effff640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f47f406b4d0 0x7f47f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.602+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f48000ac7d0 con 0x7f480815bb40 2024-03-06T07:28:30.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.602+0000 7f48077fe640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f47f406b4d0 0x7f47f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.603+0000 7f48077fe640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f47f406b4d0 0x7f47f406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f47f8006fd0 tx=0x7f47f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.607+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f480004d030 con 0x7f480815bb40 2024-03-06T07:28:30.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.780+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4808068ff0 con 0x7f480815bb40 2024-03-06T07:28:30.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.783+0000 7f47effff640 1 -- 172.21.15.72:0/3037273760 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f480007b8d0 con 0x7f480815bb40 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f47f406b4d0 msgr2=0x7f47f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f47f406b4d0 0x7f47f406d990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f47f8006fd0 tx=0x7f47f8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 msgr2=0x7f48080af4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 0x7f48080af4e0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f4800009f40 tx=0x7f4800004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 shutdown_connections 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f47f406b4d0 0x7f47f406d990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f480815bb40 0x7f48080af4e0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f48080b4970 0x7f480814ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 --2- 172.21.15.72:0/3037273760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f480806d5c0 0x7f48080b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 >> 172.21.15.72:0/3037273760 conn(0x7f480806b190 msgr2=0x7f48081597d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 shutdown_connections 2024-03-06T07:28:30.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.785+0000 7f480e11e640 1 -- 172.21.15.72:0/3037273760 wait complete. 2024-03-06T07:28:30.798 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-03-06T07:28:30.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.896+0000 7f94d68d4640 1 Processor -- start 2024-03-06T07:28:30.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.896+0000 7f94d68d4640 1 -- start start 2024-03-06T07:28:30.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d68d4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f94d01424f0 0x7f94d00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d68d4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f94d00cd7a0 0x7f94d015cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d68d4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d00cc980 con 0x7f94d006d330 2024-03-06T07:28:30.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d68d4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00cd220 con 0x7f94d00cd7a0 2024-03-06T07:28:30.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d68d4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d00ccc60 con 0x7f94d01424f0 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94cf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f94d00cd7a0 0x7f94d015cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d4e71640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f94d006d330 0x7f94d0071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55460/0 (socket says 172.21.15.72:55460) 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94d4e71640 1 -- 172.21.15.72:0/389833213 learned_addr learned my addr 172.21.15.72:0/389833213 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94cffff640 1 --2- 172.21.15.72:0/389833213 >> v2:172.21.15.146:3300/0 conn(0x7f94d01424f0 0x7f94d00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94cf7fe640 1 -- 172.21.15.72:0/389833213 >> v2:172.21.15.146:3300/0 conn(0x7f94d01424f0 msgr2=0x7f94d00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94cf7fe640 1 --2- 172.21.15.72:0/389833213 >> v2:172.21.15.146:3300/0 conn(0x7f94d01424f0 0x7f94d00cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94cf7fe640 1 -- 172.21.15.72:0/389833213 >> v1:172.21.15.72:6789/0 conn(0x7f94d006d330 legacy=0x7f94d0071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94ceffd640 1 -- 172.21.15.72:0/389833213 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123217315 0 0) 0x7f94d00cc980 con 0x7f94d006d330 2024-03-06T07:28:30.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.897+0000 7f94cf7fe640 1 -- 172.21.15.72:0/389833213 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d00ccf40 con 0x7f94d00cd7a0 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.898+0000 7f94cf7fe640 1 --2- 172.21.15.72:0/389833213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d015cef0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f94d006c6d0 tx=0x7f94c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5499e35f3411af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.898+0000 7f94ceffd640 1 -- 172.21.15.72:0/389833213 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4035020 con 0x7f94d00cd7a0 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.898+0000 7f94ceffd640 1 -- 172.21.15.72:0/389833213 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4031830 con 0x7f94d00cd7a0 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.898+0000 7f94ceffd640 1 -- 172.21.15.72:0/389833213 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4031c00 con 0x7f94d00cd7a0 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.898+0000 7f94d68d4640 1 -- 172.21.15.72:0/389833213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 msgr2=0x7f94d015cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.898+0000 7f94d68d4640 1 --2- 172.21.15.72:0/389833213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d015cef0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f94d006c6d0 tx=0x7f94c402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.899+0000 7f94d68d4640 1 -- 172.21.15.72:0/389833213 shutdown_connections 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.899+0000 7f94d68d4640 1 --2- 172.21.15.72:0/389833213 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d015cef0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.899+0000 7f94d68d4640 1 --2- 172.21.15.72:0/389833213 >> v2:172.21.15.146:3300/0 conn(0x7f94d01424f0 0x7f94d00cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.899+0000 7f94d68d4640 1 -- 172.21.15.72:0/389833213 >> 172.21.15.72:0/389833213 conn(0x7f94d00cf8a0 msgr2=0x7f94d00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.899+0000 7f94d68d4640 1 -- 172.21.15.72:0/389833213 shutdown_connections 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.899+0000 7f94d68d4640 1 -- 172.21.15.72:0/389833213 wait complete. 2024-03-06T07:28:30.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.900+0000 7f94d68d4640 1 Processor -- start 2024-03-06T07:28:30.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.900+0000 7f94d68d4640 1 -- start start 2024-03-06T07:28:30.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d006d330 0x7f94d00ab400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d00b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d01424f0 0x7f94d00acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d006d920 con 0x7f94d00cd7a0 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d006d7a0 con 0x7f94d006d330 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d006daa0 con 0x7f94d01424f0 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d00b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d00b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45812/0 (socket says 172.21.15.72:45812) 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 -- 172.21.15.72:0/1574456428 learned_addr learned my addr 172.21.15.72:0/1574456428 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 -- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d01424f0 msgr2=0x7f94d00acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d01424f0 0x7f94d00acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 -- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d006d330 msgr2=0x7f94d00ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cffff640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d006d330 0x7f94d00ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d006d330 0x7f94d00ab400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 -- 172.21.15.72:0/1574456428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0157f10 con 0x7f94d00cd7a0 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94cf7fe640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d00b0880 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009890 tx=0x7f94c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4041070 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c403aa30 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d0159730 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4035040 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.901+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d0159cf0 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.902+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d01580e0 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.903+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f94c4038020 con 0x7f94d00cd7a0 2024-03-06T07:28:30.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.903+0000 7f94ccff9640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f94bc06b4d0 0x7f94bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:30.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.903+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f94c40ab9b0 con 0x7f94d00cd7a0 2024-03-06T07:28:30.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.903+0000 7f94cffff640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f94bc06b4d0 0x7f94bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:30.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.904+0000 7f94cffff640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f94bc06b4d0 0x7f94bc06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006fd0 tx=0x7f94c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:30.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:30.907+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f94c407aab0 con 0x7f94d00cd7a0 2024-03-06T07:28:31.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.148+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) v1 -- 0x7f94d00709f0 con 0x7f94d00cd7a0 2024-03-06T07:28:31.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.469+0000 7f94ccff9640 1 -- 172.21.15.72:0/1574456428 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v355) v1 ==== 142+0+0 (secure 0 0 0) 0x7f94c407a3b0 con 0x7f94d00cd7a0 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f94bc06b4d0 msgr2=0x7f94bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f94bc06b4d0 0x7f94bc06d990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f94c0006fd0 tx=0x7f94c0008040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 msgr2=0x7f94d00b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d00b0880 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009890 tx=0x7f94c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 shutdown_connections 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f94bc06b4d0 0x7f94bc06d990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d01424f0 0x7f94d00acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94d00cd7a0 0x7f94d00b0880 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 --2- 172.21.15.72:0/1574456428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d006d330 0x7f94d00ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 >> 172.21.15.72:0/1574456428 conn(0x7f94d00cf8a0 msgr2=0x7f94d015cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 shutdown_connections 2024-03-06T07:28:31.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.471+0000 7f94d68d4640 1 -- 172.21.15.72:0/1574456428 wait complete. 2024-03-06T07:28:31.484 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-03-06T07:28:31.485 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:28:31.485 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:28:31.485 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:31.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.583+0000 7fa47a9b5640 1 Processor -- start 2024-03-06T07:28:31.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.583+0000 7fa47a9b5640 1 -- start start 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa47a9b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 0x7fa47415e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa47a9b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47415ea30 con 0x7fa47406b220 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa47a9b5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47415ed10 con 0x7fa47406b600 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa47a9b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa474161b70 con 0x7fa4740ce920 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa473fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa47406b220 0x7fa474071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55474/0 (socket says 172.21.15.72:55474) 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa473fff640 1 -- 172.21.15.72:0/2771051755 learned_addr learned my addr 172.21.15.72:0/2771051755 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770433119 0 0) 0x7fa47415ea30 con 0x7fa47406b220 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c003520 con 0x7fa47406b220 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa4737fe640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 0x7fa47415e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.584+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699920281 0 0) 0x7fa45c003520 con 0x7fa47406b220 2024-03-06T07:28:31.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 msgr2=0x7fa47415e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 0x7fa47415e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 >> v1:172.21.15.146:6789/0 conn(0x7fa47406b600 legacy=0x7fa474162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47415eff0 con 0x7fa47406b220 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fa464001670 con 0x7fa47406b220 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 >> v1:172.21.15.72:6789/0 conn(0x7fa47406b220 legacy=0x7fa474071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 0x7fa45c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa45c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45c004d40 0x7fa45c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa474161b70 con 0x7fa45c004060 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c00bc00 con 0x7fa45c004d40 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c00be00 con 0x7fa4740ce920 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa4737fe640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa45c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa478f52640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45c004d40 0x7fa45c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa473fff640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 0x7fa45c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa4737fe640 1 -- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 msgr2=0x7fa45c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa4737fe640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4740ce920 0x7fa45c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa4737fe640 1 -- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45c004d40 msgr2=0x7fa45c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa4737fe640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45c004d40 0x7fa45c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.585+0000 7fa4737fe640 1 -- 172.21.15.72:0/2771051755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c003520 con 0x7fa45c004060 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa4737fe640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa45c004800 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa46800ba30 tx=0x7fa45c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445c43fdf977ee24 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468036020 con 0x7fa45c004060 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa468032640 con 0x7fa45c004060 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa472ffd640 1 -- 172.21.15.72:0/2771051755 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46800b600 con 0x7fa45c004060 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa47a9b5640 1 -- 172.21.15.72:0/2771051755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 msgr2=0x7fa45c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/2771051755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa45c004800 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fa46800ba30 tx=0x7fa45c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa478f52640 1 -- 172.21.15.72:0/2771051755 reap_dead start 2024-03-06T07:28:31.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa47a9b5640 1 -- 172.21.15.72:0/2771051755 shutdown_connections 2024-03-06T07:28:31.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa47a9b5640 1 -- 172.21.15.72:0/2771051755 >> 172.21.15.72:0/2771051755 conn(0x7fa47406c0c0 msgr2=0x7fa47406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa47a9b5640 1 -- 172.21.15.72:0/2771051755 shutdown_connections 2024-03-06T07:28:31.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.586+0000 7fa47a9b5640 1 -- 172.21.15.72:0/2771051755 wait complete. 2024-03-06T07:28:31.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 Processor -- start 2024-03-06T07:28:31.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 -- start start 2024-03-06T07:28:31.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa47414ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa474147970 0x7fa4741e7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4741e7da0 0x7fa47406e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa474072090 con 0x7fa45c004060 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4740ce200 con 0x7fa4741e7da0 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.588+0000 7fa47a9b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa474147db0 con 0x7fa474147970 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa47414ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa47414ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45840/0 (socket says 172.21.15.72:45840) 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 -- 172.21.15.72:0/1909266114 learned_addr learned my addr 172.21.15.72:0/1909266114 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa478f52640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4741e7da0 0x7fa47406e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa4737fe640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa474147970 0x7fa4741e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 -- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa474147970 msgr2=0x7fa4741e7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa474147970 0x7fa4741e7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 -- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4741e7da0 msgr2=0x7fa47406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4741e7da0 0x7fa47406e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 -- 172.21.15.72:0/1909266114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4741e8320 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa473fff640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa47414ac40 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa4640017e0 tx=0x7fa46400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46400c6c0 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa464018070 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4741e8600 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46400ca80 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.589+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47414a440 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.590+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47406dae0 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.591+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa46400cc20 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.591+0000 7fa470ff9640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa46006b4d0 0x7fa46006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.591+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fa464089e70 con 0x7fa45c004060 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.591+0000 7fa4737fe640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa46006b4d0 0x7fa46006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.591+0000 7fa4737fe640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa46006b4d0 0x7fa46006d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa46800ba00 tx=0x7fa46803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.595+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa47406dae0 con 0x7fa45c004060 2024-03-06T07:28:31.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.768+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa47404f330 con 0x7fa45c004060 2024-03-06T07:28:31.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.769+0000 7fa470ff9640 1 -- 172.21.15.72:0/1909266114 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+13432 (secure 0 0 0) 0x7fa464058ee0 con 0x7fa45c004060 2024-03-06T07:28:31.771 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:31.771 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":119,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:28:30.461224+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6835/3088662828":"2024-03-07T07:28:27.451456+0000","172.21.15.72:6834/3088662828":"2024-03-07T07:28:27.451456+0000","172.21.15.72:6835/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6834/1152083148":"2024-03-07T07:27:25.269515+0000","172.21.15.72:6837/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.72:6836/3618393393":"2024-03-07T07:27:25.094353+0000","172.21.15.146:6835/118157918":"2024-03-07T07:27:25.085724+0000","172.21.15.146:6833/118157918":"2024-03-07T07:27:25.085724+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa46006b4d0 msgr2=0x7fa46006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa46006b4d0 0x7fa46006d990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa46800ba00 tx=0x7fa46803d040 comp rx=0 tx=0).stop 2024-03-06T07:28:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 msgr2=0x7fa47414ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa47414ac40 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fa4640017e0 tx=0x7fa46400eb90 comp rx=0 tx=0).stop 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 shutdown_connections 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa46006b4d0 0x7fa46006d990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4741e7da0 0x7fa47406e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa474147970 0x7fa4741e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 --2- 172.21.15.72:0/1909266114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c004060 0x7fa47414ac40 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 >> 172.21.15.72:0/1909266114 conn(0x7fa47406c0c0 msgr2=0x7fa474160290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 shutdown_connections 2024-03-06T07:28:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.771+0000 7fa47a9b5640 1 -- 172.21.15.72:0/1909266114 wait complete. 2024-03-06T07:28:31.784 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:31.784 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:31.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.884+0000 7fdf334e0640 1 Processor -- start 2024-03-06T07:28:31.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.884+0000 7fdf334e0640 1 -- start start 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf334e0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdf2c0cfa90 0x7fdf2c0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf334e0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdf2c0b33a0 0x7fdf2c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf334e0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf2c158ac0 con 0x7fdf2c1586e0 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf334e0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2c15b2a0 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf334e0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf2c15b540 con 0x7fdf2c0b33a0 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf30a7b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdf2c0cfa90 0x7fdf2c0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf30a7b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdf2c0cfa90 0x7fdf2c0cff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45842/0 (socket says 172.21.15.72:45842) 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.885+0000 7fdf30a7b640 1 -- 172.21.15.72:0/1894135523 learned_addr learned my addr 172.21.15.72:0/1894135523 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf31a7d640 1 --2- 172.21.15.72:0/1894135523 >> v2:172.21.15.146:3300/0 conn(0x7fdf2c0b33a0 0x7fdf2c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf30a7b640 1 -- 172.21.15.72:0/1894135523 >> v2:172.21.15.146:3300/0 conn(0x7fdf2c0b33a0 msgr2=0x7fdf2c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf30a7b640 1 --2- 172.21.15.72:0/1894135523 >> v2:172.21.15.146:3300/0 conn(0x7fdf2c0b33a0 0x7fdf2c0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf30a7b640 1 -- 172.21.15.72:0/1894135523 >> v1:172.21.15.146:6789/0 conn(0x7fdf2c1586e0 legacy=0x7fdf2c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:31.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf30a7b640 1 -- 172.21.15.72:0/1894135523 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c15b820 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf31a7d640 1 --2- 172.21.15.72:0/1894135523 >> v2:172.21.15.146:3300/0 conn(0x7fdf2c0b33a0 0x7fdf2c0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf30a7b640 1 --2- 172.21.15.72:0/1894135523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c0cff40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c0b2ae0 tx=0x7fdf1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83be344669bd40e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf23fff640 1 -- 172.21.15.72:0/1894135523 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c035020 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.886+0000 7fdf23fff640 1 -- 172.21.15.72:0/1894135523 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c031870 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf23fff640 1 -- 172.21.15.72:0/1894135523 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c031c40 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 -- 172.21.15.72:0/1894135523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 msgr2=0x7fdf2c0cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 --2- 172.21.15.72:0/1894135523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c0cff40 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c0b2ae0 tx=0x7fdf1c02b910 comp rx=0 tx=0).stop 2024-03-06T07:28:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 -- 172.21.15.72:0/1894135523 shutdown_connections 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 --2- 172.21.15.72:0/1894135523 >> v2:172.21.15.146:3300/0 conn(0x7fdf2c0b33a0 0x7fdf2c0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 --2- 172.21.15.72:0/1894135523 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c0cff40 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 -- 172.21.15.72:0/1894135523 >> 172.21.15.72:0/1894135523 conn(0x7fdf2c06b190 msgr2=0x7fdf2c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 -- 172.21.15.72:0/1894135523 shutdown_connections 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.887+0000 7fdf334e0640 1 -- 172.21.15.72:0/1894135523 wait complete. 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.888+0000 7fdf334e0640 1 Processor -- start 2024-03-06T07:28:31.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.888+0000 7fdf334e0640 1 -- start start 2024-03-06T07:28:31.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf334e0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf2c0b33a0 0x7fdf2c0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf334e0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf334e0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf2c1586e0 0x7fdf2c06cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf334e0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c0af150 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf334e0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c15bde0 con 0x7fdf2c0b33a0 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf334e0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c071350 con 0x7fdf2c1586e0 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45846/0 (socket says 172.21.15.72:45846) 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 -- 172.21.15.72:0/3492371332 learned_addr learned my addr 172.21.15.72:0/3492371332 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 -- 172.21.15.72:0/3492371332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf2c1586e0 msgr2=0x7fdf2c06cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf2c1586e0 0x7fdf2c06cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 -- 172.21.15.72:0/3492371332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf2c0b33a0 msgr2=0x7fdf2c0b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf2c0b33a0 0x7fdf2c0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 -- 172.21.15.72:0/3492371332 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c06c810 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.889+0000 7fdf30a7b640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c141000 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.890+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c040070 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.890+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf1c03a420 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.890+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c06caf0 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.890+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c035040 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.890+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c0b04d0 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.891+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c0b2930 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.891+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdf1c04b060 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.892+0000 7fdf21ffb640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf1006b4d0 0x7fdf1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.892+0000 7fdf3127c640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf1006b4d0 0x7fdf1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.892+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fdf1c0abfd0 con 0x7fdf2c0cfa90 2024-03-06T07:28:31.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.892+0000 7fdf3127c640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf1006b4d0 0x7fdf1006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c14b7b0 tx=0x7fdf28007d30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:31.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:31.896+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdf2c0b2930 con 0x7fdf2c0cfa90 2024-03-06T07:28:32.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.068+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf2c0732b0 con 0x7fdf2c0cfa90 2024-03-06T07:28:32.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.068+0000 7fdf21ffb640 1 -- 172.21.15.72:0/3492371332 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v119) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf1c07b040 con 0x7fdf2c0cfa90 2024-03-06T07:28:32.070 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:28:32.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf1006b4d0 msgr2=0x7fdf1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf1006b4d0 0x7fdf1006d990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c14b7b0 tx=0x7fdf28007d30 comp rx=0 tx=0).stop 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 msgr2=0x7fdf2c141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c141000 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c000c00 tx=0x7fdf1c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 shutdown_connections 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdf1006b4d0 0x7fdf1006d990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf2c1586e0 0x7fdf2c06cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdf2c0cfa90 0x7fdf2c141000 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 --2- 172.21.15.72:0/3492371332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf2c0b33a0 0x7fdf2c0b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 >> 172.21.15.72:0/3492371332 conn(0x7fdf2c06b190 msgr2=0x7fdf2c14c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 shutdown_connections 2024-03-06T07:28:32.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.070+0000 7fdf334e0640 1 -- 172.21.15.72:0/3492371332 wait complete. 2024-03-06T07:28:32.083 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:28:32.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.181+0000 7fcf8e155640 1 Processor -- start 2024-03-06T07:28:32.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.181+0000 7fcf8e155640 1 -- start start 2024-03-06T07:28:32.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf8e155640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcf881515e0 0x7fcf880cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf8e155640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcf880cd7a0 0x7fcf880cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf8e155640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf88071f40 con 0x7fcf8815b430 2024-03-06T07:28:32.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf8e155640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8815b8e0 con 0x7fcf880cd7a0 2024-03-06T07:28:32.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf8e155640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf88070a70 con 0x7fcf881515e0 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf87fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcf880cd7a0 0x7fcf880cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf7ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcf8815b430 0x7fcf88071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55490/0 (socket says 172.21.15.72:55490) 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf7ffff640 1 -- 172.21.15.72:0/3696258344 learned_addr learned my addr 172.21.15.72:0/3696258344 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf877fe640 1 --2- 172.21.15.72:0/3696258344 >> v2:172.21.15.146:3300/0 conn(0x7fcf881515e0 0x7fcf880cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf87fff640 1 -- 172.21.15.72:0/3696258344 >> v2:172.21.15.146:3300/0 conn(0x7fcf881515e0 msgr2=0x7fcf880cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf87fff640 1 --2- 172.21.15.72:0/3696258344 >> v2:172.21.15.146:3300/0 conn(0x7fcf881515e0 0x7fcf880cce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf86ffd640 1 -- 172.21.15.72:0/3696258344 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37542039 0 0) 0x7fcf88071f40 con 0x7fcf8815b430 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf87fff640 1 -- 172.21.15.72:0/3696258344 >> v1:172.21.15.72:6789/0 conn(0x7fcf8815b430 legacy=0x7fcf88071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.182+0000 7fcf87fff640 1 -- 172.21.15.72:0/3696258344 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88070d50 con 0x7fcf880cd7a0 2024-03-06T07:28:32.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.183+0000 7fcf87fff640 1 --2- 172.21.15.72:0/3696258344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf880cdb80 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fcf8806eb50 tx=0x7fcf8002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0ed1547b671f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.183+0000 7fcf86ffd640 1 -- 172.21.15.72:0/3696258344 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80039070 con 0x7fcf880cd7a0 2024-03-06T07:28:32.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.183+0000 7fcf86ffd640 1 -- 172.21.15.72:0/3696258344 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8002cc80 con 0x7fcf880cd7a0 2024-03-06T07:28:32.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.183+0000 7fcf86ffd640 1 -- 172.21.15.72:0/3696258344 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf80034660 con 0x7fcf880cd7a0 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 -- 172.21.15.72:0/3696258344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 msgr2=0x7fcf880cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3696258344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf880cdb80 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fcf8806eb50 tx=0x7fcf8002bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 -- 172.21.15.72:0/3696258344 shutdown_connections 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3696258344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf880cdb80 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3696258344 >> v2:172.21.15.146:3300/0 conn(0x7fcf881515e0 0x7fcf880cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 -- 172.21.15.72:0/3696258344 >> 172.21.15.72:0/3696258344 conn(0x7fcf880cf8a0 msgr2=0x7fcf880cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 -- 172.21.15.72:0/3696258344 shutdown_connections 2024-03-06T07:28:32.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.184+0000 7fcf8e155640 1 -- 172.21.15.72:0/3696258344 wait complete. 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 Processor -- start 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 -- start start 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf881453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf881515e0 0x7fcf88145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf8815b430 0x7fcf8815dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880ce0c0 con 0x7fcf880cd7a0 2024-03-06T07:28:32.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88071310 con 0x7fcf8815b430 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.185+0000 7fcf8e155640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf880cc3a0 con 0x7fcf881515e0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf881453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf881453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45864/0 (socket says 172.21.15.72:45864) 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 -- 172.21.15.72:0/3076101859 learned_addr learned my addr 172.21.15.72:0/3076101859 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 -- 172.21.15.72:0/3076101859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf881515e0 msgr2=0x7fcf88145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf881515e0 0x7fcf88145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 -- 172.21.15.72:0/3076101859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf8815b430 msgr2=0x7fcf8815dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf8815b430 0x7fcf8815dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 -- 172.21.15.72:0/3076101859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88152770 con 0x7fcf880cd7a0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf877fe640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf881453f0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800c7f0 tx=0x7fcf7800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78015070 con 0x7fcf880cd7a0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf78004240 con 0x7fcf880cd7a0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf881500e0 con 0x7fcf880cd7a0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78007620 con 0x7fcf880cd7a0 2024-03-06T07:28:32.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.186+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881506a0 con 0x7fcf880cd7a0 2024-03-06T07:28:32.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.187+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf88068840 con 0x7fcf880cd7a0 2024-03-06T07:28:32.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.188+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcf780077c0 con 0x7fcf880cd7a0 2024-03-06T07:28:32.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.188+0000 7fcf84ff9640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf6806b4d0 0x7fcf6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.188+0000 7fcf7ffff640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf6806b4d0 0x7fcf6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.188+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fcf78089ed0 con 0x7fcf880cd7a0 2024-03-06T07:28:32.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.188+0000 7fcf7ffff640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf6806b4d0 0x7fcf6806d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcf70005880 tx=0x7fcf70007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.192+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcf78058f40 con 0x7fcf880cd7a0 2024-03-06T07:28:32.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.429+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7fcf8804f330 con 0x7fcf880cd7a0 2024-03-06T07:28:32.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.473+0000 7fcf84ff9640 1 -- 172.21.15.72:0/3076101859 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v356) v1 ==== 143+0+0 (secure 0 0 0) 0x7fcf78058840 con 0x7fcf880cd7a0 2024-03-06T07:28:32.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf6806b4d0 msgr2=0x7fcf6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf6806b4d0 0x7fcf6806d990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fcf70005880 tx=0x7fcf70007a00 comp rx=0 tx=0).stop 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 msgr2=0x7fcf881453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf881453f0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800c7f0 tx=0x7fcf7800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 shutdown_connections 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcf6806b4d0 0x7fcf6806d990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf8815b430 0x7fcf8815dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf881515e0 0x7fcf88145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 --2- 172.21.15.72:0/3076101859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcf880cd7a0 0x7fcf881453f0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 >> 172.21.15.72:0/3076101859 conn(0x7fcf880cf8a0 msgr2=0x7fcf8814ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 shutdown_connections 2024-03-06T07:28:32.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.475+0000 7fcf8e155640 1 -- 172.21.15.72:0/3076101859 wait complete. 2024-03-06T07:28:32.488 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-06T07:28:32.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 Processor -- start 2024-03-06T07:28:32.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 -- start start 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f80aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99f80aef80 0x7f99f80cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80d03f0 con 0x7f99f80cffd0 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f8158ac0 con 0x7f99f80aef80 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.589+0000 7f99fce66640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f815b2a0 con 0x7f99f81586e0 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f5d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f99f80aef80 0x7f99f80cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f6d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f99f80cffd0 0x7f99f806da20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59082/0 (socket says 172.21.15.72:59082) 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f6d9d640 1 -- 172.21.15.72:0/2089598350 learned_addr learned my addr 172.21.15.72:0/2089598350 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f559a640 1 -- 172.21.15.72:0/2089598350 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287929374 0 0) 0x7f99f80d03f0 con 0x7f99f80cffd0 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f559a640 1 -- 172.21.15.72:0/2089598350 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e0003520 con 0x7f99f80cffd0 2024-03-06T07:28:32.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f5d9b640 1 -- 172.21.15.72:0/2089598350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 msgr2=0x7f99f80aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f659c640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f80aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f5d9b640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f80aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f5d9b640 1 -- 172.21.15.72:0/2089598350 >> v1:172.21.15.72:6789/0 conn(0x7f99f80cffd0 legacy=0x7f99f806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f5d9b640 1 -- 172.21.15.72:0/2089598350 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f815b540 con 0x7f99f80aef80 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.590+0000 7f99f659c640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f80aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99f5d9b640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80aef80 0x7f99f80cfa90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5762be18c9d31e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99f559a640 1 -- 172.21.15.72:0/2089598350 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec035020 con 0x7f99f80aef80 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99f559a640 1 -- 172.21.15.72:0/2089598350 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec031830 con 0x7f99f80aef80 2024-03-06T07:28:32.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99f559a640 1 -- 172.21.15.72:0/2089598350 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec031c00 con 0x7f99f80aef80 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99fce66640 1 -- 172.21.15.72:0/2089598350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80aef80 msgr2=0x7f99f80cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99fce66640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80aef80 0x7f99f80cfa90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99fce66640 1 -- 172.21.15.72:0/2089598350 shutdown_connections 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99fce66640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80aef80 0x7f99f80cfa90 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99fce66640 1 --2- 172.21.15.72:0/2089598350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f80aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.591+0000 7f99fce66640 1 -- 172.21.15.72:0/2089598350 >> 172.21.15.72:0/2089598350 conn(0x7f99f806b190 msgr2=0x7f99f806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.592+0000 7f99fce66640 1 -- 172.21.15.72:0/2089598350 shutdown_connections 2024-03-06T07:28:32.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.592+0000 7f99fce66640 1 -- 172.21.15.72:0/2089598350 wait complete. 2024-03-06T07:28:32.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.593+0000 7f99fce66640 1 Processor -- start 2024-03-06T07:28:32.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.593+0000 7f99fce66640 1 -- start start 2024-03-06T07:28:32.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80aef80 0x7f99f8146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 0x7f99f81f5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f814b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f815bde0 con 0x7f99f80cffd0 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f815bb00 con 0x7f99f80aef80 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f8071350 con 0x7f99f81586e0 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 0x7f99f81f5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 0x7f99f81f5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59006/0 (socket says 172.21.15.72:59006) 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 -- 172.21.15.72:0/4147833110 learned_addr learned my addr 172.21.15.72:0/4147833110 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f6d9d640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f814b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 -- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 msgr2=0x7f99f814b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f659c640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80aef80 0x7f99f8146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f814b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 -- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80aef80 msgr2=0x7f99f8146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80aef80 0x7f99f8146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 -- 172.21.15.72:0/4147833110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f81f5e90 con 0x7f99f80cffd0 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f659c640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80aef80 0x7f99f8146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99f5d9b640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 0x7f99f81f5950 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f99ec02bee0 tx=0x7f99ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec035040 con 0x7f99f80cffd0 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec03b920 con 0x7f99f80cffd0 2024-03-06T07:28:32.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f814e750 con 0x7f99f80cffd0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec03bd10 con 0x7f99f80cffd0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.594+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f814a690 con 0x7f99f80cffd0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.595+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f80732b0 con 0x7f99f80cffd0 2024-03-06T07:28:32.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.596+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f99ec038020 con 0x7f99f80cffd0 2024-03-06T07:28:32.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.596+0000 7f99e6ffd640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f99d806b4d0 0x7f99d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:32.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.596+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f99ec0ac130 con 0x7f99f80cffd0 2024-03-06T07:28:32.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.596+0000 7f99f659c640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f99d806b4d0 0x7f99d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:32.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.596+0000 7f99f659c640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f99d806b4d0 0x7f99d806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f99e8007f40 tx=0x7f99e8007490 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:32.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.600+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99ec07b1b0 con 0x7f99f80cffd0 2024-03-06T07:28:32.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.770+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f99f80ce9b0 con 0x7f99f80cffd0 2024-03-06T07:28:32.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.775+0000 7f99e6ffd640 1 -- 172.21.15.72:0/4147833110 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f99ec031800 con 0x7f99f80cffd0 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f99d806b4d0 msgr2=0x7f99d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f99d806b4d0 0x7f99d806d990 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f99e8007f40 tx=0x7f99e8007490 comp rx=0 tx=0).stop 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 msgr2=0x7f99f81f5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 0x7f99f81f5950 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f99ec02bee0 tx=0x7f99ec0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 shutdown_connections 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f99d806b4d0 0x7f99d806d990 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f81586e0 0x7f99f814b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99f80cffd0 0x7f99f81f5950 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 --2- 172.21.15.72:0/4147833110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80aef80 0x7f99f8146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:32.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.777+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 >> 172.21.15.72:0/4147833110 conn(0x7f99f806b190 msgr2=0x7f99f80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:32.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.778+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 shutdown_connections 2024-03-06T07:28:32.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:32.778+0000 7f99fce66640 1 -- 172.21.15.72:0/4147833110 wait complete. 2024-03-06T07:28:32.791 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:28:32.791 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:32.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:28:32.878 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:28:32.878 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:28:32.878 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:32.893 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:32.893 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:28:32.893 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:32.936 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:32.936 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:28:32.936 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:32.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:28:32.951 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:28:32.951 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:28:32.993 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:28:32.993 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-06T07:28:32.994 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:33.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.091+0000 7f91b9217640 1 Processor -- start 2024-03-06T07:28:33.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.091+0000 7f91b9217640 1 -- start start 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b9217640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 0x7f91b40cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b9217640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b40cc980 con 0x7f91b406da40 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b9217640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b40ccc60 con 0x7f91b41424f0 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b9217640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40cd220 con 0x7f91b40cd7a0 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91abfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f91b406da40 0x7f91b4071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59084/0 (socket says 172.21.15.72:59084) 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91abfff640 1 -- 172.21.15.72:0/2107267683 learned_addr learned my addr 172.21.15.72:0/2107267683 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034347423 0 0) 0x7f91b40cc980 con 0x7f91b406da40 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a0003520 con 0x7f91b406da40 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b359e640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 0x7f91b40cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4050029259 0 0) 0x7f91a0003520 con 0x7f91b406da40 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 msgr2=0x7f91b40cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 0x7f91b40cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 >> v1:172.21.15.146:6789/0 conn(0x7f91b41424f0 legacy=0x7f91b406d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b40ccf40 con 0x7f91b406da40 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f919c001670 con 0x7f91b406da40 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 >> v1:172.21.15.72:6789/0 conn(0x7f91b406da40 legacy=0x7f91b4071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 0x7f91a00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91a0004010 0x7f91a0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a0004d40 0x7f91a00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40cd220 con 0x7f91a0004010 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000bb80 con 0x7f91a0004d40 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000bd80 con 0x7f91b40cd7a0 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.092+0000 7f91abfff640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91a0004010 0x7f91a0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91b2d9d640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 0x7f91a00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91b359e640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a0004d40 0x7f91a00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91abfff640 1 -- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 msgr2=0x7f91a00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91abfff640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91b40cd7a0 0x7f91a00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91abfff640 1 -- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a0004d40 msgr2=0x7f91a00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91abfff640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a0004d40 0x7f91a00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91abfff640 1 -- 172.21.15.72:0/2107267683 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0003520 con 0x7f91a0004010 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91b359e640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a0004d40 0x7f91a00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.093+0000 7f91abfff640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91a0004010 0x7f91a0004800 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f919c00c210 tx=0x7f91a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=358af8dd978602bf server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c007d30 con 0x7f91a0004010 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c030070 con 0x7f91a0004010 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b259c640 1 -- 172.21.15.72:0/2107267683 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c02cda0 con 0x7f91a0004010 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b9217640 1 -- 172.21.15.72:0/2107267683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91a0004010 msgr2=0x7f91a0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b9217640 1 --2- 172.21.15.72:0/2107267683 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91a0004010 0x7f91a0004800 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f919c00c210 tx=0x7f91a0005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b359e640 1 -- 172.21.15.72:0/2107267683 reap_dead start 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b9217640 1 -- 172.21.15.72:0/2107267683 shutdown_connections 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b9217640 1 -- 172.21.15.72:0/2107267683 >> 172.21.15.72:0/2107267683 conn(0x7f91b40cf8a0 msgr2=0x7f91b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b9217640 1 -- 172.21.15.72:0/2107267683 shutdown_connections 2024-03-06T07:28:33.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.094+0000 7f91b9217640 1 -- 172.21.15.72:0/2107267683 wait complete. 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.095+0000 7f91b9217640 1 Processor -- start 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.095+0000 7f91b9217640 1 -- start start 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b9217640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a0004010 0x7f91b4140ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b9217640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 0x7f91b40a9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b9217640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b406afb0 0x7f91b406b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b9217640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40cc470 con 0x7f91b4157370 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b9217640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40cc2f0 con 0x7f91b406afb0 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b9217640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b40cc5f0 con 0x7f91a0004010 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 0x7f91b40a9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 0x7f91b40a9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59024/0 (socket says 172.21.15.72:59024) 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91abfff640 1 -- 172.21.15.72:0/2013219220 learned_addr learned my addr 172.21.15.72:0/2013219220 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b2d9d640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a0004010 0x7f91b4140ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91b359e640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b406afb0 0x7f91b406b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.096+0000 7f91abfff640 1 -- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a0004010 msgr2=0x7f91b4140ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91abfff640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a0004010 0x7f91b4140ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91abfff640 1 -- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b406afb0 msgr2=0x7f91b406b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91abfff640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b406afb0 0x7f91b406b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91abfff640 1 -- 172.21.15.72:0/2013219220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b413fc40 con 0x7f91b4157370 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91b2d9d640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a0004010 0x7f91b4140ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91b359e640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b406afb0 0x7f91b406b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91abfff640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 0x7f91b40a9df0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f919c02c570 tx=0x7f919c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c004bf0 con 0x7f91b4157370 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919c004d90 con 0x7f91b4157370 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b413ff20 con 0x7f91b4157370 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919c007270 con 0x7f91b4157370 2024-03-06T07:28:33.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.097+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b41404e0 con 0x7f91b4157370 2024-03-06T07:28:33.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.098+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b404f330 con 0x7f91b4157370 2024-03-06T07:28:33.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.098+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f919c047020 con 0x7f91b4157370 2024-03-06T07:28:33.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.099+0000 7f91ab7fe640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f919406b500 0x7f919406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.099+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f919c0ab270 con 0x7f91b4157370 2024-03-06T07:28:33.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.099+0000 7f91b2d9d640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f919406b500 0x7f919406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.099+0000 7f91b2d9d640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f919406b500 0x7f919406d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f91a4000c00 tx=0x7f91a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.103+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f919c07a2e0 con 0x7f91b4157370 2024-03-06T07:28:33.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.311+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91b40d0620 con 0x7f91b4157370 2024-03-06T07:28:33.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.311+0000 7f91ab7fe640 1 -- 172.21.15.72:0/2013219220 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f919c079be0 con 0x7f91b4157370 2024-03-06T07:28:33.314 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:33.315 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6833/519045994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":519045994},{"type":"v1","addr":"172.21.15.146:6833","nonce":519045994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":6329,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1063494679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1063494679},{"type":"v1","addr":"172.21.15.146:6835","nonce":1063494679}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/508281432","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6836","nonce":508281432},{"type":"v1","addr":"172.21.15.72:6837","nonce":508281432}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":8559,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6835/1620401597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":1620401597},{"type":"v1","addr":"172.21.15.72:6835","nonce":1620401597}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":74}],"filesystems":[]} 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 74 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f919406b500 msgr2=0x7f919406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f919406b500 0x7f919406d9c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f91a4000c00 tx=0x7f91a4009040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 msgr2=0x7f91b40a9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 0x7f91b40a9df0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f919c02c570 tx=0x7f919c001a40 comp rx=0 tx=0).stop 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 shutdown_connections 2024-03-06T07:28:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f919406b500 0x7f919406d9c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91b406afb0 0x7f91b406b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f91b4157370 0x7f91b40a9df0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 --2- 172.21.15.72:0/2013219220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a0004010 0x7f91b4140ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 >> 172.21.15.72:0/2013219220 conn(0x7f91b40cf8a0 msgr2=0x7f91b4160400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 shutdown_connections 2024-03-06T07:28:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.313+0000 7f91b9217640 1 -- 172.21.15.72:0/2013219220 wait complete. 2024-03-06T07:28:33.327 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-06T07:28:33.327 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-06T07:28:33.327 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:28:33.327 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:33.328 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-06T07:28:33.328 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-06T07:28:33.328 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:28:33.328 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:33.328 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-06T07:28:33.328 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-06T07:28:33.328 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:33.329 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:33.329 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-06T07:28:33.329 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-06T07:28:33.329 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:33.329 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:33.336 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:28:33.336 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-06T07:28:33.337 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:28:33.337 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-06T07:28:33.340 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:33.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-06T07:28:33.341 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:33.341 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-06T07:28:33.379 INFO:tasks.ceph.mds.a:Started 2024-03-06T07:28:33.379 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:33.380 INFO:tasks.ceph.mds.c:Started 2024-03-06T07:28:33.380 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:33.383 INFO:tasks.ceph.mds.b:Started 2024-03-06T07:28:33.383 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:33.383 INFO:tasks.ceph.mds.d:Started 2024-03-06T07:28:33.384 DEBUG:teuthology.parallel:result is None 2024-03-06T07:28:33.384 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-06T07:28:33.456 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-03-06T07:28:33.456 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:28:33.469 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-03-06T07:28:33.469 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-03-06T07:28:33.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.488+0000 7f3ea8c37640 1 Processor -- start 2024-03-06T07:28:33.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.488+0000 7f3ea8c37640 1 -- start start 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea8c37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 0x7f3ea40ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea8c37640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea40d25e0 con 0x7f3ea40d0d90 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea8c37640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea40d2880 con 0x7f3ea415e000 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea8c37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40c7c50 con 0x7f3ea40c90f0 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3e9bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3ea40d0d90 0x7f3ea40ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59096/0 (socket says 172.21.15.72:59096) 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3e9bfff640 1 -- 172.21.15.72:0/852131083 learned_addr learned my addr 172.21.15.72:0/852131083 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea2d9d640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 0x7f3ea40ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440053515 0 0) 0x7f3ea40d25e0 con 0x7f3ea40d0d90 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e90003520 con 0x7f3ea40d0d90 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278917474 0 0) 0x7f3e90003520 con 0x7f3ea40d0d90 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 msgr2=0x7f3ea40ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 0x7f3ea40ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 >> v1:172.21.15.146:6789/0 conn(0x7f3ea415e000 legacy=0x7f3ea40d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40d2b40 con 0x7f3ea40d0d90 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3e8c001670 con 0x7f3ea40d0d90 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 >> v1:172.21.15.72:6789/0 conn(0x7f3ea40d0d90 legacy=0x7f3ea40ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 0x7f3e900039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e90003f60 0x7f3e90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e90004d40 0x7f3e900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40c7c50 con 0x7f3e90003f60 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9000bb30 con 0x7f3e90004d40 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.489+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9000bd30 con 0x7f3ea40c90f0 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e90003f60 0x7f3e90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3ea259c640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 0x7f3e900039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3ea2d9d640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e90004d40 0x7f3e900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3e9bfff640 1 -- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 msgr2=0x7f3e900039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ea40c90f0 0x7f3e900039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3e9bfff640 1 -- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e90004d40 msgr2=0x7f3e900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e90004d40 0x7f3e900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.490+0000 7f3e9bfff640 1 -- 172.21.15.72:0/852131083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e90003520 con 0x7f3e90003f60 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e90003f60 0x7f3e90004800 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00c210 tx=0x7f3e90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17dbb894552a790 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c007d30 con 0x7f3e90003f60 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c030070 con 0x7f3e90003f60 2024-03-06T07:28:33.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea1d9b640 1 -- 172.21.15.72:0/852131083 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c02cda0 con 0x7f3e90003f60 2024-03-06T07:28:33.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea8c37640 1 -- 172.21.15.72:0/852131083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e90003f60 msgr2=0x7f3e90004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/852131083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e90003f60 0x7f3e90004800 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00c210 tx=0x7f3e90005830 comp rx=0 tx=0).stop 2024-03-06T07:28:33.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea2d9d640 1 -- 172.21.15.72:0/852131083 reap_dead start 2024-03-06T07:28:33.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea8c37640 1 -- 172.21.15.72:0/852131083 shutdown_connections 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea8c37640 1 -- 172.21.15.72:0/852131083 >> 172.21.15.72:0/852131083 conn(0x7f3ea40d3cf0 msgr2=0x7f3ea40d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea8c37640 1 -- 172.21.15.72:0/852131083 shutdown_connections 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.491+0000 7f3ea8c37640 1 -- 172.21.15.72:0/852131083 wait complete. 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 Processor -- start 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 -- start start 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e90003f60 0x7f3ea4156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 0x7f3ea40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ea41f7130 0x7f3ea41f75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea415e3e0 con 0x7f3ea40b1b70 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40d22d0 con 0x7f3ea41f7130 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3ea8c37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40d4810 con 0x7f3e90003f60 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3e9bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 0x7f3ea40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3e9bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 0x7f3ea40b1ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59072/0 (socket says 172.21.15.72:59072) 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.493+0000 7f3e9bfff640 1 -- 172.21.15.72:0/2171979914 learned_addr learned my addr 172.21.15.72:0/2171979914 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:33.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3ea2d9d640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ea41f7130 0x7f3ea41f75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3ea259c640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e90003f60 0x7f3ea4156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9bfff640 1 -- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e90003f60 msgr2=0x7f3ea4156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e90003f60 0x7f3ea4156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9bfff640 1 -- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ea41f7130 msgr2=0x7f3ea41f75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ea41f7130 0x7f3ea41f75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9bfff640 1 -- 172.21.15.72:0/2171979914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea4156a70 con 0x7f3ea40b1b70 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3ea2d9d640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ea41f7130 0x7f3ea41f75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9bfff640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 0x7f3ea40b1ff0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00c330 tx=0x7f3e8c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c045070 con 0x7f3ea40b1b70 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c000ed0 con 0x7f3ea40b1b70 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c004e30 con 0x7f3ea40b1b70 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea40b2710 con 0x7f3ea40b1b70 2024-03-06T07:28:33.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.494+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea40b2cd0 con 0x7f3ea40b1b70 2024-03-06T07:28:33.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.495+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea404f330 con 0x7f3ea40b1b70 2024-03-06T07:28:33.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.495+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3e8c007070 con 0x7f3ea40b1b70 2024-03-06T07:28:33.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.496+0000 7f3e9affd640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8406b500 0x7f3e8406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.496+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f3e8c0ab0b0 con 0x7f3ea40b1b70 2024-03-06T07:28:33.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.496+0000 7f3ea259c640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8406b500 0x7f3e8406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.496+0000 7f3ea259c640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8406b500 0x7f3e8406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3e94005590 tx=0x7f3e94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.500+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3e8c07a290 con 0x7f3ea40b1b70 2024-03-06T07:28:33.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.681+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3ea40d1d70 con 0x7f3ea40b1b70 2024-03-06T07:28:33.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.785+0000 7f3e9affd640 1 -- 172.21.15.72:0/2171979914 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v123) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3e8c079b90 con 0x7f3ea40b1b70 2024-03-06T07:28:33.787 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-03-06T07:28:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8406b500 msgr2=0x7f3e8406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8406b500 0x7f3e8406d9c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f3e94005590 tx=0x7f3e94009040 comp rx=0 tx=0).stop 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 msgr2=0x7f3ea40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 0x7f3ea40b1ff0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00c330 tx=0x7f3e8c001a40 comp rx=0 tx=0).stop 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 shutdown_connections 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3e8406b500 0x7f3e8406d9c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ea41f7130 0x7f3ea41f75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ea40b1b70 0x7f3ea40b1ff0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 --2- 172.21.15.72:0/2171979914 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e90003f60 0x7f3ea4156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 >> 172.21.15.72:0/2171979914 conn(0x7f3ea40d3cf0 msgr2=0x7f3ea406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 shutdown_connections 2024-03-06T07:28:33.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.787+0000 7f3ea8c37640 1 -- 172.21.15.72:0/2171979914 wait complete. 2024-03-06T07:28:33.801 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-06T07:28:33.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.898+0000 7f5d61fd0640 1 Processor -- start 2024-03-06T07:28:33.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.899+0000 7f5d61fd0640 1 -- start start 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.899+0000 7f5d61fd0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c13dc10 0x7f5d5c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.899+0000 7f5d61fd0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5d5c15bf30 0x7f5d5c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.899+0000 7f5d61fd0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c15f580 con 0x7f5d5c15e850 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.899+0000 7f5d61fd0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c0cf3f0 con 0x7f5d5c15bf30 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.899+0000 7f5d61fd0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c15b210 con 0x7f5d5c13dc10 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d52ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5d5c15bf30 0x7f5d5c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d5bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5d5c15e850 0x7f5d5c15eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59100/0 (socket says 172.21.15.72:59100) 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d5bfff640 1 -- 172.21.15.72:0/3202570164 learned_addr learned my addr 172.21.15.72:0/3202570164 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d5b7fe640 1 --2- 172.21.15.72:0/3202570164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c13dc10 0x7f5d5c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d52ffd640 1 -- 172.21.15.72:0/3202570164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c13dc10 msgr2=0x7f5d5c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:33.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d52ffd640 1 --2- 172.21.15.72:0/3202570164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c13dc10 0x7f5d5c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d52ffd640 1 -- 172.21.15.72:0/3202570164 >> v1:172.21.15.72:6789/0 conn(0x7f5d5c15e850 legacy=0x7f5d5c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d5affd640 1 -- 172.21.15.72:0/3202570164 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142205989 0 0) 0x7f5d5c15f580 con 0x7f5d5c15e850 2024-03-06T07:28:33.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.900+0000 7f5d52ffd640 1 -- 172.21.15.72:0/3202570164 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c15b450 con 0x7f5d5c15bf30 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d52ffd640 1 --2- 172.21.15.72:0/3202570164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c0b2e90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f5d440097a0 tx=0x7f5d4402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90388aee9fc7b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d5affd640 1 -- 172.21.15.72:0/3202570164 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d440315a0 con 0x7f5d5c15bf30 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d5affd640 1 -- 172.21.15.72:0/3202570164 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d44037070 con 0x7f5d5c15bf30 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d5affd640 1 -- 172.21.15.72:0/3202570164 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44031960 con 0x7f5d5c15bf30 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 -- 172.21.15.72:0/3202570164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 msgr2=0x7f5d5c0b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/3202570164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c0b2e90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f5d440097a0 tx=0x7f5d4402b840 comp rx=0 tx=0).stop 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 -- 172.21.15.72:0/3202570164 shutdown_connections 2024-03-06T07:28:33.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/3202570164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c0b2e90 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/3202570164 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c13dc10 0x7f5d5c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 -- 172.21.15.72:0/3202570164 >> 172.21.15.72:0/3202570164 conn(0x7f5d5c0cf8a0 msgr2=0x7f5d5c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 -- 172.21.15.72:0/3202570164 shutdown_connections 2024-03-06T07:28:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.901+0000 7f5d61fd0640 1 -- 172.21.15.72:0/3202570164 wait complete. 2024-03-06T07:28:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.902+0000 7f5d61fd0640 1 Processor -- start 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 -- start start 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5c13dc10 0x7f5d5c1376c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c15e850 0x7f5d5c151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c0cd7a0 con 0x7f5d5c15bf30 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c13dff0 con 0x7f5d5c13dc10 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d61fd0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c0cd920 con 0x7f5d5c15e850 2024-03-06T07:28:33.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d52ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d52ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59092/0 (socket says 172.21.15.72:59092) 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.903+0000 7f5d52ffd640 1 -- 172.21.15.72:0/713357886 learned_addr learned my addr 172.21.15.72:0/713357886 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d5bfff640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c15e850 0x7f5d5c151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d5b7fe640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5c13dc10 0x7f5d5c1376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d52ffd640 1 -- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c15e850 msgr2=0x7f5d5c151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d52ffd640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c15e850 0x7f5d5c151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d52ffd640 1 -- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5c13dc10 msgr2=0x7f5d5c1376c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d52ffd640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5c13dc10 0x7f5d5c1376c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d52ffd640 1 -- 172.21.15.72:0/713357886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c15a740 con 0x7f5d5c15bf30 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d5bfff640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c15e850 0x7f5d5c151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d5b7fe640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5c13dc10 0x7f5d5c1376c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d52ffd640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c155b90 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f5d44034040 tx=0x7f5d440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44032840 con 0x7f5d5c15bf30 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d44032a00 con 0x7f5d5c15bf30 2024-03-06T07:28:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c159420 con 0x7f5d5c15bf30 2024-03-06T07:28:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44032df0 con 0x7f5d5c15bf30 2024-03-06T07:28:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.904+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c159c60 con 0x7f5d5c15bf30 2024-03-06T07:28:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.905+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c04f330 con 0x7f5d5c15bf30 2024-03-06T07:28:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.906+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5d4404e430 con 0x7f5d5c15bf30 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.906+0000 7f5d58ff9640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5d3c06b4d0 0x7f5d3c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.906+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5d4402f030 con 0x7f5d5c15bf30 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.906+0000 7f5d5b7fe640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5d3c06b4d0 0x7f5d3c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:33.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.906+0000 7f5d5b7fe640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5d3c06b4d0 0x7f5d3c06d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c006fd0 tx=0x7f5d4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:33.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:33.910+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5d5c159420 con 0x7f5d5c15bf30 2024-03-06T07:28:34.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.145+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5d5c06eb00 con 0x7f5d5c15bf30 2024-03-06T07:28:34.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.145+0000 7f5d58ff9640 1 -- 172.21.15.72:0/713357886 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5d44040070 con 0x7f5d5c15bf30 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl6rfTNBAA9lymz+Z5K5KwboqfeRM/Rg==", 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.148 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyfiSBRAA9Y5oSfKZbz3g335/xS9AWg==", 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-03-06T07:28:34.149 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehllICXABAASUBFdRvm2E7KticIuTin6Q==", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehluw/6DBAAhMhz4f18PvYDwHLVFjf76A==", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.150 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBXGehl176AMRAAcbScP/rlITiN3hkxon15Kg==", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBaGehll8CuLxAA491n+oIJQ3zARROjOFuFYw==", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.151 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBdGehl0lg1LhAAGBN2pCT3zcw+0bk7Di2apg==", 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.152 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBgGehldrZ2LBAAQc+gHPgQMPpcP8zzitvv2w==", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBmGehlmOqDNhAA9RczIxlZ7BEZpn7SA2d7Ag==", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.153 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBpGehlPHG6NBAAgqwMTfatdL4kkAxRzef0cQ==", 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBsGehlbcQ3MhAAhyQ0bJOqbPRq8LXZivHyrg==", 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.154 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBvGehlFhHgLxAAPNy8FE31WHWP1UgpLssmiQ==", 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.155 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw==", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow r", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlwNJsGhAA5W/DdAlDCK2Oe++4sdJjdg==", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:28:34.156 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehla4ADFxAALCAb24ynBI7laBRU4pB+6Q==", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBUGehlfndAIBAARvC5xnhtH3vre5iPQiYttQ==", 2024-03-06T07:28:34.157 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl/oAuGBAAlsyuFTAQVw0aAqG60+Pa7Q==", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.158 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehlMJNOMRAAlzaQZRjjUQAG2p+ymBqvBw==", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTGehl7G+WLRAA7Iw6UF08CmwV1kxmbrSLng==", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-03-06T07:28:34.159 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-03-06T07:28:34.160 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-03-06T07:28:34.160 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.160 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:34.160 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-06T07:28:34.160 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5d3c06b4d0 msgr2=0x7f5d3c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5d3c06b4d0 0x7f5d3c06d990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c006fd0 tx=0x7f5d4c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 msgr2=0x7f5d5c155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c155b90 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f5d44034040 tx=0x7f5d440046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 shutdown_connections 2024-03-06T07:28:34.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5d3c06b4d0 0x7f5d3c06d990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d5c15e850 0x7f5d5c151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d5c15bf30 0x7f5d5c155b90 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 --2- 172.21.15.72:0/713357886 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5c13dc10 0x7f5d5c1376c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.147+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 >> 172.21.15.72:0/713357886 conn(0x7f5d5c0cf8a0 msgr2=0x7f5d5c06afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.148+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 shutdown_connections 2024-03-06T07:28:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.148+0000 7f5d61fd0640 1 -- 172.21.15.72:0/713357886 wait complete. 2024-03-06T07:28:34.162 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:34.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.259+0000 7f1407fd7640 1 Processor -- start 2024-03-06T07:28:34.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.260+0000 7f1407fd7640 1 -- start start 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.260+0000 7f1407fd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 0x7f14000ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.260+0000 7f1407fd7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14000cd450 con 0x7f14000b4f70 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.260+0000 7f1407fd7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14000cd730 con 0x7f140006c6d0 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.260+0000 7f1407fd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140015e9f0 con 0x7f1400066d90 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1405d73640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f14000b4f70 0x7f14000b5350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59112/0 (socket says 172.21.15.72:59112) 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1405d73640 1 -- 172.21.15.72:0/1550674741 learned_addr learned my addr 172.21.15.72:0/1550674741 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139904459 0 0) 0x7f14000cd450 con 0x7f14000b4f70 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 0x7f14000ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec003520 con 0x7f14000b4f70 2024-03-06T07:28:34.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839996828 0 0) 0x7f13ec003520 con 0x7f14000b4f70 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 msgr2=0x7f14000ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 0x7f14000ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 >> v1:172.21.15.146:6789/0 conn(0x7f140006c6d0 legacy=0x7f140015ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140015e680 con 0x7f14000b4f70 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f13f0001670 con 0x7f14000b4f70 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 >> v1:172.21.15.72:6789/0 conn(0x7f14000b4f70 legacy=0x7f14000b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13ec0039e0 0x7f13ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 0x7f13ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec004d40 0x7f13ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140015e9f0 con 0x7f13ec004d40 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec00bb50 con 0x7f13ec0039e0 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec00bd50 con 0x7f1400066d90 2024-03-06T07:28:34.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec004d40 0x7f13ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1405d73640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13ec0039e0 0x7f13ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1405572640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 0x7f13ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 -- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 msgr2=0x7f13ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1400066d90 0x7f13ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 -- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13ec0039e0 msgr2=0x7f13ec003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13ec0039e0 0x7f13ec003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.261+0000 7f1406574640 1 -- 172.21.15.72:0/1550674741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec003520 con 0x7f13ec004d40 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.262+0000 7f1406574640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec004d40 0x7f13ec0051c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f13f8007330 tx=0x7f13ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9714e84fdeee7da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.262+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8039070 con 0x7f13ec004d40 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.262+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f802cca0 con 0x7f13ec004d40 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.262+0000 7f1404d71640 1 -- 172.21.15.72:0/1550674741 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f8009be0 con 0x7f13ec004d40 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1407fd7640 1 -- 172.21.15.72:0/1550674741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec004d40 msgr2=0x7f13ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1407fd7640 1 --2- 172.21.15.72:0/1550674741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec004d40 0x7f13ec0051c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f13f8007330 tx=0x7f13ec005830 comp rx=0 tx=0).stop 2024-03-06T07:28:34.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1406574640 1 -- 172.21.15.72:0/1550674741 reap_dead start 2024-03-06T07:28:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1407fd7640 1 -- 172.21.15.72:0/1550674741 shutdown_connections 2024-03-06T07:28:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1407fd7640 1 -- 172.21.15.72:0/1550674741 >> 172.21.15.72:0/1550674741 conn(0x7f14000724d0 msgr2=0x7f14000728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1407fd7640 1 -- 172.21.15.72:0/1550674741 shutdown_connections 2024-03-06T07:28:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.263+0000 7f1407fd7640 1 -- 172.21.15.72:0/1550674741 wait complete. 2024-03-06T07:28:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.264+0000 7f1407fd7640 1 Processor -- start 2024-03-06T07:28:34.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.264+0000 7f1407fd7640 1 -- start start 2024-03-06T07:28:34.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13ec004d40 0x7f14001620d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 0x7f140015f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f140014a4f0 0x7f140006e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14000ccf60 con 0x7f14000cdcf0 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14000ccde0 con 0x7f140014a4f0 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14000cd0e0 con 0x7f13ec004d40 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 0x7f140015f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 0x7f140015f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59114/0 (socket says 172.21.15.72:59114) 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 -- 172.21.15.72:0/56239880 learned_addr learned my addr 172.21.15.72:0/56239880 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1406574640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f140014a4f0 0x7f140006e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405d73640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13ec004d40 0x7f14001620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 -- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13ec004d40 msgr2=0x7f14001620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13ec004d40 0x7f14001620d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 -- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f140014a4f0 msgr2=0x7f140006e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f140014a4f0 0x7f140006e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 -- 172.21.15.72:0/56239880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140014c000 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1406574640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f140014a4f0 0x7f140006e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405d73640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13ec004d40 0x7f14001620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1405572640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 0x7f140015f5f0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00f850 tx=0x7f13fc00fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc00fe40 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc018070 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140014c2e0 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.265+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc00d730 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.266+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140015fb30 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.267+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140004f330 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.267+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f13fc00d8d0 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.267+0000 7f13f67fc640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f13e406b4d0 0x7f13e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.267+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f13fc089cc0 con 0x7f14000cdcf0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.267+0000 7f1405d73640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f13e406b4d0 0x7f13e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.268+0000 7f1405d73640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f13e406b4d0 0x7f13e406d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f140015bdf0 tx=0x7f13f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.272+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f13fc058f20 con 0x7f14000cdcf0 2024-03-06T07:28:34.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.444+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f140006dd50 con 0x7f14000cdcf0 2024-03-06T07:28:34.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.445+0000 7f13f67fc640 1 -- 172.21.15.72:0/56239880 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v123) v1 ==== 74+0+12913 (secure 0 0 0) 0x7f13fc058820 con 0x7f14000cdcf0 2024-03-06T07:28:34.447 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:34.447 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":123,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:28:33.783338+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f13e406b4d0 msgr2=0x7f13e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f13e406b4d0 0x7f13e406d990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f140015bdf0 tx=0x7f13f0009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 msgr2=0x7f140015f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 0x7f140015f5f0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00f850 tx=0x7f13fc00fd10 comp rx=0 tx=0).stop 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 shutdown_connections 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f13e406b4d0 0x7f13e406d990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f140014a4f0 0x7f140006e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14000cdcf0 0x7f140015f5f0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 --2- 172.21.15.72:0/56239880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13ec004d40 0x7f14001620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 >> 172.21.15.72:0/56239880 conn(0x7f14000724d0 msgr2=0x7f14000b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 shutdown_connections 2024-03-06T07:28:34.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.447+0000 7f1407fd7640 1 -- 172.21.15.72:0/56239880 wait complete. 2024-03-06T07:28:34.461 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:28:34.461 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:28:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.562+0000 7f9f945c9640 1 Processor -- start 2024-03-06T07:28:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.562+0000 7f9f945c9640 1 -- start start 2024-03-06T07:28:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f945c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cfa90 0x7f9f8c0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f945c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f8c0b33a0 0x7f9f8c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f945c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c158ac0 con 0x7f9f8c1586e0 2024-03-06T07:28:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f945c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c15b2a0 con 0x7f9f8c0b33a0 2024-03-06T07:28:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f945c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c15b540 con 0x7f9f8c0cfa90 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f92b66640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f8c0b33a0 0x7f9f8c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f92365640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f9f8c1586e0 0x7f9f8c0aea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:36386/0 (socket says 172.21.15.72:36386) 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f91b64640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cfa90 0x7f9f8c0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f92365640 1 -- 172.21.15.72:0/2148194230 learned_addr learned my addr 172.21.15.72:0/2148194230 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.563+0000 7f9f92b66640 1 -- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cfa90 msgr2=0x7f9f8c0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.564+0000 7f9f92b66640 1 --2- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cfa90 0x7f9f8c0cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.564+0000 7f9f92b66640 1 -- 172.21.15.72:0/2148194230 >> v1:172.21.15.146:6789/0 conn(0x7f9f8c1586e0 legacy=0x7f9f8c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.564+0000 7f9f91363640 1 -- 172.21.15.72:0/2148194230 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006950460 0 0) 0x7f9f8c158ac0 con 0x7f9f8c1586e0 2024-03-06T07:28:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.564+0000 7f9f92b66640 1 -- 172.21.15.72:0/2148194230 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c15b820 con 0x7f9f8c0b33a0 2024-03-06T07:28:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.564+0000 7f9f92b66640 1 --2- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0b33a0 0x7f9f8c0b37f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009d90 tx=0x7f9f8402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b9d10d91b07f26a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f91363640 1 -- 172.21.15.72:0/2148194230 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84039070 con 0x7f9f8c0b33a0 2024-03-06T07:28:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f91363640 1 -- 172.21.15.72:0/2148194230 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8402cc50 con 0x7f9f8c0b33a0 2024-03-06T07:28:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f91363640 1 -- 172.21.15.72:0/2148194230 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84034440 con 0x7f9f8c0b33a0 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 -- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0b33a0 msgr2=0x7f9f8c0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 --2- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0b33a0 0x7f9f8c0b37f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009d90 tx=0x7f9f8402bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 -- 172.21.15.72:0/2148194230 shutdown_connections 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 --2- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0b33a0 0x7f9f8c0b37f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 --2- 172.21.15.72:0/2148194230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cfa90 0x7f9f8c0cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 -- 172.21.15.72:0/2148194230 >> 172.21.15.72:0/2148194230 conn(0x7f9f8c06b190 msgr2=0x7f9f8c06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 -- 172.21.15.72:0/2148194230 shutdown_connections 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.565+0000 7f9f945c9640 1 -- 172.21.15.72:0/2148194230 wait complete. 2024-03-06T07:28:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.566+0000 7f9f945c9640 1 Processor -- start 2024-03-06T07:28:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.566+0000 7f9f945c9640 1 -- start start 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f945c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0b33a0 0x7f9f8c1f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f945c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0cfa90 0x7f9f8c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f945c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 0x7f9f8c0b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f945c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0af150 con 0x7f9f8c1586e0 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f945c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c15bde0 con 0x7f9f8c0cfa90 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f945c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c071350 con 0x7f9f8c0b33a0 2024-03-06T07:28:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 0x7f9f8c0b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 0x7f9f8c0b3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59118/0 (socket says 172.21.15.72:59118) 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 -- 172.21.15.72:0/593408958 learned_addr learned my addr 172.21.15.72:0/593408958 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f91b64640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0cfa90 0x7f9f8c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92365640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0b33a0 0x7f9f8c1f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 -- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0b33a0 msgr2=0x7f9f8c1f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0b33a0 0x7f9f8c1f4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 -- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0cfa90 msgr2=0x7f9f8c06d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.567+0000 7f9f92b66640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0cfa90 0x7f9f8c06d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f92b66640 1 -- 172.21.15.72:0/593408958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c06ddc0 con 0x7f9f8c1586e0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f92b66640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 0x7f9f8c0b3780 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009d90 tx=0x7f9f84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84044020 con 0x7f9f8c1586e0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8403a040 con 0x7f9f8c1586e0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84039050 con 0x7f9f8c1586e0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c0c47e0 con 0x7f9f8c1586e0 2024-03-06T07:28:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.568+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c0c4da0 con 0x7f9f8c1586e0 2024-03-06T07:28:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.569+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c0732b0 con 0x7f9f8c1586e0 2024-03-06T07:28:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.569+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9f84003990 con 0x7f9f8c1586e0 2024-03-06T07:28:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.570+0000 7f9f7effd640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f7006b4d0 0x7f9f7006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.570+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9f840ac270 con 0x7f9f8c1586e0 2024-03-06T07:28:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.570+0000 7f9f92365640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f7006b4d0 0x7f9f7006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.570+0000 7f9f92365640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f7006b4d0 0x7f9f7006d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9f88000f60 tx=0x7f9f88009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.574+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9f8404e060 con 0x7f9f8c1586e0 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.745+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f8c0ae490 con 0x7f9f8c1586e0 2024-03-06T07:28:34.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.746+0000 7f9f7effd640 1 -- 172.21.15.72:0/593408958 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v123) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f8403d360 con 0x7f9f8c1586e0 2024-03-06T07:28:34.748 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f7006b4d0 msgr2=0x7f9f7006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f7006b4d0 0x7f9f7006d990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9f88000f60 tx=0x7f9f88009040 comp rx=0 tx=0).stop 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 msgr2=0x7f9f8c0b3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 0x7f9f8c0b3780 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009d90 tx=0x7f9f84004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 shutdown_connections 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9f7006b4d0 0x7f9f7006d990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f8c1586e0 0x7f9f8c0b3780 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0cfa90 0x7f9f8c06d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 --2- 172.21.15.72:0/593408958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0b33a0 0x7f9f8c1f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 >> 172.21.15.72:0/593408958 conn(0x7f9f8c06b190 msgr2=0x7f9f8c157960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 shutdown_connections 2024-03-06T07:28:34.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.747+0000 7f9f945c9640 1 -- 172.21.15.72:0/593408958 wait complete. 2024-03-06T07:28:34.761 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-06T07:28:34.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.859+0000 7f9b35bb5640 1 Processor -- start 2024-03-06T07:28:34.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.859+0000 7f9b35bb5640 1 -- start start 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b35bb5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9b300d00c0 0x7f9b300d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b35bb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 0x7f9b300cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b35bb5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b300d09e0 con 0x7f9b300ce1b0 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b35bb5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3006ca90 con 0x7f9b300d00c0 2024-03-06T07:28:34.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b35bb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3006df30 con 0x7f9b300cf850 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b2effd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 0x7f9b300cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b2f7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9b300d00c0 0x7f9b300d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b2effd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 0x7f9b300cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:51998/0 (socket says 172.21.15.72:51998) 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b2effd640 1 -- 172.21.15.72:0/2939541901 learned_addr learned my addr 172.21.15.72:0/2939541901 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.860+0000 7f9b2ffff640 1 --1- 172.21.15.72:0/2939541901 >> v1:172.21.15.146:6789/0 conn(0x7f9b300ce1b0 0x7f9b300ce640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:36394/0 (socket says 172.21.15.72:36394) 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.861+0000 7f9b2effd640 1 -- 172.21.15.72:0/2939541901 >> v2:172.21.15.146:3300/0 conn(0x7f9b300d00c0 msgr2=0x7f9b300d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.861+0000 7f9b2effd640 1 --2- 172.21.15.72:0/2939541901 >> v2:172.21.15.146:3300/0 conn(0x7f9b300d00c0 0x7f9b300d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.861+0000 7f9b2effd640 1 -- 172.21.15.72:0/2939541901 >> v1:172.21.15.146:6789/0 conn(0x7f9b300ce1b0 legacy=0x7f9b300ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.861+0000 7f9b2effd640 1 -- 172.21.15.72:0/2939541901 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3015e380 con 0x7f9b300cf850 2024-03-06T07:28:34.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.861+0000 7f9b2f7fe640 1 --2- 172.21.15.72:0/2939541901 >> v2:172.21.15.146:3300/0 conn(0x7f9b300d00c0 0x7f9b300d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b2effd640 1 --2- 172.21.15.72:0/2939541901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 0x7f9b300cd3c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009870 tx=0x7f9b2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=251de83d087d0201 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b2e7fc640 1 -- 172.21.15.72:0/2939541901 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b24034070 con 0x7f9b300cf850 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b2e7fc640 1 -- 172.21.15.72:0/2939541901 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b24030840 con 0x7f9b300cf850 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b2e7fc640 1 -- 172.21.15.72:0/2939541901 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b24030c10 con 0x7f9b300cf850 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 -- 172.21.15.72:0/2939541901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 msgr2=0x7f9b300cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/2939541901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 0x7f9b300cd3c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009870 tx=0x7f9b2402b980 comp rx=0 tx=0).stop 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 -- 172.21.15.72:0/2939541901 shutdown_connections 2024-03-06T07:28:34.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/2939541901 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300cf850 0x7f9b300cd3c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/2939541901 >> v2:172.21.15.146:3300/0 conn(0x7f9b300d00c0 0x7f9b300d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 -- 172.21.15.72:0/2939541901 >> 172.21.15.72:0/2939541901 conn(0x7f9b300b35a0 msgr2=0x7f9b300b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:34.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 -- 172.21.15.72:0/2939541901 shutdown_connections 2024-03-06T07:28:34.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.862+0000 7f9b35bb5640 1 -- 172.21.15.72:0/2939541901 wait complete. 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.863+0000 7f9b35bb5640 1 Processor -- start 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 -- start start 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300ce1b0 0x7f9b30150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b300cf850 0x7f9b300ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 0x7f9b300cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3006c440 con 0x7f9b300d00c0 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3006c160 con 0x7f9b300cf850 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.864+0000 7f9b35bb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3006c720 con 0x7f9b300ce1b0 2024-03-06T07:28:34.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 0x7f9b300cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 0x7f9b300cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59128/0 (socket says 172.21.15.72:59128) 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 -- 172.21.15.72:0/3601917484 learned_addr learned my addr 172.21.15.72:0/3601917484 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2f7fe640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300ce1b0 0x7f9b30150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2effd640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b300cf850 0x7f9b300ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 -- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300ce1b0 msgr2=0x7f9b30150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300ce1b0 0x7f9b30150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 -- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b300cf850 msgr2=0x7f9b300ca640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b300cf850 0x7f9b300ca640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 -- 172.21.15.72:0/3601917484 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300b5380 con 0x7f9b300d00c0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b2ffff640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 0x7f9b300cab80 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800c400 tx=0x7f9b2800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280074e0 con 0x7f9b300d00c0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b28007680 con 0x7f9b300d00c0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b300b5660 con 0x7f9b300d00c0 2024-03-06T07:28:34.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b280049e0 con 0x7f9b300d00c0 2024-03-06T07:28:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.865+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b300b5c20 con 0x7f9b300d00c0 2024-03-06T07:28:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.866+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3004f330 con 0x7f9b300d00c0 2024-03-06T07:28:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.867+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b2801a020 con 0x7f9b300d00c0 2024-03-06T07:28:34.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.867+0000 7f9b17fff640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1c06b4d0 0x7f9b1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:34.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.868+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9b280894d0 con 0x7f9b300d00c0 2024-03-06T07:28:34.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.868+0000 7f9b2f7fe640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1c06b4d0 0x7f9b1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:34.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.868+0000 7f9b2f7fe640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1c06b4d0 0x7f9b1c06d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009bd0 tx=0x7f9b2000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:34.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:34.872+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b280580e0 con 0x7f9b300d00c0 2024-03-06T07:28:35.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.045+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9b300ced90 con 0x7f9b300d00c0 2024-03-06T07:28:35.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.045+0000 7f9b17fff640 1 -- 172.21.15.72:0/3601917484 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v123) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9b2805c180 con 0x7f9b300d00c0 2024-03-06T07:28:35.047 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:35.047 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-03-06T07:28:35.047 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-03-06T07:28:35.048 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-03-06T07:28:35.048 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-03-06T07:28:35.048 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:35.048 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-03-06T07:28:35.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1c06b4d0 msgr2=0x7f9b1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1c06b4d0 0x7f9b1c06d990 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9b20009bd0 tx=0x7f9b2000b040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 msgr2=0x7f9b300cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 0x7f9b300cab80 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800c400 tx=0x7f9b2800c8c0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 shutdown_connections 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b1c06b4d0 0x7f9b1c06d990 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b300d00c0 0x7f9b300cab80 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b300cf850 0x7f9b300ca640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 --2- 172.21.15.72:0/3601917484 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b300ce1b0 0x7f9b30150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 >> 172.21.15.72:0/3601917484 conn(0x7f9b300b35a0 msgr2=0x7f9b3006fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 shutdown_connections 2024-03-06T07:28:35.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.047+0000 7f9b35bb5640 1 -- 172.21.15.72:0/3601917484 wait complete. 2024-03-06T07:28:35.060 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-06T07:28:35.060 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-06T07:28:35.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.162+0000 7f4d5e932640 1 Processor -- start 2024-03-06T07:28:35.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.162+0000 7f4d5e932640 1 -- start start 2024-03-06T07:28:35.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.162+0000 7f4d5e932640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d580b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.162+0000 7f4d5e932640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4d580b5ca0 0x7f4d5806c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.162+0000 7f4d5e932640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4d58072590 0x7f4d58072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d5e932640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d5815edf0 con 0x7f4d580b5ca0 2024-03-06T07:28:35.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d5e932640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d581618a0 con 0x7f4d58072590 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d5e932640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5806c7a0 con 0x7f4d5806be80 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4d580b5ca0 0x7f4d5806c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4d580b5ca0 0x7f4d5806c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59130/0 (socket says 172.21.15.72:59130) 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 -- 172.21.15.72:0/3811730214 learned_addr learned my addr 172.21.15.72:0/3811730214 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 -- 172.21.15.72:0/3811730214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 msgr2=0x7f4d580b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d5cecf640 1 --2- 172.21.15.72:0/3811730214 >> v2:172.21.15.146:3300/0 conn(0x7f4d58072590 0x7f4d58072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d57fff640 1 --2- 172.21.15.72:0/3811730214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d580b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 --2- 172.21.15.72:0/3811730214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d580b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 -- 172.21.15.72:0/3811730214 >> v2:172.21.15.146:3300/0 conn(0x7f4d58072590 msgr2=0x7f4d58072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 --2- 172.21.15.72:0/3811730214 >> v2:172.21.15.146:3300/0 conn(0x7f4d58072590 0x7f4d58072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.163+0000 7f4d577fe640 1 -- 172.21.15.72:0/3811730214 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58068b00 con 0x7f4d580b5ca0 2024-03-06T07:28:35.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d577fe640 1 --2- 172.21.15.72:0/3811730214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d580b5ca0 0x7f4d5806c260 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f4d58072150 tx=0x7f4d4802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3b0bbc9416a07d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d56ffd640 1 -- 172.21.15.72:0/3811730214 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4802e070 con 0x7f4d580b5ca0 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d56ffd640 1 -- 172.21.15.72:0/3811730214 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d48037070 con 0x7f4d580b5ca0 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d56ffd640 1 -- 172.21.15.72:0/3811730214 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48031a00 con 0x7f4d580b5ca0 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 -- 172.21.15.72:0/3811730214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d580b5ca0 msgr2=0x7f4d5806c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3811730214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d580b5ca0 0x7f4d5806c260 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f4d58072150 tx=0x7f4d4802b880 comp rx=0 tx=0).stop 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 -- 172.21.15.72:0/3811730214 shutdown_connections 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3811730214 >> v2:172.21.15.146:3300/0 conn(0x7f4d58072590 0x7f4d58072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3811730214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d580b5ca0 0x7f4d5806c260 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3811730214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d580b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 -- 172.21.15.72:0/3811730214 >> 172.21.15.72:0/3811730214 conn(0x7f4d580b4ed0 msgr2=0x7f4d580b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 -- 172.21.15.72:0/3811730214 shutdown_connections 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.164+0000 7f4d5e932640 1 -- 172.21.15.72:0/3811730214 wait complete. 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 Processor -- start 2024-03-06T07:28:35.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 -- start start 2024-03-06T07:28:35.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d58150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 0x7f4d580cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d580b5ca0 0x7f4d580ce210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5815e7a0 con 0x7f4d58072590 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d580b1900 con 0x7f4d580b5ca0 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d5e932640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5815ea80 con 0x7f4d5806be80 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.166+0000 7f4d577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 0x7f4d580cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 0x7f4d580cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59144/0 (socket says 172.21.15.72:59144) 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 -- 172.21.15.72:0/3882096633 learned_addr learned my addr 172.21.15.72:0/3882096633 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d57fff640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d58150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d5cecf640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d580b5ca0 0x7f4d580ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 -- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 msgr2=0x7f4d58150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d58150510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 -- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d580b5ca0 msgr2=0x7f4d580ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d580b5ca0 0x7f4d580ce210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 -- 172.21.15.72:0/3882096633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d580d0c50 con 0x7f4d58072590 2024-03-06T07:28:35.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d5cecf640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d580b5ca0 0x7f4d580ce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d577fe640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 0x7f4d580cd4f0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f4d48000c00 tx=0x7f4d480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48032880 con 0x7f4d58072590 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d48032a40 con 0x7f4d58072590 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d48032e30 con 0x7f4d58072590 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d580d0970 con 0x7f4d58072590 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.167+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d580c9800 con 0x7f4d58072590 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.168+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d5804f330 con 0x7f4d58072590 2024-03-06T07:28:35.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.169+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4d480453f0 con 0x7f4d58072590 2024-03-06T07:28:35.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.169+0000 7f4d54ff9640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d4406b4d0 0x7f4d4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.169+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4d480abea0 con 0x7f4d58072590 2024-03-06T07:28:35.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.169+0000 7f4d57fff640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d4406b4d0 0x7f4d4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.170+0000 7f4d57fff640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d4406b4d0 0x7f4d4406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00a9b0 tx=0x7f4d4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.173+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4d48040070 con 0x7f4d58072590 2024-03-06T07:28:35.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.344+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4d5815bb50 con 0x7f4d58072590 2024-03-06T07:28:35.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.787+0000 7f4d54ff9640 1 -- 172.21.15.72:0/3882096633 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v124) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4d4807b100 con 0x7f4d58072590 2024-03-06T07:28:35.789 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d4406b4d0 msgr2=0x7f4d4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d4406b4d0 0x7f4d4406d990 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00a9b0 tx=0x7f4d4c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 msgr2=0x7f4d580cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 0x7f4d580cd4f0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f4d48000c00 tx=0x7f4d480046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 shutdown_connections 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d4406b4d0 0x7f4d4406d990 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4d580b5ca0 0x7f4d580ce210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4d58072590 0x7f4d580cd4f0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 --2- 172.21.15.72:0/3882096633 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4d5806be80 0x7f4d58150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 >> 172.21.15.72:0/3882096633 conn(0x7f4d580b4ed0 msgr2=0x7f4d580b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 shutdown_connections 2024-03-06T07:28:35.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.790+0000 7f4d5e932640 1 -- 172.21.15.72:0/3882096633 wait complete. 2024-03-06T07:28:35.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-06T07:28:35.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.912+0000 7f3d35615640 1 Processor -- start 2024-03-06T07:28:35.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.912+0000 7f3d35615640 1 -- start start 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d35615640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300ad720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d35615640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d3006c5b0 0x7f3d300b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d35615640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d300b0f70 con 0x7f3d300b0760 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d35615640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d300b0bd0 con 0x7f3d3006c5b0 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d35615640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3015c520 con 0x7f3d3015aec0 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d2e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d3006c5b0 0x7f3d300b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d2f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3d300b0760 0x7f3d3006b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59120/0 (socket says 172.21.15.72:59120) 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.913+0000 7f3d2f7fe640 1 -- 172.21.15.72:0/2423221260 learned_addr learned my addr 172.21.15.72:0/2423221260 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2effd640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/2423221260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 msgr2=0x7f3d300ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:35.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300ad720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/2423221260 >> v1:172.21.15.72:6789/0 conn(0x7f3d300b0760 legacy=0x7f3d3006b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2dffb640 1 -- 172.21.15.72:0/2423221260 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347801255 0 0) 0x7f3d300b0f70 con 0x7f3d300b0760 2024-03-06T07:28:35.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/2423221260 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d300adc60 con 0x7f3d3006c5b0 2024-03-06T07:28:35.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2effd640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d300b0220 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f3d240097d0 tx=0x7f3d2402b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bab125baa12425 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.914+0000 7f3d2dffb640 1 -- 172.21.15.72:0/2423221260 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24035020 con 0x7f3d3006c5b0 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d2dffb640 1 -- 172.21.15.72:0/2423221260 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d24031800 con 0x7f3d3006c5b0 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d2dffb640 1 -- 172.21.15.72:0/2423221260 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24031bd0 con 0x7f3d3006c5b0 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 -- 172.21.15.72:0/2423221260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 msgr2=0x7f3d300b0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d300b0220 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f3d240097d0 tx=0x7f3d2402b8a0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 -- 172.21.15.72:0/2423221260 shutdown_connections 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d300b0220 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 --2- 172.21.15.72:0/2423221260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 -- 172.21.15.72:0/2423221260 >> 172.21.15.72:0/2423221260 conn(0x7f3d300b4fe0 msgr2=0x7f3d300b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:35.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 -- 172.21.15.72:0/2423221260 shutdown_connections 2024-03-06T07:28:35.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.915+0000 7f3d35615640 1 -- 172.21.15.72:0/2423221260 wait complete. 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.916+0000 7f3d35615640 1 Processor -- start 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.916+0000 7f3d35615640 1 -- start start 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d35615640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d30147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d35615640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300b0760 0x7f3d30157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d35615640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d35615640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3006b150 con 0x7f3d3006c5b0 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d35615640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3015c240 con 0x7f3d300b0760 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d35615640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d300b5db0 con 0x7f3d3015aec0 2024-03-06T07:28:35.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d30147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d30147490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59164/0 (socket says 172.21.15.72:59164) 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 -- 172.21.15.72:0/1241405655 learned_addr learned my addr 172.21.15.72:0/1241405655 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2f7fe640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300b0760 0x7f3d30157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 -- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 msgr2=0x7f3d300997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 -- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300b0760 msgr2=0x7f3d30157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.917+0000 7f3d2effd640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300b0760 0x7f3d30157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d2effd640 1 -- 172.21.15.72:0/1241405655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30147b20 con 0x7f3d3006c5b0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300b0760 0x7f3d30157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d2effd640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d30147490 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000ce00 tx=0x7f3d2000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2000f890 con 0x7f3d3006c5b0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2000fa30 con 0x7f3d3006c5b0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d30147e00 con 0x7f3d3006c5b0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2000fe40 con 0x7f3d3006c5b0 2024-03-06T07:28:35.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.918+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d301483c0 con 0x7f3d3006c5b0 2024-03-06T07:28:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.919+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3009e4e0 con 0x7f3d3006c5b0 2024-03-06T07:28:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.920+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d20018400 con 0x7f3d3006c5b0 2024-03-06T07:28:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.920+0000 7f3d177fe640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d1c06b4d0 0x7f3d1c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.920+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3d2008aef0 con 0x7f3d3006c5b0 2024-03-06T07:28:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.920+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d1c06b4d0 0x7f3d1c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:35.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.920+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d1c06b4d0 0x7f3d1c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3d24036000 tx=0x7f3d24008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:35.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:35.924+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d20014030 con 0x7f3d3006c5b0 2024-03-06T07:28:36.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.096+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3d30068ff0 con 0x7f3d3006c5b0 2024-03-06T07:28:36.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.793+0000 7f3d177fe640 1 -- 172.21.15.72:0/1241405655 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v125) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3d2000ae20 con 0x7f3d3006c5b0 2024-03-06T07:28:36.795 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-03-06T07:28:36.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d1c06b4d0 msgr2=0x7f3d1c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d1c06b4d0 0x7f3d1c06d990 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3d24036000 tx=0x7f3d24008030 comp rx=0 tx=0).stop 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 msgr2=0x7f3d30147490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d30147490 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000ce00 tx=0x7f3d2000a650 comp rx=0 tx=0).stop 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 shutdown_connections 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d1c06b4d0 0x7f3d1c06d990 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d3015aec0 0x7f3d300997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300b0760 0x7f3d30157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 --2- 172.21.15.72:0/1241405655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3006c5b0 0x7f3d30147490 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 >> 172.21.15.72:0/1241405655 conn(0x7f3d300b4fe0 msgr2=0x7f3d30072ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 shutdown_connections 2024-03-06T07:28:36.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.796+0000 7f3d35615640 1 -- 172.21.15.72:0/1241405655 wait complete. 2024-03-06T07:28:36.824 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-06T07:28:36.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.926+0000 7fad517c9640 1 Processor -- start 2024-03-06T07:28:36.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.926+0000 7fad517c9640 1 -- start start 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad517c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad4c15bc50 0x7fad4c0cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad517c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fad4c0cc280 0x7fad4c0cf970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad517c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c0aee20 0x7fad4c15f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad517c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c0b4850 con 0x7fad4c15bc50 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad517c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4c06d5b0 con 0x7fad4c0cc280 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad517c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c0cc7b0 con 0x7fad4c0aee20 2024-03-06T07:28:36.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad4c15bc50 0x7fad4c0cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad4c15bc50 0x7fad4c0cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59170/0 (socket says 172.21.15.72:59170) 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 -- 172.21.15.72:0/609837417 learned_addr learned my addr 172.21.15.72:0/609837417 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4b7fe640 1 --2- 172.21.15.72:0/609837417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c0aee20 0x7fad4c15f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4a7fc640 1 --2- 172.21.15.72:0/609837417 >> v2:172.21.15.146:3300/0 conn(0x7fad4c0cc280 0x7fad4c0cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 -- 172.21.15.72:0/609837417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c0aee20 msgr2=0x7fad4c15f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 --2- 172.21.15.72:0/609837417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c0aee20 0x7fad4c15f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 -- 172.21.15.72:0/609837417 >> v2:172.21.15.146:3300/0 conn(0x7fad4c0cc280 msgr2=0x7fad4c0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 --2- 172.21.15.72:0/609837417 >> v2:172.21.15.146:3300/0 conn(0x7fad4c0cc280 0x7fad4c0cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.927+0000 7fad4affd640 1 -- 172.21.15.72:0/609837417 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c0b4ce0 con 0x7fad4c15bc50 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.928+0000 7fad4affd640 1 --2- 172.21.15.72:0/609837417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c15bc50 0x7fad4c0cf430 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0cd6e0 tx=0x7fad3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa8e1e3a06a5b80 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.928+0000 7fad49ffb640 1 -- 172.21.15.72:0/609837417 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c02e070 con 0x7fad4c15bc50 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.928+0000 7fad49ffb640 1 -- 172.21.15.72:0/609837417 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c037070 con 0x7fad4c15bc50 2024-03-06T07:28:36.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.928+0000 7fad49ffb640 1 -- 172.21.15.72:0/609837417 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c031a00 con 0x7fad4c15bc50 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 -- 172.21.15.72:0/609837417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c15bc50 msgr2=0x7fad4c0cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 --2- 172.21.15.72:0/609837417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c15bc50 0x7fad4c0cf430 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0cd6e0 tx=0x7fad3c02b880 comp rx=0 tx=0).stop 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 -- 172.21.15.72:0/609837417 shutdown_connections 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 --2- 172.21.15.72:0/609837417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c0aee20 0x7fad4c15f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 --2- 172.21.15.72:0/609837417 >> v2:172.21.15.146:3300/0 conn(0x7fad4c0cc280 0x7fad4c0cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 --2- 172.21.15.72:0/609837417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c15bc50 0x7fad4c0cf430 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 -- 172.21.15.72:0/609837417 >> 172.21.15.72:0/609837417 conn(0x7fad4c06afb0 msgr2=0x7fad4c06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 -- 172.21.15.72:0/609837417 shutdown_connections 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.929+0000 7fad517c9640 1 -- 172.21.15.72:0/609837417 wait complete. 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.930+0000 7fad517c9640 1 Processor -- start 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.930+0000 7fad517c9640 1 -- start start 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 0x7fad4c143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad4c0cc280 0x7fad4c15ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c15bc50 0x7fad4c1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c15c030 con 0x7fad4c0aee20 2024-03-06T07:28:36.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c0ae900 con 0x7fad4c0cc280 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4c15c1b0 con 0x7fad4c15bc50 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 0x7fad4c143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 0x7fad4c143110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59186/0 (socket says 172.21.15.72:59186) 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 -- 172.21.15.72:0/865046382 learned_addr learned my addr 172.21.15.72:0/865046382 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4a7fc640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad4c0cc280 0x7fad4c15ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4b7fe640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c15bc50 0x7fad4c1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 -- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c15bc50 msgr2=0x7fad4c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c15bc50 0x7fad4c1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 -- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad4c0cc280 msgr2=0x7fad4c15ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad4c0cc280 0x7fad4c15ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 -- 172.21.15.72:0/865046382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c143810 con 0x7fad4c0aee20 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4b7fe640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c15bc50 0x7fad4c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:36.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad4affd640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 0x7fad4c143110 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fad3c000c00 tx=0x7fad3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c032670 con 0x7fad4c0aee20 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c032850 con 0x7fad4c0aee20 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4c143af0 con 0x7fad4c0aee20 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.931+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c032c40 con 0x7fad4c0aee20 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.932+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4c1440b0 con 0x7fad4c0aee20 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.933+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4c04f330 con 0x7fad4c0aee20 2024-03-06T07:28:36.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.933+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad3c0453f0 con 0x7fad4c0aee20 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.933+0000 7fad337fe640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.933+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fad3c0abee0 con 0x7fad4c0aee20 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.933+0000 7fad4a7fc640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:36.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.934+0000 7fad4a7fc640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0b3630 tx=0x7fad40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:36.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:36.938+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad3c040070 con 0x7fad4c0aee20 2024-03-06T07:28:37.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.145+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fad4c06e5b0 con 0x7fad4c0aee20 2024-03-06T07:28:37.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.795+0000 7fad337fe640 1 -- 172.21.15.72:0/865046382 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v77) v1 ==== 174+0+0 (secure 0 0 0) 0x7fad3c07a770 con 0x7fad4c0aee20 2024-03-06T07:28:37.797 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 18 and data pool 19 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad3806b4d0 msgr2=0x7fad3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad3806b4d0 0x7fad3806d990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0b3630 tx=0x7fad40008040 comp rx=0 tx=0).stop 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 msgr2=0x7fad4c143110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 0x7fad4c143110 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fad3c000c00 tx=0x7fad3c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 shutdown_connections 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad3806b4d0 0x7fad3806d990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad4c15bc50 0x7fad4c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad4c0cc280 0x7fad4c15ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 --2- 172.21.15.72:0/865046382 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad4c0aee20 0x7fad4c143110 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 >> 172.21.15.72:0/865046382 conn(0x7fad4c06afb0 msgr2=0x7fad4c0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.797+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 shutdown_connections 2024-03-06T07:28:37.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.798+0000 7fad517c9640 1 -- 172.21.15.72:0/865046382 wait complete. 2024-03-06T07:28:37.821 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:37.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.922+0000 7f53d9bf6640 1 Processor -- start 2024-03-06T07:28:37.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.922+0000 7f53d9bf6640 1 -- start start 2024-03-06T07:28:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d9bf6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 0x7f53d40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d9bf6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53d415bf30 0x7f53d40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d9bf6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d415f580 con 0x7f53d415e850 2024-03-06T07:28:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d9bf6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d40cf3f0 con 0x7f53d415bf30 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d9bf6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d415b210 con 0x7f53d413dc10 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d37fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 0x7f53d40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d37fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 0x7f53d40b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:52068/0 (socket says 172.21.15.72:52068) 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d2ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53d415bf30 0x7f53d40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d37fe640 1 -- 172.21.15.72:0/1988788557 learned_addr learned my addr 172.21.15.72:0/1988788557 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d37fe640 1 -- 172.21.15.72:0/1988788557 >> v2:172.21.15.146:3300/0 conn(0x7f53d415bf30 msgr2=0x7f53d40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d37fe640 1 --2- 172.21.15.72:0/1988788557 >> v2:172.21.15.146:3300/0 conn(0x7f53d415bf30 0x7f53d40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.923+0000 7f53d37fe640 1 -- 172.21.15.72:0/1988788557 >> v1:172.21.15.146:6789/0 conn(0x7f53d415e850 legacy=0x7f53d415eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.924+0000 7f53d37fe640 1 -- 172.21.15.72:0/1988788557 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d415b450 con 0x7f53d413dc10 2024-03-06T07:28:37.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.924+0000 7f53d37fe640 1 --2- 172.21.15.72:0/1988788557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 0x7f53d40b2950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c06189890596326d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.924+0000 7f53d27fc640 1 -- 172.21.15.72:0/1988788557 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4035020 con 0x7f53d413dc10 2024-03-06T07:28:37.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.924+0000 7f53d27fc640 1 -- 172.21.15.72:0/1988788557 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c4031640 con 0x7f53d413dc10 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.924+0000 7f53d27fc640 1 -- 172.21.15.72:0/1988788557 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c402d040 con 0x7f53d413dc10 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 -- 172.21.15.72:0/1988788557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 msgr2=0x7f53d40b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/1988788557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 0x7f53d40b2950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 -- 172.21.15.72:0/1988788557 shutdown_connections 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/1988788557 >> v2:172.21.15.146:3300/0 conn(0x7f53d415bf30 0x7f53d40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/1988788557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d413dc10 0x7f53d40b2950 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 -- 172.21.15.72:0/1988788557 >> 172.21.15.72:0/1988788557 conn(0x7f53d40cf8a0 msgr2=0x7f53d40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 -- 172.21.15.72:0/1988788557 shutdown_connections 2024-03-06T07:28:37.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.925+0000 7f53d9bf6640 1 -- 172.21.15.72:0/1988788557 wait complete. 2024-03-06T07:28:37.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.926+0000 7f53d9bf6640 1 Processor -- start 2024-03-06T07:28:37.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.926+0000 7f53d9bf6640 1 -- start start 2024-03-06T07:28:37.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d9bf6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 0x7f53d414f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d9bf6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53d415bf30 0x7f53d41500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d9bf6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d415e850 0x7f53d4150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d9bf6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40cd7a0 con 0x7f53d413dc10 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d9bf6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d413dff0 con 0x7f53d415bf30 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d9bf6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d40cd920 con 0x7f53d415e850 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 0x7f53d414f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 0x7f53d414f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59194/0 (socket says 172.21.15.72:59194) 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 -- 172.21.15.72:0/167764990 learned_addr learned my addr 172.21.15.72:0/167764990 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d3fff640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d415e850 0x7f53d4150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d2ffd640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53d415bf30 0x7f53d41500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 -- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d415e850 msgr2=0x7f53d4150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d415e850 0x7f53d4150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 -- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53d415bf30 msgr2=0x7f53d41500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53d415bf30 0x7f53d41500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.927+0000 7f53d37fe640 1 -- 172.21.15.72:0/167764990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d40d03f0 con 0x7f53d413dc10 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.928+0000 7f53d37fe640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 0x7f53d414f4c0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f53c4000c00 tx=0x7f53c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.928+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4040070 con 0x7f53d413dc10 2024-03-06T07:28:37.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.928+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c40399f0 con 0x7f53d413dc10 2024-03-06T07:28:37.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.928+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d4153990 con 0x7f53d413dc10 2024-03-06T07:28:37.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.928+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c4035040 con 0x7f53d413dc10 2024-03-06T07:28:37.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.928+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d4153f50 con 0x7f53d413dc10 2024-03-06T07:28:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.929+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d415eca0 con 0x7f53d413dc10 2024-03-06T07:28:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.929+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53c404b060 con 0x7f53d413dc10 2024-03-06T07:28:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.930+0000 7f53bbfff640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53c006b4d0 0x7f53c006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.930+0000 7f53d2ffd640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53c006b4d0 0x7f53c006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.930+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53c40aba90 con 0x7f53d413dc10 2024-03-06T07:28:37.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.930+0000 7f53d2ffd640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53c006b4d0 0x7f53c006d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f53d4155580 tx=0x7f53c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:37.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:37.934+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53d415eca0 con 0x7f53d413dc10 2024-03-06T07:28:38.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.109+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53d406eb00 con 0x7f53d413dc10 2024-03-06T07:28:38.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.110+0000 7f53bbfff640 1 -- 172.21.15.72:0/167764990 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16704 (secure 0 0 0) 0x7f53c407a910 con 0x7f53d413dc10 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:38.113 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":126,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:28:37.147322+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53c006b4d0 msgr2=0x7f53c006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53c006b4d0 0x7f53c006d990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f53d4155580 tx=0x7f53c8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 msgr2=0x7f53d414f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 0x7f53d414f4c0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f53c4000c00 tx=0x7f53c40046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 shutdown_connections 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53c006b4d0 0x7f53c006d990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53d415e850 0x7f53d4150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53d415bf30 0x7f53d41500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 --2- 172.21.15.72:0/167764990 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d413dc10 0x7f53d414f4c0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.112+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 >> 172.21.15.72:0/167764990 conn(0x7f53d40cf8a0 msgr2=0x7f53d40b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.113+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 shutdown_connections 2024-03-06T07:28:38.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.113+0000 7f53d9bf6640 1 -- 172.21.15.72:0/167764990 wait complete. 2024-03-06T07:28:38.126 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:38.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.223+0000 7ff3072a4640 1 Processor -- start 2024-03-06T07:28:38.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.224+0000 7ff3072a4640 1 -- start start 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.224+0000 7ff3072a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30015bed0 0x7ff3000726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.224+0000 7ff3072a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff30006e170 0x7ff3000ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.224+0000 7ff3072a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30006d3b0 con 0x7ff3000b50a0 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.224+0000 7ff3072a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3000ccaa0 con 0x7ff30006e170 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.224+0000 7ff3072a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30006d690 con 0x7ff30015bed0 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305841640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff30006e170 0x7ff3000ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305040640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff3000b50a0 0x7ff3000b5480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59136/0 (socket says 172.21.15.72:59136) 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305040640 1 -- 172.21.15.72:0/1132447214 learned_addr learned my addr 172.21.15.72:0/1132447214 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305841640 1 -- 172.21.15.72:0/1132447214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30015bed0 msgr2=0x7ff3000726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305841640 1 --2- 172.21.15.72:0/1132447214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30015bed0 0x7ff3000726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305841640 1 -- 172.21.15.72:0/1132447214 >> v1:172.21.15.72:6789/0 conn(0x7ff3000b50a0 legacy=0x7ff3000b5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305841640 1 -- 172.21.15.72:0/1132447214 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30006d970 con 0x7ff30006e170 2024-03-06T07:28:38.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff2f7fff640 1 -- 172.21.15.72:0/1132447214 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096414567 0 0) 0x7ff30006d3b0 con 0x7ff3000b50a0 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.225+0000 7ff305841640 1 --2- 172.21.15.72:0/1132447214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff30006e170 0x7ff3000ce140 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009d90 tx=0x7ff2f802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0e4b1bca947f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff2f7fff640 1 -- 172.21.15.72:0/1132447214 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8039070 con 0x7ff30006e170 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff2f7fff640 1 -- 172.21.15.72:0/1132447214 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f802cc50 con 0x7ff30006e170 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff2f7fff640 1 -- 172.21.15.72:0/1132447214 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8034440 con 0x7ff30006e170 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 -- 172.21.15.72:0/1132447214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff30006e170 msgr2=0x7ff3000ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 --2- 172.21.15.72:0/1132447214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff30006e170 0x7ff3000ce140 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009d90 tx=0x7ff2f802bea0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 -- 172.21.15.72:0/1132447214 shutdown_connections 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 --2- 172.21.15.72:0/1132447214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff30006e170 0x7ff3000ce140 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 --2- 172.21.15.72:0/1132447214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30015bed0 0x7ff3000726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 -- 172.21.15.72:0/1132447214 >> 172.21.15.72:0/1132447214 conn(0x7ff3000b1a90 msgr2=0x7ff3000b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 -- 172.21.15.72:0/1132447214 shutdown_connections 2024-03-06T07:28:38.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.226+0000 7ff3072a4640 1 -- 172.21.15.72:0/1132447214 wait complete. 2024-03-06T07:28:38.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.227+0000 7ff3072a4640 1 Processor -- start 2024-03-06T07:28:38.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.227+0000 7ff3072a4640 1 -- start start 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff3072a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30006e170 0x7ff300152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff3072a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 0x7ff300148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff3072a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30015e410 0x7ff300071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff3072a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30006f5b0 con 0x7ff3000b50a0 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff3072a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff300072be0 con 0x7ff30015e410 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff3072a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30006f730 con 0x7ff30006e170 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff30483f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 0x7ff300148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff30483f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 0x7ff300148e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59200/0 (socket says 172.21.15.72:59200) 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff30483f640 1 -- 172.21.15.72:0/2679484973 learned_addr learned my addr 172.21.15.72:0/2679484973 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff305040640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30006e170 0x7ff300152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff30483f640 1 -- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30006e170 msgr2=0x7ff300152390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff30483f640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30006e170 0x7ff300152390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.228+0000 7ff305841640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30015e410 0x7ff300071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff30483f640 1 -- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30015e410 msgr2=0x7ff300071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff30483f640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30015e410 0x7ff300071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff30483f640 1 -- 172.21.15.72:0/2679484973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3001493b0 con 0x7ff3000b50a0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff30483f640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 0x7ff300148e70 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00b440 tx=0x7ff2fc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc015020 con 0x7ff3000b50a0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2fc011890 con 0x7ff3000b50a0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2fc011c80 con 0x7ff3000b50a0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30014f050 con 0x7ff3000b50a0 2024-03-06T07:28:38.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.229+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30014f610 con 0x7ff3000b50a0 2024-03-06T07:28:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.230+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30004f330 con 0x7ff3000b50a0 2024-03-06T07:28:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.230+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff2fc018020 con 0x7ff3000b50a0 2024-03-06T07:28:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.231+0000 7ff2f5ffb640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.231+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2fc08a260 con 0x7ff3000b50a0 2024-03-06T07:28:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.231+0000 7ff305040640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.231+0000 7ff305040640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0000f60 tx=0x7ff2f0007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.235+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff2fc01f020 con 0x7ff3000b50a0 2024-03-06T07:28:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.406+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff30006eae0 con 0x7ff3000b50a0 2024-03-06T07:28:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.407+0000 7ff2f5ffb640 1 -- 172.21.15.72:0/2679484973 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16704 (secure 0 0 0) 0x7ff2fc00d040 con 0x7ff3000b50a0 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:38.410 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":126,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:28:37.147322+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 msgr2=0x7ff2e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0000f60 tx=0x7ff2f0007b50 comp rx=0 tx=0).stop 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 msgr2=0x7ff300148e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 0x7ff300148e70 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7ff2fc00b440 tx=0x7ff2fc00b900 comp rx=0 tx=0).stop 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 shutdown_connections 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff2e406b4d0 0x7ff2e406d990 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30015e410 0x7ff300071c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3000b50a0 0x7ff300148e70 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 --2- 172.21.15.72:0/2679484973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30006e170 0x7ff300152390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 >> 172.21.15.72:0/2679484973 conn(0x7ff3000b1a90 msgr2=0x7ff300158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 shutdown_connections 2024-03-06T07:28:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.409+0000 7ff3072a4640 1 -- 172.21.15.72:0/2679484973 wait complete. 2024-03-06T07:28:38.423 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.525+0000 7f3f0af65640 1 Processor -- start 2024-03-06T07:28:38.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.525+0000 7f3f0af65640 1 -- start start 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f0af65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0415def0 0x7f3f040b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f0af65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f0406cab0 0x7f3f040ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f0af65640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3f040ce880 0x7f3f040cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f0af65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f040cecf0 con 0x7f3f0406cab0 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f0af65640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f040cf1c0 con 0x7f3f040ce880 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f0af65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f040cc760 con 0x7f3f0415def0 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f0406cab0 0x7f3f040ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3f0406cab0 0x7f3f040ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59208/0 (socket says 172.21.15.72:59208) 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 -- 172.21.15.72:0/3336981327 learned_addr learned my addr 172.21.15.72:0/3336981327 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:38.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f09502640 1 --2- 172.21.15.72:0/3336981327 >> v2:172.21.15.146:3300/0 conn(0x7f3f040ce880 0x7f3f040cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f08d01640 1 --2- 172.21.15.72:0/3336981327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0415def0 0x7f3f040b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 -- 172.21.15.72:0/3336981327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0415def0 msgr2=0x7f3f040b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 --2- 172.21.15.72:0/3336981327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0415def0 0x7f3f040b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 -- 172.21.15.72:0/3336981327 >> v2:172.21.15.146:3300/0 conn(0x7f3f040ce880 msgr2=0x7f3f040cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 --2- 172.21.15.72:0/3336981327 >> v2:172.21.15.146:3300/0 conn(0x7f3f040ce880 0x7f3f040cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.526+0000 7f3f03fff640 1 -- 172.21.15.72:0/3336981327 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0406d030 con 0x7f3f0406cab0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.527+0000 7f3f03fff640 1 --2- 172.21.15.72:0/3336981327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0406cab0 0x7f3f040ce340 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f3f04071e70 tx=0x7f3ef002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f74bc39f411d156 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.527+0000 7f3f037fe640 1 -- 172.21.15.72:0/3336981327 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0034070 con 0x7f3f0406cab0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.527+0000 7f3f037fe640 1 -- 172.21.15.72:0/3336981327 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef0030850 con 0x7f3f0406cab0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.527+0000 7f3f037fe640 1 -- 172.21.15.72:0/3336981327 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0030c20 con 0x7f3f0406cab0 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 -- 172.21.15.72:0/3336981327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0406cab0 msgr2=0x7f3f040ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3336981327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0406cab0 0x7f3f040ce340 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f3f04071e70 tx=0x7f3ef002b990 comp rx=0 tx=0).stop 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 -- 172.21.15.72:0/3336981327 shutdown_connections 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3336981327 >> v2:172.21.15.146:3300/0 conn(0x7f3f040ce880 0x7f3f040cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3336981327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0406cab0 0x7f3f040ce340 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3336981327 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0415def0 0x7f3f040b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 -- 172.21.15.72:0/3336981327 >> 172.21.15.72:0/3336981327 conn(0x7f3f040b3690 msgr2=0x7f3f040b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 -- 172.21.15.72:0/3336981327 shutdown_connections 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.528+0000 7f3f0af65640 1 -- 172.21.15.72:0/3336981327 wait complete. 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.529+0000 7f3f0af65640 1 Processor -- start 2024-03-06T07:28:38.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.529+0000 7f3f0af65640 1 -- start start 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0406cab0 0x7f3f040c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f040ce880 0x7f3f0415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 0x7f3f040b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f040cc3f0 con 0x7f3f0415def0 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f040cc110 con 0x7f3f040ce880 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0415e2d0 con 0x7f3f0406cab0 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 0x7f3f040b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 0x7f3f040b7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59216/0 (socket says 172.21.15.72:59216) 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 -- 172.21.15.72:0/3177907654 learned_addr learned my addr 172.21.15.72:0/3177907654 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f08d01640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0406cab0 0x7f3f040c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f03fff640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f040ce880 0x7f3f0415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 -- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0406cab0 msgr2=0x7f3f040c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0406cab0 0x7f3f040c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 -- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f040ce880 msgr2=0x7f3f0415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f040ce880 0x7f3f0415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 -- 172.21.15.72:0/3177907654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040b6270 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f08d01640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0406cab0 0x7f3f040c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f03fff640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f040ce880 0x7f3f0415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f09502640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 0x7f3f040b7030 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3efc007e40 tx=0x7f3efc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc014620 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc019070 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f040b6550 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc014a20 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.530+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f040b6b10 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.531+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0406c850 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.532+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3efc014bc0 con 0x7f3f0415def0 2024-03-06T07:28:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.532+0000 7f3f017fa640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3ee806b4d0 0x7f3ee806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.532+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3efc08aad0 con 0x7f3f0415def0 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.532+0000 7f3f08d01640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3ee806b4d0 0x7f3ee806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.532+0000 7f3f08d01640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3ee806b4d0 0x7f3ee806d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006fd0 tx=0x7f3ef8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.536+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3f0406c850 con 0x7f3f0415def0 2024-03-06T07:28:38.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.743+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3f0409fcf0 con 0x7f3f0415def0 2024-03-06T07:28:38.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.816+0000 7f3f017fa640 1 -- 172.21.15.72:0/3177907654 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v78) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3efc059950 con 0x7f3f0415def0 2024-03-06T07:28:38.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3ee806b4d0 msgr2=0x7f3ee806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3ee806b4d0 0x7f3ee806d990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8006fd0 tx=0x7f3ef8008040 comp rx=0 tx=0).stop 2024-03-06T07:28:38.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 msgr2=0x7f3f040b7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 0x7f3f040b7030 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3efc007e40 tx=0x7f3efc00e4b0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 shutdown_connections 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3ee806b4d0 0x7f3ee806d990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f0415def0 0x7f3f040b7030 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3f040ce880 0x7f3f0415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 --2- 172.21.15.72:0/3177907654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3f0406cab0 0x7f3f040c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 >> 172.21.15.72:0/3177907654 conn(0x7f3f040b3690 msgr2=0x7f3f0406fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 shutdown_connections 2024-03-06T07:28:38.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.818+0000 7f3f0af65640 1 -- 172.21.15.72:0/3177907654 wait complete. 2024-03-06T07:28:38.832 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-06T07:28:38.832 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-06T07:28:38.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 Processor -- start 2024-03-06T07:28:38.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 -- start start 2024-03-06T07:28:38.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1281614d0 0x7fb1280d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1280d1910 0x7fb1280d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1281618b0 con 0x7fb1280d22e0 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280cf400 con 0x7fb1281614d0 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12eb13640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1280d2710 con 0x7fb1280d1910 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12c8af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1281614d0 0x7fb1280d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.934+0000 7fb12c8af640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb1281614d0 0x7fb1280d4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59232/0 (socket says 172.21.15.72:59232) 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb12c8af640 1 -- 172.21.15.72:0/2189082759 learned_addr learned my addr 172.21.15.72:0/2189082759 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb127fff640 1 --2- 172.21.15.72:0/2189082759 >> v2:172.21.15.146:3300/0 conn(0x7fb1280d1910 0x7fb1280d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb12c8af640 1 -- 172.21.15.72:0/2189082759 >> v2:172.21.15.146:3300/0 conn(0x7fb1280d1910 msgr2=0x7fb1280d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb12c8af640 1 --2- 172.21.15.72:0/2189082759 >> v2:172.21.15.146:3300/0 conn(0x7fb1280d1910 0x7fb1280d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb12c8af640 1 -- 172.21.15.72:0/2189082759 >> v1:172.21.15.146:6789/0 conn(0x7fb1280d22e0 legacy=0x7fb1280d06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb12c8af640 1 -- 172.21.15.72:0/2189082759 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280cf6a0 con 0x7fb1281614d0 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb127fff640 1 --2- 172.21.15.72:0/2189082759 >> v2:172.21.15.146:3300/0 conn(0x7fb1280d1910 0x7fb1280d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:38.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.935+0000 7fb12c8af640 1 --2- 172.21.15.72:0/2189082759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1281614d0 0x7fb1280d4860 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb12808c470 tx=0x7fb11c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bdd7612c1a4370 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb1277fe640 1 -- 172.21.15.72:0/2189082759 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c035020 con 0x7fb1281614d0 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb1277fe640 1 -- 172.21.15.72:0/2189082759 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb11c031640 con 0x7fb1281614d0 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb1277fe640 1 -- 172.21.15.72:0/2189082759 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11c02d040 con 0x7fb1281614d0 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 -- 172.21.15.72:0/2189082759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1281614d0 msgr2=0x7fb1280d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 --2- 172.21.15.72:0/2189082759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1281614d0 0x7fb1280d4860 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fb12808c470 tx=0x7fb11c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 -- 172.21.15.72:0/2189082759 shutdown_connections 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 --2- 172.21.15.72:0/2189082759 >> v2:172.21.15.146:3300/0 conn(0x7fb1280d1910 0x7fb1280d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 --2- 172.21.15.72:0/2189082759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1281614d0 0x7fb1280d4860 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 -- 172.21.15.72:0/2189082759 >> 172.21.15.72:0/2189082759 conn(0x7fb1280d3d40 msgr2=0x7fb1280d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 -- 172.21.15.72:0/2189082759 shutdown_connections 2024-03-06T07:28:38.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.936+0000 7fb12eb13640 1 -- 172.21.15.72:0/2189082759 wait complete. 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.937+0000 7fb12eb13640 1 Processor -- start 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.937+0000 7fb12eb13640 1 -- start start 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12eb13640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1280d1910 0x7fb1280c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12eb13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 0x7fb1280b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12eb13640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1281614d0 0x7fb1280b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12eb13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280b0910 con 0x7fb1280d22e0 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12eb13640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280cfc60 con 0x7fb1281614d0 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12eb13640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1280b1e40 con 0x7fb1280d1910 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 0x7fb1280b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 0x7fb1280b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59246/0 (socket says 172.21.15.72:59246) 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 -- 172.21.15.72:0/4268190028 learned_addr learned my addr 172.21.15.72:0/4268190028 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12c8af640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1280d1910 0x7fb1280c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12d0b0640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1281614d0 0x7fb1280b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 -- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1280d1910 msgr2=0x7fb1280c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1280d1910 0x7fb1280c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 -- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1281614d0 msgr2=0x7fb1280b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1281614d0 0x7fb1280b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 -- 172.21.15.72:0/4268190028 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1280ca200 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb12c8af640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1280d1910 0x7fb1280c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb127fff640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 0x7fb1280b6a80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb11400c820 tx=0x7fb11400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.939+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114015070 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.940+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb114007a30 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.940+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114007e00 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.940+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1280c9c40 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.940+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1280cb780 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.941+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1280d05b0 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.941+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb114018070 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.941+0000 7fb1257fa640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10c06b4d0 0x7fb10c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.941+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb11408a1e0 con 0x7fb1280d22e0 2024-03-06T07:28:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.941+0000 7fb12c8af640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10c06b4d0 0x7fb10c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:38.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.942+0000 7fb12c8af640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10c06b4d0 0x7fb10c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb11c036000 tx=0x7fb11c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:38.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:38.946+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb1280d05b0 con 0x7fb1280d22e0 2024-03-06T07:28:39.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.163+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb128068ff0 con 0x7fb1280d22e0 2024-03-06T07:28:39.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.823+0000 7fb1257fa640 1 -- 172.21.15.72:0/4268190028 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v79) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb114059060 con 0x7fb1280d22e0 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10c06b4d0 msgr2=0x7fb10c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10c06b4d0 0x7fb10c06d990 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb11c036000 tx=0x7fb11c03c040 comp rx=0 tx=0).stop 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 msgr2=0x7fb1280b6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 0x7fb1280b6a80 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb11400c820 tx=0x7fb11400cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 shutdown_connections 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb10c06b4d0 0x7fb10c06d990 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1281614d0 0x7fb1280b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1280d22e0 0x7fb1280b6a80 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 --2- 172.21.15.72:0/4268190028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1280d1910 0x7fb1280c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 >> 172.21.15.72:0/4268190028 conn(0x7fb1280d3d40 msgr2=0x7fb1280cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 shutdown_connections 2024-03-06T07:28:39.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.825+0000 7fb12eb13640 1 -- 172.21.15.72:0/4268190028 wait complete. 2024-03-06T07:28:39.838 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-06T07:28:39.839 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:39.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.936+0000 7f1fba10c640 1 Processor -- start 2024-03-06T07:28:39.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.937+0000 7f1fba10c640 1 -- start start 2024-03-06T07:28:39.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fba10c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fb415bdd0 0x7f1fb40c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fba10c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb40cebe0 con 0x7f1fb40d16e0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fba10c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb40ceec0 con 0x7f1fb415fa10 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fba10c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb40ce980 con 0x7f1fb415bdd0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1fb40d16e0 0x7f1fb40d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59148/0 (socket says 172.21.15.72:59148) 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb3fff640 1 -- 172.21.15.72:0/4088562069 learned_addr learned my addr 172.21.15.72:0/4088562069 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710014204 0 0) 0x7f1fb40cebe0 con 0x7f1fb40d16e0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/4088562069 >> v2:172.21.15.146:3300/0 conn(0x7f1fb415bdd0 0x7f1fb40c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f9c003520 con 0x7f1fb40d16e0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758446145 0 0) 0x7f1f9c003520 con 0x7f1fb40d16e0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 >> v2:172.21.15.146:3300/0 conn(0x7f1fb415bdd0 msgr2=0x7f1fb40c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 --2- 172.21.15.72:0/4088562069 >> v2:172.21.15.146:3300/0 conn(0x7f1fb415bdd0 0x7f1fb40c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 >> v1:172.21.15.146:6789/0 conn(0x7f1fb415fa10 legacy=0x7f1fb40ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40cf1a0 con 0x7f1fb40d16e0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1fac001a10 con 0x7f1fb40d16e0 2024-03-06T07:28:39.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 >> v1:172.21.15.72:6789/0 conn(0x7f1fb40d16e0 legacy=0x7f1fb40d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1f9c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.938+0000 7f1fb27fc640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c004800 0x7f1f9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb27fc640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c0051c0 0x7f1f9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb40ce980 con 0x7f1f9c0039b0 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c00c160 con 0x7f1f9c0051c0 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9c00c360 con 0x7f1f9c004800 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1f9c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb3fff640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c0051c0 0x7f1f9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb37fe640 1 -- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c004800 msgr2=0x7f1f9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f9c004800 0x7f1f9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb37fe640 1 -- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c0051c0 msgr2=0x7f1f9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f9c0051c0 0x7f1f9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb37fe640 1 -- 172.21.15.72:0/4088562069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9c003520 con 0x7f1f9c0039b0 2024-03-06T07:28:39.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.939+0000 7f1fb3fff640 1 -- 172.21.15.72:0/4088562069 reap_dead start 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1f9c003dd0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009ac0 tx=0x7f1f9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e979b2b8c3bd6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4039070 con 0x7f1f9c0039b0 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa402c6b0 con 0x7f1f9c0039b0 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fb27fc640 1 -- 172.21.15.72:0/4088562069 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4009690 con 0x7f1f9c0039b0 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 -- 172.21.15.72:0/4088562069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 msgr2=0x7f1f9c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1f9c003dd0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4009ac0 tx=0x7f1f9c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 -- 172.21.15.72:0/4088562069 shutdown_connections 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 --2- 172.21.15.72:0/4088562069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1f9c003dd0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 -- 172.21.15.72:0/4088562069 >> 172.21.15.72:0/4088562069 conn(0x7f1fb409a9f0 msgr2=0x7f1fb409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 -- 172.21.15.72:0/4088562069 shutdown_connections 2024-03-06T07:28:39.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.940+0000 7f1fba10c640 1 -- 172.21.15.72:0/4088562069 wait complete. 2024-03-06T07:28:39.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.941+0000 7f1fba10c640 1 Processor -- start 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 -- start start 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1fb40cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fb415c330 0x7f1fb40b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fb40c8df0 0x7f1fb40c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb415c1b0 con 0x7f1f9c0039b0 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb40cc5d0 con 0x7f1fb415c330 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fba10c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb40ce270 con 0x7f1fb40c8df0 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fb37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1fb40cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fb37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1fb40cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59262/0 (socket says 172.21.15.72:59262) 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fb37fe640 1 -- 172.21.15.72:0/2468485486 learned_addr learned my addr 172.21.15.72:0/2468485486 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fb2ffd640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fb415c330 0x7f1fb40b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.942+0000 7f1fb3fff640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fb40c8df0 0x7f1fb40c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb37fe640 1 -- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fb40c8df0 msgr2=0x7f1fb40c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fb40c8df0 0x7f1fb40c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb37fe640 1 -- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fb415c330 msgr2=0x7f1fb40b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fb415c330 0x7f1fb40b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb37fe640 1 -- 172.21.15.72:0/2468485486 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb40cbaf0 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb2ffd640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fb415c330 0x7f1fb40b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fb37fe640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1fb40cb5b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4037040 tx=0x7f1fa4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4042020 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa403a040 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fb415c8f0 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4039070 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.943+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb415ceb0 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.944+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb404f330 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.944+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1fa4045020 con 0x7f1f9c0039b0 2024-03-06T07:28:39.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.945+0000 7f1f9bfff640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fa006b500 0x7f1fa006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:39.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.945+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fa40aaf60 con 0x7f1f9c0039b0 2024-03-06T07:28:39.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.945+0000 7f1fb2ffd640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fa006b500 0x7f1fa006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:39.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.945+0000 7f1fb2ffd640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fa006b500 0x7f1fa006d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40ce6b0 tx=0x7f1fa8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:39.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:39.949+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1fa4079de0 con 0x7f1f9c0039b0 2024-03-06T07:28:40.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.241+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fb40b7a40 con 0x7f1f9c0039b0 2024-03-06T07:28:40.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.241+0000 7f1f9bfff640 1 -- 172.21.15.72:0/2468485486 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f1fa40796e0 con 0x7f1f9c0039b0 2024-03-06T07:28:40.243 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:40.243 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2751164744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2751164744},{"type":"v1","addr":"172.21.15.146:6834","nonce":2751164744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/406165267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":406165267},{"type":"v1","addr":"172.21.15.146:6835","nonce":406165267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":8637,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:39.164534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:28:40.244 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 79 2024-03-06T07:28:40.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fa006b500 msgr2=0x7f1fa006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fa006b500 0x7f1fa006d9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40ce6b0 tx=0x7f1fa8009040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 msgr2=0x7f1fb40cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1fb40cb5b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1fa4037040 tx=0x7f1fa4004730 comp rx=0 tx=0).stop 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 shutdown_connections 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1fa006b500 0x7f1fa006d9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fb40c8df0 0x7f1fb40c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fb415c330 0x7f1fb40b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 --2- 172.21.15.72:0/2468485486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f9c0039b0 0x7f1fb40cb5b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 >> 172.21.15.72:0/2468485486 conn(0x7f1fb409a9f0 msgr2=0x7f1fb40c6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 shutdown_connections 2024-03-06T07:28:40.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.243+0000 7f1fba10c640 1 -- 172.21.15.72:0/2468485486 wait complete. 2024-03-06T07:28:40.257 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:28:40.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.359+0000 7fcc2d1d6640 1 Processor -- start 2024-03-06T07:28:40.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.359+0000 7fcc2d1d6640 1 -- start start 2024-03-06T07:28:40.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2d1d6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcc280d22b0 0x7fcc280d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2d1d6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcc280b41f0 0x7fcc280b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2d1d6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280b3430 con 0x7fcc2815dfe0 2024-03-06T07:28:40.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2d1d6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc280d2c50 con 0x7fcc280b41f0 2024-03-06T07:28:40.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2d1d6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc280b31e0 con 0x7fcc280d22b0 2024-03-06T07:28:40.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2759e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcc280b41f0 0x7fcc280b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc26d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcc2815dfe0 0x7fcc280cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59152/0 (socket says 172.21.15.72:59152) 2024-03-06T07:28:40.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc26d9d640 1 -- 172.21.15.72:0/2564941777 learned_addr learned my addr 172.21.15.72:0/2564941777 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:40.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2659c640 1 --2- 172.21.15.72:0/2564941777 >> v2:172.21.15.146:3300/0 conn(0x7fcc280d22b0 0x7fcc280d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2759e640 1 -- 172.21.15.72:0/2564941777 >> v2:172.21.15.146:3300/0 conn(0x7fcc280d22b0 msgr2=0x7fcc280d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2759e640 1 --2- 172.21.15.72:0/2564941777 >> v2:172.21.15.146:3300/0 conn(0x7fcc280d22b0 0x7fcc280d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2759e640 1 -- 172.21.15.72:0/2564941777 >> v1:172.21.15.72:6789/0 conn(0x7fcc2815dfe0 legacy=0x7fcc280cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc25d9b640 1 -- 172.21.15.72:0/2564941777 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740026589 0 0) 0x7fcc280b3430 con 0x7fcc2815dfe0 2024-03-06T07:28:40.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2759e640 1 -- 172.21.15.72:0/2564941777 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280b3710 con 0x7fcc280b41f0 2024-03-06T07:28:40.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.360+0000 7fcc2659c640 1 --2- 172.21.15.72:0/2564941777 >> v2:172.21.15.146:3300/0 conn(0x7fcc280d22b0 0x7fcc280d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.361+0000 7fcc2759e640 1 --2- 172.21.15.72:0/2564941777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280b41f0 0x7fcc280b4680 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009d90 tx=0x7fcc2002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd23c4f53a76d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.361+0000 7fcc25d9b640 1 -- 172.21.15.72:0/2564941777 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20039070 con 0x7fcc280b41f0 2024-03-06T07:28:40.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.361+0000 7fcc25d9b640 1 -- 172.21.15.72:0/2564941777 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2002cc60 con 0x7fcc280b41f0 2024-03-06T07:28:40.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.361+0000 7fcc25d9b640 1 -- 172.21.15.72:0/2564941777 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20034660 con 0x7fcc280b41f0 2024-03-06T07:28:40.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/2564941777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280b41f0 msgr2=0x7fcc280b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/2564941777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280b41f0 0x7fcc280b4680 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009d90 tx=0x7fcc2002beb0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/2564941777 shutdown_connections 2024-03-06T07:28:40.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/2564941777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280b41f0 0x7fcc280b4680 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/2564941777 >> v2:172.21.15.146:3300/0 conn(0x7fcc280d22b0 0x7fcc280d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/2564941777 >> 172.21.15.72:0/2564941777 conn(0x7fcc280d3cd0 msgr2=0x7fcc280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/2564941777 shutdown_connections 2024-03-06T07:28:40.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.362+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/2564941777 wait complete. 2024-03-06T07:28:40.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.364+0000 7fcc2d1d6640 1 Processor -- start 2024-03-06T07:28:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.364+0000 7fcc2d1d6640 1 -- start start 2024-03-06T07:28:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2d1d6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc280b41f0 0x7fcc280b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2d1d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 0x7fcc2815ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2d1d6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2815dfe0 0x7fcc280b93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2d1d6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280cfa00 con 0x7fcc280d22b0 2024-03-06T07:28:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2d1d6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280b3d60 con 0x7fcc2815dfe0 2024-03-06T07:28:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2d1d6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc280d1c70 con 0x7fcc280b41f0 2024-03-06T07:28:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 0x7fcc2815ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 0x7fcc2815ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59278/0 (socket says 172.21.15.72:59278) 2024-03-06T07:28:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 -- 172.21.15.72:0/1571745249 learned_addr learned my addr 172.21.15.72:0/1571745249 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2759e640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2815dfe0 0x7fcc280b93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc26d9d640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc280b41f0 0x7fcc280b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 -- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc280b41f0 msgr2=0x7fcc280b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc280b41f0 0x7fcc280b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 -- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2815dfe0 msgr2=0x7fcc280b93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2815dfe0 0x7fcc280b93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.365+0000 7fcc2659c640 1 -- 172.21.15.72:0/1571745249 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280cbd00 con 0x7fcc280d22b0 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.366+0000 7fcc2659c640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 0x7fcc2815ad70 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800c820 tx=0x7fcc1800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.366+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18015070 con 0x7fcc280d22b0 2024-03-06T07:28:40.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.366+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc18007a30 con 0x7fcc280d22b0 2024-03-06T07:28:40.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.366+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc18007e00 con 0x7fcc280d22b0 2024-03-06T07:28:40.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.366+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc280caf40 con 0x7fcc280d22b0 2024-03-06T07:28:40.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.366+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc280cb500 con 0x7fcc280d22b0 2024-03-06T07:28:40.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.367+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2804f330 con 0x7fcc280d22b0 2024-03-06T07:28:40.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.367+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcc18018070 con 0x7fcc280d22b0 2024-03-06T07:28:40.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.368+0000 7fcc0f7fe640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc1406b4d0 0x7fcc1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.368+0000 7fcc26d9d640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc1406b4d0 0x7fcc1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.368+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc1808a1e0 con 0x7fcc280d22b0 2024-03-06T07:28:40.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.368+0000 7fcc26d9d640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc1406b4d0 0x7fcc1406d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c000f60 tx=0x7fcc1c007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.372+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcc18059060 con 0x7fcc280d22b0 2024-03-06T07:28:40.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.553+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc280cdcd0 con 0x7fcc280d22b0 2024-03-06T07:28:40.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.554+0000 7fcc0f7fe640 1 -- 172.21.15.72:0/1571745249 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16691 (secure 0 0 0) 0x7fcc18058960 con 0x7fcc280d22b0 2024-03-06T07:28:40.556 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:40.556 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":127,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:28:38.796027+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:28:40.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc1406b4d0 msgr2=0x7fcc1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc1406b4d0 0x7fcc1406d990 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c000f60 tx=0x7fcc1c007c00 comp rx=0 tx=0).stop 2024-03-06T07:28:40.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 msgr2=0x7fcc2815ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 0x7fcc2815ad70 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800c820 tx=0x7fcc1800cc20 comp rx=0 tx=0).stop 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 shutdown_connections 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcc1406b4d0 0x7fcc1406d990 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2815dfe0 0x7fcc280b93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcc280d22b0 0x7fcc2815ad70 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 --2- 172.21.15.72:0/1571745249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc280b41f0 0x7fcc280b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 >> 172.21.15.72:0/1571745249 conn(0x7fcc280d3cd0 msgr2=0x7fcc2806cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 shutdown_connections 2024-03-06T07:28:40.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.556+0000 7fcc2d1d6640 1 -- 172.21.15.72:0/1571745249 wait complete. 2024-03-06T07:28:40.570 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-06T07:28:40.570 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:40.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.669+0000 7fce02615640 1 Processor -- start 2024-03-06T07:28:40.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.669+0000 7fce02615640 1 -- start start 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fce02615640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 0x7fcdfc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fce02615640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcdfc0cf830 0x7fcdfc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fce02615640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc0b3430 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fce02615640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc0d2c50 con 0x7fcdfc0cf830 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fce02615640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0b31e0 con 0x7fcdfc15dfe0 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fcdfb7fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcdfc0cf830 0x7fcdfc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 0x7fcdfc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fcdfbfff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 0x7fcdfc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:52146/0 (socket says 172.21.15.72:52146) 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.670+0000 7fcdfbfff640 1 -- 172.21.15.72:0/639161209 learned_addr learned my addr 172.21.15.72:0/639161209 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.671+0000 7fcdfbfff640 1 -- 172.21.15.72:0/639161209 >> v2:172.21.15.146:3300/0 conn(0x7fcdfc0cf830 msgr2=0x7fcdfc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:40.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.671+0000 7fcdfbfff640 1 --2- 172.21.15.72:0/639161209 >> v2:172.21.15.146:3300/0 conn(0x7fcdfc0cf830 0x7fcdfc0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.671+0000 7fcdfbfff640 1 -- 172.21.15.72:0/639161209 >> v1:172.21.15.146:6789/0 conn(0x7fcdfc0d27f0 legacy=0x7fcdfc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.671+0000 7fcdfbfff640 1 -- 172.21.15.72:0/639161209 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc0b3710 con 0x7fcdfc15dfe0 2024-03-06T07:28:40.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.671+0000 7fcdfb7fe640 1 --2- 172.21.15.72:0/639161209 >> v2:172.21.15.146:3300/0 conn(0x7fcdfc0cf830 0x7fcdfc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.671+0000 7fcdfbfff640 1 --2- 172.21.15.72:0/639161209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 0x7fcdfc0cf2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009870 tx=0x7fcdec02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e854c0e48742b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fcdfaffd640 1 -- 172.21.15.72:0/639161209 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec034070 con 0x7fcdfc15dfe0 2024-03-06T07:28:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fcdfaffd640 1 -- 172.21.15.72:0/639161209 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec030840 con 0x7fcdfc15dfe0 2024-03-06T07:28:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fcdfaffd640 1 -- 172.21.15.72:0/639161209 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec030c10 con 0x7fcdfc15dfe0 2024-03-06T07:28:40.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 -- 172.21.15.72:0/639161209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 msgr2=0x7fcdfc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 --2- 172.21.15.72:0/639161209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 0x7fcdfc0cf2f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcdec009870 tx=0x7fcdec02b980 comp rx=0 tx=0).stop 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 -- 172.21.15.72:0/639161209 shutdown_connections 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 --2- 172.21.15.72:0/639161209 >> v2:172.21.15.146:3300/0 conn(0x7fcdfc0cf830 0x7fcdfc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 --2- 172.21.15.72:0/639161209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc15dfe0 0x7fcdfc0cf2f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 -- 172.21.15.72:0/639161209 >> 172.21.15.72:0/639161209 conn(0x7fcdfc0d3cd0 msgr2=0x7fcdfc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 -- 172.21.15.72:0/639161209 shutdown_connections 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.672+0000 7fce02615640 1 -- 172.21.15.72:0/639161209 wait complete. 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.673+0000 7fce02615640 1 Processor -- start 2024-03-06T07:28:40.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.673+0000 7fce02615640 1 -- start start 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce02615640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0cf830 0x7fcdfc14d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce02615640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 0x7fcdfc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce02615640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc15dfe0 0x7fcdfc15fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce02615640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0d1c70 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce02615640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0b3d60 con 0x7fcdfc15dfe0 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce02615640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0d1df0 con 0x7fcdfc0cf830 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fcdfb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 0x7fcdfc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fcdfb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 0x7fcdfc0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59292/0 (socket says 172.21.15.72:59292) 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fcdfb7fe640 1 -- 172.21.15.72:0/3757669739 learned_addr learned my addr 172.21.15.72:0/3757669739 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fcdfbfff640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0cf830 0x7fcdfc14d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.674+0000 7fce00bb2640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc15dfe0 0x7fcdfc15fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfb7fe640 1 -- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0cf830 msgr2=0x7fcdfc14d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfb7fe640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0cf830 0x7fcdfc14d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfb7fe640 1 -- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc15dfe0 msgr2=0x7fcdfc15fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfb7fe640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc15dfe0 0x7fcdfc15fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfb7fe640 1 -- 172.21.15.72:0/3757669739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc15e970 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfbfff640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0cf830 0x7fcdfc14d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fce00bb2640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc15dfe0 0x7fcdfc15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdfb7fe640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 0x7fcdfc0cd5e0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000e9b0 tx=0x7fcdf000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0017070 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf000ca00 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc15ec50 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf000cdf0 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.675+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc15f210 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.676+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc04f330 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.677+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcdf001a070 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.677+0000 7fcdf8ff9640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcde806b4d0 0x7fcde806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.677+0000 7fcdfbfff640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcde806b4d0 0x7fcde806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.677+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdf008a190 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.678+0000 7fcdfbfff640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcde806b4d0 0x7fcde806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcdec036000 tx=0x7fcdec03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:40.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.682+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcdf0059010 con 0x7fcdfc0d27f0 2024-03-06T07:28:40.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.853+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcdfc0ceac0 con 0x7fcde806b4d0 2024-03-06T07:28:40.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.859+0000 7fcdf8ff9640 1 -- 172.21.15.72:0/3757669739 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247075 (secure 0 0 0) 0x7fcdfc0ceac0 con 0x7fcde806b4d0 2024-03-06T07:28:40.864 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:40.865 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcde806b4d0 msgr2=0x7fcde806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcde806b4d0 0x7fcde806d990 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fcdec036000 tx=0x7fcdec03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 msgr2=0x7fcdfc0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 0x7fcdfc0cd5e0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7fcdf000e9b0 tx=0x7fcdf000ee70 comp rx=0 tx=0).stop 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 shutdown_connections 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcde806b4d0 0x7fcde806d990 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcdfc15dfe0 0x7fcdfc15fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdfc0d27f0 0x7fcdfc0cd5e0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 --2- 172.21.15.72:0/3757669739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcdfc0cf830 0x7fcdfc14d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 >> 172.21.15.72:0/3757669739 conn(0x7fcdfc0d3cd0 msgr2=0x7fcdfc0d0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 shutdown_connections 2024-03-06T07:28:40.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:40.862+0000 7fce02615640 1 -- 172.21.15.72:0/3757669739 wait complete. 2024-03-06T07:28:40.875 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-03-06T07:28:40.023279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329344,"kb_used_data":13292,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645376,"statfs":{"total":773094113280,"available":772756865024,"internally_reserved":0,"allocated":13611008,"data_stored":6587681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.234114"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806567+0000","last_change":"2024-03-06T07:28:36.806567+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806567+0000","last_undegraded":"2024-03-06T07:28:36.806567+0000","last_fullsized":"2024-03-06T07:28:36.806567+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806490+0000","last_change":"2024-03-06T07:28:36.806490+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806490+0000","last_undegraded":"2024-03-06T07:28:36.806490+0000","last_fullsized":"2024-03-06T07:28:36.806490+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.800755+0000","last_change":"2024-03-06T07:28:36.800755+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.800755+0000","last_undegraded":"2024-03-06T07:28:36.800755+0000","last_fullsized":"2024-03-06T07:28:36.800755+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806501+0000","last_change":"2024-03-06T07:28:36.806501+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806501+0000","last_undegraded":"2024-03-06T07:28:36.806501+0000","last_fullsized":"2024-03-06T07:28:36.806501+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804424+0000","last_change":"2024-03-06T07:28:36.804424+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804424+0000","last_undegraded":"2024-03-06T07:28:36.804424+0000","last_fullsized":"2024-03-06T07:28:36.804424+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.807930+0000","last_change":"2024-03-06T07:28:36.807930+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.807930+0000","last_undegraded":"2024-03-06T07:28:36.807930+0000","last_fullsized":"2024-03-06T07:28:36.807930+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803468+0000","last_change":"2024-03-06T07:28:36.803468+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803468+0000","last_undegraded":"2024-03-06T07:28:36.803468+0000","last_fullsized":"2024-03-06T07:28:36.803468+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801587+0000","last_change":"2024-03-06T07:28:36.801587+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801587+0000","last_undegraded":"2024-03-06T07:28:36.801587+0000","last_fullsized":"2024-03-06T07:28:36.801587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802274+0000","last_change":"2024-03-06T07:28:36.802274+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802274+0000","last_undegraded":"2024-03-06T07:28:36.802274+0000","last_fullsized":"2024-03-06T07:28:36.802274+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805084+0000","last_change":"2024-03-06T07:28:36.805084+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805084+0000","last_undegraded":"2024-03-06T07:28:36.805084+0000","last_fullsized":"2024-03-06T07:28:36.805084+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805445+0000","last_change":"2024-03-06T07:28:36.805445+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805445+0000","last_undegraded":"2024-03-06T07:28:36.805445+0000","last_fullsized":"2024-03-06T07:28:36.805445+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801140+0000","last_change":"2024-03-06T07:28:36.801140+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801140+0000","last_undegraded":"2024-03-06T07:28:36.801140+0000","last_fullsized":"2024-03-06T07:28:36.801140+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806487+0000","last_change":"2024-03-06T07:28:36.806487+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806487+0000","last_undegraded":"2024-03-06T07:28:36.806487+0000","last_fullsized":"2024-03-06T07:28:36.806487+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804641+0000","last_change":"2024-03-06T07:28:36.804641+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804641+0000","last_undegraded":"2024-03-06T07:28:36.804641+0000","last_fullsized":"2024-03-06T07:28:36.804641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805149+0000","last_change":"2024-03-06T07:28:36.805149+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805149+0000","last_undegraded":"2024-03-06T07:28:36.805149+0000","last_fullsized":"2024-03-06T07:28:36.805149+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798464+0000","last_change":"2024-03-06T07:28:36.798464+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798464+0000","last_undegraded":"2024-03-06T07:28:36.798464+0000","last_fullsized":"2024-03-06T07:28:36.798464+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803081+0000","last_change":"2024-03-06T07:28:36.803081+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803081+0000","last_undegraded":"2024-03-06T07:28:36.803081+0000","last_fullsized":"2024-03-06T07:28:36.803081+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799143+0000","last_change":"2024-03-06T07:28:36.799143+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799143+0000","last_undegraded":"2024-03-06T07:28:36.799143+0000","last_fullsized":"2024-03-06T07:28:36.799143+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804151+0000","last_change":"2024-03-06T07:28:36.804151+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804151+0000","last_undegraded":"2024-03-06T07:28:36.804151+0000","last_fullsized":"2024-03-06T07:28:36.804151+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798472+0000","last_change":"2024-03-06T07:28:36.798472+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798472+0000","last_undegraded":"2024-03-06T07:28:36.798472+0000","last_fullsized":"2024-03-06T07:28:36.798472+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805603+0000","last_change":"2024-03-06T07:28:36.805603+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805603+0000","last_undegraded":"2024-03-06T07:28:36.805603+0000","last_fullsized":"2024-03-06T07:28:36.805603+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803692+0000","last_change":"2024-03-06T07:28:36.803692+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803692+0000","last_undegraded":"2024-03-06T07:28:36.803692+0000","last_fullsized":"2024-03-06T07:28:36.803692+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801386+0000","last_change":"2024-03-06T07:28:36.801386+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801386+0000","last_undegraded":"2024-03-06T07:28:36.801386+0000","last_fullsized":"2024-03-06T07:28:36.801386+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802815+0000","last_change":"2024-03-06T07:28:36.802815+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802815+0000","last_undegraded":"2024-03-06T07:28:36.802815+0000","last_fullsized":"2024-03-06T07:28:36.802815+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804619+0000","last_change":"2024-03-06T07:28:36.804619+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804619+0000","last_undegraded":"2024-03-06T07:28:36.804619+0000","last_fullsized":"2024-03-06T07:28:36.804619+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798509+0000","last_change":"2024-03-06T07:28:36.798509+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798509+0000","last_undegraded":"2024-03-06T07:28:36.798509+0000","last_fullsized":"2024-03-06T07:28:36.798509+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804726+0000","last_change":"2024-03-06T07:28:36.804726+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804726+0000","last_undegraded":"2024-03-06T07:28:36.804726+0000","last_fullsized":"2024-03-06T07:28:36.804726+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798535+0000","last_change":"2024-03-06T07:28:36.798535+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798535+0000","last_undegraded":"2024-03-06T07:28:36.798535+0000","last_fullsized":"2024-03-06T07:28:36.798535+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804323+0000","last_change":"2024-03-06T07:28:36.804323+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804323+0000","last_undegraded":"2024-03-06T07:28:36.804323+0000","last_fullsized":"2024-03-06T07:28:36.804323+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803816+0000","last_change":"2024-03-06T07:28:36.803816+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803816+0000","last_undegraded":"2024-03-06T07:28:36.803816+0000","last_fullsized":"2024-03-06T07:28:36.803816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803599+0000","last_change":"2024-03-06T07:28:36.803599+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803599+0000","last_undegraded":"2024-03-06T07:28:36.803599+0000","last_fullsized":"2024-03-06T07:28:36.803599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806309+0000","last_change":"2024-03-06T07:28:36.806309+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806309+0000","last_undegraded":"2024-03-06T07:28:36.806309+0000","last_fullsized":"2024-03-06T07:28:36.806309+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797983+0000","last_change":"2024-03-06T07:28:36.797983+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797983+0000","last_undegraded":"2024-03-06T07:28:36.797983+0000","last_fullsized":"2024-03-06T07:28:36.797983+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815202+0000","last_change":"2024-03-06T07:28:36.812840+0000","last_active":"2024-03-06T07:28:36.815202+0000","last_peered":"2024-03-06T07:28:36.815202+0000","last_clean":"2024-03-06T07:28:36.815202+0000","last_became_active":"2024-03-06T07:28:36.812374+0000","last_became_peered":"2024-03-06T07:28:36.812374+0000","last_unstale":"2024-03-06T07:28:36.815202+0000","last_undegraded":"2024-03-06T07:28:36.815202+0000","last_fullsized":"2024-03-06T07:28:36.815202+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801659+0000","last_change":"2024-03-06T07:28:36.801659+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801659+0000","last_undegraded":"2024-03-06T07:28:36.801659+0000","last_fullsized":"2024-03-06T07:28:36.801659+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.810757+0000","last_change":"2024-03-06T07:28:36.808862+0000","last_active":"2024-03-06T07:28:36.810757+0000","last_peered":"2024-03-06T07:28:36.810757+0000","last_clean":"2024-03-06T07:28:36.810757+0000","last_became_active":"2024-03-06T07:28:36.808596+0000","last_became_peered":"2024-03-06T07:28:36.808596+0000","last_unstale":"2024-03-06T07:28:36.810757+0000","last_undegraded":"2024-03-06T07:28:36.810757+0000","last_fullsized":"2024-03-06T07:28:36.810757+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:40.196486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803967+0000","last_change":"2024-03-06T07:28:36.803967+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803967+0000","last_undegraded":"2024-03-06T07:28:36.803967+0000","last_fullsized":"2024-03-06T07:28:36.803967+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811791+0000","last_change":"2024-03-06T07:28:36.806718+0000","last_active":"2024-03-06T07:28:36.811791+0000","last_peered":"2024-03-06T07:28:36.811791+0000","last_clean":"2024-03-06T07:28:36.811791+0000","last_became_active":"2024-03-06T07:28:36.806447+0000","last_became_peered":"2024-03-06T07:28:36.806447+0000","last_unstale":"2024-03-06T07:28:36.811791+0000","last_undegraded":"2024-03-06T07:28:36.811791+0000","last_fullsized":"2024-03-06T07:28:36.811791+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:03.561828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806292+0000","last_change":"2024-03-06T07:28:36.806292+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806292+0000","last_undegraded":"2024-03-06T07:28:36.806292+0000","last_fullsized":"2024-03-06T07:28:36.806292+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817337+0000","last_change":"2024-03-06T07:28:36.812550+0000","last_active":"2024-03-06T07:28:36.817337+0000","last_peered":"2024-03-06T07:28:36.817337+0000","last_clean":"2024-03-06T07:28:36.817337+0000","last_became_active":"2024-03-06T07:28:36.812312+0000","last_became_peered":"2024-03-06T07:28:36.812312+0000","last_unstale":"2024-03-06T07:28:36.817337+0000","last_undegraded":"2024-03-06T07:28:36.817337+0000","last_fullsized":"2024-03-06T07:28:36.817337+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:52.682797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798999+0000","last_change":"2024-03-06T07:28:36.798999+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798999+0000","last_undegraded":"2024-03-06T07:28:36.798999+0000","last_fullsized":"2024-03-06T07:28:36.798999+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817263+0000","last_change":"2024-03-06T07:28:36.812168+0000","last_active":"2024-03-06T07:28:36.817263+0000","last_peered":"2024-03-06T07:28:36.817263+0000","last_clean":"2024-03-06T07:28:36.817263+0000","last_became_active":"2024-03-06T07:28:36.811914+0000","last_became_peered":"2024-03-06T07:28:36.811914+0000","last_unstale":"2024-03-06T07:28:36.817263+0000","last_undegraded":"2024-03-06T07:28:36.817263+0000","last_fullsized":"2024-03-06T07:28:36.817263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:25.915577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805322+0000","last_change":"2024-03-06T07:28:36.805322+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805322+0000","last_undegraded":"2024-03-06T07:28:36.805322+0000","last_fullsized":"2024-03-06T07:28:36.805322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813993+0000","last_change":"2024-03-06T07:28:36.808734+0000","last_active":"2024-03-06T07:28:36.813993+0000","last_peered":"2024-03-06T07:28:36.813993+0000","last_clean":"2024-03-06T07:28:36.813993+0000","last_became_active":"2024-03-06T07:28:36.808474+0000","last_became_peered":"2024-03-06T07:28:36.808474+0000","last_unstale":"2024-03-06T07:28:36.813993+0000","last_undegraded":"2024-03-06T07:28:36.813993+0000","last_fullsized":"2024-03-06T07:28:36.813993+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801103+0000","last_change":"2024-03-06T07:28:36.801103+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801103+0000","last_undegraded":"2024-03-06T07:28:36.801103+0000","last_fullsized":"2024-03-06T07:28:36.801103+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.812432+0000","last_change":"2024-03-06T07:28:36.811802+0000","last_active":"2024-03-06T07:28:36.812432+0000","last_peered":"2024-03-06T07:28:36.812432+0000","last_clean":"2024-03-06T07:28:36.812432+0000","last_became_active":"2024-03-06T07:28:36.811529+0000","last_became_peered":"2024-03-06T07:28:36.811529+0000","last_unstale":"2024-03-06T07:28:36.812432+0000","last_undegraded":"2024-03-06T07:28:36.812432+0000","last_fullsized":"2024-03-06T07:28:36.812432+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804090+0000","last_change":"2024-03-06T07:28:36.804090+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804090+0000","last_undegraded":"2024-03-06T07:28:36.804090+0000","last_fullsized":"2024-03-06T07:28:36.804090+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817205+0000","last_change":"2024-03-06T07:28:36.811162+0000","last_active":"2024-03-06T07:28:36.817205+0000","last_peered":"2024-03-06T07:28:36.817205+0000","last_clean":"2024-03-06T07:28:36.817205+0000","last_became_active":"2024-03-06T07:28:36.810868+0000","last_became_peered":"2024-03-06T07:28:36.810868+0000","last_unstale":"2024-03-06T07:28:36.817205+0000","last_undegraded":"2024-03-06T07:28:36.817205+0000","last_fullsized":"2024-03-06T07:28:36.817205+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:24.650548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804395+0000","last_change":"2024-03-06T07:28:36.804395+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804395+0000","last_undegraded":"2024-03-06T07:28:36.804395+0000","last_fullsized":"2024-03-06T07:28:36.804395+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809975+0000","last_change":"2024-03-06T07:28:36.807342+0000","last_active":"2024-03-06T07:28:36.809975+0000","last_peered":"2024-03-06T07:28:36.809975+0000","last_clean":"2024-03-06T07:28:36.809975+0000","last_became_active":"2024-03-06T07:28:36.807010+0000","last_became_peered":"2024-03-06T07:28:36.807010+0000","last_unstale":"2024-03-06T07:28:36.809975+0000","last_undegraded":"2024-03-06T07:28:36.809975+0000","last_fullsized":"2024-03-06T07:28:36.809975+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:36.673286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799792+0000","last_change":"2024-03-06T07:28:36.799792+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799792+0000","last_undegraded":"2024-03-06T07:28:36.799792+0000","last_fullsized":"2024-03-06T07:28:36.799792+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817215+0000","last_change":"2024-03-06T07:28:36.812151+0000","last_active":"2024-03-06T07:28:36.817215+0000","last_peered":"2024-03-06T07:28:36.817215+0000","last_clean":"2024-03-06T07:28:36.817215+0000","last_became_active":"2024-03-06T07:28:36.811892+0000","last_became_peered":"2024-03-06T07:28:36.811892+0000","last_unstale":"2024-03-06T07:28:36.817215+0000","last_undegraded":"2024-03-06T07:28:36.817215+0000","last_fullsized":"2024-03-06T07:28:36.817215+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803508+0000","last_change":"2024-03-06T07:28:36.803508+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803508+0000","last_undegraded":"2024-03-06T07:28:36.803508+0000","last_fullsized":"2024-03-06T07:28:36.803508+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.812096+0000","last_change":"2024-03-06T07:28:36.811322+0000","last_active":"2024-03-06T07:28:36.812096+0000","last_peered":"2024-03-06T07:28:36.812096+0000","last_clean":"2024-03-06T07:28:36.812096+0000","last_became_active":"2024-03-06T07:28:36.810594+0000","last_became_peered":"2024-03-06T07:28:36.810594+0000","last_unstale":"2024-03-06T07:28:36.812096+0000","last_undegraded":"2024-03-06T07:28:36.812096+0000","last_fullsized":"2024-03-06T07:28:36.812096+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805018+0000","last_change":"2024-03-06T07:28:36.805018+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805018+0000","last_undegraded":"2024-03-06T07:28:36.805018+0000","last_fullsized":"2024-03-06T07:28:36.805018+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809986+0000","last_change":"2024-03-06T07:28:36.808354+0000","last_active":"2024-03-06T07:28:36.809986+0000","last_peered":"2024-03-06T07:28:36.809986+0000","last_clean":"2024-03-06T07:28:36.809986+0000","last_became_active":"2024-03-06T07:28:36.808076+0000","last_became_peered":"2024-03-06T07:28:36.808076+0000","last_unstale":"2024-03-06T07:28:36.809986+0000","last_undegraded":"2024-03-06T07:28:36.809986+0000","last_fullsized":"2024-03-06T07:28:36.809986+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.762723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803359+0000","last_change":"2024-03-06T07:28:36.803359+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803359+0000","last_undegraded":"2024-03-06T07:28:36.803359+0000","last_fullsized":"2024-03-06T07:28:36.803359+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813932+0000","last_change":"2024-03-06T07:28:36.810704+0000","last_active":"2024-03-06T07:28:36.813932+0000","last_peered":"2024-03-06T07:28:36.813932+0000","last_clean":"2024-03-06T07:28:36.813932+0000","last_became_active":"2024-03-06T07:28:36.809890+0000","last_became_peered":"2024-03-06T07:28:36.809890+0000","last_unstale":"2024-03-06T07:28:36.813932+0000","last_undegraded":"2024-03-06T07:28:36.813932+0000","last_fullsized":"2024-03-06T07:28:36.813932+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:03.819520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804096+0000","last_change":"2024-03-06T07:28:36.804096+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804096+0000","last_undegraded":"2024-03-06T07:28:36.804096+0000","last_fullsized":"2024-03-06T07:28:36.804096+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817100+0000","last_change":"2024-03-06T07:28:36.810746+0000","last_active":"2024-03-06T07:28:36.817100+0000","last_peered":"2024-03-06T07:28:36.817100+0000","last_clean":"2024-03-06T07:28:36.817100+0000","last_became_active":"2024-03-06T07:28:36.810475+0000","last_became_peered":"2024-03-06T07:28:36.810475+0000","last_unstale":"2024-03-06T07:28:36.817100+0000","last_undegraded":"2024-03-06T07:28:36.817100+0000","last_fullsized":"2024-03-06T07:28:36.817100+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:41.135891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801824+0000","last_change":"2024-03-06T07:28:36.801824+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801824+0000","last_undegraded":"2024-03-06T07:28:36.801824+0000","last_fullsized":"2024-03-06T07:28:36.801824+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.810890+0000","last_change":"2024-03-06T07:28:36.809374+0000","last_active":"2024-03-06T07:28:36.810890+0000","last_peered":"2024-03-06T07:28:36.810890+0000","last_clean":"2024-03-06T07:28:36.810890+0000","last_became_active":"2024-03-06T07:28:36.807635+0000","last_became_peered":"2024-03-06T07:28:36.807635+0000","last_unstale":"2024-03-06T07:28:36.810890+0000","last_undegraded":"2024-03-06T07:28:36.810890+0000","last_fullsized":"2024-03-06T07:28:36.810890+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:40.196486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801766+0000","last_change":"2024-03-06T07:28:36.801766+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801766+0000","last_undegraded":"2024-03-06T07:28:36.801766+0000","last_fullsized":"2024-03-06T07:28:36.801766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815277+0000","last_change":"2024-03-06T07:28:36.814194+0000","last_active":"2024-03-06T07:28:36.815277+0000","last_peered":"2024-03-06T07:28:36.815277+0000","last_clean":"2024-03-06T07:28:36.815277+0000","last_became_active":"2024-03-06T07:28:36.813872+0000","last_became_peered":"2024-03-06T07:28:36.813872+0000","last_unstale":"2024-03-06T07:28:36.815277+0000","last_undegraded":"2024-03-06T07:28:36.815277+0000","last_fullsized":"2024-03-06T07:28:36.815277+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799285+0000","last_change":"2024-03-06T07:28:36.799285+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799285+0000","last_undegraded":"2024-03-06T07:28:36.799285+0000","last_fullsized":"2024-03-06T07:28:36.799285+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811218+0000","last_change":"2024-03-06T07:28:36.808348+0000","last_active":"2024-03-06T07:28:36.811218+0000","last_peered":"2024-03-06T07:28:36.811218+0000","last_clean":"2024-03-06T07:28:36.811218+0000","last_became_active":"2024-03-06T07:28:36.808035+0000","last_became_peered":"2024-03-06T07:28:36.808035+0000","last_unstale":"2024-03-06T07:28:36.811218+0000","last_undegraded":"2024-03-06T07:28:36.811218+0000","last_fullsized":"2024-03-06T07:28:36.811218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:02.210672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797961+0000","last_change":"2024-03-06T07:28:36.797961+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797961+0000","last_undegraded":"2024-03-06T07:28:36.797961+0000","last_fullsized":"2024-03-06T07:28:36.797961+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.818360+0000","last_change":"2024-03-06T07:28:36.817966+0000","last_active":"2024-03-06T07:28:36.818360+0000","last_peered":"2024-03-06T07:28:36.818360+0000","last_clean":"2024-03-06T07:28:36.818360+0000","last_became_active":"2024-03-06T07:28:36.817630+0000","last_became_peered":"2024-03-06T07:28:36.817630+0000","last_unstale":"2024-03-06T07:28:36.818360+0000","last_undegraded":"2024-03-06T07:28:36.818360+0000","last_fullsized":"2024-03-06T07:28:36.818360+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803589+0000","last_change":"2024-03-06T07:28:36.803589+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803589+0000","last_undegraded":"2024-03-06T07:28:36.803589+0000","last_fullsized":"2024-03-06T07:28:36.803589+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811657+0000","last_change":"2024-03-06T07:28:36.809671+0000","last_active":"2024-03-06T07:28:36.811657+0000","last_peered":"2024-03-06T07:28:36.811657+0000","last_clean":"2024-03-06T07:28:36.811657+0000","last_became_active":"2024-03-06T07:28:36.809405+0000","last_became_peered":"2024-03-06T07:28:36.809405+0000","last_unstale":"2024-03-06T07:28:36.811657+0000","last_undegraded":"2024-03-06T07:28:36.811657+0000","last_fullsized":"2024-03-06T07:28:36.811657+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802926+0000","last_change":"2024-03-06T07:28:36.802926+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802926+0000","last_undegraded":"2024-03-06T07:28:36.802926+0000","last_fullsized":"2024-03-06T07:28:36.802926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809915+0000","last_change":"2024-03-06T07:28:36.807565+0000","last_active":"2024-03-06T07:28:36.809915+0000","last_peered":"2024-03-06T07:28:36.809915+0000","last_clean":"2024-03-06T07:28:36.809915+0000","last_became_active":"2024-03-06T07:28:36.807266+0000","last_became_peered":"2024-03-06T07:28:36.807266+0000","last_unstale":"2024-03-06T07:28:36.809915+0000","last_undegraded":"2024-03-06T07:28:36.809915+0000","last_fullsized":"2024-03-06T07:28:36.809915+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.762723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804864+0000","last_change":"2024-03-06T07:28:36.804864+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804864+0000","last_undegraded":"2024-03-06T07:28:36.804864+0000","last_fullsized":"2024-03-06T07:28:36.804864+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.806027+0000","last_change":"2024-03-06T07:28:36.804975+0000","last_active":"2024-03-06T07:28:36.806027+0000","last_peered":"2024-03-06T07:28:36.806027+0000","last_clean":"2024-03-06T07:28:36.806027+0000","last_became_active":"2024-03-06T07:28:36.804697+0000","last_became_peered":"2024-03-06T07:28:36.804697+0000","last_unstale":"2024-03-06T07:28:36.806027+0000","last_undegraded":"2024-03-06T07:28:36.806027+0000","last_fullsized":"2024-03-06T07:28:36.806027+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803649+0000","last_change":"2024-03-06T07:28:36.803649+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803649+0000","last_undegraded":"2024-03-06T07:28:36.803649+0000","last_fullsized":"2024-03-06T07:28:36.803649+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817037+0000","last_change":"2024-03-06T07:28:36.810758+0000","last_active":"2024-03-06T07:28:36.817037+0000","last_peered":"2024-03-06T07:28:36.817037+0000","last_clean":"2024-03-06T07:28:36.817037+0000","last_became_active":"2024-03-06T07:28:36.810492+0000","last_became_peered":"2024-03-06T07:28:36.810492+0000","last_unstale":"2024-03-06T07:28:36.817037+0000","last_undegraded":"2024-03-06T07:28:36.817037+0000","last_fullsized":"2024-03-06T07:28:36.817037+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:55.327149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804063+0000","last_change":"2024-03-06T07:28:36.804063+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804063+0000","last_undegraded":"2024-03-06T07:28:36.804063+0000","last_fullsized":"2024-03-06T07:28:36.804063+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811507+0000","last_change":"2024-03-06T07:28:36.809882+0000","last_active":"2024-03-06T07:28:36.811507+0000","last_peered":"2024-03-06T07:28:36.811507+0000","last_clean":"2024-03-06T07:28:36.811507+0000","last_became_active":"2024-03-06T07:28:36.809035+0000","last_became_peered":"2024-03-06T07:28:36.809035+0000","last_unstale":"2024-03-06T07:28:36.811507+0000","last_undegraded":"2024-03-06T07:28:36.811507+0000","last_fullsized":"2024-03-06T07:28:36.811507+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:26.232645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806573+0000","last_change":"2024-03-06T07:28:36.806573+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806573+0000","last_undegraded":"2024-03-06T07:28:36.806573+0000","last_fullsized":"2024-03-06T07:28:36.806573+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.816568+0000","last_change":"2024-03-06T07:28:36.812497+0000","last_active":"2024-03-06T07:28:36.816568+0000","last_peered":"2024-03-06T07:28:36.816568+0000","last_clean":"2024-03-06T07:28:36.816568+0000","last_became_active":"2024-03-06T07:28:36.812253+0000","last_became_peered":"2024-03-06T07:28:36.812253+0000","last_unstale":"2024-03-06T07:28:36.816568+0000","last_undegraded":"2024-03-06T07:28:36.816568+0000","last_fullsized":"2024-03-06T07:28:36.816568+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.721283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802330+0000","last_change":"2024-03-06T07:28:36.802330+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802330+0000","last_undegraded":"2024-03-06T07:28:36.802330+0000","last_fullsized":"2024-03-06T07:28:36.802330+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815234+0000","last_change":"2024-03-06T07:28:36.812931+0000","last_active":"2024-03-06T07:28:36.815234+0000","last_peered":"2024-03-06T07:28:36.815234+0000","last_clean":"2024-03-06T07:28:36.815234+0000","last_became_active":"2024-03-06T07:28:36.812619+0000","last_became_peered":"2024-03-06T07:28:36.812619+0000","last_unstale":"2024-03-06T07:28:36.815234+0000","last_undegraded":"2024-03-06T07:28:36.815234+0000","last_fullsized":"2024-03-06T07:28:36.815234+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803589+0000","last_change":"2024-03-06T07:28:36.803589+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803589+0000","last_undegraded":"2024-03-06T07:28:36.803589+0000","last_fullsized":"2024-03-06T07:28:36.803589+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817286+0000","last_change":"2024-03-06T07:28:36.811623+0000","last_active":"2024-03-06T07:28:36.817286+0000","last_peered":"2024-03-06T07:28:36.817286+0000","last_clean":"2024-03-06T07:28:36.817286+0000","last_became_active":"2024-03-06T07:28:36.811361+0000","last_became_peered":"2024-03-06T07:28:36.811361+0000","last_unstale":"2024-03-06T07:28:36.817286+0000","last_undegraded":"2024-03-06T07:28:36.817286+0000","last_fullsized":"2024-03-06T07:28:36.817286+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.186633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806172+0000","last_change":"2024-03-06T07:28:36.806172+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806172+0000","last_undegraded":"2024-03-06T07:28:36.806172+0000","last_fullsized":"2024-03-06T07:28:36.806172+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811226+0000","last_change":"2024-03-06T07:28:36.807851+0000","last_active":"2024-03-06T07:28:36.811226+0000","last_peered":"2024-03-06T07:28:36.811226+0000","last_clean":"2024-03-06T07:28:36.811226+0000","last_became_active":"2024-03-06T07:28:36.807590+0000","last_became_peered":"2024-03-06T07:28:36.807590+0000","last_unstale":"2024-03-06T07:28:36.811226+0000","last_undegraded":"2024-03-06T07:28:36.811226+0000","last_fullsized":"2024-03-06T07:28:36.811226+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:19.630579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802948+0000","last_change":"2024-03-06T07:28:36.802948+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802948+0000","last_undegraded":"2024-03-06T07:28:36.802948+0000","last_fullsized":"2024-03-06T07:28:36.802948+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.814263+0000","last_change":"2024-03-06T07:28:36.808681+0000","last_active":"2024-03-06T07:28:36.814263+0000","last_peered":"2024-03-06T07:28:36.814263+0000","last_clean":"2024-03-06T07:28:36.814263+0000","last_became_active":"2024-03-06T07:28:36.808407+0000","last_became_peered":"2024-03-06T07:28:36.808407+0000","last_unstale":"2024-03-06T07:28:36.814263+0000","last_undegraded":"2024-03-06T07:28:36.814263+0000","last_fullsized":"2024-03-06T07:28:36.814263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:41.544227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798604+0000","last_change":"2024-03-06T07:28:36.798604+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798604+0000","last_undegraded":"2024-03-06T07:28:36.798604+0000","last_fullsized":"2024-03-06T07:28:36.798604+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815229+0000","last_change":"2024-03-06T07:28:36.812531+0000","last_active":"2024-03-06T07:28:36.815229+0000","last_peered":"2024-03-06T07:28:36.815229+0000","last_clean":"2024-03-06T07:28:36.815229+0000","last_became_active":"2024-03-06T07:28:36.812195+0000","last_became_peered":"2024-03-06T07:28:36.812195+0000","last_unstale":"2024-03-06T07:28:36.815229+0000","last_undegraded":"2024-03-06T07:28:36.815229+0000","last_fullsized":"2024-03-06T07:28:36.815229+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797980+0000","last_change":"2024-03-06T07:28:36.797980+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797980+0000","last_undegraded":"2024-03-06T07:28:36.797980+0000","last_fullsized":"2024-03-06T07:28:36.797980+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817218+0000","last_change":"2024-03-06T07:28:36.811478+0000","last_active":"2024-03-06T07:28:36.817218+0000","last_peered":"2024-03-06T07:28:36.817218+0000","last_clean":"2024-03-06T07:28:36.817218+0000","last_became_active":"2024-03-06T07:28:36.811250+0000","last_became_peered":"2024-03-06T07:28:36.811250+0000","last_unstale":"2024-03-06T07:28:36.817218+0000","last_undegraded":"2024-03-06T07:28:36.817218+0000","last_fullsized":"2024-03-06T07:28:36.817218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803573+0000","last_change":"2024-03-06T07:28:36.803573+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803573+0000","last_undegraded":"2024-03-06T07:28:36.803573+0000","last_fullsized":"2024-03-06T07:28:36.803573+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811488+0000","last_change":"2024-03-06T07:28:36.809706+0000","last_active":"2024-03-06T07:28:36.811488+0000","last_peered":"2024-03-06T07:28:36.811488+0000","last_clean":"2024-03-06T07:28:36.811488+0000","last_became_active":"2024-03-06T07:28:36.809345+0000","last_became_peered":"2024-03-06T07:28:36.809345+0000","last_unstale":"2024-03-06T07:28:36.811488+0000","last_undegraded":"2024-03-06T07:28:36.811488+0000","last_fullsized":"2024-03-06T07:28:36.811488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:25.869243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804620+0000","last_change":"2024-03-06T07:28:36.804620+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804620+0000","last_undegraded":"2024-03-06T07:28:36.804620+0000","last_fullsized":"2024-03-06T07:28:36.804620+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815176+0000","last_change":"2024-03-06T07:28:36.812624+0000","last_active":"2024-03-06T07:28:36.815176+0000","last_peered":"2024-03-06T07:28:36.815176+0000","last_clean":"2024-03-06T07:28:36.815176+0000","last_became_active":"2024-03-06T07:28:36.812331+0000","last_became_peered":"2024-03-06T07:28:36.812331+0000","last_unstale":"2024-03-06T07:28:36.815176+0000","last_undegraded":"2024-03-06T07:28:36.815176+0000","last_fullsized":"2024-03-06T07:28:36.815176+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":359,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.805855+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:28:36.805855+0000","last_peered":"2024-03-06T07:28:36.805855+0000","last_clean":"2024-03-06T07:28:36.805855+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:28:36.805855+0000","last_undegraded":"2024-03-06T07:28:36.805855+0000","last_fullsized":"2024-03-06T07:28:36.805855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":2154496,"data_stored":1270158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":679790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":2154496,"data_stored":1270158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:40.879 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:41.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.015+0000 7ff3542b5640 1 Processor -- start 2024-03-06T07:28:41.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.015+0000 7ff3542b5640 1 -- start start 2024-03-06T07:28:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff3542b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff34c15dfe0 0x7ff34c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff3542b5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff34c0d29c0 0x7ff34c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff3542b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34c0b3430 con 0x7ff34c0cf830 2024-03-06T07:28:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff3542b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0b31e0 con 0x7ff34c15dfe0 2024-03-06T07:28:41.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff3542b5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34c0b3d60 con 0x7ff34c0d29c0 2024-03-06T07:28:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff352051640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff34c15dfe0 0x7ff34c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff351850640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff34c0cf830 0x7ff34c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59154/0 (socket says 172.21.15.72:59154) 2024-03-06T07:28:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff351850640 1 -- 172.21.15.72:0/2518427455 learned_addr learned my addr 172.21.15.72:0/2518427455 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.016+0000 7ff352852640 1 --2- 172.21.15.72:0/2518427455 >> v2:172.21.15.146:3300/0 conn(0x7ff34c0d29c0 0x7ff34c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff352051640 1 -- 172.21.15.72:0/2518427455 >> v2:172.21.15.146:3300/0 conn(0x7ff34c0d29c0 msgr2=0x7ff34c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff352051640 1 --2- 172.21.15.72:0/2518427455 >> v2:172.21.15.146:3300/0 conn(0x7ff34c0d29c0 0x7ff34c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff352051640 1 -- 172.21.15.72:0/2518427455 >> v1:172.21.15.72:6789/0 conn(0x7ff34c0cf830 legacy=0x7ff34c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff352051640 1 -- 172.21.15.72:0/2518427455 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c0b3710 con 0x7ff34c15dfe0 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff35104f640 1 -- 172.21.15.72:0/2518427455 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707279162 0 0) 0x7ff34c0b3430 con 0x7ff34c0cf830 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff352051640 1 --2- 172.21.15.72:0/2518427455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0cf2f0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0b3cd0 tx=0x7ff34802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e95b58d734afd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.017+0000 7ff35104f640 1 -- 172.21.15.72:0/2518427455 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348035020 con 0x7ff34c15dfe0 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff35104f640 1 -- 172.21.15.72:0/2518427455 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348031830 con 0x7ff34c15dfe0 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff35104f640 1 -- 172.21.15.72:0/2518427455 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348031c00 con 0x7ff34c15dfe0 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 -- 172.21.15.72:0/2518427455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c15dfe0 msgr2=0x7ff34c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2518427455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0cf2f0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0b3cd0 tx=0x7ff34802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 -- 172.21.15.72:0/2518427455 shutdown_connections 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2518427455 >> v2:172.21.15.146:3300/0 conn(0x7ff34c0d29c0 0x7ff34c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2518427455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0cf2f0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 -- 172.21.15.72:0/2518427455 >> 172.21.15.72:0/2518427455 conn(0x7ff34c0d3cd0 msgr2=0x7ff34c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 -- 172.21.15.72:0/2518427455 shutdown_connections 2024-03-06T07:28:41.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.018+0000 7ff3542b5640 1 -- 172.21.15.72:0/2518427455 wait complete. 2024-03-06T07:28:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.020+0000 7ff3542b5640 1 Processor -- start 2024-03-06T07:28:41.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.020+0000 7ff3542b5640 1 -- start start 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff3542b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 0x7ff34c15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff3542b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff34c0d29c0 0x7ff34c1f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff3542b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff3542b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0d1df0 con 0x7ff34c0cf830 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff3542b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c0d1c70 con 0x7ff34c15dfe0 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff3542b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff34c15e3c0 con 0x7ff34c0d29c0 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff352051640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 0x7ff34c15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff352051640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 0x7ff34c15bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59310/0 (socket says 172.21.15.72:59310) 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.021+0000 7ff352051640 1 -- 172.21.15.72:0/2156158036 learned_addr learned my addr 172.21.15.72:0/2156158036 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352051640 1 -- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff34c0d29c0 msgr2=0x7ff34c1f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff351850640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff34c0d29c0 0x7ff34c1f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352051640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff34c0d29c0 0x7ff34c1f4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352852640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352051640 1 -- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff34c15dfe0 msgr2=0x7ff34c0b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352051640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352051640 1 -- 172.21.15.72:0/2156158036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c1f7280 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff352051640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 0x7ff34c15bb70 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ff348000c00 tx=0x7ff3480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348041070 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348004250 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff34c1f7560 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff348035040 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.022+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34c1f7b20 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.023+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34c1f7d60 con 0x7ff34c0cf830 2024-03-06T07:28:41.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.024+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff348038020 con 0x7ff34c0cf830 2024-03-06T07:28:41.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.024+0000 7ff342ffd640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff33006b500 0x7ff33006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.024+0000 7ff351850640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff33006b500 0x7ff33006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.024+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3480abb30 con 0x7ff34c0cf830 2024-03-06T07:28:41.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.024+0000 7ff351850640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff33006b500 0x7ff33006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff33c007fd0 tx=0x7ff33c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.028+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff34c1f7d60 con 0x7ff34c0cf830 2024-03-06T07:28:41.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.239+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff34c04f330 con 0x7ff33006b500 2024-03-06T07:28:41.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.245+0000 7ff342ffd640 1 -- 172.21.15.72:0/2156158036 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247075 (secure 0 0 0) 0x7ff34c04f330 con 0x7ff33006b500 2024-03-06T07:28:41.251 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:41.252 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:28:41.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff33006b500 msgr2=0x7ff33006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff33006b500 0x7ff33006d9c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7ff33c007fd0 tx=0x7ff33c009040 comp rx=0 tx=0).stop 2024-03-06T07:28:41.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 msgr2=0x7ff34c15bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 0x7ff34c15bb70 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ff348000c00 tx=0x7ff3480046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 shutdown_connections 2024-03-06T07:28:41.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff33006b500 0x7ff33006d9c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff34c15dfe0 0x7ff34c0b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff34c0d29c0 0x7ff34c1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 --2- 172.21.15.72:0/2156158036 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff34c0cf830 0x7ff34c15bb70 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 >> 172.21.15.72:0/2156158036 conn(0x7ff34c0d3cd0 msgr2=0x7ff34c0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 shutdown_connections 2024-03-06T07:28:41.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.248+0000 7ff3542b5640 1 -- 172.21.15.72:0/2156158036 wait complete. 2024-03-06T07:28:41.261 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-03-06T07:28:40.023279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329344,"kb_used_data":13292,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645376,"statfs":{"total":773094113280,"available":772756865024,"internally_reserved":0,"allocated":13611008,"data_stored":6587681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.234114"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806567+0000","last_change":"2024-03-06T07:28:36.806567+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806567+0000","last_undegraded":"2024-03-06T07:28:36.806567+0000","last_fullsized":"2024-03-06T07:28:36.806567+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806490+0000","last_change":"2024-03-06T07:28:36.806490+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806490+0000","last_undegraded":"2024-03-06T07:28:36.806490+0000","last_fullsized":"2024-03-06T07:28:36.806490+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.800755+0000","last_change":"2024-03-06T07:28:36.800755+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.800755+0000","last_undegraded":"2024-03-06T07:28:36.800755+0000","last_fullsized":"2024-03-06T07:28:36.800755+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806501+0000","last_change":"2024-03-06T07:28:36.806501+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806501+0000","last_undegraded":"2024-03-06T07:28:36.806501+0000","last_fullsized":"2024-03-06T07:28:36.806501+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804424+0000","last_change":"2024-03-06T07:28:36.804424+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804424+0000","last_undegraded":"2024-03-06T07:28:36.804424+0000","last_fullsized":"2024-03-06T07:28:36.804424+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.807930+0000","last_change":"2024-03-06T07:28:36.807930+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.807930+0000","last_undegraded":"2024-03-06T07:28:36.807930+0000","last_fullsized":"2024-03-06T07:28:36.807930+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803468+0000","last_change":"2024-03-06T07:28:36.803468+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803468+0000","last_undegraded":"2024-03-06T07:28:36.803468+0000","last_fullsized":"2024-03-06T07:28:36.803468+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801587+0000","last_change":"2024-03-06T07:28:36.801587+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801587+0000","last_undegraded":"2024-03-06T07:28:36.801587+0000","last_fullsized":"2024-03-06T07:28:36.801587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802274+0000","last_change":"2024-03-06T07:28:36.802274+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802274+0000","last_undegraded":"2024-03-06T07:28:36.802274+0000","last_fullsized":"2024-03-06T07:28:36.802274+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805084+0000","last_change":"2024-03-06T07:28:36.805084+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805084+0000","last_undegraded":"2024-03-06T07:28:36.805084+0000","last_fullsized":"2024-03-06T07:28:36.805084+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805445+0000","last_change":"2024-03-06T07:28:36.805445+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805445+0000","last_undegraded":"2024-03-06T07:28:36.805445+0000","last_fullsized":"2024-03-06T07:28:36.805445+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801140+0000","last_change":"2024-03-06T07:28:36.801140+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801140+0000","last_undegraded":"2024-03-06T07:28:36.801140+0000","last_fullsized":"2024-03-06T07:28:36.801140+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806487+0000","last_change":"2024-03-06T07:28:36.806487+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806487+0000","last_undegraded":"2024-03-06T07:28:36.806487+0000","last_fullsized":"2024-03-06T07:28:36.806487+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804641+0000","last_change":"2024-03-06T07:28:36.804641+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804641+0000","last_undegraded":"2024-03-06T07:28:36.804641+0000","last_fullsized":"2024-03-06T07:28:36.804641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805149+0000","last_change":"2024-03-06T07:28:36.805149+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805149+0000","last_undegraded":"2024-03-06T07:28:36.805149+0000","last_fullsized":"2024-03-06T07:28:36.805149+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798464+0000","last_change":"2024-03-06T07:28:36.798464+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798464+0000","last_undegraded":"2024-03-06T07:28:36.798464+0000","last_fullsized":"2024-03-06T07:28:36.798464+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803081+0000","last_change":"2024-03-06T07:28:36.803081+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803081+0000","last_undegraded":"2024-03-06T07:28:36.803081+0000","last_fullsized":"2024-03-06T07:28:36.803081+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799143+0000","last_change":"2024-03-06T07:28:36.799143+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799143+0000","last_undegraded":"2024-03-06T07:28:36.799143+0000","last_fullsized":"2024-03-06T07:28:36.799143+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804151+0000","last_change":"2024-03-06T07:28:36.804151+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804151+0000","last_undegraded":"2024-03-06T07:28:36.804151+0000","last_fullsized":"2024-03-06T07:28:36.804151+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798472+0000","last_change":"2024-03-06T07:28:36.798472+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798472+0000","last_undegraded":"2024-03-06T07:28:36.798472+0000","last_fullsized":"2024-03-06T07:28:36.798472+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805603+0000","last_change":"2024-03-06T07:28:36.805603+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805603+0000","last_undegraded":"2024-03-06T07:28:36.805603+0000","last_fullsized":"2024-03-06T07:28:36.805603+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803692+0000","last_change":"2024-03-06T07:28:36.803692+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803692+0000","last_undegraded":"2024-03-06T07:28:36.803692+0000","last_fullsized":"2024-03-06T07:28:36.803692+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801386+0000","last_change":"2024-03-06T07:28:36.801386+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801386+0000","last_undegraded":"2024-03-06T07:28:36.801386+0000","last_fullsized":"2024-03-06T07:28:36.801386+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802815+0000","last_change":"2024-03-06T07:28:36.802815+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802815+0000","last_undegraded":"2024-03-06T07:28:36.802815+0000","last_fullsized":"2024-03-06T07:28:36.802815+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804619+0000","last_change":"2024-03-06T07:28:36.804619+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804619+0000","last_undegraded":"2024-03-06T07:28:36.804619+0000","last_fullsized":"2024-03-06T07:28:36.804619+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798509+0000","last_change":"2024-03-06T07:28:36.798509+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798509+0000","last_undegraded":"2024-03-06T07:28:36.798509+0000","last_fullsized":"2024-03-06T07:28:36.798509+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804726+0000","last_change":"2024-03-06T07:28:36.804726+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804726+0000","last_undegraded":"2024-03-06T07:28:36.804726+0000","last_fullsized":"2024-03-06T07:28:36.804726+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798535+0000","last_change":"2024-03-06T07:28:36.798535+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798535+0000","last_undegraded":"2024-03-06T07:28:36.798535+0000","last_fullsized":"2024-03-06T07:28:36.798535+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804323+0000","last_change":"2024-03-06T07:28:36.804323+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804323+0000","last_undegraded":"2024-03-06T07:28:36.804323+0000","last_fullsized":"2024-03-06T07:28:36.804323+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803816+0000","last_change":"2024-03-06T07:28:36.803816+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803816+0000","last_undegraded":"2024-03-06T07:28:36.803816+0000","last_fullsized":"2024-03-06T07:28:36.803816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803599+0000","last_change":"2024-03-06T07:28:36.803599+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803599+0000","last_undegraded":"2024-03-06T07:28:36.803599+0000","last_fullsized":"2024-03-06T07:28:36.803599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806309+0000","last_change":"2024-03-06T07:28:36.806309+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806309+0000","last_undegraded":"2024-03-06T07:28:36.806309+0000","last_fullsized":"2024-03-06T07:28:36.806309+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797983+0000","last_change":"2024-03-06T07:28:36.797983+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797983+0000","last_undegraded":"2024-03-06T07:28:36.797983+0000","last_fullsized":"2024-03-06T07:28:36.797983+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815202+0000","last_change":"2024-03-06T07:28:36.812840+0000","last_active":"2024-03-06T07:28:36.815202+0000","last_peered":"2024-03-06T07:28:36.815202+0000","last_clean":"2024-03-06T07:28:36.815202+0000","last_became_active":"2024-03-06T07:28:36.812374+0000","last_became_peered":"2024-03-06T07:28:36.812374+0000","last_unstale":"2024-03-06T07:28:36.815202+0000","last_undegraded":"2024-03-06T07:28:36.815202+0000","last_fullsized":"2024-03-06T07:28:36.815202+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801659+0000","last_change":"2024-03-06T07:28:36.801659+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801659+0000","last_undegraded":"2024-03-06T07:28:36.801659+0000","last_fullsized":"2024-03-06T07:28:36.801659+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.810757+0000","last_change":"2024-03-06T07:28:36.808862+0000","last_active":"2024-03-06T07:28:36.810757+0000","last_peered":"2024-03-06T07:28:36.810757+0000","last_clean":"2024-03-06T07:28:36.810757+0000","last_became_active":"2024-03-06T07:28:36.808596+0000","last_became_peered":"2024-03-06T07:28:36.808596+0000","last_unstale":"2024-03-06T07:28:36.810757+0000","last_undegraded":"2024-03-06T07:28:36.810757+0000","last_fullsized":"2024-03-06T07:28:36.810757+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:40.196486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803967+0000","last_change":"2024-03-06T07:28:36.803967+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803967+0000","last_undegraded":"2024-03-06T07:28:36.803967+0000","last_fullsized":"2024-03-06T07:28:36.803967+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811791+0000","last_change":"2024-03-06T07:28:36.806718+0000","last_active":"2024-03-06T07:28:36.811791+0000","last_peered":"2024-03-06T07:28:36.811791+0000","last_clean":"2024-03-06T07:28:36.811791+0000","last_became_active":"2024-03-06T07:28:36.806447+0000","last_became_peered":"2024-03-06T07:28:36.806447+0000","last_unstale":"2024-03-06T07:28:36.811791+0000","last_undegraded":"2024-03-06T07:28:36.811791+0000","last_fullsized":"2024-03-06T07:28:36.811791+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:03.561828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806292+0000","last_change":"2024-03-06T07:28:36.806292+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806292+0000","last_undegraded":"2024-03-06T07:28:36.806292+0000","last_fullsized":"2024-03-06T07:28:36.806292+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817337+0000","last_change":"2024-03-06T07:28:36.812550+0000","last_active":"2024-03-06T07:28:36.817337+0000","last_peered":"2024-03-06T07:28:36.817337+0000","last_clean":"2024-03-06T07:28:36.817337+0000","last_became_active":"2024-03-06T07:28:36.812312+0000","last_became_peered":"2024-03-06T07:28:36.812312+0000","last_unstale":"2024-03-06T07:28:36.817337+0000","last_undegraded":"2024-03-06T07:28:36.817337+0000","last_fullsized":"2024-03-06T07:28:36.817337+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:52.682797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798999+0000","last_change":"2024-03-06T07:28:36.798999+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798999+0000","last_undegraded":"2024-03-06T07:28:36.798999+0000","last_fullsized":"2024-03-06T07:28:36.798999+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817263+0000","last_change":"2024-03-06T07:28:36.812168+0000","last_active":"2024-03-06T07:28:36.817263+0000","last_peered":"2024-03-06T07:28:36.817263+0000","last_clean":"2024-03-06T07:28:36.817263+0000","last_became_active":"2024-03-06T07:28:36.811914+0000","last_became_peered":"2024-03-06T07:28:36.811914+0000","last_unstale":"2024-03-06T07:28:36.817263+0000","last_undegraded":"2024-03-06T07:28:36.817263+0000","last_fullsized":"2024-03-06T07:28:36.817263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:25.915577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805322+0000","last_change":"2024-03-06T07:28:36.805322+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805322+0000","last_undegraded":"2024-03-06T07:28:36.805322+0000","last_fullsized":"2024-03-06T07:28:36.805322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813993+0000","last_change":"2024-03-06T07:28:36.808734+0000","last_active":"2024-03-06T07:28:36.813993+0000","last_peered":"2024-03-06T07:28:36.813993+0000","last_clean":"2024-03-06T07:28:36.813993+0000","last_became_active":"2024-03-06T07:28:36.808474+0000","last_became_peered":"2024-03-06T07:28:36.808474+0000","last_unstale":"2024-03-06T07:28:36.813993+0000","last_undegraded":"2024-03-06T07:28:36.813993+0000","last_fullsized":"2024-03-06T07:28:36.813993+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801103+0000","last_change":"2024-03-06T07:28:36.801103+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801103+0000","last_undegraded":"2024-03-06T07:28:36.801103+0000","last_fullsized":"2024-03-06T07:28:36.801103+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.812432+0000","last_change":"2024-03-06T07:28:36.811802+0000","last_active":"2024-03-06T07:28:36.812432+0000","last_peered":"2024-03-06T07:28:36.812432+0000","last_clean":"2024-03-06T07:28:36.812432+0000","last_became_active":"2024-03-06T07:28:36.811529+0000","last_became_peered":"2024-03-06T07:28:36.811529+0000","last_unstale":"2024-03-06T07:28:36.812432+0000","last_undegraded":"2024-03-06T07:28:36.812432+0000","last_fullsized":"2024-03-06T07:28:36.812432+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804090+0000","last_change":"2024-03-06T07:28:36.804090+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804090+0000","last_undegraded":"2024-03-06T07:28:36.804090+0000","last_fullsized":"2024-03-06T07:28:36.804090+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817205+0000","last_change":"2024-03-06T07:28:36.811162+0000","last_active":"2024-03-06T07:28:36.817205+0000","last_peered":"2024-03-06T07:28:36.817205+0000","last_clean":"2024-03-06T07:28:36.817205+0000","last_became_active":"2024-03-06T07:28:36.810868+0000","last_became_peered":"2024-03-06T07:28:36.810868+0000","last_unstale":"2024-03-06T07:28:36.817205+0000","last_undegraded":"2024-03-06T07:28:36.817205+0000","last_fullsized":"2024-03-06T07:28:36.817205+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:24.650548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804395+0000","last_change":"2024-03-06T07:28:36.804395+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804395+0000","last_undegraded":"2024-03-06T07:28:36.804395+0000","last_fullsized":"2024-03-06T07:28:36.804395+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809975+0000","last_change":"2024-03-06T07:28:36.807342+0000","last_active":"2024-03-06T07:28:36.809975+0000","last_peered":"2024-03-06T07:28:36.809975+0000","last_clean":"2024-03-06T07:28:36.809975+0000","last_became_active":"2024-03-06T07:28:36.807010+0000","last_became_peered":"2024-03-06T07:28:36.807010+0000","last_unstale":"2024-03-06T07:28:36.809975+0000","last_undegraded":"2024-03-06T07:28:36.809975+0000","last_fullsized":"2024-03-06T07:28:36.809975+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:36.673286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799792+0000","last_change":"2024-03-06T07:28:36.799792+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799792+0000","last_undegraded":"2024-03-06T07:28:36.799792+0000","last_fullsized":"2024-03-06T07:28:36.799792+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817215+0000","last_change":"2024-03-06T07:28:36.812151+0000","last_active":"2024-03-06T07:28:36.817215+0000","last_peered":"2024-03-06T07:28:36.817215+0000","last_clean":"2024-03-06T07:28:36.817215+0000","last_became_active":"2024-03-06T07:28:36.811892+0000","last_became_peered":"2024-03-06T07:28:36.811892+0000","last_unstale":"2024-03-06T07:28:36.817215+0000","last_undegraded":"2024-03-06T07:28:36.817215+0000","last_fullsized":"2024-03-06T07:28:36.817215+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803508+0000","last_change":"2024-03-06T07:28:36.803508+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803508+0000","last_undegraded":"2024-03-06T07:28:36.803508+0000","last_fullsized":"2024-03-06T07:28:36.803508+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.812096+0000","last_change":"2024-03-06T07:28:36.811322+0000","last_active":"2024-03-06T07:28:36.812096+0000","last_peered":"2024-03-06T07:28:36.812096+0000","last_clean":"2024-03-06T07:28:36.812096+0000","last_became_active":"2024-03-06T07:28:36.810594+0000","last_became_peered":"2024-03-06T07:28:36.810594+0000","last_unstale":"2024-03-06T07:28:36.812096+0000","last_undegraded":"2024-03-06T07:28:36.812096+0000","last_fullsized":"2024-03-06T07:28:36.812096+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805018+0000","last_change":"2024-03-06T07:28:36.805018+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805018+0000","last_undegraded":"2024-03-06T07:28:36.805018+0000","last_fullsized":"2024-03-06T07:28:36.805018+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809986+0000","last_change":"2024-03-06T07:28:36.808354+0000","last_active":"2024-03-06T07:28:36.809986+0000","last_peered":"2024-03-06T07:28:36.809986+0000","last_clean":"2024-03-06T07:28:36.809986+0000","last_became_active":"2024-03-06T07:28:36.808076+0000","last_became_peered":"2024-03-06T07:28:36.808076+0000","last_unstale":"2024-03-06T07:28:36.809986+0000","last_undegraded":"2024-03-06T07:28:36.809986+0000","last_fullsized":"2024-03-06T07:28:36.809986+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.762723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803359+0000","last_change":"2024-03-06T07:28:36.803359+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803359+0000","last_undegraded":"2024-03-06T07:28:36.803359+0000","last_fullsized":"2024-03-06T07:28:36.803359+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813932+0000","last_change":"2024-03-06T07:28:36.810704+0000","last_active":"2024-03-06T07:28:36.813932+0000","last_peered":"2024-03-06T07:28:36.813932+0000","last_clean":"2024-03-06T07:28:36.813932+0000","last_became_active":"2024-03-06T07:28:36.809890+0000","last_became_peered":"2024-03-06T07:28:36.809890+0000","last_unstale":"2024-03-06T07:28:36.813932+0000","last_undegraded":"2024-03-06T07:28:36.813932+0000","last_fullsized":"2024-03-06T07:28:36.813932+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:03.819520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804096+0000","last_change":"2024-03-06T07:28:36.804096+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804096+0000","last_undegraded":"2024-03-06T07:28:36.804096+0000","last_fullsized":"2024-03-06T07:28:36.804096+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817100+0000","last_change":"2024-03-06T07:28:36.810746+0000","last_active":"2024-03-06T07:28:36.817100+0000","last_peered":"2024-03-06T07:28:36.817100+0000","last_clean":"2024-03-06T07:28:36.817100+0000","last_became_active":"2024-03-06T07:28:36.810475+0000","last_became_peered":"2024-03-06T07:28:36.810475+0000","last_unstale":"2024-03-06T07:28:36.817100+0000","last_undegraded":"2024-03-06T07:28:36.817100+0000","last_fullsized":"2024-03-06T07:28:36.817100+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:41.135891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801824+0000","last_change":"2024-03-06T07:28:36.801824+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801824+0000","last_undegraded":"2024-03-06T07:28:36.801824+0000","last_fullsized":"2024-03-06T07:28:36.801824+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.810890+0000","last_change":"2024-03-06T07:28:36.809374+0000","last_active":"2024-03-06T07:28:36.810890+0000","last_peered":"2024-03-06T07:28:36.810890+0000","last_clean":"2024-03-06T07:28:36.810890+0000","last_became_active":"2024-03-06T07:28:36.807635+0000","last_became_peered":"2024-03-06T07:28:36.807635+0000","last_unstale":"2024-03-06T07:28:36.810890+0000","last_undegraded":"2024-03-06T07:28:36.810890+0000","last_fullsized":"2024-03-06T07:28:36.810890+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:40.196486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801766+0000","last_change":"2024-03-06T07:28:36.801766+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801766+0000","last_undegraded":"2024-03-06T07:28:36.801766+0000","last_fullsized":"2024-03-06T07:28:36.801766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815277+0000","last_change":"2024-03-06T07:28:36.814194+0000","last_active":"2024-03-06T07:28:36.815277+0000","last_peered":"2024-03-06T07:28:36.815277+0000","last_clean":"2024-03-06T07:28:36.815277+0000","last_became_active":"2024-03-06T07:28:36.813872+0000","last_became_peered":"2024-03-06T07:28:36.813872+0000","last_unstale":"2024-03-06T07:28:36.815277+0000","last_undegraded":"2024-03-06T07:28:36.815277+0000","last_fullsized":"2024-03-06T07:28:36.815277+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799285+0000","last_change":"2024-03-06T07:28:36.799285+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799285+0000","last_undegraded":"2024-03-06T07:28:36.799285+0000","last_fullsized":"2024-03-06T07:28:36.799285+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811218+0000","last_change":"2024-03-06T07:28:36.808348+0000","last_active":"2024-03-06T07:28:36.811218+0000","last_peered":"2024-03-06T07:28:36.811218+0000","last_clean":"2024-03-06T07:28:36.811218+0000","last_became_active":"2024-03-06T07:28:36.808035+0000","last_became_peered":"2024-03-06T07:28:36.808035+0000","last_unstale":"2024-03-06T07:28:36.811218+0000","last_undegraded":"2024-03-06T07:28:36.811218+0000","last_fullsized":"2024-03-06T07:28:36.811218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:02.210672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797961+0000","last_change":"2024-03-06T07:28:36.797961+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797961+0000","last_undegraded":"2024-03-06T07:28:36.797961+0000","last_fullsized":"2024-03-06T07:28:36.797961+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.818360+0000","last_change":"2024-03-06T07:28:36.817966+0000","last_active":"2024-03-06T07:28:36.818360+0000","last_peered":"2024-03-06T07:28:36.818360+0000","last_clean":"2024-03-06T07:28:36.818360+0000","last_became_active":"2024-03-06T07:28:36.817630+0000","last_became_peered":"2024-03-06T07:28:36.817630+0000","last_unstale":"2024-03-06T07:28:36.818360+0000","last_undegraded":"2024-03-06T07:28:36.818360+0000","last_fullsized":"2024-03-06T07:28:36.818360+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803589+0000","last_change":"2024-03-06T07:28:36.803589+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803589+0000","last_undegraded":"2024-03-06T07:28:36.803589+0000","last_fullsized":"2024-03-06T07:28:36.803589+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811657+0000","last_change":"2024-03-06T07:28:36.809671+0000","last_active":"2024-03-06T07:28:36.811657+0000","last_peered":"2024-03-06T07:28:36.811657+0000","last_clean":"2024-03-06T07:28:36.811657+0000","last_became_active":"2024-03-06T07:28:36.809405+0000","last_became_peered":"2024-03-06T07:28:36.809405+0000","last_unstale":"2024-03-06T07:28:36.811657+0000","last_undegraded":"2024-03-06T07:28:36.811657+0000","last_fullsized":"2024-03-06T07:28:36.811657+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802926+0000","last_change":"2024-03-06T07:28:36.802926+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802926+0000","last_undegraded":"2024-03-06T07:28:36.802926+0000","last_fullsized":"2024-03-06T07:28:36.802926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809915+0000","last_change":"2024-03-06T07:28:36.807565+0000","last_active":"2024-03-06T07:28:36.809915+0000","last_peered":"2024-03-06T07:28:36.809915+0000","last_clean":"2024-03-06T07:28:36.809915+0000","last_became_active":"2024-03-06T07:28:36.807266+0000","last_became_peered":"2024-03-06T07:28:36.807266+0000","last_unstale":"2024-03-06T07:28:36.809915+0000","last_undegraded":"2024-03-06T07:28:36.809915+0000","last_fullsized":"2024-03-06T07:28:36.809915+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.762723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804864+0000","last_change":"2024-03-06T07:28:36.804864+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804864+0000","last_undegraded":"2024-03-06T07:28:36.804864+0000","last_fullsized":"2024-03-06T07:28:36.804864+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.806027+0000","last_change":"2024-03-06T07:28:36.804975+0000","last_active":"2024-03-06T07:28:36.806027+0000","last_peered":"2024-03-06T07:28:36.806027+0000","last_clean":"2024-03-06T07:28:36.806027+0000","last_became_active":"2024-03-06T07:28:36.804697+0000","last_became_peered":"2024-03-06T07:28:36.804697+0000","last_unstale":"2024-03-06T07:28:36.806027+0000","last_undegraded":"2024-03-06T07:28:36.806027+0000","last_fullsized":"2024-03-06T07:28:36.806027+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803649+0000","last_change":"2024-03-06T07:28:36.803649+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803649+0000","last_undegraded":"2024-03-06T07:28:36.803649+0000","last_fullsized":"2024-03-06T07:28:36.803649+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817037+0000","last_change":"2024-03-06T07:28:36.810758+0000","last_active":"2024-03-06T07:28:36.817037+0000","last_peered":"2024-03-06T07:28:36.817037+0000","last_clean":"2024-03-06T07:28:36.817037+0000","last_became_active":"2024-03-06T07:28:36.810492+0000","last_became_peered":"2024-03-06T07:28:36.810492+0000","last_unstale":"2024-03-06T07:28:36.817037+0000","last_undegraded":"2024-03-06T07:28:36.817037+0000","last_fullsized":"2024-03-06T07:28:36.817037+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:55.327149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804063+0000","last_change":"2024-03-06T07:28:36.804063+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804063+0000","last_undegraded":"2024-03-06T07:28:36.804063+0000","last_fullsized":"2024-03-06T07:28:36.804063+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811507+0000","last_change":"2024-03-06T07:28:36.809882+0000","last_active":"2024-03-06T07:28:36.811507+0000","last_peered":"2024-03-06T07:28:36.811507+0000","last_clean":"2024-03-06T07:28:36.811507+0000","last_became_active":"2024-03-06T07:28:36.809035+0000","last_became_peered":"2024-03-06T07:28:36.809035+0000","last_unstale":"2024-03-06T07:28:36.811507+0000","last_undegraded":"2024-03-06T07:28:36.811507+0000","last_fullsized":"2024-03-06T07:28:36.811507+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:26.232645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806573+0000","last_change":"2024-03-06T07:28:36.806573+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806573+0000","last_undegraded":"2024-03-06T07:28:36.806573+0000","last_fullsized":"2024-03-06T07:28:36.806573+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.816568+0000","last_change":"2024-03-06T07:28:36.812497+0000","last_active":"2024-03-06T07:28:36.816568+0000","last_peered":"2024-03-06T07:28:36.816568+0000","last_clean":"2024-03-06T07:28:36.816568+0000","last_became_active":"2024-03-06T07:28:36.812253+0000","last_became_peered":"2024-03-06T07:28:36.812253+0000","last_unstale":"2024-03-06T07:28:36.816568+0000","last_undegraded":"2024-03-06T07:28:36.816568+0000","last_fullsized":"2024-03-06T07:28:36.816568+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.721283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802330+0000","last_change":"2024-03-06T07:28:36.802330+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802330+0000","last_undegraded":"2024-03-06T07:28:36.802330+0000","last_fullsized":"2024-03-06T07:28:36.802330+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815234+0000","last_change":"2024-03-06T07:28:36.812931+0000","last_active":"2024-03-06T07:28:36.815234+0000","last_peered":"2024-03-06T07:28:36.815234+0000","last_clean":"2024-03-06T07:28:36.815234+0000","last_became_active":"2024-03-06T07:28:36.812619+0000","last_became_peered":"2024-03-06T07:28:36.812619+0000","last_unstale":"2024-03-06T07:28:36.815234+0000","last_undegraded":"2024-03-06T07:28:36.815234+0000","last_fullsized":"2024-03-06T07:28:36.815234+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803589+0000","last_change":"2024-03-06T07:28:36.803589+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803589+0000","last_undegraded":"2024-03-06T07:28:36.803589+0000","last_fullsized":"2024-03-06T07:28:36.803589+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817286+0000","last_change":"2024-03-06T07:28:36.811623+0000","last_active":"2024-03-06T07:28:36.817286+0000","last_peered":"2024-03-06T07:28:36.817286+0000","last_clean":"2024-03-06T07:28:36.817286+0000","last_became_active":"2024-03-06T07:28:36.811361+0000","last_became_peered":"2024-03-06T07:28:36.811361+0000","last_unstale":"2024-03-06T07:28:36.817286+0000","last_undegraded":"2024-03-06T07:28:36.817286+0000","last_fullsized":"2024-03-06T07:28:36.817286+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.186633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806172+0000","last_change":"2024-03-06T07:28:36.806172+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806172+0000","last_undegraded":"2024-03-06T07:28:36.806172+0000","last_fullsized":"2024-03-06T07:28:36.806172+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811226+0000","last_change":"2024-03-06T07:28:36.807851+0000","last_active":"2024-03-06T07:28:36.811226+0000","last_peered":"2024-03-06T07:28:36.811226+0000","last_clean":"2024-03-06T07:28:36.811226+0000","last_became_active":"2024-03-06T07:28:36.807590+0000","last_became_peered":"2024-03-06T07:28:36.807590+0000","last_unstale":"2024-03-06T07:28:36.811226+0000","last_undegraded":"2024-03-06T07:28:36.811226+0000","last_fullsized":"2024-03-06T07:28:36.811226+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:19.630579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802948+0000","last_change":"2024-03-06T07:28:36.802948+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802948+0000","last_undegraded":"2024-03-06T07:28:36.802948+0000","last_fullsized":"2024-03-06T07:28:36.802948+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.814263+0000","last_change":"2024-03-06T07:28:36.808681+0000","last_active":"2024-03-06T07:28:36.814263+0000","last_peered":"2024-03-06T07:28:36.814263+0000","last_clean":"2024-03-06T07:28:36.814263+0000","last_became_active":"2024-03-06T07:28:36.808407+0000","last_became_peered":"2024-03-06T07:28:36.808407+0000","last_unstale":"2024-03-06T07:28:36.814263+0000","last_undegraded":"2024-03-06T07:28:36.814263+0000","last_fullsized":"2024-03-06T07:28:36.814263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:41.544227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798604+0000","last_change":"2024-03-06T07:28:36.798604+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798604+0000","last_undegraded":"2024-03-06T07:28:36.798604+0000","last_fullsized":"2024-03-06T07:28:36.798604+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815229+0000","last_change":"2024-03-06T07:28:36.812531+0000","last_active":"2024-03-06T07:28:36.815229+0000","last_peered":"2024-03-06T07:28:36.815229+0000","last_clean":"2024-03-06T07:28:36.815229+0000","last_became_active":"2024-03-06T07:28:36.812195+0000","last_became_peered":"2024-03-06T07:28:36.812195+0000","last_unstale":"2024-03-06T07:28:36.815229+0000","last_undegraded":"2024-03-06T07:28:36.815229+0000","last_fullsized":"2024-03-06T07:28:36.815229+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797980+0000","last_change":"2024-03-06T07:28:36.797980+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797980+0000","last_undegraded":"2024-03-06T07:28:36.797980+0000","last_fullsized":"2024-03-06T07:28:36.797980+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817218+0000","last_change":"2024-03-06T07:28:36.811478+0000","last_active":"2024-03-06T07:28:36.817218+0000","last_peered":"2024-03-06T07:28:36.817218+0000","last_clean":"2024-03-06T07:28:36.817218+0000","last_became_active":"2024-03-06T07:28:36.811250+0000","last_became_peered":"2024-03-06T07:28:36.811250+0000","last_unstale":"2024-03-06T07:28:36.817218+0000","last_undegraded":"2024-03-06T07:28:36.817218+0000","last_fullsized":"2024-03-06T07:28:36.817218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803573+0000","last_change":"2024-03-06T07:28:36.803573+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803573+0000","last_undegraded":"2024-03-06T07:28:36.803573+0000","last_fullsized":"2024-03-06T07:28:36.803573+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811488+0000","last_change":"2024-03-06T07:28:36.809706+0000","last_active":"2024-03-06T07:28:36.811488+0000","last_peered":"2024-03-06T07:28:36.811488+0000","last_clean":"2024-03-06T07:28:36.811488+0000","last_became_active":"2024-03-06T07:28:36.809345+0000","last_became_peered":"2024-03-06T07:28:36.809345+0000","last_unstale":"2024-03-06T07:28:36.811488+0000","last_undegraded":"2024-03-06T07:28:36.811488+0000","last_fullsized":"2024-03-06T07:28:36.811488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:25.869243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804620+0000","last_change":"2024-03-06T07:28:36.804620+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804620+0000","last_undegraded":"2024-03-06T07:28:36.804620+0000","last_fullsized":"2024-03-06T07:28:36.804620+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815176+0000","last_change":"2024-03-06T07:28:36.812624+0000","last_active":"2024-03-06T07:28:36.815176+0000","last_peered":"2024-03-06T07:28:36.815176+0000","last_clean":"2024-03-06T07:28:36.815176+0000","last_became_active":"2024-03-06T07:28:36.812331+0000","last_became_peered":"2024-03-06T07:28:36.812331+0000","last_unstale":"2024-03-06T07:28:36.815176+0000","last_undegraded":"2024-03-06T07:28:36.815176+0000","last_fullsized":"2024-03-06T07:28:36.815176+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":359,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.805855+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:28:36.805855+0000","last_peered":"2024-03-06T07:28:36.805855+0000","last_clean":"2024-03-06T07:28:36.805855+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:28:36.805855+0000","last_undegraded":"2024-03-06T07:28:36.805855+0000","last_fullsized":"2024-03-06T07:28:36.805855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":2154496,"data_stored":1270158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":679790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":2154496,"data_stored":1270158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:41.265 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:41.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.400+0000 7f71bba37640 1 Processor -- start 2024-03-06T07:28:41.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.401+0000 7f71bba37640 1 -- start start 2024-03-06T07:28:41.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.401+0000 7f71bba37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 0x7f71b415b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.401+0000 7f71bba37640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b4071f40 con 0x7f71b41515e0 2024-03-06T07:28:41.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.401+0000 7f71bba37640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40cd2c0 con 0x7f71b40cc340 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.401+0000 7f71bba37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4070a70 con 0x7f71b40cce40 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b97d3640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f71b41515e0 0x7f71b4071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59166/0 (socket says 172.21.15.72:59166) 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b97d3640 1 -- 172.21.15.72:0/87885712 learned_addr learned my addr 172.21.15.72:0/87885712 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b8fd2640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 0x7f71b415b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924365737 0 0) 0x7f71b4071f40 con 0x7f71b41515e0 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f719c003520 con 0x7f71b41515e0 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544609381 0 0) 0x7f719c003520 con 0x7f71b41515e0 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 msgr2=0x7f71b415b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 0x7f71b415b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 >> v1:172.21.15.146:6789/0 conn(0x7f71b40cc340 legacy=0x7f71b40cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4070d50 con 0x7f71b41515e0 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f71a4001670 con 0x7f71b41515e0 2024-03-06T07:28:41.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 >> v1:172.21.15.72:6789/0 conn(0x7f71b41515e0 legacy=0x7f71b4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f719c0039e0 0x7f719c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 0x7f719c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c004d40 0x7f719c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4070a70 con 0x7f719c0039e0 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c00b9f0 con 0x7f719c004d40 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719c00bbf0 con 0x7f71b40cce40 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b97d3640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f719c0039e0 0x7f719c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b8fd2640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 0x7f719c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.402+0000 7f71b9fd4640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c004d40 0x7f719c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.403+0000 7f71b97d3640 1 -- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 msgr2=0x7f719c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.403+0000 7f71b97d3640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b40cce40 0x7f719c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.403+0000 7f71b97d3640 1 -- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c004d40 msgr2=0x7f719c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.403+0000 7f71b97d3640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c004d40 0x7f719c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.403+0000 7f71b97d3640 1 -- 172.21.15.72:0/87885712 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c003520 con 0x7f719c0039e0 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.403+0000 7f71b97d3640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f719c0039e0 0x7f719c003dc0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c240 tx=0x7f719c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1bcfe2676b2a2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4007d10 con 0x7f719c0039e0 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4030070 con 0x7f719c0039e0 2024-03-06T07:28:41.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71b3fff640 1 -- 172.21.15.72:0/87885712 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4005d30 con 0x7f719c0039e0 2024-03-06T07:28:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71bba37640 1 -- 172.21.15.72:0/87885712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f719c0039e0 msgr2=0x7f719c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71bba37640 1 --2- 172.21.15.72:0/87885712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f719c0039e0 0x7f719c003dc0 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c240 tx=0x7f719c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71b9fd4640 1 -- 172.21.15.72:0/87885712 reap_dead start 2024-03-06T07:28:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71bba37640 1 -- 172.21.15.72:0/87885712 shutdown_connections 2024-03-06T07:28:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71bba37640 1 -- 172.21.15.72:0/87885712 >> 172.21.15.72:0/87885712 conn(0x7f71b40cf8a0 msgr2=0x7f71b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71bba37640 1 -- 172.21.15.72:0/87885712 shutdown_connections 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.404+0000 7f71bba37640 1 -- 172.21.15.72:0/87885712 wait complete. 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.405+0000 7f71bba37640 1 Processor -- start 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.405+0000 7f71bba37640 1 -- start start 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71bba37640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c0039e0 0x7f71b414c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71bba37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 0x7f71b415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71bba37640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b4153a30 0x7f71b4153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71bba37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40cc770 con 0x7f71b415ec00 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71bba37640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4071310 con 0x7f719c0039e0 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71bba37640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40cc8f0 con 0x7f71b4153a30 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 0x7f71b415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 0x7f71b415c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59328/0 (socket says 172.21.15.72:59328) 2024-03-06T07:28:41.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 -- 172.21.15.72:0/4148191090 learned_addr learned my addr 172.21.15.72:0/4148191090 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b97d3640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c0039e0 0x7f71b414c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b9fd4640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b4153a30 0x7f71b4153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 -- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b4153a30 msgr2=0x7f71b4153ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b4153a30 0x7f71b4153ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 -- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c0039e0 msgr2=0x7f71b414c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.406+0000 7f71b8fd2640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c0039e0 0x7f71b414c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71b8fd2640 1 -- 172.21.15.72:0/4148191090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b415cf80 con 0x7f71b415ec00 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71b97d3640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c0039e0 0x7f71b414c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71b8fd2640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 0x7f71b415c680 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f71a80113f0 tx=0x7f71a80118b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8020070 con 0x7f71b415ec00 2024-03-06T07:28:41.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a8007540 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b415d260 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8007930 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.407+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b415d820 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.408+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4068840 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.409+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71a8023070 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.409+0000 7f71b1ffb640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71a006b4d0 0x7f71a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.409+0000 7f71b97d3640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71a006b4d0 0x7f71a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.409+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71a808c620 con 0x7f71b415ec00 2024-03-06T07:28:41.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.410+0000 7f71b97d3640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71a006b4d0 0x7f71a006d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f71a4034000 tx=0x7f71a4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.413+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71a805aee0 con 0x7f71b415ec00 2024-03-06T07:28:41.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.587+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71b404f330 con 0x7f71a006b4d0 2024-03-06T07:28:41.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.592+0000 7f71b1ffb640 1 -- 172.21.15.72:0/4148191090 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+247075 (secure 0 0 0) 0x7f71b404f330 con 0x7f71a006b4d0 2024-03-06T07:28:41.597 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:41.598 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71a006b4d0 msgr2=0x7f71a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71a006b4d0 0x7f71a006d990 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f71a4034000 tx=0x7f71a4007000 comp rx=0 tx=0).stop 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 msgr2=0x7f71b415c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 0x7f71b415c680 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f71a80113f0 tx=0x7f71a80118b0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 shutdown_connections 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71a006b4d0 0x7f71a006d990 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71b4153a30 0x7f71b4153ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f71b415ec00 0x7f71b415c680 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.595+0000 7f71bba37640 1 --2- 172.21.15.72:0/4148191090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f719c0039e0 0x7f71b414c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.596+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 >> 172.21.15.72:0/4148191090 conn(0x7f71b40cf8a0 msgr2=0x7f71b41502f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.596+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 shutdown_connections 2024-03-06T07:28:41.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.596+0000 7f71bba37640 1 -- 172.21.15.72:0/4148191090 wait complete. 2024-03-06T07:28:41.609 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-03-06T07:28:40.023279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329344,"kb_used_data":13292,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645376,"statfs":{"total":773094113280,"available":772756865024,"internally_reserved":0,"allocated":13611008,"data_stored":6587681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.234114"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806567+0000","last_change":"2024-03-06T07:28:36.806567+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806567+0000","last_undegraded":"2024-03-06T07:28:36.806567+0000","last_fullsized":"2024-03-06T07:28:36.806567+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806490+0000","last_change":"2024-03-06T07:28:36.806490+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806490+0000","last_undegraded":"2024-03-06T07:28:36.806490+0000","last_fullsized":"2024-03-06T07:28:36.806490+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.800755+0000","last_change":"2024-03-06T07:28:36.800755+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.800755+0000","last_undegraded":"2024-03-06T07:28:36.800755+0000","last_fullsized":"2024-03-06T07:28:36.800755+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806501+0000","last_change":"2024-03-06T07:28:36.806501+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806501+0000","last_undegraded":"2024-03-06T07:28:36.806501+0000","last_fullsized":"2024-03-06T07:28:36.806501+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804424+0000","last_change":"2024-03-06T07:28:36.804424+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804424+0000","last_undegraded":"2024-03-06T07:28:36.804424+0000","last_fullsized":"2024-03-06T07:28:36.804424+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.807930+0000","last_change":"2024-03-06T07:28:36.807930+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.807930+0000","last_undegraded":"2024-03-06T07:28:36.807930+0000","last_fullsized":"2024-03-06T07:28:36.807930+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803468+0000","last_change":"2024-03-06T07:28:36.803468+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803468+0000","last_undegraded":"2024-03-06T07:28:36.803468+0000","last_fullsized":"2024-03-06T07:28:36.803468+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801587+0000","last_change":"2024-03-06T07:28:36.801587+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801587+0000","last_undegraded":"2024-03-06T07:28:36.801587+0000","last_fullsized":"2024-03-06T07:28:36.801587+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802274+0000","last_change":"2024-03-06T07:28:36.802274+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802274+0000","last_undegraded":"2024-03-06T07:28:36.802274+0000","last_fullsized":"2024-03-06T07:28:36.802274+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805084+0000","last_change":"2024-03-06T07:28:36.805084+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805084+0000","last_undegraded":"2024-03-06T07:28:36.805084+0000","last_fullsized":"2024-03-06T07:28:36.805084+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805445+0000","last_change":"2024-03-06T07:28:36.805445+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805445+0000","last_undegraded":"2024-03-06T07:28:36.805445+0000","last_fullsized":"2024-03-06T07:28:36.805445+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801140+0000","last_change":"2024-03-06T07:28:36.801140+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801140+0000","last_undegraded":"2024-03-06T07:28:36.801140+0000","last_fullsized":"2024-03-06T07:28:36.801140+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806487+0000","last_change":"2024-03-06T07:28:36.806487+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806487+0000","last_undegraded":"2024-03-06T07:28:36.806487+0000","last_fullsized":"2024-03-06T07:28:36.806487+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804641+0000","last_change":"2024-03-06T07:28:36.804641+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804641+0000","last_undegraded":"2024-03-06T07:28:36.804641+0000","last_fullsized":"2024-03-06T07:28:36.804641+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805149+0000","last_change":"2024-03-06T07:28:36.805149+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805149+0000","last_undegraded":"2024-03-06T07:28:36.805149+0000","last_fullsized":"2024-03-06T07:28:36.805149+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798464+0000","last_change":"2024-03-06T07:28:36.798464+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798464+0000","last_undegraded":"2024-03-06T07:28:36.798464+0000","last_fullsized":"2024-03-06T07:28:36.798464+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803081+0000","last_change":"2024-03-06T07:28:36.803081+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803081+0000","last_undegraded":"2024-03-06T07:28:36.803081+0000","last_fullsized":"2024-03-06T07:28:36.803081+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799143+0000","last_change":"2024-03-06T07:28:36.799143+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799143+0000","last_undegraded":"2024-03-06T07:28:36.799143+0000","last_fullsized":"2024-03-06T07:28:36.799143+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804151+0000","last_change":"2024-03-06T07:28:36.804151+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804151+0000","last_undegraded":"2024-03-06T07:28:36.804151+0000","last_fullsized":"2024-03-06T07:28:36.804151+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798472+0000","last_change":"2024-03-06T07:28:36.798472+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798472+0000","last_undegraded":"2024-03-06T07:28:36.798472+0000","last_fullsized":"2024-03-06T07:28:36.798472+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805603+0000","last_change":"2024-03-06T07:28:36.805603+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805603+0000","last_undegraded":"2024-03-06T07:28:36.805603+0000","last_fullsized":"2024-03-06T07:28:36.805603+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803692+0000","last_change":"2024-03-06T07:28:36.803692+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803692+0000","last_undegraded":"2024-03-06T07:28:36.803692+0000","last_fullsized":"2024-03-06T07:28:36.803692+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801386+0000","last_change":"2024-03-06T07:28:36.801386+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801386+0000","last_undegraded":"2024-03-06T07:28:36.801386+0000","last_fullsized":"2024-03-06T07:28:36.801386+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802815+0000","last_change":"2024-03-06T07:28:36.802815+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802815+0000","last_undegraded":"2024-03-06T07:28:36.802815+0000","last_fullsized":"2024-03-06T07:28:36.802815+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804619+0000","last_change":"2024-03-06T07:28:36.804619+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804619+0000","last_undegraded":"2024-03-06T07:28:36.804619+0000","last_fullsized":"2024-03-06T07:28:36.804619+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798509+0000","last_change":"2024-03-06T07:28:36.798509+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798509+0000","last_undegraded":"2024-03-06T07:28:36.798509+0000","last_fullsized":"2024-03-06T07:28:36.798509+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804726+0000","last_change":"2024-03-06T07:28:36.804726+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804726+0000","last_undegraded":"2024-03-06T07:28:36.804726+0000","last_fullsized":"2024-03-06T07:28:36.804726+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798535+0000","last_change":"2024-03-06T07:28:36.798535+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798535+0000","last_undegraded":"2024-03-06T07:28:36.798535+0000","last_fullsized":"2024-03-06T07:28:36.798535+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804323+0000","last_change":"2024-03-06T07:28:36.804323+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804323+0000","last_undegraded":"2024-03-06T07:28:36.804323+0000","last_fullsized":"2024-03-06T07:28:36.804323+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803816+0000","last_change":"2024-03-06T07:28:36.803816+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803816+0000","last_undegraded":"2024-03-06T07:28:36.803816+0000","last_fullsized":"2024-03-06T07:28:36.803816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803599+0000","last_change":"2024-03-06T07:28:36.803599+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803599+0000","last_undegraded":"2024-03-06T07:28:36.803599+0000","last_fullsized":"2024-03-06T07:28:36.803599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806309+0000","last_change":"2024-03-06T07:28:36.806309+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806309+0000","last_undegraded":"2024-03-06T07:28:36.806309+0000","last_fullsized":"2024-03-06T07:28:36.806309+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797983+0000","last_change":"2024-03-06T07:28:36.797983+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797983+0000","last_undegraded":"2024-03-06T07:28:36.797983+0000","last_fullsized":"2024-03-06T07:28:36.797983+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815202+0000","last_change":"2024-03-06T07:28:36.812840+0000","last_active":"2024-03-06T07:28:36.815202+0000","last_peered":"2024-03-06T07:28:36.815202+0000","last_clean":"2024-03-06T07:28:36.815202+0000","last_became_active":"2024-03-06T07:28:36.812374+0000","last_became_peered":"2024-03-06T07:28:36.812374+0000","last_unstale":"2024-03-06T07:28:36.815202+0000","last_undegraded":"2024-03-06T07:28:36.815202+0000","last_fullsized":"2024-03-06T07:28:36.815202+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801659+0000","last_change":"2024-03-06T07:28:36.801659+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801659+0000","last_undegraded":"2024-03-06T07:28:36.801659+0000","last_fullsized":"2024-03-06T07:28:36.801659+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.810757+0000","last_change":"2024-03-06T07:28:36.808862+0000","last_active":"2024-03-06T07:28:36.810757+0000","last_peered":"2024-03-06T07:28:36.810757+0000","last_clean":"2024-03-06T07:28:36.810757+0000","last_became_active":"2024-03-06T07:28:36.808596+0000","last_became_peered":"2024-03-06T07:28:36.808596+0000","last_unstale":"2024-03-06T07:28:36.810757+0000","last_undegraded":"2024-03-06T07:28:36.810757+0000","last_fullsized":"2024-03-06T07:28:36.810757+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:40.196486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803967+0000","last_change":"2024-03-06T07:28:36.803967+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803967+0000","last_undegraded":"2024-03-06T07:28:36.803967+0000","last_fullsized":"2024-03-06T07:28:36.803967+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811791+0000","last_change":"2024-03-06T07:28:36.806718+0000","last_active":"2024-03-06T07:28:36.811791+0000","last_peered":"2024-03-06T07:28:36.811791+0000","last_clean":"2024-03-06T07:28:36.811791+0000","last_became_active":"2024-03-06T07:28:36.806447+0000","last_became_peered":"2024-03-06T07:28:36.806447+0000","last_unstale":"2024-03-06T07:28:36.811791+0000","last_undegraded":"2024-03-06T07:28:36.811791+0000","last_fullsized":"2024-03-06T07:28:36.811791+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:03.561828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806292+0000","last_change":"2024-03-06T07:28:36.806292+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806292+0000","last_undegraded":"2024-03-06T07:28:36.806292+0000","last_fullsized":"2024-03-06T07:28:36.806292+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817337+0000","last_change":"2024-03-06T07:28:36.812550+0000","last_active":"2024-03-06T07:28:36.817337+0000","last_peered":"2024-03-06T07:28:36.817337+0000","last_clean":"2024-03-06T07:28:36.817337+0000","last_became_active":"2024-03-06T07:28:36.812312+0000","last_became_peered":"2024-03-06T07:28:36.812312+0000","last_unstale":"2024-03-06T07:28:36.817337+0000","last_undegraded":"2024-03-06T07:28:36.817337+0000","last_fullsized":"2024-03-06T07:28:36.817337+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:52.682797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798999+0000","last_change":"2024-03-06T07:28:36.798999+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798999+0000","last_undegraded":"2024-03-06T07:28:36.798999+0000","last_fullsized":"2024-03-06T07:28:36.798999+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817263+0000","last_change":"2024-03-06T07:28:36.812168+0000","last_active":"2024-03-06T07:28:36.817263+0000","last_peered":"2024-03-06T07:28:36.817263+0000","last_clean":"2024-03-06T07:28:36.817263+0000","last_became_active":"2024-03-06T07:28:36.811914+0000","last_became_peered":"2024-03-06T07:28:36.811914+0000","last_unstale":"2024-03-06T07:28:36.817263+0000","last_undegraded":"2024-03-06T07:28:36.817263+0000","last_fullsized":"2024-03-06T07:28:36.817263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:46:25.915577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805322+0000","last_change":"2024-03-06T07:28:36.805322+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805322+0000","last_undegraded":"2024-03-06T07:28:36.805322+0000","last_fullsized":"2024-03-06T07:28:36.805322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813993+0000","last_change":"2024-03-06T07:28:36.808734+0000","last_active":"2024-03-06T07:28:36.813993+0000","last_peered":"2024-03-06T07:28:36.813993+0000","last_clean":"2024-03-06T07:28:36.813993+0000","last_became_active":"2024-03-06T07:28:36.808474+0000","last_became_peered":"2024-03-06T07:28:36.808474+0000","last_unstale":"2024-03-06T07:28:36.813993+0000","last_undegraded":"2024-03-06T07:28:36.813993+0000","last_fullsized":"2024-03-06T07:28:36.813993+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801103+0000","last_change":"2024-03-06T07:28:36.801103+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801103+0000","last_undegraded":"2024-03-06T07:28:36.801103+0000","last_fullsized":"2024-03-06T07:28:36.801103+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.812432+0000","last_change":"2024-03-06T07:28:36.811802+0000","last_active":"2024-03-06T07:28:36.812432+0000","last_peered":"2024-03-06T07:28:36.812432+0000","last_clean":"2024-03-06T07:28:36.812432+0000","last_became_active":"2024-03-06T07:28:36.811529+0000","last_became_peered":"2024-03-06T07:28:36.811529+0000","last_unstale":"2024-03-06T07:28:36.812432+0000","last_undegraded":"2024-03-06T07:28:36.812432+0000","last_fullsized":"2024-03-06T07:28:36.812432+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804090+0000","last_change":"2024-03-06T07:28:36.804090+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804090+0000","last_undegraded":"2024-03-06T07:28:36.804090+0000","last_fullsized":"2024-03-06T07:28:36.804090+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817205+0000","last_change":"2024-03-06T07:28:36.811162+0000","last_active":"2024-03-06T07:28:36.817205+0000","last_peered":"2024-03-06T07:28:36.817205+0000","last_clean":"2024-03-06T07:28:36.817205+0000","last_became_active":"2024-03-06T07:28:36.810868+0000","last_became_peered":"2024-03-06T07:28:36.810868+0000","last_unstale":"2024-03-06T07:28:36.817205+0000","last_undegraded":"2024-03-06T07:28:36.817205+0000","last_fullsized":"2024-03-06T07:28:36.817205+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:08:24.650548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804395+0000","last_change":"2024-03-06T07:28:36.804395+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804395+0000","last_undegraded":"2024-03-06T07:28:36.804395+0000","last_fullsized":"2024-03-06T07:28:36.804395+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809975+0000","last_change":"2024-03-06T07:28:36.807342+0000","last_active":"2024-03-06T07:28:36.809975+0000","last_peered":"2024-03-06T07:28:36.809975+0000","last_clean":"2024-03-06T07:28:36.809975+0000","last_became_active":"2024-03-06T07:28:36.807010+0000","last_became_peered":"2024-03-06T07:28:36.807010+0000","last_unstale":"2024-03-06T07:28:36.809975+0000","last_undegraded":"2024-03-06T07:28:36.809975+0000","last_fullsized":"2024-03-06T07:28:36.809975+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:44:36.673286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799792+0000","last_change":"2024-03-06T07:28:36.799792+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799792+0000","last_undegraded":"2024-03-06T07:28:36.799792+0000","last_fullsized":"2024-03-06T07:28:36.799792+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817215+0000","last_change":"2024-03-06T07:28:36.812151+0000","last_active":"2024-03-06T07:28:36.817215+0000","last_peered":"2024-03-06T07:28:36.817215+0000","last_clean":"2024-03-06T07:28:36.817215+0000","last_became_active":"2024-03-06T07:28:36.811892+0000","last_became_peered":"2024-03-06T07:28:36.811892+0000","last_unstale":"2024-03-06T07:28:36.817215+0000","last_undegraded":"2024-03-06T07:28:36.817215+0000","last_fullsized":"2024-03-06T07:28:36.817215+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803508+0000","last_change":"2024-03-06T07:28:36.803508+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803508+0000","last_undegraded":"2024-03-06T07:28:36.803508+0000","last_fullsized":"2024-03-06T07:28:36.803508+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.812096+0000","last_change":"2024-03-06T07:28:36.811322+0000","last_active":"2024-03-06T07:28:36.812096+0000","last_peered":"2024-03-06T07:28:36.812096+0000","last_clean":"2024-03-06T07:28:36.812096+0000","last_became_active":"2024-03-06T07:28:36.810594+0000","last_became_peered":"2024-03-06T07:28:36.810594+0000","last_unstale":"2024-03-06T07:28:36.812096+0000","last_undegraded":"2024-03-06T07:28:36.812096+0000","last_fullsized":"2024-03-06T07:28:36.812096+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805018+0000","last_change":"2024-03-06T07:28:36.805018+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805018+0000","last_undegraded":"2024-03-06T07:28:36.805018+0000","last_fullsized":"2024-03-06T07:28:36.805018+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809986+0000","last_change":"2024-03-06T07:28:36.808354+0000","last_active":"2024-03-06T07:28:36.809986+0000","last_peered":"2024-03-06T07:28:36.809986+0000","last_clean":"2024-03-06T07:28:36.809986+0000","last_became_active":"2024-03-06T07:28:36.808076+0000","last_became_peered":"2024-03-06T07:28:36.808076+0000","last_unstale":"2024-03-06T07:28:36.809986+0000","last_undegraded":"2024-03-06T07:28:36.809986+0000","last_fullsized":"2024-03-06T07:28:36.809986+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.762723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803359+0000","last_change":"2024-03-06T07:28:36.803359+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803359+0000","last_undegraded":"2024-03-06T07:28:36.803359+0000","last_fullsized":"2024-03-06T07:28:36.803359+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813932+0000","last_change":"2024-03-06T07:28:36.810704+0000","last_active":"2024-03-06T07:28:36.813932+0000","last_peered":"2024-03-06T07:28:36.813932+0000","last_clean":"2024-03-06T07:28:36.813932+0000","last_became_active":"2024-03-06T07:28:36.809890+0000","last_became_peered":"2024-03-06T07:28:36.809890+0000","last_unstale":"2024-03-06T07:28:36.813932+0000","last_undegraded":"2024-03-06T07:28:36.813932+0000","last_fullsized":"2024-03-06T07:28:36.813932+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:03.819520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804096+0000","last_change":"2024-03-06T07:28:36.804096+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804096+0000","last_undegraded":"2024-03-06T07:28:36.804096+0000","last_fullsized":"2024-03-06T07:28:36.804096+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817100+0000","last_change":"2024-03-06T07:28:36.810746+0000","last_active":"2024-03-06T07:28:36.817100+0000","last_peered":"2024-03-06T07:28:36.817100+0000","last_clean":"2024-03-06T07:28:36.817100+0000","last_became_active":"2024-03-06T07:28:36.810475+0000","last_became_peered":"2024-03-06T07:28:36.810475+0000","last_unstale":"2024-03-06T07:28:36.817100+0000","last_undegraded":"2024-03-06T07:28:36.817100+0000","last_fullsized":"2024-03-06T07:28:36.817100+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:06:41.135891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801824+0000","last_change":"2024-03-06T07:28:36.801824+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801824+0000","last_undegraded":"2024-03-06T07:28:36.801824+0000","last_fullsized":"2024-03-06T07:28:36.801824+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.810890+0000","last_change":"2024-03-06T07:28:36.809374+0000","last_active":"2024-03-06T07:28:36.810890+0000","last_peered":"2024-03-06T07:28:36.810890+0000","last_clean":"2024-03-06T07:28:36.810890+0000","last_became_active":"2024-03-06T07:28:36.807635+0000","last_became_peered":"2024-03-06T07:28:36.807635+0000","last_unstale":"2024-03-06T07:28:36.810890+0000","last_undegraded":"2024-03-06T07:28:36.810890+0000","last_fullsized":"2024-03-06T07:28:36.810890+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:04:40.196486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.801766+0000","last_change":"2024-03-06T07:28:36.801766+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.801766+0000","last_undegraded":"2024-03-06T07:28:36.801766+0000","last_fullsized":"2024-03-06T07:28:36.801766+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815277+0000","last_change":"2024-03-06T07:28:36.814194+0000","last_active":"2024-03-06T07:28:36.815277+0000","last_peered":"2024-03-06T07:28:36.815277+0000","last_clean":"2024-03-06T07:28:36.815277+0000","last_became_active":"2024-03-06T07:28:36.813872+0000","last_became_peered":"2024-03-06T07:28:36.813872+0000","last_unstale":"2024-03-06T07:28:36.815277+0000","last_undegraded":"2024-03-06T07:28:36.815277+0000","last_fullsized":"2024-03-06T07:28:36.815277+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.799285+0000","last_change":"2024-03-06T07:28:36.799285+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.799285+0000","last_undegraded":"2024-03-06T07:28:36.799285+0000","last_fullsized":"2024-03-06T07:28:36.799285+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811218+0000","last_change":"2024-03-06T07:28:36.808348+0000","last_active":"2024-03-06T07:28:36.811218+0000","last_peered":"2024-03-06T07:28:36.811218+0000","last_clean":"2024-03-06T07:28:36.811218+0000","last_became_active":"2024-03-06T07:28:36.808035+0000","last_became_peered":"2024-03-06T07:28:36.808035+0000","last_unstale":"2024-03-06T07:28:36.811218+0000","last_undegraded":"2024-03-06T07:28:36.811218+0000","last_fullsized":"2024-03-06T07:28:36.811218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:16:02.210672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797961+0000","last_change":"2024-03-06T07:28:36.797961+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797961+0000","last_undegraded":"2024-03-06T07:28:36.797961+0000","last_fullsized":"2024-03-06T07:28:36.797961+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.818360+0000","last_change":"2024-03-06T07:28:36.817966+0000","last_active":"2024-03-06T07:28:36.818360+0000","last_peered":"2024-03-06T07:28:36.818360+0000","last_clean":"2024-03-06T07:28:36.818360+0000","last_became_active":"2024-03-06T07:28:36.817630+0000","last_became_peered":"2024-03-06T07:28:36.817630+0000","last_unstale":"2024-03-06T07:28:36.818360+0000","last_undegraded":"2024-03-06T07:28:36.818360+0000","last_fullsized":"2024-03-06T07:28:36.818360+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803589+0000","last_change":"2024-03-06T07:28:36.803589+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803589+0000","last_undegraded":"2024-03-06T07:28:36.803589+0000","last_fullsized":"2024-03-06T07:28:36.803589+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811657+0000","last_change":"2024-03-06T07:28:36.809671+0000","last_active":"2024-03-06T07:28:36.811657+0000","last_peered":"2024-03-06T07:28:36.811657+0000","last_clean":"2024-03-06T07:28:36.811657+0000","last_became_active":"2024-03-06T07:28:36.809405+0000","last_became_peered":"2024-03-06T07:28:36.809405+0000","last_unstale":"2024-03-06T07:28:36.811657+0000","last_undegraded":"2024-03-06T07:28:36.811657+0000","last_fullsized":"2024-03-06T07:28:36.811657+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802926+0000","last_change":"2024-03-06T07:28:36.802926+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802926+0000","last_undegraded":"2024-03-06T07:28:36.802926+0000","last_fullsized":"2024-03-06T07:28:36.802926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.809915+0000","last_change":"2024-03-06T07:28:36.807565+0000","last_active":"2024-03-06T07:28:36.809915+0000","last_peered":"2024-03-06T07:28:36.809915+0000","last_clean":"2024-03-06T07:28:36.809915+0000","last_became_active":"2024-03-06T07:28:36.807266+0000","last_became_peered":"2024-03-06T07:28:36.807266+0000","last_unstale":"2024-03-06T07:28:36.809915+0000","last_undegraded":"2024-03-06T07:28:36.809915+0000","last_fullsized":"2024-03-06T07:28:36.809915+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:56:00.762723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804864+0000","last_change":"2024-03-06T07:28:36.804864+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804864+0000","last_undegraded":"2024-03-06T07:28:36.804864+0000","last_fullsized":"2024-03-06T07:28:36.804864+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.806027+0000","last_change":"2024-03-06T07:28:36.804975+0000","last_active":"2024-03-06T07:28:36.806027+0000","last_peered":"2024-03-06T07:28:36.806027+0000","last_clean":"2024-03-06T07:28:36.806027+0000","last_became_active":"2024-03-06T07:28:36.804697+0000","last_became_peered":"2024-03-06T07:28:36.804697+0000","last_unstale":"2024-03-06T07:28:36.806027+0000","last_undegraded":"2024-03-06T07:28:36.806027+0000","last_fullsized":"2024-03-06T07:28:36.806027+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803649+0000","last_change":"2024-03-06T07:28:36.803649+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803649+0000","last_undegraded":"2024-03-06T07:28:36.803649+0000","last_fullsized":"2024-03-06T07:28:36.803649+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817037+0000","last_change":"2024-03-06T07:28:36.810758+0000","last_active":"2024-03-06T07:28:36.817037+0000","last_peered":"2024-03-06T07:28:36.817037+0000","last_clean":"2024-03-06T07:28:36.817037+0000","last_became_active":"2024-03-06T07:28:36.810492+0000","last_became_peered":"2024-03-06T07:28:36.810492+0000","last_unstale":"2024-03-06T07:28:36.817037+0000","last_undegraded":"2024-03-06T07:28:36.817037+0000","last_fullsized":"2024-03-06T07:28:36.817037+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:55.327149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804063+0000","last_change":"2024-03-06T07:28:36.804063+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804063+0000","last_undegraded":"2024-03-06T07:28:36.804063+0000","last_fullsized":"2024-03-06T07:28:36.804063+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811507+0000","last_change":"2024-03-06T07:28:36.809882+0000","last_active":"2024-03-06T07:28:36.811507+0000","last_peered":"2024-03-06T07:28:36.811507+0000","last_clean":"2024-03-06T07:28:36.811507+0000","last_became_active":"2024-03-06T07:28:36.809035+0000","last_became_peered":"2024-03-06T07:28:36.809035+0000","last_unstale":"2024-03-06T07:28:36.811507+0000","last_undegraded":"2024-03-06T07:28:36.811507+0000","last_fullsized":"2024-03-06T07:28:36.811507+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:37:26.232645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806573+0000","last_change":"2024-03-06T07:28:36.806573+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806573+0000","last_undegraded":"2024-03-06T07:28:36.806573+0000","last_fullsized":"2024-03-06T07:28:36.806573+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.816568+0000","last_change":"2024-03-06T07:28:36.812497+0000","last_active":"2024-03-06T07:28:36.816568+0000","last_peered":"2024-03-06T07:28:36.816568+0000","last_clean":"2024-03-06T07:28:36.816568+0000","last_became_active":"2024-03-06T07:28:36.812253+0000","last_became_peered":"2024-03-06T07:28:36.812253+0000","last_unstale":"2024-03-06T07:28:36.816568+0000","last_undegraded":"2024-03-06T07:28:36.816568+0000","last_fullsized":"2024-03-06T07:28:36.816568+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.721283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802330+0000","last_change":"2024-03-06T07:28:36.802330+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802330+0000","last_undegraded":"2024-03-06T07:28:36.802330+0000","last_fullsized":"2024-03-06T07:28:36.802330+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815234+0000","last_change":"2024-03-06T07:28:36.812931+0000","last_active":"2024-03-06T07:28:36.815234+0000","last_peered":"2024-03-06T07:28:36.815234+0000","last_clean":"2024-03-06T07:28:36.815234+0000","last_became_active":"2024-03-06T07:28:36.812619+0000","last_became_peered":"2024-03-06T07:28:36.812619+0000","last_unstale":"2024-03-06T07:28:36.815234+0000","last_undegraded":"2024-03-06T07:28:36.815234+0000","last_fullsized":"2024-03-06T07:28:36.815234+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803589+0000","last_change":"2024-03-06T07:28:36.803589+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803589+0000","last_undegraded":"2024-03-06T07:28:36.803589+0000","last_fullsized":"2024-03-06T07:28:36.803589+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817286+0000","last_change":"2024-03-06T07:28:36.811623+0000","last_active":"2024-03-06T07:28:36.817286+0000","last_peered":"2024-03-06T07:28:36.817286+0000","last_clean":"2024-03-06T07:28:36.817286+0000","last_became_active":"2024-03-06T07:28:36.811361+0000","last_became_peered":"2024-03-06T07:28:36.811361+0000","last_unstale":"2024-03-06T07:28:36.817286+0000","last_undegraded":"2024-03-06T07:28:36.817286+0000","last_fullsized":"2024-03-06T07:28:36.817286+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:06:50.186633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806172+0000","last_change":"2024-03-06T07:28:36.806172+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806172+0000","last_undegraded":"2024-03-06T07:28:36.806172+0000","last_fullsized":"2024-03-06T07:28:36.806172+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811226+0000","last_change":"2024-03-06T07:28:36.807851+0000","last_active":"2024-03-06T07:28:36.811226+0000","last_peered":"2024-03-06T07:28:36.811226+0000","last_clean":"2024-03-06T07:28:36.811226+0000","last_became_active":"2024-03-06T07:28:36.807590+0000","last_became_peered":"2024-03-06T07:28:36.807590+0000","last_unstale":"2024-03-06T07:28:36.811226+0000","last_undegraded":"2024-03-06T07:28:36.811226+0000","last_fullsized":"2024-03-06T07:28:36.811226+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:19.630579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802948+0000","last_change":"2024-03-06T07:28:36.802948+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802948+0000","last_undegraded":"2024-03-06T07:28:36.802948+0000","last_fullsized":"2024-03-06T07:28:36.802948+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.814263+0000","last_change":"2024-03-06T07:28:36.808681+0000","last_active":"2024-03-06T07:28:36.814263+0000","last_peered":"2024-03-06T07:28:36.814263+0000","last_clean":"2024-03-06T07:28:36.814263+0000","last_became_active":"2024-03-06T07:28:36.808407+0000","last_became_peered":"2024-03-06T07:28:36.808407+0000","last_unstale":"2024-03-06T07:28:36.814263+0000","last_undegraded":"2024-03-06T07:28:36.814263+0000","last_fullsized":"2024-03-06T07:28:36.814263+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:41.544227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.798604+0000","last_change":"2024-03-06T07:28:36.798604+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.798604+0000","last_undegraded":"2024-03-06T07:28:36.798604+0000","last_fullsized":"2024-03-06T07:28:36.798604+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815229+0000","last_change":"2024-03-06T07:28:36.812531+0000","last_active":"2024-03-06T07:28:36.815229+0000","last_peered":"2024-03-06T07:28:36.815229+0000","last_clean":"2024-03-06T07:28:36.815229+0000","last_became_active":"2024-03-06T07:28:36.812195+0000","last_became_peered":"2024-03-06T07:28:36.812195+0000","last_unstale":"2024-03-06T07:28:36.815229+0000","last_undegraded":"2024-03-06T07:28:36.815229+0000","last_fullsized":"2024-03-06T07:28:36.815229+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.797980+0000","last_change":"2024-03-06T07:28:36.797980+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.797980+0000","last_undegraded":"2024-03-06T07:28:36.797980+0000","last_fullsized":"2024-03-06T07:28:36.797980+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.817218+0000","last_change":"2024-03-06T07:28:36.811478+0000","last_active":"2024-03-06T07:28:36.817218+0000","last_peered":"2024-03-06T07:28:36.817218+0000","last_clean":"2024-03-06T07:28:36.817218+0000","last_became_active":"2024-03-06T07:28:36.811250+0000","last_became_peered":"2024-03-06T07:28:36.811250+0000","last_unstale":"2024-03-06T07:28:36.817218+0000","last_undegraded":"2024-03-06T07:28:36.817218+0000","last_fullsized":"2024-03-06T07:28:36.817218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803573+0000","last_change":"2024-03-06T07:28:36.803573+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803573+0000","last_undegraded":"2024-03-06T07:28:36.803573+0000","last_fullsized":"2024-03-06T07:28:36.803573+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.811488+0000","last_change":"2024-03-06T07:28:36.809706+0000","last_active":"2024-03-06T07:28:36.811488+0000","last_peered":"2024-03-06T07:28:36.811488+0000","last_clean":"2024-03-06T07:28:36.811488+0000","last_became_active":"2024-03-06T07:28:36.809345+0000","last_became_peered":"2024-03-06T07:28:36.809345+0000","last_unstale":"2024-03-06T07:28:36.811488+0000","last_undegraded":"2024-03-06T07:28:36.811488+0000","last_fullsized":"2024-03-06T07:28:36.811488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:34:25.869243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804620+0000","last_change":"2024-03-06T07:28:36.804620+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804620+0000","last_undegraded":"2024-03-06T07:28:36.804620+0000","last_fullsized":"2024-03-06T07:28:36.804620+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.815176+0000","last_change":"2024-03-06T07:28:36.812624+0000","last_active":"2024-03-06T07:28:36.815176+0000","last_peered":"2024-03-06T07:28:36.815176+0000","last_clean":"2024-03-06T07:28:36.815176+0000","last_became_active":"2024-03-06T07:28:36.812331+0000","last_became_peered":"2024-03-06T07:28:36.812331+0000","last_unstale":"2024-03-06T07:28:36.815176+0000","last_undegraded":"2024-03-06T07:28:36.815176+0000","last_fullsized":"2024-03-06T07:28:36.815176+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":359,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.805855+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:28:36.805855+0000","last_peered":"2024-03-06T07:28:36.805855+0000","last_clean":"2024-03-06T07:28:36.805855+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:28:36.805855+0000","last_undegraded":"2024-03-06T07:28:36.805855+0000","last_fullsized":"2024-03-06T07:28:36.805855+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41604,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330236,"statfs":{"total":96636764160,"available":96594161664,"internally_reserved":0,"allocated":2154496,"data_stored":1270158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1560576,"data_stored":679790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41024,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330816,"statfs":{"total":96636764160,"available":96594755584,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41020,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330820,"statfs":{"total":96636764160,"available":96594759680,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":2154496,"data_stored":1270158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3f is not active+clean 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806567+0000', 'last_change': '2024-03-06T07:28:36.806567+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806567+0000', 'last_undegraded': '2024-03-06T07:28:36.806567+0000', 'last_fullsized': '2024-03-06T07:28:36.806567+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3e is not active+clean 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806490+0000', 'last_change': '2024-03-06T07:28:36.806490+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806490+0000', 'last_undegraded': '2024-03-06T07:28:36.806490+0000', 'last_fullsized': '2024-03-06T07:28:36.806490+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3d is not active+clean 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.800755+0000', 'last_change': '2024-03-06T07:28:36.800755+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.800755+0000', 'last_undegraded': '2024-03-06T07:28:36.800755+0000', 'last_fullsized': '2024-03-06T07:28:36.800755+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3c is not active+clean 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806501+0000', 'last_change': '2024-03-06T07:28:36.806501+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806501+0000', 'last_undegraded': '2024-03-06T07:28:36.806501+0000', 'last_fullsized': '2024-03-06T07:28:36.806501+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.613 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3b is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804424+0000', 'last_change': '2024-03-06T07:28:36.804424+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804424+0000', 'last_undegraded': '2024-03-06T07:28:36.804424+0000', 'last_fullsized': '2024-03-06T07:28:36.804424+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3a is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.807930+0000', 'last_change': '2024-03-06T07:28:36.807930+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.807930+0000', 'last_undegraded': '2024-03-06T07:28:36.807930+0000', 'last_fullsized': '2024-03-06T07:28:36.807930+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.39 is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803468+0000', 'last_change': '2024-03-06T07:28:36.803468+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803468+0000', 'last_undegraded': '2024-03-06T07:28:36.803468+0000', 'last_fullsized': '2024-03-06T07:28:36.803468+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.38 is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801587+0000', 'last_change': '2024-03-06T07:28:36.801587+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801587+0000', 'last_undegraded': '2024-03-06T07:28:36.801587+0000', 'last_fullsized': '2024-03-06T07:28:36.801587+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.37 is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.802274+0000', 'last_change': '2024-03-06T07:28:36.802274+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.802274+0000', 'last_undegraded': '2024-03-06T07:28:36.802274+0000', 'last_fullsized': '2024-03-06T07:28:36.802274+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.36 is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.805084+0000', 'last_change': '2024-03-06T07:28:36.805084+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.805084+0000', 'last_undegraded': '2024-03-06T07:28:36.805084+0000', 'last_fullsized': '2024-03-06T07:28:36.805084+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.35 is not active+clean 2024-03-06T07:28:41.614 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.805445+0000', 'last_change': '2024-03-06T07:28:36.805445+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.805445+0000', 'last_undegraded': '2024-03-06T07:28:36.805445+0000', 'last_fullsized': '2024-03-06T07:28:36.805445+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.34 is not active+clean 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801140+0000', 'last_change': '2024-03-06T07:28:36.801140+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801140+0000', 'last_undegraded': '2024-03-06T07:28:36.801140+0000', 'last_fullsized': '2024-03-06T07:28:36.801140+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.33 is not active+clean 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806487+0000', 'last_change': '2024-03-06T07:28:36.806487+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806487+0000', 'last_undegraded': '2024-03-06T07:28:36.806487+0000', 'last_fullsized': '2024-03-06T07:28:36.806487+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.32 is not active+clean 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804641+0000', 'last_change': '2024-03-06T07:28:36.804641+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804641+0000', 'last_undegraded': '2024-03-06T07:28:36.804641+0000', 'last_fullsized': '2024-03-06T07:28:36.804641+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.31 is not active+clean 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.805149+0000', 'last_change': '2024-03-06T07:28:36.805149+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.805149+0000', 'last_undegraded': '2024-03-06T07:28:36.805149+0000', 'last_fullsized': '2024-03-06T07:28:36.805149+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.30 is not active+clean 2024-03-06T07:28:41.615 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.798464+0000', 'last_change': '2024-03-06T07:28:36.798464+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.798464+0000', 'last_undegraded': '2024-03-06T07:28:36.798464+0000', 'last_fullsized': '2024-03-06T07:28:36.798464+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2f is not active+clean 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803081+0000', 'last_change': '2024-03-06T07:28:36.803081+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803081+0000', 'last_undegraded': '2024-03-06T07:28:36.803081+0000', 'last_fullsized': '2024-03-06T07:28:36.803081+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2e is not active+clean 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.799143+0000', 'last_change': '2024-03-06T07:28:36.799143+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.799143+0000', 'last_undegraded': '2024-03-06T07:28:36.799143+0000', 'last_fullsized': '2024-03-06T07:28:36.799143+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2d is not active+clean 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804151+0000', 'last_change': '2024-03-06T07:28:36.804151+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804151+0000', 'last_undegraded': '2024-03-06T07:28:36.804151+0000', 'last_fullsized': '2024-03-06T07:28:36.804151+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2c is not active+clean 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.798472+0000', 'last_change': '2024-03-06T07:28:36.798472+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.798472+0000', 'last_undegraded': '2024-03-06T07:28:36.798472+0000', 'last_fullsized': '2024-03-06T07:28:36.798472+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.616 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2b is not active+clean 2024-03-06T07:28:41.617 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.805603+0000', 'last_change': '2024-03-06T07:28:36.805603+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.805603+0000', 'last_undegraded': '2024-03-06T07:28:36.805603+0000', 'last_fullsized': '2024-03-06T07:28:36.805603+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.617 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2a is not active+clean 2024-03-06T07:28:41.617 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803692+0000', 'last_change': '2024-03-06T07:28:36.803692+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803692+0000', 'last_undegraded': '2024-03-06T07:28:36.803692+0000', 'last_fullsized': '2024-03-06T07:28:36.803692+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:41.617 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.29 is not active+clean 2024-03-06T07:28:41.617 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801386+0000', 'last_change': '2024-03-06T07:28:36.801386+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801386+0000', 'last_undegraded': '2024-03-06T07:28:36.801386+0000', 'last_fullsized': '2024-03-06T07:28:36.801386+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.618 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.28 is not active+clean 2024-03-06T07:28:41.618 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.802815+0000', 'last_change': '2024-03-06T07:28:36.802815+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.802815+0000', 'last_undegraded': '2024-03-06T07:28:36.802815+0000', 'last_fullsized': '2024-03-06T07:28:36.802815+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-06T07:28:41.618 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.27 is not active+clean 2024-03-06T07:28:41.618 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804619+0000', 'last_change': '2024-03-06T07:28:36.804619+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804619+0000', 'last_undegraded': '2024-03-06T07:28:36.804619+0000', 'last_fullsized': '2024-03-06T07:28:36.804619+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.618 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.26 is not active+clean 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.798509+0000', 'last_change': '2024-03-06T07:28:36.798509+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.798509+0000', 'last_undegraded': '2024-03-06T07:28:36.798509+0000', 'last_fullsized': '2024-03-06T07:28:36.798509+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.25 is not active+clean 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804726+0000', 'last_change': '2024-03-06T07:28:36.804726+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804726+0000', 'last_undegraded': '2024-03-06T07:28:36.804726+0000', 'last_fullsized': '2024-03-06T07:28:36.804726+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.24 is not active+clean 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.798535+0000', 'last_change': '2024-03-06T07:28:36.798535+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.798535+0000', 'last_undegraded': '2024-03-06T07:28:36.798535+0000', 'last_fullsized': '2024-03-06T07:28:36.798535+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.23 is not active+clean 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804323+0000', 'last_change': '2024-03-06T07:28:36.804323+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804323+0000', 'last_undegraded': '2024-03-06T07:28:36.804323+0000', 'last_fullsized': '2024-03-06T07:28:36.804323+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.22 is not active+clean 2024-03-06T07:28:41.619 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803816+0000', 'last_change': '2024-03-06T07:28:36.803816+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803816+0000', 'last_undegraded': '2024-03-06T07:28:36.803816+0000', 'last_fullsized': '2024-03-06T07:28:36.803816+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.21 is not active+clean 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803599+0000', 'last_change': '2024-03-06T07:28:36.803599+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803599+0000', 'last_undegraded': '2024-03-06T07:28:36.803599+0000', 'last_fullsized': '2024-03-06T07:28:36.803599+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.20 is not active+clean 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806309+0000', 'last_change': '2024-03-06T07:28:36.806309+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806309+0000', 'last_undegraded': '2024-03-06T07:28:36.806309+0000', 'last_fullsized': '2024-03-06T07:28:36.806309+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1e is not active+clean 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.797983+0000', 'last_change': '2024-03-06T07:28:36.797983+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.797983+0000', 'last_undegraded': '2024-03-06T07:28:36.797983+0000', 'last_fullsized': '2024-03-06T07:28:36.797983+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1f is not active+clean 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801659+0000', 'last_change': '2024-03-06T07:28:36.801659+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801659+0000', 'last_undegraded': '2024-03-06T07:28:36.801659+0000', 'last_fullsized': '2024-03-06T07:28:36.801659+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1c is not active+clean 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803967+0000', 'last_change': '2024-03-06T07:28:36.803967+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803967+0000', 'last_undegraded': '2024-03-06T07:28:36.803967+0000', 'last_fullsized': '2024-03-06T07:28:36.803967+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.620 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1d is not active+clean 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806292+0000', 'last_change': '2024-03-06T07:28:36.806292+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806292+0000', 'last_undegraded': '2024-03-06T07:28:36.806292+0000', 'last_fullsized': '2024-03-06T07:28:36.806292+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1a is not active+clean 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.798999+0000', 'last_change': '2024-03-06T07:28:36.798999+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.798999+0000', 'last_undegraded': '2024-03-06T07:28:36.798999+0000', 'last_fullsized': '2024-03-06T07:28:36.798999+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1b is not active+clean 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.805322+0000', 'last_change': '2024-03-06T07:28:36.805322+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.805322+0000', 'last_undegraded': '2024-03-06T07:28:36.805322+0000', 'last_fullsized': '2024-03-06T07:28:36.805322+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.18 is not active+clean 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801103+0000', 'last_change': '2024-03-06T07:28:36.801103+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801103+0000', 'last_undegraded': '2024-03-06T07:28:36.801103+0000', 'last_fullsized': '2024-03-06T07:28:36.801103+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.19 is not active+clean 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804090+0000', 'last_change': '2024-03-06T07:28:36.804090+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804090+0000', 'last_undegraded': '2024-03-06T07:28:36.804090+0000', 'last_fullsized': '2024-03-06T07:28:36.804090+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.16 is not active+clean 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804395+0000', 'last_change': '2024-03-06T07:28:36.804395+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804395+0000', 'last_undegraded': '2024-03-06T07:28:36.804395+0000', 'last_fullsized': '2024-03-06T07:28:36.804395+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.621 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.17 is not active+clean 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.799792+0000', 'last_change': '2024-03-06T07:28:36.799792+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.799792+0000', 'last_undegraded': '2024-03-06T07:28:36.799792+0000', 'last_fullsized': '2024-03-06T07:28:36.799792+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.14 is not active+clean 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803508+0000', 'last_change': '2024-03-06T07:28:36.803508+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803508+0000', 'last_undegraded': '2024-03-06T07:28:36.803508+0000', 'last_fullsized': '2024-03-06T07:28:36.803508+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.15 is not active+clean 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.805018+0000', 'last_change': '2024-03-06T07:28:36.805018+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.805018+0000', 'last_undegraded': '2024-03-06T07:28:36.805018+0000', 'last_fullsized': '2024-03-06T07:28:36.805018+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.13 is not active+clean 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803359+0000', 'last_change': '2024-03-06T07:28:36.803359+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803359+0000', 'last_undegraded': '2024-03-06T07:28:36.803359+0000', 'last_fullsized': '2024-03-06T07:28:36.803359+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.10 is not active+clean 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804096+0000', 'last_change': '2024-03-06T07:28:36.804096+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804096+0000', 'last_undegraded': '2024-03-06T07:28:36.804096+0000', 'last_fullsized': '2024-03-06T07:28:36.804096+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.11 is not active+clean 2024-03-06T07:28:41.622 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801824+0000', 'last_change': '2024-03-06T07:28:36.801824+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801824+0000', 'last_undegraded': '2024-03-06T07:28:36.801824+0000', 'last_fullsized': '2024-03-06T07:28:36.801824+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.e is not active+clean 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.801766+0000', 'last_change': '2024-03-06T07:28:36.801766+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.801766+0000', 'last_undegraded': '2024-03-06T07:28:36.801766+0000', 'last_fullsized': '2024-03-06T07:28:36.801766+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.f is not active+clean 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.799285+0000', 'last_change': '2024-03-06T07:28:36.799285+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.799285+0000', 'last_undegraded': '2024-03-06T07:28:36.799285+0000', 'last_fullsized': '2024-03-06T07:28:36.799285+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.c is not active+clean 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.797961+0000', 'last_change': '2024-03-06T07:28:36.797961+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.797961+0000', 'last_undegraded': '2024-03-06T07:28:36.797961+0000', 'last_fullsized': '2024-03-06T07:28:36.797961+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.d is not active+clean 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803589+0000', 'last_change': '2024-03-06T07:28:36.803589+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803589+0000', 'last_undegraded': '2024-03-06T07:28:36.803589+0000', 'last_fullsized': '2024-03-06T07:28:36.803589+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.a is not active+clean 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.802926+0000', 'last_change': '2024-03-06T07:28:36.802926+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.802926+0000', 'last_undegraded': '2024-03-06T07:28:36.802926+0000', 'last_fullsized': '2024-03-06T07:28:36.802926+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.623 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.b is not active+clean 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804864+0000', 'last_change': '2024-03-06T07:28:36.804864+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804864+0000', 'last_undegraded': '2024-03-06T07:28:36.804864+0000', 'last_fullsized': '2024-03-06T07:28:36.804864+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.8 is not active+clean 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803649+0000', 'last_change': '2024-03-06T07:28:36.803649+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803649+0000', 'last_undegraded': '2024-03-06T07:28:36.803649+0000', 'last_fullsized': '2024-03-06T07:28:36.803649+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.9 is not active+clean 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804063+0000', 'last_change': '2024-03-06T07:28:36.804063+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804063+0000', 'last_undegraded': '2024-03-06T07:28:36.804063+0000', 'last_fullsized': '2024-03-06T07:28:36.804063+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.6 is not active+clean 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806573+0000', 'last_change': '2024-03-06T07:28:36.806573+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806573+0000', 'last_undegraded': '2024-03-06T07:28:36.806573+0000', 'last_fullsized': '2024-03-06T07:28:36.806573+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.7 is not active+clean 2024-03-06T07:28:41.624 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.802330+0000', 'last_change': '2024-03-06T07:28:36.802330+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.802330+0000', 'last_undegraded': '2024-03-06T07:28:36.802330+0000', 'last_fullsized': '2024-03-06T07:28:36.802330+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-06T07:28:41.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.4 is not active+clean 2024-03-06T07:28:41.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803589+0000', 'last_change': '2024-03-06T07:28:36.803589+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803589+0000', 'last_undegraded': '2024-03-06T07:28:36.803589+0000', 'last_fullsized': '2024-03-06T07:28:36.803589+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.5 is not active+clean 2024-03-06T07:28:41.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.806172+0000', 'last_change': '2024-03-06T07:28:36.806172+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.806172+0000', 'last_undegraded': '2024-03-06T07:28:36.806172+0000', 'last_fullsized': '2024-03-06T07:28:36.806172+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-06T07:28:41.625 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.2 is not active+clean 2024-03-06T07:28:41.625 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.802948+0000', 'last_change': '2024-03-06T07:28:36.802948+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.802948+0000', 'last_undegraded': '2024-03-06T07:28:36.802948+0000', 'last_fullsized': '2024-03-06T07:28:36.802948+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.3 is not active+clean 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.798604+0000', 'last_change': '2024-03-06T07:28:36.798604+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.798604+0000', 'last_undegraded': '2024-03-06T07:28:36.798604+0000', 'last_fullsized': '2024-03-06T07:28:36.798604+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.0 is not active+clean 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.797980+0000', 'last_change': '2024-03-06T07:28:36.797980+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.797980+0000', 'last_undegraded': '2024-03-06T07:28:36.797980+0000', 'last_fullsized': '2024-03-06T07:28:36.797980+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.1 is not active+clean 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.803573+0000', 'last_change': '2024-03-06T07:28:36.803573+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.803573+0000', 'last_undegraded': '2024-03-06T07:28:36.803573+0000', 'last_fullsized': '2024-03-06T07:28:36.803573+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:PG 19.12 is not active+clean 2024-03-06T07:28:41.626 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '19.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 125, 'state': 'creating+peering', 'last_fresh': '2024-03-06T07:28:36.804620+0000', 'last_change': '2024-03-06T07:28:36.804620+0000', 'last_active': '2024-03-06T07:28:36.789256+0000', 'last_peered': '2024-03-06T07:28:36.789256+0000', 'last_clean': '2024-03-06T07:28:36.789256+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-06T07:28:36.804620+0000', 'last_undegraded': '2024-03-06T07:28:36.804620+0000', 'last_fullsized': '2024-03-06T07:28:36.804620+0000', 'mapping_epoch': 125, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 125, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'last_clean_scrub_stamp': '2024-03-06T07:28:36.789256+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-06T07:28:41.627 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-06T07:28:41.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.747+0000 7f7b5dd7a640 1 Processor -- start 2024-03-06T07:28:41.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.747+0000 7f7b5dd7a640 1 -- start start 2024-03-06T07:28:41.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b5dd7a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7b580cc920 0x7f7b5815eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b5dd7a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b5806e000 0x7f7b58072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b5dd7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5806e770 con 0x7f7b580b4a50 2024-03-06T07:28:41.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b5dd7a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5815e660 con 0x7f7b5806e000 2024-03-06T07:28:41.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b5dd7a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5806e490 con 0x7f7b580cc920 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b5806e000 0x7f7b58072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b5806e000 0x7f7b58072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59332/0 (socket says 172.21.15.72:59332) 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 -- 172.21.15.72:0/1441834474 learned_addr learned my addr 172.21.15.72:0/1441834474 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b577fe640 1 --2- 172.21.15.72:0/1441834474 >> v2:172.21.15.146:3300/0 conn(0x7f7b580cc920 0x7f7b5815eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 -- 172.21.15.72:0/1441834474 >> v2:172.21.15.146:3300/0 conn(0x7f7b580cc920 msgr2=0x7f7b5815eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 --2- 172.21.15.72:0/1441834474 >> v2:172.21.15.146:3300/0 conn(0x7f7b580cc920 0x7f7b5815eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 -- 172.21.15.72:0/1441834474 >> v1:172.21.15.146:6789/0 conn(0x7f7b580b4a50 legacy=0x7f7b58150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.748+0000 7f7b4effd640 1 -- 172.21.15.72:0/1441834474 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5815e010 con 0x7f7b5806e000 2024-03-06T07:28:41.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.749+0000 7f7b4effd640 1 --2- 172.21.15.72:0/1441834474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b5806e000 0x7f7b58072070 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7b580737d0 tx=0x7f7b4002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6395829f4b10cd5a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.749+0000 7f7b56ffd640 1 -- 172.21.15.72:0/1441834474 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4003b070 con 0x7f7b5806e000 2024-03-06T07:28:41.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.749+0000 7f7b56ffd640 1 -- 172.21.15.72:0/1441834474 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b40036440 con 0x7f7b5806e000 2024-03-06T07:28:41.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.749+0000 7f7b56ffd640 1 -- 172.21.15.72:0/1441834474 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40036810 con 0x7f7b5806e000 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/1441834474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b5806e000 msgr2=0x7f7b58072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/1441834474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b5806e000 0x7f7b58072070 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7b580737d0 tx=0x7f7b4002e0e0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/1441834474 shutdown_connections 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/1441834474 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b5806e000 0x7f7b58072070 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/1441834474 >> v2:172.21.15.146:3300/0 conn(0x7f7b580cc920 0x7f7b5815eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/1441834474 >> 172.21.15.72:0/1441834474 conn(0x7f7b580b1c80 msgr2=0x7f7b580cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/1441834474 shutdown_connections 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.750+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/1441834474 wait complete. 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.752+0000 7f7b5dd7a640 1 Processor -- start 2024-03-06T07:28:41.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.752+0000 7f7b5dd7a640 1 -- start start 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b5dd7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b580b4a50 0x7f7b581f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b5dd7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 0x7f7b5814d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b5dd7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b5814f640 0x7f7b58148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b5dd7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580cce80 con 0x7f7b580cc920 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b5dd7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b580ccd00 con 0x7f7b5814f640 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b5dd7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58070940 con 0x7f7b580b4a50 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b4effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 0x7f7b5814d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b4effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 0x7f7b5814d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59340/0 (socket says 172.21.15.72:59340) 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b4effd640 1 -- 172.21.15.72:0/2076654742 learned_addr learned my addr 172.21.15.72:0/2076654742 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b577fe640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b580b4a50 0x7f7b581f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.753+0000 7f7b57fff640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b5814f640 0x7f7b58148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b4effd640 1 -- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b580b4a50 msgr2=0x7f7b581f0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b4effd640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b580b4a50 0x7f7b581f0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b4effd640 1 -- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b5814f640 msgr2=0x7f7b58148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b4effd640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b5814f640 0x7f7b58148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b4effd640 1 -- 172.21.15.72:0/2076654742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5815a270 con 0x7f7b580cc920 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b577fe640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b580b4a50 0x7f7b581f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b57fff640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b5814f640 0x7f7b58148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b4effd640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 0x7f7b5814d970 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7b4000b2c0 tx=0x7f7b400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b40045020 con 0x7f7b580cc920 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b400346c0 con 0x7f7b580cc920 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5814fc00 con 0x7f7b580cc920 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4003b040 con 0x7f7b580cc920 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.754+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b581501c0 con 0x7f7b580cc920 2024-03-06T07:28:41.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.755+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5804f330 con 0x7f7b580cc920 2024-03-06T07:28:41.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.756+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7b40048020 con 0x7f7b580cc920 2024-03-06T07:28:41.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.756+0000 7f7b54ff9640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3806b4d0 0x7f7b3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:41.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.756+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b400aecf0 con 0x7f7b580cc920 2024-03-06T07:28:41.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.756+0000 7f7b577fe640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3806b4d0 0x7f7b3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:41.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.757+0000 7f7b577fe640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3806b4d0 0x7f7b3806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7b48006fd0 tx=0x7f7b48008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:41.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:41.760+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7b400b3070 con 0x7f7b580cc920 2024-03-06T07:28:42.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.010+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7b580b40d0 con 0x7f7b580cc920 2024-03-06T07:28:42.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.011+0000 7f7b54ff9640 1 -- 172.21.15.72:0/2076654742 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1035 (secure 0 0 0) 0x7f7b4004f030 con 0x7f7b580cc920 2024-03-06T07:28:42.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:42.013 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":127,"num_osds":8,"num_up_osds":8,"osd_up_since":1709709696,"num_in_osds":8,"osd_in_since":1709709687,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":337248256,"bytes_avail":772756865024,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":79,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"a","status":"up:active","gid":8640}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-03-06T07:28:28.021254+0000","services":{}},"progress_events":{}} 2024-03-06T07:28:42.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3806b4d0 msgr2=0x7f7b3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3806b4d0 0x7f7b3806d990 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7b48006fd0 tx=0x7f7b48008040 comp rx=0 tx=0).stop 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 msgr2=0x7f7b5814d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 0x7f7b5814d970 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7b4000b2c0 tx=0x7f7b400046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 shutdown_connections 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7b3806b4d0 0x7f7b3806d990 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b5814f640 0x7f7b58148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b580cc920 0x7f7b5814d970 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 --2- 172.21.15.72:0/2076654742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b580b4a50 0x7f7b581f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 >> 172.21.15.72:0/2076654742 conn(0x7f7b580b1c80 msgr2=0x7f7b5806cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 shutdown_connections 2024-03-06T07:28:42.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.013+0000 7f7b5dd7a640 1 -- 172.21.15.72:0/2076654742 wait complete. 2024-03-06T07:28:42.026 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-06T07:28:42.026 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:42.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 Processor -- start 2024-03-06T07:28:42.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 -- start start 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f720015bb20 0x7f72000d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72000cfff0 0x7f72000d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7200161980 con 0x7f72000ce4b0 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7200161c60 con 0x7f720015bb20 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.130+0000 7f720620e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7200161f40 con 0x7f72000cfff0 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f720015bb20 0x7f72000d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f720015bb20 0x7f72000d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59354/0 (socket says 172.21.15.72:59354) 2024-03-06T07:28:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 -- 172.21.15.72:0/245074976 learned_addr learned my addr 172.21.15.72:0/245074976 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 -- 172.21.15.72:0/245074976 >> v2:172.21.15.146:3300/0 conn(0x7f72000cfff0 msgr2=0x7f72000d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71feffd640 1 --2- 172.21.15.72:0/245074976 >> v2:172.21.15.146:3300/0 conn(0x7f72000cfff0 0x7f72000d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 --2- 172.21.15.72:0/245074976 >> v2:172.21.15.146:3300/0 conn(0x7f72000cfff0 0x7f72000d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 -- 172.21.15.72:0/245074976 >> v1:172.21.15.146:6789/0 conn(0x7f72000ce4b0 legacy=0x7f72000ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:42.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.131+0000 7f71ff7fe640 1 -- 172.21.15.72:0/245074976 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7200162220 con 0x7f720015bb20 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.132+0000 7f71ff7fe640 1 --2- 172.21.15.72:0/245074976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000d1df0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7200071290 tx=0x7f71f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1443921cef4d69d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.132+0000 7f71fe7fc640 1 -- 172.21.15.72:0/245074976 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f0035020 con 0x7f720015bb20 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.132+0000 7f71fe7fc640 1 -- 172.21.15.72:0/245074976 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f0031870 con 0x7f720015bb20 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.132+0000 7f71fe7fc640 1 -- 172.21.15.72:0/245074976 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f0031c40 con 0x7f720015bb20 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 -- 172.21.15.72:0/245074976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 msgr2=0x7f72000d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 --2- 172.21.15.72:0/245074976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000d1df0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f7200071290 tx=0x7f71f002b910 comp rx=0 tx=0).stop 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 -- 172.21.15.72:0/245074976 shutdown_connections 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 --2- 172.21.15.72:0/245074976 >> v2:172.21.15.146:3300/0 conn(0x7f72000cfff0 0x7f72000d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 --2- 172.21.15.72:0/245074976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000d1df0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 -- 172.21.15.72:0/245074976 >> 172.21.15.72:0/245074976 conn(0x7f720009a9f0 msgr2=0x7f720009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 -- 172.21.15.72:0/245074976 shutdown_connections 2024-03-06T07:28:42.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.133+0000 7f720620e640 1 -- 172.21.15.72:0/245074976 wait complete. 2024-03-06T07:28:42.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.134+0000 7f720620e640 1 Processor -- start 2024-03-06T07:28:42.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.134+0000 7f720620e640 1 -- start start 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f720620e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72000ce4b0 0x7f72000cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f720620e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72000cfff0 0x7f72001e81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f720620e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f720620e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72000d24b0 con 0x7f720015bb20 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f720620e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72000d2330 con 0x7f72000cfff0 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f720620e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72000d1980 con 0x7f72000ce4b0 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000b6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59358/0 (socket says 172.21.15.72:59358) 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 -- 172.21.15.72:0/2962562890 learned_addr learned my addr 172.21.15.72:0/2962562890 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 -- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72000ce4b0 msgr2=0x7f72000cb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71feffd640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72000cfff0 0x7f72001e81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71ff7fe640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72000ce4b0 0x7f72000cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72000ce4b0 0x7f72000cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 -- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72000cfff0 msgr2=0x7f72001e81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72000cfff0 0x7f72001e81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 -- 172.21.15.72:0/2962562890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000ca5e0 con 0x7f720015bb20 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.135+0000 7f71fffff640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000b6d20 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f71f800c780 tx=0x7f71f800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.136+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8017070 con 0x7f720015bb20 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.136+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8003da0 con 0x7f720015bb20 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.136+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72000ca8c0 con 0x7f720015bb20 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.136+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80052b0 con 0x7f720015bb20 2024-03-06T07:28:42.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.136+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72000cae80 con 0x7f720015bb20 2024-03-06T07:28:42.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.137+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720004f330 con 0x7f720015bb20 2024-03-06T07:28:42.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.137+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f71f801a070 con 0x7f720015bb20 2024-03-06T07:28:42.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.138+0000 7f71e7fff640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71ec06b4d0 0x7f71ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:42.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.138+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71f8089880 con 0x7f720015bb20 2024-03-06T07:28:42.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.138+0000 7f71ff7fe640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71ec06b4d0 0x7f71ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:42.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.138+0000 7f71ff7fe640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71ec06b4d0 0x7f71ec06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f71f0036000 tx=0x7f71f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:42.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.142+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f71f8058700 con 0x7f720015bb20 2024-03-06T07:28:42.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.316+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72000b1820 con 0x7f71ec06b4d0 2024-03-06T07:28:42.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.322+0000 7f71e7fff640 1 -- 172.21.15.72:0/2962562890 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251571 (secure 0 0 0) 0x7f72000b1820 con 0x7f71ec06b4d0 2024-03-06T07:28:42.327 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:42.328 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71ec06b4d0 msgr2=0x7f71ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71ec06b4d0 0x7f71ec06d990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f71f0036000 tx=0x7f71f0008030 comp rx=0 tx=0).stop 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 msgr2=0x7f72000b6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000b6d20 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f71f800c780 tx=0x7f71f800cc40 comp rx=0 tx=0).stop 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 shutdown_connections 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f71ec06b4d0 0x7f71ec06d990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f720015bb20 0x7f72000b6d20 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72000cfff0 0x7f72001e81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 --2- 172.21.15.72:0/2962562890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72000ce4b0 0x7f72000cb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 >> 172.21.15.72:0/2962562890 conn(0x7f720009a9f0 msgr2=0x7f720006f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 shutdown_connections 2024-03-06T07:28:42.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:42.326+0000 7f720620e640 1 -- 172.21.15.72:0/2962562890 wait complete. 2024-03-06T07:28:42.339 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-03-06T07:28:42.023559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":201,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":165,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329592,"kb_used_data":13564,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645128,"statfs":{"total":773094113280,"available":772756611072,"internally_reserved":0,"allocated":13889536,"data_stored":6708510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000940"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806567+0000","last_change":"2024-03-06T07:28:36.806567+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806567+0000","last_undegraded":"2024-03-06T07:28:36.806567+0000","last_fullsized":"2024-03-06T07:28:36.806567+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801488+0000","last_change":"2024-03-06T07:28:37.821388+0000","last_active":"2024-03-06T07:28:38.801488+0000","last_peered":"2024-03-06T07:28:38.801488+0000","last_clean":"2024-03-06T07:28:38.801488+0000","last_became_active":"2024-03-06T07:28:37.820763+0000","last_became_peered":"2024-03-06T07:28:37.820763+0000","last_unstale":"2024-03-06T07:28:38.801488+0000","last_undegraded":"2024-03-06T07:28:38.801488+0000","last_fullsized":"2024-03-06T07:28:38.801488+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:47.914373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800456+0000","last_change":"2024-03-06T07:28:37.807180+0000","last_active":"2024-03-06T07:28:38.800456+0000","last_peered":"2024-03-06T07:28:38.800456+0000","last_clean":"2024-03-06T07:28:38.800456+0000","last_became_active":"2024-03-06T07:28:37.806885+0000","last_became_peered":"2024-03-06T07:28:37.806885+0000","last_unstale":"2024-03-06T07:28:38.800456+0000","last_undegraded":"2024-03-06T07:28:38.800456+0000","last_fullsized":"2024-03-06T07:28:38.800456+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:30.395699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806501+0000","last_change":"2024-03-06T07:28:36.806501+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806501+0000","last_undegraded":"2024-03-06T07:28:36.806501+0000","last_fullsized":"2024-03-06T07:28:36.806501+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801307+0000","last_change":"2024-03-06T07:28:37.813718+0000","last_active":"2024-03-06T07:28:39.801307+0000","last_peered":"2024-03-06T07:28:39.801307+0000","last_clean":"2024-03-06T07:28:39.801307+0000","last_became_active":"2024-03-06T07:28:37.813446+0000","last_became_peered":"2024-03-06T07:28:37.813446+0000","last_unstale":"2024-03-06T07:28:39.801307+0000","last_undegraded":"2024-03-06T07:28:39.801307+0000","last_fullsized":"2024-03-06T07:28:39.801307+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:24.990126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803550+0000","last_change":"2024-03-06T07:28:37.818530+0000","last_active":"2024-03-06T07:28:39.803550+0000","last_peered":"2024-03-06T07:28:39.803550+0000","last_clean":"2024-03-06T07:28:39.803550+0000","last_became_active":"2024-03-06T07:28:37.818198+0000","last_became_peered":"2024-03-06T07:28:37.818198+0000","last_unstale":"2024-03-06T07:28:39.803550+0000","last_undegraded":"2024-03-06T07:28:39.803550+0000","last_fullsized":"2024-03-06T07:28:39.803550+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.430067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803374+0000","last_change":"2024-03-06T07:28:37.812791+0000","last_active":"2024-03-06T07:28:39.803374+0000","last_peered":"2024-03-06T07:28:39.803374+0000","last_clean":"2024-03-06T07:28:39.803374+0000","last_became_active":"2024-03-06T07:28:37.812509+0000","last_became_peered":"2024-03-06T07:28:37.812509+0000","last_unstale":"2024-03-06T07:28:39.803374+0000","last_undegraded":"2024-03-06T07:28:39.803374+0000","last_fullsized":"2024-03-06T07:28:39.803374+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:43.391321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800451+0000","last_change":"2024-03-06T07:28:37.825862+0000","last_active":"2024-03-06T07:28:38.800451+0000","last_peered":"2024-03-06T07:28:38.800451+0000","last_clean":"2024-03-06T07:28:38.800451+0000","last_became_active":"2024-03-06T07:28:37.825352+0000","last_became_peered":"2024-03-06T07:28:37.825352+0000","last_unstale":"2024-03-06T07:28:38.800451+0000","last_undegraded":"2024-03-06T07:28:38.800451+0000","last_fullsized":"2024-03-06T07:28:38.800451+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:21.866852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800535+0000","last_change":"2024-03-06T07:28:37.810958+0000","last_active":"2024-03-06T07:28:38.800535+0000","last_peered":"2024-03-06T07:28:38.800535+0000","last_clean":"2024-03-06T07:28:38.800535+0000","last_became_active":"2024-03-06T07:28:37.809932+0000","last_became_peered":"2024-03-06T07:28:37.809932+0000","last_unstale":"2024-03-06T07:28:38.800535+0000","last_undegraded":"2024-03-06T07:28:38.800535+0000","last_fullsized":"2024-03-06T07:28:38.800535+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.430067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801698+0000","last_change":"2024-03-06T07:28:37.818998+0000","last_active":"2024-03-06T07:28:38.801698+0000","last_peered":"2024-03-06T07:28:38.801698+0000","last_clean":"2024-03-06T07:28:38.801698+0000","last_became_active":"2024-03-06T07:28:37.818743+0000","last_became_peered":"2024-03-06T07:28:37.818743+0000","last_unstale":"2024-03-06T07:28:38.801698+0000","last_undegraded":"2024-03-06T07:28:38.801698+0000","last_fullsized":"2024-03-06T07:28:38.801698+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:44.697288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998650+0000","last_change":"2024-03-06T07:28:37.818817+0000","last_active":"2024-03-06T07:28:38.998650+0000","last_peered":"2024-03-06T07:28:38.998650+0000","last_clean":"2024-03-06T07:28:38.998650+0000","last_became_active":"2024-03-06T07:28:37.818560+0000","last_became_peered":"2024-03-06T07:28:37.818560+0000","last_unstale":"2024-03-06T07:28:38.998650+0000","last_undegraded":"2024-03-06T07:28:38.998650+0000","last_fullsized":"2024-03-06T07:28:38.998650+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:37.951470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800217+0000","last_change":"2024-03-06T07:28:37.810389+0000","last_active":"2024-03-06T07:28:38.800217+0000","last_peered":"2024-03-06T07:28:38.800217+0000","last_clean":"2024-03-06T07:28:38.800217+0000","last_became_active":"2024-03-06T07:28:37.810109+0000","last_became_peered":"2024-03-06T07:28:37.810109+0000","last_unstale":"2024-03-06T07:28:38.800217+0000","last_undegraded":"2024-03-06T07:28:38.800217+0000","last_fullsized":"2024-03-06T07:28:38.800217+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:04.971325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.806487+0000","last_change":"2024-03-06T07:28:36.806487+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.806487+0000","last_undegraded":"2024-03-06T07:28:36.806487+0000","last_fullsized":"2024-03-06T07:28:36.806487+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801383+0000","last_change":"2024-03-06T07:28:37.820392+0000","last_active":"2024-03-06T07:28:38.801383+0000","last_peered":"2024-03-06T07:28:38.801383+0000","last_clean":"2024-03-06T07:28:38.801383+0000","last_became_active":"2024-03-06T07:28:37.819941+0000","last_became_peered":"2024-03-06T07:28:37.819941+0000","last_unstale":"2024-03-06T07:28:38.801383+0000","last_undegraded":"2024-03-06T07:28:38.801383+0000","last_fullsized":"2024-03-06T07:28:38.801383+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:56.329155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.805149+0000","last_change":"2024-03-06T07:28:36.805149+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.805149+0000","last_undegraded":"2024-03-06T07:28:36.805149+0000","last_fullsized":"2024-03-06T07:28:36.805149+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800475+0000","last_change":"2024-03-06T07:28:37.820286+0000","last_active":"2024-03-06T07:28:38.800475+0000","last_peered":"2024-03-06T07:28:38.800475+0000","last_clean":"2024-03-06T07:28:38.800475+0000","last_became_active":"2024-03-06T07:28:37.820027+0000","last_became_peered":"2024-03-06T07:28:37.820027+0000","last_unstale":"2024-03-06T07:28:38.800475+0000","last_undegraded":"2024-03-06T07:28:38.800475+0000","last_fullsized":"2024-03-06T07:28:38.800475+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:19.569381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803800+0000","last_change":"2024-03-06T07:28:37.813885+0000","last_active":"2024-03-06T07:28:39.803800+0000","last_peered":"2024-03-06T07:28:39.803800+0000","last_clean":"2024-03-06T07:28:39.803800+0000","last_became_active":"2024-03-06T07:28:37.813622+0000","last_became_peered":"2024-03-06T07:28:37.813622+0000","last_unstale":"2024-03-06T07:28:39.803800+0000","last_undegraded":"2024-03-06T07:28:39.803800+0000","last_fullsized":"2024-03-06T07:28:39.803800+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:45.463350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800622+0000","last_change":"2024-03-06T07:28:37.821840+0000","last_active":"2024-03-06T07:28:38.800622+0000","last_peered":"2024-03-06T07:28:38.800622+0000","last_clean":"2024-03-06T07:28:38.800622+0000","last_became_active":"2024-03-06T07:28:37.821555+0000","last_became_peered":"2024-03-06T07:28:37.821555+0000","last_unstale":"2024-03-06T07:28:38.800622+0000","last_undegraded":"2024-03-06T07:28:38.800622+0000","last_fullsized":"2024-03-06T07:28:38.800622+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:50.757145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998635+0000","last_change":"2024-03-06T07:28:37.813180+0000","last_active":"2024-03-06T07:28:38.998635+0000","last_peered":"2024-03-06T07:28:38.998635+0000","last_clean":"2024-03-06T07:28:38.998635+0000","last_became_active":"2024-03-06T07:28:37.812934+0000","last_became_peered":"2024-03-06T07:28:37.812934+0000","last_unstale":"2024-03-06T07:28:38.998635+0000","last_undegraded":"2024-03-06T07:28:38.998635+0000","last_fullsized":"2024-03-06T07:28:38.998635+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:22.911067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800947+0000","last_change":"2024-03-06T07:28:37.817427+0000","last_active":"2024-03-06T07:28:38.800947+0000","last_peered":"2024-03-06T07:28:38.800947+0000","last_clean":"2024-03-06T07:28:38.800947+0000","last_became_active":"2024-03-06T07:28:37.817166+0000","last_became_peered":"2024-03-06T07:28:37.817166+0000","last_unstale":"2024-03-06T07:28:38.800947+0000","last_undegraded":"2024-03-06T07:28:38.800947+0000","last_fullsized":"2024-03-06T07:28:38.800947+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:54.190073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801280+0000","last_change":"2024-03-06T07:28:37.814034+0000","last_active":"2024-03-06T07:28:39.801280+0000","last_peered":"2024-03-06T07:28:39.801280+0000","last_clean":"2024-03-06T07:28:39.801280+0000","last_became_active":"2024-03-06T07:28:37.813747+0000","last_became_peered":"2024-03-06T07:28:37.813747+0000","last_unstale":"2024-03-06T07:28:39.801280+0000","last_undegraded":"2024-03-06T07:28:39.801280+0000","last_fullsized":"2024-03-06T07:28:39.801280+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:36.861383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998322+0000","last_change":"2024-03-06T07:28:37.818450+0000","last_active":"2024-03-06T07:28:38.998322+0000","last_peered":"2024-03-06T07:28:38.998322+0000","last_clean":"2024-03-06T07:28:38.998322+0000","last_became_active":"2024-03-06T07:28:37.818195+0000","last_became_peered":"2024-03-06T07:28:37.818195+0000","last_unstale":"2024-03-06T07:28:38.998322+0000","last_undegraded":"2024-03-06T07:28:38.998322+0000","last_fullsized":"2024-03-06T07:28:38.998322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:40.275303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800257+0000","last_change":"2024-03-06T07:28:37.810549+0000","last_active":"2024-03-06T07:28:38.800257+0000","last_peered":"2024-03-06T07:28:38.800257+0000","last_clean":"2024-03-06T07:28:38.800257+0000","last_became_active":"2024-03-06T07:28:37.809880+0000","last_became_peered":"2024-03-06T07:28:37.809880+0000","last_unstale":"2024-03-06T07:28:38.800257+0000","last_undegraded":"2024-03-06T07:28:38.800257+0000","last_fullsized":"2024-03-06T07:28:38.800257+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:59.096651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998291+0000","last_change":"2024-03-06T07:28:37.812050+0000","last_active":"2024-03-06T07:28:38.998291+0000","last_peered":"2024-03-06T07:28:38.998291+0000","last_clean":"2024-03-06T07:28:38.998291+0000","last_became_active":"2024-03-06T07:28:37.811773+0000","last_became_peered":"2024-03-06T07:28:37.811773+0000","last_unstale":"2024-03-06T07:28:38.998291+0000","last_undegraded":"2024-03-06T07:28:38.998291+0000","last_fullsized":"2024-03-06T07:28:38.998291+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:37.879363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801643+0000","last_change":"2024-03-06T07:28:37.816377+0000","last_active":"2024-03-06T07:28:38.801643+0000","last_peered":"2024-03-06T07:28:38.801643+0000","last_clean":"2024-03-06T07:28:38.801643+0000","last_became_active":"2024-03-06T07:28:37.816100+0000","last_became_peered":"2024-03-06T07:28:37.816100+0000","last_unstale":"2024-03-06T07:28:38.801643+0000","last_undegraded":"2024-03-06T07:28:38.801643+0000","last_fullsized":"2024-03-06T07:28:38.801643+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:16.108047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800770+0000","last_change":"2024-03-06T07:28:37.820322+0000","last_active":"2024-03-06T07:28:38.800770+0000","last_peered":"2024-03-06T07:28:38.800770+0000","last_clean":"2024-03-06T07:28:38.800770+0000","last_became_active":"2024-03-06T07:28:37.820033+0000","last_became_peered":"2024-03-06T07:28:37.820033+0000","last_unstale":"2024-03-06T07:28:38.800770+0000","last_undegraded":"2024-03-06T07:28:38.800770+0000","last_fullsized":"2024-03-06T07:28:38.800770+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:24.104089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804726+0000","last_change":"2024-03-06T07:28:36.804726+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804726+0000","last_undegraded":"2024-03-06T07:28:36.804726+0000","last_fullsized":"2024-03-06T07:28:36.804726+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569432+0000","last_change":"2024-03-06T07:28:37.825936+0000","last_active":"2024-03-06T07:28:39.569432+0000","last_peered":"2024-03-06T07:28:39.569432+0000","last_clean":"2024-03-06T07:28:39.569432+0000","last_became_active":"2024-03-06T07:28:37.825175+0000","last_became_peered":"2024-03-06T07:28:37.825175+0000","last_unstale":"2024-03-06T07:28:39.569432+0000","last_undegraded":"2024-03-06T07:28:39.569432+0000","last_fullsized":"2024-03-06T07:28:39.569432+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.430067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804323+0000","last_change":"2024-03-06T07:28:36.804323+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804323+0000","last_undegraded":"2024-03-06T07:28:36.804323+0000","last_fullsized":"2024-03-06T07:28:36.804323+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803816+0000","last_change":"2024-03-06T07:28:36.803816+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803816+0000","last_undegraded":"2024-03-06T07:28:36.803816+0000","last_fullsized":"2024-03-06T07:28:36.803816+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801396+0000","last_change":"2024-03-06T07:28:37.809101+0000","last_active":"2024-03-06T07:28:39.801396+0000","last_peered":"2024-03-06T07:28:39.801396+0000","last_clean":"2024-03-06T07:28:39.801396+0000","last_became_active":"2024-03-06T07:28:37.808827+0000","last_became_peered":"2024-03-06T07:28:37.808827+0000","last_unstale":"2024-03-06T07:28:39.801396+0000","last_undegraded":"2024-03-06T07:28:39.801396+0000","last_fullsized":"2024-03-06T07:28:39.801396+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:41.270619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803390+0000","last_change":"2024-03-06T07:28:37.823717+0000","last_active":"2024-03-06T07:28:39.803390+0000","last_peered":"2024-03-06T07:28:39.803390+0000","last_clean":"2024-03-06T07:28:39.803390+0000","last_became_active":"2024-03-06T07:28:37.823383+0000","last_became_peered":"2024-03-06T07:28:37.823383+0000","last_unstale":"2024-03-06T07:28:39.803390+0000","last_undegraded":"2024-03-06T07:28:39.803390+0000","last_fullsized":"2024-03-06T07:28:39.803390+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:21.866852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800756+0000","last_change":"2024-03-06T07:28:37.816614+0000","last_active":"2024-03-06T07:28:38.800756+0000","last_peered":"2024-03-06T07:28:38.800756+0000","last_clean":"2024-03-06T07:28:38.800756+0000","last_became_active":"2024-03-06T07:28:37.816301+0000","last_became_peered":"2024-03-06T07:28:37.816301+0000","last_unstale":"2024-03-06T07:28:38.800756+0000","last_undegraded":"2024-03-06T07:28:38.800756+0000","last_fullsized":"2024-03-06T07:28:38.800756+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:21.702361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"126'4","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801710+0000","last_change":"2024-03-06T07:28:36.812840+0000","last_active":"2024-03-06T07:28:39.801710+0000","last_peered":"2024-03-06T07:28:39.801710+0000","last_clean":"2024-03-06T07:28:39.801710+0000","last_became_active":"2024-03-06T07:28:36.812374+0000","last_became_peered":"2024-03-06T07:28:36.812374+0000","last_unstale":"2024-03-06T07:28:39.801710+0000","last_undegraded":"2024-03-06T07:28:39.801710+0000","last_fullsized":"2024-03-06T07:28:39.801710+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:43.695281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800656+0000","last_change":"2024-03-06T07:28:37.804103+0000","last_active":"2024-03-06T07:28:38.800656+0000","last_peered":"2024-03-06T07:28:38.800656+0000","last_clean":"2024-03-06T07:28:38.800656+0000","last_became_active":"2024-03-06T07:28:37.803742+0000","last_became_peered":"2024-03-06T07:28:37.803742+0000","last_unstale":"2024-03-06T07:28:38.800656+0000","last_undegraded":"2024-03-06T07:28:38.800656+0000","last_fullsized":"2024-03-06T07:28:38.800656+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:56.972667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569487+0000","last_change":"2024-03-06T07:28:36.808862+0000","last_active":"2024-03-06T07:28:39.569487+0000","last_peered":"2024-03-06T07:28:39.569487+0000","last_clean":"2024-03-06T07:28:39.569487+0000","last_became_active":"2024-03-06T07:28:36.808596+0000","last_became_peered":"2024-03-06T07:28:36.808596+0000","last_unstale":"2024-03-06T07:28:39.569487+0000","last_undegraded":"2024-03-06T07:28:39.569487+0000","last_fullsized":"2024-03-06T07:28:39.569487+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:55.327149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801373+0000","last_change":"2024-03-06T07:28:37.810239+0000","last_active":"2024-03-06T07:28:39.801373+0000","last_peered":"2024-03-06T07:28:39.801373+0000","last_clean":"2024-03-06T07:28:39.801373+0000","last_became_active":"2024-03-06T07:28:37.809329+0000","last_became_peered":"2024-03-06T07:28:37.809329+0000","last_unstale":"2024-03-06T07:28:39.801373+0000","last_undegraded":"2024-03-06T07:28:39.801373+0000","last_fullsized":"2024-03-06T07:28:39.801373+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:19.165526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998744+0000","last_change":"2024-03-06T07:28:36.806718+0000","last_active":"2024-03-06T07:28:38.998744+0000","last_peered":"2024-03-06T07:28:38.998744+0000","last_clean":"2024-03-06T07:28:38.998744+0000","last_became_active":"2024-03-06T07:28:36.806447+0000","last_became_peered":"2024-03-06T07:28:36.806447+0000","last_unstale":"2024-03-06T07:28:38.998744+0000","last_undegraded":"2024-03-06T07:28:38.998744+0000","last_fullsized":"2024-03-06T07:28:38.998744+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:54.145543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801599+0000","last_change":"2024-03-06T07:28:37.821948+0000","last_active":"2024-03-06T07:28:39.801599+0000","last_peered":"2024-03-06T07:28:39.801599+0000","last_clean":"2024-03-06T07:28:39.801599+0000","last_became_active":"2024-03-06T07:28:37.821414+0000","last_became_peered":"2024-03-06T07:28:37.821414+0000","last_unstale":"2024-03-06T07:28:39.801599+0000","last_undegraded":"2024-03-06T07:28:39.801599+0000","last_fullsized":"2024-03-06T07:28:39.801599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:25.449099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801572+0000","last_change":"2024-03-06T07:28:36.812550+0000","last_active":"2024-03-06T07:28:38.801572+0000","last_peered":"2024-03-06T07:28:38.801572+0000","last_clean":"2024-03-06T07:28:38.801572+0000","last_became_active":"2024-03-06T07:28:36.812312+0000","last_became_peered":"2024-03-06T07:28:36.812312+0000","last_unstale":"2024-03-06T07:28:38.801572+0000","last_undegraded":"2024-03-06T07:28:38.801572+0000","last_fullsized":"2024-03-06T07:28:38.801572+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:22.791648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800504+0000","last_change":"2024-03-06T07:28:37.818177+0000","last_active":"2024-03-06T07:28:38.800504+0000","last_peered":"2024-03-06T07:28:38.800504+0000","last_clean":"2024-03-06T07:28:38.800504+0000","last_became_active":"2024-03-06T07:28:37.817863+0000","last_became_peered":"2024-03-06T07:28:37.817863+0000","last_unstale":"2024-03-06T07:28:38.800504+0000","last_undegraded":"2024-03-06T07:28:38.800504+0000","last_fullsized":"2024-03-06T07:28:38.800504+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:48.725241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801524+0000","last_change":"2024-03-06T07:28:36.812168+0000","last_active":"2024-03-06T07:28:38.801524+0000","last_peered":"2024-03-06T07:28:38.801524+0000","last_clean":"2024-03-06T07:28:38.801524+0000","last_became_active":"2024-03-06T07:28:36.811914+0000","last_became_peered":"2024-03-06T07:28:36.811914+0000","last_unstale":"2024-03-06T07:28:38.801524+0000","last_undegraded":"2024-03-06T07:28:38.801524+0000","last_fullsized":"2024-03-06T07:28:38.801524+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:57.566949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803578+0000","last_change":"2024-03-06T07:28:37.812896+0000","last_active":"2024-03-06T07:28:39.803578+0000","last_peered":"2024-03-06T07:28:39.803578+0000","last_clean":"2024-03-06T07:28:39.803578+0000","last_became_active":"2024-03-06T07:28:37.812577+0000","last_became_peered":"2024-03-06T07:28:37.812577+0000","last_unstale":"2024-03-06T07:28:39.803578+0000","last_undegraded":"2024-03-06T07:28:39.803578+0000","last_fullsized":"2024-03-06T07:28:39.803578+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:30.395699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800472+0000","last_change":"2024-03-06T07:28:36.808734+0000","last_active":"2024-03-06T07:28:38.800472+0000","last_peered":"2024-03-06T07:28:38.800472+0000","last_clean":"2024-03-06T07:28:38.800472+0000","last_became_active":"2024-03-06T07:28:36.808474+0000","last_became_peered":"2024-03-06T07:28:36.808474+0000","last_unstale":"2024-03-06T07:28:38.800472+0000","last_undegraded":"2024-03-06T07:28:38.800472+0000","last_fullsized":"2024-03-06T07:28:38.800472+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800624+0000","last_change":"2024-03-06T07:28:37.824916+0000","last_active":"2024-03-06T07:28:38.800624+0000","last_peered":"2024-03-06T07:28:38.800624+0000","last_clean":"2024-03-06T07:28:38.800624+0000","last_became_active":"2024-03-06T07:28:37.824538+0000","last_became_peered":"2024-03-06T07:28:37.824538+0000","last_unstale":"2024-03-06T07:28:38.800624+0000","last_undegraded":"2024-03-06T07:28:38.800624+0000","last_fullsized":"2024-03-06T07:28:38.800624+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:45.463350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803298+0000","last_change":"2024-03-06T07:28:36.811802+0000","last_active":"2024-03-06T07:28:39.803298+0000","last_peered":"2024-03-06T07:28:39.803298+0000","last_clean":"2024-03-06T07:28:39.803298+0000","last_became_active":"2024-03-06T07:28:36.811529+0000","last_became_peered":"2024-03-06T07:28:36.811529+0000","last_unstale":"2024-03-06T07:28:39.803298+0000","last_undegraded":"2024-03-06T07:28:39.803298+0000","last_fullsized":"2024-03-06T07:28:39.803298+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801398+0000","last_change":"2024-03-06T07:28:37.824620+0000","last_active":"2024-03-06T07:28:38.801398+0000","last_peered":"2024-03-06T07:28:38.801398+0000","last_clean":"2024-03-06T07:28:38.801398+0000","last_became_active":"2024-03-06T07:28:37.824374+0000","last_became_peered":"2024-03-06T07:28:37.824374+0000","last_unstale":"2024-03-06T07:28:38.801398+0000","last_undegraded":"2024-03-06T07:28:38.801398+0000","last_fullsized":"2024-03-06T07:28:38.801398+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:07.051996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801332+0000","last_change":"2024-03-06T07:28:36.811162+0000","last_active":"2024-03-06T07:28:38.801332+0000","last_peered":"2024-03-06T07:28:38.801332+0000","last_clean":"2024-03-06T07:28:38.801332+0000","last_became_active":"2024-03-06T07:28:36.810868+0000","last_became_peered":"2024-03-06T07:28:36.810868+0000","last_unstale":"2024-03-06T07:28:38.801332+0000","last_undegraded":"2024-03-06T07:28:38.801332+0000","last_fullsized":"2024-03-06T07:28:38.801332+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:02.364218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.804395+0000","last_change":"2024-03-06T07:28:36.804395+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.804395+0000","last_undegraded":"2024-03-06T07:28:36.804395+0000","last_fullsized":"2024-03-06T07:28:36.804395+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569692+0000","last_change":"2024-03-06T07:28:36.807342+0000","last_active":"2024-03-06T07:28:39.569692+0000","last_peered":"2024-03-06T07:28:39.569692+0000","last_clean":"2024-03-06T07:28:39.569692+0000","last_became_active":"2024-03-06T07:28:36.807010+0000","last_became_peered":"2024-03-06T07:28:36.807010+0000","last_unstale":"2024-03-06T07:28:39.569692+0000","last_undegraded":"2024-03-06T07:28:39.569692+0000","last_fullsized":"2024-03-06T07:28:39.569692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569607+0000","last_change":"2024-03-06T07:28:37.821338+0000","last_active":"2024-03-06T07:28:39.569607+0000","last_peered":"2024-03-06T07:28:39.569607+0000","last_clean":"2024-03-06T07:28:39.569607+0000","last_became_active":"2024-03-06T07:28:37.821053+0000","last_became_peered":"2024-03-06T07:28:37.821053+0000","last_unstale":"2024-03-06T07:28:39.569607+0000","last_undegraded":"2024-03-06T07:28:39.569607+0000","last_fullsized":"2024-03-06T07:28:39.569607+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:30.395699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800832+0000","last_change":"2024-03-06T07:28:36.812151+0000","last_active":"2024-03-06T07:28:38.800832+0000","last_peered":"2024-03-06T07:28:38.800832+0000","last_clean":"2024-03-06T07:28:38.800832+0000","last_became_active":"2024-03-06T07:28:36.811892+0000","last_became_peered":"2024-03-06T07:28:36.811892+0000","last_unstale":"2024-03-06T07:28:38.800832+0000","last_undegraded":"2024-03-06T07:28:38.800832+0000","last_fullsized":"2024-03-06T07:28:38.800832+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:03.969318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801527+0000","last_change":"2024-03-06T07:28:37.823826+0000","last_active":"2024-03-06T07:28:39.801527+0000","last_peered":"2024-03-06T07:28:39.801527+0000","last_clean":"2024-03-06T07:28:39.801527+0000","last_became_active":"2024-03-06T07:28:37.823542+0000","last_became_peered":"2024-03-06T07:28:37.823542+0000","last_unstale":"2024-03-06T07:28:39.801527+0000","last_undegraded":"2024-03-06T07:28:39.801527+0000","last_fullsized":"2024-03-06T07:28:39.801527+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:33.493872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"126'3","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569336+0000","last_change":"2024-03-06T07:28:36.811322+0000","last_active":"2024-03-06T07:28:39.569336+0000","last_peered":"2024-03-06T07:28:39.569336+0000","last_clean":"2024-03-06T07:28:39.569336+0000","last_became_active":"2024-03-06T07:28:36.810594+0000","last_became_peered":"2024-03-06T07:28:36.810594+0000","last_unstale":"2024-03-06T07:28:39.569336+0000","last_undegraded":"2024-03-06T07:28:39.569336+0000","last_fullsized":"2024-03-06T07:28:39.569336+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801460+0000","last_change":"2024-03-06T07:28:37.807298+0000","last_active":"2024-03-06T07:28:39.801460+0000","last_peered":"2024-03-06T07:28:39.801460+0000","last_clean":"2024-03-06T07:28:39.801460+0000","last_became_active":"2024-03-06T07:28:37.807020+0000","last_became_peered":"2024-03-06T07:28:37.807020+0000","last_unstale":"2024-03-06T07:28:39.801460+0000","last_undegraded":"2024-03-06T07:28:39.801460+0000","last_fullsized":"2024-03-06T07:28:39.801460+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:44.220669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"126'6","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800205+0000","last_change":"2024-03-06T07:28:36.808354+0000","last_active":"2024-03-06T07:28:38.800205+0000","last_peered":"2024-03-06T07:28:38.800205+0000","last_clean":"2024-03-06T07:28:38.800205+0000","last_became_active":"2024-03-06T07:28:36.808076+0000","last_became_peered":"2024-03-06T07:28:36.808076+0000","last_unstale":"2024-03-06T07:28:38.800205+0000","last_undegraded":"2024-03-06T07:28:38.800205+0000","last_fullsized":"2024-03-06T07:28:38.800205+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.803359+0000","last_change":"2024-03-06T07:28:36.803359+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.803359+0000","last_undegraded":"2024-03-06T07:28:36.803359+0000","last_fullsized":"2024-03-06T07:28:36.803359+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.813932+0000","last_change":"2024-03-06T07:28:36.810704+0000","last_active":"2024-03-06T07:28:36.813932+0000","last_peered":"2024-03-06T07:28:36.813932+0000","last_clean":"2024-03-06T07:28:36.813932+0000","last_became_active":"2024-03-06T07:28:36.809890+0000","last_became_peered":"2024-03-06T07:28:36.809890+0000","last_unstale":"2024-03-06T07:28:36.813932+0000","last_undegraded":"2024-03-06T07:28:36.813932+0000","last_fullsized":"2024-03-06T07:28:36.813932+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:43:03.819520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801239+0000","last_change":"2024-03-06T07:28:37.824548+0000","last_active":"2024-03-06T07:28:38.801239+0000","last_peered":"2024-03-06T07:28:38.801239+0000","last_clean":"2024-03-06T07:28:38.801239+0000","last_became_active":"2024-03-06T07:28:37.824290+0000","last_became_peered":"2024-03-06T07:28:37.824290+0000","last_unstale":"2024-03-06T07:28:38.801239+0000","last_undegraded":"2024-03-06T07:28:38.801239+0000","last_fullsized":"2024-03-06T07:28:38.801239+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:51.117603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801508+0000","last_change":"2024-03-06T07:28:36.810746+0000","last_active":"2024-03-06T07:28:38.801508+0000","last_peered":"2024-03-06T07:28:38.801508+0000","last_clean":"2024-03-06T07:28:38.801508+0000","last_became_active":"2024-03-06T07:28:36.810475+0000","last_became_peered":"2024-03-06T07:28:36.810475+0000","last_unstale":"2024-03-06T07:28:38.801508+0000","last_undegraded":"2024-03-06T07:28:38.801508+0000","last_fullsized":"2024-03-06T07:28:38.801508+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:27.539843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800380+0000","last_change":"2024-03-06T07:28:37.810787+0000","last_active":"2024-03-06T07:28:38.800380+0000","last_peered":"2024-03-06T07:28:38.800380+0000","last_clean":"2024-03-06T07:28:38.800380+0000","last_became_active":"2024-03-06T07:28:37.810266+0000","last_became_peered":"2024-03-06T07:28:37.810266+0000","last_unstale":"2024-03-06T07:28:38.800380+0000","last_undegraded":"2024-03-06T07:28:38.800380+0000","last_fullsized":"2024-03-06T07:28:38.800380+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:43.391321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"126'8","reported_seq":22,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800628+0000","last_change":"2024-03-06T07:28:36.809374+0000","last_active":"2024-03-06T07:28:38.800628+0000","last_peered":"2024-03-06T07:28:38.800628+0000","last_clean":"2024-03-06T07:28:38.800628+0000","last_became_active":"2024-03-06T07:28:36.807635+0000","last_became_peered":"2024-03-06T07:28:36.807635+0000","last_unstale":"2024-03-06T07:28:38.800628+0000","last_undegraded":"2024-03-06T07:28:38.800628+0000","last_fullsized":"2024-03-06T07:28:38.800628+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800569+0000","last_change":"2024-03-06T07:28:37.805400+0000","last_active":"2024-03-06T07:28:38.800569+0000","last_peered":"2024-03-06T07:28:38.800569+0000","last_clean":"2024-03-06T07:28:38.800569+0000","last_became_active":"2024-03-06T07:28:37.804321+0000","last_became_peered":"2024-03-06T07:28:37.804321+0000","last_unstale":"2024-03-06T07:28:38.800569+0000","last_undegraded":"2024-03-06T07:28:38.800569+0000","last_fullsized":"2024-03-06T07:28:38.800569+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:43.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569735+0000","last_change":"2024-03-06T07:28:36.814194+0000","last_active":"2024-03-06T07:28:39.569735+0000","last_peered":"2024-03-06T07:28:39.569735+0000","last_clean":"2024-03-06T07:28:39.569735+0000","last_became_active":"2024-03-06T07:28:36.813872+0000","last_became_peered":"2024-03-06T07:28:36.813872+0000","last_unstale":"2024-03-06T07:28:39.569735+0000","last_undegraded":"2024-03-06T07:28:39.569735+0000","last_fullsized":"2024-03-06T07:28:39.569735+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569606+0000","last_change":"2024-03-06T07:28:37.812580+0000","last_active":"2024-03-06T07:28:39.569606+0000","last_peered":"2024-03-06T07:28:39.569606+0000","last_clean":"2024-03-06T07:28:39.569606+0000","last_became_active":"2024-03-06T07:28:37.812301+0000","last_became_peered":"2024-03-06T07:28:37.812301+0000","last_unstale":"2024-03-06T07:28:39.569606+0000","last_undegraded":"2024-03-06T07:28:39.569606+0000","last_fullsized":"2024-03-06T07:28:39.569606+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:43.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998453+0000","last_change":"2024-03-06T07:28:36.808348+0000","last_active":"2024-03-06T07:28:38.998453+0000","last_peered":"2024-03-06T07:28:38.998453+0000","last_clean":"2024-03-06T07:28:38.998453+0000","last_became_active":"2024-03-06T07:28:36.808035+0000","last_became_peered":"2024-03-06T07:28:36.808035+0000","last_unstale":"2024-03-06T07:28:38.998453+0000","last_undegraded":"2024-03-06T07:28:38.998453+0000","last_fullsized":"2024-03-06T07:28:38.998453+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:19.839068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800965+0000","last_change":"2024-03-06T07:28:37.821013+0000","last_active":"2024-03-06T07:28:38.800965+0000","last_peered":"2024-03-06T07:28:38.800965+0000","last_clean":"2024-03-06T07:28:38.800965+0000","last_became_active":"2024-03-06T07:28:37.820756+0000","last_became_peered":"2024-03-06T07:28:37.820756+0000","last_unstale":"2024-03-06T07:28:38.800965+0000","last_undegraded":"2024-03-06T07:28:38.800965+0000","last_fullsized":"2024-03-06T07:28:38.800965+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:36.861383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803712+0000","last_change":"2024-03-06T07:28:36.817966+0000","last_active":"2024-03-06T07:28:39.803712+0000","last_peered":"2024-03-06T07:28:39.803712+0000","last_clean":"2024-03-06T07:28:39.803712+0000","last_became_active":"2024-03-06T07:28:36.817630+0000","last_became_peered":"2024-03-06T07:28:36.817630+0000","last_unstale":"2024-03-06T07:28:39.803712+0000","last_undegraded":"2024-03-06T07:28:39.803712+0000","last_fullsized":"2024-03-06T07:28:39.803712+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801709+0000","last_change":"2024-03-06T07:28:37.825233+0000","last_active":"2024-03-06T07:28:38.801709+0000","last_peered":"2024-03-06T07:28:38.801709+0000","last_clean":"2024-03-06T07:28:38.801709+0000","last_became_active":"2024-03-06T07:28:37.824969+0000","last_became_peered":"2024-03-06T07:28:37.824969+0000","last_unstale":"2024-03-06T07:28:38.801709+0000","last_undegraded":"2024-03-06T07:28:38.801709+0000","last_fullsized":"2024-03-06T07:28:38.801709+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:53.262994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569417+0000","last_change":"2024-03-06T07:28:36.809671+0000","last_active":"2024-03-06T07:28:39.569417+0000","last_peered":"2024-03-06T07:28:39.569417+0000","last_clean":"2024-03-06T07:28:39.569417+0000","last_became_active":"2024-03-06T07:28:36.809405+0000","last_became_peered":"2024-03-06T07:28:36.809405+0000","last_unstale":"2024-03-06T07:28:39.569417+0000","last_undegraded":"2024-03-06T07:28:39.569417+0000","last_fullsized":"2024-03-06T07:28:39.569417+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802926+0000","last_change":"2024-03-06T07:28:36.802926+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802926+0000","last_undegraded":"2024-03-06T07:28:36.802926+0000","last_fullsized":"2024-03-06T07:28:36.802926+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569339+0000","last_change":"2024-03-06T07:28:36.807565+0000","last_active":"2024-03-06T07:28:39.569339+0000","last_peered":"2024-03-06T07:28:39.569339+0000","last_clean":"2024-03-06T07:28:39.569339+0000","last_became_active":"2024-03-06T07:28:36.807266+0000","last_became_peered":"2024-03-06T07:28:36.807266+0000","last_unstale":"2024-03-06T07:28:39.569339+0000","last_undegraded":"2024-03-06T07:28:39.569339+0000","last_fullsized":"2024-03-06T07:28:39.569339+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803855+0000","last_change":"2024-03-06T07:28:37.818446+0000","last_active":"2024-03-06T07:28:39.803855+0000","last_peered":"2024-03-06T07:28:39.803855+0000","last_clean":"2024-03-06T07:28:39.803855+0000","last_became_active":"2024-03-06T07:28:37.818122+0000","last_became_peered":"2024-03-06T07:28:37.818122+0000","last_unstale":"2024-03-06T07:28:39.803855+0000","last_undegraded":"2024-03-06T07:28:39.803855+0000","last_fullsized":"2024-03-06T07:28:39.803855+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:56.972667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800569+0000","last_change":"2024-03-06T07:28:36.804975+0000","last_active":"2024-03-06T07:28:38.800569+0000","last_peered":"2024-03-06T07:28:38.800569+0000","last_clean":"2024-03-06T07:28:38.800569+0000","last_became_active":"2024-03-06T07:28:36.804697+0000","last_became_peered":"2024-03-06T07:28:36.804697+0000","last_unstale":"2024-03-06T07:28:38.800569+0000","last_undegraded":"2024-03-06T07:28:38.800569+0000","last_fullsized":"2024-03-06T07:28:38.800569+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801249+0000","last_change":"2024-03-06T07:28:37.817957+0000","last_active":"2024-03-06T07:28:38.801249+0000","last_peered":"2024-03-06T07:28:38.801249+0000","last_clean":"2024-03-06T07:28:38.801249+0000","last_became_active":"2024-03-06T07:28:37.817689+0000","last_became_peered":"2024-03-06T07:28:37.817689+0000","last_unstale":"2024-03-06T07:28:38.801249+0000","last_undegraded":"2024-03-06T07:28:38.801249+0000","last_fullsized":"2024-03-06T07:28:38.801249+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:20.967736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800295+0000","last_change":"2024-03-06T07:28:36.810758+0000","last_active":"2024-03-06T07:28:38.800295+0000","last_peered":"2024-03-06T07:28:38.800295+0000","last_clean":"2024-03-06T07:28:38.800295+0000","last_became_active":"2024-03-06T07:28:36.810492+0000","last_became_peered":"2024-03-06T07:28:36.810492+0000","last_unstale":"2024-03-06T07:28:38.800295+0000","last_undegraded":"2024-03-06T07:28:38.800295+0000","last_fullsized":"2024-03-06T07:28:38.800295+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:52.260988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801251+0000","last_change":"2024-03-06T07:28:37.808444+0000","last_active":"2024-03-06T07:28:39.801251+0000","last_peered":"2024-03-06T07:28:39.801251+0000","last_clean":"2024-03-06T07:28:39.801251+0000","last_became_active":"2024-03-06T07:28:37.808181+0000","last_became_peered":"2024-03-06T07:28:37.808181+0000","last_unstale":"2024-03-06T07:28:39.801251+0000","last_undegraded":"2024-03-06T07:28:39.801251+0000","last_fullsized":"2024-03-06T07:28:39.801251+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:54.190073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998218+0000","last_change":"2024-03-06T07:28:36.809882+0000","last_active":"2024-03-06T07:28:38.998218+0000","last_peered":"2024-03-06T07:28:38.998218+0000","last_clean":"2024-03-06T07:28:38.998218+0000","last_became_active":"2024-03-06T07:28:36.809035+0000","last_became_peered":"2024-03-06T07:28:36.809035+0000","last_unstale":"2024-03-06T07:28:38.998218+0000","last_undegraded":"2024-03-06T07:28:38.998218+0000","last_fullsized":"2024-03-06T07:28:38.998218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:47.288248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803906+0000","last_change":"2024-03-06T07:28:37.813974+0000","last_active":"2024-03-06T07:28:39.803906+0000","last_peered":"2024-03-06T07:28:39.803906+0000","last_clean":"2024-03-06T07:28:39.803906+0000","last_became_active":"2024-03-06T07:28:37.813672+0000","last_became_peered":"2024-03-06T07:28:37.813672+0000","last_unstale":"2024-03-06T07:28:39.803906+0000","last_undegraded":"2024-03-06T07:28:39.803906+0000","last_fullsized":"2024-03-06T07:28:39.803906+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:33.184108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-06T07:28:36.816568+0000","last_change":"2024-03-06T07:28:36.812497+0000","last_active":"2024-03-06T07:28:36.816568+0000","last_peered":"2024-03-06T07:28:36.816568+0000","last_clean":"2024-03-06T07:28:36.816568+0000","last_became_active":"2024-03-06T07:28:36.812253+0000","last_became_peered":"2024-03-06T07:28:36.812253+0000","last_unstale":"2024-03-06T07:28:36.816568+0000","last_undegraded":"2024-03-06T07:28:36.816568+0000","last_fullsized":"2024-03-06T07:28:36.816568+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:22:33.721283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":1,"reported_epoch":125,"state":"creating+peering","last_fresh":"2024-03-06T07:28:36.802330+0000","last_change":"2024-03-06T07:28:36.802330+0000","last_active":"2024-03-06T07:28:36.789256+0000","last_peered":"2024-03-06T07:28:36.789256+0000","last_clean":"2024-03-06T07:28:36.789256+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-06T07:28:36.802330+0000","last_undegraded":"2024-03-06T07:28:36.802330+0000","last_fullsized":"2024-03-06T07:28:36.802330+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"127'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801489+0000","last_change":"2024-03-06T07:28:36.812931+0000","last_active":"2024-03-06T07:28:39.801489+0000","last_peered":"2024-03-06T07:28:39.801489+0000","last_clean":"2024-03-06T07:28:39.801489+0000","last_became_active":"2024-03-06T07:28:36.812619+0000","last_became_peered":"2024-03-06T07:28:36.812619+0000","last_unstale":"2024-03-06T07:28:39.801489+0000","last_undegraded":"2024-03-06T07:28:39.801489+0000","last_fullsized":"2024-03-06T07:28:39.801489+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:52.260988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801521+0000","last_change":"2024-03-06T07:28:37.821218+0000","last_active":"2024-03-06T07:28:38.801521+0000","last_peered":"2024-03-06T07:28:38.801521+0000","last_clean":"2024-03-06T07:28:38.801521+0000","last_became_active":"2024-03-06T07:28:37.820515+0000","last_became_peered":"2024-03-06T07:28:37.820515+0000","last_unstale":"2024-03-06T07:28:38.801521+0000","last_undegraded":"2024-03-06T07:28:38.801521+0000","last_fullsized":"2024-03-06T07:28:38.801521+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:38.940310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801671+0000","last_change":"2024-03-06T07:28:36.811623+0000","last_active":"2024-03-06T07:28:38.801671+0000","last_peered":"2024-03-06T07:28:38.801671+0000","last_clean":"2024-03-06T07:28:38.801671+0000","last_became_active":"2024-03-06T07:28:36.811361+0000","last_became_peered":"2024-03-06T07:28:36.811361+0000","last_unstale":"2024-03-06T07:28:38.801671+0000","last_undegraded":"2024-03-06T07:28:38.801671+0000","last_fullsized":"2024-03-06T07:28:38.801671+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:25.324827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803746+0000","last_change":"2024-03-06T07:28:37.818590+0000","last_active":"2024-03-06T07:28:39.803746+0000","last_peered":"2024-03-06T07:28:39.803746+0000","last_clean":"2024-03-06T07:28:39.803746+0000","last_became_active":"2024-03-06T07:28:37.818314+0000","last_became_peered":"2024-03-06T07:28:37.818314+0000","last_unstale":"2024-03-06T07:28:39.803746+0000","last_undegraded":"2024-03-06T07:28:39.803746+0000","last_fullsized":"2024-03-06T07:28:39.803746+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:43.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998524+0000","last_change":"2024-03-06T07:28:36.807851+0000","last_active":"2024-03-06T07:28:38.998524+0000","last_peered":"2024-03-06T07:28:38.998524+0000","last_clean":"2024-03-06T07:28:38.998524+0000","last_became_active":"2024-03-06T07:28:36.807590+0000","last_became_peered":"2024-03-06T07:28:36.807590+0000","last_unstale":"2024-03-06T07:28:38.998524+0000","last_undegraded":"2024-03-06T07:28:38.998524+0000","last_fullsized":"2024-03-06T07:28:38.998524+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:09.184127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801240+0000","last_change":"2024-03-06T07:28:37.810622+0000","last_active":"2024-03-06T07:28:39.801240+0000","last_peered":"2024-03-06T07:28:39.801240+0000","last_clean":"2024-03-06T07:28:39.801240+0000","last_became_active":"2024-03-06T07:28:37.810366+0000","last_became_peered":"2024-03-06T07:28:37.810366+0000","last_unstale":"2024-03-06T07:28:39.801240+0000","last_undegraded":"2024-03-06T07:28:39.801240+0000","last_fullsized":"2024-03-06T07:28:39.801240+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:24.104089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801547+0000","last_change":"2024-03-06T07:28:36.808681+0000","last_active":"2024-03-06T07:28:38.801547+0000","last_peered":"2024-03-06T07:28:38.801547+0000","last_clean":"2024-03-06T07:28:38.801547+0000","last_became_active":"2024-03-06T07:28:36.808407+0000","last_became_peered":"2024-03-06T07:28:36.808407+0000","last_unstale":"2024-03-06T07:28:38.801547+0000","last_undegraded":"2024-03-06T07:28:38.801547+0000","last_fullsized":"2024-03-06T07:28:38.801547+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:09.820534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569407+0000","last_change":"2024-03-06T07:28:37.821485+0000","last_active":"2024-03-06T07:28:39.569407+0000","last_peered":"2024-03-06T07:28:39.569407+0000","last_clean":"2024-03-06T07:28:39.569407+0000","last_became_active":"2024-03-06T07:28:37.821170+0000","last_became_peered":"2024-03-06T07:28:37.821170+0000","last_unstale":"2024-03-06T07:28:39.569407+0000","last_undegraded":"2024-03-06T07:28:39.569407+0000","last_fullsized":"2024-03-06T07:28:39.569407+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:21.866852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801533+0000","last_change":"2024-03-06T07:28:36.812531+0000","last_active":"2024-03-06T07:28:39.801533+0000","last_peered":"2024-03-06T07:28:39.801533+0000","last_clean":"2024-03-06T07:28:39.801533+0000","last_became_active":"2024-03-06T07:28:36.812195+0000","last_became_peered":"2024-03-06T07:28:36.812195+0000","last_unstale":"2024-03-06T07:28:39.801533+0000","last_undegraded":"2024-03-06T07:28:39.801533+0000","last_fullsized":"2024-03-06T07:28:39.801533+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:03.969318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800475+0000","last_change":"2024-03-06T07:28:37.822792+0000","last_active":"2024-03-06T07:28:38.800475+0000","last_peered":"2024-03-06T07:28:38.800475+0000","last_clean":"2024-03-06T07:28:38.800475+0000","last_became_active":"2024-03-06T07:28:37.822452+0000","last_became_peered":"2024-03-06T07:28:37.822452+0000","last_unstale":"2024-03-06T07:28:38.800475+0000","last_undegraded":"2024-03-06T07:28:38.800475+0000","last_fullsized":"2024-03-06T07:28:38.800475+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:33.184108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800396+0000","last_change":"2024-03-06T07:28:36.811478+0000","last_active":"2024-03-06T07:28:38.800396+0000","last_peered":"2024-03-06T07:28:38.800396+0000","last_clean":"2024-03-06T07:28:38.800396+0000","last_became_active":"2024-03-06T07:28:36.811250+0000","last_became_peered":"2024-03-06T07:28:36.811250+0000","last_unstale":"2024-03-06T07:28:38.800396+0000","last_undegraded":"2024-03-06T07:28:38.800396+0000","last_fullsized":"2024-03-06T07:28:38.800396+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:43.695281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801296+0000","last_change":"2024-03-06T07:28:37.823153+0000","last_active":"2024-03-06T07:28:38.801296+0000","last_peered":"2024-03-06T07:28:38.801296+0000","last_clean":"2024-03-06T07:28:38.801296+0000","last_became_active":"2024-03-06T07:28:37.822205+0000","last_became_peered":"2024-03-06T07:28:37.822205+0000","last_unstale":"2024-03-06T07:28:38.801296+0000","last_undegraded":"2024-03-06T07:28:38.801296+0000","last_fullsized":"2024-03-06T07:28:38.801296+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:47.073356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998338+0000","last_change":"2024-03-06T07:28:36.809706+0000","last_active":"2024-03-06T07:28:38.998338+0000","last_peered":"2024-03-06T07:28:38.998338+0000","last_clean":"2024-03-06T07:28:38.998338+0000","last_became_active":"2024-03-06T07:28:36.809345+0000","last_became_peered":"2024-03-06T07:28:36.809345+0000","last_unstale":"2024-03-06T07:28:38.998338+0000","last_undegraded":"2024-03-06T07:28:38.998338+0000","last_fullsized":"2024-03-06T07:28:38.998338+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:00.510053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801164+0000","last_change":"2024-03-06T07:28:37.810948+0000","last_active":"2024-03-06T07:28:39.801164+0000","last_peered":"2024-03-06T07:28:39.801164+0000","last_clean":"2024-03-06T07:28:39.801164+0000","last_became_active":"2024-03-06T07:28:37.810064+0000","last_became_peered":"2024-03-06T07:28:37.810064+0000","last_unstale":"2024-03-06T07:28:39.801164+0000","last_undegraded":"2024-03-06T07:28:39.801164+0000","last_fullsized":"2024-03-06T07:28:39.801164+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:21.702361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801501+0000","last_change":"2024-03-06T07:28:36.812624+0000","last_active":"2024-03-06T07:28:39.801501+0000","last_peered":"2024-03-06T07:28:39.801501+0000","last_clean":"2024-03-06T07:28:39.801501+0000","last_became_active":"2024-03-06T07:28:36.812331+0000","last_became_peered":"2024-03-06T07:28:36.812331+0000","last_unstale":"2024-03-06T07:28:39.801501+0000","last_undegraded":"2024-03-06T07:28:39.801501+0000","last_fullsized":"2024-03-06T07:28:39.801501+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:58.094644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801001+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:28:39.801001+0000","last_peered":"2024-03-06T07:28:39.801001+0000","last_clean":"2024-03-06T07:28:39.801001+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:28:39.801001+0000","last_undegraded":"2024-03-06T07:28:39.801001+0000","last_fullsized":"2024-03-06T07:28:39.801001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2203648,"data_stored":1285483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1597440,"data_stored":694477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574936,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":692977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41052,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330788,"statfs":{"total":96636764160,"available":96594726912,"internally_reserved":0,"allocated":1589248,"data_stored":692935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41064,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330776,"statfs":{"total":96636764160,"available":96594714624,"internally_reserved":0,"allocated":1593344,"data_stored":693025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41016,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330824,"statfs":{"total":96636764160,"available":96594763776,"internally_reserved":0,"allocated":1548288,"data_stored":673515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1585152,"data_stored":692865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41628,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330212,"statfs":{"total":96636764160,"available":96594137088,"internally_reserved":0,"allocated":2179072,"data_stored":1283233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:43.785 INFO:tasks.ceph.mds.a.smithi072.stderr:2024-03-06T07:28:43.783+0000 7fd7429fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-06T07:28:45.343 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:28:45.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.443+0000 7fe216ff7640 1 Processor -- start 2024-03-06T07:28:45.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.443+0000 7fe216ff7640 1 -- start start 2024-03-06T07:28:45.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe216ff7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2100d22b0 0x7fe2100d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe216ff7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100b3430 con 0x7fe21015dfe0 2024-03-06T07:28:45.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe216ff7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2100b3710 con 0x7fe2100b41f0 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe216ff7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2100d2c50 con 0x7fe2100d22b0 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe20ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2100d22b0 0x7fe2100d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe214d93640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe21015dfe0 0x7fe2100cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58042/0 (socket says 172.21.15.72:58042) 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe214d93640 1 -- 172.21.15.72:0/1927338103 learned_addr learned my addr 172.21.15.72:0/1927338103 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe20ffff640 1 -- 172.21.15.72:0/1927338103 >> v1:172.21.15.146:6789/0 conn(0x7fe2100b41f0 legacy=0x7fe2100b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe20ffff640 1 -- 172.21.15.72:0/1927338103 >> v1:172.21.15.72:6789/0 conn(0x7fe21015dfe0 legacy=0x7fe2100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe20ffff640 1 -- 172.21.15.72:0/1927338103 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2100b39f0 con 0x7fe2100d22b0 2024-03-06T07:28:45.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.444+0000 7fe20f7fe640 1 -- 172.21.15.72:0/1927338103 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163150061 0 0) 0x7fe2100b3430 con 0x7fe21015dfe0 2024-03-06T07:28:45.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.445+0000 7fe20ffff640 1 --2- 172.21.15.72:0/1927338103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2100d22b0 0x7fe2100d2710 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fe2040097d0 tx=0x7fe20402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67320e92722ceee server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.445+0000 7fe20f7fe640 1 -- 172.21.15.72:0/1927338103 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204035020 con 0x7fe2100d22b0 2024-03-06T07:28:45.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.445+0000 7fe20f7fe640 1 -- 172.21.15.72:0/1927338103 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe204031830 con 0x7fe2100d22b0 2024-03-06T07:28:45.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.445+0000 7fe20f7fe640 1 -- 172.21.15.72:0/1927338103 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204031c00 con 0x7fe2100d22b0 2024-03-06T07:28:45.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 -- 172.21.15.72:0/1927338103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2100d22b0 msgr2=0x7fe2100d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1927338103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2100d22b0 0x7fe2100d2710 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fe2040097d0 tx=0x7fe20402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 -- 172.21.15.72:0/1927338103 shutdown_connections 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1927338103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2100d22b0 0x7fe2100d2710 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 -- 172.21.15.72:0/1927338103 >> 172.21.15.72:0/1927338103 conn(0x7fe2100d3cd0 msgr2=0x7fe2100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 -- 172.21.15.72:0/1927338103 shutdown_connections 2024-03-06T07:28:45.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.446+0000 7fe216ff7640 1 -- 172.21.15.72:0/1927338103 wait complete. 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.447+0000 7fe216ff7640 1 Processor -- start 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.447+0000 7fe216ff7640 1 -- start start 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe216ff7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2100b41f0 0x7fe2101f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe216ff7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 0x7fe2100cc740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe216ff7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe21015e750 0x7fe2100b6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe216ff7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100b3d60 con 0x7fe21015dfe0 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe216ff7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100b31e0 con 0x7fe2100b41f0 2024-03-06T07:28:45.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe216ff7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2100d1c70 con 0x7fe21015e750 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 0x7fe2100cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 0x7fe2100cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34876/0 (socket says 172.21.15.72:34876) 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 -- 172.21.15.72:0/1685492264 learned_addr learned my addr 172.21.15.72:0/1685492264 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe214d93640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2100b41f0 0x7fe2101f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe215594640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe21015e750 0x7fe2100b6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 -- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe21015e750 msgr2=0x7fe2100b6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe21015e750 0x7fe2100b6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 -- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2100b41f0 msgr2=0x7fe2101f38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2100b41f0 0x7fe2101f38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.448+0000 7fe20ffff640 1 -- 172.21.15.72:0/1685492264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2101f4180 con 0x7fe21015dfe0 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.449+0000 7fe20ffff640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 0x7fe2100cc740 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe20402be60 tx=0x7fe2040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.449+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20403b9c0 con 0x7fe21015dfe0 2024-03-06T07:28:45.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.449+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20403bb60 con 0x7fe21015dfe0 2024-03-06T07:28:45.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.449+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2040453f0 con 0x7fe21015dfe0 2024-03-06T07:28:45.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.449+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2101f4460 con 0x7fe21015dfe0 2024-03-06T07:28:45.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.449+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2101f3df0 con 0x7fe21015dfe0 2024-03-06T07:28:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.450+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2100b68a0 con 0x7fe21015dfe0 2024-03-06T07:28:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.451+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe20404c060 con 0x7fe21015dfe0 2024-03-06T07:28:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.451+0000 7fe20d7fa640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.451+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2040abf00 con 0x7fe21015dfe0 2024-03-06T07:28:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.451+0000 7fe214d93640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.452+0000 7fe214d93640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe208007ed0 tx=0x7fe208007e60 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.455+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe204040070 con 0x7fe21015dfe0 2024-03-06T07:28:45.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.629+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2100ca3a0 con 0x7fe1fc06b4d0 2024-03-06T07:28:45.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.635+0000 7fe20d7fa640 1 -- 172.21.15.72:0/1685492264 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252483 (secure 0 0 0) 0x7fe2100ca3a0 con 0x7fe1fc06b4d0 2024-03-06T07:28:45.639 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:45.640 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1fc06b4d0 msgr2=0x7fe1fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fe208007ed0 tx=0x7fe208007e60 comp rx=0 tx=0).stop 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 msgr2=0x7fe2100cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 0x7fe2100cc740 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fe20402be60 tx=0x7fe2040046c0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 shutdown_connections 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe1fc06b4d0 0x7fe1fc06d990 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe21015e750 0x7fe2100b6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21015dfe0 0x7fe2100cc740 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 --2- 172.21.15.72:0/1685492264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe2100b41f0 0x7fe2101f38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 >> 172.21.15.72:0/1685492264 conn(0x7fe2100d3cd0 msgr2=0x7fe21006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 shutdown_connections 2024-03-06T07:28:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.638+0000 7fe216ff7640 1 -- 172.21.15.72:0/1685492264 wait complete. 2024-03-06T07:28:45.651 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-03-06T07:28:44.024149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":182,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":329632,"kb_used_data":13608,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754645088,"statfs":{"total":773094113280,"available":772756570112,"internally_reserved":0,"allocated":13934592,"data_stored":6728476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.228887"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392504+0000","last_change":"2024-03-06T07:28:37.809758+0000","last_active":"2024-03-06T07:28:40.392504+0000","last_peered":"2024-03-06T07:28:40.392504+0000","last_clean":"2024-03-06T07:28:40.392504+0000","last_became_active":"2024-03-06T07:28:37.809502+0000","last_became_peered":"2024-03-06T07:28:37.809502+0000","last_unstale":"2024-03-06T07:28:40.392504+0000","last_undegraded":"2024-03-06T07:28:40.392504+0000","last_fullsized":"2024-03-06T07:28:40.392504+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:05:42.546233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801488+0000","last_change":"2024-03-06T07:28:37.821388+0000","last_active":"2024-03-06T07:28:38.801488+0000","last_peered":"2024-03-06T07:28:38.801488+0000","last_clean":"2024-03-06T07:28:38.801488+0000","last_became_active":"2024-03-06T07:28:37.820763+0000","last_became_peered":"2024-03-06T07:28:37.820763+0000","last_unstale":"2024-03-06T07:28:38.801488+0000","last_undegraded":"2024-03-06T07:28:38.801488+0000","last_fullsized":"2024-03-06T07:28:38.801488+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:47.914373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800456+0000","last_change":"2024-03-06T07:28:37.807180+0000","last_active":"2024-03-06T07:28:38.800456+0000","last_peered":"2024-03-06T07:28:38.800456+0000","last_clean":"2024-03-06T07:28:38.800456+0000","last_became_active":"2024-03-06T07:28:37.806885+0000","last_became_peered":"2024-03-06T07:28:37.806885+0000","last_unstale":"2024-03-06T07:28:38.800456+0000","last_undegraded":"2024-03-06T07:28:38.800456+0000","last_fullsized":"2024-03-06T07:28:38.800456+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:30.395699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392296+0000","last_change":"2024-03-06T07:28:37.808028+0000","last_active":"2024-03-06T07:28:40.392296+0000","last_peered":"2024-03-06T07:28:40.392296+0000","last_clean":"2024-03-06T07:28:40.392296+0000","last_became_active":"2024-03-06T07:28:37.807667+0000","last_became_peered":"2024-03-06T07:28:37.807667+0000","last_unstale":"2024-03-06T07:28:40.392296+0000","last_undegraded":"2024-03-06T07:28:40.392296+0000","last_fullsized":"2024-03-06T07:28:40.392296+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:13:22.873536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801307+0000","last_change":"2024-03-06T07:28:37.813718+0000","last_active":"2024-03-06T07:28:39.801307+0000","last_peered":"2024-03-06T07:28:39.801307+0000","last_clean":"2024-03-06T07:28:39.801307+0000","last_became_active":"2024-03-06T07:28:37.813446+0000","last_became_peered":"2024-03-06T07:28:37.813446+0000","last_unstale":"2024-03-06T07:28:39.801307+0000","last_undegraded":"2024-03-06T07:28:39.801307+0000","last_fullsized":"2024-03-06T07:28:39.801307+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:50:24.990126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803550+0000","last_change":"2024-03-06T07:28:37.818530+0000","last_active":"2024-03-06T07:28:39.803550+0000","last_peered":"2024-03-06T07:28:39.803550+0000","last_clean":"2024-03-06T07:28:39.803550+0000","last_became_active":"2024-03-06T07:28:37.818198+0000","last_became_peered":"2024-03-06T07:28:37.818198+0000","last_unstale":"2024-03-06T07:28:39.803550+0000","last_undegraded":"2024-03-06T07:28:39.803550+0000","last_fullsized":"2024-03-06T07:28:39.803550+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.430067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803374+0000","last_change":"2024-03-06T07:28:37.812791+0000","last_active":"2024-03-06T07:28:39.803374+0000","last_peered":"2024-03-06T07:28:39.803374+0000","last_clean":"2024-03-06T07:28:39.803374+0000","last_became_active":"2024-03-06T07:28:37.812509+0000","last_became_peered":"2024-03-06T07:28:37.812509+0000","last_unstale":"2024-03-06T07:28:39.803374+0000","last_undegraded":"2024-03-06T07:28:39.803374+0000","last_fullsized":"2024-03-06T07:28:39.803374+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:43.391321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800451+0000","last_change":"2024-03-06T07:28:37.825862+0000","last_active":"2024-03-06T07:28:38.800451+0000","last_peered":"2024-03-06T07:28:38.800451+0000","last_clean":"2024-03-06T07:28:38.800451+0000","last_became_active":"2024-03-06T07:28:37.825352+0000","last_became_peered":"2024-03-06T07:28:37.825352+0000","last_unstale":"2024-03-06T07:28:38.800451+0000","last_undegraded":"2024-03-06T07:28:38.800451+0000","last_fullsized":"2024-03-06T07:28:38.800451+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:21.866852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800535+0000","last_change":"2024-03-06T07:28:37.810958+0000","last_active":"2024-03-06T07:28:38.800535+0000","last_peered":"2024-03-06T07:28:38.800535+0000","last_clean":"2024-03-06T07:28:38.800535+0000","last_became_active":"2024-03-06T07:28:37.809932+0000","last_became_peered":"2024-03-06T07:28:37.809932+0000","last_unstale":"2024-03-06T07:28:38.800535+0000","last_undegraded":"2024-03-06T07:28:38.800535+0000","last_fullsized":"2024-03-06T07:28:38.800535+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.430067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801698+0000","last_change":"2024-03-06T07:28:37.818998+0000","last_active":"2024-03-06T07:28:38.801698+0000","last_peered":"2024-03-06T07:28:38.801698+0000","last_clean":"2024-03-06T07:28:38.801698+0000","last_became_active":"2024-03-06T07:28:37.818743+0000","last_became_peered":"2024-03-06T07:28:37.818743+0000","last_unstale":"2024-03-06T07:28:38.801698+0000","last_undegraded":"2024-03-06T07:28:38.801698+0000","last_fullsized":"2024-03-06T07:28:38.801698+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:44.697288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998650+0000","last_change":"2024-03-06T07:28:37.818817+0000","last_active":"2024-03-06T07:28:38.998650+0000","last_peered":"2024-03-06T07:28:38.998650+0000","last_clean":"2024-03-06T07:28:38.998650+0000","last_became_active":"2024-03-06T07:28:37.818560+0000","last_became_peered":"2024-03-06T07:28:37.818560+0000","last_unstale":"2024-03-06T07:28:38.998650+0000","last_undegraded":"2024-03-06T07:28:38.998650+0000","last_fullsized":"2024-03-06T07:28:38.998650+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:05:37.951470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800217+0000","last_change":"2024-03-06T07:28:37.810389+0000","last_active":"2024-03-06T07:28:38.800217+0000","last_peered":"2024-03-06T07:28:38.800217+0000","last_clean":"2024-03-06T07:28:38.800217+0000","last_became_active":"2024-03-06T07:28:37.810109+0000","last_became_peered":"2024-03-06T07:28:37.810109+0000","last_unstale":"2024-03-06T07:28:38.800217+0000","last_undegraded":"2024-03-06T07:28:38.800217+0000","last_fullsized":"2024-03-06T07:28:38.800217+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:04.971325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392116+0000","last_change":"2024-03-06T07:28:37.810714+0000","last_active":"2024-03-06T07:28:40.392116+0000","last_peered":"2024-03-06T07:28:40.392116+0000","last_clean":"2024-03-06T07:28:40.392116+0000","last_became_active":"2024-03-06T07:28:37.810467+0000","last_became_peered":"2024-03-06T07:28:37.810467+0000","last_unstale":"2024-03-06T07:28:40.392116+0000","last_undegraded":"2024-03-06T07:28:40.392116+0000","last_fullsized":"2024-03-06T07:28:40.392116+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:56:26.142130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801383+0000","last_change":"2024-03-06T07:28:37.820392+0000","last_active":"2024-03-06T07:28:38.801383+0000","last_peered":"2024-03-06T07:28:38.801383+0000","last_clean":"2024-03-06T07:28:38.801383+0000","last_became_active":"2024-03-06T07:28:37.819941+0000","last_became_peered":"2024-03-06T07:28:37.819941+0000","last_unstale":"2024-03-06T07:28:38.801383+0000","last_undegraded":"2024-03-06T07:28:38.801383+0000","last_fullsized":"2024-03-06T07:28:38.801383+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:56.329155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.391971+0000","last_change":"2024-03-06T07:28:37.804862+0000","last_active":"2024-03-06T07:28:40.391971+0000","last_peered":"2024-03-06T07:28:40.391971+0000","last_clean":"2024-03-06T07:28:40.391971+0000","last_became_active":"2024-03-06T07:28:37.804583+0000","last_became_peered":"2024-03-06T07:28:37.804583+0000","last_unstale":"2024-03-06T07:28:40.391971+0000","last_undegraded":"2024-03-06T07:28:40.391971+0000","last_fullsized":"2024-03-06T07:28:40.391971+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:55:26.039470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800475+0000","last_change":"2024-03-06T07:28:37.820286+0000","last_active":"2024-03-06T07:28:38.800475+0000","last_peered":"2024-03-06T07:28:38.800475+0000","last_clean":"2024-03-06T07:28:38.800475+0000","last_became_active":"2024-03-06T07:28:37.820027+0000","last_became_peered":"2024-03-06T07:28:37.820027+0000","last_unstale":"2024-03-06T07:28:38.800475+0000","last_undegraded":"2024-03-06T07:28:38.800475+0000","last_fullsized":"2024-03-06T07:28:38.800475+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:00:19.569381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803800+0000","last_change":"2024-03-06T07:28:37.813885+0000","last_active":"2024-03-06T07:28:39.803800+0000","last_peered":"2024-03-06T07:28:39.803800+0000","last_clean":"2024-03-06T07:28:39.803800+0000","last_became_active":"2024-03-06T07:28:37.813622+0000","last_became_peered":"2024-03-06T07:28:37.813622+0000","last_unstale":"2024-03-06T07:28:39.803800+0000","last_undegraded":"2024-03-06T07:28:39.803800+0000","last_fullsized":"2024-03-06T07:28:39.803800+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:45.463350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800622+0000","last_change":"2024-03-06T07:28:37.821840+0000","last_active":"2024-03-06T07:28:38.800622+0000","last_peered":"2024-03-06T07:28:38.800622+0000","last_clean":"2024-03-06T07:28:38.800622+0000","last_became_active":"2024-03-06T07:28:37.821555+0000","last_became_peered":"2024-03-06T07:28:37.821555+0000","last_unstale":"2024-03-06T07:28:38.800622+0000","last_undegraded":"2024-03-06T07:28:38.800622+0000","last_fullsized":"2024-03-06T07:28:38.800622+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:17:50.757145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998635+0000","last_change":"2024-03-06T07:28:37.813180+0000","last_active":"2024-03-06T07:28:38.998635+0000","last_peered":"2024-03-06T07:28:38.998635+0000","last_clean":"2024-03-06T07:28:38.998635+0000","last_became_active":"2024-03-06T07:28:37.812934+0000","last_became_peered":"2024-03-06T07:28:37.812934+0000","last_unstale":"2024-03-06T07:28:38.998635+0000","last_undegraded":"2024-03-06T07:28:38.998635+0000","last_fullsized":"2024-03-06T07:28:38.998635+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:22.911067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800947+0000","last_change":"2024-03-06T07:28:37.817427+0000","last_active":"2024-03-06T07:28:38.800947+0000","last_peered":"2024-03-06T07:28:38.800947+0000","last_clean":"2024-03-06T07:28:38.800947+0000","last_became_active":"2024-03-06T07:28:37.817166+0000","last_became_peered":"2024-03-06T07:28:37.817166+0000","last_unstale":"2024-03-06T07:28:38.800947+0000","last_undegraded":"2024-03-06T07:28:38.800947+0000","last_fullsized":"2024-03-06T07:28:38.800947+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:54.190073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801280+0000","last_change":"2024-03-06T07:28:37.814034+0000","last_active":"2024-03-06T07:28:39.801280+0000","last_peered":"2024-03-06T07:28:39.801280+0000","last_clean":"2024-03-06T07:28:39.801280+0000","last_became_active":"2024-03-06T07:28:37.813747+0000","last_became_peered":"2024-03-06T07:28:37.813747+0000","last_unstale":"2024-03-06T07:28:39.801280+0000","last_undegraded":"2024-03-06T07:28:39.801280+0000","last_fullsized":"2024-03-06T07:28:39.801280+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:36.861383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998322+0000","last_change":"2024-03-06T07:28:37.818450+0000","last_active":"2024-03-06T07:28:38.998322+0000","last_peered":"2024-03-06T07:28:38.998322+0000","last_clean":"2024-03-06T07:28:38.998322+0000","last_became_active":"2024-03-06T07:28:37.818195+0000","last_became_peered":"2024-03-06T07:28:37.818195+0000","last_unstale":"2024-03-06T07:28:38.998322+0000","last_undegraded":"2024-03-06T07:28:38.998322+0000","last_fullsized":"2024-03-06T07:28:38.998322+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:48:40.275303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800257+0000","last_change":"2024-03-06T07:28:37.810549+0000","last_active":"2024-03-06T07:28:38.800257+0000","last_peered":"2024-03-06T07:28:38.800257+0000","last_clean":"2024-03-06T07:28:38.800257+0000","last_became_active":"2024-03-06T07:28:37.809880+0000","last_became_peered":"2024-03-06T07:28:37.809880+0000","last_unstale":"2024-03-06T07:28:38.800257+0000","last_undegraded":"2024-03-06T07:28:38.800257+0000","last_fullsized":"2024-03-06T07:28:38.800257+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:59.096651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998291+0000","last_change":"2024-03-06T07:28:37.812050+0000","last_active":"2024-03-06T07:28:38.998291+0000","last_peered":"2024-03-06T07:28:38.998291+0000","last_clean":"2024-03-06T07:28:38.998291+0000","last_became_active":"2024-03-06T07:28:37.811773+0000","last_became_peered":"2024-03-06T07:28:37.811773+0000","last_unstale":"2024-03-06T07:28:38.998291+0000","last_undegraded":"2024-03-06T07:28:38.998291+0000","last_fullsized":"2024-03-06T07:28:38.998291+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:09:37.879363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801643+0000","last_change":"2024-03-06T07:28:37.816377+0000","last_active":"2024-03-06T07:28:38.801643+0000","last_peered":"2024-03-06T07:28:38.801643+0000","last_clean":"2024-03-06T07:28:38.801643+0000","last_became_active":"2024-03-06T07:28:37.816100+0000","last_became_peered":"2024-03-06T07:28:37.816100+0000","last_unstale":"2024-03-06T07:28:38.801643+0000","last_undegraded":"2024-03-06T07:28:38.801643+0000","last_fullsized":"2024-03-06T07:28:38.801643+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:16.108047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800770+0000","last_change":"2024-03-06T07:28:37.820322+0000","last_active":"2024-03-06T07:28:38.800770+0000","last_peered":"2024-03-06T07:28:38.800770+0000","last_clean":"2024-03-06T07:28:38.800770+0000","last_became_active":"2024-03-06T07:28:37.820033+0000","last_became_peered":"2024-03-06T07:28:37.820033+0000","last_unstale":"2024-03-06T07:28:38.800770+0000","last_undegraded":"2024-03-06T07:28:38.800770+0000","last_fullsized":"2024-03-06T07:28:38.800770+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:24.104089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392416+0000","last_change":"2024-03-06T07:28:37.809735+0000","last_active":"2024-03-06T07:28:40.392416+0000","last_peered":"2024-03-06T07:28:40.392416+0000","last_clean":"2024-03-06T07:28:40.392416+0000","last_became_active":"2024-03-06T07:28:37.809462+0000","last_became_peered":"2024-03-06T07:28:37.809462+0000","last_unstale":"2024-03-06T07:28:40.392416+0000","last_undegraded":"2024-03-06T07:28:40.392416+0000","last_fullsized":"2024-03-06T07:28:40.392416+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:31:45.845287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569432+0000","last_change":"2024-03-06T07:28:37.825936+0000","last_active":"2024-03-06T07:28:39.569432+0000","last_peered":"2024-03-06T07:28:39.569432+0000","last_clean":"2024-03-06T07:28:39.569432+0000","last_became_active":"2024-03-06T07:28:37.825175+0000","last_became_peered":"2024-03-06T07:28:37.825175+0000","last_unstale":"2024-03-06T07:28:39.569432+0000","last_undegraded":"2024-03-06T07:28:39.569432+0000","last_fullsized":"2024-03-06T07:28:39.569432+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:09:11.430067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392145+0000","last_change":"2024-03-06T07:28:37.810285+0000","last_active":"2024-03-06T07:28:40.392145+0000","last_peered":"2024-03-06T07:28:40.392145+0000","last_clean":"2024-03-06T07:28:40.392145+0000","last_became_active":"2024-03-06T07:28:37.809988+0000","last_became_peered":"2024-03-06T07:28:37.809988+0000","last_unstale":"2024-03-06T07:28:40.392145+0000","last_undegraded":"2024-03-06T07:28:40.392145+0000","last_fullsized":"2024-03-06T07:28:40.392145+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:11:16.558500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392059+0000","last_change":"2024-03-06T07:28:37.804909+0000","last_active":"2024-03-06T07:28:40.392059+0000","last_peered":"2024-03-06T07:28:40.392059+0000","last_clean":"2024-03-06T07:28:40.392059+0000","last_became_active":"2024-03-06T07:28:37.804639+0000","last_became_peered":"2024-03-06T07:28:37.804639+0000","last_unstale":"2024-03-06T07:28:40.392059+0000","last_undegraded":"2024-03-06T07:28:40.392059+0000","last_fullsized":"2024-03-06T07:28:40.392059+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:18:55.133784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801396+0000","last_change":"2024-03-06T07:28:37.809101+0000","last_active":"2024-03-06T07:28:39.801396+0000","last_peered":"2024-03-06T07:28:39.801396+0000","last_clean":"2024-03-06T07:28:39.801396+0000","last_became_active":"2024-03-06T07:28:37.808827+0000","last_became_peered":"2024-03-06T07:28:37.808827+0000","last_unstale":"2024-03-06T07:28:39.801396+0000","last_undegraded":"2024-03-06T07:28:39.801396+0000","last_fullsized":"2024-03-06T07:28:39.801396+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:00:41.270619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803390+0000","last_change":"2024-03-06T07:28:37.823717+0000","last_active":"2024-03-06T07:28:39.803390+0000","last_peered":"2024-03-06T07:28:39.803390+0000","last_clean":"2024-03-06T07:28:39.803390+0000","last_became_active":"2024-03-06T07:28:37.823383+0000","last_became_peered":"2024-03-06T07:28:37.823383+0000","last_unstale":"2024-03-06T07:28:39.803390+0000","last_undegraded":"2024-03-06T07:28:39.803390+0000","last_fullsized":"2024-03-06T07:28:39.803390+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:21.866852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800756+0000","last_change":"2024-03-06T07:28:37.816614+0000","last_active":"2024-03-06T07:28:38.800756+0000","last_peered":"2024-03-06T07:28:38.800756+0000","last_clean":"2024-03-06T07:28:38.800756+0000","last_became_active":"2024-03-06T07:28:37.816301+0000","last_became_peered":"2024-03-06T07:28:37.816301+0000","last_unstale":"2024-03-06T07:28:38.800756+0000","last_undegraded":"2024-03-06T07:28:38.800756+0000","last_fullsized":"2024-03-06T07:28:38.800756+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:21.702361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"126'4","reported_seq":18,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801710+0000","last_change":"2024-03-06T07:28:36.812840+0000","last_active":"2024-03-06T07:28:39.801710+0000","last_peered":"2024-03-06T07:28:39.801710+0000","last_clean":"2024-03-06T07:28:39.801710+0000","last_became_active":"2024-03-06T07:28:36.812374+0000","last_became_peered":"2024-03-06T07:28:36.812374+0000","last_unstale":"2024-03-06T07:28:39.801710+0000","last_undegraded":"2024-03-06T07:28:39.801710+0000","last_fullsized":"2024-03-06T07:28:39.801710+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:43.695281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800656+0000","last_change":"2024-03-06T07:28:37.804103+0000","last_active":"2024-03-06T07:28:38.800656+0000","last_peered":"2024-03-06T07:28:38.800656+0000","last_clean":"2024-03-06T07:28:38.800656+0000","last_became_active":"2024-03-06T07:28:37.803742+0000","last_became_peered":"2024-03-06T07:28:37.803742+0000","last_unstale":"2024-03-06T07:28:38.800656+0000","last_undegraded":"2024-03-06T07:28:38.800656+0000","last_fullsized":"2024-03-06T07:28:38.800656+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:56.972667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569487+0000","last_change":"2024-03-06T07:28:36.808862+0000","last_active":"2024-03-06T07:28:39.569487+0000","last_peered":"2024-03-06T07:28:39.569487+0000","last_clean":"2024-03-06T07:28:39.569487+0000","last_became_active":"2024-03-06T07:28:36.808596+0000","last_became_peered":"2024-03-06T07:28:36.808596+0000","last_unstale":"2024-03-06T07:28:39.569487+0000","last_undegraded":"2024-03-06T07:28:39.569487+0000","last_fullsized":"2024-03-06T07:28:39.569487+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:39:55.327149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801373+0000","last_change":"2024-03-06T07:28:37.810239+0000","last_active":"2024-03-06T07:28:39.801373+0000","last_peered":"2024-03-06T07:28:39.801373+0000","last_clean":"2024-03-06T07:28:39.801373+0000","last_became_active":"2024-03-06T07:28:37.809329+0000","last_became_peered":"2024-03-06T07:28:37.809329+0000","last_unstale":"2024-03-06T07:28:39.801373+0000","last_undegraded":"2024-03-06T07:28:39.801373+0000","last_fullsized":"2024-03-06T07:28:39.801373+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:26:19.165526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998744+0000","last_change":"2024-03-06T07:28:36.806718+0000","last_active":"2024-03-06T07:28:38.998744+0000","last_peered":"2024-03-06T07:28:38.998744+0000","last_clean":"2024-03-06T07:28:38.998744+0000","last_became_active":"2024-03-06T07:28:36.806447+0000","last_became_peered":"2024-03-06T07:28:36.806447+0000","last_unstale":"2024-03-06T07:28:38.998744+0000","last_undegraded":"2024-03-06T07:28:38.998744+0000","last_fullsized":"2024-03-06T07:28:38.998744+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:42:54.145543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801599+0000","last_change":"2024-03-06T07:28:37.821948+0000","last_active":"2024-03-06T07:28:39.801599+0000","last_peered":"2024-03-06T07:28:39.801599+0000","last_clean":"2024-03-06T07:28:39.801599+0000","last_became_active":"2024-03-06T07:28:37.821414+0000","last_became_peered":"2024-03-06T07:28:37.821414+0000","last_unstale":"2024-03-06T07:28:39.801599+0000","last_undegraded":"2024-03-06T07:28:39.801599+0000","last_fullsized":"2024-03-06T07:28:39.801599+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:19:25.449099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801572+0000","last_change":"2024-03-06T07:28:36.812550+0000","last_active":"2024-03-06T07:28:38.801572+0000","last_peered":"2024-03-06T07:28:38.801572+0000","last_clean":"2024-03-06T07:28:38.801572+0000","last_became_active":"2024-03-06T07:28:36.812312+0000","last_became_peered":"2024-03-06T07:28:36.812312+0000","last_unstale":"2024-03-06T07:28:38.801572+0000","last_undegraded":"2024-03-06T07:28:38.801572+0000","last_fullsized":"2024-03-06T07:28:38.801572+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T19:16:22.791648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800504+0000","last_change":"2024-03-06T07:28:37.818177+0000","last_active":"2024-03-06T07:28:38.800504+0000","last_peered":"2024-03-06T07:28:38.800504+0000","last_clean":"2024-03-06T07:28:38.800504+0000","last_became_active":"2024-03-06T07:28:37.817863+0000","last_became_peered":"2024-03-06T07:28:37.817863+0000","last_unstale":"2024-03-06T07:28:38.800504+0000","last_undegraded":"2024-03-06T07:28:38.800504+0000","last_fullsized":"2024-03-06T07:28:38.800504+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:19:48.725241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801524+0000","last_change":"2024-03-06T07:28:36.812168+0000","last_active":"2024-03-06T07:28:38.801524+0000","last_peered":"2024-03-06T07:28:38.801524+0000","last_clean":"2024-03-06T07:28:38.801524+0000","last_became_active":"2024-03-06T07:28:36.811914+0000","last_became_peered":"2024-03-06T07:28:36.811914+0000","last_unstale":"2024-03-06T07:28:38.801524+0000","last_undegraded":"2024-03-06T07:28:38.801524+0000","last_fullsized":"2024-03-06T07:28:38.801524+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:33:57.566949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803578+0000","last_change":"2024-03-06T07:28:37.812896+0000","last_active":"2024-03-06T07:28:39.803578+0000","last_peered":"2024-03-06T07:28:39.803578+0000","last_clean":"2024-03-06T07:28:39.803578+0000","last_became_active":"2024-03-06T07:28:37.812577+0000","last_became_peered":"2024-03-06T07:28:37.812577+0000","last_unstale":"2024-03-06T07:28:39.803578+0000","last_undegraded":"2024-03-06T07:28:39.803578+0000","last_fullsized":"2024-03-06T07:28:39.803578+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:30.395699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800472+0000","last_change":"2024-03-06T07:28:36.808734+0000","last_active":"2024-03-06T07:28:38.800472+0000","last_peered":"2024-03-06T07:28:38.800472+0000","last_clean":"2024-03-06T07:28:38.800472+0000","last_became_active":"2024-03-06T07:28:36.808474+0000","last_became_peered":"2024-03-06T07:28:36.808474+0000","last_unstale":"2024-03-06T07:28:38.800472+0000","last_undegraded":"2024-03-06T07:28:38.800472+0000","last_fullsized":"2024-03-06T07:28:38.800472+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800624+0000","last_change":"2024-03-06T07:28:37.824916+0000","last_active":"2024-03-06T07:28:38.800624+0000","last_peered":"2024-03-06T07:28:38.800624+0000","last_clean":"2024-03-06T07:28:38.800624+0000","last_became_active":"2024-03-06T07:28:37.824538+0000","last_became_peered":"2024-03-06T07:28:37.824538+0000","last_unstale":"2024-03-06T07:28:38.800624+0000","last_undegraded":"2024-03-06T07:28:38.800624+0000","last_fullsized":"2024-03-06T07:28:38.800624+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:04:45.463350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803298+0000","last_change":"2024-03-06T07:28:36.811802+0000","last_active":"2024-03-06T07:28:39.803298+0000","last_peered":"2024-03-06T07:28:39.803298+0000","last_clean":"2024-03-06T07:28:39.803298+0000","last_became_active":"2024-03-06T07:28:36.811529+0000","last_became_peered":"2024-03-06T07:28:36.811529+0000","last_unstale":"2024-03-06T07:28:39.803298+0000","last_undegraded":"2024-03-06T07:28:39.803298+0000","last_fullsized":"2024-03-06T07:28:39.803298+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:19.965730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801398+0000","last_change":"2024-03-06T07:28:37.824620+0000","last_active":"2024-03-06T07:28:38.801398+0000","last_peered":"2024-03-06T07:28:38.801398+0000","last_clean":"2024-03-06T07:28:38.801398+0000","last_became_active":"2024-03-06T07:28:37.824374+0000","last_became_peered":"2024-03-06T07:28:37.824374+0000","last_unstale":"2024-03-06T07:28:38.801398+0000","last_undegraded":"2024-03-06T07:28:38.801398+0000","last_fullsized":"2024-03-06T07:28:38.801398+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:07.051996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801332+0000","last_change":"2024-03-06T07:28:36.811162+0000","last_active":"2024-03-06T07:28:38.801332+0000","last_peered":"2024-03-06T07:28:38.801332+0000","last_clean":"2024-03-06T07:28:38.801332+0000","last_became_active":"2024-03-06T07:28:36.810868+0000","last_became_peered":"2024-03-06T07:28:36.810868+0000","last_unstale":"2024-03-06T07:28:38.801332+0000","last_undegraded":"2024-03-06T07:28:38.801332+0000","last_fullsized":"2024-03-06T07:28:38.801332+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:55:02.364218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392298+0000","last_change":"2024-03-06T07:28:37.807469+0000","last_active":"2024-03-06T07:28:40.392298+0000","last_peered":"2024-03-06T07:28:40.392298+0000","last_clean":"2024-03-06T07:28:40.392298+0000","last_became_active":"2024-03-06T07:28:37.806799+0000","last_became_peered":"2024-03-06T07:28:37.806799+0000","last_unstale":"2024-03-06T07:28:40.392298+0000","last_undegraded":"2024-03-06T07:28:40.392298+0000","last_fullsized":"2024-03-06T07:28:40.392298+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:15:28.656211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569692+0000","last_change":"2024-03-06T07:28:36.807342+0000","last_active":"2024-03-06T07:28:39.569692+0000","last_peered":"2024-03-06T07:28:39.569692+0000","last_clean":"2024-03-06T07:28:39.569692+0000","last_became_active":"2024-03-06T07:28:36.807010+0000","last_became_peered":"2024-03-06T07:28:36.807010+0000","last_unstale":"2024-03-06T07:28:39.569692+0000","last_undegraded":"2024-03-06T07:28:39.569692+0000","last_fullsized":"2024-03-06T07:28:39.569692+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:32:06.049990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569607+0000","last_change":"2024-03-06T07:28:37.821338+0000","last_active":"2024-03-06T07:28:39.569607+0000","last_peered":"2024-03-06T07:28:39.569607+0000","last_clean":"2024-03-06T07:28:39.569607+0000","last_became_active":"2024-03-06T07:28:37.821053+0000","last_became_peered":"2024-03-06T07:28:37.821053+0000","last_unstale":"2024-03-06T07:28:39.569607+0000","last_undegraded":"2024-03-06T07:28:39.569607+0000","last_fullsized":"2024-03-06T07:28:39.569607+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:22:30.395699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800832+0000","last_change":"2024-03-06T07:28:36.812151+0000","last_active":"2024-03-06T07:28:38.800832+0000","last_peered":"2024-03-06T07:28:38.800832+0000","last_clean":"2024-03-06T07:28:38.800832+0000","last_became_active":"2024-03-06T07:28:36.811892+0000","last_became_peered":"2024-03-06T07:28:36.811892+0000","last_unstale":"2024-03-06T07:28:38.800832+0000","last_undegraded":"2024-03-06T07:28:38.800832+0000","last_fullsized":"2024-03-06T07:28:38.800832+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:03.969318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801527+0000","last_change":"2024-03-06T07:28:37.823826+0000","last_active":"2024-03-06T07:28:39.801527+0000","last_peered":"2024-03-06T07:28:39.801527+0000","last_clean":"2024-03-06T07:28:39.801527+0000","last_became_active":"2024-03-06T07:28:37.823542+0000","last_became_peered":"2024-03-06T07:28:37.823542+0000","last_unstale":"2024-03-06T07:28:39.801527+0000","last_undegraded":"2024-03-06T07:28:39.801527+0000","last_fullsized":"2024-03-06T07:28:39.801527+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:21:33.493872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"126'3","reported_seq":17,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569336+0000","last_change":"2024-03-06T07:28:36.811322+0000","last_active":"2024-03-06T07:28:39.569336+0000","last_peered":"2024-03-06T07:28:39.569336+0000","last_clean":"2024-03-06T07:28:39.569336+0000","last_became_active":"2024-03-06T07:28:36.810594+0000","last_became_peered":"2024-03-06T07:28:36.810594+0000","last_unstale":"2024-03-06T07:28:39.569336+0000","last_undegraded":"2024-03-06T07:28:39.569336+0000","last_fullsized":"2024-03-06T07:28:39.569336+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801460+0000","last_change":"2024-03-06T07:28:37.807298+0000","last_active":"2024-03-06T07:28:39.801460+0000","last_peered":"2024-03-06T07:28:39.801460+0000","last_clean":"2024-03-06T07:28:39.801460+0000","last_became_active":"2024-03-06T07:28:37.807020+0000","last_became_peered":"2024-03-06T07:28:37.807020+0000","last_unstale":"2024-03-06T07:28:39.801460+0000","last_undegraded":"2024-03-06T07:28:39.801460+0000","last_fullsized":"2024-03-06T07:28:39.801460+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:56:44.220669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"126'6","reported_seq":20,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800205+0000","last_change":"2024-03-06T07:28:36.808354+0000","last_active":"2024-03-06T07:28:38.800205+0000","last_peered":"2024-03-06T07:28:38.800205+0000","last_clean":"2024-03-06T07:28:38.800205+0000","last_became_active":"2024-03-06T07:28:36.808076+0000","last_became_peered":"2024-03-06T07:28:36.808076+0000","last_unstale":"2024-03-06T07:28:38.800205+0000","last_undegraded":"2024-03-06T07:28:38.800205+0000","last_fullsized":"2024-03-06T07:28:38.800205+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:50.115596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392258+0000","last_change":"2024-03-06T07:28:37.807386+0000","last_active":"2024-03-06T07:28:40.392258+0000","last_peered":"2024-03-06T07:28:40.392258+0000","last_clean":"2024-03-06T07:28:40.392258+0000","last_became_active":"2024-03-06T07:28:37.807033+0000","last_became_peered":"2024-03-06T07:28:37.807033+0000","last_unstale":"2024-03-06T07:28:40.392258+0000","last_undegraded":"2024-03-06T07:28:40.392258+0000","last_fullsized":"2024-03-06T07:28:40.392258+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:44:42.000809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392163+0000","last_change":"2024-03-06T07:28:36.810704+0000","last_active":"2024-03-06T07:28:40.392163+0000","last_peered":"2024-03-06T07:28:40.392163+0000","last_clean":"2024-03-06T07:28:40.392163+0000","last_became_active":"2024-03-06T07:28:36.809890+0000","last_became_peered":"2024-03-06T07:28:36.809890+0000","last_unstale":"2024-03-06T07:28:40.392163+0000","last_undegraded":"2024-03-06T07:28:40.392163+0000","last_fullsized":"2024-03-06T07:28:40.392163+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:39:40.671401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801239+0000","last_change":"2024-03-06T07:28:37.824548+0000","last_active":"2024-03-06T07:28:38.801239+0000","last_peered":"2024-03-06T07:28:38.801239+0000","last_clean":"2024-03-06T07:28:38.801239+0000","last_became_active":"2024-03-06T07:28:37.824290+0000","last_became_peered":"2024-03-06T07:28:37.824290+0000","last_unstale":"2024-03-06T07:28:38.801239+0000","last_undegraded":"2024-03-06T07:28:38.801239+0000","last_fullsized":"2024-03-06T07:28:38.801239+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:23:51.117603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801508+0000","last_change":"2024-03-06T07:28:36.810746+0000","last_active":"2024-03-06T07:28:38.801508+0000","last_peered":"2024-03-06T07:28:38.801508+0000","last_clean":"2024-03-06T07:28:38.801508+0000","last_became_active":"2024-03-06T07:28:36.810475+0000","last_became_peered":"2024-03-06T07:28:36.810475+0000","last_unstale":"2024-03-06T07:28:38.801508+0000","last_undegraded":"2024-03-06T07:28:38.801508+0000","last_fullsized":"2024-03-06T07:28:38.801508+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:59:27.539843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800380+0000","last_change":"2024-03-06T07:28:37.810787+0000","last_active":"2024-03-06T07:28:38.800380+0000","last_peered":"2024-03-06T07:28:38.800380+0000","last_clean":"2024-03-06T07:28:38.800380+0000","last_became_active":"2024-03-06T07:28:37.810266+0000","last_became_peered":"2024-03-06T07:28:37.810266+0000","last_unstale":"2024-03-06T07:28:38.800380+0000","last_undegraded":"2024-03-06T07:28:38.800380+0000","last_fullsized":"2024-03-06T07:28:38.800380+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:34:43.391321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"126'8","reported_seq":22,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800628+0000","last_change":"2024-03-06T07:28:36.809374+0000","last_active":"2024-03-06T07:28:38.800628+0000","last_peered":"2024-03-06T07:28:38.800628+0000","last_clean":"2024-03-06T07:28:38.800628+0000","last_became_active":"2024-03-06T07:28:36.807635+0000","last_became_peered":"2024-03-06T07:28:36.807635+0000","last_unstale":"2024-03-06T07:28:38.800628+0000","last_undegraded":"2024-03-06T07:28:38.800628+0000","last_fullsized":"2024-03-06T07:28:38.800628+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:34:55.017814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800569+0000","last_change":"2024-03-06T07:28:37.805400+0000","last_active":"2024-03-06T07:28:38.800569+0000","last_peered":"2024-03-06T07:28:38.800569+0000","last_clean":"2024-03-06T07:28:38.800569+0000","last_became_active":"2024-03-06T07:28:37.804321+0000","last_became_peered":"2024-03-06T07:28:37.804321+0000","last_unstale":"2024-03-06T07:28:38.800569+0000","last_undegraded":"2024-03-06T07:28:38.800569+0000","last_fullsized":"2024-03-06T07:28:38.800569+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:43.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569735+0000","last_change":"2024-03-06T07:28:36.814194+0000","last_active":"2024-03-06T07:28:39.569735+0000","last_peered":"2024-03-06T07:28:39.569735+0000","last_clean":"2024-03-06T07:28:39.569735+0000","last_became_active":"2024-03-06T07:28:36.813872+0000","last_became_peered":"2024-03-06T07:28:36.813872+0000","last_unstale":"2024-03-06T07:28:39.569735+0000","last_undegraded":"2024-03-06T07:28:39.569735+0000","last_fullsized":"2024-03-06T07:28:39.569735+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:37.938304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569606+0000","last_change":"2024-03-06T07:28:37.812580+0000","last_active":"2024-03-06T07:28:39.569606+0000","last_peered":"2024-03-06T07:28:39.569606+0000","last_clean":"2024-03-06T07:28:39.569606+0000","last_became_active":"2024-03-06T07:28:37.812301+0000","last_became_peered":"2024-03-06T07:28:37.812301+0000","last_unstale":"2024-03-06T07:28:39.569606+0000","last_undegraded":"2024-03-06T07:28:39.569606+0000","last_fullsized":"2024-03-06T07:28:39.569606+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:43.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998453+0000","last_change":"2024-03-06T07:28:36.808348+0000","last_active":"2024-03-06T07:28:38.998453+0000","last_peered":"2024-03-06T07:28:38.998453+0000","last_clean":"2024-03-06T07:28:38.998453+0000","last_became_active":"2024-03-06T07:28:36.808035+0000","last_became_peered":"2024-03-06T07:28:36.808035+0000","last_unstale":"2024-03-06T07:28:38.998453+0000","last_undegraded":"2024-03-06T07:28:38.998453+0000","last_fullsized":"2024-03-06T07:28:38.998453+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:31:19.839068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800965+0000","last_change":"2024-03-06T07:28:37.821013+0000","last_active":"2024-03-06T07:28:38.800965+0000","last_peered":"2024-03-06T07:28:38.800965+0000","last_clean":"2024-03-06T07:28:38.800965+0000","last_became_active":"2024-03-06T07:28:37.820756+0000","last_became_peered":"2024-03-06T07:28:37.820756+0000","last_unstale":"2024-03-06T07:28:38.800965+0000","last_undegraded":"2024-03-06T07:28:38.800965+0000","last_fullsized":"2024-03-06T07:28:38.800965+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:24:36.861383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"126'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803712+0000","last_change":"2024-03-06T07:28:36.817966+0000","last_active":"2024-03-06T07:28:39.803712+0000","last_peered":"2024-03-06T07:28:39.803712+0000","last_clean":"2024-03-06T07:28:39.803712+0000","last_became_active":"2024-03-06T07:28:36.817630+0000","last_became_peered":"2024-03-06T07:28:36.817630+0000","last_unstale":"2024-03-06T07:28:39.803712+0000","last_undegraded":"2024-03-06T07:28:39.803712+0000","last_fullsized":"2024-03-06T07:28:39.803712+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:46.071349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801709+0000","last_change":"2024-03-06T07:28:37.825233+0000","last_active":"2024-03-06T07:28:38.801709+0000","last_peered":"2024-03-06T07:28:38.801709+0000","last_clean":"2024-03-06T07:28:38.801709+0000","last_became_active":"2024-03-06T07:28:37.824969+0000","last_became_peered":"2024-03-06T07:28:37.824969+0000","last_unstale":"2024-03-06T07:28:38.801709+0000","last_undegraded":"2024-03-06T07:28:38.801709+0000","last_fullsized":"2024-03-06T07:28:38.801709+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:53.262994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569417+0000","last_change":"2024-03-06T07:28:36.809671+0000","last_active":"2024-03-06T07:28:39.569417+0000","last_peered":"2024-03-06T07:28:39.569417+0000","last_clean":"2024-03-06T07:28:39.569417+0000","last_became_active":"2024-03-06T07:28:36.809405+0000","last_became_peered":"2024-03-06T07:28:36.809405+0000","last_unstale":"2024-03-06T07:28:39.569417+0000","last_undegraded":"2024-03-06T07:28:39.569417+0000","last_fullsized":"2024-03-06T07:28:39.569417+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:38:46.912367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392147+0000","last_change":"2024-03-06T07:28:37.810557+0000","last_active":"2024-03-06T07:28:40.392147+0000","last_peered":"2024-03-06T07:28:40.392147+0000","last_clean":"2024-03-06T07:28:40.392147+0000","last_became_active":"2024-03-06T07:28:37.810303+0000","last_became_peered":"2024-03-06T07:28:37.810303+0000","last_unstale":"2024-03-06T07:28:40.392147+0000","last_undegraded":"2024-03-06T07:28:40.392147+0000","last_fullsized":"2024-03-06T07:28:40.392147+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:28:27.842812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569339+0000","last_change":"2024-03-06T07:28:36.807565+0000","last_active":"2024-03-06T07:28:39.569339+0000","last_peered":"2024-03-06T07:28:39.569339+0000","last_clean":"2024-03-06T07:28:39.569339+0000","last_became_active":"2024-03-06T07:28:36.807266+0000","last_became_peered":"2024-03-06T07:28:36.807266+0000","last_unstale":"2024-03-06T07:28:39.569339+0000","last_undegraded":"2024-03-06T07:28:39.569339+0000","last_fullsized":"2024-03-06T07:28:39.569339+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803855+0000","last_change":"2024-03-06T07:28:37.818446+0000","last_active":"2024-03-06T07:28:39.803855+0000","last_peered":"2024-03-06T07:28:39.803855+0000","last_clean":"2024-03-06T07:28:39.803855+0000","last_became_active":"2024-03-06T07:28:37.818122+0000","last_became_peered":"2024-03-06T07:28:37.818122+0000","last_unstale":"2024-03-06T07:28:39.803855+0000","last_undegraded":"2024-03-06T07:28:39.803855+0000","last_fullsized":"2024-03-06T07:28:39.803855+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:48:56.972667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800569+0000","last_change":"2024-03-06T07:28:36.804975+0000","last_active":"2024-03-06T07:28:38.800569+0000","last_peered":"2024-03-06T07:28:38.800569+0000","last_clean":"2024-03-06T07:28:38.800569+0000","last_became_active":"2024-03-06T07:28:36.804697+0000","last_became_peered":"2024-03-06T07:28:36.804697+0000","last_unstale":"2024-03-06T07:28:38.800569+0000","last_undegraded":"2024-03-06T07:28:38.800569+0000","last_fullsized":"2024-03-06T07:28:38.800569+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:35:15.106041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801249+0000","last_change":"2024-03-06T07:28:37.817957+0000","last_active":"2024-03-06T07:28:38.801249+0000","last_peered":"2024-03-06T07:28:38.801249+0000","last_clean":"2024-03-06T07:28:38.801249+0000","last_became_active":"2024-03-06T07:28:37.817689+0000","last_became_peered":"2024-03-06T07:28:37.817689+0000","last_unstale":"2024-03-06T07:28:38.801249+0000","last_undegraded":"2024-03-06T07:28:38.801249+0000","last_fullsized":"2024-03-06T07:28:38.801249+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:47:20.967736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800295+0000","last_change":"2024-03-06T07:28:36.810758+0000","last_active":"2024-03-06T07:28:38.800295+0000","last_peered":"2024-03-06T07:28:38.800295+0000","last_clean":"2024-03-06T07:28:38.800295+0000","last_became_active":"2024-03-06T07:28:36.810492+0000","last_became_peered":"2024-03-06T07:28:36.810492+0000","last_unstale":"2024-03-06T07:28:38.800295+0000","last_undegraded":"2024-03-06T07:28:38.800295+0000","last_fullsized":"2024-03-06T07:28:38.800295+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:52.260988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801251+0000","last_change":"2024-03-06T07:28:37.808444+0000","last_active":"2024-03-06T07:28:39.801251+0000","last_peered":"2024-03-06T07:28:39.801251+0000","last_clean":"2024-03-06T07:28:39.801251+0000","last_became_active":"2024-03-06T07:28:37.808181+0000","last_became_peered":"2024-03-06T07:28:37.808181+0000","last_unstale":"2024-03-06T07:28:39.801251+0000","last_undegraded":"2024-03-06T07:28:39.801251+0000","last_fullsized":"2024-03-06T07:28:39.801251+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T18:58:54.190073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998218+0000","last_change":"2024-03-06T07:28:36.809882+0000","last_active":"2024-03-06T07:28:38.998218+0000","last_peered":"2024-03-06T07:28:38.998218+0000","last_clean":"2024-03-06T07:28:38.998218+0000","last_became_active":"2024-03-06T07:28:36.809035+0000","last_became_peered":"2024-03-06T07:28:36.809035+0000","last_unstale":"2024-03-06T07:28:38.998218+0000","last_undegraded":"2024-03-06T07:28:38.998218+0000","last_fullsized":"2024-03-06T07:28:38.998218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:56:47.288248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803906+0000","last_change":"2024-03-06T07:28:37.813974+0000","last_active":"2024-03-06T07:28:39.803906+0000","last_peered":"2024-03-06T07:28:39.803906+0000","last_clean":"2024-03-06T07:28:39.803906+0000","last_became_active":"2024-03-06T07:28:37.813672+0000","last_became_peered":"2024-03-06T07:28:37.813672+0000","last_unstale":"2024-03-06T07:28:39.803906+0000","last_undegraded":"2024-03-06T07:28:39.803906+0000","last_fullsized":"2024-03-06T07:28:39.803906+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:33.184108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392538+0000","last_change":"2024-03-06T07:28:36.812497+0000","last_active":"2024-03-06T07:28:40.392538+0000","last_peered":"2024-03-06T07:28:40.392538+0000","last_clean":"2024-03-06T07:28:40.392538+0000","last_became_active":"2024-03-06T07:28:36.812253+0000","last_became_peered":"2024-03-06T07:28:36.812253+0000","last_unstale":"2024-03-06T07:28:40.392538+0000","last_undegraded":"2024-03-06T07:28:40.392538+0000","last_fullsized":"2024-03-06T07:28:40.392538+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T15:51:57.487138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:40.392410+0000","last_change":"2024-03-06T07:28:37.811271+0000","last_active":"2024-03-06T07:28:40.392410+0000","last_peered":"2024-03-06T07:28:40.392410+0000","last_clean":"2024-03-06T07:28:40.392410+0000","last_became_active":"2024-03-06T07:28:37.811043+0000","last_became_peered":"2024-03-06T07:28:37.811043+0000","last_unstale":"2024-03-06T07:28:40.392410+0000","last_undegraded":"2024-03-06T07:28:40.392410+0000","last_fullsized":"2024-03-06T07:28:40.392410+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:27:28.374494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"127'1","reported_seq":15,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801489+0000","last_change":"2024-03-06T07:28:36.812931+0000","last_active":"2024-03-06T07:28:39.801489+0000","last_peered":"2024-03-06T07:28:39.801489+0000","last_clean":"2024-03-06T07:28:39.801489+0000","last_became_active":"2024-03-06T07:28:36.812619+0000","last_became_peered":"2024-03-06T07:28:36.812619+0000","last_unstale":"2024-03-06T07:28:39.801489+0000","last_undegraded":"2024-03-06T07:28:39.801489+0000","last_fullsized":"2024-03-06T07:28:39.801489+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:16:52.260988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801521+0000","last_change":"2024-03-06T07:28:37.821218+0000","last_active":"2024-03-06T07:28:38.801521+0000","last_peered":"2024-03-06T07:28:38.801521+0000","last_clean":"2024-03-06T07:28:38.801521+0000","last_became_active":"2024-03-06T07:28:37.820515+0000","last_became_peered":"2024-03-06T07:28:37.820515+0000","last_unstale":"2024-03-06T07:28:38.801521+0000","last_undegraded":"2024-03-06T07:28:38.801521+0000","last_fullsized":"2024-03-06T07:28:38.801521+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T10:21:38.940310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801671+0000","last_change":"2024-03-06T07:28:36.811623+0000","last_active":"2024-03-06T07:28:38.801671+0000","last_peered":"2024-03-06T07:28:38.801671+0000","last_clean":"2024-03-06T07:28:38.801671+0000","last_became_active":"2024-03-06T07:28:36.811361+0000","last_became_peered":"2024-03-06T07:28:36.811361+0000","last_unstale":"2024-03-06T07:28:38.801671+0000","last_undegraded":"2024-03-06T07:28:38.801671+0000","last_fullsized":"2024-03-06T07:28:38.801671+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:04:25.324827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.803746+0000","last_change":"2024-03-06T07:28:37.818590+0000","last_active":"2024-03-06T07:28:39.803746+0000","last_peered":"2024-03-06T07:28:39.803746+0000","last_clean":"2024-03-06T07:28:39.803746+0000","last_became_active":"2024-03-06T07:28:37.818314+0000","last_became_peered":"2024-03-06T07:28:37.818314+0000","last_unstale":"2024-03-06T07:28:39.803746+0000","last_undegraded":"2024-03-06T07:28:39.803746+0000","last_fullsized":"2024-03-06T07:28:39.803746+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T13:14:43.646551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998524+0000","last_change":"2024-03-06T07:28:36.807851+0000","last_active":"2024-03-06T07:28:38.998524+0000","last_peered":"2024-03-06T07:28:38.998524+0000","last_clean":"2024-03-06T07:28:38.998524+0000","last_became_active":"2024-03-06T07:28:36.807590+0000","last_became_peered":"2024-03-06T07:28:36.807590+0000","last_unstale":"2024-03-06T07:28:38.998524+0000","last_undegraded":"2024-03-06T07:28:38.998524+0000","last_fullsized":"2024-03-06T07:28:38.998524+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:51:09.184127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801240+0000","last_change":"2024-03-06T07:28:37.810622+0000","last_active":"2024-03-06T07:28:39.801240+0000","last_peered":"2024-03-06T07:28:39.801240+0000","last_clean":"2024-03-06T07:28:39.801240+0000","last_became_active":"2024-03-06T07:28:37.810366+0000","last_became_peered":"2024-03-06T07:28:37.810366+0000","last_unstale":"2024-03-06T07:28:39.801240+0000","last_undegraded":"2024-03-06T07:28:39.801240+0000","last_fullsized":"2024-03-06T07:28:39.801240+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:34:24.104089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801547+0000","last_change":"2024-03-06T07:28:36.808681+0000","last_active":"2024-03-06T07:28:38.801547+0000","last_peered":"2024-03-06T07:28:38.801547+0000","last_clean":"2024-03-06T07:28:38.801547+0000","last_became_active":"2024-03-06T07:28:36.808407+0000","last_became_peered":"2024-03-06T07:28:36.808407+0000","last_unstale":"2024-03-06T07:28:38.801547+0000","last_undegraded":"2024-03-06T07:28:38.801547+0000","last_fullsized":"2024-03-06T07:28:38.801547+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T09:24:09.820534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.569407+0000","last_change":"2024-03-06T07:28:37.821485+0000","last_active":"2024-03-06T07:28:39.569407+0000","last_peered":"2024-03-06T07:28:39.569407+0000","last_clean":"2024-03-06T07:28:39.569407+0000","last_became_active":"2024-03-06T07:28:37.821170+0000","last_became_peered":"2024-03-06T07:28:37.821170+0000","last_unstale":"2024-03-06T07:28:39.569407+0000","last_undegraded":"2024-03-06T07:28:39.569407+0000","last_fullsized":"2024-03-06T07:28:39.569407+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T14:09:21.866852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801533+0000","last_change":"2024-03-06T07:28:36.812531+0000","last_active":"2024-03-06T07:28:39.801533+0000","last_peered":"2024-03-06T07:28:39.801533+0000","last_clean":"2024-03-06T07:28:39.801533+0000","last_became_active":"2024-03-06T07:28:36.812195+0000","last_became_peered":"2024-03-06T07:28:36.812195+0000","last_unstale":"2024-03-06T07:28:39.801533+0000","last_undegraded":"2024-03-06T07:28:39.801533+0000","last_fullsized":"2024-03-06T07:28:39.801533+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:03.969318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800475+0000","last_change":"2024-03-06T07:28:37.822792+0000","last_active":"2024-03-06T07:28:38.800475+0000","last_peered":"2024-03-06T07:28:38.800475+0000","last_clean":"2024-03-06T07:28:38.800475+0000","last_became_active":"2024-03-06T07:28:37.822452+0000","last_became_peered":"2024-03-06T07:28:37.822452+0000","last_unstale":"2024-03-06T07:28:38.800475+0000","last_undegraded":"2024-03-06T07:28:38.800475+0000","last_fullsized":"2024-03-06T07:28:38.800475+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:50:33.184108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.800396+0000","last_change":"2024-03-06T07:28:36.811478+0000","last_active":"2024-03-06T07:28:38.800396+0000","last_peered":"2024-03-06T07:28:38.800396+0000","last_clean":"2024-03-06T07:28:38.800396+0000","last_became_active":"2024-03-06T07:28:36.811250+0000","last_became_peered":"2024-03-06T07:28:36.811250+0000","last_unstale":"2024-03-06T07:28:38.800396+0000","last_undegraded":"2024-03-06T07:28:38.800396+0000","last_fullsized":"2024-03-06T07:28:38.800396+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T11:58:43.695281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.801296+0000","last_change":"2024-03-06T07:28:37.823153+0000","last_active":"2024-03-06T07:28:38.801296+0000","last_peered":"2024-03-06T07:28:38.801296+0000","last_clean":"2024-03-06T07:28:38.801296+0000","last_became_active":"2024-03-06T07:28:37.822205+0000","last_became_peered":"2024-03-06T07:28:37.822205+0000","last_unstale":"2024-03-06T07:28:38.801296+0000","last_undegraded":"2024-03-06T07:28:38.801296+0000","last_fullsized":"2024-03-06T07:28:38.801296+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T08:34:47.073356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":14,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:38.998338+0000","last_change":"2024-03-06T07:28:36.809706+0000","last_active":"2024-03-06T07:28:38.998338+0000","last_peered":"2024-03-06T07:28:38.998338+0000","last_clean":"2024-03-06T07:28:38.998338+0000","last_became_active":"2024-03-06T07:28:36.809345+0000","last_became_peered":"2024-03-06T07:28:36.809345+0000","last_unstale":"2024-03-06T07:28:38.998338+0000","last_undegraded":"2024-03-06T07:28:38.998338+0000","last_fullsized":"2024-03-06T07:28:38.998338+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T16:15:00.510053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":12,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801164+0000","last_change":"2024-03-06T07:28:37.810948+0000","last_active":"2024-03-06T07:28:39.801164+0000","last_peered":"2024-03-06T07:28:39.801164+0000","last_clean":"2024-03-06T07:28:39.801164+0000","last_became_active":"2024-03-06T07:28:37.810064+0000","last_became_peered":"2024-03-06T07:28:37.810064+0000","last_unstale":"2024-03-06T07:28:39.801164+0000","last_undegraded":"2024-03-06T07:28:39.801164+0000","last_fullsized":"2024-03-06T07:28:39.801164+0000","mapping_epoch":125,"log_start":"0'0","ondisk_log_start":"0'0","created":125,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:36.789256+0000","last_clean_scrub_stamp":"2024-03-06T07:28:36.789256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T12:15:21.702361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"126'2","reported_seq":16,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801501+0000","last_change":"2024-03-06T07:28:36.812624+0000","last_active":"2024-03-06T07:28:39.801501+0000","last_peered":"2024-03-06T07:28:39.801501+0000","last_clean":"2024-03-06T07:28:39.801501+0000","last_became_active":"2024-03-06T07:28:36.812331+0000","last_became_peered":"2024-03-06T07:28:36.812331+0000","last_unstale":"2024-03-06T07:28:39.801501+0000","last_undegraded":"2024-03-06T07:28:39.801501+0000","last_fullsized":"2024-03-06T07:28:39.801501+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:28:35.787250+0000","last_clean_scrub_stamp":"2024-03-06T07:28:35.787250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T07:36:58.094644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":363,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-03-06T07:28:39.801001+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:28:39.801001+0000","last_peered":"2024-03-06T07:28:39.801001+0000","last_clean":"2024-03-06T07:28:39.801001+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:28:39.801001+0000","last_undegraded":"2024-03-06T07:28:39.801001+0000","last_fullsized":"2024-03-06T07:28:39.801001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2203648,"data_stored":1285483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41068,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330772,"statfs":{"total":96636764160,"available":96594710528,"internally_reserved":0,"allocated":1597440,"data_stored":694477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574936,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":692977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41052,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330788,"statfs":{"total":96636764160,"available":96594726912,"internally_reserved":0,"allocated":1589248,"data_stored":692935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41064,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330776,"statfs":{"total":96636764160,"available":96594714624,"internally_reserved":0,"allocated":1593344,"data_stored":693025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1593344,"data_stored":693481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":1585152,"data_stored":692865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41628,"kb_used_data":2128,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330212,"statfs":{"total":96636764160,"available":96594137088,"internally_reserved":0,"allocated":2179072,"data_stored":1283233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:28:45.655 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-06T07:28:45.655 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-06T07:28:45.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.797+0000 7ff80120e640 1 Processor -- start 2024-03-06T07:28:45.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.797+0000 7ff80120e640 1 -- start start 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff80120e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 0x7ff7fc0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff80120e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 0x7ff7fc0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff80120e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7fc0cd1e0 con 0x7ff7fc15c050 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff80120e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7fc0cd4b0 con 0x7ff7fc06fd10 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff80120e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc0cd790 con 0x7ff7fc0d0af0 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff7fb59e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 0x7ff7fc0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff7f259c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 0x7ff7fc0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff7f259c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 0x7ff7fc0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:59854/0 (socket says 172.21.15.72:59854) 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff7fb59e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 0x7ff7fc0ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:34952/0 (socket says 172.21.15.72:34952) 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff7f259c640 1 -- 172.21.15.72:0/3194414258 learned_addr learned my addr 172.21.15.72:0/3194414258 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.798+0000 7ff7fad9d640 1 --1- 172.21.15.72:0/3194414258 >> v1:172.21.15.146:6789/0 conn(0x7ff7fc15c050 0x7ff7fc06f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:45864/0 (socket says 172.21.15.72:45864) 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.799+0000 7ff7fb59e640 1 -- 172.21.15.72:0/3194414258 >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 msgr2=0x7ff7fc0cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.799+0000 7ff7fb59e640 1 --2- 172.21.15.72:0/3194414258 >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 0x7ff7fc0cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.799+0000 7ff7fb59e640 1 -- 172.21.15.72:0/3194414258 >> v1:172.21.15.146:6789/0 conn(0x7ff7fc15c050 legacy=0x7ff7fc06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.799+0000 7ff7fb59e640 1 -- 172.21.15.72:0/3194414258 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0cda70 con 0x7ff7fc0d0af0 2024-03-06T07:28:45.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.799+0000 7ff7f259c640 1 --2- 172.21.15.72:0/3194414258 >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 0x7ff7fc0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:28:45.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.800+0000 7ff7fb59e640 1 --2- 172.21.15.72:0/3194414258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 0x7ff7fc0ccca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009d90 tx=0x7ff7f402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46085dd60670f04b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.800+0000 7ff7fa59c640 1 -- 172.21.15.72:0/3194414258 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4039070 con 0x7ff7fc0d0af0 2024-03-06T07:28:45.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.800+0000 7ff7fa59c640 1 -- 172.21.15.72:0/3194414258 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f402cbc0 con 0x7ff7fc0d0af0 2024-03-06T07:28:45.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.800+0000 7ff7fa59c640 1 -- 172.21.15.72:0/3194414258 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4034440 con 0x7ff7fc0d0af0 2024-03-06T07:28:45.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.800+0000 7ff80120e640 1 -- 172.21.15.72:0/3194414258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 msgr2=0x7ff7fc0ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.800+0000 7ff80120e640 1 --2- 172.21.15.72:0/3194414258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 0x7ff7fc0ccca0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009d90 tx=0x7ff7f402be10 comp rx=0 tx=0).stop 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.801+0000 7ff80120e640 1 -- 172.21.15.72:0/3194414258 shutdown_connections 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.801+0000 7ff80120e640 1 --2- 172.21.15.72:0/3194414258 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc0d0af0 0x7ff7fc0ccca0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.801+0000 7ff80120e640 1 --2- 172.21.15.72:0/3194414258 >> v2:172.21.15.146:3300/0 conn(0x7ff7fc06fd10 0x7ff7fc0cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.801+0000 7ff80120e640 1 -- 172.21.15.72:0/3194414258 >> 172.21.15.72:0/3194414258 conn(0x7ff7fc0b3800 msgr2=0x7ff7fc0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.801+0000 7ff80120e640 1 -- 172.21.15.72:0/3194414258 shutdown_connections 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.801+0000 7ff80120e640 1 -- 172.21.15.72:0/3194414258 wait complete. 2024-03-06T07:28:45.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.803+0000 7ff80120e640 1 Processor -- start 2024-03-06T07:28:45.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.803+0000 7ff80120e640 1 -- start start 2024-03-06T07:28:45.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff80120e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc06fd10 0x7ff7fc0cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff80120e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7fc0d0af0 0x7ff7fc06c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff80120e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 0x7ff7fc152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff80120e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc165c30 con 0x7ff7fc15c050 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff80120e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc165ab0 con 0x7ff7fc0d0af0 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff80120e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7fc165db0 con 0x7ff7fc06fd10 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff7fb59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 0x7ff7fc152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff7fb59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 0x7ff7fc152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34890/0 (socket says 172.21.15.72:34890) 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff7fb59e640 1 -- 172.21.15.72:0/658027999 learned_addr learned my addr 172.21.15.72:0/658027999 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff7fad9d640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc06fd10 0x7ff7fc0cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff7f259c640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7fc0d0af0 0x7ff7fc06c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.804+0000 7ff7fb59e640 1 -- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc06fd10 msgr2=0x7ff7fc0cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7fb59e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc06fd10 0x7ff7fc0cb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7fb59e640 1 -- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7fc0d0af0 msgr2=0x7ff7fc06c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7fb59e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7fc0d0af0 0x7ff7fc06c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7fb59e640 1 -- 172.21.15.72:0/658027999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc06ce20 con 0x7ff7fc15c050 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7fb59e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 0x7ff7fc152ac0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009eb0 tx=0x7ff7f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4039050 con 0x7ff7fc15c050 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f4045070 con 0x7ff7fc15c050 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7fc06d100 con 0x7ff7fc15c050 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f4009450 con 0x7ff7fc15c050 2024-03-06T07:28:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.805+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7fc06d6c0 con 0x7ff7fc15c050 2024-03-06T07:28:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.806+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7fc09fe60 con 0x7ff7fc15c050 2024-03-06T07:28:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.807+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff7f403e850 con 0x7ff7fc15c050 2024-03-06T07:28:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.807+0000 7ff7f3fff640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e006b4d0 0x7ff7e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.807+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7f40ad040 con 0x7ff7fc15c050 2024-03-06T07:28:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.807+0000 7ff7fad9d640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e006b4d0 0x7ff7e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:45.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.808+0000 7ff7fad9d640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e006b4d0 0x7ff7e006d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0cc1b0 tx=0x7ff7ec007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:45.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:45.813+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff7f407b990 con 0x7ff7fc15c050 2024-03-06T07:28:46.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.062+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff7fc074040 con 0x7ff7fc15c050 2024-03-06T07:28:46.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.064+0000 7ff7f3fff640 1 -- 172.21.15.72:0/658027999 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff7f407fa30 con 0x7ff7fc15c050 2024-03-06T07:28:46.066 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-06T07:28:46.066 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBUGehlyCU7ERAAnRZb/ml2sxqXvAlXk/SnHw== 2024-03-06T07:28:46.066 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-03-06T07:28:46.066 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-03-06T07:28:46.067 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-06T07:28:46.067 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-03-06T07:28:46.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e006b4d0 msgr2=0x7ff7e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e006b4d0 0x7ff7e006d990 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc0cc1b0 tx=0x7ff7ec007e80 comp rx=0 tx=0).stop 2024-03-06T07:28:46.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 msgr2=0x7ff7fc152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 0x7ff7fc152ac0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7ff7f4009eb0 tx=0x7ff7f4004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 shutdown_connections 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7e006b4d0 0x7ff7e006d990 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7fc15c050 0x7ff7fc152ac0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7fc0d0af0 0x7ff7fc06c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 --2- 172.21.15.72:0/658027999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7fc06fd10 0x7ff7fc0cb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 >> 172.21.15.72:0/658027999 conn(0x7ff7fc0b3800 msgr2=0x7ff7fc0aeae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 shutdown_connections 2024-03-06T07:28:46.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.067+0000 7ff80120e640 1 -- 172.21.15.72:0/658027999 wait complete. 2024-03-06T07:28:46.081 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:46.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.180+0000 7f7713f81640 1 Processor -- start 2024-03-06T07:28:46.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.180+0000 7f7713f81640 1 -- start start 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.180+0000 7f7713f81640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f770c0d04a0 0x7f770c0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.180+0000 7f7713f81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770c161980 con 0x7f770c0cfff0 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.180+0000 7f7713f81640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f770c161c60 con 0x7f770c15bb20 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.180+0000 7f7713f81640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770c0d2500 con 0x7f770c0d04a0 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f771251e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f770c0d04a0 0x7f770c0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f771151c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f770c0cfff0 0x7f770c0cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58052/0 (socket says 172.21.15.72:58052) 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f771151c640 1 -- 172.21.15.72:0/3320656872 learned_addr learned my addr 172.21.15.72:0/3320656872 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f771251e640 1 -- 172.21.15.72:0/3320656872 >> v1:172.21.15.146:6789/0 conn(0x7f770c15bb20 legacy=0x7f770c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f771251e640 1 -- 172.21.15.72:0/3320656872 >> v1:172.21.15.72:6789/0 conn(0x7f770c0cfff0 legacy=0x7f770c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f7710d1b640 1 -- 172.21.15.72:0/3320656872 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696991381 0 0) 0x7f770c161980 con 0x7f770c0cfff0 2024-03-06T07:28:46.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.181+0000 7f771251e640 1 -- 172.21.15.72:0/3320656872 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c161f40 con 0x7f770c0d04a0 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.182+0000 7f771251e640 1 --2- 172.21.15.72:0/3320656872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c0d04a0 0x7f770c0ccfc0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f7704009d90 tx=0x7f770402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25acdc85d8d0b908 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.182+0000 7f7710d1b640 1 -- 172.21.15.72:0/3320656872 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704039070 con 0x7f770c0d04a0 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.182+0000 7f7710d1b640 1 -- 172.21.15.72:0/3320656872 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770402cd00 con 0x7f770c0d04a0 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.182+0000 7f7710d1b640 1 -- 172.21.15.72:0/3320656872 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704034660 con 0x7f770c0d04a0 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 -- 172.21.15.72:0/3320656872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c0d04a0 msgr2=0x7f770c0ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 --2- 172.21.15.72:0/3320656872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c0d04a0 0x7f770c0ccfc0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f7704009d90 tx=0x7f770402bf50 comp rx=0 tx=0).stop 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 -- 172.21.15.72:0/3320656872 shutdown_connections 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 --2- 172.21.15.72:0/3320656872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c0d04a0 0x7f770c0ccfc0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 -- 172.21.15.72:0/3320656872 >> 172.21.15.72:0/3320656872 conn(0x7f770c09a9f0 msgr2=0x7f770c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 -- 172.21.15.72:0/3320656872 shutdown_connections 2024-03-06T07:28:46.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.183+0000 7f7713f81640 1 -- 172.21.15.72:0/3320656872 wait complete. 2024-03-06T07:28:46.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.184+0000 7f7713f81640 1 Processor -- start 2024-03-06T07:28:46.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.184+0000 7f7713f81640 1 -- start start 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.185+0000 7f7713f81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f770c0cfff0 0x7f770c0cbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.185+0000 7f7713f81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f770c0d04a0 0x7f770c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.185+0000 7f7713f81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 0x7f770c0c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.185+0000 7f7713f81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c0d08d0 con 0x7f770c15bb20 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.185+0000 7f7713f81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c162500 con 0x7f770c0d04a0 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.185+0000 7f7713f81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770c0d1980 con 0x7f770c0cfff0 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 0x7f770c0c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 0x7f770c0c77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34910/0 (socket says 172.21.15.72:34910) 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 -- 172.21.15.72:0/3434787023 learned_addr learned my addr 172.21.15.72:0/3434787023 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f7711d1d640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f770c0cfff0 0x7f770c0cbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771151c640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f770c0d04a0 0x7f770c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 -- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f770c0cfff0 msgr2=0x7f770c0cbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f770c0cfff0 0x7f770c0cbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 -- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f770c0d04a0 msgr2=0x7f770c0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f770c0d04a0 0x7f770c0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 -- 172.21.15.72:0/3434787023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c0caaf0 con 0x7f770c15bb20 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f7711d1d640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f770c0cfff0 0x7f770c0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771151c640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f770c0d04a0 0x7f770c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f771251e640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 0x7f770c0c77c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7704037040 tx=0x7f7704004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7704039050 con 0x7f770c15bb20 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f770403a040 con 0x7f770c15bb20 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770403e610 con 0x7f770c15bb20 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770c0cadd0 con 0x7f770c15bb20 2024-03-06T07:28:46.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.186+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770c0cb390 con 0x7f770c15bb20 2024-03-06T07:28:46.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.187+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770c04f330 con 0x7f770c15bb20 2024-03-06T07:28:46.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.188+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7704047020 con 0x7f770c15bb20 2024-03-06T07:28:46.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.188+0000 7f770a7fc640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76f806b500 0x7f76f806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.188+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77040ac4f0 con 0x7f770c15bb20 2024-03-06T07:28:46.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.188+0000 7f7711d1d640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76f806b500 0x7f76f806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.189+0000 7f7711d1d640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76f806b500 0x7f76f806d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f770c0c8350 tx=0x7f76fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.192+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f770404e020 con 0x7f770c15bb20 2024-03-06T07:28:46.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.407+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f770c0cd030 con 0x7f770c15bb20 2024-03-06T07:28:46.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.407+0000 7f770a7fc640 1 -- 172.21.15.72:0/3434787023 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4982 (secure 0 0 0) 0x7f770407b430 con 0x7f770c15bb20 2024-03-06T07:28:46.410 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:46.410 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":79,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2751164744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2751164744},{"type":"v1","addr":"172.21.15.146:6834","nonce":2751164744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/406165267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":406165267},{"type":"v1","addr":"172.21.15.146:6835","nonce":406165267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":8637,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:39.164534+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:28:46.410 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 79 2024-03-06T07:28:46.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76f806b500 msgr2=0x7f76f806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76f806b500 0x7f76f806d9c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f770c0c8350 tx=0x7f76fc009040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 msgr2=0x7f770c0c77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 0x7f770c0c77c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f7704037040 tx=0x7f7704004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 shutdown_connections 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f76f806b500 0x7f76f806d9c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f770c15bb20 0x7f770c0c77c0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f770c0d04a0 0x7f770c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 --2- 172.21.15.72:0/3434787023 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f770c0cfff0 0x7f770c0cbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 >> 172.21.15.72:0/3434787023 conn(0x7f770c09a9f0 msgr2=0x7f770c09add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 shutdown_connections 2024-03-06T07:28:46.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.410+0000 7f7713f81640 1 -- 172.21.15.72:0/3434787023 wait complete. 2024-03-06T07:28:46.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:28:39.164534+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:28:46.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-06T07:28:46.825 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-06T07:28:46.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.924+0000 7f61727d7640 1 Processor -- start 2024-03-06T07:28:46.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.924+0000 7f61727d7640 1 -- start start 2024-03-06T07:28:46.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f61727d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c0b4460 0x7f616c0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f61727d7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f616c162970 0x7f616c0b3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f61727d7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f616c161cf0 0x7f616c162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f61727d7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c161b00 con 0x7f616c161cf0 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f61727d7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0cecc0 con 0x7f616c162970 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f61727d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d22c0 con 0x7f616c0b4460 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f616c161cf0 0x7f616c162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f616c161cf0 0x7f616c162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34916/0 (socket says 172.21.15.72:34916) 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 -- 172.21.15.72:0/2890156671 learned_addr learned my addr 172.21.15.72:0/2890156671 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f616bfff640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c0b4460 0x7f616c0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f616b7fe640 1 --2- 172.21.15.72:0/2890156671 >> v2:172.21.15.146:3300/0 conn(0x7f616c162970 0x7f616c0b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 -- 172.21.15.72:0/2890156671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c0b4460 msgr2=0x7f616c0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c0b4460 0x7f616c0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 -- 172.21.15.72:0/2890156671 >> v2:172.21.15.146:3300/0 conn(0x7f616c162970 msgr2=0x7f616c0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 --2- 172.21.15.72:0/2890156671 >> v2:172.21.15.146:3300/0 conn(0x7f616c162970 0x7f616c0b3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f6170d74640 1 -- 172.21.15.72:0/2890156671 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0d25a0 con 0x7f616c161cf0 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f616bfff640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c0b4460 0x7f616c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.925+0000 7f616b7fe640 1 --2- 172.21.15.72:0/2890156671 >> v2:172.21.15.146:3300/0 conn(0x7f616c162970 0x7f616c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.926+0000 7f6170d74640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c161cf0 0x7f616c162180 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f616c0b6840 tx=0x7f616402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabf544463713ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.926+0000 7f616affd640 1 -- 172.21.15.72:0/2890156671 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616402ca70 con 0x7f616c161cf0 2024-03-06T07:28:46.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.926+0000 7f616affd640 1 -- 172.21.15.72:0/2890156671 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616402cc10 con 0x7f616c161cf0 2024-03-06T07:28:46.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.926+0000 7f616affd640 1 -- 172.21.15.72:0/2890156671 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640343f0 con 0x7f616c161cf0 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 -- 172.21.15.72:0/2890156671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c161cf0 msgr2=0x7f616c162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c161cf0 0x7f616c162180 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f616c0b6840 tx=0x7f616402bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 -- 172.21.15.72:0/2890156671 shutdown_connections 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c161cf0 0x7f616c162180 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 --2- 172.21.15.72:0/2890156671 >> v2:172.21.15.146:3300/0 conn(0x7f616c162970 0x7f616c0b3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 --2- 172.21.15.72:0/2890156671 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c0b4460 0x7f616c0d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 -- 172.21.15.72:0/2890156671 >> 172.21.15.72:0/2890156671 conn(0x7f616c0d3ce0 msgr2=0x7f616c0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 -- 172.21.15.72:0/2890156671 shutdown_connections 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.927+0000 7f61727d7640 1 -- 172.21.15.72:0/2890156671 wait complete. 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.929+0000 7f61727d7640 1 Processor -- start 2024-03-06T07:28:46.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.929+0000 7f61727d7640 1 -- start start 2024-03-06T07:28:46.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f61727d7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 0x7f616c0c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f61727d7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c161cf0 0x7f616c0b96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f61727d7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f616c162970 0x7f616c0c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f61727d7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0ce950 con 0x7f616c0b4460 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f61727d7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0d2b60 con 0x7f616c162970 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f61727d7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0b3ac0 con 0x7f616c161cf0 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 0x7f616c0c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 0x7f616c0c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34930/0 (socket says 172.21.15.72:34930) 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 -- 172.21.15.72:0/1219021223 learned_addr learned my addr 172.21.15.72:0/1219021223 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c161cf0 0x7f616c0b96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f6170d74640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f616c162970 0x7f616c0c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 -- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c161cf0 msgr2=0x7f616c0b96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c161cf0 0x7f616c0b96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 -- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f616c162970 msgr2=0x7f616c0c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f616c162970 0x7f616c0c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 -- 172.21.15.72:0/1219021223 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0cae50 con 0x7f616c0b4460 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f6170d74640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f616c162970 0x7f616c0c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.930+0000 7f616bfff640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 0x7f616c0c83b0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c7f0 tx=0x7f615c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.931+0000 7f6168ff9640 1 -- 172.21.15.72:0/1219021223 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c015070 con 0x7f616c0b4460 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.931+0000 7f6168ff9640 1 -- 172.21.15.72:0/1219021223 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c007820 con 0x7f616c0b4460 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.931+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c0cb0f0 con 0x7f616c0b4460 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.931+0000 7f6168ff9640 1 -- 172.21.15.72:0/1219021223 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c007bd0 con 0x7f616c0b4460 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.931+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0c9860 con 0x7f616c0b4460 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.932+0000 7f6168ff9640 1 -- 172.21.15.72:0/1219021223 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f615c018070 con 0x7f616c0b4460 2024-03-06T07:28:46.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.932+0000 7f6168ff9640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f615806b4d0 0x7f615806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.933+0000 7f6168ff9640 1 -- 172.21.15.72:0/1219021223 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f615c08a290 con 0x7f616c0b4460 2024-03-06T07:28:46.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.933+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f615806b4d0 0x7f615806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.933+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f615806b4d0 0x7f615806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f616c0b2d20 tx=0x7f6160008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.934+0000 7f61753bf740 1 Processor -- start 2024-03-06T07:28:46.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.934+0000 7f61753bf740 1 -- start start 2024-03-06T07:28:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6d8f870 0x5558b6d8fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f61753bf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 0x5558b6d95360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dbfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f61753bf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b6cf96b0 con 0x5558b6d938c0 2024-03-06T07:28:46.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b6ca2a60 con 0x5558b6d8f870 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558b6cf5750 con 0x5558b6dbf600 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f616b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 0x5558b6d95360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f616b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 0x5558b6d95360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34944/0 (socket says 172.21.15.72:34944) 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.935+0000 7f616b7fe640 1 -- 172.21.15.72:0/2850952737 learned_addr learned my addr 172.21.15.72:0/2850952737 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616bfff640 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6d8f870 0x5558b6d8fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f6170d74640 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dbfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616b7fe640 1 -- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 msgr2=0x5558b6dbfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616b7fe640 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dbfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616b7fe640 1 -- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6d8f870 msgr2=0x5558b6d8fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616b7fe640 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6d8f870 0x5558b6d8fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616b7fe640 1 -- 172.21.15.72:0/2850952737 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558b6cc0f30 con 0x5558b6d938c0 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616bfff640 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6d8f870 0x5558b6d8fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f616b7fe640 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 0x5558b6d95360 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f616000f8e0 tx=0x7f6160012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748f22c2702b5387 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f6151ffb640 1 -- 172.21.15.72:0/2850952737 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616001b050 con 0x5558b6d938c0 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f6151ffb640 1 -- 172.21.15.72:0/2850952737 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6160025070 con 0x5558b6d938c0 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.936+0000 7f6151ffb640 1 -- 172.21.15.72:0/2850952737 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6160014e20 con 0x5558b6d938c0 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 -- 172.21.15.72:0/2850952737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 msgr2=0x5558b6d95360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 0x5558b6d95360 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f616000f8e0 tx=0x7f6160012000 comp rx=0 tx=0).stop 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 -- 172.21.15.72:0/2850952737 shutdown_connections 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dbfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6d938c0 0x5558b6d95360 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 --2- 172.21.15.72:0/2850952737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6d8f870 0x5558b6d8fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 -- 172.21.15.72:0/2850952737 >> 172.21.15.72:0/2850952737 conn(0x5558b6d918f0 msgr2=0x5558b6d931b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 -- 172.21.15.72:0/2850952737 shutdown_connections 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 -- 172.21.15.72:0/2850952737 wait complete. 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 -- start start 2024-03-06T07:28:46.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.937+0000 7f61753bf740 1 Processor -- start 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6db94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc2dc0 0x5558b6dc12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc1830 0x5558b6db9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b6d93d50 con 0x5558b6dbf600 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b6cb5ea0 con 0x5558b6dc2dc0 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558b6d94030 con 0x5558b6dc1830 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f61753bf740 10 client.0 _ll_register_callbacks cb 0x5558b6cb5870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6db94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6db94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34954/0 (socket says 172.21.15.72:34954) 2024-03-06T07:28:46.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616bfff640 1 -- 172.21.15.72:0/765535536 learned_addr learned my addr 172.21.15.72:0/765535536 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616bfff640 1 -- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc1830 msgr2=0x5558b6db9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616bfff640 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc1830 0x5558b6db9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616bfff640 1 -- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc2dc0 msgr2=0x5558b6dc12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.938+0000 7f616b7fe640 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc2dc0 0x5558b6dc12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f616bfff640 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc2dc0 0x5558b6dc12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f616bfff640 1 -- 172.21.15.72:0/765535536 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558b6cb6270 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f616bfff640 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6db94e0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f615c0587a0 tx=0x7f615c058b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11ca55b3ae91659 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f614affd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f614affd640 1 -- 172.21.15.72:0/765535536 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c01f040 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f614affd640 1 -- 172.21.15.72:0/765535536 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f615c060f20 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6160025070 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f614affd640 1 -- 172.21.15.72:0/765535536 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f615c08a0c0 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 10 client.8802 fetch_fsmap learned FSMap version 79 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x5558b6a0e2d0 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f614affd640 1 -- 172.21.15.72:0/765535536 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6160025070 con 0x5558b6dbf600 2024-03-06T07:28:46.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f614affd640 1 -- 172.21.15.72:0/765535536 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 79) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f615c057ef0 con 0x5558b6dbf600 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 10 client.8802 fetch_fsmap finished waiting for FSMap version 79 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 10 client.8802 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 20 client.8802 populate_metadata read hostname 'smithi072' 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6dee700 0x5558b6df9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 4 client.8802 mds_command: new command op to 8640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 --> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558b6d06bf0 con 0x5558b6dee700 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.939+0000 7f616b7fe640 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6dee700 0x5558b6df9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.940+0000 7f616b7fe640 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6dee700 0x5558b6df9250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b967118555d8eb7 server_cookie=649c8b7830a06822 in_seq=0 out_seq=0 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.940+0000 7f614affd640 10 client.8802 ms_handle_connect on v2:172.21.15.72:6835/1633306088 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f614affd640 1 -- 172.21.15.72:0/765535536 <== mds.0 v2:172.21.15.72:6835/1633306088 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5558b6d06bf0 con 0x5558b6dee700 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f614affd640 10 client.8802 handle_command_reply: tid=0 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 client.8802 shutdown 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 msgr2=0x5558b6db94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6db94e0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f615c0587a0 tx=0x7f615c058b60 comp rx=0 tx=0).stop 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 shutdown_connections 2024-03-06T07:28:46.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6dee700 0x5558b6df9250 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc1830 0x5558b6db9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc2dc0 0x5558b6dc12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 --2- 172.21.15.72:0/765535536 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6db94e0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 >> 172.21.15.72:0/765535536 conn(0x5558b6d8f870 msgr2=0x5558b6d8e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f614affd640 0 client.8802 ms_handle_reset on v2:172.21.15.72:6835/1633306088 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 shutdown_connections 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 1 -- 172.21.15.72:0/765535536 wait complete. 2024-03-06T07:28:46.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.941+0000 7f61753bf740 20 client.8802 trim_cache size 0 max 16384 2024-03-06T07:28:46.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 Processor -- start 2024-03-06T07:28:46.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 -- start start 2024-03-06T07:28:46.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6dd6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dd74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc2dc0 0x5558b6dda7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1f7e90 con 0x5558b6dbf600 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1f7290 con 0x5558b6dc1830 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c1f7890 con 0x5558b6dc2dc0 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f616bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6dd6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f616bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6dd6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34970/0 (socket says 172.21.15.72:34970) 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.958+0000 7f616bfff640 1 -- 172.21.15.72:0/1760543215 learned_addr learned my addr 172.21.15.72:0/1760543215 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f6170d74640 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc2dc0 0x5558b6dda7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dd74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616bfff640 1 -- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc2dc0 msgr2=0x5558b6dda7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616bfff640 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc2dc0 0x5558b6dda7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616bfff640 1 -- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 msgr2=0x5558b6dd74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616bfff640 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dd74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616bfff640 1 -- 172.21.15.72:0/1760543215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558b6ad86b0 con 0x5558b6dbf600 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dd74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f616bfff640 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6dd6f90 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f615c054f00 tx=0x7f615c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c2b2acadb69251 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61517fa640 1 -- 172.21.15.72:0/1760543215 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c01f040 con 0x5558b6dbf600 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61517fa640 1 -- 172.21.15.72:0/1760543215 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c060d20 con 0x5558b6dbf600 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61517fa640 1 -- 172.21.15.72:0/1760543215 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c05db10 con 0x5558b6dbf600 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 -- 172.21.15.72:0/1760543215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 msgr2=0x5558b6dd6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6dd6f90 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f615c054f00 tx=0x7f615c01c000 comp rx=0 tx=0).stop 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 -- 172.21.15.72:0/1760543215 shutdown_connections 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dc2dc0 0x5558b6dda7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dd74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 --2- 172.21.15.72:0/1760543215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dbf600 0x5558b6dd6f90 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 -- 172.21.15.72:0/1760543215 >> 172.21.15.72:0/1760543215 conn(0x5558b6d8f870 msgr2=0x5558b6db92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.959+0000 7f61753bf740 1 -- 172.21.15.72:0/1760543215 shutdown_connections 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.960+0000 7f61753bf740 1 -- 172.21.15.72:0/1760543215 wait complete. 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.960+0000 7f61753bf740 1 -- start start 2024-03-06T07:28:46.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.960+0000 7f61753bf740 1 Processor -- start 2024-03-06T07:28:46.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.961+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dec8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.961+0000 7f61753bf740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f61753bf740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 0x5558b6e4e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f61753bf740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b6dd51b0 con 0x5558b6dc2dc0 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558b6dc0af0 con 0x5558b6dc1830 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f61753bf740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558b6dba7c0 con 0x5558b6dbf600 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f61753bf740 10 client.0 _ll_register_callbacks cb 0x7f615c0591a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 0x5558b6e4e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 0x5558b6e4e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34980/0 (socket says 172.21.15.72:34980) 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 -- 172.21.15.72:0/1352657322 learned_addr learned my addr 172.21.15.72:0/1352657322 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f616bfff640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dec8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 -- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 msgr2=0x5558b6dec8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dec8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 -- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 msgr2=0x5558b6dece10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dece10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 -- 172.21.15.72:0/1352657322 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558b6dbcad0 con 0x5558b6dc2dc0 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f616bfff640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f616b7fe640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:46.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f6170d74640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 0x5558b6e4e2e0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f6164009f40 tx=0x7f6164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe425c74f4b7436 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f614affd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f614affd640 1 -- 172.21.15.72:0/1352657322 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164009510 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f614affd640 1 -- 172.21.15.72:0/1352657322 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6164034920 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f615c060d20 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.962+0000 7f614affd640 1 -- 172.21.15.72:0/1352657322 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=79) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6164034b20 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 10 client.8808 fetch_fsmap learned FSMap version 79 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=79,monmap=2+}) v3 -- 0x5558b6d94090 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f614affd640 1 -- 172.21.15.72:0/1352657322 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c060d20 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f614affd640 1 -- 172.21.15.72:0/1352657322 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 79) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6164008040 con 0x5558b6dc2dc0 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 10 client.8808 fetch_fsmap finished waiting for FSMap version 79 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 10 client.8808 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 20 client.8808 populate_metadata read hostname 'smithi072' 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6e4eca0 0x5558b6dbe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 4 client.8808 mds_command: new command op to 8640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-06T07:28:46.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 --> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5558b6d92050 con 0x5558b6e4eca0 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f616bfff640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6e4eca0 0x5558b6dbe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f614affd640 10 client.8808 ms_handle_connect on v2:172.21.15.72:6835/1633306088 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f616bfff640 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6e4eca0 0x5558b6dbe470 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b606e5c625af2a0 server_cookie=b14023f48ef3452e in_seq=0 out_seq=0 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f614affd640 1 -- 172.21.15.72:0/1352657322 <== mds.0 v2:172.21.15.72:6835/1633306088 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x5558b6d92050 con 0x5558b6e4eca0 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f614affd640 10 client.8808 handle_command_reply: tid=0 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.963+0000 7f61753bf740 1 client.8808 shutdown 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 msgr2=0x5558b6e4e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 0x5558b6e4e2e0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f6164009f40 tx=0x7f6164004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 shutdown_connections 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:6835/1633306088,v1:172.21.15.72:6837/1633306088] conn(0x5558b6e4eca0 0x5558b6dbe470 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5558b6dc2dc0 0x5558b6e4e2e0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5558b6dc1830 0x5558b6dece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f614affd640 0 client.8808 ms_handle_reset on v2:172.21.15.72:6835/1633306088 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 --2- 172.21.15.72:0/1352657322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5558b6dbf600 0x5558b6dec8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 >> 172.21.15.72:0/1352657322 conn(0x5558b6d8f870 msgr2=0x5558b6db92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 shutdown_connections 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 1 -- 172.21.15.72:0/1352657322 wait complete. 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61753bf740 20 client.8808 trim_cache size 0 max 16384 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "0927e84a-942e-4592-98d9-697c1cf6243f", 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 8640, 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 9.1682476190000006, 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 79, 2024-03-06T07:28:46.972 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 127, 2024-03-06T07:28:46.973 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 127, 2024-03-06T07:28:46.973 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 13.507390195999999 2024-03-06T07:28:46.973 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:28:46.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f615806b4d0 msgr2=0x7f615806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f615806b4d0 0x7f615806d990 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f616c0b2d20 tx=0x7f6160008040 comp rx=0 tx=0).stop 2024-03-06T07:28:46.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 msgr2=0x7f616c0c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 0x7f616c0c83b0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c7f0 tx=0x7f615c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 shutdown_connections 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f615806b4d0 0x7f615806d990 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f616c162970 0x7f616c0c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f616c161cf0 0x7f616c0b96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 --2- 172.21.15.72:0/1219021223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f616c0b4460 0x7f616c0c83b0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 >> 172.21.15.72:0/1219021223 conn(0x7f616c0d3ce0 msgr2=0x7f616c06cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 shutdown_connections 2024-03-06T07:28:46.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:46.964+0000 7f61727d7640 1 -- 172.21.15.72:0/1219021223 wait complete. 2024-03-06T07:28:46.977 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-06T07:28:47.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.078+0000 7fe0d1cf4640 1 Processor -- start 2024-03-06T07:28:47.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.078+0000 7fe0d1cf4640 1 -- start start 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0d1cf4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0cc0d22b0 0x7fe0cc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0d1cf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0b41f0 0x7fe0cc0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0d1cf4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0cc0b3430 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0d1cf4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0cc0d2c50 con 0x7fe0cc0d22b0 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0d1cf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0b31e0 con 0x7fe0cc0b41f0 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0caffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0cc0d22b0 0x7fe0cc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0caffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0cc0d22b0 0x7fe0cc0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:34994/0 (socket says 172.21.15.72:34994) 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0caffd640 1 -- 172.21.15.72:0/3065553257 learned_addr learned my addr 172.21.15.72:0/3065553257 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0caffd640 1 -- 172.21.15.72:0/3065553257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0b41f0 msgr2=0x7fe0cc0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.079+0000 7fe0cbfff640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0b41f0 0x7fe0cc0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.080+0000 7fe0caffd640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0b41f0 0x7fe0cc0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.080+0000 7fe0caffd640 1 -- 172.21.15.72:0/3065553257 >> v1:172.21.15.146:6789/0 conn(0x7fe0cc15dfe0 legacy=0x7fe0cc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.080+0000 7fe0caffd640 1 -- 172.21.15.72:0/3065553257 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0b3710 con 0x7fe0cc0d22b0 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.080+0000 7fe0cbfff640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0b41f0 0x7fe0cc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:47.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.080+0000 7fe0caffd640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc0d22b0 0x7fe0cc0d2710 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009870 tx=0x7fe0bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab09e18eff4a1cf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.082+0000 7fe0ca7fc640 1 -- 172.21.15.72:0/3065553257 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc035020 con 0x7fe0cc0d22b0 2024-03-06T07:28:47.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.082+0000 7fe0ca7fc640 1 -- 172.21.15.72:0/3065553257 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc031640 con 0x7fe0cc0d22b0 2024-03-06T07:28:47.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.082+0000 7fe0ca7fc640 1 -- 172.21.15.72:0/3065553257 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0bc02d040 con 0x7fe0cc0d22b0 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/3065553257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc0d22b0 msgr2=0x7fe0cc0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc0d22b0 0x7fe0cc0d2710 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009870 tx=0x7fe0bc02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/3065553257 shutdown_connections 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0b41f0 0x7fe0cc0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/3065553257 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc0d22b0 0x7fe0cc0d2710 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/3065553257 >> 172.21.15.72:0/3065553257 conn(0x7fe0cc0d3cd0 msgr2=0x7fe0cc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/3065553257 shutdown_connections 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.083+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/3065553257 wait complete. 2024-03-06T07:28:47.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.084+0000 7fe0d1cf4640 1 Processor -- start 2024-03-06T07:28:47.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 -- start start 2024-03-06T07:28:47.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0cc0b41f0 0x7fe0cc161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0d22b0 0x7fe0cc0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 0x7fe0cc1f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0cfa00 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0b3d60 con 0x7fe0cc0b41f0 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0d1cf4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0cc0d1c70 con 0x7fe0cc0d22b0 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0cbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 0x7fe0cc1f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0cbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 0x7fe0cc1f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35004/0 (socket says 172.21.15.72:35004) 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0cbfff640 1 -- 172.21.15.72:0/4091955526 learned_addr learned my addr 172.21.15.72:0/4091955526 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0caffd640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0d22b0 0x7fe0cc0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.085+0000 7fe0cb7fe640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0cc0b41f0 0x7fe0cc161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0cbfff640 1 -- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0d22b0 msgr2=0x7fe0cc0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0cbfff640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0d22b0 0x7fe0cc0b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0cbfff640 1 -- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0cc0b41f0 msgr2=0x7fe0cc161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0cbfff640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0cc0b41f0 0x7fe0cc161260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0cbfff640 1 -- 172.21.15.72:0/4091955526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc0b6de0 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0cbfff640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 0x7fe0cc1f3240 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400cef0 tx=0x7fe0c400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c401a070 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c400d860 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0cc0b97f0 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c400dc50 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.086+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0cc0cc800 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.087+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0cc04f330 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.087+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe0c401d070 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.088+0000 7fe0b3fff640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe0b806b4d0 0x7fe0b806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:47.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.088+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0c408aa20 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.088+0000 7fe0cb7fe640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe0b806b4d0 0x7fe0b806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:47.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.088+0000 7fe0cb7fe640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe0b806b4d0 0x7fe0b806d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0006fd0 tx=0x7fe0c0009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:47.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.092+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe0c4059310 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.345+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe0cc1f3780 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.346+0000 7fe0b3fff640 1 -- 172.21.15.72:0/4091955526 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v79) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe0c405d3b0 con 0x7fe0cc15dfe0 2024-03-06T07:28:47.348 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-06T07:28:47.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe0b806b4d0 msgr2=0x7fe0b806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe0b806b4d0 0x7fe0b806d990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0006fd0 tx=0x7fe0c0009060 comp rx=0 tx=0).stop 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 msgr2=0x7fe0cc1f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 0x7fe0cc1f3240 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400cef0 tx=0x7fe0c400a710 comp rx=0 tx=0).stop 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 shutdown_connections 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe0b806b4d0 0x7fe0b806d990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0cc15dfe0 0x7fe0cc1f3240 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0cc0d22b0 0x7fe0cc0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 --2- 172.21.15.72:0/4091955526 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0cc0b41f0 0x7fe0cc161260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 >> 172.21.15.72:0/4091955526 conn(0x7fe0cc0d3cd0 msgr2=0x7fe0cc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 shutdown_connections 2024-03-06T07:28:47.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:47.348+0000 7fe0d1cf4640 1 -- 172.21.15.72:0/4091955526 wait complete. 2024-03-06T07:28:47.361 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-06T07:28:47.361 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-06T07:28:47.361 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-03-06T07:28:47.362 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-06T07:28:47.362 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:47.362 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-06T07:28:47.362 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-06T07:28:47.362 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-06T07:28:47.362 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:47.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.362 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-06T07:28:47.378 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42268sec preferred_lft 42268sec 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::e8e9:65ff:fecc:2943/64 scope link 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 16:de:a0:e0:13:fe brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:47.379 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::14de:a0ff:fee0:13fe/64 scope link 2024-03-06T07:28:47.380 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.380 INFO:teuthology.orchestra.run.smithi072.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-06T07:28:47.380 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether f2:02:5c:90:0a:45 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:47.380 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::f002:5cff:fe90:a45/64 scope link 2024-03-06T07:28:47.380 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-03-06T07:28:47.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.380 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-06T07:28:47.443 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-03-06T07:28:47.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.445 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-03-06T07:28:47.500 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-03-06T07:28:47.501 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-06T07:28:47.501 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-06T07:28:47.501 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-06T07:28:47.501 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-06T07:28:47.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.501 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec bash -c ' 2024-03-06T07:28:47.501 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-03-06T07:28:47.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-06T07:28:47.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-06T07:28:47.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-06T07:28:47.502 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:28:47.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.596 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:28:47.611 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:28:47.611 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:28:47.611 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:47.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.611 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-03-06T07:28:47.681 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-06T07:28:47.682 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:47.682 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:47.736 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-06T07:28:47.736 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-06T07:28:47.737 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:47.791 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:28:47.857 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:28:47.913 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:28:47.914 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:47.915 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:47.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:47.916 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:28:47.971 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-06T07:28:47.971 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-06T07:28:48.013 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-03-06T07:28:48.040 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-03-06T07:28:48.096 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-06T07:28:48.097 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:28:48.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.098 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-03-06T07:28:48.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-03-06T07:28:48.098+0000 7fae97eb3480 -1 init, newargv = 0x55695b22c440 newargc=15 2024-03-06T07:28:48.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[78448]: starting ceph client 2024-03-06T07:28:48.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[78448]: starting fuse 2024-03-06T07:28:48.114 INFO:teuthology.orchestra.run.smithi072.stdout:41 2024-03-06T07:28:48.115 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-06T07:28:48.115 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:28:48.116 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-03-06T07:28:48.117 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:28:48.234 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.78448.asok 2024-03-06T07:28:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:28:48.244 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78448.asok 2024-03-06T07:28:48.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.245 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78448.asok status 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd", 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1739-g1bc250c7 (1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd) squid (dev)", 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "78448", 2024-03-06T07:28:48.380 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 8823, 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 8823 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3583430901 2024-03-06T07:28:48.381 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3583430901 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.8823 192.168.0.1:0/3583430901", 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/3583430901", 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 79, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 127, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-03-06T07:28:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:28:48.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.388 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-03-06T07:28:48.388 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.404 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-03-06T07:28:48.404 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:48.404 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-06T07:28:48.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-03-06T07:28:48.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 Processor -- start 2024-03-06T07:28:48.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 -- start start 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff59c0d4860 0x7ff59c0d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c0b0e30 0x7ff59c0b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c1618b0 con 0x7ff59c1614d0 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59c0d4cf0 con 0x7ff59c0d4860 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.614+0000 7ff5a31f6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0cf400 con 0x7ff59c0b0e30 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff59bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff59c0d4860 0x7ff59c0d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff5a0f92640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff59c1614d0 0x7ff59c0d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58068/0 (socket says 172.21.15.72:58068) 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff5a0f92640 1 -- 172.21.15.72:0/1324679105 learned_addr learned my addr 172.21.15.72:0/1324679105 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff5a1793640 1 --2- 172.21.15.72:0/1324679105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c0b0e30 0x7ff59c0b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff59bfff640 1 -- 172.21.15.72:0/1324679105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c0b0e30 msgr2=0x7ff59c0b12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff59bfff640 1 --2- 172.21.15.72:0/1324679105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c0b0e30 0x7ff59c0b12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff59bfff640 1 -- 172.21.15.72:0/1324679105 >> v1:172.21.15.72:6789/0 conn(0x7ff59c1614d0 legacy=0x7ff59c0d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff59bfff640 1 -- 172.21.15.72:0/1324679105 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0cf6a0 con 0x7ff59c0d4860 2024-03-06T07:28:48.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.615+0000 7ff59b7fe640 1 -- 172.21.15.72:0/1324679105 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893575096 0 0) 0x7ff59c1618b0 con 0x7ff59c1614d0 2024-03-06T07:28:48.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff59bfff640 1 --2- 172.21.15.72:0/1324679105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0d4860 0x7ff59c0d2020 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff5900097a0 tx=0x7ff59002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cf8e1861b804df server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff59b7fe640 1 -- 172.21.15.72:0/1324679105 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590035020 con 0x7ff59c0d4860 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff59b7fe640 1 -- 172.21.15.72:0/1324679105 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590031830 con 0x7ff59c0d4860 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff59b7fe640 1 -- 172.21.15.72:0/1324679105 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590031c00 con 0x7ff59c0d4860 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 -- 172.21.15.72:0/1324679105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0d4860 msgr2=0x7ff59c0d2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/1324679105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0d4860 0x7ff59c0d2020 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff5900097a0 tx=0x7ff59002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 -- 172.21.15.72:0/1324679105 shutdown_connections 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/1324679105 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c0b0e30 0x7ff59c0b12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/1324679105 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0d4860 0x7ff59c0d2020 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 -- 172.21.15.72:0/1324679105 >> 172.21.15.72:0/1324679105 conn(0x7ff59c0d3d40 msgr2=0x7ff59c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 -- 172.21.15.72:0/1324679105 shutdown_connections 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.616+0000 7ff5a31f6640 1 -- 172.21.15.72:0/1324679105 wait complete. 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 Processor -- start 2024-03-06T07:28:48.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 -- start start 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 0x7ff59c15de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff59c0d4860 0x7ff59c0c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c1614d0 0x7ff59c0b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0b0910 con 0x7ff59c0b0e30 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0cfc60 con 0x7ff59c0d4860 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.618+0000 7ff5a31f6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c0d2560 con 0x7ff59c1614d0 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 0x7ff59c15de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 0x7ff59c15de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35030/0 (socket says 172.21.15.72:35030) 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 -- 172.21.15.72:0/821412571 learned_addr learned my addr 172.21.15.72:0/821412571 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a1793640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c1614d0 0x7ff59c0b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff59bfff640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff59c0d4860 0x7ff59c0c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 -- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c1614d0 msgr2=0x7ff59c0b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c1614d0 0x7ff59c0b8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 -- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff59c0d4860 msgr2=0x7ff59c0c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff59c0d4860 0x7ff59c0c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 -- 172.21.15.72:0/821412571 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c1f3b50 con 0x7ff59c0b0e30 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a1793640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c1614d0 0x7ff59c0b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff59bfff640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff59c0d4860 0x7ff59c0c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a0f92640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 0x7ff59c15de10 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00c430 tx=0x7ff58c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0074e0 con 0x7ff59c0b0e30 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c0047e0 con 0x7ff59c0b0e30 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c0cab90 con 0x7ff59c0b0e30 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c004bd0 con 0x7ff59c0b0e30 2024-03-06T07:28:48.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.619+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c0cc420 con 0x7ff59c0b0e30 2024-03-06T07:28:48.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.620+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c068ff0 con 0x7ff59c0b0e30 2024-03-06T07:28:48.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.621+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff58c011070 con 0x7ff59c0b0e30 2024-03-06T07:28:48.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.621+0000 7ff5997fa640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff58806b4d0 0x7ff58806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:48.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.621+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff58c089e70 con 0x7ff59c0b0e30 2024-03-06T07:28:48.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.621+0000 7ff59bfff640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff58806b4d0 0x7ff58806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:48.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.621+0000 7ff59bfff640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff58806b4d0 0x7ff58806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff590036000 tx=0x7ff590008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:48.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.625+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff58c058cf0 con 0x7ff59c0b0e30 2024-03-06T07:28:48.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:48.912+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7ff59c0d2060 con 0x7ff59c0b0e30 2024-03-06T07:28:49.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.090+0000 7ff5997fa640 1 -- 172.21.15.72:0/821412571 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v80) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff58c0585f0 con 0x7ff59c0b0e30 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff58806b4d0 msgr2=0x7ff58806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff58806b4d0 0x7ff58806d990 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff590036000 tx=0x7ff590008030 comp rx=0 tx=0).stop 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 msgr2=0x7ff59c15de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 0x7ff59c15de10 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00c430 tx=0x7ff58c00c8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 shutdown_connections 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff58806b4d0 0x7ff58806d990 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff59c1614d0 0x7ff59c0b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff59c0d4860 0x7ff59c0c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 --2- 172.21.15.72:0/821412571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff59c0b0e30 0x7ff59c15de10 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 >> 172.21.15.72:0/821412571 conn(0x7ff59c0d3d40 msgr2=0x7ff59c0b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 shutdown_connections 2024-03-06T07:28:49.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.093+0000 7ff5a31f6640 1 -- 172.21.15.72:0/821412571 wait complete. 2024-03-06T07:28:49.112 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:49.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.214+0000 7ff8a730d640 1 Processor -- start 2024-03-06T07:28:49.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.215+0000 7ff8a730d640 1 -- start start 2024-03-06T07:28:49.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.215+0000 7ff8a730d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8a015dfe0 0x7ff8a00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.215+0000 7ff8a730d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00b3430 con 0x7ff8a00d29c0 2024-03-06T07:28:49.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.215+0000 7ff8a730d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00b3710 con 0x7ff8a00cf830 2024-03-06T07:28:49.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.215+0000 7ff8a730d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a00b31e0 con 0x7ff8a015dfe0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff8a58aa640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff8a00d29c0 0x7ff8a00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58070/0 (socket says 172.21.15.72:58070) 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff8a58aa640 1 -- 172.21.15.72:0/103671056 learned_addr learned my addr 172.21.15.72:0/103671056 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff8a50a9640 1 --2- 172.21.15.72:0/103671056 >> v2:172.21.15.146:3300/0 conn(0x7ff8a015dfe0 0x7ff8a00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774196664 0 0) 0x7ff8a00b3430 con 0x7ff8a00d29c0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88c003520 con 0x7ff8a00d29c0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2465115720 0 0) 0x7ff88c003520 con 0x7ff8a00d29c0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 >> v2:172.21.15.146:3300/0 conn(0x7ff8a015dfe0 msgr2=0x7ff8a00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 --2- 172.21.15.72:0/103671056 >> v2:172.21.15.146:3300/0 conn(0x7ff8a015dfe0 0x7ff8a00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 >> v1:172.21.15.146:6789/0 conn(0x7ff8a00cf830 legacy=0x7ff8a00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00b39f0 con 0x7ff8a00d29c0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff89c001670 con 0x7ff8a00d29c0 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 >> v1:172.21.15.72:6789/0 conn(0x7ff8a00d29c0 legacy=0x7ff8a00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c0039b0 0x7ff88c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff88c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c0051c0 0x7ff88c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00b31e0 con 0x7ff88c004800 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c00c090 con 0x7ff88c0051c0 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c00c290 con 0x7ff88c0039b0 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff88c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff8a50a9640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c0039b0 0x7ff88c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.216+0000 7ff8a58aa640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c0051c0 0x7ff88c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a48a8640 1 -- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c0039b0 msgr2=0x7ff88c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c0039b0 0x7ff88c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a48a8640 1 -- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c0051c0 msgr2=0x7ff88c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:49.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff88c0051c0 0x7ff88c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a48a8640 1 -- 172.21.15.72:0/103671056 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c003520 con 0x7ff88c004800 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a58aa640 1 -- 172.21.15.72:0/103671056 reap_dead start 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a50a9640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff88c0039b0 0x7ff88c003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff88c004c80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7ff890001720 tx=0x7ff88c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=833521f69dbf46e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890001370 con 0x7ff88c004800 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8900052d0 con 0x7ff88c004800 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.217+0000 7ff897fff640 1 -- 172.21.15.72:0/103671056 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890003830 con 0x7ff88c004800 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 -- 172.21.15.72:0/103671056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 msgr2=0x7ff88c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff88c004c80 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7ff890001720 tx=0x7ff88c005830 comp rx=0 tx=0).stop 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 -- 172.21.15.72:0/103671056 shutdown_connections 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 --2- 172.21.15.72:0/103671056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff88c004c80 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 -- 172.21.15.72:0/103671056 >> 172.21.15.72:0/103671056 conn(0x7ff8a00d3cd0 msgr2=0x7ff8a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 -- 172.21.15.72:0/103671056 shutdown_connections 2024-03-06T07:28:49.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.218+0000 7ff8a730d640 1 -- 172.21.15.72:0/103671056 wait complete. 2024-03-06T07:28:49.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.219+0000 7ff8a730d640 1 Processor -- start 2024-03-06T07:28:49.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.219+0000 7ff8a730d640 1 -- start start 2024-03-06T07:28:49.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a730d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff8a01f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a730d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a00b7710 0x7ff8a0160ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a730d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a015c160 0x7ff8a00b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a730d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015e3c0 con 0x7ff88c004800 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a730d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a00b3d60 con 0x7ff8a015c160 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a730d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a015e540 con 0x7ff8a00b7710 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff8a01f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff8a01f3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35044/0 (socket says 172.21.15.72:35044) 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 -- 172.21.15.72:0/665327281 learned_addr learned my addr 172.21.15.72:0/665327281 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a58aa640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a015c160 0x7ff8a00b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a00b7710 0x7ff8a0160ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 -- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a00b7710 msgr2=0x7ff8a0160ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a00b7710 0x7ff8a0160ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 -- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a015c160 msgr2=0x7ff8a00b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a015c160 0x7ff8a00b81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a50a9640 1 -- 172.21.15.72:0/665327281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a015e850 con 0x7ff88c004800 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.220+0000 7ff8a58aa640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a015c160 0x7ff8a00b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.221+0000 7ff8a50a9640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff8a01f3fa0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7ff898009f90 tx=0x7ff898010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.221+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89801a030 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.221+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898003990 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.221+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898003d60 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.221+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a0161310 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.221+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a015edd0 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.222+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a004f330 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.222+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff89801d070 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.223+0000 7ff895ffb640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff88406b500 0x7ff88406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.223+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff89808af20 con 0x7ff88c004800 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.223+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff88406b500 0x7ff88406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:49.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.223+0000 7ff8a48a8640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff88406b500 0x7ff88406d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff890001720 tx=0x7ff890009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:49.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.231+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff89801de40 con 0x7ff88c004800 2024-03-06T07:28:49.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.461+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8a00b09c0 con 0x7ff88c004800 2024-03-06T07:28:49.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.462+0000 7ff895ffb640 1 -- 172.21.15.72:0/665327281 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4998 (secure 0 0 0) 0x7ff898059e30 con 0x7ff88c004800 2024-03-06T07:28:49.464 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:49.464 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":81,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2751164744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2751164744},{"type":"v1","addr":"172.21.15.146:6834","nonce":2751164744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/406165267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":406165267},{"type":"v1","addr":"172.21.15.146:6835","nonce":406165267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:49.094055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:creating","state_seq":1,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:28:49.464 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 81 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff88406b500 msgr2=0x7ff88406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff88406b500 0x7ff88406d9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff890001720 tx=0x7ff890009030 comp rx=0 tx=0).stop 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 msgr2=0x7ff8a01f3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff8a01f3fa0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7ff898009f90 tx=0x7ff898010590 comp rx=0 tx=0).stop 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 shutdown_connections 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff88406b500 0x7ff88406d9c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8a015c160 0x7ff8a00b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8a00b7710 0x7ff8a0160ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 --2- 172.21.15.72:0/665327281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff88c004800 0x7ff8a01f3fa0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:49.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 >> 172.21.15.72:0/665327281 conn(0x7ff8a00d3cd0 msgr2=0x7ff8a006cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:49.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 shutdown_connections 2024-03-06T07:28:49.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:49.464+0000 7ff8a730d640 1 -- 172.21.15.72:0/665327281 wait complete. 2024-03-06T07:28:49.477 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-03-06T07:28:51.448 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:51.451 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:28:53.785 INFO:tasks.ceph.mds.a.smithi072.stderr:2024-03-06T07:28:53.784+0000 7fd7429fd640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-06T07:28:54.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.577+0000 7f361ce8f640 1 Processor -- start 2024-03-06T07:28:54.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.578+0000 7f361ce8f640 1 -- start start 2024-03-06T07:28:54.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.578+0000 7f361ce8f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f36180d22b0 0x7f36180d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.578+0000 7f361ce8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180b41f0 0x7f36180b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.578+0000 7f361ce8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36180b3430 con 0x7f361815dfe0 2024-03-06T07:28:54.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.578+0000 7f361ce8f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36180d2c50 con 0x7f36180d22b0 2024-03-06T07:28:54.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.578+0000 7f361ce8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36180b31e0 con 0x7f36180b41f0 2024-03-06T07:28:54.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f36180d22b0 0x7f36180d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f36180d22b0 0x7f36180d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49184/0 (socket says 172.21.15.72:49184) 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 -- 172.21.15.72:0/4035252099 learned_addr learned my addr 172.21.15.72:0/4035252099 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3616d9d640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180b41f0 0x7f36180b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 -- 172.21.15.72:0/4035252099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180b41f0 msgr2=0x7f36180b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180b41f0 0x7f36180b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 -- 172.21.15.72:0/4035252099 >> v1:172.21.15.146:6789/0 conn(0x7f361815dfe0 legacy=0x7f36180cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3615d9b640 1 -- 172.21.15.72:0/4035252099 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180b3710 con 0x7f36180d22b0 2024-03-06T07:28:54.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.579+0000 7f3616d9d640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180b41f0 0x7f36180b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:54.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.580+0000 7f3615d9b640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180d22b0 0x7f36180d2710 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f360c009870 tx=0x7f360c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262187a0ae19004e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.580+0000 7f361559a640 1 -- 172.21.15.72:0/4035252099 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c035020 con 0x7f36180d22b0 2024-03-06T07:28:54.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.580+0000 7f361559a640 1 -- 172.21.15.72:0/4035252099 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c031640 con 0x7f36180d22b0 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.580+0000 7f361559a640 1 -- 172.21.15.72:0/4035252099 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c02d040 con 0x7f36180d22b0 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 -- 172.21.15.72:0/4035252099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180d22b0 msgr2=0x7f36180d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180d22b0 0x7f36180d2710 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f360c009870 tx=0x7f360c02b8f0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 -- 172.21.15.72:0/4035252099 shutdown_connections 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180b41f0 0x7f36180b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 --2- 172.21.15.72:0/4035252099 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180d22b0 0x7f36180d2710 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 -- 172.21.15.72:0/4035252099 >> 172.21.15.72:0/4035252099 conn(0x7f36180d3cd0 msgr2=0x7f36180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 -- 172.21.15.72:0/4035252099 shutdown_connections 2024-03-06T07:28:54.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.581+0000 7f361ce8f640 1 -- 172.21.15.72:0/4035252099 wait complete. 2024-03-06T07:28:54.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.583+0000 7f361ce8f640 1 Processor -- start 2024-03-06T07:28:54.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.583+0000 7f361ce8f640 1 -- start start 2024-03-06T07:28:54.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361ce8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 0x7f3618161260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361ce8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180d22b0 0x7f36180b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361ce8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f361815dfe0 0x7f36181f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361ce8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36180cfa00 con 0x7f36180b41f0 2024-03-06T07:28:54.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361ce8f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36180b3d60 con 0x7f361815dfe0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361ce8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36180d1c70 con 0x7f36180d22b0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 0x7f3618161260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 0x7f3618161260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49192/0 (socket says 172.21.15.72:49192) 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 -- 172.21.15.72:0/3267635551 learned_addr learned my addr 172.21.15.72:0/3267635551 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f3616d9d640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f361815dfe0 0x7f36181f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f3615d9b640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180d22b0 0x7f36180b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 -- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180d22b0 msgr2=0x7f36180b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180d22b0 0x7f36180b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 -- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f361815dfe0 msgr2=0x7f36181f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f361815dfe0 0x7f36181f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.584+0000 7f361659c640 1 -- 172.21.15.72:0/3267635551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36180b6de0 con 0x7f36180b41f0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.585+0000 7f361659c640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 0x7f3618161260 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f360800b3f0 tx=0x7f360800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.585+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608015020 con 0x7f36180b41f0 2024-03-06T07:28:54.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.585+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608011830 con 0x7f36180b41f0 2024-03-06T07:28:54.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.585+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608011c20 con 0x7f36180b41f0 2024-03-06T07:28:54.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.585+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36180b97f0 con 0x7f36180b41f0 2024-03-06T07:28:54.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.585+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36180cc800 con 0x7f36180b41f0 2024-03-06T07:28:54.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.586+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361804f330 con 0x7f36180b41f0 2024-03-06T07:28:54.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.586+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3608018020 con 0x7f36180b41f0 2024-03-06T07:28:54.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.587+0000 7f3606ffd640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f35f806b4d0 0x7f35f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.587+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3608089e90 con 0x7f36180b41f0 2024-03-06T07:28:54.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.587+0000 7f3615d9b640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f35f806b4d0 0x7f35f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.587+0000 7f3615d9b640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f35f806b4d0 0x7f35f806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f360c036000 tx=0x7f360c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.591+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f360801f030 con 0x7f36180b41f0 2024-03-06T07:28:54.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.805+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36181f3780 con 0x7f36180b41f0 2024-03-06T07:28:54.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.805+0000 7f3606ffd640 1 -- 172.21.15.72:0/3267635551 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f36080172c0 con 0x7f36180b41f0 2024-03-06T07:28:54.807 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:54.807 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2751164744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2751164744},{"type":"v1","addr":"172.21.15.146:6834","nonce":2751164744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/406165267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":406165267},{"type":"v1","addr":"172.21.15.146:6835","nonce":406165267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:50.096107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8637,8640]},"id":9}]} 2024-03-06T07:28:54.808 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 82 2024-03-06T07:28:54.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f35f806b4d0 msgr2=0x7f35f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f35f806b4d0 0x7f35f806d990 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f360c036000 tx=0x7f360c008030 comp rx=0 tx=0).stop 2024-03-06T07:28:54.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 msgr2=0x7f3618161260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 0x7f3618161260 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f360800b3f0 tx=0x7f360800b8b0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 shutdown_connections 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f35f806b4d0 0x7f35f806d990 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f361815dfe0 0x7f36181f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36180d22b0 0x7f36180b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 --2- 172.21.15.72:0/3267635551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36180b41f0 0x7f3618161260 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 >> 172.21.15.72:0/3267635551 conn(0x7f36180d3cd0 msgr2=0x7f361806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 shutdown_connections 2024-03-06T07:28:54.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.807+0000 7f361ce8f640 1 -- 172.21.15.72:0/3267635551 wait complete. 2024-03-06T07:28:54.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:54.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.920+0000 7f7d493e9640 1 Processor -- start 2024-03-06T07:28:54.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.920+0000 7f7d493e9640 1 -- start start 2024-03-06T07:28:54.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d493e9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d440d16e0 0x7f7d440d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d493e9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d440cebe0 con 0x7f7d4415bdd0 2024-03-06T07:28:54.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d493e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d440ceec0 con 0x7f7d4415fa10 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d493e9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d440ce980 con 0x7f7d440d16e0 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d437fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d440d16e0 0x7f7d440d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d437fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7d440d16e0 0x7f7d440d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49206/0 (socket says 172.21.15.72:49206) 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d437fe640 1 -- 172.21.15.72:0/1671258002 learned_addr learned my addr 172.21.15.72:0/1671258002 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d42ffd640 1 --1- 172.21.15.72:0/1671258002 >> v1:172.21.15.72:6789/0 conn(0x7f7d4415bdd0 0x7f7d440c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49272/0 (socket says 172.21.15.72:49272) 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d437fe640 1 -- 172.21.15.72:0/1671258002 >> v1:172.21.15.146:6789/0 conn(0x7f7d4415fa10 legacy=0x7f7d440ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d437fe640 1 -- 172.21.15.72:0/1671258002 >> v1:172.21.15.72:6789/0 conn(0x7f7d4415bdd0 legacy=0x7f7d440c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.921+0000 7f7d437fe640 1 -- 172.21.15.72:0/1671258002 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440cf1a0 con 0x7f7d440d16e0 2024-03-06T07:28:54.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.922+0000 7f7d437fe640 1 --2- 172.21.15.72:0/1671258002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d440d16e0 0x7f7d440d1af0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009e50 tx=0x7f7d3c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9ba9c522df8190 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.922+0000 7f7d427fc640 1 -- 172.21.15.72:0/1671258002 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c02ca80 con 0x7f7d440d16e0 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.922+0000 7f7d427fc640 1 -- 172.21.15.72:0/1671258002 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c02cc20 con 0x7f7d440d16e0 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.922+0000 7f7d427fc640 1 -- 172.21.15.72:0/1671258002 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c0343f0 con 0x7f7d440d16e0 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 -- 172.21.15.72:0/1671258002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d440d16e0 msgr2=0x7f7d440d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1671258002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d440d16e0 0x7f7d440d1af0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009e50 tx=0x7f7d3c02bee0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 -- 172.21.15.72:0/1671258002 shutdown_connections 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1671258002 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d440d16e0 0x7f7d440d1af0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 -- 172.21.15.72:0/1671258002 >> 172.21.15.72:0/1671258002 conn(0x7f7d4409a9f0 msgr2=0x7f7d4409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 -- 172.21.15.72:0/1671258002 shutdown_connections 2024-03-06T07:28:54.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.923+0000 7f7d493e9640 1 -- 172.21.15.72:0/1671258002 wait complete. 2024-03-06T07:28:54.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.924+0000 7f7d493e9640 1 Processor -- start 2024-03-06T07:28:54.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.924+0000 7f7d493e9640 1 -- start start 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.925+0000 7f7d493e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d440d16e0 0x7f7d4415b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.925+0000 7f7d493e9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d4415bdd0 0x7f7d440cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.925+0000 7f7d493e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 0x7f7d4415e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.925+0000 7f7d493e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4415c1b0 con 0x7f7d4415fa10 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.925+0000 7f7d493e9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440cc5d0 con 0x7f7d4415bdd0 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.925+0000 7f7d493e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d440d2030 con 0x7f7d440d16e0 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 0x7f7d4415e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 0x7f7d4415e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49218/0 (socket says 172.21.15.72:49218) 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 -- 172.21.15.72:0/1108497271 learned_addr learned my addr 172.21.15.72:0/1108497271 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:54.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d42ffd640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d440d16e0 0x7f7d4415b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d3bfff640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d4415bdd0 0x7f7d440cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 -- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d440d16e0 msgr2=0x7f7d4415b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d440d16e0 0x7f7d4415b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 -- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d4415bdd0 msgr2=0x7f7d440cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d4415bdd0 0x7f7d440cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 -- 172.21.15.72:0/1108497271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440cb580 con 0x7f7d4415fa10 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d437fe640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 0x7f7d4415e930 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c036040 tx=0x7f7d3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c043020 con 0x7f7d4415fa10 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c03e450 con 0x7f7d4415fa10 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d440cb860 con 0x7f7d4415fa10 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c03e820 con 0x7f7d4415fa10 2024-03-06T07:28:54.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.926+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4415c330 con 0x7f7d4415fa10 2024-03-06T07:28:54.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.927+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4404f330 con 0x7f7d4415fa10 2024-03-06T07:28:54.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.928+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7d3c046020 con 0x7f7d4415fa10 2024-03-06T07:28:54.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.928+0000 7f7d3b7fe640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d2406b4d0 0x7f7d2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:54.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.929+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d3c0aca90 con 0x7f7d4415fa10 2024-03-06T07:28:54.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.929+0000 7f7d42ffd640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d2406b4d0 0x7f7d2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:54.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.929+0000 7f7d42ffd640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d2406b4d0 0x7f7d2406d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d34007fd0 tx=0x7f7d34007c40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:54.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:54.933+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7d3c04d030 con 0x7f7d4415fa10 2024-03-06T07:28:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.148+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d440b7a40 con 0x7f7d4415fa10 2024-03-06T07:28:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.148+0000 7f7d3b7fe640 1 -- 172.21.15.72:0/1108497271 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5001 (secure 0 0 0) 0x7f7d3c07b9a0 con 0x7f7d4415fa10 2024-03-06T07:28:55.150 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:55.150 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":82,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6566,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6834/2751164744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2751164744},{"type":"v1","addr":"172.21.15.146:6834","nonce":2751164744}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/406165267","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6833","nonce":406165267},{"type":"v1","addr":"172.21.15.146:6835","nonce":406165267}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:50.096107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8637,8640]},"id":9}]} 2024-03-06T07:28:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 82 2024-03-06T07:28:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d2406b4d0 msgr2=0x7f7d2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d2406b4d0 0x7f7d2406d990 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d34007fd0 tx=0x7f7d34007c40 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 msgr2=0x7f7d4415e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 0x7f7d4415e930 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c036040 tx=0x7f7d3c004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 shutdown_connections 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7d2406b4d0 0x7f7d2406d990 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d4415fa10 0x7f7d4415e930 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d4415bdd0 0x7f7d440cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 --2- 172.21.15.72:0/1108497271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d440d16e0 0x7f7d4415b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 >> 172.21.15.72:0/1108497271 conn(0x7f7d4409a9f0 msgr2=0x7f7d440b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 shutdown_connections 2024-03-06T07:28:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.150+0000 7f7d493e9640 1 -- 172.21.15.72:0/1108497271 wait complete. 2024-03-06T07:28:55.163 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-03-06T07:28:55.163 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-06T07:28:55.163 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:55.175 INFO:tasks.ceph.mds.b:Stopped 2024-03-06T07:28:55.175 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-03-06T07:28:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.279+0000 7faa4826b640 1 Processor -- start 2024-03-06T07:28:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.279+0000 7faa4826b640 1 -- start start 2024-03-06T07:28:55.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa4826b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa400b4460 0x7faa400d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa4826b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa40161cf0 0x7faa40162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa4826b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa400d22c0 con 0x7faa40162970 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa4826b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa40161b00 con 0x7faa40161cf0 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa4826b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa400cecc0 con 0x7faa400b4460 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa40161cf0 0x7faa40162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faa40161cf0 0x7faa40162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49222/0 (socket says 172.21.15.72:49222) 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 -- 172.21.15.72:0/2394704227 learned_addr learned my addr 172.21.15.72:0/2394704227 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa45806640 1 --1- 172.21.15.72:0/2394704227 >> v1:172.21.15.72:6789/0 conn(0x7faa40162970 0x7faa400b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49284/0 (socket says 172.21.15.72:49284) 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 -- 172.21.15.72:0/2394704227 >> v2:172.21.15.146:3300/0 conn(0x7faa400b4460 msgr2=0x7faa400d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 --2- 172.21.15.72:0/2394704227 >> v2:172.21.15.146:3300/0 conn(0x7faa400b4460 0x7faa400d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 -- 172.21.15.72:0/2394704227 >> v1:172.21.15.72:6789/0 conn(0x7faa40162970 legacy=0x7faa400b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.280+0000 7faa46808640 1 -- 172.21.15.72:0/2394704227 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400d25a0 con 0x7faa40161cf0 2024-03-06T07:28:55.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.281+0000 7faa46808640 1 --2- 172.21.15.72:0/2394704227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40161cf0 0x7faa40162180 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7faa4008ddd0 tx=0x7faa3802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d2a74844818f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.281+0000 7faa45005640 1 -- 172.21.15.72:0/2394704227 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3802c8d0 con 0x7faa40161cf0 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.281+0000 7faa45005640 1 -- 172.21.15.72:0/2394704227 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3802ca70 con 0x7faa40161cf0 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.281+0000 7faa45005640 1 -- 172.21.15.72:0/2394704227 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3802ce40 con 0x7faa40161cf0 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 -- 172.21.15.72:0/2394704227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40161cf0 msgr2=0x7faa40162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 --2- 172.21.15.72:0/2394704227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40161cf0 0x7faa40162180 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7faa4008ddd0 tx=0x7faa3802bdb0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 -- 172.21.15.72:0/2394704227 shutdown_connections 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 --2- 172.21.15.72:0/2394704227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40161cf0 0x7faa40162180 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 --2- 172.21.15.72:0/2394704227 >> v2:172.21.15.146:3300/0 conn(0x7faa400b4460 0x7faa400d4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 -- 172.21.15.72:0/2394704227 >> 172.21.15.72:0/2394704227 conn(0x7faa400d3ce0 msgr2=0x7faa400d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 -- 172.21.15.72:0/2394704227 shutdown_connections 2024-03-06T07:28:55.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.282+0000 7faa4826b640 1 -- 172.21.15.72:0/2394704227 wait complete. 2024-03-06T07:28:55.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.283+0000 7faa4826b640 1 Processor -- start 2024-03-06T07:28:55.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.283+0000 7faa4826b640 1 -- start start 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa4826b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa400b4460 0x7faa4015dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa4826b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa40161cf0 0x7faa400b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa4826b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 0x7faa400b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa4826b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa400ce950 con 0x7faa40162970 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa4826b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa400d2b60 con 0x7faa400b4460 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa4826b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa400b4840 con 0x7faa40161cf0 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 0x7faa400b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 0x7faa400b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49226/0 (socket says 172.21.15.72:49226) 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 -- 172.21.15.72:0/3950795380 learned_addr learned my addr 172.21.15.72:0/3950795380 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46007640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa400b4460 0x7faa4015dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 -- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa40161cf0 msgr2=0x7faa400b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa40161cf0 0x7faa400b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 -- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa400b4460 msgr2=0x7faa4015dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa400b4460 0x7faa4015dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 -- 172.21.15.72:0/3950795380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa400b1ae0 con 0x7faa40162970 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46007640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa400b4460 0x7faa4015dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.284+0000 7faa46808640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 0x7faa400b3050 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7faa38009f40 tx=0x7faa38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.285+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa38043020 con 0x7faa40162970 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.285+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3803e450 con 0x7faa40162970 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.285+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa400c87f0 con 0x7faa40162970 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.285+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3803e840 con 0x7faa40162970 2024-03-06T07:28:55.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.285+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa400c9090 con 0x7faa40162970 2024-03-06T07:28:55.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.286+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4004f330 con 0x7faa40162970 2024-03-06T07:28:55.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.286+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faa38046020 con 0x7faa40162970 2024-03-06T07:28:55.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.286+0000 7faa36ffd640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa2406b4d0 0x7faa2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:55.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.287+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa380acae0 con 0x7faa40162970 2024-03-06T07:28:55.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.287+0000 7faa46007640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa2406b4d0 0x7faa2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:55.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.287+0000 7faa46007640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa2406b4d0 0x7faa2406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00a870 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:55.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.295+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faa3804d030 con 0x7faa40162970 2024-03-06T07:28:55.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:55.674+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7faa4006b180 con 0x7faa40162970 2024-03-06T07:28:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.111+0000 7faa36ffd640 1 -- 172.21.15.72:0/3950795380 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6566 v83) v1 ==== 95+0+0 (secure 0 0 0) 0x7faa3807b9f0 con 0x7faa40162970 2024-03-06T07:28:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 6566 2024-03-06T07:28:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa2406b4d0 msgr2=0x7faa2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa2406b4d0 0x7faa2406d990 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00a870 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2024-03-06T07:28:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 msgr2=0x7faa400b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 0x7faa400b3050 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7faa38009f40 tx=0x7faa38004a40 comp rx=0 tx=0).stop 2024-03-06T07:28:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 shutdown_connections 2024-03-06T07:28:56.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faa2406b4d0 0x7faa2406d990 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa40162970 0x7faa400b3050 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa40161cf0 0x7faa400b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 --2- 172.21.15.72:0/3950795380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa400b4460 0x7faa4015dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 >> 172.21.15.72:0/3950795380 conn(0x7faa400d3ce0 msgr2=0x7faa4006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 shutdown_connections 2024-03-06T07:28:56.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.114+0000 7faa4826b640 1 -- 172.21.15.72:0/3950795380 wait complete. 2024-03-06T07:28:56.131 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-06T07:28:56.131 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:28:56.141 INFO:tasks.ceph.mds.d:Stopped 2024-03-06T07:28:56.141 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-03-06T07:28:56.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 Processor -- start 2024-03-06T07:28:56.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 -- start start 2024-03-06T07:28:56.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6c2815e000 0x7f6c280d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6c280d4810 0x7f6c280d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c280d25e0 con 0x7f6c280c80e0 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c280c7c50 con 0x7f6c280d4810 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c2d8c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c280d2880 con 0x7f6c2815e000 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c267fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6c280d4810 0x7f6c280d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c277fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6c280c80e0 0x7f6c280c8570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49300/0 (socket says 172.21.15.72:49300) 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c267fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6c280d4810 0x7f6c280d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49238/0 (socket says 172.21.15.72:49238) 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.267+0000 7f6c277fe640 1 -- 172.21.15.72:0/387740355 learned_addr learned my addr 172.21.15.72:0/387740355 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c26ffd640 1 --2- 172.21.15.72:0/387740355 >> v2:172.21.15.146:3300/0 conn(0x7f6c2815e000 0x7f6c280d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c267fc640 1 -- 172.21.15.72:0/387740355 >> v2:172.21.15.146:3300/0 conn(0x7f6c2815e000 msgr2=0x7f6c280d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c267fc640 1 --2- 172.21.15.72:0/387740355 >> v2:172.21.15.146:3300/0 conn(0x7f6c2815e000 0x7f6c280d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c267fc640 1 -- 172.21.15.72:0/387740355 >> v1:172.21.15.72:6789/0 conn(0x7f6c280c80e0 legacy=0x7f6c280c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c267fc640 1 -- 172.21.15.72:0/387740355 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c280d2b40 con 0x7f6c280d4810 2024-03-06T07:28:56.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c26ffd640 1 --2- 172.21.15.72:0/387740355 >> v2:172.21.15.146:3300/0 conn(0x7f6c2815e000 0x7f6c280d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:28:56.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.268+0000 7f6c267fc640 1 --2- 172.21.15.72:0/387740355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280d4810 0x7f6c280d0bc0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f6c2808c360 tx=0x7f6c1c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8bf72ddff25c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c25ffb640 1 -- 172.21.15.72:0/387740355 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c034070 con 0x7f6c280d4810 2024-03-06T07:28:56.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c25ffb640 1 -- 172.21.15.72:0/387740355 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c030610 con 0x7f6c280d4810 2024-03-06T07:28:56.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c25ffb640 1 -- 172.21.15.72:0/387740355 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c0309e0 con 0x7f6c280d4810 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/387740355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280d4810 msgr2=0x7f6c280d0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/387740355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280d4810 0x7f6c280d0bc0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f6c2808c360 tx=0x7f6c1c02b960 comp rx=0 tx=0).stop 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/387740355 shutdown_connections 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/387740355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280d4810 0x7f6c280d0bc0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/387740355 >> v2:172.21.15.146:3300/0 conn(0x7f6c2815e000 0x7f6c280d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/387740355 >> 172.21.15.72:0/387740355 conn(0x7f6c280d3cf0 msgr2=0x7f6c280d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/387740355 shutdown_connections 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.269+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/387740355 wait complete. 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.270+0000 7f6c2d8c9640 1 Processor -- start 2024-03-06T07:28:56.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 -- start start 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 0x7f6c280cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c280d4810 0x7f6c28154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c2815e000 0x7f6c2806c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2815e3e0 con 0x7f6c280c80e0 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280d22d0 con 0x7f6c280d4810 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c2d8c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c280cf7d0 con 0x7f6c2815e000 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c26ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 0x7f6c280cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c26ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 0x7f6c280cd910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49240/0 (socket says 172.21.15.72:49240) 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.271+0000 7f6c26ffd640 1 -- 172.21.15.72:0/2816936651 learned_addr learned my addr 172.21.15.72:0/2816936651 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c267fc640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c280d4810 0x7f6c28154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c277fe640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c2815e000 0x7f6c2806c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c26ffd640 1 -- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c2815e000 msgr2=0x7f6c2806c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c26ffd640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c2815e000 0x7f6c2806c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c26ffd640 1 -- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c280d4810 msgr2=0x7f6c28154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c26ffd640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c280d4810 0x7f6c28154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c26ffd640 1 -- 172.21.15.72:0/2816936651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c28154760 con 0x7f6c280c80e0 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c267fc640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c280d4810 0x7f6c28154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c26ffd640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 0x7f6c280cd910 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800c820 tx=0x7f6c1800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18015070 con 0x7f6c280c80e0 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c18007a30 con 0x7f6c280c80e0 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18007de0 con 0x7f6c280c80e0 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c280b3c90 con 0x7f6c280c80e0 2024-03-06T07:28:56.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c280b4250 con 0x7f6c280c80e0 2024-03-06T07:28:56.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.273+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2804f330 con 0x7f6c280c80e0 2024-03-06T07:28:56.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.274+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6c18018070 con 0x7f6c280c80e0 2024-03-06T07:28:56.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.274+0000 7f6c0ffff640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c1406b4d0 0x7f6c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:56.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.275+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c1808a0a0 con 0x7f6c280c80e0 2024-03-06T07:28:56.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.275+0000 7f6c267fc640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c1406b4d0 0x7f6c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:56.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.275+0000 7f6c267fc640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c1406b4d0 0x7f6c1406d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c036000 tx=0x7f6c1c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:56.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.280+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6c18058fb0 con 0x7f6c280c80e0 2024-03-06T07:28:56.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:56.552+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f6c2806d560 con 0x7f6c280c80e0 2024-03-06T07:28:57.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.120+0000 7f6c0ffff640 1 -- 172.21.15.72:0/2816936651 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7384 v84) v1 ==== 95+0+0 (secure 0 0 0) 0x7f6c180588b0 con 0x7f6c280c80e0 2024-03-06T07:28:57.122 INFO:teuthology.orchestra.run.smithi072.stderr:failed mds gid 7384 2024-03-06T07:28:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c1406b4d0 msgr2=0x7f6c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c1406b4d0 0x7f6c1406d990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c036000 tx=0x7f6c1c03b040 comp rx=0 tx=0).stop 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 msgr2=0x7f6c280cd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 0x7f6c280cd910 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f6c1800c820 tx=0x7f6c1800cce0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 shutdown_connections 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6c1406b4d0 0x7f6c1406d990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c2815e000 0x7f6c2806c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c280d4810 0x7f6c28154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 --2- 172.21.15.72:0/2816936651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6c280c80e0 0x7f6c280cd910 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 >> 172.21.15.72:0/2816936651 conn(0x7f6c280d3cf0 msgr2=0x7f6c280c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 shutdown_connections 2024-03-06T07:28:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.122+0000 7f6c2d8c9640 1 -- 172.21.15.72:0/2816936651 wait complete. 2024-03-06T07:28:57.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:57.134 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-06T07:28:57.135 DEBUG:teuthology.orchestra.run.smithi072:> mkdir pin 2024-03-06T07:28:57.135 DEBUG:teuthology.orchestra.run.smithi072:> touch pin/placeholder 2024-03-06T07:28:57.135 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-03-06T07:28:57.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:28:57.160 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-03-06T07:28:57.219 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:57.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.397+0000 7f389870c640 1 Processor -- start 2024-03-06T07:28:57.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.397+0000 7f389870c640 1 -- start start 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f389870c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f389015dfc0 0x7f389006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f389870c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f389006e480 0x7f389006d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f389870c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389006d7a0 0x7f38900cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f389870c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389015e3a0 con 0x7f389006e480 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f389870c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389006dbf0 con 0x7f389015dfc0 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f389870c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389006bb60 con 0x7f389006d7a0 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f3895ca7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f389006e480 0x7f389006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f3895ca7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f389006e480 0x7f389006d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49254/0 (socket says 172.21.15.72:49254) 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.398+0000 7f3895ca7640 1 -- 172.21.15.72:0/1123457811 learned_addr learned my addr 172.21.15.72:0/1123457811 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f3896ca9640 1 --2- 172.21.15.72:0/1123457811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389006d7a0 0x7f38900cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f38964a8640 1 --2- 172.21.15.72:0/1123457811 >> v2:172.21.15.146:3300/0 conn(0x7f389015dfc0 0x7f389006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f3895ca7640 1 -- 172.21.15.72:0/1123457811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389006d7a0 msgr2=0x7f38900cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f3895ca7640 1 --2- 172.21.15.72:0/1123457811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389006d7a0 0x7f38900cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f3895ca7640 1 -- 172.21.15.72:0/1123457811 >> v2:172.21.15.146:3300/0 conn(0x7f389015dfc0 msgr2=0x7f389006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f3895ca7640 1 --2- 172.21.15.72:0/1123457811 >> v2:172.21.15.146:3300/0 conn(0x7f389015dfc0 0x7f389006df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.399+0000 7f3895ca7640 1 -- 172.21.15.72:0/1123457811 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389006be40 con 0x7f389006e480 2024-03-06T07:28:57.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.400+0000 7f3895ca7640 1 --2- 172.21.15.72:0/1123457811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006e480 0x7f389006d260 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f389006a1c0 tx=0x7f388002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8170b6c6194b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.400+0000 7f38954a6640 1 -- 172.21.15.72:0/1123457811 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388002e070 con 0x7f389006e480 2024-03-06T07:28:57.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.400+0000 7f38954a6640 1 -- 172.21.15.72:0/1123457811 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3880037070 con 0x7f389006e480 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.400+0000 7f38954a6640 1 -- 172.21.15.72:0/1123457811 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880031a00 con 0x7f389006e480 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 -- 172.21.15.72:0/1123457811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006e480 msgr2=0x7f389006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 --2- 172.21.15.72:0/1123457811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006e480 0x7f389006d260 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f389006a1c0 tx=0x7f388002b880 comp rx=0 tx=0).stop 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 -- 172.21.15.72:0/1123457811 shutdown_connections 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 --2- 172.21.15.72:0/1123457811 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389006d7a0 0x7f38900cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 --2- 172.21.15.72:0/1123457811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006e480 0x7f389006d260 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 --2- 172.21.15.72:0/1123457811 >> v2:172.21.15.146:3300/0 conn(0x7f389015dfc0 0x7f389006df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 -- 172.21.15.72:0/1123457811 >> 172.21.15.72:0/1123457811 conn(0x7f38900d00e0 msgr2=0x7f38900d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 -- 172.21.15.72:0/1123457811 shutdown_connections 2024-03-06T07:28:57.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.401+0000 7f389870c640 1 -- 172.21.15.72:0/1123457811 wait complete. 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.402+0000 7f389870c640 1 Processor -- start 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.402+0000 7f389870c640 1 -- start start 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f389870c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 0x7f38900aff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f389870c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f389006e480 0x7f38900b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f389870c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389015dfc0 0x7f3890149270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f389870c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38900cf100 con 0x7f389006d7a0 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f389870c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389006c400 con 0x7f389006e480 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f389870c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900cf280 con 0x7f389015dfc0 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 0x7f38900aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 0x7f38900aff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49262/0 (socket says 172.21.15.72:49262) 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 -- 172.21.15.72:0/972238530 learned_addr learned my addr 172.21.15.72:0/972238530 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:57.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f3895ca7640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f389006e480 0x7f38900b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 -- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389015dfc0 msgr2=0x7f3890149270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389015dfc0 0x7f3890149270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 -- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f389006e480 msgr2=0x7f38900b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f389006e480 0x7f38900b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 -- 172.21.15.72:0/972238530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389009b200 con 0x7f389006d7a0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.403+0000 7f38964a8640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 0x7f38900aff80 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c7f0 tx=0x7f388c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.404+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c015070 con 0x7f389006d7a0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.404+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c004220 con 0x7f389006d7a0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.404+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c007620 con 0x7f389006d7a0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.404+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389009ac40 con 0x7f389006d7a0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.404+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38900b0690 con 0x7f389006d7a0 2024-03-06T07:28:57.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.405+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389004f330 con 0x7f389006d7a0 2024-03-06T07:28:57.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.405+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f388c0079d0 con 0x7f389006d7a0 2024-03-06T07:28:57.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.406+0000 7f3886ffd640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f387406b4d0 0x7f387406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:57.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.406+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f388c08a4a0 con 0x7f389006d7a0 2024-03-06T07:28:57.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.406+0000 7f3895ca7640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f387406b4d0 0x7f387406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:57.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.406+0000 7f3895ca7640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f387406b4d0 0x7f387406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3880036000 tx=0x7f3880008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.410+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f389009b200 con 0x7f389006d7a0 2024-03-06T07:28:57.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.643+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f389006d300 con 0x7f389006d7a0 2024-03-06T07:28:57.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.643+0000 7f3886ffd640 1 -- 172.21.15.72:0/972238530 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f388c0593b0 con 0x7f389006d7a0 2024-03-06T07:28:57.645 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:28:57.645 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:50.096107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8637,8640]},"id":9}]} 2024-03-06T07:28:57.645 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 84 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f387406b4d0 msgr2=0x7f387406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f387406b4d0 0x7f387406d990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3880036000 tx=0x7f3880008030 comp rx=0 tx=0).stop 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 msgr2=0x7f38900aff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 0x7f38900aff80 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c7f0 tx=0x7f388c00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 shutdown_connections 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f387406b4d0 0x7f387406d990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f389015dfc0 0x7f3890149270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f389006e480 0x7f38900b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 --2- 172.21.15.72:0/972238530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f389006d7a0 0x7f38900aff80 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 >> 172.21.15.72:0/972238530 conn(0x7f38900d00e0 msgr2=0x7f38900d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:57.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 shutdown_connections 2024-03-06T07:28:57.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:57.645+0000 7f389870c640 1 -- 172.21.15.72:0/972238530 wait complete. 2024-03-06T07:28:57.661 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:28:57.776 INFO:teuthology.orchestra.run.smithi072.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:28:57.776 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:28:57.777 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:28:59.778 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:28:59.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.877+0000 7efc55caf640 1 Processor -- start 2024-03-06T07:28:59.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 -- start start 2024-03-06T07:28:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efc500cf830 0x7efc500d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc500d27f0 0x7efc500d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc500d2c50 con 0x7efc500d27f0 2024-03-06T07:28:59.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc500b31e0 con 0x7efc500cf830 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.878+0000 7efc55caf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b3d60 con 0x7efc5015dfe0 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc500d27f0 0x7efc500d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc500d27f0 0x7efc500d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49264/0 (socket says 172.21.15.72:49264) 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 -- 172.21.15.72:0/519306655 learned_addr learned my addr 172.21.15.72:0/519306655 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4effd640 1 --2- 172.21.15.72:0/519306655 >> v2:172.21.15.146:3300/0 conn(0x7efc500cf830 0x7efc500d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4f7fe640 1 --2- 172.21.15.72:0/519306655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 -- 172.21.15.72:0/519306655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 msgr2=0x7efc500cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 --2- 172.21.15.72:0/519306655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 -- 172.21.15.72:0/519306655 >> v2:172.21.15.146:3300/0 conn(0x7efc500cf830 msgr2=0x7efc500d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 --2- 172.21.15.72:0/519306655 >> v2:172.21.15.146:3300/0 conn(0x7efc500cf830 0x7efc500d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.879+0000 7efc4ffff640 1 -- 172.21.15.72:0/519306655 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500b3430 con 0x7efc500d27f0 2024-03-06T07:28:59.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.880+0000 7efc4ffff640 1 --2- 172.21.15.72:0/519306655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500d27f0 0x7efc500d0300 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7efc5008cba0 tx=0x7efc4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d48078432ad149e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.880+0000 7efc4e7fc640 1 -- 172.21.15.72:0/519306655 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4802ca70 con 0x7efc500d27f0 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.880+0000 7efc4e7fc640 1 -- 172.21.15.72:0/519306655 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4802cc10 con 0x7efc500d27f0 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.880+0000 7efc4e7fc640 1 -- 172.21.15.72:0/519306655 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc480343f0 con 0x7efc500d27f0 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 -- 172.21.15.72:0/519306655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500d27f0 msgr2=0x7efc500d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 --2- 172.21.15.72:0/519306655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500d27f0 0x7efc500d0300 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7efc5008cba0 tx=0x7efc4802bed0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 -- 172.21.15.72:0/519306655 shutdown_connections 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 --2- 172.21.15.72:0/519306655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500d27f0 0x7efc500d0300 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 --2- 172.21.15.72:0/519306655 >> v2:172.21.15.146:3300/0 conn(0x7efc500cf830 0x7efc500d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 --2- 172.21.15.72:0/519306655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 -- 172.21.15.72:0/519306655 >> 172.21.15.72:0/519306655 conn(0x7efc500d3cd0 msgr2=0x7efc500d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 -- 172.21.15.72:0/519306655 shutdown_connections 2024-03-06T07:28:59.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.881+0000 7efc55caf640 1 -- 172.21.15.72:0/519306655 wait complete. 2024-03-06T07:28:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.883+0000 7efc55caf640 1 Processor -- start 2024-03-06T07:28:59.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.883+0000 7efc55caf640 1 -- start start 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 0x7efc501f71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc500d27f0 0x7efc500cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500d1c70 con 0x7efc500cf830 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc500b39f0 con 0x7efc500d27f0 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc500d1df0 con 0x7efc5015dfe0 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 0x7efc501f71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 0x7efc501f71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49274/0 (socket says 172.21.15.72:49274) 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 -- 172.21.15.72:0/3465382866 learned_addr learned my addr 172.21.15.72:0/3465382866 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 -- 172.21.15.72:0/3465382866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 msgr2=0x7efc500b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 -- 172.21.15.72:0/3465382866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc500d27f0 msgr2=0x7efc500cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:28:59.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc500d27f0 0x7efc500cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 -- 172.21.15.72:0/3465382866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc500b8970 con 0x7efc500cf830 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc4f7fe640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 0x7efc501f71e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e870 tx=0x7efc4000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40017070 con 0x7efc500cf830 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4000c940 con 0x7efc500cf830 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc501f7910 con 0x7efc500cf830 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4000cd30 con 0x7efc500cf830 2024-03-06T07:28:59.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.884+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc501f7ed0 con 0x7efc500cf830 2024-03-06T07:28:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.886+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5004f330 con 0x7efc500cf830 2024-03-06T07:28:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.886+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc4001a070 con 0x7efc500cf830 2024-03-06T07:28:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.886+0000 7efc37fff640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc3c06b500 0x7efc3c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:28:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.886+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc40089fe0 con 0x7efc500cf830 2024-03-06T07:28:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.886+0000 7efc4effd640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc3c06b500 0x7efc3c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:28:59.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.887+0000 7efc4effd640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc3c06b500 0x7efc3c06d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7efc5015b110 tx=0x7efc44009500 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:28:59.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:28:59.895+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc500b8970 con 0x7efc500cf830 2024-03-06T07:29:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.192+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc500ce870 con 0x7efc500cf830 2024-03-06T07:29:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.192+0000 7efc37fff640 1 -- 172.21.15.72:0/3465382866 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7efc40058ef0 con 0x7efc500cf830 2024-03-06T07:29:00.194 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:00.195 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:50.096107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8637,8640]},"id":9}]} 2024-03-06T07:29:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 84 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc3c06b500 msgr2=0x7efc3c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc3c06b500 0x7efc3c06d9c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7efc5015b110 tx=0x7efc44009500 comp rx=0 tx=0).stop 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 msgr2=0x7efc501f71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 0x7efc501f71e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7efc4000e870 tx=0x7efc4000ed30 comp rx=0 tx=0).stop 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 shutdown_connections 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efc3c06b500 0x7efc3c06d9c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc5015dfe0 0x7efc500b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc500d27f0 0x7efc500cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 --2- 172.21.15.72:0/3465382866 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc500cf830 0x7efc501f71e0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:00.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.194+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 >> 172.21.15.72:0/3465382866 conn(0x7efc500d3cd0 msgr2=0x7efc500cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:00.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.195+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 shutdown_connections 2024-03-06T07:29:00.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:00.195+0000 7efc55caf640 1 -- 172.21.15.72:0/3465382866 wait complete. 2024-03-06T07:29:00.208 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:00.325 INFO:teuthology.orchestra.run.smithi072.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:00.325 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:00.325 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:02.326 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:02.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.435+0000 7f258ca5c640 1 Processor -- start 2024-03-06T07:29:02.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.435+0000 7f258ca5c640 1 -- start start 2024-03-06T07:29:02.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258ca5c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f258815e090 0x7f25880d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258ca5c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25880d2920 0x7f25880b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258ca5c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f258815e470 con 0x7f25880b3b20 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258ca5c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25880d18f0 con 0x7f258815e090 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258ca5c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25880d1b90 con 0x7f25880d2920 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f258815e090 0x7f25880d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f258815e090 0x7f25880d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43360/0 (socket says 172.21.15.72:43360) 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 -- 172.21.15.72:0/3066185245 learned_addr learned my addr 172.21.15.72:0/3066185245 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f2585d9b640 1 --2- 172.21.15.72:0/3066185245 >> v2:172.21.15.146:3300/0 conn(0x7f25880d2920 0x7f25880b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 -- 172.21.15.72:0/3066185245 >> v2:172.21.15.146:3300/0 conn(0x7f25880d2920 msgr2=0x7f25880b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 --2- 172.21.15.72:0/3066185245 >> v2:172.21.15.146:3300/0 conn(0x7f25880d2920 0x7f25880b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 -- 172.21.15.72:0/3066185245 >> v1:172.21.15.146:6789/0 conn(0x7f25880b3b20 legacy=0x7f25880d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f258659c640 1 -- 172.21.15.72:0/3066185245 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880ceb50 con 0x7f258815e090 2024-03-06T07:29:02.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.436+0000 7f2585d9b640 1 --2- 172.21.15.72:0/3066185245 >> v2:172.21.15.146:3300/0 conn(0x7f25880d2920 0x7f25880b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.437+0000 7f258659c640 1 --2- 172.21.15.72:0/3066185245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f258815e090 0x7f25880d47a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f25880d1700 tx=0x7f257802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ff385f89a29827 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.437+0000 7f258559a640 1 -- 172.21.15.72:0/3066185245 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578034070 con 0x7f258815e090 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.437+0000 7f258559a640 1 -- 172.21.15.72:0/3066185245 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2578030860 con 0x7f258815e090 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.437+0000 7f258559a640 1 -- 172.21.15.72:0/3066185245 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2578030c30 con 0x7f258815e090 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 -- 172.21.15.72:0/3066185245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f258815e090 msgr2=0x7f25880d47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 --2- 172.21.15.72:0/3066185245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f258815e090 0x7f25880d47a0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f25880d1700 tx=0x7f257802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 -- 172.21.15.72:0/3066185245 shutdown_connections 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 --2- 172.21.15.72:0/3066185245 >> v2:172.21.15.146:3300/0 conn(0x7f25880d2920 0x7f25880b35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 --2- 172.21.15.72:0/3066185245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f258815e090 0x7f25880d47a0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 -- 172.21.15.72:0/3066185245 >> 172.21.15.72:0/3066185245 conn(0x7f25880d3c80 msgr2=0x7f25880d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 -- 172.21.15.72:0/3066185245 shutdown_connections 2024-03-06T07:29:02.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.438+0000 7f258ca5c640 1 -- 172.21.15.72:0/3066185245 wait complete. 2024-03-06T07:29:02.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.439+0000 7f258ca5c640 1 Processor -- start 2024-03-06T07:29:02.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.439+0000 7f258ca5c640 1 -- start start 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258ca5c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25880d2920 0x7f258815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258ca5c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f258815e090 0x7f25880cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258ca5c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 0x7f25881608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258ca5c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25880ce800 con 0x7f25880cc3a0 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258ca5c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25880b2df0 con 0x7f258815e090 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258ca5c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25880d0d60 con 0x7f25880d2920 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 0x7f25881608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 0x7f25881608e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43376/0 (socket says 172.21.15.72:43376) 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 -- 172.21.15.72:0/814771152 learned_addr learned my addr 172.21.15.72:0/814771152 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:02.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2585d9b640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f258815e090 0x7f25880cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f258659c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25880d2920 0x7f258815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 -- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25880d2920 msgr2=0x7f258815af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25880d2920 0x7f258815af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 -- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f258815e090 msgr2=0x7f25880cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f258815e090 0x7f25880cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 -- 172.21.15.72:0/814771152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25880cccd0 con 0x7f25880cc3a0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2585d9b640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f258815e090 0x7f25880cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.440+0000 7f2586d9d640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 0x7f25881608e0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f258000c730 tx=0x7f258000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.441+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580003b70 con 0x7f25880cc3a0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.441+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2580003d10 con 0x7f25880cc3a0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.441+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25880ccfb0 con 0x7f25880cc3a0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.441+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258000aa90 con 0x7f25880cc3a0 2024-03-06T07:29:02.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.441+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25880cd570 con 0x7f25880cc3a0 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.442+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f258804f330 con 0x7f25880cc3a0 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.442+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f258000ac30 con 0x7f25880cc3a0 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.443+0000 7f2576ffd640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f256806b4d0 0x7f256806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.443+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f258008a070 con 0x7f25880cc3a0 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.443+0000 7f258659c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f256806b4d0 0x7f256806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:02.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.443+0000 7f258659c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f256806b4d0 0x7f256806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2578036000 tx=0x7f257803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:02.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.447+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2580058f80 con 0x7f25880cc3a0 2024-03-06T07:29:02.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.779+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f258815bed0 con 0x7f25880cc3a0 2024-03-06T07:29:02.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.780+0000 7f2576ffd640 1 -- 172.21.15.72:0/814771152 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3417 (secure 0 0 0) 0x7f2580058880 con 0x7f25880cc3a0 2024-03-06T07:29:02.782 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:02.782 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":84,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:28:50.096107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8637,8640]},"id":9}]} 2024-03-06T07:29:02.782 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 84 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f256806b4d0 msgr2=0x7f256806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f256806b4d0 0x7f256806d990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2578036000 tx=0x7f257803b040 comp rx=0 tx=0).stop 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 msgr2=0x7f25881608e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 0x7f25881608e0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f258000c730 tx=0x7f258000cbf0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 shutdown_connections 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f256806b4d0 0x7f256806d990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25880cc3a0 0x7f25881608e0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f258815e090 0x7f25880cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 --2- 172.21.15.72:0/814771152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25880d2920 0x7f258815af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 >> 172.21.15.72:0/814771152 conn(0x7f25880d3c80 msgr2=0x7f25880d0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 shutdown_connections 2024-03-06T07:29:02.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:02.782+0000 7f258ca5c640 1 -- 172.21.15.72:0/814771152 wait complete. 2024-03-06T07:29:02.795 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:02.911 INFO:teuthology.orchestra.run.smithi072.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-06T07:29:02.911 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:02.911 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-03-06T07:29:04.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:05.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.010+0000 7faef1c1c640 1 Processor -- start 2024-03-06T07:29:05.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.010+0000 7faef1c1c640 1 -- start start 2024-03-06T07:29:05.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faef1c1c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faeec0d22b0 0x7faeec0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faef1c1c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec0b3430 con 0x7faeec0b41f0 2024-03-06T07:29:05.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faef1c1c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec0b3710 con 0x7faeec15dfe0 2024-03-06T07:29:05.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faef1c1c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeec0d2c50 con 0x7faeec0d22b0 2024-03-06T07:29:05.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faee2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faeec0d22b0 0x7faeec0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faee2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7faeec0d22b0 0x7faeec0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43392/0 (socket says 172.21.15.72:43392) 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faee2ffd640 1 -- 172.21.15.72:0/4229198373 learned_addr learned my addr 172.21.15.72:0/4229198373 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faeebfff640 1 --1- 172.21.15.72:0/4229198373 >> v1:172.21.15.72:6789/0 conn(0x7faeec0b41f0 0x7faeec0b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53188/0 (socket says 172.21.15.72:53188) 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faee2ffd640 1 -- 172.21.15.72:0/4229198373 >> v1:172.21.15.146:6789/0 conn(0x7faeec15dfe0 legacy=0x7faeec0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faeeaffd640 1 -- 172.21.15.72:0/4229198373 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356109669 0 0) 0x7faeec0b3430 con 0x7faeec0b41f0 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faee2ffd640 1 -- 172.21.15.72:0/4229198373 >> v1:172.21.15.72:6789/0 conn(0x7faeec0b41f0 legacy=0x7faeec0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.011+0000 7faee2ffd640 1 -- 172.21.15.72:0/4229198373 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0b39f0 con 0x7faeec0d22b0 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.012+0000 7faee2ffd640 1 --2- 172.21.15.72:0/4229198373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec0d2710 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7faed40098a0 tx=0x7faed402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b1ba2859a5f1d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.012+0000 7faeeaffd640 1 -- 172.21.15.72:0/4229198373 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed402f840 con 0x7faeec0d22b0 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.012+0000 7faeeaffd640 1 -- 172.21.15.72:0/4229198373 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed402f9e0 con 0x7faeec0d22b0 2024-03-06T07:29:05.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.012+0000 7faeeaffd640 1 -- 172.21.15.72:0/4229198373 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed402fdb0 con 0x7faeec0d22b0 2024-03-06T07:29:05.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 -- 172.21.15.72:0/4229198373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 msgr2=0x7faeec0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 --2- 172.21.15.72:0/4229198373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec0d2710 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7faed40098a0 tx=0x7faed402ba20 comp rx=0 tx=0).stop 2024-03-06T07:29:05.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 -- 172.21.15.72:0/4229198373 shutdown_connections 2024-03-06T07:29:05.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 --2- 172.21.15.72:0/4229198373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec0d2710 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 -- 172.21.15.72:0/4229198373 >> 172.21.15.72:0/4229198373 conn(0x7faeec0d3cd0 msgr2=0x7faeec0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 -- 172.21.15.72:0/4229198373 shutdown_connections 2024-03-06T07:29:05.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.013+0000 7faef1c1c640 1 -- 172.21.15.72:0/4229198373 wait complete. 2024-03-06T07:29:05.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.015+0000 7faef1c1c640 1 Processor -- start 2024-03-06T07:29:05.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.015+0000 7faef1c1c640 1 -- start start 2024-03-06T07:29:05.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faef1c1c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeec0b41f0 0x7faeec1f4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faef1c1c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec1f4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faef1c1c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeec15dfe0 0x7faeec0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faef1c1c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0b3d60 con 0x7faeec0d22b0 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faef1c1c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0b31e0 con 0x7faeec0b41f0 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faef1c1c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0d1c70 con 0x7faeec15dfe0 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faee2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec1f4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faee2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec1f4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43406/0 (socket says 172.21.15.72:43406) 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faee2ffd640 1 -- 172.21.15.72:0/456410584 learned_addr learned my addr 172.21.15.72:0/456410584 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faeeb7fe640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeec0b41f0 0x7faeec1f4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.016+0000 7faeebfff640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeec15dfe0 0x7faeec0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee2ffd640 1 -- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeec15dfe0 msgr2=0x7faeec0cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee2ffd640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeec15dfe0 0x7faeec0cd010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee2ffd640 1 -- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeec0b41f0 msgr2=0x7faeec1f4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee2ffd640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeec0b41f0 0x7faeec1f4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee2ffd640 1 -- 172.21.15.72:0/456410584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec1611c0 con 0x7faeec0d22b0 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee2ffd640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec1f4510 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7faed4000c00 tx=0x7faed4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4041400 con 0x7faeec0d22b0 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed4046070 con 0x7faeec0d22b0 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec1614a0 con 0x7faeec0d22b0 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4041800 con 0x7faeec0d22b0 2024-03-06T07:29:05.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.017+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec15e9d0 con 0x7faeec0d22b0 2024-03-06T07:29:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.018+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec04f330 con 0x7faeec0d22b0 2024-03-06T07:29:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.019+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7faed4040070 con 0x7faeec0d22b0 2024-03-06T07:29:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.019+0000 7faee8ff9640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.019+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faed40aca90 con 0x7faeec0d22b0 2024-03-06T07:29:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.019+0000 7faeeb7fe640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:05.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.020+0000 7faeeb7fe640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faecc06b4d0 0x7faecc06d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7faedc000f60 tx=0x7faedc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:05.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.024+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7faed4033030 con 0x7faeec0d22b0 2024-03-06T07:29:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.362+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeec0b1890 con 0x7faeec0d22b0 2024-03-06T07:29:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.363+0000 7faee8ff9640 1 -- 172.21.15.72:0/456410584 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3418 (secure 0 0 0) 0x7faed407b910 con 0x7faeec0d22b0 2024-03-06T07:29:05.365 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:05.365 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":85,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:03.460007+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:active","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640,8637]},"id":9}]} 2024-03-06T07:29:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 85 2024-03-06T07:29:05.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faecc06b4d0 msgr2=0x7faecc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faecc06b4d0 0x7faecc06d990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7faedc000f60 tx=0x7faedc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 msgr2=0x7faeec1f4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec1f4510 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7faed4000c00 tx=0x7faed4004730 comp rx=0 tx=0).stop 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 shutdown_connections 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7faecc06b4d0 0x7faecc06d990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faeec15dfe0 0x7faeec0cd010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faeec0d22b0 0x7faeec1f4510 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 --2- 172.21.15.72:0/456410584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faeec0b41f0 0x7faeec1f4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 >> 172.21.15.72:0/456410584 conn(0x7faeec0d3cd0 msgr2=0x7faeec06bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 shutdown_connections 2024-03-06T07:29:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:05.366+0000 7faef1c1c640 1 -- 172.21.15.72:0/456410584 wait complete. 2024-03-06T07:29:05.383 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-06T07:29:05.502 INFO:teuthology.orchestra.run.smithi072.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"10","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-06T07:29:05.503 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "10" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-06T07:29:05.503 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-03-06T07:29:05.503 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:29:05.503 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> stdin-killer -- python3 -c ' 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> import fcntl 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> import sys 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> import time 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:29:05.519 DEBUG:teuthology.orchestra.run.smithi072:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> while True: 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> try: 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> if os.read(0, 4096) == b"": 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> break 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> except BlockingIOError: 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> pass 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> time.sleep(2) 2024-03-06T07:29:05.520 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-03-06T07:29:05.561 DEBUG:teuthology.orchestra.run.smithi072:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:29:05.579 INFO:teuthology.orchestra.run.smithi072.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-03-06T07:29:05.651 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:06.579 DEBUG:teuthology.orchestra.run.smithi072:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:06.594 INFO:teuthology.orchestra.run.smithi072.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 29h/41d Inode: 2199023255552 Links: 2 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1301/ ubuntu) 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-06 07:29:05.644128274 +0000 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-06 07:29:05.644128274 +0000 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-06 07:29:05.644128274 +0000 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: - 2024-03-06T07:29:06.595 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-03-06T07:29:06.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:29:06.596 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-03-06T07:29:06.653 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-03-06T07:29:06.826 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"num_strays":1}} 2024-03-06T07:29:06.826 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-03-06T07:29:06.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-03-06T07:29:06.965 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_created":1}} 2024-03-06T07:29:06.965 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-03-06T07:29:06.965 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-03-06T07:29:07.114 INFO:teuthology.orchestra.run.smithi072.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-03-06T07:29:07.114 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-03-06T07:29:07.115 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-03-06T07:29:07.266 INFO:teuthology.orchestra.run.smithi072.stdout:{"message":"","return_code":0} 2024-03-06T07:29:07.266 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-03-06T07:29:07.267 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-03-06T07:29:07.388 INFO:teuthology.orchestra.run.smithi072.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":328,"bytes":65087}},"duration":0.007636983} 2024-03-06T07:29:07.388 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 65087, "items": 328 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.007636983, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-03-06T07:29:07.388 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-03-06T07:29:07.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.494+0000 7f8442de9640 1 Processor -- start 2024-03-06T07:29:07.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.494+0000 7f8442de9640 1 -- start start 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8442de9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f843c15ee10 0x7f843c168d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8442de9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843c0cea90 con 0x7f843c06b6c0 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8442de9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f843c161bc0 con 0x7f843c15bbb0 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8442de9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f843c161970 con 0x7f843c15ee10 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f843bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f843c06b6c0 0x7f843c15e700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53198/0 (socket says 172.21.15.72:53198) 2024-03-06T07:29:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f843bfff640 1 -- 172.21.15.72:0/3760913328 learned_addr learned my addr 172.21.15.72:0/3760913328 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8441386640 1 --2- 172.21.15.72:0/3760913328 >> v2:172.21.15.72:3300/0 conn(0x7f843c15ee10 0x7f843c168d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8441386640 1 -- 172.21.15.72:0/3760913328 >> v1:172.21.15.146:6789/0 conn(0x7f843c15bbb0 legacy=0x7f843c06afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8441386640 1 -- 172.21.15.72:0/3760913328 >> v1:172.21.15.72:6789/0 conn(0x7f843c06b6c0 legacy=0x7f843c15e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.495+0000 7f8441386640 1 -- 172.21.15.72:0/3760913328 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c161ea0 con 0x7f843c15ee10 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.496+0000 7f8441386640 1 --2- 172.21.15.72:0/3760913328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c168d00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f8434009d90 tx=0x7f843402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b864a1e0ce549d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.496+0000 7f843b7fe640 1 -- 172.21.15.72:0/3760913328 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434039070 con 0x7f843c15ee10 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.496+0000 7f843b7fe640 1 -- 172.21.15.72:0/3760913328 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843402cd00 con 0x7f843c15ee10 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.496+0000 7f843b7fe640 1 -- 172.21.15.72:0/3760913328 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434034660 con 0x7f843c15ee10 2024-03-06T07:29:07.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 -- 172.21.15.72:0/3760913328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 msgr2=0x7f843c168d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 --2- 172.21.15.72:0/3760913328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c168d00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f8434009d90 tx=0x7f843402bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 -- 172.21.15.72:0/3760913328 shutdown_connections 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 --2- 172.21.15.72:0/3760913328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c168d00 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 -- 172.21.15.72:0/3760913328 >> 172.21.15.72:0/3760913328 conn(0x7f843c0b2660 msgr2=0x7f843c0b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 -- 172.21.15.72:0/3760913328 shutdown_connections 2024-03-06T07:29:07.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.497+0000 7f8442de9640 1 -- 172.21.15.72:0/3760913328 wait complete. 2024-03-06T07:29:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.499+0000 7f8442de9640 1 Processor -- start 2024-03-06T07:29:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.499+0000 7f8442de9640 1 -- start start 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f8442de9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f843c06b6c0 0x7f843c0cafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f8442de9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c0cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f8442de9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f843c14cfd0 0x7f843c15a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f8442de9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c072cb0 con 0x7f843c15ee10 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f8442de9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843c1624f0 con 0x7f843c06b6c0 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f8442de9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f843c169240 con 0x7f843c14cfd0 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f843bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c0cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f843bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c0cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43426/0 (socket says 172.21.15.72:43426) 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.500+0000 7f843bfff640 1 -- 172.21.15.72:0/1324900457 learned_addr learned my addr 172.21.15.72:0/1324900457 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f8440b85640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f843c06b6c0 0x7f843c0cafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f843bfff640 1 -- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f843c14cfd0 msgr2=0x7f843c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f8441386640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f843c14cfd0 0x7f843c15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f843bfff640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f843c14cfd0 0x7f843c15a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f843bfff640 1 -- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f843c06b6c0 msgr2=0x7f843c0cafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f843bfff640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f843c06b6c0 0x7f843c0cafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f843bfff640 1 -- 172.21.15.72:0/1324900457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f843c0cb610 con 0x7f843c15ee10 2024-03-06T07:29:07.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f8440b85640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f843c06b6c0 0x7f843c0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f843bfff640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c0cc9b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f842800c4d0 tx=0x7f842800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428017070 con 0x7f843c15ee10 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8428005ce0 con 0x7f843c15ee10 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f843c0cb8f0 con 0x7f843c15ee10 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8428007270 con 0x7f843c15ee10 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.501+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843c0cbeb0 con 0x7f843c15ee10 2024-03-06T07:29:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.502+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843c0733b0 con 0x7f843c15ee10 2024-03-06T07:29:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.502+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f842801a070 con 0x7f843c15ee10 2024-03-06T07:29:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.503+0000 7f84397fa640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f842006b500 0x7f842006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.503+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8428089440 con 0x7f843c15ee10 2024-03-06T07:29:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.503+0000 7f8440b85640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f842006b500 0x7f842006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.504+0000 7f8440b85640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f842006b500 0x7f842006d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f843c153bf0 tx=0x7f8430009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:07.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.507+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f843c0cea90 con 0x7f843c15ee10 2024-03-06T07:29:07.653 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:07.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:07.724+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f843c068ff0 con 0x7f843c15ee10 2024-03-06T07:29:08.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.148+0000 7f84397fa640 1 -- 172.21.15.72:0/1324900457 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v86) v1 ==== 105+0+0 (secure 0 0 0) 0x7f8428057c70 con 0x7f843c15ee10 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f842006b500 msgr2=0x7f842006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f842006b500 0x7f842006d9c0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f843c153bf0 tx=0x7f8430009040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 msgr2=0x7f843c0cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c0cc9b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f842800c4d0 tx=0x7f842800c990 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 shutdown_connections 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f842006b500 0x7f842006d9c0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f843c14cfd0 0x7f843c15a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f843c15ee10 0x7f843c0cc9b0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 --2- 172.21.15.72:0/1324900457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f843c06b6c0 0x7f843c0cafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.150+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 >> 172.21.15.72:0/1324900457 conn(0x7f843c0b2660 msgr2=0x7f843c0cef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.151+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 shutdown_connections 2024-03-06T07:29:08.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.151+0000 7f8442de9640 1 -- 172.21.15.72:0/1324900457 wait complete. 2024-03-06T07:29:08.163 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:08.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 Processor -- start 2024-03-06T07:29:08.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 -- start start 2024-03-06T07:29:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f297415e3a0 0x7f297406d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f297406deb0 0x7f2974162030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2974161ba0 con 0x7f297415e3a0 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297415eb40 con 0x7f297406deb0 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.264+0000 7f29791e4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297415ee20 con 0x7f2974158600 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f297415e3a0 0x7f297406d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f297415e3a0 0x7f297406d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43438/0 (socket says 172.21.15.72:43438) 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 -- 172.21.15.72:0/860790680 learned_addr learned my addr 172.21.15.72:0/860790680 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f297359e640 1 --2- 172.21.15.72:0/860790680 >> v2:172.21.15.146:3300/0 conn(0x7f297406deb0 0x7f2974162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 -- 172.21.15.72:0/860790680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 msgr2=0x7f2974072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f2972d9d640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 -- 172.21.15.72:0/860790680 >> v2:172.21.15.146:3300/0 conn(0x7f297406deb0 msgr2=0x7f2974162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 --2- 172.21.15.72:0/860790680 >> v2:172.21.15.146:3300/0 conn(0x7f297406deb0 0x7f2974162030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f296bfff640 1 -- 172.21.15.72:0/860790680 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297415f100 con 0x7f297415e3a0 2024-03-06T07:29:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f297359e640 1 --2- 172.21.15.72:0/860790680 >> v2:172.21.15.146:3300/0 conn(0x7f297406deb0 0x7f2974162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.265+0000 7f2972d9d640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f296bfff640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297415e3a0 0x7f297406d970 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f29740b0510 tx=0x7f295c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769242d31e7b0bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f297259c640 1 -- 172.21.15.72:0/860790680 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c02e070 con 0x7f297415e3a0 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f297259c640 1 -- 172.21.15.72:0/860790680 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c037070 con 0x7f297415e3a0 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f297259c640 1 -- 172.21.15.72:0/860790680 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c031a00 con 0x7f297415e3a0 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 -- 172.21.15.72:0/860790680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297415e3a0 msgr2=0x7f297406d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297415e3a0 0x7f297406d970 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f29740b0510 tx=0x7f295c02b880 comp rx=0 tx=0).stop 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 -- 172.21.15.72:0/860790680 shutdown_connections 2024-03-06T07:29:08.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 --2- 172.21.15.72:0/860790680 >> v2:172.21.15.146:3300/0 conn(0x7f297406deb0 0x7f2974162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297415e3a0 0x7f297406d970 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 --2- 172.21.15.72:0/860790680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 -- 172.21.15.72:0/860790680 >> 172.21.15.72:0/860790680 conn(0x7f297406c0c0 msgr2=0x7f297406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 -- 172.21.15.72:0/860790680 shutdown_connections 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.266+0000 7f29791e4640 1 -- 172.21.15.72:0/860790680 wait complete. 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.268+0000 7f29791e4640 1 Processor -- start 2024-03-06T07:29:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.268+0000 7f29791e4640 1 -- start start 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.268+0000 7f29791e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 0x7f2974152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.268+0000 7f29791e4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.268+0000 7f29791e4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297415e3a0 0x7f2974150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.268+0000 7f29791e4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29741626f0 con 0x7f297406deb0 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f29791e4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2974162570 con 0x7f297415e3a0 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f29791e4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2974162870 con 0x7f2974158600 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 0x7f2974152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 0x7f2974152ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43452/0 (socket says 172.21.15.72:43452) 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 -- 172.21.15.72:0/699768579 learned_addr learned my addr 172.21.15.72:0/699768579 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f296bfff640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f297359e640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297415e3a0 0x7f2974150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 -- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 msgr2=0x7f2974150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 -- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297415e3a0 msgr2=0x7f2974150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297415e3a0 0x7f2974150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 -- 172.21.15.72:0/699768579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29740b1680 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f2972d9d640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 0x7f2974152ca0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f296400c7f0 tx=0x7f296400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964015070 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2964007820 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964007bd0 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29740b1960 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.269+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29740b1f20 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.271+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297404f330 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.271+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2964018070 con 0x7f297406deb0 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.271+0000 7f296b7fe640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f295406b4d0 0x7f295406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.271+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f296408a2d0 con 0x7f297406deb0 2024-03-06T07:29:08.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.271+0000 7f296bfff640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f295406b4d0 0x7f295406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:08.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.272+0000 7f296bfff640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f295406b4d0 0x7f295406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f295c036000 tx=0x7f295c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:08.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.275+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2964059150 con 0x7f297406deb0 2024-03-06T07:29:08.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.481+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29740d06f0 con 0x7f297406deb0 2024-03-06T07:29:08.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.482+0000 7f296b7fe640 1 -- 172.21.15.72:0/699768579 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f2964058a50 con 0x7f297406deb0 2024-03-06T07:29:08.484 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:08.484 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:08.485 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f295406b4d0 msgr2=0x7f295406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f295406b4d0 0x7f295406d990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f295c036000 tx=0x7f295c03d040 comp rx=0 tx=0).stop 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 msgr2=0x7f2974152ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 0x7f2974152ca0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f296400c7f0 tx=0x7f296400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 shutdown_connections 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f295406b4d0 0x7f295406d990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297415e3a0 0x7f2974150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2974158600 0x7f2974150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 --2- 172.21.15.72:0/699768579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f297406deb0 0x7f2974152ca0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:08.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 >> 172.21.15.72:0/699768579 conn(0x7f297406c0c0 msgr2=0x7f29740adf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:08.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 shutdown_connections 2024-03-06T07:29:08.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:08.484+0000 7f29791e4640 1 -- 172.21.15.72:0/699768579 wait complete. 2024-03-06T07:29:08.698 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:08.698 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:09.655 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:09.699 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:09.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 Processor -- start 2024-03-06T07:29:09.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 -- start start 2024-03-06T07:29:09.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd640b4170 0x7fbd640b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640d4760 0x7fbd640d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd640d25d0 con 0x7fbd640b4dd0 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd640d4bf0 con 0x7fbd640b4170 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.799+0000 7fbd693eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd640ce9e0 con 0x7fbd640d4760 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd62ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd640b4170 0x7fbd640b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd627fc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640d4760 0x7fbd640d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd637fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbd640b4dd0 0x7fbd640cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:44946/0 (socket says 172.21.15.72:44946) 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd62ffd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd640b4170 0x7fbd640b6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:37340/0 (socket says 172.21.15.72:37340) 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd637fe640 1 -- 172.21.15.72:0/5203311 learned_addr learned my addr 172.21.15.72:0/5203311 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd62ffd640 1 -- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640d4760 msgr2=0x7fbd640d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd62ffd640 1 --2- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640d4760 0x7fbd640d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd62ffd640 1 -- 172.21.15.72:0/5203311 >> v1:172.21.15.146:6789/0 conn(0x7fbd640b4dd0 legacy=0x7fbd640cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd61ffb640 1 -- 172.21.15.72:0/5203311 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262997465 0 0) 0x7fbd640d25d0 con 0x7fbd640b4dd0 2024-03-06T07:29:09.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd62ffd640 1 -- 172.21.15.72:0/5203311 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640d2870 con 0x7fbd640b4170 2024-03-06T07:29:09.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.800+0000 7fbd627fc640 1 --2- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640d4760 0x7fbd640d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:09.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.801+0000 7fbd62ffd640 1 --2- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4170 0x7fbd640b6a90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbd540097a0 tx=0x7fbd5402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e89d7b27d517f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.801+0000 7fbd61ffb640 1 -- 172.21.15.72:0/5203311 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54035020 con 0x7fbd640b4170 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.801+0000 7fbd61ffb640 1 -- 172.21.15.72:0/5203311 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd54031830 con 0x7fbd640b4170 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.801+0000 7fbd61ffb640 1 -- 172.21.15.72:0/5203311 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54031c00 con 0x7fbd640b4170 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 -- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4170 msgr2=0x7fbd640b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 --2- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4170 0x7fbd640b6a90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbd540097a0 tx=0x7fbd5402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 -- 172.21.15.72:0/5203311 shutdown_connections 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 --2- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640d4760 0x7fbd640d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 --2- 172.21.15.72:0/5203311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4170 0x7fbd640b6a90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 -- 172.21.15.72:0/5203311 >> 172.21.15.72:0/5203311 conn(0x7fbd640d3c40 msgr2=0x7fbd640d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 -- 172.21.15.72:0/5203311 shutdown_connections 2024-03-06T07:29:09.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.802+0000 7fbd693eb640 1 -- 172.21.15.72:0/5203311 wait complete. 2024-03-06T07:29:09.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.803+0000 7fbd693eb640 1 Processor -- start 2024-03-06T07:29:09.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.803+0000 7fbd693eb640 1 -- start start 2024-03-06T07:29:09.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd693eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640b4170 0x7fbd64156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd693eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4dd0 0x7fbd640b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd693eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 0x7fbd640b91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd693eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd640b47d0 con 0x7fbd640b8d50 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd693eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd640b4550 con 0x7fbd640b4dd0 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd693eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd640b4950 con 0x7fbd640b4170 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 0x7fbd640b91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd62ffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640b4170 0x7fbd64156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 0x7fbd640b91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43464/0 (socket says 172.21.15.72:43464) 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 -- 172.21.15.72:0/3664915673 learned_addr learned my addr 172.21.15.72:0/3664915673 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd627fc640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4dd0 0x7fbd640b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 -- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640b4170 msgr2=0x7fbd64156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640b4170 0x7fbd64156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 -- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4dd0 msgr2=0x7fbd640b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:09.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4dd0 0x7fbd640b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd637fe640 1 -- 172.21.15.72:0/3664915673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd640cbb90 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.804+0000 7fbd62ffd640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640b4170 0x7fbd64156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.805+0000 7fbd637fe640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 0x7fbd640b91e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00c7a0 tx=0x7fbd5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.805+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c018070 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.805+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5c00a640 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.805+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd640cad10 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.805+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c00aab0 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.805+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd640cb2d0 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.806+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6404f330 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.807+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbd5c01b070 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.807+0000 7fbd4b7fe640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd5006b500 0x7fbd5006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.807+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd5c0896c0 con 0x7fbd640b8d50 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.807+0000 7fbd62ffd640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd5006b500 0x7fbd5006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:09.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.808+0000 7fbd62ffd640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd5006b500 0x7fbd5006d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbd54036000 tx=0x7fbd5403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:09.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:09.813+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbd5c057f80 con 0x7fbd640b8d50 2024-03-06T07:29:10.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.024+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd64160a20 con 0x7fbd640b8d50 2024-03-06T07:29:10.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.025+0000 7fbd4b7fe640 1 -- 172.21.15.72:0/3664915673 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fbd5c05c020 con 0x7fbd640b8d50 2024-03-06T07:29:10.027 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:10.027 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:10.028 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:10.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd5006b500 msgr2=0x7fbd5006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd5006b500 0x7fbd5006d9c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbd54036000 tx=0x7fbd5403d040 comp rx=0 tx=0).stop 2024-03-06T07:29:10.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 msgr2=0x7fbd640b91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 0x7fbd640b91e0 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00c7a0 tx=0x7fbd5c00cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 shutdown_connections 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbd5006b500 0x7fbd5006d9c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd640b8d50 0x7fbd640b91e0 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd640b4dd0 0x7fbd640b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 --2- 172.21.15.72:0/3664915673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd640b4170 0x7fbd64156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 >> 172.21.15.72:0/3664915673 conn(0x7fbd640d3c40 msgr2=0x7fbd641611b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 shutdown_connections 2024-03-06T07:29:10.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:10.027+0000 7fbd693eb640 1 -- 172.21.15.72:0/3664915673 wait complete. 2024-03-06T07:29:10.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:10.242 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:11.242 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:11.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.342+0000 7f7650c88640 1 Processor -- start 2024-03-06T07:29:11.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.342+0000 7f7650c88640 1 -- start start 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7650c88640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f764c0cce40 0x7f764c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7650c88640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f764c0cd7a0 0x7f764c0cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7650c88640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f764c071f40 con 0x7f764c1515e0 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7650c88640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764c0cd2f0 con 0x7f764c0cce40 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7650c88640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764c070a70 con 0x7f764c0cd7a0 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f764c0cce40 0x7f764c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f764c0cce40 0x7f764c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43474/0 (socket says 172.21.15.72:43474) 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 -- 172.21.15.72:0/421578581 learned_addr learned my addr 172.21.15.72:0/421578581 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f764ad9d640 1 --2- 172.21.15.72:0/421578581 >> v2:172.21.15.146:3300/0 conn(0x7f764c0cd7a0 0x7f764c0cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 -- 172.21.15.72:0/421578581 >> v2:172.21.15.146:3300/0 conn(0x7f764c0cd7a0 msgr2=0x7f764c0cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 --2- 172.21.15.72:0/421578581 >> v2:172.21.15.146:3300/0 conn(0x7f764c0cd7a0 0x7f764c0cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 -- 172.21.15.72:0/421578581 >> v1:172.21.15.146:6789/0 conn(0x7f764c1515e0 legacy=0x7f764c071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.344+0000 7f7649d9b640 1 -- 172.21.15.72:0/421578581 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c070d50 con 0x7f764c0cce40 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.345+0000 7f7649d9b640 1 --2- 172.21.15.72:0/421578581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c15b430 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f764c0cc110 tx=0x7f764002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73d447531c16826 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.345+0000 7f764959a640 1 -- 172.21.15.72:0/421578581 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640035020 con 0x7f764c0cce40 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.345+0000 7f764959a640 1 -- 172.21.15.72:0/421578581 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7640031640 con 0x7f764c0cce40 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.345+0000 7f764959a640 1 -- 172.21.15.72:0/421578581 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764002d040 con 0x7f764c0cce40 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 -- 172.21.15.72:0/421578581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 msgr2=0x7f764c15b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 --2- 172.21.15.72:0/421578581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c15b430 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f764c0cc110 tx=0x7f764002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 -- 172.21.15.72:0/421578581 shutdown_connections 2024-03-06T07:29:11.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 --2- 172.21.15.72:0/421578581 >> v2:172.21.15.146:3300/0 conn(0x7f764c0cd7a0 0x7f764c0cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 --2- 172.21.15.72:0/421578581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c15b430 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 -- 172.21.15.72:0/421578581 >> 172.21.15.72:0/421578581 conn(0x7f764c0cf8a0 msgr2=0x7f764c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 -- 172.21.15.72:0/421578581 shutdown_connections 2024-03-06T07:29:11.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.346+0000 7f7650c88640 1 -- 172.21.15.72:0/421578581 wait complete. 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.347+0000 7f7650c88640 1 Processor -- start 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.348+0000 7f7650c88640 1 -- start start 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c1477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f764c0cd7a0 0x7f764c154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f764c1515e0 0x7f764c15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c0ce0c0 con 0x7f764c0cce40 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c071310 con 0x7f764c1515e0 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764c0b3f90 con 0x7f764c0cd7a0 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c1477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c1477e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43486/0 (socket says 172.21.15.72:43486) 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 -- 172.21.15.72:0/1512173643 learned_addr learned my addr 172.21.15.72:0/1512173643 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:11.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7649d9b640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f764c0cd7a0 0x7f764c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764ad9d640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f764c1515e0 0x7f764c15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 -- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f764c0cd7a0 msgr2=0x7f764c154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f764c0cd7a0 0x7f764c154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 -- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f764c1515e0 msgr2=0x7f764c15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f764c1515e0 0x7f764c15d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 -- 172.21.15.72:0/1512173643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c158e90 con 0x7f764c0cce40 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764ad9d640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f764c1515e0 0x7f764c15d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f764a59c640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c1477e0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f763c00ba50 tx=0x7f763c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c019070 con 0x7f764c0cce40 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c00cc20 con 0x7f764c0cce40 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764c159130 con 0x7f764c0cce40 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c014680 con 0x7f764c0cce40 2024-03-06T07:29:11.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.349+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764c1596f0 con 0x7f764c0cce40 2024-03-06T07:29:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.351+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764c068840 con 0x7f764c0cce40 2024-03-06T07:29:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.351+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f763c01c070 con 0x7f764c0cce40 2024-03-06T07:29:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.351+0000 7f763affd640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f762c06b4d0 0x7f762c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.351+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f763c089e80 con 0x7f764c0cce40 2024-03-06T07:29:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.351+0000 7f7649d9b640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f762c06b4d0 0x7f762c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:11.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.352+0000 7f7649d9b640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f762c06b4d0 0x7f762c06d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7640036000 tx=0x7f764003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:11.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.356+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f763c0586b0 con 0x7f764c0cce40 2024-03-06T07:29:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.624+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f764c04f330 con 0x7f764c0cce40 2024-03-06T07:29:11.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.624+0000 7f763affd640 1 -- 172.21.15.72:0/1512173643 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f763c05c750 con 0x7f764c0cce40 2024-03-06T07:29:11.627 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:11.627 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:11.627 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f762c06b4d0 msgr2=0x7f762c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f762c06b4d0 0x7f762c06d990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f7640036000 tx=0x7f764003c040 comp rx=0 tx=0).stop 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 msgr2=0x7f764c1477e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c1477e0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f763c00ba50 tx=0x7f763c00bf10 comp rx=0 tx=0).stop 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 shutdown_connections 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f762c06b4d0 0x7f762c06d990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f764c1515e0 0x7f764c15d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f764c0cd7a0 0x7f764c154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 --2- 172.21.15.72:0/1512173643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f764c0cce40 0x7f764c1477e0 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:11.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 >> 172.21.15.72:0/1512173643 conn(0x7f764c0cf8a0 msgr2=0x7f764c14e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:11.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 shutdown_connections 2024-03-06T07:29:11.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:11.626+0000 7f7650c88640 1 -- 172.21.15.72:0/1512173643 wait complete. 2024-03-06T07:29:11.656 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:11.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:11.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:12.842 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:12.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.941+0000 7fad2bb0c640 1 Processor -- start 2024-03-06T07:29:12.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.941+0000 7fad2bb0c640 1 -- start start 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad2bb0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 0x7fad2406afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad2bb0c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2415eb40 con 0x7fad24072730 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad2bb0c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2415ee20 con 0x7fad24158600 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad2bb0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad24161ba0 con 0x7fad2415e3a0 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad290a7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fad24072730 0x7fad24162030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52114/0 (socket says 172.21.15.72:52114) 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad290a7640 1 -- 172.21.15.72:0/731752799 learned_addr learned my addr 172.21.15.72:0/731752799 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad2a0a9640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 0x7fad2406afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006087468 0 0) 0x7fad2415eb40 con 0x7fad24072730 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad10003520 con 0x7fad24072730 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47692973 0 0) 0x7fad10003520 con 0x7fad24072730 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 msgr2=0x7fad2406afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 0x7fad2406afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 >> v1:172.21.15.146:6789/0 conn(0x7fad24158600 legacy=0x7fad2406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2415f100 con 0x7fad24072730 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fad14001670 con 0x7fad24072730 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 >> v1:172.21.15.72:6789/0 conn(0x7fad24072730 legacy=0x7fad24162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 0x7fad100039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad10003f40 0x7fad10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.942+0000 7fad288a6640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad10004d40 0x7fad100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24161ba0 con 0x7fad10003f40 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad1000bad0 con 0x7fad10004d40 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1000bcd0 con 0x7fad2415e3a0 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad290a7640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad10003f40 0x7fad10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad298a8640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 0x7fad100039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad2a0a9640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad10004d40 0x7fad100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad290a7640 1 -- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 msgr2=0x7fad100039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad290a7640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad2415e3a0 0x7fad100039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad290a7640 1 -- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad10004d40 msgr2=0x7fad100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad290a7640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad10004d40 0x7fad100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.943+0000 7fad290a7640 1 -- 172.21.15.72:0/731752799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad10003520 con 0x7fad10003f40 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad290a7640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad10003f40 0x7fad10004800 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c240 tx=0x7fad10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ab931e57e6f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad14007d30 con 0x7fad10003f40 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad14030070 con 0x7fad10003f40 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad288a6640 1 -- 172.21.15.72:0/731752799 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1402cd80 con 0x7fad10003f40 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2bb0c640 1 -- 172.21.15.72:0/731752799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad10003f40 msgr2=0x7fad10004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/731752799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad10003f40 0x7fad10004800 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fad1400c240 tx=0x7fad10005830 comp rx=0 tx=0).stop 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2a0a9640 1 -- 172.21.15.72:0/731752799 reap_dead start 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2bb0c640 1 -- 172.21.15.72:0/731752799 shutdown_connections 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2bb0c640 1 -- 172.21.15.72:0/731752799 >> 172.21.15.72:0/731752799 conn(0x7fad2406c0c0 msgr2=0x7fad2406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2bb0c640 1 -- 172.21.15.72:0/731752799 shutdown_connections 2024-03-06T07:29:12.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.944+0000 7fad2bb0c640 1 -- 172.21.15.72:0/731752799 wait complete. 2024-03-06T07:29:12.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.945+0000 7fad2bb0c640 1 Processor -- start 2024-03-06T07:29:12.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 -- start start 2024-03-06T07:29:12.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad10003f40 0x7fad2414d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad24154480 0x7fad24160d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 0x7fad240b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2406e080 con 0x7fad24152e70 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24072be0 con 0x7fad24154480 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2bb0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad241533e0 con 0x7fad10003f40 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2a0a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 0x7fad240b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2a0a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 0x7fad240b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59134/0 (socket says 172.21.15.72:59134) 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.946+0000 7fad2a0a9640 1 -- 172.21.15.72:0/260774428 learned_addr learned my addr 172.21.15.72:0/260774428 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad298a8640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad10003f40 0x7fad2414d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad290a7640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad24154480 0x7fad24160d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2a0a9640 1 -- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad10003f40 msgr2=0x7fad2414d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2a0a9640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad10003f40 0x7fad2414d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2a0a9640 1 -- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad24154480 msgr2=0x7fad24160d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2a0a9640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad24154480 0x7fad24160d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:12.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2a0a9640 1 -- 172.21.15.72:0/260774428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad241536c0 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2a0a9640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 0x7fad240b3300 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fad24071b20 tx=0x7fad1c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00fce0 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c00fe80 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad241539a0 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c015670 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.947+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad24153f60 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.948+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2415ef20 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.948+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad1c015810 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.949+0000 7fad1a7fc640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad0806b500 0x7fad0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.949+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad1c08b0b0 con 0x7fad24152e70 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.949+0000 7fad298a8640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad0806b500 0x7fad0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:12.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.949+0000 7fad298a8640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad0806b500 0x7fad0806d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fad241618a0 tx=0x7fad20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:12.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:12.953+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad2415ef20 con 0x7fad24152e70 2024-03-06T07:29:13.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.176+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad24154160 con 0x7fad24152e70 2024-03-06T07:29:13.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.177+0000 7fad1a7fc640 1 -- 172.21.15.72:0/260774428 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fad24154160 con 0x7fad24152e70 2024-03-06T07:29:13.179 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:13.180 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:13.180 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:13.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad0806b500 msgr2=0x7fad0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad0806b500 0x7fad0806d9c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fad241618a0 tx=0x7fad20009040 comp rx=0 tx=0).stop 2024-03-06T07:29:13.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 msgr2=0x7fad240b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 0x7fad240b3300 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fad24071b20 tx=0x7fad1c00f1b0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 shutdown_connections 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad0806b500 0x7fad0806d9c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad24152e70 0x7fad240b3300 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad24154480 0x7fad24160d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 --2- 172.21.15.72:0/260774428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad10003f40 0x7fad2414d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 >> 172.21.15.72:0/260774428 conn(0x7fad2406c0c0 msgr2=0x7fad2406c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 shutdown_connections 2024-03-06T07:29:13.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:13.179+0000 7fad2bb0c640 1 -- 172.21.15.72:0/260774428 wait complete. 2024-03-06T07:29:13.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:13.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:14.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.495+0000 7f67087f7640 1 Processor -- start 2024-03-06T07:29:14.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.495+0000 7f67087f7640 1 -- start start 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f67087f7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f670015dfe0 0x7f67000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f67087f7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000b3430 con 0x7f67000d29c0 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f67087f7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67000b3710 con 0x7f67000cf830 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f67087f7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67000b31e0 con 0x7f670015dfe0 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f6706d94640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f67000d29c0 0x7f67000b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52120/0 (socket says 172.21.15.72:52120) 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f6706d94640 1 -- 172.21.15.72:0/3123002024 learned_addr learned my addr 172.21.15.72:0/3123002024 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591137886 0 0) 0x7f67000b3430 con 0x7f67000d29c0 2024-03-06T07:29:14.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66ec003520 con 0x7f67000d29c0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.496+0000 7f6706593640 1 --2- 172.21.15.72:0/3123002024 >> v2:172.21.15.146:3300/0 conn(0x7f670015dfe0 0x7f67000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 204163063 0 0) 0x7f66ec003520 con 0x7f67000d29c0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 >> v2:172.21.15.146:3300/0 conn(0x7f670015dfe0 msgr2=0x7f67000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 --2- 172.21.15.72:0/3123002024 >> v2:172.21.15.146:3300/0 conn(0x7f670015dfe0 0x7f67000cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 >> v1:172.21.15.146:6789/0 conn(0x7f67000cf830 legacy=0x7f67000d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000b39f0 con 0x7f67000d29c0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f66f80019f0 con 0x7f67000d29c0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 >> v1:172.21.15.72:6789/0 conn(0x7f67000d29c0 legacy=0x7f67000b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66ec0039b0 0x7f66ec003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f66ec0048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66ec0050a0 0x7f66ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67000b31e0 con 0x7f66ec0043d0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec00bfe0 con 0x7f66ec0039b0 2024-03-06T07:29:14.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ec00c1e0 con 0x7f66ec0050a0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705d92640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f66ec0048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6706593640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66ec0039b0 0x7f66ec003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6706d94640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66ec0050a0 0x7f66ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705d92640 1 -- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66ec0050a0 msgr2=0x7f66ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705d92640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66ec0050a0 0x7f66ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705d92640 1 -- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66ec0039b0 msgr2=0x7f66ec003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705d92640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66ec0039b0 0x7f66ec003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6706d94640 1 -- 172.21.15.72:0/3123002024 reap_dead start 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.497+0000 7f6705d92640 1 -- 172.21.15.72:0/3123002024 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec003520 con 0x7f66ec0043d0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.498+0000 7f6705d92640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f66ec0048a0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f66fc001830 tx=0x7f66ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496cc86463feac77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.498+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc001430 con 0x7f66ec0043d0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.498+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc032620 con 0x7f66ec0043d0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.498+0000 7f6705591640 1 -- 172.21.15.72:0/3123002024 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc032a10 con 0x7f66ec0043d0 2024-03-06T07:29:14.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.498+0000 7f67087f7640 1 -- 172.21.15.72:0/3123002024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 msgr2=0x7f66ec0048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.498+0000 7f67087f7640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f66ec0048a0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f66fc001830 tx=0x7f66ec005830 comp rx=0 tx=0).stop 2024-03-06T07:29:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.499+0000 7f67087f7640 1 -- 172.21.15.72:0/3123002024 shutdown_connections 2024-03-06T07:29:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.499+0000 7f67087f7640 1 --2- 172.21.15.72:0/3123002024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f66ec0048a0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.499+0000 7f67087f7640 1 -- 172.21.15.72:0/3123002024 >> 172.21.15.72:0/3123002024 conn(0x7f67000d3cd0 msgr2=0x7f67000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.499+0000 7f67087f7640 1 -- 172.21.15.72:0/3123002024 shutdown_connections 2024-03-06T07:29:14.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.499+0000 7f67087f7640 1 -- 172.21.15.72:0/3123002024 wait complete. 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 Processor -- start 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 -- start start 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f670015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6700161040 0x7f67000b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67000b8f10 0x7f67001f70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670015e3c0 con 0x7f66ec0043d0 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67000b3d60 con 0x7f67000b8f10 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.500+0000 7f67087f7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f670015e540 con 0x7f6700161040 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f670015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f670015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59158/0 (socket says 172.21.15.72:59158) 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 -- 172.21.15.72:0/2729502487 learned_addr learned my addr 172.21.15.72:0/2729502487 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6705d92640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6700161040 0x7f67000b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706d94640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67000b8f10 0x7f67001f70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 -- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6700161040 msgr2=0x7f67000b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6700161040 0x7f67000b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 -- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67000b8f10 msgr2=0x7f67001f70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67000b8f10 0x7f67001f70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 -- 172.21.15.72:0/2729502487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670015c7c0 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f6706593640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f670015c280 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f66f40118f0 tx=0x7f66f4011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f4019070 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f4004240 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f670015f130 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f400f7c0 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.501+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f670015f6b0 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.502+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670004f330 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.503+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f66f401c070 con 0x7f66ec0043d0 2024-03-06T07:29:14.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.503+0000 7f66f2ffd640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f66e406b4d0 0x7f66e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:14.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.503+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66f408a5e0 con 0x7f66ec0043d0 2024-03-06T07:29:14.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.503+0000 7f6705d92640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f66e406b4d0 0x7f66e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:14.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.504+0000 7f6705d92640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f66e406b4d0 0x7f66e406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f66fc035000 tx=0x7f66fc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:14.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.507+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f670015c7c0 con 0x7f66ec0043d0 2024-03-06T07:29:14.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.722+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f670006bfa0 con 0x7f66ec0043d0 2024-03-06T07:29:14.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.722+0000 7f66f2ffd640 1 -- 172.21.15.72:0/2729502487 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f66f4059460 con 0x7f66ec0043d0 2024-03-06T07:29:14.725 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:14.725 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:14.725 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.724+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f66e406b4d0 msgr2=0x7f66e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.724+0000 7f67087f7640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f66e406b4d0 0x7f66e406d990 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f66fc035000 tx=0x7f66fc030000 comp rx=0 tx=0).stop 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.724+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 msgr2=0x7f670015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.724+0000 7f67087f7640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f670015c280 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f66f40118f0 tx=0x7f66f4011db0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 shutdown_connections 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f66e406b4d0 0x7f66e406d990 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67000b8f10 0x7f67001f70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6700161040 0x7f67000b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 --2- 172.21.15.72:0/2729502487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66ec0043d0 0x7f670015c280 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:14.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 >> 172.21.15.72:0/2729502487 conn(0x7f67000d3cd0 msgr2=0x7f670015d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:14.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 shutdown_connections 2024-03-06T07:29:14.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:14.725+0000 7f67087f7640 1 -- 172.21.15.72:0/2729502487 wait complete. 2024-03-06T07:29:14.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:14.938 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:15.661 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:15.939 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:16.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.039+0000 7f5e7b55e640 1 Processor -- start 2024-03-06T07:29:16.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.039+0000 7f5e7b55e640 1 -- start start 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e7b55e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e740cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e7b55e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e740cf830 0x7f5e740d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e7b55e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e740b3430 con 0x7f5e740d27f0 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e7b55e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e740d2c50 con 0x7f5e740cf830 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e7b55e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740b31e0 con 0x7f5e7415dfe0 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e78af9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5e740cf830 0x7f5e740d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e792fa640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e740cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e79afb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5e740d27f0 0x7f5e740d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:33742/0 (socket says 172.21.15.72:33742) 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.040+0000 7f5e79afb640 1 -- 172.21.15.72:0/2270179867 learned_addr learned my addr 172.21.15.72:0/2270179867 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:16.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.041+0000 7f5e78af9640 1 -- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 msgr2=0x7f5e740cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:16.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.041+0000 7f5e78af9640 1 --2- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e740cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.041+0000 7f5e78af9640 1 -- 172.21.15.72:0/2270179867 >> v1:172.21.15.146:6789/0 conn(0x7f5e740d27f0 legacy=0x7f5e740d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.041+0000 7f5e73fff640 1 -- 172.21.15.72:0/2270179867 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033260902 0 0) 0x7f5e740b3430 con 0x7f5e740d27f0 2024-03-06T07:29:16.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.041+0000 7f5e78af9640 1 -- 172.21.15.72:0/2270179867 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740b3710 con 0x7f5e740cf830 2024-03-06T07:29:16.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.041+0000 7f5e792fa640 1 --2- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e740cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:16.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e78af9640 1 --2- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e740d22b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5e680097a0 tx=0x7f5e6802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2126df9279d9cf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e73fff640 1 -- 172.21.15.72:0/2270179867 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68035020 con 0x7f5e740cf830 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e73fff640 1 -- 172.21.15.72:0/2270179867 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68031830 con 0x7f5e740cf830 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e73fff640 1 -- 172.21.15.72:0/2270179867 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68031c00 con 0x7f5e740cf830 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 -- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 msgr2=0x7f5e740d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e740d22b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5e680097a0 tx=0x7f5e6802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 -- 172.21.15.72:0/2270179867 shutdown_connections 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e740d22b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/2270179867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e740cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 -- 172.21.15.72:0/2270179867 >> 172.21.15.72:0/2270179867 conn(0x7f5e740d3cd0 msgr2=0x7f5e740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 -- 172.21.15.72:0/2270179867 shutdown_connections 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.042+0000 7f5e7b55e640 1 -- 172.21.15.72:0/2270179867 wait complete. 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.043+0000 7f5e7b55e640 1 Processor -- start 2024-03-06T07:29:16.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 -- start start 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e74161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 0x7f5e7415e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e7415ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740d1c70 con 0x7f5e740d27f0 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740b3d60 con 0x7f5e740cf830 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e7b55e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e740d1df0 con 0x7f5e7415dfe0 2024-03-06T07:29:16.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e78af9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 0x7f5e7415e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e78af9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 0x7f5e7415e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59166/0 (socket says 172.21.15.72:59166) 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.044+0000 7f5e78af9640 1 -- 172.21.15.72:0/3245133088 learned_addr learned my addr 172.21.15.72:0/3245133088 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e792fa640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e74161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e79afb640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e7415ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e78af9640 1 -- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 msgr2=0x7f5e7415ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e78af9640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e7415ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e78af9640 1 -- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 msgr2=0x7f5e74161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e78af9640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e74161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e78af9640 1 -- 172.21.15.72:0/3245133088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740b6c90 con 0x7f5e740d27f0 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e78af9640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 0x7f5e7415e7d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68035070 con 0x7f5e740d27f0 2024-03-06T07:29:16.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e740cc360 con 0x7f5e740d27f0 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e68004230 con 0x7f5e740d27f0 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6803a630 con 0x7f5e740d27f0 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.045+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e740cc920 con 0x7f5e740d27f0 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.047+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7404f330 con 0x7f5e740d27f0 2024-03-06T07:29:16.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.047+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5e68038020 con 0x7f5e740d27f0 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.047+0000 7f5e71ffb640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.047+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e680abef0 con 0x7f5e740d27f0 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.047+0000 7f5e792fa640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.047+0000 7f5e792fa640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5e741603b0 tx=0x7f5e6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:16.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.051+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5e68040070 con 0x7f5e740d27f0 2024-03-06T07:29:16.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.293+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e7415d4c0 con 0x7f5e740d27f0 2024-03-06T07:29:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.294+0000 7f5e71ffb640 1 -- 172.21.15.72:0/3245133088 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f5e6807acf0 con 0x7f5e740d27f0 2024-03-06T07:29:16.296 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:16.296 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:16.296 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:16.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 msgr2=0x7f5e6006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f5e741603b0 tx=0x7f5e6c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:16.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 msgr2=0x7f5e7415e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:16.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 0x7f5e7415e7d0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f5e68000c00 tx=0x7f5e680046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 shutdown_connections 2024-03-06T07:29:16.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5e6006b4d0 0x7f5e6006d990 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5e7415dfe0 0x7f5e7415ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e740d27f0 0x7f5e7415e7d0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 --2- 172.21.15.72:0/3245133088 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5e740cf830 0x7f5e74161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:16.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 >> 172.21.15.72:0/3245133088 conn(0x7f5e740d3cd0 msgr2=0x7f5e7406bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:16.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 shutdown_connections 2024-03-06T07:29:16.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:16.296+0000 7f5e7b55e640 1 -- 172.21.15.72:0/3245133088 wait complete. 2024-03-06T07:29:16.510 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:16.510 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:17.511 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:17.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.610+0000 7fd380ad4640 1 Processor -- start 2024-03-06T07:29:17.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.610+0000 7fd380ad4640 1 -- start start 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd380ad4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd37c15dfe0 0x7fd37c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd380ad4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0cf830 0x7fd37c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd380ad4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd37c0d27f0 0x7fd37c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd380ad4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37c0d2c50 con 0x7fd37c15dfe0 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd380ad4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37c0b31e0 con 0x7fd37c0d27f0 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd380ad4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0b3d60 con 0x7fd37c0cf830 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd37c15dfe0 0x7fd37c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd37c15dfe0 0x7fd37c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59182/0 (socket says 172.21.15.72:59182) 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 -- 172.21.15.72:0/1237777011 learned_addr learned my addr 172.21.15.72:0/1237777011 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 -- 172.21.15.72:0/1237777011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0cf830 msgr2=0x7fd37c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd371d9b640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0cf830 0x7fd37c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37ad9d640 1 --2- 172.21.15.72:0/1237777011 >> v2:172.21.15.146:3300/0 conn(0x7fd37c0d27f0 0x7fd37c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0cf830 0x7fd37c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 -- 172.21.15.72:0/1237777011 >> v2:172.21.15.146:3300/0 conn(0x7fd37c0d27f0 msgr2=0x7fd37c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 --2- 172.21.15.72:0/1237777011 >> v2:172.21.15.146:3300/0 conn(0x7fd37c0d27f0 0x7fd37c0d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37a59c640 1 -- 172.21.15.72:0/1237777011 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c0b3430 con 0x7fd37c15dfe0 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd371d9b640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0cf830 0x7fd37c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.611+0000 7fd37ad9d640 1 --2- 172.21.15.72:0/1237777011 >> v2:172.21.15.146:3300/0 conn(0x7fd37c0d27f0 0x7fd37c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.612+0000 7fd37a59c640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c15dfe0 0x7fd37c0cf2f0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd37c067ad0 tx=0x7fd36c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a1761fde224602 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.612+0000 7fd379d9b640 1 -- 172.21.15.72:0/1237777011 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c035020 con 0x7fd37c15dfe0 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.612+0000 7fd379d9b640 1 -- 172.21.15.72:0/1237777011 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c031870 con 0x7fd37c15dfe0 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.612+0000 7fd379d9b640 1 -- 172.21.15.72:0/1237777011 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c031c40 con 0x7fd37c15dfe0 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 -- 172.21.15.72:0/1237777011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c15dfe0 msgr2=0x7fd37c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c15dfe0 0x7fd37c0cf2f0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fd37c067ad0 tx=0x7fd36c02b910 comp rx=0 tx=0).stop 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 -- 172.21.15.72:0/1237777011 shutdown_connections 2024-03-06T07:29:17.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 --2- 172.21.15.72:0/1237777011 >> v2:172.21.15.146:3300/0 conn(0x7fd37c0d27f0 0x7fd37c0d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0cf830 0x7fd37c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 --2- 172.21.15.72:0/1237777011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c15dfe0 0x7fd37c0cf2f0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 -- 172.21.15.72:0/1237777011 >> 172.21.15.72:0/1237777011 conn(0x7fd37c0d3cd0 msgr2=0x7fd37c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 -- 172.21.15.72:0/1237777011 shutdown_connections 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.613+0000 7fd380ad4640 1 -- 172.21.15.72:0/1237777011 wait complete. 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.615+0000 7fd380ad4640 1 Processor -- start 2024-03-06T07:29:17.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.615+0000 7fd380ad4640 1 -- start start 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd380ad4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 0x7fd37c15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd380ad4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0d27f0 0x7fd37c0cc510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd380ad4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c15dfe0 0x7fd37c1605e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd380ad4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0d1c70 con 0x7fd37c0cf830 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd380ad4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0b39f0 con 0x7fd37c15dfe0 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd380ad4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0d1df0 con 0x7fd37c0d27f0 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 0x7fd37c15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 0x7fd37c15a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59196/0 (socket says 172.21.15.72:59196) 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 -- 172.21.15.72:0/3472970082 learned_addr learned my addr 172.21.15.72:0/3472970082 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37ad9d640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c15dfe0 0x7fd37c1605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd371d9b640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0d27f0 0x7fd37c0cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 -- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0d27f0 msgr2=0x7fd37c0cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0d27f0 0x7fd37c0cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 -- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c15dfe0 msgr2=0x7fd37c1605e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c15dfe0 0x7fd37c1605e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37a59c640 1 -- 172.21.15.72:0/3472970082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c15f540 con 0x7fd37c0cf830 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd37ad9d640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c15dfe0 0x7fd37c1605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.616+0000 7fd371d9b640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0d27f0 0x7fd37c0cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.617+0000 7fd37a59c640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 0x7fd37c15a120 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fd36c02b950 tx=0x7fd36c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.617+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c035040 con 0x7fd37c0cf830 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.617+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c045480 con 0x7fd37c0cf830 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.617+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c045870 con 0x7fd37c0cf830 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.617+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37c15f820 con 0x7fd37c0cf830 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.617+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37c15fde0 con 0x7fd37c0cf830 2024-03-06T07:29:17.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.618+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37c04f330 con 0x7fd37c0cf830 2024-03-06T07:29:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.618+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd36c038020 con 0x7fd37c0cf830 2024-03-06T07:29:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.619+0000 7fd3737fe640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd35c06b4d0 0x7fd35c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.619+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd36c0ac870 con 0x7fd37c0cf830 2024-03-06T07:29:17.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.619+0000 7fd371d9b640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd35c06b4d0 0x7fd35c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:17.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.619+0000 7fd371d9b640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd35c06b4d0 0x7fd35c06d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0d0a80 tx=0x7fd364008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:17.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.623+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd36c07b860 con 0x7fd37c0cf830 2024-03-06T07:29:17.663 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:17.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.855+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd37c0747a0 con 0x7fd37c0cf830 2024-03-06T07:29:17.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.855+0000 7fd3737fe640 1 -- 172.21.15.72:0/3472970082 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fd36c07b160 con 0x7fd37c0cf830 2024-03-06T07:29:17.858 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:17.858 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:17.858 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd35c06b4d0 msgr2=0x7fd35c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd35c06b4d0 0x7fd35c06d990 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0d0a80 tx=0x7fd364008040 comp rx=0 tx=0).stop 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 msgr2=0x7fd37c15a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 0x7fd37c15a120 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fd36c02b950 tx=0x7fd36c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 shutdown_connections 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd35c06b4d0 0x7fd35c06d990 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd37c15dfe0 0x7fd37c1605e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd37c0d27f0 0x7fd37c0cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 --2- 172.21.15.72:0/3472970082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd37c0cf830 0x7fd37c15a120 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 >> 172.21.15.72:0/3472970082 conn(0x7fd37c0d3cd0 msgr2=0x7fd37c06bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 shutdown_connections 2024-03-06T07:29:17.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:17.857+0000 7fd380ad4640 1 -- 172.21.15.72:0/3472970082 wait complete. 2024-03-06T07:29:18.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:18.072 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:19.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:19.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.170+0000 7f82f9343640 1 Processor -- start 2024-03-06T07:29:19.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f9343640 1 -- start start 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f9343640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f82f40afb20 0x7f82f415c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f9343640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40b4e30 con 0x7f82f415ab00 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f9343640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82f40b5110 con 0x7f82f40af6c0 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f9343640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82f415b310 con 0x7f82f40afb20 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f82f40afb20 0x7f82f415c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.171+0000 7f82f2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f82f415ab00 0x7f82f40b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52126/0 (socket says 172.21.15.72:52126) 2024-03-06T07:29:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.172+0000 7f82f2ffd640 1 -- 172.21.15.72:0/4042893255 learned_addr learned my addr 172.21.15.72:0/4042893255 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.172+0000 7f82f37fe640 1 -- 172.21.15.72:0/4042893255 >> v1:172.21.15.146:6789/0 conn(0x7f82f40af6c0 legacy=0x7f82f415bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.172+0000 7f82f37fe640 1 -- 172.21.15.72:0/4042893255 >> v1:172.21.15.72:6789/0 conn(0x7f82f415ab00 legacy=0x7f82f40b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.172+0000 7f82f37fe640 1 -- 172.21.15.72:0/4042893255 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f415b5f0 con 0x7f82f40afb20 2024-03-06T07:29:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.172+0000 7f82f27fc640 1 -- 172.21.15.72:0/4042893255 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029771794 0 0) 0x7f82f40b4e30 con 0x7f82f415ab00 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f37fe640 1 --2- 172.21.15.72:0/4042893255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40afb20 0x7f82f415c220 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009d90 tx=0x7f82ec02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d7f861f68a547f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f27fc640 1 -- 172.21.15.72:0/4042893255 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec039070 con 0x7f82f40afb20 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f27fc640 1 -- 172.21.15.72:0/4042893255 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec02cd00 con 0x7f82f40afb20 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f27fc640 1 -- 172.21.15.72:0/4042893255 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec034660 con 0x7f82f40afb20 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 -- 172.21.15.72:0/4042893255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40afb20 msgr2=0x7f82f415c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 --2- 172.21.15.72:0/4042893255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40afb20 0x7f82f415c220 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009d90 tx=0x7f82ec02bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 -- 172.21.15.72:0/4042893255 shutdown_connections 2024-03-06T07:29:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 --2- 172.21.15.72:0/4042893255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40afb20 0x7f82f415c220 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 -- 172.21.15.72:0/4042893255 >> 172.21.15.72:0/4042893255 conn(0x7f82f406c0c0 msgr2=0x7f82f406c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 -- 172.21.15.72:0/4042893255 shutdown_connections 2024-03-06T07:29:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.173+0000 7f82f9343640 1 -- 172.21.15.72:0/4042893255 wait complete. 2024-03-06T07:29:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.175+0000 7f82f9343640 1 Processor -- start 2024-03-06T07:29:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.175+0000 7f82f9343640 1 -- start start 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f9343640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 0x7f82f40b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f9343640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82f40afb20 0x7f82f40b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f9343640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82f415ab00 0x7f82f41f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f9343640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4071b20 con 0x7f82f40af6c0 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f9343640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f40b4ac0 con 0x7f82f415ab00 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f9343640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f4071ca0 con 0x7f82f40afb20 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 0x7f82f40b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 0x7f82f40b1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59220/0 (socket says 172.21.15.72:59220) 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 -- 172.21.15.72:0/547961329 learned_addr learned my addr 172.21.15.72:0/547961329 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82ebfff640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82f40afb20 0x7f82f40b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f37fe640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82f415ab00 0x7f82f41f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 -- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82f40afb20 msgr2=0x7f82f40b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82f40afb20 0x7f82f40b06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 -- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82f415ab00 msgr2=0x7f82f41f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82f415ab00 0x7f82f41f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f2ffd640 1 -- 172.21.15.72:0/547961329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f40ce0f0 con 0x7f82f40af6c0 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.176+0000 7f82f37fe640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82f415ab00 0x7f82f41f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.177+0000 7f82f2ffd640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 0x7f82f40b1390 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f82e400c3f0 tx=0x7f82e400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.179+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e40074e0 con 0x7f82f40af6c0 2024-03-06T07:29:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.179+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e40047e0 con 0x7f82f40af6c0 2024-03-06T07:29:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.179+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f414edd0 con 0x7f82f40af6c0 2024-03-06T07:29:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.179+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4004c10 con 0x7f82f40af6c0 2024-03-06T07:29:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.179+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f40ce3d0 con 0x7f82f40af6c0 2024-03-06T07:29:19.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.180+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f404f330 con 0x7f82f40af6c0 2024-03-06T07:29:19.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.180+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f82e4011070 con 0x7f82f40af6c0 2024-03-06T07:29:19.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.180+0000 7f82eb7fe640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82d406b4d0 0x7f82d406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:19.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.181+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82e4089e00 con 0x7f82f40af6c0 2024-03-06T07:29:19.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.181+0000 7f82ebfff640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82d406b4d0 0x7f82d406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:19.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.181+0000 7f82ebfff640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82d406b4d0 0x7f82d406d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f82f41557c0 tx=0x7f82dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:19.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.185+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f82e401a2b0 con 0x7f82f40af6c0 2024-03-06T07:29:19.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.399+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82f414efa0 con 0x7f82f40af6c0 2024-03-06T07:29:19.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.400+0000 7f82eb7fe640 1 -- 172.21.15.72:0/547961329 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f82e4058c80 con 0x7f82f40af6c0 2024-03-06T07:29:19.402 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:19.402 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:19.402 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82d406b4d0 msgr2=0x7f82d406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82d406b4d0 0x7f82d406d990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f82f41557c0 tx=0x7f82dc009040 comp rx=0 tx=0).stop 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 msgr2=0x7f82f40b1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 0x7f82f40b1390 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f82e400c3f0 tx=0x7f82e400c8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 shutdown_connections 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f82d406b4d0 0x7f82d406d990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82f415ab00 0x7f82f41f4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82f40afb20 0x7f82f40b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 --2- 172.21.15.72:0/547961329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f40af6c0 0x7f82f40b1390 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:19.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.401+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 >> 172.21.15.72:0/547961329 conn(0x7f82f406c0c0 msgr2=0x7f82f40b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:19.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.402+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 shutdown_connections 2024-03-06T07:29:19.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:19.402+0000 7f82f9343640 1 -- 172.21.15.72:0/547961329 wait complete. 2024-03-06T07:29:19.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:19.620 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:19.665 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:20.621 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:20.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.720+0000 7fde5cf7a640 1 Processor -- start 2024-03-06T07:29:20.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.720+0000 7fde5cf7a640 1 -- start start 2024-03-06T07:29:20.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.721+0000 7fde5cf7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde580d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.721+0000 7fde5cf7a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde580b3430 con 0x7fde580b41f0 2024-03-06T07:29:20.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.721+0000 7fde5cf7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde580b3710 con 0x7fde5815dfe0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.721+0000 7fde5cf7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde580d2c50 con 0x7fde580d22b0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.721+0000 7fde56d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fde580b41f0 0x7fde580b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52136/0 (socket says 172.21.15.72:52136) 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.721+0000 7fde56d9d640 1 -- 172.21.15.72:0/3821289167 learned_addr learned my addr 172.21.15.72:0/3821289167 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde4ffff640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde580d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302285579 0 0) 0x7fde580b3430 con 0x7fde580b41f0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde44003520 con 0x7fde580b41f0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259953095 0 0) 0x7fde44003520 con 0x7fde580b41f0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 msgr2=0x7fde580d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde580d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 >> v1:172.21.15.146:6789/0 conn(0x7fde5815dfe0 legacy=0x7fde580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580b39f0 con 0x7fde580b41f0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fde500019f0 con 0x7fde580b41f0 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 >> v1:172.21.15.72:6789/0 conn(0x7fde580b41f0 legacy=0x7fde580b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde440039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde44003f20 0x7fde44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde44004d40 0x7fde440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4400b9f0 con 0x7fde44004d40 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4400bba0 con 0x7fde44003f20 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde4400bda0 con 0x7fde580d22b0 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde56d9d640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde44004d40 0x7fde440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde5659c640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde440039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde4ffff640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde44003f20 0x7fde44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde56d9d640 1 -- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 msgr2=0x7fde440039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde56d9d640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde440039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde56d9d640 1 -- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde44003f20 msgr2=0x7fde44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde56d9d640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde44003f20 0x7fde44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde56d9d640 1 -- 172.21.15.72:0/3821289167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde44003520 con 0x7fde44004d40 2024-03-06T07:29:20.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde4ffff640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde44003f20 0x7fde44004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.722+0000 7fde5659c640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde580d22b0 0x7fde440039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.723+0000 7fde56d9d640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde44004d40 0x7fde440051c0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fde5000a800 tx=0x7fde44005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7077507b9319c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.723+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5000a450 con 0x7fde44004d40 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.723+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde50034650 con 0x7fde44004d40 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.723+0000 7fde55d9b640 1 -- 172.21.15.72:0/3821289167 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5002ce60 con 0x7fde44004d40 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde5cf7a640 1 -- 172.21.15.72:0/3821289167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde44004d40 msgr2=0x7fde440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/3821289167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde44004d40 0x7fde440051c0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fde5000a800 tx=0x7fde44005830 comp rx=0 tx=0).stop 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde56d9d640 1 -- 172.21.15.72:0/3821289167 reap_dead start 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde5cf7a640 1 -- 172.21.15.72:0/3821289167 shutdown_connections 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde5cf7a640 1 -- 172.21.15.72:0/3821289167 >> 172.21.15.72:0/3821289167 conn(0x7fde580d3cd0 msgr2=0x7fde580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde5cf7a640 1 -- 172.21.15.72:0/3821289167 shutdown_connections 2024-03-06T07:29:20.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.724+0000 7fde5cf7a640 1 -- 172.21.15.72:0/3821289167 wait complete. 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.725+0000 7fde5cf7a640 1 Processor -- start 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.725+0000 7fde5cf7a640 1 -- start start 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5cf7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 0x7fde581f4a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5cf7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde581f4f80 0x7fde580cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5cf7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde580cc9b0 0x7fde580b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5cf7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde580b3d60 con 0x7fde580b5c60 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5cf7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde580b31e0 con 0x7fde580cc9b0 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5cf7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde580d1c70 con 0x7fde581f4f80 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 0x7fde581f4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 0x7fde581f4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59236/0 (socket says 172.21.15.72:59236) 2024-03-06T07:29:20.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 -- 172.21.15.72:0/1927034370 learned_addr learned my addr 172.21.15.72:0/1927034370 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 -- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde581f4f80 msgr2=0x7fde580cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde581f4f80 0x7fde580cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 -- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde580cc9b0 msgr2=0x7fde580b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde56d9d640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde580cc9b0 0x7fde580b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde580cc9b0 0x7fde580b7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.726+0000 7fde5659c640 1 -- 172.21.15.72:0/1927034370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde580cb410 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.727+0000 7fde5659c640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 0x7fde581f4a40 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fde4800e800 tx=0x7fde4800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.727+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde480052e0 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.727+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde48007020 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.727+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde580cb6b0 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.727+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde480071f0 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.727+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde580cbc70 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.728+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde5804f330 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.728+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fde4801a020 con 0x7fde580b5c60 2024-03-06T07:29:20.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.728+0000 7fde4effd640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde3806b4d0 0x7fde3806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:20.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.729+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde48089970 con 0x7fde580b5c60 2024-03-06T07:29:20.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.729+0000 7fde4ffff640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde3806b4d0 0x7fde3806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:20.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.729+0000 7fde4ffff640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde3806b4d0 0x7fde3806d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fde580ccfb0 tx=0x7fde40008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:20.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.737+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fde48058880 con 0x7fde580b5c60 2024-03-06T07:29:20.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.952+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde580b2a40 con 0x7fde580b5c60 2024-03-06T07:29:20.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.953+0000 7fde4effd640 1 -- 172.21.15.72:0/1927034370 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fde48058180 con 0x7fde580b5c60 2024-03-06T07:29:20.955 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:20.955 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:20.955 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde3806b4d0 msgr2=0x7fde3806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde3806b4d0 0x7fde3806d990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fde580ccfb0 tx=0x7fde40008040 comp rx=0 tx=0).stop 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 msgr2=0x7fde581f4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 0x7fde581f4a40 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fde4800e800 tx=0x7fde4800ecc0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 shutdown_connections 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fde3806b4d0 0x7fde3806d990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde580cc9b0 0x7fde580b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde581f4f80 0x7fde580cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 --2- 172.21.15.72:0/1927034370 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fde580b5c60 0x7fde581f4a40 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:20.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 >> 172.21.15.72:0/1927034370 conn(0x7fde580d3cd0 msgr2=0x7fde5815d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:20.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 shutdown_connections 2024-03-06T07:29:20.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:20.955+0000 7fde5cf7a640 1 -- 172.21.15.72:0/1927034370 wait complete. 2024-03-06T07:29:21.170 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:21.170 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:21.625 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:21.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:21.666 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:22.171 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:22.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.271+0000 7fb3b0e88640 1 Processor -- start 2024-03-06T07:29:22.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.271+0000 7fb3b0e88640 1 -- start start 2024-03-06T07:29:22.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3b0e88640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3ac0d2710 0x7fb3ac0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3b0e88640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac0b3430 con 0x7fb3ac15dfe0 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3b0e88640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac0b3710 con 0x7fb3ac0d22b0 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3b0e88640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b31e0 con 0x7fb3ac0d2710 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3aad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3ac0d2710 0x7fb3ac0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3aad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3ac0d2710 0x7fb3ac0b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59248/0 (socket says 172.21.15.72:59248) 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3aad9d640 1 -- 172.21.15.72:0/482009453 learned_addr learned my addr 172.21.15.72:0/482009453 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.272+0000 7fb3aa59c640 1 --1- 172.21.15.72:0/482009453 >> v1:172.21.15.72:6789/0 conn(0x7fb3ac15dfe0 0x7fb3ac0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52146/0 (socket says 172.21.15.72:52146) 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3aad9d640 1 -- 172.21.15.72:0/482009453 >> v1:172.21.15.146:6789/0 conn(0x7fb3ac0d22b0 legacy=0x7fb3ac0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3aad9d640 1 -- 172.21.15.72:0/482009453 >> v1:172.21.15.72:6789/0 conn(0x7fb3ac15dfe0 legacy=0x7fb3ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3a959a640 1 -- 172.21.15.72:0/482009453 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689418597 0 0) 0x7fb3ac0b3430 con 0x7fb3ac15dfe0 2024-03-06T07:29:22.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3aad9d640 1 -- 172.21.15.72:0/482009453 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0b39f0 con 0x7fb3ac0d2710 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3aad9d640 1 --2- 172.21.15.72:0/482009453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0b4900 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009e80 tx=0x7fb39c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370e0c101a3c6865 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3a959a640 1 -- 172.21.15.72:0/482009453 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c02caf0 con 0x7fb3ac0d2710 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3a959a640 1 -- 172.21.15.72:0/482009453 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c02cc90 con 0x7fb3ac0d2710 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.273+0000 7fb3a959a640 1 -- 172.21.15.72:0/482009453 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c034610 con 0x7fb3ac0d2710 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 -- 172.21.15.72:0/482009453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 msgr2=0x7fb3ac0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/482009453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0b4900 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009e80 tx=0x7fb39c02bf50 comp rx=0 tx=0).stop 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 -- 172.21.15.72:0/482009453 shutdown_connections 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/482009453 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0b4900 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 -- 172.21.15.72:0/482009453 >> 172.21.15.72:0/482009453 conn(0x7fb3ac0d3cd0 msgr2=0x7fb3ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 -- 172.21.15.72:0/482009453 shutdown_connections 2024-03-06T07:29:22.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.274+0000 7fb3b0e88640 1 -- 172.21.15.72:0/482009453 wait complete. 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.275+0000 7fb3b0e88640 1 Processor -- start 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.275+0000 7fb3b0e88640 1 -- start start 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3b0e88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ac0d22b0 0x7fb3ac0b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3b0e88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3b0e88640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ac15dfe0 0x7fb3ac159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3b0e88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0cfa00 con 0x7fb3ac0d2710 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3b0e88640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b3d60 con 0x7fb3ac15dfe0 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3b0e88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0d1c70 con 0x7fb3ac0d22b0 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3a9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3a9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59264/0 (socket says 172.21.15.72:59264) 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3a9d9b640 1 -- 172.21.15.72:0/3224414043 learned_addr learned my addr 172.21.15.72:0/3224414043 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3aad9d640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ac15dfe0 0x7fb3ac159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3a9d9b640 1 -- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ac0d22b0 msgr2=0x7fb3ac0b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.276+0000 7fb3aa59c640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ac0d22b0 0x7fb3ac0b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3a9d9b640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ac0d22b0 0x7fb3ac0b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3a9d9b640 1 -- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ac15dfe0 msgr2=0x7fb3ac159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3a9d9b640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ac15dfe0 0x7fb3ac159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3a9d9b640 1 -- 172.21.15.72:0/3224414043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac1599b0 con 0x7fb3ac0d2710 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3aad9d640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ac15dfe0 0x7fb3ac159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3aa59c640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ac0d22b0 0x7fb3ac0b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3a9d9b640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0ca690 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000c460 tx=0x7fb3a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0016020 con 0x7fb3ac0d2710 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a0012870 con 0x7fb3ac0d2710 2024-03-06T07:29:22.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac0b9220 con 0x7fb3ac0d2710 2024-03-06T07:29:22.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0012c60 con 0x7fb3ac0d2710 2024-03-06T07:29:22.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.277+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac15f370 con 0x7fb3ac0d2710 2024-03-06T07:29:22.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.278+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac0cabd0 con 0x7fb3ac0d2710 2024-03-06T07:29:22.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.278+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3a0019020 con 0x7fb3ac0d2710 2024-03-06T07:29:22.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.279+0000 7fb39affd640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb38c06b4d0 0x7fb38c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:22.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.279+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3a0089bf0 con 0x7fb3ac0d2710 2024-03-06T07:29:22.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.279+0000 7fb3aa59c640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb38c06b4d0 0x7fb38c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:22.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.279+0000 7fb3aa59c640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb38c06b4d0 0x7fb38c06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4000f60 tx=0x7fb3a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:22.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.287+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3a0020070 con 0x7fb3ac0d2710 2024-03-06T07:29:22.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.501+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3ac04f330 con 0x7fb3ac0d2710 2024-03-06T07:29:22.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.501+0000 7fb39affd640 1 -- 172.21.15.72:0/3224414043 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fb3a0058b00 con 0x7fb3ac0d2710 2024-03-06T07:29:22.504 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:22.504 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:22.504 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:22.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb38c06b4d0 msgr2=0x7fb38c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb38c06b4d0 0x7fb38c06d990 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4000f60 tx=0x7fb3a4009040 comp rx=0 tx=0).stop 2024-03-06T07:29:22.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 msgr2=0x7fb3ac0ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0ca690 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fb3a000c460 tx=0x7fb3a000c920 comp rx=0 tx=0).stop 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 shutdown_connections 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb38c06b4d0 0x7fb38c06d990 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ac15dfe0 0x7fb3ac159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ac0d2710 0x7fb3ac0ca690 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 --2- 172.21.15.72:0/3224414043 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ac0d22b0 0x7fb3ac0b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 >> 172.21.15.72:0/3224414043 conn(0x7fb3ac0d3cd0 msgr2=0x7fb3ac06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 shutdown_connections 2024-03-06T07:29:22.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:22.504+0000 7fb3b0e88640 1 -- 172.21.15.72:0/3224414043 wait complete. 2024-03-06T07:29:22.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:22.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:23.669 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:23.719 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:23.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.819+0000 7f7a0d481640 1 Processor -- start 2024-03-06T07:29:23.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.819+0000 7f7a0d481640 1 -- start start 2024-03-06T07:29:23.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a0d481640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7a080d00c0 0x7f7a080d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a0d481640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7a080cf850 0x7f7a080cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a0d481640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a080d09e0 con 0x7f7a080ce1b0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a0d481640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0806ca90 con 0x7f7a080d00c0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a0d481640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0806df30 con 0x7f7a080cf850 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a06ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7a080d00c0 0x7f7a080d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a06ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7a080d00c0 0x7f7a080d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44746/0 (socket says 172.21.15.72:44746) 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.820+0000 7f7a06ffd640 1 -- 172.21.15.72:0/3492892378 learned_addr learned my addr 172.21.15.72:0/3492892378 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a067fc640 1 --2- 172.21.15.72:0/3492892378 >> v2:172.21.15.146:3300/0 conn(0x7f7a080cf850 0x7f7a080cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a06ffd640 1 -- 172.21.15.72:0/3492892378 >> v2:172.21.15.146:3300/0 conn(0x7f7a080cf850 msgr2=0x7f7a080cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a06ffd640 1 --2- 172.21.15.72:0/3492892378 >> v2:172.21.15.146:3300/0 conn(0x7f7a080cf850 0x7f7a080cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a06ffd640 1 -- 172.21.15.72:0/3492892378 >> v1:172.21.15.146:6789/0 conn(0x7f7a080ce1b0 legacy=0x7f7a080ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a06ffd640 1 -- 172.21.15.72:0/3492892378 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0815e380 con 0x7f7a080d00c0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a067fc640 1 --2- 172.21.15.72:0/3492892378 >> v2:172.21.15.146:3300/0 conn(0x7f7a080cf850 0x7f7a080cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.821+0000 7f7a06ffd640 1 --2- 172.21.15.72:0/3492892378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080d00c0 0x7f7a080d04a0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7a08071de0 tx=0x7f79f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d36f3d830beb6c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a05ffb640 1 -- 172.21.15.72:0/3492892378 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0035020 con 0x7f7a080d00c0 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a05ffb640 1 -- 172.21.15.72:0/3492892378 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f0031870 con 0x7f7a080d00c0 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a05ffb640 1 -- 172.21.15.72:0/3492892378 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0031c40 con 0x7f7a080d00c0 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 -- 172.21.15.72:0/3492892378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080d00c0 msgr2=0x7f7a080d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 --2- 172.21.15.72:0/3492892378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080d00c0 0x7f7a080d04a0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f7a08071de0 tx=0x7f79f002b910 comp rx=0 tx=0).stop 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 -- 172.21.15.72:0/3492892378 shutdown_connections 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 --2- 172.21.15.72:0/3492892378 >> v2:172.21.15.146:3300/0 conn(0x7f7a080cf850 0x7f7a080cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 --2- 172.21.15.72:0/3492892378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080d00c0 0x7f7a080d04a0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 -- 172.21.15.72:0/3492892378 >> 172.21.15.72:0/3492892378 conn(0x7f7a080b35a0 msgr2=0x7f7a080b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 -- 172.21.15.72:0/3492892378 shutdown_connections 2024-03-06T07:29:23.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.822+0000 7f7a0d481640 1 -- 172.21.15.72:0/3492892378 wait complete. 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.824+0000 7f7a0d481640 1 Processor -- start 2024-03-06T07:29:23.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.824+0000 7f7a0d481640 1 -- start start 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.825+0000 7f7a0d481640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 0x7f7a080c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.825+0000 7f7a0d481640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a080cf850 0x7f7a08149b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.825+0000 7f7a0d481640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a080d00c0 0x7f7a0815e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.825+0000 7f7a0d481640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0806c440 con 0x7f7a080ce1b0 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.825+0000 7f7a0d481640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0806c160 con 0x7f7a080cf850 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a0d481640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a0806c720 con 0x7f7a080d00c0 2024-03-06T07:29:23.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 0x7f7a080c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 0x7f7a080c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44758/0 (socket says 172.21.15.72:44758) 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 -- 172.21.15.72:0/1248934903 learned_addr learned my addr 172.21.15.72:0/1248934903 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a067fc640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a080cf850 0x7f7a08149b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 -- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a080d00c0 msgr2=0x7f7a0815e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a080d00c0 0x7f7a0815e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 -- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a080cf850 msgr2=0x7f7a08149b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a080cf850 0x7f7a08149b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 -- 172.21.15.72:0/1248934903 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a081617d0 con 0x7f7a080ce1b0 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a067fc640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a080cf850 0x7f7a08149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a06ffd640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 0x7f7a080c7f30 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f79f0000c00 tx=0x7f79f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f00443f0 con 0x7f7a080ce1b0 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f0044590 con 0x7f7a080ce1b0 2024-03-06T07:29:23.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0044980 con 0x7f7a080ce1b0 2024-03-06T07:29:23.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a080ae8b0 con 0x7f7a080ce1b0 2024-03-06T07:29:23.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.826+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a080aee70 con 0x7f7a080ce1b0 2024-03-06T07:29:23.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.827+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a0804f330 con 0x7f7a080ce1b0 2024-03-06T07:29:23.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.828+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f79f004b060 con 0x7f7a080ce1b0 2024-03-06T07:29:23.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.828+0000 7f79ff7fe640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f79e806b4d0 0x7f79e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:23.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.828+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79f00ac4c0 con 0x7f7a080ce1b0 2024-03-06T07:29:23.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.828+0000 7f7a067fc640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f79e806b4d0 0x7f79e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:23.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.829+0000 7f7a067fc640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f79e806b4d0 0x7f79e806d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f79f8006fd0 tx=0x7f79f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:23.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:23.833+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f79f003f070 con 0x7f7a080ce1b0 2024-03-06T07:29:24.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.089+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a0814a0c0 con 0x7f7a080ce1b0 2024-03-06T07:29:24.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.089+0000 7f79ff7fe640 1 -- 172.21.15.72:0/1248934903 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f79f002f320 con 0x7f7a080ce1b0 2024-03-06T07:29:24.092 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:24.092 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:24.092 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f79e806b4d0 msgr2=0x7f79e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f79e806b4d0 0x7f79e806d990 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f79f8006fd0 tx=0x7f79f8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 msgr2=0x7f7a080c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 0x7f7a080c7f30 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f79f0000c00 tx=0x7f79f00046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 shutdown_connections 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f79e806b4d0 0x7f79e806d990 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a080d00c0 0x7f7a0815e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a080cf850 0x7f7a08149b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 --2- 172.21.15.72:0/1248934903 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7a080ce1b0 0x7f7a080c7f30 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.091+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 >> 172.21.15.72:0/1248934903 conn(0x7f7a080b35a0 msgr2=0x7f7a080b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.092+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 shutdown_connections 2024-03-06T07:29:24.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:24.092+0000 7f7a0d481640 1 -- 172.21.15.72:0/1248934903 wait complete. 2024-03-06T07:29:24.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:24.306 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:25.308 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:25.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.407+0000 7efc1cdf7640 1 Processor -- start 2024-03-06T07:29:25.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.407+0000 7efc1cdf7640 1 -- start start 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc1cdf7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc180cf850 0x7efc180cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc1cdf7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc1806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc1cdf7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1815e380 con 0x7efc180d00c0 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc1cdf7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1806ca90 con 0x7efc180cf850 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc1cdf7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc180cdc10 con 0x7efc1806cf20 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc15d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc180cf850 0x7efc180cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc15d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efc180cf850 0x7efc180cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44764/0 (socket says 172.21.15.72:44764) 2024-03-06T07:29:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.408+0000 7efc15d9b640 1 -- 172.21.15.72:0/297629281 learned_addr learned my addr 172.21.15.72:0/297629281 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc15d9b640 1 -- 172.21.15.72:0/297629281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 msgr2=0x7efc1806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc15d9b640 1 --2- 172.21.15.72:0/297629281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc1806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc15d9b640 1 -- 172.21.15.72:0/297629281 >> v1:172.21.15.72:6789/0 conn(0x7efc180d00c0 legacy=0x7efc180d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc15d9b640 1 -- 172.21.15.72:0/297629281 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1806c160 con 0x7efc180cf850 2024-03-06T07:29:25.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc15d9b640 1 --2- 172.21.15.72:0/297629281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180cd3c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7efc08009870 tx=0x7efc0802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d43ea08d664f355 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc1559a640 1 -- 172.21.15.72:0/297629281 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08034070 con 0x7efc180cf850 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc1559a640 1 -- 172.21.15.72:0/297629281 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08030840 con 0x7efc180cf850 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.409+0000 7efc1559a640 1 -- 172.21.15.72:0/297629281 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08030c10 con 0x7efc180cf850 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 -- 172.21.15.72:0/297629281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 msgr2=0x7efc180cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/297629281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180cd3c0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7efc08009870 tx=0x7efc0802b980 comp rx=0 tx=0).stop 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 -- 172.21.15.72:0/297629281 shutdown_connections 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/297629281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc1806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/297629281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180cd3c0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 -- 172.21.15.72:0/297629281 >> 172.21.15.72:0/297629281 conn(0x7efc180b35a0 msgr2=0x7efc180b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 -- 172.21.15.72:0/297629281 shutdown_connections 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.410+0000 7efc1cdf7640 1 -- 172.21.15.72:0/297629281 wait complete. 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.411+0000 7efc1cdf7640 1 Processor -- start 2024-03-06T07:29:25.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.411+0000 7efc1cdf7640 1 -- start start 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1cdf7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc180c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1cdf7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1cdf7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc180d00c0 0x7efc180c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1cdf7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc180cd900 con 0x7efc180cf850 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1cdf7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1806c720 con 0x7efc180d00c0 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1cdf7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc180b1030 con 0x7efc1806cf20 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc15d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc15d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44766/0 (socket says 172.21.15.72:44766) 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc15d9b640 1 -- 172.21.15.72:0/598230939 learned_addr learned my addr 172.21.15.72:0/598230939 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc1659c640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc180c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.412+0000 7efc16d9d640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc180d00c0 0x7efc180c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc15d9b640 1 -- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 msgr2=0x7efc180c55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc15d9b640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc180c55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc15d9b640 1 -- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc180d00c0 msgr2=0x7efc180c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc15d9b640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc180d00c0 0x7efc180c7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc15d9b640 1 -- 172.21.15.72:0/598230939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc180c9220 con 0x7efc180cf850 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc1659c640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc180c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc16d9d640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc180d00c0 0x7efc180c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc15d9b640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180c5ae0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08048070 con 0x7efc180cf850 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc08039d90 con 0x7efc180cf850 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc08034040 con 0x7efc180cf850 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc180c9500 con 0x7efc180cf850 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.413+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc180c9ac0 con 0x7efc180cf850 2024-03-06T07:29:25.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.414+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1804f330 con 0x7efc180cf850 2024-03-06T07:29:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.414+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efc0804b070 con 0x7efc180cf850 2024-03-06T07:29:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.415+0000 7efc06ffd640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efbf806b4d0 0x7efbf806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.415+0000 7efc1659c640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efbf806b4d0 0x7efbf806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.415+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc080ac320 con 0x7efc180cf850 2024-03-06T07:29:25.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.415+0000 7efc1659c640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efbf806b4d0 0x7efbf806d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7efc0c0015e0 tx=0x7efc0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:25.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.419+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efc0807b2c0 con 0x7efc180cf850 2024-03-06T07:29:25.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.633+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc180b3ab0 con 0x7efc180cf850 2024-03-06T07:29:25.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.633+0000 7efc06ffd640 1 -- 172.21.15.72:0/598230939 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7efc0807abc0 con 0x7efc180cf850 2024-03-06T07:29:25.636 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:25.636 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:25.636 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efbf806b4d0 msgr2=0x7efbf806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efbf806b4d0 0x7efbf806d990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7efc0c0015e0 tx=0x7efc0c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 msgr2=0x7efc180c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180c5ae0 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc08004730 comp rx=0 tx=0).stop 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 shutdown_connections 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efbf806b4d0 0x7efbf806d990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc180d00c0 0x7efc180c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc180cf850 0x7efc180c5ae0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 --2- 172.21.15.72:0/598230939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc1806cf20 0x7efc180c55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 >> 172.21.15.72:0/598230939 conn(0x7efc180b35a0 msgr2=0x7efc180b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:25.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 shutdown_connections 2024-03-06T07:29:25.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:25.635+0000 7efc1cdf7640 1 -- 172.21.15.72:0/598230939 wait complete. 2024-03-06T07:29:25.670 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:25.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:25.850 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:26.851 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:26.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.950+0000 7f6417e4e640 1 Processor -- start 2024-03-06T07:29:26.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.950+0000 7f6417e4e640 1 -- start start 2024-03-06T07:29:26.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f6417e4e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f641015dfe0 0x7f64100cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f6417e4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100cf830 0x7f64100d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f6417e4e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100b3430 con 0x7f64100d27f0 2024-03-06T07:29:26.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f6417e4e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64100d2c50 con 0x7f641015dfe0 2024-03-06T07:29:26.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f6417e4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64100b31e0 con 0x7f64100cf830 2024-03-06T07:29:26.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f64153e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100cf830 0x7f64100d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f6415bea640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f641015dfe0 0x7f64100cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f64163eb640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f64100d27f0 0x7f64100d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:34270/0 (socket says 172.21.15.72:34270) 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.951+0000 7f64163eb640 1 -- 172.21.15.72:0/2659853864 learned_addr learned my addr 172.21.15.72:0/2659853864 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.952+0000 7f64153e9640 1 -- 172.21.15.72:0/2659853864 >> v2:172.21.15.146:3300/0 conn(0x7f641015dfe0 msgr2=0x7f64100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.952+0000 7f64153e9640 1 --2- 172.21.15.72:0/2659853864 >> v2:172.21.15.146:3300/0 conn(0x7f641015dfe0 0x7f64100cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.952+0000 7f64153e9640 1 -- 172.21.15.72:0/2659853864 >> v1:172.21.15.146:6789/0 conn(0x7f64100d27f0 legacy=0x7f64100d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.952+0000 7f64153e9640 1 -- 172.21.15.72:0/2659853864 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100b3710 con 0x7f64100cf830 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.952+0000 7f6414be8640 1 -- 172.21.15.72:0/2659853864 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377178517 0 0) 0x7f64100b3430 con 0x7f64100d27f0 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.952+0000 7f6415bea640 1 --2- 172.21.15.72:0/2659853864 >> v2:172.21.15.146:3300/0 conn(0x7f641015dfe0 0x7f64100cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.953+0000 7f64153e9640 1 --2- 172.21.15.72:0/2659853864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100cf830 0x7f64100d22b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f64000097a0 tx=0x7f640002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eccb7b1892f4bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.953+0000 7f6414be8640 1 -- 172.21.15.72:0/2659853864 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400035020 con 0x7f64100cf830 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.953+0000 7f6414be8640 1 -- 172.21.15.72:0/2659853864 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6400031830 con 0x7f64100cf830 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.953+0000 7f6414be8640 1 -- 172.21.15.72:0/2659853864 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6400031c00 con 0x7f64100cf830 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.953+0000 7f6417e4e640 1 -- 172.21.15.72:0/2659853864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100cf830 msgr2=0x7f64100d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.953+0000 7f6417e4e640 1 --2- 172.21.15.72:0/2659853864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100cf830 0x7f64100d22b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f64000097a0 tx=0x7f640002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.954+0000 7f6417e4e640 1 -- 172.21.15.72:0/2659853864 shutdown_connections 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.954+0000 7f6417e4e640 1 --2- 172.21.15.72:0/2659853864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100cf830 0x7f64100d22b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.954+0000 7f6417e4e640 1 --2- 172.21.15.72:0/2659853864 >> v2:172.21.15.146:3300/0 conn(0x7f641015dfe0 0x7f64100cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.954+0000 7f6417e4e640 1 -- 172.21.15.72:0/2659853864 >> 172.21.15.72:0/2659853864 conn(0x7f64100d3cd0 msgr2=0x7f64100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.954+0000 7f6417e4e640 1 -- 172.21.15.72:0/2659853864 shutdown_connections 2024-03-06T07:29:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.954+0000 7f6417e4e640 1 -- 172.21.15.72:0/2659853864 wait complete. 2024-03-06T07:29:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.955+0000 7f6417e4e640 1 Processor -- start 2024-03-06T07:29:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.955+0000 7f6417e4e640 1 -- start start 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6417e4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64100cf830 0x7f641014e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6417e4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100d27f0 0x7f641006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6417e4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 0x7f641006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6417e4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100d1c70 con 0x7f641015dfe0 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6417e4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100b3d60 con 0x7f64100cf830 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6417e4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64100d1df0 con 0x7f64100d27f0 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 0x7f641006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 0x7f641006bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44774/0 (socket says 172.21.15.72:44774) 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 -- 172.21.15.72:0/674294848 learned_addr learned my addr 172.21.15.72:0/674294848 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64153e9640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100d27f0 0x7f641006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f6415bea640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64100cf830 0x7f641014e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 -- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100d27f0 msgr2=0x7f641006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100d27f0 0x7f641006ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 -- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64100cf830 msgr2=0x7f641014e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64100cf830 0x7f641014e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 -- 172.21.15.72:0/674294848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100b9840 con 0x7f641015dfe0 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64163eb640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 0x7f641006bf40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f640800c7a0 tx=0x7f640800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.957+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6408018070 con 0x7f641015dfe0 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.958+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640800a640 con 0x7f641015dfe0 2024-03-06T07:29:26.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.958+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64100b9b20 con 0x7f641015dfe0 2024-03-06T07:29:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.958+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640800aab0 con 0x7f641015dfe0 2024-03-06T07:29:26.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.958+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f641015ea50 con 0x7f641015dfe0 2024-03-06T07:29:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.959+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f641004f330 con 0x7f641015dfe0 2024-03-06T07:29:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.959+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f640801b070 con 0x7f641015dfe0 2024-03-06T07:29:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.959+0000 7f64067fc640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63f406b4d0 0x7f63f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.959+0000 7f6415bea640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63f406b4d0 0x7f63f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.959+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6408089700 con 0x7f641015dfe0 2024-03-06T07:29:26.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.960+0000 7f6415bea640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63f406b4d0 0x7f63f406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f640c009560 tx=0x7f640c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:26.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:26.964+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6408057fc0 con 0x7f641015dfe0 2024-03-06T07:29:27.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.343+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64100d0ed0 con 0x7f641015dfe0 2024-03-06T07:29:27.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.343+0000 7f64067fc640 1 -- 172.21.15.72:0/674294848 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f640805c060 con 0x7f641015dfe0 2024-03-06T07:29:27.345 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:27.346 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:27.346 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:27.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63f406b4d0 msgr2=0x7f63f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63f406b4d0 0x7f63f406d990 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f640c009560 tx=0x7f640c009190 comp rx=0 tx=0).stop 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 msgr2=0x7f641006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 0x7f641006bf40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f640800c7a0 tx=0x7f640800cc60 comp rx=0 tx=0).stop 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 shutdown_connections 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f63f406b4d0 0x7f63f406d990 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f641015dfe0 0x7f641006bf40 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64100d27f0 0x7f641006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 --2- 172.21.15.72:0/674294848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64100cf830 0x7f641014e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 >> 172.21.15.72:0/674294848 conn(0x7f64100d3cd0 msgr2=0x7f64100d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 shutdown_connections 2024-03-06T07:29:27.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:27.345+0000 7f6417e4e640 1 -- 172.21.15.72:0/674294848 wait complete. 2024-03-06T07:29:27.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:27.559 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:27.673 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:28.561 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:28.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.661+0000 7ff187084640 1 Processor -- start 2024-03-06T07:29:28.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.661+0000 7ff187084640 1 -- start start 2024-03-06T07:29:28.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff187084640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff1800b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff187084640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1800b3430 con 0x7ff1800d22b0 2024-03-06T07:29:28.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff187084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1800b3710 con 0x7ff18015dfe0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff187084640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1800b31e0 con 0x7ff1800d2710 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff1800d22b0 0x7ff1800b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54722/0 (socket says 172.21.15.72:54722) 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17ffff640 1 -- 172.21.15.72:0/3389966240 learned_addr learned my addr 172.21.15.72:0/3389966240 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564388020 0 0) 0x7ff1800b3430 con 0x7ff1800d22b0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff170003520 con 0x7ff1800d22b0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff185621640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff1800b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227066270 0 0) 0x7ff170003520 con 0x7ff1800d22b0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 msgr2=0x7ff1800b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff1800b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 >> v1:172.21.15.146:6789/0 conn(0x7ff18015dfe0 legacy=0x7ff1800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1800b39f0 con 0x7ff1800d22b0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff16c001670 con 0x7ff1800d22b0 2024-03-06T07:29:28.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 >> v1:172.21.15.72:6789/0 conn(0x7ff1800d22b0 legacy=0x7ff1800b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.662+0000 7ff17f7fe640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff170003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff17f7fe640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff170004030 0x7ff170004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff17f7fe640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1800b31e0 con 0x7ff170004d40 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000bb80 con 0x7ff170004030 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17000bd80 con 0x7ff1800d2710 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff185621640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff17ffff640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff170004030 0x7ff170004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff184e20640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff170003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff185621640 1 -- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 msgr2=0x7ff170003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff185621640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff170003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff185621640 1 -- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff170004030 msgr2=0x7ff170004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:28.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff185621640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff170004030 0x7ff170004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff185621640 1 -- 172.21.15.72:0/3389966240 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff170003520 con 0x7ff170004d40 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.663+0000 7ff184e20640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1800d2710 0x7ff170003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff185621640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1700051c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7ff178007360 tx=0x7ff170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8136a40d4fc062d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178039070 con 0x7ff170004d40 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17802ccd0 con 0x7ff170004d40 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff17f7fe640 1 -- 172.21.15.72:0/3389966240 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178009be0 con 0x7ff170004d40 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff187084640 1 -- 172.21.15.72:0/3389966240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 msgr2=0x7ff1700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff187084640 1 --2- 172.21.15.72:0/3389966240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1700051c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7ff178007360 tx=0x7ff170005830 comp rx=0 tx=0).stop 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff185621640 1 -- 172.21.15.72:0/3389966240 reap_dead start 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff187084640 1 -- 172.21.15.72:0/3389966240 shutdown_connections 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff187084640 1 -- 172.21.15.72:0/3389966240 >> 172.21.15.72:0/3389966240 conn(0x7ff1800d3cd0 msgr2=0x7ff1800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff187084640 1 -- 172.21.15.72:0/3389966240 shutdown_connections 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.664+0000 7ff187084640 1 -- 172.21.15.72:0/3389966240 wait complete. 2024-03-06T07:29:28.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.665+0000 7ff187084640 1 Processor -- start 2024-03-06T07:29:28.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 -- start start 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1801f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff180151fb0 0x7ff180160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff180160bd0 0x7ff18009fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1800cfa00 con 0x7ff170004d40 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1800b3d60 con 0x7ff180160bd0 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff187084640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1800d1c70 con 0x7ff180151fb0 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff184e20640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1801f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff184e20640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1801f47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44788/0 (socket says 172.21.15.72:44788) 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.666+0000 7ff184e20640 1 -- 172.21.15.72:0/1966665926 learned_addr learned my addr 172.21.15.72:0/1966665926 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff17ffff640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff180151fb0 0x7ff180160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff185621640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff180160bd0 0x7ff18009fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff184e20640 1 -- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff180151fb0 msgr2=0x7ff180160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff184e20640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff180151fb0 0x7ff180160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff184e20640 1 -- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff180160bd0 msgr2=0x7ff18009fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff184e20640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff180160bd0 0x7ff18009fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff184e20640 1 -- 172.21.15.72:0/1966665926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1800a0b50 con 0x7ff170004d40 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff17ffff640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff180151fb0 0x7ff180160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff185621640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff180160bd0 0x7ff18009fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff184e20640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1801f47d0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7ff17400e600 tx=0x7ff17400eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1740052e0 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff174018070 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff174003810 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18009dc70 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.667+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18009e230 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.668+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff180160fe0 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.668+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1740039b0 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.669+0000 7ff17d7fa640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff16406b500 0x7ff16406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.669+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff174089f40 con 0x7ff170004d40 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.669+0000 7ff17ffff640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff16406b500 0x7ff16406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:28.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.669+0000 7ff17ffff640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff16406b500 0x7ff16406d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff1800cde50 tx=0x7ff16c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:28.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.673+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff180160fe0 con 0x7ff170004d40 2024-03-06T07:29:28.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.890+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1800b9120 con 0x7ff170004d40 2024-03-06T07:29:28.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.891+0000 7ff17d7fa640 1 -- 172.21.15.72:0/1966665926 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7ff174058dc0 con 0x7ff170004d40 2024-03-06T07:29:28.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:28.893 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:28.893 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.893+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff16406b500 msgr2=0x7ff16406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.893+0000 7ff187084640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff16406b500 0x7ff16406d9c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7ff1800cde50 tx=0x7ff16c009040 comp rx=0 tx=0).stop 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.893+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 msgr2=0x7ff1801f47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.893+0000 7ff187084640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1801f47d0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7ff17400e600 tx=0x7ff17400eac0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.893+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 shutdown_connections 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff16406b500 0x7ff16406d9c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff180160bd0 0x7ff18009fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff180151fb0 0x7ff180160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 --2- 172.21.15.72:0/1966665926 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff170004d40 0x7ff1801f47d0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 >> 172.21.15.72:0/1966665926 conn(0x7ff1800d3cd0 msgr2=0x7ff1800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 shutdown_connections 2024-03-06T07:29:28.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:28.894+0000 7ff187084640 1 -- 172.21.15.72:0/1966665926 wait complete. 2024-03-06T07:29:29.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:29.111 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:29.675 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:30.112 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.214+0000 7f9869393640 1 Processor -- start 2024-03-06T07:29:30.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.214+0000 7f9869393640 1 -- start start 2024-03-06T07:29:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9869393640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98640d4760 0x7f98640b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9869393640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f98640d0a80 0x7f98640cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9869393640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98640d25d0 con 0x7f98640b4170 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9869393640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98640d4bf0 con 0x7f98640d0a80 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9869393640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98640ce9e0 con 0x7f98640d4760 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9862ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f98640b4170 0x7f98640d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54732/0 (socket says 172.21.15.72:54732) 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9862ffd640 1 -- 172.21.15.72:0/1401262012 learned_addr learned my addr 172.21.15.72:0/1401262012 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f98637fe640 1 --2- 172.21.15.72:0/1401262012 >> v2:172.21.15.146:3300/0 conn(0x7f98640d0a80 0x7f98640cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f98627fc640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98640d4760 0x7f98640b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081968403 0 0) 0x7f98640d25d0 con 0x7f98640b4170 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.215+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f984c003520 con 0x7f98640b4170 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775764662 0 0) 0x7f984c003520 con 0x7f98640b4170 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98640d4760 msgr2=0x7f98640b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98640d4760 0x7f98640b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 >> v2:172.21.15.146:3300/0 conn(0x7f98640d0a80 msgr2=0x7f98640cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 --2- 172.21.15.72:0/1401262012 >> v2:172.21.15.146:3300/0 conn(0x7f98640d0a80 0x7f98640cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98640d2870 con 0x7f98640b4170 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9858001670 con 0x7f98640b4170 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 >> v1:172.21.15.72:6789/0 conn(0x7f98640b4170 legacy=0x7f98640d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c004800 0x7f984c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f984c005180 0x7f984c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640d4bf0 con 0x7f984c0039b0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640ce9e0 con 0x7f984c005180 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c004630 con 0x7f984c004800 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9862ffd640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f98627fc640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c004800 0x7f984c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f98637fe640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f984c005180 0x7f984c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9862ffd640 1 -- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c004800 msgr2=0x7f984c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9862ffd640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c004800 0x7f984c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9862ffd640 1 -- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f984c005180 msgr2=0x7f984c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9862ffd640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f984c005180 0x7f984c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f9862ffd640 1 -- 172.21.15.72:0/1401262012 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c003520 con 0x7f984c0039b0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.216+0000 7f98637fe640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f984c005180 0x7f984c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.217+0000 7f9862ffd640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f985800c210 tx=0x7f984c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f98bfb0443edaee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.217+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9858007d70 con 0x7f984c0039b0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.217+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9858030070 con 0x7f984c0039b0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.217+0000 7f9861ffb640 1 -- 172.21.15.72:0/1401262012 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985802cd60 con 0x7f984c0039b0 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f9869393640 1 -- 172.21.15.72:0/1401262012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f984c0039b0 msgr2=0x7f984c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f9869393640 1 --2- 172.21.15.72:0/1401262012 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f984c0039b0 0x7f984c003dd0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f985800c210 tx=0x7f984c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f98637fe640 1 -- 172.21.15.72:0/1401262012 reap_dead start 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f9869393640 1 -- 172.21.15.72:0/1401262012 shutdown_connections 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f9869393640 1 -- 172.21.15.72:0/1401262012 >> 172.21.15.72:0/1401262012 conn(0x7f98640d3c40 msgr2=0x7f98640d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f9869393640 1 -- 172.21.15.72:0/1401262012 shutdown_connections 2024-03-06T07:29:30.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.218+0000 7f9869393640 1 -- 172.21.15.72:0/1401262012 wait complete. 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.219+0000 7f9869393640 1 Processor -- start 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.219+0000 7f9869393640 1 -- start start 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c0039b0 0x7f98641553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 0x7f98640cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98640c8950 0x7f98640c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640b4dd0 con 0x7f98640b3600 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98640b4550 con 0x7f98640c8950 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98640b4f50 con 0x7f984c0039b0 2024-03-06T07:29:30.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 0x7f98640cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 0x7f98640cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44798/0 (socket says 172.21.15.72:44798) 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 -- 172.21.15.72:0/922010959 learned_addr learned my addr 172.21.15.72:0/922010959 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 -- 172.21.15.72:0/922010959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c0039b0 msgr2=0x7f98641553a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c0039b0 0x7f98641553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 -- 172.21.15.72:0/922010959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98640c8950 msgr2=0x7f98640c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98640c8950 0x7f98640c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 -- 172.21.15.72:0/922010959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98640cc020 con 0x7f98640b3600 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f98627fc640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 0x7f98640cb720 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f9854011570 tx=0x7f9854011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854015930 con 0x7f98640b3600 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9854015ad0 con 0x7f98640b3600 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985401d400 con 0x7f98640b3600 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.220+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98640cc300 con 0x7f98640b3600 2024-03-06T07:29:30.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.221+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98640cc8c0 con 0x7f98640b3600 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.222+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98640b3d20 con 0x7f98640b3600 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.222+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f985401d5a0 con 0x7f98640b3600 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.222+0000 7f984b7fe640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.222+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f985408aa90 con 0x7f98640b3600 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.222+0000 7f9862ffd640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.223+0000 7f9862ffd640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9858034000 tx=0x7f985802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:30.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.226+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98640b3d20 con 0x7f98640b3600 2024-03-06T07:29:30.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.439+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f986404f330 con 0x7f98640b3600 2024-03-06T07:29:30.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.440+0000 7f984b7fe640 1 -- 172.21.15.72:0/922010959 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f9854019030 con 0x7f98640b3600 2024-03-06T07:29:30.442 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:30.442 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:30.442 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f985006b4d0 msgr2=0x7f985006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f985006b4d0 0x7f985006d990 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9858034000 tx=0x7f985802e000 comp rx=0 tx=0).stop 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 msgr2=0x7f98640cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 0x7f98640cb720 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f9854011570 tx=0x7f9854011a30 comp rx=0 tx=0).stop 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 shutdown_connections 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f985006b4d0 0x7f985006d990 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98640c8950 0x7f98640c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98640b3600 0x7f98640cb720 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.441+0000 7f9869393640 1 --2- 172.21.15.72:0/922010959 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f984c0039b0 0x7f98641553a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:30.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.442+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 >> 172.21.15.72:0/922010959 conn(0x7f98640d3c40 msgr2=0x7f98640b7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:30.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.442+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 shutdown_connections 2024-03-06T07:29:30.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:30.442+0000 7f9869393640 1 -- 172.21.15.72:0/922010959 wait complete. 2024-03-06T07:29:30.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:30.656 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:31.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:31.676 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:31.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.768+0000 7f9b6a7ae640 1 Processor -- start 2024-03-06T07:29:31.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.768+0000 7f9b6a7ae640 1 -- start start 2024-03-06T07:29:31.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b6a7ae640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9b6415dfe0 0x7f9b640cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b6a7ae640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b640cf830 0x7f9b640d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b6a7ae640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640b3430 con 0x7f9b640d27f0 2024-03-06T07:29:31.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b6a7ae640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b640b31e0 con 0x7f9b640cf830 2024-03-06T07:29:31.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b6a7ae640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b640b3d60 con 0x7f9b6415dfe0 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b640cf830 0x7f9b640d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b640cf830 0x7f9b640d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44810/0 (socket says 172.21.15.72:44810) 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 -- 172.21.15.72:0/3120238421 learned_addr learned my addr 172.21.15.72:0/3120238421 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 -- 172.21.15.72:0/3120238421 >> v2:172.21.15.146:3300/0 conn(0x7f9b6415dfe0 msgr2=0x7f9b640cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 --2- 172.21.15.72:0/3120238421 >> v2:172.21.15.146:3300/0 conn(0x7f9b6415dfe0 0x7f9b640cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 -- 172.21.15.72:0/3120238421 >> v1:172.21.15.146:6789/0 conn(0x7f9b640d27f0 legacy=0x7f9b640d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:31.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.769+0000 7f9b637fe640 1 -- 172.21.15.72:0/3120238421 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b640b3710 con 0x7f9b640cf830 2024-03-06T07:29:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b637fe640 1 --2- 172.21.15.72:0/3120238421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640cf830 0x7f9b640d22b0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009870 tx=0x7f9b5402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c7f7936d81d441 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b62ffd640 1 -- 172.21.15.72:0/3120238421 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54035020 con 0x7f9b640cf830 2024-03-06T07:29:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b62ffd640 1 -- 172.21.15.72:0/3120238421 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b54031870 con 0x7f9b640cf830 2024-03-06T07:29:31.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b62ffd640 1 -- 172.21.15.72:0/3120238421 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b54031c40 con 0x7f9b640cf830 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/3120238421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640cf830 msgr2=0x7f9b640d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/3120238421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640cf830 0x7f9b640d22b0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009870 tx=0x7f9b5402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/3120238421 shutdown_connections 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/3120238421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b640cf830 0x7f9b640d22b0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/3120238421 >> v2:172.21.15.146:3300/0 conn(0x7f9b6415dfe0 0x7f9b640cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/3120238421 >> 172.21.15.72:0/3120238421 conn(0x7f9b640d3cd0 msgr2=0x7f9b640d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.770+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/3120238421 shutdown_connections 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.771+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/3120238421 wait complete. 2024-03-06T07:29:31.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 Processor -- start 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 -- start start 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b640cf830 0x7f9b6414e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b640d27f0 0x7f9b640b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 0x7f9b640b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640d1df0 con 0x7f9b6415dfe0 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640d1c70 con 0x7f9b640d27f0 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.772+0000 7f9b6a7ae640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6415e3c0 con 0x7f9b640cf830 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 0x7f9b640b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 0x7f9b640b24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44826/0 (socket says 172.21.15.72:44826) 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 -- 172.21.15.72:0/1817957052 learned_addr learned my addr 172.21.15.72:0/1817957052 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:31.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b63fff640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b640cf830 0x7f9b6414e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b637fe640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b640d27f0 0x7f9b640b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 -- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b640cf830 msgr2=0x7f9b6414e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b640cf830 0x7f9b6414e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 -- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b640d27f0 msgr2=0x7f9b640b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b640d27f0 0x7f9b640b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 -- 172.21.15.72:0/1817957052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6414e7e0 con 0x7f9b6415dfe0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b637fe640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b640d27f0 0x7f9b640b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b63fff640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b640cf830 0x7f9b6414e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b68d4b640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 0x7f9b640b24b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00c780 tx=0x7f9b5c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c017070 con 0x7f9b6415dfe0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5c003da0 con 0x7f9b6415dfe0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c0052b0 con 0x7f9b6415dfe0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6415f1b0 con 0x7f9b6415dfe0 2024-03-06T07:29:31.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.773+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6415f770 con 0x7f9b6415dfe0 2024-03-06T07:29:31.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.774+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6404f330 con 0x7f9b6415dfe0 2024-03-06T07:29:31.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.775+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9b5c005450 con 0x7f9b6415dfe0 2024-03-06T07:29:31.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.775+0000 7f9b60ff9640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b4406b4d0 0x7f9b4406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:31.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.775+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b5c0897e0 con 0x7f9b6415dfe0 2024-03-06T07:29:31.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.775+0000 7f9b63fff640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b4406b4d0 0x7f9b4406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:31.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.776+0000 7f9b63fff640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b4406b4d0 0x7f9b4406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009930 tx=0x7f9b4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:31.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.780+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9b5c058660 con 0x7f9b6415dfe0 2024-03-06T07:29:31.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.995+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b640b2f60 con 0x7f9b6415dfe0 2024-03-06T07:29:31.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.995+0000 7f9b60ff9640 1 -- 172.21.15.72:0/1817957052 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f9b5c057f60 con 0x7f9b6415dfe0 2024-03-06T07:29:31.998 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:31.998 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:31.998 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b4406b4d0 msgr2=0x7f9b4406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b4406b4d0 0x7f9b4406d990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009930 tx=0x7f9b4c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 msgr2=0x7f9b640b24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 0x7f9b640b24b0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c00c780 tx=0x7f9b5c00cc40 comp rx=0 tx=0).stop 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 shutdown_connections 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9b4406b4d0 0x7f9b4406d990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b6415dfe0 0x7f9b640b24b0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b640d27f0 0x7f9b640b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 --2- 172.21.15.72:0/1817957052 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b640cf830 0x7f9b6414e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 >> 172.21.15.72:0/1817957052 conn(0x7f9b640d3cd0 msgr2=0x7f9b640cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:32.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 shutdown_connections 2024-03-06T07:29:32.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:31.997+0000 7f9b6a7ae640 1 -- 172.21.15.72:0/1817957052 wait complete. 2024-03-06T07:29:32.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:32.211 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:33.212 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:33.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.312+0000 7fbdfd655640 1 Processor -- start 2024-03-06T07:29:33.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdfd655640 1 -- start start 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdfd655640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbdf80cf850 0x7fbdf80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdfd655640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdf806cf20 0x7fbdf806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdfd655640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf815e380 con 0x7fbdf80d00c0 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdfd655640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf806ca90 con 0x7fbdf80cf850 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdfd655640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf80cdc10 con 0x7fbdf806cf20 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.313+0000 7fbdf67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbdf80cf850 0x7fbdf80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbdf80cf850 0x7fbdf80cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47226/0 (socket says 172.21.15.72:47226) 2024-03-06T07:29:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf67fc640 1 -- 172.21.15.72:0/2727819056 learned_addr learned my addr 172.21.15.72:0/2727819056 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf77fe640 1 --2- 172.21.15.72:0/2727819056 >> v2:172.21.15.146:3300/0 conn(0x7fbdf806cf20 0x7fbdf806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf67fc640 1 -- 172.21.15.72:0/2727819056 >> v2:172.21.15.146:3300/0 conn(0x7fbdf806cf20 msgr2=0x7fbdf806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf67fc640 1 --2- 172.21.15.72:0/2727819056 >> v2:172.21.15.146:3300/0 conn(0x7fbdf806cf20 0x7fbdf806d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf67fc640 1 -- 172.21.15.72:0/2727819056 >> v1:172.21.15.146:6789/0 conn(0x7fbdf80d00c0 legacy=0x7fbdf80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.314+0000 7fbdf67fc640 1 -- 172.21.15.72:0/2727819056 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf806c160 con 0x7fbdf80cf850 2024-03-06T07:29:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdf67fc640 1 --2- 172.21.15.72:0/2727819056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf80cf850 0x7fbdf80cd3c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8071f50 tx=0x7fbdec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0620e3e69b8383 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdf5ffb640 1 -- 172.21.15.72:0/2727819056 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec034070 con 0x7fbdf80cf850 2024-03-06T07:29:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdf5ffb640 1 -- 172.21.15.72:0/2727819056 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdec030860 con 0x7fbdf80cf850 2024-03-06T07:29:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdf5ffb640 1 -- 172.21.15.72:0/2727819056 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdec030c30 con 0x7fbdf80cf850 2024-03-06T07:29:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 -- 172.21.15.72:0/2727819056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf80cf850 msgr2=0x7fbdf80cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2727819056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf80cf850 0x7fbdf80cd3c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8071f50 tx=0x7fbdec02b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 -- 172.21.15.72:0/2727819056 shutdown_connections 2024-03-06T07:29:33.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2727819056 >> v2:172.21.15.146:3300/0 conn(0x7fbdf806cf20 0x7fbdf806d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2727819056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf80cf850 0x7fbdf80cd3c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 -- 172.21.15.72:0/2727819056 >> 172.21.15.72:0/2727819056 conn(0x7fbdf80b35a0 msgr2=0x7fbdf80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 -- 172.21.15.72:0/2727819056 shutdown_connections 2024-03-06T07:29:33.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.315+0000 7fbdfd655640 1 -- 172.21.15.72:0/2727819056 wait complete. 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 Processor -- start 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 -- start start 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 0x7fbdf8159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf80cf850 0x7fbdf815a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf80d00c0 0x7fbdf80ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80cd900 con 0x7fbdf806cf20 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf806c720 con 0x7fbdf80d00c0 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdfd655640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf80b1030 con 0x7fbdf80cf850 2024-03-06T07:29:33.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.317+0000 7fbdf6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 0x7fbdf8159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 0x7fbdf8159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47240/0 (socket says 172.21.15.72:47240) 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 -- 172.21.15.72:0/2501817288 learned_addr learned my addr 172.21.15.72:0/2501817288 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf67fc640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf80cf850 0x7fbdf815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf77fe640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf80d00c0 0x7fbdf80ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 -- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf80cf850 msgr2=0x7fbdf815a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf80cf850 0x7fbdf815a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 -- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf80d00c0 msgr2=0x7fbdf80ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf80d00c0 0x7fbdf80ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 -- 172.21.15.72:0/2501817288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf81f5050 con 0x7fbdf806cf20 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdf6ffd640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 0x7fbdf8159b40 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbde800c3c0 tx=0x7fbde800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80052d0 con 0x7fbdf806cf20 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8005490 con 0x7fbdf806cf20 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf81f5330 con 0x7fbdf806cf20 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8007260 con 0x7fbdf806cf20 2024-03-06T07:29:33.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.318+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf81f58f0 con 0x7fbdf806cf20 2024-03-06T07:29:33.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.319+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf804f330 con 0x7fbdf806cf20 2024-03-06T07:29:33.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.320+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbde8007400 con 0x7fbdf806cf20 2024-03-06T07:29:33.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.320+0000 7fbddf7fe640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbde406b500 0x7fbde406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:33.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.320+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbde8089600 con 0x7fbdf806cf20 2024-03-06T07:29:33.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.320+0000 7fbdf67fc640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbde406b500 0x7fbde406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:33.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.321+0000 7fbdf67fc640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbde406b500 0x7fbde406d9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbdec036000 tx=0x7fbdec03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:33.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.324+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbde80584d0 con 0x7fbdf806cf20 2024-03-06T07:29:33.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.600+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf80ced60 con 0x7fbdf806cf20 2024-03-06T07:29:33.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.600+0000 7fbddf7fe640 1 -- 172.21.15.72:0/2501817288 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fbde8057dd0 con 0x7fbdf806cf20 2024-03-06T07:29:33.603 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:33.603 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:33.603 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:33.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbde406b500 msgr2=0x7fbde406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbde406b500 0x7fbde406d9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fbdec036000 tx=0x7fbdec03b040 comp rx=0 tx=0).stop 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 msgr2=0x7fbdf8159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 0x7fbdf8159b40 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7fbde800c3c0 tx=0x7fbde800c880 comp rx=0 tx=0).stop 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 shutdown_connections 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbde406b500 0x7fbde406d9c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf80d00c0 0x7fbdf80ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf80cf850 0x7fbdf815a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 --2- 172.21.15.72:0/2501817288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbdf806cf20 0x7fbdf8159b40 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 >> 172.21.15.72:0/2501817288 conn(0x7fbdf80b35a0 msgr2=0x7fbdf80702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 shutdown_connections 2024-03-06T07:29:33.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:33.602+0000 7fbdfd655640 1 -- 172.21.15.72:0/2501817288 wait complete. 2024-03-06T07:29:33.679 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:33.817 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:33.817 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:34.817 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:34.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.916+0000 7f375c4b8640 1 Processor -- start 2024-03-06T07:29:34.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.916+0000 7f375c4b8640 1 -- start start 2024-03-06T07:29:34.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375c4b8640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f37540d22b0 0x7f37540d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375c4b8640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37540b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375c4b8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37540b3430 con 0x7f375415dfe0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375c4b8640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37540d2c50 con 0x7f37540d22b0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375c4b8640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37540b31e0 con 0x7f37540b41f0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375a254640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f375415dfe0 0x7f37540cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56120/0 (socket says 172.21.15.72:56120) 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375a254640 1 -- 172.21.15.72:0/3199158448 learned_addr learned my addr 172.21.15.72:0/3199158448 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f375aa55640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37540b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f3759a53640 1 --2- 172.21.15.72:0/3199158448 >> v2:172.21.15.146:3300/0 conn(0x7f37540d22b0 0x7f37540d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.917+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733497938 0 0) 0x7f37540b3430 con 0x7f375415dfe0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3740003520 con 0x7f375415dfe0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2051511190 0 0) 0x7f3740003520 con 0x7f375415dfe0 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 msgr2=0x7f37540b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37540b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 >> v2:172.21.15.146:3300/0 conn(0x7f37540d22b0 msgr2=0x7f37540d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 --2- 172.21.15.72:0/3199158448 >> v2:172.21.15.146:3300/0 conn(0x7f37540d22b0 0x7f37540d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540b3710 con 0x7f375415dfe0 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3744001670 con 0x7f375415dfe0 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 >> v1:172.21.15.72:6789/0 conn(0x7f375415dfe0 legacy=0x7f37540cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37400039b0 0x7f3740003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740004800 0x7f3740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540b31e0 con 0x7f3740004800 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374000b9c0 con 0x7f37400039b0 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374000bb40 con 0x7f37540b41f0 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759a53640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740004800 0x7f3740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f375a254640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37400039b0 0x7f3740003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f375aa55640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759a53640 1 -- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 msgr2=0x7f37400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759a53640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759a53640 1 -- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37400039b0 msgr2=0x7f3740003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759a53640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37400039b0 0x7f3740003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f3759a53640 1 -- 172.21.15.72:0/3199158448 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3740003520 con 0x7f3740004800 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.918+0000 7f375aa55640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37540b41f0 0x7f37400051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.919+0000 7f3759a53640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740004800 0x7f3740004c80 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ac0 tx=0x7f3740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2677667e37f6c52d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750039070 con 0x7f3740004800 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375002c710 con 0x7f3740004800 2024-03-06T07:29:34.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f3759252640 1 -- 172.21.15.72:0/3199158448 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750009690 con 0x7f3740004800 2024-03-06T07:29:34.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375c4b8640 1 -- 172.21.15.72:0/3199158448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740004800 msgr2=0x7f3740004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375c4b8640 1 --2- 172.21.15.72:0/3199158448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3740004800 0x7f3740004c80 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f3750009ac0 tx=0x7f3740005830 comp rx=0 tx=0).stop 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375aa55640 1 -- 172.21.15.72:0/3199158448 reap_dead start 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375c4b8640 1 -- 172.21.15.72:0/3199158448 shutdown_connections 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375c4b8640 1 -- 172.21.15.72:0/3199158448 >> 172.21.15.72:0/3199158448 conn(0x7f37540d3cd0 msgr2=0x7f37540d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375c4b8640 1 -- 172.21.15.72:0/3199158448 shutdown_connections 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.920+0000 7f375c4b8640 1 -- 172.21.15.72:0/3199158448 wait complete. 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.921+0000 7f375c4b8640 1 Processor -- start 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.921+0000 7f375c4b8640 1 -- start start 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375c4b8640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3740004800 0x7f37540cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375c4b8640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f375415e3c0 0x7f37540ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375c4b8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 0x7f375406c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375c4b8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540cfa00 con 0x7f375406be40 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375c4b8640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37540b3d60 con 0x7f375415e3c0 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375c4b8640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37540d1c70 con 0x7f3740004800 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 0x7f375406c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 0x7f375406c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47258/0 (socket says 172.21.15.72:47258) 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 -- 172.21.15.72:0/1116525330 learned_addr learned my addr 172.21.15.72:0/1116525330 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f3759a53640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f375415e3c0 0x7f37540ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375a254640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3740004800 0x7f37540cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 -- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3740004800 msgr2=0x7f37540cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3740004800 0x7f37540cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 -- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f375415e3c0 msgr2=0x7f37540ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f375415e3c0 0x7f37540ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.922+0000 7f375aa55640 1 -- 172.21.15.72:0/1116525330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375415eb30 con 0x7f375406be40 2024-03-06T07:29:34.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.923+0000 7f375aa55640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 0x7f375406c2c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f375408c940 tx=0x7f374c010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.923+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c003990 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.923+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374c003b50 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.923+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374c004020 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.923+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f375415ee10 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.923+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f375415f3d0 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.924+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37540d46a0 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.924+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f374c005d10 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.925+0000 7f374affd640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f373806b4d0 0x7f373806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.925+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f374c08a9c0 con 0x7f375406be40 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.925+0000 7f375a254640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f373806b4d0 0x7f373806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:34.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.925+0000 7f375a254640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f373806b4d0 0x7f373806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3744000fb0 tx=0x7f3744009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:34.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:34.933+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37540d46a0 con 0x7f375406be40 2024-03-06T07:29:35.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.144+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f375404f330 con 0x7f375406be40 2024-03-06T07:29:35.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.145+0000 7f374affd640 1 -- 172.21.15.72:0/1116525330 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f374c025020 con 0x7f375406be40 2024-03-06T07:29:35.147 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:35.147 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:35.147 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:35.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f373806b4d0 msgr2=0x7f373806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f373806b4d0 0x7f373806d990 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3744000fb0 tx=0x7f3744009040 comp rx=0 tx=0).stop 2024-03-06T07:29:35.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 msgr2=0x7f375406c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:35.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 0x7f375406c2c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f375408c940 tx=0x7f374c010590 comp rx=0 tx=0).stop 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 shutdown_connections 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f373806b4d0 0x7f373806d990 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f375406be40 0x7f375406c2c0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f375415e3c0 0x7f37540ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 --2- 172.21.15.72:0/1116525330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3740004800 0x7f37540cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 >> 172.21.15.72:0/1116525330 conn(0x7f37540d3cd0 msgr2=0x7f37540b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 shutdown_connections 2024-03-06T07:29:35.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:35.147+0000 7f375c4b8640 1 -- 172.21.15.72:0/1116525330 wait complete. 2024-03-06T07:29:35.361 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:35.361 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:35.680 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:36.362 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:36.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.460+0000 7f6faeb13640 1 Processor -- start 2024-03-06T07:29:36.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.460+0000 7f6faeb13640 1 -- start start 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.460+0000 7f6faeb13640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6fa80cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.460+0000 7f6faeb13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa815e380 con 0x7f6fa806cf20 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.460+0000 7f6faeb13640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa80cdc10 con 0x7f6fa80d00c0 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.460+0000 7f6faeb13640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa806ca90 con 0x7f6fa80cf850 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fad0b0640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6fa806cf20 0x7f6fa806d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56130/0 (socket says 172.21.15.72:56130) 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fad0b0640 1 -- 172.21.15.72:0/1859288881 learned_addr learned my addr 172.21.15.72:0/1859288881 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:36.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa7fff640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6fa80cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938862297 0 0) 0x7f6fa815e380 con 0x7f6fa806cf20 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f90003520 con 0x7f6fa806cf20 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2176910867 0 0) 0x7f6f90003520 con 0x7f6fa806cf20 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 msgr2=0x7f6fa80cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6fa80cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 >> v1:172.21.15.146:6789/0 conn(0x7f6fa80d00c0 legacy=0x7f6fa80d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa806c160 con 0x7f6fa806cf20 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f6f98001670 con 0x7f6fa806cf20 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 >> v1:172.21.15.72:6789/0 conn(0x7f6fa806cf20 legacy=0x7f6fa806d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6f90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f90004800 0x7f6f90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6f900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa806ca90 con 0x7f6f900039e0 2024-03-06T07:29:36.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9000ba70 con 0x7f6f90004800 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9000bc70 con 0x7f6fa80cf850 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fac8af640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6f90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fa7fff640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f90004800 0x7f6f90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.461+0000 7f6fad0b0640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6f900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fac8af640 1 -- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 msgr2=0x7f6f900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fac8af640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6f900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fac8af640 1 -- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f90004800 msgr2=0x7f6f90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fac8af640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f90004800 0x7f6f90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fac8af640 1 -- 172.21.15.72:0/1859288881 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f90003520 con 0x7f6f900039e0 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fad0b0640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80cf850 0x7f6f900051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fa7fff640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f90004800 0x7f6f90004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fac8af640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6f90003e00 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0001e20 tx=0x7f6f90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5719cb8960c7e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0001a70 con 0x7f6f900039e0 2024-03-06T07:29:36.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0003b70 con 0x7f6f900039e0 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.462+0000 7f6fa77fe640 1 -- 172.21.15.72:0/1859288881 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00050d0 con 0x7f6f900039e0 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6faeb13640 1 -- 172.21.15.72:0/1859288881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 msgr2=0x7f6f90003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6faeb13640 1 --2- 172.21.15.72:0/1859288881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6f90003e00 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0001e20 tx=0x7f6f90005830 comp rx=0 tx=0).stop 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6fad0b0640 1 -- 172.21.15.72:0/1859288881 reap_dead start 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6faeb13640 1 -- 172.21.15.72:0/1859288881 shutdown_connections 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6faeb13640 1 -- 172.21.15.72:0/1859288881 >> 172.21.15.72:0/1859288881 conn(0x7f6fa80b35a0 msgr2=0x7f6fa80b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6faeb13640 1 -- 172.21.15.72:0/1859288881 shutdown_connections 2024-03-06T07:29:36.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.463+0000 7f6faeb13640 1 -- 172.21.15.72:0/1859288881 wait complete. 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.464+0000 7f6faeb13640 1 Processor -- start 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.464+0000 7f6faeb13640 1 -- start start 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6faeb13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6fa815c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6faeb13640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fa815b9b0 0x7f6fa815c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6faeb13640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80c6480 0x7f6fa80c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6faeb13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa80b1030 con 0x7f6f900039e0 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6faeb13640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa806c720 con 0x7f6fa815b9b0 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6faeb13640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa8159710 con 0x7f6fa80c6480 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6fa815c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6fa815c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47262/0 (socket says 172.21.15.72:47262) 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 -- 172.21.15.72:0/178738801 learned_addr learned my addr 172.21.15.72:0/178738801 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:36.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fa7fff640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fa815b9b0 0x7f6fa815c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fad0b0640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80c6480 0x7f6fa80c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 -- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80c6480 msgr2=0x7f6fa80c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80c6480 0x7f6fa80c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 -- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fa815b9b0 msgr2=0x7f6fa815c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fa815b9b0 0x7f6fa815c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.465+0000 7f6fac8af640 1 -- 172.21.15.72:0/178738801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa80c78a0 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.466+0000 7f6fac8af640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6fa815c3b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0001f40 tx=0x7f6fa00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.466+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0003970 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.466+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0006b50 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.466+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa80c7b40 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.466+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0007030 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.466+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa80c8100 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.467+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa804f330 con 0x7f6f900039e0 2024-03-06T07:29:36.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.468+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f6fa00071d0 con 0x7f6f900039e0 2024-03-06T07:29:36.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.468+0000 7f6fa57fa640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f9406b500 0x7f6f9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:36.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.468+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6fa00abce0 con 0x7f6f900039e0 2024-03-06T07:29:36.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.468+0000 7f6fa7fff640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f9406b500 0x7f6f9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:36.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.469+0000 7f6fa7fff640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f9406b500 0x7f6f9406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6fa815cf00 tx=0x7f6f9c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:36.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.473+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6fa0033020 con 0x7f6f900039e0 2024-03-06T07:29:36.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.688+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fa80c5850 con 0x7f6f900039e0 2024-03-06T07:29:36.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.689+0000 7f6fa57fa640 1 -- 172.21.15.72:0/178738801 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f6fa007abb0 con 0x7f6f900039e0 2024-03-06T07:29:36.691 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:36.691 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:36.691 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.690+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f9406b500 msgr2=0x7f6f9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.690+0000 7f6faeb13640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f9406b500 0x7f6f9406d9c0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f6fa815cf00 tx=0x7f6f9c008040 comp rx=0 tx=0).stop 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.690+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 msgr2=0x7f6fa815c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.690+0000 7f6faeb13640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6fa815c3b0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0001f40 tx=0x7f6fa00014a0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 shutdown_connections 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f6f9406b500 0x7f6f9406d9c0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa80c6480 0x7f6fa80c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fa815b9b0 0x7f6fa815c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 --2- 172.21.15.72:0/178738801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f900039e0 0x7f6fa815c3b0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:36.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 >> 172.21.15.72:0/178738801 conn(0x7f6fa80b35a0 msgr2=0x7f6fa80d04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:36.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 shutdown_connections 2024-03-06T07:29:36.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:36.691+0000 7f6faeb13640 1 -- 172.21.15.72:0/178738801 wait complete. 2024-03-06T07:29:36.909 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:36.909 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:37.683 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:37.910 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.009+0000 7f61fc993640 1 Processor -- start 2024-03-06T07:29:38.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.009+0000 7f61fc993640 1 -- start start 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61fc993640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f80b3060 0x7f61f815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61fc993640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f61f815bf30 0x7f61f815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61fc993640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f815f2d0 con 0x7f61f813dc10 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61fc993640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f80cf3f0 con 0x7f61f815bf30 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61fc993640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f815b210 con 0x7f61f80b3060 2024-03-06T07:29:38.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f61f815bf30 0x7f61f815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f61f813dc10 0x7f61f80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56136/0 (socket says 172.21.15.72:56136) 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f61f815bf30 0x7f61f815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47266/0 (socket says 172.21.15.72:47266) 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f659c640 1 -- 172.21.15.72:0/3178052804 learned_addr learned my addr 172.21.15.72:0/3178052804 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f6d9d640 1 -- 172.21.15.72:0/3178052804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f80b3060 msgr2=0x7f61f815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f6d9d640 1 --2- 172.21.15.72:0/3178052804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f80b3060 0x7f61f815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f6d9d640 1 -- 172.21.15.72:0/3178052804 >> v1:172.21.15.72:6789/0 conn(0x7f61f813dc10 legacy=0x7f61f80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f6d9d640 1 -- 172.21.15.72:0/3178052804 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f815f580 con 0x7f61f815bf30 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.010+0000 7f61f559a640 1 -- 172.21.15.72:0/3178052804 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085033479 0 0) 0x7f61f815f2d0 con 0x7f61f813dc10 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.011+0000 7f61f6d9d640 1 --2- 172.21.15.72:0/3178052804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f815bf30 0x7f61f815ed90 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f61e800bfa0 tx=0x7f61e802e010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dea445286cd1d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.011+0000 7f61f559a640 1 -- 172.21.15.72:0/3178052804 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e802e9b0 con 0x7f61f815bf30 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.011+0000 7f61f559a640 1 -- 172.21.15.72:0/3178052804 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e802eb50 con 0x7f61f815bf30 2024-03-06T07:29:38.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.011+0000 7f61f559a640 1 -- 172.21.15.72:0/3178052804 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e8009540 con 0x7f61f815bf30 2024-03-06T07:29:38.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 -- 172.21.15.72:0/3178052804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f815bf30 msgr2=0x7f61f815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 --2- 172.21.15.72:0/3178052804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f815bf30 0x7f61f815ed90 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f61e800bfa0 tx=0x7f61e802e010 comp rx=0 tx=0).stop 2024-03-06T07:29:38.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 -- 172.21.15.72:0/3178052804 shutdown_connections 2024-03-06T07:29:38.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 --2- 172.21.15.72:0/3178052804 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f815bf30 0x7f61f815ed90 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 --2- 172.21.15.72:0/3178052804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f80b3060 0x7f61f815e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 -- 172.21.15.72:0/3178052804 >> 172.21.15.72:0/3178052804 conn(0x7f61f80cf8a0 msgr2=0x7f61f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 -- 172.21.15.72:0/3178052804 shutdown_connections 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.012+0000 7f61fc993640 1 -- 172.21.15.72:0/3178052804 wait complete. 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.013+0000 7f61fc993640 1 Processor -- start 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.013+0000 7f61fc993640 1 -- start start 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61fc993640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 0x7f61f8152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61fc993640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61f813dc10 0x7f61f813bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61fc993640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f815bf30 0x7f61f813a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61fc993640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f813dff0 con 0x7f61f80b3060 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61fc993640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f815b730 con 0x7f61f813dc10 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61fc993640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f80cd7a0 con 0x7f61f815bf30 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 0x7f61f8152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 0x7f61f8152380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47278/0 (socket says 172.21.15.72:47278) 2024-03-06T07:29:38.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 -- 172.21.15.72:0/3548572457 learned_addr learned my addr 172.21.15.72:0/3548572457 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f6d9d640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f815bf30 0x7f61f813a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f5d9b640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61f813dc10 0x7f61f813bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 -- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f815bf30 msgr2=0x7f61f813a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f815bf30 0x7f61f813a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 -- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61f813dc10 msgr2=0x7f61f813bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61f813dc10 0x7f61f813bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f659c640 1 -- 172.21.15.72:0/3548572457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f8152a80 con 0x7f61f80b3060 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f6d9d640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f815bf30 0x7f61f813a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.014+0000 7f61f5d9b640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61f813dc10 0x7f61f813bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.015+0000 7f61f659c640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 0x7f61f8152380 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c500 tx=0x7f61ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.015+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec018070 con 0x7f61f80b3060 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.015+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec007580 con 0x7f61f80b3060 2024-03-06T07:29:38.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.015+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec004a20 con 0x7f61f80b3060 2024-03-06T07:29:38.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.015+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f8152d60 con 0x7f61f80b3060 2024-03-06T07:29:38.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.015+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f8153320 con 0x7f61f80b3060 2024-03-06T07:29:38.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.016+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f804f330 con 0x7f61f80b3060 2024-03-06T07:29:38.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.016+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f61ec01b070 con 0x7f61f80b3060 2024-03-06T07:29:38.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.017+0000 7f61e6ffd640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f61d806b4d0 0x7f61d806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:38.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.017+0000 7f61f5d9b640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f61d806b4d0 0x7f61d806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:38.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.017+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f61ec0894c0 con 0x7f61f80b3060 2024-03-06T07:29:38.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.017+0000 7f61f5d9b640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f61d806b4d0 0x7f61d806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f61f0006fd0 tx=0x7f61f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:38.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.021+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f61ec058390 con 0x7f61f80b3060 2024-03-06T07:29:38.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.362+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61f806eb00 con 0x7f61f80b3060 2024-03-06T07:29:38.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.363+0000 7f61e6ffd640 1 -- 172.21.15.72:0/3548572457 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7f61ec057c90 con 0x7f61f80b3060 2024-03-06T07:29:38.365 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:38.365 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:38.366 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f61d806b4d0 msgr2=0x7f61d806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f61d806b4d0 0x7f61d806d990 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f61f0006fd0 tx=0x7f61f0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 msgr2=0x7f61f8152380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 0x7f61f8152380 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c500 tx=0x7f61ec00c9c0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 shutdown_connections 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f61d806b4d0 0x7f61d806d990 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61f815bf30 0x7f61f813a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61f813dc10 0x7f61f813bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 --2- 172.21.15.72:0/3548572457 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f61f80b3060 0x7f61f8152380 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:38.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 >> 172.21.15.72:0/3548572457 conn(0x7f61f80cf8a0 msgr2=0x7f61f806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:38.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 shutdown_connections 2024-03-06T07:29:38.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:38.365+0000 7f61fc993640 1 -- 172.21.15.72:0/3548572457 wait complete. 2024-03-06T07:29:38.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:38.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:39.580 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:39.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.680+0000 7fba6cb5e640 1 Processor -- start 2024-03-06T07:29:39.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.680+0000 7fba6cb5e640 1 -- start start 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba6cb5e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba6815dfe0 0x7fba680cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba6cb5e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 0x7fba680b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba6cb5e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680b3430 con 0x7fba680cf830 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba6cb5e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba680b31e0 con 0x7fba6815dfe0 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba6cb5e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba680b3d60 con 0x7fba680d29c0 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba5ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fba680cf830 0x7fba680d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56144/0 (socket says 172.21.15.72:56144) 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba5ffff640 1 -- 172.21.15.72:0/2800281909 learned_addr learned my addr 172.21.15.72:0/2800281909 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba66d9d640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 0x7fba680b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba6659c640 1 --2- 172.21.15.72:0/2800281909 >> v2:172.21.15.146:3300/0 conn(0x7fba6815dfe0 0x7fba680cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472606557 0 0) 0x7fba680b3430 con 0x7fba680cf830 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba54003520 con 0x7fba680cf830 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339254138 0 0) 0x7fba54003520 con 0x7fba680cf830 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 msgr2=0x7fba680b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 0x7fba680b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 >> v2:172.21.15.146:3300/0 conn(0x7fba6815dfe0 msgr2=0x7fba680cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 --2- 172.21.15.72:0/2800281909 >> v2:172.21.15.146:3300/0 conn(0x7fba6815dfe0 0x7fba680cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680b3710 con 0x7fba680cf830 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.681+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fba50001670 con 0x7fba680cf830 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 >> v1:172.21.15.72:6789/0 conn(0x7fba680cf830 legacy=0x7fba680d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 0x7fba54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba54004d40 0x7fba540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680b31e0 con 0x7fba540039b0 2024-03-06T07:29:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680b3d60 con 0x7fba54004d40 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5400b9c0 con 0x7fba680d29c0 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba6659c640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba5ffff640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 0x7fba54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba66d9d640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba54004d40 0x7fba540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba6659c640 1 -- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 msgr2=0x7fba54004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba6659c640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba680d29c0 0x7fba54004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba6659c640 1 -- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba54004d40 msgr2=0x7fba540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba6659c640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba54004d40 0x7fba540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.682+0000 7fba6659c640 1 -- 172.21.15.72:0/2800281909 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54003520 con 0x7fba540039b0 2024-03-06T07:29:39.687 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6659c640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba54003dd0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fba58009af0 tx=0x7fba54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80facc521f0f558c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58039070 con 0x7fba540039b0 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5802c710 con 0x7fba540039b0 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba65d9b640 1 -- 172.21.15.72:0/2800281909 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58009690 con 0x7fba540039b0 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6cb5e640 1 -- 172.21.15.72:0/2800281909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 msgr2=0x7fba54003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/2800281909 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba54003dd0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fba58009af0 tx=0x7fba54005830 comp rx=0 tx=0).stop 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba66d9d640 1 -- 172.21.15.72:0/2800281909 reap_dead start 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6cb5e640 1 -- 172.21.15.72:0/2800281909 shutdown_connections 2024-03-06T07:29:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6cb5e640 1 -- 172.21.15.72:0/2800281909 >> 172.21.15.72:0/2800281909 conn(0x7fba680d3cd0 msgr2=0x7fba680d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6cb5e640 1 -- 172.21.15.72:0/2800281909 shutdown_connections 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.683+0000 7fba6cb5e640 1 -- 172.21.15.72:0/2800281909 wait complete. 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.685+0000 7fba6cb5e640 1 Processor -- start 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.685+0000 7fba6cb5e640 1 -- start start 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6cb5e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba6815bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6cb5e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6815f0f0 0x7fba6815f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6cb5e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ca160 0x7fba680ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6cb5e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680d1df0 con 0x7fba540039b0 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6cb5e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680d1c70 con 0x7fba680ca160 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6cb5e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6815e3c0 con 0x7fba6815f0f0 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba6815bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba6815bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47292/0 (socket says 172.21.15.72:47292) 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 -- 172.21.15.72:0/1761047736 learned_addr learned my addr 172.21.15.72:0/1761047736 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:39.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 -- 172.21.15.72:0/1761047736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6815f0f0 msgr2=0x7fba6815f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6815f0f0 0x7fba6815f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 -- 172.21.15.72:0/1761047736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ca160 msgr2=0x7fba680ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ca160 0x7fba680ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.686+0000 7fba6659c640 1 -- 172.21.15.72:0/1761047736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba681f4b70 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.687+0000 7fba6659c640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba6815bda0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fba58037040 tx=0x7fba58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.687+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58039050 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.687+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58045070 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.687+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba681f4e50 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.687+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba580044e0 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.687+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6815fe00 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.688+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba6804f330 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.688+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fba58003a50 con 0x7fba540039b0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.689+0000 7fba5effd640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b500 0x7fba4806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.689+0000 7fba5ffff640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b500 0x7fba4806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:39.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.689+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fba580abc30 con 0x7fba540039b0 2024-03-06T07:29:39.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.689+0000 7fba5ffff640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b500 0x7fba4806d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fba50000f60 tx=0x7fba50000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:39.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.693+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fba5804e020 con 0x7fba540039b0 2024-03-06T07:29:39.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.908+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba680b2650 con 0x7fba540039b0 2024-03-06T07:29:39.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.908+0000 7fba5effd640 1 -- 172.21.15.72:0/1761047736 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+3415 (secure 0 0 0) 0x7fba5807ac20 con 0x7fba540039b0 2024-03-06T07:29:39.910 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:39.910 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":87,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:08.151093+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:39.911 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 87 2024-03-06T07:29:39.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b500 msgr2=0x7fba4806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b500 0x7fba4806d9c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fba50000f60 tx=0x7fba50000ed0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 msgr2=0x7fba6815bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba6815bda0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fba58037040 tx=0x7fba58004730 comp rx=0 tx=0).stop 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 shutdown_connections 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fba4806b500 0x7fba4806d9c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba680ca160 0x7fba680ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6815f0f0 0x7fba6815f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 --2- 172.21.15.72:0/1761047736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba540039b0 0x7fba6815bda0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 >> 172.21.15.72:0/1761047736 conn(0x7fba680d3cd0 msgr2=0x7fba6806cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 shutdown_connections 2024-03-06T07:29:39.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:39.910+0000 7fba6cb5e640 1 -- 172.21.15.72:0/1761047736 wait complete. 2024-03-06T07:29:40.124 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:08.151093+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:40.124 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:41.126 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:41.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.225+0000 7f5b22d77640 1 Processor -- start 2024-03-06T07:29:41.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b22d77640 1 -- start start 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b22d77640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c0d04a0 0x7f5b1c0ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b22d77640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c0d0910 con 0x7f5b1c0cfff0 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b22d77640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c161980 con 0x7f5b1c15bb20 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b22d77640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0d2500 con 0x7f5b1c0d04a0 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b1bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5b1c0cfff0 0x7f5b1c0cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:56158/0 (socket says 172.21.15.72:56158) 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.226+0000 7f5b1bfff640 1 -- 172.21.15.72:0/19994799 learned_addr learned my addr 172.21.15.72:0/19994799 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b21314640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c0d04a0 0x7f5b1c0ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715093099 0 0) 0x7f5b1c0d0910 con 0x7f5b1c0cfff0 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c003520 con 0x7f5b1c0cfff0 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 184404000 0 0) 0x7f5b1c161980 con 0x7f5b1c15bb20 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c0d0910 con 0x7f5b1c15bb20 2024-03-06T07:29:41.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4228868351 0 0) 0x7f5b0c003520 con 0x7f5b1c0cfff0 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c0d04a0 msgr2=0x7f5b1c0ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c0d04a0 0x7f5b1c0ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 >> v1:172.21.15.146:6789/0 conn(0x7f5b1c15bb20 legacy=0x7f5b1c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c161c60 con 0x7f5b1c0cfff0 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f5b08001670 con 0x7f5b1c0cfff0 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 >> v1:172.21.15.72:6789/0 conn(0x7f5b1c0cfff0 legacy=0x7f5b1c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b0c003b70 0x7f5b0c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b0c004800 0x7f5b0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0d2500 con 0x7f5b0c0051c0 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c00bfe0 con 0x7f5b0c004800 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c00c1e0 con 0x7f5b0c003b70 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b21314640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c0d04a0 0x7f5b1c0ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:41.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b21314640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b20b13640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b0c003b70 0x7f5b0c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.227+0000 7f5b1bfff640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b0c004800 0x7f5b0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b21314640 1 -- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b0c003b70 msgr2=0x7f5b0c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b21314640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b0c003b70 0x7f5b0c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b21314640 1 -- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b0c004800 msgr2=0x7f5b0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b21314640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b0c004800 0x7f5b0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b21314640 1 -- 172.21.15.72:0/19994799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c003520 con 0x7f5b0c0051c0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b21314640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b0c00b9c0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0d0880 tx=0x7f5b0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5040792c34400c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.228+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14009d40 con 0x7f5b0c0051c0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1402ccf0 con 0x7f5b0c0051c0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b1b7fe640 1 -- 172.21.15.72:0/19994799 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14003bb0 con 0x7f5b0c0051c0 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b22d77640 1 -- 172.21.15.72:0/19994799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 msgr2=0x7f5b0c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b22d77640 1 --2- 172.21.15.72:0/19994799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b0c00b9c0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c0d0880 tx=0x7f5b0c005830 comp rx=0 tx=0).stop 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b21314640 1 -- 172.21.15.72:0/19994799 reap_dead start 2024-03-06T07:29:41.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b22d77640 1 -- 172.21.15.72:0/19994799 shutdown_connections 2024-03-06T07:29:41.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b22d77640 1 -- 172.21.15.72:0/19994799 >> 172.21.15.72:0/19994799 conn(0x7f5b1c09a9f0 msgr2=0x7f5b1c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b22d77640 1 -- 172.21.15.72:0/19994799 shutdown_connections 2024-03-06T07:29:41.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.229+0000 7f5b22d77640 1 -- 172.21.15.72:0/19994799 wait complete. 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.230+0000 7f5b22d77640 1 Processor -- start 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.230+0000 7f5b22d77640 1 -- start start 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b22d77640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b1c0b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b22d77640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1c149d20 0x7f5b1c0c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b22d77640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c14a180 0x7f5b1c0c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b22d77640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c162500 con 0x7f5b0c0051c0 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b22d77640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c162220 con 0x7f5b1c149d20 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b22d77640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c0d1980 con 0x7f5b1c14a180 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b20b13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b1c0b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b20b13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b1c0b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47306/0 (socket says 172.21.15.72:47306) 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b20b13640 1 -- 172.21.15.72:0/2066191564 learned_addr learned my addr 172.21.15.72:0/2066191564 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b20b13640 1 -- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c14a180 msgr2=0x7f5b1c0c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b21314640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c14a180 0x7f5b1c0c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.231+0000 7f5b1bfff640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1c149d20 0x7f5b1c0c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b20b13640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c14a180 0x7f5b1c0c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b20b13640 1 -- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1c149d20 msgr2=0x7f5b1c0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b20b13640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1c149d20 0x7f5b1c0c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b20b13640 1 -- 172.21.15.72:0/2066191564 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c1e9590 con 0x7f5b0c0051c0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b21314640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c14a180 0x7f5b1c0c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b1bfff640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1c149d20 0x7f5b1c0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b20b13640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b1c0b4700 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f5b1000f7f0 tx=0x7f5b1000fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1000fea0 con 0x7f5b0c0051c0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b10018070 con 0x7f5b0c0051c0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c1e9870 con 0x7f5b0c0051c0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1000d6d0 con 0x7f5b0c0051c0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.232+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c1e9e30 con 0x7f5b0c0051c0 2024-03-06T07:29:41.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.233+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c1ea170 con 0x7f5b0c0051c0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.233+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b1000d870 con 0x7f5b0c0051c0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.234+0000 7f5b197fa640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b0006b500 0x7f5b0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.234+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5b1008a1c0 con 0x7f5b0c0051c0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.234+0000 7f5b1bfff640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b0006b500 0x7f5b0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.234+0000 7f5b1bfff640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b0006b500 0x7f5b0006d9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c14abd0 tx=0x7f5b08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:41.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.238+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b10058f70 con 0x7f5b0c0051c0 2024-03-06T07:29:41.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.448+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b1c147a20 con 0x7f5b0c0051c0 2024-03-06T07:29:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.449+0000 7f5b197fa640 1 -- 172.21.15.72:0/2066191564 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5b10058870 con 0x7f5b0c0051c0 2024-03-06T07:29:41.451 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:41.451 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:41.451 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b0006b500 msgr2=0x7f5b0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b0006b500 0x7f5b0006d9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c14abd0 tx=0x7f5b08009040 comp rx=0 tx=0).stop 2024-03-06T07:29:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 msgr2=0x7f5b1c0b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b1c0b4700 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f5b1000f7f0 tx=0x7f5b1000fcb0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 shutdown_connections 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b0006b500 0x7f5b0006d9c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b1c14a180 0x7f5b1c0c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1c149d20 0x7f5b1c0c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 --2- 172.21.15.72:0/2066191564 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b0c0051c0 0x7f5b1c0b4700 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 >> 172.21.15.72:0/2066191564 conn(0x7f5b1c09a9f0 msgr2=0x7f5b1c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 shutdown_connections 2024-03-06T07:29:41.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:41.451+0000 7f5b22d77640 1 -- 172.21.15.72:0/2066191564 wait complete. 2024-03-06T07:29:41.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:41.665 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:41.686 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:42.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:42.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.764+0000 7f5bb2066640 1 Processor -- start 2024-03-06T07:29:42.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.764+0000 7f5bb2066640 1 -- start start 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5bb2066640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 0x7f5bac0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5bb2066640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5bac15ef60 0x7f5bac0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5bb2066640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bac15f580 con 0x7f5bac15bf30 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5bb2066640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bac0cf3f0 con 0x7f5bac15ef60 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5bb2066640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac15b210 con 0x7f5bac13dc10 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5ba2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5bac15bf30 0x7f5bac15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47250/0 (socket says 172.21.15.72:47250) 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5ba2ffd640 1 -- 172.21.15.72:0/274248224 learned_addr learned my addr 172.21.15.72:0/274248224 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140945009 0 0) 0x7f5bac15f580 con 0x7f5bac15bf30 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b98003520 con 0x7f5bac15bf30 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5babfff640 1 --2- 172.21.15.72:0/274248224 >> v2:172.21.15.146:3300/0 conn(0x7f5bac15ef60 0x7f5bac0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.765+0000 7f5bab7fe640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 0x7f5bac0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3486965422 0 0) 0x7f5b98003520 con 0x7f5bac15bf30 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 msgr2=0x7f5bac0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 0x7f5bac0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 >> v2:172.21.15.146:3300/0 conn(0x7f5bac15ef60 msgr2=0x7f5bac0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 --2- 172.21.15.72:0/274248224 >> v2:172.21.15.146:3300/0 conn(0x7f5bac15ef60 0x7f5bac0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac15b450 con 0x7f5bac15bf30 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f5b94001670 con 0x7f5bac15bf30 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 >> v1:172.21.15.72:6789/0 conn(0x7f5bac15bf30 legacy=0x7f5bac15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 0x7f5b980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5b98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b98004d40 0x7f5b980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac0cf3f0 con 0x7f5b98003f30 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac15b210 con 0x7f5b98004d40 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9800ba10 con 0x7f5bac13dc10 2024-03-06T07:29:42.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5ba2ffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5b98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5bab7fe640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 0x7f5b980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5babfff640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b98004d40 0x7f5b980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5ba2ffd640 1 -- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 msgr2=0x7f5b980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5ba2ffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac13dc10 0x7f5b980039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5ba2ffd640 1 -- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b98004d40 msgr2=0x7f5b980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5ba2ffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b98004d40 0x7f5b980051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.766+0000 7f5ba2ffd640 1 -- 172.21.15.72:0/274248224 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b98003520 con 0x7f5b98003f30 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.767+0000 7f5ba2ffd640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5b98004800 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400c210 tx=0x7f5b98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c524ef2d631484b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.767+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94007d50 con 0x7f5b98003f30 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.767+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b94030070 con 0x7f5b98003f30 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.767+0000 7f5baaffd640 1 -- 172.21.15.72:0/274248224 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9402cd20 con 0x7f5b98003f30 2024-03-06T07:29:42.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5bb2066640 1 -- 172.21.15.72:0/274248224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 msgr2=0x7f5b98004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5bb2066640 1 --2- 172.21.15.72:0/274248224 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5b98004800 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f5b9400c210 tx=0x7f5b98005830 comp rx=0 tx=0).stop 2024-03-06T07:29:42.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5babfff640 1 -- 172.21.15.72:0/274248224 reap_dead start 2024-03-06T07:29:42.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5bb2066640 1 -- 172.21.15.72:0/274248224 shutdown_connections 2024-03-06T07:29:42.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5bb2066640 1 -- 172.21.15.72:0/274248224 >> 172.21.15.72:0/274248224 conn(0x7f5bac0cf8a0 msgr2=0x7f5bac0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5bb2066640 1 -- 172.21.15.72:0/274248224 shutdown_connections 2024-03-06T07:29:42.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.768+0000 7f5bb2066640 1 -- 172.21.15.72:0/274248224 wait complete. 2024-03-06T07:29:42.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.769+0000 7f5bb2066640 1 Processor -- start 2024-03-06T07:29:42.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.769+0000 7f5bb2066640 1 -- start start 2024-03-06T07:29:42.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5bac132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bac151f90 0x7f5bac159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac159ea0 0x7f5bac1545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac0b2950 con 0x7f5b98003f30 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac13dff0 con 0x7f5bac151f90 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac0b2ad0 con 0x7f5bac159ea0 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5bac132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5bac132130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46270/0 (socket says 172.21.15.72:46270) 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 -- 172.21.15.72:0/2840811234 learned_addr learned my addr 172.21.15.72:0/2840811234 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 -- 172.21.15.72:0/2840811234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac159ea0 msgr2=0x7f5bac1545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac159ea0 0x7f5bac1545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 -- 172.21.15.72:0/2840811234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bac151f90 msgr2=0x7f5bac159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bac151f90 0x7f5bac159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 -- 172.21.15.72:0/2840811234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac14f7a0 con 0x7f5b98003f30 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bab7fe640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5bac132130 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c011940 tx=0x7f5b9c011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00fa10 con 0x7f5b98003f30 2024-03-06T07:29:42.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9c00fbb0 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bac1553d0 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c014440 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.770+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bac155990 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.771+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bac15a2c0 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.772+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b9c012040 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.772+0000 7f5ba8ff9640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b8c06b4a0 0x7f5b8c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.773+0000 7f5ba2ffd640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b8c06b4a0 0x7f5b8c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.773+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5b9c08a720 con 0x7f5b98003f30 2024-03-06T07:29:42.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.773+0000 7f5ba2ffd640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b8c06b4a0 0x7f5b8c06d960 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5b94034000 tx=0x7f5b94039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:42.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.777+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b9c019030 con 0x7f5b98003f30 2024-03-06T07:29:42.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.987+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bac04f330 con 0x7f5b98003f30 2024-03-06T07:29:42.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.988+0000 7f5ba8ff9640 1 -- 172.21.15.72:0/2840811234 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5b9c059560 con 0x7f5b98003f30 2024-03-06T07:29:42.990 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:42.990 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:42.990 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:42.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b8c06b4a0 msgr2=0x7f5b8c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b8c06b4a0 0x7f5b8c06d960 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f5b94034000 tx=0x7f5b94039040 comp rx=0 tx=0).stop 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 msgr2=0x7f5bac132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5bac132130 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c011940 tx=0x7f5b9c011e00 comp rx=0 tx=0).stop 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 shutdown_connections 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b8c06b4a0 0x7f5b8c06d960 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bac159ea0 0x7f5bac1545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bac151f90 0x7f5bac159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 --2- 172.21.15.72:0/2840811234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b98003f30 0x7f5bac132130 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 >> 172.21.15.72:0/2840811234 conn(0x7f5bac0cf8a0 msgr2=0x7f5bac0707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 shutdown_connections 2024-03-06T07:29:42.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:42.990+0000 7f5bb2066640 1 -- 172.21.15.72:0/2840811234 wait complete. 2024-03-06T07:29:43.208 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:43.208 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:43.689 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:44.209 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:44.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.308+0000 7f961fe0e640 1 Processor -- start 2024-03-06T07:29:44.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.309+0000 7f961fe0e640 1 -- start start 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.309+0000 7f961fe0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f961815dfe0 0x7f96180cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.309+0000 7f961fe0e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96180cf830 0x7f96180d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.309+0000 7f961fe0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96180b3430 con 0x7f96180d27f0 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.309+0000 7f961fe0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96180b31e0 con 0x7f961815dfe0 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.309+0000 7f961fe0e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96180b3d60 con 0x7f96180cf830 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f961815dfe0 0x7f96180cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f961815dfe0 0x7f96180cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46274/0 (socket says 172.21.15.72:46274) 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961e3ab640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f96180d27f0 0x7f96180d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47256/0 (socket says 172.21.15.72:47256) 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 -- 172.21.15.72:0/3232192485 learned_addr learned my addr 172.21.15.72:0/3232192485 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:44.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961d3a9640 1 --2- 172.21.15.72:0/3232192485 >> v2:172.21.15.146:3300/0 conn(0x7f96180cf830 0x7f96180d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 -- 172.21.15.72:0/3232192485 >> v2:172.21.15.146:3300/0 conn(0x7f96180cf830 msgr2=0x7f96180d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 --2- 172.21.15.72:0/3232192485 >> v2:172.21.15.146:3300/0 conn(0x7f96180cf830 0x7f96180d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 -- 172.21.15.72:0/3232192485 >> v1:172.21.15.72:6789/0 conn(0x7f96180d27f0 legacy=0x7f96180d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961cba8640 1 -- 172.21.15.72:0/3232192485 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650120649 0 0) 0x7f96180b3430 con 0x7f96180d27f0 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961dbaa640 1 -- 172.21.15.72:0/3232192485 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96180b3710 con 0x7f961815dfe0 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.310+0000 7f961d3a9640 1 --2- 172.21.15.72:0/3232192485 >> v2:172.21.15.146:3300/0 conn(0x7f96180cf830 0x7f96180d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961dbaa640 1 --2- 172.21.15.72:0/3232192485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f961815dfe0 0x7f96180cf2f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f9614009870 tx=0x7f961402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74033b7b30998e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961cba8640 1 -- 172.21.15.72:0/3232192485 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614034070 con 0x7f961815dfe0 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961cba8640 1 -- 172.21.15.72:0/3232192485 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9614030860 con 0x7f961815dfe0 2024-03-06T07:29:44.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961cba8640 1 -- 172.21.15.72:0/3232192485 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614030c30 con 0x7f961815dfe0 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 -- 172.21.15.72:0/3232192485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f961815dfe0 msgr2=0x7f96180cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 --2- 172.21.15.72:0/3232192485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f961815dfe0 0x7f96180cf2f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f9614009870 tx=0x7f961402b9a0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 -- 172.21.15.72:0/3232192485 shutdown_connections 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 --2- 172.21.15.72:0/3232192485 >> v2:172.21.15.146:3300/0 conn(0x7f96180cf830 0x7f96180d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 --2- 172.21.15.72:0/3232192485 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f961815dfe0 0x7f96180cf2f0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 -- 172.21.15.72:0/3232192485 >> 172.21.15.72:0/3232192485 conn(0x7f96180d3cd0 msgr2=0x7f96180d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 -- 172.21.15.72:0/3232192485 shutdown_connections 2024-03-06T07:29:44.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.311+0000 7f961fe0e640 1 -- 172.21.15.72:0/3232192485 wait complete. 2024-03-06T07:29:44.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 Processor -- start 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 -- start start 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96180cf830 0x7f96180b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 0x7f96180b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815dfe0 0x7f96180b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96180d1df0 con 0x7f96180d27f0 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96180d1c70 con 0x7f96180cf830 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961fe0e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961815e3c0 con 0x7f961815dfe0 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961d3a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 0x7f96180b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961d3a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 0x7f96180b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46288/0 (socket says 172.21.15.72:46288) 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961d3a9640 1 -- 172.21.15.72:0/2104421297 learned_addr learned my addr 172.21.15.72:0/2104421297 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.313+0000 7f961dbaa640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96180cf830 0x7f96180b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961e3ab640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815dfe0 0x7f96180b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961d3a9640 1 -- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815dfe0 msgr2=0x7f96180b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961d3a9640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815dfe0 0x7f96180b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961d3a9640 1 -- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96180cf830 msgr2=0x7f96180b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961d3a9640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96180cf830 0x7f96180b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961d3a9640 1 -- 172.21.15.72:0/2104421297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96180b8430 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961d3a9640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 0x7f96180b4610 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f960800c7f0 tx=0x7f960800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608015070 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9608007820 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96180b8710 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608007bf0 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.314+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96180b8cd0 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.315+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961804f330 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.315+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9608018070 con 0x7f96180d27f0 2024-03-06T07:29:44.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.316+0000 7f960e7fc640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f95fc06b4d0 0x7f95fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.316+0000 7f961dbaa640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f95fc06b4d0 0x7f95fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.316+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f960808a2e0 con 0x7f96180d27f0 2024-03-06T07:29:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.316+0000 7f961dbaa640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f95fc06b4d0 0x7f95fc06d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9614036000 tx=0x7f9614008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.320+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9608058ad0 con 0x7f96180d27f0 2024-03-06T07:29:44.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.535+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f961806a1b0 con 0x7f96180d27f0 2024-03-06T07:29:44.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.536+0000 7f960e7fc640 1 -- 172.21.15.72:0/2104421297 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f960805cb70 con 0x7f96180d27f0 2024-03-06T07:29:44.537 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:44.537 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:44.537 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.538+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f95fc06b4d0 msgr2=0x7f95fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.538+0000 7f961fe0e640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f95fc06b4d0 0x7f95fc06d990 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9614036000 tx=0x7f9614008030 comp rx=0 tx=0).stop 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.538+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 msgr2=0x7f96180b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.538+0000 7f961fe0e640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 0x7f96180b4610 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f960800c7f0 tx=0x7f960800ccb0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 shutdown_connections 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f95fc06b4d0 0x7f95fc06d990 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815dfe0 0x7f96180b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96180d27f0 0x7f96180b4610 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 --2- 172.21.15.72:0/2104421297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96180cf830 0x7f96180b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:44.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 >> 172.21.15.72:0/2104421297 conn(0x7f96180d3cd0 msgr2=0x7f96180d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:44.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 shutdown_connections 2024-03-06T07:29:44.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:44.539+0000 7f961fe0e640 1 -- 172.21.15.72:0/2104421297 wait complete. 2024-03-06T07:29:44.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:44.755 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:45.690 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:45.756 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:45.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.857+0000 7f51f9c4e640 1 Processor -- start 2024-03-06T07:29:45.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.857+0000 7f51f9c4e640 1 -- start start 2024-03-06T07:29:45.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f9c4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f415dfe0 0x7f51f40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f9c4e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51f40cf830 0x7f51f40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f9c4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f40b3430 con 0x7f51f40d27f0 2024-03-06T07:29:45.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f9c4e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f40d2c50 con 0x7f51f40cf830 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f9c4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40b31e0 con 0x7f51f415dfe0 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f2ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51f40cf830 0x7f51f40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f51f40d27f0 0x7f51f40d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47264/0 (socket says 172.21.15.72:47264) 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f3fff640 1 -- 172.21.15.72:0/3250332765 learned_addr learned my addr 172.21.15.72:0/3250332765 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f2ffd640 1 -- 172.21.15.72:0/3250332765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f415dfe0 msgr2=0x7f51f40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f2ffd640 1 --2- 172.21.15.72:0/3250332765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f415dfe0 0x7f51f40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f2ffd640 1 -- 172.21.15.72:0/3250332765 >> v1:172.21.15.72:6789/0 conn(0x7f51f40d27f0 legacy=0x7f51f40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f2ffd640 1 -- 172.21.15.72:0/3250332765 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f40b3710 con 0x7f51f40cf830 2024-03-06T07:29:45.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.858+0000 7f51f27fc640 1 -- 172.21.15.72:0/3250332765 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418792018 0 0) 0x7f51f40b3430 con 0x7f51f40d27f0 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.859+0000 7f51f2ffd640 1 --2- 172.21.15.72:0/3250332765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40cf830 0x7f51f40d22b0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f51e40097a0 tx=0x7f51e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d57399a9db364be server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.859+0000 7f51f27fc640 1 -- 172.21.15.72:0/3250332765 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4035020 con 0x7f51f40cf830 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f27fc640 1 -- 172.21.15.72:0/3250332765 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e4031830 con 0x7f51f40cf830 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f27fc640 1 -- 172.21.15.72:0/3250332765 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4031c00 con 0x7f51f40cf830 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 -- 172.21.15.72:0/3250332765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40cf830 msgr2=0x7f51f40d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/3250332765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40cf830 0x7f51f40d22b0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f51e40097a0 tx=0x7f51e402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 -- 172.21.15.72:0/3250332765 shutdown_connections 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/3250332765 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40cf830 0x7f51f40d22b0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/3250332765 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f415dfe0 0x7f51f40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 -- 172.21.15.72:0/3250332765 >> 172.21.15.72:0/3250332765 conn(0x7f51f40d3cd0 msgr2=0x7f51f40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:45.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 -- 172.21.15.72:0/3250332765 shutdown_connections 2024-03-06T07:29:45.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.860+0000 7f51f9c4e640 1 -- 172.21.15.72:0/3250332765 wait complete. 2024-03-06T07:29:45.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.861+0000 7f51f9c4e640 1 Processor -- start 2024-03-06T07:29:45.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.861+0000 7f51f9c4e640 1 -- start start 2024-03-06T07:29:45.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f9c4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f40cf830 0x7f51f40c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f9c4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 0x7f51f415ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f9c4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f415dfe0 0x7f51f40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f9c4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40d1c70 con 0x7f51f40d27f0 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f9c4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40b3d60 con 0x7f51f415dfe0 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f9c4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40d1df0 con 0x7f51f40cf830 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 0x7f51f415ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 0x7f51f415ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46306/0 (socket says 172.21.15.72:46306) 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f2ffd640 1 -- 172.21.15.72:0/2511443922 learned_addr learned my addr 172.21.15.72:0/2511443922 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.862+0000 7f51f37fe640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f40cf830 0x7f51f40c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f2ffd640 1 -- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f40cf830 msgr2=0x7f51f40c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f2ffd640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f40cf830 0x7f51f40c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f2ffd640 1 -- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f415dfe0 msgr2=0x7f51f40b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f2ffd640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f415dfe0 0x7f51f40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:45.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f2ffd640 1 -- 172.21.15.72:0/2511443922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f4160a00 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f37fe640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f40cf830 0x7f51f40c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f2ffd640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 0x7f51f415ff40 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f51e402bee0 tx=0x7f51e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e403ba80 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e403bc20 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e40453f0 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f4160ce0 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.863+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f40cbcf0 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.864+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f40cb800 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.864+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f51e4038020 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.865+0000 7f51dbfff640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51e006b4d0 0x7f51e006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.865+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51e40ac0b0 con 0x7f51f40d27f0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.865+0000 7f51f37fe640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51e006b4d0 0x7f51e006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:45.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.865+0000 7f51f37fe640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51e006b4d0 0x7f51e006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f51e800a870 tx=0x7f51e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:45.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:45.869+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f51e4040070 con 0x7f51f40d27f0 2024-03-06T07:29:46.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.082+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f404f330 con 0x7f51f40d27f0 2024-03-06T07:29:46.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.083+0000 7f51dbfff640 1 -- 172.21.15.72:0/2511443922 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f51e407ae50 con 0x7f51f40d27f0 2024-03-06T07:29:46.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:46.084 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:46.085 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:46.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51e006b4d0 msgr2=0x7f51e006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51e006b4d0 0x7f51e006d990 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f51e800a870 tx=0x7f51e8008040 comp rx=0 tx=0).stop 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 msgr2=0x7f51f415ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 0x7f51f415ff40 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f51e402bee0 tx=0x7f51e40046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 shutdown_connections 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f51e006b4d0 0x7f51e006d990 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f415dfe0 0x7f51f40b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51f40d27f0 0x7f51f415ff40 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 --2- 172.21.15.72:0/2511443922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f40cf830 0x7f51f40c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 >> 172.21.15.72:0/2511443922 conn(0x7f51f40d3cd0 msgr2=0x7f51f40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 shutdown_connections 2024-03-06T07:29:46.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:46.085+0000 7f51f9c4e640 1 -- 172.21.15.72:0/2511443922 wait complete. 2024-03-06T07:29:46.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:46.298 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:47.300 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:47.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.400+0000 7fb33e748640 1 Processor -- start 2024-03-06T07:29:47.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.400+0000 7fb33e748640 1 -- start start 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.400+0000 7fb33e748640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 0x7fb3380b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.400+0000 7fb33e748640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3380d25d0 con 0x7fb3380d4760 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.400+0000 7fb33e748640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3380d2870 con 0x7fb3380b4170 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.400+0000 7fb33e748640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380d4bf0 con 0x7fb3380d0a80 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb3377fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb3380d4760 0x7fb3380cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47274/0 (socket says 172.21.15.72:47274) 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb3377fe640 1 -- 172.21.15.72:0/168718631 learned_addr learned my addr 172.21.15.72:0/168718631 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754319090 0 0) 0x7fb3380d25d0 con 0x7fb3380d4760 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb320003520 con 0x7fb3380d4760 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb33cce5640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 0x7fb3380b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3603038035 0 0) 0x7fb320003520 con 0x7fb3380d4760 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 msgr2=0x7fb3380b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 0x7fb3380b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 >> v1:172.21.15.146:6789/0 conn(0x7fb3380b4170 legacy=0x7fb3380d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380d2b50 con 0x7fb3380d4760 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fb32c001670 con 0x7fb3380d4760 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 >> v1:172.21.15.72:6789/0 conn(0x7fb3380d4760 legacy=0x7fb3380cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3200039e0 0x7fb320003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 0x7fb320004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb320004d40 0x7fb3200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380d4bf0 con 0x7fb3200039e0 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32000baa0 con 0x7fb320004d40 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32000bca0 con 0x7fb3380d0a80 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.401+0000 7fb337fff640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3200039e0 0x7fb320003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb3377fe640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 0x7fb320004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb33cce5640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb320004d40 0x7fb3200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb337fff640 1 -- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 msgr2=0x7fb320004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb337fff640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3380d0a80 0x7fb320004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb337fff640 1 -- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb320004d40 msgr2=0x7fb3200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:47.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb337fff640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb320004d40 0x7fb3200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb337fff640 1 -- 172.21.15.72:0/168718631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320003520 con 0x7fb3200039e0 2024-03-06T07:29:47.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.402+0000 7fb337fff640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3200039e0 0x7fb320003e00 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb328001750 tx=0x7fb320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb6737af7d57b4a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328001370 con 0x7fb3200039e0 2024-03-06T07:29:47.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3280052d0 con 0x7fb3200039e0 2024-03-06T07:29:47.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb336ffd640 1 -- 172.21.15.72:0/168718631 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328003830 con 0x7fb3200039e0 2024-03-06T07:29:47.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33e748640 1 -- 172.21.15.72:0/168718631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3200039e0 msgr2=0x7fb320003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33e748640 1 --2- 172.21.15.72:0/168718631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3200039e0 0x7fb320003e00 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb328001750 tx=0x7fb320005830 comp rx=0 tx=0).stop 2024-03-06T07:29:47.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33cce5640 1 -- 172.21.15.72:0/168718631 reap_dead start 2024-03-06T07:29:47.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33e748640 1 -- 172.21.15.72:0/168718631 shutdown_connections 2024-03-06T07:29:47.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33e748640 1 -- 172.21.15.72:0/168718631 >> 172.21.15.72:0/168718631 conn(0x7fb3380d3c40 msgr2=0x7fb3380d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33e748640 1 -- 172.21.15.72:0/168718631 shutdown_connections 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.403+0000 7fb33e748640 1 -- 172.21.15.72:0/168718631 wait complete. 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.404+0000 7fb33e748640 1 Processor -- start 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 -- start start 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200039e0 0x7fb338156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3381570c0 0x7fb33815e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 0x7fb33815ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380b4dd0 con 0x7fb3380cd610 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380ce9e0 con 0x7fb3200039e0 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33e748640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3380b4f50 con 0x7fb3381570c0 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33cce5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 0x7fb33815ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33cce5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 0x7fb33815ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46316/0 (socket says 172.21.15.72:46316) 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.405+0000 7fb33cce5640 1 -- 172.21.15.72:0/4211990260 learned_addr learned my addr 172.21.15.72:0/4211990260 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb3377fe640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3381570c0 0x7fb33815e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb337fff640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200039e0 0x7fb338156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33cce5640 1 -- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3381570c0 msgr2=0x7fb33815e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33cce5640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3381570c0 0x7fb33815e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33cce5640 1 -- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200039e0 msgr2=0x7fb338156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33cce5640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200039e0 0x7fb338156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33cce5640 1 -- 172.21.15.72:0/4211990260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3380cc570 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb3377fe640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3381570c0 0x7fb33815e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33cce5640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 0x7fb33815ea10 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb330007d10 tx=0x7fb3300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330011e40 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33001b070 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3380cc850 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330003990 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.406+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3380cce10 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.407+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33804f330 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.408+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb330005d10 con 0x7fb3380cd610 2024-03-06T07:29:47.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.408+0000 7fb334ff9640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb32406b500 0x7fb32406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:47.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.408+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb33008a650 con 0x7fb3380cd610 2024-03-06T07:29:47.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.408+0000 7fb337fff640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb32406b500 0x7fb32406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:47.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.408+0000 7fb337fff640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb32406b500 0x7fb32406d9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb328033000 tx=0x7fb328009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:47.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.412+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb330025020 con 0x7fb3380cd610 2024-03-06T07:29:47.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.644+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3380733b0 con 0x7fb3380cd610 2024-03-06T07:29:47.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.645+0000 7fb334ff9640 1 -- 172.21.15.72:0/4211990260 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fb330014350 con 0x7fb3380cd610 2024-03-06T07:29:47.647 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:47.647 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:47.647 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.646+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb32406b500 msgr2=0x7fb32406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.646+0000 7fb33e748640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb32406b500 0x7fb32406d9c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb328033000 tx=0x7fb328009030 comp rx=0 tx=0).stop 2024-03-06T07:29:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.646+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 msgr2=0x7fb33815ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.646+0000 7fb33e748640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 0x7fb33815ea10 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fb330007d10 tx=0x7fb3300115a0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 shutdown_connections 2024-03-06T07:29:47.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb32406b500 0x7fb32406d9c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3380cd610 0x7fb33815ea10 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3381570c0 0x7fb33815e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 --2- 172.21.15.72:0/4211990260 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3200039e0 0x7fb338156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:47.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 >> 172.21.15.72:0/4211990260 conn(0x7fb3380d3c40 msgr2=0x7fb338160230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:47.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 shutdown_connections 2024-03-06T07:29:47.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:47.647+0000 7fb33e748640 1 -- 172.21.15.72:0/4211990260 wait complete. 2024-03-06T07:29:47.691 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:47.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:47.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:48.863 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:48.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.963+0000 7f42aa7a0640 1 Processor -- start 2024-03-06T07:29:48.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.963+0000 7f42aa7a0640 1 -- start start 2024-03-06T07:29:48.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42aa7a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 0x7f42a415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42aa7a0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a415f210 con 0x7f42a40b6e30 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42aa7a0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42a4162660 con 0x7f42a40b29f0 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42aa7a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40d1970 con 0x7f42a40b1200 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42a8f9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f42a40b6e30 0x7f42a4161840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47288/0 (socket says 172.21.15.72:47288) 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42a8f9d640 1 -- 172.21.15.72:0/963922899 learned_addr learned my addr 172.21.15.72:0/963922899 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f42a979e640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 0x7f42a415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235707579 0 0) 0x7f4294001110 con 0x7f42a40b6e30 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4290003520 con 0x7f42a40b6e30 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3825053727 0 0) 0x7f4290003520 con 0x7f42a40b6e30 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 msgr2=0x7f42a415ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 0x7f42a415ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 >> v1:172.21.15.146:6789/0 conn(0x7f42a40b29f0 legacy=0x7f42a4161f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40d1c50 con 0x7f42a40b6e30 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f4294001840 con 0x7f42a40b6e30 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 >> v1:172.21.15.72:6789/0 conn(0x7f42a40b6e30 legacy=0x7f42a4161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f4290003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4290004800 0x7f4290004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 0x7f42900051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40d1970 con 0x7f4290003a50 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429000bad0 con 0x7f4290004800 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.964+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429000bcd0 con 0x7f42a40b1200 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a979e640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f4290003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a8f9d640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4290004800 0x7f4290004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a9f9f640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 0x7f42900051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a979e640 1 -- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 msgr2=0x7f42900051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a979e640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40b1200 0x7f42900051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a979e640 1 -- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4290004800 msgr2=0x7f4290004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:48.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a979e640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4290004800 0x7f4290004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.965+0000 7f42a979e640 1 -- 172.21.15.72:0/963922899 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4290003520 con 0x7f4290003a50 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.966+0000 7f42a979e640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f4290003e30 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009ac0 tx=0x7f4290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3716e9e4a688f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.966+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0039070 con 0x7f4290003a50 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.966+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a002c6b0 con 0x7f4290003a50 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.966+0000 7f429bfff640 1 -- 172.21.15.72:0/963922899 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0009690 con 0x7f4290003a50 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42aa7a0640 1 -- 172.21.15.72:0/963922899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 msgr2=0x7f4290003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/963922899 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f4290003e30 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f42a0009ac0 tx=0x7f4290005830 comp rx=0 tx=0).stop 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42a9f9f640 1 -- 172.21.15.72:0/963922899 reap_dead start 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42aa7a0640 1 -- 172.21.15.72:0/963922899 shutdown_connections 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42aa7a0640 1 -- 172.21.15.72:0/963922899 >> 172.21.15.72:0/963922899 conn(0x7f42a409a9f0 msgr2=0x7f42a409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42aa7a0640 1 -- 172.21.15.72:0/963922899 shutdown_connections 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.967+0000 7f42aa7a0640 1 -- 172.21.15.72:0/963922899 wait complete. 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.968+0000 7f42aa7a0640 1 Processor -- start 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.968+0000 7f42aa7a0640 1 -- start start 2024-03-06T07:29:48.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f42a414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40cd4f0 0x7f42a40cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42a40cfbe0 0x7f42a40d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40d1640 con 0x7f4290003a50 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a40d2210 con 0x7f42a40cfbe0 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42a406ca70 con 0x7f42a40cd4f0 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f42a414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f42a414a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46340/0 (socket says 172.21.15.72:46340) 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 -- 172.21.15.72:0/1915638876 learned_addr learned my addr 172.21.15.72:0/1915638876 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 -- 172.21.15.72:0/1915638876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40cd4f0 msgr2=0x7f42a40cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40cd4f0 0x7f42a40cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 -- 172.21.15.72:0/1915638876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42a40cfbe0 msgr2=0x7f42a40d0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42a40cfbe0 0x7f42a40d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 -- 172.21.15.72:0/1915638876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a40d09b0 con 0x7f4290003a50 2024-03-06T07:29:48.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42a979e640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f42a414a590 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f42a0037040 tx=0x7f42a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0042020 con 0x7f4290003a50 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a003a040 con 0x7f4290003a50 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a40d0c90 con 0x7f4290003a50 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a0039070 con 0x7f4290003a50 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.969+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a40d05a0 con 0x7f4290003a50 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.970+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4068f80 con 0x7f4290003a50 2024-03-06T07:29:48.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.971+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f42a0045020 con 0x7f4290003a50 2024-03-06T07:29:48.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.971+0000 7f4299ffb640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f428806b500 0x7f428806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:48.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.972+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f42a00ab3d0 con 0x7f4290003a50 2024-03-06T07:29:48.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.972+0000 7f42a8f9d640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f428806b500 0x7f428806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:48.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.972+0000 7f42a8f9d640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f428806b500 0x7f428806d9c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4294001d70 tx=0x7f4294001ce0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:48.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:48.975+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f42a004c020 con 0x7f4290003a50 2024-03-06T07:29:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.185+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42a40b1730 con 0x7f4290003a50 2024-03-06T07:29:49.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.185+0000 7f4299ffb640 1 -- 172.21.15.72:0/1915638876 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f42a007a380 con 0x7f4290003a50 2024-03-06T07:29:49.188 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:49.188 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:49.188 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f428806b500 msgr2=0x7f428806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f428806b500 0x7f428806d9c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f4294001d70 tx=0x7f4294001ce0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 msgr2=0x7f42a414a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f42a414a590 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f42a0037040 tx=0x7f42a0004730 comp rx=0 tx=0).stop 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 shutdown_connections 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f428806b500 0x7f428806d9c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42a40cfbe0 0x7f42a40d0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42a40cd4f0 0x7f42a40cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 --2- 172.21.15.72:0/1915638876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4290003a50 0x7f42a414a590 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:49.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.187+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 >> 172.21.15.72:0/1915638876 conn(0x7f42a409a9f0 msgr2=0x7f42a40afc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:49.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.188+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 shutdown_connections 2024-03-06T07:29:49.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:49.188+0000 7f42aa7a0640 1 -- 172.21.15.72:0/1915638876 wait complete. 2024-03-06T07:29:49.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:49.402 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:49.694 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:50.403 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:50.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.502+0000 7fbfec288640 1 Processor -- start 2024-03-06T07:29:50.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.502+0000 7fbfec288640 1 -- start start 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfec288640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbfe415dfe0 0x7fbfe40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfec288640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe40cf830 0x7fbfe40d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfec288640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe40b3430 con 0x7fbfe40d27f0 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfec288640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe40d2c50 con 0x7fbfe415dfe0 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfec288640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40b31e0 con 0x7fbfe40cf830 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfea024640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbfe415dfe0 0x7fbfe40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfea024640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbfe415dfe0 0x7fbfe40cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46356/0 (socket says 172.21.15.72:46356) 2024-03-06T07:29:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfea024640 1 -- 172.21.15.72:0/3071380437 learned_addr learned my addr 172.21.15.72:0/3071380437 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.503+0000 7fbfe9823640 1 --2- 172.21.15.72:0/3071380437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe40cf830 0x7fbfe40d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.504+0000 7fbfea024640 1 -- 172.21.15.72:0/3071380437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe40cf830 msgr2=0x7fbfe40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.504+0000 7fbfea024640 1 --2- 172.21.15.72:0/3071380437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe40cf830 0x7fbfe40d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.504+0000 7fbfea024640 1 -- 172.21.15.72:0/3071380437 >> v1:172.21.15.146:6789/0 conn(0x7fbfe40d27f0 legacy=0x7fbfe40d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.504+0000 7fbfea024640 1 -- 172.21.15.72:0/3071380437 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe40b3710 con 0x7fbfe415dfe0 2024-03-06T07:29:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.504+0000 7fbfea024640 1 --2- 172.21.15.72:0/3071380437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe415dfe0 0x7fbfe40cf2f0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009870 tx=0x7fbfd402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908e30938bf7e4f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfe9022640 1 -- 172.21.15.72:0/3071380437 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4034070 con 0x7fbfe415dfe0 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfe9022640 1 -- 172.21.15.72:0/3071380437 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd4030840 con 0x7fbfe415dfe0 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfe9022640 1 -- 172.21.15.72:0/3071380437 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4030c10 con 0x7fbfe415dfe0 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 -- 172.21.15.72:0/3071380437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe415dfe0 msgr2=0x7fbfe40cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 --2- 172.21.15.72:0/3071380437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe415dfe0 0x7fbfe40cf2f0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009870 tx=0x7fbfd402b980 comp rx=0 tx=0).stop 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 -- 172.21.15.72:0/3071380437 shutdown_connections 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 --2- 172.21.15.72:0/3071380437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe40cf830 0x7fbfe40d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 --2- 172.21.15.72:0/3071380437 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe415dfe0 0x7fbfe40cf2f0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 -- 172.21.15.72:0/3071380437 >> 172.21.15.72:0/3071380437 conn(0x7fbfe40d3cd0 msgr2=0x7fbfe40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 -- 172.21.15.72:0/3071380437 shutdown_connections 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.505+0000 7fbfec288640 1 -- 172.21.15.72:0/3071380437 wait complete. 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.506+0000 7fbfec288640 1 Processor -- start 2024-03-06T07:29:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.506+0000 7fbfec288640 1 -- start start 2024-03-06T07:29:50.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfec288640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe40cf830 0x7fbfe414af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfec288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 0x7fbfe415f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfec288640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe415dfe0 0x7fbfe4160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfec288640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40d1c70 con 0x7fbfe40d27f0 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfec288640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40b3d60 con 0x7fbfe40cf830 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfec288640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe40d1df0 con 0x7fbfe415dfe0 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfe9823640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 0x7fbfe415f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfe9823640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 0x7fbfe415f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46372/0 (socket says 172.21.15.72:46372) 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfe9823640 1 -- 172.21.15.72:0/2944733535 learned_addr learned my addr 172.21.15.72:0/2944733535 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfea024640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe40cf830 0x7fbfe414af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.507+0000 7fbfea825640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe415dfe0 0x7fbfe4160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfe9823640 1 -- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe415dfe0 msgr2=0x7fbfe4160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfe9823640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe415dfe0 0x7fbfe4160490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfe9823640 1 -- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe40cf830 msgr2=0x7fbfe414af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfe9823640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe40cf830 0x7fbfe414af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfe9823640 1 -- 172.21.15.72:0/2944733535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe415a330 con 0x7fbfe40d27f0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfe9823640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 0x7fbfe415f530 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000e980 tx=0x7fbfe000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0017070 con 0x7fbfe40d27f0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe000ca00 con 0x7fbfe40d27f0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe40cda60 con 0x7fbfe40d27f0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe000cdd0 con 0x7fbfe40d27f0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.508+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe40ce020 con 0x7fbfe40d27f0 2024-03-06T07:29:50.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.509+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe40cde50 con 0x7fbfe40d27f0 2024-03-06T07:29:50.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.510+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfe001a070 con 0x7fbfe40d27f0 2024-03-06T07:29:50.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.510+0000 7fbfdaffd640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfc806b4d0 0x7fbfc806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:50.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.510+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbfe008a250 con 0x7fbfe40d27f0 2024-03-06T07:29:50.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.510+0000 7fbfea024640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfc806b4d0 0x7fbfc806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:50.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.511+0000 7fbfea024640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfc806b4d0 0x7fbfc806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4036000 tx=0x7fbfd403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:50.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.515+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbfe0059090 con 0x7fbfe40d27f0 2024-03-06T07:29:50.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.727+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe40cdc30 con 0x7fbfe40d27f0 2024-03-06T07:29:50.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.727+0000 7fbfdaffd640 1 -- 172.21.15.72:0/2944733535 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fbfe0058990 con 0x7fbfe40d27f0 2024-03-06T07:29:50.729 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:50.729 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:50.730 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:50.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfc806b4d0 msgr2=0x7fbfc806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfc806b4d0 0x7fbfc806d990 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4036000 tx=0x7fbfd403b040 comp rx=0 tx=0).stop 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 msgr2=0x7fbfe415f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 0x7fbfe415f530 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000e980 tx=0x7fbfe000ee40 comp rx=0 tx=0).stop 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 shutdown_connections 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfc806b4d0 0x7fbfc806d990 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe415dfe0 0x7fbfe4160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe40d27f0 0x7fbfe415f530 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 --2- 172.21.15.72:0/2944733535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe40cf830 0x7fbfe414af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 >> 172.21.15.72:0/2944733535 conn(0x7fbfe40d3cd0 msgr2=0x7fbfe406d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 shutdown_connections 2024-03-06T07:29:50.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:50.729+0000 7fbfec288640 1 -- 172.21.15.72:0/2944733535 wait complete. 2024-03-06T07:29:50.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:50.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:51.688 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:51.690 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:29:51.695 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:51.944 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:52.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.044+0000 7fed2b50c640 1 Processor -- start 2024-03-06T07:29:52.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.045+0000 7fed2b50c640 1 -- start start 2024-03-06T07:29:52.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed2b50c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed24166af0 0x7fed24168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed2b50c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2415e380 con 0x7fed240d00c0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed2b50c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240cdc10 con 0x7fed240cf850 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed2b50c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2406ca90 con 0x7fed24166af0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed292a8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fed240d00c0 0x7fed240d04a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47292/0 (socket says 172.21.15.72:47292) 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed292a8640 1 -- 172.21.15.72:0/1454349572 learned_addr learned my addr 172.21.15.72:0/1454349572 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845199382 0 0) 0x7fed2415e380 con 0x7fed240d00c0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10003520 con 0x7fed240d00c0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed29aa9640 1 --2- 172.21.15.72:0/1454349572 >> v2:172.21.15.146:3300/0 conn(0x7fed24166af0 0x7fed24168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 320986373 0 0) 0x7fed10003520 con 0x7fed240d00c0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 >> v2:172.21.15.146:3300/0 conn(0x7fed24166af0 msgr2=0x7fed24168ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 --2- 172.21.15.72:0/1454349572 >> v2:172.21.15.146:3300/0 conn(0x7fed24166af0 0x7fed24168ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 >> v1:172.21.15.146:6789/0 conn(0x7fed240cf850 legacy=0x7fed240cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2406c160 con 0x7fed240d00c0 2024-03-06T07:29:52.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fed14001670 con 0x7fed240d00c0 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.046+0000 7fed29aa9640 1 --2- 172.21.15.72:0/1454349572 >> v2:172.21.15.146:3300/0 conn(0x7fed24166af0 0x7fed24168ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 >> v1:172.21.15.72:6789/0 conn(0x7fed240d00c0 legacy=0x7fed240d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100039b0 0x7fed10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10004800 0x7fed10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed100051c0 0x7fed1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2406ca90 con 0x7fed100051c0 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000c110 con 0x7fed10004800 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1000c310 con 0x7fed100039b0 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed29aa9640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed100051c0 0x7fed1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed28aa7640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10004800 0x7fed10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed292a8640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100039b0 0x7fed10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed29aa9640 1 -- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100039b0 msgr2=0x7fed10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed29aa9640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100039b0 0x7fed10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed29aa9640 1 -- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10004800 msgr2=0x7fed10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed29aa9640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed10004800 0x7fed10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed29aa9640 1 -- 172.21.15.72:0/1454349572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed10003520 con 0x7fed100051c0 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.047+0000 7fed292a8640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100039b0 0x7fed10003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed29aa9640 1 -- 172.21.15.72:0/1454349572 reap_dead start 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed29aa9640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed100051c0 0x7fed1000b9c0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0073b0 tx=0x7fed10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1297a0bcf715825b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c039070 con 0x7fed100051c0 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c02cd90 con 0x7fed100051c0 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed1bfff640 1 -- 172.21.15.72:0/1454349572 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c009c80 con 0x7fed100051c0 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 -- 172.21.15.72:0/1454349572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed100051c0 msgr2=0x7fed1000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed100051c0 0x7fed1000b9c0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0073b0 tx=0x7fed10005830 comp rx=0 tx=0).stop 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 -- 172.21.15.72:0/1454349572 shutdown_connections 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 --2- 172.21.15.72:0/1454349572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed100051c0 0x7fed1000b9c0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 -- 172.21.15.72:0/1454349572 >> 172.21.15.72:0/1454349572 conn(0x7fed240b35a0 msgr2=0x7fed240b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 -- 172.21.15.72:0/1454349572 shutdown_connections 2024-03-06T07:29:52.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.048+0000 7fed2b50c640 1 -- 172.21.15.72:0/1454349572 wait complete. 2024-03-06T07:29:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 Processor -- start 2024-03-06T07:29:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 -- start start 2024-03-06T07:29:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100051c0 0x7fed24157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 0x7fed2415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed240c6220 0x7fed2406d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed240b1030 con 0x7fed240c7a00 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2406c720 con 0x7fed240c6220 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed2b50c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed240c81a0 con 0x7fed100051c0 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed28aa7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 0x7fed2415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed28aa7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 0x7fed2415c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46398/0 (socket says 172.21.15.72:46398) 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.050+0000 7fed28aa7640 1 -- 172.21.15.72:0/910657695 learned_addr learned my addr 172.21.15.72:0/910657695 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed29aa9640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed240c6220 0x7fed2406d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed292a8640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100051c0 0x7fed24157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed28aa7640 1 -- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100051c0 msgr2=0x7fed24157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed28aa7640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100051c0 0x7fed24157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed28aa7640 1 -- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed240c6220 msgr2=0x7fed2406d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed28aa7640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed240c6220 0x7fed2406d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed28aa7640 1 -- 172.21.15.72:0/910657695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240c7ec0 con 0x7fed240c7a00 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed28aa7640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 0x7fed2415c4d0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fed2000f620 tx=0x7fed2000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed200052e0 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed20018070 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed240c5740 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20003810 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.051+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed240c5d00 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.052+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2404f330 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.053+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fed200039b0 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.053+0000 7fed19ffb640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fed0806b500 0x7fed0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.053+0000 7fed292a8640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fed0806b500 0x7fed0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.053+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fed20089ed0 con 0x7fed240c7a00 2024-03-06T07:29:52.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.054+0000 7fed292a8640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fed0806b500 0x7fed0806d9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fed14000ff0 tx=0x7fed14007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:52.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.058+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fed200586c0 con 0x7fed240c7a00 2024-03-06T07:29:52.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.273+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed240cd230 con 0x7fed240c7a00 2024-03-06T07:29:52.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.273+0000 7fed19ffb640 1 -- 172.21.15.72:0/910657695 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fed2005c760 con 0x7fed240c7a00 2024-03-06T07:29:52.275 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:52.276 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:52.276 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:52.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fed0806b500 msgr2=0x7fed0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fed0806b500 0x7fed0806d9c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fed14000ff0 tx=0x7fed14007040 comp rx=0 tx=0).stop 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 msgr2=0x7fed2415c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 0x7fed2415c4d0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7fed2000f620 tx=0x7fed2000fae0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 shutdown_connections 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fed0806b500 0x7fed0806d9c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed240c6220 0x7fed2406d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fed240c7a00 0x7fed2415c4d0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 --2- 172.21.15.72:0/910657695 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed100051c0 0x7fed24157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 >> 172.21.15.72:0/910657695 conn(0x7fed240b35a0 msgr2=0x7fed2415ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 shutdown_connections 2024-03-06T07:29:52.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:52.275+0000 7fed2b50c640 1 -- 172.21.15.72:0/910657695 wait complete. 2024-03-06T07:29:52.489 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:52.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:53.491 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:53.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.591+0000 7f8a8006b640 1 Processor -- start 2024-03-06T07:29:53.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.591+0000 7f8a8006b640 1 -- start start 2024-03-06T07:29:53.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a8006b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8a7815bb20 0x7f8a780d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a8006b640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8a780cd8c0 0x7f8a780d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a8006b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a78161980 con 0x7f8a780cfff0 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a8006b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a78161c60 con 0x7f8a7815bb20 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a8006b640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a78161f40 con 0x7f8a780cd8c0 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8a7815bb20 0x7f8a780d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8a7815bb20 0x7f8a780d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59624/0 (socket says 172.21.15.72:59624) 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 -- 172.21.15.72:0/3752143250 learned_addr learned my addr 172.21.15.72:0/3752143250 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7e608640 1 --2- 172.21.15.72:0/3752143250 >> v2:172.21.15.146:3300/0 conn(0x7f8a780cd8c0 0x7f8a780d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 -- 172.21.15.72:0/3752143250 >> v2:172.21.15.146:3300/0 conn(0x7f8a780cd8c0 msgr2=0x7f8a780d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 --2- 172.21.15.72:0/3752143250 >> v2:172.21.15.146:3300/0 conn(0x7f8a780cd8c0 0x7f8a780d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 -- 172.21.15.72:0/3752143250 >> v1:172.21.15.146:6789/0 conn(0x7f8a780cfff0 legacy=0x7f8a780ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7de07640 1 -- 172.21.15.72:0/3752143250 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78162220 con 0x7f8a7815bb20 2024-03-06T07:29:53.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.592+0000 7f8a7e608640 1 --2- 172.21.15.72:0/3752143250 >> v2:172.21.15.146:3300/0 conn(0x7f8a780cd8c0 0x7f8a780d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.593+0000 7f8a7de07640 1 --2- 172.21.15.72:0/3752143250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a780d1df0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f8a781618c0 tx=0x7f8a7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321af5b5c787df4e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.593+0000 7f8a7ce05640 1 -- 172.21.15.72:0/3752143250 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a74035020 con 0x7f8a7815bb20 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.593+0000 7f8a7ce05640 1 -- 172.21.15.72:0/3752143250 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a74031870 con 0x7f8a7815bb20 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.593+0000 7f8a7ce05640 1 -- 172.21.15.72:0/3752143250 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a74031c40 con 0x7f8a7815bb20 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 -- 172.21.15.72:0/3752143250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 msgr2=0x7f8a780d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 --2- 172.21.15.72:0/3752143250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a780d1df0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f8a781618c0 tx=0x7f8a7402b910 comp rx=0 tx=0).stop 2024-03-06T07:29:53.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 -- 172.21.15.72:0/3752143250 shutdown_connections 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 --2- 172.21.15.72:0/3752143250 >> v2:172.21.15.146:3300/0 conn(0x7f8a780cd8c0 0x7f8a780d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 --2- 172.21.15.72:0/3752143250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a780d1df0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 -- 172.21.15.72:0/3752143250 >> 172.21.15.72:0/3752143250 conn(0x7f8a7809a9f0 msgr2=0x7f8a7809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 -- 172.21.15.72:0/3752143250 shutdown_connections 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.594+0000 7f8a8006b640 1 -- 172.21.15.72:0/3752143250 wait complete. 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.595+0000 7f8a8006b640 1 Processor -- start 2024-03-06T07:29:53.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.595+0000 7f8a8006b640 1 -- start start 2024-03-06T07:29:53.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a780cd8c0 0x7f8a781f8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a780cfff0 0x7f8a78154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a7806ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780d24b0 con 0x7f8a7815bb20 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780d2330 con 0x7f8a780cfff0 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780d1980 con 0x7f8a780cd8c0 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a7806ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a7806ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59628/0 (socket says 172.21.15.72:59628) 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 -- 172.21.15.72:0/8781651 learned_addr learned my addr 172.21.15.72:0/8781651 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7de07640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a780cd8c0 0x7f8a781f8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7d606640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a780cfff0 0x7f8a78154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 -- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a780cd8c0 msgr2=0x7f8a781f8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a780cd8c0 0x7f8a781f8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 -- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a780cfff0 msgr2=0x7f8a78154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a780cfff0 0x7f8a78154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 -- 172.21.15.72:0/8781651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78159e70 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7de07640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a780cd8c0 0x7f8a781f8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a7e608640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a7806ceb0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000cf20 tx=0x7f8a7000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7001a070 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7000d840 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a781592f0 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7000dc30 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.596+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a78159b90 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.597+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a7804f330 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.598+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8a7001d070 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.598+0000 7f8a6e7fc640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8a5c06b4d0 0x7f8a5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.598+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a7008ab00 con 0x7f8a7815bb20 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.598+0000 7f8a7de07640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8a5c06b4d0 0x7f8a5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:53.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.599+0000 7f8a7de07640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8a5c06b4d0 0x7f8a5c06d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8a74036000 tx=0x7f8a7403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:53.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.607+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8a70059940 con 0x7f8a7815bb20 2024-03-06T07:29:53.697 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:53.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.822+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a7806f090 con 0x7f8a7815bb20 2024-03-06T07:29:53.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.822+0000 7f8a6e7fc640 1 -- 172.21.15.72:0/8781651 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f8a70059240 con 0x7f8a7815bb20 2024-03-06T07:29:53.824 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:53.824 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:53.825 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8a5c06b4d0 msgr2=0x7f8a5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8a5c06b4d0 0x7f8a5c06d990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f8a74036000 tx=0x7f8a7403c040 comp rx=0 tx=0).stop 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 msgr2=0x7f8a7806ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a7806ceb0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000cf20 tx=0x7f8a7000a7d0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 shutdown_connections 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8a5c06b4d0 0x7f8a5c06d990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8a7815bb20 0x7f8a7806ceb0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a780cfff0 0x7f8a78154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 --2- 172.21.15.72:0/8781651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a780cd8c0 0x7f8a781f8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 >> 172.21.15.72:0/8781651 conn(0x7f8a7809a9f0 msgr2=0x7f8a780b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 shutdown_connections 2024-03-06T07:29:53.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:53.824+0000 7f8a8006b640 1 -- 172.21.15.72:0/8781651 wait complete. 2024-03-06T07:29:54.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:54.038 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:55.039 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:55.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.138+0000 7f2bde442640 1 Processor -- start 2024-03-06T07:29:55.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 -- start start 2024-03-06T07:29:55.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2bd815dfe0 0x7f2bd80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80cf830 0x7f2bd80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2bd80d27f0 0x7f2bd80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd80d2c50 con 0x7f2bd815dfe0 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bd80b31e0 con 0x7f2bd80d27f0 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bde442640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd80b3d60 con 0x7f2bd80cf830 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bd7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2bd815dfe0 0x7f2bd80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bd7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2bd815dfe0 0x7f2bd80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59636/0 (socket says 172.21.15.72:59636) 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.139+0000 7f2bd7fff640 1 -- 172.21.15.72:0/1170379324 learned_addr learned my addr 172.21.15.72:0/1170379324 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd7fff640 1 -- 172.21.15.72:0/1170379324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80cf830 msgr2=0x7f2bd80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd77fe640 1 --2- 172.21.15.72:0/1170379324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80cf830 0x7f2bd80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd7fff640 1 --2- 172.21.15.72:0/1170379324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80cf830 0x7f2bd80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd7fff640 1 -- 172.21.15.72:0/1170379324 >> v2:172.21.15.146:3300/0 conn(0x7f2bd80d27f0 msgr2=0x7f2bd80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd7fff640 1 --2- 172.21.15.72:0/1170379324 >> v2:172.21.15.146:3300/0 conn(0x7f2bd80d27f0 0x7f2bd80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd7fff640 1 -- 172.21.15.72:0/1170379324 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd80b3430 con 0x7f2bd815dfe0 2024-03-06T07:29:55.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.140+0000 7f2bd7fff640 1 --2- 172.21.15.72:0/1170379324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd815dfe0 0x7f2bd80cf2f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8067ad0 tx=0x7f2bc802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8708ab987852f9a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bd6ffd640 1 -- 172.21.15.72:0/1170379324 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8035020 con 0x7f2bd815dfe0 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bd6ffd640 1 -- 172.21.15.72:0/1170379324 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc8031870 con 0x7f2bd815dfe0 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bd6ffd640 1 -- 172.21.15.72:0/1170379324 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8031c40 con 0x7f2bd815dfe0 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 -- 172.21.15.72:0/1170379324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd815dfe0 msgr2=0x7f2bd80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 --2- 172.21.15.72:0/1170379324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd815dfe0 0x7f2bd80cf2f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8067ad0 tx=0x7f2bc802b910 comp rx=0 tx=0).stop 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 -- 172.21.15.72:0/1170379324 shutdown_connections 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 --2- 172.21.15.72:0/1170379324 >> v2:172.21.15.146:3300/0 conn(0x7f2bd80d27f0 0x7f2bd80d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 --2- 172.21.15.72:0/1170379324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80cf830 0x7f2bd80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 --2- 172.21.15.72:0/1170379324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd815dfe0 0x7f2bd80cf2f0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 -- 172.21.15.72:0/1170379324 >> 172.21.15.72:0/1170379324 conn(0x7f2bd80d3cd0 msgr2=0x7f2bd80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 -- 172.21.15.72:0/1170379324 shutdown_connections 2024-03-06T07:29:55.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.141+0000 7f2bde442640 1 -- 172.21.15.72:0/1170379324 wait complete. 2024-03-06T07:29:55.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.142+0000 7f2bde442640 1 Processor -- start 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 -- start start 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 0x7f2bd80cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80d27f0 0x7f2bd81f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bd815dfe0 0x7f2bd81f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd80d1c70 con 0x7f2bd80cf830 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd80b39f0 con 0x7f2bd815dfe0 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bde442640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd80d1df0 con 0x7f2bd80d27f0 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 0x7f2bd80cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 0x7f2bd80cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59644/0 (socket says 172.21.15.72:59644) 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.143+0000 7f2bd7fff640 1 -- 172.21.15.72:0/3104874522 learned_addr learned my addr 172.21.15.72:0/3104874522 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd77fe640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80d27f0 0x7f2bd81f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd7fff640 1 -- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80d27f0 msgr2=0x7f2bd81f6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd7fff640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80d27f0 0x7f2bd81f6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd7fff640 1 -- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bd815dfe0 msgr2=0x7f2bd81f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bdc9df640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bd815dfe0 0x7f2bd81f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd7fff640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bd815dfe0 0x7f2bd81f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd7fff640 1 -- 172.21.15.72:0/3104874522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd80b99f0 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd77fe640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80d27f0 0x7f2bd81f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bdc9df640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bd815dfe0 0x7f2bd81f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd7fff640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 0x7f2bd80cdcd0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f2bc802b950 tx=0x7f2bc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8040070 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc803a420 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd8160100 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8035040 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.144+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd80caf60 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.145+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd804f330 con 0x7f2bd80cf830 2024-03-06T07:29:55.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.146+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2bc8038020 con 0x7f2bd80cf830 2024-03-06T07:29:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.146+0000 7f2bd4ff9640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bc406b4d0 0x7f2bc406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.146+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2bc80ac240 con 0x7f2bd80cf830 2024-03-06T07:29:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.146+0000 7f2bd77fe640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bc406b4d0 0x7f2bc406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:55.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.147+0000 7f2bd77fe640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bc406b4d0 0x7f2bc406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8161570 tx=0x7f2bcc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:55.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.155+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2bc807b090 con 0x7f2bd80cf830 2024-03-06T07:29:55.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.364+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bd806a090 con 0x7f2bd80cf830 2024-03-06T07:29:55.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.365+0000 7f2bd4ff9640 1 -- 172.21.15.72:0/3104874522 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2bc807a990 con 0x7f2bd80cf830 2024-03-06T07:29:55.367 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:55.367 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:55.367 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:55.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bc406b4d0 msgr2=0x7f2bc406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bc406b4d0 0x7f2bc406d990 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8161570 tx=0x7f2bcc008040 comp rx=0 tx=0).stop 2024-03-06T07:29:55.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 msgr2=0x7f2bd80cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:55.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 0x7f2bd80cdcd0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f2bc802b950 tx=0x7f2bc80046c0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 shutdown_connections 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2bc406b4d0 0x7f2bc406d990 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bd815dfe0 0x7f2bd81f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bd80d27f0 0x7f2bd81f6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 --2- 172.21.15.72:0/3104874522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2bd80cf830 0x7f2bd80cdcd0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 >> 172.21.15.72:0/3104874522 conn(0x7f2bd80d3cd0 msgr2=0x7f2bd80b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 shutdown_connections 2024-03-06T07:29:55.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:55.367+0000 7f2bde442640 1 -- 172.21.15.72:0/3104874522 wait complete. 2024-03-06T07:29:55.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:55.581 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:55.700 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:56.583 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:56.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.685+0000 7fc6bae50640 1 Processor -- start 2024-03-06T07:29:56.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.685+0000 7fc6bae50640 1 -- start start 2024-03-06T07:29:56.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6bae50640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b41515e0 0x7fc6b40cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6bae50640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc6b40cd7a0 0x7fc6b40cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6bae50640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b4071f40 con 0x7fc6b415b430 2024-03-06T07:29:56.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6bae50640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b415b8b0 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6bae50640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4070a70 con 0x7fc6b41515e0 2024-03-06T07:29:56.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc6b40cd7a0 0x7fc6b40cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc6b40cd7a0 0x7fc6b40cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59660/0 (socket says 172.21.15.72:59660) 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 -- 172.21.15.72:0/153025714 learned_addr learned my addr 172.21.15.72:0/153025714 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b8bec640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b41515e0 0x7fc6b40cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 -- 172.21.15.72:0/153025714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b41515e0 msgr2=0x7fc6b40cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b41515e0 0x7fc6b40cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 -- 172.21.15.72:0/153025714 >> v1:172.21.15.146:6789/0 conn(0x7fc6b415b430 legacy=0x7fc6b4071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b93ed640 1 -- 172.21.15.72:0/153025714 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4070d50 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.686+0000 7fc6b8bec640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b41515e0 0x7fc6b40cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.687+0000 7fc6b93ed640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b40cdbd0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009e50 tx=0x7fc6ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9f7217477a7856 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.687+0000 7fc6b37fe640 1 -- 172.21.15.72:0/153025714 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac02ca70 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.687+0000 7fc6b37fe640 1 -- 172.21.15.72:0/153025714 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac02cc10 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.687+0000 7fc6b37fe640 1 -- 172.21.15.72:0/153025714 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0343f0 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 -- 172.21.15.72:0/153025714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 msgr2=0x7fc6b40cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b40cdbd0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac009e50 tx=0x7fc6ac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 -- 172.21.15.72:0/153025714 shutdown_connections 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b40cdbd0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 --2- 172.21.15.72:0/153025714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b41515e0 0x7fc6b40cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 -- 172.21.15.72:0/153025714 >> 172.21.15.72:0/153025714 conn(0x7fc6b40cf8a0 msgr2=0x7fc6b40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 -- 172.21.15.72:0/153025714 shutdown_connections 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.688+0000 7fc6bae50640 1 -- 172.21.15.72:0/153025714 wait complete. 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.689+0000 7fc6bae50640 1 Processor -- start 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.689+0000 7fc6bae50640 1 -- start start 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6bae50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b41381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6bae50640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6b41515e0 0x7fc6b414dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6bae50640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b415b430 0x7fc6b414e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6bae50640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40ce110 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6bae50640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b4071310 con 0x7fc6b41515e0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6bae50640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40cc3a0 con 0x7fc6b415b430 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b41381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b41381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59668/0 (socket says 172.21.15.72:59668) 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 -- 172.21.15.72:0/757246751 learned_addr learned my addr 172.21.15.72:0/757246751 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b93ed640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b415b430 0x7fc6b414e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b3fff640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6b41515e0 0x7fc6b414dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 -- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b415b430 msgr2=0x7fc6b414e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b415b430 0x7fc6b414e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 -- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6b41515e0 msgr2=0x7fc6b414dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6b41515e0 0x7fc6b414dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 -- 172.21.15.72:0/757246751 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b41386e0 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.690+0000 7fc6b8bec640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b41381a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c820 tx=0x7fc6a800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.691+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8015070 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.691+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a8007a30 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.691+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b415c8f0 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.691+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8007e00 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.691+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b415ceb0 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.692+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b4068840 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.692+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc6a8018070 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.692+0000 7fc6b17fa640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69806b500 0x7fc69806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.693+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc6a808a210 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.693+0000 7fc6b3fff640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69806b500 0x7fc69806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:56.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.693+0000 7fc6b3fff640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69806b500 0x7fc69806d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0006fd0 tx=0x7fc6a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:56.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.697+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc6a8058fc0 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.986+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6b414ef00 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.987+0000 7fc6b17fa640 1 -- 172.21.15.72:0/757246751 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc6b414ef00 con 0x7fc6b40cd7a0 2024-03-06T07:29:56.989 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:56.989 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:56.990 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:56.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69806b500 msgr2=0x7fc69806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69806b500 0x7fc69806d9c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0006fd0 tx=0x7fc6a0008040 comp rx=0 tx=0).stop 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 msgr2=0x7fc6b41381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b41381a0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c820 tx=0x7fc6a800cc20 comp rx=0 tx=0).stop 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 shutdown_connections 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc69806b500 0x7fc69806d9c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6b415b430 0x7fc6b414e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6b41515e0 0x7fc6b414dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 --2- 172.21.15.72:0/757246751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6b40cd7a0 0x7fc6b41381a0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 >> 172.21.15.72:0/757246751 conn(0x7fc6b40cf8a0 msgr2=0x7fc6b4072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 shutdown_connections 2024-03-06T07:29:56.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:56.989+0000 7fc6bae50640 1 -- 172.21.15.72:0/757246751 wait complete. 2024-03-06T07:29:57.204 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:57.204 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:58.205 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:58.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.303+0000 7f990e1ac640 1 Processor -- start 2024-03-06T07:29:58.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.303+0000 7f990e1ac640 1 -- start start 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f990e1ac640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f990815dfe0 0x7f99080cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f990e1ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99080d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f990e1ac640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080b3430 con 0x7f99080d27f0 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f990e1ac640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99080d2c50 con 0x7f990815dfe0 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f990e1ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080b31e0 con 0x7f99080cf830 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f99077fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f990815dfe0 0x7f99080cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f98feffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99080d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f9907fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f99080d27f0 0x7f99080d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:41076/0 (socket says 172.21.15.72:41076) 2024-03-06T07:29:58.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f9907fff640 1 -- 172.21.15.72:0/863099371 learned_addr learned my addr 172.21.15.72:0/863099371 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:29:58.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f99077fe640 1 -- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 msgr2=0x7f99080d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f99077fe640 1 --2- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99080d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f99077fe640 1 -- 172.21.15.72:0/863099371 >> v1:172.21.15.146:6789/0 conn(0x7f99080d27f0 legacy=0x7f99080d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f99077fe640 1 -- 172.21.15.72:0/863099371 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080b3710 con 0x7f990815dfe0 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.305+0000 7f98feffd640 1 --2- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99080d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.306+0000 7f99077fe640 1 --2- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990815dfe0 0x7f99080cf2f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f98f80097a0 tx=0x7f98f802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113b55a057f83539 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.306+0000 7f9906ffd640 1 -- 172.21.15.72:0/863099371 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8035020 con 0x7f990815dfe0 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.306+0000 7f9906ffd640 1 -- 172.21.15.72:0/863099371 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f8031810 con 0x7f990815dfe0 2024-03-06T07:29:58.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.306+0000 7f9906ffd640 1 -- 172.21.15.72:0/863099371 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f8031be0 con 0x7f990815dfe0 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 -- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990815dfe0 msgr2=0x7f99080cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 --2- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990815dfe0 0x7f99080cf2f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f98f80097a0 tx=0x7f98f802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 -- 172.21.15.72:0/863099371 shutdown_connections 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 --2- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99080d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 --2- 172.21.15.72:0/863099371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990815dfe0 0x7f99080cf2f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 -- 172.21.15.72:0/863099371 >> 172.21.15.72:0/863099371 conn(0x7f99080d3cd0 msgr2=0x7f99080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 -- 172.21.15.72:0/863099371 shutdown_connections 2024-03-06T07:29:58.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.307+0000 7f990e1ac640 1 -- 172.21.15.72:0/863099371 wait complete. 2024-03-06T07:29:58.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.309+0000 7f990e1ac640 1 Processor -- start 2024-03-06T07:29:58.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.309+0000 7f990e1ac640 1 -- start start 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.310+0000 7f990e1ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99081f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.310+0000 7f990e1ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99080d27f0 0x7f990814d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.310+0000 7f990e1ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 0x7f990814d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.310+0000 7f990e1ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080d1c70 con 0x7f990815dfe0 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.310+0000 7f990e1ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99080b3d60 con 0x7f99080d27f0 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.310+0000 7f990e1ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99080d1df0 con 0x7f99080cf830 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 0x7f990814d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 0x7f990814d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59670/0 (socket says 172.21.15.72:59670) 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 -- 172.21.15.72:0/1047792353 learned_addr learned my addr 172.21.15.72:0/1047792353 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:58.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f99077fe640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99081f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f98feffd640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99080d27f0 0x7f990814d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 -- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 msgr2=0x7f99081f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99081f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 -- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99080d27f0 msgr2=0x7f990814d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99080d27f0 0x7f990814d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 -- 172.21.15.72:0/1047792353 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99081f40b0 con 0x7f990815dfe0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9907fff640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 0x7f990814d860 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f990000cbc0 tx=0x7f990000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900017070 con 0x7f990815dfe0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990000ad40 con 0x7f990815dfe0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99081f4390 con 0x7f990815dfe0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9900012600 con 0x7f990815dfe0 2024-03-06T07:29:58.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.311+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990814ce00 con 0x7f990815dfe0 2024-03-06T07:29:58.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.312+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990804f330 con 0x7f990815dfe0 2024-03-06T07:29:58.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.313+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f990001a070 con 0x7f990815dfe0 2024-03-06T07:29:58.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.313+0000 7f9904ff9640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e806b4d0 0x7f98e806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:58.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.313+0000 7f99077fe640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e806b4d0 0x7f98e806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:58.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.313+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9900089e60 con 0x7f990815dfe0 2024-03-06T07:29:58.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.314+0000 7f99077fe640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e806b4d0 0x7f98e806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f98f8036000 tx=0x7f98f803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:58.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.317+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f99000586e0 con 0x7f990815dfe0 2024-03-06T07:29:58.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.527+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99080b0d40 con 0x7f990815dfe0 2024-03-06T07:29:58.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.527+0000 7f9904ff9640 1 -- 172.21.15.72:0/1047792353 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f990005c780 con 0x7f990815dfe0 2024-03-06T07:29:58.530 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:29:58.530 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:29:58.530 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e806b4d0 msgr2=0x7f98e806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e806b4d0 0x7f98e806d990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f98f8036000 tx=0x7f98f803c040 comp rx=0 tx=0).stop 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 msgr2=0x7f990814d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 0x7f990814d860 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f990000cbc0 tx=0x7f990000a430 comp rx=0 tx=0).stop 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 shutdown_connections 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e806b4d0 0x7f98e806d990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f990815dfe0 0x7f990814d860 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99080d27f0 0x7f990814d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 --2- 172.21.15.72:0/1047792353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99080cf830 0x7f99081f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:58.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 >> 172.21.15.72:0/1047792353 conn(0x7f99080d3cd0 msgr2=0x7f99080cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:58.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 shutdown_connections 2024-03-06T07:29:58.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:58.529+0000 7f990e1ac640 1 -- 172.21.15.72:0/1047792353 wait complete. 2024-03-06T07:29:58.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:29:58.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:29:59.704 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:29:59.745 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:29:59.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.844+0000 7ffad533a640 1 Processor -- start 2024-03-06T07:29:59.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.844+0000 7ffad533a640 1 -- start start 2024-03-06T07:29:59.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffad533a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffad015bdd0 0x7ffad00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffad533a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015fa10 0x7ffad00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffad533a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffad00d16e0 0x7ffad00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffad533a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00cebe0 con 0x7ffad00d16e0 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffad533a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad00ceec0 con 0x7ffad015bdd0 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffad533a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00cf1a0 con 0x7ffad015fa10 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffacf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffad00d16e0 0x7ffad00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffacf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ffad00d16e0 0x7ffad00d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59674/0 (socket says 172.21.15.72:59674) 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffacf7fe640 1 -- 172.21.15.72:0/254208467 learned_addr learned my addr 172.21.15.72:0/254208467 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffac7fff640 1 --2- 172.21.15.72:0/254208467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015fa10 0x7ffad00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.845+0000 7ffaceffd640 1 --2- 172.21.15.72:0/254208467 >> v2:172.21.15.146:3300/0 conn(0x7ffad015bdd0 0x7ffad00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffacf7fe640 1 -- 172.21.15.72:0/254208467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015fa10 msgr2=0x7ffad00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffacf7fe640 1 --2- 172.21.15.72:0/254208467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015fa10 0x7ffad00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffacf7fe640 1 -- 172.21.15.72:0/254208467 >> v2:172.21.15.146:3300/0 conn(0x7ffad015bdd0 msgr2=0x7ffad00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffacf7fe640 1 --2- 172.21.15.72:0/254208467 >> v2:172.21.15.146:3300/0 conn(0x7ffad015bdd0 0x7ffad00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffacf7fe640 1 -- 172.21.15.72:0/254208467 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00cc8e0 con 0x7ffad00d16e0 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffaceffd640 1 --2- 172.21.15.72:0/254208467 >> v2:172.21.15.146:3300/0 conn(0x7ffad015bdd0 0x7ffad00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7ffacf7fe640 1 --2- 172.21.15.72:0/254208467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00d1b30 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ffad006e3c0 tx=0x7ffac802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6adfe901792b252 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7fface7fc640 1 -- 172.21.15.72:0/254208467 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac802c9e0 con 0x7ffad00d16e0 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7fface7fc640 1 -- 172.21.15.72:0/254208467 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac802cb80 con 0x7ffad00d16e0 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.846+0000 7fface7fc640 1 -- 172.21.15.72:0/254208467 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80343f0 con 0x7ffad00d16e0 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 -- 172.21.15.72:0/254208467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 msgr2=0x7ffad00d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 --2- 172.21.15.72:0/254208467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00d1b30 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7ffad006e3c0 tx=0x7ffac802be40 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 -- 172.21.15.72:0/254208467 shutdown_connections 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 --2- 172.21.15.72:0/254208467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00d1b30 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 --2- 172.21.15.72:0/254208467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015fa10 0x7ffad00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 --2- 172.21.15.72:0/254208467 >> v2:172.21.15.146:3300/0 conn(0x7ffad015bdd0 0x7ffad00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 -- 172.21.15.72:0/254208467 >> 172.21.15.72:0/254208467 conn(0x7ffad009a9f0 msgr2=0x7ffad009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:29:59.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 -- 172.21.15.72:0/254208467 shutdown_connections 2024-03-06T07:29:59.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.847+0000 7ffad533a640 1 -- 172.21.15.72:0/254208467 wait complete. 2024-03-06T07:29:59.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.848+0000 7ffad533a640 1 Processor -- start 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.848+0000 7ffad533a640 1 -- start start 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffad533a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffad533a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015bdd0 0x7ffad015c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffad533a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad015fa10 0x7ffad014e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffad533a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00cc5d0 con 0x7ffad00d16e0 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffad533a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad00d2070 con 0x7ffad015fa10 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffad533a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad015c1b0 con 0x7ffad015bdd0 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00b6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59676/0 (socket says 172.21.15.72:59676) 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 -- 172.21.15.72:0/3890794928 learned_addr learned my addr 172.21.15.72:0/3890794928 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 -- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015bdd0 msgr2=0x7ffad015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015bdd0 0x7ffad015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 -- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad015fa10 msgr2=0x7ffad014e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:29:59.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffacf7fe640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad015fa10 0x7ffad014e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad015fa10 0x7ffad014e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.849+0000 7ffaceffd640 1 -- 172.21.15.72:0/3890794928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad006d000 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.850+0000 7ffaceffd640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00b6ef0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7ffac000c820 tx=0x7ffac000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.850+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0015070 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.850+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0007a30 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.850+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac0007de0 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.850+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad006d2e0 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.850+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad006d8a0 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.851+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad015fe90 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.851+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ffac0018070 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.851+0000 7ffac77fe640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.852+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ffac008a150 con 0x7ffad00d16e0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.852+0000 7ffac7fff640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:29:59.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.852+0000 7ffac7fff640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab006b4d0 0x7ffab006d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffad009b3a0 tx=0x7ffab8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:29:59.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:29:59.856+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ffad015fe90 con 0x7ffad00d16e0 2024-03-06T07:30:00.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.071+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffad004f330 con 0x7ffad00d16e0 2024-03-06T07:30:00.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.072+0000 7ffac77fe640 1 -- 172.21.15.72:0/3890794928 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7ffac0058f90 con 0x7ffad00d16e0 2024-03-06T07:30:00.074 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:00.074 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:00.074 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.074+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab006b4d0 msgr2=0x7ffab006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.074+0000 7ffad533a640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab006b4d0 0x7ffab006d990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ffad009b3a0 tx=0x7ffab8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.074+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 msgr2=0x7ffad00b6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.074+0000 7ffad533a640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00b6ef0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7ffac000c820 tx=0x7ffac000cce0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.074+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 shutdown_connections 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ffab006b4d0 0x7ffab006d990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffad015fa10 0x7ffad014e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffad015bdd0 0x7ffad015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 --2- 172.21.15.72:0/3890794928 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad00d16e0 0x7ffad00b6ef0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 >> 172.21.15.72:0/3890794928 conn(0x7ffad009a9f0 msgr2=0x7ffad006f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 shutdown_connections 2024-03-06T07:30:00.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:00.075+0000 7ffad533a640 1 -- 172.21.15.72:0/3890794928 wait complete. 2024-03-06T07:30:00.288 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:00.288 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:01.290 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:01.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.401+0000 7fd2ee1dd640 1 Processor -- start 2024-03-06T07:30:01.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.401+0000 7fd2ee1dd640 1 -- start start 2024-03-06T07:30:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2ee1dd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2e815dfe0 0x7fd2e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2ee1dd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80cf830 0x7fd2e80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2ee1dd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80b3430 con 0x7fd2e80d27f0 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2ee1dd640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e80d2c50 con 0x7fd2e815dfe0 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2ee1dd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80b31e0 con 0x7fd2e80cf830 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e77fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2e815dfe0 0x7fd2e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e7fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd2e80d27f0 0x7fd2e80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:41088/0 (socket says 172.21.15.72:41088) 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e7fff640 1 -- 172.21.15.72:0/2969567133 learned_addr learned my addr 172.21.15.72:0/2969567133 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e6ffd640 1 --2- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80cf830 0x7fd2e80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e67fc640 1 -- 172.21.15.72:0/2969567133 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507923201 0 0) 0x7fd2e80b3430 con 0x7fd2e80d27f0 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e67fc640 1 -- 172.21.15.72:0/2969567133 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4003520 con 0x7fd2e80d27f0 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e77fe640 1 -- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80cf830 msgr2=0x7fd2e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e77fe640 1 --2- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80cf830 0x7fd2e80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e77fe640 1 -- 172.21.15.72:0/2969567133 >> v1:172.21.15.146:6789/0 conn(0x7fd2e80d27f0 legacy=0x7fd2e80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.402+0000 7fd2e77fe640 1 -- 172.21.15.72:0/2969567133 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80b3710 con 0x7fd2e815dfe0 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.403+0000 7fd2e77fe640 1 --2- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80cf2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3025334af43539b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.403+0000 7fd2e67fc640 1 -- 172.21.15.72:0/2969567133 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8035020 con 0x7fd2e815dfe0 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.403+0000 7fd2e67fc640 1 -- 172.21.15.72:0/2969567133 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d8031830 con 0x7fd2e815dfe0 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.403+0000 7fd2e67fc640 1 -- 172.21.15.72:0/2969567133 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8031be0 con 0x7fd2e815dfe0 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e815dfe0 msgr2=0x7fd2e80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80cf2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/2969567133 shutdown_connections 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80cf830 0x7fd2e80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/2969567133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80cf2f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/2969567133 >> 172.21.15.72:0/2969567133 conn(0x7fd2e80d3cd0 msgr2=0x7fd2e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/2969567133 shutdown_connections 2024-03-06T07:30:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.404+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/2969567133 wait complete. 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.405+0000 7fd2ee1dd640 1 Processor -- start 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.406+0000 7fd2ee1dd640 1 -- start start 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2ee1dd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80cf830 0x7fd2e815ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2ee1dd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80d27f0 0x7fd2e80cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2ee1dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80b9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2ee1dd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80d1c70 con 0x7fd2e815dfe0 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2ee1dd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80b3d60 con 0x7fd2e80cf830 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2ee1dd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80d1df0 con 0x7fd2e80d27f0 2024-03-06T07:30:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80b9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80b9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:59684/0 (socket says 172.21.15.72:59684) 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 -- 172.21.15.72:0/45116137 learned_addr learned my addr 172.21.15.72:0/45116137 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e77fe640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80cf830 0x7fd2e815ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e6ffd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80d27f0 0x7fd2e80cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 -- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80d27f0 msgr2=0x7fd2e80cc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80d27f0 0x7fd2e80cc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 -- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80cf830 msgr2=0x7fd2e815ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80cf830 0x7fd2e815ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.407+0000 7fd2e7fff640 1 -- 172.21.15.72:0/45116137 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80cc750 con 0x7fd2e815dfe0 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.408+0000 7fd2e7fff640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80b9490 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000cb40 tx=0x7fd2e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.408+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0017070 con 0x7fd2e815dfe0 2024-03-06T07:30:01.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.408+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e000aca0 con 0x7fd2e815dfe0 2024-03-06T07:30:01.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.408+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e80cca30 con 0x7fd2e815dfe0 2024-03-06T07:30:01.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.408+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0012620 con 0x7fd2e815dfe0 2024-03-06T07:30:01.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.408+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e80ccff0 con 0x7fd2e815dfe0 2024-03-06T07:30:01.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.409+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e804f330 con 0x7fd2e815dfe0 2024-03-06T07:30:01.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.409+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd2e001a070 con 0x7fd2e815dfe0 2024-03-06T07:30:01.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.410+0000 7fd2dffff640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:01.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.410+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd2e0089f00 con 0x7fd2e815dfe0 2024-03-06T07:30:01.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.410+0000 7fd2e77fe640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:01.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.410+0000 7fd2e77fe640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:01.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.414+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd2e0058780 con 0x7fd2e815dfe0 2024-03-06T07:30:01.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.622+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e80b99d0 con 0x7fd2e815dfe0 2024-03-06T07:30:01.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.623+0000 7fd2dffff640 1 -- 172.21.15.72:0/45116137 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fd2e005c820 con 0x7fd2e815dfe0 2024-03-06T07:30:01.625 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:01.625 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:01.625 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:01.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 msgr2=0x7fd2c806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d803c040 comp rx=0 tx=0).stop 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 msgr2=0x7fd2e80b9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80b9490 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000cb40 tx=0x7fd2e000a060 comp rx=0 tx=0).stop 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 shutdown_connections 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd2c806b4d0 0x7fd2c806d990 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd2e815dfe0 0x7fd2e80b9490 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2e80d27f0 0x7fd2e80cc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 --2- 172.21.15.72:0/45116137 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2e80cf830 0x7fd2e815ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 >> 172.21.15.72:0/45116137 conn(0x7fd2e80d3cd0 msgr2=0x7fd2e80b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 shutdown_connections 2024-03-06T07:30:01.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:01.625+0000 7fd2ee1dd640 1 -- 172.21.15.72:0/45116137 wait complete. 2024-03-06T07:30:01.706 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:01.840 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:01.840 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:02.841 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:02.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.940+0000 7f1d5e9e9640 1 Processor -- start 2024-03-06T07:30:02.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.940+0000 7f1d5e9e9640 1 -- start start 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d5e9e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 0x7f1d580b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d5e9e9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d580b3430 con 0x7f1d5815dfe0 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d5e9e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d580b3710 con 0x7f1d580d22b0 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d5e9e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580b31e0 con 0x7f1d580d2710 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d57fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1d5815dfe0 0x7f1d580cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52338/0 (socket says 172.21.15.72:52338) 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d57fff640 1 -- 172.21.15.72:0/551938326 learned_addr learned my addr 172.21.15.72:0/551938326 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d5cf86640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 0x7f1d580b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837125205 0 0) 0x7f1d580b3430 con 0x7f1d5815dfe0 2024-03-06T07:30:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d40003520 con 0x7f1d5815dfe0 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248331477 0 0) 0x7f1d580b3710 con 0x7f1d580d22b0 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d580b3430 con 0x7f1d580d22b0 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372169046 0 0) 0x7f1d40003520 con 0x7f1d5815dfe0 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 msgr2=0x7f1d580b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 0x7f1d580b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 >> v1:172.21.15.146:6789/0 conn(0x7f1d580d22b0 legacy=0x7f1d580b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d580b39f0 con 0x7f1d5815dfe0 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1d48001670 con 0x7f1d5815dfe0 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 >> v1:172.21.15.72:6789/0 conn(0x7f1d5815dfe0 legacy=0x7f1d580cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 0x7f1d40003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d400040f0 0x7f1d40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d40004d40 0x7f1d400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580b31e0 con 0x7f1d40004d40 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4000baa0 con 0x7f1d400040f0 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.941+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4000bca0 con 0x7f1d580d2710 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d5cf86640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d40004d40 0x7f1d400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d57fff640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 0x7f1d40003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d577fe640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d400040f0 0x7f1d40004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d5cf86640 1 -- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 msgr2=0x7f1d40003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d5cf86640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d580d2710 0x7f1d40003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d5cf86640 1 -- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d400040f0 msgr2=0x7f1d40004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d5cf86640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d400040f0 0x7f1d40004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.942+0000 7f1d5cf86640 1 -- 172.21.15.72:0/551938326 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d40003520 con 0x7f1d40004d40 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5cf86640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d40004d40 0x7f1d400051c0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009fc0 tx=0x7f1d40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d36cd8a9d0701ce server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d50039070 con 0x7f1d40004d40 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5002cc20 con 0x7f1d40004d40 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d56ffd640 1 -- 172.21.15.72:0/551938326 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d50009c10 con 0x7f1d40004d40 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/551938326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d40004d40 msgr2=0x7f1d400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/551938326 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d40004d40 0x7f1d400051c0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009fc0 tx=0x7f1d40005830 comp rx=0 tx=0).stop 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5cf86640 1 -- 172.21.15.72:0/551938326 reap_dead start 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/551938326 shutdown_connections 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/551938326 >> 172.21.15.72:0/551938326 conn(0x7f1d580d3cd0 msgr2=0x7f1d580d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/551938326 shutdown_connections 2024-03-06T07:30:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.943+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/551938326 wait complete. 2024-03-06T07:30:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.944+0000 7f1d5e9e9640 1 Processor -- start 2024-03-06T07:30:02.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 -- start start 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d40004d40 0x7f1d580b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 0x7f1d58151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5815ed20 0x7f1d5815f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580cfa00 con 0x7f1d58161010 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580b3d60 con 0x7f1d5815ed20 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d5e9e9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d580d1c70 con 0x7f1d40004d40 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 0x7f1d58151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 0x7f1d58151560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38004/0 (socket says 172.21.15.72:38004) 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.945+0000 7f1d577fe640 1 -- 172.21.15.72:0/2298973112 learned_addr learned my addr 172.21.15.72:0/2298973112 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:02.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d57fff640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d40004d40 0x7f1d580b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d5cf86640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5815ed20 0x7f1d5815f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d577fe640 1 -- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d40004d40 msgr2=0x7f1d580b8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d577fe640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d40004d40 0x7f1d580b8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d577fe640 1 -- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5815ed20 msgr2=0x7f1d5815f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d577fe640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5815ed20 0x7f1d5815f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d577fe640 1 -- 172.21.15.72:0/2298973112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d580b9100 con 0x7f1d58161010 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d577fe640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 0x7f1d58151560 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00f770 tx=0x7f1d4c00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c00fe80 con 0x7f1d58161010 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4c018070 con 0x7f1d58161010 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c00d730 con 0x7f1d58161010 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d580b93a0 con 0x7f1d58161010 2024-03-06T07:30:02.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.946+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d580b9960 con 0x7f1d58161010 2024-03-06T07:30:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.947+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d580b9ba0 con 0x7f1d58161010 2024-03-06T07:30:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.948+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1d4c00d8d0 con 0x7f1d58161010 2024-03-06T07:30:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.948+0000 7f1d54ff9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d4406b500 0x7f1d4406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.948+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1d4c08a060 con 0x7f1d58161010 2024-03-06T07:30:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.948+0000 7f1d57fff640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d4406b500 0x7f1d4406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:02.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.949+0000 7f1d57fff640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d4406b500 0x7f1d4406d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1d480011e0 tx=0x7f1d48001e50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:02.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:02.957+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1d4c0588e0 con 0x7f1d58161010 2024-03-06T07:30:03.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.303+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d5804f330 con 0x7f1d58161010 2024-03-06T07:30:03.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.303+0000 7f1d54ff9640 1 -- 172.21.15.72:0/2298973112 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f1d4c05c980 con 0x7f1d58161010 2024-03-06T07:30:03.306 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:03.306 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:03.306 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d4406b500 msgr2=0x7f1d4406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d4406b500 0x7f1d4406d9c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f1d480011e0 tx=0x7f1d48001e50 comp rx=0 tx=0).stop 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 msgr2=0x7f1d58151560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 0x7f1d58151560 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c00f770 tx=0x7f1d4c00fc30 comp rx=0 tx=0).stop 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 shutdown_connections 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1d4406b500 0x7f1d4406d9c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5815ed20 0x7f1d5815f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d58161010 0x7f1d58151560 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 --2- 172.21.15.72:0/2298973112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d40004d40 0x7f1d580b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:03.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.305+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 >> 172.21.15.72:0/2298973112 conn(0x7f1d580d3cd0 msgr2=0x7f1d5806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:03.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.306+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 shutdown_connections 2024-03-06T07:30:03.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:03.306+0000 7f1d5e9e9640 1 -- 172.21.15.72:0/2298973112 wait complete. 2024-03-06T07:30:03.520 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:03.520 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:03.708 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:04.521 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:04.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.621+0000 7fd6174df640 1 Processor -- start 2024-03-06T07:30:04.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.621+0000 7fd6174df640 1 -- start start 2024-03-06T07:30:04.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.622+0000 7fd6174df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd61015dfe0 0x7fd6100cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.622+0000 7fd6174df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd6100cf830 0x7fd6100d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.622+0000 7fd6174df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6100b3430 con 0x7fd6100d27f0 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.622+0000 7fd6174df640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6100d2c50 con 0x7fd6100cf830 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.622+0000 7fd6174df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100b31e0 con 0x7fd61015dfe0 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd614a7a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd6100cf830 0x7fd6100d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd61527b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd61015dfe0 0x7fd6100cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd615a7c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd6100d27f0 0x7fd6100d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:34500/0 (socket says 172.21.15.72:34500) 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd615a7c640 1 -- 172.21.15.72:0/3792955342 learned_addr learned my addr 172.21.15.72:0/3792955342 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd61527b640 1 -- 172.21.15.72:0/3792955342 >> v2:172.21.15.146:3300/0 conn(0x7fd6100cf830 msgr2=0x7fd6100d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd61527b640 1 --2- 172.21.15.72:0/3792955342 >> v2:172.21.15.146:3300/0 conn(0x7fd6100cf830 0x7fd6100d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd61527b640 1 -- 172.21.15.72:0/3792955342 >> v1:172.21.15.146:6789/0 conn(0x7fd6100d27f0 legacy=0x7fd6100d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd61527b640 1 -- 172.21.15.72:0/3792955342 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100b3710 con 0x7fd61015dfe0 2024-03-06T07:30:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd607fff640 1 -- 172.21.15.72:0/3792955342 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335857282 0 0) 0x7fd6100b3430 con 0x7fd6100d27f0 2024-03-06T07:30:04.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.623+0000 7fd614a7a640 1 --2- 172.21.15.72:0/3792955342 >> v2:172.21.15.146:3300/0 conn(0x7fd6100cf830 0x7fd6100d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:04.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.624+0000 7fd61527b640 1 --2- 172.21.15.72:0/3792955342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd61015dfe0 0x7fd6100cf2f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0097a0 tx=0x7fd60c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bad7361422e4ec45 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.624+0000 7fd607fff640 1 -- 172.21.15.72:0/3792955342 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c035020 con 0x7fd61015dfe0 2024-03-06T07:30:04.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.624+0000 7fd607fff640 1 -- 172.21.15.72:0/3792955342 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60c031830 con 0x7fd61015dfe0 2024-03-06T07:30:04.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.624+0000 7fd607fff640 1 -- 172.21.15.72:0/3792955342 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c031c00 con 0x7fd61015dfe0 2024-03-06T07:30:04.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.624+0000 7fd6174df640 1 -- 172.21.15.72:0/3792955342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd61015dfe0 msgr2=0x7fd6100cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 --2- 172.21.15.72:0/3792955342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd61015dfe0 0x7fd6100cf2f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0097a0 tx=0x7fd60c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 -- 172.21.15.72:0/3792955342 shutdown_connections 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 --2- 172.21.15.72:0/3792955342 >> v2:172.21.15.146:3300/0 conn(0x7fd6100cf830 0x7fd6100d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 --2- 172.21.15.72:0/3792955342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd61015dfe0 0x7fd6100cf2f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 -- 172.21.15.72:0/3792955342 >> 172.21.15.72:0/3792955342 conn(0x7fd6100d3cd0 msgr2=0x7fd6100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 -- 172.21.15.72:0/3792955342 shutdown_connections 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.625+0000 7fd6174df640 1 -- 172.21.15.72:0/3792955342 wait complete. 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.626+0000 7fd6174df640 1 Processor -- start 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.626+0000 7fd6174df640 1 -- start start 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.626+0000 7fd6174df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 0x7fd6100a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.626+0000 7fd6174df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6100d27f0 0x7fd6100b8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.626+0000 7fd6174df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61015dfe0 0x7fd6101603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd6174df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100d1c70 con 0x7fd6100cf830 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd6174df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100b3d60 con 0x7fd61015dfe0 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd6174df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6100d1df0 con 0x7fd6100d27f0 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 0x7fd6100a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 0x7fd6100a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38010/0 (socket says 172.21.15.72:38010) 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 -- 172.21.15.72:0/4142931077 learned_addr learned my addr 172.21.15.72:0/4142931077 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd615a7c640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61015dfe0 0x7fd6101603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd614a7a640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6100d27f0 0x7fd6100b8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 -- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6100d27f0 msgr2=0x7fd6100b8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6100d27f0 0x7fd6100b8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 -- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61015dfe0 msgr2=0x7fd6101603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61015dfe0 0x7fd6101603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 -- 172.21.15.72:0/4142931077 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6100cbcb0 con 0x7fd6100cf830 2024-03-06T07:30:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd61527b640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 0x7fd6100a1210 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000c00 tx=0x7fd60c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c035040 con 0x7fd6100cf830 2024-03-06T07:30:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd60c004250 con 0x7fd6100cf830 2024-03-06T07:30:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6100cbf90 con 0x7fd6100cf830 2024-03-06T07:30:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.627+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60c03a640 con 0x7fd6100cf830 2024-03-06T07:30:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.628+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6100cc830 con 0x7fd6100cf830 2024-03-06T07:30:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.628+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6100cc380 con 0x7fd6100cf830 2024-03-06T07:30:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.629+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd60c038020 con 0x7fd6100cf830 2024-03-06T07:30:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.629+0000 7fd605ffb640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5f406b4d0 0x7fd5f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.629+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd60c0abe00 con 0x7fd6100cf830 2024-03-06T07:30:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.629+0000 7fd614a7a640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5f406b4d0 0x7fd5f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.630+0000 7fd614a7a640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5f406b4d0 0x7fd5f406d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd610067210 tx=0x7fd600009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:04.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.635+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd60c07a600 con 0x7fd6100cf830 2024-03-06T07:30:04.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.851+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd61008d3c0 con 0x7fd6100cf830 2024-03-06T07:30:04.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.852+0000 7fd605ffb640 1 -- 172.21.15.72:0/4142931077 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fd61008d3c0 con 0x7fd6100cf830 2024-03-06T07:30:04.856 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:04.856 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:04.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5f406b4d0 msgr2=0x7fd5f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5f406b4d0 0x7fd5f406d990 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd610067210 tx=0x7fd600009190 comp rx=0 tx=0).stop 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 msgr2=0x7fd6100a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 0x7fd6100a1210 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fd60c000c00 tx=0x7fd60c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 shutdown_connections 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd5f406b4d0 0x7fd5f406d990 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd61015dfe0 0x7fd6101603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6100d27f0 0x7fd6100b8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 --2- 172.21.15.72:0/4142931077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd6100cf830 0x7fd6100a1210 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 >> 172.21.15.72:0/4142931077 conn(0x7fd6100d3cd0 msgr2=0x7fd61006d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 shutdown_connections 2024-03-06T07:30:04.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:04.854+0000 7fd6174df640 1 -- 172.21.15.72:0/4142931077 wait complete. 2024-03-06T07:30:05.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:05.072 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:05.710 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:06.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:06.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.172+0000 7f257bb63640 1 Processor -- start 2024-03-06T07:30:06.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.172+0000 7f257bb63640 1 -- start start 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f257bb63640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f257415dfe0 0x7f25740cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f257bb63640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740b3430 con 0x7f25740d29c0 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f257bb63640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740b3710 con 0x7f25740cf830 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f257bb63640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25740b31e0 con 0x7f257415dfe0 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f257a100640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f25740d29c0 0x7f25740b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52344/0 (socket says 172.21.15.72:52344) 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f257a100640 1 -- 172.21.15.72:0/2039069207 learned_addr learned my addr 172.21.15.72:0/2039069207 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.173+0000 7f25798ff640 1 --2- 172.21.15.72:0/2039069207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f257415dfe0 0x7f25740cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553053819 0 0) 0x7f25740b3430 con 0x7f25740d29c0 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2560003520 con 0x7f25740d29c0 2024-03-06T07:30:06.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483289693 0 0) 0x7f25740b3710 con 0x7f25740cf830 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25740b3430 con 0x7f25740cf830 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25798ff640 1 -- 172.21.15.72:0/2039069207 >> v1:172.21.15.146:6789/0 conn(0x7f25740cf830 legacy=0x7f25740d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 128612398 0 0) 0x7f2560003520 con 0x7f25740d29c0 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25798ff640 1 -- 172.21.15.72:0/2039069207 >> v1:172.21.15.72:6789/0 conn(0x7f25740d29c0 legacy=0x7f25740b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.174+0000 7f25798ff640 1 -- 172.21.15.72:0/2039069207 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740b39f0 con 0x7f257415dfe0 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f25798ff640 1 --2- 172.21.15.72:0/2039069207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f257415dfe0 0x7f25740cf2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25700097d0 tx=0x7f257002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b660a86c2e394c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2570037070 con 0x7f257415dfe0 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2570030510 con 0x7f257415dfe0 2024-03-06T07:30:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f25788fd640 1 -- 172.21.15.72:0/2039069207 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0019f0 con 0x7f257415dfe0 2024-03-06T07:30:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f257bb63640 1 -- 172.21.15.72:0/2039069207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f257415dfe0 msgr2=0x7f25740cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f257bb63640 1 --2- 172.21.15.72:0/2039069207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f257415dfe0 0x7f25740cf2f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25700097d0 tx=0x7f257002b800 comp rx=0 tx=0).stop 2024-03-06T07:30:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f257bb63640 1 -- 172.21.15.72:0/2039069207 shutdown_connections 2024-03-06T07:30:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f257bb63640 1 --2- 172.21.15.72:0/2039069207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f257415dfe0 0x7f25740cf2f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.175+0000 7f257bb63640 1 -- 172.21.15.72:0/2039069207 >> 172.21.15.72:0/2039069207 conn(0x7f25740d3cd0 msgr2=0x7f25740d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.176+0000 7f257bb63640 1 -- 172.21.15.72:0/2039069207 shutdown_connections 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.176+0000 7f257bb63640 1 -- 172.21.15.72:0/2039069207 wait complete. 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 Processor -- start 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 -- start start 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cf830 0x7f2574153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25740d29c0 0x7f25740cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 0x7f257409f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f257415e3c0 con 0x7f257415dfe0 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25740b3d60 con 0x7f25740d29c0 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257bb63640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f257415e540 con 0x7f25740cf830 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.177+0000 7f257a100640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 0x7f257409f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 0x7f257409f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38026/0 (socket says 172.21.15.72:38026) 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 -- 172.21.15.72:0/1716110175 learned_addr learned my addr 172.21.15.72:0/1716110175 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f25798ff640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cf830 0x7f2574153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f25790fe640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25740d29c0 0x7f25740cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 -- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cf830 msgr2=0x7f2574153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cf830 0x7f2574153f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 -- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25740d29c0 msgr2=0x7f25740cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25740d29c0 0x7f25740cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 -- 172.21.15.72:0/1716110175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740cbba0 con 0x7f257415dfe0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257a100640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 0x7f257409f2e0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f256c001f30 tx=0x7f256c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c017070 con 0x7f257415dfe0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c012400 con 0x7f257415dfe0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25740cbe70 con 0x7f257415dfe0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c0127f0 con 0x7f257415dfe0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.178+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25740cc430 con 0x7f257415dfe0 2024-03-06T07:30:06.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.179+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25740b3710 con 0x7f257415dfe0 2024-03-06T07:30:06.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.180+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f256c01a070 con 0x7f257415dfe0 2024-03-06T07:30:06.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.180+0000 7f256a7fc640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f255806b4d0 0x7f255806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:06.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.180+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f256c08a1d0 con 0x7f257415dfe0 2024-03-06T07:30:06.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.180+0000 7f25798ff640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f255806b4d0 0x7f255806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:06.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.180+0000 7f25798ff640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f255806b4d0 0x7f255806d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2570034000 tx=0x7f2570008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:06.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.184+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25740b3710 con 0x7f257415dfe0 2024-03-06T07:30:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.398+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25740cf0b0 con 0x7f257415dfe0 2024-03-06T07:30:06.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.398+0000 7f256a7fc640 1 -- 172.21.15.72:0/1716110175 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f256c058940 con 0x7f257415dfe0 2024-03-06T07:30:06.400 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:06.400 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:06.401 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f255806b4d0 msgr2=0x7f255806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f255806b4d0 0x7f255806d990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2570034000 tx=0x7f2570008030 comp rx=0 tx=0).stop 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 msgr2=0x7f257409f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 0x7f257409f2e0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f256c001f30 tx=0x7f256c00a430 comp rx=0 tx=0).stop 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 shutdown_connections 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f255806b4d0 0x7f255806d990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f257415dfe0 0x7f257409f2e0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25740d29c0 0x7f25740cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 --2- 172.21.15.72:0/1716110175 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25740cf830 0x7f2574153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 >> 172.21.15.72:0/1716110175 conn(0x7f25740d3cd0 msgr2=0x7f257406cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 shutdown_connections 2024-03-06T07:30:06.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:06.400+0000 7f257bb63640 1 -- 172.21.15.72:0/1716110175 wait complete. 2024-03-06T07:30:06.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:06.614 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:07.615 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:07.712 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:07.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.714+0000 7fe8f918c640 1 Processor -- start 2024-03-06T07:30:07.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.714+0000 7fe8f918c640 1 -- start start 2024-03-06T07:30:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f918c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8f40cfff0 0x7fe8f40d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f918c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 0x7fe8f40ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f918c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8f4161980 con 0x7fe8f415bb20 2024-03-06T07:30:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f918c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f40d2500 con 0x7fe8f40cfff0 2024-03-06T07:30:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f918c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4161c60 con 0x7fe8f40cc8b0 2024-03-06T07:30:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f2d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe8f415bb20 0x7fe8f40d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52360/0 (socket says 172.21.15.72:52360) 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.715+0000 7fe8f2d9d640 1 -- 172.21.15.72:0/131799777 learned_addr learned my addr 172.21.15.72:0/131799777 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950095571 0 0) 0x7fe8f4161980 con 0x7fe8f415bb20 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 --2- 172.21.15.72:0/131799777 >> v2:172.21.15.146:3300/0 conn(0x7fe8f40cfff0 0x7fe8f40d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8dc003520 con 0x7fe8f415bb20 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f359e640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 0x7fe8f40ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 332470519 0 0) 0x7fe8dc003520 con 0x7fe8f415bb20 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 msgr2=0x7fe8f40ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 0x7fe8f40ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 >> v2:172.21.15.146:3300/0 conn(0x7fe8f40cfff0 msgr2=0x7fe8f40d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 --2- 172.21.15.72:0/131799777 >> v2:172.21.15.146:3300/0 conn(0x7fe8f40cfff0 0x7fe8f40d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4161f40 con 0x7fe8f415bb20 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe8e4001670 con 0x7fe8f415bb20 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 >> v1:172.21.15.72:6789/0 conn(0x7fe8f415bb20 legacy=0x7fe8f40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc0039b0 0x7fe8dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 0x7fe8dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40d2500 con 0x7fe8dc004800 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4161c60 con 0x7fe8dc0039b0 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc00b9c0 con 0x7fe8f40cc8b0 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f2d9d640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc0039b0 0x7fe8dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f359e640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 0x7fe8dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 -- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 msgr2=0x7fe8dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cc8b0 0x7fe8dc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 -- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc0039b0 msgr2=0x7fe8dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc0039b0 0x7fe8dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f259c640 1 -- 172.21.15.72:0/131799777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc003520 con 0x7fe8dc004800 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.716+0000 7fe8f2d9d640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc0039b0 0x7fe8dc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.717+0000 7fe8f259c640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8dc004c80 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009ac0 tx=0x7fe8dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09fa8c2c1865a53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.717+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8039070 con 0x7fe8dc004800 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.717+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e802c710 con 0x7fe8dc004800 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.717+0000 7fe8f1d9b640 1 -- 172.21.15.72:0/131799777 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8009690 con 0x7fe8dc004800 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f918c640 1 -- 172.21.15.72:0/131799777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 msgr2=0x7fe8dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f918c640 1 --2- 172.21.15.72:0/131799777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8dc004c80 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8009ac0 tx=0x7fe8dc005830 comp rx=0 tx=0).stop 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f359e640 1 -- 172.21.15.72:0/131799777 reap_dead start 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f918c640 1 -- 172.21.15.72:0/131799777 shutdown_connections 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f918c640 1 -- 172.21.15.72:0/131799777 >> 172.21.15.72:0/131799777 conn(0x7fe8f409a9f0 msgr2=0x7fe8f409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f918c640 1 -- 172.21.15.72:0/131799777 shutdown_connections 2024-03-06T07:30:07.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.718+0000 7fe8f918c640 1 -- 172.21.15.72:0/131799777 wait complete. 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 Processor -- start 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 -- start start 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8f4155120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cb760 0x7fe8f41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40c9510 0x7fe8f40c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40d1980 con 0x7fe8dc004800 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f4162500 con 0x7fe8f40c9510 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f918c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40d1b00 con 0x7fe8f40cb760 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8f4155120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8f4155120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38054/0 (socket says 172.21.15.72:38054) 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.720+0000 7fe8f2d9d640 1 -- 172.21.15.72:0/1905304479 learned_addr learned my addr 172.21.15.72:0/1905304479 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:07.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f359e640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40c9510 0x7fe8f40c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f259c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cb760 0x7fe8f41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f2d9d640 1 -- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cb760 msgr2=0x7fe8f41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f2d9d640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cb760 0x7fe8f41598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f2d9d640 1 -- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40c9510 msgr2=0x7fe8f40c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f2d9d640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40c9510 0x7fe8f40c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f2d9d640 1 -- 172.21.15.72:0/1905304479 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f415a240 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f2d9d640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8f4155120 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400e690 tx=0x7fe8e400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e400c6c0 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8e4018070 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f415a520 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8e400caa0 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.721+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f4155660 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.722+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f404f330 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.722+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8e400cc40 con 0x7fe8dc004800 2024-03-06T07:30:07.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.723+0000 7fe8db7fe640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8e006b500 0x7fe8e006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:07.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.723+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe8e408a010 con 0x7fe8dc004800 2024-03-06T07:30:07.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.723+0000 7fe8f259c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8e006b500 0x7fe8e006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:07.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.723+0000 7fe8f259c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8e006b500 0x7fe8e006d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8036000 tx=0x7fe8e8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:07.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.727+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8e4058ee0 con 0x7fe8dc004800 2024-03-06T07:30:07.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.935+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8f40b1820 con 0x7fe8dc004800 2024-03-06T07:30:07.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.936+0000 7fe8db7fe640 1 -- 172.21.15.72:0/1905304479 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fe8e40587e0 con 0x7fe8dc004800 2024-03-06T07:30:07.939 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:07.939 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:07.939 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:07.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8e006b500 msgr2=0x7fe8e006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8e006b500 0x7fe8e006d9c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fe8e8036000 tx=0x7fe8e8030000 comp rx=0 tx=0).stop 2024-03-06T07:30:07.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 msgr2=0x7fe8f4155120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:07.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8f4155120 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fe8e400e690 tx=0x7fe8e400eb50 comp rx=0 tx=0).stop 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 shutdown_connections 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8e006b500 0x7fe8e006d9c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8f40c9510 0x7fe8f40c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8f40cb760 0x7fe8f41598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 --2- 172.21.15.72:0/1905304479 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8dc004800 0x7fe8f4155120 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 >> 172.21.15.72:0/1905304479 conn(0x7fe8f409a9f0 msgr2=0x7fe8f40cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 shutdown_connections 2024-03-06T07:30:07.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:07.938+0000 7fe8f918c640 1 -- 172.21.15.72:0/1905304479 wait complete. 2024-03-06T07:30:08.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:08.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:09.154 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:09.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 Processor -- start 2024-03-06T07:30:09.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 -- start start 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b3415dfe0 0x7f2b340cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2b340cf830 0x7f2b340d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b340b3430 con 0x7f2b340d27f0 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b340b31e0 con 0x7f2b340cf830 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.254+0000 7f2b3aa8a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b340b3d60 con 0x7f2b3415dfe0 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2b340cf830 0x7f2b340d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2b340cf830 0x7f2b340d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38056/0 (socket says 172.21.15.72:38056) 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 -- 172.21.15.72:0/1530935842 learned_addr learned my addr 172.21.15.72:0/1530935842 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b38826640 1 --2- 172.21.15.72:0/1530935842 >> v2:172.21.15.146:3300/0 conn(0x7f2b3415dfe0 0x7f2b340cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 -- 172.21.15.72:0/1530935842 >> v2:172.21.15.146:3300/0 conn(0x7f2b3415dfe0 msgr2=0x7f2b340cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:09.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 --2- 172.21.15.72:0/1530935842 >> v2:172.21.15.146:3300/0 conn(0x7f2b3415dfe0 0x7f2b340cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 -- 172.21.15.72:0/1530935842 >> v1:172.21.15.146:6789/0 conn(0x7f2b340d27f0 legacy=0x7f2b340d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.255+0000 7f2b33fff640 1 -- 172.21.15.72:0/1530935842 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b340b3710 con 0x7f2b340cf830 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.256+0000 7f2b33fff640 1 --2- 172.21.15.72:0/1530935842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b340cf830 0x7f2b340d22b0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009870 tx=0x7f2b2802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba597676257dae18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.256+0000 7f2b337fe640 1 -- 172.21.15.72:0/1530935842 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28034070 con 0x7f2b340cf830 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.256+0000 7f2b337fe640 1 -- 172.21.15.72:0/1530935842 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b28030860 con 0x7f2b340cf830 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.256+0000 7f2b337fe640 1 -- 172.21.15.72:0/1530935842 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28030c30 con 0x7f2b340cf830 2024-03-06T07:30:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1530935842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b340cf830 msgr2=0x7f2b340d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1530935842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b340cf830 0x7f2b340d22b0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009870 tx=0x7f2b2802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1530935842 shutdown_connections 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1530935842 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b340cf830 0x7f2b340d22b0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1530935842 >> v2:172.21.15.146:3300/0 conn(0x7f2b3415dfe0 0x7f2b340cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1530935842 >> 172.21.15.72:0/1530935842 conn(0x7f2b340d3cd0 msgr2=0x7f2b340d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1530935842 shutdown_connections 2024-03-06T07:30:09.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.257+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1530935842 wait complete. 2024-03-06T07:30:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.259+0000 7f2b3aa8a640 1 Processor -- start 2024-03-06T07:30:09.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.259+0000 7f2b3aa8a640 1 -- start start 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b3aa8a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b340cf830 0x7f2b340ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b3aa8a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b340d27f0 0x7f2b340b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b3aa8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 0x7f2b3414ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b3aa8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b340d1df0 con 0x7f2b3415dfe0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b3aa8a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b340d1c70 con 0x7f2b340d27f0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b3aa8a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3415e3c0 con 0x7f2b340cf830 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.260+0000 7f2b39027640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 0x7f2b3414ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 0x7f2b3414ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38058/0 (socket says 172.21.15.72:38058) 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 -- 172.21.15.72:0/1332948576 learned_addr learned my addr 172.21.15.72:0/1332948576 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b33fff640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b340d27f0 0x7f2b340b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b38826640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b340cf830 0x7f2b340ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 -- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b340cf830 msgr2=0x7f2b340ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b340cf830 0x7f2b340ca1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 -- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b340d27f0 msgr2=0x7f2b340b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b340d27f0 0x7f2b340b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 -- 172.21.15.72:0/1332948576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b340b9150 con 0x7f2b3415dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b33fff640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b340d27f0 0x7f2b340b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b39027640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 0x7f2b3414ccf0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f2b24001eb0 tx=0x7f2b2400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24017070 con 0x7f2b3415dfe0 2024-03-06T07:30:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b240052d0 con 0x7f2b3415dfe0 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24003810 con 0x7f2b3415dfe0 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b340b9430 con 0x7f2b3415dfe0 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.261+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3415f8e0 con 0x7f2b3415dfe0 2024-03-06T07:30:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.262+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3404f330 con 0x7f2b3415dfe0 2024-03-06T07:30:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.263+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b2401a070 con 0x7f2b3415dfe0 2024-03-06T07:30:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.263+0000 7f2b317fa640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b2006b4d0 0x7f2b2006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.263+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b24089550 con 0x7f2b3415dfe0 2024-03-06T07:30:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.263+0000 7f2b38826640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b2006b4d0 0x7f2b2006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.264+0000 7f2b38826640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b2006b4d0 0x7f2b2006d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009f30 tx=0x7f2b2c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.268+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b24058420 con 0x7f2b3415dfe0 2024-03-06T07:30:09.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.588+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b340b5310 con 0x7f2b3415dfe0 2024-03-06T07:30:09.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.588+0000 7f2b317fa640 1 -- 172.21.15.72:0/1332948576 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2b24057d20 con 0x7f2b3415dfe0 2024-03-06T07:30:09.590 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:09.591 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:09.591 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b2006b4d0 msgr2=0x7f2b2006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b2006b4d0 0x7f2b2006d990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009f30 tx=0x7f2b2c00b040 comp rx=0 tx=0).stop 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 msgr2=0x7f2b3414ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 0x7f2b3414ccf0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f2b24001eb0 tx=0x7f2b2400c890 comp rx=0 tx=0).stop 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 shutdown_connections 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b2006b4d0 0x7f2b2006d990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b3415dfe0 0x7f2b3414ccf0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b340d27f0 0x7f2b340b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 --2- 172.21.15.72:0/1332948576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b340cf830 0x7f2b340ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 >> 172.21.15.72:0/1332948576 conn(0x7f2b340d3cd0 msgr2=0x7f2b340b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:09.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 shutdown_connections 2024-03-06T07:30:09.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:09.590+0000 7f2b3aa8a640 1 -- 172.21.15.72:0/1332948576 wait complete. 2024-03-06T07:30:09.713 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:09.805 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:09.805 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:10.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:10.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.905+0000 7f98fe284640 1 Processor -- start 2024-03-06T07:30:10.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fe284640 1 -- start start 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fe284640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fe284640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98f80d29c0 0x7f98f80b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fe284640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f80b3430 con 0x7f98f80cf830 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fe284640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f80b31e0 con 0x7f98f80d29c0 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fe284640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f80b3d60 con 0x7f98f815dfe0 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fc821640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98f80d29c0 0x7f98f80b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fc821640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f98f80d29c0 0x7f98f80b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38072/0 (socket says 172.21.15.72:38072) 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.906+0000 7f98fc821640 1 -- 172.21.15.72:0/3572597289 learned_addr learned my addr 172.21.15.72:0/3572597289 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.907+0000 7f98f7fff640 1 --2- 172.21.15.72:0/3572597289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.907+0000 7f98fc821640 1 -- 172.21.15.72:0/3572597289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 msgr2=0x7f98f80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.907+0000 7f98fc821640 1 --2- 172.21.15.72:0/3572597289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.907+0000 7f98fc821640 1 -- 172.21.15.72:0/3572597289 >> v1:172.21.15.146:6789/0 conn(0x7f98f80cf830 legacy=0x7f98f80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.907+0000 7f98fc821640 1 -- 172.21.15.72:0/3572597289 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f80b3710 con 0x7f98f80d29c0 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.907+0000 7f98fc821640 1 --2- 172.21.15.72:0/3572597289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d29c0 0x7f98f80b41f0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009e50 tx=0x7f98f002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01ab10bf052c13d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98f6ffd640 1 -- 172.21.15.72:0/3572597289 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f002cb00 con 0x7f98f80d29c0 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98f6ffd640 1 -- 172.21.15.72:0/3572597289 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f002cca0 con 0x7f98f80d29c0 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98f6ffd640 1 -- 172.21.15.72:0/3572597289 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0034610 con 0x7f98f80d29c0 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 -- 172.21.15.72:0/3572597289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d29c0 msgr2=0x7f98f80b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 --2- 172.21.15.72:0/3572597289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d29c0 0x7f98f80b41f0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009e50 tx=0x7f98f002bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 -- 172.21.15.72:0/3572597289 shutdown_connections 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 --2- 172.21.15.72:0/3572597289 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80d29c0 0x7f98f80b41f0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 --2- 172.21.15.72:0/3572597289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 -- 172.21.15.72:0/3572597289 >> 172.21.15.72:0/3572597289 conn(0x7f98f80d3cd0 msgr2=0x7f98f80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 -- 172.21.15.72:0/3572597289 shutdown_connections 2024-03-06T07:30:10.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.908+0000 7f98fe284640 1 -- 172.21.15.72:0/3572597289 wait complete. 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.909+0000 7f98fe284640 1 Processor -- start 2024-03-06T07:30:10.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 -- start start 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 0x7f98f80a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d29c0 0x7f98f80cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f815be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f80d1df0 con 0x7f98f80cf830 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f80d1c70 con 0x7f98f80d29c0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98fe284640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f815e3c0 con 0x7f98f815dfe0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 0x7f98f80a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 0x7f98f80a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38086/0 (socket says 172.21.15.72:38086) 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.910+0000 7f98f7fff640 1 -- 172.21.15.72:0/1825643952 learned_addr learned my addr 172.21.15.72:0/1825643952 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f77fe640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d29c0 0x7f98f80cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98fc821640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f815be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f7fff640 1 -- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 msgr2=0x7f98f815be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f7fff640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f815be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f7fff640 1 -- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d29c0 msgr2=0x7f98f80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f7fff640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d29c0 0x7f98f80cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f7fff640 1 -- 172.21.15.72:0/1825643952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f80caf80 con 0x7f98f80cf830 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f7fff640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 0x7f98f80a1690 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f98e800e870 tx=0x7f98e800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8017070 con 0x7f98f80cf830 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e800c940 con 0x7f98f80cf830 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f80cb260 con 0x7f98f80cf830 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e800cd30 con 0x7f98f80cf830 2024-03-06T07:30:10.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.911+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f80cd7a0 con 0x7f98f80cf830 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.912+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f804f330 con 0x7f98f80cf830 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.912+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f98e801a070 con 0x7f98f80cf830 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.913+0000 7f98f4ff9640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e406b4d0 0x7f98e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.913+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f98e808a430 con 0x7f98f80cf830 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.913+0000 7f98f77fe640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e406b4d0 0x7f98e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:10.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.913+0000 7f98f77fe640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e406b4d0 0x7f98e406d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f98f81f72e0 tx=0x7f98ec009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:10.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:10.917+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f98e8059300 con 0x7f98f80cf830 2024-03-06T07:30:11.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.198+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98f80ca380 con 0x7f98f80cf830 2024-03-06T07:30:11.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.198+0000 7f98f4ff9640 1 -- 172.21.15.72:0/1825643952 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f98e8058c00 con 0x7f98f80cf830 2024-03-06T07:30:11.201 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:11.201 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:11.201 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:11.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e406b4d0 msgr2=0x7f98e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e406b4d0 0x7f98e406d990 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f98f81f72e0 tx=0x7f98ec009040 comp rx=0 tx=0).stop 2024-03-06T07:30:11.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 msgr2=0x7f98f80a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 0x7f98f80a1690 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f98e800e870 tx=0x7f98e800ed30 comp rx=0 tx=0).stop 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 shutdown_connections 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f98e406b4d0 0x7f98e406d990 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98f815dfe0 0x7f98f815be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98f80d29c0 0x7f98f80cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 --2- 172.21.15.72:0/1825643952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f98f80cf830 0x7f98f80a1690 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 >> 172.21.15.72:0/1825643952 conn(0x7f98f80d3cd0 msgr2=0x7f98f80b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 shutdown_connections 2024-03-06T07:30:11.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:11.200+0000 7f98fe284640 1 -- 172.21.15.72:0/1825643952 wait complete. 2024-03-06T07:30:11.418 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:11.418 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:11.716 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:12.419 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:12.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.518+0000 7fb3f298a640 1 Processor -- start 2024-03-06T07:30:12.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.518+0000 7fb3f298a640 1 -- start start 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3f298a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec06b220 0x7fb3ec15e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3f298a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec15ea30 con 0x7fb3ec06b600 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3f298a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec15ed10 con 0x7fb3ec0ce920 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3f298a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec161b70 con 0x7fb3ec06b220 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3f0f27640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb3ec06b600 0x7fb3ec162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60590/0 (socket says 172.21.15.72:60590) 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3f0f27640 1 -- 172.21.15.72:0/2354839234 learned_addr learned my addr 172.21.15.72:0/2354839234 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3ebfff640 1 --2- 172.21.15.72:0/2354839234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec06b220 0x7fb3ec15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2857613262 0 0) 0x7fb3ec15ea30 con 0x7fb3ec06b600 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d8003520 con 0x7fb3ec06b600 2024-03-06T07:30:12.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351050387 0 0) 0x7fb3ec15ed10 con 0x7fb3ec0ce920 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec15ea30 con 0x7fb3ec0ce920 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896768690 0 0) 0x7fb3d8003520 con 0x7fb3ec06b600 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3ebfff640 1 -- 172.21.15.72:0/2354839234 >> v1:172.21.15.146:6789/0 conn(0x7fb3ec0ce920 legacy=0x7fb3ec071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3ebfff640 1 -- 172.21.15.72:0/2354839234 >> v1:172.21.15.72:6789/0 conn(0x7fb3ec06b600 legacy=0x7fb3ec162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.519+0000 7fb3ebfff640 1 -- 172.21.15.72:0/2354839234 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec15eff0 con 0x7fb3ec06b220 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3ebfff640 1 --2- 172.21.15.72:0/2354839234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec06b220 0x7fb3ec15e450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40097d0 tx=0x7fb3d402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=789f10ff43af6c54 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4037070 con 0x7fb3ec06b220 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d4030510 con 0x7fb3ec06b220 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3eaffd640 1 -- 172.21.15.72:0/2354839234 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40019f0 con 0x7fb3ec06b220 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 -- 172.21.15.72:0/2354839234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec06b220 msgr2=0x7fb3ec15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 --2- 172.21.15.72:0/2354839234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec06b220 0x7fb3ec15e450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40097d0 tx=0x7fb3d402b800 comp rx=0 tx=0).stop 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 -- 172.21.15.72:0/2354839234 shutdown_connections 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 --2- 172.21.15.72:0/2354839234 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec06b220 0x7fb3ec15e450 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 -- 172.21.15.72:0/2354839234 >> 172.21.15.72:0/2354839234 conn(0x7fb3ec06c0c0 msgr2=0x7fb3ec06c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 -- 172.21.15.72:0/2354839234 shutdown_connections 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.521+0000 7fb3f298a640 1 -- 172.21.15.72:0/2354839234 wait complete. 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 Processor -- start 2024-03-06T07:30:12.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 -- start start 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 0x7fb3ec0ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec06b600 0x7fb3ec14b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec0ce920 0x7fb3ec0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec072090 con 0x7fb3ec06b220 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec0ce200 con 0x7fb3ec06b600 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.523+0000 7fb3f298a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1e9a80 con 0x7fb3ec0ce920 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 0x7fb3ec0ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 0x7fb3ec0ae210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42940/0 (socket says 172.21.15.72:42940) 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 -- 172.21.15.72:0/1316116713 learned_addr learned my addr 172.21.15.72:0/1316116713 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3f0f27640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec0ce920 0x7fb3ec0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3eb7fe640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec06b600 0x7fb3ec14b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 -- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec0ce920 msgr2=0x7fb3ec0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec0ce920 0x7fb3ec0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 -- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec06b600 msgr2=0x7fb3ec14b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec06b600 0x7fb3ec14b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 -- 172.21.15.72:0/1316116713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0ae750 con 0x7fb3ec06b220 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3ebfff640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 0x7fb3ec0ae210 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fb3d402b840 tx=0x7fb3d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4040020 con 0x7fb3ec06b220 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d4038040 con 0x7fb3ec06b220 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4037070 con 0x7fb3ec06b220 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec1e8e70 con 0x7fb3ec06b220 2024-03-06T07:30:12.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.524+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec1e9430 con 0x7fb3ec06b220 2024-03-06T07:30:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.526+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec06c9d0 con 0x7fb3ec06b220 2024-03-06T07:30:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.526+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb3d4043020 con 0x7fb3ec06b220 2024-03-06T07:30:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.526+0000 7fb3e8ff9640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb3cc06b4d0 0x7fb3cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.526+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb3d40ab520 con 0x7fb3ec06b220 2024-03-06T07:30:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.526+0000 7fb3eb7fe640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb3cc06b4d0 0x7fb3cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:12.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.527+0000 7fb3eb7fe640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb3cc06b4d0 0x7fb3cc06d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec15b8d0 tx=0x7fb3dc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:12.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.530+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb3ec06c9d0 con 0x7fb3ec06b220 2024-03-06T07:30:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.742+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3ec04f330 con 0x7fb3ec06b220 2024-03-06T07:30:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.743+0000 7fb3e8ff9640 1 -- 172.21.15.72:0/1316116713 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fb3d404a030 con 0x7fb3ec06b220 2024-03-06T07:30:12.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:12.745 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb3cc06b4d0 msgr2=0x7fb3cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb3cc06b4d0 0x7fb3cc06d990 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec15b8d0 tx=0x7fb3dc009040 comp rx=0 tx=0).stop 2024-03-06T07:30:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 msgr2=0x7fb3ec0ae210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 0x7fb3ec0ae210 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7fb3d402b840 tx=0x7fb3d40046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 shutdown_connections 2024-03-06T07:30:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb3cc06b4d0 0x7fb3cc06d990 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3ec0ce920 0x7fb3ec0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3ec06b600 0x7fb3ec14b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 --2- 172.21.15.72:0/1316116713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec06b220 0x7fb3ec0ae210 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 >> 172.21.15.72:0/1316116713 conn(0x7fb3ec06c0c0 msgr2=0x7fb3ec06f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 shutdown_connections 2024-03-06T07:30:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:12.745+0000 7fb3f298a640 1 -- 172.21.15.72:0/1316116713 wait complete. 2024-03-06T07:30:12.959 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:12.959 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:13.718 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:13.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:14.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.059+0000 7ff8140a4640 1 Processor -- start 2024-03-06T07:30:14.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.059+0000 7ff8140a4640 1 -- start start 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff8140a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff80c15dfe0 0x7ff80c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff8140a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0cf830 0x7ff80c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff8140a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0b3430 con 0x7ff80c0d27f0 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff8140a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80c0d2c50 con 0x7ff80c15dfe0 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff8140a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0b31e0 con 0x7ff80c0cf830 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff811e40640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff80c15dfe0 0x7ff80c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff811e40640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff80c15dfe0 0x7ff80c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42952/0 (socket says 172.21.15.72:42952) 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.060+0000 7ff811e40640 1 -- 172.21.15.72:0/2735566145 learned_addr learned my addr 172.21.15.72:0/2735566145 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff81163f640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0cf830 0x7ff80c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff811e40640 1 -- 172.21.15.72:0/2735566145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0cf830 msgr2=0x7ff80c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff811e40640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0cf830 0x7ff80c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff811e40640 1 -- 172.21.15.72:0/2735566145 >> v1:172.21.15.146:6789/0 conn(0x7ff80c0d27f0 legacy=0x7ff80c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff811e40640 1 -- 172.21.15.72:0/2735566145 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0b3710 con 0x7ff80c15dfe0 2024-03-06T07:30:14.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff81163f640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0cf830 0x7ff80c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:14.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.061+0000 7ff811e40640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c15dfe0 0x7ff80c0cf2f0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff808009870 tx=0x7ff80802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a16e47b618af344 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff810e3e640 1 -- 172.21.15.72:0/2735566145 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808034070 con 0x7ff80c15dfe0 2024-03-06T07:30:14.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff810e3e640 1 -- 172.21.15.72:0/2735566145 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff808030840 con 0x7ff80c15dfe0 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff810e3e640 1 -- 172.21.15.72:0/2735566145 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808030c10 con 0x7ff80c15dfe0 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 -- 172.21.15.72:0/2735566145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c15dfe0 msgr2=0x7ff80c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c15dfe0 0x7ff80c0cf2f0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7ff808009870 tx=0x7ff80802b980 comp rx=0 tx=0).stop 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 -- 172.21.15.72:0/2735566145 shutdown_connections 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0cf830 0x7ff80c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 --2- 172.21.15.72:0/2735566145 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c15dfe0 0x7ff80c0cf2f0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 -- 172.21.15.72:0/2735566145 >> 172.21.15.72:0/2735566145 conn(0x7ff80c0d3cd0 msgr2=0x7ff80c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 -- 172.21.15.72:0/2735566145 shutdown_connections 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.062+0000 7ff8140a4640 1 -- 172.21.15.72:0/2735566145 wait complete. 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.063+0000 7ff8140a4640 1 Processor -- start 2024-03-06T07:30:14.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.063+0000 7ff8140a4640 1 -- start start 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff8140a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 0x7ff80c1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff8140a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0d27f0 0x7ff80c14d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff8140a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80c15dfe0 0x7ff80c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff8140a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0d1c70 con 0x7ff80c0cf830 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff8140a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0b3d60 con 0x7ff80c15dfe0 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff8140a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff80c0d1df0 con 0x7ff80c0d27f0 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 0x7ff80c1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 0x7ff80c1f39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42964/0 (socket says 172.21.15.72:42964) 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 -- 172.21.15.72:0/3913782004 learned_addr learned my addr 172.21.15.72:0/3913782004 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff81163f640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0d27f0 0x7ff80c14d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff812641640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80c15dfe0 0x7ff80c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 -- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0d27f0 msgr2=0x7ff80c14d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0d27f0 0x7ff80c14d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 -- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80c15dfe0 msgr2=0x7ff80c14d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80c15dfe0 0x7ff80c14d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff811e40640 1 -- 172.21.15.72:0/3913782004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c1f40b0 con 0x7ff80c0cf830 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.064+0000 7ff81163f640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0d27f0 0x7ff80c14d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.065+0000 7ff811e40640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 0x7ff80c1f39b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7ff808000c00 tx=0x7ff8080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.065+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808034040 con 0x7ff80c0cf830 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.065+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff808039d90 con 0x7ff80c0cf830 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.065+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff808043610 con 0x7ff80c0cf830 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.065+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c1f4390 con 0x7ff80c0cf830 2024-03-06T07:30:14.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.065+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c14ce00 con 0x7ff80c0cf830 2024-03-06T07:30:14.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.066+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c04f330 con 0x7ff80c0cf830 2024-03-06T07:30:14.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.066+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff80804a060 con 0x7ff80c0cf830 2024-03-06T07:30:14.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.067+0000 7ff8027fc640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f006b4d0 0x7ff7f006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:14.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.067+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff8080ac220 con 0x7ff80c0cf830 2024-03-06T07:30:14.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.067+0000 7ff81163f640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f006b4d0 0x7ff7f006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:14.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.067+0000 7ff81163f640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f006b4d0 0x7ff7f006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff80c1615c0 tx=0x7ff7fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:14.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.071+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff80803e070 con 0x7ff80c0cf830 2024-03-06T07:30:14.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.285+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff80c0b0d40 con 0x7ff80c0cf830 2024-03-06T07:30:14.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.286+0000 7ff8027fc640 1 -- 172.21.15.72:0/3913782004 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7ff80802d330 con 0x7ff80c0cf830 2024-03-06T07:30:14.288 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:14.288 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:14.288 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:14.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f006b4d0 msgr2=0x7ff7f006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f006b4d0 0x7ff7f006d990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7ff80c1615c0 tx=0x7ff7fc008040 comp rx=0 tx=0).stop 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 msgr2=0x7ff80c1f39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 0x7ff80c1f39b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7ff808000c00 tx=0x7ff8080046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 shutdown_connections 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f006b4d0 0x7ff7f006d990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff80c15dfe0 0x7ff80c14d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff80c0d27f0 0x7ff80c14d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 --2- 172.21.15.72:0/3913782004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff80c0cf830 0x7ff80c1f39b0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 >> 172.21.15.72:0/3913782004 conn(0x7ff80c0d3cd0 msgr2=0x7ff80c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 shutdown_connections 2024-03-06T07:30:14.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:14.288+0000 7ff8140a4640 1 -- 172.21.15.72:0/3913782004 wait complete. 2024-03-06T07:30:14.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:14.502 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:15.503 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:15.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 Processor -- start 2024-03-06T07:30:15.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 -- start start 2024-03-06T07:30:15.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5fc015fa10 0x7f5fc00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5fc00d16e0 0x7f5fc00d1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc00cebe0 con 0x7f5fc00d16e0 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc00ceec0 con 0x7f5fc015fa10 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.604+0000 7f5fc629a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00cf1a0 con 0x7f5fc015bdd0 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5fc00d16e0 0x7f5fc00d1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5fc00d16e0 0x7f5fc00d1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42976/0 (socket says 172.21.15.72:42976) 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 -- 172.21.15.72:0/2763395361 learned_addr learned my addr 172.21.15.72:0/2763395361 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fbf7fe640 1 --2- 172.21.15.72:0/2763395361 >> v2:172.21.15.146:3300/0 conn(0x7f5fc015fa10 0x7f5fc00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fbffff640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 -- 172.21.15.72:0/2763395361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 msgr2=0x7f5fc00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 -- 172.21.15.72:0/2763395361 >> v2:172.21.15.146:3300/0 conn(0x7f5fc015fa10 msgr2=0x7f5fc00ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 --2- 172.21.15.72:0/2763395361 >> v2:172.21.15.146:3300/0 conn(0x7f5fc015fa10 0x7f5fc00ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fc4837640 1 -- 172.21.15.72:0/2763395361 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc00cc8e0 con 0x7f5fc00d16e0 2024-03-06T07:30:15.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fbf7fe640 1 --2- 172.21.15.72:0/2763395361 >> v2:172.21.15.146:3300/0 conn(0x7f5fc015fa10 0x7f5fc00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:15.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.605+0000 7f5fbffff640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.606+0000 7f5fc4837640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00d16e0 0x7f5fc00d1b30 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5fc006e2d0 tx=0x7f5fb802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee8d3d0d3b1f13e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.606+0000 7f5fbeffd640 1 -- 172.21.15.72:0/2763395361 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb802c9e0 con 0x7f5fc00d16e0 2024-03-06T07:30:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.606+0000 7f5fbeffd640 1 -- 172.21.15.72:0/2763395361 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb802cb80 con 0x7f5fc00d16e0 2024-03-06T07:30:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.606+0000 7f5fbeffd640 1 -- 172.21.15.72:0/2763395361 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb80343f0 con 0x7f5fc00d16e0 2024-03-06T07:30:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 -- 172.21.15.72:0/2763395361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00d16e0 msgr2=0x7f5fc00d1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00d16e0 0x7f5fc00d1b30 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5fc006e2d0 tx=0x7f5fb802be40 comp rx=0 tx=0).stop 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 -- 172.21.15.72:0/2763395361 shutdown_connections 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc00d16e0 0x7f5fc00d1b30 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 --2- 172.21.15.72:0/2763395361 >> v2:172.21.15.146:3300/0 conn(0x7f5fc015fa10 0x7f5fc00ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 --2- 172.21.15.72:0/2763395361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 -- 172.21.15.72:0/2763395361 >> 172.21.15.72:0/2763395361 conn(0x7f5fc009a9f0 msgr2=0x7f5fc009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 -- 172.21.15.72:0/2763395361 shutdown_connections 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.607+0000 7f5fc629a640 1 -- 172.21.15.72:0/2763395361 wait complete. 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.608+0000 7f5fc629a640 1 Processor -- start 2024-03-06T07:30:15.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.608+0000 7f5fc629a640 1 -- start start 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc629a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc00d16e0 0x7f5fc014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc629a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc014cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc629a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 0x7f5fc006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc629a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00cc5d0 con 0x7f5fc015fa10 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc629a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc00d2070 con 0x7f5fc00d16e0 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc629a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc015c1b0 con 0x7f5fc015bdd0 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 0x7f5fc006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 0x7f5fc006f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42986/0 (socket says 172.21.15.72:42986) 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 -- 172.21.15.72:0/3793713101 learned_addr learned my addr 172.21.15.72:0/3793713101 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fbffff640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc00d16e0 0x7f5fc014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fbf7fe640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc014cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 -- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 msgr2=0x7f5fc014cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc014cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 -- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc00d16e0 msgr2=0x7f5fc014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc00d16e0 0x7f5fc014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.609+0000 7f5fc4837640 1 -- 172.21.15.72:0/3793713101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc014eb30 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.610+0000 7f5fc4837640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 0x7f5fc006f000 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8036040 tx=0x7f5fb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.610+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8043020 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.610+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb803e450 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.610+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8009510 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.610+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc014ee10 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.610+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc014f3d0 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.611+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc0072d10 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.611+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5fb8046020 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.612+0000 7f5fbcff9640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fac06b500 0x7f5fac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.612+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5fb80aca90 con 0x7f5fc015fa10 2024-03-06T07:30:15.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.612+0000 7f5fbffff640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fac06b500 0x7f5fac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:15.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.612+0000 7f5fbffff640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fac06b500 0x7f5fac06d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0006fd0 tx=0x7f5fb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:15.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.620+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5fc0072d10 con 0x7f5fc015fa10 2024-03-06T07:30:15.719 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:15.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.899+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fc00b7a40 con 0x7f5fc015fa10 2024-03-06T07:30:15.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.899+0000 7f5fbcff9640 1 -- 172.21.15.72:0/3793713101 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5fb804d030 con 0x7f5fc015fa10 2024-03-06T07:30:15.901 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:15.902 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:15.902 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:15.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fac06b500 msgr2=0x7f5fac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fac06b500 0x7f5fac06d9c0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0006fd0 tx=0x7f5fb0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:15.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 msgr2=0x7f5fc006f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:15.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 0x7f5fc006f000 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8036040 tx=0x7f5fb8004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:15.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 shutdown_connections 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5fac06b500 0x7f5fac06d9c0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5fc015fa10 0x7f5fc006f000 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fc015bdd0 0x7f5fc014cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 --2- 172.21.15.72:0/3793713101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fc00d16e0 0x7f5fc014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 >> 172.21.15.72:0/3793713101 conn(0x7f5fc009a9f0 msgr2=0x7f5fc006fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 shutdown_connections 2024-03-06T07:30:15.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:15.902+0000 7f5fc629a640 1 -- 172.21.15.72:0/3793713101 wait complete. 2024-03-06T07:30:16.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:16.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:17.118 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.219+0000 7f4c879a1640 1 Processor -- start 2024-03-06T07:30:17.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.219+0000 7f4c879a1640 1 -- start start 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.220+0000 7f4c879a1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c8015f8f0 0x7f4c800724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.220+0000 7f4c879a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8006afb0 0x7f4c8006b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.220+0000 7f4c879a1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c800ce9a0 con 0x7f4c8015bd20 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.220+0000 7f4c879a1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c80072ad0 con 0x7f4c8015f8f0 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.220+0000 7f4c879a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c80072db0 con 0x7f4c8006afb0 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c8015f8f0 0x7f4c800724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c8015f8f0 0x7f4c800724b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:54340/0 (socket says 172.21.15.72:54340) 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c8573d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4c8015bd20 0x7f4c800ce290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57334/0 (socket says 172.21.15.72:57334) 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 -- 172.21.15.72:0/1361686410 learned_addr learned my addr 172.21.15.72:0/1361686410 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c85f3e640 1 --2- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8006afb0 0x7f4c8006b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 -- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8006afb0 msgr2=0x7f4c8006b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 --2- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8006afb0 0x7f4c8006b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 -- 172.21.15.72:0/1361686410 >> v1:172.21.15.146:6789/0 conn(0x7f4c8015bd20 legacy=0x7f4c800ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.221+0000 7f4c84f3c640 1 -- 172.21.15.72:0/1361686410 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c80073090 con 0x7f4c8015f8f0 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c84f3c640 1 --2- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c800724b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009870 tx=0x7f4c7402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f31e38703da1ad89 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c7ffff640 1 -- 172.21.15.72:0/1361686410 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74034070 con 0x7f4c8015f8f0 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c7ffff640 1 -- 172.21.15.72:0/1361686410 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74030840 con 0x7f4c8015f8f0 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c7ffff640 1 -- 172.21.15.72:0/1361686410 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74030c10 con 0x7f4c8015f8f0 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 -- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 msgr2=0x7f4c800724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 --2- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c800724b0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009870 tx=0x7f4c7402b980 comp rx=0 tx=0).stop 2024-03-06T07:30:17.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 -- 172.21.15.72:0/1361686410 shutdown_connections 2024-03-06T07:30:17.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 --2- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8006afb0 0x7f4c8006b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 --2- 172.21.15.72:0/1361686410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c800724b0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 -- 172.21.15.72:0/1361686410 >> 172.21.15.72:0/1361686410 conn(0x7f4c800b1d00 msgr2=0x7f4c800b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.222+0000 7f4c879a1640 1 -- 172.21.15.72:0/1361686410 shutdown_connections 2024-03-06T07:30:17.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.223+0000 7f4c879a1640 1 -- 172.21.15.72:0/1361686410 wait complete. 2024-03-06T07:30:17.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 Processor -- start 2024-03-06T07:30:17.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 -- start start 2024-03-06T07:30:17.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 0x7f4c800c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8015bd20 0x7f4c800c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c8015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800672f0 con 0x7f4c8006afb0 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c800cca90 con 0x7f4c8015f8f0 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c879a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8015c100 con 0x7f4c8015bd20 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c8573d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 0x7f4c800c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c8573d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 0x7f4c800c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42996/0 (socket says 172.21.15.72:42996) 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c8573d640 1 -- 172.21.15.72:0/49426884 learned_addr learned my addr 172.21.15.72:0/49426884 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c85f3e640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c8015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.224+0000 7f4c84f3c640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8015bd20 0x7f4c800c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c8573d640 1 -- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8015bd20 msgr2=0x7f4c800c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c8573d640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8015bd20 0x7f4c800c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c8573d640 1 -- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 msgr2=0x7f4c8015a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c8573d640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c8015a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c8573d640 1 -- 172.21.15.72:0/49426884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c800c55e0 con 0x7f4c8006afb0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c8573d640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 0x7f4c800c6680 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800c430 tx=0x7f4c7800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c780074e0 con 0x7f4c8006afb0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c78007680 con 0x7f4c8006afb0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78004a00 con 0x7f4c8006afb0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c800c58c0 con 0x7f4c8006afb0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.225+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c800c5e80 con 0x7f4c8006afb0 2024-03-06T07:30:17.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.226+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8004f330 con 0x7f4c8006afb0 2024-03-06T07:30:17.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.227+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4c7801a020 con 0x7f4c8006afb0 2024-03-06T07:30:17.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.227+0000 7f4c7dffb640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:17.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.227+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4c78089910 con 0x7f4c8006afb0 2024-03-06T07:30:17.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.227+0000 7f4c84f3c640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:17.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.227+0000 7f4c84f3c640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4c74036000 tx=0x7f4c7403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:17.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.231+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4c800ce9a0 con 0x7f4c8006afb0 2024-03-06T07:30:17.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.445+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c800735e0 con 0x7f4c8006afb0 2024-03-06T07:30:17.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.446+0000 7f4c7dffb640 1 -- 172.21.15.72:0/49426884 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f4c78058080 con 0x7f4c8006afb0 2024-03-06T07:30:17.448 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:17.448 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:17.448 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 msgr2=0x7f4c6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4c74036000 tx=0x7f4c7403b040 comp rx=0 tx=0).stop 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 msgr2=0x7f4c800c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 0x7f4c800c6680 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f4c7800c430 tx=0x7f4c7800c8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 shutdown_connections 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4c6c06b4d0 0x7f4c6c06d990 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c8015f8f0 0x7f4c8015a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.447+0000 7f4c879a1640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c8015bd20 0x7f4c800c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.448+0000 7f4c879a1640 1 --2- 172.21.15.72:0/49426884 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c8006afb0 0x7f4c800c6680 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:17.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.448+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 >> 172.21.15.72:0/49426884 conn(0x7f4c800b1d00 msgr2=0x7f4c800cce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:17.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.448+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 shutdown_connections 2024-03-06T07:30:17.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:17.448+0000 7f4c879a1640 1 -- 172.21.15.72:0/49426884 wait complete. 2024-03-06T07:30:17.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:17.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:17.722 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:18.667 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.787+0000 7f2145e08640 1 Processor -- start 2024-03-06T07:30:18.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.787+0000 7f2145e08640 1 -- start start 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f2145e08640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f2140166db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f2145e08640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21400d0020 con 0x7f214015c320 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f2145e08640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21400b0d20 con 0x7f21400b40c0 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f2145e08640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21400b1000 con 0x7f21400d06b0 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f213f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f214015c320 0x7f21400b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60602/0 (socket says 172.21.15.72:60602) 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f213f7fe640 1 -- 172.21.15.72:0/1860579616 learned_addr learned my addr 172.21.15.72:0/1860579616 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:18.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360048618 0 0) 0x7f21400d0020 con 0x7f214015c320 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003520 con 0x7f214015c320 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.788+0000 7f213ffff640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f2140166db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 39748088 0 0) 0x7f212c003520 con 0x7f214015c320 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 msgr2=0x7f2140166db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f2140166db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 >> v1:172.21.15.146:6789/0 conn(0x7f21400b40c0 legacy=0x7f21400b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400cfa60 con 0x7f214015c320 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2130001670 con 0x7f214015c320 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 >> v1:172.21.15.72:6789/0 conn(0x7f214015c320 legacy=0x7f21400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f212c0039e0 0x7f212c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c004800 0x7f212c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f212c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21400b1000 con 0x7f212c0039e0 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c005550 con 0x7f212c004800 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c00bf00 con 0x7f21400d06b0 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213f7fe640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f212c0039e0 0x7f212c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213effd640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c004800 0x7f212c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213ffff640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f212c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213f7fe640 1 -- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 msgr2=0x7f212c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213f7fe640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f212c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213f7fe640 1 -- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c004800 msgr2=0x7f212c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213f7fe640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c004800 0x7f212c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213f7fe640 1 -- 172.21.15.72:0/1860579616 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c003520 con 0x7f212c0039e0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.789+0000 7f213ffff640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21400d06b0 0x7f212c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f213f7fe640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f212c0039e0 0x7f212c003de0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f213000c290 tx=0x7f212c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28087569e687db01 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2130007d50 con 0x7f212c0039e0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2130030070 con 0x7f212c0039e0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f213e7fc640 1 -- 172.21.15.72:0/1860579616 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213002cd80 con 0x7f212c0039e0 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f2145e08640 1 -- 172.21.15.72:0/1860579616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f212c0039e0 msgr2=0x7f212c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f2145e08640 1 --2- 172.21.15.72:0/1860579616 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f212c0039e0 0x7f212c003de0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f213000c290 tx=0x7f212c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f213ffff640 1 -- 172.21.15.72:0/1860579616 reap_dead start 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f2145e08640 1 -- 172.21.15.72:0/1860579616 shutdown_connections 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f2145e08640 1 -- 172.21.15.72:0/1860579616 >> 172.21.15.72:0/1860579616 conn(0x7f21400b35a0 msgr2=0x7f21400b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.790+0000 7f2145e08640 1 -- 172.21.15.72:0/1860579616 shutdown_connections 2024-03-06T07:30:18.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.791+0000 7f2145e08640 1 -- 172.21.15.72:0/1860579616 wait complete. 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.792+0000 7f2145e08640 1 Processor -- start 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.792+0000 7f2145e08640 1 -- start start 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c0039e0 0x7f214014cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f214014c9e0 0x7f21400b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 0x7f21400c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21400cda00 con 0x7f21400c64b0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21400cd880 con 0x7f212c0039e0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2140164da0 con 0x7f214014c9e0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 0x7f21400c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 0x7f21400c6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43024/0 (socket says 172.21.15.72:43024) 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 -- 172.21.15.72:0/4140249578 learned_addr learned my addr 172.21.15.72:0/4140249578 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213f7fe640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c0039e0 0x7f214014cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 -- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f214014c9e0 msgr2=0x7f21400b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213effd640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f214014c9e0 0x7f21400b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f214014c9e0 0x7f21400b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 -- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c0039e0 msgr2=0x7f214014cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c0039e0 0x7f214014cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:18.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 -- 172.21.15.72:0/4140249578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400c5410 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f213ffff640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 0x7f21400c6930 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f21400720d0 tx=0x7f2138009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138019070 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138007cb0 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21400c56f0 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138007e60 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.793+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21400c5cb0 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.795+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214004f330 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.795+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f213801c070 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.795+0000 7f2137fff640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f212006b500 0x7f212006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.795+0000 7f213f7fe640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f212006b500 0x7f212006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.795+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2138089cf0 con 0x7f21400c64b0 2024-03-06T07:30:18.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.796+0000 7f213f7fe640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f212006b500 0x7f212006d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2130034000 tx=0x7f2130009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:18.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:18.799+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f213808e070 con 0x7f21400c64b0 2024-03-06T07:30:19.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.014+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21400cd0c0 con 0x7f21400c64b0 2024-03-06T07:30:19.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.015+0000 7f2137fff640 1 -- 172.21.15.72:0/4140249578 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2138058b30 con 0x7f21400c64b0 2024-03-06T07:30:19.017 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:19.017 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:19.017 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.016+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f212006b500 msgr2=0x7f212006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.016+0000 7f2145e08640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f212006b500 0x7f212006d9c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2130034000 tx=0x7f2130009030 comp rx=0 tx=0).stop 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.016+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 msgr2=0x7f21400c6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.016+0000 7f2145e08640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 0x7f21400c6930 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f21400720d0 tx=0x7f2138009d30 comp rx=0 tx=0).stop 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 shutdown_connections 2024-03-06T07:30:19.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f212006b500 0x7f212006d9c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21400c64b0 0x7f21400c6930 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f214014c9e0 0x7f21400b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 --2- 172.21.15.72:0/4140249578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f212c0039e0 0x7f214014cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 >> 172.21.15.72:0/4140249578 conn(0x7f21400b35a0 msgr2=0x7f21400d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 shutdown_connections 2024-03-06T07:30:19.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:19.017+0000 7f2145e08640 1 -- 172.21.15.72:0/4140249578 wait complete. 2024-03-06T07:30:19.231 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:19.231 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:19.724 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:20.232 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:20.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.332+0000 7fd8163bf640 1 Processor -- start 2024-03-06T07:30:20.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.332+0000 7fd8163bf640 1 -- start start 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd8163bf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8100d2710 0x7fd8100b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd8163bf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8100b3430 con 0x7fd81015dfe0 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd8163bf640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8100b3710 con 0x7fd8100d22b0 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd8163bf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8100b31e0 con 0x7fd8100d2710 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd81495c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8100d2710 0x7fd8100b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd81495c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8100d2710 0x7fd8100b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43026/0 (socket says 172.21.15.72:43026) 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd80ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd81015dfe0 0x7fd8100cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60612/0 (socket says 172.21.15.72:60612) 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.333+0000 7fd81495c640 1 -- 172.21.15.72:0/1683411232 learned_addr learned my addr 172.21.15.72:0/1683411232 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.334+0000 7fd81495c640 1 -- 172.21.15.72:0/1683411232 >> v1:172.21.15.146:6789/0 conn(0x7fd8100d22b0 legacy=0x7fd8100b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:20.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.334+0000 7fd81495c640 1 -- 172.21.15.72:0/1683411232 >> v1:172.21.15.72:6789/0 conn(0x7fd81015dfe0 legacy=0x7fd8100cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.334+0000 7fd81495c640 1 -- 172.21.15.72:0/1683411232 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8100b39f0 con 0x7fd8100d2710 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.334+0000 7fd81495c640 1 --2- 172.21.15.72:0/1683411232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8100d2710 0x7fd8100b4900 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fd808009e80 tx=0x7fd80802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277533be615bdab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd80effd640 1 -- 172.21.15.72:0/1683411232 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80802c940 con 0x7fd8100d2710 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd80effd640 1 -- 172.21.15.72:0/1683411232 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80802cae0 con 0x7fd8100d2710 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd80effd640 1 -- 172.21.15.72:0/1683411232 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8080343f0 con 0x7fd8100d2710 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 -- 172.21.15.72:0/1683411232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8100d2710 msgr2=0x7fd8100b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 --2- 172.21.15.72:0/1683411232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8100d2710 0x7fd8100b4900 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fd808009e80 tx=0x7fd80802be20 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 -- 172.21.15.72:0/1683411232 shutdown_connections 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 --2- 172.21.15.72:0/1683411232 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8100d2710 0x7fd8100b4900 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 -- 172.21.15.72:0/1683411232 >> 172.21.15.72:0/1683411232 conn(0x7fd8100d3cd0 msgr2=0x7fd8100d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 -- 172.21.15.72:0/1683411232 shutdown_connections 2024-03-06T07:30:20.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.335+0000 7fd8163bf640 1 -- 172.21.15.72:0/1683411232 wait complete. 2024-03-06T07:30:20.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.336+0000 7fd8163bf640 1 Processor -- start 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 -- start start 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8100d22b0 0x7fd8100cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8100d2710 0x7fd81014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 0x7fd8100b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100cfa00 con 0x7fd81015dfe0 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100b3d60 con 0x7fd8100d2710 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd8163bf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8100d1c70 con 0x7fd8100d22b0 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd81495c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 0x7fd8100b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd81495c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 0x7fd8100b68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43036/0 (socket says 172.21.15.72:43036) 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd81495c640 1 -- 172.21.15.72:0/3787352247 learned_addr learned my addr 172.21.15.72:0/3787352247 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd80f7fe640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8100d2710 0x7fd81014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.337+0000 7fd80ffff640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8100d22b0 0x7fd8100cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd81495c640 1 -- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8100d22b0 msgr2=0x7fd8100cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd81495c640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8100d22b0 0x7fd8100cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd81495c640 1 -- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8100d2710 msgr2=0x7fd81014ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd81495c640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8100d2710 0x7fd81014ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd81495c640 1 -- 172.21.15.72:0/3787352247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8100ccd40 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd80f7fe640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8100d2710 0x7fd81014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd81495c640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 0x7fd8100b68a0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd80802c430 tx=0x7fd808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808043020 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80803e450 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8100ccfe0 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808009510 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.338+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8100cd5a0 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.339+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81004f330 con 0x7fd81015dfe0 2024-03-06T07:30:20.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.340+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd808046020 con 0x7fd81015dfe0 2024-03-06T07:30:20.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.340+0000 7fd80cff9640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:20.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.340+0000 7fd80ffff640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:20.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.340+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd8080aca00 con 0x7fd81015dfe0 2024-03-06T07:30:20.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.341+0000 7fd80ffff640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd804007fd0 tx=0x7fd804007c20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:20.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.344+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd80804d030 con 0x7fd81015dfe0 2024-03-06T07:30:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.558+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd81006a190 con 0x7fd81015dfe0 2024-03-06T07:30:20.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.559+0000 7fd80cff9640 1 -- 172.21.15.72:0/3787352247 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fd80807b8d0 con 0x7fd81015dfe0 2024-03-06T07:30:20.561 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:20.561 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:20.562 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 msgr2=0x7fd7fc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd804007fd0 tx=0x7fd804007c20 comp rx=0 tx=0).stop 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 msgr2=0x7fd8100b68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 0x7fd8100b68a0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd80802c430 tx=0x7fd808004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 shutdown_connections 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd7fc06b4d0 0x7fd7fc06d990 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd81015dfe0 0x7fd8100b68a0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8100d2710 0x7fd81014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 --2- 172.21.15.72:0/3787352247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8100d22b0 0x7fd8100cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 >> 172.21.15.72:0/3787352247 conn(0x7fd8100d3cd0 msgr2=0x7fd8100cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 shutdown_connections 2024-03-06T07:30:20.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:20.561+0000 7fd8163bf640 1 -- 172.21.15.72:0/3787352247 wait complete. 2024-03-06T07:30:20.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:20.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:21.726 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:21.777 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 Processor -- start 2024-03-06T07:30:21.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 -- start start 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd10415dfe0 0x7fd1040cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 0x7fd1040d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1040b3430 con 0x7fd1040d27f0 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1040d2c50 con 0x7fd10415dfe0 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.879+0000 7fd10ab25640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040b31e0 con 0x7fd1040cf830 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd1090c2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd1040d27f0 0x7fd1040d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60614/0 (socket says 172.21.15.72:60614) 2024-03-06T07:30:21.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd1090c2640 1 -- 172.21.15.72:0/3262885921 learned_addr learned my addr 172.21.15.72:0/3262885921 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd1088c1640 1 --2- 172.21.15.72:0/3262885921 >> v2:172.21.15.146:3300/0 conn(0x7fd10415dfe0 0x7fd1040cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 0x7fd1040d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641179567 0 0) 0x7fd1040b3430 con 0x7fd1040d27f0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0ec003520 con 0x7fd1040d27f0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723171706 0 0) 0x7fd0ec003520 con 0x7fd1040d27f0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 msgr2=0x7fd1040d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 0x7fd1040d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 >> v2:172.21.15.146:3300/0 conn(0x7fd10415dfe0 msgr2=0x7fd1040cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 --2- 172.21.15.72:0/3262885921 >> v2:172.21.15.146:3300/0 conn(0x7fd10415dfe0 0x7fd1040cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1040b3710 con 0x7fd1040d27f0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fd0fc0019f0 con 0x7fd1040d27f0 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd1088c1640 1 --2- 172.21.15.72:0/3262885921 >> v2:172.21.15.146:3300/0 conn(0x7fd10415dfe0 0x7fd1040cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 >> v1:172.21.15.72:6789/0 conn(0x7fd1040d27f0 legacy=0x7fd1040d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 0x7fd0ec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0ec003f30 0x7fd0ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec004d40 0x7fd0ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040b31e0 con 0x7fd0ec003f30 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec00b9c0 con 0x7fd0ec004d40 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec00bb40 con 0x7fd1040cf830 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0ec003f30 0x7fd0ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd1088c1640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 0x7fd0ec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.880+0000 7fd1090c2640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec004d40 0x7fd0ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.881+0000 7fd0fbfff640 1 -- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 msgr2=0x7fd0ec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.881+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1040cf830 0x7fd0ec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.881+0000 7fd0fbfff640 1 -- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec004d40 msgr2=0x7fd0ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.881+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0ec004d40 0x7fd0ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.881+0000 7fd0fbfff640 1 -- 172.21.15.72:0/3262885921 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec003520 con 0x7fd0ec003f30 2024-03-06T07:30:21.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0ec003f30 0x7fd0ec004800 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009a90 tx=0x7fd0ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3f4ebcd99aed11 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4039070 con 0x7fd0ec003f30 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f402c6b0 con 0x7fd0ec003f30 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd0fb7fe640 1 -- 172.21.15.72:0/3262885921 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4009690 con 0x7fd0ec003f30 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd10ab25640 1 -- 172.21.15.72:0/3262885921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0ec003f30 msgr2=0x7fd0ec004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd10ab25640 1 --2- 172.21.15.72:0/3262885921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0ec003f30 0x7fd0ec004800 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009a90 tx=0x7fd0ec005830 comp rx=0 tx=0).stop 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd1090c2640 1 -- 172.21.15.72:0/3262885921 reap_dead start 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd10ab25640 1 -- 172.21.15.72:0/3262885921 shutdown_connections 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd10ab25640 1 -- 172.21.15.72:0/3262885921 >> 172.21.15.72:0/3262885921 conn(0x7fd1040d3cd0 msgr2=0x7fd1040d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.882+0000 7fd10ab25640 1 -- 172.21.15.72:0/3262885921 shutdown_connections 2024-03-06T07:30:21.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.883+0000 7fd10ab25640 1 -- 172.21.15.72:0/3262885921 wait complete. 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 Processor -- start 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 -- start start 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec003f30 0x7fd10414c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 0x7fd1040b8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1040cb190 0x7fd1040cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040d1c70 con 0x7fd1040b7d10 2024-03-06T07:30:21.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040b3d60 con 0x7fd1040cb190 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.884+0000 7fd10ab25640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1040d1df0 con 0x7fd0ec003f30 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 0x7fd1040b8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 0x7fd1040b8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43044/0 (socket says 172.21.15.72:43044) 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 -- 172.21.15.72:0/2242763694 learned_addr learned my addr 172.21.15.72:0/2242763694 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd1088c1640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec003f30 0x7fd10414c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 -- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec003f30 msgr2=0x7fd10414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd1090c2640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1040cb190 0x7fd1040cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec003f30 0x7fd10414c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 -- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1040cb190 msgr2=0x7fd1040cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1040cb190 0x7fd1040cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:21.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 -- 172.21.15.72:0/2242763694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1040b86b0 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd1090c2640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1040cb190 0x7fd1040cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0fbfff640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 0x7fd1040b8170 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4037040 tx=0x7fd0f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4039050 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f4045070 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40044e0 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1040cbef0 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.885+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1040cc4b0 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.886+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1040cf5f0 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.887+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fd0f4003850 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.887+0000 7fd0f97fa640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0f006b500 0x7fd0f006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.887+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd0f40abcf0 con 0x7fd1040b7d10 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.887+0000 7fd1088c1640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0f006b500 0x7fd0f006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:21.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.888+0000 7fd1088c1640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0f006b500 0x7fd0f006d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd100009bf0 tx=0x7fd100008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:21.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:21.892+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fd1040cf5f0 con 0x7fd1040b7d10 2024-03-06T07:30:21.960 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:21.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:22.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.222+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1040d2510 con 0x7fd1040b7d10 2024-03-06T07:30:22.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.222+0000 7fd0f97fa640 1 -- 172.21.15.72:0/2242763694 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fd0f404e020 con 0x7fd1040b7d10 2024-03-06T07:30:22.225 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:22.225 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:22.225 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0f006b500 msgr2=0x7fd0f006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0f006b500 0x7fd0f006d9c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd100009bf0 tx=0x7fd100008040 comp rx=0 tx=0).stop 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 msgr2=0x7fd1040b8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 0x7fd1040b8170 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4037040 tx=0x7fd0f4004730 comp rx=0 tx=0).stop 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 shutdown_connections 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fd0f006b500 0x7fd0f006d9c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1040cb190 0x7fd1040cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1040b7d10 0x7fd1040b8170 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 --2- 172.21.15.72:0/2242763694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0ec003f30 0x7fd10414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 >> 172.21.15.72:0/2242763694 conn(0x7fd1040d3cd0 msgr2=0x7fd10415d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:22.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 shutdown_connections 2024-03-06T07:30:22.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:22.224+0000 7fd10ab25640 1 -- 172.21.15.72:0/2242763694 wait complete. 2024-03-06T07:30:22.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:22.439 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:23.440 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:23.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.540+0000 7fc6516ed640 1 Processor -- start 2024-03-06T07:30:23.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.540+0000 7fc6516ed640 1 -- start start 2024-03-06T07:30:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.541+0000 7fc6516ed640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc64c0d2710 0x7fc64c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.541+0000 7fc6516ed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0b3430 con 0x7fc64c0d22b0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.541+0000 7fc6516ed640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64c0b3710 con 0x7fc64c15dfe0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.541+0000 7fc6516ed640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64c0b31e0 con 0x7fc64c0d2710 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.541+0000 7fc64a7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc64c0d22b0 0x7fc64c0b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:40038/0 (socket says 172.21.15.72:40038) 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.541+0000 7fc64a7fc640 1 -- 172.21.15.72:0/3127161235 learned_addr learned my addr 172.21.15.72:0/3127161235 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 --2- 172.21.15.72:0/3127161235 >> v2:172.21.15.146:3300/0 conn(0x7fc64c0d2710 0x7fc64c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686957729 0 0) 0x7fc64c0b3430 con 0x7fc64c0d22b0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc634003520 con 0x7fc64c0d22b0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727777190 0 0) 0x7fc634003520 con 0x7fc64c0d22b0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 >> v2:172.21.15.146:3300/0 conn(0x7fc64c0d2710 msgr2=0x7fc64c0b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 --2- 172.21.15.72:0/3127161235 >> v2:172.21.15.146:3300/0 conn(0x7fc64c0d2710 0x7fc64c0b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 >> v1:172.21.15.146:6789/0 conn(0x7fc64c15dfe0 legacy=0x7fc64c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c0b39f0 con 0x7fc64c0d22b0 2024-03-06T07:30:23.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fc640001670 con 0x7fc64c0d22b0 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 >> v1:172.21.15.72:6789/0 conn(0x7fc64c0d22b0 legacy=0x7fc64c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340039e0 0x7fc634003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc634004800 0x7fc634004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6340051c0 0x7fc63400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0b31e0 con 0x7fc6340051c0 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63400c090 con 0x7fc6340039e0 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc63400c290 con 0x7fc634004800 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6340051c0 0x7fc63400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64affd640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340039e0 0x7fc634003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64a7fc640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc634004800 0x7fc634004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 -- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc634004800 msgr2=0x7fc634004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc634004800 0x7fc634004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 -- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340039e0 msgr2=0x7fc634003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340039e0 0x7fc634003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.542+0000 7fc64b7fe640 1 -- 172.21.15.72:0/3127161235 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634003520 con 0x7fc6340051c0 2024-03-06T07:30:23.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.543+0000 7fc64b7fe640 1 -- 172.21.15.72:0/3127161235 reap_dead start 2024-03-06T07:30:23.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.543+0000 7fc64b7fe640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6340051c0 0x7fc63400b9c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fc644009fc0 tx=0x7fc634005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8f66abd70071f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644009c10 con 0x7fc6340051c0 2024-03-06T07:30:23.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64402cca0 con 0x7fc6340051c0 2024-03-06T07:30:23.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc649ffb640 1 -- 172.21.15.72:0/3127161235 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644003bd0 con 0x7fc6340051c0 2024-03-06T07:30:23.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 -- 172.21.15.72:0/3127161235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6340051c0 msgr2=0x7fc63400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6340051c0 0x7fc63400b9c0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fc644009fc0 tx=0x7fc634005830 comp rx=0 tx=0).stop 2024-03-06T07:30:23.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 -- 172.21.15.72:0/3127161235 shutdown_connections 2024-03-06T07:30:23.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 --2- 172.21.15.72:0/3127161235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc6340051c0 0x7fc63400b9c0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 -- 172.21.15.72:0/3127161235 >> 172.21.15.72:0/3127161235 conn(0x7fc64c0d3cd0 msgr2=0x7fc64c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 -- 172.21.15.72:0/3127161235 shutdown_connections 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.544+0000 7fc6516ed640 1 -- 172.21.15.72:0/3127161235 wait complete. 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.545+0000 7fc6516ed640 1 Processor -- start 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 -- start start 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340051c0 0x7fc64c0cd650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 0x7fc64c15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c0ca160 0x7fc64c0ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0d1c70 con 0x7fc64c15ee10 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0b3d60 con 0x7fc6340051c0 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc6516ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64c0d1df0 con 0x7fc64c0ca160 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc64a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 0x7fc64c15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc64a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 0x7fc64c15b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46680/0 (socket says 172.21.15.72:46680) 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc64a7fc640 1 -- 172.21.15.72:0/102741703 learned_addr learned my addr 172.21.15.72:0/102741703 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc64affd640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340051c0 0x7fc64c0cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.546+0000 7fc64b7fe640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c0ca160 0x7fc64c0ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc64a7fc640 1 -- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c0ca160 msgr2=0x7fc64c0ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc64a7fc640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c0ca160 0x7fc64c0ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc64a7fc640 1 -- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340051c0 msgr2=0x7fc64c0cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc64a7fc640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340051c0 0x7fc64c0cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc64a7fc640 1 -- 172.21.15.72:0/102741703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c09ebd0 con 0x7fc64c15ee10 2024-03-06T07:30:23.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc64a7fc640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 0x7fc64c15b600 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fc64000e800 tx=0x7fc64000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64000c8d0 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc640018070 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64c0cdb90 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64000ccb0 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.547+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64c15f690 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.548+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64c04f330 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.548+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc640013420 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.549+0000 7fc6337fe640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc63806b500 0x7fc63806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.549+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc640089bd0 con 0x7fc64c15ee10 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.549+0000 7fc64affd640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc63806b500 0x7fc63806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:23.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.549+0000 7fc64affd640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc63806b500 0x7fc63806d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc63c000c00 tx=0x7fc63c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:23.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.557+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc64c09ebd0 con 0x7fc64c15ee10 2024-03-06T07:30:23.727 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:23.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.770+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc64c0b2fc0 con 0x7fc64c15ee10 2024-03-06T07:30:23.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.771+0000 7fc6337fe640 1 -- 172.21.15.72:0/102741703 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc6400583c0 con 0x7fc64c15ee10 2024-03-06T07:30:23.773 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:23.773 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:23.774 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:23.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc63806b500 msgr2=0x7fc63806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc63806b500 0x7fc63806d9c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc63c000c00 tx=0x7fc63c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:23.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 msgr2=0x7fc64c15b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:23.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 0x7fc64c15b600 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fc64000e800 tx=0x7fc64000ecc0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 shutdown_connections 2024-03-06T07:30:23.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc63806b500 0x7fc63806d9c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.773+0000 7fc6516ed640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc64c0ca160 0x7fc64c0ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.774+0000 7fc6516ed640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc64c15ee10 0x7fc64c15b600 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.774+0000 7fc6516ed640 1 --2- 172.21.15.72:0/102741703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6340051c0 0x7fc64c0cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.774+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 >> 172.21.15.72:0/102741703 conn(0x7fc64c0d3cd0 msgr2=0x7fc64c0b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.774+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 shutdown_connections 2024-03-06T07:30:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:23.774+0000 7fc6516ed640 1 -- 172.21.15.72:0/102741703 wait complete. 2024-03-06T07:30:23.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:23.988 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:24.989 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:25.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.088+0000 7f6182c15640 1 Processor -- start 2024-03-06T07:30:25.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.088+0000 7f6182c15640 1 -- start start 2024-03-06T07:30:25.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.089+0000 7f6182c15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0d22b0 0x7f617c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.089+0000 7f6182c15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0b3430 con 0x7f617c0b41f0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f6182c15640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c0b3710 con 0x7f617c15dfe0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f6182c15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0d2c50 con 0x7f617c0d22b0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f61811b2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f617c0b41f0 0x7f617c0b4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:40054/0 (socket says 172.21.15.72:40054) 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f61811b2640 1 -- 172.21.15.72:0/681424714 learned_addr learned my addr 172.21.15.72:0/681424714 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617bfff640 1 --2- 172.21.15.72:0/681424714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0d22b0 0x7f617c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120233610 0 0) 0x7f617c0b3430 con 0x7f617c0b41f0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c003520 con 0x7f617c0b41f0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378233661 0 0) 0x7f617c0b3710 con 0x7f617c15dfe0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f617c0b3430 con 0x7f617c15dfe0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617bfff640 1 -- 172.21.15.72:0/681424714 >> v1:172.21.15.146:6789/0 conn(0x7f617c15dfe0 legacy=0x7f617c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617bfff640 1 -- 172.21.15.72:0/681424714 >> v1:172.21.15.72:6789/0 conn(0x7f617c0b41f0 legacy=0x7f617c0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395750728 0 0) 0x7f616c003520 con 0x7f617c0b41f0 2024-03-06T07:30:25.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.090+0000 7f617bfff640 1 -- 172.21.15.72:0/681424714 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0b39f0 con 0x7f617c0d22b0 2024-03-06T07:30:25.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.091+0000 7f617bfff640 1 --2- 172.21.15.72:0/681424714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0d22b0 0x7f617c0d2710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f61680097d0 tx=0x7f616802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a72db1ceb8b719 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168037070 con 0x7f617c0d22b0 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168030510 con 0x7f617c0d22b0 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f617b7fe640 1 -- 172.21.15.72:0/681424714 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61740019f0 con 0x7f617c0d22b0 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 -- 172.21.15.72:0/681424714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0d22b0 msgr2=0x7f617c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 --2- 172.21.15.72:0/681424714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0d22b0 0x7f617c0d2710 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f61680097d0 tx=0x7f616802b800 comp rx=0 tx=0).stop 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 -- 172.21.15.72:0/681424714 shutdown_connections 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 --2- 172.21.15.72:0/681424714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0d22b0 0x7f617c0d2710 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 -- 172.21.15.72:0/681424714 >> 172.21.15.72:0/681424714 conn(0x7f617c0d3cd0 msgr2=0x7f617c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 -- 172.21.15.72:0/681424714 shutdown_connections 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.092+0000 7f6182c15640 1 -- 172.21.15.72:0/681424714 wait complete. 2024-03-06T07:30:25.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.093+0000 7f6182c15640 1 Processor -- start 2024-03-06T07:30:25.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.093+0000 7f6182c15640 1 -- start start 2024-03-06T07:30:25.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f6182c15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0b41f0 0x7f617c14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f6182c15640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f617c0d22b0 0x7f617c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f6182c15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 0x7f617c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f6182c15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0b3d60 con 0x7f617c15dfe0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f6182c15640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0b31e0 con 0x7f617c0d22b0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f6182c15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c0d1c70 con 0x7f617c0b41f0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f61811b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 0x7f617c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f61811b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 0x7f617c0b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46686/0 (socket says 172.21.15.72:46686) 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f61811b2640 1 -- 172.21.15.72:0/2137076639 learned_addr learned my addr 172.21.15.72:0/2137076639 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f617bfff640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f617c0d22b0 0x7f617c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.094+0000 7f61809b1640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0b41f0 0x7f617c14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61811b2640 1 -- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0b41f0 msgr2=0x7f617c14f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61811b2640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0b41f0 0x7f617c14f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61811b2640 1 -- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f617c0d22b0 msgr2=0x7f617c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61811b2640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f617c0d22b0 0x7f617c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61811b2640 1 -- 172.21.15.72:0/2137076639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c15ed30 con 0x7f617c15dfe0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f617bfff640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f617c0d22b0 0x7f617c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61811b2640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 0x7f617c0b2180 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f6174001f30 tx=0x7f617400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174017070 con 0x7f617c15dfe0 2024-03-06T07:30:25.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c15f010 con 0x7f617c15dfe0 2024-03-06T07:30:25.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174012400 con 0x7f617c15dfe0 2024-03-06T07:30:25.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6174012830 con 0x7f617c15dfe0 2024-03-06T07:30:25.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.095+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c160240 con 0x7f617c15dfe0 2024-03-06T07:30:25.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.096+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c04f330 con 0x7f617c15dfe0 2024-03-06T07:30:25.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.097+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f617401a070 con 0x7f617c15dfe0 2024-03-06T07:30:25.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.097+0000 7f61797fa640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f616006b4d0 0x7f616006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:25.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.097+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f617408a160 con 0x7f617c15dfe0 2024-03-06T07:30:25.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.097+0000 7f61809b1640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f616006b4d0 0x7f616006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:25.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.097+0000 7f61809b1640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f616006b4d0 0x7f616006d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f61700011e0 tx=0x7f6170009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:25.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.105+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f6174058960 con 0x7f617c15dfe0 2024-03-06T07:30:25.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.320+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f617c14f5f0 con 0x7f617c15dfe0 2024-03-06T07:30:25.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.321+0000 7f61797fa640 1 -- 172.21.15.72:0/2137076639 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f617405ca00 con 0x7f617c15dfe0 2024-03-06T07:30:25.323 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:25.323 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:25.323 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.322+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f616006b4d0 msgr2=0x7f616006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.322+0000 7f6182c15640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f616006b4d0 0x7f616006d990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f61700011e0 tx=0x7f6170009040 comp rx=0 tx=0).stop 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.322+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 msgr2=0x7f617c0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.322+0000 7f6182c15640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 0x7f617c0b2180 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f6174001f30 tx=0x7f617400a430 comp rx=0 tx=0).stop 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 shutdown_connections 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f616006b4d0 0x7f616006d990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f617c15dfe0 0x7f617c0b2180 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f617c0d22b0 0x7f617c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 --2- 172.21.15.72:0/2137076639 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f617c0b41f0 0x7f617c14f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:25.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 >> 172.21.15.72:0/2137076639 conn(0x7f617c0d3cd0 msgr2=0x7f617c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:25.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 shutdown_connections 2024-03-06T07:30:25.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:25.323+0000 7f6182c15640 1 -- 172.21.15.72:0/2137076639 wait complete. 2024-03-06T07:30:25.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:25.537 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:25.728 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:26.538 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:26.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.639+0000 7f2dacfbd640 1 Processor -- start 2024-03-06T07:30:26.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.639+0000 7f2dacfbd640 1 -- start start 2024-03-06T07:30:26.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2dacfbd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 0x7f2da80b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2dacfbd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80b3430 con 0x7f2da815dfe0 2024-03-06T07:30:26.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2dacfbd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2da80b3710 con 0x7f2da80d22b0 2024-03-06T07:30:26.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2dacfbd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80b31e0 con 0x7f2da80d2710 2024-03-06T07:30:26.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2da815dfe0 0x7f2da80cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:40062/0 (socket says 172.21.15.72:40062) 2024-03-06T07:30:26.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da659c640 1 -- 172.21.15.72:0/541386464 learned_addr learned my addr 172.21.15.72:0/541386464 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393927461 0 0) 0x7f2da80b3430 con 0x7f2da815dfe0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d94003520 con 0x7f2da815dfe0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da6d9d640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 0x7f2da80b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1630802460 0 0) 0x7f2d94003520 con 0x7f2da815dfe0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 msgr2=0x7f2da80b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 0x7f2da80b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 >> v1:172.21.15.146:6789/0 conn(0x7f2da80d22b0 legacy=0x7f2da80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da80b39f0 con 0x7f2da815dfe0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2d98001670 con 0x7f2da815dfe0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 >> v1:172.21.15.72:6789/0 conn(0x7f2da815dfe0 legacy=0x7f2da80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2d94003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d94004800 0x7f2d94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 0x7f2d940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80b31e0 con 0x7f2d940039e0 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9400baa0 con 0x7f2d94004800 2024-03-06T07:30:26.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.640+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9400bca0 con 0x7f2da80d2710 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da659c640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2d94003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da6d9d640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 0x7f2d940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2d9ffff640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d94004800 0x7f2d94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da659c640 1 -- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 msgr2=0x7f2d940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da659c640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80d2710 0x7f2d940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da659c640 1 -- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d94004800 msgr2=0x7f2d94004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da659c640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d94004800 0x7f2d94004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.641+0000 7f2da659c640 1 -- 172.21.15.72:0/541386464 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d94003520 con 0x7f2d940039e0 2024-03-06T07:30:26.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2da659c640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2d94003e20 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c210 tx=0x7f2d94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6eff84d2d713e56 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98007d10 con 0x7f2d940039e0 2024-03-06T07:30:26.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98030070 con 0x7f2d940039e0 2024-03-06T07:30:26.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2da5d9b640 1 -- 172.21.15.72:0/541386464 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98005cf0 con 0x7f2d940039e0 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2dacfbd640 1 -- 172.21.15.72:0/541386464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 msgr2=0x7f2d94003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/541386464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2d94003e20 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f2d9800c210 tx=0x7f2d94005830 comp rx=0 tx=0).stop 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2da6d9d640 1 -- 172.21.15.72:0/541386464 reap_dead start 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2dacfbd640 1 -- 172.21.15.72:0/541386464 shutdown_connections 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2dacfbd640 1 -- 172.21.15.72:0/541386464 >> 172.21.15.72:0/541386464 conn(0x7f2da80d3cd0 msgr2=0x7f2da80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2dacfbd640 1 -- 172.21.15.72:0/541386464 shutdown_connections 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.642+0000 7f2dacfbd640 1 -- 172.21.15.72:0/541386464 wait complete. 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.644+0000 7f2dacfbd640 1 Processor -- start 2024-03-06T07:30:26.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.644+0000 7f2dacfbd640 1 -- start start 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2dacfbd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2da81f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2dacfbd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80cbdf0 0x7f2da81f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2dacfbd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2da81510a0 0x7f2da81f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2dacfbd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80cfa00 con 0x7f2d940039e0 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2dacfbd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80b3d60 con 0x7f2da81510a0 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2dacfbd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80d1c70 con 0x7f2da80cbdf0 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2da81f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2da81f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46696/0 (socket says 172.21.15.72:46696) 2024-03-06T07:30:26.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 -- 172.21.15.72:0/3880813129 learned_addr learned my addr 172.21.15.72:0/3880813129 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da6d9d640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2da81510a0 0x7f2da81f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 -- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80cbdf0 msgr2=0x7f2da81f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80cbdf0 0x7f2da81f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 -- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2da81510a0 msgr2=0x7f2da81f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2da81510a0 0x7f2da81f7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.645+0000 7f2da659c640 1 -- 172.21.15.72:0/3880813129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da81612b0 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.646+0000 7f2da659c640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2da81f3470 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f2d98000f60 tx=0x7f2da80cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.646+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98007d10 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.646+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98004bf0 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.646+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da8161590 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.646+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98007070 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.646+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da8150df0 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.647+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2da804f330 con 0x7f2d940039e0 2024-03-06T07:30:26.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.647+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2d98007210 con 0x7f2d940039e0 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.648+0000 7f2d9effd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d8806b500 0x7f2d8806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.648+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2d980ab810 con 0x7f2d940039e0 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.648+0000 7f2d9ffff640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d8806b500 0x7f2d8806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:26.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.648+0000 7f2d9ffff640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d8806b500 0x7f2d8806d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2da80cc210 tx=0x7f2d90009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:26.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.652+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2d9807a6e0 con 0x7f2d940039e0 2024-03-06T07:30:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.955+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2da804f330 con 0x7f2d940039e0 2024-03-06T07:30:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.955+0000 7f2d9effd640 1 -- 172.21.15.72:0/3880813129 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2d98079fe0 con 0x7f2d940039e0 2024-03-06T07:30:26.957 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:26.957 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d8806b500 msgr2=0x7f2d8806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d8806b500 0x7f2d8806d9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f2da80cc210 tx=0x7f2d90009040 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 msgr2=0x7f2da81f3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2da81f3470 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f2d98000f60 tx=0x7f2da80cfd40 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 shutdown_connections 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2d8806b500 0x7f2d8806d9c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2da81510a0 0x7f2da81f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2da80cbdf0 0x7f2da81f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 --2- 172.21.15.72:0/3880813129 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d940039e0 0x7f2da81f3470 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 >> 172.21.15.72:0/3880813129 conn(0x7f2da80d3cd0 msgr2=0x7f2da80d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 shutdown_connections 2024-03-06T07:30:26.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:26.957+0000 7f2dacfbd640 1 -- 172.21.15.72:0/3880813129 wait complete. 2024-03-06T07:30:27.175 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:27.175 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:27.731 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:28.176 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:28.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.276+0000 7fb8180f4640 1 Processor -- start 2024-03-06T07:30:28.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.276+0000 7fb8180f4640 1 -- start start 2024-03-06T07:30:28.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb8180f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb81015fa10 0x7fb8100ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb8180f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb8100d16e0 0x7fb8100d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb8180f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8100cebe0 con 0x7fb81015bdd0 2024-03-06T07:30:28.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb8180f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8100ceec0 con 0x7fb81015fa10 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb8180f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8100cf1a0 con 0x7fb8100d16e0 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb81568f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb81015fa10 0x7fb8100ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb81568f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb81015fa10 0x7fb8100ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46712/0 (socket says 172.21.15.72:46712) 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.277+0000 7fb81568f640 1 -- 172.21.15.72:0/574081180 learned_addr learned my addr 172.21.15.72:0/574081180 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb816691640 1 --2- 172.21.15.72:0/574081180 >> v2:172.21.15.146:3300/0 conn(0x7fb8100d16e0 0x7fb8100d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb81568f640 1 -- 172.21.15.72:0/574081180 >> v2:172.21.15.146:3300/0 conn(0x7fb8100d16e0 msgr2=0x7fb8100d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb81568f640 1 --2- 172.21.15.72:0/574081180 >> v2:172.21.15.146:3300/0 conn(0x7fb8100d16e0 0x7fb8100d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb81568f640 1 -- 172.21.15.72:0/574081180 >> v1:172.21.15.72:6789/0 conn(0x7fb81015bdd0 legacy=0x7fb8100c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb81568f640 1 -- 172.21.15.72:0/574081180 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100cc8e0 con 0x7fb81015fa10 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb816691640 1 --2- 172.21.15.72:0/574081180 >> v2:172.21.15.146:3300/0 conn(0x7fb8100d16e0 0x7fb8100d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb81568f640 1 --2- 172.21.15.72:0/574081180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015fa10 0x7fb8100ce270 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fb8100c5520 tx=0x7fb80002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ebfaa5997a6f95 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.278+0000 7fb814e8e640 1 -- 172.21.15.72:0/574081180 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800035020 con 0x7fb81015fa10 2024-03-06T07:30:28.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb814e8e640 1 -- 172.21.15.72:0/574081180 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800031870 con 0x7fb81015fa10 2024-03-06T07:30:28.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb814e8e640 1 -- 172.21.15.72:0/574081180 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800031c40 con 0x7fb81015fa10 2024-03-06T07:30:28.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 -- 172.21.15.72:0/574081180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015fa10 msgr2=0x7fb8100ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 --2- 172.21.15.72:0/574081180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015fa10 0x7fb8100ce270 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fb8100c5520 tx=0x7fb80002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:28.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 -- 172.21.15.72:0/574081180 shutdown_connections 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 --2- 172.21.15.72:0/574081180 >> v2:172.21.15.146:3300/0 conn(0x7fb8100d16e0 0x7fb8100d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 --2- 172.21.15.72:0/574081180 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015fa10 0x7fb8100ce270 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 -- 172.21.15.72:0/574081180 >> 172.21.15.72:0/574081180 conn(0x7fb81009a9f0 msgr2=0x7fb81009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 -- 172.21.15.72:0/574081180 shutdown_connections 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.279+0000 7fb8180f4640 1 -- 172.21.15.72:0/574081180 wait complete. 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.280+0000 7fb8180f4640 1 Processor -- start 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.280+0000 7fb8180f4640 1 -- start start 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb8180f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8100d16e0 0x7fb81015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb8180f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 0x7fb81015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb8180f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81015fa10 0x7fb81015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb8180f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100d2030 con 0x7fb81015bdd0 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb8180f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81015c1b0 con 0x7fb81015fa10 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb8180f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100ce7b0 con 0x7fb8100d16e0 2024-03-06T07:30:28.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 0x7fb81015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 0x7fb81015c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46720/0 (socket says 172.21.15.72:46720) 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 -- 172.21.15.72:0/1387113450 learned_addr learned my addr 172.21.15.72:0/1387113450 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb816691640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81015fa10 0x7fb81015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb815e90640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8100d16e0 0x7fb81015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 -- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8100d16e0 msgr2=0x7fb81015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8100d16e0 0x7fb81015aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 -- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81015fa10 msgr2=0x7fb81015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81015fa10 0x7fb81015c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb81568f640 1 -- 172.21.15.72:0/1387113450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100cb550 con 0x7fb81015bdd0 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.281+0000 7fb815e90640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8100d16e0 0x7fb81015aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.282+0000 7fb81568f640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 0x7fb81015c330 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fb80002b950 tx=0x7fb8000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.282+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800040070 con 0x7fb81015bdd0 2024-03-06T07:30:28.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.282+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80003a420 con 0x7fb81015bdd0 2024-03-06T07:30:28.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.282+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8100cb270 con 0x7fb81015bdd0 2024-03-06T07:30:28.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.282+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800035040 con 0x7fb81015bdd0 2024-03-06T07:30:28.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.282+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8101f20d0 con 0x7fb81015bdd0 2024-03-06T07:30:28.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.283+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81004f330 con 0x7fb81015bdd0 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.283+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb800038020 con 0x7fb81015bdd0 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.283+0000 7fb8067fc640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7f406b4d0 0x7fb7f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.284+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb8000ac1b0 con 0x7fb81015bdd0 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.284+0000 7fb815e90640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7f406b4d0 0x7fb7f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:28.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.284+0000 7fb815e90640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7f406b4d0 0x7fb7f406d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb8100cb7e0 tx=0x7fb80c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:28.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.288+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb80007aff0 con 0x7fb81015bdd0 2024-03-06T07:30:28.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.504+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8100cdd60 con 0x7fb81015bdd0 2024-03-06T07:30:28.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.504+0000 7fb8067fc640 1 -- 172.21.15.72:0/1387113450 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fb80007a8f0 con 0x7fb81015bdd0 2024-03-06T07:30:28.507 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:28.507 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:28.507 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7f406b4d0 msgr2=0x7fb7f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7f406b4d0 0x7fb7f406d990 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb8100cb7e0 tx=0x7fb80c009040 comp rx=0 tx=0).stop 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 msgr2=0x7fb81015c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 0x7fb81015c330 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fb80002b950 tx=0x7fb8000046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 shutdown_connections 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb7f406b4d0 0x7fb7f406d990 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81015fa10 0x7fb81015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb81015bdd0 0x7fb81015c330 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 --2- 172.21.15.72:0/1387113450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb8100d16e0 0x7fb81015aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.506+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 >> 172.21.15.72:0/1387113450 conn(0x7fb81009a9f0 msgr2=0x7fb81009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:28.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.507+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 shutdown_connections 2024-03-06T07:30:28.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:28.507+0000 7fb8180f4640 1 -- 172.21.15.72:0/1387113450 wait complete. 2024-03-06T07:30:28.721 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:28.721 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:29.722 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:29.732 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:29.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.830+0000 7f89c2ac9640 1 Processor -- start 2024-03-06T07:30:29.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.830+0000 7f89c2ac9640 1 -- start start 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c2ac9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c2ac9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89bc0d29c0 0x7f89bc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c2ac9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc0b3430 con 0x7f89bc0cf830 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c2ac9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc0b31e0 con 0x7f89bc0d29c0 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c2ac9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0b3d60 con 0x7f89bc15dfe0 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89bc0d29c0 0x7f89bc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f89bc0d29c0 0x7f89bc0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46730/0 (socket says 172.21.15.72:46730) 2024-03-06T07:30:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 -- 172.21.15.72:0/2655344566 learned_addr learned my addr 172.21.15.72:0/2655344566 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c0865640 1 --2- 172.21.15.72:0/2655344566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 -- 172.21.15.72:0/2655344566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc15dfe0 msgr2=0x7f89bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 --2- 172.21.15.72:0/2655344566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 -- 172.21.15.72:0/2655344566 >> v1:172.21.15.146:6789/0 conn(0x7f89bc0cf830 legacy=0x7f89bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.831+0000 7f89c1066640 1 -- 172.21.15.72:0/2655344566 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0b3710 con 0x7f89bc0d29c0 2024-03-06T07:30:29.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.832+0000 7f89c1066640 1 --2- 172.21.15.72:0/2655344566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0b41f0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009e50 tx=0x7f89b402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5014d835d0d274f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.832+0000 7f89bb7fe640 1 -- 172.21.15.72:0/2655344566 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b402cb00 con 0x7f89bc0d29c0 2024-03-06T07:30:29.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.832+0000 7f89bb7fe640 1 -- 172.21.15.72:0/2655344566 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b402cca0 con 0x7f89bc0d29c0 2024-03-06T07:30:29.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.832+0000 7f89bb7fe640 1 -- 172.21.15.72:0/2655344566 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4034610 con 0x7f89bc0d29c0 2024-03-06T07:30:29.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.832+0000 7f89c2ac9640 1 -- 172.21.15.72:0/2655344566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc0d29c0 msgr2=0x7f89bc0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/2655344566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0b41f0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009e50 tx=0x7f89b402bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 -- 172.21.15.72:0/2655344566 shutdown_connections 2024-03-06T07:30:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/2655344566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0b41f0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/2655344566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc15dfe0 0x7f89bc0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 -- 172.21.15.72:0/2655344566 >> 172.21.15.72:0/2655344566 conn(0x7f89bc0d3cd0 msgr2=0x7f89bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 -- 172.21.15.72:0/2655344566 shutdown_connections 2024-03-06T07:30:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.833+0000 7f89c2ac9640 1 -- 172.21.15.72:0/2655344566 wait complete. 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.834+0000 7f89c2ac9640 1 Processor -- start 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.834+0000 7f89c2ac9640 1 -- start start 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c2ac9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc0cf830 0x7f89bc0a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c2ac9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c2ac9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 0x7f89bc15be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c2ac9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0d1df0 con 0x7f89bc15dfe0 2024-03-06T07:30:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c2ac9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0d1c70 con 0x7f89bc0d29c0 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c2ac9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc15e3c0 con 0x7f89bc0cf830 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 0x7f89bc15be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 0x7f89bc15be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46732/0 (socket says 172.21.15.72:46732) 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 -- 172.21.15.72:0/4288913200 learned_addr learned my addr 172.21.15.72:0/4288913200 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c0865640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc0cf830 0x7f89bc0a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89bbfff640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 -- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc0cf830 msgr2=0x7f89bc0a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc0cf830 0x7f89bc0a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 -- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89bc0d29c0 msgr2=0x7f89bc0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:29.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:29.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 -- 172.21.15.72:0/4288913200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc0caf80 con 0x7f89bc15dfe0 2024-03-06T07:30:29.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c0865640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc0cf830 0x7f89bc0a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89bbfff640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.835+0000 7f89c1066640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 0x7f89bc15be30 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009e50 tx=0x7f89b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.836+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4043020 con 0x7f89bc15dfe0 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.836+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89b402ce50 con 0x7f89bc15dfe0 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.836+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc0cb260 con 0x7f89bc15dfe0 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.836+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89b4009510 con 0x7f89bc15dfe0 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.836+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc0cd7a0 con 0x7f89bc15dfe0 2024-03-06T07:30:29.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.837+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc04f330 con 0x7f89bc15dfe0 2024-03-06T07:30:29.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.837+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f89b4046020 con 0x7f89bc15dfe0 2024-03-06T07:30:29.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.837+0000 7f89b97fa640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89a006b4d0 0x7f89a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:29.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.838+0000 7f89c0865640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89a006b4d0 0x7f89a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:29.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.838+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f89b40ac7d0 con 0x7f89bc15dfe0 2024-03-06T07:30:29.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.838+0000 7f89c0865640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89a006b4d0 0x7f89a006d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006fd0 tx=0x7f89b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:29.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:29.842+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f89b404d030 con 0x7f89bc15dfe0 2024-03-06T07:30:30.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.055+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89bc0ca380 con 0x7f89bc15dfe0 2024-03-06T07:30:30.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.056+0000 7f89b97fa640 1 -- 172.21.15.72:0/4288913200 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f89b407b6a0 con 0x7f89bc15dfe0 2024-03-06T07:30:30.058 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:30.058 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:30.058 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89a006b4d0 msgr2=0x7f89a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89a006b4d0 0x7f89a006d990 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f89b0006fd0 tx=0x7f89b0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 msgr2=0x7f89bc15be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 0x7f89bc15be30 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f89b4009e50 tx=0x7f89b4004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 shutdown_connections 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f89a006b4d0 0x7f89a006d990 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89bc15dfe0 0x7f89bc15be30 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89bc0d29c0 0x7f89bc0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 --2- 172.21.15.72:0/4288913200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89bc0cf830 0x7f89bc0a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:30.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 >> 172.21.15.72:0/4288913200 conn(0x7f89bc0d3cd0 msgr2=0x7f89bc0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:30.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 shutdown_connections 2024-03-06T07:30:30.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:30.059+0000 7f89c2ac9640 1 -- 172.21.15.72:0/4288913200 wait complete. 2024-03-06T07:30:30.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:30.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:31.275 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:31.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.374+0000 7fe8ef502640 1 Processor -- start 2024-03-06T07:30:31.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.374+0000 7fe8ef502640 1 -- start start 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.374+0000 7fe8ef502640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8e815dfe0 0x7fe8e80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8ef502640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80b3430 con 0x7fe8e80cf830 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8ef502640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8e80b3710 con 0x7fe8e80d29c0 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8ef502640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8e80b31e0 con 0x7fe8e815dfe0 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8eca9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe8e80cf830 0x7fe8e80d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:40078/0 (socket says 172.21.15.72:40078) 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8eca9d640 1 -- 172.21.15.72:0/496302292 learned_addr learned my addr 172.21.15.72:0/496302292 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8ed29e640 1 --2- 172.21.15.72:0/496302292 >> v2:172.21.15.146:3300/0 conn(0x7fe8e815dfe0 0x7fe8e80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173465850 0 0) 0x7fe8e80b3430 con 0x7fe8e80cf830 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0003520 con 0x7fe8e80cf830 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1493642905 0 0) 0x7fe8d0003520 con 0x7fe8e80cf830 2024-03-06T07:30:31.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 >> v2:172.21.15.146:3300/0 conn(0x7fe8e815dfe0 msgr2=0x7fe8e80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 --2- 172.21.15.72:0/496302292 >> v2:172.21.15.146:3300/0 conn(0x7fe8e815dfe0 0x7fe8e80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 >> v1:172.21.15.146:6789/0 conn(0x7fe8e80d29c0 legacy=0x7fe8e80b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e80b39f0 con 0x7fe8e80cf830 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe8dc001670 con 0x7fe8e80cf830 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 >> v1:172.21.15.72:6789/0 conn(0x7fe8e80cf830 legacy=0x7fe8e80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d00039b0 0x7fe8d0003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d0004800 0x7fe8d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8d00051c0 0x7fe8d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80b31e0 con 0x7fe8d0004800 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000c090 con 0x7fe8d00051c0 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8d000c290 con 0x7fe8d00039b0 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.375+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d0004800 0x7fe8d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8ed29e640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d00039b0 0x7fe8d0003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eda9f640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8d00051c0 0x7fe8d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eca9d640 1 -- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d00039b0 msgr2=0x7fe8d0003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d00039b0 0x7fe8d0003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eca9d640 1 -- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8d00051c0 msgr2=0x7fe8d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8d00051c0 0x7fe8d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eda9f640 1 -- 172.21.15.72:0/496302292 reap_dead start 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.376+0000 7fe8eca9d640 1 -- 172.21.15.72:0/496302292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0003520 con 0x7fe8d0004800 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.377+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d0004800 0x7fe8d0004c80 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00c210 tx=0x7fe8d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516bd23435bcb925 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.377+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc007d10 con 0x7fe8d0004800 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.377+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc030070 con 0x7fe8d0004800 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.377+0000 7fe8e7fff640 1 -- 172.21.15.72:0/496302292 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc005d50 con 0x7fe8d0004800 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 -- 172.21.15.72:0/496302292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d0004800 msgr2=0x7fe8d0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d0004800 0x7fe8d0004c80 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00c210 tx=0x7fe8d0005830 comp rx=0 tx=0).stop 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 -- 172.21.15.72:0/496302292 shutdown_connections 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 --2- 172.21.15.72:0/496302292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8d0004800 0x7fe8d0004c80 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 -- 172.21.15.72:0/496302292 >> 172.21.15.72:0/496302292 conn(0x7fe8e80d3cd0 msgr2=0x7fe8e80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 -- 172.21.15.72:0/496302292 shutdown_connections 2024-03-06T07:30:31.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.378+0000 7fe8ef502640 1 -- 172.21.15.72:0/496302292 wait complete. 2024-03-06T07:30:31.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.379+0000 7fe8ef502640 1 Processor -- start 2024-03-06T07:30:31.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.380+0000 7fe8ef502640 1 -- start start 2024-03-06T07:30:31.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d0004800 0x7fe8e81f4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 0x7fe8e80b88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8e80b8e20 0x7fe8e815b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e815e3c0 con 0x7fe8e81f7160 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e80b3d60 con 0x7fe8e80b8e20 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e815e540 con 0x7fe8d0004800 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 0x7fe8e80b88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 0x7fe8e80b88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46760/0 (socket says 172.21.15.72:46760) 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 -- 172.21.15.72:0/2652706228 learned_addr learned my addr 172.21.15.72:0/2652706228 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 -- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d0004800 msgr2=0x7fe8e81f4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d0004800 0x7fe8e81f4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eda9f640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8e80b8e20 0x7fe8e815b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 -- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8e80b8e20 msgr2=0x7fe8e815b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8e80b8e20 0x7fe8e815b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 -- 172.21.15.72:0/2652706228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8e815c2e0 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8eca9d640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 0x7fe8e80b88e0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc001970 tx=0x7fe8e80b38e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc045070 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8dc0049f0 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc007d10 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.381+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e815c5c0 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.382+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e815e720 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.383+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e804f330 con 0x7fe8e81f7160 2024-03-06T07:30:31.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.383+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8dc007070 con 0x7fe8e81f7160 2024-03-06T07:30:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.384+0000 7fe8e5ffb640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8d406b500 0x7fe8d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.384+0000 7fe8ed29e640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8d406b500 0x7fe8d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.384+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe8dc0aaed0 con 0x7fe8e81f7160 2024-03-06T07:30:31.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.384+0000 7fe8ed29e640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8d406b500 0x7fe8d406d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006e80 tx=0x7fe8e0006df0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:31.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.389+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe8dc079d10 con 0x7fe8e81f7160 2024-03-06T07:30:31.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.602+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8e80b9200 con 0x7fe8e81f7160 2024-03-06T07:30:31.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.603+0000 7fe8e5ffb640 1 -- 172.21.15.72:0/2652706228 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fe8dc079610 con 0x7fe8e81f7160 2024-03-06T07:30:31.605 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:31.605 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:31.605 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8d406b500 msgr2=0x7fe8d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8d406b500 0x7fe8d406d9c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fe8e0006e80 tx=0x7fe8e0006df0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 msgr2=0x7fe8e80b88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 0x7fe8e80b88e0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc001970 tx=0x7fe8e80b38e0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 shutdown_connections 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8d406b500 0x7fe8d406d9c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8e80b8e20 0x7fe8e815b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8e81f7160 0x7fe8e80b88e0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 --2- 172.21.15.72:0/2652706228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8d0004800 0x7fe8e81f4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 >> 172.21.15.72:0/2652706228 conn(0x7fe8e80d3cd0 msgr2=0x7fe8e80d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 shutdown_connections 2024-03-06T07:30:31.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:31.606+0000 7fe8ef502640 1 -- 172.21.15.72:0/2652706228 wait complete. 2024-03-06T07:30:31.734 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:31.819 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:31.820 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:32.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:32.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.920+0000 7fa3cd3bc640 1 Processor -- start 2024-03-06T07:30:32.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3cd3bc640 1 -- start start 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3cd3bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3c815bb20 0x7fa3c80d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3cd3bc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3c80cd8c0 0x7fa3c80d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3cd3bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c8161980 con 0x7fa3c80cfff0 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3cd3bc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c8161c60 con 0x7fa3c80cd8c0 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3cd3bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c8161f40 con 0x7fa3c815bb20 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3c77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3c80cd8c0 0x7fa3c80d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.921+0000 7fa3c77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3c80cd8c0 0x7fa3c80d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33140/0 (socket says 172.21.15.72:33140) 2024-03-06T07:30:32.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c77fe640 1 -- 172.21.15.72:0/1012054184 learned_addr learned my addr 172.21.15.72:0/1012054184 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:32.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c6ffd640 1 --2- 172.21.15.72:0/1012054184 >> v2:172.21.15.146:3300/0 conn(0x7fa3c815bb20 0x7fa3c80d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c77fe640 1 -- 172.21.15.72:0/1012054184 >> v2:172.21.15.146:3300/0 conn(0x7fa3c815bb20 msgr2=0x7fa3c80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c77fe640 1 --2- 172.21.15.72:0/1012054184 >> v2:172.21.15.146:3300/0 conn(0x7fa3c815bb20 0x7fa3c80d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c77fe640 1 -- 172.21.15.72:0/1012054184 >> v1:172.21.15.146:6789/0 conn(0x7fa3c80cfff0 legacy=0x7fa3c80ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:32.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c77fe640 1 -- 172.21.15.72:0/1012054184 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c8162220 con 0x7fa3c80cd8c0 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.922+0000 7fa3c77fe640 1 --2- 172.21.15.72:0/1012054184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c80cd8c0 0x7fa3c80d04a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80d2bc0 tx=0x7fa3c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8641737bcd05e974 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3c67fc640 1 -- 172.21.15.72:0/1012054184 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c002ca80 con 0x7fa3c80cd8c0 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3c67fc640 1 -- 172.21.15.72:0/1012054184 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c002cc20 con 0x7fa3c80cd8c0 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3c67fc640 1 -- 172.21.15.72:0/1012054184 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00343f0 con 0x7fa3c80cd8c0 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/1012054184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c80cd8c0 msgr2=0x7fa3c80d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/1012054184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c80cd8c0 0x7fa3c80d04a0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80d2bc0 tx=0x7fa3c002bee0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/1012054184 shutdown_connections 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/1012054184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c80cd8c0 0x7fa3c80d04a0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/1012054184 >> v2:172.21.15.146:3300/0 conn(0x7fa3c815bb20 0x7fa3c80d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/1012054184 >> 172.21.15.72:0/1012054184 conn(0x7fa3c809a9f0 msgr2=0x7fa3c809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:32.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/1012054184 shutdown_connections 2024-03-06T07:30:32.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.923+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/1012054184 wait complete. 2024-03-06T07:30:32.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.925+0000 7fa3cd3bc640 1 Processor -- start 2024-03-06T07:30:32.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.925+0000 7fa3cd3bc640 1 -- start start 2024-03-06T07:30:32.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.926+0000 7fa3cd3bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cd8c0 0x7fa3c80cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.926+0000 7fa3cd3bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80cfff0 0x7fa3c80c6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3cd3bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 0x7fa3c815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3cd3bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80d24b0 con 0x7fa3c815bb20 2024-03-06T07:30:32.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3cd3bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80d2330 con 0x7fa3c80cfff0 2024-03-06T07:30:32.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3cd3bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80d1980 con 0x7fa3c80cd8c0 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 0x7fa3c815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 0x7fa3c815ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33154/0 (socket says 172.21.15.72:33154) 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 -- 172.21.15.72:0/2103897349 learned_addr learned my addr 172.21.15.72:0/2103897349 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c6ffd640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cd8c0 0x7fa3c80cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3be7fc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80cfff0 0x7fa3c80c6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 -- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cd8c0 msgr2=0x7fa3c80cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cd8c0 0x7fa3c80cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 -- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80cfff0 msgr2=0x7fa3c80c6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80cfff0 0x7fa3c80c6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 -- 172.21.15.72:0/2103897349 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80c75e0 con 0x7fa3c815bb20 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c6ffd640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cd8c0 0x7fa3c80cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3be7fc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80cfff0 0x7fa3c80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3c77fe640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 0x7fa3c815ecf0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0036040 tx=0x7fa3c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0043020 con 0x7fa3c815bb20 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c003e450 con 0x7fa3c815bb20 2024-03-06T07:30:32.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c80c7020 con 0x7fa3c815bb20 2024-03-06T07:30:32.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0009510 con 0x7fa3c815bb20 2024-03-06T07:30:32.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.927+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c815e4f0 con 0x7fa3c815bb20 2024-03-06T07:30:32.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.928+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c80d2120 con 0x7fa3c815bb20 2024-03-06T07:30:32.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.929+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa3c0046020 con 0x7fa3c815bb20 2024-03-06T07:30:32.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.929+0000 7fa3bffff640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa3a806b4d0 0x7fa3a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:32.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.929+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa3c00aca50 con 0x7fa3c815bb20 2024-03-06T07:30:32.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.929+0000 7fa3c6ffd640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa3a806b4d0 0x7fa3a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:32.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.930+0000 7fa3c6ffd640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa3a806b4d0 0x7fa3a806d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8006fd0 tx=0x7fa3b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:32.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:32.934+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa3c80d2120 con 0x7fa3c815bb20 2024-03-06T07:30:33.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.150+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3c804f330 con 0x7fa3c815bb20 2024-03-06T07:30:33.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.151+0000 7fa3bffff640 1 -- 172.21.15.72:0/2103897349 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fa3c004d030 con 0x7fa3c815bb20 2024-03-06T07:30:33.152 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:33.152 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:33.152 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa3a806b4d0 msgr2=0x7fa3a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa3a806b4d0 0x7fa3a806d990 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8006fd0 tx=0x7fa3b8008040 comp rx=0 tx=0).stop 2024-03-06T07:30:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 msgr2=0x7fa3c815ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 0x7fa3c815ecf0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0036040 tx=0x7fa3c0004a40 comp rx=0 tx=0).stop 2024-03-06T07:30:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 shutdown_connections 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa3a806b4d0 0x7fa3a806d990 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3c815bb20 0x7fa3c815ecf0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80cfff0 0x7fa3c80c6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 --2- 172.21.15.72:0/2103897349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cd8c0 0x7fa3c80cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 >> 172.21.15.72:0/2103897349 conn(0x7fa3c809a9f0 msgr2=0x7fa3c809add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 shutdown_connections 2024-03-06T07:30:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:33.153+0000 7fa3cd3bc640 1 -- 172.21.15.72:0/2103897349 wait complete. 2024-03-06T07:30:33.366 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:33.366 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:33.737 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:34.367 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:34.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.494+0000 7f88013b5640 1 Processor -- start 2024-03-06T07:30:34.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.494+0000 7f88013b5640 1 -- start start 2024-03-06T07:30:34.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f88013b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 0x7f87fc0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f88013b5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f87fc0d4760 0x7f87fc0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f88013b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0d25d0 con 0x7f87fc0b4dd0 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f88013b5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87fc0d4bf0 con 0x7f87fc0d4760 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f88013b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0ce9e0 con 0x7f87fc0b4170 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f87fc0b4dd0 0x7f87fc0cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41930/0 (socket says 172.21.15.72:41930) 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fb7fe640 1 -- 172.21.15.72:0/3818719480 learned_addr learned my addr 172.21.15.72:0/3818719480 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87f3fff640 1 --2- 172.21.15.72:0/3818719480 >> v2:172.21.15.146:3300/0 conn(0x7f87fc0d4760 0x7f87fc0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87faffd640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 0x7f87fc0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645403366 0 0) 0x7f87fc0d25d0 con 0x7f87fc0b4dd0 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e8003520 con 0x7f87fc0b4dd0 2024-03-06T07:30:34.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803837559 0 0) 0x7f87e8003520 con 0x7f87fc0b4dd0 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 msgr2=0x7f87fc0b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 0x7f87fc0b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 >> v2:172.21.15.146:3300/0 conn(0x7f87fc0d4760 msgr2=0x7f87fc0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 --2- 172.21.15.72:0/3818719480 >> v2:172.21.15.146:3300/0 conn(0x7f87fc0d4760 0x7f87fc0d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc0d2870 con 0x7f87fc0b4dd0 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f87f40019f0 con 0x7f87fc0b4dd0 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 >> v1:172.21.15.72:6789/0 conn(0x7f87fc0b4dd0 legacy=0x7f87fc0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 0x7f87e80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e8003f30 0x7f87e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8004d40 0x7f87e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0d4bf0 con 0x7f87e8003f30 2024-03-06T07:30:34.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0ce9e0 con 0x7f87e8004d40 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.496+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e800b9c0 con 0x7f87fc0b4170 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87f3fff640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e8003f30 0x7f87e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87faffd640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 0x7f87e80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87fb7fe640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8004d40 0x7f87e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87f3fff640 1 -- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 msgr2=0x7f87e80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87f3fff640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc0b4170 0x7f87e80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87f3fff640 1 -- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8004d40 msgr2=0x7f87e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87f3fff640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8004d40 0x7f87e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.497+0000 7f87f3fff640 1 -- 172.21.15.72:0/3818719480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e8003520 con 0x7f87e8003f30 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f87f3fff640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e8003f30 0x7f87e8004800 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009ac0 tx=0x7f87e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5ab1bf312e78b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4039070 con 0x7f87e8003f30 2024-03-06T07:30:34.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e402c6d0 con 0x7f87e8003f30 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f87fa7fc640 1 -- 172.21.15.72:0/3818719480 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4009690 con 0x7f87e8003f30 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f88013b5640 1 -- 172.21.15.72:0/3818719480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e8003f30 msgr2=0x7f87e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f88013b5640 1 --2- 172.21.15.72:0/3818719480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87e8003f30 0x7f87e8004800 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009ac0 tx=0x7f87e8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f87fb7fe640 1 -- 172.21.15.72:0/3818719480 reap_dead start 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f88013b5640 1 -- 172.21.15.72:0/3818719480 shutdown_connections 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f88013b5640 1 -- 172.21.15.72:0/3818719480 >> 172.21.15.72:0/3818719480 conn(0x7f87fc0d3c40 msgr2=0x7f87fc0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f88013b5640 1 -- 172.21.15.72:0/3818719480 shutdown_connections 2024-03-06T07:30:34.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.498+0000 7f88013b5640 1 -- 172.21.15.72:0/3818719480 wait complete. 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.499+0000 7f88013b5640 1 Processor -- start 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 -- start start 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8003f30 0x7f87fc156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc1f3d20 0x7f87fc0cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 0x7f87fc15da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b47d0 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b4550 con 0x7f87e8003f30 2024-03-06T07:30:34.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f88013b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87fc0b4950 con 0x7f87fc1f3d20 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f87fb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 0x7f87fc15da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f87fb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 0x7f87fc15da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33162/0 (socket says 172.21.15.72:33162) 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f87fb7fe640 1 -- 172.21.15.72:0/4226601284 learned_addr learned my addr 172.21.15.72:0/4226601284 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.500+0000 7f87faffd640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8003f30 0x7f87fc156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87f3fff640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc1f3d20 0x7f87fc0cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87fb7fe640 1 -- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc1f3d20 msgr2=0x7f87fc0cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87fb7fe640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc1f3d20 0x7f87fc0cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87fb7fe640 1 -- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8003f30 msgr2=0x7f87fc156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87fb7fe640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8003f30 0x7f87fc156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87fb7fe640 1 -- 172.21.15.72:0/4226601284 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc158fd0 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87f3fff640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc1f3d20 0x7f87fc0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87faffd640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8003f30 0x7f87fc156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87fb7fe640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 0x7f87fc15da10 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f87fc06e060 tx=0x7f87f4007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f400aca0 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f4018070 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87fc159290 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f400ae60 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.501+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87fc159850 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.502+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87fc04f330 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.502+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f87f4013630 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.503+0000 7f87f37fe640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.503+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f87f408a9b0 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.503+0000 7f87faffd640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:34.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.503+0000 7f87faffd640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87dc06b4d0 0x7f87dc06d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009ae0 tx=0x7f87ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:34.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.507+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f87f4059760 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.721+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87fc0733b0 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.721+0000 7f87f37fe640 1 -- 172.21.15.72:0/4226601284 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f87f4059060 con 0x7f87fc0cc8c0 2024-03-06T07:30:34.722 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:34.722 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:34.723 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87dc06b4d0 msgr2=0x7f87dc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87dc06b4d0 0x7f87dc06d990 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f87ec009ae0 tx=0x7f87ec008040 comp rx=0 tx=0).stop 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 msgr2=0x7f87fc15da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 0x7f87fc15da10 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f87fc06e060 tx=0x7f87f4007b20 comp rx=0 tx=0).stop 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 shutdown_connections 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f87dc06b4d0 0x7f87dc06d990 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87fc0cc8c0 0x7f87fc15da10 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87fc1f3d20 0x7f87fc0cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 --2- 172.21.15.72:0/4226601284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87e8003f30 0x7f87fc156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 >> 172.21.15.72:0/4226601284 conn(0x7f87fc0d3c40 msgr2=0x7f87fc0b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 shutdown_connections 2024-03-06T07:30:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:34.723+0000 7f88013b5640 1 -- 172.21.15.72:0/4226601284 wait complete. 2024-03-06T07:30:34.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:34.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:35.739 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:35.937 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:36.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.037+0000 7fc198156640 1 Processor -- start 2024-03-06T07:30:36.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.037+0000 7fc198156640 1 -- start start 2024-03-06T07:30:36.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc198156640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc19015dfe0 0x7fc1900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc198156640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900d29c0 0x7fc1900b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc198156640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1900b3430 con 0x7fc1900cf830 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc198156640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1900b31e0 con 0x7fc19015dfe0 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc198156640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900b3d60 con 0x7fc1900d29c0 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc19015dfe0 0x7fc1900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc19015dfe0 0x7fc1900cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33166/0 (socket says 172.21.15.72:33166) 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 -- 172.21.15.72:0/3093709374 learned_addr learned my addr 172.21.15.72:0/3093709374 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 -- 172.21.15.72:0/3093709374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900d29c0 msgr2=0x7fc1900b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 --2- 172.21.15.72:0/3093709374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900d29c0 0x7fc1900b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 -- 172.21.15.72:0/3093709374 >> v1:172.21.15.146:6789/0 conn(0x7fc1900cf830 legacy=0x7fc1900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:36.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.038+0000 7fc195ef2640 1 -- 172.21.15.72:0/3093709374 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1900b3710 con 0x7fc19015dfe0 2024-03-06T07:30:36.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.039+0000 7fc195ef2640 1 --2- 172.21.15.72:0/3093709374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc1900cf2f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fc180009870 tx=0x7fc18002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1115d3ee9bc2a19e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.039+0000 7fc194ef0640 1 -- 172.21.15.72:0/3093709374 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180035020 con 0x7fc19015dfe0 2024-03-06T07:30:36.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.039+0000 7fc194ef0640 1 -- 172.21.15.72:0/3093709374 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc180031640 con 0x7fc19015dfe0 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.039+0000 7fc194ef0640 1 -- 172.21.15.72:0/3093709374 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc18002d040 con 0x7fc19015dfe0 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 -- 172.21.15.72:0/3093709374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 msgr2=0x7fc1900cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 --2- 172.21.15.72:0/3093709374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc1900cf2f0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fc180009870 tx=0x7fc18002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 -- 172.21.15.72:0/3093709374 shutdown_connections 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 --2- 172.21.15.72:0/3093709374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900d29c0 0x7fc1900b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 --2- 172.21.15.72:0/3093709374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc1900cf2f0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 -- 172.21.15.72:0/3093709374 >> 172.21.15.72:0/3093709374 conn(0x7fc1900d3cd0 msgr2=0x7fc1900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 -- 172.21.15.72:0/3093709374 shutdown_connections 2024-03-06T07:30:36.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.040+0000 7fc198156640 1 -- 172.21.15.72:0/3093709374 wait complete. 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.041+0000 7fc198156640 1 Processor -- start 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.041+0000 7fc198156640 1 -- start start 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc198156640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900cf830 0x7fc1901f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc198156640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1900d29c0 0x7fc1900cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc198156640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc190160ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc198156640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900d1df0 con 0x7fc19015dfe0 2024-03-06T07:30:36.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc198156640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1900d1c70 con 0x7fc1900d29c0 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc198156640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc19015e3c0 con 0x7fc1900cf830 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc190160ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc190160ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33176/0 (socket says 172.21.15.72:33176) 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 -- 172.21.15.72:0/2163387929 learned_addr learned my addr 172.21.15.72:0/2163387929 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc195ef2640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900cf830 0x7fc1901f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1956f1640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1900d29c0 0x7fc1900cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 -- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900cf830 msgr2=0x7fc1901f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900cf830 0x7fc1901f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 -- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1900d29c0 msgr2=0x7fc1900cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1900d29c0 0x7fc1900cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1966f3640 1 -- 172.21.15.72:0/2163387929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19015fc80 con 0x7fc19015dfe0 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc195ef2640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900cf830 0x7fc1901f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:36.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.042+0000 7fc1956f1640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1900d29c0 0x7fc1900cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:36.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.043+0000 7fc1966f3640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc190160ca0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b970 tx=0x7fc18800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.043+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188019070 con 0x7fc19015dfe0 2024-03-06T07:30:36.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.043+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc18800cb60 con 0x7fc19015dfe0 2024-03-06T07:30:36.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.043+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1900cb780 con 0x7fc19015dfe0 2024-03-06T07:30:36.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.043+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188014440 con 0x7fc19015dfe0 2024-03-06T07:30:36.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.043+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1900cbd40 con 0x7fc19015dfe0 2024-03-06T07:30:36.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.044+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19004f330 con 0x7fc19015dfe0 2024-03-06T07:30:36.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.044+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc18801c070 con 0x7fc19015dfe0 2024-03-06T07:30:36.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.045+0000 7fc1867fc640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc17406b4d0 0x7fc17406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:36.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.045+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc188089f80 con 0x7fc19015dfe0 2024-03-06T07:30:36.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.045+0000 7fc195ef2640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc17406b4d0 0x7fc17406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:36.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.045+0000 7fc195ef2640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc17406b4d0 0x7fc17406d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc180036000 tx=0x7fc18003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:36.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.049+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc188058dc0 con 0x7fc19015dfe0 2024-03-06T07:30:36.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.264+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1901f3580 con 0x7fc19015dfe0 2024-03-06T07:30:36.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.264+0000 7fc1867fc640 1 -- 172.21.15.72:0/2163387929 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc1880586c0 con 0x7fc19015dfe0 2024-03-06T07:30:36.267 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:36.267 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:36.267 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:36.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc17406b4d0 msgr2=0x7fc17406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc17406b4d0 0x7fc17406d990 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc180036000 tx=0x7fc18003c040 comp rx=0 tx=0).stop 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 msgr2=0x7fc190160ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc190160ca0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b970 tx=0x7fc18800be30 comp rx=0 tx=0).stop 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 shutdown_connections 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc17406b4d0 0x7fc17406d990 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc19015dfe0 0x7fc190160ca0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1900d29c0 0x7fc1900cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 --2- 172.21.15.72:0/2163387929 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1900cf830 0x7fc1901f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 >> 172.21.15.72:0/2163387929 conn(0x7fc1900d3cd0 msgr2=0x7fc1900d0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 shutdown_connections 2024-03-06T07:30:36.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:36.266+0000 7fc198156640 1 -- 172.21.15.72:0/2163387929 wait complete. 2024-03-06T07:30:36.480 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:36.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:37.481 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:37.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.581+0000 7f5b4f15e640 1 Processor -- start 2024-03-06T07:30:37.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.581+0000 7f5b4f15e640 1 -- start start 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4f15e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b4815dfe0 0x7f5b480cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4f15e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5b480d29c0 0x7f5b480b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4f15e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b480b3430 con 0x7f5b480cf830 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4f15e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b480b31e0 con 0x7f5b480d29c0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4f15e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480b3d60 con 0x7f5b4815dfe0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4d6fb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5b480d29c0 0x7f5b480b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4d6fb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5b480d29c0 0x7f5b480b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33182/0 (socket says 172.21.15.72:33182) 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.582+0000 7f5b4d6fb640 1 -- 172.21.15.72:0/1497204976 learned_addr learned my addr 172.21.15.72:0/1497204976 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.583+0000 7f5b4cefa640 1 --2- 172.21.15.72:0/1497204976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b4815dfe0 0x7f5b480cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.583+0000 7f5b4d6fb640 1 -- 172.21.15.72:0/1497204976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b4815dfe0 msgr2=0x7f5b480cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.583+0000 7f5b4d6fb640 1 --2- 172.21.15.72:0/1497204976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b4815dfe0 0x7f5b480cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.583+0000 7f5b4d6fb640 1 -- 172.21.15.72:0/1497204976 >> v1:172.21.15.146:6789/0 conn(0x7f5b480cf830 legacy=0x7f5b480d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.583+0000 7f5b4d6fb640 1 -- 172.21.15.72:0/1497204976 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b480b3710 con 0x7f5b480d29c0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.583+0000 7f5b4d6fb640 1 --2- 172.21.15.72:0/1497204976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480d29c0 0x7f5b480b41f0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009e50 tx=0x7f5b4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4939a1f7288b881b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b477fe640 1 -- 172.21.15.72:0/1497204976 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4002cb00 con 0x7f5b480d29c0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b477fe640 1 -- 172.21.15.72:0/1497204976 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4002cca0 con 0x7f5b480d29c0 2024-03-06T07:30:37.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b477fe640 1 -- 172.21.15.72:0/1497204976 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b40034610 con 0x7f5b480d29c0 2024-03-06T07:30:37.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1497204976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480d29c0 msgr2=0x7f5b480b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1497204976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480d29c0 0x7f5b480b41f0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009e50 tx=0x7f5b4002bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:37.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1497204976 shutdown_connections 2024-03-06T07:30:37.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1497204976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480d29c0 0x7f5b480b41f0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1497204976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b4815dfe0 0x7f5b480cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1497204976 >> 172.21.15.72:0/1497204976 conn(0x7f5b480d3cd0 msgr2=0x7f5b480d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1497204976 shutdown_connections 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.584+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1497204976 wait complete. 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.585+0000 7f5b4f15e640 1 Processor -- start 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.585+0000 7f5b4f15e640 1 -- start start 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4f15e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 0x7f5b48155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4f15e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b480d29c0 0x7f5b4815e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4f15e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b4815dfe0 0x7f5b4815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4f15e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480d1df0 con 0x7f5b480cf830 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4f15e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b480d1c70 con 0x7f5b4815dfe0 2024-03-06T07:30:37.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4f15e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4815e3c0 con 0x7f5b480d29c0 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4cefa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 0x7f5b48155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4cefa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 0x7f5b48155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33198/0 (socket says 172.21.15.72:33198) 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4cefa640 1 -- 172.21.15.72:0/1957999491 learned_addr learned my addr 172.21.15.72:0/1957999491 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b4d6fb640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b4815dfe0 0x7f5b4815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.586+0000 7f5b47fff640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b480d29c0 0x7f5b4815e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4cefa640 1 -- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b480d29c0 msgr2=0x7f5b4815e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4cefa640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b480d29c0 0x7f5b4815e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4cefa640 1 -- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b4815dfe0 msgr2=0x7f5b4815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4cefa640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b4815dfe0 0x7f5b4815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4cefa640 1 -- 172.21.15.72:0/1957999491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b481563b0 con 0x7f5b480cf830 2024-03-06T07:30:37.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4d6fb640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b4815dfe0 0x7f5b4815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4cefa640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 0x7f5b48155b90 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800d5f0 tx=0x7f5b3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b380125f0 con 0x7f5b480cf830 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b38019070 con 0x7f5b480cf830 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b480b9a50 con 0x7f5b480cf830 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b380129d0 con 0x7f5b480cf830 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.587+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4815f5e0 con 0x7f5b480cf830 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.588+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4804f330 con 0x7f5b480cf830 2024-03-06T07:30:37.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.588+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5b38012b70 con 0x7f5b480cf830 2024-03-06T07:30:37.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.589+0000 7f5b457fa640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b2806b4d0 0x7f5b2806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:37.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.589+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5b3808a5b0 con 0x7f5b480cf830 2024-03-06T07:30:37.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.589+0000 7f5b47fff640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b2806b4d0 0x7f5b2806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:37.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.589+0000 7f5b47fff640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b2806b4d0 0x7f5b2806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5b480b5030 tx=0x7f5b30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:37.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.593+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5b38059480 con 0x7f5b480cf830 2024-03-06T07:30:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:37.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.833+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b4806c570 con 0x7f5b480cf830 2024-03-06T07:30:37.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.833+0000 7f5b457fa640 1 -- 172.21.15.72:0/1957999491 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5b38058d80 con 0x7f5b480cf830 2024-03-06T07:30:37.836 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:37.836 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:37.836 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:37.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b2806b4d0 msgr2=0x7f5b2806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b2806b4d0 0x7f5b2806d990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5b480b5030 tx=0x7f5b30009040 comp rx=0 tx=0).stop 2024-03-06T07:30:37.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 msgr2=0x7f5b48155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:37.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 0x7f5b48155b90 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f5b3800d5f0 tx=0x7f5b3800dab0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 shutdown_connections 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5b2806b4d0 0x7f5b2806d990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b4815dfe0 0x7f5b4815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b480d29c0 0x7f5b4815e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 --2- 172.21.15.72:0/1957999491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5b480cf830 0x7f5b48155b90 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 >> 172.21.15.72:0/1957999491 conn(0x7f5b480d3cd0 msgr2=0x7f5b4806ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 shutdown_connections 2024-03-06T07:30:37.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:37.835+0000 7f5b4f15e640 1 -- 172.21.15.72:0/1957999491 wait complete. 2024-03-06T07:30:38.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:38.050 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:39.052 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:39.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.166+0000 7f572d223640 1 Processor -- start 2024-03-06T07:30:39.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.166+0000 7f572d223640 1 -- start start 2024-03-06T07:30:39.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f572d223640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57281515e0 0x7f57280cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f572d223640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5728071f40 con 0x7f572815b430 2024-03-06T07:30:39.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f572d223640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572815b8b0 con 0x7f57280cc340 2024-03-06T07:30:39.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f572d223640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5728070a70 con 0x7f57281515e0 2024-03-06T07:30:39.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f572659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f572815b430 0x7f5728071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41932/0 (socket says 172.21.15.72:41932) 2024-03-06T07:30:39.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f572659c640 1 -- 172.21.15.72:0/2252014051 learned_addr learned my addr 172.21.15.72:0/2252014051 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.167+0000 7f5726d9d640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57281515e0 0x7f57280cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504714111 0 0) 0x7f5728071f40 con 0x7f572815b430 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5714003520 con 0x7f572815b430 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459937138 0 0) 0x7f572815b8b0 con 0x7f57280cc340 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728071f40 con 0x7f57280cc340 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3430861284 0 0) 0x7f5714003520 con 0x7f572815b430 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57281515e0 msgr2=0x7f57280cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57281515e0 0x7f57280cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 >> v1:172.21.15.146:6789/0 conn(0x7f57280cc340 legacy=0x7f57280cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5728070d50 con 0x7f572815b430 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f5710001670 con 0x7f572815b430 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 >> v1:172.21.15.72:6789/0 conn(0x7f572815b430 legacy=0x7f5728071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5714003b70 0x7f5714003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5714004800 0x7f5714004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f571400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5728070a70 con 0x7f57140051c0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571400e1c0 con 0x7f5714003b70 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571400e3c0 con 0x7f5714004800 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5726d9d640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f57281515e0 0x7f57280cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572759e640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f571400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572659c640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5714004800 0x7f5714004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f5726d9d640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5714003b70 0x7f5714003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572759e640 1 -- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5714004800 msgr2=0x7f5714004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572759e640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5714004800 0x7f5714004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572759e640 1 -- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5714003b70 msgr2=0x7f5714003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572759e640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5714003b70 0x7f5714003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.168+0000 7f572759e640 1 -- 172.21.15.72:0/2252014051 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714003520 con 0x7f57140051c0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.169+0000 7f572759e640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f571400dbd0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f57180018c0 tx=0x7f5714005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cdd5f9da00d74f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.169+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718037070 con 0x7f57140051c0 2024-03-06T07:30:39.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.169+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57180052f0 con 0x7f57140051c0 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.169+0000 7f5725d9b640 1 -- 172.21.15.72:0/2252014051 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57180014e0 con 0x7f57140051c0 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572d223640 1 -- 172.21.15.72:0/2252014051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 msgr2=0x7f571400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572d223640 1 --2- 172.21.15.72:0/2252014051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f571400dbd0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f57180018c0 tx=0x7f5714005830 comp rx=0 tx=0).stop 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572759e640 1 -- 172.21.15.72:0/2252014051 reap_dead start 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572d223640 1 -- 172.21.15.72:0/2252014051 shutdown_connections 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572d223640 1 -- 172.21.15.72:0/2252014051 >> 172.21.15.72:0/2252014051 conn(0x7f57280cf8a0 msgr2=0x7f57280cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572d223640 1 -- 172.21.15.72:0/2252014051 shutdown_connections 2024-03-06T07:30:39.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.170+0000 7f572d223640 1 -- 172.21.15.72:0/2252014051 wait complete. 2024-03-06T07:30:39.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.172+0000 7f572d223640 1 Processor -- start 2024-03-06T07:30:39.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.172+0000 7f572d223640 1 -- start start 2024-03-06T07:30:39.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572d223640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f572813a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572d223640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572815e850 0x7f572815be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572d223640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f572815c3a0 0x7f572813c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572d223640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57280cc770 con 0x7f57140051c0 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572d223640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5728071310 con 0x7f572815e850 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572d223640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57280cc8f0 con 0x7f572815c3a0 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f5726d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f572813a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f5726d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f572813a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33220/0 (socket says 172.21.15.72:33220) 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f5726d9d640 1 -- 172.21.15.72:0/653607555 learned_addr learned my addr 172.21.15.72:0/653607555 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572659c640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572815e850 0x7f572815be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.173+0000 7f572759e640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f572815c3a0 0x7f572813c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f5726d9d640 1 -- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f572815c3a0 msgr2=0x7f572813c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f5726d9d640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f572815c3a0 0x7f572813c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f5726d9d640 1 -- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572815e850 msgr2=0x7f572815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f5726d9d640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572815e850 0x7f572815be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f5726d9d640 1 -- 172.21.15.72:0/653607555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572813b4a0 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f572759e640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f572815c3a0 0x7f572813c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f5726d9d640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f572813a8c0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f5720007f80 tx=0x7f5720011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f572001a020 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5720016470 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5720016840 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572813b780 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.174+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572813bd40 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.175+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5728068840 con 0x7f57140051c0 2024-03-06T07:30:39.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.176+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5720005d10 con 0x7f57140051c0 2024-03-06T07:30:39.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.176+0000 7f571f7fe640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:39.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.176+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f572008b160 con 0x7f57140051c0 2024-03-06T07:30:39.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.176+0000 7f572659c640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:39.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.176+0000 7f572659c640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f57280cfd60 tx=0x7f5710007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:39.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.180+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5720024030 con 0x7f57140051c0 2024-03-06T07:30:39.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.387+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f572813df30 con 0x7f57140051c0 2024-03-06T07:30:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.388+0000 7f571f7fe640 1 -- 172.21.15.72:0/653607555 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f5720059fa0 con 0x7f57140051c0 2024-03-06T07:30:39.390 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:39.390 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:39.390 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 msgr2=0x7f570806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f57280cfd60 tx=0x7f5710007040 comp rx=0 tx=0).stop 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 msgr2=0x7f572813a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f572813a8c0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f5720007f80 tx=0x7f5720011660 comp rx=0 tx=0).stop 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 shutdown_connections 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f570806b4d0 0x7f570806d990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f572815c3a0 0x7f572813c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f572815e850 0x7f572815be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 --2- 172.21.15.72:0/653607555 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57140051c0 0x7f572813a8c0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:39.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 >> 172.21.15.72:0/653607555 conn(0x7f57280cf8a0 msgr2=0x7f5728072110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:39.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 shutdown_connections 2024-03-06T07:30:39.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:39.390+0000 7f572d223640 1 -- 172.21.15.72:0/653607555 wait complete. 2024-03-06T07:30:39.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:39.604 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:39.743 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:40.605 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:40.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.705+0000 7f03d0313640 1 Processor -- start 2024-03-06T07:30:40.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.705+0000 7f03d0313640 1 -- start start 2024-03-06T07:30:40.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03d0313640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c815dfe0 0x7f03c80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03d0313640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03c80cf830 0x7f03c80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03d0313640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80b3430 con 0x7f03c80d27f0 2024-03-06T07:30:40.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03d0313640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c80d2c50 con 0x7f03c80cf830 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03d0313640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80b31e0 con 0x7f03c815dfe0 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03ce0af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c815dfe0 0x7f03c80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03cd8ae640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03c80cf830 0x7f03c80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03ce0af640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c815dfe0 0x7f03c80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:41102/0 (socket says 172.21.15.72:41102) 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03cd8ae640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f03c80cf830 0x7f03c80d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:59860/0 (socket says 172.21.15.72:59860) 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03ce8b0640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f03c80d27f0 0x7f03c80d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:46464/0 (socket says 172.21.15.72:46464) 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03ce0af640 1 -- 172.21.15.72:0/2640740352 learned_addr learned my addr 172.21.15.72:0/2640740352 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03cd8ae640 1 -- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c815dfe0 msgr2=0x7f03c80cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03cd8ae640 1 --2- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c815dfe0 0x7f03c80cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.706+0000 7f03cd8ae640 1 -- 172.21.15.72:0/2640740352 >> v1:172.21.15.146:6789/0 conn(0x7f03c80d27f0 legacy=0x7f03c80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.707+0000 7f03cd8ae640 1 -- 172.21.15.72:0/2640740352 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80b3710 con 0x7f03c80cf830 2024-03-06T07:30:40.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.707+0000 7f03cd8ae640 1 --2- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c80d22b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1eb16bf3a9b558c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.707+0000 7f03cd0ad640 1 -- 172.21.15.72:0/2640740352 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8035020 con 0x7f03c80cf830 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03cd0ad640 1 -- 172.21.15.72:0/2640740352 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b8031830 con 0x7f03c80cf830 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03cd0ad640 1 -- 172.21.15.72:0/2640740352 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031c00 con 0x7f03c80cf830 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 -- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 msgr2=0x7f03c80d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 --2- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c80d22b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f03b80097a0 tx=0x7f03b802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 -- 172.21.15.72:0/2640740352 shutdown_connections 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 --2- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c80d22b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 --2- 172.21.15.72:0/2640740352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c815dfe0 0x7f03c80cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 -- 172.21.15.72:0/2640740352 >> 172.21.15.72:0/2640740352 conn(0x7f03c80d3cd0 msgr2=0x7f03c80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 -- 172.21.15.72:0/2640740352 shutdown_connections 2024-03-06T07:30:40.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.708+0000 7f03d0313640 1 -- 172.21.15.72:0/2640740352 wait complete. 2024-03-06T07:30:40.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.709+0000 7f03d0313640 1 Processor -- start 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.709+0000 7f03d0313640 1 -- start start 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03d0313640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c8154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03d0313640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c80d27f0 0x7f03c80cb620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03d0313640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 0x7f03c80cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03d0313640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80d1c70 con 0x7f03c815dfe0 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03d0313640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80b3d60 con 0x7f03c80cf830 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03d0313640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80d1df0 con 0x7f03c80d27f0 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 0x7f03c80cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 0x7f03c80cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33232/0 (socket says 172.21.15.72:33232) 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 -- 172.21.15.72:0/2088972287 learned_addr learned my addr 172.21.15.72:0/2088972287 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03cd8ae640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c80d27f0 0x7f03c80cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce0af640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c8154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 -- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c80d27f0 msgr2=0x7f03c80cb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c80d27f0 0x7f03c80cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 -- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 msgr2=0x7f03c8154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c8154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce8b0640 1 -- 172.21.15.72:0/2088972287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80b7c50 con 0x7f03c815dfe0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.710+0000 7f03ce0af640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c8154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.711+0000 7f03ce8b0640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 0x7f03c80cbb60 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f03c400c780 tx=0x7f03c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.711+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4017070 con 0x7f03c815dfe0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.711+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c400a640 con 0x7f03c815dfe0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.711+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c80b7f30 con 0x7f03c815dfe0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.711+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c400aa30 con 0x7f03c815dfe0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.711+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c80b84f0 con 0x7f03c815dfe0 2024-03-06T07:30:40.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.712+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c804f330 con 0x7f03c815dfe0 2024-03-06T07:30:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.712+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f03c401a070 con 0x7f03c815dfe0 2024-03-06T07:30:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.713+0000 7f03beffd640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03ac06b4d0 0x7f03ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.713+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f03c4089770 con 0x7f03c815dfe0 2024-03-06T07:30:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.713+0000 7f03ce0af640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03ac06b4d0 0x7f03ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:40.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.713+0000 7f03ce0af640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03ac06b4d0 0x7f03ac06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009a20 tx=0x7f03c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:40.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.717+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f03c4057f60 con 0x7f03c815dfe0 2024-03-06T07:30:40.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.932+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03c80ceb50 con 0x7f03c815dfe0 2024-03-06T07:30:40.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.932+0000 7f03beffd640 1 -- 172.21.15.72:0/2088972287 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f03c405c000 con 0x7f03c815dfe0 2024-03-06T07:30:40.935 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:40.935 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:40.935 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:40.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03ac06b4d0 msgr2=0x7f03ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03ac06b4d0 0x7f03ac06d990 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009a20 tx=0x7f03c0008040 comp rx=0 tx=0).stop 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 msgr2=0x7f03c80cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 0x7f03c80cbb60 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f03c400c780 tx=0x7f03c400cc40 comp rx=0 tx=0).stop 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 shutdown_connections 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f03ac06b4d0 0x7f03ac06d990 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f03c815dfe0 0x7f03c80cbb60 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f03c80d27f0 0x7f03c80cb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 --2- 172.21.15.72:0/2088972287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f03c80cf830 0x7f03c8154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 >> 172.21.15.72:0/2088972287 conn(0x7f03c80d3cd0 msgr2=0x7f03c806cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 shutdown_connections 2024-03-06T07:30:40.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:40.935+0000 7f03d0313640 1 -- 172.21.15.72:0/2088972287 wait complete. 2024-03-06T07:30:41.152 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:41.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:41.745 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:42.153 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:42.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.253+0000 7f3834932640 1 Processor -- start 2024-03-06T07:30:42.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.253+0000 7f3834932640 1 -- start start 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f3834932640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 0x7f38300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f3834932640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300b3430 con 0x7f38300cf830 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f3834932640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38300b3710 con 0x7f38300d29c0 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f3834932640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38300b31e0 con 0x7f383015dfe0 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f3827fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f38300cf830 0x7f38300d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41948/0 (socket says 172.21.15.72:41948) 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f3827fff640 1 -- 172.21.15.72:0/2537460722 learned_addr learned my addr 172.21.15.72:0/2537460722 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382e59c640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 0x7f38300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962436230 0 0) 0x7f38300b3430 con 0x7f38300cf830 2024-03-06T07:30:42.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381c003520 con 0x7f38300cf830 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284481099 0 0) 0x7f38300b3710 con 0x7f38300d29c0 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38300b3430 con 0x7f38300d29c0 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1606613309 0 0) 0x7f381c003520 con 0x7f38300cf830 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 msgr2=0x7f38300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 0x7f38300cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 >> v1:172.21.15.146:6789/0 conn(0x7f38300d29c0 legacy=0x7f38300b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300b39f0 con 0x7f38300cf830 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3818001670 con 0x7f38300cf830 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.254+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 >> v1:172.21.15.72:6789/0 conn(0x7f38300cf830 legacy=0x7f38300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382dd9b640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c003b70 0x7f381c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382dd9b640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f381c004800 0x7f381c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382dd9b640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 0x7f381c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300b31e0 con 0x7f381c004800 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c00ba70 con 0x7f381c003b70 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c00bc70 con 0x7f383015dfe0 2024-03-06T07:30:42.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f3827fff640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f381c004800 0x7f381c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382e59c640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c003b70 0x7f381c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f382ed9d640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 0x7f381c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f3827fff640 1 -- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 msgr2=0x7f381c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f3827fff640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f383015dfe0 0x7f381c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f3827fff640 1 -- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c003b70 msgr2=0x7f381c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f3827fff640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f381c003b70 0x7f381c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.255+0000 7f3827fff640 1 -- 172.21.15.72:0/2537460722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c003520 con 0x7f381c004800 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3827fff640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f381c004800 0x7f381c004c80 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3818007fb0 tx=0x7f381c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7beeb8487b5025d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818007c00 con 0x7f381c004800 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3818030070 con 0x7f381c004800 2024-03-06T07:30:42.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f382dd9b640 1 -- 172.21.15.72:0/2537460722 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818005d10 con 0x7f381c004800 2024-03-06T07:30:42.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3834932640 1 -- 172.21.15.72:0/2537460722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f381c004800 msgr2=0x7f381c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3834932640 1 --2- 172.21.15.72:0/2537460722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f381c004800 0x7f381c004c80 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3818007fb0 tx=0x7f381c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f382ed9d640 1 -- 172.21.15.72:0/2537460722 reap_dead start 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3834932640 1 -- 172.21.15.72:0/2537460722 shutdown_connections 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3834932640 1 -- 172.21.15.72:0/2537460722 >> 172.21.15.72:0/2537460722 conn(0x7f38300d3cd0 msgr2=0x7f38300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3834932640 1 -- 172.21.15.72:0/2537460722 shutdown_connections 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.256+0000 7f3834932640 1 -- 172.21.15.72:0/2537460722 wait complete. 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 Processor -- start 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 -- start start 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381c004800 0x7f38300cd960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 0x7f38300cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f383006bc20 0x7f383006c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383015e3c0 con 0x7f383006d1b0 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38300b3d60 con 0x7f383006bc20 2024-03-06T07:30:42.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.258+0000 7f3834932640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383015e540 con 0x7f381c004800 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 0x7f38300cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 0x7f38300cada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33250/0 (socket says 172.21.15.72:33250) 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 -- 172.21.15.72:0/771395588 learned_addr learned my addr 172.21.15.72:0/771395588 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f382e59c640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381c004800 0x7f38300cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f382ed9d640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f383006bc20 0x7f383006c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 -- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381c004800 msgr2=0x7f38300cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381c004800 0x7f38300cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 -- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f383006bc20 msgr2=0x7f383006c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f383006bc20 0x7f383006c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 -- 172.21.15.72:0/771395588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38300cb710 con 0x7f383006d1b0 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f382ed9d640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f383006bc20 0x7f383006c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3827fff640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 0x7f38300cada0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f381800c1d0 tx=0x7f38300cfd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818045070 con 0x7f383006d1b0 2024-03-06T07:30:42.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38300cb9f0 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3818007480 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3818007c00 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.259+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3830155820 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.260+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383004f330 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.261+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f381802c770 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.261+0000 7f3826ffd640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f381006b500 0x7f381006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.261+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f38180ab040 con 0x7f383006d1b0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.261+0000 7f382e59c640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f381006b500 0x7f381006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:42.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.261+0000 7f382e59c640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f381006b500 0x7f381006d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3820009a60 tx=0x7f3820009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:42.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.269+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f38180797b0 con 0x7f383006d1b0 2024-03-06T07:30:42.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.477+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38300b43d0 con 0x7f383006d1b0 2024-03-06T07:30:42.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.477+0000 7f3826ffd640 1 -- 172.21.15.72:0/771395588 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f381807d850 con 0x7f383006d1b0 2024-03-06T07:30:42.480 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:42.480 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:42.480 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:42.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f381006b500 msgr2=0x7f381006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f381006b500 0x7f381006d9c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f3820009a60 tx=0x7f3820009190 comp rx=0 tx=0).stop 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 msgr2=0x7f38300cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 0x7f38300cada0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f381800c1d0 tx=0x7f38300cfd40 comp rx=0 tx=0).stop 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 shutdown_connections 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f381006b500 0x7f381006d9c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f383006bc20 0x7f383006c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f383006d1b0 0x7f38300cada0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 --2- 172.21.15.72:0/771395588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f381c004800 0x7f38300cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 >> 172.21.15.72:0/771395588 conn(0x7f38300d3cd0 msgr2=0x7f38300cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 shutdown_connections 2024-03-06T07:30:42.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:42.479+0000 7f3834932640 1 -- 172.21.15.72:0/771395588 wait complete. 2024-03-06T07:30:42.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:42.694 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:43.695 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:43.747 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:43.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff335dac640 1 Processor -- start 2024-03-06T07:30:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff335dac640 1 -- start start 2024-03-06T07:30:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff335dac640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff33015dfe0 0x7ff3300cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff335dac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3300b3430 con 0x7ff3300d29c0 2024-03-06T07:30:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff335dac640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3300b3710 con 0x7ff3300cf830 2024-03-06T07:30:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff335dac640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3300b31e0 con 0x7ff33015dfe0 2024-03-06T07:30:43.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff32ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff3300d29c0 0x7ff3300b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46982/0 (socket says 172.21.15.72:46982) 2024-03-06T07:30:43.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.795+0000 7ff32ffff640 1 -- 172.21.15.72:0/441152930 learned_addr learned my addr 172.21.15.72:0/441152930 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/441152930 >> v2:172.21.15.146:3300/0 conn(0x7ff33015dfe0 0x7ff3300cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987502460 0 0) 0x7ff3300b3430 con 0x7ff3300d29c0 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff318003520 con 0x7ff3300d29c0 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115181373 0 0) 0x7ff318003520 con 0x7ff3300d29c0 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 >> v2:172.21.15.146:3300/0 conn(0x7ff33015dfe0 msgr2=0x7ff3300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 --2- 172.21.15.72:0/441152930 >> v2:172.21.15.146:3300/0 conn(0x7ff33015dfe0 0x7ff3300cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 >> v1:172.21.15.146:6789/0 conn(0x7ff3300cf830 legacy=0x7ff3300d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3300b39f0 con 0x7ff3300d29c0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7ff3280019f0 con 0x7ff3300d29c0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 >> v1:172.21.15.72:6789/0 conn(0x7ff3300d29c0 legacy=0x7ff3300b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff318003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff318004800 0x7ff318004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3180051c0 0x7ff31800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300b31e0 con 0x7ff3180039b0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31800c090 con 0x7ff3180051c0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31800c290 con 0x7ff318004800 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff318003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32effd640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff318004800 0x7ff318004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32ffff640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3180051c0 0x7ff31800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 -- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff318004800 msgr2=0x7ff318004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff318004800 0x7ff318004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 -- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3180051c0 msgr2=0x7ff31800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3180051c0 0x7ff31800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32ffff640 1 -- 172.21.15.72:0/441152930 reap_dead start 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.796+0000 7ff32f7fe640 1 -- 172.21.15.72:0/441152930 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff318003520 con 0x7ff3180039b0 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.797+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff318003df0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7ff320009ac0 tx=0x7ff318005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce636bd088ed884 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.797+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320039070 con 0x7ff3180039b0 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.797+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32002c6b0 con 0x7ff3180039b0 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.797+0000 7ff32e7fc640 1 -- 172.21.15.72:0/441152930 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320009690 con 0x7ff3180039b0 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 -- 172.21.15.72:0/441152930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 msgr2=0x7ff318003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff318003df0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7ff320009ac0 tx=0x7ff318005830 comp rx=0 tx=0).stop 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 -- 172.21.15.72:0/441152930 shutdown_connections 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 --2- 172.21.15.72:0/441152930 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff318003df0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 -- 172.21.15.72:0/441152930 >> 172.21.15.72:0/441152930 conn(0x7ff3300d3cd0 msgr2=0x7ff3300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 -- 172.21.15.72:0/441152930 shutdown_connections 2024-03-06T07:30:43.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.798+0000 7ff335dac640 1 -- 172.21.15.72:0/441152930 wait complete. 2024-03-06T07:30:43.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.799+0000 7ff335dac640 1 Processor -- start 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.799+0000 7ff335dac640 1 -- start start 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff335dac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff330151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff335dac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff330151d10 0x7ff3300cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff335dac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff33015bde0 0x7ff33015c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff335dac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33015e3c0 con 0x7ff3180039b0 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff335dac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3300b3d60 con 0x7ff330151d10 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff335dac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33015e540 con 0x7ff33015bde0 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff330151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff330151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41858/0 (socket says 172.21.15.72:41858) 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 -- 172.21.15.72:0/116667680 learned_addr learned my addr 172.21.15.72:0/116667680 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 -- 172.21.15.72:0/116667680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff33015bde0 msgr2=0x7ff33015c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff33015bde0 0x7ff33015c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 -- 172.21.15.72:0/116667680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff330151d10 msgr2=0x7ff3300cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff330151d10 0x7ff3300cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:43.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 -- 172.21.15.72:0/116667680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3300b6a70 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.800+0000 7ff32f7fe640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff330151600 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7ff320000c00 tx=0x7ff320004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.801+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320043020 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.801+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32002cd90 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.801+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3300b6d50 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.801+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff320039040 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.801+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3300b7310 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.802+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33004f330 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.802+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff320046020 con 0x7ff3180039b0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.803+0000 7ff317fff640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff31c06b4d0 0x7ff31c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.803+0000 7ff32effd640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff31c06b4d0 0x7ff31c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:43.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.803+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff3200ab9d0 con 0x7ff3180039b0 2024-03-06T07:30:43.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.803+0000 7ff32effd640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff31c06b4d0 0x7ff31c06d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff324007fd0 tx=0x7ff324009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:43.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:43.811+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff32004d070 con 0x7ff3180039b0 2024-03-06T07:30:44.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.058+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3300b7830 con 0x7ff3180039b0 2024-03-06T07:30:44.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.058+0000 7ff317fff640 1 -- 172.21.15.72:0/116667680 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7ff32007a8a0 con 0x7ff3180039b0 2024-03-06T07:30:44.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:44.061 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:44.061 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:44.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff31c06b4d0 msgr2=0x7ff31c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff31c06b4d0 0x7ff31c06d990 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7ff324007fd0 tx=0x7ff324009040 comp rx=0 tx=0).stop 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 msgr2=0x7ff330151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff330151600 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7ff320000c00 tx=0x7ff320004730 comp rx=0 tx=0).stop 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 shutdown_connections 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff31c06b4d0 0x7ff31c06d990 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff33015bde0 0x7ff33015c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff330151d10 0x7ff3300cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 --2- 172.21.15.72:0/116667680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3180039b0 0x7ff330151600 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 >> 172.21.15.72:0/116667680 conn(0x7ff3300d3cd0 msgr2=0x7ff3300b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.060+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 shutdown_connections 2024-03-06T07:30:44.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:44.061+0000 7ff335dac640 1 -- 172.21.15.72:0/116667680 wait complete. 2024-03-06T07:30:44.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:44.275 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:45.276 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:45.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.374+0000 7f91a4829640 1 Processor -- start 2024-03-06T07:30:45.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.374+0000 7f91a4829640 1 -- start start 2024-03-06T07:30:45.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f91a4829640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f91a00b3060 0x7f91a015e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f91a4829640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a015f580 con 0x7f91a013dc10 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f91a4829640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a015b450 con 0x7f91a015bf30 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f91a4829640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00cf3f0 con 0x7f91a00b3060 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f91a013dc10 0x7f91a00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46996/0 (socket says 172.21.15.72:46996) 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919e59c640 1 -- 172.21.15.72:0/3145943022 learned_addr learned my addr 172.21.15.72:0/3145943022 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550949458 0 0) 0x7f91a015f580 con 0x7f91a013dc10 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f918c003520 con 0x7f91a013dc10 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919dd9b640 1 --2- 172.21.15.72:0/3145943022 >> v2:172.21.15.146:3300/0 conn(0x7f91a00b3060 0x7f91a015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200505792 0 0) 0x7f918c003520 con 0x7f91a013dc10 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 >> v2:172.21.15.146:3300/0 conn(0x7f91a00b3060 msgr2=0x7f91a015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 --2- 172.21.15.72:0/3145943022 >> v2:172.21.15.146:3300/0 conn(0x7f91a00b3060 0x7f91a015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 >> v1:172.21.15.146:6789/0 conn(0x7f91a015bf30 legacy=0x7f91a015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a015b730 con 0x7f91a013dc10 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9198001670 con 0x7f91a013dc10 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 >> v1:172.21.15.72:6789/0 conn(0x7f91a013dc10 legacy=0x7f91a00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f918c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f918c004800 0x7f918c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00cf3f0 con 0x7f918c0039b0 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918c00c1b0 con 0x7f918c0052e0 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.376+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c00c3b0 con 0x7f918c004800 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919e59c640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f918c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919dd9b640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f918c004800 0x7f918c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919ed9d640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919e59c640 1 -- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f918c004800 msgr2=0x7f918c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919e59c640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f918c004800 0x7f918c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919e59c640 1 -- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f918c0052e0 msgr2=0x7f918c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919e59c640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919e59c640 1 -- 172.21.15.72:0/3145943022 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c003520 con 0x7f918c0039b0 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919dd9b640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f918c004800 0x7f918c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919ed9d640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f918c0052e0 0x7f918c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.377+0000 7f919ed9d640 1 -- 172.21.15.72:0/3145943022 reap_dead start 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f919e59c640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f918c003dd0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9198007cb0 tx=0x7f918c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75080fc5f468a186 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198007cf0 con 0x7f918c0039b0 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198032070 con 0x7f918c0039b0 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f919d59a640 1 -- 172.21.15.72:0/3145943022 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91980058d0 con 0x7f918c0039b0 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 -- 172.21.15.72:0/3145943022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 msgr2=0x7f918c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f918c003dd0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f9198007cb0 tx=0x7f918c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 -- 172.21.15.72:0/3145943022 shutdown_connections 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 --2- 172.21.15.72:0/3145943022 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f918c003dd0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 -- 172.21.15.72:0/3145943022 >> 172.21.15.72:0/3145943022 conn(0x7f91a00cf8a0 msgr2=0x7f91a00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 -- 172.21.15.72:0/3145943022 shutdown_connections 2024-03-06T07:30:45.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.378+0000 7f91a4829640 1 -- 172.21.15.72:0/3145943022 wait complete. 2024-03-06T07:30:45.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 Processor -- start 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 -- start start 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f91a015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a015dae0 0x7f91a0144180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a01417a0 0x7f91a012ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00b2950 con 0x7f918c0039b0 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a015b210 con 0x7f91a01417a0 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f91a4829640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00b2ad0 con 0x7f91a015dae0 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.380+0000 7f919e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f91a015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f91a015c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41880/0 (socket says 172.21.15.72:41880) 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 -- 172.21.15.72:0/931724351 learned_addr learned my addr 172.21.15.72:0/931724351 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919dd9b640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a015dae0 0x7f91a0144180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919ed9d640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a01417a0 0x7f91a012ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 -- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a015dae0 msgr2=0x7f91a0144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a015dae0 0x7f91a0144180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 -- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a01417a0 msgr2=0x7f91a012ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a01417a0 0x7f91a012ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 -- 172.21.15.72:0/931724351 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a015ca40 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919dd9b640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a015dae0 0x7f91a0144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f919e59c640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f91a015c340 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f919802e720 tx=0x7f91a015b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198048070 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9198004bf0 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198001c30 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a015cd20 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.381+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a015d2e0 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.382+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a004f330 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.383+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9198007070 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.383+0000 7f9196ffd640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f918006b500 0x7f918006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.383+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f91980ad6d0 con 0x7f918c0039b0 2024-03-06T07:30:45.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.383+0000 7f919dd9b640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f918006b500 0x7f918006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.384+0000 7f919dd9b640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f918006b500 0x7f918006d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f91a006de30 tx=0x7f9188008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:45.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.387+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f919807bf10 con 0x7f918c0039b0 2024-03-06T07:30:45.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.722+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91a006eb00 con 0x7f918c0039b0 2024-03-06T07:30:45.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.722+0000 7f9196ffd640 1 -- 172.21.15.72:0/931724351 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f919807ffb0 con 0x7f918c0039b0 2024-03-06T07:30:45.725 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:45.725 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:45.725 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:45.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f918006b500 msgr2=0x7f918006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f918006b500 0x7f918006d9c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f91a006de30 tx=0x7f9188008040 comp rx=0 tx=0).stop 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 msgr2=0x7f91a015c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f91a015c340 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f919802e720 tx=0x7f91a015b620 comp rx=0 tx=0).stop 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 shutdown_connections 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f918006b500 0x7f918006d9c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f91a01417a0 0x7f91a012ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f91a015dae0 0x7f91a0144180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 --2- 172.21.15.72:0/931724351 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f918c0039b0 0x7f91a015c340 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 >> 172.21.15.72:0/931724351 conn(0x7f91a00cf8a0 msgr2=0x7f91a013caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 shutdown_connections 2024-03-06T07:30:45.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:45.725+0000 7f91a4829640 1 -- 172.21.15.72:0/931724351 wait complete. 2024-03-06T07:30:45.749 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:45.943 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:45.943 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:46.944 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:47.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.045+0000 7fad789d0640 1 Processor -- start 2024-03-06T07:30:47.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.045+0000 7fad789d0640 1 -- start start 2024-03-06T07:30:47.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.046+0000 7fad789d0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad740c4a90 0x7fad740c4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.046+0000 7fad789d0640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fad740cf9c0 0x7fad740cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.046+0000 7fad789d0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740c53b0 con 0x7fad74165d90 2024-03-06T07:30:47.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad789d0640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad740d2410 con 0x7fad740c4a90 2024-03-06T07:30:47.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad789d0640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad740d28a0 con 0x7fad740cf9c0 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad740c4a90 0x7fad740c4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad740c4a90 0x7fad740c4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41890/0 (socket says 172.21.15.72:41890) 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 -- 172.21.15.72:0/1223840855 learned_addr learned my addr 172.21.15.72:0/1223840855 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad71d9b640 1 --2- 172.21.15.72:0/1223840855 >> v2:172.21.15.146:3300/0 conn(0x7fad740cf9c0 0x7fad740cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 -- 172.21.15.72:0/1223840855 >> v2:172.21.15.146:3300/0 conn(0x7fad740cf9c0 msgr2=0x7fad740cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 --2- 172.21.15.72:0/1223840855 >> v2:172.21.15.146:3300/0 conn(0x7fad740cf9c0 0x7fad740cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 -- 172.21.15.72:0/1223840855 >> v1:172.21.15.146:6789/0 conn(0x7fad74165d90 legacy=0x7fad741661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad7259c640 1 -- 172.21.15.72:0/1223840855 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740d1800 con 0x7fad740c4a90 2024-03-06T07:30:47.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.047+0000 7fad71d9b640 1 --2- 172.21.15.72:0/1223840855 >> v2:172.21.15.146:3300/0 conn(0x7fad740cf9c0 0x7fad740cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:47.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.048+0000 7fad7259c640 1 --2- 172.21.15.72:0/1223840855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740c4e70 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fad740cc0c0 tx=0x7fad6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cc78660cbcde1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.048+0000 7fad7159a640 1 -- 172.21.15.72:0/1223840855 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c035020 con 0x7fad740c4a90 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.048+0000 7fad7159a640 1 -- 172.21.15.72:0/1223840855 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c031870 con 0x7fad740c4a90 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.048+0000 7fad7159a640 1 -- 172.21.15.72:0/1223840855 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c031c40 con 0x7fad740c4a90 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 -- 172.21.15.72:0/1223840855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 msgr2=0x7fad740c4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 --2- 172.21.15.72:0/1223840855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740c4e70 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fad740cc0c0 tx=0x7fad6c02b910 comp rx=0 tx=0).stop 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 -- 172.21.15.72:0/1223840855 shutdown_connections 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 --2- 172.21.15.72:0/1223840855 >> v2:172.21.15.146:3300/0 conn(0x7fad740cf9c0 0x7fad740cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 --2- 172.21.15.72:0/1223840855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740c4e70 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 -- 172.21.15.72:0/1223840855 >> 172.21.15.72:0/1223840855 conn(0x7fad7409a9f0 msgr2=0x7fad7409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 -- 172.21.15.72:0/1223840855 shutdown_connections 2024-03-06T07:30:47.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.049+0000 7fad789d0640 1 -- 172.21.15.72:0/1223840855 wait complete. 2024-03-06T07:30:47.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.050+0000 7fad789d0640 1 Processor -- start 2024-03-06T07:30:47.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.050+0000 7fad789d0640 1 -- start start 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad789d0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad789d0640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad740cf9c0 0x7fad740b5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad789d0640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad74165d90 0x7fad7409ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad789d0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740d20a0 con 0x7fad740c4a90 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad789d0640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad740d1dc0 con 0x7fad74165d90 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad789d0640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad740af900 con 0x7fad740cf9c0 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41892/0 (socket says 172.21.15.72:41892) 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 -- 172.21.15.72:0/3622076874 learned_addr learned my addr 172.21.15.72:0/3622076874 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 -- 172.21.15.72:0/3622076874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad740cf9c0 msgr2=0x7fad740b5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad740cf9c0 0x7fad740b5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 -- 172.21.15.72:0/3622076874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad74165d90 msgr2=0x7fad7409ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad74165d90 0x7fad7409ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 -- 172.21.15.72:0/3622076874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7409f2b0 con 0x7fad740c4a90 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.051+0000 7fad7259c640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740b46d0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fad6c000c00 tx=0x7fad6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.052+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c040070 con 0x7fad740c4a90 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.052+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c03a420 con 0x7fad740c4a90 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.052+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad740b4cb0 con 0x7fad740c4a90 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.052+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c035040 con 0x7fad740c4a90 2024-03-06T07:30:47.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.052+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad740b5270 con 0x7fad740c4a90 2024-03-06T07:30:47.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.053+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7404f330 con 0x7fad740c4a90 2024-03-06T07:30:47.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.053+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad6c038020 con 0x7fad740c4a90 2024-03-06T07:30:47.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.053+0000 7fad62ffd640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad5406b4d0 0x7fad5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:47.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.053+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad6c0ac2e0 con 0x7fad740c4a90 2024-03-06T07:30:47.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.054+0000 7fad71d9b640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad5406b4d0 0x7fad5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:47.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.054+0000 7fad71d9b640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad5406b4d0 0x7fad5406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fad740cdd20 tx=0x7fad680083d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:47.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.058+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad6c07b090 con 0x7fad740c4a90 2024-03-06T07:30:47.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.350+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad740cede0 con 0x7fad740c4a90 2024-03-06T07:30:47.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.350+0000 7fad62ffd640 1 -- 172.21.15.72:0/3622076874 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fad6c07a990 con 0x7fad740c4a90 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:47.353 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad5406b4d0 msgr2=0x7fad5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad5406b4d0 0x7fad5406d990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fad740cdd20 tx=0x7fad680083d0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 msgr2=0x7fad740b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740b46d0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fad6c000c00 tx=0x7fad6c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 shutdown_connections 2024-03-06T07:30:47.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad5406b4d0 0x7fad5406d990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad74165d90 0x7fad7409ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad740cf9c0 0x7fad740b5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 --2- 172.21.15.72:0/3622076874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad740c4a90 0x7fad740b46d0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 >> 172.21.15.72:0/3622076874 conn(0x7fad7409a9f0 msgr2=0x7fad740b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 shutdown_connections 2024-03-06T07:30:47.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:47.353+0000 7fad789d0640 1 -- 172.21.15.72:0/3622076874 wait complete. 2024-03-06T07:30:47.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:47.567 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:47.751 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:48.568 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:48.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.667+0000 7f3bd46fb640 1 Processor -- start 2024-03-06T07:30:48.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.667+0000 7f3bd46fb640 1 -- start start 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd46fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 0x7f3bcc0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd46fb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bcc0b3430 con 0x7f3bcc15dfe0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd46fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bcc0b3710 con 0x7f3bcc0b41f0 2024-03-06T07:30:48.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd46fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0d2c50 con 0x7f3bcc0d22b0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd2497640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3bcc15dfe0 0x7f3bcc0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47004/0 (socket says 172.21.15.72:47004) 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd2497640 1 -- 172.21.15.72:0/1416179525 learned_addr learned my addr 172.21.15.72:0/1416179525 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 0x7f3bcc0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532235501 0 0) 0x7f3bcc0b3430 con 0x7f3bcc15dfe0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bb8003520 con 0x7f3bcc15dfe0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2490834682 0 0) 0x7f3bb8003520 con 0x7f3bcc15dfe0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 msgr2=0x7f3bcc0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 0x7f3bcc0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 >> v1:172.21.15.146:6789/0 conn(0x7f3bcc0b41f0 legacy=0x7f3bcc0b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0b39f0 con 0x7f3bcc15dfe0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3bc8001670 con 0x7f3bcc15dfe0 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 >> v1:172.21.15.72:6789/0 conn(0x7f3bcc15dfe0 legacy=0x7f3bcc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 0x7f3bb80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bb8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bb8004d40 0x7f3bb80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb800ba70 con 0x7f3bb8003f60 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb800bc20 con 0x7f3bb8004d40 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.668+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb800be20 con 0x7f3bcc0d22b0 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bb8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd2c98640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bb8004d40 0x7f3bb80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd2497640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 0x7f3bb80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd1c96640 1 -- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 msgr2=0x7f3bb80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc0d22b0 0x7f3bb80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd1c96640 1 -- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bb8004d40 msgr2=0x7f3bb80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bb8004d40 0x7f3bb80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.669+0000 7f3bd1c96640 1 -- 172.21.15.72:0/1416179525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb8003520 con 0x7f3bb8003f60 2024-03-06T07:30:48.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.670+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bb8004800 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009ac0 tx=0x7f3bb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d739e10ad4fca5f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.670+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc039070 con 0x7f3bb8003f60 2024-03-06T07:30:48.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.670+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bbc02c6b0 con 0x7f3bb8003f60 2024-03-06T07:30:48.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.670+0000 7f3bd1495640 1 -- 172.21.15.72:0/1416179525 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc009690 con 0x7f3bb8003f60 2024-03-06T07:30:48.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd46fb640 1 -- 172.21.15.72:0/1416179525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 msgr2=0x7f3bb8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/1416179525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bb8004800 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009ac0 tx=0x7f3bb8005830 comp rx=0 tx=0).stop 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd2c98640 1 -- 172.21.15.72:0/1416179525 reap_dead start 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd46fb640 1 -- 172.21.15.72:0/1416179525 shutdown_connections 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd46fb640 1 -- 172.21.15.72:0/1416179525 >> 172.21.15.72:0/1416179525 conn(0x7f3bcc0d3cd0 msgr2=0x7f3bcc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd46fb640 1 -- 172.21.15.72:0/1416179525 shutdown_connections 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.671+0000 7f3bd46fb640 1 -- 172.21.15.72:0/1416179525 wait complete. 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.672+0000 7f3bd46fb640 1 Processor -- start 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.672+0000 7f3bd46fb640 1 -- start start 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bcc159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc1599c0 0x7f3bcc0b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bcc160ea0 0x7f3bcc0b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b3d60 con 0x7f3bb8003f60 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0b31e0 con 0x7f3bcc160ea0 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc0d1c70 con 0x7f3bcc1599c0 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bcc159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bcc159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41910/0 (socket says 172.21.15.72:41910) 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 -- 172.21.15.72:0/14641871 learned_addr learned my addr 172.21.15.72:0/14641871 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 -- 172.21.15.72:0/14641871 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc1599c0 msgr2=0x7f3bcc0b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc1599c0 0x7f3bcc0b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 -- 172.21.15.72:0/14641871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bcc160ea0 msgr2=0x7f3bcc0b9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bcc160ea0 0x7f3bcc0b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 -- 172.21.15.72:0/14641871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bcc0cdfc0 con 0x7f3bb8003f60 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd2497640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bcc159130 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800e820 tx=0x7f3bc800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800c7f0 con 0x7f3bb8003f60 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8018070 con 0x7f3bb8003f60 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc800cbd0 con 0x7f3bb8003f60 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc15f0b0 con 0x7f3bb8003f60 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.673+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bcc15f670 con 0x7f3bb8003f60 2024-03-06T07:30:48.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.674+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bcc04f330 con 0x7f3bb8003f60 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.675+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3bc801d460 con 0x7f3bb8003f60 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.675+0000 7f3bc2ffd640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3bb006b4d0 0x7f3bb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.675+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3bc808a190 con 0x7f3bb8003f60 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.675+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3bb006b4d0 0x7f3bb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:48.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.676+0000 7f3bd1c96640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3bb006b4d0 0x7f3bb006d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc036000 tx=0x7f3bbc02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:48.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.684+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3bc8059060 con 0x7f3bb8003f60 2024-03-06T07:30:48.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.912+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bcc15a7d0 con 0x7f3bb8003f60 2024-03-06T07:30:48.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.913+0000 7f3bc2ffd640 1 -- 172.21.15.72:0/14641871 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f3bc8058960 con 0x7f3bb8003f60 2024-03-06T07:30:48.915 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:48.915 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:48.915 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:48.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3bb006b4d0 msgr2=0x7f3bb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3bb006b4d0 0x7f3bb006d990 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc036000 tx=0x7f3bbc02e000 comp rx=0 tx=0).stop 2024-03-06T07:30:48.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 msgr2=0x7f3bcc159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bcc159130 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800e820 tx=0x7f3bc800ece0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 shutdown_connections 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3bb006b4d0 0x7f3bb006d990 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bcc160ea0 0x7f3bcc0b9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bcc1599c0 0x7f3bcc0b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 --2- 172.21.15.72:0/14641871 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3bb8003f60 0x7f3bcc159130 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 >> 172.21.15.72:0/14641871 conn(0x7f3bcc0d3cd0 msgr2=0x7f3bcc0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 shutdown_connections 2024-03-06T07:30:48.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:48.915+0000 7f3bd46fb640 1 -- 172.21.15.72:0/14641871 wait complete. 2024-03-06T07:30:49.129 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:49.130 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:49.753 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:50.131 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:50.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.230+0000 7f02ccaeb640 1 Processor -- start 2024-03-06T07:30:50.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.230+0000 7f02ccaeb640 1 -- start start 2024-03-06T07:30:50.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02ccaeb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c806c6d0 0x7f02c815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02ccaeb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02c8066d90 0x7f02c80ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02ccaeb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c80cd450 con 0x7f02c80b4f70 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02ccaeb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c815e9f0 con 0x7f02c8066d90 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02ccaeb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80cda10 con 0x7f02c806c6d0 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02c6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02c8066d90 0x7f02c80ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02c6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f02c8066d90 0x7f02c80ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41914/0 (socket says 172.21.15.72:41914) 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02c6d9d640 1 -- 172.21.15.72:0/337706785 learned_addr learned my addr 172.21.15.72:0/337706785 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02c5d9b640 1 --2- 172.21.15.72:0/337706785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c806c6d0 0x7f02c815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02c6d9d640 1 -- 172.21.15.72:0/337706785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c806c6d0 msgr2=0x7f02c815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.231+0000 7f02c6d9d640 1 --2- 172.21.15.72:0/337706785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c806c6d0 0x7f02c815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.232+0000 7f02c6d9d640 1 -- 172.21.15.72:0/337706785 >> v1:172.21.15.146:6789/0 conn(0x7f02c80b4f70 legacy=0x7f02c80b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.232+0000 7f02c6d9d640 1 -- 172.21.15.72:0/337706785 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c80cd730 con 0x7f02c8066d90 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.232+0000 7f02c6d9d640 1 --2- 172.21.15.72:0/337706785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c80ccb50 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009e50 tx=0x7f02c002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f5ef64a89b6df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.232+0000 7f02c559a640 1 -- 172.21.15.72:0/337706785 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c002cb00 con 0x7f02c8066d90 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.232+0000 7f02c559a640 1 -- 172.21.15.72:0/337706785 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c002cca0 con 0x7f02c8066d90 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.232+0000 7f02c559a640 1 -- 172.21.15.72:0/337706785 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0034610 con 0x7f02c8066d90 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 -- 172.21.15.72:0/337706785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 msgr2=0x7f02c80ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/337706785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c80ccb50 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f02c0009e50 tx=0x7f02c002bf60 comp rx=0 tx=0).stop 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 -- 172.21.15.72:0/337706785 shutdown_connections 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/337706785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c80ccb50 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/337706785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c806c6d0 0x7f02c815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 -- 172.21.15.72:0/337706785 >> 172.21.15.72:0/337706785 conn(0x7f02c80724d0 msgr2=0x7f02c80728e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 -- 172.21.15.72:0/337706785 shutdown_connections 2024-03-06T07:30:50.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.233+0000 7f02ccaeb640 1 -- 172.21.15.72:0/337706785 wait complete. 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.234+0000 7f02ccaeb640 1 Processor -- start 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.234+0000 7f02ccaeb640 1 -- start start 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02ccaeb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02ccaeb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c80b4f70 0x7f02c81609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02ccaeb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c81e84d0 0x7f02c81e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02ccaeb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80cd090 con 0x7f02c8066d90 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02ccaeb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c815f3c0 con 0x7f02c81e84d0 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02ccaeb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80cd210 con 0x7f02c80b4f70 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c814b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41924/0 (socket says 172.21.15.72:41924) 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 -- 172.21.15.72:0/3240336583 learned_addr learned my addr 172.21.15.72:0/3240336583 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c5d9b640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c80b4f70 0x7f02c81609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c6d9d640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c81e84d0 0x7f02c81e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 -- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c80b4f70 msgr2=0x7f02c81609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c80b4f70 0x7f02c81609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 -- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c81e84d0 msgr2=0x7f02c81e8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c81e84d0 0x7f02c81e8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c659c640 1 -- 172.21.15.72:0/3240336583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c815b530 con 0x7f02c8066d90 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.235+0000 7f02c6d9d640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c81e84d0 0x7f02c81e8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.236+0000 7f02c659c640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c814b0f0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00c400 tx=0x7f02bc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.236+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc0052d0 con 0x7f02c8066d90 2024-03-06T07:30:50.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.236+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc005490 con 0x7f02c8066d90 2024-03-06T07:30:50.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.236+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c80adf90 con 0x7f02c8066d90 2024-03-06T07:30:50.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.236+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc007260 con 0x7f02c8066d90 2024-03-06T07:30:50.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.236+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c80ae550 con 0x7f02c8066d90 2024-03-06T07:30:50.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.237+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c804f330 con 0x7f02c8066d90 2024-03-06T07:30:50.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.237+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f02bc007400 con 0x7f02c8066d90 2024-03-06T07:30:50.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.238+0000 7f02b6ffd640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f02a806b4d0 0x7f02a806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:50.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.238+0000 7f02c5d9b640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f02a806b4d0 0x7f02a806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:50.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.238+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f02bc089650 con 0x7f02c8066d90 2024-03-06T07:30:50.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.238+0000 7f02c5d9b640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f02a806b4d0 0x7f02a806d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f02c806d7d0 tx=0x7f02b8009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:50.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.242+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f02bc057ed0 con 0x7f02c8066d90 2024-03-06T07:30:50.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.452+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02c80b1860 con 0x7f02c8066d90 2024-03-06T07:30:50.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.453+0000 7f02b6ffd640 1 -- 172.21.15.72:0/3240336583 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f02bc05bf70 con 0x7f02c8066d90 2024-03-06T07:30:50.455 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:50.455 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:50.455 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:50.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f02a806b4d0 msgr2=0x7f02a806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f02a806b4d0 0x7f02a806d990 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f02c806d7d0 tx=0x7f02b8009190 comp rx=0 tx=0).stop 2024-03-06T07:30:50.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 msgr2=0x7f02c814b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:50.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c814b0f0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f02bc00c400 tx=0x7f02bc00c8c0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 shutdown_connections 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f02a806b4d0 0x7f02a806d990 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02c81e84d0 0x7f02c81e8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02c80b4f70 0x7f02c81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 --2- 172.21.15.72:0/3240336583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02c8066d90 0x7f02c814b0f0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 >> 172.21.15.72:0/3240336583 conn(0x7f02c80724d0 msgr2=0x7f02c806d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 shutdown_connections 2024-03-06T07:30:50.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:50.455+0000 7f02ccaeb640 1 -- 172.21.15.72:0/3240336583 wait complete. 2024-03-06T07:30:50.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:50.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:51.670 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:51.755 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:51.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.773+0000 7f479db81640 1 Processor -- start 2024-03-06T07:30:51.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f479db81640 1 -- start start 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f479db81640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4798166af0 0x7f4798168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f479db81640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f479815e380 con 0x7f47980cf850 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f479db81640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47980cdc10 con 0x7f47980d00c0 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f479db81640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f479806ca90 con 0x7f4798166af0 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f478ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f47980cf850 0x7f47980cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47010/0 (socket says 172.21.15.72:47010) 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.774+0000 7f478ffff640 1 -- 172.21.15.72:0/217169830 learned_addr learned my addr 172.21.15.72:0/217169830 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4797fff640 1 --2- 172.21.15.72:0/217169830 >> v2:172.21.15.146:3300/0 conn(0x7f4798166af0 0x7f4798168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4796ffd640 1 -- 172.21.15.72:0/217169830 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990533651 0 0) 0x7f479815e380 con 0x7f47980cf850 2024-03-06T07:30:51.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4796ffd640 1 -- 172.21.15.72:0/217169830 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4784003520 con 0x7f47980cf850 2024-03-06T07:30:51.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4797fff640 1 -- 172.21.15.72:0/217169830 >> v1:172.21.15.146:6789/0 conn(0x7f47980d00c0 legacy=0x7f47980d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4797fff640 1 -- 172.21.15.72:0/217169830 >> v1:172.21.15.72:6789/0 conn(0x7f47980cf850 legacy=0x7f47980cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4796ffd640 1 -- 172.21.15.72:0/217169830 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2432244961 0 0) 0x7f4784003520 con 0x7f47980cf850 2024-03-06T07:30:51.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.775+0000 7f4797fff640 1 -- 172.21.15.72:0/217169830 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f479806c160 con 0x7f4798166af0 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f4797fff640 1 --2- 172.21.15.72:0/217169830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798166af0 0x7f4798168ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4790009d90 tx=0x7f479002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af643c0efdd0ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f4796ffd640 1 -- 172.21.15.72:0/217169830 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4790039070 con 0x7f4798166af0 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f4796ffd640 1 -- 172.21.15.72:0/217169830 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f479002c940 con 0x7f4798166af0 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f4796ffd640 1 -- 172.21.15.72:0/217169830 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4780001670 con 0x7f4798166af0 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f479db81640 1 -- 172.21.15.72:0/217169830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798166af0 msgr2=0x7f4798168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f479db81640 1 --2- 172.21.15.72:0/217169830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798166af0 0x7f4798168ee0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4790009d90 tx=0x7f479002be10 comp rx=0 tx=0).stop 2024-03-06T07:30:51.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f479db81640 1 -- 172.21.15.72:0/217169830 shutdown_connections 2024-03-06T07:30:51.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.776+0000 7f479db81640 1 --2- 172.21.15.72:0/217169830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4798166af0 0x7f4798168ee0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.777+0000 7f479db81640 1 -- 172.21.15.72:0/217169830 >> 172.21.15.72:0/217169830 conn(0x7f47980b35a0 msgr2=0x7f47980b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:51.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.777+0000 7f479db81640 1 -- 172.21.15.72:0/217169830 shutdown_connections 2024-03-06T07:30:51.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.777+0000 7f479db81640 1 -- 172.21.15.72:0/217169830 wait complete. 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 Processor -- start 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 -- start start 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 0x7f47980c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47980d00c0 0x7f47981547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4798166af0 0x7f47980cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479806c720 con 0x7f47980cf850 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479806df30 con 0x7f47980d00c0 2024-03-06T07:30:51.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f479db81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47980b1030 con 0x7f4798166af0 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f47977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 0x7f47980c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f47977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 0x7f47980c8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41928/0 (socket says 172.21.15.72:41928) 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.778+0000 7f47977fe640 1 -- 172.21.15.72:0/1219218163 learned_addr learned my addr 172.21.15.72:0/1219218163 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f478ffff640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47980d00c0 0x7f47981547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f4797fff640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4798166af0 0x7f47980cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f47977fe640 1 -- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4798166af0 msgr2=0x7f47980cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f47977fe640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4798166af0 0x7f47980cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f47977fe640 1 -- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47980d00c0 msgr2=0x7f47981547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f47977fe640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47980d00c0 0x7f47981547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f47977fe640 1 -- 172.21.15.72:0/1219218163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47980c9220 con 0x7f47980cf850 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f47977fe640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 0x7f47980c8b40 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f478800c450 tx=0x7f478800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47880052d0 con 0x7f47980cf850 2024-03-06T07:30:51.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4788005470 con 0x7f47980cf850 2024-03-06T07:30:51.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4788012870 con 0x7f47980cf850 2024-03-06T07:30:51.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47980c9500 con 0x7f47980cf850 2024-03-06T07:30:51.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.779+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47980c9ac0 con 0x7f47980cf850 2024-03-06T07:30:51.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.780+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479804f330 con 0x7f47980cf850 2024-03-06T07:30:51.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.781+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4788012a10 con 0x7f47980cf850 2024-03-06T07:30:51.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.781+0000 7f4794ff9640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477806b4d0 0x7f477806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:51.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.781+0000 7f478ffff640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477806b4d0 0x7f477806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:51.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.781+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4788089eb0 con 0x7f47980cf850 2024-03-06T07:30:51.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.782+0000 7f478ffff640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477806b4d0 0x7f477806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4798166a30 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:51.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.786+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4788058cf0 con 0x7f47980cf850 2024-03-06T07:30:52.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.998+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47980d15c0 con 0x7f47980cf850 2024-03-06T07:30:52.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:51.999+0000 7f4794ff9640 1 -- 172.21.15.72:0/1219218163 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f47880585f0 con 0x7f47980cf850 2024-03-06T07:30:52.001 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:52.001 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:52.001 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477806b4d0 msgr2=0x7f477806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477806b4d0 0x7f477806d990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4798166a30 tx=0x7f4780009040 comp rx=0 tx=0).stop 2024-03-06T07:30:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 msgr2=0x7f47980c8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 0x7f47980c8b40 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f478800c450 tx=0x7f478800c910 comp rx=0 tx=0).stop 2024-03-06T07:30:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 shutdown_connections 2024-03-06T07:30:52.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f477806b4d0 0x7f477806d990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4798166af0 0x7f47980cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47980d00c0 0x7f47981547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 --2- 172.21.15.72:0/1219218163 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f47980cf850 0x7f47980c8b40 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 >> 172.21.15.72:0/1219218163 conn(0x7f47980b35a0 msgr2=0x7f47980b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 shutdown_connections 2024-03-06T07:30:52.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:52.001+0000 7f479db81640 1 -- 172.21.15.72:0/1219218163 wait complete. 2024-03-06T07:30:52.023 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:52.025 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:30:52.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:52.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:53.217 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:53.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.316+0000 7f708bd1e640 1 Processor -- start 2024-03-06T07:30:53.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.317+0000 7f708bd1e640 1 -- start start 2024-03-06T07:30:53.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.317+0000 7f708bd1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70840d4760 0x7f70840b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.317+0000 7f708bd1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70840d25d0 con 0x7f70840d0a80 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.317+0000 7f708bd1e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70840d2870 con 0x7f70840b4170 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.317+0000 7f708bd1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70840d4bf0 con 0x7f70840d4760 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f708a2bb640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f70840d0a80 0x7f70840cee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50478/0 (socket says 172.21.15.72:50478) 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f708a2bb640 1 -- 172.21.15.72:0/1675229181 learned_addr learned my addr 172.21.15.72:0/1675229181 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f70892b9640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70840d4760 0x7f70840b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104128809 0 0) 0x7f70840d25d0 con 0x7f70840d0a80 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f706c003520 con 0x7f70840d0a80 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087636841 0 0) 0x7f70840d2870 con 0x7f70840b4170 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70840d25d0 con 0x7f70840b4170 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864253627 0 0) 0x7f706c003520 con 0x7f70840d0a80 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70840d4760 msgr2=0x7f70840b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70840d4760 0x7f70840b6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 >> v1:172.21.15.146:6789/0 conn(0x7f70840b4170 legacy=0x7f70840d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70840d2b50 con 0x7f70840d0a80 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f7074001670 con 0x7f70840d0a80 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 >> v1:172.21.15.72:6789/0 conn(0x7f70840d0a80 legacy=0x7f70840cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f706c003b70 0x7f706c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f706c004800 0x7f706c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f706c005180 0x7f706c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840d4bf0 con 0x7f706c005180 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706c00bf00 con 0x7f706c003b70 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706c00c0b0 con 0x7f706c004800 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f708a2bb640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f706c005180 0x7f706c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f7089aba640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f706c003b70 0x7f706c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.318+0000 7f70892b9640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f706c004800 0x7f706c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f708a2bb640 1 -- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f706c004800 msgr2=0x7f706c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f708a2bb640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f706c004800 0x7f706c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f708a2bb640 1 -- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f706c003b70 msgr2=0x7f706c003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f708a2bb640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f706c003b70 0x7f706c003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f708a2bb640 1 -- 172.21.15.72:0/1675229181 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c003520 con 0x7f706c005180 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f708a2bb640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f706c005180 0x7f706c00b9c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f708408d0e0 tx=0x7f706c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ac609c3b456e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074007c00 con 0x7f706c005180 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7074030070 con 0x7f706c005180 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.319+0000 7f7088ab8640 1 -- 172.21.15.72:0/1675229181 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074005d30 con 0x7f706c005180 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708bd1e640 1 -- 172.21.15.72:0/1675229181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f706c005180 msgr2=0x7f706c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708bd1e640 1 --2- 172.21.15.72:0/1675229181 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f706c005180 0x7f706c00b9c0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f708408d0e0 tx=0x7f706c005830 comp rx=0 tx=0).stop 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708a2bb640 1 -- 172.21.15.72:0/1675229181 reap_dead start 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708bd1e640 1 -- 172.21.15.72:0/1675229181 shutdown_connections 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708bd1e640 1 -- 172.21.15.72:0/1675229181 >> 172.21.15.72:0/1675229181 conn(0x7f70840d3c40 msgr2=0x7f70840d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708bd1e640 1 -- 172.21.15.72:0/1675229181 shutdown_connections 2024-03-06T07:30:53.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.320+0000 7f708bd1e640 1 -- 172.21.15.72:0/1675229181 wait complete. 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.321+0000 7f708bd1e640 1 Processor -- start 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.321+0000 7f708bd1e640 1 -- start start 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708bd1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 0x7f70840cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708bd1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70840cc500 0x7f708414a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708bd1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f708415b800 0x7f708414a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708bd1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840b4dd0 con 0x7f708415b420 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708bd1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70840ce9e0 con 0x7f70840cc500 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708bd1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70840b4f50 con 0x7f708415b800 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 0x7f70840cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 0x7f70840cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47926/0 (socket says 172.21.15.72:47926) 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 -- 172.21.15.72:0/3181608968 learned_addr learned my addr 172.21.15.72:0/3181608968 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f70892b9640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70840cc500 0x7f708414a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 -- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f708415b800 msgr2=0x7f708414a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708a2bb640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f708415b800 0x7f708414a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f708415b800 0x7f708414a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 -- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70840cc500 msgr2=0x7f708414a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70840cc500 0x7f708414a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f7089aba640 1 -- 172.21.15.72:0/3181608968 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70840cc9f0 con 0x7f708415b420 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f70892b9640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70840cc500 0x7f708414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.322+0000 7f708a2bb640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f708415b800 0x7f708414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.323+0000 7f7089aba640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 0x7f70840cbfc0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f707c00fb50 tx=0x7f707c00d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.323+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c00dc30 con 0x7f708415b420 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.323+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707c018070 con 0x7f708415b420 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.323+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70840cccd0 con 0x7f708415b420 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.323+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707c003970 con 0x7f708415b420 2024-03-06T07:30:53.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.323+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70840cd290 con 0x7f708415b420 2024-03-06T07:30:53.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.324+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708404f330 con 0x7f708415b420 2024-03-06T07:30:53.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.324+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f707c003b30 con 0x7f708415b420 2024-03-06T07:30:53.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.325+0000 7f70827fc640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f707006b500 0x7f707006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:53.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.325+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f707c08a520 con 0x7f708415b420 2024-03-06T07:30:53.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.325+0000 7f70892b9640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f707006b500 0x7f707006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:53.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.325+0000 7f70892b9640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f707006b500 0x7f707006d9c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f70841f7350 tx=0x7f7078008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:53.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.329+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f707c003e20 con 0x7f708415b420 2024-03-06T07:30:53.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.537+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70840733b0 con 0x7f708415b420 2024-03-06T07:30:53.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.538+0000 7f70827fc640 1 -- 172.21.15.72:0/3181608968 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f707c0593f0 con 0x7f708415b420 2024-03-06T07:30:53.540 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:53.540 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:53.540 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:53.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f707006b500 msgr2=0x7f707006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f707006b500 0x7f707006d9c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f70841f7350 tx=0x7f7078008040 comp rx=0 tx=0).stop 2024-03-06T07:30:53.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 msgr2=0x7f70840cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 0x7f70840cbfc0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f707c00fb50 tx=0x7f707c00d520 comp rx=0 tx=0).stop 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 shutdown_connections 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f707006b500 0x7f707006d9c0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f708415b800 0x7f708414a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70840cc500 0x7f708414a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 --2- 172.21.15.72:0/3181608968 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f708415b420 0x7f70840cbfc0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 >> 172.21.15.72:0/3181608968 conn(0x7f70840d3c40 msgr2=0x7f70840b7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 shutdown_connections 2024-03-06T07:30:53.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:53.540+0000 7f708bd1e640 1 -- 172.21.15.72:0/3181608968 wait complete. 2024-03-06T07:30:53.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:53.754 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:53.757 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:54.755 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.856+0000 7fc92955e640 1 Processor -- start 2024-03-06T07:30:54.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.856+0000 7fc92955e640 1 -- start start 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc92955e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92415dfe0 0x7fc9240cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc92955e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9240d29c0 0x7fc9240b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc92955e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9240b3430 con 0x7fc9240cf830 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc92955e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9240b31e0 con 0x7fc9240d29c0 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc92955e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240b3d60 con 0x7fc92415dfe0 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9237fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9240d29c0 0x7fc9240b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc91bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc9240cf830 0x7fc9240d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50480/0 (socket says 172.21.15.72:50480) 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc91bfff640 1 -- 172.21.15.72:0/3847367087 learned_addr learned my addr 172.21.15.72:0/3847367087 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9237fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9240d29c0 0x7fc9240b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47928/0 (socket says 172.21.15.72:47928) 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9237fe640 1 -- 172.21.15.72:0/3847367087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92415dfe0 msgr2=0x7fc9240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9237fe640 1 --2- 172.21.15.72:0/3847367087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92415dfe0 0x7fc9240cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9237fe640 1 -- 172.21.15.72:0/3847367087 >> v1:172.21.15.72:6789/0 conn(0x7fc9240cf830 legacy=0x7fc9240d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9237fe640 1 -- 172.21.15.72:0/3847367087 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240b3710 con 0x7fc9240d29c0 2024-03-06T07:30:54.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.857+0000 7fc9227fc640 1 -- 172.21.15.72:0/3847367087 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911145490 0 0) 0x7fc9240b3430 con 0x7fc9240cf830 2024-03-06T07:30:54.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc9237fe640 1 --2- 172.21.15.72:0/3847367087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9240b41f0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009d90 tx=0x7fc91c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f801e303a7ba7bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc9227fc640 1 -- 172.21.15.72:0/3847367087 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c039070 con 0x7fc9240d29c0 2024-03-06T07:30:54.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc9227fc640 1 -- 172.21.15.72:0/3847367087 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc91c02cc50 con 0x7fc9240d29c0 2024-03-06T07:30:54.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc9227fc640 1 -- 172.21.15.72:0/3847367087 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c034440 con 0x7fc9240d29c0 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 -- 172.21.15.72:0/3847367087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 msgr2=0x7fc9240b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 --2- 172.21.15.72:0/3847367087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9240b41f0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fc91c009d90 tx=0x7fc91c02bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 -- 172.21.15.72:0/3847367087 shutdown_connections 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 --2- 172.21.15.72:0/3847367087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9240b41f0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 --2- 172.21.15.72:0/3847367087 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc92415dfe0 0x7fc9240cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 -- 172.21.15.72:0/3847367087 >> 172.21.15.72:0/3847367087 conn(0x7fc9240d3cd0 msgr2=0x7fc9240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 -- 172.21.15.72:0/3847367087 shutdown_connections 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.859+0000 7fc92955e640 1 -- 172.21.15.72:0/3847367087 wait complete. 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.860+0000 7fc92955e640 1 Processor -- start 2024-03-06T07:30:54.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.860+0000 7fc92955e640 1 -- start start 2024-03-06T07:30:54.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc92955e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9240cf830 0x7fc9240cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc92955e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9241500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc92955e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc92415dfe0 0x7fc9240b91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc92955e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240d1df0 con 0x7fc9240d29c0 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc92955e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9240d1c70 con 0x7fc92415dfe0 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc92955e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc92415e3c0 con 0x7fc9240cf830 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9241500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9241500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47932/0 (socket says 172.21.15.72:47932) 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 -- 172.21.15.72:0/2705800092 learned_addr learned my addr 172.21.15.72:0/2705800092 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc9237fe640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc92415dfe0 0x7fc9240b91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc922ffd640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9240cf830 0x7fc9240cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 -- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9240cf830 msgr2=0x7fc9240cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9240cf830 0x7fc9240cd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 -- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc92415dfe0 msgr2=0x7fc9240b91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc92415dfe0 0x7fc9240b91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.861+0000 7fc91bfff640 1 -- 172.21.15.72:0/2705800092 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240cb180 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc9237fe640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc92415dfe0 0x7fc9240b91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc91bfff640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9241500a0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00c4b0 tx=0x7fc90c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c018070 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90c007500 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9240cb460 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90c004a20 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.862+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9240cba20 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.863+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc92404f330 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.863+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc90c01b070 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.864+0000 7fc91b7fe640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc90406b500 0x7fc90406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.864+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc90c0894e0 con 0x7fc9240d29c0 2024-03-06T07:30:54.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.864+0000 7fc922ffd640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc90406b500 0x7fc90406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:54.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.864+0000 7fc922ffd640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc90406b500 0x7fc90406d9c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc91400a770 tx=0x7fc914008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:54.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:54.868+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc90c058470 con 0x7fc9240d29c0 2024-03-06T07:30:55.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.082+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9240ca8f0 con 0x7fc9240d29c0 2024-03-06T07:30:55.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.083+0000 7fc91b7fe640 1 -- 172.21.15.72:0/2705800092 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fc90c057d70 con 0x7fc9240d29c0 2024-03-06T07:30:55.085 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:55.085 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:55.085 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc90406b500 msgr2=0x7fc90406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc90406b500 0x7fc90406d9c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc91400a770 tx=0x7fc914008040 comp rx=0 tx=0).stop 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 msgr2=0x7fc9241500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9241500a0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fc90c00c4b0 tx=0x7fc90c00c970 comp rx=0 tx=0).stop 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 shutdown_connections 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc90406b500 0x7fc90406d9c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc92415dfe0 0x7fc9240b91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9240d29c0 0x7fc9241500a0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 --2- 172.21.15.72:0/2705800092 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9240cf830 0x7fc9240cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.084+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 >> 172.21.15.72:0/2705800092 conn(0x7fc9240d3cd0 msgr2=0x7fc92406c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:55.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.085+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 shutdown_connections 2024-03-06T07:30:55.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:55.085+0000 7fc92955e640 1 -- 172.21.15.72:0/2705800092 wait complete. 2024-03-06T07:30:55.298 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:55.299 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:55.759 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:56.300 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:56.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.399+0000 7f82675c9640 1 Processor -- start 2024-03-06T07:30:56.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.399+0000 7f82675c9640 1 -- start start 2024-03-06T07:30:56.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.400+0000 7f82675c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f82600cd610 0x7f82600d1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.400+0000 7f82675c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600d15a0 0x7f826006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.400+0000 7f82675c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82600afd40 con 0x7f826015bb90 2024-03-06T07:30:56.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.400+0000 7f82675c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82600d0bd0 con 0x7f82600cd610 2024-03-06T07:30:56.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.400+0000 7f82675c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82600b0020 con 0x7f82600d15a0 2024-03-06T07:30:56.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8264b64640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f82600cd610 0x7f82600d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265b66640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600d15a0 0x7f826006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265365640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f826015bb90 0x7f826006f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:44272/0 (socket says 172.21.15.72:44272) 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265365640 1 -- 172.21.15.72:0/3191018456 learned_addr learned my addr 172.21.15.72:0/3191018456 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8257fff640 1 -- 172.21.15.72:0/3191018456 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1271785713 0 0) 0x7f82600afd40 con 0x7f826015bb90 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8257fff640 1 -- 172.21.15.72:0/3191018456 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c003520 con 0x7f826015bb90 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265b66640 1 -- 172.21.15.72:0/3191018456 >> v2:172.21.15.146:3300/0 conn(0x7f82600cd610 msgr2=0x7f82600d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265b66640 1 --2- 172.21.15.72:0/3191018456 >> v2:172.21.15.146:3300/0 conn(0x7f82600cd610 0x7f82600d1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265b66640 1 -- 172.21.15.72:0/3191018456 >> v1:172.21.15.146:6789/0 conn(0x7f826015bb90 legacy=0x7f826006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.401+0000 7f8265b66640 1 -- 172.21.15.72:0/3191018456 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82600d0860 con 0x7f82600d15a0 2024-03-06T07:30:56.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.402+0000 7f8265b66640 1 --2- 172.21.15.72:0/3191018456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600d15a0 0x7f826006afb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8258009d90 tx=0x7f825802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5949d82a18f75c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.402+0000 7f8257fff640 1 -- 172.21.15.72:0/3191018456 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258039070 con 0x7f82600d15a0 2024-03-06T07:30:56.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.402+0000 7f8257fff640 1 -- 172.21.15.72:0/3191018456 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825802cc50 con 0x7f82600d15a0 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.402+0000 7f8257fff640 1 -- 172.21.15.72:0/3191018456 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258034440 con 0x7f82600d15a0 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 -- 172.21.15.72:0/3191018456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600d15a0 msgr2=0x7f826006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 --2- 172.21.15.72:0/3191018456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600d15a0 0x7f826006afb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8258009d90 tx=0x7f825802bea0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 -- 172.21.15.72:0/3191018456 shutdown_connections 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 --2- 172.21.15.72:0/3191018456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600d15a0 0x7f826006afb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 --2- 172.21.15.72:0/3191018456 >> v2:172.21.15.146:3300/0 conn(0x7f82600cd610 0x7f82600d1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 -- 172.21.15.72:0/3191018456 >> 172.21.15.72:0/3191018456 conn(0x7f826006c0c0 msgr2=0x7f826006c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 -- 172.21.15.72:0/3191018456 shutdown_connections 2024-03-06T07:30:56.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.403+0000 7f82675c9640 1 -- 172.21.15.72:0/3191018456 wait complete. 2024-03-06T07:30:56.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.405+0000 7f82675c9640 1 Processor -- start 2024-03-06T07:30:56.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.405+0000 7f82675c9640 1 -- start start 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f82675c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600cd610 0x7f8260143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f82675c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 0x7f82600b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f82675c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82600b0760 0x7f8260145830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f82675c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826006b670 con 0x7f826015bb90 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f82675c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826006b4f0 con 0x7f82600b0760 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f82675c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826006b7f0 con 0x7f82600cd610 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 0x7f82600b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 0x7f82600b0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47942/0 (socket says 172.21.15.72:47942) 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 -- 172.21.15.72:0/577310592 learned_addr learned my addr 172.21.15.72:0/577310592 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8265365640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600cd610 0x7f8260143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8265b66640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82600b0760 0x7f8260145830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 -- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600cd610 msgr2=0x7f8260143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600cd610 0x7f8260143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 -- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82600b0760 msgr2=0x7f8260145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82600b0760 0x7f8260145830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.406+0000 7f8264b64640 1 -- 172.21.15.72:0/577310592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82601445d0 con 0x7f826015bb90 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.407+0000 7f8264b64640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 0x7f82600b0220 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9b0 tx=0x7f825c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.407+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c017070 con 0x7f826015bb90 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.407+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c00c9a0 con 0x7f826015bb90 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.407+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82601448b0 con 0x7f826015bb90 2024-03-06T07:30:56.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.407+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c00cd90 con 0x7f826015bb90 2024-03-06T07:30:56.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.407+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8260144e70 con 0x7f826015bb90 2024-03-06T07:30:56.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.408+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8260143090 con 0x7f826015bb90 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.408+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f825c01a070 con 0x7f826015bb90 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.409+0000 7f8255ffb640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f824406b4d0 0x7f824406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.409+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f825c08a570 con 0x7f826015bb90 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.409+0000 7f8265365640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f824406b4d0 0x7f824406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:56.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.409+0000 7f8265365640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f824406b4d0 0x7f824406d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8250005880 tx=0x7f8250009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.413+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8260143090 con 0x7f826015bb90 2024-03-06T07:30:56.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.708+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f826004f330 con 0x7f826015bb90 2024-03-06T07:30:56.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.709+0000 7f8255ffb640 1 -- 172.21.15.72:0/577310592 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f825c058d60 con 0x7f826015bb90 2024-03-06T07:30:56.711 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:56.711 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:56.711 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:56.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f824406b4d0 msgr2=0x7f824406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f824406b4d0 0x7f824406d990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8250005880 tx=0x7f8250009040 comp rx=0 tx=0).stop 2024-03-06T07:30:56.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 msgr2=0x7f82600b0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:56.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 0x7f82600b0220 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9b0 tx=0x7f825c00ee70 comp rx=0 tx=0).stop 2024-03-06T07:30:56.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 shutdown_connections 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f824406b4d0 0x7f824406d990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82600b0760 0x7f8260145830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f826015bb90 0x7f82600b0220 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 --2- 172.21.15.72:0/577310592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82600cd610 0x7f8260143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 >> 172.21.15.72:0/577310592 conn(0x7f826006c0c0 msgr2=0x7f82600cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 shutdown_connections 2024-03-06T07:30:56.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:56.711+0000 7f82675c9640 1 -- 172.21.15.72:0/577310592 wait complete. 2024-03-06T07:30:56.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:56.925 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:57.761 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:57.926 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:58.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.024+0000 7f3fa7cc9640 1 Processor -- start 2024-03-06T07:30:58.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 -- start start 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa015dfe0 0x7f3fa00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3fa00cf830 0x7f3fa00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3fa00d27f0 0x7f3fa00d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa00d2c50 con 0x7f3fa00cf830 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fa00b31e0 con 0x7f3fa00d27f0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa7cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00b3d60 con 0x7f3fa015dfe0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.025+0000 7f3fa5264640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3fa00cf830 0x7f3fa00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3fa00cf830 0x7f3fa00d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47956/0 (socket says 172.21.15.72:47956) 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 -- 172.21.15.72:0/3530174056 learned_addr learned my addr 172.21.15.72:0/3530174056 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5a65640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa015dfe0 0x7f3fa00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa6266640 1 --2- 172.21.15.72:0/3530174056 >> v2:172.21.15.146:3300/0 conn(0x7f3fa00d27f0 0x7f3fa00d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 -- 172.21.15.72:0/3530174056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa015dfe0 msgr2=0x7f3fa00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa015dfe0 0x7f3fa00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 -- 172.21.15.72:0/3530174056 >> v2:172.21.15.146:3300/0 conn(0x7f3fa00d27f0 msgr2=0x7f3fa00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 --2- 172.21.15.72:0/3530174056 >> v2:172.21.15.146:3300/0 conn(0x7f3fa00d27f0 0x7f3fa00d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5264640 1 -- 172.21.15.72:0/3530174056 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa00b3430 con 0x7f3fa00cf830 2024-03-06T07:30:58.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.026+0000 7f3fa5a65640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa015dfe0 0x7f3fa00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.027+0000 7f3fa5264640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa00d22b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0067ad0 tx=0x7f3f9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5e0ea6e8b01213 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.027+0000 7f3fa4a63640 1 -- 172.21.15.72:0/3530174056 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f90035020 con 0x7f3fa00cf830 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.027+0000 7f3fa4a63640 1 -- 172.21.15.72:0/3530174056 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f90031870 con 0x7f3fa00cf830 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.027+0000 7f3fa4a63640 1 -- 172.21.15.72:0/3530174056 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f90031c40 con 0x7f3fa00cf830 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/3530174056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 msgr2=0x7f3fa00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa00d22b0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f3fa0067ad0 tx=0x7f3f9002b910 comp rx=0 tx=0).stop 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/3530174056 shutdown_connections 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/3530174056 >> v2:172.21.15.146:3300/0 conn(0x7f3fa00d27f0 0x7f3fa00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa00d22b0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/3530174056 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa015dfe0 0x7f3fa00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/3530174056 >> 172.21.15.72:0/3530174056 conn(0x7f3fa00d3cd0 msgr2=0x7f3fa00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/3530174056 shutdown_connections 2024-03-06T07:30:58.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.028+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/3530174056 wait complete. 2024-03-06T07:30:58.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.029+0000 7f3fa7cc9640 1 Processor -- start 2024-03-06T07:30:58.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.029+0000 7f3fa7cc9640 1 -- start start 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa0158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa00d27f0 0x7f3fa015ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa015dfe0 0x7f3fa00b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00d1c70 con 0x7f3fa00cf830 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00b39f0 con 0x7f3fa015dfe0 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fa00d1df0 con 0x7f3fa00d27f0 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa0158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47958/0 (socket says 172.21.15.72:47958) 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 -- 172.21.15.72:0/2074482060 learned_addr learned my addr 172.21.15.72:0/2074482060 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 -- 172.21.15.72:0/2074482060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa00d27f0 msgr2=0x7f3fa015ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa00d27f0 0x7f3fa015ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 -- 172.21.15.72:0/2074482060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa015dfe0 msgr2=0x7f3fa00b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa015dfe0 0x7f3fa00b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 -- 172.21.15.72:0/2074482060 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa0158aa0 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa5a65640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa0158560 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3f98007ef0 tx=0x7f3f9800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f9801a070 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f98003b90 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa00cdca0 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.030+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f98004030 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.031+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa00b7040 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.031+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa00b5200 con 0x7f3fa00cf830 2024-03-06T07:30:58.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.032+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3f9801d070 con 0x7f3fa00cf830 2024-03-06T07:30:58.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.032+0000 7f3f967fc640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:58.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.032+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3f9808a3d0 con 0x7f3fa00cf830 2024-03-06T07:30:58.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.032+0000 7f3fa5264640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:58.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.033+0000 7f3fa5264640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3f8406b4d0 0x7f3f8406d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3f90036000 tx=0x7f3f90030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:58.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.037+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3fa00b5200 con 0x7f3fa00cf830 2024-03-06T07:30:58.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.253+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fa004f330 con 0x7f3fa00cf830 2024-03-06T07:30:58.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.254+0000 7f3f967fc640 1 -- 172.21.15.72:0/2074482060 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f3f98010070 con 0x7f3fa00cf830 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:58.256 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3f8406b4d0 msgr2=0x7f3f8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3f8406b4d0 0x7f3f8406d990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3f90036000 tx=0x7f3f90030000 comp rx=0 tx=0).stop 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 msgr2=0x7f3fa0158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa0158560 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f3f98007ef0 tx=0x7f3f9800d4a0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 shutdown_connections 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3f8406b4d0 0x7f3f8406d990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3fa015dfe0 0x7f3fa00b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3fa00d27f0 0x7f3fa015ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 --2- 172.21.15.72:0/2074482060 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3fa00cf830 0x7f3fa0158560 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 >> 172.21.15.72:0/2074482060 conn(0x7f3fa00d3cd0 msgr2=0x7f3fa006cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 shutdown_connections 2024-03-06T07:30:58.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:58.256+0000 7f3fa7cc9640 1 -- 172.21.15.72:0/2074482060 wait complete. 2024-03-06T07:30:58.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:30:58.469 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:30:59.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:30:59.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.570+0000 7f4382eaf640 1 Processor -- start 2024-03-06T07:30:59.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.570+0000 7f4382eaf640 1 -- start start 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4382eaf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f437c15fa10 0x7f437c0ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4382eaf640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f437c0d16e0 0x7f437c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4382eaf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437c0cebe0 con 0x7f437c15bdd0 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4382eaf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c0ceec0 con 0x7f437c15fa10 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4382eaf640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f437c0cf1a0 con 0x7f437c0d16e0 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f437bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f437c15fa10 0x7f437c0ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4380c4b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f437c15bdd0 0x7f437c0c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50486/0 (socket says 172.21.15.72:50486) 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f4380c4b640 1 -- 172.21.15.72:0/4103584659 learned_addr learned my addr 172.21.15.72:0/4103584659 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:30:59.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f438144c640 1 --2- 172.21.15.72:0/4103584659 >> v2:172.21.15.146:3300/0 conn(0x7f437c0d16e0 0x7f437c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f437bfff640 1 -- 172.21.15.72:0/4103584659 >> v2:172.21.15.146:3300/0 conn(0x7f437c0d16e0 msgr2=0x7f437c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f437bfff640 1 --2- 172.21.15.72:0/4103584659 >> v2:172.21.15.146:3300/0 conn(0x7f437c0d16e0 0x7f437c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f437bfff640 1 -- 172.21.15.72:0/4103584659 >> v1:172.21.15.72:6789/0 conn(0x7f437c15bdd0 legacy=0x7f437c0c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f437b7fe640 1 -- 172.21.15.72:0/4103584659 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672097963 0 0) 0x7f437c0cebe0 con 0x7f437c15bdd0 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f437bfff640 1 -- 172.21.15.72:0/4103584659 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c0cc8e0 con 0x7f437c15fa10 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.571+0000 7f438144c640 1 --2- 172.21.15.72:0/4103584659 >> v2:172.21.15.146:3300/0 conn(0x7f437c0d16e0 0x7f437c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.572+0000 7f437bfff640 1 --2- 172.21.15.72:0/4103584659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15fa10 0x7f437c0ce270 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f437c0c5520 tx=0x7f436402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17eb553ad6a42e3e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.572+0000 7f437b7fe640 1 -- 172.21.15.72:0/4103584659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640315a0 con 0x7f437c15fa10 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.572+0000 7f437b7fe640 1 -- 172.21.15.72:0/4103584659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4364037070 con 0x7f437c15fa10 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.572+0000 7f437b7fe640 1 -- 172.21.15.72:0/4103584659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364031960 con 0x7f437c15fa10 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 -- 172.21.15.72:0/4103584659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15fa10 msgr2=0x7f437c0ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 --2- 172.21.15.72:0/4103584659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15fa10 0x7f437c0ce270 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f437c0c5520 tx=0x7f436402b840 comp rx=0 tx=0).stop 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 -- 172.21.15.72:0/4103584659 shutdown_connections 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 --2- 172.21.15.72:0/4103584659 >> v2:172.21.15.146:3300/0 conn(0x7f437c0d16e0 0x7f437c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 --2- 172.21.15.72:0/4103584659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15fa10 0x7f437c0ce270 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 -- 172.21.15.72:0/4103584659 >> 172.21.15.72:0/4103584659 conn(0x7f437c09a9f0 msgr2=0x7f437c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 -- 172.21.15.72:0/4103584659 shutdown_connections 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.573+0000 7f4382eaf640 1 -- 172.21.15.72:0/4103584659 wait complete. 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.575+0000 7f4382eaf640 1 Processor -- start 2024-03-06T07:30:59.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.575+0000 7f4382eaf640 1 -- start start 2024-03-06T07:30:59.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4382eaf640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f437c0d16e0 0x7f437c1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4382eaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 0x7f437c1f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4382eaf640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f437c15fa10 0x7f437c15cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4382eaf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0d2030 con 0x7f437c15bdd0 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4382eaf640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c15c1b0 con 0x7f437c0d16e0 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4382eaf640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f437c0ce7b0 con 0x7f437c15fa10 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 0x7f437c1f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 0x7f437c1f4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47972/0 (socket says 172.21.15.72:47972) 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 -- 172.21.15.72:0/2328324810 learned_addr learned my addr 172.21.15.72:0/2328324810 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4380c4b640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f437c0d16e0 0x7f437c1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f438144c640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f437c15fa10 0x7f437c15cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 -- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f437c15fa10 msgr2=0x7f437c15cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f437c15fa10 0x7f437c15cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 -- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f437c0d16e0 msgr2=0x7f437c1582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f437c0d16e0 0x7f437c1582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 -- 172.21.15.72:0/2328324810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c158820 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f4380c4b640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f437c0d16e0 0x7f437c1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f438144c640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f437c15fa10 0x7f437c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.576+0000 7f437bfff640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 0x7f437c1f4df0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f4364009890 tx=0x7f43640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.577+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364004230 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.577+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43640043d0 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.577+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c1f5760 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.577+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364003850 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.577+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c15c460 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.578+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c04f330 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.578+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f43640039f0 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.579+0000 7f43797fa640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f435c06b500 0x7f435c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.579+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f436402f030 con 0x7f437c15bdd0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.579+0000 7f4380c4b640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f435c06b500 0x7f435c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:30:59.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.579+0000 7f4380c4b640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f435c06b500 0x7f435c06d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f43740012e0 tx=0x7f4374009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:30:59.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.587+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4364041070 con 0x7f437c15bdd0 2024-03-06T07:30:59.762 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:30:59.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.803+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f437c0b7a40 con 0x7f437c15bdd0 2024-03-06T07:30:59.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.804+0000 7f43797fa640 1 -- 172.21.15.72:0/2328324810 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f436403ae20 con 0x7f437c15bdd0 2024-03-06T07:30:59.806 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:30:59.806 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:30:59.806 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:30:59.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f435c06b500 msgr2=0x7f435c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f435c06b500 0x7f435c06d9c0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f43740012e0 tx=0x7f4374009040 comp rx=0 tx=0).stop 2024-03-06T07:30:59.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 msgr2=0x7f437c1f4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 0x7f437c1f4df0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f4364009890 tx=0x7f43640046c0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 shutdown_connections 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f435c06b500 0x7f435c06d9c0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f437c15fa10 0x7f437c15cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f437c15bdd0 0x7f437c1f4df0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 --2- 172.21.15.72:0/2328324810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f437c0d16e0 0x7f437c1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 >> 172.21.15.72:0/2328324810 conn(0x7f437c09a9f0 msgr2=0x7f437c06f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 shutdown_connections 2024-03-06T07:30:59.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:30:59.806+0000 7f4382eaf640 1 -- 172.21.15.72:0/2328324810 wait complete. 2024-03-06T07:31:00.020 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:00.020 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:01.022 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:01.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 Processor -- start 2024-03-06T07:31:01.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 -- start start 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f569c15dfe0 0x7f569c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c0cf830 0x7f569c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0b3430 con 0x7f569c0d27f0 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c0d2c50 con 0x7f569c15dfe0 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f56a1b3a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0b31e0 con 0x7f569c0cf830 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.121+0000 7f569b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f569c15dfe0 0x7f569c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f569c15dfe0 0x7f569c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47976/0 (socket says 172.21.15.72:47976) 2024-03-06T07:31:01.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 -- 172.21.15.72:0/529289830 learned_addr learned my addr 172.21.15.72:0/529289830 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:01.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 -- 172.21.15.72:0/529289830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c0cf830 msgr2=0x7f569c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569affd640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c0cf830 0x7f569c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c0cf830 0x7f569c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 -- 172.21.15.72:0/529289830 >> v1:172.21.15.146:6789/0 conn(0x7f569c0d27f0 legacy=0x7f569c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:01.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 -- 172.21.15.72:0/529289830 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c0b3710 con 0x7f569c15dfe0 2024-03-06T07:31:01.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569affd640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c0cf830 0x7f569c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:01.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.122+0000 7f569b7fe640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c15dfe0 0x7f569c0cf2f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f568c009e50 tx=0x7f568c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e170a7e3d098da6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f569a7fc640 1 -- 172.21.15.72:0/529289830 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c02ca70 con 0x7f569c15dfe0 2024-03-06T07:31:01.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f569a7fc640 1 -- 172.21.15.72:0/529289830 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c02cc10 con 0x7f569c15dfe0 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f569a7fc640 1 -- 172.21.15.72:0/529289830 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c0343f0 con 0x7f569c15dfe0 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 -- 172.21.15.72:0/529289830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c15dfe0 msgr2=0x7f569c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c15dfe0 0x7f569c0cf2f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f568c009e50 tx=0x7f568c02bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 -- 172.21.15.72:0/529289830 shutdown_connections 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c0cf830 0x7f569c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/529289830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c15dfe0 0x7f569c0cf2f0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 -- 172.21.15.72:0/529289830 >> 172.21.15.72:0/529289830 conn(0x7f569c0d3cd0 msgr2=0x7f569c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 -- 172.21.15.72:0/529289830 shutdown_connections 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.123+0000 7f56a1b3a640 1 -- 172.21.15.72:0/529289830 wait complete. 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.124+0000 7f56a1b3a640 1 Processor -- start 2024-03-06T07:31:01.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 -- start start 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f569c0cf830 0x7f569c1f39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 0x7f569c14f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c15dfe0 0x7f569c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0d1c70 con 0x7f569c0d27f0 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0b3d60 con 0x7f569c0cf830 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f56a1b3a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0d1df0 con 0x7f569c15dfe0 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f569affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 0x7f569c14f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f569affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 0x7f569c14f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47980/0 (socket says 172.21.15.72:47980) 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.125+0000 7f569affd640 1 -- 172.21.15.72:0/44009941 learned_addr learned my addr 172.21.15.72:0/44009941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569b7fe640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f569c0cf830 0x7f569c1f39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569bfff640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c15dfe0 0x7f569c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569affd640 1 -- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c15dfe0 msgr2=0x7f569c0b2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569affd640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c15dfe0 0x7f569c0b2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569affd640 1 -- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f569c0cf830 msgr2=0x7f569c1f39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569affd640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f569c0cf830 0x7f569c1f39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569affd640 1 -- 172.21.15.72:0/44009941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c1f7c30 con 0x7f569c0d27f0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f569affd640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 0x7f569c14f090 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f568400c7f0 tx=0x7f568400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684015070 con 0x7f569c0d27f0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684007a30 con 0x7f569c0d27f0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c0cc460 con 0x7f569c0d27f0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684007e00 con 0x7f569c0d27f0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.126+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c14f5d0 con 0x7f569c0d27f0 2024-03-06T07:31:01.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.127+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c04f330 con 0x7f569c0d27f0 2024-03-06T07:31:01.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.128+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5684018070 con 0x7f569c0d27f0 2024-03-06T07:31:01.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.128+0000 7f5693fff640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f567c06b4d0 0x7f567c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:01.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.128+0000 7f569b7fe640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f567c06b4d0 0x7f567c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:01.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.128+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f568408a080 con 0x7f569c0d27f0 2024-03-06T07:31:01.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.129+0000 7f569b7fe640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f567c06b4d0 0x7f567c06d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f568c035000 tx=0x7f568c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:01.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.133+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5684058900 con 0x7f569c0d27f0 2024-03-06T07:31:01.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.350+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f569c0cebf0 con 0x7f569c0d27f0 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.351+0000 7f5693fff640 1 -- 172.21.15.72:0/44009941 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f568405c9a0 con 0x7f569c0d27f0 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f567c06b4d0 msgr2=0x7f567c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f567c06b4d0 0x7f567c06d990 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f568c035000 tx=0x7f568c030000 comp rx=0 tx=0).stop 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 msgr2=0x7f569c14f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 0x7f569c14f090 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f568400c7f0 tx=0x7f568400ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 shutdown_connections 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f567c06b4d0 0x7f567c06d990 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f569c15dfe0 0x7f569c0b2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f569c0d27f0 0x7f569c14f090 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 --2- 172.21.15.72:0/44009941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f569c0cf830 0x7f569c1f39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 >> 172.21.15.72:0/44009941 conn(0x7f569c0d3cd0 msgr2=0x7f569c0b41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 shutdown_connections 2024-03-06T07:31:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:01.353+0000 7f56a1b3a640 1 -- 172.21.15.72:0/44009941 wait complete. 2024-03-06T07:31:01.567 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:01.567 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:01.765 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:02.568 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.668+0000 7f93a3fe9640 1 Processor -- start 2024-03-06T07:31:02.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.668+0000 7f93a3fe9640 1 -- start start 2024-03-06T07:31:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a3fe9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 0x7f939c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a3fe9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f939c0d29c0 0x7f939c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a3fe9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939c0b3430 con 0x7f939c0cf830 2024-03-06T07:31:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a3fe9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939c0b31e0 con 0x7f939c0d29c0 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a3fe9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0b3d60 con 0x7f939c15dfe0 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a1584640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f939c0cf830 0x7f939c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55488/0 (socket says 172.21.15.72:55488) 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a1584640 1 -- 172.21.15.72:0/2867766761 learned_addr learned my addr 172.21.15.72:0/2867766761 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472819221 0 0) 0x7f939c0b3430 con 0x7f939c0cf830 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a1d85640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 0x7f939c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9388003520 con 0x7f939c0cf830 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a2586640 1 --2- 172.21.15.72:0/2867766761 >> v2:172.21.15.146:3300/0 conn(0x7f939c0d29c0 0x7f939c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266583755 0 0) 0x7f9388003520 con 0x7f939c0cf830 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 msgr2=0x7f939c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 0x7f939c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 >> v2:172.21.15.146:3300/0 conn(0x7f939c0d29c0 msgr2=0x7f939c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 --2- 172.21.15.72:0/2867766761 >> v2:172.21.15.146:3300/0 conn(0x7f939c0d29c0 0x7f939c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c0b3710 con 0x7f939c0cf830 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f938c001670 con 0x7f939c0cf830 2024-03-06T07:31:02.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 >> v1:172.21.15.72:6789/0 conn(0x7f939c0cf830 legacy=0x7f939c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93880039b0 0x7f9388003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 0x7f9388004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9388004d40 0x7f93880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0b31e0 con 0x7f93880039b0 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0b3d60 con 0x7f9388004d40 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.669+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f938800b9c0 con 0x7f939c15dfe0 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93880039b0 0x7f9388003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a2586640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9388004d40 0x7f93880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1584640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 0x7f9388004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 -- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 msgr2=0x7f9388004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15dfe0 0x7f9388004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 -- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9388004d40 msgr2=0x7f93880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9388004d40 0x7f93880051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 -- 172.21.15.72:0/2867766761 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9388003520 con 0x7f93880039b0 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.670+0000 7f93a1d85640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93880039b0 0x7f9388003dd0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f9398009ac0 tx=0x7f9388005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f08ce137a190f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398039070 con 0x7f93880039b0 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939802c6b0 con 0x7f93880039b0 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a0d83640 1 -- 172.21.15.72:0/2867766761 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9398009690 con 0x7f93880039b0 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a3fe9640 1 -- 172.21.15.72:0/2867766761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93880039b0 msgr2=0x7f9388003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/2867766761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f93880039b0 0x7f9388003dd0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f9398009ac0 tx=0x7f9388005830 comp rx=0 tx=0).stop 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a2586640 1 -- 172.21.15.72:0/2867766761 reap_dead start 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a3fe9640 1 -- 172.21.15.72:0/2867766761 shutdown_connections 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a3fe9640 1 -- 172.21.15.72:0/2867766761 >> 172.21.15.72:0/2867766761 conn(0x7f939c0d3cd0 msgr2=0x7f939c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a3fe9640 1 -- 172.21.15.72:0/2867766761 shutdown_connections 2024-03-06T07:31:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.671+0000 7f93a3fe9640 1 -- 172.21.15.72:0/2867766761 wait complete. 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.672+0000 7f93a3fe9640 1 Processor -- start 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 -- start start 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93880039b0 0x7f939c15c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15f0d0 0x7f939c160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 0x7f939c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0d1df0 con 0x7f939c0ccab0 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0d1c70 con 0x7f93880039b0 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a3fe9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939c15e3c0 con 0x7f939c15f0d0 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a2586640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 0x7f939c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a2586640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 0x7f939c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44418/0 (socket says 172.21.15.72:44418) 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a2586640 1 -- 172.21.15.72:0/1737112393 learned_addr learned my addr 172.21.15.72:0/1737112393 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a1584640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15f0d0 0x7f939c160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.673+0000 7f93a1d85640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93880039b0 0x7f939c15c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a2586640 1 -- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15f0d0 msgr2=0x7f939c160fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a2586640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15f0d0 0x7f939c160fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a2586640 1 -- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93880039b0 msgr2=0x7f939c15c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a2586640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93880039b0 0x7f939c15c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a2586640 1 -- 172.21.15.72:0/1737112393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c15c780 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a2586640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 0x7f939c0ccf60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f939c0b5770 tx=0x7f93940115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394011db0 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939401b070 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c15f5b0 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9394003990 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.674+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c15fb70 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.675+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c04f330 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.675+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9394005d10 con 0x7f939c0ccab0 2024-03-06T07:31:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.676+0000 7f93927fc640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f938006b500 0x7f938006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:02.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.676+0000 7f93a1d85640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f938006b500 0x7f938006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:02.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.676+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f939408a500 con 0x7f939c0ccab0 2024-03-06T07:31:02.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.676+0000 7f93a1d85640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f938006b500 0x7f938006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9398036000 tx=0x7f9398030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:02.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:02.680+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9394025020 con 0x7f939c0ccab0 2024-03-06T07:31:03.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.045+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f939c0b26e0 con 0x7f939c0ccab0 2024-03-06T07:31:03.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.045+0000 7f93927fc640 1 -- 172.21.15.72:0/1737112393 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f93940593d0 con 0x7f939c0ccab0 2024-03-06T07:31:03.048 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:03.048 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:03.048 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:31:03.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f938006b500 msgr2=0x7f938006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f938006b500 0x7f938006d9c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f9398036000 tx=0x7f9398030000 comp rx=0 tx=0).stop 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 msgr2=0x7f939c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 0x7f939c0ccf60 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f939c0b5770 tx=0x7f93940115a0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 shutdown_connections 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f938006b500 0x7f938006d9c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f939c0ccab0 0x7f939c0ccf60 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f939c15f0d0 0x7f939c160fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 --2- 172.21.15.72:0/1737112393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93880039b0 0x7f939c15c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 >> 172.21.15.72:0/1737112393 conn(0x7f939c0d3cd0 msgr2=0x7f939c06bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 shutdown_connections 2024-03-06T07:31:03.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:03.047+0000 7f93a3fe9640 1 -- 172.21.15.72:0/1737112393 wait complete. 2024-03-06T07:31:03.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:03.261 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:03.766 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:04.262 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:04.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.361+0000 7fe2aa28d640 1 Processor -- start 2024-03-06T07:31:04.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.361+0000 7fe2aa28d640 1 -- start start 2024-03-06T07:31:04.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.361+0000 7fe2aa28d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 0x7fe2a40b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.361+0000 7fe2aa28d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40b3430 con 0x7fe2a40d22b0 2024-03-06T07:31:04.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.361+0000 7fe2aa28d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2a40b3710 con 0x7fe2a415dfe0 2024-03-06T07:31:04.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.361+0000 7fe2aa28d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40b31e0 con 0x7fe2a40d2710 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe2a40d22b0 0x7fe2a40b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55504/0 (socket says 172.21.15.72:55504) 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a37fe640 1 -- 172.21.15.72:0/69150235 learned_addr learned my addr 172.21.15.72:0/69150235 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574328100 0 0) 0x7fe2a40b3430 con 0x7fe2a40d22b0 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe28c003520 con 0x7fe2a40d22b0 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a882a640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 0x7fe2a40b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1011085312 0 0) 0x7fe28c003520 con 0x7fe2a40d22b0 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 msgr2=0x7fe2a40b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 0x7fe2a40b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 >> v1:172.21.15.146:6789/0 conn(0x7fe2a415dfe0 legacy=0x7fe2a40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a40b39f0 con 0x7fe2a40d22b0 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe298001670 con 0x7fe2a40d22b0 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 >> v1:172.21.15.72:6789/0 conn(0x7fe2a40d22b0 legacy=0x7fe2a40b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28c0039e0 0x7fe28c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c004800 0x7fe28c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 0x7fe28c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40b31e0 con 0x7fe28c0039e0 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c00bb50 con 0x7fe28c004800 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c00bd50 con 0x7fe2a40d2710 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.362+0000 7fe2a3fff640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28c0039e0 0x7fe28c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a37fe640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c004800 0x7fe28c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a882a640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 0x7fe28c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a3fff640 1 -- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 msgr2=0x7fe28c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a3fff640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a40d2710 0x7fe28c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a3fff640 1 -- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c004800 msgr2=0x7fe28c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a3fff640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c004800 0x7fe28c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a3fff640 1 -- 172.21.15.72:0/69150235 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28c003520 con 0x7fe28c0039e0 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a37fe640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c004800 0x7fe28c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.363+0000 7fe2a3fff640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28c0039e0 0x7fe28c003e20 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fe294001750 tx=0x7fe28c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d6f947f3b43bca server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294001370 con 0x7fe28c0039e0 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2940052d0 con 0x7fe28c0039e0 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2a2ffd640 1 -- 172.21.15.72:0/69150235 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe294003830 con 0x7fe28c0039e0 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2aa28d640 1 -- 172.21.15.72:0/69150235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28c0039e0 msgr2=0x7fe28c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/69150235 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28c0039e0 0x7fe28c003e20 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fe294001750 tx=0x7fe28c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2a882a640 1 -- 172.21.15.72:0/69150235 reap_dead start 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2aa28d640 1 -- 172.21.15.72:0/69150235 shutdown_connections 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2aa28d640 1 -- 172.21.15.72:0/69150235 >> 172.21.15.72:0/69150235 conn(0x7fe2a40d3cd0 msgr2=0x7fe2a40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2aa28d640 1 -- 172.21.15.72:0/69150235 shutdown_connections 2024-03-06T07:31:04.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.364+0000 7fe2aa28d640 1 -- 172.21.15.72:0/69150235 wait complete. 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.365+0000 7fe2aa28d640 1 Processor -- start 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.365+0000 7fe2aa28d640 1 -- start start 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2aa28d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c0039e0 0x7fe2a41f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2aa28d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a409e870 0x7fe2a409f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2aa28d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 0x7fe2a4151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2aa28d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40cfa00 con 0x7fe2a40a1450 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2aa28d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40b3d60 con 0x7fe28c0039e0 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2aa28d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a40d1c70 con 0x7fe2a409e870 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2a882a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 0x7fe2a4151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2a882a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 0x7fe2a4151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44436/0 (socket says 172.21.15.72:44436) 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.366+0000 7fe2a882a640 1 -- 172.21.15.72:0/88012560 learned_addr learned my addr 172.21.15.72:0/88012560 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a37fe640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a409e870 0x7fe2a409f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a882a640 1 -- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a409e870 msgr2=0x7fe2a409f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a882a640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a409e870 0x7fe2a409f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a882a640 1 -- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c0039e0 msgr2=0x7fe2a41f7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a882a640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c0039e0 0x7fe2a41f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a882a640 1 -- 172.21.15.72:0/88012560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a41f7d70 con 0x7fe2a40a1450 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a882a640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 0x7fe2a4151180 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fe2a408c940 tx=0x7fe29c009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c007cf0 con 0x7fe2a40a1450 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29c019070 con 0x7fe2a40a1450 2024-03-06T07:31:04.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe29c014650 con 0x7fe2a40a1450 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a409dad0 con 0x7fe2a40a1450 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.367+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a409dd90 con 0x7fe2a40a1450 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.368+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a404f330 con 0x7fe2a40a1450 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.368+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe29c0147f0 con 0x7fe2a40a1450 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.369+0000 7fe2a0ff9640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe29006b4d0 0x7fe29006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.369+0000 7fe2a3fff640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe29006b4d0 0x7fe29006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.369+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe29c08a620 con 0x7fe2a40a1450 2024-03-06T07:31:04.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.369+0000 7fe2a3fff640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe29006b4d0 0x7fe29006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe294001750 tx=0x7fe29402f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:04.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.373+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe2a41f7d70 con 0x7fe2a40a1450 2024-03-06T07:31:04.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.624+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2a40b2c70 con 0x7fe2a40a1450 2024-03-06T07:31:04.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.624+0000 7fe2a0ff9640 1 -- 172.21.15.72:0/88012560 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7fe29c0594f0 con 0x7fe2a40a1450 2024-03-06T07:31:04.626 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:04.626 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:04.627 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:31:04.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe29006b4d0 msgr2=0x7fe29006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe29006b4d0 0x7fe29006d990 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe294001750 tx=0x7fe29402f000 comp rx=0 tx=0).stop 2024-03-06T07:31:04.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 msgr2=0x7fe2a4151180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 0x7fe2a4151180 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fe2a408c940 tx=0x7fe29c009d20 comp rx=0 tx=0).stop 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 shutdown_connections 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe29006b4d0 0x7fe29006d990 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2a40a1450 0x7fe2a4151180 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe2a409e870 0x7fe2a409f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 --2- 172.21.15.72:0/88012560 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe28c0039e0 0x7fe2a41f7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 >> 172.21.15.72:0/88012560 conn(0x7fe2a40d3cd0 msgr2=0x7fe2a406ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 shutdown_connections 2024-03-06T07:31:04.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:04.626+0000 7fe2aa28d640 1 -- 172.21.15.72:0/88012560 wait complete. 2024-03-06T07:31:04.843 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:04.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:05.768 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:05.844 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.944+0000 7f06c260e640 1 Processor -- start 2024-03-06T07:31:05.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.944+0000 7f06c260e640 1 -- start start 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06c260e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f06bc15dfe0 0x7f06bc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06c260e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0d29c0 0x7f06bc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06c260e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06bc0b3430 con 0x7f06bc0cf830 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06c260e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06bc0b31e0 con 0x7f06bc15dfe0 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06c260e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0b3d60 con 0x7f06bc0d29c0 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06bb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f06bc0cf830 0x7f06bc0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55518/0 (socket says 172.21.15.72:55518) 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.945+0000 7f06bb7fe640 1 -- 172.21.15.72:0/2760191696 learned_addr learned my addr 172.21.15.72:0/2760191696 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06bbfff640 1 --2- 172.21.15.72:0/2760191696 >> v2:172.21.15.146:3300/0 conn(0x7f06bc15dfe0 0x7f06bc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06c0bab640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0d29c0 0x7f06bc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407681744 0 0) 0x7f06bc0b3430 con 0x7f06bc0cf830 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a4003520 con 0x7f06bc0cf830 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1876782236 0 0) 0x7f06a4003520 con 0x7f06bc0cf830 2024-03-06T07:31:05.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0d29c0 msgr2=0x7f06bc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0d29c0 0x7f06bc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 >> v2:172.21.15.146:3300/0 conn(0x7f06bc15dfe0 msgr2=0x7f06bc0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 --2- 172.21.15.72:0/2760191696 >> v2:172.21.15.146:3300/0 conn(0x7f06bc15dfe0 0x7f06bc0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc0b3710 con 0x7f06bc0cf830 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f06b0001670 con 0x7f06bc0cf830 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 >> v1:172.21.15.72:6789/0 conn(0x7f06bc0cf830 legacy=0x7f06bc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a40039b0 0x7f06a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a4004800 0x7f06a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06a40051c0 0x7f06a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0b31e0 con 0x7f06a40051c0 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0b3d60 con 0x7f06a40039b0 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a400bf30 con 0x7f06a4004800 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06c0bab640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0d29c0 0x7f06bc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:05.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06c0bab640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06a40051c0 0x7f06a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06bbfff640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a40039b0 0x7f06a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.946+0000 7f06bb7fe640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a4004800 0x7f06a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.947+0000 7f06c0bab640 1 -- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a4004800 msgr2=0x7f06a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.947+0000 7f06c0bab640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a4004800 0x7f06a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.947+0000 7f06c0bab640 1 -- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a40039b0 msgr2=0x7f06a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.947+0000 7f06c0bab640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a40039b0 0x7f06a4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.947+0000 7f06c0bab640 1 -- 172.21.15.72:0/2760191696 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4003520 con 0x7f06a40051c0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c0bab640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06a40051c0 0x7f06a400b9c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009f00 tx=0x7f06a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c260d3bc3c59c037 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4032070 con 0x7f06a40051c0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b4030070 con 0x7f06a40051c0 2024-03-06T07:31:05.952 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06baffd640 1 -- 172.21.15.72:0/2760191696 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4009b50 con 0x7f06a40051c0 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c260e640 1 -- 172.21.15.72:0/2760191696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06a40051c0 msgr2=0x7f06a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c260e640 1 --2- 172.21.15.72:0/2760191696 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06a40051c0 0x7f06a400b9c0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009f00 tx=0x7f06a4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c0bab640 1 -- 172.21.15.72:0/2760191696 reap_dead start 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c260e640 1 -- 172.21.15.72:0/2760191696 shutdown_connections 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c260e640 1 -- 172.21.15.72:0/2760191696 >> 172.21.15.72:0/2760191696 conn(0x7f06bc0d3cd0 msgr2=0x7f06bc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c260e640 1 -- 172.21.15.72:0/2760191696 shutdown_connections 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.948+0000 7f06c260e640 1 -- 172.21.15.72:0/2760191696 wait complete. 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.950+0000 7f06c260e640 1 Processor -- start 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.950+0000 7f06c260e640 1 -- start start 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0a00d0 0x7f06bc160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 0x7f06bc15b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06bc0a04b0 0x7f06bc15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0d1df0 con 0x7f06bc160fa0 2024-03-06T07:31:05.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0d1c70 con 0x7f06bc0a04b0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc15e3c0 con 0x7f06bc0a00d0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 0x7f06bc15b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 0x7f06bc15b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44458/0 (socket says 172.21.15.72:44458) 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 -- 172.21.15.72:0/3241238388 learned_addr learned my addr 172.21.15.72:0/3241238388 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bbfff640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0a00d0 0x7f06bc160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c0bab640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06bc0a04b0 0x7f06bc15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 -- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0a00d0 msgr2=0x7f06bc160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0a00d0 0x7f06bc160a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 -- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06bc0a04b0 msgr2=0x7f06bc15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06bc0a04b0 0x7f06bc15bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 -- 172.21.15.72:0/3241238388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc15f9c0 con 0x7f06bc160fa0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06bb7fe640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 0x7f06bc15b750 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f06b00072c0 tx=0x7f06b0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b000f9c0 con 0x7f06bc160fa0 2024-03-06T07:31:05.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b0018070 con 0x7f06bc160fa0 2024-03-06T07:31:05.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc15fca0 con 0x7f06bc160fa0 2024-03-06T07:31:05.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b000fdc0 con 0x7f06bc160fa0 2024-03-06T07:31:05.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.952+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc160260 con 0x7f06bc160fa0 2024-03-06T07:31:05.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.954+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc04f330 con 0x7f06bc160fa0 2024-03-06T07:31:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.954+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f06b001d460 con 0x7f06bc160fa0 2024-03-06T07:31:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.955+0000 7f06b8ff9640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06a806b500 0x7f06a806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.955+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f06b008a380 con 0x7f06bc160fa0 2024-03-06T07:31:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.955+0000 7f06bbfff640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06a806b500 0x7f06a806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:05.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.955+0000 7f06bbfff640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06a806b500 0x7f06a806d9c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009b00 tx=0x7f06ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:05.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:05.959+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f06b0059250 con 0x7f06bc160fa0 2024-03-06T07:31:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.175+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06bc0b2c50 con 0x7f06bc160fa0 2024-03-06T07:31:06.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.176+0000 7f06b8ff9640 1 -- 172.21.15.72:0/3241238388 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f06b0058b50 con 0x7f06bc160fa0 2024-03-06T07:31:06.178 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:06.178 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:06.179 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:31:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06a806b500 msgr2=0x7f06a806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06a806b500 0x7f06a806d9c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009b00 tx=0x7f06ac008040 comp rx=0 tx=0).stop 2024-03-06T07:31:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 msgr2=0x7f06bc15b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 0x7f06bc15b750 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f06b00072c0 tx=0x7f06b0007780 comp rx=0 tx=0).stop 2024-03-06T07:31:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 shutdown_connections 2024-03-06T07:31:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f06a806b500 0x7f06a806d9c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06bc0a04b0 0x7f06bc15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06bc160fa0 0x7f06bc15b750 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 --2- 172.21.15.72:0/3241238388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06bc0a00d0 0x7f06bc160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 >> 172.21.15.72:0/3241238388 conn(0x7f06bc0d3cd0 msgr2=0x7f06bc0b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 shutdown_connections 2024-03-06T07:31:06.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:06.178+0000 7f06c260e640 1 -- 172.21.15.72:0/3241238388 wait complete. 2024-03-06T07:31:06.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:06.392 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:07.393 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.495+0000 7f2b7713e640 1 Processor -- start 2024-03-06T07:31:07.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.495+0000 7f2b7713e640 1 -- start start 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b7713e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b700d16e0 0x7f2b700d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b7713e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700cebe0 con 0x7f2b7015fa10 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b7713e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b700ceec0 con 0x7f2b7015bdd0 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b7713e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b700ce980 con 0x7f2b700d16e0 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b6ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2b7015fa10 0x7f2b700ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55532/0 (socket says 172.21.15.72:55532) 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b6ffff640 1 -- 172.21.15.72:0/3054199149 learned_addr learned my addr 172.21.15.72:0/3054199149 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b756db640 1 --2- 172.21.15.72:0/3054199149 >> v2:172.21.15.146:3300/0 conn(0x7f2b700d16e0 0x7f2b700d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815462802 0 0) 0x7f2b700cebe0 con 0x7f2b7015fa10 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.496+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58003520 con 0x7f2b7015fa10 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191009485 0 0) 0x7f2b58003520 con 0x7f2b7015fa10 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 >> v2:172.21.15.146:3300/0 conn(0x7f2b700d16e0 msgr2=0x7f2b700d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/3054199149 >> v2:172.21.15.146:3300/0 conn(0x7f2b700d16e0 0x7f2b700d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 >> v1:172.21.15.146:6789/0 conn(0x7f2b7015bdd0 legacy=0x7f2b700c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700cf1a0 con 0x7f2b7015fa10 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2b64001690 con 0x7f2b7015fa10 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 >> v1:172.21.15.72:6789/0 conn(0x7f2b7015fa10 legacy=0x7f2b700ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b580043b0 0x7f2b58004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b58005120 0x7f2b5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700ce980 con 0x7f2b580039b0 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5800c060 con 0x7f2b580043b0 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5800c260 con 0x7f2b58005120 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b74eda640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b6ffff640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b580043b0 0x7f2b58004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b756db640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b58005120 0x7f2b5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b74eda640 1 -- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b58005120 msgr2=0x7f2b5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b74eda640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b58005120 0x7f2b5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b74eda640 1 -- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b580043b0 msgr2=0x7f2b58004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b74eda640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b580043b0 0x7f2b58004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b74eda640 1 -- 172.21.15.72:0/3054199149 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58003520 con 0x7f2b580039b0 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.497+0000 7f2b756db640 1 -- 172.21.15.72:0/3054199149 reap_dead start 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b74eda640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003d90 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f2b60001860 tx=0x7f2b58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c114c2f9718752 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b60038070 con 0x7f2b580039b0 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b60033410 con 0x7f2b580039b0 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/3054199149 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b60001430 con 0x7f2b580039b0 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b7713e640 1 -- 172.21.15.72:0/3054199149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 msgr2=0x7f2b58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b7713e640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003d90 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f2b60001860 tx=0x7f2b58005830 comp rx=0 tx=0).stop 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b7713e640 1 -- 172.21.15.72:0/3054199149 shutdown_connections 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.498+0000 7f2b7713e640 1 --2- 172.21.15.72:0/3054199149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003d90 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.499+0000 7f2b7713e640 1 -- 172.21.15.72:0/3054199149 >> 172.21.15.72:0/3054199149 conn(0x7f2b7009a9f0 msgr2=0x7f2b7009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.499+0000 7f2b7713e640 1 -- 172.21.15.72:0/3054199149 shutdown_connections 2024-03-06T07:31:07.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.499+0000 7f2b7713e640 1 -- 172.21.15.72:0/3054199149 wait complete. 2024-03-06T07:31:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.500+0000 7f2b7713e640 1 Processor -- start 2024-03-06T07:31:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.500+0000 7f2b7713e640 1 -- start start 2024-03-06T07:31:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b7015a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b700b7360 0x7f2b700b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b700b5620 0x7f2b701f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7015c1b0 con 0x7f2b580039b0 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700cc5d0 con 0x7f2b700b5620 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b700ce270 con 0x7f2b700b7360 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b7015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b7015a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44474/0 (socket says 172.21.15.72:44474) 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 -- 172.21.15.72:0/2014889722 learned_addr learned my addr 172.21.15.72:0/2014889722 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b6ffff640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b700b7360 0x7f2b700b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b756db640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b700b5620 0x7f2b701f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 -- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b700b7360 msgr2=0x7f2b700b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b700b7360 0x7f2b700b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 -- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b700b5620 msgr2=0x7f2b701f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b700b5620 0x7f2b701f1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 -- 172.21.15.72:0/2014889722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700b62c0 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b74eda640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b7015a970 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2b60036040 tx=0x7f2b60001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b60043020 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6003e670 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b60038040 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b700b65a0 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.501+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b700b6b60 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.502+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7004f330 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.503+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2b60046020 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.503+0000 7f2b6d7fa640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b5c06b4d0 0x7f2b5c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.503+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b600abc30 con 0x7f2b580039b0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.503+0000 7f2b6ffff640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b5c06b4d0 0x7f2b5c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:07.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.504+0000 7f2b6ffff640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b5c06b4d0 0x7f2b5c06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2b700d05d0 tx=0x7f2b64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:07.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.512+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2b6004d030 con 0x7f2b580039b0 2024-03-06T07:31:07.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.726+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b700b7a40 con 0x7f2b580039b0 2024-03-06T07:31:07.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.726+0000 7f2b6d7fa640 1 -- 172.21.15.72:0/2014889722 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3463 (secure 0 0 0) 0x7f2b6007ac20 con 0x7f2b580039b0 2024-03-06T07:31:07.729 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:07.729 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":88,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:29:40.508609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:07.729 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 88 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b5c06b4d0 msgr2=0x7f2b5c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b5c06b4d0 0x7f2b5c06d990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2b700d05d0 tx=0x7f2b64009040 comp rx=0 tx=0).stop 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 msgr2=0x7f2b7015a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b7015a970 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2b60036040 tx=0x7f2b60001230 comp rx=0 tx=0).stop 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 shutdown_connections 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2b5c06b4d0 0x7f2b5c06d990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b700b5620 0x7f2b701f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b700b7360 0x7f2b700b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 --2- 172.21.15.72:0/2014889722 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b7015a970 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 >> 172.21.15.72:0/2014889722 conn(0x7f2b7009a9f0 msgr2=0x7f2b70070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 shutdown_connections 2024-03-06T07:31:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:07.729+0000 7f2b7713e640 1 -- 172.21.15.72:0/2014889722 wait complete. 2024-03-06T07:31:07.770 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:07.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:29:40.508609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:07.944 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:08.945 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:09.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 Processor -- start 2024-03-06T07:31:09.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 -- start start 2024-03-06T07:31:09.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f145c0cfff0 0x7f145c0d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 0x7f145c0ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145c161980 con 0x7f145c15bb20 2024-03-06T07:31:09.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145c0d2500 con 0x7f145c0cfff0 2024-03-06T07:31:09.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.044+0000 7f1463f1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145c161c60 con 0x7f145c0cc8b0 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1461cba640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f145c15bb20 0x7f145c0d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55546/0 (socket says 172.21.15.72:55546) 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1461cba640 1 -- 172.21.15.72:0/4132682401 learned_addr learned my addr 172.21.15.72:0/4132682401 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f14614b9640 1 --2- 172.21.15.72:0/4132682401 >> v2:172.21.15.146:3300/0 conn(0x7f145c0cfff0 0x7f145c0d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f14624bb640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 0x7f145c0ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 271484311 0 0) 0x7f145c161980 con 0x7f145c15bb20 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448003520 con 0x7f145c15bb20 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 109130913 0 0) 0x7f1448003520 con 0x7f145c15bb20 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 msgr2=0x7f145c0ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 0x7f145c0ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 >> v2:172.21.15.146:3300/0 conn(0x7f145c0cfff0 msgr2=0x7f145c0d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 --2- 172.21.15.72:0/4132682401 >> v2:172.21.15.146:3300/0 conn(0x7f145c0cfff0 0x7f145c0d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c161f40 con 0x7f145c15bb20 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f1458001670 con 0x7f145c15bb20 2024-03-06T07:31:09.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 >> v1:172.21.15.72:6789/0 conn(0x7f145c15bb20 legacy=0x7f145c0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f1448003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1448004800 0x7f1448004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 0x7f14480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0d2500 con 0x7f14480039b0 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c161c60 con 0x7f1448004800 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144800b9c0 con 0x7f145c0cc8b0 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f1461cba640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f1448003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.045+0000 7f14614b9640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1448004800 0x7f1448004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f14624bb640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 0x7f14480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f1461cba640 1 -- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 msgr2=0x7f14480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f1461cba640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0cc8b0 0x7f14480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f1461cba640 1 -- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1448004800 msgr2=0x7f1448004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:09.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f1461cba640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1448004800 0x7f1448004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f1461cba640 1 -- 172.21.15.72:0/4132682401 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1448003520 con 0x7f14480039b0 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.046+0000 7f1461cba640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f1448003dd0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f1458001050 tx=0x7f1448005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9b7e68a0a3abf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458007d70 con 0x7f14480039b0 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458030070 con 0x7f14480039b0 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1460cb8640 1 -- 172.21.15.72:0/4132682401 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145802cd80 con 0x7f14480039b0 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1463f1e640 1 -- 172.21.15.72:0/4132682401 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 msgr2=0x7f1448003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1463f1e640 1 --2- 172.21.15.72:0/4132682401 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f1448003dd0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f1458001050 tx=0x7f1448005830 comp rx=0 tx=0).stop 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f14624bb640 1 -- 172.21.15.72:0/4132682401 reap_dead start 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1463f1e640 1 -- 172.21.15.72:0/4132682401 shutdown_connections 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1463f1e640 1 -- 172.21.15.72:0/4132682401 >> 172.21.15.72:0/4132682401 conn(0x7f145c09a9f0 msgr2=0x7f145c09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1463f1e640 1 -- 172.21.15.72:0/4132682401 shutdown_connections 2024-03-06T07:31:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.047+0000 7f1463f1e640 1 -- 172.21.15.72:0/4132682401 wait complete. 2024-03-06T07:31:09.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.048+0000 7f1463f1e640 1 Processor -- start 2024-03-06T07:31:09.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.048+0000 7f1463f1e640 1 -- start start 2024-03-06T07:31:09.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1463f1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f145c0b6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1463f1e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0af940 0x7f145c0c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1463f1e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f145c0afdc0 0x7f145c1f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1463f1e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0d1980 con 0x7f14480039b0 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1463f1e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145c162500 con 0x7f145c0afdc0 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1463f1e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145c0d1b00 con 0x7f145c0af940 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f145c0b6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f145c0b6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44498/0 (socket says 172.21.15.72:44498) 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 -- 172.21.15.72:0/1913734264 learned_addr learned my addr 172.21.15.72:0/1913734264 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 -- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0af940 msgr2=0x7f145c0c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0af940 0x7f145c0c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f14624bb640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f145c0afdc0 0x7f145c1f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 -- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f145c0afdc0 msgr2=0x7f145c1f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f145c0afdc0 0x7f145c1f73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.049+0000 7f1461cba640 1 -- 172.21.15.72:0/1913734264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c0b4a30 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.050+0000 7f1461cba640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f145c0b6f30 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f145800c300 tx=0x7f1458000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.050+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458004bf0 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.050+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458004d90 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.050+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458007270 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.050+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f145c0b4d10 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.050+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f145c1f6ec0 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.051+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145c04f330 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.051+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1458007410 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.052+0000 7f14527fc640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f144006b4d0 0x7f144006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.052+0000 7f14614b9640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f144006b4d0 0x7f144006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.052+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f14580ab700 con 0x7f14480039b0 2024-03-06T07:31:09.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.052+0000 7f14614b9640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f144006b4d0 0x7f144006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f144c006b10 tx=0x7f144c006aa0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:09.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.056+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1458047350 con 0x7f14480039b0 2024-03-06T07:31:09.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.291+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f145c0747a0 con 0x7f14480039b0 2024-03-06T07:31:09.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.291+0000 7f14527fc640 1 -- 172.21.15.72:0/1913734264 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f145807a5d0 con 0x7f14480039b0 2024-03-06T07:31:09.294 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:09.294 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:09.294 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:09.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f144006b4d0 msgr2=0x7f144006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f144006b4d0 0x7f144006d990 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f144c006b10 tx=0x7f144c006aa0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 msgr2=0x7f145c0b6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f145c0b6f30 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f145800c300 tx=0x7f1458000fe0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 shutdown_connections 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f144006b4d0 0x7f144006d990 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f145c0afdc0 0x7f145c1f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f145c0af940 0x7f145c0c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 --2- 172.21.15.72:0/1913734264 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14480039b0 0x7f145c0b6f30 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 >> 172.21.15.72:0/1913734264 conn(0x7f145c09a9f0 msgr2=0x7f145c06eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 shutdown_connections 2024-03-06T07:31:09.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:09.293+0000 7f1463f1e640 1 -- 172.21.15.72:0/1913734264 wait complete. 2024-03-06T07:31:09.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:09.507 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:09.773 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:10.508 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:10.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.606+0000 7fcb7e481640 1 Processor -- start 2024-03-06T07:31:10.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.606+0000 7fcb7e481640 1 -- start start 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7e481640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb7815dfe0 0x7fcb780cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7e481640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb780d29c0 0x7fcb780b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7e481640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780b3430 con 0x7fcb780cf830 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7e481640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb780b31e0 con 0x7fcb780d29c0 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7e481640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780b3d60 con 0x7fcb7815dfe0 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7ca1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb780d29c0 0x7fcb780b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb6f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcb780cf830 0x7fcb780d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55556/0 (socket says 172.21.15.72:55556) 2024-03-06T07:31:10.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb7ca1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb780d29c0 0x7fcb780b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44510/0 (socket says 172.21.15.72:44510) 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.607+0000 7fcb6f7fe640 1 -- 172.21.15.72:0/2412001491 learned_addr learned my addr 172.21.15.72:0/2412001491 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb77fff640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb7815dfe0 0x7fcb780cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/2412001491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb7815dfe0 msgr2=0x7fcb780cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb7ca1e640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb7815dfe0 0x7fcb780cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/2412001491 >> v1:172.21.15.72:6789/0 conn(0x7fcb780cf830 legacy=0x7fcb780d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb777fe640 1 -- 172.21.15.72:0/2412001491 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432176969 0 0) 0x7fcb780b3430 con 0x7fcb780cf830 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/2412001491 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780b3710 con 0x7fcb780d29c0 2024-03-06T07:31:10.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.608+0000 7fcb77fff640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb7815dfe0 0x7fcb780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:10.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.609+0000 7fcb7ca1e640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb780d29c0 0x7fcb780b41f0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009d90 tx=0x7fcb7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917dc95dec69aff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.609+0000 7fcb777fe640 1 -- 172.21.15.72:0/2412001491 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70039070 con 0x7fcb780d29c0 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.609+0000 7fcb777fe640 1 -- 172.21.15.72:0/2412001491 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7002cc50 con 0x7fcb780d29c0 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.609+0000 7fcb777fe640 1 -- 172.21.15.72:0/2412001491 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70034440 con 0x7fcb780d29c0 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.609+0000 7fcb7e481640 1 -- 172.21.15.72:0/2412001491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb780d29c0 msgr2=0x7fcb780b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.609+0000 7fcb7e481640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb780d29c0 0x7fcb780b41f0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009d90 tx=0x7fcb7002bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.610+0000 7fcb7e481640 1 -- 172.21.15.72:0/2412001491 shutdown_connections 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.610+0000 7fcb7e481640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb780d29c0 0x7fcb780b41f0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.610+0000 7fcb7e481640 1 --2- 172.21.15.72:0/2412001491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb7815dfe0 0x7fcb780cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.610+0000 7fcb7e481640 1 -- 172.21.15.72:0/2412001491 >> 172.21.15.72:0/2412001491 conn(0x7fcb780d3cd0 msgr2=0x7fcb780d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.610+0000 7fcb7e481640 1 -- 172.21.15.72:0/2412001491 shutdown_connections 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.610+0000 7fcb7e481640 1 -- 172.21.15.72:0/2412001491 wait complete. 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.611+0000 7fcb7e481640 1 Processor -- start 2024-03-06T07:31:10.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.611+0000 7fcb7e481640 1 -- start start 2024-03-06T07:31:10.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb780cf830 0x7fcb780ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb780d29c0 0x7fcb7815f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 0x7fcb780cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780d1df0 con 0x7fcb7815dfe0 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb780d1c70 con 0x7fcb780d29c0 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7815e3c0 con 0x7fcb780cf830 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 0x7fcb780cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 0x7fcb780cb440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44522/0 (socket says 172.21.15.72:44522) 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/849977102 learned_addr learned my addr 172.21.15.72:0/849977102 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb77fff640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb780cf830 0x7fcb780ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb6f7fe640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb780d29c0 0x7fcb7815f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb780cf830 msgr2=0x7fcb780ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb780cf830 0x7fcb780ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb780d29c0 msgr2=0x7fcb7815f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb780d29c0 0x7fcb7815f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 -- 172.21.15.72:0/849977102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7815faa0 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb6f7fe640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb780d29c0 0x7fcb7815f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7ca1e640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 0x7fcb780cb440 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009d90 tx=0x7fcb70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70044020 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb7003a040 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7815fd80 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb70039050 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.612+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb780caf20 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.613+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb780b3360 con 0x7fcb7815dfe0 2024-03-06T07:31:10.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.614+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcb70003990 con 0x7fcb7815dfe0 2024-03-06T07:31:10.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.614+0000 7fcb757fa640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5806b4d0 0x7fcb5806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:10.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.614+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcb700ac6f0 con 0x7fcb7815dfe0 2024-03-06T07:31:10.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.614+0000 7fcb77fff640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5806b4d0 0x7fcb5806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:10.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.615+0000 7fcb77fff640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5806b4d0 0x7fcb5806d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800a950 tx=0x7fcb68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:10.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.619+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcb780b3360 con 0x7fcb7815dfe0 2024-03-06T07:31:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.829+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb7804f330 con 0x7fcb7815dfe0 2024-03-06T07:31:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.829+0000 7fcb757fa640 1 -- 172.21.15.72:0/849977102 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fcb7004e060 con 0x7fcb7815dfe0 2024-03-06T07:31:10.832 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:10.832 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5806b4d0 msgr2=0x7fcb5806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5806b4d0 0x7fcb5806d990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fcb6800a950 tx=0x7fcb68008040 comp rx=0 tx=0).stop 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 msgr2=0x7fcb780cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 0x7fcb780cb440 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fcb70009d90 tx=0x7fcb70004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 shutdown_connections 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fcb5806b4d0 0x7fcb5806d990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb7815dfe0 0x7fcb780cb440 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb780d29c0 0x7fcb7815f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 --2- 172.21.15.72:0/849977102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb780cf830 0x7fcb780ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.831+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 >> 172.21.15.72:0/849977102 conn(0x7fcb780d3cd0 msgr2=0x7fcb780b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:10.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.832+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 shutdown_connections 2024-03-06T07:31:10.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:10.832+0000 7fcb7e481640 1 -- 172.21.15.72:0/849977102 wait complete. 2024-03-06T07:31:11.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:11.046 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:11.775 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:12.047 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:12.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.158+0000 7f8fa8088640 1 Processor -- start 2024-03-06T07:31:12.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.158+0000 7f8fa8088640 1 -- start start 2024-03-06T07:31:12.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa8088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa8088640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fa00cf830 0x7f8fa00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa8088640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00b3430 con 0x7f8fa00d27f0 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa8088640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00d2c50 con 0x7f8fa00cf830 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa8088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00b31e0 con 0x7f8fa015dfe0 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5e24640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5623640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fa00cf830 0x7f8fa00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa6625640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8fa00d27f0 0x7f8fa00d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:57562/0 (socket says 172.21.15.72:57562) 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa6625640 1 -- 172.21.15.72:0/3306731144 learned_addr learned my addr 172.21.15.72:0/3306731144 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa4e22640 1 -- 172.21.15.72:0/3306731144 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057139408 0 0) 0x7f8fa00b3430 con 0x7f8fa00d27f0 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa4e22640 1 -- 172.21.15.72:0/3306731144 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003520 con 0x7f8fa00d27f0 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5623640 1 -- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa015dfe0 msgr2=0x7f8fa00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5623640 1 --2- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5623640 1 -- 172.21.15.72:0/3306731144 >> v1:172.21.15.146:6789/0 conn(0x7f8fa00d27f0 legacy=0x7f8fa00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5623640 1 -- 172.21.15.72:0/3306731144 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00b3710 con 0x7f8fa00cf830 2024-03-06T07:31:12.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.159+0000 7f8fa5e24640 1 --2- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa5623640 1 --2- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00d22b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8f900097a0 tx=0x7f8f9002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6e0bdec3e296295 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa4e22640 1 -- 172.21.15.72:0/3306731144 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90035020 con 0x7f8fa00cf830 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa4e22640 1 -- 172.21.15.72:0/3306731144 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f90031830 con 0x7f8fa00cf830 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa4e22640 1 -- 172.21.15.72:0/3306731144 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90031be0 con 0x7f8fa00cf830 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 -- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 msgr2=0x7f8fa00d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 --2- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00d22b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8f900097a0 tx=0x7f8f9002b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 -- 172.21.15.72:0/3306731144 shutdown_connections 2024-03-06T07:31:12.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 --2- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00d22b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 --2- 172.21.15.72:0/3306731144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa015dfe0 0x7f8fa00cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 -- 172.21.15.72:0/3306731144 >> 172.21.15.72:0/3306731144 conn(0x7f8fa00d3cd0 msgr2=0x7f8fa00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 -- 172.21.15.72:0/3306731144 shutdown_connections 2024-03-06T07:31:12.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.161+0000 7f8fa8088640 1 -- 172.21.15.72:0/3306731144 wait complete. 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.162+0000 7f8fa8088640 1 Processor -- start 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 -- start start 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa00d27f0 0x7f8fa015fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00d1c70 con 0x7f8fa015dfe0 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00b3d60 con 0x7f8fa00cf830 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa8088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00d1df0 con 0x7f8fa00d27f0 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa6625640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa6625640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44528/0 (socket says 172.21.15.72:44528) 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa6625640 1 -- 172.21.15.72:0/129826300 learned_addr learned my addr 172.21.15.72:0/129826300 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa5e24640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.163+0000 7f8fa5623640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa00d27f0 0x7f8fa015fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa6625640 1 -- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa00d27f0 msgr2=0x7f8fa015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa6625640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa00d27f0 0x7f8fa015fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa6625640 1 -- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 msgr2=0x7f8fa00cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa6625640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa6625640 1 -- 172.21.15.72:0/129826300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa0151460 con 0x7f8fa015dfe0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa6625640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0160530 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800cb70 tx=0x7f8f9800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98017070 con 0x7f8fa015dfe0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9800ad40 con 0x7f8fa015dfe0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa00b9b80 con 0x7f8fa015dfe0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f98012620 con 0x7f8fa015dfe0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.164+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa00b92e0 con 0x7f8fa015dfe0 2024-03-06T07:31:12.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.165+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa00b8d90 con 0x7f8fa015dfe0 2024-03-06T07:31:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.166+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8f9801a070 con 0x7f8fa015dfe0 2024-03-06T07:31:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.166+0000 7f8f967fc640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f8406b4d0 0x7f8f8406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.166+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8f9808a470 con 0x7f8fa015dfe0 2024-03-06T07:31:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.166+0000 7f8fa5e24640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f8406b4d0 0x7f8f8406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:12.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.167+0000 7f8fa5e24640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f8406b4d0 0x7f8f8406d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009560 tx=0x7f8f9c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:12.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.170+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8f98021020 con 0x7f8fa015dfe0 2024-03-06T07:31:12.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.384+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa004f330 con 0x7f8fa015dfe0 2024-03-06T07:31:12.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.384+0000 7f8f967fc640 1 -- 172.21.15.72:0/129826300 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f8f98058c60 con 0x7f8fa015dfe0 2024-03-06T07:31:12.386 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:12.387 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:12.387 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:12.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f8406b4d0 msgr2=0x7f8f8406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f8406b4d0 0x7f8f8406d990 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c009560 tx=0x7f8f9c009190 comp rx=0 tx=0).stop 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 msgr2=0x7f8fa0160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0160530 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800cb70 tx=0x7f8f9800a430 comp rx=0 tx=0).stop 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 shutdown_connections 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8f8406b4d0 0x7f8f8406d990 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fa015dfe0 0x7f8fa0160530 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fa00d27f0 0x7f8fa015fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 --2- 172.21.15.72:0/129826300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fa00cf830 0x7f8fa00cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 >> 172.21.15.72:0/129826300 conn(0x7f8fa00d3cd0 msgr2=0x7f8fa00d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 shutdown_connections 2024-03-06T07:31:12.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:12.386+0000 7f8fa8088640 1 -- 172.21.15.72:0/129826300 wait complete. 2024-03-06T07:31:12.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:12.600 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:13.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.701+0000 7fbd0d8f4640 1 Processor -- start 2024-03-06T07:31:13.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.701+0000 7fbd0d8f4640 1 -- start start 2024-03-06T07:31:13.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.701+0000 7fbd0d8f4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbd080d22b0 0x7fbd080d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.701+0000 7fbd0d8f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080b3430 con 0x7fbd0815dfe0 2024-03-06T07:31:13.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd0d8f4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd080b3710 con 0x7fbd080b41f0 2024-03-06T07:31:13.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd0d8f4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd080d2c50 con 0x7fbd080d22b0 2024-03-06T07:31:13.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd06ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbd0815dfe0 0x7fbd080cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50962/0 (socket says 172.21.15.72:50962) 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd06ffd640 1 -- 172.21.15.72:0/2441981194 learned_addr learned my addr 172.21.15.72:0/2441981194 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513813173 0 0) 0x7fbd080b3430 con 0x7fbd0815dfe0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcf4003520 con 0x7fbd0815dfe0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbcfffff640 1 --2- 172.21.15.72:0/2441981194 >> v2:172.21.15.146:3300/0 conn(0x7fbd080d22b0 0x7fbd080d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909236049 0 0) 0x7fbcf4003520 con 0x7fbd0815dfe0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 >> v2:172.21.15.146:3300/0 conn(0x7fbd080d22b0 msgr2=0x7fbd080d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 --2- 172.21.15.72:0/2441981194 >> v2:172.21.15.146:3300/0 conn(0x7fbd080d22b0 0x7fbd080d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 >> v1:172.21.15.146:6789/0 conn(0x7fbd080b41f0 legacy=0x7fbd080b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd080b39f0 con 0x7fbd0815dfe0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fbcf8001670 con 0x7fbd0815dfe0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 >> v1:172.21.15.72:6789/0 conn(0x7fbd0815dfe0 legacy=0x7fbd080cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40039b0 0x7fbcf4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcf4004800 0x7fbcf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf40051c0 0x7fbcf400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf400c0e0 con 0x7fbcf40051c0 2024-03-06T07:31:13.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf400c290 con 0x7fbcf4004800 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf400c490 con 0x7fbcf40039b0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.702+0000 7fbd077fe640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf40051c0 0x7fbcf400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbcfffff640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcf4004800 0x7fbcf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd06ffd640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40039b0 0x7fbcf4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 -- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40039b0 msgr2=0x7fbcf4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40039b0 0x7fbcf4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 -- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcf4004800 msgr2=0x7fbcf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcf4004800 0x7fbcf4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 -- 172.21.15.72:0/2441981194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf4003520 con 0x7fbcf40051c0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbcfffff640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbcf4004800 0x7fbcf4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 -- 172.21.15.72:0/2441981194 reap_dead start 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.703+0000 7fbd077fe640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf40051c0 0x7fbcf400b9c0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fbd00001e50 tx=0x7fbcf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8782a0d404114fd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00001a70 con 0x7fbcf40051c0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd00003bb0 con 0x7fbcf40051c0 2024-03-06T07:31:13.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd067fc640 1 -- 172.21.15.72:0/2441981194 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd000050d0 con 0x7fbcf40051c0 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/2441981194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf40051c0 msgr2=0x7fbcf400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf40051c0 0x7fbcf400b9c0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7fbd00001e50 tx=0x7fbcf4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/2441981194 shutdown_connections 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/2441981194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbcf40051c0 0x7fbcf400b9c0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/2441981194 >> 172.21.15.72:0/2441981194 conn(0x7fbd080d3cd0 msgr2=0x7fbd080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/2441981194 shutdown_connections 2024-03-06T07:31:13.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.704+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/2441981194 wait complete. 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.706+0000 7fbd0d8f4640 1 Processor -- start 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.706+0000 7fbd0d8f4640 1 -- start start 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd0d8f4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40051c0 0x7fbd080cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd0d8f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 0x7fbd080a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd0d8f4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08152d30 0x7fbd081531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd0d8f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080b3d60 con 0x7fbd081f3c50 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd0d8f4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd080b31e0 con 0x7fbd08152d30 2024-03-06T07:31:13.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd0d8f4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd0815e3c0 con 0x7fbcf40051c0 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 0x7fbd080a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 0x7fbd080a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40690/0 (socket says 172.21.15.72:40690) 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 -- 172.21.15.72:0/965774794 learned_addr learned my addr 172.21.15.72:0/965774794 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd077fe640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08152d30 0x7fbd081531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbd06ffd640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40051c0 0x7fbd080cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 -- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40051c0 msgr2=0x7fbd080cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40051c0 0x7fbd080cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 -- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08152d30 msgr2=0x7fbd081531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08152d30 0x7fbd081531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 -- 172.21.15.72:0/965774794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd080cb6a0 con 0x7fbd081f3c50 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.707+0000 7fbcfffff640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 0x7fbd080a03d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000f550 tx=0x7fbcf000fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.708+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0013810 con 0x7fbd081f3c50 2024-03-06T07:31:13.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.708+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf001a070 con 0x7fbd081f3c50 2024-03-06T07:31:13.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.708+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd080cb980 con 0x7fbd081f3c50 2024-03-06T07:31:13.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.708+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0013bb0 con 0x7fbd081f3c50 2024-03-06T07:31:13.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.708+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd080cbf40 con 0x7fbd081f3c50 2024-03-06T07:31:13.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.709+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd080cc740 con 0x7fbd081f3c50 2024-03-06T07:31:13.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.709+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbcf0012070 con 0x7fbd081f3c50 2024-03-06T07:31:13.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.710+0000 7fbd04ff9640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce806b500 0x7fbce806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:13.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.710+0000 7fbd06ffd640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce806b500 0x7fbce806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:13.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.710+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbcf008a200 con 0x7fbd081f3c50 2024-03-06T07:31:13.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.710+0000 7fbd06ffd640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce806b500 0x7fbce806d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8001eb0 tx=0x7fbcf8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:13.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.714+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbcf001c310 con 0x7fbd081f3c50 2024-03-06T07:31:13.776 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:13.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.940+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd0806bf50 con 0x7fbd081f3c50 2024-03-06T07:31:13.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.941+0000 7fbd04ff9640 1 -- 172.21.15.72:0/965774794 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fbcf0059040 con 0x7fbd081f3c50 2024-03-06T07:31:13.943 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:13.943 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:13.943 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:13.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce806b500 msgr2=0x7fbce806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce806b500 0x7fbce806d9c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8001eb0 tx=0x7fbcf8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:13.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 msgr2=0x7fbd080a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 0x7fbd080a03d0 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000f550 tx=0x7fbcf000fa10 comp rx=0 tx=0).stop 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 shutdown_connections 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbce806b500 0x7fbce806d9c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd08152d30 0x7fbd081531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd081f3c50 0x7fbd080a03d0 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 --2- 172.21.15.72:0/965774794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbcf40051c0 0x7fbd080cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 >> 172.21.15.72:0/965774794 conn(0x7fbd080d3cd0 msgr2=0x7fbd080b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 shutdown_connections 2024-03-06T07:31:13.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:13.943+0000 7fbd0d8f4640 1 -- 172.21.15.72:0/965774794 wait complete. 2024-03-06T07:31:14.157 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:14.158 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:15.158 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:15.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.258+0000 7f1b04c6f640 1 Processor -- start 2024-03-06T07:31:15.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.258+0000 7f1b04c6f640 1 -- start start 2024-03-06T07:31:15.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.259+0000 7f1b04c6f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1b0015dfe0 0x7f1b000cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.259+0000 7f1b04c6f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b000cf830 0x7f1b000d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.259+0000 7f1b04c6f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b000d27f0 0x7f1b000d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.259+0000 7f1b04c6f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b000b31e0 con 0x7f1b000cf830 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.259+0000 7f1b04c6f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b000b3d60 con 0x7f1b0015dfe0 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.259+0000 7f1b04c6f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000b3430 con 0x7f1b000d27f0 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b000cf830 0x7f1b000d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1b000cf830 0x7f1b000d22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40696/0 (socket says 172.21.15.72:40696) 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 -- 172.21.15.72:0/177609508 learned_addr learned my addr 172.21.15.72:0/177609508 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afe59c640 1 --2- 172.21.15.72:0/177609508 >> v2:172.21.15.146:3300/0 conn(0x7f1b0015dfe0 0x7f1b000cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afed9d640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b000d27f0 0x7f1b000d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 -- 172.21.15.72:0/177609508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b000d27f0 msgr2=0x7f1b000d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b000d27f0 0x7f1b000d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 -- 172.21.15.72:0/177609508 >> v2:172.21.15.146:3300/0 conn(0x7f1b0015dfe0 msgr2=0x7f1b000cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/177609508 >> v2:172.21.15.146:3300/0 conn(0x7f1b0015dfe0 0x7f1b000cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afdd9b640 1 -- 172.21.15.72:0/177609508 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000b3710 con 0x7f1b000cf830 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afe59c640 1 --2- 172.21.15.72:0/177609508 >> v2:172.21.15.146:3300/0 conn(0x7f1b0015dfe0 0x7f1b000cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.260+0000 7f1afed9d640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b000d27f0 0x7f1b000d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000d22b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f1b00067990 tx=0x7f1af402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8339052c9c239682 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1afd59a640 1 -- 172.21.15.72:0/177609508 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4035020 con 0x7f1b000cf830 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1afd59a640 1 -- 172.21.15.72:0/177609508 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af4031870 con 0x7f1b000cf830 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1afd59a640 1 -- 172.21.15.72:0/177609508 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af4031c40 con 0x7f1b000cf830 2024-03-06T07:31:15.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 -- 172.21.15.72:0/177609508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 msgr2=0x7f1b000d22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000d22b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f1b00067990 tx=0x7f1af402b910 comp rx=0 tx=0).stop 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 -- 172.21.15.72:0/177609508 shutdown_connections 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b000d27f0 0x7f1b000d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/177609508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000d22b0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/177609508 >> v2:172.21.15.146:3300/0 conn(0x7f1b0015dfe0 0x7f1b000cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.261+0000 7f1b04c6f640 1 -- 172.21.15.72:0/177609508 >> 172.21.15.72:0/177609508 conn(0x7f1b000d3cd0 msgr2=0x7f1b000d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.262+0000 7f1b04c6f640 1 -- 172.21.15.72:0/177609508 shutdown_connections 2024-03-06T07:31:15.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.262+0000 7f1b04c6f640 1 -- 172.21.15.72:0/177609508 wait complete. 2024-03-06T07:31:15.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.263+0000 7f1b04c6f640 1 Processor -- start 2024-03-06T07:31:15.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.263+0000 7f1b04c6f640 1 -- start start 2024-03-06T07:31:15.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.263+0000 7f1b04c6f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.263+0000 7f1b04c6f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b000d27f0 0x7f1b0015c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.263+0000 7f1b04c6f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b0015dfe0 0x7f1b000cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.263+0000 7f1b04c6f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000d1df0 con 0x7f1b000cf830 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1b04c6f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000d1c70 con 0x7f1b000d27f0 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1b04c6f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0015e3c0 con 0x7f1b0015dfe0 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000cbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40702/0 (socket says 172.21.15.72:40702) 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 -- 172.21.15.72:0/2160434452 learned_addr learned my addr 172.21.15.72:0/2160434452 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afed9d640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b0015dfe0 0x7f1b000cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b000d27f0 0x7f1b0015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 -- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b0015dfe0 msgr2=0x7f1b000cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b0015dfe0 0x7f1b000cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 -- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b000d27f0 msgr2=0x7f1b0015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b000d27f0 0x7f1b0015c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 -- 172.21.15.72:0/2160434452 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b000cae40 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b000d27f0 0x7f1b0015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1afe59c640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000cbbc0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c820 tx=0x7f1af000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0015070 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0007a30 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b000cb0e0 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0007de0 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.264+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b000cb6a0 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.265+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0004f330 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.266+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1af0018070 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.266+0000 7f1aeeffd640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae006b500 0x7f1ae006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.266+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1af008a0d0 con 0x7f1b000cf830 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.266+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae006b500 0x7f1ae006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:15.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.267+0000 7f1afdd9b640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae006b500 0x7f1ae006d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1af4036000 tx=0x7f1af4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:15.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.271+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1af0058f10 con 0x7f1b000cf830 2024-03-06T07:31:15.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.485+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b000b27a0 con 0x7f1b000cf830 2024-03-06T07:31:15.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.485+0000 7f1aeeffd640 1 -- 172.21.15.72:0/2160434452 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f1af0058810 con 0x7f1b000cf830 2024-03-06T07:31:15.488 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:15.488 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:15.488 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae006b500 msgr2=0x7f1ae006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae006b500 0x7f1ae006d9c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1af4036000 tx=0x7f1af4008030 comp rx=0 tx=0).stop 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 msgr2=0x7f1b000cbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000cbbc0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f1af000c820 tx=0x7f1af000cce0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 shutdown_connections 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1ae006b500 0x7f1ae006d9c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1b0015dfe0 0x7f1b000cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1b000d27f0 0x7f1b0015c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 --2- 172.21.15.72:0/2160434452 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1b000cf830 0x7f1b000cbbc0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:15.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.487+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 >> 172.21.15.72:0/2160434452 conn(0x7f1b000d3cd0 msgr2=0x7f1b000b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:15.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.488+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 shutdown_connections 2024-03-06T07:31:15.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:15.488+0000 7f1b04c6f640 1 -- 172.21.15.72:0/2160434452 wait complete. 2024-03-06T07:31:15.701 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:15.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:15.778 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:16.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.802+0000 7fb2d7193640 1 Processor -- start 2024-03-06T07:31:16.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.802+0000 7fb2d7193640 1 -- start start 2024-03-06T07:31:16.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d7193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2d00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d7193640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00b3430 con 0x7fb2d00d29c0 2024-03-06T07:31:16.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d7193640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d00b3710 con 0x7fb2d00cf830 2024-03-06T07:31:16.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d7193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00b31e0 con 0x7fb2d015dfe0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d5730640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb2d00d29c0 0x7fb2d00b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50972/0 (socket says 172.21.15.72:50972) 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d5730640 1 -- 172.21.15.72:0/3709210412 learned_addr learned my addr 172.21.15.72:0/3709210412 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2d4f2f640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2d00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.803+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354864207 0 0) 0x7fb2d00b3430 con 0x7fb2d00d29c0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0003520 con 0x7fb2d00d29c0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2264396773 0 0) 0x7fb2c0003520 con 0x7fb2d00d29c0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 msgr2=0x7fb2d00cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2d00cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 >> v1:172.21.15.146:6789/0 conn(0x7fb2d00cf830 legacy=0x7fb2d00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00b39f0 con 0x7fb2d00d29c0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fb2c80019f0 con 0x7fb2d00d29c0 2024-03-06T07:31:16.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 >> v1:172.21.15.72:6789/0 conn(0x7fb2d00d29c0 legacy=0x7fb2d00b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2c00039e0 0x7fb2c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2c0004d40 0x7fb2c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00b31e0 con 0x7fb2c0004d40 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c000ba70 con 0x7fb2c00039e0 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c000bc70 con 0x7fb2d015dfe0 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d5730640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2c0004d40 0x7fb2c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d4f2f640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2c00039e0 0x7fb2c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cffff640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d5730640 1 -- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 msgr2=0x7fb2c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d5730640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d5730640 1 -- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2c00039e0 msgr2=0x7fb2c0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:16.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d5730640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2c00039e0 0x7fb2c0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2d5730640 1 -- 172.21.15.72:0/3709210412 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c0003520 con 0x7fb2c0004d40 2024-03-06T07:31:16.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.804+0000 7fb2cffff640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2d015dfe0 0x7fb2c0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:16.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.805+0000 7fb2d5730640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2c0004d40 0x7fb2c00051c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800a830 tx=0x7fb2c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda71850d3c250bc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c800a480 con 0x7fb2c0004d40 2024-03-06T07:31:16.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8034630 con 0x7fb2c0004d40 2024-03-06T07:31:16.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2cf7fe640 1 -- 172.21.15.72:0/3709210412 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8034a60 con 0x7fb2c0004d40 2024-03-06T07:31:16.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d7193640 1 -- 172.21.15.72:0/3709210412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2c0004d40 msgr2=0x7fb2c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d7193640 1 --2- 172.21.15.72:0/3709210412 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2c0004d40 0x7fb2c00051c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800a830 tx=0x7fb2c0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d5730640 1 -- 172.21.15.72:0/3709210412 reap_dead start 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d7193640 1 -- 172.21.15.72:0/3709210412 shutdown_connections 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d7193640 1 -- 172.21.15.72:0/3709210412 >> 172.21.15.72:0/3709210412 conn(0x7fb2d00d3cd0 msgr2=0x7fb2d00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d7193640 1 -- 172.21.15.72:0/3709210412 shutdown_connections 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.806+0000 7fb2d7193640 1 -- 172.21.15.72:0/3709210412 wait complete. 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.807+0000 7fb2d7193640 1 Processor -- start 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.807+0000 7fb2d7193640 1 -- start start 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d7193640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2c0004d40 0x7fb2d015b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d7193640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 0x7fb2d00cbf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d7193640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006ca10 0x7fb2d006ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d7193640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d015e3c0 con 0x7fb2d015c040 2024-03-06T07:31:16.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d7193640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d00b3d60 con 0x7fb2d006ca10 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d7193640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d015e540 con 0x7fb2c0004d40 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 0x7fb2d00cbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2cffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 0x7fb2d00cbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40720/0 (socket says 172.21.15.72:40720) 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2cffff640 1 -- 172.21.15.72:0/4213370762 learned_addr learned my addr 172.21.15.72:0/4213370762 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.808+0000 7fb2d4f2f640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2c0004d40 0x7fb2d015b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2d5730640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006ca10 0x7fb2d006ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cffff640 1 -- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2c0004d40 msgr2=0x7fb2d015b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cffff640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2c0004d40 0x7fb2d015b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cffff640 1 -- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006ca10 msgr2=0x7fb2d006ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cffff640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006ca10 0x7fb2d006ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cffff640 1 -- 172.21.15.72:0/4213370762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00cc820 con 0x7fb2d015c040 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cffff640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 0x7fb2d00cbf20 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00e5a0 tx=0x7fb2bc00e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc0052e0 con 0x7fb2d015c040 2024-03-06T07:31:16.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc018070 con 0x7fb2d015c040 2024-03-06T07:31:16.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc003830 con 0x7fb2d015c040 2024-03-06T07:31:16.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d00ccb00 con 0x7fb2d015c040 2024-03-06T07:31:16.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.809+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d009dd20 con 0x7fb2d015c040 2024-03-06T07:31:16.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.811+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d004f330 con 0x7fb2d015c040 2024-03-06T07:31:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.811+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb2bc0039d0 con 0x7fb2d015c040 2024-03-06T07:31:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.812+0000 7fb2cd7fa640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b4d0 0x7fb2b406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.812+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb2bc089df0 con 0x7fb2d015c040 2024-03-06T07:31:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.812+0000 7fb2d4f2f640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b4d0 0x7fb2b406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:16.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.812+0000 7fb2d4f2f640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b4d0 0x7fb2b406d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009bc0 tx=0x7fb2c4009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:16.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:16.816+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb2bc058c30 con 0x7fb2d015c040 2024-03-06T07:31:17.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.030+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d015d6c0 con 0x7fb2d015c040 2024-03-06T07:31:17.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.031+0000 7fb2cd7fa640 1 -- 172.21.15.72:0/4213370762 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fb2bc058530 con 0x7fb2d015c040 2024-03-06T07:31:17.033 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:17.033 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:17.033 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.033+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b4d0 msgr2=0x7fb2b406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.033+0000 7fb2d7193640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b4d0 0x7fb2b406d990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb2c4009bc0 tx=0x7fb2c4009190 comp rx=0 tx=0).stop 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.033+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 msgr2=0x7fb2d00cbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.033+0000 7fb2d7193640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 0x7fb2d00cbf20 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00e5a0 tx=0x7fb2bc00e9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 shutdown_connections 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb2b406b4d0 0x7fb2b406d990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2d006ca10 0x7fb2d006ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb2d015c040 0x7fb2d00cbf20 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 --2- 172.21.15.72:0/4213370762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2c0004d40 0x7fb2d015b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 >> 172.21.15.72:0/4213370762 conn(0x7fb2d00d3cd0 msgr2=0x7fb2d00b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:17.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 shutdown_connections 2024-03-06T07:31:17.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:17.034+0000 7fb2d7193640 1 -- 172.21.15.72:0/4213370762 wait complete. 2024-03-06T07:31:17.248 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:17.248 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:17.781 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:18.248 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:18.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 Processor -- start 2024-03-06T07:31:18.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 -- start start 2024-03-06T07:31:18.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015bdd0 0x7f37c00c5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37c015fa10 0x7f37c00ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c00cebe0 con 0x7f37c00d16e0 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37c00ceec0 con 0x7f37c015fa10 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37c52eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00cf1a0 con 0x7f37c015bdd0 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.349+0000 7f37b7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37c015fa10 0x7f37c00ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37c015fa10 0x7f37c00ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40734/0 (socket says 172.21.15.72:40734) 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 -- 172.21.15.72:0/195631802 learned_addr learned my addr 172.21.15.72:0/195631802 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37beffd640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015bdd0 0x7f37c00c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 -- 172.21.15.72:0/195631802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015bdd0 msgr2=0x7f37c00c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015bdd0 0x7f37c00c5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 -- 172.21.15.72:0/195631802 >> v1:172.21.15.146:6789/0 conn(0x7f37c00d16e0 legacy=0x7f37c00d1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 -- 172.21.15.72:0/195631802 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c00cc8e0 con 0x7f37c015fa10 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37beffd640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015bdd0 0x7f37c00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.350+0000 7f37b7fff640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015fa10 0x7f37c00ce270 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009870 tx=0x7f37a802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e9666c10e22b7a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37be7fc640 1 -- 172.21.15.72:0/195631802 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a80317c0 con 0x7f37c015fa10 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37be7fc640 1 -- 172.21.15.72:0/195631802 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a8037070 con 0x7f37c015fa10 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37be7fc640 1 -- 172.21.15.72:0/195631802 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8031b80 con 0x7f37c015fa10 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 -- 172.21.15.72:0/195631802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015fa10 msgr2=0x7f37c00ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015fa10 0x7f37c00ce270 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009870 tx=0x7f37a802b860 comp rx=0 tx=0).stop 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 -- 172.21.15.72:0/195631802 shutdown_connections 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015fa10 0x7f37c00ce270 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 --2- 172.21.15.72:0/195631802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015bdd0 0x7f37c00c5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 -- 172.21.15.72:0/195631802 >> 172.21.15.72:0/195631802 conn(0x7f37c009a9f0 msgr2=0x7f37c009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 -- 172.21.15.72:0/195631802 shutdown_connections 2024-03-06T07:31:18.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.351+0000 7f37c52eb640 1 -- 172.21.15.72:0/195631802 wait complete. 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.352+0000 7f37c52eb640 1 Processor -- start 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.352+0000 7f37c52eb640 1 -- start start 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37c52eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c00d16e0 0x7f37c015d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37c52eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 0x7f37c015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37c52eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015fa10 0x7f37c015c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37c52eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c015c1b0 con 0x7f37c015bdd0 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37c52eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00cc5d0 con 0x7f37c00d16e0 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37c52eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c00ce7b0 con 0x7f37c015fa10 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 0x7f37c015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 0x7f37c015c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40744/0 (socket says 172.21.15.72:40744) 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 -- 172.21.15.72:0/2728580495 learned_addr learned my addr 172.21.15.72:0/2728580495 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37bf7fe640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015fa10 0x7f37c015c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37beffd640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c00d16e0 0x7f37c015d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 -- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015fa10 msgr2=0x7f37c015c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015fa10 0x7f37c015c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 -- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c00d16e0 msgr2=0x7f37c015d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c00d16e0 0x7f37c015d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.353+0000 7f37b7fff640 1 -- 172.21.15.72:0/2728580495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c015eb80 con 0x7f37c015bdd0 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37bf7fe640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015fa10 0x7f37c015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:18.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37b7fff640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 0x7f37c015c330 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f37a802b8a0 tx=0x7f37a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8032860 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a8032a20 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c015df70 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a8032e10 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.354+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c015e530 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.355+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c004f330 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.355+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37a804c060 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.355+0000 7f37b77fe640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.356+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37a80aba40 con 0x7f37c015bdd0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.356+0000 7f37beffd640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:18.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.356+0000 7f37beffd640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f37c006d750 tx=0x7f37b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:18.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.360+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37a8040070 con 0x7f37c015bdd0 2024-03-06T07:31:18.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.638+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37c01597d0 con 0x7f37c015bdd0 2024-03-06T07:31:18.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.638+0000 7f37b77fe640 1 -- 172.21.15.72:0/2728580495 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f37a80abe20 con 0x7f37c015bdd0 2024-03-06T07:31:18.641 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:18.641 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:18.641 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:18.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 msgr2=0x7f37a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f37c006d750 tx=0x7f37b0008040 comp rx=0 tx=0).stop 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 msgr2=0x7f37c015c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 0x7f37c015c330 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f37a802b8a0 tx=0x7f37a80046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 shutdown_connections 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37c015fa10 0x7f37c015c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c015bdd0 0x7f37c015c330 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 --2- 172.21.15.72:0/2728580495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37c00d16e0 0x7f37c015d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 >> 172.21.15.72:0/2728580495 conn(0x7f37c009a9f0 msgr2=0x7f37c009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:18.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 shutdown_connections 2024-03-06T07:31:18.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:18.641+0000 7f37c52eb640 1 -- 172.21.15.72:0/2728580495 wait complete. 2024-03-06T07:31:18.855 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:18.855 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:19.783 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:19.856 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.972+0000 7fdddc9e2640 1 Processor -- start 2024-03-06T07:31:19.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.972+0000 7fdddc9e2640 1 -- start start 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.973+0000 7fdddc9e2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fddd815e090 0x7fddd80d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.973+0000 7fdddc9e2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddd80b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.973+0000 7fdddc9e2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd815e470 con 0x7fddd80b3b20 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.973+0000 7fdddc9e2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd80d18f0 con 0x7fddd815e090 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.973+0000 7fdddc9e2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80d1b90 con 0x7fddd80d2920 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd6d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fddd80b3b20 0x7fddd80d0650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50988/0 (socket says 172.21.15.72:50988) 2024-03-06T07:31:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd6d9d640 1 -- 172.21.15.72:0/1310065626 learned_addr learned my addr 172.21.15.72:0/1310065626 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddd80b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd659c640 1 --2- 172.21.15.72:0/1310065626 >> v2:172.21.15.146:3300/0 conn(0x7fddd815e090 0x7fddd80d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385226125 0 0) 0x7fddd815e470 con 0x7fddd80b3b20 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003520 con 0x7fddd80b3b20 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436750470 0 0) 0x7fddc0003520 con 0x7fddd80b3b20 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 msgr2=0x7fddd80b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddd80b35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 >> v2:172.21.15.146:3300/0 conn(0x7fddd815e090 msgr2=0x7fddd80d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 --2- 172.21.15.72:0/1310065626 >> v2:172.21.15.146:3300/0 conn(0x7fddd815e090 0x7fddd80d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80ceb50 con 0x7fddd80b3b20 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fddc8001670 con 0x7fddd80b3b20 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 >> v1:172.21.15.72:6789/0 conn(0x7fddd80b3b20 legacy=0x7fddd80d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddc0004d40 0x7fddc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80d18f0 con 0x7fddc00039b0 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80d1b90 con 0x7fddc0004d40 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc000b9c0 con 0x7fddd80d2920 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd659c640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.974+0000 7fddd6d9d640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddc0004d40 0x7fddc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd659c640 1 -- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 msgr2=0x7fddc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd659c640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddc0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd659c640 1 -- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddc0004d40 msgr2=0x7fddc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd659c640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddc0004d40 0x7fddc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd659c640 1 -- 172.21.15.72:0/1310065626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0003520 con 0x7fddc00039b0 2024-03-06T07:31:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80d2920 0x7fddc0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd659c640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddc0003dd0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009ac0 tx=0x7fddc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4ecaf28ca276e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0039070 con 0x7fddc00039b0 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd002c710 con 0x7fddc00039b0 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.975+0000 7fddd559a640 1 -- 172.21.15.72:0/1310065626 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0009690 con 0x7fddc00039b0 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fdddc9e2640 1 -- 172.21.15.72:0/1310065626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 msgr2=0x7fddc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/1310065626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddc0003dd0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009ac0 tx=0x7fddc0005830 comp rx=0 tx=0).stop 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fddd6d9d640 1 -- 172.21.15.72:0/1310065626 reap_dead start 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fdddc9e2640 1 -- 172.21.15.72:0/1310065626 shutdown_connections 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fdddc9e2640 1 -- 172.21.15.72:0/1310065626 >> 172.21.15.72:0/1310065626 conn(0x7fddd80d3c80 msgr2=0x7fddd80d4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fdddc9e2640 1 -- 172.21.15.72:0/1310065626 shutdown_connections 2024-03-06T07:31:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.976+0000 7fdddc9e2640 1 -- 172.21.15.72:0/1310065626 wait complete. 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.977+0000 7fdddc9e2640 1 Processor -- start 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.977+0000 7fdddc9e2640 1 -- start start 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fdddc9e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddd80cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fdddc9e2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80ccaa0 0x7fddd8153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fdddc9e2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd815e660 0x7fddd8154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fdddc9e2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80ce800 con 0x7fddc00039b0 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fdddc9e2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80b2df0 con 0x7fddd815e660 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fdddc9e2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd80d0d60 con 0x7fddd80ccaa0 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddd80cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddd80cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40756/0 (socket says 172.21.15.72:40756) 2024-03-06T07:31:19.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 -- 172.21.15.72:0/2287052300 learned_addr learned my addr 172.21.15.72:0/2287052300 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 -- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80ccaa0 msgr2=0x7fddd8153aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80ccaa0 0x7fddd8153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd6d9d640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd815e660 0x7fddd8154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80ccaa0 0x7fddd8153aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 -- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd815e660 msgr2=0x7fddd8154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd815e660 0x7fddd8154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd659c640 1 -- 172.21.15.72:0/2287052300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd80cd2f0 con 0x7fddc00039b0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80ccaa0 0x7fddd8153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.978+0000 7fddd6d9d640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd815e660 0x7fddd8154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.979+0000 7fddd659c640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddd80cbb20 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009be0 tx=0x7fddd0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.979+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0042020 con 0x7fddc00039b0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.979+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd003a040 con 0x7fddc00039b0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.979+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd80cd5d0 con 0x7fddc00039b0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.979+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0039070 con 0x7fddc00039b0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.979+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd81547e0 con 0x7fddc00039b0 2024-03-06T07:31:19.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.980+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd804f330 con 0x7fddc00039b0 2024-03-06T07:31:19.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.980+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddd0045020 con 0x7fddc00039b0 2024-03-06T07:31:19.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.981+0000 7fddc6ffd640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb806b500 0x7fddb806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:19.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.981+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb806b500 0x7fddb806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:19.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.981+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fddd00ab690 con 0x7fddc00039b0 2024-03-06T07:31:19.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.981+0000 7fddd5d9b640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb806b500 0x7fddb806d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fddd80cc2d0 tx=0x7fddcc00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:19.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:19.985+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fddd004c070 con 0x7fddc00039b0 2024-03-06T07:31:20.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.194+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddd80cd7a0 con 0x7fddc00039b0 2024-03-06T07:31:20.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.194+0000 7fddc6ffd640 1 -- 172.21.15.72:0/2287052300 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fddd007a4d0 con 0x7fddc00039b0 2024-03-06T07:31:20.197 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:20.197 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:20.197 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:20.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb806b500 msgr2=0x7fddb806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb806b500 0x7fddb806d9c0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fddd80cc2d0 tx=0x7fddcc00b040 comp rx=0 tx=0).stop 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 msgr2=0x7fddd80cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddd80cbb20 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fddd0009be0 tx=0x7fddd0004730 comp rx=0 tx=0).stop 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 shutdown_connections 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb806b500 0x7fddb806d9c0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd815e660 0x7fddd8154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd80ccaa0 0x7fddd8153aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 --2- 172.21.15.72:0/2287052300 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddc00039b0 0x7fddd80cbb20 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 >> 172.21.15.72:0/2287052300 conn(0x7fddd80d3c80 msgr2=0x7fddd80b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 shutdown_connections 2024-03-06T07:31:20.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:20.196+0000 7fdddc9e2640 1 -- 172.21.15.72:0/2287052300 wait complete. 2024-03-06T07:31:20.414 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:20.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:21.415 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:21.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.516+0000 7f3028a8f640 1 Processor -- start 2024-03-06T07:31:21.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.516+0000 7f3028a8f640 1 -- start start 2024-03-06T07:31:21.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3028a8f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30240d2710 0x7f30240b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3028a8f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240b3430 con 0x7f302415dfe0 2024-03-06T07:31:21.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3028a8f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30240b3710 con 0x7f30240d22b0 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3028a8f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30240b31e0 con 0x7f30240d2710 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3022d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30240d2710 0x7f30240b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f302259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f302415dfe0 0x7f30240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50998/0 (socket says 172.21.15.72:50998) 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3022d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f30240d2710 0x7f30240b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40758/0 (socket says 172.21.15.72:40758) 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f302259c640 1 -- 172.21.15.72:0/3163682509 learned_addr learned my addr 172.21.15.72:0/3163682509 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3022d9d640 1 -- 172.21.15.72:0/3163682509 >> v1:172.21.15.146:6789/0 conn(0x7f30240d22b0 legacy=0x7f30240b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3022d9d640 1 -- 172.21.15.72:0/3163682509 >> v1:172.21.15.72:6789/0 conn(0x7f302415dfe0 legacy=0x7f30240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.517+0000 7f3022d9d640 1 -- 172.21.15.72:0/3163682509 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240b39f0 con 0x7f30240d2710 2024-03-06T07:31:21.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.518+0000 7f3022d9d640 1 --2- 172.21.15.72:0/3163682509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30240d2710 0x7f30240b4900 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f30140097d0 tx=0x7f301402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8247acb9561ce9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.518+0000 7f302159a640 1 -- 172.21.15.72:0/3163682509 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014035020 con 0x7f30240d2710 2024-03-06T07:31:21.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.518+0000 7f302159a640 1 -- 172.21.15.72:0/3163682509 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3014031830 con 0x7f30240d2710 2024-03-06T07:31:21.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.518+0000 7f302159a640 1 -- 172.21.15.72:0/3163682509 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014031c00 con 0x7f30240d2710 2024-03-06T07:31:21.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 -- 172.21.15.72:0/3163682509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30240d2710 msgr2=0x7f30240b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 --2- 172.21.15.72:0/3163682509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30240d2710 0x7f30240b4900 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f30140097d0 tx=0x7f301402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 -- 172.21.15.72:0/3163682509 shutdown_connections 2024-03-06T07:31:21.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 --2- 172.21.15.72:0/3163682509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f30240d2710 0x7f30240b4900 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 -- 172.21.15.72:0/3163682509 >> 172.21.15.72:0/3163682509 conn(0x7f30240d3cd0 msgr2=0x7f30240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 -- 172.21.15.72:0/3163682509 shutdown_connections 2024-03-06T07:31:21.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.519+0000 7f3028a8f640 1 -- 172.21.15.72:0/3163682509 wait complete. 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.520+0000 7f3028a8f640 1 Processor -- start 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.520+0000 7f3028a8f640 1 -- start start 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3028a8f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30240d22b0 0x7f30240cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3028a8f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30240d2710 0x7f30240b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3028a8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 0x7f30240b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3028a8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240cfa00 con 0x7f302415dfe0 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3028a8f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240b3d60 con 0x7f30240d2710 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3028a8f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30240d1c70 con 0x7f30240d22b0 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 0x7f30240b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 0x7f30240b6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40774/0 (socket says 172.21.15.72:40774) 2024-03-06T07:31:21.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 -- 172.21.15.72:0/2658716272 learned_addr learned my addr 172.21.15.72:0/2658716272 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 -- 172.21.15.72:0/2658716272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30240d22b0 msgr2=0x7f30240cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30240d22b0 0x7f30240cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 -- 172.21.15.72:0/2658716272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30240d2710 msgr2=0x7f30240b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30240d2710 0x7f30240b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 -- 172.21.15.72:0/2658716272 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240cd7b0 con 0x7f302415dfe0 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.521+0000 7f3022d9d640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 0x7f30240b6de0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f30140097d0 tx=0x7f30140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.522+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014041070 con 0x7f302415dfe0 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.522+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301403a8d0 con 0x7f302415dfe0 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.522+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30240cc1d0 con 0x7f302415dfe0 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.522+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3014035040 con 0x7f302415dfe0 2024-03-06T07:31:21.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.522+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30240cc790 con 0x7f302415dfe0 2024-03-06T07:31:21.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.523+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302404f330 con 0x7f302415dfe0 2024-03-06T07:31:21.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.523+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3014038020 con 0x7f302415dfe0 2024-03-06T07:31:21.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.524+0000 7f301affd640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f300406b4d0 0x7f300406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:21.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.524+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f30140abc60 con 0x7f302415dfe0 2024-03-06T07:31:21.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.524+0000 7f302259c640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f300406b4d0 0x7f300406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:21.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.524+0000 7f302259c640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f300406b4d0 0x7f300406d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f301c0012e0 tx=0x7f301c001590 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:21.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.528+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f301407aaa0 con 0x7f302415dfe0 2024-03-06T07:31:21.784 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:21.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.815+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30240ce840 con 0x7f302415dfe0 2024-03-06T07:31:21.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.815+0000 7f301affd640 1 -- 172.21.15.72:0/2658716272 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f301407a3a0 con 0x7f302415dfe0 2024-03-06T07:31:21.816 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:21.817 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:21.817 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:21.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f300406b4d0 msgr2=0x7f300406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f300406b4d0 0x7f300406d990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f301c0012e0 tx=0x7f301c001590 comp rx=0 tx=0).stop 2024-03-06T07:31:21.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 msgr2=0x7f30240b6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:21.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 0x7f30240b6de0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f30140097d0 tx=0x7f30140046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 shutdown_connections 2024-03-06T07:31:21.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f300406b4d0 0x7f300406d990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f302415dfe0 0x7f30240b6de0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30240d2710 0x7f30240b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 --2- 172.21.15.72:0/2658716272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30240d22b0 0x7f30240cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:21.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 >> 172.21.15.72:0/2658716272 conn(0x7f30240d3cd0 msgr2=0x7f30240cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:21.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 shutdown_connections 2024-03-06T07:31:21.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:21.818+0000 7f3028a8f640 1 -- 172.21.15.72:0/2658716272 wait complete. 2024-03-06T07:31:22.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:22.032 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:22.146 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:22.148 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:23.032 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:23.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 Processor -- start 2024-03-06T07:31:23.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 -- start start 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f58ec0b4170 0x7f58ec0b6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58ec0d0a80 0x7f58ec0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec0d25d0 con 0x7f58ec0d4760 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec0d4bf0 con 0x7f58ec0b4170 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58f16df640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec0ce9e0 con 0x7f58ec0d0a80 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.134+0000 7f58eaffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f58ec0b4170 0x7f58ec0b6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58e3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f58ec0d4760 0x7f58ec0d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:51546/0 (socket says 172.21.15.72:51546) 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58e3fff640 1 -- 172.21.15.72:0/3916286535 learned_addr learned my addr 172.21.15.72:0/3916286535 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58eb7fe640 1 --2- 172.21.15.72:0/3916286535 >> v2:172.21.15.146:3300/0 conn(0x7f58ec0d0a80 0x7f58ec0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58eaffd640 1 -- 172.21.15.72:0/3916286535 >> v2:172.21.15.146:3300/0 conn(0x7f58ec0d0a80 msgr2=0x7f58ec0cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58eaffd640 1 --2- 172.21.15.72:0/3916286535 >> v2:172.21.15.146:3300/0 conn(0x7f58ec0d0a80 0x7f58ec0cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58eaffd640 1 -- 172.21.15.72:0/3916286535 >> v1:172.21.15.72:6789/0 conn(0x7f58ec0d4760 legacy=0x7f58ec0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58eaffd640 1 -- 172.21.15.72:0/3916286535 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0d2870 con 0x7f58ec0b4170 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.135+0000 7f58ea7fc640 1 -- 172.21.15.72:0/3916286535 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896544802 0 0) 0x7f58ec0d25d0 con 0x7f58ec0d4760 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58eaffd640 1 --2- 172.21.15.72:0/3916286535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec0b6a90 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f58ec08df00 tx=0x7f58dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6272ee5dc5346cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58ea7fc640 1 -- 172.21.15.72:0/3916286535 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc035020 con 0x7f58ec0b4170 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58ea7fc640 1 -- 172.21.15.72:0/3916286535 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc031830 con 0x7f58ec0b4170 2024-03-06T07:31:23.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58ea7fc640 1 -- 172.21.15.72:0/3916286535 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc031c00 con 0x7f58ec0b4170 2024-03-06T07:31:23.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 -- 172.21.15.72:0/3916286535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 msgr2=0x7f58ec0b6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 --2- 172.21.15.72:0/3916286535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec0b6a90 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f58ec08df00 tx=0x7f58dc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 -- 172.21.15.72:0/3916286535 shutdown_connections 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 --2- 172.21.15.72:0/3916286535 >> v2:172.21.15.146:3300/0 conn(0x7f58ec0d0a80 0x7f58ec0cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 --2- 172.21.15.72:0/3916286535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec0b6a90 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 -- 172.21.15.72:0/3916286535 >> 172.21.15.72:0/3916286535 conn(0x7f58ec0d3c40 msgr2=0x7f58ec0d4050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 -- 172.21.15.72:0/3916286535 shutdown_connections 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.136+0000 7f58f16df640 1 -- 172.21.15.72:0/3916286535 wait complete. 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.137+0000 7f58f16df640 1 Processor -- start 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 -- start start 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec15f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58ec0d0a80 0x7f58ec0a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58ec0a0c50 0x7f58ec0b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0b4dd0 con 0x7f58ec0b4170 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0b4550 con 0x7f58ec0d0a80 2024-03-06T07:31:23.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58f16df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec0b4f50 con 0x7f58ec0a0c50 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58eaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec15f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58eaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec15f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33134/0 (socket says 172.21.15.72:33134) 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.138+0000 7f58eaffd640 1 -- 172.21.15.72:0/2285618362 learned_addr learned my addr 172.21.15.72:0/2285618362 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eaffd640 1 -- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58ec0a0c50 msgr2=0x7f58ec0b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eb7fe640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58ec0a0c50 0x7f58ec0b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58e3fff640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58ec0d0a80 0x7f58ec0a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eaffd640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58ec0a0c50 0x7f58ec0b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eaffd640 1 -- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58ec0d0a80 msgr2=0x7f58ec0a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eaffd640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58ec0d0a80 0x7f58ec0a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eaffd640 1 -- 172.21.15.72:0/2285618362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec1602c0 con 0x7f58ec0b4170 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eb7fe640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58ec0a0c50 0x7f58ec0b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58e3fff640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58ec0d0a80 0x7f58ec0a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58eaffd640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec15f6e0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f58dc000c00 tx=0x7f58dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc035070 con 0x7f58ec0b4170 2024-03-06T07:31:23.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58dc004250 con 0x7f58ec0b4170 2024-03-06T07:31:23.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec0b2420 con 0x7f58ec0b4170 2024-03-06T07:31:23.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58dc03a6f0 con 0x7f58ec0b4170 2024-03-06T07:31:23.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.139+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec09dfb0 con 0x7f58ec0b4170 2024-03-06T07:31:23.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.140+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec0a1060 con 0x7f58ec0b4170 2024-03-06T07:31:23.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.141+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f58dc038020 con 0x7f58ec0b4170 2024-03-06T07:31:23.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.141+0000 7f58e37fe640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f58cc06b4d0 0x7f58cc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:23.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.141+0000 7f58e3fff640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f58cc06b4d0 0x7f58cc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:23.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.141+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f58dc0abfd0 con 0x7f58ec0b4170 2024-03-06T07:31:23.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.141+0000 7f58e3fff640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f58cc06b4d0 0x7f58cc06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f58ec06bf00 tx=0x7f58d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:23.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.146+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f58ec0a1060 con 0x7f58ec0b4170 2024-03-06T07:31:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.463+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58ec04f330 con 0x7f58ec0b4170 2024-03-06T07:31:23.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.463+0000 7f58e37fe640 1 -- 172.21.15.72:0/2285618362 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f58dc040070 con 0x7f58ec0b4170 2024-03-06T07:31:23.464 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:23.465 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f58cc06b4d0 msgr2=0x7f58cc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f58cc06b4d0 0x7f58cc06d990 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f58ec06bf00 tx=0x7f58d4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 msgr2=0x7f58ec15f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec15f6e0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f58dc000c00 tx=0x7f58dc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 shutdown_connections 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f58cc06b4d0 0x7f58cc06d990 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58ec0a0c50 0x7f58ec0b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58ec0d0a80 0x7f58ec0a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 --2- 172.21.15.72:0/2285618362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58ec0b4170 0x7f58ec15f6e0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 >> 172.21.15.72:0/2285618362 conn(0x7f58ec0d3c40 msgr2=0x7f58ec0ccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 shutdown_connections 2024-03-06T07:31:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:23.466+0000 7f58f16df640 1 -- 172.21.15.72:0/2285618362 wait complete. 2024-03-06T07:31:23.679 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:23.679 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:23.786 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:24.680 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:24.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.780+0000 7f9473484640 1 Processor -- start 2024-03-06T07:31:24.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.781+0000 7f9473484640 1 -- start start 2024-03-06T07:31:24.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.781+0000 7f9473484640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c15dfe0 0x7f946c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.781+0000 7f9473484640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f946c0d29c0 0x7f946c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.781+0000 7f9473484640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0b3430 con 0x7f946c0cf830 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.781+0000 7f9473484640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f946c0b31e0 con 0x7f946c0d29c0 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.781+0000 7f9473484640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0b3d60 con 0x7f946c15dfe0 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9471a21640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f946c0d29c0 0x7f946c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9470a1f640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f946c0cf830 0x7f946c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:55504/0 (socket says 172.21.15.72:55504) 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9471220640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c15dfe0 0x7f946c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9470a1f640 1 -- 172.21.15.72:0/1420377999 learned_addr learned my addr 172.21.15.72:0/1420377999 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9471a21640 1 -- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c15dfe0 msgr2=0x7f946c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9471a21640 1 --2- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c15dfe0 0x7f946c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9471a21640 1 -- 172.21.15.72:0/1420377999 >> v1:172.21.15.146:6789/0 conn(0x7f946c0cf830 legacy=0x7f946c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.782+0000 7f9471a21640 1 -- 172.21.15.72:0/1420377999 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c0b3710 con 0x7f946c0d29c0 2024-03-06T07:31:24.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.783+0000 7f9471a21640 1 --2- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0d29c0 0x7f946c0b41f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9464009d90 tx=0x7f946402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ad9aa1cc4f97795 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9463fff640 1 -- 172.21.15.72:0/1420377999 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464039070 con 0x7f946c0d29c0 2024-03-06T07:31:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9463fff640 1 -- 172.21.15.72:0/1420377999 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946402cc50 con 0x7f946c0d29c0 2024-03-06T07:31:24.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9463fff640 1 -- 172.21.15.72:0/1420377999 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464034440 con 0x7f946c0d29c0 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9473484640 1 -- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0d29c0 msgr2=0x7f946c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9473484640 1 --2- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0d29c0 0x7f946c0b41f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9464009d90 tx=0x7f946402bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9473484640 1 -- 172.21.15.72:0/1420377999 shutdown_connections 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9473484640 1 --2- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0d29c0 0x7f946c0b41f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9473484640 1 --2- 172.21.15.72:0/1420377999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c15dfe0 0x7f946c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.788+0000 7f9473484640 1 -- 172.21.15.72:0/1420377999 >> 172.21.15.72:0/1420377999 conn(0x7f946c0d3cd0 msgr2=0x7f946c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.789+0000 7f9473484640 1 -- 172.21.15.72:0/1420377999 shutdown_connections 2024-03-06T07:31:24.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.789+0000 7f9473484640 1 -- 172.21.15.72:0/1420377999 wait complete. 2024-03-06T07:31:24.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.790+0000 7f9473484640 1 Processor -- start 2024-03-06T07:31:24.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.790+0000 7f9473484640 1 -- start start 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0cf830 0x7f946c0cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c0d29c0 0x7f946c0cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 0x7f946c14f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0d1df0 con 0x7f946c15dfe0 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0d1c70 con 0x7f946c0cf830 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c15e3c0 con 0x7f946c0d29c0 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 0x7f946c14f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 0x7f946c14f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33142/0 (socket says 172.21.15.72:33142) 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 -- 172.21.15.72:0/321077874 learned_addr learned my addr 172.21.15.72:0/321077874 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471220640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0cf830 0x7f946c0cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9470a1f640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c0d29c0 0x7f946c0cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 -- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c0d29c0 msgr2=0x7f946c0cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c0d29c0 0x7f946c0cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 -- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0cf830 msgr2=0x7f946c0cdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0cf830 0x7f946c0cdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 -- 172.21.15.72:0/321077874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c14f6f0 con 0x7f946c15dfe0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9470a1f640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c0d29c0 0x7f946c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9471a21640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 0x7f946c14f1b0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f9464009d90 tx=0x7f9464004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464039050 con 0x7f946c15dfe0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9464045070 con 0x7f946c15dfe0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c0b8350 con 0x7f946c15dfe0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.792+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464009450 con 0x7f946c15dfe0 2024-03-06T07:31:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.793+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c0b8910 con 0x7f946c15dfe0 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.794+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c04f330 con 0x7f946c15dfe0 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.794+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f946403e610 con 0x7f946c15dfe0 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.794+0000 7f9461ffb640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.795+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f94640acbd0 con 0x7f946c15dfe0 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.795+0000 7f9471220640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:24.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.795+0000 7f9471220640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9468009aa0 tx=0x7f9468009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:24.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:24.799+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f946c14f6f0 con 0x7f946c15dfe0 2024-03-06T07:31:25.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.011+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f946c06a190 con 0x7f946c15dfe0 2024-03-06T07:31:25.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.012+0000 7f9461ffb640 1 -- 172.21.15.72:0/321077874 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f946407baa0 con 0x7f946c15dfe0 2024-03-06T07:31:25.014 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:25.014 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:25.014 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f945006b4d0 msgr2=0x7f945006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f945006b4d0 0x7f945006d990 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9468009aa0 tx=0x7f9468009190 comp rx=0 tx=0).stop 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 msgr2=0x7f946c14f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 0x7f946c14f1b0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f9464009d90 tx=0x7f9464004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 shutdown_connections 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f945006b4d0 0x7f945006d990 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f946c15dfe0 0x7f946c14f1b0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f946c0d29c0 0x7f946c0cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 --2- 172.21.15.72:0/321077874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f946c0cf830 0x7f946c0cdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:25.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 >> 172.21.15.72:0/321077874 conn(0x7f946c0d3cd0 msgr2=0x7f946c0cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:25.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 shutdown_connections 2024-03-06T07:31:25.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:25.014+0000 7f9473484640 1 -- 172.21.15.72:0/321077874 wait complete. 2024-03-06T07:31:25.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:25.228 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:25.788 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:26.229 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:26.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.327+0000 7f34b2116640 1 Processor -- start 2024-03-06T07:31:26.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.327+0000 7f34b2116640 1 -- start start 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34b2116640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f34ac15bb20 0x7f34ac0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34b2116640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0cfff0 0x7f34ac0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34b2116640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac161980 con 0x7f34ac0ce4b0 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34b2116640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac161c60 con 0x7f34ac15bb20 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34b2116640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac161f40 con 0x7f34ac0cfff0 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f34ac15bb20 0x7f34ac0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f34ac15bb20 0x7f34ac0d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33156/0 (socket says 172.21.15.72:33156) 2024-03-06T07:31:26.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 -- 172.21.15.72:0/337472378 learned_addr learned my addr 172.21.15.72:0/337472378 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34aaffd640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0cfff0 0x7f34ac0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 -- 172.21.15.72:0/337472378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0cfff0 msgr2=0x7f34ac0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0cfff0 0x7f34ac0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 -- 172.21.15.72:0/337472378 >> v1:172.21.15.146:6789/0 conn(0x7f34ac0ce4b0 legacy=0x7f34ac0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34ab7fe640 1 -- 172.21.15.72:0/337472378 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac162220 con 0x7f34ac15bb20 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.328+0000 7f34aaffd640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0cfff0 0x7f34ac0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.329+0000 7f34ab7fe640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0d1df0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f349c009870 tx=0x7f349c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64ae19da31c2bea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.329+0000 7f34aa7fc640 1 -- 172.21.15.72:0/337472378 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c034070 con 0x7f34ac15bb20 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.329+0000 7f34aa7fc640 1 -- 172.21.15.72:0/337472378 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349c030840 con 0x7f34ac15bb20 2024-03-06T07:31:26.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.329+0000 7f34aa7fc640 1 -- 172.21.15.72:0/337472378 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349c030c10 con 0x7f34ac15bb20 2024-03-06T07:31:26.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.329+0000 7f34b2116640 1 -- 172.21.15.72:0/337472378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 msgr2=0x7f34ac0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.329+0000 7f34b2116640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0d1df0 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f349c009870 tx=0x7f349c02b980 comp rx=0 tx=0).stop 2024-03-06T07:31:26.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.330+0000 7f34b2116640 1 -- 172.21.15.72:0/337472378 shutdown_connections 2024-03-06T07:31:26.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.330+0000 7f34b2116640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0cfff0 0x7f34ac0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.330+0000 7f34b2116640 1 --2- 172.21.15.72:0/337472378 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0d1df0 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.330+0000 7f34b2116640 1 -- 172.21.15.72:0/337472378 >> 172.21.15.72:0/337472378 conn(0x7f34ac09a9f0 msgr2=0x7f34ac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.330+0000 7f34b2116640 1 -- 172.21.15.72:0/337472378 shutdown_connections 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.330+0000 7f34b2116640 1 -- 172.21.15.72:0/337472378 wait complete. 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.331+0000 7f34b2116640 1 Processor -- start 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.331+0000 7f34b2116640 1 -- start start 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0ce4b0 0x7f34ac0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34ac0cfff0 0x7f34ac0c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0d24b0 con 0x7f34ac15bb20 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0d2330 con 0x7f34ac0cfff0 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0d1980 con 0x7f34ac0ce4b0 2024-03-06T07:31:26.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0c5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33160/0 (socket says 172.21.15.72:33160) 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 -- 172.21.15.72:0/2860747876 learned_addr learned my addr 172.21.15.72:0/2860747876 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34aaffd640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34ac0cfff0 0x7f34ac0c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34ab7fe640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0ce4b0 0x7f34ac0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 -- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0ce4b0 msgr2=0x7f34ac0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0ce4b0 0x7f34ac0cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 -- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34ac0cfff0 msgr2=0x7f34ac0c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34ac0cfff0 0x7f34ac0c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 -- 172.21.15.72:0/2860747876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac0cd6e0 con 0x7f34ac15bb20 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34abfff640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0c5680 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c7a0 tx=0x7f34a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4003b70 con 0x7f34ac15bb20 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a4003d30 con 0x7f34ac15bb20 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac06c1d0 con 0x7f34ac15bb20 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a400aa90 con 0x7f34ac15bb20 2024-03-06T07:31:26.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.332+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac06c790 con 0x7f34ac15bb20 2024-03-06T07:31:26.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.333+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac04f330 con 0x7f34ac15bb20 2024-03-06T07:31:26.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.334+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f34a400ac30 con 0x7f34ac15bb20 2024-03-06T07:31:26.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.334+0000 7f3493fff640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f349806b500 0x7f349806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:26.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.334+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f34a408a150 con 0x7f34ac15bb20 2024-03-06T07:31:26.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.334+0000 7f34ab7fe640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f349806b500 0x7f349806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:26.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.335+0000 7f34ab7fe640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f349806b500 0x7f349806d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f349c036000 tx=0x7f349c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:26.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.339+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f34a4058f90 con 0x7f34ac15bb20 2024-03-06T07:31:26.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.609+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34ac09b150 con 0x7f34ac15bb20 2024-03-06T07:31:26.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.610+0000 7f3493fff640 1 -- 172.21.15.72:0/2860747876 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f34a4058890 con 0x7f34ac15bb20 2024-03-06T07:31:26.612 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:26.612 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:26.613 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f349806b500 msgr2=0x7f349806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f349806b500 0x7f349806d9c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f349c036000 tx=0x7f349c03b040 comp rx=0 tx=0).stop 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 msgr2=0x7f34ac0c5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0c5680 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c7a0 tx=0x7f34a400cc60 comp rx=0 tx=0).stop 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 shutdown_connections 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f349806b500 0x7f349806d9c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f34ac15bb20 0x7f34ac0c5680 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34ac0cfff0 0x7f34ac0c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 --2- 172.21.15.72:0/2860747876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34ac0ce4b0 0x7f34ac0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:26.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 >> 172.21.15.72:0/2860747876 conn(0x7f34ac09a9f0 msgr2=0x7f34ac0b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:26.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 shutdown_connections 2024-03-06T07:31:26.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:26.612+0000 7f34b2116640 1 -- 172.21.15.72:0/2860747876 wait complete. 2024-03-06T07:31:26.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:26.826 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:27.791 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:27.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:27.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 Processor -- start 2024-03-06T07:31:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 -- start start 2024-03-06T07:31:27.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c15dfe0 0x7fe00c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe00c0cf830 0x7fe00c0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe00c0d27f0 0x7fe00c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0d2c50 con 0x7fe00c0d27f0 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0b31e0 con 0x7fe00c0cf830 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.927+0000 7fe0143a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0b3d60 con 0x7fe00c15dfe0 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe00c0d27f0 0x7fe00c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe00c0d27f0 0x7fe00c0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33164/0 (socket says 172.21.15.72:33164) 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 -- 172.21.15.72:0/3100762975 learned_addr learned my addr 172.21.15.72:0/3100762975 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 -- 172.21.15.72:0/3100762975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c15dfe0 msgr2=0x7fe00c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 --2- 172.21.15.72:0/3100762975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c15dfe0 0x7fe00c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 -- 172.21.15.72:0/3100762975 >> v2:172.21.15.146:3300/0 conn(0x7fe00c0cf830 msgr2=0x7fe00c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01193c640 1 --2- 172.21.15.72:0/3100762975 >> v2:172.21.15.146:3300/0 conn(0x7fe00c0cf830 0x7fe00c0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 --2- 172.21.15.72:0/3100762975 >> v2:172.21.15.146:3300/0 conn(0x7fe00c0cf830 0x7fe00c0d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01293e640 1 -- 172.21.15.72:0/3100762975 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c0b3430 con 0x7fe00c0d27f0 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.928+0000 7fe01193c640 1 --2- 172.21.15.72:0/3100762975 >> v2:172.21.15.146:3300/0 conn(0x7fe00c0cf830 0x7fe00c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe01293e640 1 --2- 172.21.15.72:0/3100762975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0d0300 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fe00c08cba0 tx=0x7fe00402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbc89a3d2f07ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe01113b640 1 -- 172.21.15.72:0/3100762975 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00402ca70 con 0x7fe00c0d27f0 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe01113b640 1 -- 172.21.15.72:0/3100762975 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00402cc10 con 0x7fe00c0d27f0 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe01113b640 1 -- 172.21.15.72:0/3100762975 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0040343f0 con 0x7fe00c0d27f0 2024-03-06T07:31:27.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 -- 172.21.15.72:0/3100762975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c0d27f0 msgr2=0x7fe00c0d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3100762975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0d0300 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fe00c08cba0 tx=0x7fe00402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 -- 172.21.15.72:0/3100762975 shutdown_connections 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3100762975 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0d0300 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3100762975 >> v2:172.21.15.146:3300/0 conn(0x7fe00c0cf830 0x7fe00c0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3100762975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c15dfe0 0x7fe00c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 -- 172.21.15.72:0/3100762975 >> 172.21.15.72:0/3100762975 conn(0x7fe00c0d3cd0 msgr2=0x7fe00c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.929+0000 7fe0143a1640 1 -- 172.21.15.72:0/3100762975 shutdown_connections 2024-03-06T07:31:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.930+0000 7fe0143a1640 1 -- 172.21.15.72:0/3100762975 wait complete. 2024-03-06T07:31:27.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 Processor -- start 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 -- start start 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c0cf830 0x7fe00c1f3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 0x7fe00c0ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0d1c70 con 0x7fe00c15dfe0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0b39f0 con 0x7fe00c0d27f0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe0143a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0d1df0 con 0x7fe00c0cf830 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe01293e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 0x7fe00c0ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe01293e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 0x7fe00c0ccb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33174/0 (socket says 172.21.15.72:33174) 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.931+0000 7fe01293e640 1 -- 172.21.15.72:0/3143156315 learned_addr learned my addr 172.21.15.72:0/3143156315 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01193c640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01213d640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c0cf830 0x7fe00c1f3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01293e640 1 -- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c0cf830 msgr2=0x7fe00c1f3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01293e640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c0cf830 0x7fe00c1f3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01293e640 1 -- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe00c0d27f0 msgr2=0x7fe00c0cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01293e640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01293e640 1 -- 172.21.15.72:0/3143156315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c1f3d60 con 0x7fe00c15dfe0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe01293e640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 0x7fe00c0ccb40 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fe004009f40 tx=0x7fe004004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004043020 con 0x7fe00c15dfe0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00403e450 con 0x7fe00c15dfe0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c1f4040 con 0x7fe00c15dfe0 2024-03-06T07:31:27.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe00403e860 con 0x7fe00c15dfe0 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.932+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c15e4f0 con 0x7fe00c15dfe0 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.933+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c0cc270 con 0x7fe00c15dfe0 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.934+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe004046020 con 0x7fe00c15dfe0 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.934+0000 7fe002ffd640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdff006b4d0 0x7fdff006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.934+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe0040acc70 con 0x7fe00c15dfe0 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.934+0000 7fe01213d640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdff006b4d0 0x7fdff006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:27.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.935+0000 7fe01213d640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdff006b4d0 0x7fdff006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0cc640 tx=0x7fe008008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:27.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:27.943+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe00c0cc270 con 0x7fe00c15dfe0 2024-03-06T07:31:28.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.158+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe00c0ce9a0 con 0x7fe00c15dfe0 2024-03-06T07:31:28.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.159+0000 7fe002ffd640 1 -- 172.21.15.72:0/3143156315 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe00404d030 con 0x7fe00c15dfe0 2024-03-06T07:31:28.161 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:28.161 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:28.161 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdff006b4d0 msgr2=0x7fdff006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdff006b4d0 0x7fdff006d990 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0cc640 tx=0x7fe008008040 comp rx=0 tx=0).stop 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 msgr2=0x7fe00c0ccb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 0x7fe00c0ccb40 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fe004009f40 tx=0x7fe004004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 shutdown_connections 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdff006b4d0 0x7fdff006d990 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.160+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe00c15dfe0 0x7fe00c0ccb40 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.161+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe00c0d27f0 0x7fe00c0cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.161+0000 7fe0143a1640 1 --2- 172.21.15.72:0/3143156315 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe00c0cf830 0x7fe00c1f3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:28.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.161+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 >> 172.21.15.72:0/3143156315 conn(0x7fe00c0d3cd0 msgr2=0x7fe00c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:28.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.161+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 shutdown_connections 2024-03-06T07:31:28.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:28.161+0000 7fe0143a1640 1 -- 172.21.15.72:0/3143156315 wait complete. 2024-03-06T07:31:28.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:28.375 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:29.376 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:29.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.476+0000 7f39870c9640 1 Processor -- start 2024-03-06T07:31:29.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.476+0000 7f39870c9640 1 -- start start 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f39870c9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f398006da40 0x7f39800cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f39870c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800cc980 con 0x7f39801424f0 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f39870c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39800ccc60 con 0x7f39800cd7a0 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f39870c9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39800cd220 con 0x7f398006da40 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f3984e65640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f39801424f0 0x7f398006d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:51554/0 (socket says 172.21.15.72:51554) 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f3984e65640 1 -- 172.21.15.72:0/3490826345 learned_addr learned my addr 172.21.15.72:0/3490826345 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:29.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f397ffff640 1 --2- 172.21.15.72:0/3490826345 >> v2:172.21.15.146:3300/0 conn(0x7f398006da40 0x7f39800cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747479659 0 0) 0x7f39800cc980 con 0x7f39801424f0 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.477+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3968003520 con 0x7f39801424f0 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404824735 0 0) 0x7f3968003520 con 0x7f39801424f0 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 >> v2:172.21.15.146:3300/0 conn(0x7f398006da40 msgr2=0x7f39800cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 --2- 172.21.15.72:0/3490826345 >> v2:172.21.15.146:3300/0 conn(0x7f398006da40 0x7f39800cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 >> v1:172.21.15.146:6789/0 conn(0x7f39800cd7a0 legacy=0x7f3980071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800ccf40 con 0x7f39801424f0 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f3970001670 con 0x7f39801424f0 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 >> v1:172.21.15.72:6789/0 conn(0x7f39801424f0 legacy=0x7f398006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39680039b0 0x7f3968003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f3968004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39680051c0 0x7f396800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800cd220 con 0x7f3968004800 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396800c060 con 0x7f39680051c0 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396800c260 con 0x7f39680039b0 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397ffff640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f3968004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f3985666640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39680051c0 0x7f396800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f3984e65640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39680039b0 0x7f3968003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397ffff640 1 -- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39680039b0 msgr2=0x7f3968003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397ffff640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39680039b0 0x7f3968003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397ffff640 1 -- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39680051c0 msgr2=0x7f396800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397ffff640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39680051c0 0x7f396800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f397ffff640 1 -- 172.21.15.72:0/3490826345 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968003520 con 0x7f3968004800 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.478+0000 7f3985666640 1 -- 172.21.15.72:0/3490826345 reap_dead start 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.479+0000 7f397ffff640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f3968004c80 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f3974009ac0 tx=0x7f3968005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90872b72f278879 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.479+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974039070 con 0x7f3968004800 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.479+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397402c6b0 con 0x7f3968004800 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.479+0000 7f397f7fe640 1 -- 172.21.15.72:0/3490826345 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974009690 con 0x7f3968004800 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 -- 172.21.15.72:0/3490826345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 msgr2=0x7f3968004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f3968004c80 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f3974009ac0 tx=0x7f3968005830 comp rx=0 tx=0).stop 2024-03-06T07:31:29.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 -- 172.21.15.72:0/3490826345 shutdown_connections 2024-03-06T07:31:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 --2- 172.21.15.72:0/3490826345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f3968004c80 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 -- 172.21.15.72:0/3490826345 >> 172.21.15.72:0/3490826345 conn(0x7f39800cf8a0 msgr2=0x7f39800cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 -- 172.21.15.72:0/3490826345 shutdown_connections 2024-03-06T07:31:29.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.480+0000 7f39870c9640 1 -- 172.21.15.72:0/3490826345 wait complete. 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.481+0000 7f39870c9640 1 Processor -- start 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.481+0000 7f39870c9640 1 -- start start 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f39870c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f39801f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f39870c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f398013e6b0 0x7f39800b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f39870c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39801440f0 0x7f3980144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f39870c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800cc470 con 0x7f3968004800 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f39870c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39800cc2f0 con 0x7f39801440f0 2024-03-06T07:31:29.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f39870c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800cc5f0 con 0x7f398013e6b0 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f39801f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f39801f4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33200/0 (socket says 172.21.15.72:33200) 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 -- 172.21.15.72:0/432709755 learned_addr learned my addr 172.21.15.72:0/432709755 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3985666640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39801440f0 0x7f3980144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f397ffff640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f398013e6b0 0x7f39800b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 -- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f398013e6b0 msgr2=0x7f39800b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f398013e6b0 0x7f39800b3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 -- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39801440f0 msgr2=0x7f3980144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39801440f0 0x7f3980144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 -- 172.21.15.72:0/432709755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39800b4a90 con 0x7f3968004800 2024-03-06T07:31:29.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.482+0000 7f3984e65640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f39801f4f90 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f39700072c0 tx=0x7f3970007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.483+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000f960 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.483+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3970018070 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.483+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f398013eb30 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.483+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397000fd60 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.483+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39800b44d0 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.484+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398006e5d0 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.484+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f397001d460 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.484+0000 7f397d7fa640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f396c06b500 0x7f396c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.485+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f397008a1f0 con 0x7f3968004800 2024-03-06T07:31:29.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.485+0000 7f397ffff640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f396c06b500 0x7f396c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:29.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.485+0000 7f397ffff640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f396c06b500 0x7f396c06d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3974036000 tx=0x7f3974030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:29.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.489+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f398006e5d0 con 0x7f3968004800 2024-03-06T07:31:29.792 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:29.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.840+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f398004f330 con 0x7f3968004800 2024-03-06T07:31:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.841+0000 7f397d7fa640 1 -- 172.21.15.72:0/432709755 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f3970059030 con 0x7f3968004800 2024-03-06T07:31:29.843 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:29.843 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:29.843 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f396c06b500 msgr2=0x7f396c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f396c06b500 0x7f396c06d9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3974036000 tx=0x7f3974030000 comp rx=0 tx=0).stop 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 msgr2=0x7f39801f4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f39801f4f90 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f39700072c0 tx=0x7f3970007780 comp rx=0 tx=0).stop 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 shutdown_connections 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f396c06b500 0x7f396c06d9c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39801440f0 0x7f3980144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f398013e6b0 0x7f39800b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 --2- 172.21.15.72:0/432709755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3968004800 0x7f39801f4f90 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 >> 172.21.15.72:0/432709755 conn(0x7f39800cf8a0 msgr2=0x7f39800cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:29.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 shutdown_connections 2024-03-06T07:31:29.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:29.843+0000 7f39870c9640 1 -- 172.21.15.72:0/432709755 wait complete. 2024-03-06T07:31:30.057 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:30.057 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:31.057 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:31.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.177+0000 7f6788581640 1 Processor -- start 2024-03-06T07:31:31.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.177+0000 7f6788581640 1 -- start start 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6788581640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f67800d22b0 0x7f67800d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6788581640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f67800b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6788581640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67800b3430 con 0x7f678015dfe0 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6788581640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67800d2c50 con 0x7f67800d22b0 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6788581640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67800b31e0 con 0x7f67800b41f0 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6785b1c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f67800d22b0 0x7f67800d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6785b1c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f67800d22b0 0x7f67800d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33210/0 (socket says 172.21.15.72:33210) 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.178+0000 7f6785b1c640 1 -- 172.21.15.72:0/1523020182 learned_addr learned my addr 172.21.15.72:0/1523020182 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.179+0000 7f6786b1e640 1 --2- 172.21.15.72:0/1523020182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f67800b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.179+0000 7f6785b1c640 1 -- 172.21.15.72:0/1523020182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 msgr2=0x7f67800b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.179+0000 7f6785b1c640 1 --2- 172.21.15.72:0/1523020182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f67800b4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.179+0000 7f6785b1c640 1 -- 172.21.15.72:0/1523020182 >> v1:172.21.15.146:6789/0 conn(0x7f678015dfe0 legacy=0x7f67800cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.179+0000 7f6785b1c640 1 -- 172.21.15.72:0/1523020182 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67800b3710 con 0x7f67800d22b0 2024-03-06T07:31:31.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.179+0000 7f6785b1c640 1 --2- 172.21.15.72:0/1523020182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67800d22b0 0x7f67800d2710 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6770009870 tx=0x7f677002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c331da44ef21da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f678531b640 1 -- 172.21.15.72:0/1523020182 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6770035020 con 0x7f67800d22b0 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f678531b640 1 -- 172.21.15.72:0/1523020182 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6770031640 con 0x7f67800d22b0 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f678531b640 1 -- 172.21.15.72:0/1523020182 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677002d040 con 0x7f67800d22b0 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 -- 172.21.15.72:0/1523020182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67800d22b0 msgr2=0x7f67800d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 --2- 172.21.15.72:0/1523020182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67800d22b0 0x7f67800d2710 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f6770009870 tx=0x7f677002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 -- 172.21.15.72:0/1523020182 shutdown_connections 2024-03-06T07:31:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 --2- 172.21.15.72:0/1523020182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f67800b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 --2- 172.21.15.72:0/1523020182 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f67800d22b0 0x7f67800d2710 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 -- 172.21.15.72:0/1523020182 >> 172.21.15.72:0/1523020182 conn(0x7f67800d3cd0 msgr2=0x7f67800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 -- 172.21.15.72:0/1523020182 shutdown_connections 2024-03-06T07:31:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.180+0000 7f6788581640 1 -- 172.21.15.72:0/1523020182 wait complete. 2024-03-06T07:31:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.181+0000 7f6788581640 1 Processor -- start 2024-03-06T07:31:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.181+0000 7f6788581640 1 -- start start 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6788581640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f6780152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6788581640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67800d22b0 0x7f67800b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6788581640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 0x7f6780160cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6788581640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67800cfa00 con 0x7f678015dfe0 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6788581640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67800b3d60 con 0x7f67800d22b0 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6788581640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67800d1c70 con 0x7f67800b41f0 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6786b1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 0x7f6780160cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6786b1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 0x7f6780160cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33216/0 (socket says 172.21.15.72:33216) 2024-03-06T07:31:31.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6786b1e640 1 -- 172.21.15.72:0/1232747576 learned_addr learned my addr 172.21.15.72:0/1232747576 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6785b1c640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67800d22b0 0x7f67800b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f678631d640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f6780152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6786b1e640 1 -- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 msgr2=0x7f6780152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.182+0000 7f6786b1e640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f6780152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6786b1e640 1 -- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67800d22b0 msgr2=0x7f67800b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6786b1e640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67800d22b0 0x7f67800b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6786b1e640 1 -- 172.21.15.72:0/1232747576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67800b7210 con 0x7f678015dfe0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f678631d640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f6780152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6786b1e640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 0x7f6780160cb0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f677800cd70 tx=0x7f677800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778003b90 con 0x7f678015dfe0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6778003d30 con 0x7f678015dfe0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67800b74f0 con 0x7f678015dfe0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6778004240 con 0x7f678015dfe0 2024-03-06T07:31:31.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.183+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67800b7ab0 con 0x7f678015dfe0 2024-03-06T07:31:31.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.184+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678004f330 con 0x7f678015dfe0 2024-03-06T07:31:31.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.184+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f67780043e0 con 0x7f678015dfe0 2024-03-06T07:31:31.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.185+0000 7f6776ffd640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:31.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.185+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f677808adc0 con 0x7f678015dfe0 2024-03-06T07:31:31.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.185+0000 7f678631d640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:31.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.185+0000 7f678631d640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f676406b4d0 0x7f676406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f677c00a830 tx=0x7f677c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:31.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.189+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f677801d340 con 0x7f678015dfe0 2024-03-06T07:31:31.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.403+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67800ceb20 con 0x7f678015dfe0 2024-03-06T07:31:31.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.404+0000 7f6776ffd640 1 -- 172.21.15.72:0/1232747576 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f6778059c00 con 0x7f678015dfe0 2024-03-06T07:31:31.406 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:31.406 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:31.407 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f676406b4d0 msgr2=0x7f676406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f676406b4d0 0x7f676406d990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f677c00a830 tx=0x7f677c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 msgr2=0x7f6780160cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 0x7f6780160cb0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f677800cd70 tx=0x7f677800a660 comp rx=0 tx=0).stop 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 shutdown_connections 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f676406b4d0 0x7f676406d990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f678015dfe0 0x7f6780160cb0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67800d22b0 0x7f67800b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 --2- 172.21.15.72:0/1232747576 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67800b41f0 0x7f6780152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:31.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 >> 172.21.15.72:0/1232747576 conn(0x7f67800d3cd0 msgr2=0x7f678006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:31.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 shutdown_connections 2024-03-06T07:31:31.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:31.406+0000 7f6788581640 1 -- 172.21.15.72:0/1232747576 wait complete. 2024-03-06T07:31:31.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:31.619 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:31.795 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:32.621 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:32.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.720+0000 7fe1a37a1640 1 Processor -- start 2024-03-06T07:31:32.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.721+0000 7fe1a37a1640 1 -- start start 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.721+0000 7fe1a37a1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe19c15dfe0 0x7fe19c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.721+0000 7fe1a37a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.721+0000 7fe1a37a1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c0b3430 con 0x7fe19c0cf830 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.721+0000 7fe1a37a1640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19c0b31e0 con 0x7fe19c15dfe0 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.721+0000 7fe1a37a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0b3d60 con 0x7fe19c0d29c0 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe19c15dfe0 0x7fe19c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe19c15dfe0 0x7fe19c0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45144/0 (socket says 172.21.15.72:45144) 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 -- 172.21.15.72:0/1149398031 learned_addr learned my addr 172.21.15.72:0/1149398031 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a1d3e640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 -- 172.21.15.72:0/1149398031 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 msgr2=0x7fe19c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 -- 172.21.15.72:0/1149398031 >> v1:172.21.15.146:6789/0 conn(0x7fe19c0cf830 legacy=0x7fe19c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a153d640 1 -- 172.21.15.72:0/1149398031 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0b3710 con 0x7fe19c15dfe0 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.722+0000 7fe1a1d3e640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a153d640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0cf2f0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fe198009870 tx=0x7fe19802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded8fd74f46f97e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe193fff640 1 -- 172.21.15.72:0/1149398031 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198035020 con 0x7fe19c15dfe0 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe193fff640 1 -- 172.21.15.72:0/1149398031 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe198031640 con 0x7fe19c15dfe0 2024-03-06T07:31:32.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe193fff640 1 -- 172.21.15.72:0/1149398031 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19802d040 con 0x7fe19c15dfe0 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1149398031 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c15dfe0 msgr2=0x7fe19c0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0cf2f0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7fe198009870 tx=0x7fe19802b8f0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1149398031 shutdown_connections 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1149398031 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0cf2f0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1149398031 >> 172.21.15.72:0/1149398031 conn(0x7fe19c0d3cd0 msgr2=0x7fe19c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1149398031 shutdown_connections 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.723+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1149398031 wait complete. 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 Processor -- start 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 -- start start 2024-03-06T07:31:32.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 0x7fe19c1547c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c15fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0d1df0 con 0x7fe19c0cf830 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0d1c70 con 0x7fe19c15dfe0 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a37a1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c15e3c0 con 0x7fe19c0d29c0 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a153d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 0x7fe19c1547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a153d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 0x7fe19c1547c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45150/0 (socket says 172.21.15.72:45150) 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a153d640 1 -- 172.21.15.72:0/1282740019 learned_addr learned my addr 172.21.15.72:0/1282740019 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a1d3e640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.725+0000 7fe1a0d3c640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c15fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a153d640 1 -- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 msgr2=0x7fe19c15fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a153d640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c15fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a153d640 1 -- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c15dfe0 msgr2=0x7fe19c0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a153d640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a153d640 1 -- 172.21.15.72:0/1282740019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0b9b60 con 0x7fe19c0cf830 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a1d3e640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a0d3c640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a153d640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 0x7fe19c1547c0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fe19802b930 tx=0x7fe1980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198040070 con 0x7fe19c0cf830 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe198039c10 con 0x7fe19c0cf830 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c0b92c0 con 0x7fe19c0cf830 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198035040 con 0x7fe19c0cf830 2024-03-06T07:31:32.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.726+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c0b9880 con 0x7fe19c0cf830 2024-03-06T07:31:32.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.727+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c04f330 con 0x7fe19c0cf830 2024-03-06T07:31:32.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.728+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe19804b060 con 0x7fe19c0cf830 2024-03-06T07:31:32.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.728+0000 7fe191ffb640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:32.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.728+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe1980abd60 con 0x7fe19c0cf830 2024-03-06T07:31:32.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.728+0000 7fe1a0d3c640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:32.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.729+0000 7fe1a0d3c640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18006b4d0 0x7fe18006d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0ca100 tx=0x7fe18c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:32.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.733+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe19802f300 con 0x7fe19c0cf830 2024-03-06T07:31:32.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.951+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe19c154d00 con 0x7fe19c0cf830 2024-03-06T07:31:32.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.952+0000 7fe191ffb640 1 -- 172.21.15.72:0/1282740019 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe19807aba0 con 0x7fe19c0cf830 2024-03-06T07:31:32.954 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:32.954 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:32.954 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:32.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18006b4d0 msgr2=0x7fe18006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18006b4d0 0x7fe18006d990 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe19c0ca100 tx=0x7fe18c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 msgr2=0x7fe19c1547c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 0x7fe19c1547c0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fe19802b930 tx=0x7fe1980046c0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 shutdown_connections 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18006b4d0 0x7fe18006d990 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c15dfe0 0x7fe19c0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c0d29c0 0x7fe19c15fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 --2- 172.21.15.72:0/1282740019 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c0cf830 0x7fe19c1547c0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.954+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 >> 172.21.15.72:0/1282740019 conn(0x7fe19c0d3cd0 msgr2=0x7fe19c06c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.955+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 shutdown_connections 2024-03-06T07:31:32.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:32.955+0000 7fe1a37a1640 1 -- 172.21.15.72:0/1282740019 wait complete. 2024-03-06T07:31:33.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:33.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:33.797 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:34.172 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:34.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.307+0000 7f7dcdf0c640 1 Processor -- start 2024-03-06T07:31:34.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 -- start start 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dc815dfe0 0x7f7dc80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7dc80d27f0 0x7f7dc80d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc80d2c50 con 0x7f7dc815dfe0 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc80b31e0 con 0x7f7dc80d27f0 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dcdf0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80b3d60 con 0x7f7dc80cf830 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dc77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dc815dfe0 0x7f7dc80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dc77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7dc815dfe0 0x7f7dc80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45152/0 (socket says 172.21.15.72:45152) 2024-03-06T07:31:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.308+0000 7f7dc77fe640 1 -- 172.21.15.72:0/2113949619 learned_addr learned my addr 172.21.15.72:0/2113949619 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.309+0000 7f7dc77fe640 1 -- 172.21.15.72:0/2113949619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 msgr2=0x7f7dc80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.309+0000 7f7dc77fe640 1 --2- 172.21.15.72:0/2113949619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.309+0000 7f7dc77fe640 1 -- 172.21.15.72:0/2113949619 >> v2:172.21.15.146:3300/0 conn(0x7f7dc80d27f0 msgr2=0x7f7dc80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.309+0000 7f7dc77fe640 1 --2- 172.21.15.72:0/2113949619 >> v2:172.21.15.146:3300/0 conn(0x7f7dc80d27f0 0x7f7dc80d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.309+0000 7f7dc77fe640 1 -- 172.21.15.72:0/2113949619 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80b3430 con 0x7f7dc815dfe0 2024-03-06T07:31:34.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.309+0000 7f7dc77fe640 1 --2- 172.21.15.72:0/2113949619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80cf2f0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8067ad0 tx=0x7f7dc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccc7f56384715e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dc67fc640 1 -- 172.21.15.72:0/2113949619 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc002ca70 con 0x7f7dc815dfe0 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dc67fc640 1 -- 172.21.15.72:0/2113949619 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc002cc10 con 0x7f7dc815dfe0 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dc67fc640 1 -- 172.21.15.72:0/2113949619 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00343f0 con 0x7f7dc815dfe0 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/2113949619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 msgr2=0x7f7dc80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/2113949619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80cf2f0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f7dc8067ad0 tx=0x7f7dc002bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/2113949619 shutdown_connections 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/2113949619 >> v2:172.21.15.146:3300/0 conn(0x7f7dc80d27f0 0x7f7dc80d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/2113949619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/2113949619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80cf2f0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/2113949619 >> 172.21.15.72:0/2113949619 conn(0x7f7dc80d3cd0 msgr2=0x7f7dc80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/2113949619 shutdown_connections 2024-03-06T07:31:34.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.310+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/2113949619 wait complete. 2024-03-06T07:31:34.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.311+0000 7f7dcdf0c640 1 Processor -- start 2024-03-06T07:31:34.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 -- start start 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc81531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dc80d27f0 0x7f7dc80cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80b8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80d1c70 con 0x7f7dc815dfe0 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80b39f0 con 0x7f7dc80d27f0 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dcdf0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80d1df0 con 0x7f7dc80cf830 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.312+0000 7f7dc7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80b8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80b8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45158/0 (socket says 172.21.15.72:45158) 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 -- 172.21.15.72:0/1458840359 learned_addr learned my addr 172.21.15.72:0/1458840359 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc6ffd640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dc80d27f0 0x7f7dc80cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc77fe640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc81531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 -- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 msgr2=0x7f7dc81531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc81531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 -- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dc80d27f0 msgr2=0x7f7dc80cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dc80d27f0 0x7f7dc80cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 -- 172.21.15.72:0/1458840359 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc80cc890 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dc7fff640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80b8160 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009fd0 tx=0x7f7db800b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8015020 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db8011830 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc80cb7f0 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db8011c00 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.313+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc80cbdb0 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.314+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc804f330 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.315+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7db8018020 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.315+0000 7f7dbffff640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7da806b4d0 0x7f7da806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.315+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7db8089d80 con 0x7f7dc815dfe0 2024-03-06T07:31:34.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.315+0000 7f7dc77fe640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7da806b4d0 0x7f7da806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:34.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.316+0000 7f7dc77fe640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7da806b4d0 0x7f7da806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0035000 tx=0x7f7dc0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:34.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.320+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7db801f030 con 0x7f7dc815dfe0 2024-03-06T07:31:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.534+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7dc80d0db0 con 0x7f7dc815dfe0 2024-03-06T07:31:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.535+0000 7f7dbffff640 1 -- 172.21.15.72:0/1458840359 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f7db8058bc0 con 0x7f7dc815dfe0 2024-03-06T07:31:34.537 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:34.537 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7da806b4d0 msgr2=0x7f7da806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7da806b4d0 0x7f7da806d990 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0035000 tx=0x7f7dc0030000 comp rx=0 tx=0).stop 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 msgr2=0x7f7dc80b8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80b8160 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009fd0 tx=0x7f7db800b820 comp rx=0 tx=0).stop 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 shutdown_connections 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7da806b4d0 0x7f7da806d990 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7dc815dfe0 0x7f7dc80b8160 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7dc80d27f0 0x7f7dc80cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 --2- 172.21.15.72:0/1458840359 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7dc80cf830 0x7f7dc81531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.536+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 >> 172.21.15.72:0/1458840359 conn(0x7f7dc80d3cd0 msgr2=0x7f7dc80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.537+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 shutdown_connections 2024-03-06T07:31:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:34.537+0000 7f7dcdf0c640 1 -- 172.21.15.72:0/1458840359 wait complete. 2024-03-06T07:31:34.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:34.751 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:35.752 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:35.798 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:35.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.851+0000 7f9a48088640 1 Processor -- start 2024-03-06T07:31:35.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.851+0000 7f9a48088640 1 -- start start 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a48088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a4015dfe0 0x7f9a400cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a48088640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a400b3430 con 0x7f9a400cf830 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a48088640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a400b3710 con 0x7f9a400d29c0 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a48088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400b31e0 con 0x7f9a4015dfe0 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a45623640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9a400cf830 0x7f9a400d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41234/0 (socket says 172.21.15.72:41234) 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a45623640 1 -- 172.21.15.72:0/2446563944 learned_addr learned my addr 172.21.15.72:0/2446563944 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a45e24640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a4015dfe0 0x7f9a400cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151546862 0 0) 0x7f9a400b3430 con 0x7f9a400cf830 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c003520 con 0x7f9a400cf830 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049603472 0 0) 0x7f9a2c003520 con 0x7f9a400cf830 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a4015dfe0 msgr2=0x7f9a400cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a4015dfe0 0x7f9a400cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 >> v1:172.21.15.146:6789/0 conn(0x7f9a400d29c0 legacy=0x7f9a400b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a400b39f0 con 0x7f9a400cf830 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.852+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9a3c001670 con 0x7f9a400cf830 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 >> v1:172.21.15.72:6789/0 conn(0x7f9a400cf830 legacy=0x7f9a400d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a2c0039e0 0x7f9a2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2c004800 0x7f9a2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0051c0 0x7f9a2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400b31e0 con 0x7f9a2c0039e0 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c00bfb0 con 0x7f9a2c004800 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c00c180 con 0x7f9a2c0051c0 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a4015dfe0 0x7f9a400cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a2c0039e0 0x7f9a2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45623640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2c004800 0x7f9a2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a46625640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0051c0 0x7f9a2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 -- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0051c0 msgr2=0x7f9a2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0051c0 0x7f9a2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 -- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2c004800 msgr2=0x7f9a2c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a2c004800 0x7f9a2c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a45e24640 1 -- 172.21.15.72:0/2446563944 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c003520 con 0x7f9a2c0039e0 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.853+0000 7f9a46625640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0051c0 0x7f9a2c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a45e24640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a2c0039e0 0x7f9a2c003dc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009b00 tx=0x7f9a2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c64605e9255ddd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a300097d0 con 0x7f9a2c0039e0 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a30004030 con 0x7f9a2c0039e0 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a44e22640 1 -- 172.21.15.72:0/2446563944 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a30004400 con 0x7f9a2c0039e0 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a48088640 1 -- 172.21.15.72:0/2446563944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a2c0039e0 msgr2=0x7f9a2c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a48088640 1 --2- 172.21.15.72:0/2446563944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a2c0039e0 0x7f9a2c003dc0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009b00 tx=0x7f9a2c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a46625640 1 -- 172.21.15.72:0/2446563944 reap_dead start 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a48088640 1 -- 172.21.15.72:0/2446563944 shutdown_connections 2024-03-06T07:31:35.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.854+0000 7f9a48088640 1 -- 172.21.15.72:0/2446563944 >> 172.21.15.72:0/2446563944 conn(0x7f9a400d3cd0 msgr2=0x7f9a400d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:35.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.855+0000 7f9a48088640 1 -- 172.21.15.72:0/2446563944 shutdown_connections 2024-03-06T07:31:35.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.855+0000 7f9a48088640 1 -- 172.21.15.72:0/2446563944 wait complete. 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 Processor -- start 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 -- start start 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0039e0 0x7f9a400cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 0x7f9a400ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a401f6610 0x7f9a401609a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4015e3c0 con 0x7f9a401f4a70 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400b3d60 con 0x7f9a401f6610 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a48088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4015e540 con 0x7f9a2c0039e0 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.856+0000 7f9a45623640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 0x7f9a400ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 0x7f9a400ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45178/0 (socket says 172.21.15.72:45178) 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 -- 172.21.15.72:0/4053663106 learned_addr learned my addr 172.21.15.72:0/4053663106 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a46625640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a401f6610 0x7f9a401609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45e24640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0039e0 0x7f9a400cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 -- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0039e0 msgr2=0x7f9a400cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0039e0 0x7f9a400cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 -- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a401f6610 msgr2=0x7f9a401609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a401f6610 0x7f9a401609a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 -- 172.21.15.72:0/4053663106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a401f4ef0 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a45623640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 0x7f9a400ccbf0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0072c0 tx=0x7f9a3c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00f9c0 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a3c018070 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c00fdc0 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a400cd440 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.857+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a4015eb80 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.858+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4004f330 con 0x7f9a401f4a70 2024-03-06T07:31:35.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.859+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9a3c01d460 con 0x7f9a401f4a70 2024-03-06T07:31:35.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.859+0000 7f9a367fc640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a2406b4d0 0x7f9a2406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:35.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.859+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9a3c08a310 con 0x7f9a401f4a70 2024-03-06T07:31:35.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.859+0000 7f9a45e24640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a2406b4d0 0x7f9a2406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:35.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.859+0000 7f9a45e24640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a2406b4d0 0x7f9a2406d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9a30035000 tx=0x7f9a30030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:35.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:35.867+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9a401f4ef0 con 0x7f9a401f4a70 2024-03-06T07:31:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.083+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a4006c230 con 0x7f9a401f4a70 2024-03-06T07:31:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.083+0000 7f9a367fc640 1 -- 172.21.15.72:0/4053663106 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f9a3c059150 con 0x7f9a401f4a70 2024-03-06T07:31:36.086 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:36.086 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:36.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a2406b4d0 msgr2=0x7f9a2406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a2406b4d0 0x7f9a2406d990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f9a30035000 tx=0x7f9a30030000 comp rx=0 tx=0).stop 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 msgr2=0x7f9a400ccbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 0x7f9a400ccbf0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0072c0 tx=0x7f9a3c007780 comp rx=0 tx=0).stop 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 shutdown_connections 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9a2406b4d0 0x7f9a2406d990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a401f6610 0x7f9a401609a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9a401f4a70 0x7f9a400ccbf0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 --2- 172.21.15.72:0/4053663106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a2c0039e0 0x7f9a400cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 >> 172.21.15.72:0/4053663106 conn(0x7f9a400d3cd0 msgr2=0x7f9a4006b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.085+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 shutdown_connections 2024-03-06T07:31:36.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:36.086+0000 7f9a48088640 1 -- 172.21.15.72:0/4053663106 wait complete. 2024-03-06T07:31:36.300 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:36.300 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:37.301 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:37.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.400+0000 7f07defac640 1 Processor -- start 2024-03-06T07:31:37.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.400+0000 7f07defac640 1 -- start start 2024-03-06T07:31:37.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07defac640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07d80d04a0 0x7f07d80ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07defac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d8161980 con 0x7f07d80cfff0 2024-03-06T07:31:37.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07defac640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d8161c60 con 0x7f07d815bb20 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07defac640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d80d2500 con 0x7f07d80d04a0 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07dd549640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07d80d04a0 0x7f07d80ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07d7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f07d80cfff0 0x7f07d80cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41250/0 (socket says 172.21.15.72:41250) 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07d7fff640 1 -- 172.21.15.72:0/4294008978 learned_addr learned my addr 172.21.15.72:0/4294008978 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.401+0000 7f07dd549640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07d80d04a0 0x7f07d80ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45192/0 (socket says 172.21.15.72:45192) 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07dd549640 1 -- 172.21.15.72:0/4294008978 >> v1:172.21.15.146:6789/0 conn(0x7f07d815bb20 legacy=0x7f07d80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07dd549640 1 -- 172.21.15.72:0/4294008978 >> v1:172.21.15.72:6789/0 conn(0x7f07d80cfff0 legacy=0x7f07d80cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07dd549640 1 -- 172.21.15.72:0/4294008978 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8161f40 con 0x7f07d80d04a0 2024-03-06T07:31:37.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07dd549640 1 --2- 172.21.15.72:0/4294008978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80d04a0 0x7f07d80ccfc0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009d90 tx=0x7f07d002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce1c9eec39abfe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07d77fe640 1 -- 172.21.15.72:0/4294008978 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0039070 con 0x7f07d80d04a0 2024-03-06T07:31:37.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07d77fe640 1 -- 172.21.15.72:0/4294008978 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d002cc60 con 0x7f07d80d04a0 2024-03-06T07:31:37.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.402+0000 7f07d77fe640 1 -- 172.21.15.72:0/4294008978 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d0034660 con 0x7f07d80d04a0 2024-03-06T07:31:37.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 -- 172.21.15.72:0/4294008978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80d04a0 msgr2=0x7f07d80ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 --2- 172.21.15.72:0/4294008978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80d04a0 0x7f07d80ccfc0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009d90 tx=0x7f07d002beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 -- 172.21.15.72:0/4294008978 shutdown_connections 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 --2- 172.21.15.72:0/4294008978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80d04a0 0x7f07d80ccfc0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 -- 172.21.15.72:0/4294008978 >> 172.21.15.72:0/4294008978 conn(0x7f07d809a9f0 msgr2=0x7f07d809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 -- 172.21.15.72:0/4294008978 shutdown_connections 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.403+0000 7f07defac640 1 -- 172.21.15.72:0/4294008978 wait complete. 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.404+0000 7f07defac640 1 Processor -- start 2024-03-06T07:31:37.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.404+0000 7f07defac640 1 -- start start 2024-03-06T07:31:37.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07defac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 0x7f07d8159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07defac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07d80d04a0 0x7f07d80b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07defac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07d815bb20 0x7f07d80cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07defac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80d08d0 con 0x7f07d80cfff0 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07defac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d8162500 con 0x7f07d80d04a0 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07defac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80d1980 con 0x7f07d815bb20 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07dcd48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 0x7f07d8159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07dcd48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 0x7f07d8159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45196/0 (socket says 172.21.15.72:45196) 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.405+0000 7f07dcd48640 1 -- 172.21.15.72:0/612489762 learned_addr learned my addr 172.21.15.72:0/612489762 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07d7fff640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07d80d04a0 0x7f07d80b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dd549640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07d815bb20 0x7f07d80cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dcd48640 1 -- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07d815bb20 msgr2=0x7f07d80cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dcd48640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07d815bb20 0x7f07d80cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dcd48640 1 -- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07d80d04a0 msgr2=0x7f07d80b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dcd48640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07d80d04a0 0x7f07d80b6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dcd48640 1 -- 172.21.15.72:0/612489762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d80b5ad0 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07d7fff640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07d80d04a0 0x7f07d80b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dd549640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07d815bb20 0x7f07d80cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07dcd48640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 0x7f07d8159c20 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00c460 tx=0x7f07cc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc016020 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc012870 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80b4a30 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc012c60 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.406+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80b52d0 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.407+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d80b4e20 con 0x7f07d80cfff0 2024-03-06T07:31:37.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.408+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f07cc019020 con 0x7f07d80cfff0 2024-03-06T07:31:37.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.408+0000 7f07d57fa640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07bc06b4d0 0x7f07bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:37.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.408+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f07cc089cc0 con 0x7f07d80cfff0 2024-03-06T07:31:37.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.408+0000 7f07d7fff640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07bc06b4d0 0x7f07bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:37.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.408+0000 7f07d7fff640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07bc06b4d0 0x7f07bc06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f07d81f7630 tx=0x7f07c4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:37.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.412+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f07cc020070 con 0x7f07d80cfff0 2024-03-06T07:31:37.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.624+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07d804f330 con 0x7f07d80cfff0 2024-03-06T07:31:37.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.624+0000 7f07d57fa640 1 -- 172.21.15.72:0/612489762 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f07cc058b00 con 0x7f07d80cfff0 2024-03-06T07:31:37.626 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:37.627 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:37.627 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07bc06b4d0 msgr2=0x7f07bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07bc06b4d0 0x7f07bc06d990 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f07d81f7630 tx=0x7f07c4009040 comp rx=0 tx=0).stop 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 msgr2=0x7f07d8159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 0x7f07d8159c20 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00c460 tx=0x7f07cc00c920 comp rx=0 tx=0).stop 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 shutdown_connections 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f07bc06b4d0 0x7f07bc06d990 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07d815bb20 0x7f07d80cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07d80d04a0 0x7f07d80b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 --2- 172.21.15.72:0/612489762 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07d80cfff0 0x7f07d8159c20 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:37.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 >> 172.21.15.72:0/612489762 conn(0x7f07d809a9f0 msgr2=0x7f07d806ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:37.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 shutdown_connections 2024-03-06T07:31:37.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:37.627+0000 7f07defac640 1 -- 172.21.15.72:0/612489762 wait complete. 2024-03-06T07:31:37.800 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:37.842 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:37.842 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:38.844 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:38.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.942+0000 7fced7488640 1 Processor -- start 2024-03-06T07:31:38.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.942+0000 7fced7488640 1 -- start start 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.943+0000 7fced7488640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 0x7fced0072730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.943+0000 7fced7488640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced015eb40 con 0x7fced0162030 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.943+0000 7fced7488640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced015ee20 con 0x7fced015e3a0 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.943+0000 7fced7488640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced0161ba0 con 0x7fced0158600 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.943+0000 7fced5a25640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fced0162030 0x7fced006afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41258/0 (socket says 172.21.15.72:41258) 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.943+0000 7fced5a25640 1 -- 172.21.15.72:0/4023558225 learned_addr learned my addr 172.21.15.72:0/4023558225 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:38.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896817654 0 0) 0x7fced015eb40 con 0x7fced0162030 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc003520 con 0x7fced0162030 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fced5224640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 0x7fced0072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3668429175 0 0) 0x7fcebc003520 con 0x7fced0162030 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 msgr2=0x7fced0072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 0x7fced0072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 >> v1:172.21.15.146:6789/0 conn(0x7fced015e3a0 legacy=0x7fced006d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced015f100 con 0x7fced0162030 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fcecc0019f0 con 0x7fced0162030 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 >> v1:172.21.15.72:6789/0 conn(0x7fced0162030 legacy=0x7fced006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 0x7fcebc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcebc003f60 0x7fcebc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc004d40 0x7fcebc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0161ba0 con 0x7fcebc003f60 2024-03-06T07:31:38.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc00bb50 con 0x7fcebc004d40 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc00bd50 con 0x7fced0158600 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fced4a23640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcebc003f60 0x7fcebc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fced5224640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 0x7fcebc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fced5a25640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc004d40 0x7fcebc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fced4a23640 1 -- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 msgr2=0x7fcebc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.944+0000 7fced4a23640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fced0158600 0x7fcebc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fced4a23640 1 -- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc004d40 msgr2=0x7fcebc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fced4a23640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc004d40 0x7fcebc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fced4a23640 1 -- 172.21.15.72:0/4023558225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc003520 con 0x7fcebc003f60 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fced5a25640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcebc004d40 0x7fcebc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fced4a23640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcebc003f60 0x7fcebc004800 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fcec0001750 tx=0x7fcebc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c71a368bf8d8ea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0001370 con 0x7fcebc003f60 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec00052d0 con 0x7fcebc003f60 2024-03-06T07:31:38.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.945+0000 7fcec7fff640 1 -- 172.21.15.72:0/4023558225 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0003830 con 0x7fcebc003f60 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced7488640 1 -- 172.21.15.72:0/4023558225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcebc003f60 msgr2=0x7fcebc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced7488640 1 --2- 172.21.15.72:0/4023558225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcebc003f60 0x7fcebc004800 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fcec0001750 tx=0x7fcebc005830 comp rx=0 tx=0).stop 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced5a25640 1 -- 172.21.15.72:0/4023558225 reap_dead start 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced7488640 1 -- 172.21.15.72:0/4023558225 shutdown_connections 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced7488640 1 -- 172.21.15.72:0/4023558225 >> 172.21.15.72:0/4023558225 conn(0x7fced006c0c0 msgr2=0x7fced006c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced7488640 1 -- 172.21.15.72:0/4023558225 shutdown_connections 2024-03-06T07:31:38.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.946+0000 7fced7488640 1 -- 172.21.15.72:0/4023558225 wait complete. 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.947+0000 7fced7488640 1 Processor -- start 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.947+0000 7fced7488640 1 -- start start 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced7488640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcebc003f60 0x7fced0146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced7488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 0x7fced00c5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced7488640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced015cbc0 0x7fced014a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced7488640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced006e080 con 0x7fced015b580 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced7488640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced006b6c0 con 0x7fced015cbc0 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced7488640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced015c730 con 0x7fcebc003f60 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 0x7fced00c5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 0x7fced00c5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45210/0 (socket says 172.21.15.72:45210) 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 -- 172.21.15.72:0/4089532847 learned_addr learned my addr 172.21.15.72:0/4089532847 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 -- 172.21.15.72:0/4089532847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcebc003f60 msgr2=0x7fced0146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcebc003f60 0x7fced0146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 -- 172.21.15.72:0/4089532847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced015cbc0 msgr2=0x7fced014a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced015cbc0 0x7fced014a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 -- 172.21.15.72:0/4089532847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced015bb20 con 0x7fced015b580 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.948+0000 7fced4a23640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 0x7fced00c5860 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fcec0008fd0 tx=0x7fcec0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.949+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0041070 con 0x7fced015b580 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.949+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec0005480 con 0x7fced015b580 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.949+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec0001370 con 0x7fced015b580 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.949+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced015be00 con 0x7fced015b580 2024-03-06T07:31:38.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.949+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced015c3c0 con 0x7fced015b580 2024-03-06T07:31:38.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.950+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced004f330 con 0x7fced015b580 2024-03-06T07:31:38.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.950+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fcec0044070 con 0x7fced015b580 2024-03-06T07:31:38.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.951+0000 7fcec5ffb640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fceb406b500 0x7fceb406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:38.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.951+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcec00abd90 con 0x7fced015b580 2024-03-06T07:31:38.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.951+0000 7fced5224640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fceb406b500 0x7fceb406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:38.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.951+0000 7fced5224640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fceb406b500 0x7fceb406d9c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcec8009ab0 tx=0x7fcec80095d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:38.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:38.959+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fcec007ac60 con 0x7fced015b580 2024-03-06T07:31:39.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.175+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fced00d06f0 con 0x7fced015b580 2024-03-06T07:31:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.176+0000 7fcec5ffb640 1 -- 172.21.15.72:0/4089532847 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fcec007a560 con 0x7fced015b580 2024-03-06T07:31:39.178 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:39.178 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:39.178 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:39.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fceb406b500 msgr2=0x7fceb406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fceb406b500 0x7fceb406d9c0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fcec8009ab0 tx=0x7fcec80095d0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 msgr2=0x7fced00c5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 0x7fced00c5860 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fcec0008fd0 tx=0x7fcec0001120 comp rx=0 tx=0).stop 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 shutdown_connections 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fceb406b500 0x7fceb406d9c0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fced015cbc0 0x7fced014a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fced015b580 0x7fced00c5860 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 --2- 172.21.15.72:0/4089532847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcebc003f60 0x7fced0146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 >> 172.21.15.72:0/4089532847 conn(0x7fced006c0c0 msgr2=0x7fced00af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 shutdown_connections 2024-03-06T07:31:39.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:39.178+0000 7fced7488640 1 -- 172.21.15.72:0/4089532847 wait complete. 2024-03-06T07:31:39.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:39.392 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:39.802 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:40.392 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:40.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.493+0000 7f99a15e2640 1 Processor -- start 2024-03-06T07:31:40.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.493+0000 7f99a15e2640 1 -- start start 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f99a15e2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f999c0b3060 0x7f999c15e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f99a15e2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999c15f580 con 0x7f999c15bf30 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f99a15e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999c15b450 con 0x7f999c13dc10 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f99a15e2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999c0cf3f0 con 0x7f999c0b3060 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f999c15bf30 0x7f999c15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41274/0 (socket says 172.21.15.72:41274) 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999b7fe640 1 -- 172.21.15.72:0/2716611342 learned_addr learned my addr 172.21.15.72:0/2716611342 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f9993fff640 1 --2- 172.21.15.72:0/2716611342 >> v2:172.21.15.146:3300/0 conn(0x7f999c0b3060 0x7f999c15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177389120 0 0) 0x7f999c15f580 con 0x7f999c15bf30 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988003520 con 0x7f999c15bf30 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440660002 0 0) 0x7f9988003520 con 0x7f999c15bf30 2024-03-06T07:31:40.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 >> v2:172.21.15.146:3300/0 conn(0x7f999c0b3060 msgr2=0x7f999c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 --2- 172.21.15.72:0/2716611342 >> v2:172.21.15.146:3300/0 conn(0x7f999c0b3060 0x7f999c15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 >> v1:172.21.15.146:6789/0 conn(0x7f999c13dc10 legacy=0x7f999c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c15b730 con 0x7f999c15bf30 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f99940019f0 con 0x7f999c15bf30 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 >> v1:172.21.15.72:6789/0 conn(0x7f999c15bf30 legacy=0x7f999c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f9988003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9988004800 0x7f9988004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99880052e0 0x7f998800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0cf3f0 con 0x7f99880039b0 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998800c1b0 con 0x7f9988004800 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.494+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998800c3b0 con 0x7f99880052e0 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f9993fff640 1 --2- 172.21.15.72:0/2716611342 >> v2:172.21.15.146:3300/0 conn(0x7f999c0b3060 0x7f999c15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:40.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999affd640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f9988003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999b7fe640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99880052e0 0x7f998800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f9993fff640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9988004800 0x7f9988004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999affd640 1 -- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99880052e0 msgr2=0x7f998800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999affd640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99880052e0 0x7f998800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999affd640 1 -- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9988004800 msgr2=0x7f9988004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999affd640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9988004800 0x7f9988004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999affd640 1 -- 172.21.15.72:0/2716611342 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9988003520 con 0x7f99880039b0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.495+0000 7f999b7fe640 1 -- 172.21.15.72:0/2716611342 reap_dead start 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f999affd640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f9988003dd0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f998c001720 tx=0x7f9988005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec41962e5ad2f96b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c001370 con 0x7f99880039b0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0052d0 con 0x7f99880039b0 2024-03-06T07:31:40.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f999a7fc640 1 -- 172.21.15.72:0/2716611342 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c003830 con 0x7f99880039b0 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 -- 172.21.15.72:0/2716611342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 msgr2=0x7f9988003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f9988003dd0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f998c001720 tx=0x7f9988005830 comp rx=0 tx=0).stop 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 -- 172.21.15.72:0/2716611342 shutdown_connections 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 --2- 172.21.15.72:0/2716611342 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f9988003dd0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 -- 172.21.15.72:0/2716611342 >> 172.21.15.72:0/2716611342 conn(0x7f999c0cf8a0 msgr2=0x7f999c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 -- 172.21.15.72:0/2716611342 shutdown_connections 2024-03-06T07:31:40.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.496+0000 7f99a15e2640 1 -- 172.21.15.72:0/2716611342 wait complete. 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.497+0000 7f99a15e2640 1 Processor -- start 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.497+0000 7f99a15e2640 1 -- start start 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f99a15e2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f999c0aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f99a15e2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f999c143d80 0x7f999c142440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f99a15e2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c1408a0 0x7f999c071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f99a15e2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b2950 con 0x7f99880039b0 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f99a15e2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c15b210 con 0x7f999c1408a0 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f99a15e2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f999c0b2ad0 con 0x7f999c143d80 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f999affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f999c0aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f999affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f999c0aee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45234/0 (socket says 172.21.15.72:45234) 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f999affd640 1 -- 172.21.15.72:0/1754407542 learned_addr learned my addr 172.21.15.72:0/1754407542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:40.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f9993fff640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f999c143d80 0x7f999c142440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.498+0000 7f999b7fe640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c1408a0 0x7f999c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999affd640 1 -- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f999c143d80 msgr2=0x7f999c142440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999affd640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f999c143d80 0x7f999c142440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999affd640 1 -- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c1408a0 msgr2=0x7f999c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999affd640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c1408a0 0x7f999c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999affd640 1 -- 172.21.15.72:0/1754407542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c142ce0 con 0x7f99880039b0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999b7fe640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c1408a0 0x7f999c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f999affd640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f999c0aee30 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f998c000c00 tx=0x7f998c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c041020 con 0x7f99880039b0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c005480 con 0x7f99880039b0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c142fc0 con 0x7f99880039b0 2024-03-06T07:31:40.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c001370 con 0x7f99880039b0 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.499+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c143580 con 0x7f99880039b0 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.500+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c04f330 con 0x7f99880039b0 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.500+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f998c044020 con 0x7f99880039b0 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.501+0000 7f99937fe640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f997c06b4d0 0x7f997c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.501+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f998c0abc20 con 0x7f99880039b0 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.501+0000 7f9993fff640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f997c06b4d0 0x7f997c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:40.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.501+0000 7f9993fff640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f997c06b4d0 0x7f997c06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f999c1443d0 tx=0x7f9984008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:40.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.505+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f998c04b030 con 0x7f99880039b0 2024-03-06T07:31:40.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.732+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999c06eb00 con 0x7f99880039b0 2024-03-06T07:31:40.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.732+0000 7f99937fe640 1 -- 172.21.15.72:0/1754407542 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f998c07a9d0 con 0x7f99880039b0 2024-03-06T07:31:40.734 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:40.734 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:40.735 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:40.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f997c06b4d0 msgr2=0x7f997c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f997c06b4d0 0x7f997c06d990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f999c1443d0 tx=0x7f9984008040 comp rx=0 tx=0).stop 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 msgr2=0x7f999c0aee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f999c0aee30 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f998c000c00 tx=0x7f998c001120 comp rx=0 tx=0).stop 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 shutdown_connections 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f997c06b4d0 0x7f997c06d990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f999c1408a0 0x7f999c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f999c143d80 0x7f999c142440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 --2- 172.21.15.72:0/1754407542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f99880039b0 0x7f999c0aee30 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 >> 172.21.15.72:0/1754407542 conn(0x7f999c0cf8a0 msgr2=0x7f999c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 shutdown_connections 2024-03-06T07:31:40.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:40.734+0000 7f99a15e2640 1 -- 172.21.15.72:0/1754407542 wait complete. 2024-03-06T07:31:40.948 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:40.949 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:41.805 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:41.950 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:42.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.049+0000 7fb7bf4eb640 1 Processor -- start 2024-03-06T07:31:42.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.050+0000 7fb7bf4eb640 1 -- start start 2024-03-06T07:31:42.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.050+0000 7fb7bf4eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 0x7fb7b80d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.050+0000 7fb7bf4eb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb7b80cd8c0 0x7fb7b80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.050+0000 7fb7bf4eb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b8161980 con 0x7fb7b80cfff0 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.050+0000 7fb7bf4eb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b8161c60 con 0x7fb7b80cd8c0 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.050+0000 7fb7bf4eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b8161f40 con 0x7fb7b815bb20 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bca86640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb7b80cfff0 0x7fb7b80ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41284/0 (socket says 172.21.15.72:41284) 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bca86640 1 -- 172.21.15.72:0/3960934385 learned_addr learned my addr 172.21.15.72:0/3960934385 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bda88640 1 --2- 172.21.15.72:0/3960934385 >> v2:172.21.15.146:3300/0 conn(0x7fb7b80cd8c0 0x7fb7b80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bd287640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 0x7fb7b80d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328575697 0 0) 0x7fb7b8161980 con 0x7fb7b80cfff0 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a4003520 con 0x7fb7b80cfff0 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526033522 0 0) 0x7fb7a4003520 con 0x7fb7b80cfff0 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 msgr2=0x7fb7b80d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 0x7fb7b80d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 >> v2:172.21.15.146:3300/0 conn(0x7fb7b80cd8c0 msgr2=0x7fb7b80d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 --2- 172.21.15.72:0/3960934385 >> v2:172.21.15.146:3300/0 conn(0x7fb7b80cd8c0 0x7fb7b80d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8162220 con 0x7fb7b80cfff0 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fb7a8001670 con 0x7fb7b80cfff0 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 >> v1:172.21.15.72:6789/0 conn(0x7fb7b80cfff0 legacy=0x7fb7b80ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 0x7fb7a40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a4003f30 0x7fb7a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b8161c60 con 0x7fb7a4004d40 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b8161f40 con 0x7fb7a4003f30 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a400ba10 con 0x7fb7b815bb20 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bda88640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bd287640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 0x7fb7a40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.051+0000 7fb7bca86640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a4003f30 0x7fb7a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.052+0000 7fb7bda88640 1 -- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 msgr2=0x7fb7a40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.052+0000 7fb7bda88640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815bb20 0x7fb7a40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.052+0000 7fb7bda88640 1 -- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a4003f30 msgr2=0x7fb7a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.052+0000 7fb7bda88640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a4003f30 0x7fb7a4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.052+0000 7fb7bda88640 1 -- 172.21.15.72:0/3960934385 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a4003520 con 0x7fb7a4004d40 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.052+0000 7fb7bda88640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7a40051c0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000e170 tx=0x7fb7a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa96028612cef65 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0039070 con 0x7fb7a4004d40 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b002cc20 con 0x7fb7a4004d40 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7affff640 1 -- 172.21.15.72:0/3960934385 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0009c10 con 0x7fb7a4004d40 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3960934385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 msgr2=0x7fb7a40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3960934385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7a40051c0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fb7b000e170 tx=0x7fb7a4005830 comp rx=0 tx=0).stop 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bda88640 1 -- 172.21.15.72:0/3960934385 reap_dead start 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3960934385 shutdown_connections 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3960934385 >> 172.21.15.72:0/3960934385 conn(0x7fb7b809a9f0 msgr2=0x7fb7b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3960934385 shutdown_connections 2024-03-06T07:31:42.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.053+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3960934385 wait complete. 2024-03-06T07:31:42.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.055+0000 7fb7bf4eb640 1 Processor -- start 2024-03-06T07:31:42.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.055+0000 7fb7bf4eb640 1 -- start start 2024-03-06T07:31:42.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bf4eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7b81570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bf4eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815f530 0x7fb7b8158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bf4eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7b815f9b0 0x7fb7b80aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bf4eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80d1b00 con 0x7fb7a4004d40 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bf4eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80d1980 con 0x7fb7b815f9b0 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bf4eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7b80d1df0 con 0x7fb7b815f530 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7b81570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7b81570d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45262/0 (socket says 172.21.15.72:45262) 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 -- 172.21.15.72:0/3993572785 learned_addr learned my addr 172.21.15.72:0/3993572785 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bca86640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815f530 0x7fb7b8158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bda88640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7b815f9b0 0x7fb7b80aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 -- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815f530 msgr2=0x7fb7b8158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815f530 0x7fb7b8158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 -- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7b815f9b0 msgr2=0x7fb7b80aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7b815f9b0 0x7fb7b80aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 -- 172.21.15.72:0/3993572785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b8159510 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.056+0000 7fb7bd287640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7b81570d0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4011980 tx=0x7fb7b4011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.057+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b400faf0 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.057+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b400fc90 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.057+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4014440 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.057+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b815c610 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.057+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b815cbd0 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.058+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b804f330 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.058+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb7b4012040 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.059+0000 7fb7adffb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb79c06b4d0 0x7fb79c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.059+0000 7fb7bca86640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb79c06b4d0 0x7fb79c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.059+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb7b408a9b0 con 0x7fb7a4004d40 2024-03-06T07:31:42.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.059+0000 7fb7bca86640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb79c06b4d0 0x7fb79c06d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8007fd0 tx=0x7fb7a8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:42.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.063+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb7b4019030 con 0x7fb7a4004d40 2024-03-06T07:31:42.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.274+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7b80b1820 con 0x7fb7a4004d40 2024-03-06T07:31:42.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.274+0000 7fb7adffb640 1 -- 172.21.15.72:0/3993572785 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fb7b40591a0 con 0x7fb7a4004d40 2024-03-06T07:31:42.277 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:42.277 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:42.277 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:42.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb79c06b4d0 msgr2=0x7fb79c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb79c06b4d0 0x7fb79c06d990 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8007fd0 tx=0x7fb7a8009040 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 msgr2=0x7fb7b81570d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7b81570d0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4011980 tx=0x7fb7b4011e40 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 shutdown_connections 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb79c06b4d0 0x7fb79c06d990 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7b815f9b0 0x7fb7b80aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7b815f530 0x7fb7b8158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 --2- 172.21.15.72:0/3993572785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7a4004d40 0x7fb7b81570d0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 >> 172.21.15.72:0/3993572785 conn(0x7fb7b809a9f0 msgr2=0x7fb7b809ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 shutdown_connections 2024-03-06T07:31:42.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:42.276+0000 7fb7bf4eb640 1 -- 172.21.15.72:0/3993572785 wait complete. 2024-03-06T07:31:42.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:42.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:43.491 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:43.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.591+0000 7fbf3d4c9640 1 Processor -- start 2024-03-06T07:31:43.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.591+0000 7fbf3d4c9640 1 -- start start 2024-03-06T07:31:43.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf3d4c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 0x7fbf380b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf3d4c9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380b3430 con 0x7fbf380d22b0 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf3d4c9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf380b3710 con 0x7fbf3815dfe0 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf3d4c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380b31e0 con 0x7fbf380d2710 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf2ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbf380d22b0 0x7fbf380b41f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36018/0 (socket says 172.21.15.72:36018) 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf2ffff640 1 -- 172.21.15.72:0/1200430537 learned_addr learned my addr 172.21.15.72:0/1200430537 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646507016 0 0) 0x7fbf380b3430 con 0x7fbf380d22b0 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf24003520 con 0x7fbf380d22b0 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.592+0000 7fbf377fe640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 0x7fbf380b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781630096 0 0) 0x7fbf24003520 con 0x7fbf380d22b0 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 msgr2=0x7fbf380b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 0x7fbf380b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 >> v1:172.21.15.146:6789/0 conn(0x7fbf3815dfe0 legacy=0x7fbf380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380b39f0 con 0x7fbf380d22b0 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fbf20001670 con 0x7fbf380d22b0 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 >> v1:172.21.15.72:6789/0 conn(0x7fbf380d22b0 legacy=0x7fbf380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 0x7fbf24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380b31e0 con 0x7fbf240039e0 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2400baa0 con 0x7fbf24004d40 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2400bca0 con 0x7fbf380d2710 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf2ffff640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 0x7fbf24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf377fe640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf36ffd640 1 -- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 msgr2=0x7fbf24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380d2710 0x7fbf24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf36ffd640 1 -- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf24004d40 msgr2=0x7fbf240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf24004d40 0x7fbf240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.593+0000 7fbf36ffd640 1 -- 172.21.15.72:0/1200430537 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24003520 con 0x7fbf240039e0 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.594+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fbf28001720 tx=0x7fbf24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1506656bd9bfec70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.594+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28001370 con 0x7fbf240039e0 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.594+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf280052d0 con 0x7fbf240039e0 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.594+0000 7fbf367fc640 1 -- 172.21.15.72:0/1200430537 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28003830 con 0x7fbf240039e0 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.594+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/1200430537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf240039e0 msgr2=0x7fbf24003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.594+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/1200430537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf240039e0 0x7fbf24003e20 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fbf28001720 tx=0x7fbf24005830 comp rx=0 tx=0).stop 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.595+0000 7fbf377fe640 1 -- 172.21.15.72:0/1200430537 reap_dead start 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.595+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/1200430537 shutdown_connections 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.595+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/1200430537 >> 172.21.15.72:0/1200430537 conn(0x7fbf380d3cd0 msgr2=0x7fbf380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.595+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/1200430537 shutdown_connections 2024-03-06T07:31:43.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.595+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/1200430537 wait complete. 2024-03-06T07:31:43.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 Processor -- start 2024-03-06T07:31:43.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 -- start start 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf240039e0 0x7fbf3815c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380b0c80 0x7fbf380b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 0x7fbf380b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380cfa00 con 0x7fbf380b57c0 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380b3d60 con 0x7fbf240039e0 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.596+0000 7fbf3d4c9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf380d1c70 con 0x7fbf380b0c80 2024-03-06T07:31:43.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 0x7fbf380b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 0x7fbf380b5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60608/0 (socket says 172.21.15.72:60608) 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 -- 172.21.15.72:0/271846807 learned_addr learned my addr 172.21.15.72:0/271846807 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf2ffff640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380b0c80 0x7fbf380b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf240039e0 0x7fbf3815c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 -- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380b0c80 msgr2=0x7fbf380b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380b0c80 0x7fbf380b1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 -- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf240039e0 msgr2=0x7fbf3815c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf240039e0 0x7fbf3815c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 -- 172.21.15.72:0/271846807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf380b6d90 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf377fe640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 0x7fbf380b5c70 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fbf3808c940 tx=0x7fbf300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30011e10 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3001b070 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf380b64f0 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30003990 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.597+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf380b6ab0 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.598+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3804f330 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.599+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbf30005d10 con 0x7fbf380b57c0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.599+0000 7fbf2f7fe640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf1806b4d0 0x7fbf1806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.599+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf1806b4d0 0x7fbf1806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:43.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.599+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf3008a560 con 0x7fbf380b57c0 2024-03-06T07:31:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.600+0000 7fbf36ffd640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf1806b4d0 0x7fbf1806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbf28033000 tx=0x7fbf28009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:43.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.604+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbf3001d310 con 0x7fbf380b57c0 2024-03-06T07:31:43.807 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:43.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.821+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf3815d470 con 0x7fbf380b57c0 2024-03-06T07:31:43.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.821+0000 7fbf2f7fe640 1 -- 172.21.15.72:0/271846807 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fbf30059430 con 0x7fbf380b57c0 2024-03-06T07:31:43.823 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:43.823 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:43.824 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf1806b4d0 msgr2=0x7fbf1806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf1806b4d0 0x7fbf1806d990 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbf28033000 tx=0x7fbf28009030 comp rx=0 tx=0).stop 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 msgr2=0x7fbf380b5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 0x7fbf380b5c70 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fbf3808c940 tx=0x7fbf300115a0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 shutdown_connections 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbf1806b4d0 0x7fbf1806d990 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf380b57c0 0x7fbf380b5c70 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf380b0c80 0x7fbf380b1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 --2- 172.21.15.72:0/271846807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf240039e0 0x7fbf3815c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 >> 172.21.15.72:0/271846807 conn(0x7fbf380d3cd0 msgr2=0x7fbf381603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 shutdown_connections 2024-03-06T07:31:43.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:43.823+0000 7fbf3d4c9640 1 -- 172.21.15.72:0/271846807 wait complete. 2024-03-06T07:31:44.037 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:44.037 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:45.038 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:45.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.137+0000 7fb719a4e640 1 Processor -- start 2024-03-06T07:31:45.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.137+0000 7fb719a4e640 1 -- start start 2024-03-06T07:31:45.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb719a4e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb7140d22b0 0x7fb7140d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb719a4e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7140b41f0 0x7fb7140b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb719a4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7140b3430 con 0x7fb71415dfe0 2024-03-06T07:31:45.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb719a4e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7140d2c50 con 0x7fb7140b41f0 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb719a4e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7140b31e0 con 0x7fb7140d22b0 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb7137fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7140b41f0 0x7fb7140b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb712ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb71415dfe0 0x7fb7140cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36032/0 (socket says 172.21.15.72:36032) 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb712ffd640 1 -- 172.21.15.72:0/28839811 learned_addr learned my addr 172.21.15.72:0/28839811 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb7137fe640 1 -- 172.21.15.72:0/28839811 >> v2:172.21.15.146:3300/0 conn(0x7fb7140d22b0 msgr2=0x7fb7140d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb7137fe640 1 --2- 172.21.15.72:0/28839811 >> v2:172.21.15.146:3300/0 conn(0x7fb7140d22b0 0x7fb7140d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb7137fe640 1 -- 172.21.15.72:0/28839811 >> v1:172.21.15.72:6789/0 conn(0x7fb71415dfe0 legacy=0x7fb7140cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.139+0000 7fb7137fe640 1 -- 172.21.15.72:0/28839811 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7140b3710 con 0x7fb7140b41f0 2024-03-06T07:31:45.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.140+0000 7fb7137fe640 1 --2- 172.21.15.72:0/28839811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7140b41f0 0x7fb7140b4680 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009d90 tx=0x7fb70c02bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd9ec945b0ebfac server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.140+0000 7fb7127fc640 1 -- 172.21.15.72:0/28839811 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c039070 con 0x7fb7140b41f0 2024-03-06T07:31:45.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.140+0000 7fb7127fc640 1 -- 172.21.15.72:0/28839811 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c02ca20 con 0x7fb7140b41f0 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.140+0000 7fb7127fc640 1 -- 172.21.15.72:0/28839811 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c02cdf0 con 0x7fb7140b41f0 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 -- 172.21.15.72:0/28839811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7140b41f0 msgr2=0x7fb7140b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 --2- 172.21.15.72:0/28839811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7140b41f0 0x7fb7140b4680 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009d90 tx=0x7fb70c02bcf0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 -- 172.21.15.72:0/28839811 shutdown_connections 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 --2- 172.21.15.72:0/28839811 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7140b41f0 0x7fb7140b4680 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 --2- 172.21.15.72:0/28839811 >> v2:172.21.15.146:3300/0 conn(0x7fb7140d22b0 0x7fb7140d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 -- 172.21.15.72:0/28839811 >> 172.21.15.72:0/28839811 conn(0x7fb7140d3cd0 msgr2=0x7fb7140d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 -- 172.21.15.72:0/28839811 shutdown_connections 2024-03-06T07:31:45.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.141+0000 7fb719a4e640 1 -- 172.21.15.72:0/28839811 wait complete. 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.142+0000 7fb719a4e640 1 Processor -- start 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.142+0000 7fb719a4e640 1 -- start start 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb719a4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7140b41f0 0x7fb71414f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb719a4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7140d22b0 0x7fb7140cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb719a4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 0x7fb71415f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb719a4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140cfa00 con 0x7fb71415dfe0 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb719a4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140b3d60 con 0x7fb7140d22b0 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb719a4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7140d1c70 con 0x7fb7140b41f0 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 0x7fb71415f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 0x7fb71415f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60618/0 (socket says 172.21.15.72:60618) 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 -- 172.21.15.72:0/2515751796 learned_addr learned my addr 172.21.15.72:0/2515751796 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:45.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb712ffd640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7140b41f0 0x7fb71414f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb70bfff640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7140d22b0 0x7fb7140cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 -- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7140b41f0 msgr2=0x7fb71414f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7140b41f0 0x7fb71414f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 -- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7140d22b0 msgr2=0x7fb7140cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7140d22b0 0x7fb7140cb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 -- 172.21.15.72:0/2515751796 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71415e610 con 0x7fb71415dfe0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb70bfff640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7140d22b0 0x7fb7140cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.143+0000 7fb7137fe640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 0x7fb71415f6b0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fb70c037040 tx=0x7fb70c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.144+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c044020 con 0x7fb71415dfe0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.144+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c034c10 con 0x7fb71415dfe0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.144+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71415e8f0 con 0x7fb71415dfe0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.144+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c039040 con 0x7fb71415dfe0 2024-03-06T07:31:45.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.144+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71415eeb0 con 0x7fb71415dfe0 2024-03-06T07:31:45.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.145+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71404f330 con 0x7fb71415dfe0 2024-03-06T07:31:45.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.146+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb70c047020 con 0x7fb71415dfe0 2024-03-06T07:31:45.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.146+0000 7fb710ff9640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb6f406b4d0 0x7fb6f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:45.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.146+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb70c0ac830 con 0x7fb71415dfe0 2024-03-06T07:31:45.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.146+0000 7fb712ffd640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb6f406b4d0 0x7fb6f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:45.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.146+0000 7fb712ffd640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb6f406b4d0 0x7fb6f406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb704000f60 tx=0x7fb704007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:45.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.150+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb70c04e070 con 0x7fb71415dfe0 2024-03-06T07:31:45.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.551+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7140ce9b0 con 0x7fb71415dfe0 2024-03-06T07:31:45.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.551+0000 7fb710ff9640 1 -- 172.21.15.72:0/2515751796 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fb70c07b730 con 0x7fb71415dfe0 2024-03-06T07:31:45.554 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:45.554 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:45.554 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:45.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb6f406b4d0 msgr2=0x7fb6f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb6f406b4d0 0x7fb6f406d990 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb704000f60 tx=0x7fb704007c80 comp rx=0 tx=0).stop 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 msgr2=0x7fb71415f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 0x7fb71415f6b0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fb70c037040 tx=0x7fb70c004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 shutdown_connections 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb6f406b4d0 0x7fb6f406d990 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb71415dfe0 0x7fb71415f6b0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7140d22b0 0x7fb7140cb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 --2- 172.21.15.72:0/2515751796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7140b41f0 0x7fb71414f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 >> 172.21.15.72:0/2515751796 conn(0x7fb7140d3cd0 msgr2=0x7fb7140b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 shutdown_connections 2024-03-06T07:31:45.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:45.554+0000 7fb719a4e640 1 -- 172.21.15.72:0/2515751796 wait complete. 2024-03-06T07:31:45.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:45.768 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:45.809 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:46.769 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:46.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.868+0000 7fe7a4a50640 1 Processor -- start 2024-03-06T07:31:46.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.868+0000 7fe7a4a50640 1 -- start start 2024-03-06T07:31:46.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe7a4a50640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe7a015ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe7a4a50640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a015f580 con 0x7fe7a013dc10 2024-03-06T07:31:46.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe7a4a50640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a015b450 con 0x7fe7a00b3060 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe7a4a50640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00cf3f0 con 0x7fe7a015bf30 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe7a013dc10 0x7fe7a00b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36044/0 (socket says 172.21.15.72:36044) 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79e59c640 1 -- 172.21.15.72:0/4229096781 learned_addr learned my addr 172.21.15.72:0/4229096781 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53022755 0 0) 0x7fe7a015f580 con 0x7fe7a013dc10 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe78c003520 con 0x7fe7a013dc10 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79ed9d640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe7a015ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1779998986 0 0) 0x7fe78c003520 con 0x7fe7a013dc10 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 msgr2=0x7fe7a015ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe7a015ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 >> v1:172.21.15.146:6789/0 conn(0x7fe7a00b3060 legacy=0x7fe7a015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a015b730 con 0x7fe7a013dc10 2024-03-06T07:31:46.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe790001670 con 0x7fe7a013dc10 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79ed9d640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe7a015ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 >> v1:172.21.15.72:6789/0 conn(0x7fe7a013dc10 legacy=0x7fe7a00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe78c0039e0 0x7fe78c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe78c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe78c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a00cf3f0 con 0x7fe78c004800 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c0054e0 con 0x7fe78c0039e0 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.869+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78c00bf00 con 0x7fe7a015bf30 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe78c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79e59c640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe78c0039e0 0x7fe78c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79ed9d640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe78c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 -- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 msgr2=0x7fe78c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a015bf30 0x7fe78c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 -- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe78c0039e0 msgr2=0x7fe78c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe78c0039e0 0x7fe78c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 -- 172.21.15.72:0/4229096781 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe78c003520 con 0x7fe78c004800 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.870+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe78c004c80 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fe788001750 tx=0x7fe78c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc11a587650ab0b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788001370 con 0x7fe78c004800 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7880052d0 con 0x7fe78c004800 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe79d59a640 1 -- 172.21.15.72:0/4229096781 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788003830 con 0x7fe78c004800 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe7a4a50640 1 -- 172.21.15.72:0/4229096781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 msgr2=0x7fe78c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/4229096781 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe78c004c80 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fe788001750 tx=0x7fe78c005830 comp rx=0 tx=0).stop 2024-03-06T07:31:46.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe79ed9d640 1 -- 172.21.15.72:0/4229096781 reap_dead start 2024-03-06T07:31:46.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe7a4a50640 1 -- 172.21.15.72:0/4229096781 shutdown_connections 2024-03-06T07:31:46.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe7a4a50640 1 -- 172.21.15.72:0/4229096781 >> 172.21.15.72:0/4229096781 conn(0x7fe7a00cf8a0 msgr2=0x7fe7a00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:46.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe7a4a50640 1 -- 172.21.15.72:0/4229096781 shutdown_connections 2024-03-06T07:31:46.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.871+0000 7fe7a4a50640 1 -- 172.21.15.72:0/4229096781 wait complete. 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 Processor -- start 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 -- start start 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe7a00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a00adda0 0x7fe7a0156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7a01567b0 0x7fe7a0156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a015ef60 con 0x7fe78c004800 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a015b210 con 0x7fe7a01567b0 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe7a4a50640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a015f0e0 con 0x7fe7a00adda0 2024-03-06T07:31:46.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe79e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe7a00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe79e59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe7a00ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60636/0 (socket says 172.21.15.72:60636) 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe79e59c640 1 -- 172.21.15.72:0/3151224045 learned_addr learned my addr 172.21.15.72:0/3151224045 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a00adda0 0x7fe7a0156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.873+0000 7fe79ed9d640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7a01567b0 0x7fe7a0156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe79e59c640 1 -- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a00adda0 msgr2=0x7fe7a0156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe79e59c640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a00adda0 0x7fe7a0156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe79e59c640 1 -- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7a01567b0 msgr2=0x7fe7a0156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe79e59c640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7a01567b0 0x7fe7a0156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:46.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe79e59c640 1 -- 172.21.15.72:0/3151224045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a01f3e40 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe79e59c640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe7a00ad860 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fe790007370 tx=0x7fe790007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79001a070 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe79000f8c0 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a0157210 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe79000fcd0 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.874+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a013e3a0 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.875+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a0157730 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.876+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe79001d070 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.876+0000 7fe796ffd640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe78006b470 0x7fe78006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.876+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe790089b30 con 0x7fe78c004800 2024-03-06T07:31:46.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.876+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe78006b470 0x7fe78006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:46.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.876+0000 7fe79dd9b640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe78006b470 0x7fe78006d930 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe788033000 tx=0x7fe788009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:46.883 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:46.880+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe7a0157730 con 0x7fe78c004800 2024-03-06T07:31:47.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.094+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7a004f330 con 0x7fe78c004800 2024-03-06T07:31:47.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.095+0000 7fe796ffd640 1 -- 172.21.15.72:0/3151224045 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe790058320 con 0x7fe78c004800 2024-03-06T07:31:47.097 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:47.097 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:47.097 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.096+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe78006b470 msgr2=0x7fe78006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.096+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe78006b470 0x7fe78006d930 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fe788033000 tx=0x7fe788009030 comp rx=0 tx=0).stop 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.096+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 msgr2=0x7fe7a00ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.096+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe7a00ad860 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7fe790007370 tx=0x7fe790007830 comp rx=0 tx=0).stop 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 shutdown_connections 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe78006b470 0x7fe78006d930 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7a01567b0 0x7fe7a0156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7a00adda0 0x7fe7a0156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 --2- 172.21.15.72:0/3151224045 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe78c004800 0x7fe7a00ad860 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:47.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 >> 172.21.15.72:0/3151224045 conn(0x7fe7a00cf8a0 msgr2=0x7fe7a0070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:47.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 shutdown_connections 2024-03-06T07:31:47.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:47.097+0000 7fe7a4a50640 1 -- 172.21.15.72:0/3151224045 wait complete. 2024-03-06T07:31:47.311 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:47.311 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:47.811 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:48.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.412+0000 7f223e30c640 1 Processor -- start 2024-03-06T07:31:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.412+0000 7f223e30c640 1 -- start start 2024-03-06T07:31:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f223e30c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22380d2710 0x7f22380b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f223e30c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22380b3430 con 0x7f223815dfe0 2024-03-06T07:31:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f223e30c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22380b3710 con 0x7f22380d22b0 2024-03-06T07:31:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f223e30c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22380b31e0 con 0x7f22380d2710 2024-03-06T07:31:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f2237fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f223815dfe0 0x7f22380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36056/0 (socket says 172.21.15.72:36056) 2024-03-06T07:31:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f2237fff640 1 -- 172.21.15.72:0/290468487 learned_addr learned my addr 172.21.15.72:0/290468487 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f223c8a9640 1 --2- 172.21.15.72:0/290468487 >> v2:172.21.15.146:3300/0 conn(0x7f22380d2710 0x7f22380b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.413+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952717377 0 0) 0x7f22380b3430 con 0x7f223815dfe0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2220003520 con 0x7f223815dfe0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824816112 0 0) 0x7f22380b3710 con 0x7f22380d22b0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22380b3430 con 0x7f22380d22b0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 146067948 0 0) 0x7f2220003520 con 0x7f223815dfe0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 >> v2:172.21.15.146:3300/0 conn(0x7f22380d2710 msgr2=0x7f22380b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 --2- 172.21.15.72:0/290468487 >> v2:172.21.15.146:3300/0 conn(0x7f22380d2710 0x7f22380b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 >> v1:172.21.15.146:6789/0 conn(0x7f22380d22b0 legacy=0x7f22380b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22380b39f0 con 0x7f223815dfe0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f2228001670 con 0x7f223815dfe0 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 >> v1:172.21.15.72:6789/0 conn(0x7f223815dfe0 legacy=0x7f22380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2220003b70 0x7f2220003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2220004800 0x7f2220004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f222000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22380b31e0 con 0x7f22200051c0 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f222000c060 con 0x7f2220004800 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222000c260 con 0x7f2220003b70 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f223c8a9640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f222000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2237fff640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2220003b70 0x7f2220003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f22377fe640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2220004800 0x7f2220004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f223c8a9640 1 -- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2220003b70 msgr2=0x7f2220003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f223c8a9640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2220003b70 0x7f2220003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f223c8a9640 1 -- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2220004800 msgr2=0x7f2220004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f223c8a9640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2220004800 0x7f2220004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f223c8a9640 1 -- 172.21.15.72:0/290468487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2220003520 con 0x7f22200051c0 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f2237fff640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2220003b70 0x7f2220003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.414+0000 7f22377fe640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2220004800 0x7f2220004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:48.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.415+0000 7f223c8a9640 1 -- 172.21.15.72:0/290468487 reap_dead start 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.415+0000 7f223c8a9640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f222000b9c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f2230009fc0 tx=0x7f2220005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b450051714658e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.415+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230009c10 con 0x7f22200051c0 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.415+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223002cca0 con 0x7f22200051c0 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.415+0000 7f2236ffd640 1 -- 172.21.15.72:0/290468487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2230003bd0 con 0x7f22200051c0 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 -- 172.21.15.72:0/290468487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 msgr2=0x7f222000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f222000b9c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f2230009fc0 tx=0x7f2220005830 comp rx=0 tx=0).stop 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 -- 172.21.15.72:0/290468487 shutdown_connections 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 --2- 172.21.15.72:0/290468487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f222000b9c0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 -- 172.21.15.72:0/290468487 >> 172.21.15.72:0/290468487 conn(0x7f22380d3cd0 msgr2=0x7f22380d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 -- 172.21.15.72:0/290468487 shutdown_connections 2024-03-06T07:31:48.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.416+0000 7f223e30c640 1 -- 172.21.15.72:0/290468487 wait complete. 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.417+0000 7f223e30c640 1 Processor -- start 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.417+0000 7f223e30c640 1 -- start start 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f22381f3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22380b57c0 0x7f22380b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f223815f2f0 0x7f223815f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22380d1c70 con 0x7f22200051c0 2024-03-06T07:31:48.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22380b3d60 con 0x7f22380b57c0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22380d1df0 con 0x7f223815f2f0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f22381f3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f22381f3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60640/0 (socket says 172.21.15.72:60640) 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 -- 172.21.15.72:0/1995253282 learned_addr learned my addr 172.21.15.72:0/1995253282 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223c8a9640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f223815f2f0 0x7f223815f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f22377fe640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22380b57c0 0x7f22380b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 -- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f223815f2f0 msgr2=0x7f223815f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f223815f2f0 0x7f223815f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 -- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22380b57c0 msgr2=0x7f22380b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22380b57c0 0x7f22380b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 -- 172.21.15.72:0/1995253282 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238160020 con 0x7f22200051c0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2237fff640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f22381f3680 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f222800e700 tx=0x7f222800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222800c6c0 con 0x7f22200051c0 2024-03-06T07:31:48.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2228018070 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22380b7870 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222800ca60 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.418+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22380b7e30 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.419+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223804f330 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.420+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f222800cc00 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.420+0000 7f2234ff9640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f222406b500 0x7f222406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.420+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2228089ce0 con 0x7f22200051c0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.420+0000 7f22377fe640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f222406b500 0x7f222406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:48.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.421+0000 7f22377fe640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f222406b500 0x7f222406d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f222c007fd0 tx=0x7f222c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:48.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.425+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2228058b20 con 0x7f22200051c0 2024-03-06T07:31:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.639+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22380ce0d0 con 0x7f22200051c0 2024-03-06T07:31:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.640+0000 7f2234ff9640 1 -- 172.21.15.72:0/1995253282 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f2228058420 con 0x7f22200051c0 2024-03-06T07:31:48.642 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:48.642 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:48.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f222406b500 msgr2=0x7f222406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f222406b500 0x7f222406d9c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f222c007fd0 tx=0x7f222c009040 comp rx=0 tx=0).stop 2024-03-06T07:31:48.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 msgr2=0x7f22381f3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:48.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f22381f3680 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f222800e700 tx=0x7f222800ebc0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 shutdown_connections 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f222406b500 0x7f222406d9c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f223815f2f0 0x7f223815f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22380b57c0 0x7f22380b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 --2- 172.21.15.72:0/1995253282 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f22200051c0 0x7f22381f3680 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 >> 172.21.15.72:0/1995253282 conn(0x7f22380d3cd0 msgr2=0x7f22381e8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 shutdown_connections 2024-03-06T07:31:48.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:48.642+0000 7f223e30c640 1 -- 172.21.15.72:0/1995253282 wait complete. 2024-03-06T07:31:48.856 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:48.856 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:49.812 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:49.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.956+0000 7fc32b93a640 1 Processor -- start 2024-03-06T07:31:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc32b93a640 1 -- start start 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc32b93a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3240d22b0 0x7fc3240d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc32b93a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3240b41f0 0x7fc3240b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc32b93a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3240b3430 con 0x7fc32415dfe0 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc32b93a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3240d2c50 con 0x7fc3240d22b0 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc32b93a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240b31e0 con 0x7fc3240b41f0 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc328ed5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3240d22b0 0x7fc3240d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc3296d6640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc32415dfe0 0x7fc3240cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36072/0 (socket says 172.21.15.72:36072) 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.957+0000 7fc3296d6640 1 -- 172.21.15.72:0/3943401530 learned_addr learned my addr 172.21.15.72:0/3943401530 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc329ed7640 1 --2- 172.21.15.72:0/3943401530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3240b41f0 0x7fc3240b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc328ed5640 1 -- 172.21.15.72:0/3943401530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3240b41f0 msgr2=0x7fc3240b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc328ed5640 1 --2- 172.21.15.72:0/3943401530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3240b41f0 0x7fc3240b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc328ed5640 1 -- 172.21.15.72:0/3943401530 >> v1:172.21.15.72:6789/0 conn(0x7fc32415dfe0 legacy=0x7fc3240cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc31bfff640 1 -- 172.21.15.72:0/3943401530 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211477162 0 0) 0x7fc3240b3430 con 0x7fc32415dfe0 2024-03-06T07:31:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc328ed5640 1 -- 172.21.15.72:0/3943401530 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3240b3710 con 0x7fc3240d22b0 2024-03-06T07:31:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.958+0000 7fc328ed5640 1 --2- 172.21.15.72:0/3943401530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240d22b0 0x7fc3240d2710 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc3140097a0 tx=0x7fc31402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c047323d834d53ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc31bfff640 1 -- 172.21.15.72:0/3943401530 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314035020 con 0x7fc3240d22b0 2024-03-06T07:31:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc31bfff640 1 -- 172.21.15.72:0/3943401530 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc314031830 con 0x7fc3240d22b0 2024-03-06T07:31:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc31bfff640 1 -- 172.21.15.72:0/3943401530 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc314031c00 con 0x7fc3240d22b0 2024-03-06T07:31:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 -- 172.21.15.72:0/3943401530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240d22b0 msgr2=0x7fc3240d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 --2- 172.21.15.72:0/3943401530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240d22b0 0x7fc3240d2710 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc3140097a0 tx=0x7fc31402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 -- 172.21.15.72:0/3943401530 shutdown_connections 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 --2- 172.21.15.72:0/3943401530 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc3240b41f0 0x7fc3240b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 --2- 172.21.15.72:0/3943401530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240d22b0 0x7fc3240d2710 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 -- 172.21.15.72:0/3943401530 >> 172.21.15.72:0/3943401530 conn(0x7fc3240d3cd0 msgr2=0x7fc3240d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 -- 172.21.15.72:0/3943401530 shutdown_connections 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.959+0000 7fc32b93a640 1 -- 172.21.15.72:0/3943401530 wait complete. 2024-03-06T07:31:49.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.961+0000 7fc32b93a640 1 Processor -- start 2024-03-06T07:31:49.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.961+0000 7fc32b93a640 1 -- start start 2024-03-06T07:31:49.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc32b93a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 0x7fc3240cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc32b93a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3240d22b0 0x7fc3240b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc32b93a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc32415dfe0 0x7fc3240b7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc32b93a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240cfa00 con 0x7fc3240b41f0 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc32b93a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240b3d60 con 0x7fc3240d22b0 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc32b93a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3240d1c70 con 0x7fc32415dfe0 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 0x7fc3240cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 0x7fc3240cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60654/0 (socket says 172.21.15.72:60654) 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 -- 172.21.15.72:0/2857870667 learned_addr learned my addr 172.21.15.72:0/2857870667 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 -- 172.21.15.72:0/2857870667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc32415dfe0 msgr2=0x7fc3240b7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc32415dfe0 0x7fc3240b7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 -- 172.21.15.72:0/2857870667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3240d22b0 msgr2=0x7fc3240b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3240d22b0 0x7fc3240b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.962+0000 7fc3296d6640 1 -- 172.21.15.72:0/2857870667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32415cf10 con 0x7fc3240b41f0 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.963+0000 7fc3296d6640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 0x7fc3240cd870 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c460 tx=0x7fc32000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.963+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320018070 con 0x7fc3240b41f0 2024-03-06T07:31:49.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.963+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc320007500 con 0x7fc3240b41f0 2024-03-06T07:31:49.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.963+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32409ddd0 con 0x7fc3240b41f0 2024-03-06T07:31:49.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.963+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320004a20 con 0x7fc3240b41f0 2024-03-06T07:31:49.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.963+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3240cb8d0 con 0x7fc3240b41f0 2024-03-06T07:31:49.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.964+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3240cbad0 con 0x7fc3240b41f0 2024-03-06T07:31:49.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.964+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc32001b070 con 0x7fc3240b41f0 2024-03-06T07:31:49.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.964+0000 7fc319ffb640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30806b4d0 0x7fc30806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:49.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.964+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc320089520 con 0x7fc3240b41f0 2024-03-06T07:31:49.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.964+0000 7fc328ed5640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30806b4d0 0x7fc30806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:49.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.965+0000 7fc328ed5640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30806b4d0 0x7fc30806d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc314036000 tx=0x7fc314030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:49.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:49.973+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc3240cbad0 con 0x7fc3240b41f0 2024-03-06T07:31:50.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.190+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3240cb270 con 0x7fc3240b41f0 2024-03-06T07:31:50.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.190+0000 7fc319ffb640 1 -- 172.21.15.72:0/2857870667 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fc32409ddd0 con 0x7fc3240b41f0 2024-03-06T07:31:50.193 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:50.193 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:50.193 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.192+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30806b4d0 msgr2=0x7fc30806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.192+0000 7fc32b93a640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30806b4d0 0x7fc30806d990 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc314036000 tx=0x7fc314030000 comp rx=0 tx=0).stop 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.192+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 msgr2=0x7fc3240cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.192+0000 7fc32b93a640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 0x7fc3240cd870 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc32000c460 tx=0x7fc32000c920 comp rx=0 tx=0).stop 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 shutdown_connections 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc30806b4d0 0x7fc30806d990 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc32415dfe0 0x7fc3240b7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc3240d22b0 0x7fc3240b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 --2- 172.21.15.72:0/2857870667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3240b41f0 0x7fc3240cd870 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:50.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 >> 172.21.15.72:0/2857870667 conn(0x7fc3240d3cd0 msgr2=0x7fc3240b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:50.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 shutdown_connections 2024-03-06T07:31:50.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:50.193+0000 7fc32b93a640 1 -- 172.21.15.72:0/2857870667 wait complete. 2024-03-06T07:31:50.406 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:50.407 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:51.407 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:51.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.506+0000 7ff210f8a640 1 Processor -- start 2024-03-06T07:31:51.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.506+0000 7ff210f8a640 1 -- start start 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff210f8a640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff20c15dfe0 0x7ff20c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff210f8a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff20c0d29c0 0x7ff20c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff210f8a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c0b3430 con 0x7ff20c0cf830 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff210f8a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c0b31e0 con 0x7ff20c0d29c0 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff210f8a640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c0b3d60 con 0x7ff20c15dfe0 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff20ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff20c0d29c0 0x7ff20c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff201d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff20c0cf830 0x7ff20c0d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36086/0 (socket says 172.21.15.72:36086) 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.507+0000 7ff201d9b640 1 -- 172.21.15.72:0/2704129882 learned_addr learned my addr 172.21.15.72:0/2704129882 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20a59c640 1 --2- 172.21.15.72:0/2704129882 >> v2:172.21.15.146:3300/0 conn(0x7ff20c15dfe0 0x7ff20c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20ad9d640 1 -- 172.21.15.72:0/2704129882 >> v2:172.21.15.146:3300/0 conn(0x7ff20c15dfe0 msgr2=0x7ff20c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20ad9d640 1 --2- 172.21.15.72:0/2704129882 >> v2:172.21.15.146:3300/0 conn(0x7ff20c15dfe0 0x7ff20c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20ad9d640 1 -- 172.21.15.72:0/2704129882 >> v1:172.21.15.72:6789/0 conn(0x7ff20c0cf830 legacy=0x7ff20c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff209d9b640 1 -- 172.21.15.72:0/2704129882 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532876141 0 0) 0x7ff20c0b3430 con 0x7ff20c0cf830 2024-03-06T07:31:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20ad9d640 1 -- 172.21.15.72:0/2704129882 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c0b3710 con 0x7ff20c0d29c0 2024-03-06T07:31:51.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20a59c640 1 --2- 172.21.15.72:0/2704129882 >> v2:172.21.15.146:3300/0 conn(0x7ff20c15dfe0 0x7ff20c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff20ad9d640 1 --2- 172.21.15.72:0/2704129882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0d29c0 0x7ff20c0b41f0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7ff20c067990 tx=0x7ff20402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cd60284d9550fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff209d9b640 1 -- 172.21.15.72:0/2704129882 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204039070 con 0x7ff20c0d29c0 2024-03-06T07:31:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff209d9b640 1 -- 172.21.15.72:0/2704129882 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20402cc80 con 0x7ff20c0d29c0 2024-03-06T07:31:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.508+0000 7ff209d9b640 1 -- 172.21.15.72:0/2704129882 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204034660 con 0x7ff20c0d29c0 2024-03-06T07:31:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 -- 172.21.15.72:0/2704129882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0d29c0 msgr2=0x7ff20c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 --2- 172.21.15.72:0/2704129882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0d29c0 0x7ff20c0b41f0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7ff20c067990 tx=0x7ff20402bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 -- 172.21.15.72:0/2704129882 shutdown_connections 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 --2- 172.21.15.72:0/2704129882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0d29c0 0x7ff20c0b41f0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 --2- 172.21.15.72:0/2704129882 >> v2:172.21.15.146:3300/0 conn(0x7ff20c15dfe0 0x7ff20c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 -- 172.21.15.72:0/2704129882 >> 172.21.15.72:0/2704129882 conn(0x7ff20c0d3cd0 msgr2=0x7ff20c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 -- 172.21.15.72:0/2704129882 shutdown_connections 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.509+0000 7ff210f8a640 1 -- 172.21.15.72:0/2704129882 wait complete. 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.510+0000 7ff210f8a640 1 Processor -- start 2024-03-06T07:31:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.510+0000 7ff210f8a640 1 -- start start 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff210f8a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 0x7ff20c151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff210f8a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff20c0d29c0 0x7ff20c160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff210f8a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff20c15dfe0 0x7ff20c160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff210f8a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0d1df0 con 0x7ff20c0cf830 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff210f8a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0d1c70 con 0x7ff20c15dfe0 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff210f8a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15e3c0 con 0x7ff20c0d29c0 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff20a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 0x7ff20c151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff20a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 0x7ff20c151ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60672/0 (socket says 172.21.15.72:60672) 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff20a59c640 1 -- 172.21.15.72:0/3556537935 learned_addr learned my addr 172.21.15.72:0/3556537935 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.511+0000 7ff20a59c640 1 -- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff20c0d29c0 msgr2=0x7ff20c160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff20a59c640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff20c0d29c0 0x7ff20c160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff20a59c640 1 -- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff20c15dfe0 msgr2=0x7ff20c160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff20ad9d640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff20c15dfe0 0x7ff20c160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff20a59c640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff20c15dfe0 0x7ff20c160670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff20a59c640 1 -- 172.21.15.72:0/3556537935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c1525b0 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff20a59c640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 0x7ff20c151ab0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00c7f0 tx=0x7ff1fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc015070 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc007820 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c160ca0 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc007bd0 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.512+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c161260 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.513+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c04f330 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.513+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff1fc018070 con 0x7ff20c0cf830 2024-03-06T07:31:51.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.514+0000 7ff2037fe640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:51.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.514+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff1fc08a340 con 0x7ff20c0cf830 2024-03-06T07:31:51.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.514+0000 7ff201d9b640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:51.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.514+0000 7ff201d9b640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4007fd0 tx=0x7ff1f4007920 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:51.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.518+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff1fc059180 con 0x7ff20c0cf830 2024-03-06T07:31:51.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.727+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20c0b4b30 con 0x7ff20c0cf830 2024-03-06T07:31:51.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.728+0000 7ff2037fe640 1 -- 172.21.15.72:0/3556537935 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7ff1fc058a80 con 0x7ff20c0cf830 2024-03-06T07:31:51.730 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:51.730 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:51.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1ec06b4d0 msgr2=0x7ff1ec06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4007fd0 tx=0x7ff1f4007920 comp rx=0 tx=0).stop 2024-03-06T07:31:51.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 msgr2=0x7ff20c151ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 0x7ff20c151ab0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc00c7f0 tx=0x7ff1fc00ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 shutdown_connections 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff1ec06b4d0 0x7ff1ec06d990 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff20c15dfe0 0x7ff20c160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff20c0d29c0 0x7ff20c160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 --2- 172.21.15.72:0/3556537935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff20c0cf830 0x7ff20c151ab0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 >> 172.21.15.72:0/3556537935 conn(0x7ff20c0d3cd0 msgr2=0x7ff20c06c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 shutdown_connections 2024-03-06T07:31:51.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:51.730+0000 7ff210f8a640 1 -- 172.21.15.72:0/3556537935 wait complete. 2024-03-06T07:31:51.814 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:51.944 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:51.945 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:52.217 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:52.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:31:52.945 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:53.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 Processor -- start 2024-03-06T07:31:53.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 -- start start 2024-03-06T07:31:53.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd2c0d22b0 0x7efd2c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efd2c0b41f0 0x7efd2c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd2c0b3430 con 0x7efd2c15dfe0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd2c0d2c50 con 0x7efd2c0d22b0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd3311e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd2c0b31e0 con 0x7efd2c0b41f0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.045+0000 7efd2bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd2c0d22b0 0x7efd2c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd2bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efd2c0d22b0 0x7efd2c0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57834/0 (socket says 172.21.15.72:57834) 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd2bfff640 1 -- 172.21.15.72:0/145512083 learned_addr learned my addr 172.21.15.72:0/145512083 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd316bb640 1 --2- 172.21.15.72:0/145512083 >> v2:172.21.15.146:3300/0 conn(0x7efd2c0b41f0 0x7efd2c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd2bfff640 1 -- 172.21.15.72:0/145512083 >> v2:172.21.15.146:3300/0 conn(0x7efd2c0b41f0 msgr2=0x7efd2c0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd2bfff640 1 --2- 172.21.15.72:0/145512083 >> v2:172.21.15.146:3300/0 conn(0x7efd2c0b41f0 0x7efd2c0b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd2bfff640 1 -- 172.21.15.72:0/145512083 >> v1:172.21.15.146:6789/0 conn(0x7efd2c15dfe0 legacy=0x7efd2c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.046+0000 7efd2bfff640 1 -- 172.21.15.72:0/145512083 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c0b3710 con 0x7efd2c0d22b0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd2bfff640 1 --2- 172.21.15.72:0/145512083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c0d2710 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7efd18009870 tx=0x7efd1802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f474ade46221e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd2b7fe640 1 -- 172.21.15.72:0/145512083 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18034070 con 0x7efd2c0d22b0 2024-03-06T07:31:53.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd2b7fe640 1 -- 172.21.15.72:0/145512083 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd18030860 con 0x7efd2c0d22b0 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd2b7fe640 1 -- 172.21.15.72:0/145512083 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18030c30 con 0x7efd2c0d22b0 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 -- 172.21.15.72:0/145512083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 msgr2=0x7efd2c0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 --2- 172.21.15.72:0/145512083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c0d2710 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7efd18009870 tx=0x7efd1802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 -- 172.21.15.72:0/145512083 shutdown_connections 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 --2- 172.21.15.72:0/145512083 >> v2:172.21.15.146:3300/0 conn(0x7efd2c0b41f0 0x7efd2c0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 --2- 172.21.15.72:0/145512083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c0d2710 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 -- 172.21.15.72:0/145512083 >> 172.21.15.72:0/145512083 conn(0x7efd2c0d3cd0 msgr2=0x7efd2c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 -- 172.21.15.72:0/145512083 shutdown_connections 2024-03-06T07:31:53.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.047+0000 7efd3311e640 1 -- 172.21.15.72:0/145512083 wait complete. 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.048+0000 7efd3311e640 1 Processor -- start 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 -- start start 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd2c0b41f0 0x7efd2c1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c1f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd2c15dfe0 0x7efd2c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c0cfa00 con 0x7efd2c0d22b0 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c0b3d60 con 0x7efd2c0b41f0 2024-03-06T07:31:53.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd3311e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c0d1c70 con 0x7efd2c15dfe0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd2bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c1f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd2bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c1f29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57850/0 (socket says 172.21.15.72:57850) 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.049+0000 7efd2bfff640 1 -- 172.21.15.72:0/1771359374 learned_addr learned my addr 172.21.15.72:0/1771359374 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd30eba640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd2c0b41f0 0x7efd2c1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd316bb640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd2c15dfe0 0x7efd2c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd2bfff640 1 -- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd2c15dfe0 msgr2=0x7efd2c1f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd2bfff640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd2c15dfe0 0x7efd2c1f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd2bfff640 1 -- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd2c0b41f0 msgr2=0x7efd2c1599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd2bfff640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd2c0b41f0 0x7efd2c1599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd2bfff640 1 -- 172.21.15.72:0/1771359374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c15a2f0 con 0x7efd2c0d22b0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd316bb640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd2c15dfe0 0x7efd2c1f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd2bfff640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c1f29f0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7efd18000c00 tx=0x7efd18004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18048070 con 0x7efd2c0d22b0 2024-03-06T07:31:53.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd18030660 con 0x7efd2c0d22b0 2024-03-06T07:31:53.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2c1581a0 con 0x7efd2c0d22b0 2024-03-06T07:31:53.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18034040 con 0x7efd2c0d22b0 2024-03-06T07:31:53.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.050+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2c15a590 con 0x7efd2c0d22b0 2024-03-06T07:31:53.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.051+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd2c04f330 con 0x7efd2c0d22b0 2024-03-06T07:31:53.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.052+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7efd1804b070 con 0x7efd2c0d22b0 2024-03-06T07:31:53.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.052+0000 7efd297fa640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd1006b4d0 0x7efd1006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:53.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.052+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7efd180ac100 con 0x7efd2c0d22b0 2024-03-06T07:31:53.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.053+0000 7efd30eba640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd1006b4d0 0x7efd1006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:53.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.053+0000 7efd30eba640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd1006b4d0 0x7efd1006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efd200016e0 tx=0x7efd20009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:53.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.057+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7efd18030830 con 0x7efd2c0d22b0 2024-03-06T07:31:53.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.273+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd2c0747a0 con 0x7efd2c0d22b0 2024-03-06T07:31:53.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.273+0000 7efd297fa640 1 -- 172.21.15.72:0/1771359374 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7efd1807aaa0 con 0x7efd2c0d22b0 2024-03-06T07:31:53.276 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:53.276 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:53.276 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:53.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd1006b4d0 msgr2=0x7efd1006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd1006b4d0 0x7efd1006d990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7efd200016e0 tx=0x7efd20009040 comp rx=0 tx=0).stop 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 msgr2=0x7efd2c1f29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c1f29f0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7efd18000c00 tx=0x7efd18004730 comp rx=0 tx=0).stop 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 shutdown_connections 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7efd1006b4d0 0x7efd1006d990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efd2c15dfe0 0x7efd2c1f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efd2c0d22b0 0x7efd2c1f29f0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 --2- 172.21.15.72:0/1771359374 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efd2c0b41f0 0x7efd2c1599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 >> 172.21.15.72:0/1771359374 conn(0x7efd2c0d3cd0 msgr2=0x7efd2c0d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 shutdown_connections 2024-03-06T07:31:53.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:53.276+0000 7efd3311e640 1 -- 172.21.15.72:0/1771359374 wait complete. 2024-03-06T07:31:53.494 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:53.494 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:53.817 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:54.495 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:54.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.594+0000 7f56e9a83640 1 Processor -- start 2024-03-06T07:31:54.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.594+0000 7f56e9a83640 1 -- start start 2024-03-06T07:31:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.595+0000 7f56e9a83640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56e415dfe0 0x7f56e40cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.595+0000 7f56e9a83640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.595+0000 7f56e9a83640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e40b3430 con 0x7f56e40cf830 2024-03-06T07:31:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.595+0000 7f56e9a83640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e40b31e0 con 0x7f56e415dfe0 2024-03-06T07:31:54.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.595+0000 7f56e9a83640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40b3d60 con 0x7f56e40d29c0 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e37fe640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f56e415dfe0 0x7f56e40cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e2ffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f56e40cf830 0x7f56e40d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:37806/0 (socket says 172.21.15.72:37806) 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e2ffd640 1 -- 172.21.15.72:0/2401059430 learned_addr learned my addr 172.21.15.72:0/2401059430 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e3fff640 1 -- 172.21.15.72:0/2401059430 >> v2:172.21.15.146:3300/0 conn(0x7f56e415dfe0 msgr2=0x7f56e40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e3fff640 1 --2- 172.21.15.72:0/2401059430 >> v2:172.21.15.146:3300/0 conn(0x7f56e415dfe0 0x7f56e40cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e3fff640 1 -- 172.21.15.72:0/2401059430 >> v1:172.21.15.146:6789/0 conn(0x7f56e40cf830 legacy=0x7f56e40d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e3fff640 1 -- 172.21.15.72:0/2401059430 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40b3710 con 0x7f56e40d29c0 2024-03-06T07:31:54.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.596+0000 7f56e37fe640 1 --2- 172.21.15.72:0/2401059430 >> v2:172.21.15.146:3300/0 conn(0x7f56e415dfe0 0x7f56e40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.597+0000 7f56e3fff640 1 --2- 172.21.15.72:0/2401059430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40b41f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009d90 tx=0x7f56dc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20649ae2d857517f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.597+0000 7f56e27fc640 1 -- 172.21.15.72:0/2401059430 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc039070 con 0x7f56e40d29c0 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.597+0000 7f56e27fc640 1 -- 172.21.15.72:0/2401059430 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc02cc50 con 0x7f56e40d29c0 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.597+0000 7f56e27fc640 1 -- 172.21.15.72:0/2401059430 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc034440 con 0x7f56e40d29c0 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 -- 172.21.15.72:0/2401059430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 msgr2=0x7f56e40b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2401059430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40b41f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009d90 tx=0x7f56dc02bea0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 -- 172.21.15.72:0/2401059430 shutdown_connections 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2401059430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40b41f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2401059430 >> v2:172.21.15.146:3300/0 conn(0x7f56e415dfe0 0x7f56e40cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 -- 172.21.15.72:0/2401059430 >> 172.21.15.72:0/2401059430 conn(0x7f56e40d3cd0 msgr2=0x7f56e40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 -- 172.21.15.72:0/2401059430 shutdown_connections 2024-03-06T07:31:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.598+0000 7f56e9a83640 1 -- 172.21.15.72:0/2401059430 wait complete. 2024-03-06T07:31:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.599+0000 7f56e9a83640 1 Processor -- start 2024-03-06T07:31:54.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.599+0000 7f56e9a83640 1 -- start start 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e9a83640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cf830 0x7f56e40b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e9a83640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e9a83640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 0x7f56e4152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e9a83640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40d1df0 con 0x7f56e415dfe0 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e9a83640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40d1c70 con 0x7f56e40cf830 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e9a83640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e415e3c0 con 0x7f56e40d29c0 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 0x7f56e4152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 0x7f56e4152020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57858/0 (socket says 172.21.15.72:57858) 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 -- 172.21.15.72:0/2560125953 learned_addr learned my addr 172.21.15.72:0/2560125953 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e2ffd640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e37fe640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cf830 0x7f56e40b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 -- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 msgr2=0x7f56e40cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 -- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cf830 msgr2=0x7f56e40b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cf830 0x7f56e40b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 -- 172.21.15.72:0/2560125953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40b81a0 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e37fe640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cf830 0x7f56e40b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.600+0000 7f56e3fff640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 0x7f56e4152020 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009d90 tx=0x7f56dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.601+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc039050 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.601+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56dc045070 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.601+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e40ccea0 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.601+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56dc009450 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.601+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e40cd460 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.602+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e404f330 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.602+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f56dc03e610 con 0x7f56e415dfe0 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.602+0000 7f56cbfff640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f56d006b500 0x7f56d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:54.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.603+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f56dc0acc20 con 0x7f56e415dfe0 2024-03-06T07:31:54.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.603+0000 7f56e37fe640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f56d006b500 0x7f56d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:54.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.603+0000 7f56e37fe640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f56d006b500 0x7f56d006d9c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009a20 tx=0x7f56d8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:54.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.611+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f56dc07baf0 con 0x7f56e415dfe0 2024-03-06T07:31:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.825+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56e40b5490 con 0x7f56e415dfe0 2024-03-06T07:31:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.826+0000 7f56cbfff640 1 -- 172.21.15.72:0/2560125953 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f56dc07b3f0 con 0x7f56e415dfe0 2024-03-06T07:31:54.828 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:54.828 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:54.829 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:54.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f56d006b500 msgr2=0x7f56d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f56d006b500 0x7f56d006d9c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009a20 tx=0x7f56d8008040 comp rx=0 tx=0).stop 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 msgr2=0x7f56e4152020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 0x7f56e4152020 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009d90 tx=0x7f56dc004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 shutdown_connections 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f56d006b500 0x7f56d006d9c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f56e415dfe0 0x7f56e4152020 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56e40d29c0 0x7f56e40cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 --2- 172.21.15.72:0/2560125953 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56e40cf830 0x7f56e40b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 >> 172.21.15.72:0/2560125953 conn(0x7f56e40d3cd0 msgr2=0x7f56e406cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 shutdown_connections 2024-03-06T07:31:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:54.828+0000 7f56e9a83640 1 -- 172.21.15.72:0/2560125953 wait complete. 2024-03-06T07:31:55.042 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:55.042 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:55.819 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:56.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.143+0000 7f72640a4640 1 Processor -- start 2024-03-06T07:31:56.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.143+0000 7f72640a4640 1 -- start start 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f72640a4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f725c0d22b0 0x7f725c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f72640a4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f725c0b41f0 0x7f725c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f72640a4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f725c0b3430 con 0x7f725c15dfe0 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f72640a4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725c0d2c50 con 0x7f725c0b41f0 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f72640a4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f725c0b31e0 con 0x7f725c0d22b0 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f7262641640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f725c0b41f0 0x7f725c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f7261e40640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f725c15dfe0 0x7f725c0cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59072/0 (socket says 172.21.15.72:59072) 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.144+0000 7f7261e40640 1 -- 172.21.15.72:0/2258124294 learned_addr learned my addr 172.21.15.72:0/2258124294 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7262641640 1 -- 172.21.15.72:0/2258124294 >> v2:172.21.15.146:3300/0 conn(0x7f725c0d22b0 msgr2=0x7f725c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7262641640 1 --2- 172.21.15.72:0/2258124294 >> v2:172.21.15.146:3300/0 conn(0x7f725c0d22b0 0x7f725c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7262641640 1 -- 172.21.15.72:0/2258124294 >> v1:172.21.15.72:6789/0 conn(0x7f725c15dfe0 legacy=0x7f725c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7262641640 1 -- 172.21.15.72:0/2258124294 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c0b3710 con 0x7f725c0b41f0 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7262641640 1 --2- 172.21.15.72:0/2258124294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c0b41f0 0x7f725c0b4680 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f7258009d90 tx=0x7f725802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355f20c81b468bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7260e3e640 1 -- 172.21.15.72:0/2258124294 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258039070 con 0x7f725c0b41f0 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7260e3e640 1 -- 172.21.15.72:0/2258124294 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725802cc60 con 0x7f725c0b41f0 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.145+0000 7f7260e3e640 1 -- 172.21.15.72:0/2258124294 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258034660 con 0x7f725c0b41f0 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 -- 172.21.15.72:0/2258124294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c0b41f0 msgr2=0x7f725c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 --2- 172.21.15.72:0/2258124294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c0b41f0 0x7f725c0b4680 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f7258009d90 tx=0x7f725802beb0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 -- 172.21.15.72:0/2258124294 shutdown_connections 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 --2- 172.21.15.72:0/2258124294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c0b41f0 0x7f725c0b4680 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 --2- 172.21.15.72:0/2258124294 >> v2:172.21.15.146:3300/0 conn(0x7f725c0d22b0 0x7f725c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 -- 172.21.15.72:0/2258124294 >> 172.21.15.72:0/2258124294 conn(0x7f725c0d3cd0 msgr2=0x7f725c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 -- 172.21.15.72:0/2258124294 shutdown_connections 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.146+0000 7f72640a4640 1 -- 172.21.15.72:0/2258124294 wait complete. 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.147+0000 7f72640a4640 1 Processor -- start 2024-03-06T07:31:56.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.147+0000 7f72640a4640 1 -- start start 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f72640a4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f725c0b41f0 0x7f725c1f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f72640a4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f725c0d22b0 0x7f725c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f72640a4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 0x7f725c1f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f72640a4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c0cfa00 con 0x7f725c15dfe0 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f72640a4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c0b3d60 con 0x7f725c0b41f0 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f72640a4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c0d1c70 con 0x7f725c0d22b0 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 0x7f725c1f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 0x7f725c1f32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57870/0 (socket says 172.21.15.72:57870) 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 -- 172.21.15.72:0/2556944376 learned_addr learned my addr 172.21.15.72:0/2556944376 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7261e40640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f725c0b41f0 0x7f725c1f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f726163f640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f725c0d22b0 0x7f725c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 -- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f725c0d22b0 msgr2=0x7f725c0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f725c0d22b0 0x7f725c0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 -- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f725c0b41f0 msgr2=0x7f725c1f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f725c0b41f0 0x7f725c1f6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7262641640 1 -- 172.21.15.72:0/2556944376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725c1f7680 con 0x7f725c15dfe0 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f7261e40640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f725c0b41f0 0x7f725c1f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.148+0000 7f726163f640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f725c0d22b0 0x7f725c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.149+0000 7f7262641640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 0x7f725c1f32e0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f7258009d90 tx=0x7f7258004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.149+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258039050 con 0x7f725c15dfe0 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.149+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725803a040 con 0x7f725c15dfe0 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.149+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725803e610 con 0x7f725c15dfe0 2024-03-06T07:31:56.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.149+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725c1f7960 con 0x7f725c15dfe0 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.149+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725c1f3be0 con 0x7f725c15dfe0 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.150+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725c1f7e80 con 0x7f725c15dfe0 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.150+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7258047020 con 0x7f725c15dfe0 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.150+0000 7f72527fc640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.151+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f72580ac6b0 con 0x7f725c15dfe0 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.151+0000 7f7261e40640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:56.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.151+0000 7f7261e40640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f724006b4d0 0x7f724006d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7254000f60 tx=0x7f7254009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:56.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.155+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f725c1f7e80 con 0x7f725c15dfe0 2024-03-06T07:31:56.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.370+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f725c04f330 con 0x7f725c15dfe0 2024-03-06T07:31:56.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.371+0000 7f72527fc640 1 -- 172.21.15.72:0/2556944376 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f725804e020 con 0x7f725c15dfe0 2024-03-06T07:31:56.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:56.373 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:56.373 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:56.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f724006b4d0 msgr2=0x7f724006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f724006b4d0 0x7f724006d990 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7254000f60 tx=0x7f7254009040 comp rx=0 tx=0).stop 2024-03-06T07:31:56.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 msgr2=0x7f725c1f32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:56.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 0x7f725c1f32e0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f7258009d90 tx=0x7f7258004a40 comp rx=0 tx=0).stop 2024-03-06T07:31:56.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 shutdown_connections 2024-03-06T07:31:56.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f724006b4d0 0x7f724006d990 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f725c15dfe0 0x7f725c1f32e0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f725c0d22b0 0x7f725c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 --2- 172.21.15.72:0/2556944376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f725c0b41f0 0x7f725c1f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:56.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 >> 172.21.15.72:0/2556944376 conn(0x7f725c0d3cd0 msgr2=0x7f725c0d0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:56.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 shutdown_connections 2024-03-06T07:31:56.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:56.373+0000 7f72640a4640 1 -- 172.21.15.72:0/2556944376 wait complete. 2024-03-06T07:31:56.587 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:56.587 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:57.588 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:57.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.687+0000 7fb0c7701640 1 Processor -- start 2024-03-06T07:31:57.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.687+0000 7fb0c7701640 1 -- start start 2024-03-06T07:31:57.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c7701640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0c015dfe0 0x7fb0c00cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c7701640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00cf830 0x7fb0c00d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c7701640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb0c00d27f0 0x7fb0c00d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c7701640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c00d2c50 con 0x7fb0c015dfe0 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c7701640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c00b31e0 con 0x7fb0c00d27f0 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c7701640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b3d60 con 0x7fb0c00cf830 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0c015dfe0 0x7fb0c00cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0c015dfe0 0x7fb0c00cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57882/0 (socket says 172.21.15.72:57882) 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 -- 172.21.15.72:0/87688159 learned_addr learned my addr 172.21.15.72:0/87688159 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 -- 172.21.15.72:0/87688159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00cf830 msgr2=0x7fb0c00d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 --2- 172.21.15.72:0/87688159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00cf830 0x7fb0c00d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 -- 172.21.15.72:0/87688159 >> v2:172.21.15.146:3300/0 conn(0x7fb0c00d27f0 msgr2=0x7fb0c00d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 --2- 172.21.15.72:0/87688159 >> v2:172.21.15.146:3300/0 conn(0x7fb0c00d27f0 0x7fb0c00d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.688+0000 7fb0c549d640 1 -- 172.21.15.72:0/87688159 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00b3430 con 0x7fb0c015dfe0 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.689+0000 7fb0c549d640 1 --2- 172.21.15.72:0/87688159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c00cf2f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0067ad0 tx=0x7fb0b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892aad18fcec2c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.689+0000 7fb0b7fff640 1 -- 172.21.15.72:0/87688159 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0035020 con 0x7fb0c015dfe0 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.689+0000 7fb0b7fff640 1 -- 172.21.15.72:0/87688159 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0031870 con 0x7fb0c015dfe0 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.689+0000 7fb0b7fff640 1 -- 172.21.15.72:0/87688159 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0031c40 con 0x7fb0c015dfe0 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 -- 172.21.15.72:0/87688159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 msgr2=0x7fb0c00cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 --2- 172.21.15.72:0/87688159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c00cf2f0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0067ad0 tx=0x7fb0b002b910 comp rx=0 tx=0).stop 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 -- 172.21.15.72:0/87688159 shutdown_connections 2024-03-06T07:31:57.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 --2- 172.21.15.72:0/87688159 >> v2:172.21.15.146:3300/0 conn(0x7fb0c00d27f0 0x7fb0c00d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 --2- 172.21.15.72:0/87688159 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00cf830 0x7fb0c00d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 --2- 172.21.15.72:0/87688159 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c00cf2f0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 -- 172.21.15.72:0/87688159 >> 172.21.15.72:0/87688159 conn(0x7fb0c00d3cd0 msgr2=0x7fb0c00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 -- 172.21.15.72:0/87688159 shutdown_connections 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.690+0000 7fb0c7701640 1 -- 172.21.15.72:0/87688159 wait complete. 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.691+0000 7fb0c7701640 1 Processor -- start 2024-03-06T07:31:57.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.691+0000 7fb0c7701640 1 -- start start 2024-03-06T07:31:57.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0c00cf830 0x7fb0c0159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00d27f0 0x7fb0c00cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c0160480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00d1c70 con 0x7fb0c015dfe0 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b39f0 con 0x7fb0c00cf830 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00d1df0 con 0x7fb0c00d27f0 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c0160480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c0160480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57888/0 (socket says 172.21.15.72:57888) 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 -- 172.21.15.72:0/215290325 learned_addr learned my addr 172.21.15.72:0/215290325 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 -- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00d27f0 msgr2=0x7fb0c00cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c4c9c640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00d27f0 0x7fb0c00cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c549d640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0c00cf830 0x7fb0c0159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00d27f0 0x7fb0c00cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 -- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0c00cf830 msgr2=0x7fb0c0159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0c00cf830 0x7fb0c0159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 -- 172.21.15.72:0/215290325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00b8570 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c5c9e640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c0160480 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800b970 tx=0x7fb0b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8019070 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b800cb60 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c00b8850 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.692+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8014440 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.693+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c015a280 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.694+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c004f330 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.694+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb0b801c070 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.694+0000 7fb0b5ffb640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0a406b4d0 0x7fb0a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.695+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb0b8089f80 con 0x7fb0c015dfe0 2024-03-06T07:31:57.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.695+0000 7fb0c549d640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0a406b4d0 0x7fb0a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:57.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.695+0000 7fb0c549d640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0a406b4d0 0x7fb0a406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0036000 tx=0x7fb0b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:57.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.699+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb0b8058dc0 con 0x7fb0c015dfe0 2024-03-06T07:31:57.820 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:31:57.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.971+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c00747a0 con 0x7fb0c015dfe0 2024-03-06T07:31:57.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.971+0000 7fb0b5ffb640 1 -- 172.21.15.72:0/215290325 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fb0b80586c0 con 0x7fb0c015dfe0 2024-03-06T07:31:57.973 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:57.974 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:57.974 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0a406b4d0 msgr2=0x7fb0a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0a406b4d0 0x7fb0a406d990 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0036000 tx=0x7fb0b003c040 comp rx=0 tx=0).stop 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 msgr2=0x7fb0c0160480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c0160480 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fb0b800b970 tx=0x7fb0b800be30 comp rx=0 tx=0).stop 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 shutdown_connections 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb0a406b4d0 0x7fb0a406d990 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0c015dfe0 0x7fb0c0160480 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0c00d27f0 0x7fb0c00cd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 --2- 172.21.15.72:0/215290325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0c00cf830 0x7fb0c0159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 >> 172.21.15.72:0/215290325 conn(0x7fb0c00d3cd0 msgr2=0x7fb0c006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 shutdown_connections 2024-03-06T07:31:57.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:57.973+0000 7fb0c7701640 1 -- 172.21.15.72:0/215290325 wait complete. 2024-03-06T07:31:58.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:58.189 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:59.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:31:59.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.289+0000 7f1a7871e640 1 Processor -- start 2024-03-06T07:31:59.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.289+0000 7f1a7871e640 1 -- start start 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.290+0000 7f1a7871e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.290+0000 7f1a7871e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1a700cf830 0x7f1a700d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.290+0000 7f1a7871e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a700d27f0 0x7f1a700d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.290+0000 7f1a7871e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a700d2c50 con 0x7f1a700d27f0 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.290+0000 7f1a7871e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a700b31e0 con 0x7f1a700cf830 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.290+0000 7f1a7871e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a700b3d60 con 0x7f1a7015dfe0 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a700d27f0 0x7f1a700d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1a700d27f0 0x7f1a700d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57894/0 (socket says 172.21.15.72:57894) 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 -- 172.21.15.72:0/2685159702 learned_addr learned my addr 172.21.15.72:0/2685159702 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 -- 172.21.15.72:0/2685159702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 msgr2=0x7f1a700cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a764ba640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a75cb9640 1 --2- 172.21.15.72:0/2685159702 >> v2:172.21.15.146:3300/0 conn(0x7f1a700cf830 0x7f1a700d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 -- 172.21.15.72:0/2685159702 >> v2:172.21.15.146:3300/0 conn(0x7f1a700cf830 msgr2=0x7f1a700d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 --2- 172.21.15.72:0/2685159702 >> v2:172.21.15.146:3300/0 conn(0x7f1a700cf830 0x7f1a700d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a76cbb640 1 -- 172.21.15.72:0/2685159702 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a700b3430 con 0x7f1a700d27f0 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a764ba640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.291+0000 7f1a75cb9640 1 --2- 172.21.15.72:0/2685159702 >> v2:172.21.15.146:3300/0 conn(0x7f1a700cf830 0x7f1a700d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a76cbb640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700d0300 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f1a7008cba0 tx=0x7f1a6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808add8737826d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a754b8640 1 -- 172.21.15.72:0/2685159702 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6802ca70 con 0x7f1a700d27f0 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a754b8640 1 -- 172.21.15.72:0/2685159702 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6802cc10 con 0x7f1a700d27f0 2024-03-06T07:31:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a754b8640 1 -- 172.21.15.72:0/2685159702 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a680343f0 con 0x7f1a700d27f0 2024-03-06T07:31:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 -- 172.21.15.72:0/2685159702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 msgr2=0x7f1a700d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700d0300 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f1a7008cba0 tx=0x7f1a6802bed0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 -- 172.21.15.72:0/2685159702 shutdown_connections 2024-03-06T07:31:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700d0300 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 --2- 172.21.15.72:0/2685159702 >> v2:172.21.15.146:3300/0 conn(0x7f1a700cf830 0x7f1a700d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 --2- 172.21.15.72:0/2685159702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 -- 172.21.15.72:0/2685159702 >> 172.21.15.72:0/2685159702 conn(0x7f1a700d3cd0 msgr2=0x7f1a700d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 -- 172.21.15.72:0/2685159702 shutdown_connections 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.292+0000 7f1a7871e640 1 -- 172.21.15.72:0/2685159702 wait complete. 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.293+0000 7f1a7871e640 1 Processor -- start 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 -- start start 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a700cf830 0x7f1a701567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a700d1c70 con 0x7f1a700d27f0 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a700b39f0 con 0x7f1a700cf830 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a7871e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a700d1df0 con 0x7f1a7015dfe0 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a75cb9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a75cb9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57906/0 (socket says 172.21.15.72:57906) 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.294+0000 7f1a75cb9640 1 -- 172.21.15.72:0/469818293 learned_addr learned my addr 172.21.15.72:0/469818293 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a75cb9640 1 -- 172.21.15.72:0/469818293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 msgr2=0x7f1a700b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a75cb9640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a75cb9640 1 -- 172.21.15.72:0/469818293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a700cf830 msgr2=0x7f1a701567e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a75cb9640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a700cf830 0x7f1a701567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a75cb9640 1 -- 172.21.15.72:0/469818293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a700b90c0 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a75cb9640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700b68d0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000c7f0 tx=0x7f1a6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a60015070 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a60007820 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a700b93a0 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a60007bf0 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.295+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a700b9960 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.296+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a700b9ba0 con 0x7f1a700d27f0 2024-03-06T07:31:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.296+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1a60018070 con 0x7f1a700d27f0 2024-03-06T07:31:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.297+0000 7f1a66ffd640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a5406b4d0 0x7f1a5406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:31:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.297+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1a6008a1b0 con 0x7f1a700d27f0 2024-03-06T07:31:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.297+0000 7f1a764ba640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a5406b4d0 0x7f1a5406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:31:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.297+0000 7f1a764ba640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a5406b4d0 0x7f1a5406d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c006fd0 tx=0x7f1a6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:31:59.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.301+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1a700b9ba0 con 0x7f1a700d27f0 2024-03-06T07:31:59.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.516+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a70156d20 con 0x7f1a700d27f0 2024-03-06T07:31:59.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.516+0000 7f1a66ffd640 1 -- 172.21.15.72:0/469818293 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f1a60058f60 con 0x7f1a700d27f0 2024-03-06T07:31:59.518 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:31:59.519 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:31:59.519 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a5406b4d0 msgr2=0x7f1a5406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a5406b4d0 0x7f1a5406d990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c006fd0 tx=0x7f1a6c008040 comp rx=0 tx=0).stop 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 msgr2=0x7f1a700b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700b68d0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000c7f0 tx=0x7f1a6000ccb0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 shutdown_connections 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1a5406b4d0 0x7f1a5406d990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a7015dfe0 0x7f1a700b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a700d27f0 0x7f1a700b68d0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 --2- 172.21.15.72:0/469818293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a700cf830 0x7f1a701567e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.518+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 >> 172.21.15.72:0/469818293 conn(0x7f1a700d3cd0 msgr2=0x7f1a7006ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.519+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 shutdown_connections 2024-03-06T07:31:59.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:31:59.519+0000 7f1a7871e640 1 -- 172.21.15.72:0/469818293 wait complete. 2024-03-06T07:31:59.733 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:31:59.733 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:31:59.822 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:00.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:00.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.835+0000 7f4db4265640 1 Processor -- start 2024-03-06T07:32:00.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.835+0000 7f4db4265640 1 -- start start 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db4265640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac15bb20 0x7f4dac0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db4265640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4dac0cd8c0 0x7f4dac0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db4265640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac161980 con 0x7f4dac0cfff0 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db4265640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dac161c60 con 0x7f4dac0cd8c0 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db4265640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac161f40 con 0x7f4dac15bb20 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db1800640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4dac0cfff0 0x7f4dac0ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59074/0 (socket says 172.21.15.72:59074) 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db1800640 1 -- 172.21.15.72:0/80149476 learned_addr learned my addr 172.21.15.72:0/80149476 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db2802640 1 --2- 172.21.15.72:0/80149476 >> v2:172.21.15.72:3300/0 conn(0x7f4dac0cd8c0 0x7f4dac0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db2001640 1 --2- 172.21.15.72:0/80149476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac15bb20 0x7f4dac0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.836+0000 7f4db0fff640 1 -- 172.21.15.72:0/80149476 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345075085 0 0) 0x7f4dac161980 con 0x7f4dac0cfff0 2024-03-06T07:32:00.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.837+0000 7f4db0fff640 1 -- 172.21.15.72:0/80149476 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d98003520 con 0x7f4dac0cfff0 2024-03-06T07:32:00.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.837+0000 7f4db2802640 1 -- 172.21.15.72:0/80149476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac15bb20 msgr2=0x7f4dac0d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:00.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.837+0000 7f4db2802640 1 --2- 172.21.15.72:0/80149476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac15bb20 0x7f4dac0d1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.837+0000 7f4db2802640 1 -- 172.21.15.72:0/80149476 >> v1:172.21.15.72:6789/0 conn(0x7f4dac0cfff0 legacy=0x7f4dac0ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.837+0000 7f4db2802640 1 -- 172.21.15.72:0/80149476 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac162220 con 0x7f4dac0cd8c0 2024-03-06T07:32:00.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.837+0000 7f4db2802640 1 --2- 172.21.15.72:0/80149476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac0cd8c0 0x7f4dac0d0470 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f4da40097a0 tx=0x7f4da402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781e1bb2a0e25e33 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db0fff640 1 -- 172.21.15.72:0/80149476 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da40315a0 con 0x7f4dac0cd8c0 2024-03-06T07:32:00.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db0fff640 1 -- 172.21.15.72:0/80149476 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da4037070 con 0x7f4dac0cd8c0 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db0fff640 1 -- 172.21.15.72:0/80149476 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4031940 con 0x7f4dac0cd8c0 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 -- 172.21.15.72:0/80149476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac0cd8c0 msgr2=0x7f4dac0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 --2- 172.21.15.72:0/80149476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac0cd8c0 0x7f4dac0d0470 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f4da40097a0 tx=0x7f4da402b840 comp rx=0 tx=0).stop 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 -- 172.21.15.72:0/80149476 shutdown_connections 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 --2- 172.21.15.72:0/80149476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac0cd8c0 0x7f4dac0d0470 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 --2- 172.21.15.72:0/80149476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac15bb20 0x7f4dac0d1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 -- 172.21.15.72:0/80149476 >> 172.21.15.72:0/80149476 conn(0x7f4dac09a9f0 msgr2=0x7f4dac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 -- 172.21.15.72:0/80149476 shutdown_connections 2024-03-06T07:32:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.838+0000 7f4db4265640 1 -- 172.21.15.72:0/80149476 wait complete. 2024-03-06T07:32:00.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.840+0000 7f4db4265640 1 Processor -- start 2024-03-06T07:32:00.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.840+0000 7f4db4265640 1 -- start start 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.841+0000 7f4db4265640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0cd8c0 0x7f4dac151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0cfff0 0x7f4dac0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 0x7f4dac15cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0d24b0 con 0x7f4dac15bb20 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0d2330 con 0x7f4dac0cfff0 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dac0d1980 con 0x7f4dac0cd8c0 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 0x7f4dac15cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 0x7f4dac15cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57928/0 (socket says 172.21.15.72:57928) 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 -- 172.21.15.72:0/3895264538 learned_addr learned my addr 172.21.15.72:0/3895264538 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2001640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0cd8c0 0x7f4dac151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db1800640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0cfff0 0x7f4dac0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 -- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0cd8c0 msgr2=0x7f4dac151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0cd8c0 0x7f4dac151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 -- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0cfff0 msgr2=0x7f4dac0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0cfff0 0x7f4dac0c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 -- 172.21.15.72:0/3895264538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0c6cf0 con 0x7f4dac15bb20 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db2802640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 0x7f4dac15cf50 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f4da4034040 tx=0x7f4da40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4032630 con 0x7f4dac15bb20 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40327d0 con 0x7f4dac15bb20 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac0c7490 con 0x7f4dac15bb20 2024-03-06T07:32:00.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da4032bc0 con 0x7f4dac15bb20 2024-03-06T07:32:00.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.842+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac0c7a50 con 0x7f4dac15bb20 2024-03-06T07:32:00.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.843+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac04f330 con 0x7f4dac15bb20 2024-03-06T07:32:00.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.844+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4da404c020 con 0x7f4dac15bb20 2024-03-06T07:32:00.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.844+0000 7f4da27fc640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d9006b4d0 0x7f4d9006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:00.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.844+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4da402f020 con 0x7f4dac15bb20 2024-03-06T07:32:00.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.844+0000 7f4db2001640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d9006b4d0 0x7f4d9006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:00.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.845+0000 7f4db2001640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d9006b4d0 0x7f4d9006d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f4da80074a0 tx=0x7f4da800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:00.849+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4da4040070 con 0x7f4dac15bb20 2024-03-06T07:32:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.099+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dac06e240 con 0x7f4dac15bb20 2024-03-06T07:32:01.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.099+0000 7f4da27fc640 1 -- 172.21.15.72:0/3895264538 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f4da407abd0 con 0x7f4dac15bb20 2024-03-06T07:32:01.102 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:01.102 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:01.102 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:01.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d9006b4d0 msgr2=0x7f4d9006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d9006b4d0 0x7f4d9006d990 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f4da80074a0 tx=0x7f4da800b040 comp rx=0 tx=0).stop 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 msgr2=0x7f4dac15cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 0x7f4dac15cf50 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f4da4034040 tx=0x7f4da40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 shutdown_connections 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4d9006b4d0 0x7f4d9006d990 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4dac15bb20 0x7f4dac15cf50 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dac0cfff0 0x7f4dac0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 --2- 172.21.15.72:0/3895264538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dac0cd8c0 0x7f4dac151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 >> 172.21.15.72:0/3895264538 conn(0x7f4dac09a9f0 msgr2=0x7f4dac0cc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 shutdown_connections 2024-03-06T07:32:01.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:01.101+0000 7f4db4265640 1 -- 172.21.15.72:0/3895264538 wait complete. 2024-03-06T07:32:01.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:01.315 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:02.316 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:02.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.415+0000 7ff988fbd640 1 Processor -- start 2024-03-06T07:32:02.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.415+0000 7ff988fbd640 1 -- start start 2024-03-06T07:32:02.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff988fbd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9840d22b0 0x7ff9840d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff988fbd640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9840b41f0 0x7ff9840b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff988fbd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9840b3430 con 0x7ff98415dfe0 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff988fbd640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9840d2c50 con 0x7ff9840b41f0 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff988fbd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840b31e0 con 0x7ff9840d22b0 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff982d9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9840b41f0 0x7ff9840b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff98259c640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7ff98415dfe0 0x7ff9840cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:37818/0 (socket says 172.21.15.72:37818) 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff981d9b640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9840d22b0 0x7ff9840d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.417+0000 7ff98259c640 1 -- 172.21.15.72:0/3823415603 learned_addr learned my addr 172.21.15.72:0/3823415603 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.418+0000 7ff981d9b640 1 -- 172.21.15.72:0/3823415603 >> v2:172.21.15.146:3300/0 conn(0x7ff9840b41f0 msgr2=0x7ff9840b4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.418+0000 7ff981d9b640 1 --2- 172.21.15.72:0/3823415603 >> v2:172.21.15.146:3300/0 conn(0x7ff9840b41f0 0x7ff9840b4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.418+0000 7ff981d9b640 1 -- 172.21.15.72:0/3823415603 >> v1:172.21.15.146:6789/0 conn(0x7ff98415dfe0 legacy=0x7ff9840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.418+0000 7ff981d9b640 1 -- 172.21.15.72:0/3823415603 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840b3710 con 0x7ff9840d22b0 2024-03-06T07:32:02.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.418+0000 7ff982d9d640 1 --2- 172.21.15.72:0/3823415603 >> v2:172.21.15.146:3300/0 conn(0x7ff9840b41f0 0x7ff9840b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.418+0000 7ff981d9b640 1 --2- 172.21.15.72:0/3823415603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9840d22b0 0x7ff9840d2710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0097a0 tx=0x7ff96c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b30752207f5e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff98159a640 1 -- 172.21.15.72:0/3823415603 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c035020 con 0x7ff9840d22b0 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff98159a640 1 -- 172.21.15.72:0/3823415603 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c031830 con 0x7ff9840d22b0 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff98159a640 1 -- 172.21.15.72:0/3823415603 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c031c00 con 0x7ff9840d22b0 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 -- 172.21.15.72:0/3823415603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9840d22b0 msgr2=0x7ff9840d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 --2- 172.21.15.72:0/3823415603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9840d22b0 0x7ff9840d2710 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff96c0097a0 tx=0x7ff96c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 -- 172.21.15.72:0/3823415603 shutdown_connections 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 --2- 172.21.15.72:0/3823415603 >> v2:172.21.15.146:3300/0 conn(0x7ff9840b41f0 0x7ff9840b4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 --2- 172.21.15.72:0/3823415603 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff9840d22b0 0x7ff9840d2710 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 -- 172.21.15.72:0/3823415603 >> 172.21.15.72:0/3823415603 conn(0x7ff9840d3cd0 msgr2=0x7ff9840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.419+0000 7ff988fbd640 1 -- 172.21.15.72:0/3823415603 shutdown_connections 2024-03-06T07:32:02.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.420+0000 7ff988fbd640 1 -- 172.21.15.72:0/3823415603 wait complete. 2024-03-06T07:32:02.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.421+0000 7ff988fbd640 1 Processor -- start 2024-03-06T07:32:02.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.421+0000 7ff988fbd640 1 -- start start 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff988fbd640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9840b41f0 0x7ff9841f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff988fbd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 0x7ff9840cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff988fbd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98415dfe0 0x7ff9840cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff988fbd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840cfa00 con 0x7ff9840d22b0 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff988fbd640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840b3d60 con 0x7ff9840b41f0 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff988fbd640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9840d1c70 con 0x7ff98415dfe0 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 0x7ff9840cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 0x7ff9840cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57936/0 (socket says 172.21.15.72:57936) 2024-03-06T07:32:02.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 -- 172.21.15.72:0/550297885 learned_addr learned my addr 172.21.15.72:0/550297885 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff982d9d640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98415dfe0 0x7ff9840cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff98259c640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9840b41f0 0x7ff9841f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 -- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98415dfe0 msgr2=0x7ff9840cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98415dfe0 0x7ff9840cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 -- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9840b41f0 msgr2=0x7ff9841f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9840b41f0 0x7ff9841f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 -- 172.21.15.72:0/550297885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840cdbd0 con 0x7ff9840d22b0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.423+0000 7ff981d9b640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 0x7ff9840cce60 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7ff96c02bee0 tx=0x7ff96c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.424+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c035040 con 0x7ff9840d22b0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.424+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff96c03b860 con 0x7ff9840d22b0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.424+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9840cdeb0 con 0x7ff9840d22b0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.424+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96c03bc50 con 0x7ff9840d22b0 2024-03-06T07:32:02.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.424+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9840cd8e0 con 0x7ff9840d22b0 2024-03-06T07:32:02.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.425+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98404f330 con 0x7ff9840d22b0 2024-03-06T07:32:02.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.425+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff96c038020 con 0x7ff9840d22b0 2024-03-06T07:32:02.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.425+0000 7ff97affd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff96406b500 0x7ff96406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:02.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.426+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff96c0ac1a0 con 0x7ff9840d22b0 2024-03-06T07:32:02.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.426+0000 7ff98259c640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff96406b500 0x7ff96406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:02.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.426+0000 7ff98259c640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff96406b500 0x7ff96406d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff9740012e0 tx=0x7ff974001990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:02.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.430+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff96c031800 con 0x7ff9840d22b0 2024-03-06T07:32:02.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.645+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9840b09c0 con 0x7ff9840d22b0 2024-03-06T07:32:02.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.645+0000 7ff97affd640 1 -- 172.21.15.72:0/550297885 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7ff96c07aff0 con 0x7ff9840d22b0 2024-03-06T07:32:02.648 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:02.648 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:02.648 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff96406b500 msgr2=0x7ff96406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff96406b500 0x7ff96406d9c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff9740012e0 tx=0x7ff974001990 comp rx=0 tx=0).stop 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 msgr2=0x7ff9840cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 0x7ff9840cce60 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7ff96c02bee0 tx=0x7ff96c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 shutdown_connections 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff96406b500 0x7ff96406d9c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98415dfe0 0x7ff9840cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9840d22b0 0x7ff9840cce60 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 --2- 172.21.15.72:0/550297885 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9840b41f0 0x7ff9841f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:02.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.647+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 >> 172.21.15.72:0/550297885 conn(0x7ff9840d3cd0 msgr2=0x7ff98406cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:02.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.648+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 shutdown_connections 2024-03-06T07:32:02.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:02.648+0000 7ff988fbd640 1 -- 172.21.15.72:0/550297885 wait complete. 2024-03-06T07:32:02.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:02.862 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:03.827 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:03.862 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.961+0000 7f8bafc9d640 1 Processor -- start 2024-03-06T07:32:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.961+0000 7f8bafc9d640 1 -- start start 2024-03-06T07:32:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bafc9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8ba815dfe0 0x7f8ba80cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bafc9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8ba80cf830 0x7f8ba80d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bafc9d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba80b3430 con 0x7f8ba80d27f0 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bafc9d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba80b31e0 con 0x7f8ba815dfe0 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bafc9d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba80b3d60 con 0x7f8ba80cf830 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8ba815dfe0 0x7f8ba80cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8ba815dfe0 0x7f8ba80cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58682/0 (socket says 172.21.15.72:58682) 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 -- 172.21.15.72:0/1876918775 learned_addr learned my addr 172.21.15.72:0/1876918775 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bad238640 1 --2- 172.21.15.72:0/1876918775 >> v2:172.21.15.146:3300/0 conn(0x7f8ba80cf830 0x7f8ba80d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 -- 172.21.15.72:0/1876918775 >> v2:172.21.15.146:3300/0 conn(0x7f8ba80cf830 msgr2=0x7f8ba80d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 --2- 172.21.15.72:0/1876918775 >> v2:172.21.15.146:3300/0 conn(0x7f8ba80cf830 0x7f8ba80d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 -- 172.21.15.72:0/1876918775 >> v1:172.21.15.146:6789/0 conn(0x7f8ba80d27f0 legacy=0x7f8ba80d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.962+0000 7f8bada39640 1 -- 172.21.15.72:0/1876918775 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba80b3710 con 0x7f8ba815dfe0 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.963+0000 7f8bada39640 1 --2- 172.21.15.72:0/1876918775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba815dfe0 0x7f8ba80cf2f0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009870 tx=0x7f8ba402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3baaa96f1409c93c server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.963+0000 7f8baca37640 1 -- 172.21.15.72:0/1876918775 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4035020 con 0x7f8ba815dfe0 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.963+0000 7f8baca37640 1 -- 172.21.15.72:0/1876918775 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4031870 con 0x7f8ba815dfe0 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.963+0000 7f8baca37640 1 -- 172.21.15.72:0/1876918775 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4031c40 con 0x7f8ba815dfe0 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 -- 172.21.15.72:0/1876918775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba815dfe0 msgr2=0x7f8ba80cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/1876918775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba815dfe0 0x7f8ba80cf2f0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4009870 tx=0x7f8ba402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 -- 172.21.15.72:0/1876918775 shutdown_connections 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/1876918775 >> v2:172.21.15.146:3300/0 conn(0x7f8ba80cf830 0x7f8ba80d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/1876918775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba815dfe0 0x7f8ba80cf2f0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 -- 172.21.15.72:0/1876918775 >> 172.21.15.72:0/1876918775 conn(0x7f8ba80d3cd0 msgr2=0x7f8ba80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 -- 172.21.15.72:0/1876918775 shutdown_connections 2024-03-06T07:32:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.964+0000 7f8bafc9d640 1 -- 172.21.15.72:0/1876918775 wait complete. 2024-03-06T07:32:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.965+0000 7f8bafc9d640 1 Processor -- start 2024-03-06T07:32:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.965+0000 7f8bafc9d640 1 -- start start 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 0x7f8ba81610a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ba80d27f0 0x7f8ba8149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ba815dfe0 0x7f8ba814a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80d1df0 con 0x7f8ba80cf830 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80d1c70 con 0x7f8ba80d27f0 2024-03-06T07:32:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba815e3c0 con 0x7f8ba815dfe0 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 0x7f8ba81610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 0x7f8ba81610a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58688/0 (socket says 172.21.15.72:58688) 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 -- 172.21.15.72:0/2542943082 learned_addr learned my addr 172.21.15.72:0/2542943082 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 -- 172.21.15.72:0/2542943082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ba815dfe0 msgr2=0x7f8ba814a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ba815dfe0 0x7f8ba814a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 -- 172.21.15.72:0/2542943082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ba80d27f0 msgr2=0x7f8ba8149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ba80d27f0 0x7f8ba8149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 -- 172.21.15.72:0/2542943082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba80ccbd0 con 0x7f8ba80cf830 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bada39640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 0x7f8ba81610a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000c00 tx=0x7f8ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4040070 con 0x7f8ba80cf830 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba4030d00 con 0x7f8ba80cf830 2024-03-06T07:32:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba80cceb0 con 0x7f8ba80cf830 2024-03-06T07:32:03.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4035040 con 0x7f8ba80cf830 2024-03-06T07:32:03.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.966+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba80cd470 con 0x7f8ba80cf830 2024-03-06T07:32:03.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.967+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba804f330 con 0x7f8ba80cf830 2024-03-06T07:32:03.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.968+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8ba404b060 con 0x7f8ba80cf830 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.969+0000 7f8b9e7fc640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b8c06b4d0 0x7f8b8c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.969+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8ba40abfb0 con 0x7f8ba80cf830 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.969+0000 7f8bad238640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b8c06b4d0 0x7f8b8c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:03.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.969+0000 7f8bad238640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b8c06b4d0 0x7f8b8c06d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8b98006fd0 tx=0x7f8b98008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:03.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:03.973+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8ba407ae80 con 0x7f8ba80cf830 2024-03-06T07:32:04.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.183+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ba81615e0 con 0x7f8ba80cf830 2024-03-06T07:32:04.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.184+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/2542943082 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f8ba407a780 con 0x7f8ba80cf830 2024-03-06T07:32:04.186 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:04.186 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:04.187 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:04.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b8c06b4d0 msgr2=0x7f8b8c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b8c06b4d0 0x7f8b8c06d990 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8b98006fd0 tx=0x7f8b98008040 comp rx=0 tx=0).stop 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 msgr2=0x7f8ba81610a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 0x7f8ba81610a0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000c00 tx=0x7f8ba40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 shutdown_connections 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8b8c06b4d0 0x7f8b8c06d990 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ba815dfe0 0x7f8ba814a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ba80d27f0 0x7f8ba8149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 --2- 172.21.15.72:0/2542943082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80cf830 0x7f8ba81610a0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 >> 172.21.15.72:0/2542943082 conn(0x7f8ba80d3cd0 msgr2=0x7f8ba80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 shutdown_connections 2024-03-06T07:32:04.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:04.186+0000 7f8bafc9d640 1 -- 172.21.15.72:0/2542943082 wait complete. 2024-03-06T07:32:04.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:04.404 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:05.405 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:05.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.504+0000 7f4a55802640 1 Processor -- start 2024-03-06T07:32:05.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.504+0000 7f4a55802640 1 -- start start 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a55802640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015b9b0 0x7f4a500c7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a55802640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a500ced50 0x7f4a500cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a55802640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a500c78e0 con 0x7f4a500d1900 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a55802640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a500d0480 con 0x7f4a500ced50 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a55802640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a500cfe30 con 0x7f4a5015b9b0 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4e7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a500ced50 0x7f4a500cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4effd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015b9b0 0x7f4a500c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4f7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4a500d1900 0x7f4a500d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:37354/0 (socket says 172.21.15.72:37354) 2024-03-06T07:32:05.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4f7fe640 1 -- 172.21.15.72:0/3315505326 learned_addr learned my addr 172.21.15.72:0/3315505326 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:05.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4effd640 1 -- 172.21.15.72:0/3315505326 >> v2:172.21.15.146:3300/0 conn(0x7f4a500ced50 msgr2=0x7f4a500cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:05.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4effd640 1 --2- 172.21.15.72:0/3315505326 >> v2:172.21.15.146:3300/0 conn(0x7f4a500ced50 0x7f4a500cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4effd640 1 -- 172.21.15.72:0/3315505326 >> v1:172.21.15.146:6789/0 conn(0x7f4a500d1900 legacy=0x7f4a500d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4effd640 1 -- 172.21.15.72:0/3315505326 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500d0110 con 0x7f4a5015b9b0 2024-03-06T07:32:05.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.505+0000 7f4a4e7fc640 1 --2- 172.21.15.72:0/3315505326 >> v2:172.21.15.146:3300/0 conn(0x7f4a500ced50 0x7f4a500cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a4effd640 1 --2- 172.21.15.72:0/3315505326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015b9b0 0x7f4a500c7300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a480097a0 tx=0x7f4a4802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d68ba42ff965726f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a4dffb640 1 -- 172.21.15.72:0/3315505326 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48035020 con 0x7f4a5015b9b0 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a4dffb640 1 -- 172.21.15.72:0/3315505326 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a48031810 con 0x7f4a5015b9b0 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a4dffb640 1 -- 172.21.15.72:0/3315505326 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48031be0 con 0x7f4a5015b9b0 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 -- 172.21.15.72:0/3315505326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015b9b0 msgr2=0x7f4a500c7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 --2- 172.21.15.72:0/3315505326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015b9b0 0x7f4a500c7300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4a480097a0 tx=0x7f4a4802b8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 -- 172.21.15.72:0/3315505326 shutdown_connections 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 --2- 172.21.15.72:0/3315505326 >> v2:172.21.15.146:3300/0 conn(0x7f4a500ced50 0x7f4a500cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 --2- 172.21.15.72:0/3315505326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a5015b9b0 0x7f4a500c7300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 -- 172.21.15.72:0/3315505326 >> 172.21.15.72:0/3315505326 conn(0x7f4a5009a9f0 msgr2=0x7f4a5009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 -- 172.21.15.72:0/3315505326 shutdown_connections 2024-03-06T07:32:05.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.507+0000 7f4a55802640 1 -- 172.21.15.72:0/3315505326 wait complete. 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.509+0000 7f4a55802640 1 Processor -- start 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.509+0000 7f4a55802640 1 -- start start 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.509+0000 7f4a55802640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 0x7f4a500b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.509+0000 7f4a55802640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d1900 0x7f4a50147100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a55802640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5015b9b0 0x7f4a5015e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a55802640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a500cf330 con 0x7f4a500ced50 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a55802640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a500cf1b0 con 0x7f4a5015b9b0 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a55802640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a500cc5e0 con 0x7f4a500d1900 2024-03-06T07:32:05.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 0x7f4a500b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 0x7f4a500b6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58690/0 (socket says 172.21.15.72:58690) 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 -- 172.21.15.72:0/1626716392 learned_addr learned my addr 172.21.15.72:0/1626716392 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4e7fc640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d1900 0x7f4a50147100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4f7fe640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5015b9b0 0x7f4a5015e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 -- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d1900 msgr2=0x7f4a50147100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d1900 0x7f4a50147100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 -- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5015b9b0 msgr2=0x7f4a5015e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5015b9b0 0x7f4a5015e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 -- 172.21.15.72:0/1626716392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500b5cd0 con 0x7f4a500ced50 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a4effd640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 0x7f4a500b6d70 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f4a4802b8f0 tx=0x7f4a480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48041070 con 0x7f4a500ced50 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4803ad00 con 0x7f4a500ced50 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48035040 con 0x7f4a500ced50 2024-03-06T07:32:05.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a500b5fb0 con 0x7f4a500ced50 2024-03-06T07:32:05.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.510+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a500b6570 con 0x7f4a500ced50 2024-03-06T07:32:05.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.511+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a50068ff0 con 0x7f4a500ced50 2024-03-06T07:32:05.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.512+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4a4804c060 con 0x7f4a500ced50 2024-03-06T07:32:05.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.513+0000 7f4a47fff640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3006b500 0x7f4a3006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:05.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.513+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a480abcb0 con 0x7f4a500ced50 2024-03-06T07:32:05.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.513+0000 7f4a4e7fc640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3006b500 0x7f4a3006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:05.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.513+0000 7f4a4e7fc640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3006b500 0x7f4a3006d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009560 tx=0x7f4a38009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:05.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.517+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4a4807a4b0 con 0x7f4a500ced50 2024-03-06T07:32:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:05.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.840+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a50067ff0 con 0x7f4a500ced50 2024-03-06T07:32:05.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.841+0000 7f4a47fff640 1 -- 172.21.15.72:0/1626716392 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f4a4807e550 con 0x7f4a500ced50 2024-03-06T07:32:05.843 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:05.843 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:05.843 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:05.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3006b500 msgr2=0x7f4a3006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3006b500 0x7f4a3006d9c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009560 tx=0x7f4a38009190 comp rx=0 tx=0).stop 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 msgr2=0x7f4a500b6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 0x7f4a500b6d70 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f4a4802b8f0 tx=0x7f4a480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 shutdown_connections 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4a3006b500 0x7f4a3006d9c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a5015b9b0 0x7f4a5015e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a500d1900 0x7f4a50147100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 --2- 172.21.15.72:0/1626716392 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a500ced50 0x7f4a500b6d70 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 >> 172.21.15.72:0/1626716392 conn(0x7f4a5009a9f0 msgr2=0x7f4a5006fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 shutdown_connections 2024-03-06T07:32:05.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:05.843+0000 7f4a55802640 1 -- 172.21.15.72:0/1626716392 wait complete. 2024-03-06T07:32:06.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:06.058 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:07.059 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:07.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.158+0000 7fe0b0ec2640 1 Processor -- start 2024-03-06T07:32:07.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.158+0000 7fe0b0ec2640 1 -- start start 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0b0ec2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0ac0d22b0 0x7fe0ac0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0b0ec2640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0ac0b41f0 0x7fe0ac0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0b0ec2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0b3430 con 0x7fe0ac15dfe0 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0b0ec2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac0d2c50 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0b0ec2640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac0b31e0 con 0x7fe0ac0b41f0 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0ac0d22b0 0x7fe0ac0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe0ac0d22b0 0x7fe0ac0d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58692/0 (socket says 172.21.15.72:58692) 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 -- 172.21.15.72:0/1757610407 learned_addr learned my addr 172.21.15.72:0/1757610407 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:07.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 -- 172.21.15.72:0/1757610407 >> v2:172.21.15.146:3300/0 conn(0x7fe0ac0b41f0 msgr2=0x7fe0ac0b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 --2- 172.21.15.72:0/1757610407 >> v2:172.21.15.146:3300/0 conn(0x7fe0ac0b41f0 0x7fe0ac0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 -- 172.21.15.72:0/1757610407 >> v1:172.21.15.146:6789/0 conn(0x7fe0ac15dfe0 legacy=0x7fe0ac0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.159+0000 7fe0a3fff640 1 -- 172.21.15.72:0/1757610407 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac0b3710 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.160+0000 7fe0a3fff640 1 --2- 172.21.15.72:0/1757610407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0d2710 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fe094009870 tx=0x7fe09402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dda3a9e2d152220 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.160+0000 7fe0a9d9b640 1 -- 172.21.15.72:0/1757610407 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094035020 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.160+0000 7fe0a9d9b640 1 -- 172.21.15.72:0/1757610407 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe094031870 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.160+0000 7fe0a9d9b640 1 -- 172.21.15.72:0/1757610407 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094031c40 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.160+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/1757610407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 msgr2=0x7fe0ac0d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.160+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/1757610407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0d2710 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fe094009870 tx=0x7fe09402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.161+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/1757610407 shutdown_connections 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.161+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/1757610407 >> v2:172.21.15.146:3300/0 conn(0x7fe0ac0b41f0 0x7fe0ac0b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.161+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/1757610407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0d2710 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.161+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/1757610407 >> 172.21.15.72:0/1757610407 conn(0x7fe0ac0d3cd0 msgr2=0x7fe0ac0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.161+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/1757610407 shutdown_connections 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.161+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/1757610407 wait complete. 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.162+0000 7fe0b0ec2640 1 Processor -- start 2024-03-06T07:32:07.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.162+0000 7fe0b0ec2640 1 -- start start 2024-03-06T07:32:07.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0ac0b41f0 0x7fe0ac156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0ac15dfe0 0x7fe0ac0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0cfa00 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0b3d60 con 0x7fe0ac0b41f0 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0d1c70 con 0x7fe0ac15dfe0 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58702/0 (socket says 172.21.15.72:58702) 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 -- 172.21.15.72:0/3308195677 learned_addr learned my addr 172.21.15.72:0/3308195677 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 -- 172.21.15.72:0/3308195677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0ac15dfe0 msgr2=0x7fe0ac0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0ac15dfe0 0x7fe0ac0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 -- 172.21.15.72:0/3308195677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0ac0b41f0 msgr2=0x7fe0ac156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0ac0b41f0 0x7fe0ac156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 -- 172.21.15.72:0/3308195677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac0cc960 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a3fff640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0cbdf0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fe094009870 tx=0x7fe0940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09403ad60 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0940443f0 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac0ccc40 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0940447e0 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.163+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac0cd200 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.164+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac04f330 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.165+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe09404b060 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.165+0000 7fe0a2ffd640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe08c06b4d0 0x7fe08c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:07.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.165+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe0940ac450 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.165+0000 7fe0aa59c640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe08c06b4d0 0x7fe08c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:07.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.166+0000 7fe0aa59c640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe08c06b4d0 0x7fe08c06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009650 tx=0x7fe09c006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:07.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.169+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe09403f070 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.380+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0ac15d4c0 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.381+0000 7fe0a2ffd640 1 -- 172.21.15.72:0/3308195677 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fe09402f320 con 0x7fe0ac0d22b0 2024-03-06T07:32:07.383 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:07.383 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:07.384 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe08c06b4d0 msgr2=0x7fe08c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe08c06b4d0 0x7fe08c06d990 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009650 tx=0x7fe09c006ba0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 msgr2=0x7fe0ac0cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0cbdf0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fe094009870 tx=0x7fe0940046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 shutdown_connections 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe08c06b4d0 0x7fe08c06d990 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0ac15dfe0 0x7fe0ac0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0ac0d22b0 0x7fe0ac0cbdf0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 --2- 172.21.15.72:0/3308195677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0ac0b41f0 0x7fe0ac156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 >> 172.21.15.72:0/3308195677 conn(0x7fe0ac0d3cd0 msgr2=0x7fe0ac06cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:07.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 shutdown_connections 2024-03-06T07:32:07.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:07.384+0000 7fe0b0ec2640 1 -- 172.21.15.72:0/3308195677 wait complete. 2024-03-06T07:32:07.598 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:07.598 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:07.831 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:08.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:08.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.697+0000 7f887d084640 1 Processor -- start 2024-03-06T07:32:08.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.697+0000 7f887d084640 1 -- start start 2024-03-06T07:32:08.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.697+0000 7f887d084640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 0x7f88780afb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.697+0000 7f887d084640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780b4e30 con 0x7f887815ab00 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887d084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780b5110 con 0x7f887815bb10 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887d084640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88780b0060 con 0x7f88780af6c0 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f8876d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f887815ab00 0x7f88780b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45028/0 (socket says 172.21.15.72:45028) 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f8876d9d640 1 -- 172.21.15.72:0/194521756 learned_addr learned my addr 172.21.15.72:0/194521756 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f886e59c640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 0x7f88780afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158449856 0 0) 0x7f88780b4e30 con 0x7f887815ab00 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8864003520 con 0x7f887815ab00 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2276496418 0 0) 0x7f8864003520 con 0x7f887815ab00 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 msgr2=0x7f88780afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 0x7f88780afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 >> v1:172.21.15.146:6789/0 conn(0x7f887815bb10 legacy=0x7f887815bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887815b310 con 0x7f887815ab00 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f8868001670 con 0x7f887815ab00 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 >> v1:172.21.15.72:6789/0 conn(0x7f887815ab00 legacy=0x7f88780b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88640039e0 0x7f8864003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 0x7f8864004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8864004d40 0x7f88640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886400b9c0 con 0x7f88640039e0 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886400bb40 con 0x7f8864004d40 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886400bd40 con 0x7f88780af6c0 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.698+0000 7f8876d9d640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88640039e0 0x7f8864003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f886e59c640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 0x7f8864004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f887759e640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8864004d40 0x7f88640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f8876d9d640 1 -- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 msgr2=0x7f8864004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f8876d9d640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88780af6c0 0x7f8864004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f8876d9d640 1 -- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8864004d40 msgr2=0x7f88640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f8876d9d640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8864004d40 0x7f88640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f8876d9d640 1 -- 172.21.15.72:0/194521756 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864003520 con 0x7f88640039e0 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.699+0000 7f8876d9d640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88640039e0 0x7f8864003dc0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f886800c240 tx=0x7f8864005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c78975e3052e4bd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868007d10 con 0x7f88640039e0 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8868030070 con 0x7f88640039e0 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887659c640 1 -- 172.21.15.72:0/194521756 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868005d30 con 0x7f88640039e0 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887d084640 1 -- 172.21.15.72:0/194521756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88640039e0 msgr2=0x7f8864003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887d084640 1 --2- 172.21.15.72:0/194521756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88640039e0 0x7f8864003dc0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f886800c240 tx=0x7f8864005830 comp rx=0 tx=0).stop 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887759e640 1 -- 172.21.15.72:0/194521756 reap_dead start 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887d084640 1 -- 172.21.15.72:0/194521756 shutdown_connections 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887d084640 1 -- 172.21.15.72:0/194521756 >> 172.21.15.72:0/194521756 conn(0x7f887806c0c0 msgr2=0x7f887806c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887d084640 1 -- 172.21.15.72:0/194521756 shutdown_connections 2024-03-06T07:32:08.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.700+0000 7f887d084640 1 -- 172.21.15.72:0/194521756 wait complete. 2024-03-06T07:32:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.701+0000 7f887d084640 1 Processor -- start 2024-03-06T07:32:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 -- start start 2024-03-06T07:32:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88640039e0 0x7f8878157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88780cffb0 0x7f8878154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 0x7f887814f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8878071b20 con 0x7f88780d0430 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88780b4ac0 con 0x7f88780cffb0 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887d084640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8878071ca0 con 0x7f88640039e0 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 0x7f887814f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887759e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 0x7f887814f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58732/0 (socket says 172.21.15.72:58732) 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.702+0000 7f887759e640 1 -- 172.21.15.72:0/3350655214 learned_addr learned my addr 172.21.15.72:0/3350655214 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f8876d9d640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88640039e0 0x7f8878157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f886e59c640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88780cffb0 0x7f8878154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887759e640 1 -- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88640039e0 msgr2=0x7f8878157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887759e640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88640039e0 0x7f8878157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887759e640 1 -- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88780cffb0 msgr2=0x7f8878154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887759e640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88780cffb0 0x7f8878154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887759e640 1 -- 172.21.15.72:0/3350655214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780ad860 con 0x7f88780d0430 2024-03-06T07:32:08.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887759e640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 0x7f887814f4b0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f887815bef0 tx=0x7f887000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870003970 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8870018070 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88780adb40 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8870003d70 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.703+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88780ae100 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.704+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887804f330 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.705+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f88700050d0 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.705+0000 7f886ffff640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f885806b4d0 0x7f885806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.705+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8870089eb0 con 0x7f88780d0430 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.705+0000 7f8876d9d640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f885806b4d0 0x7f885806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:08.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.706+0000 7f8876d9d640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f885806b4d0 0x7f885806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8868034000 tx=0x7f8868007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:08.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.714+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8870058730 con 0x7f88780d0430 2024-03-06T07:32:08.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.929+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f887815d6c0 con 0x7f88780d0430 2024-03-06T07:32:08.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.930+0000 7f886ffff640 1 -- 172.21.15.72:0/3350655214 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f887005c7d0 con 0x7f88780d0430 2024-03-06T07:32:08.932 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:08.932 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:08.933 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.932+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f885806b4d0 msgr2=0x7f885806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.932+0000 7f887d084640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f885806b4d0 0x7f885806d990 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f8868034000 tx=0x7f8868007000 comp rx=0 tx=0).stop 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.932+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 msgr2=0x7f887814f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.932+0000 7f887d084640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 0x7f887814f4b0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f887815bef0 tx=0x7f887000eb90 comp rx=0 tx=0).stop 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 shutdown_connections 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f885806b4d0 0x7f885806d990 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88780d0430 0x7f887814f4b0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88780cffb0 0x7f8878154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 --2- 172.21.15.72:0/3350655214 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88640039e0 0x7f8878157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:08.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 >> 172.21.15.72:0/3350655214 conn(0x7f887806c0c0 msgr2=0x7f88780b1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:08.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 shutdown_connections 2024-03-06T07:32:08.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:08.933+0000 7f887d084640 1 -- 172.21.15.72:0/3350655214 wait complete. 2024-03-06T07:32:09.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:09.147 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:09.833 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:10.148 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:10.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.248+0000 7f73985ac640 1 Processor -- start 2024-03-06T07:32:10.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.248+0000 7f73985ac640 1 -- start start 2024-03-06T07:32:10.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.249+0000 7f73985ac640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 0x7f73900cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.249+0000 7f73985ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f73900b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.249+0000 7f73985ac640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73900b3430 con 0x7f73900cf830 2024-03-06T07:32:10.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.249+0000 7f73985ac640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73900b31e0 con 0x7f739015dfe0 2024-03-06T07:32:10.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.249+0000 7f73985ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73900b3d60 con 0x7f73900d29c0 2024-03-06T07:32:10.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f73900b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396348640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 0x7f73900cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f73900b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.72:55342/0 (socket says 172.21.15.72:55342) 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396348640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 0x7f73900cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.72:33290/0 (socket says 172.21.15.72:33290) 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7395b47640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f73900cf830 0x7f73900d22b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:37378/0 (socket says 172.21.15.72:37378) 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 -- 172.21.15.72:0/1898193798 learned_addr learned my addr 172.21.15.72:0/1898193798 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 -- 172.21.15.72:0/1898193798 >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 msgr2=0x7f73900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 --2- 172.21.15.72:0/1898193798 >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 0x7f73900cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 -- 172.21.15.72:0/1898193798 >> v1:172.21.15.146:6789/0 conn(0x7f73900cf830 legacy=0x7f73900d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396b49640 1 -- 172.21.15.72:0/1898193798 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73900b3710 con 0x7f73900d29c0 2024-03-06T07:32:10.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.250+0000 7f7396348640 1 --2- 172.21.15.72:0/1898193798 >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 0x7f73900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.251+0000 7f7396b49640 1 --2- 172.21.15.72:0/1898193798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f73900b41f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7388009e50 tx=0x7f738802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d77cb57f9af022e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.251+0000 7f7395346640 1 -- 172.21.15.72:0/1898193798 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738802cb00 con 0x7f73900d29c0 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.251+0000 7f7395346640 1 -- 172.21.15.72:0/1898193798 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738802cca0 con 0x7f73900d29c0 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.251+0000 7f7395346640 1 -- 172.21.15.72:0/1898193798 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388034610 con 0x7f73900d29c0 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 -- 172.21.15.72:0/1898193798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 msgr2=0x7f73900b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 --2- 172.21.15.72:0/1898193798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f73900b41f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7388009e50 tx=0x7f738802bf60 comp rx=0 tx=0).stop 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 -- 172.21.15.72:0/1898193798 shutdown_connections 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 --2- 172.21.15.72:0/1898193798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f73900b41f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 --2- 172.21.15.72:0/1898193798 >> v2:172.21.15.146:3300/0 conn(0x7f739015dfe0 0x7f73900cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 -- 172.21.15.72:0/1898193798 >> 172.21.15.72:0/1898193798 conn(0x7f73900d3cd0 msgr2=0x7f73900d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 -- 172.21.15.72:0/1898193798 shutdown_connections 2024-03-06T07:32:10.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.252+0000 7f73985ac640 1 -- 172.21.15.72:0/1898193798 wait complete. 2024-03-06T07:32:10.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.253+0000 7f73985ac640 1 Processor -- start 2024-03-06T07:32:10.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.253+0000 7f73985ac640 1 -- start start 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.253+0000 7f73985ac640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900cf830 0x7f73900b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f7390160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 0x7f7390150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73900d1df0 con 0x7f739015dfe0 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73900d1c70 con 0x7f73900cf830 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f739015e3c0 con 0x7f73900d29c0 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 0x7f7390150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 0x7f7390150c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58736/0 (socket says 172.21.15.72:58736) 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 -- 172.21.15.72:0/3849948310 learned_addr learned my addr 172.21.15.72:0/3849948310 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396348640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900cf830 0x7f73900b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7395b47640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f7390160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 -- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 msgr2=0x7f7390160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f7390160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 -- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900cf830 msgr2=0x7f73900b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900cf830 0x7f73900b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 -- 172.21.15.72:0/3849948310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73900cada0 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7395b47640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f7390160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7396b49640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 0x7f7390150c10 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f7388009f70 tx=0x7f7388004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388043020 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738802ce50 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7388009510 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73900cb080 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.254+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73900cb920 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.255+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739004f330 con 0x7f739015dfe0 2024-03-06T07:32:10.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.256+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7388046020 con 0x7f739015dfe0 2024-03-06T07:32:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.257+0000 7f7386ffd640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f737406b4d0 0x7f737406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.257+0000 7f7396348640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f737406b4d0 0x7f737406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.257+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f73880ac7a0 con 0x7f739015dfe0 2024-03-06T07:32:10.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.257+0000 7f7396348640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f737406b4d0 0x7f737406d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f738c009560 tx=0x7f738c009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:10.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.261+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f738804d030 con 0x7f739015dfe0 2024-03-06T07:32:10.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.481+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73900b09c0 con 0x7f739015dfe0 2024-03-06T07:32:10.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.481+0000 7f7386ffd640 1 -- 172.21.15.72:0/3849948310 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f738807b560 con 0x7f739015dfe0 2024-03-06T07:32:10.482 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:10.483 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:10.483 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:10.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f737406b4d0 msgr2=0x7f737406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f737406b4d0 0x7f737406d990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f738c009560 tx=0x7f738c009190 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 msgr2=0x7f7390150c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 0x7f7390150c10 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f7388009f70 tx=0x7f7388004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 shutdown_connections 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f737406b4d0 0x7f737406d990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f739015dfe0 0x7f7390150c10 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f73900d29c0 0x7f7390160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 --2- 172.21.15.72:0/3849948310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f73900cf830 0x7f73900b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 >> 172.21.15.72:0/3849948310 conn(0x7f73900d3cd0 msgr2=0x7f73900d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 shutdown_connections 2024-03-06T07:32:10.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:10.483+0000 7f73985ac640 1 -- 172.21.15.72:0/3849948310 wait complete. 2024-03-06T07:32:10.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:10.697 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:11.698 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:11.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.799+0000 7fec65d7a640 1 Processor -- start 2024-03-06T07:32:11.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.799+0000 7fec65d7a640 1 -- start start 2024-03-06T07:32:11.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec65d7a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec600d22b0 0x7fec600d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec65d7a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600b3430 con 0x7fec600b41f0 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec65d7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec600b3710 con 0x7fec6015dfe0 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec65d7a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec600d2c50 con 0x7fec600d22b0 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec600d22b0 0x7fec600d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec600d22b0 0x7fec600d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58738/0 (socket says 172.21.15.72:58738) 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5effd640 1 -- 172.21.15.72:0/642559186 learned_addr learned my addr 172.21.15.72:0/642559186 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fec600b41f0 0x7fec600b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45038/0 (socket says 172.21.15.72:45038) 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5effd640 1 -- 172.21.15.72:0/642559186 >> v1:172.21.15.146:6789/0 conn(0x7fec6015dfe0 legacy=0x7fec600cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5effd640 1 -- 172.21.15.72:0/642559186 >> v1:172.21.15.72:6789/0 conn(0x7fec600b41f0 legacy=0x7fec600b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5effd640 1 -- 172.21.15.72:0/642559186 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600b39f0 con 0x7fec600d22b0 2024-03-06T07:32:11.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.800+0000 7fec5e7fc640 1 -- 172.21.15.72:0/642559186 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880085443 0 0) 0x7fec600b3430 con 0x7fec600b41f0 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.801+0000 7fec5effd640 1 --2- 172.21.15.72:0/642559186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec600d2710 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fec540098a0 tx=0x7fec5402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d8c7d31eed10d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.801+0000 7fec5e7fc640 1 -- 172.21.15.72:0/642559186 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5402f840 con 0x7fec600d22b0 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.801+0000 7fec5e7fc640 1 -- 172.21.15.72:0/642559186 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5402f9e0 con 0x7fec600d22b0 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.801+0000 7fec5e7fc640 1 -- 172.21.15.72:0/642559186 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5402fdb0 con 0x7fec600d22b0 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 -- 172.21.15.72:0/642559186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 msgr2=0x7fec600d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 --2- 172.21.15.72:0/642559186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec600d2710 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fec540098a0 tx=0x7fec5402ba20 comp rx=0 tx=0).stop 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 -- 172.21.15.72:0/642559186 shutdown_connections 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 --2- 172.21.15.72:0/642559186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec600d2710 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 -- 172.21.15.72:0/642559186 >> 172.21.15.72:0/642559186 conn(0x7fec600d3cd0 msgr2=0x7fec600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 -- 172.21.15.72:0/642559186 shutdown_connections 2024-03-06T07:32:11.804 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.802+0000 7fec65d7a640 1 -- 172.21.15.72:0/642559186 wait complete. 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.803+0000 7fec65d7a640 1 Processor -- start 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.803+0000 7fec65d7a640 1 -- start start 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec65d7a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec600b41f0 0x7fec600b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec65d7a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec601510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec65d7a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec6015dfe0 0x7fec600b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec65d7a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec600b3d60 con 0x7fec600d22b0 2024-03-06T07:32:11.806 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec65d7a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec600b31e0 con 0x7fec6015dfe0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec65d7a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec600d1c70 con 0x7fec600b41f0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec601510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec601510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58740/0 (socket says 172.21.15.72:58740) 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 -- 172.21.15.72:0/2027308522 learned_addr learned my addr 172.21.15.72:0/2027308522 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 -- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec600b41f0 msgr2=0x7fec600b68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5ffff640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec6015dfe0 0x7fec600b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec600b41f0 0x7fec600b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 -- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec6015dfe0 msgr2=0x7fec600b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec6015dfe0 0x7fec600b8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 -- 172.21.15.72:0/2027308522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec600b9380 con 0x7fec600d22b0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.804+0000 7fec5effd640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec601510b0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fec540090b0 tx=0x7fec54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.805+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54047070 con 0x7fec600d22b0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.805+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec54037cc0 con 0x7fec600d22b0 2024-03-06T07:32:11.807 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.805+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec600b9660 con 0x7fec600d22b0 2024-03-06T07:32:11.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.805+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54004270 con 0x7fec600d22b0 2024-03-06T07:32:11.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.805+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec600b6e10 con 0x7fec600d22b0 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.806+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6004f330 con 0x7fec600d22b0 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.807+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fec5404a070 con 0x7fec600d22b0 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.807+0000 7fec47fff640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec4c06b4d0 0x7fec4c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.807+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fec540ac370 con 0x7fec600d22b0 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.807+0000 7fec5f7fe640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec4c06b4d0 0x7fec4c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:11.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.808+0000 7fec5f7fe640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec4c06b4d0 0x7fec4c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fec50000f60 tx=0x7fec500079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:11.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:11.812+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fec54033030 con 0x7fec600d22b0 2024-03-06T07:32:11.834 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:12.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.024+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec600cb370 con 0x7fec600d22b0 2024-03-06T07:32:12.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.025+0000 7fec47fff640 1 -- 172.21.15.72:0/2027308522 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fec5407b1b0 con 0x7fec600d22b0 2024-03-06T07:32:12.026 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:12.026 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:12.026 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec4c06b4d0 msgr2=0x7fec4c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec4c06b4d0 0x7fec4c06d990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fec50000f60 tx=0x7fec500079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 msgr2=0x7fec601510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec601510b0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fec540090b0 tx=0x7fec54004730 comp rx=0 tx=0).stop 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 shutdown_connections 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fec4c06b4d0 0x7fec4c06d990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec6015dfe0 0x7fec600b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec600d22b0 0x7fec601510b0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 --2- 172.21.15.72:0/2027308522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec600b41f0 0x7fec600b68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:12.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 >> 172.21.15.72:0/2027308522 conn(0x7fec600d3cd0 msgr2=0x7fec600d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:12.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 shutdown_connections 2024-03-06T07:32:12.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:12.027+0000 7fec65d7a640 1 -- 172.21.15.72:0/2027308522 wait complete. 2024-03-06T07:32:12.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:12.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:13.241 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:13.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.340+0000 7f3d90c2a640 1 Processor -- start 2024-03-06T07:32:13.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.340+0000 7f3d90c2a640 1 -- start start 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d90c2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c1515e0 0x7f3d8c0cce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d90c2a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d8c15b430 0x7f3d8c071830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d90c2a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8c15b8b0 con 0x7f3d8c0cd7a0 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d90c2a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8c070a70 con 0x7f3d8c15b430 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d90c2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c070d50 con 0x7f3d8c1515e0 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d89d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d8c15b430 0x7f3d8c071830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d89d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d8c15b430 0x7f3d8c071830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55360/0 (socket says 172.21.15.72:55360) 2024-03-06T07:32:13.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.341+0000 7f3d89d9b640 1 -- 172.21.15.72:0/2498793488 learned_addr learned my addr 172.21.15.72:0/2498793488 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:13.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.342+0000 7f3d8a59c640 1 --2- 172.21.15.72:0/2498793488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c1515e0 0x7f3d8c0cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.342+0000 7f3d89d9b640 1 -- 172.21.15.72:0/2498793488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c1515e0 msgr2=0x7f3d8c0cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:13.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.342+0000 7f3d89d9b640 1 --2- 172.21.15.72:0/2498793488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c1515e0 0x7f3d8c0cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.342+0000 7f3d89d9b640 1 -- 172.21.15.72:0/2498793488 >> v1:172.21.15.146:6789/0 conn(0x7f3d8c0cd7a0 legacy=0x7f3d8c0cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.342+0000 7f3d89d9b640 1 -- 172.21.15.72:0/2498793488 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c071030 con 0x7f3d8c15b430 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.342+0000 7f3d89d9b640 1 --2- 172.21.15.72:0/2498793488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c15b430 0x7f3d8c071830 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000d150 tx=0x7f3d8002e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3f4f4c45e482ff server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d8959a640 1 -- 172.21.15.72:0/2498793488 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80034070 con 0x7f3d8c15b430 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d8959a640 1 -- 172.21.15.72:0/2498793488 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80038400 con 0x7f3d8c15b430 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d8959a640 1 -- 172.21.15.72:0/2498793488 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d800387d0 con 0x7f3d8c15b430 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 -- 172.21.15.72:0/2498793488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c15b430 msgr2=0x7f3d8c071830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/2498793488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c15b430 0x7f3d8c071830 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000d150 tx=0x7f3d8002e160 comp rx=0 tx=0).stop 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 -- 172.21.15.72:0/2498793488 shutdown_connections 2024-03-06T07:32:13.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/2498793488 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c15b430 0x7f3d8c071830 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/2498793488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c1515e0 0x7f3d8c0cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 -- 172.21.15.72:0/2498793488 >> 172.21.15.72:0/2498793488 conn(0x7f3d8c0cf8a0 msgr2=0x7f3d8c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 -- 172.21.15.72:0/2498793488 shutdown_connections 2024-03-06T07:32:13.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.343+0000 7f3d90c2a640 1 -- 172.21.15.72:0/2498793488 wait complete. 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 Processor -- start 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 -- start start 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c0cd7a0 0x7f3d8c146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 0x7f3d8c14c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d8c15b430 0x7f3d8c14ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c071ef0 con 0x7f3d8c1515e0 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c071d70 con 0x7f3d8c15b430 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.345+0000 7f3d90c2a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0cc3a0 con 0x7f3d8c0cd7a0 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 0x7f3d8c14c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 0x7f3d8c14c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55372/0 (socket says 172.21.15.72:55372) 2024-03-06T07:32:13.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 -- 172.21.15.72:0/1861818309 learned_addr learned my addr 172.21.15.72:0/1861818309 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d8a59c640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c0cd7a0 0x7f3d8c146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d8ad9d640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d8c15b430 0x7f3d8c14ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 -- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c0cd7a0 msgr2=0x7f3d8c146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c0cd7a0 0x7f3d8c146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 -- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d8c15b430 msgr2=0x7f3d8c14ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d8c15b430 0x7f3d8c14ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 -- 172.21.15.72:0/1861818309 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c1efef0 con 0x7f3d8c1515e0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d89d9b640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 0x7f3d8c14c4e0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f3d8003a040 tx=0x7f3d80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80047070 con 0x7f3d8c1515e0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80037970 con 0x7f3d8c1515e0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c1f0190 con 0x7f3d8c1515e0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8000b720 con 0x7f3d8c1515e0 2024-03-06T07:32:13.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.346+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c1f0750 con 0x7f3d8c1515e0 2024-03-06T07:32:13.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.347+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c068840 con 0x7f3d8c1515e0 2024-03-06T07:32:13.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.348+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3d8002ed10 con 0x7f3d8c1515e0 2024-03-06T07:32:13.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.349+0000 7f3d7affd640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:13.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.349+0000 7f3d8a59c640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:13.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.349+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3d800ae650 con 0x7f3d8c1515e0 2024-03-06T07:32:13.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.349+0000 7f3d8a59c640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3d84009b40 tx=0x7f3d84009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:13.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.353+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f3d8007ce40 con 0x7f3d8c1515e0 2024-03-06T07:32:13.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.644+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d8c0b4c30 con 0x7f3d8c1515e0 2024-03-06T07:32:13.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.644+0000 7f3d7affd640 1 -- 172.21.15.72:0/1861818309 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f3d80080ee0 con 0x7f3d8c1515e0 2024-03-06T07:32:13.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:13.647 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:13.647 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:13.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d6c06b4d0 msgr2=0x7f3d6c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3d84009b40 tx=0x7f3d84009190 comp rx=0 tx=0).stop 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 msgr2=0x7f3d8c14c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 0x7f3d8c14c4e0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f3d8003a040 tx=0x7f3d80004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 shutdown_connections 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f3d6c06b4d0 0x7f3d6c06d990 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d8c15b430 0x7f3d8c14ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d8c1515e0 0x7f3d8c14c4e0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 --2- 172.21.15.72:0/1861818309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d8c0cd7a0 0x7f3d8c146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 >> 172.21.15.72:0/1861818309 conn(0x7f3d8c0cf8a0 msgr2=0x7f3d8c0ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 shutdown_connections 2024-03-06T07:32:13.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:13.646+0000 7f3d90c2a640 1 -- 172.21.15.72:0/1861818309 wait complete. 2024-03-06T07:32:13.836 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:13.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:13.860 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:14.861 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:14.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.962+0000 7f37c3583640 1 Processor -- start 2024-03-06T07:32:14.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.962+0000 7f37c3583640 1 -- start start 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c3583640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37bc15bb20 0x7f37bc0d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c3583640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0cfff0 0x7f37bc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c3583640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f37bc0ce4b0 0x7f37bc0ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c3583640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37bc161980 con 0x7f37bc15bb20 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c3583640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37bc161c60 con 0x7f37bc0ce4b0 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c3583640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc161f40 con 0x7f37bc0cfff0 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37bc15bb20 0x7f37bc0d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37bc15bb20 0x7f37bc0d1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55378/0 (socket says 172.21.15.72:55378) 2024-03-06T07:32:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 -- 172.21.15.72:0/3974675380 learned_addr learned my addr 172.21.15.72:0/3974675380 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c1b20640 1 --2- 172.21.15.72:0/3974675380 >> v2:172.21.15.146:3300/0 conn(0x7f37bc0ce4b0 0x7f37bc0ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c0b1e640 1 --2- 172.21.15.72:0/3974675380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0cfff0 0x7f37bc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 -- 172.21.15.72:0/3974675380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0cfff0 msgr2=0x7f37bc0d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 --2- 172.21.15.72:0/3974675380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0cfff0 0x7f37bc0d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 -- 172.21.15.72:0/3974675380 >> v2:172.21.15.146:3300/0 conn(0x7f37bc0ce4b0 msgr2=0x7f37bc0ce940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 --2- 172.21.15.72:0/3974675380 >> v2:172.21.15.146:3300/0 conn(0x7f37bc0ce4b0 0x7f37bc0ce940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.963+0000 7f37c131f640 1 -- 172.21.15.72:0/3974675380 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc162220 con 0x7f37bc15bb20 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.964+0000 7f37c131f640 1 --2- 172.21.15.72:0/3974675380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15bb20 0x7f37bc0d1df0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0b13b0 tx=0x7f37b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16341e7c96aaddea server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.964+0000 7f37b3fff640 1 -- 172.21.15.72:0/3974675380 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8035020 con 0x7f37bc15bb20 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.964+0000 7f37b3fff640 1 -- 172.21.15.72:0/3974675380 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37b8031870 con 0x7f37bc15bb20 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.964+0000 7f37b3fff640 1 -- 172.21.15.72:0/3974675380 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b8031c40 con 0x7f37bc15bb20 2024-03-06T07:32:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.964+0000 7f37c3583640 1 -- 172.21.15.72:0/3974675380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15bb20 msgr2=0x7f37bc0d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 --2- 172.21.15.72:0/3974675380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15bb20 0x7f37bc0d1df0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0b13b0 tx=0x7f37b802b910 comp rx=0 tx=0).stop 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 -- 172.21.15.72:0/3974675380 shutdown_connections 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 --2- 172.21.15.72:0/3974675380 >> v2:172.21.15.146:3300/0 conn(0x7f37bc0ce4b0 0x7f37bc0ce940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 --2- 172.21.15.72:0/3974675380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0cfff0 0x7f37bc0d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 --2- 172.21.15.72:0/3974675380 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc15bb20 0x7f37bc0d1df0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 -- 172.21.15.72:0/3974675380 >> 172.21.15.72:0/3974675380 conn(0x7f37bc09a9f0 msgr2=0x7f37bc09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 -- 172.21.15.72:0/3974675380 shutdown_connections 2024-03-06T07:32:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.965+0000 7f37c3583640 1 -- 172.21.15.72:0/3974675380 wait complete. 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.966+0000 7f37c3583640 1 Processor -- start 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.966+0000 7f37c3583640 1 -- start start 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.966+0000 7f37c3583640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0ce4b0 0x7f37bc0cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.966+0000 7f37c3583640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 0x7f37bc15edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c3583640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc15bb20 0x7f37bc0c98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c3583640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc0d24b0 con 0x7f37bc0cfff0 2024-03-06T07:32:14.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c3583640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc0d2330 con 0x7f37bc15bb20 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c3583640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc0d1980 con 0x7f37bc0ce4b0 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 0x7f37bc15edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 0x7f37bc15edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55392/0 (socket says 172.21.15.72:55392) 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 -- 172.21.15.72:0/2326569303 learned_addr learned my addr 172.21.15.72:0/2326569303 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c131f640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0ce4b0 0x7f37bc0cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 -- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0ce4b0 msgr2=0x7f37bc0cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0ce4b0 0x7f37bc0cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c1b20640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc15bb20 0x7f37bc0c98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 -- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc15bb20 msgr2=0x7f37bc0c98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc15bb20 0x7f37bc0c98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 -- 172.21.15.72:0/2326569303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc0c5690 con 0x7f37bc0cfff0 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c0b1e640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 0x7f37bc15edd0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00d800 tx=0x7f37ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac00b6c0 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ac019070 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc0c5970 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.967+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ac00ba80 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.968+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc0c5f30 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.968+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc0c6410 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.969+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f37ac00bc20 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.969+0000 7f37b1ffb640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.969+0000 7f37c131f640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.969+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f37ac08a710 con 0x7f37bc0cfff0 2024-03-06T07:32:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.970+0000 7f37c131f640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f37b8036000 tx=0x7f37b8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:14.974+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f37bc0c6410 con 0x7f37bc0cfff0 2024-03-06T07:32:15.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.186+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37bc04f330 con 0x7f37bc0cfff0 2024-03-06T07:32:15.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.186+0000 7f37b1ffb640 1 -- 172.21.15.72:0/2326569303 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f37ac059560 con 0x7f37bc0cfff0 2024-03-06T07:32:15.189 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:15.189 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:15.189 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:15.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 msgr2=0x7f37a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f37b8036000 tx=0x7f37b8008030 comp rx=0 tx=0).stop 2024-03-06T07:32:15.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 msgr2=0x7f37bc15edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 0x7f37bc15edd0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f37ac00d800 tx=0x7f37ac00dcc0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 shutdown_connections 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f37a006b4d0 0x7f37a006d990 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37bc15bb20 0x7f37bc0c98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37bc0cfff0 0x7f37bc15edd0 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 --2- 172.21.15.72:0/2326569303 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37bc0ce4b0 0x7f37bc0cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.188+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 >> 172.21.15.72:0/2326569303 conn(0x7f37bc09a9f0 msgr2=0x7f37bc15c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.189+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 shutdown_connections 2024-03-06T07:32:15.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:15.189+0000 7f37c3583640 1 -- 172.21.15.72:0/2326569303 wait complete. 2024-03-06T07:32:15.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:15.403 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:15.837 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:16.404 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 Processor -- start 2024-03-06T07:32:16.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 -- start start 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15dfe0 0x7f2c7c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c7c0d29c0 0x7f2c7c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7c0b3430 con 0x7f2c7c0cf830 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c0b31e0 con 0x7f2c7c0d29c0 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c82d13640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c7c0b3d60 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.503+0000 7f2c812b0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c7c0d29c0 0x7f2c7c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c812b0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2c7c0d29c0 0x7f2c7c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55406/0 (socket says 172.21.15.72:55406) 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c812b0640 1 -- 172.21.15.72:0/2688653223 learned_addr learned my addr 172.21.15.72:0/2688653223 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c80aaf640 1 --2- 172.21.15.72:0/2688653223 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15dfe0 0x7f2c7c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c812b0640 1 -- 172.21.15.72:0/2688653223 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15dfe0 msgr2=0x7f2c7c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:16.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c812b0640 1 --2- 172.21.15.72:0/2688653223 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15dfe0 0x7f2c7c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c812b0640 1 -- 172.21.15.72:0/2688653223 >> v1:172.21.15.72:6789/0 conn(0x7f2c7c0cf830 legacy=0x7f2c7c0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c812b0640 1 -- 172.21.15.72:0/2688653223 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c0b3710 con 0x7f2c7c0d29c0 2024-03-06T07:32:16.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.504+0000 7f2c737fe640 1 -- 172.21.15.72:0/2688653223 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262457408 0 0) 0x7f2c7c0b3430 con 0x7f2c7c0cf830 2024-03-06T07:32:16.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c812b0640 1 --2- 172.21.15.72:0/2688653223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0b41f0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c067990 tx=0x7f2c7402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827d8cc5775612fc server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c737fe640 1 -- 172.21.15.72:0/2688653223 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7402cb20 con 0x7f2c7c0d29c0 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c737fe640 1 -- 172.21.15.72:0/2688653223 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7402ccc0 con 0x7f2c7c0d29c0 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c737fe640 1 -- 172.21.15.72:0/2688653223 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74034610 con 0x7f2c7c0d29c0 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 -- 172.21.15.72:0/2688653223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c0d29c0 msgr2=0x7f2c7c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 --2- 172.21.15.72:0/2688653223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0b41f0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c067990 tx=0x7f2c7402bf80 comp rx=0 tx=0).stop 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 -- 172.21.15.72:0/2688653223 shutdown_connections 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 --2- 172.21.15.72:0/2688653223 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0b41f0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 --2- 172.21.15.72:0/2688653223 >> v2:172.21.15.146:3300/0 conn(0x7f2c7c15dfe0 0x7f2c7c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 -- 172.21.15.72:0/2688653223 >> 172.21.15.72:0/2688653223 conn(0x7f2c7c0d3cd0 msgr2=0x7f2c7c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 -- 172.21.15.72:0/2688653223 shutdown_connections 2024-03-06T07:32:16.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.505+0000 7f2c82d13640 1 -- 172.21.15.72:0/2688653223 wait complete. 2024-03-06T07:32:16.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.506+0000 7f2c82d13640 1 Processor -- start 2024-03-06T07:32:16.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.506+0000 7f2c82d13640 1 -- start start 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c82d13640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c0cf830 0x7f2c7c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c82d13640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c82d13640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 0x7f2c7c1e8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c82d13640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0d1df0 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c82d13640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c0d1c70 con 0x7f2c7c0d29c0 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c82d13640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7c15e3c0 con 0x7f2c7c0cf830 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c812b0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 0x7f2c7c1e8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c812b0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 0x7f2c7c1e8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55414/0 (socket says 172.21.15.72:55414) 2024-03-06T07:32:16.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.507+0000 7f2c812b0640 1 -- 172.21.15.72:0/1675963962 learned_addr learned my addr 172.21.15.72:0/1675963962 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c80aaf640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c0cf830 0x7f2c7c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c73fff640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c812b0640 1 -- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c0cf830 msgr2=0x7f2c7c160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c812b0640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c0cf830 0x7f2c7c160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c812b0640 1 -- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0d29c0 msgr2=0x7f2c7c0ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c812b0640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c812b0640 1 -- 172.21.15.72:0/1675963962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7c160830 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c80aaf640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c0cf830 0x7f2c7c160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c73fff640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c812b0640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 0x7f2c7c1e8270 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2c74036040 tx=0x7f2c74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74043030 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c74003990 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7c161300 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c74009510 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.508+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7c1f76c0 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.509+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7c04f330 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.510+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f2c7402cb30 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.510+0000 7f2c717fa640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:16.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.510+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2c740ac640 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.510+0000 7f2c80aaf640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:16.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.511+0000 7f2c80aaf640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f2c78006fd0 tx=0x7f2c78008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:16.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.515+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f2c7404d020 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.732+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7c0b3390 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.732+0000 7f2c717fa640 1 -- 172.21.15.72:0/1675963962 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f2c7407b3e0 con 0x7f2c7c15dfe0 2024-03-06T07:32:16.734 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:16.734 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:16.735 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:16.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 msgr2=0x7f2c6806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f2c78006fd0 tx=0x7f2c78008040 comp rx=0 tx=0).stop 2024-03-06T07:32:16.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 msgr2=0x7f2c7c1e8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 0x7f2c7c1e8270 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2c74036040 tx=0x7f2c74004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 shutdown_connections 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f2c6806b4d0 0x7f2c6806d990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2c7c15dfe0 0x7f2c7c1e8270 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c7c0d29c0 0x7f2c7c0ccc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 --2- 172.21.15.72:0/1675963962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c7c0cf830 0x7f2c7c160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 >> 172.21.15.72:0/1675963962 conn(0x7f2c7c0d3cd0 msgr2=0x7f2c7c0d22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 shutdown_connections 2024-03-06T07:32:16.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:16.734+0000 7f2c82d13640 1 -- 172.21.15.72:0/1675963962 wait complete. 2024-03-06T07:32:16.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-06T07:28:37.147617+0000', 'modified': '2024-03-06T07:31:08.461708+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8640, 'mds_1': 8637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8637': {'gid': 8637, 'name': 'c', 'rank': 1, 'incarnation': 81, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.72:6836/2047054467', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 2047054467}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 2047054467}]}, 'join_fscid': -1, 'laggy_since': '2024-03-06T07:29:40.508609+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8640': {'gid': 8640, 'name': 'a', 'rank': 0, 'incarnation': 77, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/1633306088', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 1633306088}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 1633306088}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8640, 'qdb_cluster': [8640]} 2024-03-06T07:32:16.949 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8637:up:stopping 2024-03-06T07:32:17.840 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:17.950 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 9, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 9, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-06T07:28:37.147617+0000", "damaged": [], "data_pools": [ 19 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8637": { "addr": "172.21.15.72:6836/2047054467", "addrs": { "addrvec": [ { "addr": "172.21.15.72:6834", "nonce": 2047054467, "type": "v2" }, { "addr": "172.21.15.72:6836", "nonce": 2047054467, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8637, "incarnation": 81, "join_fscid": -1, "laggy_since": "2024-03-06T07:29:40.508609+0000", "name": "c", "rank": 1, "state": "up:stopping", "state_seq": 5 }, "gid_8640": { "addr": "172.21.15.72:6837/1633306088", "addrs": { "addrvec": [ { "addr": "172.21.15.72:6835", "nonce": 1633306088, "type": "v2" }, { "addr": "172.21.15.72:6837", "nonce": 1633306088, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8640, "incarnation": 77, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 18, "modified": "2024-03-06T07:31:08.461708+0000", "qdb_cluster": [ 8640 ], "qdb_leader": 8640, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8640, "mds_1": 8637 } } } ], "standbys": [] } 2024-03-06T07:32:17.950 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-06T07:32:17.951 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:18.000 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-06T07:32:18.041 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-06T07:32:18.041 INFO:tasks.cephfs.mount:Terminating background process 2024-03-06T07:32:18.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:18 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-03-06T07:32:19.841 INFO:teuthology.orchestra.run.smithi072.stderr:open_dir_background: keeping dir open 2024-03-06T07:32:19.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-06T07:32:19.851 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:32:19.851 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:19.906 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:20.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.044+0000 7f96b62ed640 1 Processor -- start 2024-03-06T07:32:20.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.044+0000 7f96b62ed640 1 -- start start 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96b62ed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f96b00cce40 0x7f96b015b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96b62ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cd7a0 0x7f96b00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96b62ed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b0071f40 con 0x7f96b01515e0 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96b62ed640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b00cd2f0 con 0x7f96b00cce40 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96b62ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0070a70 con 0x7f96b00cd7a0 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96af7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f96b00cce40 0x7f96b015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96affff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f96b01515e0 0x7f96b0071830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53608/0 (socket says 172.21.15.72:53608) 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96affff640 1 -- 172.21.15.72:0/3348908973 learned_addr learned my addr 172.21.15.72:0/3348908973 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96b488a640 1 --2- 172.21.15.72:0/3348908973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cd7a0 0x7f96b00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96af7fe640 1 -- 172.21.15.72:0/3348908973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cd7a0 msgr2=0x7f96b00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96af7fe640 1 --2- 172.21.15.72:0/3348908973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cd7a0 0x7f96b00cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96af7fe640 1 -- 172.21.15.72:0/3348908973 >> v1:172.21.15.72:6789/0 conn(0x7f96b01515e0 legacy=0x7f96b0071830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96aeffd640 1 -- 172.21.15.72:0/3348908973 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554198781 0 0) 0x7f96b0071f40 con 0x7f96b01515e0 2024-03-06T07:32:20.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.045+0000 7f96af7fe640 1 -- 172.21.15.72:0/3348908973 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b0070d50 con 0x7f96b00cce40 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.046+0000 7f96af7fe640 1 --2- 172.21.15.72:0/3348908973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b00cce40 0x7f96b015b430 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f96a000a940 tx=0x7f96a002cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88e98e2b177d9a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.046+0000 7f96aeffd640 1 -- 172.21.15.72:0/3348908973 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a003a070 con 0x7f96b00cce40 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.046+0000 7f96aeffd640 1 -- 172.21.15.72:0/3348908973 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a002db30 con 0x7f96b00cce40 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.046+0000 7f96aeffd640 1 -- 172.21.15.72:0/3348908973 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a000b040 con 0x7f96b00cce40 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 -- 172.21.15.72:0/3348908973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b00cce40 msgr2=0x7f96b015b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 --2- 172.21.15.72:0/3348908973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b00cce40 0x7f96b015b430 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f96a000a940 tx=0x7f96a002cd80 comp rx=0 tx=0).stop 2024-03-06T07:32:20.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 -- 172.21.15.72:0/3348908973 shutdown_connections 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 --2- 172.21.15.72:0/3348908973 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cd7a0 0x7f96b00cdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 --2- 172.21.15.72:0/3348908973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b00cce40 0x7f96b015b430 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 -- 172.21.15.72:0/3348908973 >> 172.21.15.72:0/3348908973 conn(0x7f96b00cf8a0 msgr2=0x7f96b00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 -- 172.21.15.72:0/3348908973 shutdown_connections 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.047+0000 7f96b62ed640 1 -- 172.21.15.72:0/3348908973 wait complete. 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.048+0000 7f96b62ed640 1 Processor -- start 2024-03-06T07:32:20.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.048+0000 7f96b62ed640 1 -- start start 2024-03-06T07:32:20.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b62ed640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cce40 0x7f96b013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b62ed640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b00cd7a0 0x7f96b00ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b62ed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 0x7f96b00ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b62ed640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00ce0f0 con 0x7f96b01515e0 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b62ed640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b0071310 con 0x7f96b00cd7a0 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b62ed640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b00cc3a0 con 0x7f96b00cce40 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 0x7f96b00ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 0x7f96b00ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55430/0 (socket says 172.21.15.72:55430) 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 -- 172.21.15.72:0/2745959144 learned_addr learned my addr 172.21.15.72:0/2745959144 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 -- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cce40 msgr2=0x7f96b013fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cce40 0x7f96b013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 -- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b00cd7a0 msgr2=0x7f96b00ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96af7fe640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b00cd7a0 0x7f96b00ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b00cd7a0 0x7f96b00ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96b488a640 1 -- 172.21.15.72:0/2745959144 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b006d480 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.049+0000 7f96af7fe640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b00cd7a0 0x7f96b00ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.050+0000 7f96b488a640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 0x7f96b00ca740 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f96a8007e40 tx=0x7f96a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.050+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8014620 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.050+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8019070 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.050+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b006d760 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.050+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8014a20 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.050+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b006dd20 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.051+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b0068840 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.051+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f96a8014bc0 con 0x7f96b01515e0 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.051+0000 7f96acff9640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f969c06b4d0 0x7f969c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.052+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f96a808ab80 con 0x7f96b01515e0 2024-03-06T07:32:20.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.052+0000 7f96affff640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f969c06b4d0 0x7f969c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.052+0000 7f96affff640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f969c06b4d0 0x7f969c06d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005880 tx=0x7f96a4007a00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.056+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f96a80599c0 con 0x7f96b01515e0 2024-03-06T07:32:20.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.233+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96b013dba0 con 0x7f96b01515e0 2024-03-06T07:32:20.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.234+0000 7f96acff9640 1 -- 172.21.15.72:0/2745959144 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16733 (secure 0 0 0) 0x7f96b013dba0 con 0x7f96b01515e0 2024-03-06T07:32:20.236 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:20.236 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":131,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:29:33.212437+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.21","mappings":[{"from":2,"to":0}]},{"pgid":"19.31","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f969c06b4d0 msgr2=0x7f969c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f969c06b4d0 0x7f969c06d990 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f96a4005880 tx=0x7f96a4007a00 comp rx=0 tx=0).stop 2024-03-06T07:32:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 msgr2=0x7f96b00ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 0x7f96b00ca740 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f96a8007e40 tx=0x7f96a800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 shutdown_connections 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f969c06b4d0 0x7f969c06d990 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96b01515e0 0x7f96b00ca740 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b00cd7a0 0x7f96b00ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 --2- 172.21.15.72:0/2745959144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b00cce40 0x7f96b013fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 >> 172.21.15.72:0/2745959144 conn(0x7f96b00cf8a0 msgr2=0x7f96b014e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 shutdown_connections 2024-03-06T07:32:20.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.236+0000 7f96b62ed640 1 -- 172.21.15.72:0/2745959144 wait complete. 2024-03-06T07:32:20.250 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:35.363937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:36.115427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:20.250 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:20.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.352+0000 7fa724a39640 1 Processor -- start 2024-03-06T07:32:20.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.352+0000 7fa724a39640 1 -- start start 2024-03-06T07:32:20.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa724a39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200d2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa724a39640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa7200b0e30 0x7fa7200b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa724a39640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7201618b0 con 0x7fa7201614d0 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa724a39640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7200d4cc0 con 0x7fa7200b0e30 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa724a39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200cf400 con 0x7fa7200d4860 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa7200b0e30 0x7fa7200b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa7200b0e30 0x7fa7200b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55438/0 (socket says 172.21.15.72:55438) 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 -- 172.21.15.72:0/408034211 learned_addr learned my addr 172.21.15.72:0/408034211 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71dd9b640 1 --2- 172.21.15.72:0/408034211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200d2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 -- 172.21.15.72:0/408034211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 msgr2=0x7fa7200d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 --2- 172.21.15.72:0/408034211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200d2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 -- 172.21.15.72:0/408034211 >> v1:172.21.15.146:6789/0 conn(0x7fa7201614d0 legacy=0x7fa7200d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.353+0000 7fa71ed9d640 1 -- 172.21.15.72:0/408034211 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7200cf6a0 con 0x7fa7200b0e30 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.354+0000 7fa71ed9d640 1 --2- 172.21.15.72:0/408034211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7200b0e30 0x7fa7200b12c0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa71002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e20b85173e2081 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa71d59a640 1 -- 172.21.15.72:0/408034211 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710035020 con 0x7fa7200b0e30 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa71d59a640 1 -- 172.21.15.72:0/408034211 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa710031640 con 0x7fa7200b0e30 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa71d59a640 1 -- 172.21.15.72:0/408034211 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71002d040 con 0x7fa7200b0e30 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 -- 172.21.15.72:0/408034211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7200b0e30 msgr2=0x7fa7200b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 --2- 172.21.15.72:0/408034211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7200b0e30 0x7fa7200b12c0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa71002b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 -- 172.21.15.72:0/408034211 shutdown_connections 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 --2- 172.21.15.72:0/408034211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7200b0e30 0x7fa7200b12c0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 --2- 172.21.15.72:0/408034211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200d2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 -- 172.21.15.72:0/408034211 >> 172.21.15.72:0/408034211 conn(0x7fa7200d3d40 msgr2=0x7fa7200d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 -- 172.21.15.72:0/408034211 shutdown_connections 2024-03-06T07:32:20.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.355+0000 7fa724a39640 1 -- 172.21.15.72:0/408034211 wait complete. 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 Processor -- start 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 -- start start 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7200b0e30 0x7fa720154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200b8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 0x7fa7200cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200b0910 con 0x7fa7201614d0 2024-03-06T07:32:20.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200cfc60 con 0x7fa7200b0e30 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa724a39640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200d2560 con 0x7fa7200d4860 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa71ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 0x7fa7200cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa71ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 0x7fa7200cce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55442/0 (socket says 172.21.15.72:55442) 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.357+0000 7fa71ed9d640 1 -- 172.21.15.72:0/4060411962 learned_addr learned my addr 172.21.15.72:0/4060411962 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71dd9b640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200b8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71e59c640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7200b0e30 0x7fa720154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71ed9d640 1 -- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 msgr2=0x7fa7200b8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71ed9d640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200b8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71ed9d640 1 -- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7200b0e30 msgr2=0x7fa720154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71ed9d640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7200b0e30 0x7fa720154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71ed9d640 1 -- 172.21.15.72:0/4060411962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7200b8f70 con 0x7fa7201614d0 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71e59c640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7200b0e30 0x7fa720154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:20.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa71ed9d640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 0x7fa7200cce20 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa7100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710040070 con 0x7fa7201614d0 2024-03-06T07:32:20.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa710039cd0 con 0x7fa7201614d0 2024-03-06T07:32:20.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7200b6e30 con 0x7fa7201614d0 2024-03-06T07:32:20.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710035040 con 0x7fa7201614d0 2024-03-06T07:32:20.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.358+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7200b73f0 con 0x7fa7201614d0 2024-03-06T07:32:20.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.359+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa720068ff0 con 0x7fa7201614d0 2024-03-06T07:32:20.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.360+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fa710038020 con 0x7fa7201614d0 2024-03-06T07:32:20.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.360+0000 7fa70effd640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa70006b4d0 0x7fa70006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.360+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7100abd50 con 0x7fa7201614d0 2024-03-06T07:32:20.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.360+0000 7fa71e59c640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa70006b4d0 0x7fa70006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.361+0000 7fa71e59c640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa70006b4d0 0x7fa70006d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa7200c77e0 tx=0x7fa718008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.364+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fa7100af030 con 0x7fa7201614d0 2024-03-06T07:32:20.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.539+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa7200b85d0 con 0x7fa7201614d0 2024-03-06T07:32:20.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.539+0000 7fa70effd640 1 -- 172.21.15.72:0/4060411962 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa71007ab90 con 0x7fa7201614d0 2024-03-06T07:32:20.541 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:32:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa70006b4d0 msgr2=0x7fa70006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa70006b4d0 0x7fa70006d990 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa7200c77e0 tx=0x7fa718008040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 msgr2=0x7fa7200cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 0x7fa7200cce20 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa7100046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 shutdown_connections 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fa70006b4d0 0x7fa70006d990 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7201614d0 0x7fa7200cce20 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7200d4860 0x7fa7200b8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 --2- 172.21.15.72:0/4060411962 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7200b0e30 0x7fa720154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 >> 172.21.15.72:0/4060411962 conn(0x7fa7200d3d40 msgr2=0x7fa7200cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 shutdown_connections 2024-03-06T07:32:20.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.541+0000 7fa724a39640 1 -- 172.21.15.72:0/4060411962 wait complete. 2024-03-06T07:32:20.554 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:20.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.655+0000 7f7e327c7640 1 Processor -- start 2024-03-06T07:32:20.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.655+0000 7f7e327c7640 1 -- start start 2024-03-06T07:32:20.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e327c7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e2c1614d0 0x7f7e2c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e327c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d24b0 0x7f7e2c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e327c7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c1618b0 con 0x7f7e2c0d1910 2024-03-06T07:32:20.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e327c7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2c0cf400 con 0x7f7e2c1614d0 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e327c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0cf6a0 con 0x7f7e2c0d24b0 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e237fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e2c0d1910 0x7f7e2c0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53616/0 (socket says 172.21.15.72:53616) 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e237fe640 1 -- 172.21.15.72:0/3831765632 learned_addr learned my addr 172.21.15.72:0/3831765632 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e30d64640 1 --2- 172.21.15.72:0/3831765632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d24b0 0x7f7e2c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e2bfff640 1 --2- 172.21.15.72:0/3831765632 >> v2:172.21.15.146:3300/0 conn(0x7f7e2c1614d0 0x7f7e2c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e2b7fe640 1 -- 172.21.15.72:0/3831765632 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227082408 0 0) 0x7f7e2c1618b0 con 0x7f7e2c0d1910 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.656+0000 7f7e2b7fe640 1 -- 172.21.15.72:0/3831765632 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18003520 con 0x7f7e2c0d1910 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.657+0000 7f7e30d64640 1 -- 172.21.15.72:0/3831765632 >> v2:172.21.15.146:3300/0 conn(0x7f7e2c1614d0 msgr2=0x7f7e2c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.657+0000 7f7e30d64640 1 --2- 172.21.15.72:0/3831765632 >> v2:172.21.15.146:3300/0 conn(0x7f7e2c1614d0 0x7f7e2c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.657+0000 7f7e30d64640 1 -- 172.21.15.72:0/3831765632 >> v1:172.21.15.72:6789/0 conn(0x7f7e2c0d1910 legacy=0x7f7e2c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.657+0000 7f7e30d64640 1 -- 172.21.15.72:0/3831765632 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0cf980 con 0x7f7e2c0d24b0 2024-03-06T07:32:20.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.658+0000 7f7e30d64640 1 --2- 172.21.15.72:0/3831765632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d24b0 0x7f7e2c0b0e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009d90 tx=0x7f7e2402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d045131a0cc139a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.658+0000 7f7e2b7fe640 1 -- 172.21.15.72:0/3831765632 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24039070 con 0x7f7e2c0d24b0 2024-03-06T07:32:20.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.658+0000 7f7e2b7fe640 1 -- 172.21.15.72:0/3831765632 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e2402cc50 con 0x7f7e2c0d24b0 2024-03-06T07:32:20.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.658+0000 7f7e2b7fe640 1 -- 172.21.15.72:0/3831765632 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24034440 con 0x7f7e2c0d24b0 2024-03-06T07:32:20.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 -- 172.21.15.72:0/3831765632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d24b0 msgr2=0x7f7e2c0b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 --2- 172.21.15.72:0/3831765632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d24b0 0x7f7e2c0b0e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009d90 tx=0x7f7e2402bea0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 -- 172.21.15.72:0/3831765632 shutdown_connections 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 --2- 172.21.15.72:0/3831765632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d24b0 0x7f7e2c0b0e30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 --2- 172.21.15.72:0/3831765632 >> v2:172.21.15.146:3300/0 conn(0x7f7e2c1614d0 0x7f7e2c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 -- 172.21.15.72:0/3831765632 >> 172.21.15.72:0/3831765632 conn(0x7f7e2c0d3d40 msgr2=0x7f7e2c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 -- 172.21.15.72:0/3831765632 shutdown_connections 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.659+0000 7f7e327c7640 1 -- 172.21.15.72:0/3831765632 wait complete. 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.660+0000 7f7e327c7640 1 Processor -- start 2024-03-06T07:32:20.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.660+0000 7f7e327c7640 1 -- start start 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d1910 0x7f7e2c1f4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2c0d24b0 0x7f7e2c14d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 0x7f7e2c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b1370 con 0x7f7e2c1614d0 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b0910 con 0x7f7e2c0d24b0 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c0b14f0 con 0x7f7e2c0d1910 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 0x7f7e2c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 0x7f7e2c14d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55448/0 (socket says 172.21.15.72:55448) 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 -- 172.21.15.72:0/4076723996 learned_addr learned my addr 172.21.15.72:0/4076723996 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e2bfff640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d1910 0x7f7e2c1f4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e237fe640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2c0d24b0 0x7f7e2c14d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 -- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d1910 msgr2=0x7f7e2c1f4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d1910 0x7f7e2c1f4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 -- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2c0d24b0 msgr2=0x7f7e2c14d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2c0d24b0 0x7f7e2c14d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 -- 172.21.15.72:0/4076723996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c1f4b90 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e30d64640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 0x7f7e2c14d720 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009d90 tx=0x7f7e24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24039050 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e24045070 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c0b3790 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.661+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e24009450 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.662+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c0b3d50 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.663+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c0c70b0 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.663+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e2403e610 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.663+0000 7f7e297fa640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.663+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e240accb0 con 0x7f7e2c1614d0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.663+0000 7f7e2bfff640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.664+0000 7f7e2bfff640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0d2130 tx=0x7f7e1c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.668+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e2c1f4b90 con 0x7f7e2c1614d0 2024-03-06T07:32:20.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.845+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e2c068ff0 con 0x7f7e2c1614d0 2024-03-06T07:32:20.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.845+0000 7f7e297fa640 1 -- 172.21.15.72:0/4076723996 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e2407ba70 con 0x7f7e2c1614d0 2024-03-06T07:32:20.847 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:32:20.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e0c06b4d0 msgr2=0x7f7e0c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c0d2130 tx=0x7f7e1c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 msgr2=0x7f7e2c14d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 0x7f7e2c14d720 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009d90 tx=0x7f7e24004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 shutdown_connections 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7e0c06b4d0 0x7f7e0c06d990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e2c1614d0 0x7f7e2c14d720 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e2c0d24b0 0x7f7e2c14d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 --2- 172.21.15.72:0/4076723996 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e2c0d1910 0x7f7e2c1f4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 >> 172.21.15.72:0/4076723996 conn(0x7f7e2c0d3d40 msgr2=0x7f7e2c160140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 shutdown_connections 2024-03-06T07:32:20.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.847+0000 7f7e327c7640 1 -- 172.21.15.72:0/4076723996 wait complete. 2024-03-06T07:32:20.861 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:20.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.963+0000 7fad99956640 1 Processor -- start 2024-03-06T07:32:20.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.963+0000 7fad99956640 1 -- start start 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad99956640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fad940b1e40 0x7fad94168b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad99956640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad940d2730 con 0x7fad941614d0 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad99956640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad941618b0 con 0x7fad940d4860 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad99956640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad940d4cf0 con 0x7fad940b1e40 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad92ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fad941614d0 0x7fad940d1910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53624/0 (socket says 172.21.15.72:53624) 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad92ffd640 1 -- 172.21.15.72:0/3407073798 learned_addr learned my addr 172.21.15.72:0/3407073798 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad937fe640 1 --2- 172.21.15.72:0/3407073798 >> v2:172.21.15.146:3300/0 conn(0x7fad940b1e40 0x7fad94168b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.964+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402251654 0 0) 0x7fad940d2730 con 0x7fad941614d0 2024-03-06T07:32:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad80003520 con 0x7fad941614d0 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517484265 0 0) 0x7fad80003520 con 0x7fad941614d0 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 >> v2:172.21.15.146:3300/0 conn(0x7fad940b1e40 msgr2=0x7fad94168b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 --2- 172.21.15.72:0/3407073798 >> v2:172.21.15.146:3300/0 conn(0x7fad940b1e40 0x7fad94168b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 >> v1:172.21.15.146:6789/0 conn(0x7fad940d4860 legacy=0x7fad940d2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad940cf400 con 0x7fad941614d0 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fad84001670 con 0x7fad941614d0 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad937fe640 1 --2- 172.21.15.72:0/3407073798 >> v2:172.21.15.146:3300/0 conn(0x7fad940b1e40 0x7fad94168b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 >> v1:172.21.15.72:6789/0 conn(0x7fad941614d0 legacy=0x7fad940d1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad800039b0 0x7fad80003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004800 0x7fad80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad80005210 0x7fad8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940d4cf0 con 0x7fad80004800 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8000c140 con 0x7fad80005210 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8000c340 con 0x7fad800039b0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad8bfff640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004800 0x7fad80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad937fe640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad80005210 0x7fad8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad92ffd640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad800039b0 0x7fad80003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad8bfff640 1 -- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad800039b0 msgr2=0x7fad80003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad8bfff640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad800039b0 0x7fad80003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad8bfff640 1 -- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad80005210 msgr2=0x7fad8000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad8bfff640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad80005210 0x7fad8000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad8bfff640 1 -- 172.21.15.72:0/3407073798 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad80003520 con 0x7fad80004800 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad937fe640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad80005210 0x7fad8000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.965+0000 7fad937fe640 1 -- 172.21.15.72:0/3407073798 reap_dead start 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.966+0000 7fad8bfff640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004800 0x7fad80004c80 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fad7c001750 tx=0x7fad80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250e82c6231dc88b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c001370 con 0x7fad80004800 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c0052d0 con 0x7fad80004800 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad927fc640 1 -- 172.21.15.72:0/3407073798 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c003830 con 0x7fad80004800 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 -- 172.21.15.72:0/3407073798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004800 msgr2=0x7fad80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004800 0x7fad80004c80 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fad7c001750 tx=0x7fad80005830 comp rx=0 tx=0).stop 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 -- 172.21.15.72:0/3407073798 shutdown_connections 2024-03-06T07:32:20.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 --2- 172.21.15.72:0/3407073798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad80004800 0x7fad80004c80 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 -- 172.21.15.72:0/3407073798 >> 172.21.15.72:0/3407073798 conn(0x7fad940d3d40 msgr2=0x7fad940d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:20.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 -- 172.21.15.72:0/3407073798 shutdown_connections 2024-03-06T07:32:20.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.967+0000 7fad99956640 1 -- 172.21.15.72:0/3407073798 wait complete. 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.968+0000 7fad99956640 1 Processor -- start 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.968+0000 7fad99956640 1 -- start start 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad99956640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad80004800 0x7fad94159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad99956640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad940b86a0 0x7fad940b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad99956640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 0x7fad940c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad99956640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940cfc60 con 0x7fad940b9040 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad99956640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad940cf980 con 0x7fad940b86a0 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad99956640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad940b0910 con 0x7fad80004800 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad937fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 0x7fad940c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad937fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 0x7fad940c8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55476/0 (socket says 172.21.15.72:55476) 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad937fe640 1 -- 172.21.15.72:0/3319936112 learned_addr learned my addr 172.21.15.72:0/3319936112 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad937fe640 1 -- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad80004800 msgr2=0x7fad94159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad937fe640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad80004800 0x7fad94159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad937fe640 1 -- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad940b86a0 msgr2=0x7fad940b8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.969+0000 7fad8bfff640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad940b86a0 0x7fad940b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad937fe640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad940b86a0 0x7fad940b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad937fe640 1 -- 172.21.15.72:0/3319936112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad940b9890 con 0x7fad940b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad937fe640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 0x7fad940c8c20 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fad940cadc0 tx=0x7fad8c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c011db0 con 0x7fad940b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c01b070 con 0x7fad940b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad940cb820 con 0x7fad940b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c003990 con 0x7fad940b9040 2024-03-06T07:32:20.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.970+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad940cbde0 con 0x7fad940b9040 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.971+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad94068ff0 con 0x7fad940b9040 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.971+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fad8c003b30 con 0x7fad940b9040 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.972+0000 7fad90ff9640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad7406b4d0 0x7fad7406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.972+0000 7fad92ffd640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad7406b4d0 0x7fad7406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.972+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad8c08ac80 con 0x7fad940b9040 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.972+0000 7fad92ffd640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad7406b4d0 0x7fad7406d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fad84001eb0 tx=0x7fad84001d80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:20.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:20.976+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fad8c003e20 con 0x7fad940b9040 2024-03-06T07:32:21.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.155+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad940b5550 con 0x7fad940b9040 2024-03-06T07:32:21.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.155+0000 7fad90ff9640 1 -- 172.21.15.72:0/3319936112 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7fad8c059ac0 con 0x7fad940b9040 2024-03-06T07:32:21.157 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:32:21.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad7406b4d0 msgr2=0x7fad7406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad7406b4d0 0x7fad7406d990 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fad84001eb0 tx=0x7fad84001d80 comp rx=0 tx=0).stop 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 msgr2=0x7fad940c8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 0x7fad940c8c20 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fad940cadc0 tx=0x7fad8c0115a0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 shutdown_connections 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fad7406b4d0 0x7fad7406d990 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad940b9040 0x7fad940c8c20 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad940b86a0 0x7fad940b8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 --2- 172.21.15.72:0/3319936112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad80004800 0x7fad94159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 >> 172.21.15.72:0/3319936112 conn(0x7fad940d3d40 msgr2=0x7fad94160800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 shutdown_connections 2024-03-06T07:32:21.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.157+0000 7fad99956640 1 -- 172.21.15.72:0/3319936112 wait complete. 2024-03-06T07:32:21.174 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-06T07:32:21.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.301+0000 7f53c3de9640 1 Processor -- start 2024-03-06T07:32:21.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.301+0000 7f53c3de9640 1 -- start start 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.302+0000 7f53c3de9640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53bc162970 0x7f53bc0d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.302+0000 7f53c3de9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0d22c0 con 0x7f53bc0b4460 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.302+0000 7f53c3de9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc0d25a0 con 0x7f53bc161cf0 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.302+0000 7f53c3de9640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc161b00 con 0x7f53bc162970 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.302+0000 7f53c1b85640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f53bc0b4460 0x7f53bc0b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53626/0 (socket says 172.21.15.72:53626) 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.302+0000 7f53c1b85640 1 -- 172.21.15.72:0/802430525 learned_addr learned my addr 172.21.15.72:0/802430525 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911124619 0 0) 0x7f53bc0d22c0 con 0x7f53bc0b4460 2024-03-06T07:32:21.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a8003520 con 0x7f53bc0b4460 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1384640 1 --2- 172.21.15.72:0/802430525 >> v2:172.21.15.146:3300/0 conn(0x7f53bc162970 0x7f53bc0d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 715552508 0 0) 0x7f53a8003520 con 0x7f53bc0b4460 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 >> v2:172.21.15.146:3300/0 conn(0x7f53bc162970 msgr2=0x7f53bc0d4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 --2- 172.21.15.72:0/802430525 >> v2:172.21.15.146:3300/0 conn(0x7f53bc162970 0x7f53bc0d4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 >> v1:172.21.15.146:6789/0 conn(0x7f53bc161cf0 legacy=0x7f53bc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0d2880 con 0x7f53bc0b4460 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f53b0001670 con 0x7f53bc0b4460 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 >> v1:172.21.15.72:6789/0 conn(0x7f53bc0b4460 legacy=0x7f53bc0b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a80039b0 0x7f53a8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53a8004800 0x7f53a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80051c0 0x7f53a800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc161b00 con 0x7f53a80039b0 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a800c0f0 con 0x7f53a80051c0 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a800c2f0 con 0x7f53a8004800 2024-03-06T07:32:21.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1b85640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a80039b0 0x7f53a8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c2386640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80051c0 0x7f53a800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1384640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53a8004800 0x7f53a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1b85640 1 -- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53a8004800 msgr2=0x7f53a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1b85640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53a8004800 0x7f53a8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1b85640 1 -- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80051c0 msgr2=0x7f53a800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1b85640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80051c0 0x7f53a800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c1b85640 1 -- 172.21.15.72:0/802430525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8003520 con 0x7f53a80039b0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c2386640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80051c0 0x7f53a800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.303+0000 7f53c2386640 1 -- 172.21.15.72:0/802430525 reap_dead start 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.304+0000 7f53c1b85640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a80039b0 0x7f53a8003db0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f53b000c210 tx=0x7f53a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0cd86b5878f465 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.304+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0007d30 con 0x7f53a80039b0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.304+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0030070 con 0x7f53a80039b0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.304+0000 7f53c0b83640 1 -- 172.21.15.72:0/802430525 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0005d70 con 0x7f53a80039b0 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 -- 172.21.15.72:0/802430525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a80039b0 msgr2=0x7f53a8003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a80039b0 0x7f53a8003db0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f53b000c210 tx=0x7f53a8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:21.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 -- 172.21.15.72:0/802430525 shutdown_connections 2024-03-06T07:32:21.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 --2- 172.21.15.72:0/802430525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53a80039b0 0x7f53a8003db0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 -- 172.21.15.72:0/802430525 >> 172.21.15.72:0/802430525 conn(0x7f53bc0d3ce0 msgr2=0x7f53bc0d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 -- 172.21.15.72:0/802430525 shutdown_connections 2024-03-06T07:32:21.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.305+0000 7f53c3de9640 1 -- 172.21.15.72:0/802430525 wait complete. 2024-03-06T07:32:21.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.306+0000 7f53c3de9640 1 Processor -- start 2024-03-06T07:32:21.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.306+0000 7f53c3de9640 1 -- start start 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c3de9640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80039b0 0x7f53bc156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c3de9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 0x7f53bc0b86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c3de9640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53bc0b1bc0 0x7f53bc0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c3de9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0ce950 con 0x7f53bc0b8270 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c3de9640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0cecc0 con 0x7f53a80039b0 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c3de9640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0b4840 con 0x7f53bc0b1bc0 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 0x7f53bc0b86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 0x7f53bc0b86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55498/0 (socket says 172.21.15.72:55498) 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 -- 172.21.15.72:0/2298013931 learned_addr learned my addr 172.21.15.72:0/2298013931 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1b85640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80039b0 0x7f53bc156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c2386640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53bc0b1bc0 0x7f53bc0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 -- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53bc0b1bc0 msgr2=0x7f53bc0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53bc0b1bc0 0x7f53bc0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 -- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80039b0 msgr2=0x7f53bc156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80039b0 0x7f53bc156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1384640 1 -- 172.21.15.72:0/2298013931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0c8770 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c1b85640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80039b0 0x7f53bc156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.307+0000 7f53c2386640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53bc0b1bc0 0x7f53bc0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.308+0000 7f53c1384640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 0x7f53bc0b86d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f53b800f600 tx=0x7f53b800fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.308+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8019020 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.308+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b8014870 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.308+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc0c7bf0 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.308+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8014c20 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.308+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc0c81b0 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.309+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc04f330 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.309+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f53b801c020 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.310+0000 7f53ae7fc640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.310+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f53b808a960 con 0x7f53bc0b8270 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.310+0000 7f53c1b85640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:21.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.310+0000 7f53c1b85640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53a006b4d0 0x7f53a006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f53b0034000 tx=0x7f53b0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:21.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.314+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f53b8023020 con 0x7f53bc0b8270 2024-03-06T07:32:21.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.503+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f53bc06b180 con 0x7f53bc0b8270 2024-03-06T07:32:21.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.504+0000 7f53ae7fc640 1 -- 172.21.15.72:0/2298013931 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v131) v1 ==== 80+0+0 (secure 0 0 0) 0x7f53b8026350 con 0x7f53bc0b8270 2024-03-06T07:32:21.506 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.505+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53a006b4d0 msgr2=0x7f53a006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.505+0000 7f53c3de9640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53a006b4d0 0x7f53a006d990 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f53b0034000 tx=0x7f53b0009030 comp rx=0 tx=0).stop 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.505+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 msgr2=0x7f53bc0b86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.505+0000 7f53c3de9640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 0x7f53bc0b86d0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f53b800f600 tx=0x7f53b800fac0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 shutdown_connections 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f53a006b4d0 0x7f53a006d990 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53bc0b1bc0 0x7f53bc0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53bc0b8270 0x7f53bc0b86d0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 --2- 172.21.15.72:0/2298013931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53a80039b0 0x7f53bc156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 >> 172.21.15.72:0/2298013931 conn(0x7f53bc0d3ce0 msgr2=0x7f53bc06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 shutdown_connections 2024-03-06T07:32:21.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:21.506+0000 7f53c3de9640 1 -- 172.21.15.72:0/2298013931 wait complete. 2024-03-06T07:32:21.519 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-03-06T07:32:21.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:21.519 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-06T07:32:21.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[78448]: fuse finished with error 0 and tester_r 0 2024-03-06T07:32:21.546 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:32:21.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:21.546 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-06T07:32:21.715 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:32:21.715 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:21.734 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:21.734 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:32:21.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:21.734 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-06T07:32:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:21.792 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-06T07:32:21.793 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:32:21.793 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:21.807 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:21.807 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:32:21.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:21.808 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-06T07:32:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:21.862 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-06T07:32:21.862 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:32:21.863 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:21.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:21.881 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:32:21.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:21.881 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-06T07:32:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:32:21.937 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-06T07:32:21.938 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.041+0000 7fbaca088640 1 Processor -- start 2024-03-06T07:32:22.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.041+0000 7fbaca088640 1 -- start start 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbaca088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac413dc10 0x7fbac40b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbaca088640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbac415bf30 0x7fbac40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbaca088640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbac415e850 0x7fbac415eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbaca088640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac40cf3f0 con 0x7fbac415e850 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbaca088640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac415b210 con 0x7fbac415bf30 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbaca088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac415f580 con 0x7fbac413dc10 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.042+0000 7fbac3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbac415e850 0x7fbac415eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbac415e850 0x7fbac415eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55510/0 (socket says 172.21.15.72:55510) 2024-03-06T07:32:22.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 -- 172.21.15.72:0/514171823 learned_addr learned my addr 172.21.15.72:0/514171823 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac37fe640 1 --2- 172.21.15.72:0/514171823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac413dc10 0x7fbac40b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbabaffd640 1 --2- 172.21.15.72:0/514171823 >> v2:172.21.15.146:3300/0 conn(0x7fbac415bf30 0x7fbac40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 -- 172.21.15.72:0/514171823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac413dc10 msgr2=0x7fbac40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 --2- 172.21.15.72:0/514171823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac413dc10 0x7fbac40b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 -- 172.21.15.72:0/514171823 >> v2:172.21.15.146:3300/0 conn(0x7fbac415bf30 msgr2=0x7fbac40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 --2- 172.21.15.72:0/514171823 >> v2:172.21.15.146:3300/0 conn(0x7fbac415bf30 0x7fbac40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.043+0000 7fbac3fff640 1 -- 172.21.15.72:0/514171823 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac415b450 con 0x7fbac415e850 2024-03-06T07:32:22.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbac3fff640 1 --2- 172.21.15.72:0/514171823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac415eca0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fbac4068890 tx=0x7fbabc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9caff96cd7834cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbac2ffd640 1 -- 172.21.15.72:0/514171823 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc02c9e0 con 0x7fbac415e850 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbac2ffd640 1 -- 172.21.15.72:0/514171823 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc02cb80 con 0x7fbac415e850 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbac2ffd640 1 -- 172.21.15.72:0/514171823 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc0343f0 con 0x7fbac415e850 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 -- 172.21.15.72:0/514171823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 msgr2=0x7fbac415eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 --2- 172.21.15.72:0/514171823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac415eca0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fbac4068890 tx=0x7fbabc02be40 comp rx=0 tx=0).stop 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 -- 172.21.15.72:0/514171823 shutdown_connections 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 --2- 172.21.15.72:0/514171823 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac415eca0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 --2- 172.21.15.72:0/514171823 >> v2:172.21.15.146:3300/0 conn(0x7fbac415bf30 0x7fbac40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 --2- 172.21.15.72:0/514171823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac413dc10 0x7fbac40b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 -- 172.21.15.72:0/514171823 >> 172.21.15.72:0/514171823 conn(0x7fbac40cf8a0 msgr2=0x7fbac40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.044+0000 7fbaca088640 1 -- 172.21.15.72:0/514171823 shutdown_connections 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.045+0000 7fbaca088640 1 -- 172.21.15.72:0/514171823 wait complete. 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 Processor -- start 2024-03-06T07:32:22.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 -- start start 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac413dc10 0x7fbac4140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac415bf30 0x7fbac41f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac41441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac415f1e0 con 0x7fbac415e850 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac413dff0 con 0x7fbac413dc10 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.046+0000 7fbaca088640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac415f360 con 0x7fbac415bf30 2024-03-06T07:32:22.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac41441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac41441d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55516/0 (socket says 172.21.15.72:55516) 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 -- 172.21.15.72:0/1486671458 learned_addr learned my addr 172.21.15.72:0/1486671458 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac37fe640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac413dc10 0x7fbac4140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbabaffd640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac415bf30 0x7fbac41f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 -- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac415bf30 msgr2=0x7fbac41f1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac415bf30 0x7fbac41f1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 -- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac413dc10 msgr2=0x7fbac4140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac413dc10 0x7fbac4140900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 -- 172.21.15.72:0/1486671458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac413fba0 con 0x7fbac415e850 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac3fff640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac41441d0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fbabc036040 tx=0x7fbabc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc043020 con 0x7fbac415e850 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc03e450 con 0x7fbac415e850 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac4154390 con 0x7fbac415e850 2024-03-06T07:32:22.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc009510 con 0x7fbac415e850 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.047+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4140100 con 0x7fbac415e850 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.048+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac404f330 con 0x7fbac415e850 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.049+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbabc046020 con 0x7fbac415e850 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.049+0000 7fbac0ff9640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbaa806b4d0 0x7fbaa806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.049+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbabc0ac8a0 con 0x7fbac415e850 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.049+0000 7fbac37fe640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbaa806b4d0 0x7fbaa806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.050+0000 7fbac37fe640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbaa806b4d0 0x7fbaa806d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbac413d970 tx=0x7fbaac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.053+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbabc04d030 con 0x7fbac415e850 2024-03-06T07:32:22.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.265+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac406eb00 con 0x7fbac415e850 2024-03-06T07:32:22.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.265+0000 7fbac0ff9640 1 -- 172.21.15.72:0/1486671458 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fbabc0ac020 con 0x7fbac415e850 2024-03-06T07:32:22.268 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:22.268 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:22.268 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbaa806b4d0 msgr2=0x7fbaa806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbaa806b4d0 0x7fbaa806d990 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fbac413d970 tx=0x7fbaac008040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 msgr2=0x7fbac41441d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac41441d0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fbabc036040 tx=0x7fbabc004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 shutdown_connections 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbaa806b4d0 0x7fbaa806d990 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac415e850 0x7fbac41441d0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac415bf30 0x7fbac41f1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 --2- 172.21.15.72:0/1486671458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac413dc10 0x7fbac4140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 >> 172.21.15.72:0/1486671458 conn(0x7fbac40cf8a0 msgr2=0x7fbac4071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 shutdown_connections 2024-03-06T07:32:22.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.267+0000 7fbaca088640 1 -- 172.21.15.72:0/1486671458 wait complete. 2024-03-06T07:32:22.280 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:22.281 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:22.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.420+0000 7f4e3d33e640 1 Processor -- start 2024-03-06T07:32:22.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.420+0000 7f4e3d33e640 1 -- start start 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e3d33e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e3813dc10 0x7f4e380b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e3d33e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3815f580 con 0x7f4e3815bf30 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e3d33e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3815b450 con 0x7f4e3815ef60 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e3d33e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e380cf3f0 con 0x7f4e3813dc10 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e36ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e3813dc10 0x7f4e380b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e367fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4e3815bf30 0x7f4e3815e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53632/0 (socket says 172.21.15.72:53632) 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.421+0000 7f4e367fc640 1 -- 172.21.15.72:0/1504927122 learned_addr learned my addr 172.21.15.72:0/1504927122 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.422+0000 7f4e36ffd640 1 -- 172.21.15.72:0/1504927122 >> v1:172.21.15.146:6789/0 conn(0x7f4e3815ef60 legacy=0x7f4e380708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.422+0000 7f4e36ffd640 1 -- 172.21.15.72:0/1504927122 >> v1:172.21.15.72:6789/0 conn(0x7f4e3815bf30 legacy=0x7f4e3815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.422+0000 7f4e35ffb640 1 -- 172.21.15.72:0/1504927122 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864924285 0 0) 0x7f4e3815f580 con 0x7f4e3815bf30 2024-03-06T07:32:22.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.422+0000 7f4e36ffd640 1 -- 172.21.15.72:0/1504927122 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3815b730 con 0x7f4e3813dc10 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e36ffd640 1 --2- 172.21.15.72:0/1504927122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3813dc10 0x7f4e380b2950 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f4e280097a0 tx=0x7f4e2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5e81eed9d1490d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e35ffb640 1 -- 172.21.15.72:0/1504927122 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28035020 con 0x7f4e3813dc10 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e35ffb640 1 -- 172.21.15.72:0/1504927122 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e28031830 con 0x7f4e3813dc10 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e35ffb640 1 -- 172.21.15.72:0/1504927122 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28031c00 con 0x7f4e3813dc10 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e3d33e640 1 -- 172.21.15.72:0/1504927122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3813dc10 msgr2=0x7f4e380b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/1504927122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3813dc10 0x7f4e380b2950 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f4e280097a0 tx=0x7f4e2802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e3d33e640 1 -- 172.21.15.72:0/1504927122 shutdown_connections 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/1504927122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3813dc10 0x7f4e380b2950 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.423+0000 7f4e3d33e640 1 -- 172.21.15.72:0/1504927122 >> 172.21.15.72:0/1504927122 conn(0x7f4e380cf8a0 msgr2=0x7f4e380cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.424+0000 7f4e3d33e640 1 -- 172.21.15.72:0/1504927122 shutdown_connections 2024-03-06T07:32:22.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.424+0000 7f4e3d33e640 1 -- 172.21.15.72:0/1504927122 wait complete. 2024-03-06T07:32:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.425+0000 7f4e3d33e640 1 Processor -- start 2024-03-06T07:32:22.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.426+0000 7f4e3d33e640 1 -- start start 2024-03-06T07:32:22.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e3d33e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e3813dc10 0x7f4e381f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e3d33e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3815bf30 0x7f4e38143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e3d33e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 0x7f4e38141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e3d33e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e380b2e90 con 0x7f4e3815ef60 2024-03-06T07:32:22.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e3d33e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3815b210 con 0x7f4e3813dc10 2024-03-06T07:32:22.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e3d33e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e380b3010 con 0x7f4e3815bf30 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 0x7f4e38141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 0x7f4e38141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55534/0 (socket says 172.21.15.72:55534) 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 -- 172.21.15.72:0/2835493534 learned_addr learned my addr 172.21.15.72:0/2835493534 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e36ffd640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e3813dc10 0x7f4e381f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e367fc640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3815bf30 0x7f4e38143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 -- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3815bf30 msgr2=0x7f4e38143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3815bf30 0x7f4e38143d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 -- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e3813dc10 msgr2=0x7f4e381f4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e3813dc10 0x7f4e381f4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 -- 172.21.15.72:0/2835493534 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3813f030 con 0x7f4e3815ef60 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e36ffd640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e3813dc10 0x7f4e381f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e377fe640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 0x7f4e38141740 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000c6e0 tx=0x7f4e3000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30003b70 con 0x7f4e3815ef60 2024-03-06T07:32:22.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.427+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30003d30 con 0x7f4e3815ef60 2024-03-06T07:32:22.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.428+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3813f310 con 0x7f4e3815ef60 2024-03-06T07:32:22.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.428+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30005310 con 0x7f4e3815ef60 2024-03-06T07:32:22.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.428+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3813f8d0 con 0x7f4e3815ef60 2024-03-06T07:32:22.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.428+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3804f330 con 0x7f4e3815ef60 2024-03-06T07:32:22.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.429+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4e300054b0 con 0x7f4e3815ef60 2024-03-06T07:32:22.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.429+0000 7f4e1f7fe640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e2406b500 0x7f4e2406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.429+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4e30089ff0 con 0x7f4e3815ef60 2024-03-06T07:32:22.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.429+0000 7f4e36ffd640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e2406b500 0x7f4e2406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.430+0000 7f4e36ffd640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e2406b500 0x7f4e2406d9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4e28036000 tx=0x7f4e28008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.434+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4e30058ec0 con 0x7f4e3815ef60 2024-03-06T07:32:22.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.608+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e3806eb00 con 0x7f4e3815ef60 2024-03-06T07:32:22.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.609+0000 7f4e1f7fe640 1 -- 172.21.15.72:0/2835493534 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16733 (secure 0 0 0) 0x7f4e300587c0 con 0x7f4e3815ef60 2024-03-06T07:32:22.611 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:22.612 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":131,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:29:33.212437+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.21","mappings":[{"from":2,"to":0}]},{"pgid":"19.31","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:22.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e2406b500 msgr2=0x7f4e2406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e2406b500 0x7f4e2406d9c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f4e28036000 tx=0x7f4e28008030 comp rx=0 tx=0).stop 2024-03-06T07:32:22.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 msgr2=0x7f4e38141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 0x7f4e38141740 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000c6e0 tx=0x7f4e3000cba0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 shutdown_connections 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4e2406b500 0x7f4e2406d9c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e3815ef60 0x7f4e38141740 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3815bf30 0x7f4e38143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 --2- 172.21.15.72:0/2835493534 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e3813dc10 0x7f4e381f4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 >> 172.21.15.72:0/2835493534 conn(0x7f4e380cf8a0 msgr2=0x7f4e380af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 shutdown_connections 2024-03-06T07:32:22.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.611+0000 7f4e3d33e640 1 -- 172.21.15.72:0/2835493534 wait complete. 2024-03-06T07:32:22.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-06T07:21:37.990747+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-06T07:28:35.363937+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-06T07:28:36.115427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '127', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-06T07:32:22.625 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-06T07:32:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.727+0000 7f9481af4640 1 Processor -- start 2024-03-06T07:32:22.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.727+0000 7f9481af4640 1 -- start start 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f9481af4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f947c15c050 0x7f947c06f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f9481af4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f947c06fb40 0x7f947c0cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f9481af4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947c0cd4b0 con 0x7f947c0ccca0 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f9481af4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947c06ffb0 con 0x7f947c06fb40 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f9481af4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947c0cd790 con 0x7f947c15c050 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f947affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f947c06fb40 0x7f947c0cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f947affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f947c06fb40 0x7f947c0cc760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57372/0 (socket says 172.21.15.72:57372) 2024-03-06T07:32:22.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.728+0000 7f947affd640 1 -- 172.21.15.72:0/649054358 learned_addr learned my addr 172.21.15.72:0/649054358 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:22.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.729+0000 7f947b7fe640 1 --2- 172.21.15.72:0/649054358 >> v2:172.21.15.146:3300/0 conn(0x7f947c15c050 0x7f947c06f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.729+0000 7f947affd640 1 -- 172.21.15.72:0/649054358 >> v2:172.21.15.146:3300/0 conn(0x7f947c15c050 msgr2=0x7f947c06f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:22.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.729+0000 7f947affd640 1 --2- 172.21.15.72:0/649054358 >> v2:172.21.15.146:3300/0 conn(0x7f947c15c050 0x7f947c06f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.729+0000 7f947affd640 1 -- 172.21.15.72:0/649054358 >> v1:172.21.15.146:6789/0 conn(0x7f947c0ccca0 legacy=0x7f947c0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.729+0000 7f947affd640 1 -- 172.21.15.72:0/649054358 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0cda70 con 0x7f947c06fb40 2024-03-06T07:32:22.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.729+0000 7f947affd640 1 --2- 172.21.15.72:0/649054358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c06fb40 0x7f947c0cc760 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f947c070ef0 tx=0x7f947002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b149cefc06fd10 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f947a7fc640 1 -- 172.21.15.72:0/649054358 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470035020 con 0x7f947c06fb40 2024-03-06T07:32:22.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f947a7fc640 1 -- 172.21.15.72:0/649054358 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9470031870 con 0x7f947c06fb40 2024-03-06T07:32:22.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f947a7fc640 1 -- 172.21.15.72:0/649054358 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470031c40 con 0x7f947c06fb40 2024-03-06T07:32:22.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 -- 172.21.15.72:0/649054358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c06fb40 msgr2=0x7f947c0cc760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 --2- 172.21.15.72:0/649054358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c06fb40 0x7f947c0cc760 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f947c070ef0 tx=0x7f947002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 -- 172.21.15.72:0/649054358 shutdown_connections 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 --2- 172.21.15.72:0/649054358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c06fb40 0x7f947c0cc760 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 --2- 172.21.15.72:0/649054358 >> v2:172.21.15.146:3300/0 conn(0x7f947c15c050 0x7f947c06f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 -- 172.21.15.72:0/649054358 >> 172.21.15.72:0/649054358 conn(0x7f947c0b3800 msgr2=0x7f947c0b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 -- 172.21.15.72:0/649054358 shutdown_connections 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.730+0000 7f9481af4640 1 -- 172.21.15.72:0/649054358 wait complete. 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.732+0000 7f9481af4640 1 Processor -- start 2024-03-06T07:32:22.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.732+0000 7f9481af4640 1 -- start start 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.733+0000 7f9481af4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f947c06fb40 0x7f947c0b7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.733+0000 7f9481af4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 0x7f947c0b5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.733+0000 7f9481af4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c15c050 0x7f947c1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.733+0000 7f9481af4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0d01f0 con 0x7f947c0ccca0 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.733+0000 7f9481af4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0cd130 con 0x7f947c15c050 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.733+0000 7f9481af4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f947c0d0370 con 0x7f947c06fb40 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 0x7f947c0b5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 0x7f947c0b5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57382/0 (socket says 172.21.15.72:57382) 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 -- 172.21.15.72:0/3217429895 learned_addr learned my addr 172.21.15.72:0/3217429895 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947b7fe640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f947c06fb40 0x7f947c0b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947bfff640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c15c050 0x7f947c1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 -- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f947c06fb40 msgr2=0x7f947c0b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f947c06fb40 0x7f947c0b7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 -- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c15c050 msgr2=0x7f947c1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c15c050 0x7f947c1592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 -- 172.21.15.72:0/3217429895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c0b4cd0 con 0x7f947c0ccca0 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f947affd640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 0x7f947c0b5d70 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f947002b950 tx=0x7f94700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470040070 con 0x7f947c0ccca0 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9470030d00 con 0x7f947c0ccca0 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c0b4fb0 con 0x7f947c0ccca0 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9470035040 con 0x7f947c0ccca0 2024-03-06T07:32:22.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.734+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c0b5570 con 0x7f947c0ccca0 2024-03-06T07:32:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.735+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c09fe60 con 0x7f947c0ccca0 2024-03-06T07:32:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.736+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9470038020 con 0x7f947c0ccca0 2024-03-06T07:32:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.736+0000 7f9463fff640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f946806b4d0 0x7f946806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.736+0000 7f947b7fe640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f946806b4d0 0x7f946806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.736+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f94700abf60 con 0x7f947c0ccca0 2024-03-06T07:32:22.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.737+0000 7f947b7fe640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f946806b4d0 0x7f946806d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f947c1f54a0 tx=0x7f946c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:22.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.741+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f947007ada0 con 0x7f947c0ccca0 2024-03-06T07:32:22.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.912+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f947c068ff0 con 0x7f947c0ccca0 2024-03-06T07:32:22.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.913+0000 7f9463fff640 1 -- 172.21.15.72:0/3217429895 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f947007a6a0 con 0x7f947c0ccca0 2024-03-06T07:32:22.915 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.914+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f946806b4d0 msgr2=0x7f946806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f946806b4d0 0x7f946806d990 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f947c1f54a0 tx=0x7f946c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 msgr2=0x7f947c0b5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 0x7f947c0b5d70 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f947002b950 tx=0x7f94700046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 shutdown_connections 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f946806b4d0 0x7f946806d990 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f947c15c050 0x7f947c1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f947c0ccca0 0x7f947c0b5d70 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 --2- 172.21.15.72:0/3217429895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f947c06fb40 0x7f947c0b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:22.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 >> 172.21.15.72:0/3217429895 conn(0x7f947c0b3800 msgr2=0x7f947c0d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:22.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 shutdown_connections 2024-03-06T07:32:22.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:22.915+0000 7f9481af4640 1 -- 172.21.15.72:0/3217429895 wait complete. 2024-03-06T07:32:22.928 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-06T07:32:23.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.033+0000 7fca9ae4e640 1 Processor -- start 2024-03-06T07:32:23.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.033+0000 7fca9ae4e640 1 -- start start 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca9ae4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 0x7fca940b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca9ae4e640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca940cea50 0x7fca940ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca9ae4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9406d030 con 0x7fca9406cab0 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca9ae4e640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca940cf1c0 con 0x7fca940cea50 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca9ae4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca940cc760 con 0x7fca9415def0 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca93fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fca9406cab0 0x7fca940ce340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50038/0 (socket says 172.21.15.72:50038) 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca93fff640 1 -- 172.21.15.72:0/2905637391 learned_addr learned my addr 172.21.15.72:0/2905637391 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca993eb640 1 --2- 172.21.15.72:0/2905637391 >> v2:172.21.15.146:3300/0 conn(0x7fca940cea50 0x7fca940ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca98bea640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 0x7fca940b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.034+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155228012 0 0) 0x7fca9406d030 con 0x7fca9406cab0 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca80003520 con 0x7fca9406cab0 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1557597459 0 0) 0x7fca80003520 con 0x7fca9406cab0 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 msgr2=0x7fca940b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 0x7fca940b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 >> v2:172.21.15.146:3300/0 conn(0x7fca940cea50 msgr2=0x7fca940ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 --2- 172.21.15.72:0/2905637391 >> v2:172.21.15.146:3300/0 conn(0x7fca940cea50 0x7fca940ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9406d310 con 0x7fca9406cab0 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fca7c001670 con 0x7fca9406cab0 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 >> v1:172.21.15.72:6789/0 conn(0x7fca9406cab0 legacy=0x7fca940ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 0x7fca800039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca80003f30 0x7fca80004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80004d40 0x7fca800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940cc760 con 0x7fca80003f30 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000b9c0 con 0x7fca80004d40 2024-03-06T07:32:23.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8000bb70 con 0x7fca9415def0 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca993eb640 1 --2- 172.21.15.72:0/2905637391 >> v2:172.21.15.146:3300/0 conn(0x7fca940cea50 0x7fca940ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca93fff640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca80003f30 0x7fca80004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca98bea640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 0x7fca800039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca993eb640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80004d40 0x7fca800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca93fff640 1 -- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 msgr2=0x7fca800039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca93fff640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca9415def0 0x7fca800039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca93fff640 1 -- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80004d40 msgr2=0x7fca800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca93fff640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80004d40 0x7fca800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.035+0000 7fca93fff640 1 -- 172.21.15.72:0/2905637391 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca80003520 con 0x7fca80003f30 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.036+0000 7fca93fff640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca80003f30 0x7fca80004800 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c240 tx=0x7fca80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4672f93432c2f3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.036+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c007d60 con 0x7fca80003f30 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.036+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c030070 con 0x7fca80003f30 2024-03-06T07:32:23.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.036+0000 7fca937fe640 1 -- 172.21.15.72:0/2905637391 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c02cd70 con 0x7fca80003f30 2024-03-06T07:32:23.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca9ae4e640 1 -- 172.21.15.72:0/2905637391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca80003f30 msgr2=0x7fca80004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/2905637391 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca80003f30 0x7fca80004800 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c240 tx=0x7fca80005830 comp rx=0 tx=0).stop 2024-03-06T07:32:23.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca993eb640 1 -- 172.21.15.72:0/2905637391 reap_dead start 2024-03-06T07:32:23.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca9ae4e640 1 -- 172.21.15.72:0/2905637391 shutdown_connections 2024-03-06T07:32:23.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca9ae4e640 1 -- 172.21.15.72:0/2905637391 >> 172.21.15.72:0/2905637391 conn(0x7fca940b3690 msgr2=0x7fca940b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca9ae4e640 1 -- 172.21.15.72:0/2905637391 shutdown_connections 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.037+0000 7fca9ae4e640 1 -- 172.21.15.72:0/2905637391 wait complete. 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.038+0000 7fca9ae4e640 1 Processor -- start 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.038+0000 7fca9ae4e640 1 -- start start 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca9ae4e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80003f30 0x7fca940c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca9ae4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 0x7fca941607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca9ae4e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca940c9670 0x7fca94160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca9ae4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9415e2d0 con 0x7fca94147e50 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca9ae4e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca940cc3f0 con 0x7fca80003f30 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca9ae4e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9415e450 con 0x7fca940c9670 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 0x7fca941607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 0x7fca941607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57386/0 (socket says 172.21.15.72:57386) 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 -- 172.21.15.72:0/3070936480 learned_addr learned my addr 172.21.15.72:0/3070936480 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca993eb640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca940c9670 0x7fca94160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca98bea640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80003f30 0x7fca940c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 -- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca940c9670 msgr2=0x7fca94160ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca940c9670 0x7fca94160ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 -- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80003f30 msgr2=0x7fca940c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80003f30 0x7fca940c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 -- 172.21.15.72:0/3070936480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca941f39f0 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca93fff640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 0x7fca941607a0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c240 tx=0x7fca7c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c007d60 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.039+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7c004bf0 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.040+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7c007070 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.040+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca94148330 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.040+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca941488f0 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.040+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9406b380 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.041+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fca7c047020 con 0x7fca94147e50 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.041+0000 7fca917fa640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca7406b500 0x7fca7406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.041+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fca7c0ab4c0 con 0x7fca94147e50 2024-03-06T07:32:23.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.041+0000 7fca98bea640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca7406b500 0x7fca7406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.042+0000 7fca98bea640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca7406b500 0x7fca7406d9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fca84009a20 tx=0x7fca84008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.050+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fca9406b380 con 0x7fca94147e50 2024-03-06T07:32:23.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.226+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca9409fcf0 con 0x7fca94147e50 2024-03-06T07:32:23.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.227+0000 7fca917fa640 1 -- 172.21.15.72:0/3070936480 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca7c07a270 con 0x7fca94147e50 2024-03-06T07:32:23.229 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-03-06T07:32:23.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca7406b500 msgr2=0x7fca7406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca7406b500 0x7fca7406d9c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fca84009a20 tx=0x7fca84008040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 msgr2=0x7fca941607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 0x7fca941607a0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c240 tx=0x7fca7c001a40 comp rx=0 tx=0).stop 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 shutdown_connections 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fca7406b500 0x7fca7406d9c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca940c9670 0x7fca94160ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca94147e50 0x7fca941607a0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 --2- 172.21.15.72:0/3070936480 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca80003f30 0x7fca940c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 >> 172.21.15.72:0/3070936480 conn(0x7fca940b3690 msgr2=0x7fca940ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 shutdown_connections 2024-03-06T07:32:23.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.229+0000 7fca9ae4e640 1 -- 172.21.15.72:0/3070936480 wait complete. 2024-03-06T07:32:23.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-06T07:32:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.349+0000 7fedfea15640 1 Processor -- start 2024-03-06T07:32:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.349+0000 7fedfea15640 1 -- start start 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedfea15640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fedf80d06e0 0x7fedf815fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedfea15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fedf80cf200 0x7fedf80cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedfea15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80d47f0 con 0x7fedf80d0ac0 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedfea15640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf80ced70 con 0x7fedf80cf200 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedfea15640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf80d1ab0 con 0x7fedf80d06e0 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedfcfb2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fedf80cf200 0x7fedf80cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedf77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fedf80d0ac0 0x7fedf80b47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50052/0 (socket says 172.21.15.72:50052) 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.350+0000 7fedf77fe640 1 -- 172.21.15.72:0/2210308210 learned_addr learned my addr 172.21.15.72:0/2210308210 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedf7fff640 1 --2- 172.21.15.72:0/2210308210 >> v2:172.21.15.146:3300/0 conn(0x7fedf80d06e0 0x7fedf815fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedfcfb2640 1 -- 172.21.15.72:0/2210308210 >> v2:172.21.15.146:3300/0 conn(0x7fedf80d06e0 msgr2=0x7fedf815fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedfcfb2640 1 --2- 172.21.15.72:0/2210308210 >> v2:172.21.15.146:3300/0 conn(0x7fedf80d06e0 0x7fedf815fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedfcfb2640 1 -- 172.21.15.72:0/2210308210 >> v1:172.21.15.72:6789/0 conn(0x7fedf80d0ac0 legacy=0x7fedf80b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedfcfb2640 1 -- 172.21.15.72:0/2210308210 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80d1d90 con 0x7fedf80cf200 2024-03-06T07:32:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedf6ffd640 1 -- 172.21.15.72:0/2210308210 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651618258 0 0) 0x7fedf80d47f0 con 0x7fedf80d0ac0 2024-03-06T07:32:23.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedfcfb2640 1 --2- 172.21.15.72:0/2210308210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80cf200 0x7fedf80cf650 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fedf80b6850 tx=0x7fedf002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38682567993f8393 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.351+0000 7fedf6ffd640 1 -- 172.21.15.72:0/2210308210 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0039070 con 0x7fedf80cf200 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedf6ffd640 1 -- 172.21.15.72:0/2210308210 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf002cc80 con 0x7fedf80cf200 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedf6ffd640 1 -- 172.21.15.72:0/2210308210 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0034660 con 0x7fedf80cf200 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 -- 172.21.15.72:0/2210308210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80cf200 msgr2=0x7fedf80cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 --2- 172.21.15.72:0/2210308210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80cf200 0x7fedf80cf650 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fedf80b6850 tx=0x7fedf002bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 -- 172.21.15.72:0/2210308210 shutdown_connections 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 --2- 172.21.15.72:0/2210308210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80cf200 0x7fedf80cf650 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 --2- 172.21.15.72:0/2210308210 >> v2:172.21.15.146:3300/0 conn(0x7fedf80d06e0 0x7fedf815fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 -- 172.21.15.72:0/2210308210 >> 172.21.15.72:0/2210308210 conn(0x7fedf80d3cd0 msgr2=0x7fedf80d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 -- 172.21.15.72:0/2210308210 shutdown_connections 2024-03-06T07:32:23.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.352+0000 7fedfea15640 1 -- 172.21.15.72:0/2210308210 wait complete. 2024-03-06T07:32:23.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.354+0000 7fedfea15640 1 Processor -- start 2024-03-06T07:32:23.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.354+0000 7fedfea15640 1 -- start start 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfea15640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedf80cf200 0x7fedf80b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfea15640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedf80d06e0 0x7fedf8153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfea15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 0x7fedf806c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfea15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80b44d0 con 0x7fedf80d0ac0 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfea15640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80ce990 con 0x7fedf80cf200 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfea15640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf80d0f30 con 0x7fedf80d06e0 2024-03-06T07:32:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 0x7fedf806c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 0x7fedf806c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57410/0 (socket says 172.21.15.72:57410) 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 -- 172.21.15.72:0/3985125399 learned_addr learned my addr 172.21.15.72:0/3985125399 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedf7fff640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedf80cf200 0x7fedf80b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedf77fe640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedf80d06e0 0x7fedf8153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 -- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedf80d06e0 msgr2=0x7fedf8153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedf80d06e0 0x7fedf8153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 -- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedf80cf200 msgr2=0x7fedf80b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedf80cf200 0x7fedf80b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.355+0000 7fedfcfb2640 1 -- 172.21.15.72:0/3985125399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80b0900 con 0x7fedf80d0ac0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.356+0000 7fedfcfb2640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 0x7fedf806c760 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fedf0037040 tx=0x7fedf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.356+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0044020 con 0x7fedf80d0ac0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.356+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf003a040 con 0x7fedf80d0ac0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.356+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0039050 con 0x7fedf80d0ac0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.356+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf80b0ba0 con 0x7fedf80d0ac0 2024-03-06T07:32:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.356+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf80b1160 con 0x7fedf80d0ac0 2024-03-06T07:32:23.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.357+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf80a1c80 con 0x7fedf80d0ac0 2024-03-06T07:32:23.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.357+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fedf0003b90 con 0x7fedf80d0ac0 2024-03-06T07:32:23.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.358+0000 7fedf4ff9640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fede406b4d0 0x7fede406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.358+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fedf00acae0 con 0x7fedf80d0ac0 2024-03-06T07:32:23.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.358+0000 7fedf7fff640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fede406b4d0 0x7fede406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.358+0000 7fedf7fff640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fede406b4d0 0x7fede406d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fede8006fd0 tx=0x7fede8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.362+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fedf004e020 con 0x7fedf80d0ac0 2024-03-06T07:32:23.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.539+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fedf80d2090 con 0x7fedf80d0ac0 2024-03-06T07:32:23.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.540+0000 7fedf4ff9640 1 -- 172.21.15.72:0/3985125399 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7fedf007b9b0 con 0x7fedf80d0ac0 2024-03-06T07:32:23.542 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fede406b4d0 msgr2=0x7fede406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fede406b4d0 0x7fede406d990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fede8006fd0 tx=0x7fede8008040 comp rx=0 tx=0).stop 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 msgr2=0x7fedf806c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 0x7fedf806c760 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fedf0037040 tx=0x7fedf0004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 shutdown_connections 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fede406b4d0 0x7fede406d990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fedf80d0ac0 0x7fedf806c760 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fedf80d06e0 0x7fedf8153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 --2- 172.21.15.72:0/3985125399 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fedf80cf200 0x7fedf80b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.541+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 >> 172.21.15.72:0/3985125399 conn(0x7fedf80d3cd0 msgr2=0x7fedf815dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.542+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 shutdown_connections 2024-03-06T07:32:23.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.542+0000 7fedfea15640 1 -- 172.21.15.72:0/3985125399 wait complete. 2024-03-06T07:32:23.555 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.653+0000 7f9c8b488640 1 Processor -- start 2024-03-06T07:32:23.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.653+0000 7f9c8b488640 1 -- start start 2024-03-06T07:32:23.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c8b488640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9c8415dfe0 0x7f9c840cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c8b488640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c840d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c8b488640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c840b3430 con 0x7f9c840d27f0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c8b488640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c840d2c50 con 0x7f9c8415dfe0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c8b488640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840b31e0 con 0x7f9c840cf830 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c89a25640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9c840d27f0 0x7f9c840d0300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50054/0 (socket says 172.21.15.72:50054) 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c89a25640 1 -- 172.21.15.72:0/2112596767 learned_addr learned my addr 172.21.15.72:0/2112596767 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c89224640 1 --2- 172.21.15.72:0/2112596767 >> v2:172.21.15.146:3300/0 conn(0x7f9c8415dfe0 0x7f9c840cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168307916 0 0) 0x7f9c840b3430 con 0x7f9c840d27f0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c88a23640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c840d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.654+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c70003520 con 0x7f9c840d27f0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177491777 0 0) 0x7f9c70003520 con 0x7f9c840d27f0 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 msgr2=0x7f9c840d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c840d22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 >> v2:172.21.15.146:3300/0 conn(0x7f9c8415dfe0 msgr2=0x7f9c840cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 --2- 172.21.15.72:0/2112596767 >> v2:172.21.15.146:3300/0 conn(0x7f9c8415dfe0 0x7f9c840cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c840b3710 con 0x7f9c840d27f0 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f9c7c0019f0 con 0x7f9c840d27f0 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 >> v1:172.21.15.72:6789/0 conn(0x7f9c840d27f0 legacy=0x7f9c840d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c700039b0 0x7f9c70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c70004800 0x7f9c70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840b31e0 con 0x7f9c700039b0 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7000b9c0 con 0x7f9c70004800 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7000bb40 con 0x7f9c840cf830 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89224640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c700039b0 0x7f9c70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c88a23640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c70004800 0x7f9c70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89a25640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89224640 1 -- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 msgr2=0x7f9c700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89224640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89224640 1 -- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c70004800 msgr2=0x7f9c70004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:23.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89224640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c70004800 0x7f9c70004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89224640 1 -- 172.21.15.72:0/2112596767 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c70003520 con 0x7f9c700039b0 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c88a23640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c70004800 0x7f9c70004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.655+0000 7f9c89a25640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cf830 0x7f9c700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.656+0000 7f9c89224640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c700039b0 0x7f9c70003dd0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009ac0 tx=0x7f9c70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de43c510ed89b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.656+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80039070 con 0x7f9c700039b0 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.656+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8002c710 con 0x7f9c700039b0 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.656+0000 7f9c7bfff640 1 -- 172.21.15.72:0/2112596767 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c80009690 con 0x7f9c700039b0 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.656+0000 7f9c8b488640 1 -- 172.21.15.72:0/2112596767 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c700039b0 msgr2=0x7f9c70003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.656+0000 7f9c8b488640 1 --2- 172.21.15.72:0/2112596767 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c700039b0 0x7f9c70003dd0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009ac0 tx=0x7f9c70005830 comp rx=0 tx=0).stop 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.657+0000 7f9c89a25640 1 -- 172.21.15.72:0/2112596767 reap_dead start 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.657+0000 7f9c8b488640 1 -- 172.21.15.72:0/2112596767 shutdown_connections 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.657+0000 7f9c8b488640 1 -- 172.21.15.72:0/2112596767 >> 172.21.15.72:0/2112596767 conn(0x7f9c840d3cd0 msgr2=0x7f9c840d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.657+0000 7f9c8b488640 1 -- 172.21.15.72:0/2112596767 shutdown_connections 2024-03-06T07:32:23.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.657+0000 7f9c8b488640 1 -- 172.21.15.72:0/2112596767 wait complete. 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.658+0000 7f9c8b488640 1 Processor -- start 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.658+0000 7f9c8b488640 1 -- start start 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.658+0000 7f9c8b488640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c700039b0 0x7f9c8415d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.658+0000 7f9c8b488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 0x7f9c840ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.658+0000 7f9c8b488640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cd290 0x7f9c840cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c8b488640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840d1c70 con 0x7f9c840b8330 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c8b488640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840b3d60 con 0x7f9c700039b0 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c8b488640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840d1df0 con 0x7f9c840cd290 2024-03-06T07:32:23.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 0x7f9c840ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 0x7f9c840ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57434/0 (socket says 172.21.15.72:57434) 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 -- 172.21.15.72:0/469753533 learned_addr learned my addr 172.21.15.72:0/469753533 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c89224640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c700039b0 0x7f9c8415d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c89a25640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cd290 0x7f9c840cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 -- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cd290 msgr2=0x7f9c840cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cd290 0x7f9c840cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 -- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c700039b0 msgr2=0x7f9c8415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c700039b0 0x7f9c8415d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 -- 172.21.15.72:0/469753533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c840cc0e0 con 0x7f9c840b8330 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c89224640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c700039b0 0x7f9c8415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c88a23640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 0x7f9c840ccb70 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f9c740115e0 tx=0x7f9c74011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7400f8f0 con 0x7f9c840b8330 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7400fa90 con 0x7f9c840b8330 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c840b8810 con 0x7f9c840b8330 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7400fe40 con 0x7f9c840b8330 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.659+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c840b8dd0 con 0x7f9c840b8330 2024-03-06T07:32:23.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.660+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c8404f330 con 0x7f9c840b8330 2024-03-06T07:32:23.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.661+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c74015460 con 0x7f9c840b8330 2024-03-06T07:32:23.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.661+0000 7f9c79ffb640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c6806b500 0x7f9c6806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:23.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.661+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9c7408ab90 con 0x7f9c840b8330 2024-03-06T07:32:23.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.661+0000 7f9c89224640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c6806b500 0x7f9c6806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:23.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.662+0000 7f9c89224640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c6806b500 0x7f9c6806d9c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9c80036000 tx=0x7f9c80030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:23.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.666+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9c840cc0e0 con 0x7f9c840b8330 2024-03-06T07:32:23.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.979+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c840d1a30 con 0x7f9c840b8330 2024-03-06T07:32:23.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.979+0000 7f9c79ffb640 1 -- 172.21.15.72:0/469753533 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f9c74019030 con 0x7f9c840b8330 2024-03-06T07:32:23.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:23.982 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:23.982 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:23.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c6806b500 msgr2=0x7f9c6806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c6806b500 0x7f9c6806d9c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9c80036000 tx=0x7f9c80030000 comp rx=0 tx=0).stop 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 msgr2=0x7f9c840ccb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 0x7f9c840ccb70 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f9c740115e0 tx=0x7f9c74011aa0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 shutdown_connections 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c6806b500 0x7f9c6806d9c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c840cd290 0x7f9c840cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9c840b8330 0x7f9c840ccb70 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 --2- 172.21.15.72:0/469753533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c700039b0 0x7f9c8415d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 >> 172.21.15.72:0/469753533 conn(0x7f9c840d3cd0 msgr2=0x7f9c840cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 shutdown_connections 2024-03-06T07:32:23.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:23.981+0000 7f9c8b488640 1 -- 172.21.15.72:0/469753533 wait complete. 2024-03-06T07:32:23.994 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:24.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.094+0000 7f4aeb10c640 1 Processor -- start 2024-03-06T07:32:24.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.094+0000 7f4aeb10c640 1 -- start start 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.094+0000 7f4aeb10c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ae40d22b0 0x7f4ae40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.094+0000 7f4aeb10c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae40b3430 con 0x7f4ae40b41f0 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.094+0000 7f4aeb10c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae40b3710 con 0x7f4ae415dfe0 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.094+0000 7f4aeb10c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae40d2c50 con 0x7f4ae40d22b0 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ae40d22b0 0x7f4ae40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae96a9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4ae40b41f0 0x7f4ae40b4680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50062/0 (socket says 172.21.15.72:50062) 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae96a9640 1 -- 172.21.15.72:0/4067894493 learned_addr learned my addr 172.21.15.72:0/4067894493 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae3fff640 1 -- 172.21.15.72:0/4067894493 >> v1:172.21.15.146:6789/0 conn(0x7f4ae415dfe0 legacy=0x7f4ae40cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae3fff640 1 -- 172.21.15.72:0/4067894493 >> v1:172.21.15.72:6789/0 conn(0x7f4ae40b41f0 legacy=0x7f4ae40b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae3fff640 1 -- 172.21.15.72:0/4067894493 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae40b39f0 con 0x7f4ae40d22b0 2024-03-06T07:32:24.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.095+0000 7f4ae37fe640 1 -- 172.21.15.72:0/4067894493 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253240762 0 0) 0x7f4ae40b3430 con 0x7f4ae40b41f0 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4ae3fff640 1 --2- 172.21.15.72:0/4067894493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40d2710 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80097d0 tx=0x7f4ad802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24fbe2ea95ee4085 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4ae37fe640 1 -- 172.21.15.72:0/4067894493 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8035020 con 0x7f4ae40d22b0 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4ae37fe640 1 -- 172.21.15.72:0/4067894493 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad8031830 con 0x7f4ae40d22b0 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4ae37fe640 1 -- 172.21.15.72:0/4067894493 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad8031c00 con 0x7f4ae40d22b0 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 -- 172.21.15.72:0/4067894493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 msgr2=0x7f4ae40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/4067894493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40d2710 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80097d0 tx=0x7f4ad802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 -- 172.21.15.72:0/4067894493 shutdown_connections 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/4067894493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40d2710 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 -- 172.21.15.72:0/4067894493 >> 172.21.15.72:0/4067894493 conn(0x7f4ae40d3cd0 msgr2=0x7f4ae40d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 -- 172.21.15.72:0/4067894493 shutdown_connections 2024-03-06T07:32:24.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.096+0000 7f4aeb10c640 1 -- 172.21.15.72:0/4067894493 wait complete. 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 Processor -- start 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 -- start start 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ae40b41f0 0x7f4ae4161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40b68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ae415dfe0 0x7f4ae415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae40b3d60 con 0x7f4ae40d22b0 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae40b31e0 con 0x7f4ae40b41f0 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.098+0000 7f4aeb10c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae40d1c70 con 0x7f4ae415dfe0 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40b68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40b68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57456/0 (socket says 172.21.15.72:57456) 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3707802093 learned_addr learned my addr 172.21.15.72:0/3707802093 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:24.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae8ea8640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ae40b41f0 0x7f4ae4161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae96a9640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ae415dfe0 0x7f4ae415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ae415dfe0 msgr2=0x7f4ae415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ae415dfe0 0x7f4ae415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ae40b41f0 msgr2=0x7f4ae4161200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ae40b41f0 0x7f4ae4161200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3707802093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae40b7000 con 0x7f4ae40d22b0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae96a9640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ae415dfe0 0x7f4ae415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae8ea8640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ae40b41f0 0x7f4ae4161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae3fff640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40b68d0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80098f0 tx=0x7f4ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad803ba80 con 0x7f4ae40d22b0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ad803bc20 con 0x7f4ae40d22b0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae40b72e0 con 0x7f4ae40d22b0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ad80453f0 con 0x7f4ae40d22b0 2024-03-06T07:32:24.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.099+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae40b78a0 con 0x7f4ae40d22b0 2024-03-06T07:32:24.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.100+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae404f330 con 0x7f4ae40d22b0 2024-03-06T07:32:24.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.101+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f4ad804c060 con 0x7f4ae40d22b0 2024-03-06T07:32:24.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.101+0000 7f4ae17fa640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4ad006b4d0 0x7f4ad006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.101+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4ad80abfc0 con 0x7f4ae40d22b0 2024-03-06T07:32:24.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.101+0000 7f4ae8ea8640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4ad006b4d0 0x7f4ad006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.102+0000 7f4ae8ea8640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4ad006b4d0 0x7f4ad006d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00c420 tx=0x7f4adc009570 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.105+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f4ad8040070 con 0x7f4ae40d22b0 2024-03-06T07:32:24.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.444+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ae415c090 con 0x7f4ae40d22b0 2024-03-06T07:32:24.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.445+0000 7f4ae17fa640 1 -- 172.21.15.72:0/3707802093 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16733 (secure 0 0 0) 0x7f4ad802f2e0 con 0x7f4ae40d22b0 2024-03-06T07:32:24.447 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:24.447 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":131,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:29:33.212437+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.21","mappings":[{"from":2,"to":0}]},{"pgid":"19.31","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4ad006b4d0 msgr2=0x7f4ad006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4ad006b4d0 0x7f4ad006d990 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00c420 tx=0x7f4adc009570 comp rx=0 tx=0).stop 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 msgr2=0x7f4ae40b68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40b68d0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80098f0 tx=0x7f4ad80046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 shutdown_connections 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f4ad006b4d0 0x7f4ad006d990 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ae415dfe0 0x7f4ae415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ae40d22b0 0x7f4ae40b68d0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 --2- 172.21.15.72:0/3707802093 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ae40b41f0 0x7f4ae4161200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 >> 172.21.15.72:0/3707802093 conn(0x7f4ae40d3cd0 msgr2=0x7f4ae406ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 shutdown_connections 2024-03-06T07:32:24.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.447+0000 7f4aeb10c640 1 -- 172.21.15.72:0/3707802093 wait complete. 2024-03-06T07:32:24.464 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:24.464 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.563+0000 7f1c38683640 1 Processor -- start 2024-03-06T07:32:24.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.563+0000 7f1c38683640 1 -- start start 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.564+0000 7f1c38683640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c300d22b0 0x7f1c300d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.564+0000 7f1c38683640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300b41f0 0x7f1c300b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.564+0000 7f1c38683640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c300b3430 con 0x7f1c3015dfe0 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.564+0000 7f1c38683640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c300d2c50 con 0x7f1c300d22b0 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.564+0000 7f1c38683640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c300b31e0 con 0x7f1c300b41f0 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c300d22b0 0x7f1c300d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1c300d22b0 0x7f1c300d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57470/0 (socket says 172.21.15.72:57470) 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 -- 172.21.15.72:0/1179616074 learned_addr learned my addr 172.21.15.72:0/1179616074 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:24.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c36c20640 1 --2- 172.21.15.72:0/1179616074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300b41f0 0x7f1c300b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 -- 172.21.15.72:0/1179616074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300b41f0 msgr2=0x7f1c300b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 --2- 172.21.15.72:0/1179616074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300b41f0 0x7f1c300b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 -- 172.21.15.72:0/1179616074 >> v1:172.21.15.146:6789/0 conn(0x7f1c3015dfe0 legacy=0x7f1c300cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.565+0000 7f1c35c1e640 1 -- 172.21.15.72:0/1179616074 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c300b3710 con 0x7f1c300d22b0 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c35c1e640 1 --2- 172.21.15.72:0/1179616074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300d22b0 0x7f1c300d2710 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009870 tx=0x7f1c2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a49089ed8872dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c3541d640 1 -- 172.21.15.72:0/1179616074 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20034070 con 0x7f1c300d22b0 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c3541d640 1 -- 172.21.15.72:0/1179616074 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20030840 con 0x7f1c300d22b0 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c3541d640 1 -- 172.21.15.72:0/1179616074 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c20030c10 con 0x7f1c300d22b0 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 -- 172.21.15.72:0/1179616074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300d22b0 msgr2=0x7f1c300d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 --2- 172.21.15.72:0/1179616074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300d22b0 0x7f1c300d2710 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009870 tx=0x7f1c2002b980 comp rx=0 tx=0).stop 2024-03-06T07:32:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 -- 172.21.15.72:0/1179616074 shutdown_connections 2024-03-06T07:32:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 --2- 172.21.15.72:0/1179616074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c300b41f0 0x7f1c300b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 --2- 172.21.15.72:0/1179616074 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300d22b0 0x7f1c300d2710 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 -- 172.21.15.72:0/1179616074 >> 172.21.15.72:0/1179616074 conn(0x7f1c300d3cd0 msgr2=0x7f1c300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.566+0000 7f1c38683640 1 -- 172.21.15.72:0/1179616074 shutdown_connections 2024-03-06T07:32:24.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.567+0000 7f1c38683640 1 -- 172.21.15.72:0/1179616074 wait complete. 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 Processor -- start 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 -- start start 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 0x7f1c300b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300d22b0 0x7f1c301f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c3015dfe0 0x7f1c301f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c300cfa00 con 0x7f1c300b41f0 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c300b3d60 con 0x7f1c300d22b0 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c38683640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c300d1c70 con 0x7f1c3015dfe0 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c3641f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 0x7f1c300b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c3641f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 0x7f1c300b6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57474/0 (socket says 172.21.15.72:57474) 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.568+0000 7f1c3641f640 1 -- 172.21.15.72:0/1896622237 learned_addr learned my addr 172.21.15.72:0/1896622237 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c36c20640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c3015dfe0 0x7f1c301f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c35c1e640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300d22b0 0x7f1c301f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c3641f640 1 -- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c3015dfe0 msgr2=0x7f1c301f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c3641f640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c3015dfe0 0x7f1c301f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c3641f640 1 -- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300d22b0 msgr2=0x7f1c301f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c3641f640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300d22b0 0x7f1c301f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c3641f640 1 -- 172.21.15.72:0/1896622237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c300b9490 con 0x7f1c300b41f0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c3641f640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 0x7f1c300b6910 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00c3f0 tx=0x7f1c2c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c0052d0 con 0x7f1c300b41f0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c2c005490 con 0x7f1c300b41f0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c300b9770 con 0x7f1c300b41f0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2c007260 con 0x7f1c300b41f0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.569+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c300b1130 con 0x7f1c300b41f0 2024-03-06T07:32:24.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.570+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3004f330 con 0x7f1c300b41f0 2024-03-06T07:32:24.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.571+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1c2c007400 con 0x7f1c300b41f0 2024-03-06T07:32:24.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.571+0000 7f1c26ffd640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1406b4d0 0x7f1c1406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.571+0000 7f1c35c1e640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1406b4d0 0x7f1c1406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.571+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1c2c0895c0 con 0x7f1c300b41f0 2024-03-06T07:32:24.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.572+0000 7f1c35c1e640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1406b4d0 0x7f1c1406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1c20036000 tx=0x7f1c20008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.575+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f1c2c057e40 con 0x7f1c300b41f0 2024-03-06T07:32:24.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.791+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c300b2f40 con 0x7f1c300b41f0 2024-03-06T07:32:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.792+0000 7f1c26ffd640 1 -- 172.21.15.72:0/1896622237 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7f1c2c05bee0 con 0x7f1c300b41f0 2024-03-06T07:32:24.794 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:24.794 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:24.794 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.793+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1406b4d0 msgr2=0x7f1c1406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.793+0000 7f1c38683640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1406b4d0 0x7f1c1406d990 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f1c20036000 tx=0x7f1c20008030 comp rx=0 tx=0).stop 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.793+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 msgr2=0x7f1c300b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.793+0000 7f1c38683640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 0x7f1c300b6910 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c00c3f0 tx=0x7f1c2c00c8b0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 shutdown_connections 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1c1406b4d0 0x7f1c1406d990 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c3015dfe0 0x7f1c301f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c300d22b0 0x7f1c301f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 --2- 172.21.15.72:0/1896622237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c300b41f0 0x7f1c300b6910 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 >> 172.21.15.72:0/1896622237 conn(0x7f1c300d3cd0 msgr2=0x7f1c3006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 shutdown_connections 2024-03-06T07:32:24.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.794+0000 7f1c38683640 1 -- 172.21.15.72:0/1896622237 wait complete. 2024-03-06T07:32:24.807 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:24.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.909+0000 7fbffc08f640 1 Processor -- start 2024-03-06T07:32:24.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.909+0000 7fbffc08f640 1 -- start start 2024-03-06T07:32:24.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.910+0000 7fbffc08f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbff40d04a0 0x7fbff40ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.910+0000 7fbffc08f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff4161980 con 0x7fbff415bb20 2024-03-06T07:32:24.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.910+0000 7fbffc08f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff4161c60 con 0x7fbff40cfff0 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.910+0000 7fbffc08f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff40d2500 con 0x7fbff40d04a0 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.910+0000 7fbff9e2b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbff415bb20 0x7fbff40d1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50078/0 (socket says 172.21.15.72:50078) 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.910+0000 7fbff9e2b640 1 -- 172.21.15.72:0/2864792066 learned_addr learned my addr 172.21.15.72:0/2864792066 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 --2- 172.21.15.72:0/2864792066 >> v2:172.21.15.146:3300/0 conn(0x7fbff40d04a0 0x7fbff40ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792943213 0 0) 0x7fbff4161980 con 0x7fbff415bb20 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe0003520 con 0x7fbff415bb20 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661525722 0 0) 0x7fbff4161c60 con 0x7fbff40cfff0 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbff4161980 con 0x7fbff40cfff0 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1080436648 0 0) 0x7fbfe0003520 con 0x7fbff415bb20 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 >> v2:172.21.15.146:3300/0 conn(0x7fbff40d04a0 msgr2=0x7fbff40ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 --2- 172.21.15.72:0/2864792066 >> v2:172.21.15.146:3300/0 conn(0x7fbff40d04a0 0x7fbff40ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 >> v1:172.21.15.146:6789/0 conn(0x7fbff40cfff0 legacy=0x7fbff40cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff4161f40 con 0x7fbff415bb20 2024-03-06T07:32:24.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fbff0001670 con 0x7fbff415bb20 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 >> v1:172.21.15.72:6789/0 conn(0x7fbff415bb20 legacy=0x7fbff40d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe0003b90 0x7fbfe0003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe0004800 0x7fbfe0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe00051c0 0x7fbfe000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40d2500 con 0x7fbfe00051c0 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe000c090 con 0x7fbfe0003b90 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe000c290 con 0x7fbfe0004800 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe00051c0 0x7fbfe000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff9e2b640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe0003b90 0x7fbfe0003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff962a640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe0004800 0x7fbfe0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 -- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe0004800 msgr2=0x7fbfe0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfe0004800 0x7fbfe0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 -- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe0003b90 msgr2=0x7fbfe0003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:24.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe0003b90 0x7fbfe0003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbffa62c640 1 -- 172.21.15.72:0/2864792066 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0003520 con 0x7fbfe00051c0 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.911+0000 7fbff9e2b640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe0003b90 0x7fbfe0003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.912+0000 7fbffa62c640 1 -- 172.21.15.72:0/2864792066 reap_dead start 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.912+0000 7fbffa62c640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe00051c0 0x7fbfe000b9c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009fc0 tx=0x7fbfe0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821f584aa73ff60d server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.912+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec009c10 con 0x7fbfe00051c0 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.912+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec02cca0 con 0x7fbfe00051c0 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.912+0000 7fbff8e29640 1 -- 172.21.15.72:0/2864792066 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec003bd0 con 0x7fbfe00051c0 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 -- 172.21.15.72:0/2864792066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe00051c0 msgr2=0x7fbfe000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe00051c0 0x7fbfe000b9c0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009fc0 tx=0x7fbfe0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 -- 172.21.15.72:0/2864792066 shutdown_connections 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 --2- 172.21.15.72:0/2864792066 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfe00051c0 0x7fbfe000b9c0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 -- 172.21.15.72:0/2864792066 >> 172.21.15.72:0/2864792066 conn(0x7fbff409a9f0 msgr2=0x7fbff409ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 -- 172.21.15.72:0/2864792066 shutdown_connections 2024-03-06T07:32:24.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.913+0000 7fbffc08f640 1 -- 172.21.15.72:0/2864792066 wait complete. 2024-03-06T07:32:24.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 Processor -- start 2024-03-06T07:32:24.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 -- start start 2024-03-06T07:32:24.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe00051c0 0x7fbff40af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 0x7fbff41f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff4159220 0x7fbff406be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40d08d0 con 0x7fbff4158da0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff4162500 con 0x7fbfe00051c0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbffc08f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff40d1980 con 0x7fbff4159220 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbff962a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 0x7fbff41f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbff962a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 0x7fbff41f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57486/0 (socket says 172.21.15.72:57486) 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.915+0000 7fbff962a640 1 -- 172.21.15.72:0/62886064 learned_addr learned my addr 172.21.15.72:0/62886064 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff9e2b640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe00051c0 0x7fbff40af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbffa62c640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff4159220 0x7fbff406be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff962a640 1 -- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff4159220 msgr2=0x7fbff406be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff962a640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff4159220 0x7fbff406be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff962a640 1 -- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe00051c0 msgr2=0x7fbff40af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff962a640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe00051c0 0x7fbff40af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff962a640 1 -- 172.21.15.72:0/62886064 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff41f4510 con 0x7fbff4158da0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbff962a640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 0x7fbff41f3ac0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400e7e0 tx=0x7fbfe400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40050d0 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe4018070 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff41f47f0 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40054b0 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.916+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff40aed70 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.917+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff40aef70 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.918+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbfe400c6c0 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.918+0000 7fbfea7fc640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfd806b4d0 0x7fbfd806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.918+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbfe4089ee0 con 0x7fbff4158da0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.918+0000 7fbff9e2b640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfd806b4d0 0x7fbfd806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:24.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.919+0000 7fbff9e2b640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfd806b4d0 0x7fbfd806d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000ff0 tx=0x7fbff0007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:24.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:24.922+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbff40aef70 con 0x7fbff4158da0 2024-03-06T07:32:25.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.137+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbff40b1820 con 0x7fbff4158da0 2024-03-06T07:32:25.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.137+0000 7fbfea7fc640 1 -- 172.21.15.72:0/62886064 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+3462 (secure 0 0 0) 0x7fbfe4058790 con 0x7fbff4158da0 2024-03-06T07:32:25.139 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:25.139 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":89,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-06T07:28:37.147617+0000","modified":"2024-03-06T07:31:08.461708+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8640,"mds_1":8637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8637":{"gid":8637,"name":"c","rank":1,"incarnation":81,"state":"up:stopping","state_seq":5,"addr":"172.21.15.72:6836/2047054467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":2047054467},{"type":"v1","addr":"172.21.15.72:6836","nonce":2047054467}]},"join_fscid":-1,"laggy_since":"2024-03-06T07:29:40.508609+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8640":{"gid":8640,"name":"a","rank":0,"incarnation":77,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/1633306088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1633306088},{"type":"v1","addr":"172.21.15.72:6837","nonce":1633306088}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8640,"qdb_cluster":[8640]},"id":9}]} 2024-03-06T07:32:25.140 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 89 2024-03-06T07:32:25.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfd806b4d0 msgr2=0x7fbfd806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfd806b4d0 0x7fbfd806d990 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fbff0000ff0 tx=0x7fbff0007040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 msgr2=0x7fbff41f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 0x7fbff41f3ac0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400e7e0 tx=0x7fbfe400eca0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 shutdown_connections 2024-03-06T07:32:25.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbfd806b4d0 0x7fbfd806d990 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff4159220 0x7fbff406be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbff4158da0 0x7fbff41f3ac0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 --2- 172.21.15.72:0/62886064 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfe00051c0 0x7fbff40af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 >> 172.21.15.72:0/62886064 conn(0x7fbff409a9f0 msgr2=0x7fbff4159cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 shutdown_connections 2024-03-06T07:32:25.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.139+0000 7fbffc08f640 1 -- 172.21.15.72:0/62886064 wait complete. 2024-03-06T07:32:25.152 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:25.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.251+0000 7f5ed508d640 1 Processor -- start 2024-03-06T07:32:25.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.251+0000 7f5ed508d640 1 -- start start 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ed508d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ed00cfff0 0x7f5ed00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ed508d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ed00cc8b0 0x7f5ed00ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ed508d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed0161980 con 0x7f5ed015bb20 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ed508d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed00d2500 con 0x7f5ed00cfff0 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ed508d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed0161c60 con 0x7f5ed00cc8b0 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ece59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ed00cfff0 0x7f5ed00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ece59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5ed00cfff0 0x7f5ed00d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57494/0 (socket says 172.21.15.72:57494) 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.252+0000 7f5ece59c640 1 -- 172.21.15.72:0/2681320196 learned_addr learned my addr 172.21.15.72:0/2681320196 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.253+0000 7f5ecf59e640 1 --2- 172.21.15.72:0/2681320196 >> v2:172.21.15.146:3300/0 conn(0x7f5ed00cc8b0 0x7f5ed00ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.253+0000 7f5ece59c640 1 -- 172.21.15.72:0/2681320196 >> v2:172.21.15.146:3300/0 conn(0x7f5ed00cc8b0 msgr2=0x7f5ed00ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.253+0000 7f5ece59c640 1 --2- 172.21.15.72:0/2681320196 >> v2:172.21.15.146:3300/0 conn(0x7f5ed00cc8b0 0x7f5ed00ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.253+0000 7f5ece59c640 1 -- 172.21.15.72:0/2681320196 >> v1:172.21.15.146:6789/0 conn(0x7f5ed015bb20 legacy=0x7f5ed00d1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.253+0000 7f5ece59c640 1 -- 172.21.15.72:0/2681320196 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0161f40 con 0x7f5ed00cfff0 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.253+0000 7f5ece59c640 1 --2- 172.21.15.72:0/2681320196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed00cfff0 0x7f5ed00d04a0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f5ed01618c0 tx=0x7f5eb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8e3c577403044 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ecdd9b640 1 -- 172.21.15.72:0/2681320196 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8034070 con 0x7f5ed00cfff0 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ecdd9b640 1 -- 172.21.15.72:0/2681320196 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb8030860 con 0x7f5ed00cfff0 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ecdd9b640 1 -- 172.21.15.72:0/2681320196 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8030c30 con 0x7f5ed00cfff0 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 -- 172.21.15.72:0/2681320196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed00cfff0 msgr2=0x7f5ed00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 --2- 172.21.15.72:0/2681320196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed00cfff0 0x7f5ed00d04a0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f5ed01618c0 tx=0x7f5eb802b9a0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 -- 172.21.15.72:0/2681320196 shutdown_connections 2024-03-06T07:32:25.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 --2- 172.21.15.72:0/2681320196 >> v2:172.21.15.146:3300/0 conn(0x7f5ed00cc8b0 0x7f5ed00ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 --2- 172.21.15.72:0/2681320196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed00cfff0 0x7f5ed00d04a0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 -- 172.21.15.72:0/2681320196 >> 172.21.15.72:0/2681320196 conn(0x7f5ed009a9f0 msgr2=0x7f5ed009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 -- 172.21.15.72:0/2681320196 shutdown_connections 2024-03-06T07:32:25.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.254+0000 7f5ed508d640 1 -- 172.21.15.72:0/2681320196 wait complete. 2024-03-06T07:32:25.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 Processor -- start 2024-03-06T07:32:25.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 -- start start 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ed00cc8b0 0x7f5ed0151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ed00cfff0 0x7f5ed0157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 0x7f5ed00b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00d1980 con 0x7f5ed015bb20 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed0162500 con 0x7f5ed00cc8b0 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ed508d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00d1b00 con 0x7f5ed00cfff0 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ecf59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 0x7f5ed00b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ecf59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 0x7f5ed00b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57498/0 (socket says 172.21.15.72:57498) 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.256+0000 7f5ecf59e640 1 -- 172.21.15.72:0/729744328 learned_addr learned my addr 172.21.15.72:0/729744328 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5eced9d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ed00cc8b0 0x7f5ed0151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ece59c640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ed00cfff0 0x7f5ed0157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ecf59e640 1 -- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ed00cfff0 msgr2=0x7f5ed0157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ecf59e640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ed00cfff0 0x7f5ed0157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ecf59e640 1 -- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ed00cc8b0 msgr2=0x7f5ed0151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ecf59e640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ed00cc8b0 0x7f5ed0151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ecf59e640 1 -- 172.21.15.72:0/729744328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed01e91c0 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5eced9d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ed00cc8b0 0x7f5ed0151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ece59c640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ed00cfff0 0x7f5ed0157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ecf59e640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 0x7f5ed00b0310 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000d5d0 tx=0x7f5ec000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00125f0 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec0019070 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed01e94a0 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec00129f0 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.257+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed01e9a60 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.258+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed004f330 con 0x7f5ed015bb20 2024-03-06T07:32:25.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.259+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f5ec0012b90 con 0x7f5ed015bb20 2024-03-06T07:32:25.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.259+0000 7f5ec77fe640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5eb006b4d0 0x7f5eb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.259+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5ec008a580 con 0x7f5ed015bb20 2024-03-06T07:32:25.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.259+0000 7f5eced9d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5eb006b4d0 0x7f5eb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.259+0000 7f5eced9d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5eb006b4d0 0x7f5eb006d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8001d60 tx=0x7f5ec8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.263+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f5ec00593d0 con 0x7f5ed015bb20 2024-03-06T07:32:25.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.440+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ed00b1820 con 0x7f5ed015bb20 2024-03-06T07:32:25.443 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.441+0000 7f5ec77fe640 1 -- 172.21.15.72:0/729744328 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16733 (secure 0 0 0) 0x7f5ec0058cd0 con 0x7f5ed015bb20 2024-03-06T07:32:25.443 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:25.443 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":131,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:29:33.212437+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-06T07:28:35.363937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-06T07:28:36.115427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"127","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.21","mappings":[{"from":2,"to":0}]},{"pgid":"19.31","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5eb006b4d0 msgr2=0x7f5eb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5eb006b4d0 0x7f5eb006d990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8001d60 tx=0x7f5ec8009040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 msgr2=0x7f5ed00b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 0x7f5ed00b0310 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000d5d0 tx=0x7f5ec000da90 comp rx=0 tx=0).stop 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 shutdown_connections 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f5eb006b4d0 0x7f5eb006d990 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5ed015bb20 0x7f5ed00b0310 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ed00cfff0 0x7f5ed0157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 --2- 172.21.15.72:0/729744328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ed00cc8b0 0x7f5ed0151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 >> 172.21.15.72:0/729744328 conn(0x7f5ed009a9f0 msgr2=0x7f5ed009add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 shutdown_connections 2024-03-06T07:32:25.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.443+0000 7f5ed508d640 1 -- 172.21.15.72:0/729744328 wait complete. 2024-03-06T07:32:25.456 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-06T07:32:25.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.568+0000 7f25cb084640 1 Processor -- start 2024-03-06T07:32:25.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.568+0000 7f25cb084640 1 -- start start 2024-03-06T07:32:25.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25cb084640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 0x7f25c4165b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25cb084640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c406cd60 con 0x7f25c4158880 2024-03-06T07:32:25.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25cb084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c406d030 con 0x7f25c415b380 2024-03-06T07:32:25.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25cb084640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c406d310 con 0x7f25c4070fe0 2024-03-06T07:32:25.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25c8e20640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f25c4158880 0x7f25c406c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50088/0 (socket says 172.21.15.72:50088) 2024-03-06T07:32:25.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25c8e20640 1 -- 172.21.15.72:0/1654110881 learned_addr learned my addr 172.21.15.72:0/1654110881 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25c9621640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 0x7f25c4165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764158913 0 0) 0x7f25c406cd60 con 0x7f25c4158880 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.569+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25b0003520 con 0x7f25c4158880 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1015822243 0 0) 0x7f25b0003520 con 0x7f25c4158880 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 msgr2=0x7f25c4165b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 0x7f25c4165b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 >> v1:172.21.15.146:6789/0 conn(0x7f25c415b380 legacy=0x7f25c40708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c406d5f0 con 0x7f25c4158880 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f25bc0019f0 con 0x7f25c4158880 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 >> v1:172.21.15.72:6789/0 conn(0x7f25c4158880 legacy=0x7f25c406c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b00039e0 0x7f25b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 0x7f25b0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c406d310 con 0x7f25b0004d40 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b000dc00 con 0x7f25b00039e0 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b000de00 con 0x7f25c4070fe0 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c9621640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c8e20640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b00039e0 0x7f25b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c3fff640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 0x7f25b0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c9621640 1 -- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 msgr2=0x7f25b0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c9621640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c4070fe0 0x7f25b0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c9621640 1 -- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b00039e0 msgr2=0x7f25b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c9621640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25b00039e0 0x7f25b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.570+0000 7f25c9621640 1 -- 172.21.15.72:0/1654110881 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b0003520 con 0x7f25b0004d40 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25c9621640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25b00051c0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f25c406e430 tx=0x7f25b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d49ee7b8c692603 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac039070 con 0x7f25b0004d40 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25ac02c630 con 0x7f25b0004d40 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25c37fe640 1 -- 172.21.15.72:0/1654110881 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ac009690 con 0x7f25b0004d40 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25cb084640 1 -- 172.21.15.72:0/1654110881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 msgr2=0x7f25b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25cb084640 1 --2- 172.21.15.72:0/1654110881 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25b00051c0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f25c406e430 tx=0x7f25b0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.571+0000 7f25c9621640 1 -- 172.21.15.72:0/1654110881 reap_dead start 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.572+0000 7f25cb084640 1 -- 172.21.15.72:0/1654110881 shutdown_connections 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.572+0000 7f25cb084640 1 -- 172.21.15.72:0/1654110881 >> 172.21.15.72:0/1654110881 conn(0x7f25c40b1c80 msgr2=0x7f25c40cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.572+0000 7f25cb084640 1 -- 172.21.15.72:0/1654110881 shutdown_connections 2024-03-06T07:32:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.572+0000 7f25cb084640 1 -- 172.21.15.72:0/1654110881 wait complete. 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.573+0000 7f25cb084640 1 Processor -- start 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.573+0000 7f25cb084640 1 -- start start 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25c40ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c40adda0 0x7f25c41f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c41f3d10 0x7f25c41f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40ce7d0 con 0x7f25b0004d40 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c406dbb0 con 0x7f25c40adda0 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c40ce950 con 0x7f25c41f3d10 2024-03-06T07:32:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25c40ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25c40ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57528/0 (socket says 172.21.15.72:57528) 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 -- 172.21.15.72:0/1963285184 learned_addr learned my addr 172.21.15.72:0/1963285184 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c9621640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c41f3d10 0x7f25c41f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c3fff640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c40adda0 0x7f25c41f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 -- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c41f3d10 msgr2=0x7f25c41f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c41f3d10 0x7f25c41f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 -- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c40adda0 msgr2=0x7f25c41f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c40adda0 0x7f25c41f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 -- 172.21.15.72:0/1963285184 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c40ae520 con 0x7f25b0004d40 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c8e20640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25c40ad860 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f25bc001f20 tx=0x7f25bc00a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc017020 con 0x7f25b0004d40 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc0133f0 con 0x7f25b0004d40 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc013800 con 0x7f25b0004d40 2024-03-06T07:32:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c40ae800 con 0x7f25b0004d40 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.574+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c40aedc0 con 0x7f25b0004d40 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.576+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c404f330 con 0x7f25b0004d40 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.576+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f25bc01a020 con 0x7f25b0004d40 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.576+0000 7f25c17fa640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f25a406b4d0 0x7f25a406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.576+0000 7f25c3fff640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f25a406b4d0 0x7f25a406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.576+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f25bc08ae30 con 0x7f25b0004d40 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.577+0000 7f25c3fff640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f25a406b4d0 0x7f25a406d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f25c415a610 tx=0x7f25b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.581+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f25bc021020 con 0x7f25b0004d40 2024-03-06T07:32:25.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.865+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f25c40d0300 con 0x7f25b0004d40 2024-03-06T07:32:25.867 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-03-06T07:32:25.865+0000 7f7b93250640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-06T07:32:25.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.866+0000 7f25c17fa640 1 -- 172.21.15.72:0/1963285184 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) v1 ==== 161+0+0 (secure 0 0 0) 0x7f25bc059c70 con 0x7f25b0004d40 2024-03-06T07:32:25.868 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-06T07:32:25.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.870+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f25a406b4d0 msgr2=0x7f25a406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.870+0000 7f25cb084640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f25a406b4d0 0x7f25a406d990 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f25c415a610 tx=0x7f25b4009040 comp rx=0 tx=0).stop 2024-03-06T07:32:25.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.870+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 msgr2=0x7f25c40ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.870+0000 7f25cb084640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25c40ad860 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f25bc001f20 tx=0x7f25bc00a730 comp rx=0 tx=0).stop 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 shutdown_connections 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f25a406b4d0 0x7f25a406d990 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c41f3d10 0x7f25c41f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c40adda0 0x7f25c41f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 --2- 172.21.15.72:0/1963285184 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f25b0004d40 0x7f25c40ad860 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 >> 172.21.15.72:0/1963285184 conn(0x7f25c40b1c80 msgr2=0x7f25c40cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 shutdown_connections 2024-03-06T07:32:25.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.871+0000 7f25cb084640 1 -- 172.21.15.72:0/1963285184 wait complete. 2024-03-06T07:32:25.885 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-06T07:32:25.908 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-03-06T07:32:25.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.985+0000 7f365ff3e640 1 Processor -- start 2024-03-06T07:32:25.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365ff3e640 1 -- start start 2024-03-06T07:32:25.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365ff3e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f365815bb40 0x7f36580b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365ff3e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f365806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365ff3e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365815ff90 con 0x7f365806d5c0 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365ff3e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365806c910 con 0x7f365815bb40 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365ff3e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3658160270 con 0x7f365815f7d0 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365dcda640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f365815bb40 0x7f36580b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365e4db640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f365806d5c0 0x7f36580cfa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50118/0 (socket says 172.21.15.72:50118) 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.986+0000 7f365e4db640 1 -- 172.21.15.72:0/1583570243 learned_addr learned my addr 172.21.15.72:0/1583570243 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365d4d9640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f365806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365dcda640 1 -- 172.21.15.72:0/1583570243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 msgr2=0x7f365806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365dcda640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f365806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365dcda640 1 -- 172.21.15.72:0/1583570243 >> v1:172.21.15.72:6789/0 conn(0x7f365806d5c0 legacy=0x7f36580cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365dcda640 1 -- 172.21.15.72:0/1583570243 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3658160550 con 0x7f365815bb40 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365ccd8640 1 -- 172.21.15.72:0/1583570243 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302747351 0 0) 0x7f365815ff90 con 0x7f365806d5c0 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365d4d9640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f365806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:25.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.987+0000 7f365dcda640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365815bb40 0x7f36580b4970 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f36480097a0 tx=0x7f364802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945305305ba9b535 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ccd8640 1 -- 172.21.15.72:0/1583570243 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648035020 con 0x7f365815bb40 2024-03-06T07:32:25.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ccd8640 1 -- 172.21.15.72:0/1583570243 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3648031830 con 0x7f365815bb40 2024-03-06T07:32:25.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ccd8640 1 -- 172.21.15.72:0/1583570243 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648031c00 con 0x7f365815bb40 2024-03-06T07:32:25.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 -- 172.21.15.72:0/1583570243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365815bb40 msgr2=0x7f36580b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365815bb40 0x7f36580b4970 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f36480097a0 tx=0x7f364802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 -- 172.21.15.72:0/1583570243 shutdown_connections 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f365806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 --2- 172.21.15.72:0/1583570243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365815bb40 0x7f36580b4970 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 -- 172.21.15.72:0/1583570243 >> 172.21.15.72:0/1583570243 conn(0x7f365806b190 msgr2=0x7f365806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 -- 172.21.15.72:0/1583570243 shutdown_connections 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.988+0000 7f365ff3e640 1 -- 172.21.15.72:0/1583570243 wait complete. 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.989+0000 7f365ff3e640 1 Processor -- start 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.989+0000 7f365ff3e640 1 -- start start 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365ff3e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 0x7f365814d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365ff3e640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f365815bb40 0x7f365814d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365ff3e640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f36581580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365ff3e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365815bf20 con 0x7f365806d5c0 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365ff3e640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365815fc30 con 0x7f365815bb40 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365ff3e640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365815c0a0 con 0x7f365815f7d0 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 0x7f365814d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 0x7f365814d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57562/0 (socket says 172.21.15.72:57562) 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 -- 172.21.15.72:0/2987675471 learned_addr learned my addr 172.21.15.72:0/2987675471 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 -- 172.21.15.72:0/2987675471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 msgr2=0x7f36581580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f36581580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 -- 172.21.15.72:0/2987675471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f365815bb40 msgr2=0x7f365814d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f365815bb40 0x7f365814d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.990+0000 7f365dcda640 1 -- 172.21.15.72:0/2987675471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f365815ee90 con 0x7f365806d5c0 2024-03-06T07:32:25.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.991+0000 7f365dcda640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 0x7f365814d5c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f364802bee0 tx=0x7f36480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.991+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648041070 con 0x7f365806d5c0 2024-03-06T07:32:25.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.991+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364803b920 con 0x7f365806d5c0 2024-03-06T07:32:25.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.991+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648035040 con 0x7f365806d5c0 2024-03-06T07:32:25.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.991+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365815c2f0 con 0x7f365806d5c0 2024-03-06T07:32:25.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.991+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365815f170 con 0x7f365806d5c0 2024-03-06T07:32:25.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.992+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36580732b0 con 0x7f365806d5c0 2024-03-06T07:32:25.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.992+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3648038020 con 0x7f365806d5c0 2024-03-06T07:32:25.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.993+0000 7f364e7fc640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f363c06b4d0 0x7f363c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:25.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.993+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5939+0+0 (secure 0 0 0) 0x7f36480abca0 con 0x7f365806d5c0 2024-03-06T07:32:25.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.993+0000 7f365d4d9640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f363c06b4d0 0x7f363c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:25.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.993+0000 7f365d4d9640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f363c06b4d0 0x7f363c06d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3654006e80 tx=0x7f3654008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:25.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:25.997+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f364807aa40 con 0x7f365806d5c0 2024-03-06T07:32:26.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.205+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3658068ff0 con 0x7f365806d5c0 2024-03-06T07:32:26.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.871+0000 7f364e7fc640 1 -- 172.21.15.72:0/2987675471 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v91) v1 ==== 104+0+0 (secure 0 0 0) 0x7f364807a340 con 0x7f365806d5c0 2024-03-06T07:32:26.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f363c06b4d0 msgr2=0x7f363c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f363c06b4d0 0x7f363c06d990 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f3654006e80 tx=0x7f3654008040 comp rx=0 tx=0).stop 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 msgr2=0x7f365814d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 0x7f365814d5c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f364802bee0 tx=0x7f36480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 shutdown_connections 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f363c06b4d0 0x7f363c06d990 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f365815f7d0 0x7f36581580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f365815bb40 0x7f365814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 --2- 172.21.15.72:0/2987675471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f365806d5c0 0x7f365814d5c0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 >> 172.21.15.72:0/2987675471 conn(0x7f365806b190 msgr2=0x7f36580ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 shutdown_connections 2024-03-06T07:32:26.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:26.874+0000 7f365ff3e640 1 -- 172.21.15.72:0/2987675471 wait complete. 2024-03-06T07:32:26.889 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-06T07:32:27.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.035+0000 7fc9d6215640 1 Processor -- start 2024-03-06T07:32:27.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.035+0000 7fc9d6215640 1 -- start start 2024-03-06T07:32:27.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9d6215640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc9d00b3fd0 0x7fc9d00b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9d6215640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9d00b2e90 0x7fc9d015dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9d6215640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d00b4490 con 0x7fc9d015a490 2024-03-06T07:32:27.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9d6215640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d00cf4a0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9d6215640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9d00b4750 con 0x7fc9d00b3fd0 2024-03-06T07:32:27.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc9d00b2e90 0x7fc9d015dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cf7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc9d015a490 0x7fc9d006df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50126/0 (socket says 172.21.15.72:50126) 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cf7fe640 1 -- 172.21.15.72:0/3904210927 learned_addr learned my addr 172.21.15.72:0/3904210927 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cffff640 1 -- 172.21.15.72:0/3904210927 >> v2:172.21.15.146:3300/0 conn(0x7fc9d00b3fd0 msgr2=0x7fc9d00b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cffff640 1 --2- 172.21.15.72:0/3904210927 >> v2:172.21.15.146:3300/0 conn(0x7fc9d00b3fd0 0x7fc9d00b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cffff640 1 -- 172.21.15.72:0/3904210927 >> v1:172.21.15.72:6789/0 conn(0x7fc9d015a490 legacy=0x7fc9d006df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9ce7fc640 1 -- 172.21.15.72:0/3904210927 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771798142 0 0) 0x7fc9d00b4490 con 0x7fc9d015a490 2024-03-06T07:32:27.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.036+0000 7fc9cffff640 1 -- 172.21.15.72:0/3904210927 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d00b4a30 con 0x7fc9d00b2e90 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9cffff640 1 --2- 172.21.15.72:0/3904210927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d015dec0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0070670 tx=0x7fc9c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2c38f983f3899e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9ce7fc640 1 -- 172.21.15.72:0/3904210927 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8039070 con 0x7fc9d00b2e90 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9ce7fc640 1 -- 172.21.15.72:0/3904210927 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c802cc80 con 0x7fc9d00b2e90 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9ce7fc640 1 -- 172.21.15.72:0/3904210927 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8034660 con 0x7fc9d00b2e90 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9d6215640 1 -- 172.21.15.72:0/3904210927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 msgr2=0x7fc9d015dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9d6215640 1 --2- 172.21.15.72:0/3904210927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d015dec0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0070670 tx=0x7fc9c802bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9d6215640 1 -- 172.21.15.72:0/3904210927 shutdown_connections 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9d6215640 1 --2- 172.21.15.72:0/3904210927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d015dec0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9d6215640 1 --2- 172.21.15.72:0/3904210927 >> v2:172.21.15.146:3300/0 conn(0x7fc9d00b3fd0 0x7fc9d00b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.037+0000 7fc9d6215640 1 -- 172.21.15.72:0/3904210927 >> 172.21.15.72:0/3904210927 conn(0x7fc9d00b1650 msgr2=0x7fc9d00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.038+0000 7fc9d6215640 1 -- 172.21.15.72:0/3904210927 shutdown_connections 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.038+0000 7fc9d6215640 1 -- 172.21.15.72:0/3904210927 wait complete. 2024-03-06T07:32:27.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.039+0000 7fc9d6215640 1 Processor -- start 2024-03-06T07:32:27.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.039+0000 7fc9d6215640 1 -- start start 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d0146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9d00b3fd0 0x7fc9d01476b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9d015a490 0x7fc9d014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00b3440 con 0x7fc9d00b2e90 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00b32c0 con 0x7fc9d00b3fd0 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d00b35c0 con 0x7fc9d015a490 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d0146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d0146c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57570/0 (socket says 172.21.15.72:57570) 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 -- 172.21.15.72:0/2842235843 learned_addr learned my addr 172.21.15.72:0/2842235843 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 -- 172.21.15.72:0/2842235843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9d015a490 msgr2=0x7fc9d014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9d015a490 0x7fc9d014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 -- 172.21.15.72:0/2842235843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9d00b3fd0 msgr2=0x7fc9d01476b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9d00b3fd0 0x7fc9d01476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 -- 172.21.15.72:0/2842235843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d0147bf0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9cf7fe640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d0146c30 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400c430 tx=0x7fc9c400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c40074e0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c40047e0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d01495a0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4004bd0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.040+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d0149b60 con 0x7fc9d00b2e90 2024-03-06T07:32:27.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.041+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d00687d0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.042+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc9c4011070 con 0x7fc9d00b2e90 2024-03-06T07:32:27.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.042+0000 7fc9b7fff640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9bc06b4d0 0x7fc9bc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:27.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.042+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5939+0+0 (secure 0 0 0) 0x7fc9c4089f60 con 0x7fc9d00b2e90 2024-03-06T07:32:27.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.042+0000 7fc9ceffd640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9bc06b4d0 0x7fc9bc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:27.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.042+0000 7fc9ceffd640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9bc06b4d0 0x7fc9bc06d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0150390 tx=0x7fc9c0006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:27.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.046+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc9d0147bf0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.218+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc9d00ae070 con 0x7fc9d00b2e90 2024-03-06T07:32:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.873+0000 7fc9b7fff640 1 -- 172.21.15.72:0/2842235843 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc9c4058cf0 con 0x7fc9d00b2e90 2024-03-06T07:32:27.875 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-03-06T07:32:27.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9bc06b4d0 msgr2=0x7fc9bc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9bc06b4d0 0x7fc9bc06d990 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0150390 tx=0x7fc9c0006c90 comp rx=0 tx=0).stop 2024-03-06T07:32:27.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 msgr2=0x7fc9d0146c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:27.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d0146c30 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400c430 tx=0x7fc9c400c8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 shutdown_connections 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc9bc06b4d0 0x7fc9bc06d990 secure :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0150390 tx=0x7fc9c0006c90 comp rx=0 tx=0).stop 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc9d015a490 0x7fc9d014a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc9d00b3fd0 0x7fc9d01476b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 --2- 172.21.15.72:0/2842235843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc9d00b2e90 0x7fc9d0146c30 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.877+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 >> 172.21.15.72:0/2842235843 conn(0x7fc9d00b1650 msgr2=0x7fc9d00ce810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.879+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 shutdown_connections 2024-03-06T07:32:27.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.879+0000 7fc9d6215640 1 -- 172.21.15.72:0/2842235843 wait complete. 2024-03-06T07:32:27.894 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-06T07:32:28.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.998+0000 7ff81bb81640 1 Processor -- start 2024-03-06T07:32:28.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.998+0000 7ff81bb81640 1 -- start start 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.998+0000 7ff81bb81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81bb81640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81bb81640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff81406e4d0 con 0x7ff8140b2e90 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81bb81640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8140b4490 con 0x7ff8140b3fd0 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81bb81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140cf4a0 con 0x7ff81415a490 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57584/0 (socket says 172.21.15.72:57584) 2024-03-06T07:32:28.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 -- 172.21.15.72:0/2974707466 learned_addr learned my addr 172.21.15.72:0/2974707466 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:28.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 -- 172.21.15.72:0/2974707466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 msgr2=0x7ff81406df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 --2- 172.21.15.72:0/2974707466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 -- 172.21.15.72:0/2974707466 >> v1:172.21.15.146:6789/0 conn(0x7ff8140b2e90 legacy=0x7ff8140708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:28.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:27.999+0000 7ff81911c640 1 -- 172.21.15.72:0/2974707466 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8140b4750 con 0x7ff8140b3fd0 2024-03-06T07:32:28.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81911c640 1 --2- 172.21.15.72:0/2974707466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7ff804009870 tx=0x7ff80402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e8266618c1f190 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81891b640 1 -- 172.21.15.72:0/2974707466 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804035020 con 0x7ff8140b3fd0 2024-03-06T07:32:28.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81891b640 1 -- 172.21.15.72:0/2974707466 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff804031640 con 0x7ff8140b3fd0 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81891b640 1 -- 172.21.15.72:0/2974707466 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80402d040 con 0x7ff8140b3fd0 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81bb81640 1 -- 172.21.15.72:0/2974707466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b3fd0 msgr2=0x7ff8140b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2974707466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7ff804009870 tx=0x7ff80402b8f0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81bb81640 1 -- 172.21.15.72:0/2974707466 shutdown_connections 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2974707466 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b3fd0 0x7ff8140b2950 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2974707466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81406df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.000+0000 7ff81bb81640 1 -- 172.21.15.72:0/2974707466 >> 172.21.15.72:0/2974707466 conn(0x7ff8140b1650 msgr2=0x7ff8140b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.001+0000 7ff81bb81640 1 -- 172.21.15.72:0/2974707466 shutdown_connections 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.001+0000 7ff81bb81640 1 -- 172.21.15.72:0/2974707466 wait complete. 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 Processor -- start 2024-03-06T07:32:28.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 -- start start 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 0x7ff814144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8140b3fd0 0x7ff81414ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81415c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140cf260 con 0x7ff8140b2e90 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140cf0e0 con 0x7ff8140b3fd0 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81bb81640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140b3320 con 0x7ff81415a490 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81991d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 0x7ff814144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81991d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 0x7ff814144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57588/0 (socket says 172.21.15.72:57588) 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.002+0000 7ff81991d640 1 -- 172.21.15.72:0/2235108807 learned_addr learned my addr 172.21.15.72:0/2235108807 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81991d640 1 -- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 msgr2=0x7ff81415c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81a11e640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81991d640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81415c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81911c640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8140b3fd0 0x7ff81414ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81991d640 1 -- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8140b3fd0 msgr2=0x7ff81414ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81991d640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8140b3fd0 0x7ff81414ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81991d640 1 -- 172.21.15.72:0/2235108807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8141481c0 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81991d640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 0x7ff814144930 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b410 tx=0x7ff81000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810015020 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff810011850 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff814147160 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810011c40 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.003+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8141476e0 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.004+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8140687d0 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.004+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7ff810018020 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.005+0000 7ff80a7fc640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f806b4d0 0x7ff7f806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.005+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5501+0+0 (secure 0 0 0) 0x7ff810089f40 con 0x7ff8140b2e90 2024-03-06T07:32:28.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.005+0000 7ff81911c640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f806b4d0 0x7ff7f806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:28.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.005+0000 7ff81911c640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f806b4d0 0x7ff7f806d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff804036000 tx=0x7ff80403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:28.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.009+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7ff81001f020 con 0x7ff8140b2e90 2024-03-06T07:32:28.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.185+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff8140b01e0 con 0x7ff8140b2e90 2024-03-06T07:32:28.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.878+0000 7ff80a7fc640 1 -- 172.21.15.72:0/2235108807 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff8100172c0 con 0x7ff8140b2e90 2024-03-06T07:32:28.881 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-03-06T07:32:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f806b4d0 msgr2=0x7ff7f806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f806b4d0 0x7ff7f806d990 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff804036000 tx=0x7ff80403c040 comp rx=0 tx=0).stop 2024-03-06T07:32:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 msgr2=0x7ff814144930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:28.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 0x7ff814144930 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7ff81000b410 tx=0x7ff81000b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 shutdown_connections 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7ff7f806b4d0 0x7ff7f806d990 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff81415a490 0x7ff81415c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8140b3fd0 0x7ff81414ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 --2- 172.21.15.72:0/2235108807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8140b2e90 0x7ff814144930 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 >> 172.21.15.72:0/2235108807 conn(0x7ff8140b1650 msgr2=0x7ff81414ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 shutdown_connections 2024-03-06T07:32:28.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.882+0000 7ff81bb81640 1 -- 172.21.15.72:0/2235108807 wait complete. 2024-03-06T07:32:28.897 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-06T07:32:29.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.999+0000 7fbef25eb640 1 Processor -- start 2024-03-06T07:32:29.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:28.999+0000 7fbef25eb640 1 -- start start 2024-03-06T07:32:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbef25eb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbeec0cfa90 0x7fbeec0cff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbef25eb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeec158ac0 con 0x7fbeec0b33a0 2024-03-06T07:32:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbef25eb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeec15b2a0 con 0x7fbeec1586e0 2024-03-06T07:32:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbef25eb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeec15b540 con 0x7fbeec0cfa90 2024-03-06T07:32:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbeeb7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbeec0cfa90 0x7fbeec0cff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbef0b88640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbeec0b33a0 0x7fbeec0b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50142/0 (socket says 172.21.15.72:50142) 2024-03-06T07:32:29.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbef0b88640 1 -- 172.21.15.72:0/4202659680 learned_addr learned my addr 172.21.15.72:0/4202659680 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:29.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/4202659680 >> v1:172.21.15.146:6789/0 conn(0x7fbeec1586e0 legacy=0x7fbeec0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/4202659680 >> v1:172.21.15.72:6789/0 conn(0x7fbeec0b33a0 legacy=0x7fbeec0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbeeaffd640 1 -- 172.21.15.72:0/4202659680 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061217510 0 0) 0x7fbeec158ac0 con 0x7fbeec0b33a0 2024-03-06T07:32:29.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.000+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/4202659680 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec15b820 con 0x7fbeec0cfa90 2024-03-06T07:32:29.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbeeb7fe640 1 --2- 172.21.15.72:0/4202659680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec0cff40 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0097a0 tx=0x7fbedc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d402ff18d177415 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbeeaffd640 1 -- 172.21.15.72:0/4202659680 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc035020 con 0x7fbeec0cfa90 2024-03-06T07:32:29.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbeeaffd640 1 -- 172.21.15.72:0/4202659680 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc031830 con 0x7fbeec0cfa90 2024-03-06T07:32:29.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbeeaffd640 1 -- 172.21.15.72:0/4202659680 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc031c00 con 0x7fbeec0cfa90 2024-03-06T07:32:29.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbef25eb640 1 -- 172.21.15.72:0/4202659680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 msgr2=0x7fbeec0cff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbef25eb640 1 --2- 172.21.15.72:0/4202659680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec0cff40 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0097a0 tx=0x7fbedc02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbef25eb640 1 -- 172.21.15.72:0/4202659680 shutdown_connections 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.001+0000 7fbef25eb640 1 --2- 172.21.15.72:0/4202659680 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec0cff40 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.002+0000 7fbef25eb640 1 -- 172.21.15.72:0/4202659680 >> 172.21.15.72:0/4202659680 conn(0x7fbeec06b190 msgr2=0x7fbeec06b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.002+0000 7fbef25eb640 1 -- 172.21.15.72:0/4202659680 shutdown_connections 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.002+0000 7fbef25eb640 1 -- 172.21.15.72:0/4202659680 wait complete. 2024-03-06T07:32:29.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.003+0000 7fbef25eb640 1 Processor -- start 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.003+0000 7fbef25eb640 1 -- start start 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.003+0000 7fbef25eb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbeec0b33a0 0x7fbeec141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.003+0000 7fbef25eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.003+0000 7fbef25eb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbeec1586e0 0x7fbeec150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbef25eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0af150 con 0x7fbeec0cfa90 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbef25eb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec15bde0 con 0x7fbeec1586e0 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbef25eb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec071350 con 0x7fbeec0b33a0 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec150520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57606/0 (socket says 172.21.15.72:57606) 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/3205742307 learned_addr learned my addr 172.21.15.72:0/3205742307 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/3205742307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbeec0b33a0 msgr2=0x7fbeec141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbeec0b33a0 0x7fbeec141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/3205742307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbeec1586e0 msgr2=0x7fbeec150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbeec1586e0 0x7fbeec150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 -- 172.21.15.72:0/3205742307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec143e30 con 0x7fbeec0cfa90 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbeeb7fe640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec150520 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0098c0 tx=0x7fbedc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc004230 con 0x7fbeec0cfa90 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbedc0043d0 con 0x7fbeec0cfa90 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeec06d620 con 0x7fbeec0cfa90 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbedc03cad0 con 0x7fbeec0cfa90 2024-03-06T07:32:29.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.004+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec1443d0 con 0x7fbeec0cfa90 2024-03-06T07:32:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.005+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec0732b0 con 0x7fbeec0cfa90 2024-03-06T07:32:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.006+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbedc038020 con 0x7fbeec0cfa90 2024-03-06T07:32:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.006+0000 7fbee8ff9640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbed806b500 0x7fbed806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.006+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbedc0ac170 con 0x7fbeec0cfa90 2024-03-06T07:32:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.006+0000 7fbeebfff640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbed806b500 0x7fbed806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.006+0000 7fbeebfff640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbed806b500 0x7fbed806d9c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009650 tx=0x7fbee0006ba0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.010+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbedc041070 con 0x7fbeec0cfa90 2024-03-06T07:32:29.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.190+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbeec068ff0 con 0x7fbeec0cfa90 2024-03-06T07:32:29.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.194+0000 7fbee8ff9640 1 -- 172.21.15.72:0/3205742307 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbedc07b290 con 0x7fbeec0cfa90 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbed806b500 msgr2=0x7fbed806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbed806b500 0x7fbed806d9c0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fbee0009650 tx=0x7fbee0006ba0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 msgr2=0x7fbeec150520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec150520 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0098c0 tx=0x7fbedc0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 shutdown_connections 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbed806b500 0x7fbed806d9c0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbeec1586e0 0x7fbeec150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbeec0cfa90 0x7fbeec150520 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 --2- 172.21.15.72:0/3205742307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbeec0b33a0 0x7fbeec141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 >> 172.21.15.72:0/3205742307 conn(0x7fbeec06b190 msgr2=0x7fbeec0b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 shutdown_connections 2024-03-06T07:32:29.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.196+0000 7fbef25eb640 1 -- 172.21.15.72:0/3205742307 wait complete. 2024-03-06T07:32:29.209 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.307+0000 7f8fdee88640 1 Processor -- start 2024-03-06T07:32:29.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.307+0000 7f8fdee88640 1 -- start start 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdee88640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8fd80b3060 0x7f8fd815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdee88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 0x7f8fd815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdee88640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd815f2d0 con 0x7f8fd813dc10 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdee88640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd80cf3f0 con 0x7f8fd80b3060 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdee88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd815b210 con 0x7f8fd815bf30 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdcc24640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8fd813dc10 0x7f8fd80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50152/0 (socket says 172.21.15.72:50152) 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdcc24640 1 -- 172.21.15.72:0/346718754 learned_addr learned my addr 172.21.15.72:0/346718754 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fdd425640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 0x7f8fd815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd7fff640 1 --2- 172.21.15.72:0/346718754 >> v2:172.21.15.146:3300/0 conn(0x7f8fd80b3060 0x7f8fd815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835255493 0 0) 0x7f8fd815f2d0 con 0x7f8fd813dc10 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc8003520 con 0x7f8fd813dc10 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809943214 0 0) 0x7f8fc8003520 con 0x7f8fd813dc10 2024-03-06T07:32:29.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 msgr2=0x7f8fd815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 0x7f8fd815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 >> v2:172.21.15.146:3300/0 conn(0x7f8fd80b3060 msgr2=0x7f8fd815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 --2- 172.21.15.72:0/346718754 >> v2:172.21.15.146:3300/0 conn(0x7f8fd80b3060 0x7f8fd815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd815f580 con 0x7f8fd813dc10 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f8fcc001670 con 0x7f8fd813dc10 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 >> v1:172.21.15.72:6789/0 conn(0x7f8fd813dc10 legacy=0x7f8fd80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 0x7f8fc80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fc8003f30 0x7f8fc8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fc8004d40 0x7f8fc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd80cf3f0 con 0x7f8fc8004d40 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd815b210 con 0x7f8fc8003f30 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.308+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc800ba10 con 0x7f8fd815bf30 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fc8004d40 0x7f8fc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdcc24640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 0x7f8fc80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fd7fff640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fc8003f30 0x7f8fc8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 -- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 msgr2=0x7f8fc80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd815bf30 0x7f8fc80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 -- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fc8003f30 msgr2=0x7f8fc8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fc8003f30 0x7f8fc8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 -- 172.21.15.72:0/346718754 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc8003520 con 0x7f8fc8004d40 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fd7fff640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fc8003f30 0x7f8fc8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.309+0000 7f8fdd425640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fc8004d40 0x7f8fc80051c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000e170 tx=0x7f8fc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa6d3f732b366a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0039070 con 0x7f8fc8004d40 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd002cbf0 con 0x7f8fc8004d40 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fd77fe640 1 -- 172.21.15.72:0/346718754 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0009c10 con 0x7f8fc8004d40 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdee88640 1 -- 172.21.15.72:0/346718754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fc8004d40 msgr2=0x7f8fc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdee88640 1 --2- 172.21.15.72:0/346718754 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fc8004d40 0x7f8fc80051c0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000e170 tx=0x7f8fc8005830 comp rx=0 tx=0).stop 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdd425640 1 -- 172.21.15.72:0/346718754 reap_dead start 2024-03-06T07:32:29.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdee88640 1 -- 172.21.15.72:0/346718754 shutdown_connections 2024-03-06T07:32:29.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdee88640 1 -- 172.21.15.72:0/346718754 >> 172.21.15.72:0/346718754 conn(0x7f8fd80cf8a0 msgr2=0x7f8fd80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdee88640 1 -- 172.21.15.72:0/346718754 shutdown_connections 2024-03-06T07:32:29.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.310+0000 7f8fdee88640 1 -- 172.21.15.72:0/346718754 wait complete. 2024-03-06T07:32:29.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.311+0000 7f8fdee88640 1 Processor -- start 2024-03-06T07:32:29.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.311+0000 7f8fdee88640 1 -- start start 2024-03-06T07:32:29.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdee88640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd80cda10 0x7f8fd813ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdee88640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd80cddf0 0x7f8fd81f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdee88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 0x7f8fd8155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdee88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd813dff0 con 0x7f8fd8154290 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdee88640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd815b730 con 0x7f8fd80cda10 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdee88640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd815ed90 con 0x7f8fd80cddf0 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdd425640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 0x7f8fd8155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdd425640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 0x7f8fd8155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57618/0 (socket says 172.21.15.72:57618) 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdd425640 1 -- 172.21.15.72:0/2156849428 learned_addr learned my addr 172.21.15.72:0/2156849428 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fd7fff640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd80cddf0 0x7f8fd81f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.312+0000 7f8fdcc24640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd80cda10 0x7f8fd813ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdd425640 1 -- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd80cddf0 msgr2=0x7f8fd81f4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdd425640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd80cddf0 0x7f8fd81f4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdd425640 1 -- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd80cda10 msgr2=0x7f8fd813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdd425640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd80cda10 0x7f8fd813ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdd425640 1 -- 172.21.15.72:0/2156849428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd813f4b0 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdcc24640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd80cda10 0x7f8fd813ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdd425640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 0x7f8fd8155eb0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000e170 tx=0x7f8fd0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0039070 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fd003a040 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd813f790 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0009c10 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.313+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd8153d70 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.314+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd813f990 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.314+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f8fd0046070 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.315+0000 7f8fd57fa640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fbc06b500 0x7f8fbc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.315+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8fd00aba20 con 0x7f8fd8154290 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.315+0000 7f8fdcc24640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fbc06b500 0x7f8fbc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:29.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.315+0000 7f8fdcc24640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fbc06b500 0x7f8fbc06d9c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc007fd0 tx=0x7f8fcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:29.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.319+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f8fd007ab40 con 0x7f8fd8154290 2024-03-06T07:32:29.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.558+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) v1 -- 0x7f8fd804f330 con 0x7f8fd8154290 2024-03-06T07:32:29.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.880+0000 7f8fd57fa640 1 -- 172.21.15.72:0/2156849428 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v553) v1 ==== 140+0+0 (secure 0 0 0) 0x7f8fd007a440 con 0x7f8fd8154290 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.882+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fbc06b500 msgr2=0x7f8fbc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.882+0000 7f8fdee88640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fbc06b500 0x7f8fbc06d9c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc007fd0 tx=0x7f8fcc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.882+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 msgr2=0x7f8fd8155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.882+0000 7f8fdee88640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 0x7f8fd8155eb0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000e170 tx=0x7f8fd0009750 comp rx=0 tx=0).stop 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 shutdown_connections 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f8fbc06b500 0x7f8fbc06d9c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8fd8154290 0x7f8fd8155eb0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8fd80cddf0 0x7f8fd81f4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 --2- 172.21.15.72:0/2156849428 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8fd80cda10 0x7f8fd813ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 >> 172.21.15.72:0/2156849428 conn(0x7f8fd80cf8a0 msgr2=0x7f8fd8071670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:29.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 shutdown_connections 2024-03-06T07:32:29.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:29.883+0000 7f8fdee88640 1 -- 172.21.15.72:0/2156849428 wait complete. 2024-03-06T07:32:29.897 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ERROR 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:ERROR: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 734, in test_migrate_unlinked_dir 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:29.898 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:Ran 8 tests in 614.351s 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:ERROR: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/test_strays.py", line 734, in test_migrate_unlinked_dir 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: self.fs.wait_for_daemons(timeout=120) 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: raise RuntimeError("Timed out waiting for MDS daemons to become healthy") 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner:RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-06T07:32:29.899 INFO:tasks.cephfs_test_runner: 2024-03-06T07:32:29.900 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:30.093 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dd16d3fdc1494e8b93c9c82217f1cca4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1bc250c71294e65f0a1e8e3ebd8b2a618c6b57bd/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-03-06T07:32:30.095 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-06T07:32:30.104 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-06T07:32:30.141 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-06T07:32:30.142 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:32:30.142 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:30.161 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:30.161 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-06T07:32:30.161 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:30.216 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056331,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2223 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248812k,nr_inodes=812203,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497628k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-06T07:32:30.216 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:32:30.216 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:30.235 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:30.236 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-03-06T07:32:30.236 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-06T07:32:30.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18517 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-06T07:32:30.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.294 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-03-06T07:32:30.352 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-06T07:32:30.352 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-06T07:32:30.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.353 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-06T07:32:30.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-06T07:32:30.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-03-06T07:32:30.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.593 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-03-06T07:32:30.608 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-06T07:32:30.609 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-06T07:32:30.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.609 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-06T07:32:30.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.676 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-06T07:32:30.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-06T07:32:30.744 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-03-06T07:32:30.838 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-06T07:32:30.848 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-06T07:32:30.848 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.948+0000 7fe1a1ef4640 1 Processor -- start 2024-03-06T07:32:30.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.948+0000 7fe1a1ef4640 1 -- start start 2024-03-06T07:32:30.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe1a1ef4640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe19c13dc10 0x7fe19c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe1a1ef4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 0x7fe19c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe1a1ef4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c15f580 con 0x7fe19c15bf30 2024-03-06T07:32:30.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe1a1ef4640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19c0cf3f0 con 0x7fe19c13dc10 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe1a1ef4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c15b210 con 0x7fe19c15ef60 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe19affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe19c15bf30 0x7fe19c15e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50162/0 (socket says 172.21.15.72:50162) 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe19affd640 1 -- 172.21.15.72:0/1687688112 learned_addr learned my addr 172.21.15.72:0/1687688112 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365086405 0 0) 0x7fe19c15f580 con 0x7fe19c15bf30 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.949+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe184003520 con 0x7fe19c15bf30 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19bfff640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 0x7fe19c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19b7fe640 1 --2- 172.21.15.72:0/1687688112 >> v2:172.21.15.146:3300/0 conn(0x7fe19c13dc10 0x7fe19c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127937413 0 0) 0x7fe184003520 con 0x7fe19c15bf30 2024-03-06T07:32:30.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 msgr2=0x7fe19c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 0x7fe19c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 >> v2:172.21.15.146:3300/0 conn(0x7fe19c13dc10 msgr2=0x7fe19c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 --2- 172.21.15.72:0/1687688112 >> v2:172.21.15.146:3300/0 conn(0x7fe19c13dc10 0x7fe19c0b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c15b450 con 0x7fe19c15bf30 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fe190001670 con 0x7fe19c15bf30 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 >> v1:172.21.15.72:6789/0 conn(0x7fe19c15bf30 legacy=0x7fe19c15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 0x7fe1840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe184003f30 0x7fe184004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe184004d40 0x7fe1840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0cf3f0 con 0x7fe184003f30 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c15b210 con 0x7fe184004d40 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18400ba10 con 0x7fe19c15ef60 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19affd640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe184003f30 0x7fe184004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19b7fe640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 0x7fe1840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19bfff640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe184004d40 0x7fe1840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19affd640 1 -- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 msgr2=0x7fe1840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19affd640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c15ef60 0x7fe1840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19affd640 1 -- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe184004d40 msgr2=0x7fe1840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19affd640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe184004d40 0x7fe1840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.950+0000 7fe19affd640 1 -- 172.21.15.72:0/1687688112 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe184003520 con 0x7fe184003f30 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.951+0000 7fe19affd640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe184003f30 0x7fe184004800 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c210 tx=0x7fe184005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913665a5cfbfb47b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.951+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190007d50 con 0x7fe184003f30 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.951+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190030070 con 0x7fe184003f30 2024-03-06T07:32:30.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.951+0000 7fe19a7fc640 1 -- 172.21.15.72:0/1687688112 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19002cd20 con 0x7fe184003f30 2024-03-06T07:32:30.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/1687688112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe184003f30 msgr2=0x7fe184004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/1687688112 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe184003f30 0x7fe184004800 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c210 tx=0x7fe184005830 comp rx=0 tx=0).stop 2024-03-06T07:32:30.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe19bfff640 1 -- 172.21.15.72:0/1687688112 reap_dead start 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/1687688112 shutdown_connections 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/1687688112 >> 172.21.15.72:0/1687688112 conn(0x7fe19c0cf8a0 msgr2=0x7fe19c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/1687688112 shutdown_connections 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.952+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/1687688112 wait complete. 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.953+0000 7fe1a1ef4640 1 Processor -- start 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.953+0000 7fe1a1ef4640 1 -- start start 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe1a1ef4640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe184003f30 0x7fe19c13ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe1a1ef4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 0x7fe19c143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe1a1ef4640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c1441f0 0x7fe19c15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe1a1ef4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0b2e90 con 0x7fe19c13e6a0 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe1a1ef4640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c13dff0 con 0x7fe19c1441f0 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe1a1ef4640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0b3010 con 0x7fe184003f30 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 0x7fe19c143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 0x7fe19c143cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57632/0 (socket says 172.21.15.72:57632) 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 -- 172.21.15.72:0/2788308863 learned_addr learned my addr 172.21.15.72:0/2788308863 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19bfff640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c1441f0 0x7fe19c15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19b7fe640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe184003f30 0x7fe19c13ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 -- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe184003f30 msgr2=0x7fe19c13ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe184003f30 0x7fe19c13ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 -- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c1441f0 msgr2=0x7fe19c15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c1441f0 0x7fe19c15d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 -- 172.21.15.72:0/2788308863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c1f01e0 con 0x7fe19c13e6a0 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.954+0000 7fe19affd640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 0x7fe19c143cb0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c330 tx=0x7fe19c15b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.955+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190045070 con 0x7fe19c13e6a0 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.955+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190004bf0 con 0x7fe19c13e6a0 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.955+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190001cb0 con 0x7fe19c13e6a0 2024-03-06T07:32:30.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.955+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c13eb20 con 0x7fe19c13e6a0 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.955+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c142ef0 con 0x7fe19c13e6a0 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.956+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c04f330 con 0x7fe19c13e6a0 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.956+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe190048070 con 0x7fe19c13e6a0 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.956+0000 7fe183fff640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18806b470 0x7fe18806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.957+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe1900ab190 con 0x7fe19c13e6a0 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.957+0000 7fe19b7fe640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18806b470 0x7fe18806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:30.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.957+0000 7fe19b7fe640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18806b470 0x7fe18806d930 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009b00 tx=0x7fe18c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:30.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:30.961+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe19007a340 con 0x7fe19c13e6a0 2024-03-06T07:32:31.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.131+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe19c14f000 con 0x7fe18806b470 2024-03-06T07:32:31.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.132+0000 7fe183fff640 1 -- 172.21.15.72:0/2788308863 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13422 (secure 0 0 0) 0x7fe19c14f000 con 0x7fe18806b470 2024-03-06T07:32:31.134 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:31.134 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:32:31.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18806b470 msgr2=0x7fe18806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18806b470 0x7fe18806d930 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009b00 tx=0x7fe18c008040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 msgr2=0x7fe19c143cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 0x7fe19c143cb0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c330 tx=0x7fe19c15b620 comp rx=0 tx=0).stop 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 shutdown_connections 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe18806b470 0x7fe18806d930 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe19c1441f0 0x7fe19c15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe19c13e6a0 0x7fe19c143cb0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 --2- 172.21.15.72:0/2788308863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe184003f30 0x7fe19c13ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 >> 172.21.15.72:0/2788308863 conn(0x7fe19c0cf8a0 msgr2=0x7fe19c06db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 shutdown_connections 2024-03-06T07:32:31.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.134+0000 7fe1a1ef4640 1 -- 172.21.15.72:0/2788308863 wait complete. 2024-03-06T07:32:31.147 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-03-06T07:32:30.062944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330160,"kb_used_data":14168,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644560,"statfs":{"total":773094113280,"available":772756029440,"internally_reserved":0,"allocated":14508032,"data_stored":7019930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33900,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-150,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000884"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":373,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-06T07:29:42.019042+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:29:42.019042+0000","last_peered":"2024-03-06T07:29:42.019042+0000","last_clean":"2024-03-06T07:29:42.019042+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:29:42.019042+0000","last_undegraded":"2024-03-06T07:29:42.019042+0000","last_fullsized":"2024-03-06T07:29:42.019042+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574980,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41712,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330128,"statfs":{"total":96636764160,"available":96594051072,"internally_reserved":0,"allocated":2269184,"data_stored":1326420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41140,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330700,"statfs":{"total":96636764160,"available":96594636800,"internally_reserved":0,"allocated":1679360,"data_stored":741747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574981,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41128,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330712,"statfs":{"total":96636764160,"available":96594649088,"internally_reserved":0,"allocated":1667072,"data_stored":723876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1650688,"data_stored":723072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574982,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41144,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1683456,"data_stored":737429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41104,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330736,"statfs":{"total":96636764160,"available":96594673664,"internally_reserved":0,"allocated":1646592,"data_stored":717397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41132,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330708,"statfs":{"total":96636764160,"available":96594644992,"internally_reserved":0,"allocated":1671168,"data_stored":736619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41688,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330152,"statfs":{"total":96636764160,"available":96594075648,"internally_reserved":0,"allocated":2240512,"data_stored":1313370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:31.147 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:31.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.245+0000 7f9bc169a640 1 Processor -- start 2024-03-06T07:32:31.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bc169a640 1 -- start start 2024-03-06T07:32:31.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bc169a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9bbc15dfe0 0x7f9bbc0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bc169a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0cf830 0x7f9bbc0d22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bc169a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bbc0b3430 con 0x7f9bbc0d27f0 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bc169a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bbc0d2c50 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bc169a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0b31e0 con 0x7f9bbc0cf830 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.246+0000 7f9bbaffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9bbc15dfe0 0x7f9bbc0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9bbc15dfe0 0x7f9bbc0cf2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57646/0 (socket says 172.21.15.72:57646) 2024-03-06T07:32:31.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 -- 172.21.15.72:0/904805340 learned_addr learned my addr 172.21.15.72:0/904805340 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:31.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 -- 172.21.15.72:0/904805340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0cf830 msgr2=0x7f9bbc0d22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 --2- 172.21.15.72:0/904805340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0cf830 0x7f9bbc0d22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 -- 172.21.15.72:0/904805340 >> v1:172.21.15.146:6789/0 conn(0x7f9bbc0d27f0 legacy=0x7f9bbc0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 -- 172.21.15.72:0/904805340 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0b3710 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.247+0000 7f9bbaffd640 1 --2- 172.21.15.72:0/904805340 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc0cf2f0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009e50 tx=0x7f9bac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b259a9456f8c42b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bb9ffb640 1 -- 172.21.15.72:0/904805340 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac02ca70 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bb9ffb640 1 -- 172.21.15.72:0/904805340 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac02cc10 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bb9ffb640 1 -- 172.21.15.72:0/904805340 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0343f0 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 -- 172.21.15.72:0/904805340 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 msgr2=0x7f9bbc0cf2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 --2- 172.21.15.72:0/904805340 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc0cf2f0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f9bac009e50 tx=0x7f9bac02bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 -- 172.21.15.72:0/904805340 shutdown_connections 2024-03-06T07:32:31.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 --2- 172.21.15.72:0/904805340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0cf830 0x7f9bbc0d22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 --2- 172.21.15.72:0/904805340 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc0cf2f0 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 -- 172.21.15.72:0/904805340 >> 172.21.15.72:0/904805340 conn(0x7f9bbc0d3cd0 msgr2=0x7f9bbc0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 -- 172.21.15.72:0/904805340 shutdown_connections 2024-03-06T07:32:31.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.248+0000 7f9bc169a640 1 -- 172.21.15.72:0/904805340 wait complete. 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.249+0000 7f9bc169a640 1 Processor -- start 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 -- start start 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bbc0cf830 0x7f9bbc14dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0d27f0 0x7f9bbc15f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc160670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0d1c70 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0b3d60 con 0x7f9bbc0cf830 2024-03-06T07:32:31.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bc169a640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bbc0d1df0 con 0x7f9bbc0d27f0 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bbb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bbb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc160670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57662/0 (socket says 172.21.15.72:57662) 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bbb7fe640 1 -- 172.21.15.72:0/3648319746 learned_addr learned my addr 172.21.15.72:0/3648319746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bbaffd640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bbc0cf830 0x7f9bbc14dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.250+0000 7f9bba7fc640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0d27f0 0x7f9bbc15f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bbb7fe640 1 -- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0d27f0 msgr2=0x7f9bbc15f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bbb7fe640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0d27f0 0x7f9bbc15f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bbb7fe640 1 -- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bbc0cf830 msgr2=0x7f9bbc14dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bbb7fe640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bbc0cf830 0x7f9bbc14dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bbb7fe640 1 -- 172.21.15.72:0/3648319746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bbc0cada0 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bbb7fe640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc160670 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400c3c0 tx=0x7f9bb400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb4017070 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb40035f0 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bbc0cb080 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb40039e0 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.251+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bbc160150 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.252+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bbc160350 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.253+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9bb401a070 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.253+0000 7f9ba37fe640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ba806b4d0 0x7f9ba806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.253+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9bb40891d0 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.253+0000 7f9bbaffd640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ba806b4d0 0x7f9ba806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.253+0000 7f9bbaffd640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ba806b4d0 0x7f9ba806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9bac035000 tx=0x7f9bac030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.257+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9bbc160350 con 0x7f9bbc15dfe0 2024-03-06T07:32:31.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.423+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bbc04f330 con 0x7f9ba806b4d0 2024-03-06T07:32:31.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.424+0000 7f9ba37fe640 1 -- 172.21.15.72:0/3648319746 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13422 (secure 0 0 0) 0x7f9bbc04f330 con 0x7f9ba806b4d0 2024-03-06T07:32:31.426 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:31.426 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:32:31.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ba806b4d0 msgr2=0x7f9ba806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ba806b4d0 0x7f9ba806d990 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f9bac035000 tx=0x7f9bac030000 comp rx=0 tx=0).stop 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 msgr2=0x7f9bbc160670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc160670 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400c3c0 tx=0x7f9bb400c7c0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 shutdown_connections 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9ba806b4d0 0x7f9ba806d990 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9bbc15dfe0 0x7f9bbc160670 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bbc0d27f0 0x7f9bbc15f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 --2- 172.21.15.72:0/3648319746 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bbc0cf830 0x7f9bbc14dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 >> 172.21.15.72:0/3648319746 conn(0x7f9bbc0d3cd0 msgr2=0x7f9bbc0b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 shutdown_connections 2024-03-06T07:32:31.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.426+0000 7f9bc169a640 1 -- 172.21.15.72:0/3648319746 wait complete. 2024-03-06T07:32:31.439 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-03-06T07:32:30.062944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330160,"kb_used_data":14168,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644560,"statfs":{"total":773094113280,"available":772756029440,"internally_reserved":0,"allocated":14508032,"data_stored":7019930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33900,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-150,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000884"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":373,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-06T07:29:42.019042+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:29:42.019042+0000","last_peered":"2024-03-06T07:29:42.019042+0000","last_clean":"2024-03-06T07:29:42.019042+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:29:42.019042+0000","last_undegraded":"2024-03-06T07:29:42.019042+0000","last_fullsized":"2024-03-06T07:29:42.019042+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574980,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41712,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330128,"statfs":{"total":96636764160,"available":96594051072,"internally_reserved":0,"allocated":2269184,"data_stored":1326420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41140,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330700,"statfs":{"total":96636764160,"available":96594636800,"internally_reserved":0,"allocated":1679360,"data_stored":741747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574981,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41128,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330712,"statfs":{"total":96636764160,"available":96594649088,"internally_reserved":0,"allocated":1667072,"data_stored":723876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1650688,"data_stored":723072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574982,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41144,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1683456,"data_stored":737429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41104,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330736,"statfs":{"total":96636764160,"available":96594673664,"internally_reserved":0,"allocated":1646592,"data_stored":717397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41132,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330708,"statfs":{"total":96636764160,"available":96594644992,"internally_reserved":0,"allocated":1671168,"data_stored":736619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41688,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330152,"statfs":{"total":96636764160,"available":96594075648,"internally_reserved":0,"allocated":2240512,"data_stored":1313370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:31.439 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-06T07:32:31.440 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:31.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.537+0000 7fc7fddf6640 1 Processor -- start 2024-03-06T07:32:31.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.537+0000 7fc7fddf6640 1 -- start start 2024-03-06T07:32:31.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7fddf6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7f80b3060 0x7fc7f815e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7fddf6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f815f580 con 0x7fc7f813dc10 2024-03-06T07:32:31.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7fddf6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f815b450 con 0x7fc7f815bf30 2024-03-06T07:32:31.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7fddf6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f80cf3f0 con 0x7fc7f80b3060 2024-03-06T07:32:31.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc7f813dc10 0x7fc7f80b2950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50170/0 (socket says 172.21.15.72:50170) 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3969650877 learned_addr learned my addr 172.21.15.72:0/3969650877 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611097799 0 0) 0x7fc7f815f580 con 0x7fc7f813dc10 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0003520 con 0x7fc7f813dc10 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f6ffd640 1 --2- 172.21.15.72:0/3969650877 >> v2:172.21.15.146:3300/0 conn(0x7fc7f80b3060 0x7fc7f815e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074604239 0 0) 0x7fc7e0003520 con 0x7fc7f813dc10 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 >> v2:172.21.15.146:3300/0 conn(0x7fc7f80b3060 msgr2=0x7fc7f815e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 --2- 172.21.15.72:0/3969650877 >> v2:172.21.15.146:3300/0 conn(0x7fc7f80b3060 0x7fc7f815e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 >> v1:172.21.15.146:6789/0 conn(0x7fc7f815bf30 legacy=0x7fc7f815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f815b730 con 0x7fc7f813dc10 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7fc7e8001670 con 0x7fc7f813dc10 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 >> v1:172.21.15.72:6789/0 conn(0x7fc7f813dc10 legacy=0x7fc7f80b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7e0004800 0x7fc7e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7e00051c0 0x7fc7e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80cf3f0 con 0x7fc7e00039b0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e000c160 con 0x7fc7e00051c0 2024-03-06T07:32:31.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e000c360 con 0x7fc7e0004800 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.538+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f6ffd640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7e0004800 0x7fc7e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f7fff640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7e00051c0 0x7fc7e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7e0004800 msgr2=0x7fc7e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7e0004800 0x7fc7e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7e00051c0 msgr2=0x7fc7e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7e00051c0 0x7fc7e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f7fff640 1 -- 172.21.15.72:0/3969650877 reap_dead start 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3969650877 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e0003520 con 0x7fc7e00039b0 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7e0003dd0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800c240 tx=0x7fc7e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd008b272993eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8007cf0 con 0x7fc7e00039b0 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8030070 con 0x7fc7e00039b0 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.539+0000 7fc7f67fc640 1 -- 172.21.15.72:0/3969650877 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8005d30 con 0x7fc7e00039b0 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3969650877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 msgr2=0x7fc7e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7e0003dd0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800c240 tx=0x7fc7e0005830 comp rx=0 tx=0).stop 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3969650877 shutdown_connections 2024-03-06T07:32:31.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3969650877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7e0003dd0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3969650877 >> 172.21.15.72:0/3969650877 conn(0x7fc7f80cf8a0 msgr2=0x7fc7f80cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3969650877 shutdown_connections 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.540+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3969650877 wait complete. 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.541+0000 7fc7fddf6640 1 Processor -- start 2024-03-06T07:32:31.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.541+0000 7fc7fddf6640 1 -- start start 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7fddf6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7f813d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7fddf6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7f813e200 0x7fc7f81539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7fddf6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f806d2c0 0x7fc7f806d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7fddf6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80b2950 con 0x7fc7e00039b0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7fddf6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f815b210 con 0x7fc7f813e200 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7fddf6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f80b2ad0 con 0x7fc7f806d2c0 2024-03-06T07:32:31.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7f813d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7f813d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57668/0 (socket says 172.21.15.72:57668) 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3924947404 learned_addr learned my addr 172.21.15.72:0/3924947404 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3924947404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f806d2c0 msgr2=0x7fc7f806d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f806d2c0 0x7fc7f806d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3924947404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7f813e200 msgr2=0x7fc7f81539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7f813e200 0x7fc7f81539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.542+0000 7fc7f77fe640 1 -- 172.21.15.72:0/3924947404 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f81f1c20 con 0x7fc7e00039b0 2024-03-06T07:32:31.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.543+0000 7fc7f77fe640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7f813d260 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800c240 tx=0x7fc7f815f750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.543+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8045070 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.543+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8004bf0 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.543+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8001c30 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.543+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f813e770 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.543+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f813ed30 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.544+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f804f330 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.544+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fc7e8007070 con 0x7fc7e00039b0 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.545+0000 7fc7dffff640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc7e406b4d0 0x7fc7e406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.545+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc7e80aaf50 con 0x7fc7e00039b0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.545+0000 7fc7f6ffd640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc7e406b4d0 0x7fc7e406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.545+0000 7fc7f6ffd640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc7e406b4d0 0x7fc7e406d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8134970 tx=0x7fc7ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.549+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fc7f81f1c20 con 0x7fc7e00039b0 2024-03-06T07:32:31.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.719+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7f80af870 con 0x7fc7e406b4d0 2024-03-06T07:32:31.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.720+0000 7fc7dffff640 1 -- 172.21.15.72:0/3924947404 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13422 (secure 0 0 0) 0x7fc7f80af870 con 0x7fc7e406b4d0 2024-03-06T07:32:31.722 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:31.722 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.721+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc7e406b4d0 msgr2=0x7fc7e406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.721+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc7e406b4d0 0x7fc7e406d990 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8134970 tx=0x7fc7ec008040 comp rx=0 tx=0).stop 2024-03-06T07:32:31.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.721+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 msgr2=0x7fc7f813d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.721+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7f813d260 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800c240 tx=0x7fc7f815f750 comp rx=0 tx=0).stop 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 shutdown_connections 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fc7e406b4d0 0x7fc7e406d990 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f806d2c0 0x7fc7f806d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7f813e200 0x7fc7f81539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 --2- 172.21.15.72:0/3924947404 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc7e00039b0 0x7fc7f813d260 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 >> 172.21.15.72:0/3924947404 conn(0x7fc7f80cf8a0 msgr2=0x7fc7f80cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 shutdown_connections 2024-03-06T07:32:31.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.722+0000 7fc7fddf6640 1 -- 172.21.15.72:0/3924947404 wait complete. 2024-03-06T07:32:31.734 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-03-06T07:32:30.062944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330160,"kb_used_data":14168,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644560,"statfs":{"total":773094113280,"available":772756029440,"internally_reserved":0,"allocated":14508032,"data_stored":7019930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33900,"num_objects":-44,"num_object_clones":0,"num_object_copies":-88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-4,"num_read_kb":0,"num_write":-150,"num_write_kb":-84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000884"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":373,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-06T07:29:42.019042+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:29:42.019042+0000","last_peered":"2024-03-06T07:29:42.019042+0000","last_clean":"2024-03-06T07:29:42.019042+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:29:42.019042+0000","last_undegraded":"2024-03-06T07:29:42.019042+0000","last_fullsized":"2024-03-06T07:29:42.019042+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574980,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41712,"kb_used_data":2216,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330128,"statfs":{"total":96636764160,"available":96594051072,"internally_reserved":0,"allocated":2269184,"data_stored":1326420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41140,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330700,"statfs":{"total":96636764160,"available":96594636800,"internally_reserved":0,"allocated":1679360,"data_stored":741747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574981,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41128,"kb_used_data":1628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330712,"statfs":{"total":96636764160,"available":96594649088,"internally_reserved":0,"allocated":1667072,"data_stored":723876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574981,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":1650688,"data_stored":723072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574982,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41144,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1683456,"data_stored":737429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574981,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41104,"kb_used_data":1608,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330736,"statfs":{"total":96636764160,"available":96594673664,"internally_reserved":0,"allocated":1646592,"data_stored":717397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41132,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330708,"statfs":{"total":96636764160,"available":96594644992,"internally_reserved":0,"allocated":1671168,"data_stored":736619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41688,"kb_used_data":2188,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330152,"statfs":{"total":96636764160,"available":96594075648,"internally_reserved":0,"allocated":2240512,"data_stored":1313370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:31.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-06T07:32:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.833+0000 7fdc50f0c640 1 Processor -- start 2024-03-06T07:32:31.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.833+0000 7fdc50f0c640 1 -- start start 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc50f0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 0x7fdc4c0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc50f0c640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdc4c15bf30 0x7fdc4c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc50f0c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc4c15e850 0x7fdc4c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc50f0c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c0cf3f0 con 0x7fdc4c15e850 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc50f0c640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c15b210 con 0x7fdc4c15bf30 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc50f0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c15f580 con 0x7fdc4c13dc10 2024-03-06T07:32:31.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc4ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc4c15e850 0x7fdc4c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc4ad9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdc4c15e850 0x7fdc4c15eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57684/0 (socket says 172.21.15.72:57684) 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc4ad9d640 1 -- 172.21.15.72:0/2801093448 learned_addr learned my addr 172.21.15.72:0/2801093448 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc4ad9d640 1 -- 172.21.15.72:0/2801093448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 msgr2=0x7fdc4c0b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.834+0000 7fdc4ad9d640 1 --2- 172.21.15.72:0/2801093448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 0x7fdc4c0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc4ad9d640 1 -- 172.21.15.72:0/2801093448 >> v2:172.21.15.146:3300/0 conn(0x7fdc4c15bf30 msgr2=0x7fdc4c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc4ad9d640 1 --2- 172.21.15.72:0/2801093448 >> v2:172.21.15.146:3300/0 conn(0x7fdc4c15bf30 0x7fdc4c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc4ad9d640 1 -- 172.21.15.72:0/2801093448 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c15b450 con 0x7fdc4c15e850 2024-03-06T07:32:31.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc4ad9d640 1 --2- 172.21.15.72:0/2801093448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15e850 0x7fdc4c15eca0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c068890 tx=0x7fdc4402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f01706eb2f3ae8b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc49d9b640 1 -- 172.21.15.72:0/2801093448 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4402c9e0 con 0x7fdc4c15e850 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc49d9b640 1 -- 172.21.15.72:0/2801093448 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4402cb80 con 0x7fdc4c15e850 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.835+0000 7fdc49d9b640 1 -- 172.21.15.72:0/2801093448 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc440343f0 con 0x7fdc4c15e850 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 -- 172.21.15.72:0/2801093448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15e850 msgr2=0x7fdc4c15eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/2801093448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15e850 0x7fdc4c15eca0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c068890 tx=0x7fdc4402be40 comp rx=0 tx=0).stop 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 -- 172.21.15.72:0/2801093448 shutdown_connections 2024-03-06T07:32:31.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/2801093448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15e850 0x7fdc4c15eca0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/2801093448 >> v2:172.21.15.146:3300/0 conn(0x7fdc4c15bf30 0x7fdc4c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/2801093448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 0x7fdc4c0b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 -- 172.21.15.72:0/2801093448 >> 172.21.15.72:0/2801093448 conn(0x7fdc4c0cf8a0 msgr2=0x7fdc4c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 -- 172.21.15.72:0/2801093448 shutdown_connections 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.836+0000 7fdc50f0c640 1 -- 172.21.15.72:0/2801093448 wait complete. 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.837+0000 7fdc50f0c640 1 Processor -- start 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.837+0000 7fdc50f0c640 1 -- start start 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc50f0c640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 0x7fdc4c141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc50f0c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 0x7fdc4c1f40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc50f0c640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc4c15e850 0x7fdc4c1f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc50f0c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c15f1e0 con 0x7fdc4c15bf30 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc50f0c640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c13dff0 con 0x7fdc4c15e850 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc50f0c640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c15f360 con 0x7fdc4c13dc10 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 0x7fdc4c1f40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 0x7fdc4c1f40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:57688/0 (socket says 172.21.15.72:57688) 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 -- 172.21.15.72:0/633329481 learned_addr learned my addr 172.21.15.72:0/633329481 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:31.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 -- 172.21.15.72:0/633329481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 msgr2=0x7fdc4c141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 0x7fdc4c141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 -- 172.21.15.72:0/633329481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc4c15e850 msgr2=0x7fdc4c1f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc4c15e850 0x7fdc4c1f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.838+0000 7fdc43fff640 1 -- 172.21.15.72:0/633329481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c1535d0 con 0x7fdc4c15bf30 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.839+0000 7fdc43fff640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 0x7fdc4c1f40f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400b4a0 tx=0x7fdc3400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.839+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34014070 con 0x7fdc4c15bf30 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.839+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc34010610 con 0x7fdc4c15bf30 2024-03-06T07:32:31.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.839+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c142830 con 0x7fdc4c15bf30 2024-03-06T07:32:31.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.839+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34010a00 con 0x7fdc4c15bf30 2024-03-06T07:32:31.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.839+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c153870 con 0x7fdc4c15bf30 2024-03-06T07:32:31.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.840+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c04f330 con 0x7fdc4c15bf30 2024-03-06T07:32:31.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.840+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fdc34017070 con 0x7fdc4c15bf30 2024-03-06T07:32:31.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.841+0000 7fdc42ffd640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:31.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.841+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fdc340898a0 con 0x7fdc4c15bf30 2024-03-06T07:32:31.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.841+0000 7fdc4a59c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:31.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.841+0000 7fdc4a59c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009670 tx=0x7fdc3c006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:31.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:31.845+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fdc34058a50 con 0x7fdc4c15bf30 2024-03-06T07:32:32.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.017+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc4c0cca40 con 0x7fdc4c15bf30 2024-03-06T07:32:32.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.018+0000 7fdc42ffd640 1 -- 172.21.15.72:0/633329481 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+13172 (secure 0 0 0) 0x7fdc34058350 con 0x7fdc4c15bf30 2024-03-06T07:32:32.020 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:32.021 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":134,"fsid":"0927e84a-942e-4592-98d9-697c1cf6243f","created":"2024-03-06T07:21:25.702296+0000","modified":"2024-03-06T07:32:28.874674+0000","last_up_change":"2024-03-06T07:21:36.924037+0000","last_in_change":"2024-03-06T07:21:27.895074+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-06T07:21:37.990747+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e430101d-cbf3-4e3e-a12e-d68c57daef06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6818","nonce":4178159476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6823","nonce":4178159476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6829","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6831","nonce":4178159476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6825","nonce":4178159476},{"type":"v1","addr":"172.21.15.72:6827","nonce":4178159476}]},"public_addr":"172.21.15.72:6818/4178159476","cluster_addr":"172.21.15.72:6823/4178159476","heartbeat_back_addr":"172.21.15.72:6831/4178159476","heartbeat_front_addr":"172.21.15.72:6827/4178159476","state":["exists","up"]},{"osd":1,"uuid":"7cd0d2fb-bff1-41c8-b382-01b0b22d939b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6801","nonce":2878132174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6803","nonce":2878132174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6807","nonce":2878132174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2878132174},{"type":"v1","addr":"172.21.15.72:6805","nonce":2878132174}]},"public_addr":"172.21.15.72:6801/2878132174","cluster_addr":"172.21.15.72:6803/2878132174","heartbeat_back_addr":"172.21.15.72:6807/2878132174","heartbeat_front_addr":"172.21.15.72:6805/2878132174","state":["exists","up"]},{"osd":2,"uuid":"a076999a-a24b-4415-beb9-6ca9c85bbf86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6817","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6819","nonce":3054266462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6821","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6822","nonce":3054266462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6830","nonce":3054266462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3054266462},{"type":"v1","addr":"172.21.15.72:6826","nonce":3054266462}]},"public_addr":"172.21.15.72:6819/3054266462","cluster_addr":"172.21.15.72:6822/3054266462","heartbeat_back_addr":"172.21.15.72:6830/3054266462","heartbeat_front_addr":"172.21.15.72:6826/3054266462","state":["exists","up"]},{"osd":3,"uuid":"79189137-86fa-4d90-8872-0a05f646a09d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6809","nonce":4245662075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6811","nonce":4245662075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6815","nonce":4245662075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":4245662075},{"type":"v1","addr":"172.21.15.72:6813","nonce":4245662075}]},"public_addr":"172.21.15.72:6809/4245662075","cluster_addr":"172.21.15.72:6811/4245662075","heartbeat_back_addr":"172.21.15.72:6815/4245662075","heartbeat_front_addr":"172.21.15.72:6813/4245662075","state":["exists","up"]},{"osd":4,"uuid":"86e81164-5d05-4c30-a056-850177548536","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6822","nonce":3084745005}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6825","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6827","nonce":3084745005}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6831","nonce":3084745005}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":3084745005},{"type":"v1","addr":"172.21.15.146:6829","nonce":3084745005}]},"public_addr":"172.21.15.146:6822/3084745005","cluster_addr":"172.21.15.146:6827/3084745005","heartbeat_back_addr":"172.21.15.146:6831/3084745005","heartbeat_front_addr":"172.21.15.146:6829/3084745005","state":["exists","up"]},{"osd":5,"uuid":"351b9f65-50b0-442c-a118-e53c5589de7c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6801","nonce":2545401618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6803","nonce":2545401618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6807","nonce":2545401618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2545401618},{"type":"v1","addr":"172.21.15.146:6805","nonce":2545401618}]},"public_addr":"172.21.15.146:6801/2545401618","cluster_addr":"172.21.15.146:6803/2545401618","heartbeat_back_addr":"172.21.15.146:6807/2545401618","heartbeat_front_addr":"172.21.15.146:6805/2545401618","state":["exists","up"]},{"osd":6,"uuid":"524bb873-aab9-45af-b6b4-41fb9a95709a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":125,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6809","nonce":1113968375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6811","nonce":1113968375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6815","nonce":1113968375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":1113968375},{"type":"v1","addr":"172.21.15.146:6813","nonce":1113968375}]},"public_addr":"172.21.15.146:6809/1113968375","cluster_addr":"172.21.15.146:6811/1113968375","heartbeat_back_addr":"172.21.15.146:6815/1113968375","heartbeat_front_addr":"172.21.15.146:6813/1113968375","state":["exists","up"]},{"osd":7,"uuid":"dbeeecd3-529d-4582-9aa7-8e26cbf8683a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6817","nonce":1306818419}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6819","nonce":1306818419}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6826","nonce":1306818419}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6821","nonce":1306818419},{"type":"v1","addr":"172.21.15.146:6823","nonce":1306818419}]},"public_addr":"172.21.15.146:6817/1306818419","cluster_addr":"172.21.15.146:6819/1306818419","heartbeat_back_addr":"172.21.15.146:6826/1306818419","heartbeat_front_addr":"172.21.15.146:6823/1306818419","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.057879+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.021917+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.009973+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.012432+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.061497+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.997311+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:32.994288+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-06T07:21:33.034892+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6837/1633306088":"2024-03-07T07:32:25.866467+0000","172.21.15.72:6835/1633306088":"2024-03-07T07:32:25.866467+0000","172.21.15.72:6836/2047054467":"2024-03-07T07:32:25.866456+0000","172.21.15.72:6834/2047054467":"2024-03-07T07:32:25.866456+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-06T07:32:32.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdc2c06b4d0 msgr2=0x7fdc2c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009670 tx=0x7fdc3c006bf0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 msgr2=0x7fdc4c1f40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 0x7fdc4c1f40f0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400b4a0 tx=0x7fdc3400b960 comp rx=0 tx=0).stop 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 shutdown_connections 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fdc2c06b4d0 0x7fdc2c06d990 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdc4c15e850 0x7fdc4c1f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc4c15bf30 0x7fdc4c1f40f0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 --2- 172.21.15.72:0/633329481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdc4c13dc10 0x7fdc4c141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 >> 172.21.15.72:0/633329481 conn(0x7fdc4c0cf8a0 msgr2=0x7fdc4c0b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 shutdown_connections 2024-03-06T07:32:32.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:32.020+0000 7fdc50f0c640 1 -- 172.21.15.72:0/633329481 wait complete. 2024-03-06T07:32:33.034 INFO:tasks.ceph:Scrubbing osd.0 2024-03-06T07:32:33.035 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:33.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.147+0000 7f1eb166f640 1 Processor -- start 2024-03-06T07:32:33.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.147+0000 7f1eb166f640 1 -- start start 2024-03-06T07:32:33.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eb166f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac15bdd0 0x7f1eac1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eb166f640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1eac0d0130 0x7f1eac15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eb166f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eac06f420 con 0x7f1eac0701e0 2024-03-06T07:32:33.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eb166f640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eac06f700 con 0x7f1eac0d0130 2024-03-06T07:32:33.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eb166f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac06f9e0 con 0x7f1eac15bdd0 2024-03-06T07:32:33.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eaaffd640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac15bdd0 0x7f1eac1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eaa7fc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1eac0d0130 0x7f1eac15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eab7fe640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1eac0701e0 0x7f1eac070610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:33338/0 (socket says 172.21.15.72:33338) 2024-03-06T07:32:33.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.148+0000 7f1eab7fe640 1 -- 172.21.15.72:0/105396058 learned_addr learned my addr 172.21.15.72:0/105396058 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:33.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.149+0000 7f1eaa7fc640 1 -- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac15bdd0 msgr2=0x7f1eac1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.149+0000 7f1eaa7fc640 1 --2- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac15bdd0 0x7f1eac1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.149+0000 7f1eaa7fc640 1 -- 172.21.15.72:0/105396058 >> v1:172.21.15.146:6789/0 conn(0x7f1eac0701e0 legacy=0x7f1eac070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.149+0000 7f1ea9ffb640 1 -- 172.21.15.72:0/105396058 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348276377 0 0) 0x7f1eac06f420 con 0x7f1eac0701e0 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.149+0000 7f1eaa7fc640 1 -- 172.21.15.72:0/105396058 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac06fcc0 con 0x7f1eac0d0130 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.149+0000 7f1eaaffd640 1 --2- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac15bdd0 0x7f1eac1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eaa7fc640 1 --2- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac0d0130 0x7f1eac15d330 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00097a0 tx=0x7f1ea002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=236ea00bd9e0cd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1ea9ffb640 1 -- 172.21.15.72:0/105396058 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0031780 con 0x7f1eac0d0130 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1ea9ffb640 1 -- 172.21.15.72:0/105396058 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea0037070 con 0x7f1eac0d0130 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1ea9ffb640 1 -- 172.21.15.72:0/105396058 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea0031b40 con 0x7f1eac0d0130 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 -- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac0d0130 msgr2=0x7f1eac15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 --2- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac0d0130 0x7f1eac15d330 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00097a0 tx=0x7f1ea002b820 comp rx=0 tx=0).stop 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 -- 172.21.15.72:0/105396058 shutdown_connections 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 --2- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac0d0130 0x7f1eac15d330 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 --2- 172.21.15.72:0/105396058 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac15bdd0 0x7f1eac1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 -- 172.21.15.72:0/105396058 >> 172.21.15.72:0/105396058 conn(0x7f1eac09a9f0 msgr2=0x7f1eac09ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 -- 172.21.15.72:0/105396058 shutdown_connections 2024-03-06T07:32:33.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.150+0000 7f1eb166f640 1 -- 172.21.15.72:0/105396058 wait complete. 2024-03-06T07:32:33.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 Processor -- start 2024-03-06T07:32:33.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 -- start start 2024-03-06T07:32:33.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eac0701e0 0x7f1eac15c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac0d0130 0x7f1eac0b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac15bdd0 0x7f1eac0b5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac15d9f0 con 0x7f1eac0701e0 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac15d870 con 0x7f1eac15bdd0 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.152+0000 7f1eb166f640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eac0d08d0 con 0x7f1eac0d0130 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eac0701e0 0x7f1eac15c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eac0701e0 0x7f1eac15c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54870/0 (socket says 172.21.15.72:54870) 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 -- 172.21.15.72:0/3880767496 learned_addr learned my addr 172.21.15.72:0/3880767496 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaa7fc640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac0d0130 0x7f1eac0b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eab7fe640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac15bdd0 0x7f1eac0b5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 -- 172.21.15.72:0/3880767496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac0d0130 msgr2=0x7f1eac0b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1eac0d0130 0x7f1eac0b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 -- 172.21.15.72:0/3880767496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac15bdd0 msgr2=0x7f1eac0b5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1eac15bdd0 0x7f1eac0b5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 -- 172.21.15.72:0/3880767496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eac1f2ab0 con 0x7f1eac0701e0 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eaaffd640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eac0701e0 0x7f1eac15c930 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00e990 tx=0x7f1e9c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c017070 con 0x7f1eac0701e0 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c00c9e0 con 0x7f1eac0701e0 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac1f2210 con 0x7f1eac0701e0 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c00cdb0 con 0x7f1eac0701e0 2024-03-06T07:32:33.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.153+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eac1f27d0 con 0x7f1eac0701e0 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.154+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1eac04f330 con 0x7f1eac0701e0 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.155+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1e9c01a070 con 0x7f1eac0701e0 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.155+0000 7f1e937fe640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e9806b4d0 0x7f1e9806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.155+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1e9c08a0f0 con 0x7f1eac0701e0 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.155+0000 7f1eaa7fc640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e9806b4d0 0x7f1e9806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.156+0000 7f1e937fe640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1e98071430 0x7f1e98073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.156+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1e9c026ea0 con 0x7f1e98071430 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.156+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e9c08a580 con 0x7f1eac0701e0 2024-03-06T07:32:33.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.156+0000 7f1eab7fe640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1e98071430 0x7f1e98073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.156+0000 7f1eaa7fc640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e9806b4d0 0x7f1e9806d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0036000 tx=0x7f1ea0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.156+0000 7f1eab7fe640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1e98071430 0x7f1e98073850 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.157+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== osd.0 v2:172.21.15.72:6816/4178159476 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1e9c026ea0 con 0x7f1e98071430 2024-03-06T07:32:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 --> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1eac070860 con 0x7f1e98071430 2024-03-06T07:32:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1e937fe640 1 -- 172.21.15.72:0/3880767496 <== osd.0 v2:172.21.15.72:6816/4178159476 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1eac070860 con 0x7f1e98071430 2024-03-06T07:32:33.173 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:33.173 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:33.173 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:33.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1e98071430 msgr2=0x7f1e98073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6816/4178159476,v1:172.21.15.72:6818/4178159476] conn(0x7f1e98071430 0x7f1e98073850 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e9806b4d0 msgr2=0x7f1e9806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1e9806b4d0 0x7f1e9806d990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0036000 tx=0x7f1ea0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:33.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eac0701e0 msgr2=0x7f1eac15c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eb166f640 1 --2- 172.21.15.72:0/3880767496 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eac0701e0 0x7f1eac15c930 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00e990 tx=0x7f1e9c00ee50 comp rx=0 tx=0).stop 2024-03-06T07:32:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.170+0000 7f1eab7fe640 1 -- 172.21.15.72:0/3880767496 reap_dead start 2024-03-06T07:32:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.171+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 shutdown_connections 2024-03-06T07:32:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.171+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 >> 172.21.15.72:0/3880767496 conn(0x7f1eac09a9f0 msgr2=0x7f1eac1e4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.171+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 shutdown_connections 2024-03-06T07:32:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.171+0000 7f1eb166f640 1 -- 172.21.15.72:0/3880767496 wait complete. 2024-03-06T07:32:33.183 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-06T07:32:33.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.281+0000 7f0efd664640 1 Processor -- start 2024-03-06T07:32:33.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.281+0000 7f0efd664640 1 -- start start 2024-03-06T07:32:33.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0efd664640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0ef8070fe0 0x7f0ef81658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0efd664640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef806d370 con 0x7f0ef81444c0 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0efd664640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef806d650 con 0x7f0ef815b2d0 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0efd664640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef81448a0 con 0x7f0ef8070fe0 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0ef77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0ef8070fe0 0x7f0ef81658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0ef6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0ef81444c0 0x7f0ef806ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53116/0 (socket says 172.21.15.72:53116) 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0ef6ffd640 1 -- 172.21.15.72:0/1444430013 learned_addr learned my addr 172.21.15.72:0/1444430013 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0ef77fe640 1 -- 172.21.15.72:0/1444430013 >> v1:172.21.15.146:6789/0 conn(0x7f0ef815b2d0 legacy=0x7f0ef80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0ef77fe640 1 -- 172.21.15.72:0/1444430013 >> v1:172.21.15.72:6789/0 conn(0x7f0ef81444c0 legacy=0x7f0ef806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.282+0000 7f0ef77fe640 1 -- 172.21.15.72:0/1444430013 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef806d930 con 0x7f0ef8070fe0 2024-03-06T07:32:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.283+0000 7f0ef77fe640 1 --2- 172.21.15.72:0/1444430013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef8070fe0 0x7f0ef81658d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00097a0 tx=0x7f0ef002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87265ddbb358b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.283+0000 7f0ef5ffb640 1 -- 172.21.15.72:0/1444430013 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00315a0 con 0x7f0ef8070fe0 2024-03-06T07:32:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.283+0000 7f0ef5ffb640 1 -- 172.21.15.72:0/1444430013 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef0037070 con 0x7f0ef8070fe0 2024-03-06T07:32:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.283+0000 7f0ef5ffb640 1 -- 172.21.15.72:0/1444430013 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0031960 con 0x7f0ef8070fe0 2024-03-06T07:32:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 -- 172.21.15.72:0/1444430013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef8070fe0 msgr2=0x7f0ef81658d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 --2- 172.21.15.72:0/1444430013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef8070fe0 0x7f0ef81658d0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00097a0 tx=0x7f0ef002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 -- 172.21.15.72:0/1444430013 shutdown_connections 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 --2- 172.21.15.72:0/1444430013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef8070fe0 0x7f0ef81658d0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 -- 172.21.15.72:0/1444430013 >> 172.21.15.72:0/1444430013 conn(0x7f0ef80b1c80 msgr2=0x7f0ef80cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 -- 172.21.15.72:0/1444430013 shutdown_connections 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.284+0000 7f0efd664640 1 -- 172.21.15.72:0/1444430013 wait complete. 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.285+0000 7f0efd664640 1 Processor -- start 2024-03-06T07:32:33.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.285+0000 7f0efd664640 1 -- start start 2024-03-06T07:32:33.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0efd664640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ef8070fe0 0x7f0ef815a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0efd664640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 0x7f0ef813f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0efd664640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ef815b2d0 0x7f0ef813fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0efd664640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef806def0 con 0x7f0ef81444c0 2024-03-06T07:32:33.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0efd664640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef806d110 con 0x7f0ef815b2d0 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0efd664640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef80cea70 con 0x7f0ef8070fe0 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 0x7f0ef813f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 0x7f0ef813f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54888/0 (socket says 172.21.15.72:54888) 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 -- 172.21.15.72:0/2043643325 learned_addr learned my addr 172.21.15.72:0/2043643325 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef77fe640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ef815b2d0 0x7f0ef813fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef6ffd640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ef8070fe0 0x7f0ef815a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 -- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ef8070fe0 msgr2=0x7f0ef815a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ef8070fe0 0x7f0ef815a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 -- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ef815b2d0 msgr2=0x7f0ef813fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ef815b2d0 0x7f0ef813fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 -- 172.21.15.72:0/2043643325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef81e5230 con 0x7f0ef81444c0 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0ef67fc640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 0x7f0ef813f830 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00b420 tx=0x7f0eec00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.286+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec011830 con 0x7f0ef81444c0 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.287+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eec016070 con 0x7f0ef81444c0 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.287+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef81e5510 con 0x7f0ef81444c0 2024-03-06T07:32:33.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.287+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eec011c10 con 0x7f0ef81444c0 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.287+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef81e5ad0 con 0x7f0ef81444c0 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.288+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef804f330 con 0x7f0ef81444c0 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.288+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f0eec01b460 con 0x7f0ef81444c0 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.288+0000 7f0edf7fe640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee406b4d0 0x7f0ee406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.289+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0eec089900 con 0x7f0ef81444c0 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.289+0000 7f0ef6ffd640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee406b4d0 0x7f0ee406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.289+0000 7f0ef6ffd640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee406b4d0 0x7f0ee406d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80012e0 tx=0x7f0ee8001a30 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.293+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f0eec020020 con 0x7f0ef81444c0 2024-03-06T07:32:33.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.467+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ef806e780 con 0x7f0ee406b4d0 2024-03-06T07:32:33.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.468+0000 7f0edf7fe640 1 -- 172.21.15.72:0/2043643325 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ef806e780 con 0x7f0ee406b4d0 2024-03-06T07:32:33.470 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 0 to deep-scrub 2024-03-06T07:32:33.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee406b4d0 msgr2=0x7f0ee406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee406b4d0 0x7f0ee406d990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80012e0 tx=0x7f0ee8001a30 comp rx=0 tx=0).stop 2024-03-06T07:32:33.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 msgr2=0x7f0ef813f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 0x7f0ef813f830 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f0eec00b420 tx=0x7f0eec00b8e0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 shutdown_connections 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f0ee406b4d0 0x7f0ee406d990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ef815b2d0 0x7f0ef813fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ef81444c0 0x7f0ef813f830 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 --2- 172.21.15.72:0/2043643325 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0ef8070fe0 0x7f0ef815a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 >> 172.21.15.72:0/2043643325 conn(0x7f0ef80b1c80 msgr2=0x7f0ef806e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 shutdown_connections 2024-03-06T07:32:33.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.470+0000 7f0efd664640 1 -- 172.21.15.72:0/2043643325 wait complete. 2024-03-06T07:32:33.482 INFO:tasks.ceph:Scrubbing osd.1 2024-03-06T07:32:33.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:33.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.584+0000 7f362fb65640 1 Processor -- start 2024-03-06T07:32:33.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.584+0000 7f362fb65640 1 -- start start 2024-03-06T07:32:33.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362fb65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f362815ac00 0x7f362806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362fb65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f362815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362fb65640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362815afe0 con 0x7f36280708b0 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362fb65640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36280cdc10 con 0x7f362815ac00 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362fb65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36280cd000 con 0x7f3628070d10 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d901640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f362815ac00 0x7f362806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d100640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f36280708b0 0x7f362815e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53132/0 (socket says 172.21.15.72:53132) 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d100640 1 -- 172.21.15.72:0/2896877299 learned_addr learned my addr 172.21.15.72:0/2896877299 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362e102640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f362815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d901640 1 -- 172.21.15.72:0/2896877299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 msgr2=0x7f362815eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d901640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f362815eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d901640 1 -- 172.21.15.72:0/2896877299 >> v1:172.21.15.72:6789/0 conn(0x7f36280708b0 legacy=0x7f362815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362d901640 1 -- 172.21.15.72:0/2896877299 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36280cd2e0 con 0x7f362815ac00 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.585+0000 7f362e102640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f362815eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.586+0000 7f362d901640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362815ac00 0x7f362806c910 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f3624009d90 tx=0x7f362402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db74bebc2d5c44b server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.586+0000 7f362c8ff640 1 -- 172.21.15.72:0/2896877299 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624039070 con 0x7f362815ac00 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.586+0000 7f362c8ff640 1 -- 172.21.15.72:0/2896877299 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362402cbc0 con 0x7f362815ac00 2024-03-06T07:32:33.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.586+0000 7f362c8ff640 1 -- 172.21.15.72:0/2896877299 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624034440 con 0x7f362815ac00 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 -- 172.21.15.72:0/2896877299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362815ac00 msgr2=0x7f362806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362815ac00 0x7f362806c910 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f3624009d90 tx=0x7f362402be10 comp rx=0 tx=0).stop 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 -- 172.21.15.72:0/2896877299 shutdown_connections 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f362815eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 --2- 172.21.15.72:0/2896877299 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f362815ac00 0x7f362806c910 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 -- 172.21.15.72:0/2896877299 >> 172.21.15.72:0/2896877299 conn(0x7f36280b1650 msgr2=0x7f36280b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 -- 172.21.15.72:0/2896877299 shutdown_connections 2024-03-06T07:32:33.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.587+0000 7f362fb65640 1 -- 172.21.15.72:0/2896877299 wait complete. 2024-03-06T07:32:33.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.588+0000 7f362fb65640 1 Processor -- start 2024-03-06T07:32:33.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.588+0000 7f362fb65640 1 -- start start 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362fb65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36280708b0 0x7f3628158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362fb65640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f36280ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362fb65640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f362815ac00 0x7f36280acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362fb65640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3628071190 con 0x7f36280708b0 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362fb65640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36280cd8a0 con 0x7f362815ac00 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362fb65640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362806ce50 con 0x7f3628070d10 2024-03-06T07:32:33.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36280708b0 0x7f3628158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36280708b0 0x7f3628158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54908/0 (socket says 172.21.15.72:54908) 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 -- 172.21.15.72:0/674995740 learned_addr learned my addr 172.21.15.72:0/674995740 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d100640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f36280ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362e102640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f362815ac00 0x7f36280acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 -- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 msgr2=0x7f36280ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3628070d10 0x7f36280ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 -- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f362815ac00 msgr2=0x7f36280acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f362815ac00 0x7f36280acd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 -- 172.21.15.72:0/674995740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36280ab250 con 0x7f36280708b0 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362e102640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f362815ac00 0x7f36280acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f362d901640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36280708b0 0x7f3628158a00 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f3624037040 tx=0x7f3624004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.589+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3624039050 con 0x7f36280708b0 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.590+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3624045070 con 0x7f36280708b0 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.590+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362403e850 con 0x7f36280708b0 2024-03-06T07:32:33.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.590+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3628158f40 con 0x7f36280708b0 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.590+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36280aa490 con 0x7f36280708b0 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.590+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f362804f330 con 0x7f36280708b0 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.591+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f362403e9f0 con 0x7f36280708b0 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.591+0000 7f361e7fc640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f360c06b4d0 0x7f360c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.591+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f36240ad5f0 con 0x7f36280708b0 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.591+0000 7f362d100640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f360c06b4d0 0x7f360c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.591+0000 7f361e7fc640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f360c071430 0x7f360c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.592+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36240091a0 con 0x7f360c071430 2024-03-06T07:32:33.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.592+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36240ada80 con 0x7f36280708b0 2024-03-06T07:32:33.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.592+0000 7f362e102640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f360c071430 0x7f360c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.592+0000 7f362d100640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f360c06b4d0 0x7f360c06d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f36280b11e0 tx=0x7f3618009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.592+0000 7f362e102640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f360c071430 0x7f360c073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.593+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== osd.1 v2:172.21.15.72:6800/2878132174 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36240091a0 con 0x7f360c071430 2024-03-06T07:32:33.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.605+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 --> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3628159170 con 0x7f360c071430 2024-03-06T07:32:33.608 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f361e7fc640 1 -- 172.21.15.72:0/674995740 <== osd.1 v2:172.21.15.72:6800/2878132174 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3628159170 con 0x7f360c071430 2024-03-06T07:32:33.608 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:33.608 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:33.609 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:33.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f360c071430 msgr2=0x7f360c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6800/2878132174,v1:172.21.15.72:6801/2878132174] conn(0x7f360c071430 0x7f360c073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f360c06b4d0 msgr2=0x7f360c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f360c06b4d0 0x7f360c06d990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f36280b11e0 tx=0x7f3618009040 comp rx=0 tx=0).stop 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36280708b0 msgr2=0x7f3628158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 --2- 172.21.15.72:0/674995740 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36280708b0 0x7f3628158a00 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f3624037040 tx=0x7f3624004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362e102640 1 -- 172.21.15.72:0/674995740 reap_dead start 2024-03-06T07:32:33.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 shutdown_connections 2024-03-06T07:32:33.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 >> 172.21.15.72:0/674995740 conn(0x7f36280b1650 msgr2=0x7f36280af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 shutdown_connections 2024-03-06T07:32:33.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.606+0000 7f362fb65640 1 -- 172.21.15.72:0/674995740 wait complete. 2024-03-06T07:32:33.618 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-06T07:32:33.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.716+0000 7f142a720640 1 Processor -- start 2024-03-06T07:32:33.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f142a720640 1 -- start start 2024-03-06T07:32:33.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f142a720640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14241444c0 0x7f142406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f142a720640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f14240cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f142a720640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f142406d370 con 0x7f142415b2d0 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f142a720640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14241448a0 con 0x7f14241444c0 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f142a720640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142406d650 con 0x7f1424070fe0 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.717+0000 7f1423fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14241444c0 0x7f142406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14241444c0 0x7f142406ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54912/0 (socket says 172.21.15.72:54912) 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 -- 172.21.15.72:0/1171418667 learned_addr learned my addr 172.21.15.72:0/1171418667 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:33.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1428cbd640 1 --2- 172.21.15.72:0/1171418667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f14240cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 -- 172.21.15.72:0/1171418667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 msgr2=0x7f14240cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:33.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 --2- 172.21.15.72:0/1171418667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f14240cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 -- 172.21.15.72:0/1171418667 >> v1:172.21.15.146:6789/0 conn(0x7f142415b2d0 legacy=0x7f14240708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 -- 172.21.15.72:0/1171418667 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142406d930 con 0x7f14241444c0 2024-03-06T07:32:33.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.718+0000 7f1423fff640 1 --2- 172.21.15.72:0/1171418667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f142406ca00 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f140c009870 tx=0x7f140c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4ae74e8c06129a server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f1422ffd640 1 -- 172.21.15.72:0/1171418667 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c034070 con 0x7f14241444c0 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f1422ffd640 1 -- 172.21.15.72:0/1171418667 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c030840 con 0x7f14241444c0 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f1422ffd640 1 -- 172.21.15.72:0/1171418667 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c030c10 con 0x7f14241444c0 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 -- 172.21.15.72:0/1171418667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 msgr2=0x7f142406ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 --2- 172.21.15.72:0/1171418667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f142406ca00 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f140c009870 tx=0x7f140c02b980 comp rx=0 tx=0).stop 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 -- 172.21.15.72:0/1171418667 shutdown_connections 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 --2- 172.21.15.72:0/1171418667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f14240cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 --2- 172.21.15.72:0/1171418667 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f142406ca00 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 -- 172.21.15.72:0/1171418667 >> 172.21.15.72:0/1171418667 conn(0x7f14240b1c80 msgr2=0x7f14240cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 -- 172.21.15.72:0/1171418667 shutdown_connections 2024-03-06T07:32:33.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.719+0000 7f142a720640 1 -- 172.21.15.72:0/1171418667 wait complete. 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.720+0000 7f142a720640 1 Processor -- start 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 -- start start 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f142413b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f14240cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f142415b2d0 0x7f14240c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142406cf40 con 0x7f14241444c0 2024-03-06T07:32:33.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142406def0 con 0x7f142415b2d0 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f142a720640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142406d0c0 con 0x7f1424070fe0 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f14237fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f14240cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f14237fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f14240cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54918/0 (socket says 172.21.15.72:54918) 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f14237fe640 1 -- 172.21.15.72:0/2178781256 learned_addr learned my addr 172.21.15.72:0/2178781256 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f1428cbd640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f142415b2d0 0x7f14240c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.721+0000 7f1423fff640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f142413b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f14237fe640 1 -- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 msgr2=0x7f142413b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f14237fe640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f142413b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f14237fe640 1 -- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f142415b2d0 msgr2=0x7f14240c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f14237fe640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f142415b2d0 0x7f14240c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f14237fe640 1 -- 172.21.15.72:0/2178781256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142413ba70 con 0x7f14241444c0 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f14237fe640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f14240cbb80 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f141400c3c0 tx=0x7f141400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140052d0 con 0x7f14241444c0 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1414005490 con 0x7f14241444c0 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14240cc0c0 con 0x7f14241444c0 2024-03-06T07:32:33.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414007260 con 0x7f14241444c0 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.722+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14240cadc0 con 0x7f14241444c0 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.723+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142404f330 con 0x7f14241444c0 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.723+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1414007400 con 0x7f14241444c0 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.724+0000 7f1420ff9640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f140406b500 0x7f140406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.724+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1414089300 con 0x7f14241444c0 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.724+0000 7f1423fff640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f140406b500 0x7f140406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:33.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.724+0000 7f1423fff640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f140406b500 0x7f140406d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f140c036000 tx=0x7f140c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:33.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.728+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f142413ba70 con 0x7f14241444c0 2024-03-06T07:32:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.903+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f142406e460 con 0x7f140406b500 2024-03-06T07:32:33.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.903+0000 7f1420ff9640 1 -- 172.21.15.72:0/2178781256 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f142406e460 con 0x7f140406b500 2024-03-06T07:32:33.906 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 1 to deep-scrub 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f140406b500 msgr2=0x7f140406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f140406b500 0x7f140406d9c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f140c036000 tx=0x7f140c008030 comp rx=0 tx=0).stop 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 msgr2=0x7f14240cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f14240cbb80 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f141400c3c0 tx=0x7f141400c880 comp rx=0 tx=0).stop 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 shutdown_connections 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f140406b500 0x7f140406d9c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f142415b2d0 0x7f14240c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14241444c0 0x7f14240cbb80 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 --2- 172.21.15.72:0/2178781256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1424070fe0 0x7f142413b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 >> 172.21.15.72:0/2178781256 conn(0x7f14240b1c80 msgr2=0x7f142415ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 shutdown_connections 2024-03-06T07:32:33.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:33.905+0000 7f142a720640 1 -- 172.21.15.72:0/2178781256 wait complete. 2024-03-06T07:32:33.918 INFO:tasks.ceph:Scrubbing osd.2 2024-03-06T07:32:33.918 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:34.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.020+0000 7f215e0df640 1 Processor -- start 2024-03-06T07:32:34.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.020+0000 7f215e0df640 1 -- start start 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f215e0df640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f215815ac00 0x7f215806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f215e0df640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215815f5c0 con 0x7f21580708b0 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f215e0df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21580711c0 con 0x7f2158070d10 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f215e0df640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21580cdc10 con 0x7f215815ac00 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f21577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f215815ac00 0x7f215806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f2156ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f21580708b0 0x7f215815e7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53138/0 (socket says 172.21.15.72:53138) 2024-03-06T07:32:34.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.021+0000 7f2156ffd640 1 -- 172.21.15.72:0/3777657790 learned_addr learned my addr 172.21.15.72:0/3777657790 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:34.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21577fe640 1 -- 172.21.15.72:0/3777657790 >> v1:172.21.15.146:6789/0 conn(0x7f2158070d10 legacy=0x7f215815eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21577fe640 1 -- 172.21.15.72:0/3777657790 >> v1:172.21.15.72:6789/0 conn(0x7f21580708b0 legacy=0x7f215815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21567fc640 1 -- 172.21.15.72:0/3777657790 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485527789 0 0) 0x7f215815f5c0 con 0x7f21580708b0 2024-03-06T07:32:34.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21577fe640 1 -- 172.21.15.72:0/3777657790 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215815afe0 con 0x7f215815ac00 2024-03-06T07:32:34.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21577fe640 1 --2- 172.21.15.72:0/3777657790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f215815ac00 0x7f215806c910 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f21480097a0 tx=0x7f214802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aeae3d33a7bd55f server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21567fc640 1 -- 172.21.15.72:0/3777657790 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21480315a0 con 0x7f215815ac00 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.022+0000 7f21567fc640 1 -- 172.21.15.72:0/3777657790 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2148037070 con 0x7f215815ac00 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f21567fc640 1 -- 172.21.15.72:0/3777657790 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148031960 con 0x7f215815ac00 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 -- 172.21.15.72:0/3777657790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f215815ac00 msgr2=0x7f215806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 --2- 172.21.15.72:0/3777657790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f215815ac00 0x7f215806c910 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f21480097a0 tx=0x7f214802b840 comp rx=0 tx=0).stop 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 -- 172.21.15.72:0/3777657790 shutdown_connections 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 --2- 172.21.15.72:0/3777657790 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f215815ac00 0x7f215806c910 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 -- 172.21.15.72:0/3777657790 >> 172.21.15.72:0/3777657790 conn(0x7f21580b1650 msgr2=0x7f21580b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 -- 172.21.15.72:0/3777657790 shutdown_connections 2024-03-06T07:32:34.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.023+0000 7f215e0df640 1 -- 172.21.15.72:0/3777657790 wait complete. 2024-03-06T07:32:34.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.024+0000 7f215e0df640 1 Processor -- start 2024-03-06T07:32:34.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.024+0000 7f215e0df640 1 -- start start 2024-03-06T07:32:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f215e0df640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21580708b0 0x7f21581f0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f215e0df640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2158070d10 0x7f215815d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f215e0df640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f215815ac00 0x7f215815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f215e0df640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21580cd5c0 con 0x7f21580708b0 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f215e0df640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21580cd2e0 con 0x7f215815ac00 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f215e0df640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21580cd8a0 con 0x7f2158070d10 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21580708b0 0x7f21581f0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21580708b0 0x7f21581f0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54940/0 (socket says 172.21.15.72:54940) 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 -- 172.21.15.72:0/2623038189 learned_addr learned my addr 172.21.15.72:0/2623038189 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f2156ffd640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2158070d10 0x7f215815d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f2157fff640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f215815ac00 0x7f215815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 -- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2158070d10 msgr2=0x7f215815d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2158070d10 0x7f215815d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 -- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f215815ac00 msgr2=0x7f215815dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f215815ac00 0x7f215815dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f21577fe640 1 -- 172.21.15.72:0/2623038189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215814c3f0 con 0x7f21580708b0 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f2157fff640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f215815ac00 0x7f215815dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.025+0000 7f2156ffd640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2158070d10 0x7f215815d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.026+0000 7f21577fe640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21580708b0 0x7f21581f0060 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f21480097a0 tx=0x7f21480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.026+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148006e10 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.026+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2148004030 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.026+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f215815bd90 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.026+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2148004400 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.026+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215815c350 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.027+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f215804f330 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.027+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f214804e020 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f213ffff640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f214406b4d0 0x7f214406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f214802f020 con 0x7f21580708b0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f2156ffd640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f214406b4d0 0x7f214406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f213ffff640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f2144071430 0x7f2144073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2148048ec0 con 0x7f2144071430 2024-03-06T07:32:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21480ae3a0 con 0x7f21580708b0 2024-03-06T07:32:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f2157fff640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f2144071430 0x7f2144073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f2156ffd640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f214406b4d0 0x7f214406d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f215815f0d0 tx=0x7f214c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.028+0000 7f2157fff640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f2144071430 0x7f2144073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.029+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== osd.2 v2:172.21.15.72:6817/3054266462 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2148048ec0 con 0x7f2144071430 2024-03-06T07:32:34.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.042+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 --> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f215815c7e0 con 0x7f2144071430 2024-03-06T07:32:34.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.042+0000 7f213ffff640 1 -- 172.21.15.72:0/2623038189 <== osd.2 v2:172.21.15.72:6817/3054266462 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f215815c7e0 con 0x7f2144071430 2024-03-06T07:32:34.044 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:34.045 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:34.045 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.042+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f2144071430 msgr2=0x7f2144073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.042+0000 7f215e0df640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6817/3054266462,v1:172.21.15.72:6819/3054266462] conn(0x7f2144071430 0x7f2144073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.042+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f214406b4d0 msgr2=0x7f214406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f214406b4d0 0x7f214406d990 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f215815f0d0 tx=0x7f214c009040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21580708b0 msgr2=0x7f21581f0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 --2- 172.21.15.72:0/2623038189 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f21580708b0 0x7f21581f0060 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f21480097a0 tx=0x7f21480046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f2157fff640 1 -- 172.21.15.72:0/2623038189 reap_dead start 2024-03-06T07:32:34.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 shutdown_connections 2024-03-06T07:32:34.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 >> 172.21.15.72:0/2623038189 conn(0x7f21580b1650 msgr2=0x7f21580b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 shutdown_connections 2024-03-06T07:32:34.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.043+0000 7f215e0df640 1 -- 172.21.15.72:0/2623038189 wait complete. 2024-03-06T07:32:34.055 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-06T07:32:34.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.154+0000 7fb9280d6640 1 Processor -- start 2024-03-06T07:32:34.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.154+0000 7fb9280d6640 1 -- start start 2024-03-06T07:32:34.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb9280d6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9200b0240 0x7fb92006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb9280d6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb92006b4f0 0x7fb9200cd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb9280d6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb920161cf0 con 0x7fb9201586c0 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb9280d6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb920158aa0 con 0x7fb9200b0240 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb9280d6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92006b960 con 0x7fb92006b4f0 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9200b0240 0x7fb92006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9200b0240 0x7fb92006afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54942/0 (socket says 172.21.15.72:54942) 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 -- 172.21.15.72:0/3806553237 learned_addr learned my addr 172.21.15.72:0/3806553237 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb926673640 1 --2- 172.21.15.72:0/3806553237 >> v2:172.21.15.146:3300/0 conn(0x7fb92006b4f0 0x7fb9200cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 -- 172.21.15.72:0/3806553237 >> v2:172.21.15.146:3300/0 conn(0x7fb92006b4f0 msgr2=0x7fb9200cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 --2- 172.21.15.72:0/3806553237 >> v2:172.21.15.146:3300/0 conn(0x7fb92006b4f0 0x7fb9200cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 -- 172.21.15.72:0/3806553237 >> v1:172.21.15.146:6789/0 conn(0x7fb9201586c0 legacy=0x7fb92015efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.155+0000 7fb925671640 1 -- 172.21.15.72:0/3806553237 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb920161fd0 con 0x7fb9200b0240 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.156+0000 7fb925671640 1 --2- 172.21.15.72:0/3806553237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb92006afb0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fb920067ce0 tx=0x7fb91002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfe06ae7edbf894 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.156+0000 7fb924e70640 1 -- 172.21.15.72:0/3806553237 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910035020 con 0x7fb9200b0240 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.156+0000 7fb924e70640 1 -- 172.21.15.72:0/3806553237 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910031870 con 0x7fb9200b0240 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.156+0000 7fb924e70640 1 -- 172.21.15.72:0/3806553237 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910031c40 con 0x7fb9200b0240 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 -- 172.21.15.72:0/3806553237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 msgr2=0x7fb92006afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 --2- 172.21.15.72:0/3806553237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb92006afb0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fb920067ce0 tx=0x7fb91002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 -- 172.21.15.72:0/3806553237 shutdown_connections 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 --2- 172.21.15.72:0/3806553237 >> v2:172.21.15.146:3300/0 conn(0x7fb92006b4f0 0x7fb9200cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 --2- 172.21.15.72:0/3806553237 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb92006afb0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 -- 172.21.15.72:0/3806553237 >> 172.21.15.72:0/3806553237 conn(0x7fb9200d09d0 msgr2=0x7fb9200d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 -- 172.21.15.72:0/3806553237 shutdown_connections 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.157+0000 7fb9280d6640 1 -- 172.21.15.72:0/3806553237 wait complete. 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.158+0000 7fb9280d6640 1 Processor -- start 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.158+0000 7fb9280d6640 1 -- start start 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.158+0000 7fb9280d6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb92006b4f0 0x7fb92013dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb920140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9201586c0 0x7fb9200b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb920162870 con 0x7fb9200b0240 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb920162590 con 0x7fb9201586c0 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9200719e0 con 0x7fb92006b4f0 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb920140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb920140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54950/0 (socket says 172.21.15.72:54950) 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 -- 172.21.15.72:0/1732363082 learned_addr learned my addr 172.21.15.72:0/1732363082 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925e72640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb92006b4f0 0x7fb92013dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb926673640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9201586c0 0x7fb9200b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 -- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb92006b4f0 msgr2=0x7fb92013dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb92006b4f0 0x7fb92013dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 -- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9201586c0 msgr2=0x7fb9200b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9201586c0 0x7fb9200b4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 -- 172.21.15.72:0/1732363082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92013e520 con 0x7fb9200b0240 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb925671640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb920140cf0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fb910000c00 tx=0x7fb9100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910040070 con 0x7fb9200b0240 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb910030d00 con 0x7fb9200b0240 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9201e7b00 con 0x7fb9200b0240 2024-03-06T07:32:34.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb910035040 con 0x7fb9200b0240 2024-03-06T07:32:34.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.159+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9201e80c0 con 0x7fb9200b0240 2024-03-06T07:32:34.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.160+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92004f330 con 0x7fb9200b0240 2024-03-06T07:32:34.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.161+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb91004b060 con 0x7fb9200b0240 2024-03-06T07:32:34.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.161+0000 7fb9167fc640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb90406b4d0 0x7fb90406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.161+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb9100abb90 con 0x7fb9200b0240 2024-03-06T07:32:34.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.161+0000 7fb925e72640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb90406b4d0 0x7fb90406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.162+0000 7fb925e72640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb90406b4d0 0x7fb90406d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb91c006fd0 tx=0x7fb91c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.166+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb91007a6f0 con 0x7fb9200b0240 2024-03-06T07:32:34.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.337+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9200d17a0 con 0x7fb90406b4d0 2024-03-06T07:32:34.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.337+0000 7fb9167fc640 1 -- 172.21.15.72:0/1732363082 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb9200d17a0 con 0x7fb90406b4d0 2024-03-06T07:32:34.339 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 2 to deep-scrub 2024-03-06T07:32:34.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb90406b4d0 msgr2=0x7fb90406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb90406b4d0 0x7fb90406d990 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fb91c006fd0 tx=0x7fb91c009060 comp rx=0 tx=0).stop 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 msgr2=0x7fb920140cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb920140cf0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fb910000c00 tx=0x7fb9100046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 shutdown_connections 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb90406b4d0 0x7fb90406d990 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9201586c0 0x7fb9200b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9200b0240 0x7fb920140cf0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 --2- 172.21.15.72:0/1732363082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb92006b4f0 0x7fb92013dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 >> 172.21.15.72:0/1732363082 conn(0x7fb9200d09d0 msgr2=0x7fb9200cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 shutdown_connections 2024-03-06T07:32:34.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.339+0000 7fb9280d6640 1 -- 172.21.15.72:0/1732363082 wait complete. 2024-03-06T07:32:34.352 INFO:tasks.ceph:Scrubbing osd.3 2024-03-06T07:32:34.352 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:34.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.453+0000 7fddd3bb5640 1 Processor -- start 2024-03-06T07:32:34.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.454+0000 7fddd3bb5640 1 -- start start 2024-03-06T07:32:34.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.454+0000 7fddd3bb5640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fddcc0708b0 0x7fddcc070d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.454+0000 7fddd3bb5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc15afe0 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.454+0000 7fddd3bb5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc0cd000 con 0x7fddcc15ac00 2024-03-06T07:32:34.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.454+0000 7fddd3bb5640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddcc071250 con 0x7fddcc0708b0 2024-03-06T07:32:34.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd2152640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fddcc0ce0a0 0x7fddcc0ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53142/0 (socket says 172.21.15.72:53142) 2024-03-06T07:32:34.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd2152640 1 -- 172.21.15.72:0/156747254 learned_addr learned my addr 172.21.15.72:0/156747254 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd1150640 1 --2- 172.21.15.72:0/156747254 >> v2:172.21.15.146:3300/0 conn(0x7fddcc0708b0 0x7fddcc070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260808999 0 0) 0x7fddcc15afe0 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb8003520 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214117848 0 0) 0x7fddcc0cd000 con 0x7fddcc15ac00 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddcc15afe0 con 0x7fddcc15ac00 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd1150640 1 -- 172.21.15.72:0/156747254 >> v1:172.21.15.146:6789/0 conn(0x7fddcc15ac00 legacy=0x7fddcc06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd1150640 1 -- 172.21.15.72:0/156747254 >> v1:172.21.15.72:6789/0 conn(0x7fddcc0ce0a0 legacy=0x7fddcc0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987743875 0 0) 0x7fddb8003520 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.455+0000 7fddd1150640 1 -- 172.21.15.72:0/156747254 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0cd2e0 con 0x7fddcc0708b0 2024-03-06T07:32:34.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.456+0000 7fddd1150640 1 --2- 172.21.15.72:0/156747254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc0708b0 0x7fddcc070d10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097a0 tx=0x7fddc802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d65578c8b1873172 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.456+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8037070 con 0x7fddcc0708b0 2024-03-06T07:32:34.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.456+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc8030510 con 0x7fddcc0708b0 2024-03-06T07:32:34.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.456+0000 7fddd094f640 1 -- 172.21.15.72:0/156747254 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc40019f0 con 0x7fddcc0708b0 2024-03-06T07:32:34.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 -- 172.21.15.72:0/156747254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc0708b0 msgr2=0x7fddcc070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 --2- 172.21.15.72:0/156747254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc0708b0 0x7fddcc070d10 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fddc80097a0 tx=0x7fddc802b790 comp rx=0 tx=0).stop 2024-03-06T07:32:34.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 -- 172.21.15.72:0/156747254 shutdown_connections 2024-03-06T07:32:34.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 --2- 172.21.15.72:0/156747254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc0708b0 0x7fddcc070d10 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 -- 172.21.15.72:0/156747254 >> 172.21.15.72:0/156747254 conn(0x7fddcc0b1650 msgr2=0x7fddcc0b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 -- 172.21.15.72:0/156747254 shutdown_connections 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.457+0000 7fddd3bb5640 1 -- 172.21.15.72:0/156747254 wait complete. 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.458+0000 7fddd3bb5640 1 Processor -- start 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.458+0000 7fddd3bb5640 1 -- start start 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.458+0000 7fddd3bb5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddcc0708b0 0x7fddcc14f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.458+0000 7fddd3bb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddcc0ce0a0 0x7fddcc15d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.458+0000 7fddd3bb5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc15ac00 0x7fddcc15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd3bb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0cd8a0 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd3bb5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0cdc10 con 0x7fddcc15ac00 2024-03-06T07:32:34.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd3bb5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddcc0cf5d0 con 0x7fddcc0708b0 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddcc0ce0a0 0x7fddcc15d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddcc0ce0a0 0x7fddcc15d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54954/0 (socket says 172.21.15.72:54954) 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 -- 172.21.15.72:0/1157466895 learned_addr learned my addr 172.21.15.72:0/1157466895 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1951640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddcc0708b0 0x7fddcc14f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd2152640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc15ac00 0x7fddcc15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 -- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddcc0708b0 msgr2=0x7fddcc14f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddcc0708b0 0x7fddcc14f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 -- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc15ac00 msgr2=0x7fddcc15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddcc15ac00 0x7fddcc15d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 -- 172.21.15.72:0/1157466895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc158440 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1951640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddcc0708b0 0x7fddcc14f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd1150640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddcc0ce0a0 0x7fddcc15d2d0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fddc80098c0 tx=0x7fddc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8041070 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc803c870 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc158720 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc8037040 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.459+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc15b400 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.460+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fddcc15b210 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fddc8044070 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddc27fc640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb006b4d0 0x7fddb006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fddc80aba60 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddd1951640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb006b4d0 0x7fddb006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddc27fc640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7fddb0071430 0x7fddb0073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fddc8030510 con 0x7fddb0071430 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddc80b02c0 con 0x7fddcc0ce0a0 2024-03-06T07:32:34.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.461+0000 7fddd2152640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7fddb0071430 0x7fddb0073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.462+0000 7fddd1951640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb006b4d0 0x7fddb006d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0011e0 tx=0x7fddbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.462+0000 7fddd2152640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7fddb0071430 0x7fddb0073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.462+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== osd.3 v2:172.21.15.72:6808/4245662075 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fddc8030510 con 0x7fddb0071430 2024-03-06T07:32:34.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.475+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 --> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fddcc04f330 con 0x7fddb0071430 2024-03-06T07:32:34.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.475+0000 7fddc27fc640 1 -- 172.21.15.72:0/1157466895 <== osd.3 v2:172.21.15.72:6808/4245662075 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fddcc04f330 con 0x7fddb0071430 2024-03-06T07:32:34.478 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:34.478 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:34.478 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:34.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7fddb0071430 msgr2=0x7fddb0073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6808/4245662075,v1:172.21.15.72:6809/4245662075] conn(0x7fddb0071430 0x7fddb0073850 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb006b4d0 msgr2=0x7fddb006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fddb006b4d0 0x7fddb006d990 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fddbc0011e0 tx=0x7fddbc009040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddcc0ce0a0 msgr2=0x7fddcc15d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 --2- 172.21.15.72:0/1157466895 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fddcc0ce0a0 0x7fddcc15d2d0 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fddc80098c0 tx=0x7fddc80046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd2152640 1 -- 172.21.15.72:0/1157466895 reap_dead start 2024-03-06T07:32:34.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 shutdown_connections 2024-03-06T07:32:34.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 >> 172.21.15.72:0/1157466895 conn(0x7fddcc0b1650 msgr2=0x7fddcc0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 shutdown_connections 2024-03-06T07:32:34.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.476+0000 7fddd3bb5640 1 -- 172.21.15.72:0/1157466895 wait complete. 2024-03-06T07:32:34.489 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-06T07:32:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.586+0000 7f85c5116640 1 Processor -- start 2024-03-06T07:32:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.586+0000 7f85c5116640 1 -- start start 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85c5116640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f85c00cec30 0x7f85c00cf010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85c5116640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85c00723f0 0x7f85c00ad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85c5116640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c0072880 0x7f85c00adda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85c5116640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c00d0850 con 0x7f85c00723f0 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85c5116640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c00cfff0 con 0x7f85c00cec30 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85c5116640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cf550 con 0x7f85c0072880 2024-03-06T07:32:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85c00723f0 0x7f85c00ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85c00723f0 0x7f85c00ad860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54968/0 (socket says 172.21.15.72:54968) 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 -- 172.21.15.72:0/290313796 learned_addr learned my addr 172.21.15.72:0/290313796 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85bf59e640 1 --2- 172.21.15.72:0/290313796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c0072880 0x7f85c00adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85bed9d640 1 --2- 172.21.15.72:0/290313796 >> v2:172.21.15.146:3300/0 conn(0x7f85c00cec30 0x7f85c00cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 -- 172.21.15.72:0/290313796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c0072880 msgr2=0x7f85c00adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 --2- 172.21.15.72:0/290313796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c0072880 0x7f85c00adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 -- 172.21.15.72:0/290313796 >> v2:172.21.15.146:3300/0 conn(0x7f85c00cec30 msgr2=0x7f85c00cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 --2- 172.21.15.72:0/290313796 >> v2:172.21.15.146:3300/0 conn(0x7f85c00cec30 0x7f85c00cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.587+0000 7f85be59c640 1 -- 172.21.15.72:0/290313796 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c015e520 con 0x7f85c00723f0 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.588+0000 7f85be59c640 1 --2- 172.21.15.72:0/290313796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c00723f0 0x7f85c00ad860 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f85c00737d0 tx=0x7f85b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe8bd59722f5098 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.588+0000 7f85bdd9b640 1 -- 172.21.15.72:0/290313796 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4035020 con 0x7f85c00723f0 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.588+0000 7f85bdd9b640 1 -- 172.21.15.72:0/290313796 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4031870 con 0x7f85c00723f0 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.588+0000 7f85bdd9b640 1 -- 172.21.15.72:0/290313796 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4031c40 con 0x7f85c00723f0 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.588+0000 7f85c5116640 1 -- 172.21.15.72:0/290313796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c00723f0 msgr2=0x7f85c00ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.588+0000 7f85c5116640 1 --2- 172.21.15.72:0/290313796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c00723f0 0x7f85c00ad860 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f85c00737d0 tx=0x7f85b402b910 comp rx=0 tx=0).stop 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 -- 172.21.15.72:0/290313796 shutdown_connections 2024-03-06T07:32:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 --2- 172.21.15.72:0/290313796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c0072880 0x7f85c00adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 --2- 172.21.15.72:0/290313796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c00723f0 0x7f85c00ad860 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 --2- 172.21.15.72:0/290313796 >> v2:172.21.15.146:3300/0 conn(0x7f85c00cec30 0x7f85c00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 -- 172.21.15.72:0/290313796 >> 172.21.15.72:0/290313796 conn(0x7f85c00b1c80 msgr2=0x7f85c00cf8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 -- 172.21.15.72:0/290313796 shutdown_connections 2024-03-06T07:32:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.589+0000 7f85c5116640 1 -- 172.21.15.72:0/290313796 wait complete. 2024-03-06T07:32:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.590+0000 7f85c5116640 1 Processor -- start 2024-03-06T07:32:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.590+0000 7f85c5116640 1 -- start start 2024-03-06T07:32:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85c5116640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c00723f0 0x7f85c01f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85c5116640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 0x7f85c01f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85c5116640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c00cec30 0x7f85c015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85c5116640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cc4d0 con 0x7f85c0072880 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85c5116640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00d0570 con 0x7f85c00723f0 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85c5116640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cc7b0 con 0x7f85c00cec30 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 0x7f85c01f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 0x7f85c01f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54974/0 (socket says 172.21.15.72:54974) 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 -- 172.21.15.72:0/888465273 learned_addr learned my addr 172.21.15.72:0/888465273 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85bed9d640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c00723f0 0x7f85c01f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85bf59e640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c00cec30 0x7f85c015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 -- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c00cec30 msgr2=0x7f85c015d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c00cec30 0x7f85c015d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 -- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c00723f0 msgr2=0x7f85c01f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c00723f0 0x7f85c01f84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 -- 172.21.15.72:0/888465273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c015ad60 con 0x7f85c0072880 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.591+0000 7f85be59c640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 0x7f85c01f1b40 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f85b402b950 tx=0x7f85b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.592+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4040070 con 0x7f85c0072880 2024-03-06T07:32:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.592+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b4030d00 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.592+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c015b040 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.592+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b4035040 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.592+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c0147f60 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.593+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c004f330 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.593+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f85b4038020 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.593+0000 7f85a77fe640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f85ac06b4d0 0x7f85ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.593+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f85b40abd30 con 0x7f85c0072880 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.593+0000 7f85bed9d640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f85ac06b4d0 0x7f85ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.594+0000 7f85bed9d640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f85ac06b4d0 0x7f85ac06d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f85c015e860 tx=0x7f85b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.598+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f85b407a800 con 0x7f85c0072880 2024-03-06T07:32:34.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.769+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f85c00cc110 con 0x7f85ac06b4d0 2024-03-06T07:32:34.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.769+0000 7f85a77fe640 1 -- 172.21.15.72:0/888465273 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f85c00cc110 con 0x7f85ac06b4d0 2024-03-06T07:32:34.771 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 3 to deep-scrub 2024-03-06T07:32:34.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f85ac06b4d0 msgr2=0x7f85ac06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f85ac06b4d0 0x7f85ac06d990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f85c015e860 tx=0x7f85b0008040 comp rx=0 tx=0).stop 2024-03-06T07:32:34.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 msgr2=0x7f85c01f1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 0x7f85c01f1b40 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f85b402b950 tx=0x7f85b40046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 shutdown_connections 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f85ac06b4d0 0x7f85ac06d990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c00cec30 0x7f85c015d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85c0072880 0x7f85c01f1b40 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 --2- 172.21.15.72:0/888465273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c00723f0 0x7f85c01f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 >> 172.21.15.72:0/888465273 conn(0x7f85c00b1c80 msgr2=0x7f85c006b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 shutdown_connections 2024-03-06T07:32:34.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.771+0000 7f85c5116640 1 -- 172.21.15.72:0/888465273 wait complete. 2024-03-06T07:32:34.784 INFO:tasks.ceph:Scrubbing osd.4 2024-03-06T07:32:34.784 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:34.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.886+0000 7f1bd6b63640 1 Processor -- start 2024-03-06T07:32:34.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bd6b63640 1 -- start start 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bd6b63640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1bd015dfa0 0x7f1bd00d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bd6b63640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd015e380 con 0x7f1bd00d4630 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bd6b63640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd00d1e30 con 0x7f1bd00d29f0 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bd6b63640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd00d2100 con 0x7f1bd015dfa0 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bd48ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1bd015dfa0 0x7f1bd00d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.887+0000 7f1bcffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1bd00d4630 0x7f1bd00d22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53156/0 (socket says 172.21.15.72:53156) 2024-03-06T07:32:34.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bd48ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1bd015dfa0 0x7f1bd00d18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54988/0 (socket says 172.21.15.72:54988) 2024-03-06T07:32:34.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bcffff640 1 -- 172.21.15.72:0/1381283565 learned_addr learned my addr 172.21.15.72:0/1381283565 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:34.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bd48ff640 1 -- 172.21.15.72:0/1381283565 >> v1:172.21.15.146:6789/0 conn(0x7f1bd00d29f0 legacy=0x7f1bd00cb190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:34.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bd48ff640 1 -- 172.21.15.72:0/1381283565 >> v1:172.21.15.72:6789/0 conn(0x7f1bd00d4630 legacy=0x7f1bd00d22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bd48ff640 1 -- 172.21.15.72:0/1381283565 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd00cf470 con 0x7f1bd015dfa0 2024-03-06T07:32:34.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bcf7fe640 1 -- 172.21.15.72:0/1381283565 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432880727 0 0) 0x7f1bd015e380 con 0x7f1bd00d4630 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.888+0000 7f1bd48ff640 1 --2- 172.21.15.72:0/1381283565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd00d18f0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00097a0 tx=0x7f1bc002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4808e67e6086ac8e server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bcf7fe640 1 -- 172.21.15.72:0/1381283565 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc00315a0 con 0x7f1bd015dfa0 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bcf7fe640 1 -- 172.21.15.72:0/1381283565 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc0037070 con 0x7f1bd015dfa0 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bcf7fe640 1 -- 172.21.15.72:0/1381283565 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0031960 con 0x7f1bd015dfa0 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 -- 172.21.15.72:0/1381283565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 msgr2=0x7f1bd00d18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 --2- 172.21.15.72:0/1381283565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd00d18f0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00097a0 tx=0x7f1bc002b840 comp rx=0 tx=0).stop 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 -- 172.21.15.72:0/1381283565 shutdown_connections 2024-03-06T07:32:34.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 --2- 172.21.15.72:0/1381283565 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd00d18f0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 -- 172.21.15.72:0/1381283565 >> 172.21.15.72:0/1381283565 conn(0x7f1bd009d000 msgr2=0x7f1bd00d3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 -- 172.21.15.72:0/1381283565 shutdown_connections 2024-03-06T07:32:34.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.889+0000 7f1bd6b63640 1 -- 172.21.15.72:0/1381283565 wait complete. 2024-03-06T07:32:34.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.890+0000 7f1bd6b63640 1 Processor -- start 2024-03-06T07:32:34.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.890+0000 7f1bd6b63640 1 -- start start 2024-03-06T07:32:34.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd6b63640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bd00d29f0 0x7f1bd009f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd6b63640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bd00d4630 0x7f1bd015bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd6b63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd014eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd6b63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd00cb8a0 con 0x7f1bd015dfa0 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd6b63640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd00cfa30 con 0x7f1bd00d29f0 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd6b63640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd00cba20 con 0x7f1bd00d4630 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd5100640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd014eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd5100640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd014eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55004/0 (socket says 172.21.15.72:55004) 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.891+0000 7f1bd5100640 1 -- 172.21.15.72:0/3475650541 learned_addr learned my addr 172.21.15.72:0/3475650541 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd48ff640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bd00d29f0 0x7f1bd009f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd5100640 1 -- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bd00d4630 msgr2=0x7f1bd015bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd5100640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bd00d4630 0x7f1bd015bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd5100640 1 -- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bd00d29f0 msgr2=0x7f1bd009f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd5100640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bd00d29f0 0x7f1bd009f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd5100640 1 -- 172.21.15.72:0/3475650541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd009f570 con 0x7f1bd015dfa0 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd48ff640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bd00d29f0 0x7f1bd009f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd5100640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd014eed0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800c780 tx=0x7f1bc800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc8017070 con 0x7f1bd015dfa0 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc8003da0 con 0x7f1bd015dfa0 2024-03-06T07:32:34.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd014cf70 con 0x7f1bd015dfa0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80052b0 con 0x7f1bd015dfa0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.892+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd014d530 con 0x7f1bd015dfa0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.893+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1bd004f330 con 0x7f1bd015dfa0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.894+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f1bc8006b10 con 0x7f1bd015dfa0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.894+0000 7f1bcd7fa640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.894+0000 7f1bd48ff640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.894+0000 7f1bd48ff640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0036000 tx=0x7f1bc0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.894+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1bc808f050 con 0x7f1bd015dfa0 2024-03-06T07:32:34.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.895+0000 7f1bcd7fa640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1bbc0713a0 0x7f1bbc0737c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:34.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.895+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1bbc073e70 con 0x7f1bbc0713a0 2024-03-06T07:32:34.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.895+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bc8054110 con 0x7f1bd015dfa0 2024-03-06T07:32:34.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.895+0000 7f1bcffff640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1bbc0713a0 0x7f1bbc0737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:34.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.895+0000 7f1bcffff640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1bbc0713a0 0x7f1bbc0737c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:34.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.896+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== osd.4 v2:172.21.15.146:6820/3084745005 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1bbc073e70 con 0x7f1bbc0713a0 2024-03-06T07:32:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.909+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 --> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1bd00d07c0 con 0x7f1bbc0713a0 2024-03-06T07:32:34.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bcd7fa640 1 -- 172.21.15.72:0/3475650541 <== osd.4 v2:172.21.15.146:6820/3084745005 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f1bd00d07c0 con 0x7f1bbc0713a0 2024-03-06T07:32:34.912 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:34.912 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:34.912 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:34.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1bbc0713a0 msgr2=0x7f1bbc0737c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.146:6820/3084745005,v1:172.21.15.146:6822/3084745005] conn(0x7f1bbc0713a0 0x7f1bbc0737c0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:34.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1bbc06b4d0 msgr2=0x7f1bbc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f1bbc06b4d0 0x7f1bbc06d990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0036000 tx=0x7f1bc0008030 comp rx=0 tx=0).stop 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 msgr2=0x7f1bd014eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 --2- 172.21.15.72:0/3475650541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bd015dfa0 0x7f1bd014eed0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800c780 tx=0x7f1bc800cc40 comp rx=0 tx=0).stop 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd5100640 1 -- 172.21.15.72:0/3475650541 reap_dead start 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 shutdown_connections 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 >> 172.21.15.72:0/3475650541 conn(0x7f1bd009d000 msgr2=0x7f1bd006b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 shutdown_connections 2024-03-06T07:32:34.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:34.910+0000 7f1bd6b63640 1 -- 172.21.15.72:0/3475650541 wait complete. 2024-03-06T07:32:34.922 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-06T07:32:35.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.020+0000 7feeeb4fb640 1 Processor -- start 2024-03-06T07:32:35.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feeeb4fb640 1 -- start start 2024-03-06T07:32:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feeeb4fb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feee40b4460 0x7feee40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feeeb4fb640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feee4161cf0 0x7feee4162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feeeb4fb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee40d22c0 con 0x7feee4162970 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feeeb4fb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee4161b00 con 0x7feee40b4460 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feeeb4fb640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feee40cecc0 con 0x7feee4161cf0 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feee9297640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feee40b4460 0x7feee40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feee9297640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7feee40b4460 0x7feee40d4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55016/0 (socket says 172.21.15.72:55016) 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.021+0000 7feee9297640 1 -- 172.21.15.72:0/3131862161 learned_addr learned my addr 172.21.15.72:0/3131862161 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.022+0000 7feee9a98640 1 --2- 172.21.15.72:0/3131862161 >> v2:172.21.15.146:3300/0 conn(0x7feee4161cf0 0x7feee4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.022+0000 7feee9297640 1 -- 172.21.15.72:0/3131862161 >> v2:172.21.15.146:3300/0 conn(0x7feee4161cf0 msgr2=0x7feee4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.022+0000 7feee9297640 1 --2- 172.21.15.72:0/3131862161 >> v2:172.21.15.146:3300/0 conn(0x7feee4161cf0 0x7feee4162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.022+0000 7feee9297640 1 -- 172.21.15.72:0/3131862161 >> v1:172.21.15.146:6789/0 conn(0x7feee4162970 legacy=0x7feee40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.022+0000 7feee9297640 1 -- 172.21.15.72:0/3131862161 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee40d25a0 con 0x7feee40b4460 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.022+0000 7feee9297640 1 --2- 172.21.15.72:0/3131862161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee40d4800 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7feee408c900 tx=0x7feed402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7e1ea86626cdbd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feee3fff640 1 -- 172.21.15.72:0/3131862161 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed402ca80 con 0x7feee40b4460 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feee3fff640 1 -- 172.21.15.72:0/3131862161 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed402cc20 con 0x7feee40b4460 2024-03-06T07:32:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feee3fff640 1 -- 172.21.15.72:0/3131862161 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40343f0 con 0x7feee40b4460 2024-03-06T07:32:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3131862161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 msgr2=0x7feee40d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3131862161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee40d4800 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7feee408c900 tx=0x7feed402bee0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3131862161 shutdown_connections 2024-03-06T07:32:35.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3131862161 >> v2:172.21.15.146:3300/0 conn(0x7feee4161cf0 0x7feee4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3131862161 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee40d4800 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3131862161 >> 172.21.15.72:0/3131862161 conn(0x7feee40d3ce0 msgr2=0x7feee40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3131862161 shutdown_connections 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.023+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3131862161 wait complete. 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.024+0000 7feeeb4fb640 1 Processor -- start 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.024+0000 7feeeb4fb640 1 -- start start 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feeeb4fb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee4159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feeeb4fb640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feee4161cf0 0x7feee40c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feeeb4fb640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feee4162970 0x7feee40b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feeeb4fb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee40ce950 con 0x7feee40b4460 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feeeb4fb640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee40d2b60 con 0x7feee4161cf0 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feeeb4fb640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee40b4840 con 0x7feee4162970 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee4159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee4159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55030/0 (socket says 172.21.15.72:55030) 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 -- 172.21.15.72:0/3649017313 learned_addr learned my addr 172.21.15.72:0/3649017313 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 -- 172.21.15.72:0/3649017313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feee4162970 msgr2=0x7feee40b1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feee4162970 0x7feee40b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 -- 172.21.15.72:0/3649017313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feee4161cf0 msgr2=0x7feee40c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feee4161cf0 0x7feee40c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.025+0000 7feee9297640 1 -- 172.21.15.72:0/3649017313 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee4159550 con 0x7feee40b4460 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.026+0000 7feee9297640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee4159010 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7feed4000f80 tx=0x7feed4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.026+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4043020 con 0x7feee40b4460 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.026+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed403e450 con 0x7feee40b4460 2024-03-06T07:32:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.026+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed403e840 con 0x7feee40b4460 2024-03-06T07:32:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.026+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee40caf50 con 0x7feee40b4460 2024-03-06T07:32:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.026+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee40cb510 con 0x7feee40b4460 2024-03-06T07:32:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.027+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee404f330 con 0x7feee40b4460 2024-03-06T07:32:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.027+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7feed4046020 con 0x7feee40b4460 2024-03-06T07:32:35.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.028+0000 7feee1ffb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feed006b4d0 0x7feed006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.028+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7feed40ac820 con 0x7feee40b4460 2024-03-06T07:32:35.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.028+0000 7feee8a96640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feed006b4d0 0x7feed006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.028+0000 7feee8a96640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feed006b4d0 0x7feed006d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feee40b3060 tx=0x7feed80079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.032+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7feed404d030 con 0x7feee40b4460 2024-03-06T07:32:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.206+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7feee40d30f0 con 0x7feed006b4d0 2024-03-06T07:32:35.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.206+0000 7feee1ffb640 1 -- 172.21.15.72:0/3649017313 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7feee40d30f0 con 0x7feed006b4d0 2024-03-06T07:32:35.209 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 4 to deep-scrub 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feed006b4d0 msgr2=0x7feed006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feed006b4d0 0x7feed006d990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7feee40b3060 tx=0x7feed80079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 msgr2=0x7feee4159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee4159010 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7feed4000f80 tx=0x7feed4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 shutdown_connections 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7feed006b4d0 0x7feed006d990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feee4162970 0x7feee40b1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feee4161cf0 0x7feee40c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 --2- 172.21.15.72:0/3649017313 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feee40b4460 0x7feee4159010 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 >> 172.21.15.72:0/3649017313 conn(0x7feee40d3ce0 msgr2=0x7feee40d0160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 shutdown_connections 2024-03-06T07:32:35.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.208+0000 7feeeb4fb640 1 -- 172.21.15.72:0/3649017313 wait complete. 2024-03-06T07:32:35.221 INFO:tasks.ceph:Scrubbing osd.5 2024-03-06T07:32:35.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.323+0000 7fe90d748640 1 Processor -- start 2024-03-06T07:32:35.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.323+0000 7fe90d748640 1 -- start start 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe90d748640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe90815a490 0x7fe90806df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe90d748640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe9080b3fd0 0x7fe9080b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe90d748640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90806e4d0 con 0x7fe9080b2e90 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe90d748640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9080b4490 con 0x7fe9080b3fd0 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe90d748640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9080cf4a0 con 0x7fe90815a490 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe9067fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe9080b3fd0 0x7fe9080b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe9067fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe9080b3fd0 0x7fe9080b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55042/0 (socket says 172.21.15.72:55042) 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe9067fc640 1 -- 172.21.15.72:0/3338736520 learned_addr learned my addr 172.21.15.72:0/3338736520 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe906ffd640 1 --2- 172.21.15.72:0/3338736520 >> v2:172.21.15.146:3300/0 conn(0x7fe90815a490 0x7fe90806df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe9067fc640 1 -- 172.21.15.72:0/3338736520 >> v2:172.21.15.146:3300/0 conn(0x7fe90815a490 msgr2=0x7fe90806df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe9067fc640 1 --2- 172.21.15.72:0/3338736520 >> v2:172.21.15.146:3300/0 conn(0x7fe90815a490 0x7fe90806df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.324+0000 7fe9067fc640 1 -- 172.21.15.72:0/3338736520 >> v1:172.21.15.146:6789/0 conn(0x7fe9080b2e90 legacy=0x7fe9080708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.325+0000 7fe9067fc640 1 -- 172.21.15.72:0/3338736520 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9080b4750 con 0x7fe9080b3fd0 2024-03-06T07:32:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.325+0000 7fe9067fc640 1 --2- 172.21.15.72:0/3338736520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b3fd0 0x7fe9080b2950 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fe90806bf40 tx=0x7fe8fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca78a8f2455eb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.325+0000 7fe905ffb640 1 -- 172.21.15.72:0/3338736520 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc035020 con 0x7fe9080b3fd0 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.325+0000 7fe905ffb640 1 -- 172.21.15.72:0/3338736520 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8fc031870 con 0x7fe9080b3fd0 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.325+0000 7fe905ffb640 1 -- 172.21.15.72:0/3338736520 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc031c40 con 0x7fe9080b3fd0 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 -- 172.21.15.72:0/3338736520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b3fd0 msgr2=0x7fe9080b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 --2- 172.21.15.72:0/3338736520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b3fd0 0x7fe9080b2950 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7fe90806bf40 tx=0x7fe8fc02b910 comp rx=0 tx=0).stop 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 -- 172.21.15.72:0/3338736520 shutdown_connections 2024-03-06T07:32:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 --2- 172.21.15.72:0/3338736520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b3fd0 0x7fe9080b2950 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 --2- 172.21.15.72:0/3338736520 >> v2:172.21.15.146:3300/0 conn(0x7fe90815a490 0x7fe90806df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 -- 172.21.15.72:0/3338736520 >> 172.21.15.72:0/3338736520 conn(0x7fe9080b1650 msgr2=0x7fe9080b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 -- 172.21.15.72:0/3338736520 shutdown_connections 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.326+0000 7fe90d748640 1 -- 172.21.15.72:0/3338736520 wait complete. 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.327+0000 7fe90d748640 1 Processor -- start 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.327+0000 7fe90d748640 1 -- start start 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe90d748640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b2e90 0x7fe908145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe90d748640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9080b3fd0 0x7fe9081444e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe90d748640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe90815a490 0x7fe90815ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe90d748640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9080cf260 con 0x7fe9080b2e90 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe90d748640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9080cf0e0 con 0x7fe90815a490 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe90d748640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9080b32c0 con 0x7fe9080b3fd0 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b2e90 0x7fe908145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b2e90 0x7fe908145ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55044/0 (socket says 172.21.15.72:55044) 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 -- 172.21.15.72:0/1877075563 learned_addr learned my addr 172.21.15.72:0/1877075563 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe9067fc640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9080b3fd0 0x7fe9081444e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe9077fe640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe90815a490 0x7fe90815ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 -- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9080b3fd0 msgr2=0x7fe9081444e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9080b3fd0 0x7fe9081444e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 -- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe90815a490 msgr2=0x7fe90815ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe90815a490 0x7fe90815ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.328+0000 7fe906ffd640 1 -- 172.21.15.72:0/1877075563 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908144e30 con 0x7fe9080b2e90 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe906ffd640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b2e90 0x7fe908145ed0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8007e40 tx=0x7fe8f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8014620 con 0x7fe9080b2e90 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f8019070 con 0x7fe9080b2e90 2024-03-06T07:32:35.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908145110 con 0x7fe9080b2e90 2024-03-06T07:32:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f8014a00 con 0x7fe9080b2e90 2024-03-06T07:32:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9081459b0 con 0x7fe9080b2e90 2024-03-06T07:32:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.329+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe908145500 con 0x7fe9080b2e90 2024-03-06T07:32:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.330+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe8f8014ba0 con 0x7fe9080b2e90 2024-03-06T07:32:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.330+0000 7fe8ef7fe640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8f406b4d0 0x7fe8f406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe9067fc640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8f406b4d0 0x7fe8f406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe8f808a750 con 0x7fe9080b2e90 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe8ef7fe640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7fe8f4071430 0x7fe8f4073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8f8022ec0 con 0x7fe8f4071430 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8f8091050 con 0x7fe9080b2e90 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe9067fc640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8f406b4d0 0x7fe8f406d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc036000 tx=0x7fe8fc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe9077fe640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7fe8f4071430 0x7fe8f4073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.331+0000 7fe9077fe640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7fe8f4071430 0x7fe8f4073850 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.332+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== osd.5 v2:172.21.15.146:6800/2545401618 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe8f8022ec0 con 0x7fe8f4071430 2024-03-06T07:32:35.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.345+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 --> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe90804f330 con 0x7fe8f4071430 2024-03-06T07:32:35.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe8ef7fe640 1 -- 172.21.15.72:0/1877075563 <== osd.5 v2:172.21.15.146:6800/2545401618 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe90804f330 con 0x7fe8f4071430 2024-03-06T07:32:35.348 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:35.348 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:35.348 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:35.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7fe8f4071430 msgr2=0x7fe8f4073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.146:6800/2545401618,v1:172.21.15.146:6801/2545401618] conn(0x7fe8f4071430 0x7fe8f4073850 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8f406b4d0 msgr2=0x7fe8f406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe8f406b4d0 0x7fe8f406d990 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc036000 tx=0x7fe8fc03c040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b2e90 msgr2=0x7fe908145ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 --2- 172.21.15.72:0/1877075563 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe9080b2e90 0x7fe908145ed0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8007e40 tx=0x7fe8f800e4b0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe9077fe640 1 -- 172.21.15.72:0/1877075563 reap_dead start 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 shutdown_connections 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 >> 172.21.15.72:0/1877075563 conn(0x7fe9080b1650 msgr2=0x7fe9080708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 shutdown_connections 2024-03-06T07:32:35.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.346+0000 7fe90d748640 1 -- 172.21.15.72:0/1877075563 wait complete. 2024-03-06T07:32:35.358 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-06T07:32:35.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.457+0000 7fe6b0288640 1 Processor -- start 2024-03-06T07:32:35.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.457+0000 7fe6b0288640 1 -- start start 2024-03-06T07:32:35.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6b0288640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6a8162970 0x7fe6a80d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6b0288640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe6a8161cf0 0x7fe6a8162180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6b0288640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a80d22c0 con 0x7fe6a80b4460 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6b0288640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a8161b00 con 0x7fe6a8162970 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6b0288640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a80cecc0 con 0x7fe6a8161cf0 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6ae024640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe6a80b4460 0x7fe6a80b3580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53170/0 (socket says 172.21.15.72:53170) 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6ae024640 1 -- 172.21.15.72:0/2759390715 learned_addr learned my addr 172.21.15.72:0/2759390715 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.458+0000 7fe6ad823640 1 --2- 172.21.15.72:0/2759390715 >> v2:172.21.15.72:3300/0 conn(0x7fe6a8162970 0x7fe6a80d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad022640 1 -- 172.21.15.72:0/2759390715 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338976328 0 0) 0x7fe6a80d22c0 con 0x7fe6a80b4460 2024-03-06T07:32:35.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad022640 1 -- 172.21.15.72:0/2759390715 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe694003520 con 0x7fe6a80b4460 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ae825640 1 --2- 172.21.15.72:0/2759390715 >> v2:172.21.15.146:3300/0 conn(0x7fe6a8161cf0 0x7fe6a8162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad823640 1 -- 172.21.15.72:0/2759390715 >> v2:172.21.15.146:3300/0 conn(0x7fe6a8161cf0 msgr2=0x7fe6a8162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad823640 1 --2- 172.21.15.72:0/2759390715 >> v2:172.21.15.146:3300/0 conn(0x7fe6a8161cf0 0x7fe6a8162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad823640 1 -- 172.21.15.72:0/2759390715 >> v1:172.21.15.72:6789/0 conn(0x7fe6a80b4460 legacy=0x7fe6a80b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad823640 1 -- 172.21.15.72:0/2759390715 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a80d25a0 con 0x7fe6a8162970 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ae825640 1 --2- 172.21.15.72:0/2759390715 >> v2:172.21.15.146:3300/0 conn(0x7fe6a8161cf0 0x7fe6a8162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad823640 1 --2- 172.21.15.72:0/2759390715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8162970 0x7fe6a80d4800 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fe6a808c900 tx=0x7fe69802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fea40b858d5e3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.459+0000 7fe6ad022640 1 -- 172.21.15.72:0/2759390715 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698035020 con 0x7fe6a8162970 2024-03-06T07:32:35.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6ad022640 1 -- 172.21.15.72:0/2759390715 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe698031830 con 0x7fe6a8162970 2024-03-06T07:32:35.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6ad022640 1 -- 172.21.15.72:0/2759390715 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698031be0 con 0x7fe6a8162970 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 -- 172.21.15.72:0/2759390715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8162970 msgr2=0x7fe6a80d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 --2- 172.21.15.72:0/2759390715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8162970 0x7fe6a80d4800 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fe6a808c900 tx=0x7fe69802b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 -- 172.21.15.72:0/2759390715 shutdown_connections 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 --2- 172.21.15.72:0/2759390715 >> v2:172.21.15.146:3300/0 conn(0x7fe6a8161cf0 0x7fe6a8162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 --2- 172.21.15.72:0/2759390715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8162970 0x7fe6a80d4800 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 -- 172.21.15.72:0/2759390715 >> 172.21.15.72:0/2759390715 conn(0x7fe6a80d3ce0 msgr2=0x7fe6a80d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 -- 172.21.15.72:0/2759390715 shutdown_connections 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.460+0000 7fe6b0288640 1 -- 172.21.15.72:0/2759390715 wait complete. 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.461+0000 7fe6b0288640 1 Processor -- start 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.461+0000 7fe6b0288640 1 -- start start 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6b0288640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6a80b4460 0x7fe6a80b87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6b0288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 0x7fe6a80c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6b0288640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6a8162970 0x7fe6a80c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6b0288640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a80ce950 con 0x7fe6a8161cf0 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6b0288640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a80d2b60 con 0x7fe6a8162970 2024-03-06T07:32:35.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6b0288640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a80b4840 con 0x7fe6a80b4460 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 0x7fe6a80c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 0x7fe6a80c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55066/0 (socket says 172.21.15.72:55066) 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 -- 172.21.15.72:0/949122741 learned_addr learned my addr 172.21.15.72:0/949122741 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 -- 172.21.15.72:0/949122741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6a80b4460 msgr2=0x7fe6a80b87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6a80b4460 0x7fe6a80b87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 -- 172.21.15.72:0/949122741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6a8162970 msgr2=0x7fe6a80c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6a8162970 0x7fe6a80c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.462+0000 7fe6ad823640 1 -- 172.21.15.72:0/949122741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a80b7730 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.463+0000 7fe6ad823640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 0x7fe6a80c7810 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fe698000c00 tx=0x7fe6980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.463+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698041070 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.463+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69803b860 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.463+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a80b7a10 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.463+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe698035040 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.463+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a80b7fd0 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.464+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a80b37d0 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.464+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fe698038020 con 0x7fe6a8161cf0 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.464+0000 7fe69effd640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe68c06b4d0 0x7fe68c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.465+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe6980ab7d0 con 0x7fe6a8161cf0 2024-03-06T07:32:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.465+0000 7fe6ae024640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe68c06b4d0 0x7fe68c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.465+0000 7fe6ae024640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe68c06b4d0 0x7fe68c06d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40012e0 tx=0x7fe6a40015b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.469+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fe698031800 con 0x7fe6a8161cf0 2024-03-06T07:32:35.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.641+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6a804f330 con 0x7fe68c06b4d0 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.642+0000 7fe69effd640 1 -- 172.21.15.72:0/949122741 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe6a804f330 con 0x7fe68c06b4d0 2024-03-06T07:32:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 5 to deep-scrub 2024-03-06T07:32:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe68c06b4d0 msgr2=0x7fe68c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe68c06b4d0 0x7fe68c06d990 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40012e0 tx=0x7fe6a40015b0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 msgr2=0x7fe6a80c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 0x7fe6a80c7810 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fe698000c00 tx=0x7fe6980046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 shutdown_connections 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fe68c06b4d0 0x7fe68c06d990 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6a8162970 0x7fe6a80c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a8161cf0 0x7fe6a80c7810 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 --2- 172.21.15.72:0/949122741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6a80b4460 0x7fe6a80b87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 >> 172.21.15.72:0/949122741 conn(0x7fe6a80d3ce0 msgr2=0x7fe6a815fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 shutdown_connections 2024-03-06T07:32:35.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.644+0000 7fe6b0288640 1 -- 172.21.15.72:0/949122741 wait complete. 2024-03-06T07:32:35.657 INFO:tasks.ceph:Scrubbing osd.6 2024-03-06T07:32:35.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:35.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.760+0000 7f305e1bc640 1 Processor -- start 2024-03-06T07:32:35.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.760+0000 7f305e1bc640 1 -- start start 2024-03-06T07:32:35.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.760+0000 7f305e1bc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f305815e180 0x7f30580cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f305e1bc640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f30580cfe00 0x7f30580ce4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f305e1bc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580d0250 con 0x7f305815f290 2024-03-06T07:32:35.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f305e1bc640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305815f7c0 con 0x7f305815e180 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f305e1bc640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580ccc60 con 0x7f30580cfe00 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f30577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f305815e180 0x7f30580cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f3056ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f305815f290 0x7f30580cf6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53178/0 (socket says 172.21.15.72:53178) 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f3056ffd640 1 -- 172.21.15.72:0/4096276493 learned_addr learned my addr 172.21.15.72:0/4096276493 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f3057fff640 1 --2- 172.21.15.72:0/4096276493 >> v2:172.21.15.146:3300/0 conn(0x7f30580cfe00 0x7f30580ce4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f30577fe640 1 -- 172.21.15.72:0/4096276493 >> v2:172.21.15.146:3300/0 conn(0x7f30580cfe00 msgr2=0x7f30580ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f30577fe640 1 --2- 172.21.15.72:0/4096276493 >> v2:172.21.15.146:3300/0 conn(0x7f30580cfe00 0x7f30580ce4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f30577fe640 1 -- 172.21.15.72:0/4096276493 >> v1:172.21.15.72:6789/0 conn(0x7f305815f290 legacy=0x7f30580cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.761+0000 7f30577fe640 1 -- 172.21.15.72:0/4096276493 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580ccf40 con 0x7f305815e180 2024-03-06T07:32:35.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f30577fe640 1 --2- 172.21.15.72:0/4096276493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815e180 0x7f30580cd740 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f3058074bf0 tx=0x7f304802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f516af91746a857 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f30567fc640 1 -- 172.21.15.72:0/4096276493 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480315a0 con 0x7f305815e180 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f30567fc640 1 -- 172.21.15.72:0/4096276493 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3048037070 con 0x7f305815e180 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f30567fc640 1 -- 172.21.15.72:0/4096276493 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048031960 con 0x7f305815e180 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 -- 172.21.15.72:0/4096276493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815e180 msgr2=0x7f30580cd740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 --2- 172.21.15.72:0/4096276493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815e180 0x7f30580cd740 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f3058074bf0 tx=0x7f304802b840 comp rx=0 tx=0).stop 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 -- 172.21.15.72:0/4096276493 shutdown_connections 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 --2- 172.21.15.72:0/4096276493 >> v2:172.21.15.146:3300/0 conn(0x7f30580cfe00 0x7f30580ce4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 --2- 172.21.15.72:0/4096276493 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815e180 0x7f30580cd740 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 -- 172.21.15.72:0/4096276493 >> 172.21.15.72:0/4096276493 conn(0x7f30580d1b20 msgr2=0x7f30580d1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 -- 172.21.15.72:0/4096276493 shutdown_connections 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.762+0000 7f305e1bc640 1 -- 172.21.15.72:0/4096276493 wait complete. 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 Processor -- start 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 -- start start 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30580cfe00 0x7f305814aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305815e180 0x7f305814af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815f290 0x7f30581f75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580cc910 con 0x7f305815f290 2024-03-06T07:32:35.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580cdc80 con 0x7f30580cfe00 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.764+0000 7f305e1bc640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30580b2ee0 con 0x7f305815e180 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815f290 0x7f30581f75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815f290 0x7f30581f75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55096/0 (socket says 172.21.15.72:55096) 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 -- 172.21.15.72:0/2851306674 learned_addr learned my addr 172.21.15.72:0/2851306674 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f30577fe640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30580cfe00 0x7f305814aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3056ffd640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305815e180 0x7f305814af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 -- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305815e180 msgr2=0x7f305814af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305815e180 0x7f305814af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 -- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30580cfe00 msgr2=0x7f305814aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30580cfe00 0x7f305814aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 -- 172.21.15.72:0/2851306674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30581e8fe0 con 0x7f305815f290 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3056ffd640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f305815e180 0x7f305814af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f3057fff640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815f290 0x7f30581f75e0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f305000cc30 tx=0x7f305000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000ce40 con 0x7f305815f290 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3050003990 con 0x7f305815f290 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050003d40 con 0x7f305815f290 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30581e92c0 con 0x7f305815f290 2024-03-06T07:32:35.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.765+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3058148c50 con 0x7f305815f290 2024-03-06T07:32:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.766+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f305804f330 con 0x7f305815f290 2024-03-06T07:32:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f3050003ee0 con 0x7f305815f290 2024-03-06T07:32:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f303ffff640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304406b4d0 0x7f304406d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f305008a340 con 0x7f305815f290 2024-03-06T07:32:35.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f30577fe640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304406b4d0 0x7f304406d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f303ffff640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f3044071430 0x7f3044073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f305000de50 con 0x7f3044071430 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3050091050 con 0x7f305815f290 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f3056ffd640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f3044071430 0x7f3044073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.767+0000 7f30577fe640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304406b4d0 0x7f304406d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3048036000 tx=0x7f304803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.768+0000 7f3056ffd640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f3044071430 0x7f3044073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.769+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== osd.6 v2:172.21.15.146:6808/1113968375 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f305000de50 con 0x7f3044071430 2024-03-06T07:32:35.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 --> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3058072d80 con 0x7f3044071430 2024-03-06T07:32:35.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f303ffff640 1 -- 172.21.15.72:0/2851306674 <== osd.6 v2:172.21.15.146:6808/1113968375 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3058072d80 con 0x7f3044071430 2024-03-06T07:32:35.784 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:35.784 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:35.785 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:35.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f3044071430 msgr2=0x7f3044073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.146:6808/1113968375,v1:172.21.15.146:6809/1113968375] conn(0x7f3044071430 0x7f3044073850 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304406b4d0 msgr2=0x7f304406d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f304406b4d0 0x7f304406d990 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3048036000 tx=0x7f304803d040 comp rx=0 tx=0).stop 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815f290 msgr2=0x7f30581f75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 --2- 172.21.15.72:0/2851306674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f305815f290 0x7f30581f75e0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f305000cc30 tx=0x7f305000a5c0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f3057fff640 1 -- 172.21.15.72:0/2851306674 reap_dead start 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 shutdown_connections 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.782+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 >> 172.21.15.72:0/2851306674 conn(0x7f30580d1b20 msgr2=0x7f305815bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.783+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 shutdown_connections 2024-03-06T07:32:35.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.783+0000 7f305e1bc640 1 -- 172.21.15.72:0/2851306674 wait complete. 2024-03-06T07:32:35.795 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-06T07:32:35.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.894+0000 7f97b1c9d640 1 Processor -- start 2024-03-06T07:32:35.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.894+0000 7f97b1c9d640 1 -- start start 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97b1c9d640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f97ac161670 0x7f97ac072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97b1c9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f97ac0b0ea0 0x7f97ac072ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97b1c9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ac066d00 con 0x7f97ac072a60 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97b1c9d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ac0b6170 con 0x7f97ac0b0ea0 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97b1c9d640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ac0ada80 con 0x7f97ac161670 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97abfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f97ac0b0ea0 0x7f97ac072ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97aaffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f97ac072a60 0x7f97ac0adf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53186/0 (socket says 172.21.15.72:53186) 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97aaffd640 1 -- 172.21.15.72:0/68383422 learned_addr learned my addr 172.21.15.72:0/68383422 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97abfff640 1 -- 172.21.15.72:0/68383422 >> v2:172.21.15.146:3300/0 conn(0x7f97ac161670 msgr2=0x7f97ac072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97abfff640 1 --2- 172.21.15.72:0/68383422 >> v2:172.21.15.146:3300/0 conn(0x7f97ac161670 0x7f97ac072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97abfff640 1 -- 172.21.15.72:0/68383422 >> v1:172.21.15.72:6789/0 conn(0x7f97ac072a60 legacy=0x7f97ac0adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97aa7fc640 1 -- 172.21.15.72:0/68383422 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727458941 0 0) 0x7f97ac066d00 con 0x7f97ac072a60 2024-03-06T07:32:35.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.895+0000 7f97abfff640 1 -- 172.21.15.72:0/68383422 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac066fc0 con 0x7f97ac0b0ea0 2024-03-06T07:32:35.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.896+0000 7f97abfff640 1 --2- 172.21.15.72:0/68383422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac072ef0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f97ac070a80 tx=0x7f97a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63e15d553210648 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.896+0000 7f97aa7fc640 1 -- 172.21.15.72:0/68383422 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4039070 con 0x7f97ac0b0ea0 2024-03-06T07:32:35.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.896+0000 7f97aa7fc640 1 -- 172.21.15.72:0/68383422 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a402cc80 con 0x7f97ac0b0ea0 2024-03-06T07:32:35.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.896+0000 7f97aa7fc640 1 -- 172.21.15.72:0/68383422 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4034660 con 0x7f97ac0b0ea0 2024-03-06T07:32:35.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 -- 172.21.15.72:0/68383422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac0b0ea0 msgr2=0x7f97ac072ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/68383422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac072ef0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f97ac070a80 tx=0x7f97a402bed0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 -- 172.21.15.72:0/68383422 shutdown_connections 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/68383422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac072ef0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/68383422 >> v2:172.21.15.146:3300/0 conn(0x7f97ac161670 0x7f97ac072520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 -- 172.21.15.72:0/68383422 >> 172.21.15.72:0/68383422 conn(0x7f97ac0b4e60 msgr2=0x7f97ac0b5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 -- 172.21.15.72:0/68383422 shutdown_connections 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.897+0000 7f97b1c9d640 1 -- 172.21.15.72:0/68383422 wait complete. 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.898+0000 7f97b1c9d640 1 Processor -- start 2024-03-06T07:32:35.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.898+0000 7f97b1c9d640 1 -- start start 2024-03-06T07:32:35.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97b1c9d640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97ac072a60 0x7f97ac1f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97b1c9d640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97b1c9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 0x7f97ac0b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97b1c9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac06b330 con 0x7f97ac161670 2024-03-06T07:32:35.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97b1c9d640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac161a50 con 0x7f97ac0b0ea0 2024-03-06T07:32:35.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97b1c9d640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ac06b610 con 0x7f97ac072a60 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 0x7f97ac0b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 0x7f97ac0b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55116/0 (socket says 172.21.15.72:55116) 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 -- 172.21.15.72:0/2318159191 learned_addr learned my addr 172.21.15.72:0/2318159191 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97aaffd640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97ab7fe640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97ac072a60 0x7f97ac1f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 -- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97ac072a60 msgr2=0x7f97ac1f7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97ac072a60 0x7f97ac1f7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 -- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97ac0b0ea0 msgr2=0x7f97ac148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 -- 172.21.15.72:0/2318159191 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ac148880 con 0x7f97ac161670 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.899+0000 7f97abfff640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 0x7f97ac0b1880 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f97a402c4e0 tx=0x7f97a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.901+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4044020 con 0x7f97ac161670 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.901+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a403f470 con 0x7f97ac161670 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.901+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ac1f6ea0 con 0x7f97ac161670 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.901+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a4039040 con 0x7f97ac161670 2024-03-06T07:32:35.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.901+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ac155d50 con 0x7f97ac161670 2024-03-06T07:32:35.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.902+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ac04f330 con 0x7f97ac161670 2024-03-06T07:32:35.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.902+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f97a4047020 con 0x7f97ac161670 2024-03-06T07:32:35.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.902+0000 7f9793fff640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f979806b500 0x7f979806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:35.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.903+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f97a40ac750 con 0x7f97ac161670 2024-03-06T07:32:35.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.903+0000 7f97ab7fe640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f979806b500 0x7f979806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:35.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.903+0000 7f97ab7fe640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f979806b500 0x7f979806d9c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f979c0098e0 tx=0x7f979c009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:35.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:35.907+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f97a404e070 con 0x7f97ac161670 2024-03-06T07:32:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.081+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f97ac155760 con 0x7f979806b500 2024-03-06T07:32:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.081+0000 7f9793fff640 1 -- 172.21.15.72:0/2318159191 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f97ac155760 con 0x7f979806b500 2024-03-06T07:32:36.083 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 6 to deep-scrub 2024-03-06T07:32:36.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f979806b500 msgr2=0x7f979806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f979806b500 0x7f979806d9c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f979c0098e0 tx=0x7f979c009060 comp rx=0 tx=0).stop 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 msgr2=0x7f97ac0b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 0x7f97ac0b1880 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f97a402c4e0 tx=0x7f97a4004a40 comp rx=0 tx=0).stop 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 shutdown_connections 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f979806b500 0x7f979806d9c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97ac161670 0x7f97ac0b1880 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f97ac0b0ea0 0x7f97ac148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 --2- 172.21.15.72:0/2318159191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f97ac072a60 0x7f97ac1f7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 >> 172.21.15.72:0/2318159191 conn(0x7f97ac0b4e60 msgr2=0x7f97ac0cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 shutdown_connections 2024-03-06T07:32:36.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.083+0000 7f97b1c9d640 1 -- 172.21.15.72:0/2318159191 wait complete. 2024-03-06T07:32:36.096 INFO:tasks.ceph:Scrubbing osd.7 2024-03-06T07:32:36.097 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-06T07:32:36.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.198+0000 7f4956df6640 1 Processor -- start 2024-03-06T07:32:36.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.198+0000 7f4956df6640 1 -- start start 2024-03-06T07:32:36.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4956df6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f495015e050 0x7f49500cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4956df6640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49500ce170 0x7f49500cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4956df6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f495006afb0 0x7f495006b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4956df6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495015e430 con 0x7f495015e050 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4956df6640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49500cd440 con 0x7f49500ce170 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4956df6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49500cdce0 con 0x7f495006afb0 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f495015e050 0x7f49500cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f495015e050 0x7f49500cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55118/0 (socket says 172.21.15.72:55118) 2024-03-06T07:32:36.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 -- 172.21.15.72:0/1971955748 learned_addr learned my addr 172.21.15.72:0/1971955748 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:36.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 -- 172.21.15.72:0/1971955748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f495006afb0 msgr2=0x7f495006b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 --2- 172.21.15.72:0/1971955748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f495006afb0 0x7f495006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 -- 172.21.15.72:0/1971955748 >> v2:172.21.15.146:3300/0 conn(0x7f49500ce170 msgr2=0x7f49500cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 --2- 172.21.15.72:0/1971955748 >> v2:172.21.15.146:3300/0 conn(0x7f49500ce170 0x7f49500cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.199+0000 7f4954b92640 1 -- 172.21.15.72:0/1971955748 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49500d1860 con 0x7f495015e050 2024-03-06T07:32:36.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.200+0000 7f4954b92640 1 --2- 172.21.15.72:0/1971955748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49500cf0b0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f4950067d60 tx=0x7f494002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289054ec152dfd20 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.200+0000 7f494f7fe640 1 -- 172.21.15.72:0/1971955748 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940035020 con 0x7f495015e050 2024-03-06T07:32:36.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.200+0000 7f494f7fe640 1 -- 172.21.15.72:0/1971955748 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4940031870 con 0x7f495015e050 2024-03-06T07:32:36.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.200+0000 7f494f7fe640 1 -- 172.21.15.72:0/1971955748 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4940031c40 con 0x7f495015e050 2024-03-06T07:32:36.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 -- 172.21.15.72:0/1971955748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 msgr2=0x7f49500cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 --2- 172.21.15.72:0/1971955748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49500cf0b0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f4950067d60 tx=0x7f494002b910 comp rx=0 tx=0).stop 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 -- 172.21.15.72:0/1971955748 shutdown_connections 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 --2- 172.21.15.72:0/1971955748 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f495006afb0 0x7f495006b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 --2- 172.21.15.72:0/1971955748 >> v2:172.21.15.146:3300/0 conn(0x7f49500ce170 0x7f49500cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 --2- 172.21.15.72:0/1971955748 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49500cf0b0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 -- 172.21.15.72:0/1971955748 >> 172.21.15.72:0/1971955748 conn(0x7f49500d0a30 msgr2=0x7f49500d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 -- 172.21.15.72:0/1971955748 shutdown_connections 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.201+0000 7f4956df6640 1 -- 172.21.15.72:0/1971955748 wait complete. 2024-03-06T07:32:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.202+0000 7f4956df6640 1 Processor -- start 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.202+0000 7f4956df6640 1 -- start start 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4956df6640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f495006afb0 0x7f4950160a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4956df6640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49500ce170 0x7f4950155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4956df6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49501585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4956df6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49500cd970 con 0x7f495015e050 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4956df6640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49500cd690 con 0x7f495006afb0 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4956df6640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49500719a0 con 0x7f49500ce170 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49501585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49501585f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55120/0 (socket says 172.21.15.72:55120) 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 -- 172.21.15.72:0/2792948797 learned_addr learned my addr 172.21.15.72:0/2792948797 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4954b92640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f495006afb0 0x7f4950160a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f494ffff640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49500ce170 0x7f4950155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 -- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49500ce170 msgr2=0x7f4950155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49500ce170 0x7f4950155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 -- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f495006afb0 msgr2=0x7f4950160a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f495006afb0 0x7f4950160a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 -- 172.21.15.72:0/2792948797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49501590d0 con 0x7f495015e050 2024-03-06T07:32:36.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.203+0000 7f4955393640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49501585f0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f494800b970 tx=0x7f494800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.204+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948019070 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.204+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494800cac0 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.204+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49501593b0 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.204+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4948014440 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.204+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495015ea50 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.205+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f495015a240 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.205+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f494801c070 con 0x7f495015e050 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494d7fa640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f493c06b4d0 0x7f493c06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4948089b60 con 0x7f495015e050 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f4954b92640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f493c06b4d0 0x7f493c06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494d7fa640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f493c071430 0x7f493c073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f493c073f00 con 0x7f493c071430 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f494808e2c0 con 0x7f495015e050 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494ffff640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f493c071430 0x7f493c073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f4954b92640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f493c06b4d0 0x7f493c06d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4940036000 tx=0x7f494003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.206+0000 7f494ffff640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f493c071430 0x7f493c073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.207+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== osd.7 v2:172.21.15.146:6816/1306818419 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f493c073f00 con 0x7f493c071430 2024-03-06T07:32:36.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.220+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 --> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f495004f330 con 0x7f493c071430 2024-03-06T07:32:36.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f494d7fa640 1 -- 172.21.15.72:0/2792948797 <== osd.7 v2:172.21.15.146:6816/1306818419 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f495004f330 con 0x7f493c071430 2024-03-06T07:32:36.223 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-06T07:32:36.223 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-06T07:32:36.223 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-06T07:32:36.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f493c071430 msgr2=0x7f493c073850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.146:6816/1306818419,v1:172.21.15.146:6817/1306818419] conn(0x7f493c071430 0x7f493c073850 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f493c06b4d0 msgr2=0x7f493c06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f493c06b4d0 0x7f493c06d990 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f4940036000 tx=0x7f494003c040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 msgr2=0x7f49501585f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 --2- 172.21.15.72:0/2792948797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f495015e050 0x7f49501585f0 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f494800b970 tx=0x7f494800bd70 comp rx=0 tx=0).stop 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4955393640 1 -- 172.21.15.72:0/2792948797 reap_dead start 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 shutdown_connections 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 >> 172.21.15.72:0/2792948797 conn(0x7f49500d0a30 msgr2=0x7f49500b06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 shutdown_connections 2024-03-06T07:32:36.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.221+0000 7f4956df6640 1 -- 172.21.15.72:0/2792948797 wait complete. 2024-03-06T07:32:36.234 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.331+0000 7f9cac6b5640 1 Processor -- start 2024-03-06T07:32:36.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.331+0000 7f9cac6b5640 1 -- start start 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9cac6b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9ca4162970 0x7f9ca40d4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9cac6b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40d22c0 con 0x7f9ca4161cf0 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9cac6b5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca40d25a0 con 0x7f9ca40b4460 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9cac6b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca4161b00 con 0x7f9ca4162970 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9ca9c50640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9ca4162970 0x7f9ca40d4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9caac52640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9ca4161cf0 0x7f9ca40cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53196/0 (socket says 172.21.15.72:53196) 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.332+0000 7f9caac52640 1 -- 172.21.15.72:0/2197631735 learned_addr learned my addr 172.21.15.72:0/2197631735 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:36.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.333+0000 7f9ca9c50640 1 -- 172.21.15.72:0/2197631735 >> v1:172.21.15.146:6789/0 conn(0x7f9ca40b4460 legacy=0x7f9ca40b3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.333+0000 7f9ca9c50640 1 -- 172.21.15.72:0/2197631735 >> v1:172.21.15.72:6789/0 conn(0x7f9ca4161cf0 legacy=0x7f9ca40cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.333+0000 7f9ca944f640 1 -- 172.21.15.72:0/2197631735 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120956311 0 0) 0x7f9ca40d22c0 con 0x7f9ca4161cf0 2024-03-06T07:32:36.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.333+0000 7f9ca9c50640 1 -- 172.21.15.72:0/2197631735 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca40d2880 con 0x7f9ca4162970 2024-03-06T07:32:36.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.333+0000 7f9ca9c50640 1 --2- 172.21.15.72:0/2197631735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4162970 0x7f9ca40d4800 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f9c940097a0 tx=0x7f9c9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cff8c5526b0bdcd server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9ca944f640 1 -- 172.21.15.72:0/2197631735 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94035020 con 0x7f9ca4162970 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9ca944f640 1 -- 172.21.15.72:0/2197631735 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c94031830 con 0x7f9ca4162970 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9ca944f640 1 -- 172.21.15.72:0/2197631735 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c94031c00 con 0x7f9ca4162970 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 -- 172.21.15.72:0/2197631735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4162970 msgr2=0x7f9ca40d4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/2197631735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4162970 0x7f9ca40d4800 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7f9c940097a0 tx=0x7f9c9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 -- 172.21.15.72:0/2197631735 shutdown_connections 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/2197631735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4162970 0x7f9ca40d4800 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 -- 172.21.15.72:0/2197631735 >> 172.21.15.72:0/2197631735 conn(0x7f9ca40d3ce0 msgr2=0x7f9ca40d40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 -- 172.21.15.72:0/2197631735 shutdown_connections 2024-03-06T07:32:36.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.334+0000 7f9cac6b5640 1 -- 172.21.15.72:0/2197631735 wait complete. 2024-03-06T07:32:36.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.335+0000 7f9cac6b5640 1 Processor -- start 2024-03-06T07:32:36.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.335+0000 7f9cac6b5640 1 -- start start 2024-03-06T07:32:36.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9cac6b5640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca40b4460 0x7f9ca40c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9cac6b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 0x7f9ca40c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9cac6b5640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca4162970 0x7f9ca41565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9cac6b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40ce950 con 0x7f9ca4161cf0 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9cac6b5640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40cecc0 con 0x7f9ca4162970 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9cac6b5640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca40b4840 con 0x7f9ca40b4460 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 0x7f9ca40c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 0x7f9ca40c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55148/0 (socket says 172.21.15.72:55148) 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 -- 172.21.15.72:0/3987071941 learned_addr learned my addr 172.21.15.72:0/3987071941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 -- 172.21.15.72:0/3987071941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca40b4460 msgr2=0x7f9ca40c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca40b4460 0x7f9ca40c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 -- 172.21.15.72:0/3987071941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca4162970 msgr2=0x7f9ca41565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca4162970 0x7f9ca41565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.336+0000 7f9ca9c50640 1 -- 172.21.15.72:0/3987071941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca4156b00 con 0x7f9ca4161cf0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.337+0000 7f9ca9c50640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 0x7f9ca40c7810 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f9c940098c0 tx=0x7f9c940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.337+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9403ba80 con 0x7f9ca4161cf0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.337+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9403bc20 con 0x7f9ca4161cf0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.337+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca40b76d0 con 0x7f9ca4161cf0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.337+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c940453f0 con 0x7f9ca4161cf0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.337+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca40c8420 con 0x7f9ca4161cf0 2024-03-06T07:32:36.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.338+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca40b9be0 con 0x7f9ca4161cf0 2024-03-06T07:32:36.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.338+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f9c94038020 con 0x7f9ca4161cf0 2024-03-06T07:32:36.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.339+0000 7f9c9affd640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.339+0000 7f9caa451640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.339+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9c940abd80 con 0x7f9ca4161cf0 2024-03-06T07:32:36.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.339+0000 7f9caa451640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000ab90 tx=0x7f9ca0009230 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.343+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f9ca40b9be0 con 0x7f9ca4161cf0 2024-03-06T07:32:36.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.514+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ca404f330 con 0x7f9c8806b4d0 2024-03-06T07:32:36.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.515+0000 7f9c9affd640 1 -- 172.21.15.72:0/3987071941 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9ca404f330 con 0x7f9c8806b4d0 2024-03-06T07:32:36.517 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 7 to deep-scrub 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c8806b4d0 msgr2=0x7f9c8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c8806b4d0 0x7f9c8806d990 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000ab90 tx=0x7f9ca0009230 comp rx=0 tx=0).stop 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 msgr2=0x7f9ca40c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 0x7f9ca40c7810 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f9c940098c0 tx=0x7f9c940046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 shutdown_connections 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f9c8806b4d0 0x7f9c8806d990 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ca4162970 0x7f9ca41565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ca4161cf0 0x7f9ca40c7810 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 --2- 172.21.15.72:0/3987071941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ca40b4460 0x7f9ca40c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 >> 172.21.15.72:0/3987071941 conn(0x7f9ca40d3ce0 msgr2=0x7f9ca40d0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 shutdown_connections 2024-03-06T07:32:36.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.517+0000 7f9cac6b5640 1 -- 172.21.15.72:0/3987071941 wait complete. 2024-03-06T07:32:36.530 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:36.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.627+0000 7f7e1fdf7640 1 Processor -- start 2024-03-06T07:32:36.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.627+0000 7f7e1fdf7640 1 -- start start 2024-03-06T07:32:36.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1fdf7640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e181424f0 0x7f7e180cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1fdf7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 0x7f7e1815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1fdf7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e180cc980 con 0x7f7e1806d330 2024-03-06T07:32:36.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1fdf7640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180cd220 con 0x7f7e181424f0 2024-03-06T07:32:36.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1fdf7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180ccc60 con 0x7f7e180cd7a0 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1e394640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e1806d330 0x7f7e18071b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53206/0 (socket says 172.21.15.72:53206) 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1e394640 1 -- 172.21.15.72:0/3670764280 learned_addr learned my addr 172.21.15.72:0/3670764280 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3670764280 >> v2:172.21.15.146:3300/0 conn(0x7f7e181424f0 0x7f7e180cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1d392640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 0x7f7e1815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582822930 0 0) 0x7f7e180cc980 con 0x7f7e1806d330 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.628+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e04003520 con 0x7f7e1806d330 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15270109 0 0) 0x7f7e04003520 con 0x7f7e1806d330 2024-03-06T07:32:36.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 msgr2=0x7f7e1815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 0x7f7e1815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 >> v2:172.21.15.146:3300/0 conn(0x7f7e181424f0 msgr2=0x7f7e180cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 --2- 172.21.15.72:0/3670764280 >> v2:172.21.15.146:3300/0 conn(0x7f7e181424f0 0x7f7e180cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e180ccf40 con 0x7f7e1806d330 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3389178991 0 0) 0x7f7e10001670 con 0x7f7e1806d330 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 >> v1:172.21.15.72:6789/0 conn(0x7f7e1806d330 legacy=0x7f7e18071b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 0x7f7e04004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180cd220 con 0x7f7e040039b0 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e180ccc60 con 0x7f7e04004d40 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0400b9c0 con 0x7f7e180cd7a0 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1d392640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 0x7f7e04004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1e394640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1db93640 1 -- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 msgr2=0x7f7e04004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e180cd7a0 0x7f7e04004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1db93640 1 -- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e04004d40 msgr2=0x7f7e040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e04004d40 0x7f7e040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.629+0000 7f7e1db93640 1 -- 172.21.15.72:0/3670764280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04003520 con 0x7f7e040039b0 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e04003dd0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009fc0 tx=0x7f7e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929dc97b9fb8d222 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14039070 con 0x7f7e040039b0 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1402cc60 con 0x7f7e040039b0 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1cb91640 1 -- 172.21.15.72:0/3670764280 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14009c10 con 0x7f7e040039b0 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3670764280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 msgr2=0x7f7e04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3670764280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e04003dd0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009fc0 tx=0x7f7e04005830 comp rx=0 tx=0).stop 2024-03-06T07:32:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1e394640 1 -- 172.21.15.72:0/3670764280 reap_dead start 2024-03-06T07:32:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3670764280 shutdown_connections 2024-03-06T07:32:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3670764280 >> 172.21.15.72:0/3670764280 conn(0x7f7e180cf8a0 msgr2=0x7f7e180cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.630+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3670764280 shutdown_connections 2024-03-06T07:32:36.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.631+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3670764280 wait complete. 2024-03-06T07:32:36.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 Processor -- start 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 -- start start 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e181400a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e1814e920 0x7f7e1813c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e1813f1b0 0x7f7e181f1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1815d070 con 0x7f7e040039b0 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1815cef0 con 0x7f7e1814e920 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1fdf7640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1815d1f0 con 0x7f7e1813f1b0 2024-03-06T07:32:36.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1db93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e181400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1db93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e181400a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55158/0 (socket says 172.21.15.72:55158) 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.632+0000 7f7e1db93640 1 -- 172.21.15.72:0/3255158869 learned_addr learned my addr 172.21.15.72:0/3255158869 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1d392640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e1814e920 0x7f7e1813c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1e394640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e1813f1b0 0x7f7e181f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1db93640 1 -- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e1813f1b0 msgr2=0x7f7e181f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e1813f1b0 0x7f7e181f1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1db93640 1 -- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e1814e920 msgr2=0x7f7e1813c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e1814e920 0x7f7e1813c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1db93640 1 -- 172.21.15.72:0/3255158869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18140920 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1d392640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e1814e920 0x7f7e1813c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1db93640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e181400a0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f7e14037040 tx=0x7f7e14009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14039050 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e14045070 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1814be30 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1403e610 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.633+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e181405e0 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.634+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1804f330 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.635+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f7e1403e7b0 con 0x7f7e040039b0 2024-03-06T07:32:36.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.635+0000 7f7e0e7fc640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.635+0000 7f7e1d392640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.635+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7e140aca70 con 0x7f7e040039b0 2024-03-06T07:32:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.635+0000 7f7e1d392640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7e180cea50 tx=0x7f7e08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:36.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.639+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f7e1407bc20 con 0x7f7e040039b0 2024-03-06T07:32:36.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.808+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e18074580 con 0x7f7dfc06b4d0 2024-03-06T07:32:36.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.809+0000 7f7e0e7fc640 1 -- 172.21.15.72:0/3255158869 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13485 (secure 0 0 0) 0x7f7e18074580 con 0x7f7dfc06b4d0 2024-03-06T07:32:36.812 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:36.813 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:32:36.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dfc06b4d0 msgr2=0x7f7dfc06d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f7e180cea50 tx=0x7f7e08008040 comp rx=0 tx=0).stop 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 msgr2=0x7f7e181400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e181400a0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f7e14037040 tx=0x7f7e14009750 comp rx=0 tx=0).stop 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 shutdown_connections 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f7dfc06b4d0 0x7f7dfc06d990 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e1813f1b0 0x7f7e181f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e1814e920 0x7f7e1813c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 --2- 172.21.15.72:0/3255158869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e040039b0 0x7f7e181400a0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 >> 172.21.15.72:0/3255158869 conn(0x7f7e180cf8a0 msgr2=0x7f7e1806afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 shutdown_connections 2024-03-06T07:32:36.815 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:36.811+0000 7f7e1fdf7640 1 -- 172.21.15.72:0/3255158869 wait complete. 2024-03-06T07:32:36.823 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-03-06T07:32:36.063911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330312,"kb_used_data":14280,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644408,"statfs":{"total":773094113280,"available":772755873792,"internally_reserved":0,"allocated":14622720,"data_stored":7059408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.189135"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":379,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-03-06T07:32:28.883294+0000","last_change":"2024-03-06T07:21:39.964052+0000","last_active":"2024-03-06T07:32:28.883294+0000","last_peered":"2024-03-06T07:32:28.883294+0000","last_clean":"2024-03-06T07:32:28.883294+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:32:28.883294+0000","last_undegraded":"2024-03-06T07:32:28.883294+0000","last_fullsized":"2024-03-06T07:32:28.883294+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-06T07:21:38.927883+0000","last_clean_scrub_stamp":"2024-03-06T07:21:38.927883+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:58:55.814125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574981,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2273280,"data_stored":1325202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41724,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330116,"statfs":{"total":96636764160,"available":96594038784,"internally_reserved":0,"allocated":2273280,"data_stored":1325202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:36.826 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-06T07:21:38.927883+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=21, tm_sec=38, tm_wday=2, tm_yday=66, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=6, tm_hour=7, tm_min=32, tm_sec=32, tm_wday=2, tm_yday=66, tm_isdst=0) 2024-03-06T07:32:36.826 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-06T07:32:52.444 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:52.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:32:56.827 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-06T07:32:56.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.937+0000 7fb4ea102640 1 Processor -- start 2024-03-06T07:32:56.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.937+0000 7fb4ea102640 1 -- start start 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4ea102640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb4e415bf30 0x7fb4e415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4ea102640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e415f580 con 0x7fb4e40b3060 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4ea102640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e415b450 con 0x7fb4e413dc10 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4ea102640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e40cf3f0 con 0x7fb4e415bf30 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4e3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb4e415bf30 0x7fb4e415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4e2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb4e40b3060 0x7fb4e415e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43948/0 (socket says 172.21.15.72:43948) 2024-03-06T07:32:56.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.938+0000 7fb4e2ffd640 1 -- 172.21.15.72:0/216201233 learned_addr learned my addr 172.21.15.72:0/216201233 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:56.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e3fff640 1 -- 172.21.15.72:0/216201233 >> v1:172.21.15.146:6789/0 conn(0x7fb4e413dc10 legacy=0x7fb4e40b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:56.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e3fff640 1 -- 172.21.15.72:0/216201233 >> v1:172.21.15.72:6789/0 conn(0x7fb4e40b3060 legacy=0x7fb4e415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e27fc640 1 -- 172.21.15.72:0/216201233 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521669161 0 0) 0x7fb4e415f580 con 0x7fb4e40b3060 2024-03-06T07:32:56.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e3fff640 1 -- 172.21.15.72:0/216201233 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e415b730 con 0x7fb4e415bf30 2024-03-06T07:32:56.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e3fff640 1 --2- 172.21.15.72:0/216201233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e415bf30 0x7fb4e415ef60 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009d90 tx=0x7fb4dc02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5460106b8c5a828 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e27fc640 1 -- 172.21.15.72:0/216201233 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc039070 con 0x7fb4e415bf30 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e27fc640 1 -- 172.21.15.72:0/216201233 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc02cbd0 con 0x7fb4e415bf30 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.939+0000 7fb4e27fc640 1 -- 172.21.15.72:0/216201233 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc034440 con 0x7fb4e415bf30 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 -- 172.21.15.72:0/216201233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e415bf30 msgr2=0x7fb4e415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 --2- 172.21.15.72:0/216201233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e415bf30 0x7fb4e415ef60 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc009d90 tx=0x7fb4dc02be20 comp rx=0 tx=0).stop 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 -- 172.21.15.72:0/216201233 shutdown_connections 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 --2- 172.21.15.72:0/216201233 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e415bf30 0x7fb4e415ef60 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 -- 172.21.15.72:0/216201233 >> 172.21.15.72:0/216201233 conn(0x7fb4e40cf8a0 msgr2=0x7fb4e40cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:56.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 -- 172.21.15.72:0/216201233 shutdown_connections 2024-03-06T07:32:56.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.940+0000 7fb4ea102640 1 -- 172.21.15.72:0/216201233 wait complete. 2024-03-06T07:32:56.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.941+0000 7fb4ea102640 1 Processor -- start 2024-03-06T07:32:56.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.941+0000 7fb4ea102640 1 -- start start 2024-03-06T07:32:56.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4ea102640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4e40b3060 0x7fb4e4159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4ea102640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 0x7fb4e4140700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4ea102640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4e415bf30 0x7fb4e4140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4ea102640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e40718e0 con 0x7fb4e413dc10 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4ea102640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e415b210 con 0x7fb4e415bf30 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4ea102640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e4071a60 con 0x7fb4e40b3060 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 0x7fb4e4140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 0x7fb4e4140700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46944/0 (socket says 172.21.15.72:46944) 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 -- 172.21.15.72:0/4223409992 learned_addr learned my addr 172.21.15.72:0/4223409992 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 -- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4e40b3060 msgr2=0x7fb4e4159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4e40b3060 0x7fb4e4159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 -- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4e415bf30 msgr2=0x7fb4e4140c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e3fff640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4e415bf30 0x7fb4e4140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.942+0000 7fb4e2ffd640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4e415bf30 0x7fb4e4140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4e2ffd640 1 -- 172.21.15.72:0/4223409992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e4141180 con 0x7fb4e413dc10 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4e2ffd640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 0x7fb4e4140700 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800c4d0 tx=0x7fb4d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8018070 con 0x7fb4e413dc10 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d8007500 con 0x7fb4e413dc10 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d8004a20 con 0x7fb4e413dc10 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e413f2f0 con 0x7fb4e413dc10 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.943+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e413fb90 con 0x7fb4e413dc10 2024-03-06T07:32:56.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.944+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e404f330 con 0x7fb4e413dc10 2024-03-06T07:32:56.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.944+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fb4d801b070 con 0x7fb4e413dc10 2024-03-06T07:32:56.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.945+0000 7fb4cbfff640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb4d006b4d0 0x7fb4d006d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:56.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.945+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb4d8089270 con 0x7fb4e413dc10 2024-03-06T07:32:56.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.945+0000 7fb4e37fe640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb4d006b4d0 0x7fb4d006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:56.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.945+0000 7fb4e37fe640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb4d006b4d0 0x7fb4d006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb4e416adf0 tx=0x7fb4d40079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:56.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:56.950+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fb4d8058390 con 0x7fb4e413dc10 2024-03-06T07:32:57.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.117+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 --> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4e40b1200 con 0x7fb4d006b4d0 2024-03-06T07:32:57.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.118+0000 7fb4cbfff640 1 -- 172.21.15.72:0/4223409992 <== mgr.4104 v2:172.21.15.72:6832/333891523 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+13379 (secure 0 0 0) 0x7fb4e40b1200 con 0x7fb4d006b4d0 2024-03-06T07:32:57.121 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:57.121 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-06T07:32:57.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb4d006b4d0 msgr2=0x7fb4d006d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb4d006b4d0 0x7fb4d006d990 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb4e416adf0 tx=0x7fb4d40079b0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 msgr2=0x7fb4e4140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 0x7fb4e4140700 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fb4d800c4d0 tx=0x7fb4d800c990 comp rx=0 tx=0).stop 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 shutdown_connections 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fb4d006b4d0 0x7fb4d006d990 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4e415bf30 0x7fb4e4140c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4e413dc10 0x7fb4e4140700 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 --2- 172.21.15.72:0/4223409992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4e40b3060 0x7fb4e4159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 >> 172.21.15.72:0/4223409992 conn(0x7fb4e40cf8a0 msgr2=0x7fb4e40b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 shutdown_connections 2024-03-06T07:32:57.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.120+0000 7fb4ea102640 1 -- 172.21.15.72:0/4223409992 wait complete. 2024-03-06T07:32:57.133 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-03-06T07:32:56.067865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":330296,"kb_used_data":14280,"kb_used_omap":12,"kb_used_meta":315891,"kb_avail":754644424,"statfs":{"total":773094113280,"available":772755890176,"internally_reserved":0,"allocated":14622720,"data_stored":7059408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":323472945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002321"},"pg_stats":[{"pgid":"1.0","version":"134'119","reported_seq":390,"reported_epoch":134,"state":"active+clean","last_fresh":"2024-03-06T07:32:36.763614+0000","last_change":"2024-03-06T07:32:36.763651+0000","last_active":"2024-03-06T07:32:36.763614+0000","last_peered":"2024-03-06T07:32:36.763614+0000","last_clean":"2024-03-06T07:32:36.763614+0000","last_became_active":"2024-03-06T07:21:39.963024+0000","last_became_peered":"2024-03-06T07:21:39.963024+0000","last_unstale":"2024-03-06T07:32:36.763614+0000","last_undegraded":"2024-03-06T07:32:36.763614+0000","last_fullsized":"2024-03-06T07:32:36.763614+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'119","last_scrub_stamp":"2024-03-06T07:32:36.763516+0000","last_deep_scrub":"134'119","last_deep_scrub_stamp":"2024-03-06T07:32:36.763516+0000","last_clean_scrub_stamp":"2024-03-06T07:32:36.763516+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-07T17:15:00.996906+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574985,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41716,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330124,"statfs":{"total":96636764160,"available":96594046976,"internally_reserved":0,"allocated":2273280,"data_stored":1325202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574987,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":41144,"kb_used_data":1640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":1679360,"data_stored":734834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574986,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41716,"kb_used_data":2220,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330124,"statfs":{"total":96636764160,"available":96594046976,"internally_reserved":0,"allocated":2273280,"data_stored":1325202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-06T07:32:57.134 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-06T07:32:57.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.231+0000 7fbea3713640 1 Processor -- start 2024-03-06T07:32:57.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.232+0000 7fbea3713640 1 -- start start 2024-03-06T07:32:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.232+0000 7fbea3713640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe9c1614d0 0x7fbe9c0d4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.232+0000 7fbea3713640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.232+0000 7fbea3713640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c1618b0 con 0x7fbe9c0d1910 2024-03-06T07:32:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.232+0000 7fbea3713640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe9c0cf400 con 0x7fbe9c1614d0 2024-03-06T07:32:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.232+0000 7fbea3713640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0cf6a0 con 0x7fbe9c0d24b0 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea14af640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbe9c1614d0 0x7fbe9c0d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea0cae640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbe9c0d1910 0x7fbe9c0d1da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:59676/0 (socket says 172.21.15.72:59676) 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea0cae640 1 -- 172.21.15.72:0/1441537677 learned_addr learned my addr 172.21.15.72:0/1441537677 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea1cb0640 1 --2- 172.21.15.72:0/1441537677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea1cb0640 1 -- 172.21.15.72:0/1441537677 >> v2:172.21.15.146:3300/0 conn(0x7fbe9c1614d0 msgr2=0x7fbe9c0d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea1cb0640 1 --2- 172.21.15.72:0/1441537677 >> v2:172.21.15.146:3300/0 conn(0x7fbe9c1614d0 0x7fbe9c0d4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea1cb0640 1 -- 172.21.15.72:0/1441537677 >> v1:172.21.15.146:6789/0 conn(0x7fbe9c0d1910 legacy=0x7fbe9c0d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea1cb0640 1 -- 172.21.15.72:0/1441537677 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0cf980 con 0x7fbe9c0d24b0 2024-03-06T07:32:57.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.233+0000 7fbea14af640 1 --2- 172.21.15.72:0/1441537677 >> v2:172.21.15.146:3300/0 conn(0x7fbe9c1614d0 0x7fbe9c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-06T07:32:57.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea1cb0640 1 --2- 172.21.15.72:0/1441537677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b0e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe940097a0 tx=0x7fbe9402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e62d0bad96067e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbe9bfff640 1 -- 172.21.15.72:0/1441537677 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94035020 con 0x7fbe9c0d24b0 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbe9bfff640 1 -- 172.21.15.72:0/1441537677 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe94031830 con 0x7fbe9c0d24b0 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbe9bfff640 1 -- 172.21.15.72:0/1441537677 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94031c00 con 0x7fbe9c0d24b0 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea3713640 1 -- 172.21.15.72:0/1441537677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c0d24b0 msgr2=0x7fbe9c0b0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea3713640 1 --2- 172.21.15.72:0/1441537677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b0e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbe940097a0 tx=0x7fbe9402b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea3713640 1 -- 172.21.15.72:0/1441537677 shutdown_connections 2024-03-06T07:32:57.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea3713640 1 --2- 172.21.15.72:0/1441537677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b0e30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea3713640 1 --2- 172.21.15.72:0/1441537677 >> v2:172.21.15.146:3300/0 conn(0x7fbe9c1614d0 0x7fbe9c0d4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.234+0000 7fbea3713640 1 -- 172.21.15.72:0/1441537677 >> 172.21.15.72:0/1441537677 conn(0x7fbe9c0d3d40 msgr2=0x7fbe9c0d4150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.235+0000 7fbea3713640 1 -- 172.21.15.72:0/1441537677 shutdown_connections 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.235+0000 7fbea3713640 1 -- 172.21.15.72:0/1441537677 wait complete. 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 Processor -- start 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 -- start start 2024-03-06T07:32:57.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 0x7fbe9c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c1614d0 0x7fbe9c1521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0b1370 con 0x7fbe9c0d1910 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0b0910 con 0x7fbe9c0d24b0 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.236+0000 7fbea3713640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0b14f0 con 0x7fbe9c1614d0 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 0x7fbe9c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 0x7fbe9c0c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46950/0 (socket says 172.21.15.72:46950) 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 -- 172.21.15.72:0/2903897853 learned_addr learned my addr 172.21.15.72:0/2903897853 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:57.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea0cae640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea1cb0640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c1614d0 0x7fbe9c1521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 -- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c1614d0 msgr2=0x7fbe9c1521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c1614d0 0x7fbe9c1521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 -- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe9c0d24b0 msgr2=0x7fbe9c0b96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 -- 172.21.15.72:0/2903897853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c152700 con 0x7fbe9c0d1910 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea14af640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 0x7fbe9c0c8b90 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009ec0 tx=0x7fbe8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c018020 con 0x7fbe9c0d1910 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c014680 con 0x7fbe9c0d1910 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c0c9270 con 0x7fbe9c0d1910 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.237+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c014a50 con 0x7fbe9c0d1910 2024-03-06T07:32:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.238+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c0c9830 con 0x7fbe9c0d1910 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.239+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c068ff0 con 0x7fbe9c0d1910 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.239+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7fbe8c005d10 con 0x7fbe9c0d1910 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.239+0000 7fbe99ffb640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe8806b4d0 0x7fbe8806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.239+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbe8c08aad0 con 0x7fbe9c0d1910 2024-03-06T07:32:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.239+0000 7fbea0cae640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe8806b4d0 0x7fbe8806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.240+0000 7fbea0cae640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe8806b4d0 0x7fbe8806d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0b7130 tx=0x7fbe90007e80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.244+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7fbe8c022030 con 0x7fbe9c0d1910 2024-03-06T07:32:57.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.415+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbe9c0d20b0 con 0x7fbe9c0d1910 2024-03-06T07:32:57.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.418+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v34)=0 v34) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbe9c152700 con 0x7fbe9c0d1910 2024-03-06T07:32:57.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.419+0000 7fbe99ffb640 1 -- 172.21.15.72:0/2903897853 <== mon.0 v2:172.21.15.72:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbe8c059630 con 0x7fbe9c0d1910 2024-03-06T07:32:57.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe8806b4d0 msgr2=0x7fbe8806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe8806b4d0 0x7fbe8806d990 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0b7130 tx=0x7fbe90007e80 comp rx=0 tx=0).stop 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 msgr2=0x7fbe9c0c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 0x7fbe9c0c8b90 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c009ec0 tx=0x7fbe8c00c490 comp rx=0 tx=0).stop 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 shutdown_connections 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7fbe8806b4d0 0x7fbe8806d990 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbe9c1614d0 0x7fbe9c1521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbe9c0d24b0 0x7fbe9c0b96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 --2- 172.21.15.72:0/2903897853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbe9c0d1910 0x7fbe9c0c8b90 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 >> 172.21.15.72:0/2903897853 conn(0x7fbe9c0d3d40 msgr2=0x7fbe9c06c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 shutdown_connections 2024-03-06T07:32:57.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.421+0000 7fbea3713640 1 -- 172.21.15.72:0/2903897853 wait complete. 2024-03-06T07:32:57.434 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-06T07:32:57.435 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-06T07:32:57.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 Processor -- start 2024-03-06T07:32:57.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 -- start start 2024-03-06T07:32:57.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c1424f0 0x7f278c0cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f278c06da40 0x7f278c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0cc980 con 0x7f278c0cd7a0 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278c0cd220 con 0x7f278c06da40 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f2791cb1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c0ccc60 con 0x7f278c1424f0 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278b7fe640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c1424f0 0x7f278c0cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278affd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f278c0cd7a0 0x7f278c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.72:59690/0 (socket says 172.21.15.72:59690) 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278bfff640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f278c06da40 0x7f278c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278affd640 1 -- 172.21.15.72:0/2363662563 learned_addr learned my addr 172.21.15.72:0/2363662563 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-06T07:32:57.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278a7fc640 1 -- 172.21.15.72:0/2363662563 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744402244 0 0) 0x7f278c0cc980 con 0x7f278c0cd7a0 2024-03-06T07:32:57.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278b7fe640 1 -- 172.21.15.72:0/2363662563 >> v2:172.21.15.146:3300/0 conn(0x7f278c06da40 msgr2=0x7f278c15cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-06T07:32:57.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278b7fe640 1 --2- 172.21.15.72:0/2363662563 >> v2:172.21.15.146:3300/0 conn(0x7f278c06da40 0x7f278c15cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278b7fe640 1 -- 172.21.15.72:0/2363662563 >> v1:172.21.15.146:6789/0 conn(0x7f278c0cd7a0 legacy=0x7f278c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.534+0000 7f278b7fe640 1 -- 172.21.15.72:0/2363662563 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c0ccf40 con 0x7f278c1424f0 2024-03-06T07:32:57.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.535+0000 7f278b7fe640 1 --2- 172.21.15.72:0/2363662563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c1424f0 0x7f278c0cc2f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43757ff92074b4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.535+0000 7f278a7fc640 1 -- 172.21.15.72:0/2363662563 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c035020 con 0x7f278c1424f0 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.535+0000 7f278a7fc640 1 -- 172.21.15.72:0/2363662563 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f277c031830 con 0x7f278c1424f0 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.535+0000 7f278a7fc640 1 -- 172.21.15.72:0/2363662563 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c031c20 con 0x7f278c1424f0 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 -- 172.21.15.72:0/2363662563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c1424f0 msgr2=0x7f278c0cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 --2- 172.21.15.72:0/2363662563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c1424f0 0x7f278c0cc2f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c02b8d0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 -- 172.21.15.72:0/2363662563 shutdown_connections 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 --2- 172.21.15.72:0/2363662563 >> v2:172.21.15.146:3300/0 conn(0x7f278c06da40 0x7f278c15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 --2- 172.21.15.72:0/2363662563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c1424f0 0x7f278c0cc2f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 -- 172.21.15.72:0/2363662563 >> 172.21.15.72:0/2363662563 conn(0x7f278c0cf8a0 msgr2=0x7f278c0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 -- 172.21.15.72:0/2363662563 shutdown_connections 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.536+0000 7f2791cb1640 1 -- 172.21.15.72:0/2363662563 wait complete. 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.537+0000 7f2791cb1640 1 Processor -- start 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.537+0000 7f2791cb1640 1 -- start start 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f2791cb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 0x7f278c0ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f2791cb1640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c0cd7a0 0x7f278c0ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f2791cb1640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f278c1424f0 0x7f278c1f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f2791cb1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c071ce0 con 0x7f278c06da40 2024-03-06T07:32:57.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f2791cb1640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c071b60 con 0x7f278c1424f0 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f2791cb1640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278c071e60 con 0x7f278c0cd7a0 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 0x7f278c0ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 0x7f278c0ab0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46964/0 (socket says 172.21.15.72:46964) 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 -- 172.21.15.72:0/4219252138 learned_addr learned my addr 172.21.15.72:0/4219252138 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278affd640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c0cd7a0 0x7f278c0ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278bfff640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f278c1424f0 0x7f278c1f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 -- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c0cd7a0 msgr2=0x7f278c0ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c0cd7a0 0x7f278c0ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 -- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f278c1424f0 msgr2=0x7f278c1f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f278c1424f0 0x7f278c1f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 -- 172.21.15.72:0/4219252138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c0a9450 con 0x7f278c06da40 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278bfff640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f278c1424f0 0x7f278c1f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278affd640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c0cd7a0 0x7f278c0ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.538+0000 7f278b7fe640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 0x7f278c0ab0e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.539+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c041070 con 0x7f278c06da40 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.539+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f277c004230 con 0x7f278c06da40 2024-03-06T07:32:57.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.539+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c1f5bc0 con 0x7f278c06da40 2024-03-06T07:32:57.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.539+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c035040 con 0x7f278c06da40 2024-03-06T07:32:57.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.539+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c1f6180 con 0x7f278c06da40 2024-03-06T07:32:57.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.540+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c04f330 con 0x7f278c06da40 2024-03-06T07:32:57.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.540+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 89780+0+0 (secure 0 0 0) 0x7f277c038020 con 0x7f278c06da40 2024-03-06T07:32:57.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.540+0000 7f2773fff640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f277806b4d0 0x7f277806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-06T07:32:57.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.540+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f277c0ab9c0 con 0x7f278c06da40 2024-03-06T07:32:57.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.540+0000 7f278affd640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f277806b4d0 0x7f277806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-06T07:32:57.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.541+0000 7f278affd640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f277806b4d0 0x7f277806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f278c0ab620 tx=0x7f2780009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-06T07:32:57.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.545+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183808 (secure 0 0 0) 0x7f277c0ade20 con 0x7f278c06da40 2024-03-06T07:32:57.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.751+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f278c145b30 con 0x7f278c06da40 2024-03-06T07:32:57.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.751+0000 7f2773fff640 1 -- 172.21.15.72:0/4219252138 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+1326 (secure 0 0 0) 0x7f277c031800 con 0x7f278c06da40 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":91,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9765,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/1157612702","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":1157612702},{"type":"v1","addr":"172.21.15.72:6837","nonce":1157612702}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[]} 2024-03-06T07:32:57.753 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 91 2024-03-06T07:32:57.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f277806b4d0 msgr2=0x7f277806d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f277806b4d0 0x7f277806d990 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f278c0ab620 tx=0x7f2780009040 comp rx=0 tx=0).stop 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 msgr2=0x7f278c0ab0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 0x7f278c0ab0e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f277c0097a0 tx=0x7f277c0046c0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 shutdown_connections 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:6832/333891523,v1:172.21.15.72:6833/333891523] conn(0x7f277806b4d0 0x7f277806d990 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f278c1424f0 0x7f278c1f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f278c0cd7a0 0x7f278c0ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 --2- 172.21.15.72:0/4219252138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f278c06da40 0x7f278c0ab0e0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-06T07:32:57.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 >> 172.21.15.72:0/4219252138 conn(0x7f278c0cf8a0 msgr2=0x7f278c15efb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-06T07:32:57.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 shutdown_connections 2024-03-06T07:32:57.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-06T07:32:57.753+0000 7f2791cb1640 1 -- 172.21.15.72:0/4219252138 wait complete. 2024-03-06T07:32:57.765 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-06T07:32:57.765 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-06T07:32:57.766 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-06T07:32:57.766 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:57.774 INFO:tasks.ceph.mds.a:Stopped 2024-03-06T07:32:57.774 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-06T07:32:57.774 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:57.825 INFO:tasks.ceph.mds.c:Stopped 2024-03-06T07:32:57.825 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-03-06T07:32:57.828 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-03-06T07:32:57.834 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-06T07:32:57.834 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-06T07:32:57.834 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:57.892 INFO:tasks.ceph.osd.0:Stopped 2024-03-06T07:32:57.892 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-06T07:32:57.893 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:57.959 INFO:tasks.ceph.osd.1:Stopped 2024-03-06T07:32:57.960 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-06T07:32:57.960 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.031 INFO:tasks.ceph.osd.2:Stopped 2024-03-06T07:32:58.031 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-06T07:32:58.031 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.100 INFO:tasks.ceph.osd.3:Stopped 2024-03-06T07:32:58.101 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-06T07:32:58.101 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.131 INFO:tasks.ceph.osd.4:Stopped 2024-03-06T07:32:58.131 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-06T07:32:58.131 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.203 INFO:tasks.ceph.osd.5:Stopped 2024-03-06T07:32:58.203 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-06T07:32:58.203 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.275 INFO:tasks.ceph.osd.6:Stopped 2024-03-06T07:32:58.275 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-06T07:32:58.275 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.342 INFO:tasks.ceph.osd.7:Stopped 2024-03-06T07:32:58.342 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-06T07:32:58.342 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-06T07:32:58.343 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.370 INFO:tasks.ceph.mgr.y:Stopped 2024-03-06T07:32:58.370 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-06T07:32:58.370 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.409 INFO:tasks.ceph.mgr.x:Stopped 2024-03-06T07:32:58.409 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-06T07:32:58.409 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-06T07:32:58.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.421 INFO:tasks.ceph.mon.a:Stopped 2024-03-06T07:32:58.421 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-06T07:32:58.421 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.461 INFO:tasks.ceph.mon.b:Stopped 2024-03-06T07:32:58.462 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-06T07:32:58.462 INFO:teuthology.orchestra.run:waiting for 300 2024-03-06T07:32:58.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-06T07:32:58.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-03-06T07:32:58.515 INFO:tasks.ceph.mon.c:Stopped 2024-03-06T07:32:58.515 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-06T07:32:58.515 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-06T07:32:58.546 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:32:58.547 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-06T07:32:58.786 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:32:58.786 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-06T07:32:58.868 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:32:58.868 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-06T07:32:58.958 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:32:58.959 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-06T07:32:59.009 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:32:59.009 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-06T07:32:59.196 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:32:59.196 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-06T07:32:59.274 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:32:59.274 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-06T07:32:59.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:32:59.334 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-06T07:32:59.390 INFO:tasks.ceph:Archiving mon data... 2024-03-06T07:32:59.391 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/data/mon.a.tgz 2024-03-06T07:32:59.391 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-06T07:32:59.407 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.EG6W8OgR3f 2024-03-06T07:32:59.407 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.EG6W8OgR3f 2024-03-06T07:32:59.564 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.EG6W8OgR3f 2024-03-06T07:32:59.649 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.EG6W8OgR3f is 392KB 2024-03-06T07:32:59.746 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.EG6W8OgR3f 2024-03-06T07:32:59.760 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/data/mon.b.tgz 2024-03-06T07:32:59.760 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-06T07:32:59.776 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.c8vGaAdXGR 2024-03-06T07:32:59.777 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.c8vGaAdXGR 2024-03-06T07:32:59.936 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.c8vGaAdXGR 2024-03-06T07:33:00.020 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.c8vGaAdXGR is 397KB 2024-03-06T07:33:00.117 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.c8vGaAdXGR 2024-03-06T07:33:00.132 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/data/mon.c.tgz 2024-03-06T07:33:00.132 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-03-06T07:33:00.186 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.aOj7zkw5HA 2024-03-06T07:33:00.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.aOj7zkw5HA 2024-03-06T07:33:00.325 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.aOj7zkw5HA 2024-03-06T07:33:00.409 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.aOj7zkw5HA is 397KB 2024-03-06T07:33:00.505 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.aOj7zkw5HA 2024-03-06T07:33:00.520 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-06T07:33:00.520 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:33:00.523 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-06T07:33:00.604 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:33:00.641 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-06T07:33:00.641 INFO:tasks.ceph:Archiving crash dumps... 2024-03-06T07:33:00.643 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/remote/smithi072/crash 2024-03-06T07:33:00.644 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:33:00.672 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/remote/smithi146/crash 2024-03-06T07:33:00.673 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-06T07:33:00.708 INFO:tasks.ceph:Shutting down logrotate 2024-03-06T07:33:00.708 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:00.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:00.817 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:00.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-06T07:33:00.870 INFO:tasks.ceph:Compressing logs... 2024-03-06T07:33:00.870 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:33:00.886 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-06T07:33:00.969 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-06T07:33:00.969 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37848.log 2024-03-06T07:33:00.969 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37928.log 2024-03-06T07:33:00.969 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.37848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37888.log 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37848.log.gz/var/log/ceph/ceph-client.admin.37928.log: 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37928.log.gz 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.37888.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37968.log 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37888.log.gz 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.37968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44310.log 2024-03-06T07:33:00.970 INFO:teuthology.orchestra.run.smithi146.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37968.log.gz 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38008.log 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5/var/log/ceph/ceph-client.1.44310.log: --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.38008.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.7.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.38008.log.gz 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr: -5 89.6% -- replaced with /var/log/ceph/ceph-client.1.44310.log.gz 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38128.log 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5/var/log/ceph/ceph-osd.5.log: --verbose -- /var/log/ceph/ceph-client.admin.38048.log 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.38128.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-06T07:33:00.971 INFO:teuthology.orchestra.run.smithi146.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38128.log.gz 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.38048.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38048.log.gz 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44762.log 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.3.44762.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38088.log 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: 90.5% -- replaced with /var/log/ceph/ceph-client.3.44762.log.gz 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-06T07:33:00.972 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.38088.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38088.log.gz 2024-03-06T07:33:00.973 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-06T07:33:00.973 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:33:00.973 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-06T07:33:00.973 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.d.log: 94.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-06T07:33:00.973 INFO:teuthology.orchestra.run.smithi146.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-06T07:33:01.082 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.c.log: 92.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-06T07:33:01.237 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73062.log 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83645.log 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52762.log 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73062.log: /var/log/ceph/ceph-client.admin.83645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51075.log 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52762.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73062.log.gz 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48710.log 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83645.log.gz 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52762.log.gz 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82829.log 2024-03-06T07:33:01.238 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52342.log 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51075.log.gz 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48710.log.gz 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49676.log 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52342.log: gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.82829.log.gz 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.0.65540.log 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58778.log 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52342.log.gz 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.65540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50699.log 2024-03-06T07:33:01.239 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49676.log.gz 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83855.log 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83141.log 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58778.log.gz 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83855.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50699.log.gz 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62075.log 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83855.log.gz 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60139.log 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62075.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83141.log.gz 2024-03-06T07:33:01.240 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84503.log 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69246.log 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84503.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60139.log.gz 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.62075.log.gz 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43875.log 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84503.log.gz 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45338.log 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69246.log.gz 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56736.log 2024-03-06T07:33:01.241 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47999.log 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43875.log.gz 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56736.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45338.log.gz 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67877.log 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56736.log.gz 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70703.log 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58878.log 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47999.log.gz 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70703.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67877.log.gz 2024-03-06T07:33:01.242 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70703.log.gz 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85570.log 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61907.log 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58878.log.gz 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56111.log 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61907.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85570.log.gz 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44805.log 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61907.log.gz 2024-03-06T07:33:01.243 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57556.log 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56111.log.gz 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56321.log 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84377.log 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57556.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44805.log.gz 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61949.log 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57556.log.gz 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84377.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56321.log.gz 2024-03-06T07:33:01.244 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70661.log 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61949.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84377.log.gz 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86710.log 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56069.log 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70661.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61949.log.gz 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86710.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.44330.log 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70661.log.gz 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63420.log 2024-03-06T07:33:01.245 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56069.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86710.log.gz 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44330.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56069.log.gz 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86021.log 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78656.log 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44330.log.gz 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63420.log: /var/log/ceph/ceph-client.admin.86021.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63420.log.gz 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57510.log 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.86021.log.gz 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58694.log 2024-03-06T07:33:01.246 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78656.log: /var/log/ceph/ceph-client.admin.57510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59248.log 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58694.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78656.log.gz 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51656.log 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57510.log.gz 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59248.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58694.log.gz 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84792.log 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51656.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59248.log.gz 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85937.log 2024-03-06T07:33:01.247 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51656.log.gz 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51289.log 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85937.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84792.log.gz 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71464.log 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85937.log.gz 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.51289.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63672.log 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58500.log 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51289.log.gz 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr: 95.5% -- replaced with /var/log/ceph/ceph-client.0.65540.log.gz 2024-03-06T07:33:01.248 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71464.log.gz 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55436.log 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76924.log 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58500.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63672.log.gz 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57976.log 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.55436.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58500.log.gz 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49844.log 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57976.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76924.log.gz 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79917.log 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57976.log.gz 2024-03-06T07:33:01.249 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31876.log 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49844.log: /var/log/ceph/ceph-client.admin.79917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84903.log 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49844.log.gz 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.tmp-client.admin.31876.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79917.log.gz 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44795.log 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31876.log.gz 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50619.log 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3%/var/log/ceph/ceph-client.admin.44795.log: -- replaced with /var/log/ceph/ceph-client.admin.84903.log.gz 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56862.log 2024-03-06T07:33:01.250 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44795.log.gz 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48878.log 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50619.log: /var/log/ceph/ceph-client.admin.56862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53811.log 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50619.log.gz 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56862.log.gz 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51159.log 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48878.log.gz 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70955.log 2024-03-06T07:33:01.251 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84461.log 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51159.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53811.log.gz 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69534.log 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51159.log.gz 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77531.log 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84461.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70955.log.gz 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54338.log 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84461.log.gz 2024-03-06T07:33:01.252 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69534.log.gz 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75026.log 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77531.log: /var/log/ceph/ceph-client.admin.54338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74877.log 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77531.log.gz 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54338.log.gz 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67028.log 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86189.log 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74877.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75026.log.gz 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67028.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74877.log.gz 2024-03-06T07:33:01.253 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45139.log 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67028.log.gz 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.86189.log: -- /var/log/ceph/ceph-client.admin.70372.log 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57682.log 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.86189.log.gz 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70372.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71552.log 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70372.log.gz 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47243.log 2024-03-06T07:33:01.254 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57682.log.gz 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55593.log 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71552.log: /var/log/ceph/ceph-client.admin.47243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56694.log 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71552.log.gz 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47243.log.gz 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66505.log 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49760.log 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56694.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55593.log.gz 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66505.log: 89.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.56694.log.gz 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55985.log 2024-03-06T07:33:01.255 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66505.log.gz 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77266.log 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49760.log: /var/log/ceph/ceph-client.admin.55985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60704.log 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49760.log.gz 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77266.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55985.log.gz 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53414.log 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60704.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77266.log.gz 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75212.log 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60704.log.gz 2024-03-06T07:33:01.256 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37951.log 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53414.log: /var/log/ceph/ceph-client.admin.75212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57426.log 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53414.log.gz 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37951.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75212.log.gz 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66902.log 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37951.log.gz 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51596.log 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57426.log: /var/log/ceph/ceph-client.admin.66902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81760.log 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57426.log.gz 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84167.log 2024-03-06T07:33:01.257 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66902.log.gz 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51596.log.gz 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86231.log 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77661.log 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.81760.log.gz 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84167.log: /var/log/ceph/ceph-client.admin.86231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67326.log 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84167.log.gz 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86231.log.gz 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76233.log 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77661.log.gz 2024-03-06T07:33:01.258 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.67326.log: -- /var/log/ceph/ceph-client.admin.48752.log 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82222.log 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67326.log.gz 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48752.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76233.log.gz 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86483.log 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48752.log.gz 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71338.log 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82222.log.gz 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44414.log 2024-03-06T07:33:01.259 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86483.log: /var/log/ceph/ceph-client.admin.71338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79563.log 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.86483.log.gz 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44414.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71338.log.gz 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53162.log 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84209.log 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79563.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44414.log.gz 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74375.log 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79563.log.gz 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53162.log.gz 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53372.log 2024-03-06T07:33:01.260 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58736.log 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84209.log.gz 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53372.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74375.log.gz 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48125.log 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53372.log.gz 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73384.log 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72507.log 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58736.log.gz 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48125.log: /var/log/ceph/ceph-client.admin.73384.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48125.log.gz 2024-03-06T07:33:01.261 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53288.log 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73384.log.gz 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65206.log 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.72507.log: /var/log/ceph/ceph-client.admin.53288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63015.log 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53288.log.gz 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64307.log 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62599.log 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63015.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.65206.log.gz 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86567.log 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63015.log.gz 2024-03-06T07:33:01.262 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51512.log 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64307.log.gz 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62599.log.gz 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76966.log 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57766.log 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51512.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.86567.log.gz 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85257.log 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51512.log.gz 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57766.log: gzip -5 --verbose 88.3% -- /var/log/ceph/ceph-client.admin.52883.log -- replaced with /var/log/ceph/ceph-client.admin.76966.log.gz 2024-03-06T07:33:01.263 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85257.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57766.log.gz 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64145.log 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85257.log.gz 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66253.log 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86063.log 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52883.log.gz 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66253.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64145.log.gz 2024-03-06T07:33:01.264 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.63294.log 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86063.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66253.log.gz 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70330.log 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86063.log.gz 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75870.log 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63294.log: /var/log/ceph/ceph-client.admin.70330.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.63294.log.gz -5 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.85173.log 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70330.log.gz 2024-03-06T07:33:01.265 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66860.log 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75870.log: /var/log/ceph/ceph-client.admin.85173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65773.log 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75870.log.gz 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66860.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85173.log.gz 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72978.log 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.66860.log.gz 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53540.log 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65773.log.gz 2024-03-06T07:33:01.266 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50949.log 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72978.log: /var/log/ceph/ceph-client.admin.53540.log: gzip -5 94.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.0.72507.log.gz /var/log/ceph/ceph-client.admin.82180.log 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72978.log.gz 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53540.log.gz 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80295.log 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45131.log 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50949.log.gz 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82180.log: /var/log/ceph/ceph-client.admin.80295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44456.log 2024-03-06T07:33:01.267 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82180.log.gz 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45131.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80295.log.gz 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63462.log 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67123.log 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45131.log.gz 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63462.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44456.log.gz 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70997.log 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63462.log.gz 2024-03-06T07:33:01.268 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82871.log 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67123.log: /var/log/ceph/ceph-client.admin.70997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47915.log 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67123.log.gz 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82871.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70997.log.gz 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44800.log 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.82871.log.gz 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72248.log 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47915.log.gz 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76275.log 2024-03-06T07:33:01.269 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44800.log: /var/log/ceph/ceph-client.admin.72248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69925.log 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44800.log.gz 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76275.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72248.log.gz 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.55436.log.gz 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44288.log 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80211.log 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69925.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76275.log.gz 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85769.log 2024-03-06T07:33:01.270 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69925.log.gz 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80211.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44288.log.gz -5 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.79479.log 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80211.log.gz 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76191.log 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85769.log.gz 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44801.log 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81508.log 2024-03-06T07:33:01.271 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79479.log.gz 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76191.log.gz 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84624.log 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44801.log.gz 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75691.log 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86105.log 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81508.log.gz 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84624.log.gz 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83519.log 2024-03-06T07:33:01.272 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75691.log.gz 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59819.log 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64563.log 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86105.log.gz 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83519.log.gz/var/log/ceph/ceph-client.admin.59819.log: 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73246.log 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59819.log.gz 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47411.log 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83477.log 2024-03-06T07:33:01.273 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64563.log.gz 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73246.log.gz 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80775.log 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47411.log.gz 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.83477.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.86628.log 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48548.log 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83477.log.gz 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80775.log.gz 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62557.log 2024-03-06T07:33:01.274 INFO:teuthology.orchestra.run.smithi072.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.86628.log.gz 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49330.log 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62557.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48548.log.gz 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77955.log 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62557.log.gz 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61297.log 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49330.log.gz 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57850.log 2024-03-06T07:33:01.275 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57203.log 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77955.log.gz 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57850.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61297.log.gz 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78189.log 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57203.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.66944.log -- replaced with /var/log/ceph/ceph-client.admin.57850.log.gz 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56153.log 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57203.log.gz 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78189.log.gz 2024-03-06T07:33:01.276 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65164.log 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66944.log.gz 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53456.log 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47327.log 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56153.log.gz 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65164.log.gz 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66058.log 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53456.log.gz 2024-03-06T07:33:01.277 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50096.log 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54733.log 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47327.log.gz 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66058.log.gz 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76317.log 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50096.log.gz 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81466.log 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70558.log 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54733.log.gz 2024-03-06T07:33:01.278 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76317.log.gz 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67751.log 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81466.log.gz 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59004.log 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61445.log 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70558.log.gz 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67751.log.gz 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46070.log 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59004.log.gz 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66986.log 2024-03-06T07:33:01.279 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74559.log 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61445.log.gz 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46070.log.gz 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85811.log 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66986.log.gz 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59861.log 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.59507.log 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74559.log.gz 2024-03-06T07:33:01.280 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59861.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85811.log.gz 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49718.log 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59861.log.gz 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.59507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80547.log 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61213.log 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49718.log.gz 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76882.log 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54194.log 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80547.log.gz 2024-03-06T07:33:01.281 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61213.log: /var/log/ceph/ceph-client.admin.76882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50390.log 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61213.log.gz 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56526.log 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54194.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76882.log.gz 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75753.log 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54194.log.gz 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76569.log 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3%/var/log/ceph/ceph-client.admin.56526.log: -- replaced with /var/log/ceph/ceph-client.admin.50390.log.gz 2024-03-06T07:33:01.282 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75753.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56526.log.gz 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73120.log 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75753.log.gz 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69742.log 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76569.log: /var/log/ceph/ceph-client.admin.73120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59186.log 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76569.log.gz 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69742.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73120.log.gz 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80902.log 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69742.log.gz 2024-03-06T07:33:01.283 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79875.log 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59186.log: /var/log/ceph/ceph-client.admin.80902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50054.log 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59186.log.gz 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79875.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80902.log.gzgzip 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61487.log 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73813.log 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50054.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79875.log.gz 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61487.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50054.log.gz 2024-03-06T07:33:01.284 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60914.log 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60223.log 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73813.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61487.log.gz 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56568.log 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60914.log.gz 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60746.log 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.73813.log.gz 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45842.log 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60223.log.gz 2024-03-06T07:33:01.285 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56568.log.gz 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57640.log 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53685.log 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45842.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60746.log.gz 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57640.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.45842.log.gz --verbose 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.73288.log 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57640.log.gz 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79791.log 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53685.log: /var/log/ceph/ceph-client.admin.73288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75255.log 2024-03-06T07:33:01.286 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53685.log.gz 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73288.log.gz 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45884.log 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79791.log.gz 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48506.log 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75255.log: /var/log/ceph/ceph-client.admin.45884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72783.log 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45884.log.gz 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.75255.log.gz 2024-03-06T07:33:01.287 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57287.log 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62675.log 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72783.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48506.log.gz 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57287.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72783.log.gz 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80986.log 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57287.log.gz 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65272.log 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62675.log: /var/log/ceph/ceph-client.admin.80986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52107.log 2024-03-06T07:33:01.288 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62675.log.gz 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65272.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80986.log.gz 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75468.log 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.52107.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65272.log.gz 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43917.log 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53895.log 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75468.log.gz 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71134.log 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53895.log: gzip -5 88.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43917.log.gz /var/log/ceph/ceph-client.admin.72936.log 2024-03-06T07:33:01.289 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53895.log.gz 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77061.log 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72936.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82619.log 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71134.log.gz 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72936.log.gz 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55111.log 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78605.log 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77061.log.gz 2024-03-06T07:33:01.290 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82619.log.gz 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61571.log 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48262.log 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78605.log: /var/log/ceph/ceph-client.admin.61571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85005.log 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78605.log.gz 91.1% -- replaced with /var/log/ceph/ceph-client.admin.55111.log.gz 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66100.log 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61571.log.gz 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85005.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48262.log.gz 2024-03-06T07:33:01.291 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63966.log 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.85005.log.gz 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54985.log 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.63966.log: -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66100.log.gz 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.77224.log 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54985.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63966.log.gz 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50348.log 2024-03-06T07:33:01.292 INFO:teuthology.orchestra.run.smithi072.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.54985.log.gz 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60830.log 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77573.log 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77224.log.gz 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50348.log.gz 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54112.log 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77573.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60830.log.gz 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49970.log 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77573.log.gz 2024-03-06T07:33:01.293 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80043.log 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54112.log.gz 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76485.log 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80043.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49970.log.gz 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44582.log 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80043.log.gz 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67709.log 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62931.log 2024-03-06T07:33:01.294 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76485.log.gz 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52539.log 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44582.log.gz 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62931.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67709.log.gz 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86273.log 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62931.log.gz 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.50432.log/var/log/ceph/ceph-client.admin.52539.log: 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86273.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52539.log.gz 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66295.log 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.86273.log.gz 2024-03-06T07:33:01.295 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47285.log 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50432.log.gz 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46597.log 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47285.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66295.log.gz 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66631.log 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47285.log.gz 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78081.log 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46597.log: /var/log/ceph/ceph-client.admin.66631.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46597.log.gz 2024-03-06T07:33:01.296 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.67368.log 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78081.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66631.log.gz 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78081.log.gz 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.81322.log 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77615.log 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81322.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67368.log.gz 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43959.log 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81322.log.gz 2024-03-06T07:33:01.297 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58018.log 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80860.log 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.77615.log.gz 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43959.log.gz 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61781.log 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83225.log 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61781.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80860.log.gz 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78845.log 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.58018.log.gz 2024-03-06T07:33:01.298 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.83225.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82577.log 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61781.log.gz 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47133.log 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83225.log.gz 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78845.log.gz 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78761.log 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53009.log 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47133.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82577.log.gz 2024-03-06T07:33:01.299 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85528.log 88.7% 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47133.log.gz 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78761.log.gz 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45134.log 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76840.log 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53009.log.gz 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85528.log: /var/log/ceph/ceph-client.admin.45134.log: gzip -5 --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85528.log.gz 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.82661.log 2024-03-06T07:33:01.300 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45134.log.gz 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76840.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76840.log.gz 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83267.log 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82661.log.gz 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83813.log 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63756.log 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83267.log: gzip/var/log/ceph/ceph-client.admin.83813.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64349.log 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83267.log.gz 2024-03-06T07:33:01.301 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63756.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83813.log.gz 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64433.log 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63756.log.gz 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74815.log 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64433.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64349.log.gz 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.66379.log 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74815.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64433.log.gz 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62973.log 2024-03-06T07:33:01.302 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74815.log.gz 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44001.log 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62973.log: gzip -5 --verbose 88.5% -- /var/log/ceph/ceph-client.admin.45142.log 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66379.log.gz 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44001.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62973.log.gz 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47831.log 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44001.log.gz 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55027.log 2024-03-06T07:33:01.303 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49088.log 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9%/var/log/ceph/ceph-client.admin.47831.log: -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76798.log 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47831.log.gz 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49088.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55027.log.gz 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68649.log 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5%/var/log/ceph/ceph-client.admin.76798.log: -- replaced with /var/log/ceph/ceph-client.admin.49088.log.gz 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70162.log 2024-03-06T07:33:01.304 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67793.log 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68649.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76798.log.gz 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70162.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68649.log.gz 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.60265.log 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67793.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70162.log.gz 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80505.log 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67793.log.gz 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56652.log 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60265.log: /var/log/ceph/ceph-client.admin.80505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69680.log 2024-03-06T07:33:01.305 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60265.log.gz 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80505.log.gz 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38320.log 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82913.log 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56652.log.gz 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38320.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.69680.log.gz 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47579.log 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38320.log.gz 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51245.log 2024-03-06T07:33:01.306 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53330.log 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47579.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82913.log.gz 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51245.log: gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47579.log.gz 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.60662.log 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51245.log.gz 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71636.log 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53330.log.gz 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73725.log 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60662.log.gz 2024-03-06T07:33:01.307 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52663.log 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.73725.log: /var/log/ceph/ceph-client.admin.62819.log 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71636.log.gz 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52663.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73725.log.gz 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56363.log 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62819.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52663.log.gz 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73483.log 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62819.log.gz 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62033.log 2024-03-06T07:33:01.308 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44498.log 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56363.log.gz 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73483.log.gz 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79731.log 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44498.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62033.log.gz 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47537.log 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44498.log.gz 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67919.log 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47537.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.79731.log.gz 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60181.log 2024-03-06T07:33:01.309 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67919.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47537.log.gz 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.79833.log 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67919.log.gz 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66776.log 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79833.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60181.log.gz 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71678.log 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79833.log.gz 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83999.log 2024-03-06T07:33:01.310 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59046.log 87.4% 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66776.log.gz 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64008.log 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71678.log.gz 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr:gzip 88.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.83999.log.gz --verbose 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.49430.log 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45716.log 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64008.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59046.log.gz 2024-03-06T07:33:01.311 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52282.log 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64008.log.gz 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49430.log.gz 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82324.log 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45716.log.gz 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose/var/log/ceph/ceph-client.admin.52282.log: -- /var/log/ceph/ceph-client.admin.60055.log 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82324.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52282.log.gz 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.71508.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82324.log.gz 2024-03-06T07:33:01.312 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74441.log 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60055.log.gz 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.62400.log 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71508.log.gz 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69494.log 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77997.log 2024-03-06T07:33:01.313 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.62400.log: /var/log/ceph/ceph-client.admin.69494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78039.log 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77997.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69494.log.gz 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr:gzip 92.5% -- replaced with /var/log/ceph/ceph-client.admin.74441.log.gz 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63840.log 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77997.log.gz 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78039.log: gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78039.log.gz 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.50577.log 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55874.log 2024-03-06T07:33:01.314 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63840.log.gz 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56195.log 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50577.log.gz 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81280.log 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5/var/log/ceph/ceph-client.admin.56195.log: --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55874.log.gz 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.61009.log 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81280.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56195.log.gz 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84666.log 2024-03-06T07:33:01.315 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81280.log.gz 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82408.log 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84666.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85853.log 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61009.log.gz 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84666.log.gz 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59777.log 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85853.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43832.log 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82408.log.gz 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59777.log: gzip -5 88.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.85853.log.gz -- 2024-03-06T07:33:01.316 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.63336.log 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2%/var/log/ceph/ceph-client.admin.43832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80169.log 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59777.log.gz 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43832.log.gz 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr:gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63336.log.gz 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80169.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83309.log 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50991.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79046.log 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80169.log.gz 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53600.log 2024-03-06T07:33:01.317 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86315.log 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83309.log.gz 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61529.log 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79046.log.gz 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53600.log: /var/log/ceph/ceph-client.admin.86315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82096.log 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4%/var/log/ceph/ceph-client.admin.61529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57468.log 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.86315.log.gz 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53600.log.gz 2024-03-06T07:33:01.318 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61529.log.gz/var/log/ceph/ceph-client.admin.82096.log: gzip 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49004.log 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70456.log 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82096.log.gz/var/log/ceph/ceph-client.admin.49004.log: 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69967.log 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57468.log.gz 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49004.log.gz 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70456.log: --verbose -- /var/log/ceph/ceph-client.admin.63252.log 2024-03-06T07:33:01.319 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69967.log: 93.1% -- replaced with /var/log/ceph/ceph-client.0.59507.log.gz 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70456.log.gz 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66211.log 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.69967.log.gz 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49246.log 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48920.log 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63252.log.gz 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54901.log 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66211.log.gz 2024-03-06T07:33:01.320 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48920.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49246.log.gz 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45590.log 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48920.log.gz 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55748.log 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82366.log 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54901.log.gz 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55748.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56483.log 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45590.log.gz 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55748.log.gz 2024-03-06T07:33:01.321 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60097.log 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82366.log.gz 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74753.log 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56483.log: /var/log/ceph/ceph-client.admin.60097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49634.log 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56483.log.gz 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74753.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60097.log.gz 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58920.log 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49634.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74753.log.gz 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74313.log 2024-03-06T07:33:01.322 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49634.log.gz 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54691.log 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66421.log 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74313.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58920.log.gz 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54691.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74313.log.gz 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70608.log 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54691.log.gz 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50907.log 2024-03-06T07:33:01.323 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50821.log 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66421.log.gz 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50907.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64050.log 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70608.log.gz 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50907.log.gz 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52841.log 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.50821.log.gz -5 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.66016.log 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64050.log: /var/log/ceph/ceph-client.admin.52841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67961.log 2024-03-06T07:33:01.324 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64050.log.gz 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66016.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52841.log.gz 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83393.log 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53246.log 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67961.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66016.log.gz 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61357.log 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67961.log.gz 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.83393.log.gz 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.53246.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49928.log 2024-03-06T07:33:01.325 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45138.log 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53246.log.gz 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49928.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61357.log.gz 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43613.log 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45138.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49928.log.gz 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47621.log 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45138.log.gz 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70288.log 2024-03-06T07:33:01.326 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43613.log: /var/log/ceph/ceph-client.admin.47621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63924.log 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43613.log.gz 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47621.log.gz 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86525.log 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70288.log.gz 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63924.log: /var/log/ceph/ceph-client.admin.86525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45422.log 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63924.log.gz 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.86525.log.gz 2024-03-06T07:33:01.327 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70913.log 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77703.log 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70913.log: gzip -5 --verbose -- 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45422.log.gz 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.54817.log 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.70913.log.gz 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69883.log 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77703.log.gz 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68541.log 2024-03-06T07:33:01.328 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54817.log.gz 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69883.log: 92.7% -- replaced with /var/log/ceph/ceph-client.0.62400.log.gz 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47453.log 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69883.log.gz 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73162.log 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68541.log: /var/log/ceph/ceph-client.admin.47453.log: gzip -5 --verbose 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68541.log.gz 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.52601.log 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47453.log.gz 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69432.log 2024-03-06T07:33:01.329 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73162.log.gz 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61697.log 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52601.log.gz 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61171.log 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61865.log 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69432.log.gz 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61171.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.61697.log.gz 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.71888.log 2024-03-06T07:33:01.330 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61171.log.gz 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81196.log 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50306.log/var/log/ceph/ceph-client.admin.71888.log: 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61865.log.gz 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51340.log 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.71888.log.gz 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81196.log.gz 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82703.log 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53769.log 2024-03-06T07:33:01.331 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51340.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50306.log.gz 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82703.log: gzip -5 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51340.log.gz --verbose 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.54603.log 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82703.log.gz 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84545.log 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74602.log 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53769.log.gz 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54603.log: /var/log/ceph/ceph-client.admin.84545.log: gzip -5 --verbose -- 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54603.log.gz 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.67835.log 2024-03-06T07:33:01.332 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84545.log.gz 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45380.log 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.74602.log.gz 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70120.log 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45380.log: gzip 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67835.log.gz 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53643.log 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70120.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45380.log.gz 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83099.log 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70120.log.gz 2024-03-06T07:33:01.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81112.log 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53643.log: /var/log/ceph/ceph-client.admin.83099.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53643.log.gz 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64391.log 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81112.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.83099.log.gz 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61255.log 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81112.log.gz 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61255.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64391.log.gz 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60620.log 2024-03-06T07:33:01.334 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52477.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61255.log.gz 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.51470.log 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57724.log 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51470.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60620.log.gz 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71804.log 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57724.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51470.log.gz 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76672.log 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71804.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57724.log.gz 2024-03-06T07:33:01.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77829.log 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76672.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71804.log.gz 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70246.log 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76672.log.gz 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83183.log 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70246.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.77829.log.gz -5 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84750.log 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83183.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63504.log 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70246.log.gz 2024-03-06T07:33:01.336 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83183.log.gz 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82012.log 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70009.log 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84750.log.gz 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63504.log.gz 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52415.log 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71214.log 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70009.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82012.log.gz 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.78448.log 2024-03-06T07:33:01.337 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70009.log.gz 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52415.log.gz 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63378.log 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63882.log 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.78448.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71214.log.gz 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62141.log 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63378.log.gz 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56237.log 2024-03-06T07:33:01.338 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.62141.log: -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63882.log.gz 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49802.log 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56237.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62141.log.gz 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50779.log 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56237.log.gz 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44087.log 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49802.log.gz 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82787.log 2024-03-06T07:33:01.339 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50779.log.gz 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60349.log 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.82787.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45674.log 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44087.log.gz 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60349.log: gzip -5 --verbose -- 88.2% /var/log/ceph/ceph-client.admin.49162.log 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82787.log.gz 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45674.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60349.log.gz 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48836.log 2024-03-06T07:33:01.340 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45674.log.gz 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60872.log 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48836.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49162.log.gz 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57037.log 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60872.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48836.log.gz 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85401.log 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60872.log.gz 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.57037.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55832.log 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85401.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57037.log.gz 2024-03-06T07:33:01.341 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65835.log 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85401.log.gz 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83039.log 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55832.log: /var/log/ceph/ceph-client.admin.65835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65914.log 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55832.log.gz 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83039.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65835.log.gz 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72098.log 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83039.log.gz 2024-03-06T07:33:01.342 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.65914.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45141.log 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72098.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65914.log.gz 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr: 94.5% -- replaced with /var/log/ceph/ceph-client.0.78448.log.gz 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56778.log 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.72098.log.gz 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38201.log 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56778.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45141.log.gz 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48962.log 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56778.log.gz 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85131.log 2024-03-06T07:33:01.343 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48962.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38201.log.gz 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64828.log 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48962.log.gz 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71762.log 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54943.log 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.85131.log.gz 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71762.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64828.log.gz 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75150.log 2024-03-06T07:33:01.344 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71762.log.gz 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55069.log 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69062.log 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75150.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54943.log.gz 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51698.log 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75150.log.gz 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69062.log: gzip -5 88.7% --verbose -- /var/log/ceph/ceph-client.admin.65080.log 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55069.log.gz 2024-03-06T07:33:01.345 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63630.log 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr: 91.1% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51698.log.gz 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65080.log: -- replaced with /var/log/ceph/ceph-client.admin.69062.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72014.log 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63630.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65080.log.gz 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45132.log 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6%/var/log/ceph/ceph-client.admin.72014.log: -- replaced with /var/log/ceph/ceph-client.admin.63630.log.gz 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64225.log 2024-03-06T07:33:01.346 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45132.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72014.log.gz 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49388.log 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45132.log.gz 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60788.log 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64225.log.gz 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81406.log 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60788.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49388.log.gz 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54380.log 2024-03-06T07:33:01.347 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60788.log.gz 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83435.log 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81406.log: /var/log/ceph/ceph-client.admin.54380.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.56405.log 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81406.log.gz 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83435.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54380.log.gz 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67497.log 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83435.log.gz 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51117.log 2024-03-06T07:33:01.348 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84335.log 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67497.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56405.log.gz 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76714.log 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67497.log.gz 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84335.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51117.log.gz 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54519.log 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84335.log.gz 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77436.log 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85047.log 2024-03-06T07:33:01.349 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54519.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76714.log.gz 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85979.log 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54519.log.gz 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85047.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77436.log.gz 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67541.log 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.85047.log.gz 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85979.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.85979.log.gz 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48632.log 2024-03-06T07:33:01.350 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67541.log.gz 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63546.log 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48794.log 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63546.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48632.log.gz 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57375.log 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63546.log.gz 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77745.log 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48794.log.gz 2024-03-06T07:33:01.351 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79647.log 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57375.log.gz 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61823.log 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77745.log.gz 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68328.log 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.79647.log.gz 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71039.log 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.68328.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61823.log.gz 2024-03-06T07:33:01.352 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59124.log 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83603.log 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71039.log.gz 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45548.log 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59124.log.gz 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75912.log 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83603.log: /var/log/ceph/ceph-client.admin.45548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57808.log 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75912.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45548.log.gz 2024-03-06T07:33:01.353 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83603.log.gz 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75912.log.gz 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48590.log 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71422.log 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57808.log.gz 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81154.log 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48590.log.gz 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49288.log 2024-03-06T07:33:01.354 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71422.log: /var/log/ceph/ceph-client.admin.81154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85341.log 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49288.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81154.log.gz 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71422.log.gz 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49288.log.gz 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76038.log 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61401.log 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.85341.log.gz/var/log/ceph/ceph-client.admin.76038.log: 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73771.log 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.76038.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.46475.log 2024-03-06T07:33:01.355 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61401.log: /var/log/ceph/ceph-client.admin.73771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44806.log 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.46475.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61401.log.gz 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73771.log.gz 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83729.log 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75344.log 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.46475.log.gz 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44806.log.gz 2024-03-06T07:33:01.356 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82138.log 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69184.log 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83729.log.gz 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82138.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75344.log.gz 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64954.log 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50180.log 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82138.log.gz 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69184.log.gz 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64954.log: gzip -5 --verbose -- 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64954.log.gz 2024-03-06T07:33:01.357 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.49886.log 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44131.log 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50180.log.gz 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81886.log 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3%/var/log/ceph/ceph-client.admin.44131.log: -- replaced with /var/log/ceph/ceph-client.admin.49886.log.gz 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64996.log 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44131.log.gz 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68971.log 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81886.log: /var/log/ceph/ceph-client.admin.64996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73663.log 2024-03-06T07:33:01.358 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68971.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64996.log.gz 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81886.log.gz 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68971.log.gz 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85895.log 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60578.log 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73663.log.gz 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48083.log 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.85895.log.gz 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83771.log 2024-03-06T07:33:01.359 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60578.log: /var/log/ceph/ceph-client.admin.48083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75088.log 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83771.log: 87.4% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48083.log.gz 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60578.log.gz 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83771.log.gz 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63798.log 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.75088.log: /var/log/ceph/ceph-client.admin.82745.log 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63798.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61655.log 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75088.log.gz 2024-03-06T07:33:01.360 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63798.log.gz 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68929.log 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77308.log 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.82745.log.gz 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68929.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61655.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51848.log 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68929.log.gz 2024-03-06T07:33:01.361 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82997.log 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.77308.log.gz/var/log/ceph/ceph-client.admin.51848.log: 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47957.log 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51848.log.gz 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65038.log 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80379.log 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82997.log.gz 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47957.log: /var/log/ceph/ceph-client.admin.65038.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47957.log.gz 2024-03-06T07:33:01.362 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65038.log.gz 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82054.log 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83561.log 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80379.log.gz 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57245.log 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82054.log.gz 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54236.log 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83561.log.gz 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77787.log 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76756.log 2024-03-06T07:33:01.363 INFO:teuthology.orchestra.run.smithi072.stderr: 87.3%/var/log/ceph/ceph-client.admin.54236.log: -- replaced with /var/log/ceph/ceph-client.admin.57245.log.gz/var/log/ceph/ceph-client.admin.77787.log: 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54236.log.gz 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77787.log.gz 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75996.log 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68069.log 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6%/var/log/ceph/ceph-client.admin.75996.log: -- replaced with /var/log/ceph/ceph-client.admin.76756.log.gz 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54032.log 2024-03-06T07:33:01.364 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.75996.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43521.log 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68069.log: /var/log/ceph/ceph-client.admin.54032.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68069.log.gz 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54032.log.gz 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71930.log 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43521.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43521.log.gz 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64698.log 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65122.log 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71930.log.gz 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52967.log 2024-03-06T07:33:01.365 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64698.log.gz 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77392.log 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.65122.log.gz 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45758.log 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52967.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.51424.log 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77392.log: /var/log/ceph/ceph-client.admin.45758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47873.log 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3%/var/log/ceph/ceph-client.admin.51424.log: -- replaced with /var/log/ceph/ceph-client.admin.77392.log.gz 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45758.log.gz 2024-03-06T07:33:01.366 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80691.log 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51424.log.gz 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56279.log 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84251.log 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47873.log.gz/var/log/ceph/ceph-client.admin.80691.log: 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60013.log 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80691.log.gz 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84251.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56279.log.gz 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66691.log 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84251.log.gz 2024-03-06T07:33:01.367 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.60013.log: -- /var/log/ceph/ceph-client.admin.81676.log 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66691.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60013.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73485.log 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66691.log.gz 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48304.log 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81676.log: /var/log/ceph/ceph-client.admin.73485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59903.log 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48304.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81676.log.gz 2024-03-06T07:33:01.368 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73485.log.gz 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81844.log 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48304.log.gz 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47789.log 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59903.log: /var/log/ceph/ceph-client.admin.81844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53937.log 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59903.log.gz 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47789.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81844.log.gz 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53937.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86668.log 2024-03-06T07:33:01.369 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47789.log.gz 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53937.log.gz 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56610.log 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79689.log 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86668.log.gz 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74920.log 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79689.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56610.log.gz 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79689.log.gz 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61991.log 2024-03-06T07:33:01.370 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80607.log 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr: 89.6%/var/log/ceph/ceph-client.admin.61991.log: -- replaced with /var/log/ceph/ceph-client.admin.74920.log.gz 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82282.log 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61991.log.gz 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57331.log 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82282.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80607.log.gz 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82282.log.gz 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50222.log 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66337.log 2024-03-06T07:33:01.371 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57331.log: /var/log/ceph/ceph-client.admin.50222.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57331.log.gz 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50222.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51740.log 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81028.log 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66337.log: /var/log/ceph/ceph-client.admin.51740.log: 87.8% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66337.log.gz 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51740.log.gz 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84083.log 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47747.log 2024-03-06T07:33:01.372 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81028.log: /var/log/ceph/ceph-client.admin.84083.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81028.log.gz 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84083.log.gz 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84708.log 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55177.log 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84708.log: /var/log/ceph/ceph-client.admin.47747.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47747.log.gz 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84708.log.gz 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74674.log 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80817.log 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55177.log: /var/log/ceph/ceph-client.admin.74674.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55177.log.gz 2024-03-06T07:33:01.373 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74674.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82451.log 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82535.log 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80817.log: /var/log/ceph/ceph-client.admin.82451.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80817.log.gz 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82451.log.gz 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51201.log 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5/var/log/ceph/ceph-client.admin.82535.log: --verbose -- /var/log/ceph/ceph-client.admin.75406.log 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51201.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82535.log.gz 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81238.log 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51201.log.gz/var/log/ceph/ceph-client.admin.75406.log: 2024-03-06T07:33:01.374 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.75406.log.gz 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44200.log 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75954.log 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.81238.log.gz 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69370.log 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44200.log.gz 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75954.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75954.log.gz 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70498.log 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54424.log 2024-03-06T07:33:01.375 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69370.log.gz 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75592.log 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70498.log.gz 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54424.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54424.log.gz 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71296.log 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78719.log 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75592.log.gz 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44372.log 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71296.log.gz 2024-03-06T07:33:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78719.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78719.log.gz 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52706.log 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60493.log 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44372.log.gz 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53498.log 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60493.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.52706.log.gz 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60493.log.gz 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63714.log 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59735.log 2024-03-06T07:33:01.377 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53498.log.gz 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79108.log 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59735.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63714.log.gz 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.59735.log.gz 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69841.log 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62777.log 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79108.log.gz 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72056.log 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62777.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69841.log.gz 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62777.log.gz 2024-03-06T07:33:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83915.log 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47705.log 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72056.log.gz 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60433.log 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47705.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83915.log.gz 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47705.log.gz 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60391.log 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66734.log 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60433.log.gz 2024-03-06T07:33:01.379 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84041.log 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3%/var/log/ceph/ceph-client.admin.66734.log: -- replaced with /var/log/ceph/ceph-client.admin.60391.log.gz 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66734.log.gz 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64786.log 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79959.log 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84041.log.gz 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45632.log 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79959.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64786.log.gz 2024-03-06T07:33:01.380 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.79959.log.gz 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77480.log 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80463.log 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45632.log.gz 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64912.log 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80463.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77480.log.gz 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80463.log.gz 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71972.log 2024-03-06T07:33:01.381 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80337.log 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64912.log.gz 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80253.log 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80337.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71972.log.gz 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80337.log.gz 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44624.log 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76443.log 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80253.log.gz 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44624.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-06T07:33:01.382 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76443.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44624.log.gz 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76443.log.gz 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54859.log 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81070.log 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54859.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54859.log.gz 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55790.log 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81070.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81070.log.gz 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57892.log 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55790.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55790.log.gz 2024-03-06T07:33:01.383 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71594.log 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57892.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57892.log.gz 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58820.log 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71594.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71594.log.gz 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84963.log 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58820.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58820.log.gz 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48220.log 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84963.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84963.log.gz 2024-03-06T07:33:01.384 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47091.log 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48220.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48220.log.gz 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67583.log 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47091.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47091.log.gz 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44242.log 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67583.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67583.log.gz 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54649.log 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44242.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44242.log.gz 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50012.log 2024-03-06T07:33:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54649.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54649.log.gz 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78984.log 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50012.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50012.log.gz 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83351.log 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78984.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.78984.log.gz 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70204.log 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83351.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83351.log.gz 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72140.log 2024-03-06T07:33:01.386 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70204.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.70204.log.gz 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63588.log 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72140.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72140.log.gz 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49514.log 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63588.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63588.log.gz 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45968.log 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49514.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49514.log.gz 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77350.log 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45968.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.45968.log.gz 2024-03-06T07:33:01.387 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50138.log 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77350.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77350.log.gz 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68003.log 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50138.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50138.log.gz 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67411.log 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68003.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.68003.log.gz 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55706.log 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67411.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67411.log.gz 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80127.log 2024-03-06T07:33:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55706.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55706.log.gz 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67453.log 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80127.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.80127.log.gz 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54775.log 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67453.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67453.log.gz 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44804.log 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54775.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54775.log.gz 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60536.log 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44804.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44804.log.gz 2024-03-06T07:33:01.389 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57119.log 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60536.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60536.log.gz 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85727.log 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57119.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.57119.log.gz 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66818.log 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85727.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85727.log.gz 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54561.log 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66818.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.66818.log.gz 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67625.log 2024-03-06T07:33:01.390 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54561.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54561.log.gz 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53204.log 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67625.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67625.log.gz 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45800.log 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53204.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53204.log.gz 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62717.log 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45800.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45800.log.gz 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53853.log 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.62717.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62717.log.gz 2024-03-06T07:33:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85299.log 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53853.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53853.log.gz 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73484.log 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85299.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.85299.log.gz 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74138.log 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73484.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73484.log.gz 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73204.log 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.74138.log: 93.4% -- replaced with /var/log/ceph/ceph-client.0.74138.log.gz 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-06T07:33:01.392 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73204.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73204.log.gz 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48041.log 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.log: 86.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57598.log 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48041.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48041.log.gz 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58084.log 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57598.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57598.log.gz 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76629.log 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58084.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58084.log.gz 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45464.log 2024-03-06T07:33:01.393 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76629.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76629.log.gz 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76401.log 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45464.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45464.log.gz 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47663.log 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76401.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76401.log.gz 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73879.log 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47663.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47663.log.gz 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56957.log 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73879.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73879.log.gz 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58542.log 2024-03-06T07:33:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56957.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56957.log.gz 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61739.log 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58542.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58542.log.gz 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61739.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61739.log.gz 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79605.log 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80944.log 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79605.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79605.log.gz 2024-03-06T07:33:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48464.log 2024-03-06T07:33:01.396 INFO:teuthology.orchestra.run.smithi146.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-06T07:33:01.396 INFO:teuthology.orchestra.run.smithi146.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-06T07:33:01.407 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80944.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43790.log 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80944.log.gz 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48464.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48464.log.gz 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76149.log 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43790.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43790.log.gz 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76149.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76149.log.gz 2024-03-06T07:33:01.408 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56027.log 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79395.log 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56027.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56027.log.gz 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50534.log 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79395.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.79395.log.gz 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76527.log 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50534.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50534.log.gz 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51382.log 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76527.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.76527.log.gz 2024-03-06T07:33:01.409 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80733.log 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70871.log 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80733.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80733.log.gz 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51382.log.gz 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49556.log 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50474.log 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49556.log: 87.7% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70871.log.gz 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49556.log.gz 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82493.log 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84293.log 2024-03-06T07:33:01.410 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82493.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82493.log.gz 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50474.log.gz 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49472.log 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84293.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.84293.log.gz 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.84419.log 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49472.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49472.log.gz 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66463.log 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56820.log 2024-03-06T07:33:01.411 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66463.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.66463.log.gz 2024-03-06T07:33:01.412 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84419.log.gz 2024-03-06T07:33:01.413 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58343.log 2024-03-06T07:33:01.423 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63057.log 2024-03-06T07:33:01.427 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.58343.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56820.log.gz 2024-03-06T07:33:01.431 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48364.log 2024-03-06T07:33:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81970.log 2024-03-06T07:33:01.441 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48364.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48364.log.gz 2024-03-06T07:33:01.445 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63057.log.gz 2024-03-06T07:33:01.449 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85215.log 2024-03-06T07:33:01.463 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81970.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81970.log.gz 2024-03-06T07:33:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54278.log 2024-03-06T07:33:01.468 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85215.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.85215.log.gz 2024-03-06T07:33:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80001.log 2024-03-06T07:33:01.487 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54278.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54278.log.gz 2024-03-06T07:33:01.494 INFO:teuthology.orchestra.run.smithi072.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.58343.log.gz 2024-03-06T07:33:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85089.log 2024-03-06T07:33:01.499 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80001.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80001.log.gz 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78922.log 2024-03-06T07:33:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85089.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85089.log.gz 2024-03-06T07:33:01.526 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86147.log 2024-03-06T07:33:01.526 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78922.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78922.log.gz 2024-03-06T07:33:01.536 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51782.log 2024-03-06T07:33:01.536 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86147.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86147.log.gz 2024-03-06T07:33:01.551 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68777.log 2024-03-06T07:33:01.552 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51782.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51782.log.gz 2024-03-06T07:33:01.559 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57161.log 2024-03-06T07:33:01.569 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69618.log 2024-03-06T07:33:01.569 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57161.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57161.log.gz 2024-03-06T07:33:01.573 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68777.log.gz 2024-03-06T07:33:01.578 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45124.log 2024-03-06T07:33:01.591 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69618.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69618.log.gz 2024-03-06T07:33:01.594 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81592.log 2024-03-06T07:33:01.594 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45124.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45124.log.gz 2024-03-06T07:33:01.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61089.log 2024-03-06T07:33:01.614 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79437.log 2024-03-06T07:33:01.614 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61089.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.61089.log.gz 2024-03-06T07:33:01.616 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81592.log.gz 2024-03-06T07:33:01.623 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80649.log 2024-03-06T07:33:01.632 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49204.log 2024-03-06T07:33:01.632 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80649.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80649.log.gz 2024-03-06T07:33:01.636 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79437.log.gz 2024-03-06T07:33:01.641 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74497.log 2024-03-06T07:33:01.654 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49204.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49204.log.gz 2024-03-06T07:33:01.661 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86399.log 2024-03-06T07:33:01.661 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.74497.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74497.log.gz 2024-03-06T07:33:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45926.log 2024-03-06T07:33:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86399.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.86399.log.gz 2024-03-06T07:33:01.687 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79521.log 2024-03-06T07:33:01.697 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78123.log 2024-03-06T07:33:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79521.log: 88.8% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45926.log.gz 2024-03-06T07:33:01.702 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79521.log.gz 2024-03-06T07:33:01.712 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85444.log 2024-03-06T07:33:01.714 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78123.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.78123.log.gz 2024-03-06T07:33:01.721 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78803.log 2024-03-06T07:33:01.731 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85444.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49046.log 2024-03-06T07:33:01.735 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.78803.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.85444.log.gz 2024-03-06T07:33:01.739 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75530.log 2024-03-06T07:33:01.744 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49046.log: -- replaced with /var/log/ceph/ceph-client.admin.78803.log.gz 2024-03-06T07:33:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69308.log 2024-03-06T07:33:01.753 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49046.log.gz 2024-03-06T07:33:01.754 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75530.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75530.log.gz 2024-03-06T07:33:01.760 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86441.log 2024-03-06T07:33:01.760 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69308.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69308.log.gz 2024-03-06T07:33:01.769 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57934.log 2024-03-06T07:33:01.769 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86441.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.86441.log.gz 2024-03-06T07:33:01.783 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-06T07:33:01.785 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54468.log 2024-03-06T07:33:01.786 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57934.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57934.log.gz 2024-03-06T07:33:01.798 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79353.log 2024-03-06T07:33:01.798 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54468.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54468.log.gz 2024-03-06T07:33:01.806 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44794.log 2024-03-06T07:33:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.79353.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79353.log.gz 2024-03-06T07:33:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76359.log 2024-03-06T07:33:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44794.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44794.log.gz 2024-03-06T07:33:01.828 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-06T07:33:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71380.log 2024-03-06T07:33:01.836 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.76359.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76359.log.gz 2024-03-06T07:33:01.847 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51033.log 2024-03-06T07:33:01.847 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71380.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71380.log.gz 2024-03-06T07:33:01.861 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67203.log 2024-03-06T07:33:01.862 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51033.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51033.log.gz 2024-03-06T07:33:01.864 INFO:teuthology.orchestra.run.smithi072.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.0.52107.log.gz 2024-03-06T07:33:01.867 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68651.log 2024-03-06T07:33:01.868 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67203.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67203.log.gz 2024-03-06T07:33:01.868 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58962.log 2024-03-06T07:33:01.868 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68651.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68651.log.gz 2024-03-06T07:33:01.868 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60307.log 2024-03-06T07:33:01.868 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58962.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58962.log.gz 2024-03-06T07:33:01.876 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68849.log 2024-03-06T07:33:01.876 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60307.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60307.log.gz 2024-03-06T07:33:01.876 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77871.log 2024-03-06T07:33:01.876 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68849.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68849.log.gz 2024-03-06T07:33:01.876 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67284.log 2024-03-06T07:33:01.876 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77871.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.77871.log.gz 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44803.log 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67284.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67284.log.gz 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85486.log 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44803.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44803.log.gz 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53727.log 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.85486.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.85486.log.gz 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64870.log 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53727.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53727.log.gz 2024-03-06T07:33:01.877 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64519.log 2024-03-06T07:33:01.878 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64870.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64870.log.gz 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58652.log 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64519.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64519.log.gz 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83687.log 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58652.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58652.log.gz 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66589.log 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83687.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.83687.log.gz 2024-03-06T07:33:01.884 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64475.log 2024-03-06T07:33:01.885 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66589.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66589.log.gz 2024-03-06T07:33:01.885 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47495.log 2024-03-06T07:33:01.885 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64475.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64475.log.gz 2024-03-06T07:33:01.885 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81634.log 2024-03-06T07:33:01.885 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47495.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47495.log.gz 2024-03-06T07:33:01.885 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81364.log 2024-03-06T07:33:01.890 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81634.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81634.log.gz 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50264.log 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81364.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81364.log.gz 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69020.log 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50264.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50264.log.gz 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52925.log 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.69020.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69020.log.gz 2024-03-06T07:33:01.891 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72806.log 2024-03-06T07:33:01.892 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52925.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52925.log.gz 2024-03-06T07:33:01.892 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50863.log 2024-03-06T07:33:01.892 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72806.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.72806.log.gz 2024-03-06T07:33:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75635.log 2024-03-06T07:33:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50863.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50863.log.gz 2024-03-06T07:33:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67667.log 2024-03-06T07:33:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.75635.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.75635.log.gz 2024-03-06T07:33:01.895 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77141.log 2024-03-06T07:33:01.897 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.67667.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67667.log.gz 2024-03-06T07:33:01.897 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80421.log 2024-03-06T07:33:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.77141.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77141.log.gz 2024-03-06T07:33:01.898 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73020.log 2024-03-06T07:33:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80421.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80421.log.gz 2024-03-06T07:33:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66547.log 2024-03-06T07:33:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73020.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73020.log.gz 2024-03-06T07:33:01.900 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72894.log 2024-03-06T07:33:01.902 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.66547.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66547.log.gz 2024-03-06T07:33:01.902 INFO:teuthology.orchestra.run.smithi072.stderr: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65974.log 2024-03-06T07:33:01.902 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-06T07:33:01.902 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84125.log 2024-03-06T07:33:01.902 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72894.log.gz 2024-03-06T07:33:01.903 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.65974.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65974.log.gz 2024-03-06T07:33:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63099.log 2024-03-06T07:33:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.84125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72182.log 2024-03-06T07:33:01.912 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.84125.log.gz 2024-03-06T07:33:01.912 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81550.log 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.72182.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63099.log.gz 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81718.log 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81550.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.72182.log.gz 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81550.log.gz 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44540.log 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45506.log 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81718.log.gz 2024-03-06T07:33:01.913 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63210.log 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45506.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44540.log.gz 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77913.log 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45506.log.gz 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.63210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81928.log 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63210.log.gz/var/log/ceph/ceph-client.admin.77913.log: 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77913.log.gz 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.46969.log 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43709.log 2024-03-06T07:33:01.914 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81928.log.gz 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.2.46969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83957.log 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43709.log: 90.8% -- replaced with /var/log/ceph/ceph-client.2.46969.log.gz 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43709.log.gz 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82955.log 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.83957.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83957.log.gz 2024-03-06T07:33:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80085.log 2024-03-06T07:33:01.918 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82955.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82955.log.gz 2024-03-06T07:33:01.918 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47369.log 2024-03-06T07:33:01.918 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.80085.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80085.log.gz 2024-03-06T07:33:01.918 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53120.log 2024-03-06T07:33:01.918 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47369.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47369.log.gz 2024-03-06T07:33:01.918 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61613.log 2024-03-06T07:33:01.919 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53120.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53120.log.gz 2024-03-06T07:33:01.919 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64744.log 2024-03-06T07:33:01.919 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.61613.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61613.log.gz 2024-03-06T07:33:01.919 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70829.log 2024-03-06T07:33:01.919 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64744.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64744.log.gz 2024-03-06T07:33:01.919 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64656.log 2024-03-06T07:33:01.920 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70829.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70829.log.gz 2024-03-06T07:33:01.921 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51554.log 2024-03-06T07:33:01.921 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64656.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64656.log.gz 2024-03-06T07:33:01.921 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68650.log 2024-03-06T07:33:01.921 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51554.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51554.log.gz 2024-03-06T07:33:01.922 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86357.log 2024-03-06T07:33:01.922 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.68650.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68650.log.gz 2024-03-06T07:33:01.922 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44043.log 2024-03-06T07:33:01.923 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.86357.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86357.log.gz 2024-03-06T07:33:01.923 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73486.log 2024-03-06T07:33:01.924 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44043.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44043.log.gz 2024-03-06T07:33:01.924 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48406.log 2024-03-06T07:33:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.73486.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73486.log.gz 2024-03-06T07:33:01.925 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81802.log 2024-03-06T07:33:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48406.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48406.log.gz 2024-03-06T07:33:01.926 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64614.log 2024-03-06T07:33:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81802.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81802.log.gz 2024-03-06T07:33:01.927 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70414.log 2024-03-06T07:33:01.928 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.64614.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64614.log.gz 2024-03-06T07:33:01.928 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71720.log 2024-03-06T07:33:01.929 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.70414.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70414.log.gz 2024-03-06T07:33:01.929 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71846.log 2024-03-06T07:33:01.930 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71720.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71720.log.gz 2024-03-06T07:33:01.931 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.71846.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71846.log.gz 2024-03-06T07:33:02.006 INFO:teuthology.orchestra.run.smithi072.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-06T07:33:02.035 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-06T07:33:02.042 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-06T07:33:02.044 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-03-06T07:33:02.044 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m1.114s 2024-03-06T07:33:02.044 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m4.586s 2024-03-06T07:33:02.045 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.190s 2024-03-06T07:33:02.465 INFO:teuthology.orchestra.run.smithi072.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-06T07:33:02.595 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-06T07:33:02.642 INFO:teuthology.orchestra.run.smithi072.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-06T07:33:02.664 INFO:teuthology.orchestra.run.smithi072.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-06T07:33:03.163 INFO:teuthology.orchestra.run.smithi072.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.0.68328.log.gz 2024-03-06T07:33:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-06T07:33:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-06T07:33:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-06T07:33:03.515 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-06T07:33:03.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-06T07:33:03.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-06T07:33:03.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-06T07:33:03.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-06T07:33:03.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-03-06T07:33:09.932 INFO:teuthology.orchestra.run.smithi072.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-06T07:33:09.934 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-06T07:33:09.934 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m9.036s 2024-03-06T07:33:09.934 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m18.608s 2024-03-06T07:33:09.935 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m1.066s 2024-03-06T07:33:09.935 INFO:tasks.ceph:Archiving logs... 2024-03-06T07:33:09.935 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/remote/smithi072/log 2024-03-06T07:33:09.936 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:33:13.107 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/remote/smithi146/log 2024-03-06T07:33:13.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-06T07:33:13.773 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-06T07:33:13.784 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-06T07:33:13.785 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:33:13.788 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-06T07:33:13.875 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-06T07:33:13.884 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-06T07:33:13.884 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:33:13.887 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-06T07:33:13.900 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:33:13.903 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-03-06T07:33:13.905 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:33:13.905 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-06T07:33:13.905 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1844us[+1844us] +/- 99ms 2024-03-06T07:33:13.905 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -1060us[-1041us] +/- 53ms 2024-03-06T07:33:13.905 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +97us[ +97us] +/- 89ms 2024-03-06T07:33:13.906 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:33:13.907 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-06T07:33:13.907 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-03-06T07:33:13.908 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1859us[+1859us] +/- 99ms 2024-03-06T07:33:13.908 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -1027us[-1033us] +/- 53ms 2024-03-06T07:33:13.908 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +98us[ +93us] +/- 89ms 2024-03-06T07:33:13.908 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-06T07:33:13.909 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-06T07:33:13.924 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-06T07:33:13.924 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-06T07:33:13.944 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:33:14.042 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-06T07:33:14.117 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:33:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:14.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-06T07:33:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:14.180 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-06T07:33:14.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-06T07:33:14.255 INFO:teuthology.task.internal:Duration was 1181.751508 seconds 2024-03-06T07:33:14.256 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-06T07:33:14.286 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-06T07:33:14.287 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:33:14.290 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-06T07:33:14.332 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:33:14.336 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-06T07:33:14.744 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-06T07:33:14.744 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-03-06T07:33:14.744 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:33:14.766 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-03-06T07:33:14.766 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-06T07:33:14.796 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-06T07:33:14.796 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:33:14.808 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-06T07:33:14.866 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-06T07:33:14.866 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:33:14.878 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-06T07:33:15.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-06T07:33:15.024 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-06T07:33:15.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:33:15.051 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-06T07:33:15.078 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-06T07:33:15.090 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:15.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:15.117 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-03-06T07:33:15.144 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-03-06T07:33:15.160 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:15.187 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-06T07:33:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-06T07:33:15.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-06T07:33:15.226 INFO:teuthology.task.internal:Transferring archived files... 2024-03-06T07:33:15.227 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/remote/smithi072 2024-03-06T07:33:15.227 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:33:15.264 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060/remote/smithi146 2024-03-06T07:33:15.265 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-06T07:33:15.299 INFO:teuthology.task.internal:Removing archive directory... 2024-03-06T07:33:15.300 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:33:15.302 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-06T07:33:15.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-06T07:33:15.362 INFO:teuthology.task.internal:Not uploading archives. 2024-03-06T07:33:15.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-06T07:33:15.388 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-06T07:33:15.388 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:33:15.391 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-06T07:33:15.403 INFO:teuthology.orchestra.run.smithi072.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:33 /home/ubuntu/cephtest 2024-03-06T07:33:15.405 INFO:teuthology.orchestra.run.smithi146.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 6 07:33 /home/ubuntu/cephtest 2024-03-06T07:33:15.406 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-06T07:33:15.421 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-06T07:33:15.496 INFO:teuthology.nuke:Checking targets against current locks 2024-03-06T07:33:15.523 DEBUG:teuthology.nuke:shortname: smithi072 2024-03-06T07:33:15.524 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:33:15.549 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:40.609193', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCtvxMT4pfpfCRFYMBCFiNMlviQr8bJ6S6yeXVqeLhb68KzR5NS0ZCgwYjKleIYo7G/6Gl9YgE2jNprIz1lZFac='} 2024-03-06T07:33:15.552 DEBUG:teuthology.nuke:shortname: smithi146 2024-03-06T07:33:15.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-06T07:33:15.577 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-03-06_07:04:29-fs-wip-lusov-qdb-distro-default-smithi/7583060', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-06 07:04:40.610607', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJCTiRBUCAsvCfWLVyHQLrRgxHEjeAHsL1fnPo17LH6ZJ0FHIDmxIoBRs5NN378dcLl9HGNxpNaBHagISm2z7PM='} 2024-03-06T07:33:15.598 INFO:teuthology.orchestra.console.smithi072:Power off 2024-03-06T07:33:15.598 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:33:15.622 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-03-06T07:33:15.622 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:15.637 INFO:teuthology.orchestra.console.smithi146:Power off 2024-03-06T07:33:15.638 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-06T07:33:15.647 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-06T07:33:15.660 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-03-06T07:33:15.660 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:15.682 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-06T07:33:19.649 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:19.683 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:19.761 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-06T07:33:19.792 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-06T07:33:23.763 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:23.793 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:23.874 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-06T07:33:23.902 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-03-06T07:33:27.876 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:27.903 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-06T07:33:27.989 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-03-06T07:33:28.014 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-03-06T07:33:28.090 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-03-06T07:33:28.114 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-03-06T07:33:28.221 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/strays} duration: 1181.7515075206757 failure_reason: 'Test failure: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dd16d3fdc1494e8b93c9c82217f1cca4 status: fail success: false 2024-03-06T07:33:28.222 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-06T07:33:28.316 INFO:teuthology.run:FAIL